2024-09-15T03:18:50.922 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T03:18:50.940 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:18:51.017 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} email: skanta@redhat.com first_in_suite: false job_id: '7905270' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJlUxoBpFMPN3NKSagvoLtSCYyb0J8MtZuP5XGEVPd+ck0m4yl1yO/mpRdefjYPeFRNy49Q7y5Rgodt3LuLfDQ= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDZjzx2pYJsT+ulWbi43rj0ABxZKlsJpjsbSHQbFt4LGo5Da9WSN5n59zrjmC+nNgV3RczWisIQ7JeYmxig0rA= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T03:18:51.018 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T03:18:51.018 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T03:18:51.019 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T03:18:51.020 INFO:teuthology.task.internal:Checking packages... 2024-09-15T03:18:51.042 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T03:18:51.042 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T03:18:51.042 INFO:teuthology.packaging:ref: None 2024-09-15T03:18:51.042 INFO:teuthology.packaging:tag: None 2024-09-15T03:18:51.043 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:18:51.043 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:18:51.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:18:51.193 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T03:18:51.195 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T03:18:51.201 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T03:18:51.201 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T03:18:51.209 INFO:teuthology.task.internal:Saving configuration 2024-09-15T03:18:51.221 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T03:18:51.229 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T03:18:51.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:13:42.105021', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJlUxoBpFMPN3NKSagvoLtSCYyb0J8MtZuP5XGEVPd+ck0m4yl1yO/mpRdefjYPeFRNy49Q7y5Rgodt3LuLfDQ='} 2024-09-15T03:18:51.272 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:13:42.106798', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDZjzx2pYJsT+ulWbi43rj0ABxZKlsJpjsbSHQbFt4LGo5Da9WSN5n59zrjmC+nNgV3RczWisIQ7JeYmxig0rA='} 2024-09-15T03:18:51.272 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T03:18:51.278 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-15T03:18:51.278 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-15T03:18:51.278 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T03:18:51.384 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7779e07ac0>, signals=[15]) 2024-09-15T03:18:51.385 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T03:18:51.390 INFO:teuthology.task.internal:Opening connections... 2024-09-15T03:18:51.390 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:18:51.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:18:51.475 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:18:51.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:18:51.554 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T03:18:51.560 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-09-15T03:18:51.578 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-09-15T03:18:51.578 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-09-15T03:18:51.637 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:18:51.638 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:18:51.639 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:18:51.639 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-09-15T03:18:51.667 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-15T03:18:51.685 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-15T03:18:51.685 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-15T03:18:51.743 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-15T03:18:51.743 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-15T03:18:51.743 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-15T03:18:51.743 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:18:51.744 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:18:51.744 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-15T03:18:51.767 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T03:18:51.775 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T03:18:51.781 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T03:18:51.781 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:18:51.783 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:18:51.801 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T03:18:51.806 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T03:18:51.807 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:18:51.841 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:18:51.936 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T03:18:51.991 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T03:18:52.005 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T03:18:52.006 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T03:18:52.006 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:18:52.007 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:18:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:18:52.025 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-15T03:18:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:18:52.026 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-15T03:18:52.083 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:18:52.083 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-503.el9.x86_64 2024-09-15T03:18:52.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-09-15T03:18:52.084 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:18:52.084 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-15T03:18:52.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-15T03:18:54.597 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-15T03:18:55.584 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-15T03:18:59.144 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 20 MB 00:02 2024-09-15T03:19:00.098 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-15T03:19:04.261 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-15T03:19:04.398 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T03:19:05.894 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-15T03:19:06.157 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 19 kB 00:00 2024-09-15T03:19:06.649 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-15T03:19:07.036 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T03:19:12.154 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-09-15T03:19:14.261 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:19:14.261 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:19:14.315 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:19:14.315 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:19:14.361 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:19:14.371 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:19:14.372 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-15T03:19:14.372 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:19:14.372 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:19:14.372 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:19:14.372 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:19:14.372 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:19:14.373 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:19:14.373 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:19:14.373 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:19:14.373 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:19:14.373 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:19:14.373 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-09-15T03:19:14.374 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:19:14.374 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 79 M 2024-09-15T03:19:14.374 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 126 M 2024-09-15T03:19:14.374 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:19:14.402 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:19:14.418 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:19:14.419 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-15T03:19:14.419 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:19:14.419 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:19:14.419 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:19:14.420 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:19:14.420 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:19:14.420 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:19:14.420 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:19:14.420 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:19:14.420 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:19:14.421 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:19:14.421 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-15T03:19:14.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:19:14.421 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-15T03:19:14.421 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-15T03:19:14.421 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:19:14.547 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 298 kB/s | 23 kB 00:00 2024-09-15T03:19:14.743 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 288 kB/s | 23 kB 00:00 2024-09-15T03:19:15.097 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-15T03:19:15.414 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-15T03:19:15.427 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-09-15T03:19:15.902 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-15T03:19:16.048 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 19 MB/s | 29 MB 00:01 2024-09-15T03:19:16.048 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:19:16.048 INFO:teuthology.orchestra.run.smithi153.stdout:Total 47 MB/s | 79 MB 00:01 2024-09-15T03:19:16.269 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 19 MB/s | 29 MB 00:01 2024-09-15T03:19:16.269 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:19:16.269 INFO:teuthology.orchestra.run.smithi181.stdout:Total 43 MB/s | 79 MB 00:01 2024-09-15T03:19:16.736 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:19:16.949 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:19:17.089 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:19:17.089 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:19:17.273 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:19:17.274 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:19:18.047 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:19:18.048 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:19:18.236 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:19:18.236 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:19:20.518 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:19:20.658 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:19:21.207 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:19:21.283 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:19:21.342 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:19:21.436 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:19:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:19:24.248 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:19:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:19:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:19:29.089 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:19:29.127 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:19:29.236 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:19:29.273 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:19:33.980 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:19:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.251 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.291 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.511 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.561 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:20:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:20:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:20:12.840 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:12.840 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:20:12.840 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:20:12.840 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:20:13.412 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:13.412 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:20:13.413 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:20:13.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:20:13.715 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:20:13.760 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:20:13.848 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.848 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:20:13.848 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:20:13.848 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:20:13.848 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:20:13.848 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T03:20:13.848 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:20:13.849 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:20:13.907 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:20:13.908 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:20:13.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 82.el9 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8274670 2024-09-15T03:20:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:20:13.952 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:20:13.953 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-09-15T03:20:13.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:20:13.953 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:20:13.954 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:20:13.954 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:20:13.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:20:14.101 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:20:14.185 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:20:14.186 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:20:14.186 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:20:14.186 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:20:14.186 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:20:14.186 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T03:20:14.186 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:20:14.186 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:20:14.244 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:20:14.245 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:20:14.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-15T03:20:14.277 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-15T03:20:14.277 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-15T03:20:14.277 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-15T03:20:14.277 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:20:14.278 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-15T03:20:14.279 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:20:14.280 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:20:14.280 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:20:14.281 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:20:14.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:20:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-09-15T03:20:14.990 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-15T03:20:15.901 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:20:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-09-15T03:20:15.924 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-15T03:20:15.940 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.jiVpvR0l24 2024-09-15T03:20:15.940 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jiVpvR0l24 2024-09-15T03:20:16.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.jiVpvR0l24 2024-09-15T03:20:16.163 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.jiVpvR0l24 is 6KB 2024-09-15T03:20:16.211 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.jiVpvR0l24 2024-09-15T03:20:16.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:20:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:20:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-15T03:20:16.260 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-15T03:20:16.276 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.yh79T30I5S 2024-09-15T03:20:16.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yh79T30I5S 2024-09-15T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:20:16.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:20:16.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.yh79T30I5S 2024-09-15T03:20:16.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-09-15T03:20:16.477 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.yh79T30I5S is 6KB 2024-09-15T03:20:16.525 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.yh79T30I5S 2024-09-15T03:20:16.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:20:16.606 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:20:16.607 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:20:16.607 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:20:16.607 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:20:16.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:20:16.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-15T03:20:46.473 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T03:20:46.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-15T03:20:46.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:20:46.769 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T03:20:46.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T03:20:46.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:21:20.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-15T03:21:29.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-15T03:21:29.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:21:46.785 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T03:21:48.349 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-15T03:21:55.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T03:21:55.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:21:56.204 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-15T03:21:56.605 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T03:21:56.605 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:21:56.605 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-15T03:21:56.619 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:21:56.620 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:21:56.620 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:21:56.620 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:21:57.620 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T03:21:57.620 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:21:57.699 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-15T03:21:57.727 DEBUG:teuthology.parallel:result is None 2024-09-15T03:22:00.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-15T03:22:00.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:22:01.198 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-15T03:22:01.601 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-15T03:22:01.601 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:22:01.601 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-15T03:22:01.618 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:22:01.619 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:22:01.619 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:22:01.619 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:22:02.620 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T03:22:02.620 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:22:02.726 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-09-15T03:22:02.754 DEBUG:teuthology.parallel:result is None 2024-09-15T03:22:02.754 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T03:22:02.762 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T03:22:02.762 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:22:02.764 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:22:02.781 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T03:22:02.787 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T03:22:02.793 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T03:22:02.793 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:22:02.819 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:22:02.866 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T03:22:02.872 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T03:22:02.873 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:22:02.883 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:22:02.922 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:02.932 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:02.938 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:02.948 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:22:02.950 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T03:22:02.957 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T03:22:02.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:22:02.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:22:03.014 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T03:22:03.023 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T03:22:03.023 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:22:03.042 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:22:03.069 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:22:03.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:22:03.179 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:22:03.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:22:03.302 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:22:03.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:22:03.371 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:22:03.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:22:03.466 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:22:03.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:22:03.589 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:22:03.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:22:03.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-09-15T03:22:03.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-15T03:22:03.710 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:22:03.749 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:22:04.033 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T03:22:04.040 INFO:teuthology.task.internal:Starting timer... 2024-09-15T03:22:04.041 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T03:22:04.056 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T03:22:04.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-09-15T03:22:04.113 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-09-15T03:22:04.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-15T03:22:04.174 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-15T03:22:04.177 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T03:22:04.177 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-09-15T03:22:04.201 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-09-15T03:22:04.201 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-15T03:22:04.235 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-15T03:22:04.236 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-15T03:22:04.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:22:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:22:04.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:22:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:22:04.302 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T03:22:04.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:22:04.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:22:04.393 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T03:22:04.404 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T03:22:04.418 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T03:22:04.419 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T03:25:49.905 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-15T03:25:49.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-15T03:25:49.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:25:49.987 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-15T03:25:50.079 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-15T03:25:50.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T03:25:50.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:25:50.157 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-15T03:25:50.196 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T03:25:50.197 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T03:25:50.208 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T03:25:50.209 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:25:50.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:25:50.215 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:25:50.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:25:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:25:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:25:50.268 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:25:50.279 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-09-15T03:25:50.283 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:25:50.290 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-09-15T03:25:50.305 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:25:50.308 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-15T03:25:50.318 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:25:50.319 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-15T03:25:50.333 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:25:50.348 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:25:50.372 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:25:50.376 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:25:50.376 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-15T03:25:50.376 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.376 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.376 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.376 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.401 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:25:50.404 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:25:50.404 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-15T03:25:50.405 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.405 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.405 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.405 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:25:50.405 INFO:teuthology.run_tasks:Running task install... 2024-09-15T03:25:50.416 DEBUG:teuthology.task.install:project ceph 2024-09-15T03:25:50.416 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:25:50.416 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T03:25:50.416 INFO:teuthology.task.install:Using flavor: default 2024-09-15T03:25:50.422 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T03:25:50.422 INFO:teuthology.task.install:extra packages: [] 2024-09-15T03:25:50.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:25:50.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:25:50.983 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:46 ago on Sun 15 Sep 2024 03:25:04 AM UTC. 2024-09-15T03:25:51.003 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:46 ago on Sun 15 Sep 2024 03:25:04 AM UTC. 2024-09-15T03:25:51.070 INFO:teuthology.orchestra.run.smithi153.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:25:51.090 INFO:teuthology.orchestra.run.smithi181.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:25:51.112 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:25:51.113 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:25:51.113 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:25:51.132 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:25:51.133 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:25:51.133 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:25:51.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:25:51.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:25:51.362 INFO:teuthology.orchestra.run.smithi153.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:25:51.362 INFO:teuthology.orchestra.run.smithi153.stderr:of the main distribution, and quality may vary. 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:, 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:and packages are not held to any quality or security level. 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi181.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi181.stderr:of the main distribution, and quality may vary. 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:25:51.363 INFO:teuthology.orchestra.run.smithi181.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:25:51.364 INFO:teuthology.orchestra.run.smithi181.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:25:51.364 INFO:teuthology.orchestra.run.smithi181.stderr:, 2024-09-15T03:25:51.364 INFO:teuthology.orchestra.run.smithi181.stderr:and packages are not held to any quality or security level. 2024-09-15T03:25:51.364 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:25:51.364 INFO:teuthology.orchestra.run.smithi181.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:25:51.364 INFO:teuthology.orchestra.run.smithi181.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:25:51.449 INFO:teuthology.orchestra.run.smithi181.stdout:Repository successfully enabled. 2024-09-15T03:25:51.449 INFO:teuthology.orchestra.run.smithi153.stdout:Repository successfully enabled. 2024-09-15T03:25:51.481 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:25:51.482 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:25:51.490 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:25:51.490 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:25:51.640 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:25:51.641 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:25:51.645 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:25:51.645 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:25:51.806 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:25:51.807 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:25:51.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:25:51.823 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:25:51.823 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:25:51.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:25:51.833 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:25:51.833 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:25:51.849 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:25:51.849 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:25:51.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:25:51.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:25:51.976 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:25:51.990 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:25:52.040 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-15T03:25:52.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-15T03:25:52.054 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-09-15T03:25:52.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-09-15T03:25:52.269 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-15T03:25:52.285 INFO:teuthology.orchestra.run.smithi153.stdout:45 files removed 2024-09-15T03:25:52.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-15T03:25:52.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-09-15T03:25:53.170 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 131 kB/s | 77 kB 00:00 2024-09-15T03:25:53.189 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 132 kB/s | 77 kB 00:00 2024-09-15T03:25:53.737 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-09-15T03:25:53.774 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-09-15T03:25:54.255 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 3.7 kB/s | 1.8 kB 00:00 2024-09-15T03:25:54.291 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 3.7 kB/s | 1.8 kB 00:00 2024-09-15T03:25:54.422 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-15T03:25:54.458 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-15T03:25:54.914 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-15T03:25:55.150 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-15T03:25:56.932 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-15T03:25:57.410 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-15T03:26:01.668 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-15T03:26:02.216 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.5 MB 00:01 2024-09-15T03:26:03.244 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-15T03:26:03.591 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 130 kB/s | 19 kB 00:00 2024-09-15T03:26:04.446 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-15T03:26:04.456 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-15T03:26:09.860 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-09-15T03:26:09.912 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-15T03:26:12.355 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:26:12.368 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:12.368 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:26:12.369 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:26:12.370 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout:Install 22 Packages 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-09-15T03:26:12.371 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:12.372 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:26:12.372 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-09-15T03:26:12.372 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:26:12.385 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:12.385 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:26:12.385 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:26:12.386 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:26:12.387 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout:Install 22 Packages 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-15T03:26:12.388 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:12.389 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 58 M 2024-09-15T03:26:12.389 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:26:13.303 INFO:teuthology.orchestra.run.smithi181.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 7.8 MB/s | 5.0 MB 00:00 2024-09-15T03:26:13.386 INFO:teuthology.orchestra.run.smithi181.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 304 kB/s | 25 kB 00:00 2024-09-15T03:26:13.503 INFO:teuthology.orchestra.run.smithi181.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 5.3 MB/s | 637 kB 00:00 2024-09-15T03:26:13.595 INFO:teuthology.orchestra.run.smithi181.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 4.9 MB/s | 458 kB 00:00 2024-09-15T03:26:13.746 INFO:teuthology.orchestra.run.smithi153.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 4.7 MB/s | 5.0 MB 00:01 2024-09-15T03:26:13.838 INFO:teuthology.orchestra.run.smithi153.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 274 kB/s | 25 kB 00:00 2024-09-15T03:26:13.989 INFO:teuthology.orchestra.run.smithi153.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 4.1 MB/s | 637 kB 00:00 2024-09-15T03:26:14.062 INFO:teuthology.orchestra.run.smithi181.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 10 MB/s | 14 MB 00:01 2024-09-15T03:26:14.146 INFO:teuthology.orchestra.run.smithi181.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 8.9 MB/s | 4.9 MB 00:00 2024-09-15T03:26:14.164 INFO:teuthology.orchestra.run.smithi153.stdout:(4/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 9.8 MB/s | 14 MB 00:01 2024-09-15T03:26:14.171 INFO:teuthology.orchestra.run.smithi181.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 417 kB/s | 45 kB 00:00 2024-09-15T03:26:14.189 INFO:teuthology.orchestra.run.smithi153.stdout:(5/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.2 MB/s | 458 kB 00:00 2024-09-15T03:26:14.196 INFO:teuthology.orchestra.run.smithi181.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.3 MB/s | 120 kB 00:00 2024-09-15T03:26:14.238 INFO:teuthology.orchestra.run.smithi181.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.3 MB/s | 156 kB 00:00 2024-09-15T03:26:14.239 INFO:teuthology.orchestra.run.smithi153.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 904 kB/s | 45 kB 00:00 2024-09-15T03:26:14.263 INFO:teuthology.orchestra.run.smithi181.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 4.7 MB/s | 320 kB 00:00 2024-09-15T03:26:14.298 INFO:teuthology.orchestra.run.smithi153.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 2.0 MB/s | 120 kB 00:00 2024-09-15T03:26:14.331 INFO:teuthology.orchestra.run.smithi181.stdout:(11/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.5 MB/s | 100 kB 00:00 2024-09-15T03:26:14.356 INFO:teuthology.orchestra.run.smithi181.stdout:(12/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 2.5 MB/s | 295 kB 00:00 2024-09-15T03:26:14.356 INFO:teuthology.orchestra.run.smithi153.stdout:(8/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.6 MB/s | 156 kB 00:00 2024-09-15T03:26:14.448 INFO:teuthology.orchestra.run.smithi153.stdout:(9/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 17 MB/s | 4.9 MB 00:00 2024-09-15T03:26:14.473 INFO:teuthology.orchestra.run.smithi153.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 2.7 MB/s | 320 kB 00:00 2024-09-15T03:26:14.499 INFO:teuthology.orchestra.run.smithi153.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 5.7 MB/s | 295 kB 00:00 2024-09-15T03:26:14.516 INFO:teuthology.orchestra.run.smithi153.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 2.3 MB/s | 100 kB 00:00 2024-09-15T03:26:14.606 INFO:teuthology.orchestra.run.smithi181.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 12 MB/s | 22 MB 00:01 2024-09-15T03:26:14.631 INFO:teuthology.orchestra.run.smithi181.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 164 kB/s | 45 kB 00:00 2024-09-15T03:26:14.715 INFO:teuthology.orchestra.run.smithi181.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-09-15T03:26:14.765 INFO:teuthology.orchestra.run.smithi181.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-15T03:26:14.782 INFO:teuthology.orchestra.run.smithi181.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 631 kB/s | 42 kB 00:00 2024-09-15T03:26:14.824 INFO:teuthology.orchestra.run.smithi181.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 5.1 MB/s | 308 kB 00:00 2024-09-15T03:26:14.850 INFO:teuthology.orchestra.run.smithi181.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-09-15T03:26:14.866 INFO:teuthology.orchestra.run.smithi181.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 578 kB/s | 49 kB 00:00 2024-09-15T03:26:14.882 INFO:teuthology.orchestra.run.smithi153.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 123 kB/s | 45 kB 00:00 2024-09-15T03:26:14.917 INFO:teuthology.orchestra.run.smithi181.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-15T03:26:14.959 INFO:teuthology.orchestra.run.smithi181.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-15T03:26:14.966 INFO:teuthology.orchestra.run.smithi153.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 343 kB/s | 160 kB 00:00 2024-09-15T03:26:15.166 INFO:teuthology.orchestra.run.smithi153.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-15T03:26:15.217 INFO:teuthology.orchestra.run.smithi181.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-09-15T03:26:15.275 INFO:teuthology.orchestra.run.smithi153.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 391 kB/s | 42 kB 00:00 2024-09-15T03:26:15.309 INFO:teuthology.orchestra.run.smithi181.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 7.5 MB/s | 2.9 MB 00:00 2024-09-15T03:26:15.314 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:26:15.314 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-15T03:26:15.325 INFO:teuthology.orchestra.run.smithi153.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-15T03:26:15.350 INFO:teuthology.orchestra.run.smithi153.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-15T03:26:15.376 INFO:teuthology.orchestra.run.smithi153.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-15T03:26:15.377 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:26:15.410 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:26:15.410 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:26:15.584 INFO:teuthology.orchestra.run.smithi153.stdout:(20/24): ceph-common-17.2.7-1643.gd75f2382.el9. 7.8 MB/s | 22 MB 00:02 2024-09-15T03:26:15.609 INFO:teuthology.orchestra.run.smithi153.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 911 kB/s | 662 kB 00:00 2024-09-15T03:26:15.660 INFO:teuthology.orchestra.run.smithi153.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-09-15T03:26:15.894 INFO:teuthology.orchestra.run.smithi153.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 10 MB/s | 2.9 MB 00:00 2024-09-15T03:26:15.944 INFO:teuthology.orchestra.run.smithi153.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 8.6 MB/s | 3.1 MB 00:00 2024-09-15T03:26:15.947 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:26:15.947 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 58 MB 00:03 2024-09-15T03:26:16.010 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:26:16.044 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:26:16.044 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:26:16.051 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:26:16.051 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:26:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:26:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:26:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:26:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:26:17.745 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:26:17.820 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:26:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:26:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:26:18.026 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:26:18.072 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:26:18.109 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:26:18.149 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:26:18.491 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:26:18.556 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:26:18.582 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:26:18.604 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:26:18.646 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:26:18.682 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:26:18.696 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:26:18.725 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:26:18.762 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:26:18.774 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:26:18.797 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:26:18.824 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:26:18.852 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:26:18.886 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:26:18.901 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:26:18.939 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:26:18.952 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:26:18.969 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:26:18.986 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:26:19.018 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:26:19.311 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:26:19.389 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:26:19.406 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:26:19.442 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:26:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:26:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:26:19.541 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:26:19.561 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:26:19.584 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:26:19.600 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:26:19.621 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:26:19.685 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:26:19.711 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:26:19.777 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:26:19.833 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:26:19.869 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:26:19.902 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:26:20.195 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:26:20.231 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:26:20.321 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:26:20.394 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:26:20.425 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:26:22.002 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:26:22.055 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:26:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:26:22.634 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:26:22.736 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:26:22.736 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:26:22.736 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:26:22.737 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:26:22.886 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:26:22.930 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:26:23.469 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:26:23.499 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:26:23.595 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:26:23.595 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:26:23.596 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:26:23.596 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:23.626 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:26:32.623 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-15T03:26:32.624 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:33.483 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:26:33.483 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-09-15T03:26:33.483 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-09-15T03:26:33.483 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-09-15T03:26:33.483 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-09-15T03:26:33.483 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-09-15T03:26:33.484 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-09-15T03:26:33.484 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-09-15T03:26:33.484 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:33.934 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:26:34.010 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:26:34.010 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:34.010 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:26:34.010 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:26:34.011 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:26:34.011 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:34.013 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:26:34.095 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:26:34.096 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:26:34.798 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:26:34.873 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:26:34.873 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:34.873 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:26:34.873 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:26:34.873 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:26:34.874 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:34.875 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:26:34.975 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:26:34.975 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:26:35.441 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:26:35.441 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:26:35.442 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:26:35.443 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:26:36.262 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:26:36.263 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:26:36.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:26:36.265 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:26:36.265 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:26:36.446 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:26:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:36.449 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:26:36.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:36.449 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:26:36.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-15T03:26:37.268 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:26:37.268 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:37.268 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-09-15T03:26:37.268 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.268 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.268 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.269 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:26:37.270 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:37.271 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:26:37.271 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:37.271 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:26:37.353 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:26:37.486 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:26:37.486 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:37.486 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:26:37.486 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:37.486 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:26:37.486 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-15T03:26:37.487 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:37.488 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 48 M 2024-09-15T03:26:37.488 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 193 M 2024-09-15T03:26:37.488 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:26:37.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-09-15T03:26:37.910 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-15T03:26:37.952 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-15T03:26:38.003 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 693 kB/s | 163 kB 00:00 2024-09-15T03:26:38.190 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:29 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:26:38.328 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:26:38.329 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:38.329 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:26:38.329 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:38.329 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:26:38.329 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:26:38.329 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-09-15T03:26:38.330 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:38.331 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 48 M 2024-09-15T03:26:38.331 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 193 M 2024-09-15T03:26:38.331 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:26:38.657 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 731 kB/s | 163 kB 00:00 2024-09-15T03:26:39.224 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 385 kB/s | 304 kB 00:00 2024-09-15T03:26:39.274 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 104 kB/s | 64 kB 00:00 2024-09-15T03:26:39.361 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 48 MB 00:01 2024-09-15T03:26:39.363 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:26:39.364 INFO:teuthology.orchestra.run.smithi181.stdout:Total 26 MB/s | 48 MB 00:01 2024-09-15T03:26:39.392 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:26:39.411 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:26:39.411 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:26:39.777 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:26:39.777 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:26:40.301 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:26:40.388 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:26:40.428 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:26:40.463 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:26:40.499 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 23 MB/s | 48 MB 00:02 2024-09-15T03:26:40.501 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:26:40.501 INFO:teuthology.orchestra.run.smithi153.stdout:Total 22 MB/s | 48 MB 00:02 2024-09-15T03:26:40.530 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:26:40.550 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:26:40.550 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:26:40.920 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:26:40.920 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:26:41.440 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:26:41.508 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:26:41.548 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:26:41.576 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:26:44.900 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:26:44.977 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:26:46.028 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:26:46.098 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:26:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:26:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:26:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:26:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:26:47.433 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:26:47.433 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:47.433 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:26:47.433 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:47.433 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:47.434 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:26:47.434 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:26:47.434 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:47.434 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:26:47.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-15T03:26:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:26:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:26:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:26:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:26:48.296 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:26:48.442 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:26:48.444 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:26:48.445 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:26:48.446 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:26:48.447 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout:Install 34 Packages 2024-09-15T03:26:48.448 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:48.449 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 27 M 2024-09-15T03:26:48.449 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 96 M 2024-09-15T03:26:48.449 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:26:48.597 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:26:48.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:26:48.598 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:48.598 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:48.598 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:26:48.598 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:26:48.598 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:26:48.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-09-15T03:26:48.995 INFO:teuthology.orchestra.run.smithi181.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-15T03:26:49.179 INFO:teuthology.orchestra.run.smithi181.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.7 MB/s | 1.4 MB 00:00 2024-09-15T03:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.9 MB/s | 2.1 MB 00:00 2024-09-15T03:26:49.321 INFO:teuthology.orchestra.run.smithi181.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.6 MB/s | 240 kB 00:00 2024-09-15T03:26:49.388 INFO:teuthology.orchestra.run.smithi181.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-09-15T03:26:49.452 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:40 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:26:49.538 INFO:teuthology.orchestra.run.smithi181.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-15T03:26:49.588 INFO:teuthology.orchestra.run.smithi181.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-15T03:26:49.596 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:26:49.599 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:26:49.600 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:26:49.601 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout:Install 34 Packages 2024-09-15T03:26:49.602 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:49.603 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 27 M 2024-09-15T03:26:49.603 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 96 M 2024-09-15T03:26:49.603 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:26:49.655 INFO:teuthology.orchestra.run.smithi181.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 4.6 MB/s | 1.2 MB 00:00 2024-09-15T03:26:49.681 INFO:teuthology.orchestra.run.smithi181.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.4 MB/s | 135 kB 00:00 2024-09-15T03:26:49.706 INFO:teuthology.orchestra.run.smithi181.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-15T03:26:49.739 INFO:teuthology.orchestra.run.smithi181.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-15T03:26:49.899 INFO:teuthology.orchestra.run.smithi181.stdout:(12/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 21 MB/s | 15 MB 00:00 2024-09-15T03:26:49.982 INFO:teuthology.orchestra.run.smithi181.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 144 kB/s | 35 kB 00:00 2024-09-15T03:26:50.016 INFO:teuthology.orchestra.run.smithi181.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 557 kB/s | 172 kB 00:00 2024-09-15T03:26:50.066 INFO:teuthology.orchestra.run.smithi181.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 610 kB/s | 51 kB 00:00 2024-09-15T03:26:50.091 INFO:teuthology.orchestra.run.smithi181.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 403 kB/s | 77 kB 00:00 2024-09-15T03:26:50.116 INFO:teuthology.orchestra.run.smithi181.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-09-15T03:26:50.166 INFO:teuthology.orchestra.run.smithi181.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-15T03:26:50.192 INFO:teuthology.orchestra.run.smithi181.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 472 kB/s | 11 kB 00:00 2024-09-15T03:26:50.199 INFO:teuthology.orchestra.run.smithi153.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-15T03:26:50.217 INFO:teuthology.orchestra.run.smithi181.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-15T03:26:50.242 INFO:teuthology.orchestra.run.smithi181.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-15T03:26:50.268 INFO:teuthology.orchestra.run.smithi181.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-15T03:26:50.293 INFO:teuthology.orchestra.run.smithi181.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-15T03:26:50.318 INFO:teuthology.orchestra.run.smithi181.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-15T03:26:50.344 INFO:teuthology.orchestra.run.smithi181.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-15T03:26:50.369 INFO:teuthology.orchestra.run.smithi181.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:26:50.402 INFO:teuthology.orchestra.run.smithi181.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-15T03:26:50.415 INFO:teuthology.orchestra.run.smithi153.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.2 MB/s | 1.4 MB 00:00 2024-09-15T03:26:50.419 INFO:teuthology.orchestra.run.smithi181.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-15T03:26:50.445 INFO:teuthology.orchestra.run.smithi181.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-15T03:26:50.462 INFO:teuthology.orchestra.run.smithi181.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-15T03:26:50.466 INFO:teuthology.orchestra.run.smithi153.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-09-15T03:26:50.487 INFO:teuthology.orchestra.run.smithi181.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-15T03:26:50.507 INFO:teuthology.orchestra.run.smithi181.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.0 MB/s | 20 kB 00:00 2024-09-15T03:26:50.532 INFO:teuthology.orchestra.run.smithi181.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.6 MB/s | 230 kB 00:00 2024-09-15T03:26:50.533 INFO:teuthology.orchestra.run.smithi153.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 3.5 MB/s | 240 kB 00:00 2024-09-15T03:26:50.558 INFO:teuthology.orchestra.run.smithi181.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.4 MB/s | 427 kB 00:00 2024-09-15T03:26:50.561 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:26:50.561 INFO:teuthology.orchestra.run.smithi181.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-15T03:26:50.608 INFO:teuthology.orchestra.run.smithi153.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 9.6 MB/s | 3.9 MB 00:00 2024-09-15T03:26:50.689 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:26:50.729 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:26:50.729 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:26:50.775 INFO:teuthology.orchestra.run.smithi153.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-15T03:26:50.833 INFO:teuthology.orchestra.run.smithi153.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-15T03:26:51.009 INFO:teuthology.orchestra.run.smithi153.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 26 MB/s | 15 MB 00:00 2024-09-15T03:26:51.034 INFO:teuthology.orchestra.run.smithi153.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 674 kB/s | 135 kB 00:00 2024-09-15T03:26:51.068 INFO:teuthology.orchestra.run.smithi153.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-15T03:26:51.093 INFO:teuthology.orchestra.run.smithi153.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-15T03:26:51.141 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:26:51.141 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:26:51.161 INFO:teuthology.orchestra.run.smithi153.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 522 kB/s | 35 kB 00:00 2024-09-15T03:26:51.193 INFO:teuthology.orchestra.run.smithi153.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-15T03:26:51.219 INFO:teuthology.orchestra.run.smithi153.stdout:(14/34): python3-requests-2.25.1-8.el9.noarch.r 597 kB/s | 125 kB 00:00 2024-09-15T03:26:51.244 INFO:teuthology.orchestra.run.smithi153.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-15T03:26:51.269 INFO:teuthology.orchestra.run.smithi153.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-15T03:26:51.311 INFO:teuthology.orchestra.run.smithi153.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 651 kB/s | 43 kB 00:00 2024-09-15T03:26:51.337 INFO:teuthology.orchestra.run.smithi153.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 354 kB/s | 42 kB 00:00 2024-09-15T03:26:51.370 INFO:teuthology.orchestra.run.smithi153.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-15T03:26:51.395 INFO:teuthology.orchestra.run.smithi153.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 759 kB/s | 18 kB 00:00 2024-09-15T03:26:51.421 INFO:teuthology.orchestra.run.smithi153.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-09-15T03:26:51.446 INFO:teuthology.orchestra.run.smithi153.stdout:(22/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-09-15T03:26:51.471 INFO:teuthology.orchestra.run.smithi153.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-09-15T03:26:51.496 INFO:teuthology.orchestra.run.smithi153.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-15T03:26:51.521 INFO:teuthology.orchestra.run.smithi153.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-15T03:26:51.548 INFO:teuthology.orchestra.run.smithi153.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-09-15T03:26:51.572 INFO:teuthology.orchestra.run.smithi153.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-15T03:26:51.597 INFO:teuthology.orchestra.run.smithi153.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-15T03:26:51.614 INFO:teuthology.orchestra.run.smithi153.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T03:26:51.640 INFO:teuthology.orchestra.run.smithi153.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-15T03:26:51.665 INFO:teuthology.orchestra.run.smithi153.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-15T03:26:51.682 INFO:teuthology.orchestra.run.smithi153.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-15T03:26:51.707 INFO:teuthology.orchestra.run.smithi153.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-15T03:26:51.732 INFO:teuthology.orchestra.run.smithi153.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-15T03:26:51.737 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:26:51.737 INFO:teuthology.orchestra.run.smithi153.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-15T03:26:51.868 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:26:51.905 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:26:51.905 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:26:52.284 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:26:52.284 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:26:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:26:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:26:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:26:52.517 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:26:52.609 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:26:52.696 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:26:52.737 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:26:52.780 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:26:52.810 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:26:52.845 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:26:52.878 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:26:52.918 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:26:52.953 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:26:52.979 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:26:53.044 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:26:53.077 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:26:53.150 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:26:53.287 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:26:53.368 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:26:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:26:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:26:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:26:53.572 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:26:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:26:53.652 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:26:53.687 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:26:53.734 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:26:53.757 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:26:53.815 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:26:53.825 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:26:53.893 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:26:53.912 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:26:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:26:53.980 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:26:53.999 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:26:54.014 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:26:54.029 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:26:54.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:26:54.097 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:26:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:26:54.138 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:26:54.173 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:26:54.207 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:26:54.255 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:26:54.285 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:26:54.288 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:26:54.316 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:26:54.343 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:26:54.343 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:54.343 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:26:54.343 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:26:54.343 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:26:54.343 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:54.360 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:26:54.498 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:26:54.578 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:26:54.632 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:26:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:26:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:26:54.783 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:26:54.969 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:26:55.015 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:26:55.093 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:26:55.190 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:26:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:26:55.347 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:26:55.494 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:26:55.525 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:26:55.550 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:26:55.550 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:55.551 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:26:55.551 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:26:55.551 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:26:55.551 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:55.659 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:26:55.686 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:26:55.686 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:55.686 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:26:55.686 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:26:55.686 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:26:55.687 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:26:56.083 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:26:56.083 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:56.084 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:26:56.084 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:26:56.084 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:26:56.084 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:56.285 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:26:56.308 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:26:56.309 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:56.309 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:26:56.309 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:26:56.309 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:26:56.309 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:56.373 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:26:56.909 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:26:56.939 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:26:56.939 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:56.939 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:26:56.939 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:26:56.939 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:26:56.940 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:26:57.352 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:26:57.352 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:57.352 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:26:57.352 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:26:57.352 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:26:57.352 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:57.544 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:26:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:26:57.568 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:26:57.568 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:26:57.568 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:26:57.569 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:26:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:26:57.633 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:26:58.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:26:58.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:26:58.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:26:58.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:26:58.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:26:58.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:26:58.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:26:58.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:26:58.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:26:58.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:26:58.986 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:58.986 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:26:58.987 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:26:58.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:26:58.989 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:26:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:26:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:26:59.284 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:26:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:26:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:26:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:26:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:26:59.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-15T03:26:59.914 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:50 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:00.003 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:00.045 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:00.046 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:00.046 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:00.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:27:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:27:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:27:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:27:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:27:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:27:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:27:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:00.343 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:00.687 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:00.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-09-15T03:27:00.819 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:00.819 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:00.819 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 83 k 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 375 k 2024-09-15T03:27:00.820 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:01.000 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 460 kB/s | 83 kB 00:00 2024-09-15T03:27:01.001 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:01.001 INFO:teuthology.orchestra.run.smithi181.stdout:Total 455 kB/s | 83 kB 00:00 2024-09-15T03:27:01.001 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:01.006 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:01.007 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:01.008 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:01.009 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:01.083 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:01.270 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:01.359 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:01.401 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:01.402 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:01.402 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:01.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-09-15T03:27:01.510 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:01.597 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:01.958 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:02.019 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:02.151 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:02.152 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 83 k 2024-09-15T03:27:02.153 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 375 k 2024-09-15T03:27:02.153 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:02.158 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:02.159 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:02.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:27:02.352 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 414 kB/s | 83 kB 00:00 2024-09-15T03:27:02.353 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:02.353 INFO:teuthology.orchestra.run.smithi153.stdout:Total 411 kB/s | 83 kB 00:00 2024-09-15T03:27:02.353 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:02.358 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:02.358 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:02.360 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:02.361 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:02.438 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:02.860 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:02.920 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:02.947 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:03.052 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:03.053 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:03.053 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T03:27:03.054 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:03.054 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:03.054 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:27:03.054 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:03.054 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:03.055 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:03.055 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:03.055 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:03.055 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 137 k 2024-09-15T03:27:03.055 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 379 k 2024-09-15T03:27:03.055 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:03.258 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 674 kB/s | 137 kB 00:00 2024-09-15T03:27:03.258 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:03.259 INFO:teuthology.orchestra.run.smithi181.stdout:Total 668 kB/s | 137 kB 00:00 2024-09-15T03:27:03.259 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:03.266 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:03.266 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:03.322 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:03.323 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:03.336 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:03.471 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:03.567 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:27:03.587 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:03.587 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:03.587 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:03.587 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:03.587 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:03.587 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:03.592 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:27:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:27:03.592 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:27:03.593 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:03.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:27:04.330 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:04.469 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:04.469 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:04.469 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-15T03:27:04.469 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:04.470 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:04.471 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 137 k 2024-09-15T03:27:04.471 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 379 k 2024-09-15T03:27:04.471 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:27:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:04.535 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:04.535 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:04.683 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 649 kB/s | 137 kB 00:00 2024-09-15T03:27:04.683 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:04.684 INFO:teuthology.orchestra.run.smithi153.stdout:Total 644 kB/s | 137 kB 00:00 2024-09-15T03:27:04.684 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:04.691 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:04.691 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:04.747 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:04.748 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:04.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-15T03:27:04.892 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:04.959 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:27:04.984 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:27:04.984 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:27:04.984 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:27:04.984 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:05.326 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:05.420 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:05.463 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:05.464 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:05.464 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:05.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:27:05.947 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:27:05.947 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:05.947 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:05.948 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:06.093 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:06.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-09-15T03:27:06.223 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:27:06.224 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout:Install 7 Packages 2024-09-15T03:27:06.225 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:06.226 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.7 M 2024-09-15T03:27:06.226 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 80 M 2024-09-15T03:27:06.226 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:06.437 INFO:teuthology.orchestra.run.smithi181.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 96 kB/s | 15 kB 00:00 2024-09-15T03:27:06.454 INFO:teuthology.orchestra.run.smithi181.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1643.gd75 144 kB/s | 24 kB 00:00 2024-09-15T03:27:06.487 INFO:teuthology.orchestra.run.smithi181.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 180 kB/s | 9.0 kB 00:00 2024-09-15T03:27:06.504 INFO:teuthology.orchestra.run.smithi181.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-15T03:27:06.521 INFO:teuthology.orchestra.run.smithi181.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 609 kB/s | 41 kB 00:00 2024-09-15T03:27:06.546 INFO:teuthology.orchestra.run.smithi181.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-15T03:27:06.645 INFO:teuthology.orchestra.run.smithi181.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 9.3 MB/s | 3.4 MB 00:00 2024-09-15T03:27:06.648 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:06.649 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.6 MB/s | 3.7 MB 00:00 2024-09-15T03:27:06.649 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:06.657 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:06.657 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:06.708 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:06.739 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:06.739 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:06.798 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:06.840 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:06.841 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:06.841 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:06.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:27:07.046 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:07.123 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:27:07.157 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:27:07.212 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:27:07.244 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:27:07.272 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:27:07.470 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:07.604 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:27:07.605 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:07.606 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.7 M 2024-09-15T03:27:07.607 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 80 M 2024-09-15T03:27:07.607 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:08.086 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 178 kB/s | 24 kB 00:00 2024-09-15T03:27:08.111 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 90 kB/s | 15 kB 00:00 2024-09-15T03:27:08.153 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:27:08.178 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:27:08.204 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-15T03:27:08.229 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:27:08.267 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:27:08.301 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:27:08.337 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.6 MB/s | 3.4 MB 00:00 2024-09-15T03:27:08.341 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:08.341 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.0 MB/s | 3.7 MB 00:00 2024-09-15T03:27:08.341 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:08.349 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:08.350 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:08.447 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:08.448 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:08.750 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:27:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:27:08.894 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:27:08.926 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:27:08.957 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:27:09.363 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:27:09.364 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:27:09.364 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:27:09.364 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:27:09.364 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:27:09.364 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:27:09.364 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:27:09.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:27:09.649 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:09.649 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:09.650 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:09.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:27:09.957 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:27:10.000 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:27:10.406 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:01 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:10.539 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:27:10.540 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-15T03:27:10.541 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:10.542 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-15T03:27:10.542 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-15T03:27:10.542 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:10.838 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-15T03:27:10.872 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-15T03:27:10.947 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-15T03:27:10.989 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-15T03:27:11.022 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:27:11.022 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:27:11.022 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:27:11.023 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:27:11.023 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:27:11.023 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:27:11.023 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:27:11.030 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 927 kB/s | 38 kB 00:00 2024-09-15T03:27:11.115 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-15T03:27:11.290 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-09-15T03:27:11.306 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:27:11.306 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:11.306 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:11.306 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:11.307 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:11.365 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-15T03:27:11.390 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 889 kB/s | 245 kB 00:00 2024-09-15T03:27:11.449 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.2 MB/s | 250 kB 00:00 2024-09-15T03:27:11.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:27:11.541 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-15T03:27:11.849 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-15T03:27:11.850 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:11.850 INFO:teuthology.orchestra.run.smithi181.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-15T03:27:12.077 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:03 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:12.204 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:12.210 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:12.211 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-09-15T03:27:12.211 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:12.211 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-09-15T03:27:12.211 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:27:12.212 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:27:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:12.213 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:12.213 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-09-15T03:27:12.213 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-09-15T03:27:12.213 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:12.214 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 38 M 2024-09-15T03:27:12.214 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 212 M 2024-09-15T03:27:12.214 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:12.230 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:12.230 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:12.745 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-15T03:27:12.828 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 350 kB/s | 191 kB 00:00 2024-09-15T03:27:12.856 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-09-15T03:27:12.904 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-15T03:27:12.954 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-15T03:27:13.021 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-15T03:27:13.171 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-09-15T03:27:13.280 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-15T03:27:13.305 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 863 kB/s | 245 kB 00:00 2024-09-15T03:27:13.338 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.2 MB/s | 250 kB 00:00 2024-09-15T03:27:13.414 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-15T03:27:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:13.581 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:27:13.629 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:27:13.722 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:27:13.766 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:27:13.952 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:27:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:27:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:27:14.547 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-15T03:27:14.547 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:14.548 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 38 MB 00:02 2024-09-15T03:27:14.710 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:27:14.819 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:27:14.901 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:14.928 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:14.928 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:15.419 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:15.420 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:16.078 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:27:16.182 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:16.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:27:16.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:27:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:27:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:27:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:27:16.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:27:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:27:17.110 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:27:17.147 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:27:17.400 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:27:17.518 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:27:18.783 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:27:19.713 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:27:19.713 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:27:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:27:19.833 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:27:19.869 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:27:20.099 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:27:20.100 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:27:20.101 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:27:20.101 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:27:20.101 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:20.101 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:20.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:27:21.038 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:12 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:21.182 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:21.183 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:21.183 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:21.183 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:27:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-15T03:27:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:21.186 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-15T03:27:21.186 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-15T03:27:21.186 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:21.488 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 233 kB/s | 19 kB 00:00 2024-09-15T03:27:21.513 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 239 kB/s | 26 kB 00:00 2024-09-15T03:27:21.572 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 286 kB/s | 47 kB 00:00 2024-09-15T03:27:21.597 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-15T03:27:21.623 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-15T03:27:21.648 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-15T03:27:21.690 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T03:27:21.715 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-15T03:27:21.749 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-15T03:27:21.782 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-15T03:27:21.808 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-15T03:27:21.833 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-09-15T03:27:21.866 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-15T03:27:21.869 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:21.869 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-15T03:27:21.935 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:21.950 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:21.950 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:22.039 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:22.041 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:27:22.377 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:27:22.547 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:27:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:27:22.781 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:27:22.781 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:22.781 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:22.781 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:22.781 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:27:22.781 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:22.782 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:22.840 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:27:22.877 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:27:22.918 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:27:23.020 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:27:23.122 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:27:23.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:27:23.171 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:27:23.718 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:14 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:23.851 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:23.852 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:27:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:23.854 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:23.855 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-09-15T03:27:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:23.855 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.2 M 2024-09-15T03:27:23.855 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-09-15T03:27:23.855 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:23.897 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:27:23.959 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:27:23.993 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:27:24.038 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:27:24.062 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:27:24.067 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 289 kB/s | 47 kB 00:00 2024-09-15T03:27:24.410 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-15T03:27:24.510 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 42 kB/s | 26 kB 00:00 2024-09-15T03:27:24.810 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-15T03:27:24.910 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:27:24.910 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:27:24.910 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:27:24.910 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:27:24.911 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:27:25.202 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 200 kB/s | 159 kB 00:00 2024-09-15T03:27:25.240 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:27:25.240 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:27:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:27:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:27:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:27:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:25.242 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:25.244 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 778 kB/s | 32 kB 00:00 2024-09-15T03:27:25.269 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 568 kB/s | 14 kB 00:00 2024-09-15T03:27:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.6 MB/s | 220 kB 00:00 2024-09-15T03:27:25.320 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 344 kB/s | 279 kB 00:00 2024-09-15T03:27:25.362 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 15 MB/s | 1.0 MB 00:00 2024-09-15T03:27:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-15T03:27:25.404 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 90 kB/s | 54 kB 00:00 2024-09-15T03:27:25.420 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-09-15T03:27:25.424 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:25.424 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-09-15T03:27:25.490 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:25.506 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:25.506 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:25.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:27:25.592 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:25.593 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:26.068 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:17 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:26.103 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:26.200 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:27:26.201 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:27:26.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:27:26.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:27:26.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:27:26.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:26.204 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:26.204 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:26.204 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-15T03:27:26.204 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:26.204 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-15T03:27:26.205 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-15T03:27:26.205 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:26.260 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:27:26.322 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:27:26.377 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:27:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:27:26.456 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:27:26.558 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:27:26.659 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:27:26.707 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:27:26.768 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-15T03:27:26.818 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T03:27:26.844 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 538 kB/s | 124 kB 00:00 2024-09-15T03:27:26.869 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T03:27:26.911 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-15T03:27:27.169 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-15T03:27:27.172 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:27.172 INFO:teuthology.orchestra.run.smithi181.stdout:Total 7.2 MB/s | 7.0 MB 00:00 2024-09-15T03:27:27.243 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:27.254 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:27.254 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:27.353 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:27.354 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:27.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:27:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:27:27.530 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:27:27.575 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:27:27.599 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:27:27.658 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:27.836 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:27:27.896 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:27:28.333 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:27:28.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:27:28.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:27:28.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:27:28.417 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:27:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:27:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:27:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:27:28.686 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:27:28.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:27:28.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:27:28.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:27:28.688 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:28.688 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:28.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:27:29.342 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:27:29.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:27:29.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:27:29.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:27:29.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:27:29.343 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:27:29.509 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:20 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:29.621 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:27:29.621 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:29.621 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:29.622 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:29.643 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:27:29.644 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-09-15T03:27:29.645 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 7.0 M 2024-09-15T03:27:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-09-15T03:27:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:29.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-15T03:27:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 610 kB/s | 124 kB 00:00 2024-09-15T03:27:30.005 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-15T03:27:30.056 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-15T03:27:30.081 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-15T03:27:30.106 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-15T03:27:30.273 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-15T03:27:30.275 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:30.275 INFO:teuthology.orchestra.run.smithi153.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-15T03:27:30.346 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:30.357 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:30.357 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:30.410 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:30.456 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:30.456 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:30.544 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:30.544 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:30.544 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:30.544 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-15T03:27:30.545 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:30.546 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 893 k 2024-09-15T03:27:30.546 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.6 M 2024-09-15T03:27:30.546 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:30.663 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 820 kB/s | 80 kB 00:00 2024-09-15T03:27:30.764 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:30.847 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-15T03:27:30.847 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:30.848 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.9 MB/s | 893 kB 00:00 2024-09-15T03:27:30.859 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:30.880 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:30.880 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:30.930 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:27:30.960 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:30.960 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:30.974 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:27:31.143 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:31.284 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:27:31.356 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:27:31.501 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:27:31.577 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:27:31.633 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:27:31.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:27:32.019 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:27:32.019 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:27:32.288 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:27:32.288 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:32.289 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:32.289 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:27:32.289 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:32.289 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:27:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:27:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:27:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:27:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:27:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:27:32.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-15T03:27:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:27:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:32.692 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:32.693 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:32.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-09-15T03:27:33.140 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:24 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:33.279 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:33.280 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:33.280 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T03:27:33.280 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:33.280 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 270 k 2024-09-15T03:27:33.281 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-15T03:27:33.282 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:33.497 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:24 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:33.510 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T03:27:33.510 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:33.511 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-15T03:27:33.511 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:33.519 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:33.519 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:33.548 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:33.549 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:33.630 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:33.631 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:33.632 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-09-15T03:27:33.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:33.632 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 893 k 2024-09-15T03:27:33.632 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.6 M 2024-09-15T03:27:33.632 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:33.763 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:33.829 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:33.844 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:33.844 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:27:33.845 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:27:33.845 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:33.857 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 658 kB/s | 80 kB 00:00 2024-09-15T03:27:34.025 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-15T03:27:34.025 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:34.025 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.2 MB/s | 893 kB 00:00 2024-09-15T03:27:34.037 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:34.059 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:34.059 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:34.138 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:34.139 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:34.349 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:27:34.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:27:34.845 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:34.845 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:34.845 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:34.845 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:34.845 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:34.846 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:35.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-15T03:27:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:27:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:27:35.433 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:27:35.434 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:35.434 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:35.434 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:27:35.434 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:35.434 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:35.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-09-15T03:27:35.682 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:35.818 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:35.819 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:35.819 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T03:27:35.819 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 126 k 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 448 k 2024-09-15T03:27:35.820 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:36.264 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:27 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:36.307 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 259 kB/s | 126 kB 00:00 2024-09-15T03:27:36.307 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:36.308 INFO:teuthology.orchestra.run.smithi181.stdout:Total 258 kB/s | 126 kB 00:00 2024-09-15T03:27:36.308 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:36.313 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:36.314 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:36.376 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:36.377 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:36.405 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:36.406 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 270 k 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.3 M 2024-09-15T03:27:36.407 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:36.525 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:36.604 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.3 MB/s | 270 kB 00:00 2024-09-15T03:27:36.605 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:36.605 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 270 kB 00:00 2024-09-15T03:27:36.605 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:36.613 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:36.613 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:36.619 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:36.639 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:36.640 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:36.914 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:36.931 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:36.931 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:27:36.931 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:27:36.931 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:37.031 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:37.288 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:37.288 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:37.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-15T03:27:37.884 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:27:37.885 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:37.885 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:37.885 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:27:37.885 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:37.885 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:38.120 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:38.150 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-09-15T03:27:38.212 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:38.256 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:38.257 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:38.257 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:38.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-15T03:27:38.732 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:38.869 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:38.870 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:38.870 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:38.871 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:38.872 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 126 k 2024-09-15T03:27:38.872 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 448 k 2024-09-15T03:27:38.872 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:38.904 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:39.037 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:39.037 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 25 k 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 110 k 2024-09-15T03:27:39.038 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:39.075 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 619 kB/s | 126 kB 00:00 2024-09-15T03:27:39.077 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:39.077 INFO:teuthology.orchestra.run.smithi153.stdout:Total 613 kB/s | 126 kB 00:00 2024-09-15T03:27:39.077 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:39.082 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:39.082 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:39.146 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:39.146 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:39.193 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 161 kB/s | 25 kB 00:00 2024-09-15T03:27:39.195 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:39.195 INFO:teuthology.orchestra.run.smithi181.stdout:Total 160 kB/s | 25 kB 00:00 2024-09-15T03:27:39.195 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:39.198 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:39.198 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:39.200 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:39.200 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:39.302 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:39.415 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:39.873 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:40.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:40.002 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:40.002 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:40.002 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:40.002 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:40.002 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:40.133 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:40.133 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:40.133 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:40.133 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:40.133 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:40.133 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:40.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-15T03:27:40.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-09-15T03:27:40.741 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:40.832 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:40.875 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:40.876 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:40.876 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:40.935 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:40.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-15T03:27:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:41.072 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:41.073 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:41.073 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:41.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-09-15T03:27:41.503 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:41.593 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:41.635 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:41.636 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:41.637 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:41.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-15T03:27:41.712 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:41.845 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:41.846 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:41.846 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-15T03:27:41.846 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:41.846 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:41.846 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:27:41.846 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 25 k 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 110 k 2024-09-15T03:27:41.847 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:42.004 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 159 kB/s | 25 kB 00:00 2024-09-15T03:27:42.004 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:42.005 INFO:teuthology.orchestra.run.smithi153.stdout:Total 158 kB/s | 25 kB 00:00 2024-09-15T03:27:42.005 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:42.008 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:42.008 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:42.010 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:42.010 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:42.099 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:42.208 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:42.259 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:42.348 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:42.392 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:42.393 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:42.393 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:42.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-15T03:27:42.563 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:42.798 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:42.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:42.798 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:42.798 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:42.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:42.798 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:42.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-09-15T03:27:43.015 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:43.104 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:43.147 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:43.148 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:43.148 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:43.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-15T03:27:43.546 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:43.636 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:43.680 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:43.680 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:43.681 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:43.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-09-15T03:27:43.777 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:43.868 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:43.910 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:43.911 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:43.911 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:43.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-15T03:27:44.304 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:44.396 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:44.440 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:44.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-09-15T03:27:44.552 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:44.641 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:44.684 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:44.685 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T03:27:44.685 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:44.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-15T03:27:45.061 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:45.150 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:45.193 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:45.194 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:45.194 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:45.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-09-15T03:27:45.316 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:45.449 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 82 k 2024-09-15T03:27:45.450 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 222 k 2024-09-15T03:27:45.451 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:45.613 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 502 kB/s | 82 kB 00:00 2024-09-15T03:27:45.614 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:45.614 INFO:teuthology.orchestra.run.smithi181.stdout:Total 498 kB/s | 82 kB 00:00 2024-09-15T03:27:45.614 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:45.621 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:45.621 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:45.684 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:45.685 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:45.817 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:45.827 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:45.926 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:45.950 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:46.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-09-15T03:27:46.310 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:46.554 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:46.554 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:46.555 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:46.555 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:46.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:46.555 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:46.586 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:46.677 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:46.720 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:46.721 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:46.721 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:46.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-09-15T03:27:46.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-15T03:27:47.342 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:47.381 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:47.432 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:27:47.475 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:47.476 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-15T03:27:47.476 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:47.515 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:47.516 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:47.516 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T03:27:47.516 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.9 M 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 12 M 2024-09-15T03:27:47.517 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:47.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-09-15T03:27:47.851 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-15T03:27:47.851 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-15T03:27:47.852 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:47.861 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:47.941 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:48.098 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:48.232 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:48.232 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 k 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 222 k 2024-09-15T03:27:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:48.353 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:48.418 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:48.423 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 431 kB/s | 82 kB 00:00 2024-09-15T03:27:48.423 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:48.424 INFO:teuthology.orchestra.run.smithi153.stdout:Total 428 kB/s | 82 kB 00:00 2024-09-15T03:27:48.424 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:48.430 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:48.430 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:48.443 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:48.443 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:27:48.443 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:27:48.443 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:27:48.443 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:27:48.443 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:48.493 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:48.493 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:48.638 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:48.744 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:49.103 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:49.389 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:49.389 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:49.390 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:49.390 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:49.390 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:49.390 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:49.509 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:49.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:49.510 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:49.510 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:49.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:49.510 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:49.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-09-15T03:27:49.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-15T03:27:50.223 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:50.355 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:50.361 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:50.362 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:50.362 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-15T03:27:50.362 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:50.362 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:50.362 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:27:50.362 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.9 M 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 12 M 2024-09-15T03:27:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:50.491 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T03:27:50.492 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T03:27:50.493 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T03:27:50.493 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:50.493 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 165 k 2024-09-15T03:27:50.493 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 477 k 2024-09-15T03:27:50.493 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T03:27:50.847 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 467 kB/s | 165 kB 00:00 2024-09-15T03:27:50.847 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:50.848 INFO:teuthology.orchestra.run.smithi181.stdout:Total 464 kB/s | 165 kB 00:00 2024-09-15T03:27:50.848 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T03:27:50.856 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T03:27:50.856 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T03:27:50.921 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T03:27:50.922 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T03:27:51.083 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T03:27:51.143 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 3.8 MB/s | 2.9 MB 00:00 2024-09-15T03:27:51.144 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:51.144 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.8 MB/s | 2.9 MB 00:00 2024-09-15T03:27:51.144 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:51.153 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:51.153 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:51.189 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:51.233 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:51.234 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:51.627 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:51.647 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:51.727 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:51.752 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:51.752 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:27:51.753 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:27:51.753 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:27:51.753 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:27:51.753 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:51.887 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:51.887 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:51.887 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T03:27:51.887 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:51.887 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:51.887 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T03:27:52.168 DEBUG:teuthology.parallel:result is None 2024-09-15T03:27:52.843 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:52.843 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:52.843 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:52.843 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:52.843 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:52.843 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:53.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-09-15T03:27:53.682 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:26:09 AM UTC. 2024-09-15T03:27:53.818 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-15T03:27:53.820 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:53.821 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 165 k 2024-09-15T03:27:53.821 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 477 k 2024-09-15T03:27:53.821 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-15T03:27:54.230 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 404 kB/s | 165 kB 00:00 2024-09-15T03:27:54.232 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:27:54.232 INFO:teuthology.orchestra.run.smithi153.stdout:Total 402 kB/s | 165 kB 00:00 2024-09-15T03:27:54.232 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-15T03:27:54.238 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-15T03:27:54.238 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-15T03:27:54.302 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-15T03:27:54.302 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-15T03:27:54.466 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-15T03:27:54.577 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:55.029 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:55.302 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:27:55.302 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:55.302 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-15T03:27:55.302 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:27:55.302 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:55.302 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-15T03:27:55.543 DEBUG:teuthology.parallel:result is None 2024-09-15T03:27:55.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:27:55.693 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:27:55.715 INFO:teuthology.orchestra.run.smithi153.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:27:55.716 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:27:55.716 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:27:55.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:27:55.863 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:27:55.889 INFO:teuthology.orchestra.run.smithi181.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:27:55.889 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:27:55.889 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:27:55.891 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:27:55.891 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:55.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:27:55.926 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:55.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:27:55.959 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:27:55.960 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:55.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:27:55.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:27:56.058 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:56.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:27:56.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:27:56.154 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:27:56.154 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:56.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:27:56.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:27:56.244 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:56.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:27:56.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:27:56.337 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:27:56.337 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:56.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:27:56.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:27:56.426 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:56.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:27:56.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:27:56.521 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T03:27:56.601 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T03:27:56.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-09-15T03:27:56.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-09-15T03:27:56.629 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T03:27:56.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:27:56.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:27:56.695 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T03:27:56.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:27:56.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:27:56.770 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T03:27:56.770 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T03:27:56.770 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905270', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJlUxoBpFMPN3NKSagvoLtSCYyb0J8MtZuP5XGEVPd+ck0m4yl1yO/mpRdefjYPeFRNy49Q7y5Rgodt3LuLfDQ=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDZjzx2pYJsT+ulWbi43rj0ABxZKlsJpjsbSHQbFt4LGo5Da9WSN5n59zrjmC+nNgV3RczWisIQ7JeYmxig0rA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T03:27:56.771 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:27:56.799 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:27:56.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:27:56.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:27:56.880 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:56.880 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:27:56.938 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:27:56.938 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-09-15T03:27:56.993 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:27:56.993 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:56.993 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-15T03:27:56.993 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:56.993 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:56.994 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-15 03:27:52.007568432 +0000 2024-09-15T03:27:56.994 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-15 03:25:28.873720310 +0000 2024-09-15T03:27:56.994 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-15 03:25:28.873720310 +0000 2024-09-15T03:27:56.994 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-15 03:25:28.873720310 +0000 2024-09-15T03:27:56.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:27:57.057 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-15T03:27:57.057 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-15T03:27:57.057 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000264354 s, 1.9 MB/s 2024-09-15T03:27:57.059 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:27:57.115 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-15 03:27:52.007568432 +0000 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-15 03:25:29.207714559 +0000 2024-09-15T03:27:57.170 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-15 03:25:29.207714559 +0000 2024-09-15T03:27:57.171 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-15 03:25:29.207714559 +0000 2024-09-15T03:27:57.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:27:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-15T03:27:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-15T03:27:57.234 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000203339 s, 2.5 MB/s 2024-09-15T03:27:57.235 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:27:57.292 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-09-15T03:27:57.348 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:27:57.348 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:57.348 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-15T03:27:57.348 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:57.348 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:57.348 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-15 03:27:52.008568415 +0000 2024-09-15T03:27:57.349 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-15 03:25:29.530708997 +0000 2024-09-15T03:27:57.349 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-15 03:25:29.530708997 +0000 2024-09-15T03:27:57.349 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-15 03:25:29.530708997 +0000 2024-09-15T03:27:57.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:27:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-15T03:27:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-15T03:27:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000188325 s, 2.7 MB/s 2024-09-15T03:27:57.414 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:27:57.470 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-09-15T03:27:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:27:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:57.526 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-15T03:27:57.526 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-15 03:27:52.008568415 +0000 2024-09-15T03:27:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-15 03:25:29.842703624 +0000 2024-09-15T03:27:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-15 03:25:29.842703624 +0000 2024-09-15T03:27:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-15 03:25:29.842703624 +0000 2024-09-15T03:27:57.527 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:27:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-15T03:27:57.590 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-15T03:27:57.591 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.0001869 s, 2.7 MB/s 2024-09-15T03:27:57.592 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:27:57.649 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:27:57.649 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:57.649 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:27:57.666 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:27:57.666 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-15T03:27:57.723 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 03:27:48.700105943 +0000 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 03:25:30.533428623 +0000 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 03:25:30.533428623 +0000 2024-09-15T03:27:57.724 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 03:25:30.533428623 +0000 2024-09-15T03:27:57.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:27:57.789 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T03:27:57.789 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T03:27:57.789 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000245635 s, 2.1 MB/s 2024-09-15T03:27:57.790 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:27:57.849 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 03:27:48.701105924 +0000 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 03:25:30.866422196 +0000 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 03:25:30.867422177 +0000 2024-09-15T03:27:57.905 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 03:25:30.866422196 +0000 2024-09-15T03:27:57.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:27:57.968 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T03:27:57.968 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T03:27:57.968 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000171867 s, 3.0 MB/s 2024-09-15T03:27:57.970 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:27:58.026 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-15T03:27:58.081 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:27:58.081 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 03:27:48.701105924 +0000 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 03:25:31.203415692 +0000 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 03:25:31.203415692 +0000 2024-09-15T03:27:58.082 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 03:25:31.203415692 +0000 2024-09-15T03:27:58.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:27:58.145 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T03:27:58.145 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T03:27:58.145 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000228031 s, 2.2 MB/s 2024-09-15T03:27:58.146 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:27:58.203 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-15T03:27:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:27:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:27:58.258 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-15T03:27:58.258 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:27:58.258 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:27:58.258 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 03:27:48.701105924 +0000 2024-09-15T03:27:58.259 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 03:25:31.522409535 +0000 2024-09-15T03:27:58.259 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 03:25:31.522409535 +0000 2024-09-15T03:27:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 03:25:31.522409535 +0000 2024-09-15T03:27:58.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:27:58.321 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T03:27:58.322 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T03:27:58.322 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000215558 s, 2.4 MB/s 2024-09-15T03:27:58.323 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:27:58.379 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:27:58.379 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T03:27:58.379 INFO:tasks.ceph:Generating config... 2024-09-15T03:27:58.380 INFO:tasks.ceph:[client] debug monc = 20 2024-09-15T03:27:58.380 INFO:tasks.ceph:[client] debug ms = 1 2024-09-15T03:27:58.380 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T03:27:58.381 INFO:tasks.ceph:[global] ms type = async 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-15T03:27:58.381 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-15T03:27:58.382 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-15T03:27:58.382 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-15T03:27:58.382 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-15T03:27:58.382 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-15T03:27:58.382 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T03:27:58.382 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T03:27:58.382 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T03:27:58.383 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T03:27:58.384 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T03:27:58.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T03:27:58.464 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T03:27:58.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T03:27:58.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:27:58.569 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.153'), ('mon.c', '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]'), ('mon.b', '172.21.15.181')] 2024-09-15T03:27:58.570 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.181', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-15T03:27:58.571 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:58.571 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:27:58.625 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --addv c '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]' --add b 172.21.15.181 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = octopus 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:27:58.707 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 15 (octopus) 2024-09-15T03:27:58.708 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2024-09-15T03:27:58.708 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:27:58.708 INFO:teuthology.orchestra.run.smithi153.stdout:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:27:58.708 INFO:teuthology.orchestra.run.smithi153.stdout:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:27:58.708 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T03:27:58.709 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:27:58.764 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7dd05962-040e-4682-bebc-5f3c2ef6922e... 2024-09-15T03:27:58.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:27:58.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-09-15T03:27:58.848 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:27:58.849 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-09-15T03:27:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.851 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.181" 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2024-09-15T03:27:58.852 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay max = 1 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay probability = 0.005 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay type = mon 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject internal delays = 0.002 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 2500 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.853 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-09-15T03:27:58.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-09-15T03:27:58.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-09-15T03:27:58.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-09-15T03:27:58.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-09-15T03:27:58.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: bdev async discard = True 2024-09-15T03:27:58.856 INFO:teuthology.orchestra.run.smithi153.stdout: bdev enable discard = True 2024-09-15T03:27:58.857 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-09-15T03:27:58.857 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-09-15T03:27:58.857 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-09-15T03:27:58.857 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-09-15T03:27:58.857 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-09-15T03:27:58.857 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd memory target = 1610612736 2024-09-15T03:27:58.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 10 2024-09-15T03:27:58.859 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:27:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:27:58.861 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 25 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject missing keys = 0.05 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 5 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-09-15T03:27:58.862 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-09-15T03:27:58.863 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:27:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-15T03:27:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-15T03:27:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:27:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-15T03:27:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.181" 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: mon client directed command retry = 5 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 1 2024-09-15T03:27:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject delay max = 1 2024-09-15T03:27:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject delay probability = 0.005 2024-09-15T03:27:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject delay type = mon 2024-09-15T03:27:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject internal delays = 0.002 2024-09-15T03:27:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject socket failures = 2500 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: ms type = async 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-15T03:27:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-15T03:27:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-15T03:27:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:27:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-15T03:27:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-15T03:27:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore zero block detection = True 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2024-09-15T03:27:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2024-09-15T03:27:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock override recovery settings = True 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: osd memory target = 1610612736 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2024-09-15T03:27:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:27:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 10 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-15T03:27:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:27:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: mon min osdmap epochs = 25 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune interval = 2 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune min = 15 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune txsize = 2 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub inject missing keys = 0.05 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: paxos service trim min = 5 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-15T03:27:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-15T03:27:58.883 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-15T03:27:58.883 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T03:27:58.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T03:27:58.955 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T03:27:58.955 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:58.955 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T03:27:58.970 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:58.970 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T03:27:59.026 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:27:59.026 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:59.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:27:59.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:27:59.100 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:27:59.101 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:27:59.156 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:27:59.156 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:59.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:27:59.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:27:59.192 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:27:59.192 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:27:59.247 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T03:27:59.248 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T03:27:59.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:27:59.358 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:27:59.360 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T03:27:59.360 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T03:27:59.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T03:27:59.397 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T03:27:59.409 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T03:27:59.409 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T03:27:59.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T03:27:59.434 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:27:59.434 INFO:tasks.ceph:role: osd.0 2024-09-15T03:27:59.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:27:59.435 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:27:59.507 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:27:59.507 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:27:59.507 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:27:59.508 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:28:00.028 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-15T03:28:00.030 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-15T03:28:00.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T03:28:00.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T03:28:00.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T03:28:00.270 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:28:00.270 INFO:tasks.ceph:role: osd.1 2024-09-15T03:28:00.270 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:28:00.270 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:28:00.342 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:28:00.343 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:28:00.834 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-15T03:28:00.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-15T03:28:00.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T03:28:00.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T03:28:00.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T03:28:01.028 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:28:01.029 INFO:tasks.ceph:role: osd.2 2024-09-15T03:28:01.029 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:28:01.029 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:28:01.101 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:28:01.101 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:28:01.101 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:28:01.101 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:28:01.102 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:28:01.102 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:28:01.102 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:28:01.102 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:28:01.102 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:28:01.102 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:28:01.631 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-15T03:28:01.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-15T03:28:01.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T03:28:01.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T03:28:01.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:01.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:01.905+0000 7f5fa3612440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T03:28:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:01.907+0000 7f5fa3612440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T03:28:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:01.907+0000 7f5fa3612440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:28:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:01.907+0000 7f5fa3612440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:28:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:01.907+0000 7f5fa3612440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:28:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:01.915+0000 7f5fa3612440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T03:28:02.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:02.449+0000 7f5fa3612440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:03.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:03.217+0000 7f5fa3612440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:03.984+0000 7f5fa3612440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:04.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T03:28:04.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:04.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:04.889+0000 7f89aee34440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T03:28:04.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:04.889+0000 7f89aee34440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T03:28:04.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:04.889+0000 7f89aee34440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:28:04.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:04.889+0000 7f89aee34440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:28:04.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:04.889+0000 7f89aee34440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:28:04.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:04.890+0000 7f89aee34440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T03:28:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:05.413+0000 7f89aee34440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:06.177+0000 7f89aee34440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:06.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:06.943+0000 7f89aee34440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:07.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T03:28:07.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:07.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:07.852+0000 7f1e5031f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T03:28:07.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:07.852+0000 7f1e5031f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T03:28:07.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:07.852+0000 7f1e5031f440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:28:07.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:07.852+0000 7f1e5031f440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:28:07.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:07.852+0000 7f1e5031f440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:28:07.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:07.853+0000 7f1e5031f440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T03:28:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:08.377+0000 7f1e5031f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:09.141+0000 7f1e5031f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:09.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:09.904+0000 7f1e5031f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:10.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T03:28:10.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T03:28:10.760 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:28:10.761 INFO:tasks.ceph:role: osd.3 2024-09-15T03:28:10.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:28:10.761 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:28:10.837 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:28:11.359 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T03:28:11.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T03:28:11.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T03:28:11.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T03:28:11.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T03:28:11.566 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:28:11.566 INFO:tasks.ceph:role: osd.4 2024-09-15T03:28:11.566 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:28:11.566 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:28:11.639 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:28:12.098 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T03:28:12.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T03:28:12.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T03:28:12.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T03:28:12.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T03:28:12.284 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:28:12.284 INFO:tasks.ceph:role: osd.5 2024-09-15T03:28:12.284 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:28:12.285 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:28:12.356 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:28:12.357 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:28:12.842 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T03:28:12.844 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T03:28:12.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-15T03:28:12.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T03:28:13.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:13.128 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.127+0000 7f446f935440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T03:28:13.130 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.129+0000 7f446f935440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T03:28:13.130 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.129+0000 7f446f935440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:28:13.130 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.129+0000 7f446f935440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:28:13.130 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.129+0000 7f446f935440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:28:13.137 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.136+0000 7f446f935440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T03:28:13.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:13.662+0000 7f446f935440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:14.431 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:14.430+0000 7f446f935440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:15.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:15.196+0000 7f446f935440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:15.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T03:28:16.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:16.110 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.109+0000 7fd9e7e3b440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T03:28:16.110 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.109+0000 7fd9e7e3b440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T03:28:16.110 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.109+0000 7fd9e7e3b440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:28:16.110 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.109+0000 7fd9e7e3b440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:28:16.111 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.109+0000 7fd9e7e3b440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:28:16.111 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.110+0000 7fd9e7e3b440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T03:28:16.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:16.632+0000 7fd9e7e3b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:17.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:17.396+0000 7fd9e7e3b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:18.160 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:18.159+0000 7fd9e7e3b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:18.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T03:28:18.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:19.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.064+0000 7f46d4f27440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T03:28:19.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.065+0000 7f46d4f27440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T03:28:19.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.065+0000 7f46d4f27440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:28:19.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.065+0000 7f46d4f27440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:28:19.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.065+0000 7f46d4f27440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:28:19.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.065+0000 7f46d4f27440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T03:28:19.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:19.596+0000 7f46d4f27440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:20.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:20.361+0000 7f46d4f27440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:21.126 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:21.125+0000 7f46d4f27440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:28:21.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T03:28:21.946 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T03:28:21.946 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:28:21.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T03:28:21.977 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:28:21.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T03:28:22.049 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:28:22.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T03:28:22.119 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:22.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T03:28:22.147 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:22.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T03:28:22.217 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:22.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T03:28:22.284 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:22.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T03:28:22.354 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:22.354 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T03:28:22.410 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T03:28:22.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:28:22.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:28:22.453 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-09-15T03:28:22.454 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBBVOZm6GICNhAAYr3KCtSaePg0LZIy2pEYwg== 2024-09-15T03:28:22.454 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-09-15T03:28:22.454 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBEVOZmTqgMNRAAMCVlOIh9XLCH3sl52tEUmQ== 2024-09-15T03:28:22.454 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-09-15T03:28:22.454 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHVOZm0xTWMhAA2WsCh/dE8rWGhr+X84va1g== 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA/VOZmD6hVFRAAnS732ARNxpcYqAQ+ztrL9w== 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBNVOZmT6qnBxAApB9rkgUno1QCyuoWO9DkBA== 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBQVOZmA92NBhAA9vxGRdPFe12MS6P3x3+iUA== 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBTVOZmOxPrAxAAqIsCpO2gVfCA+v896t6mRw== 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-09-15T03:28:22.455 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA/VOZmB3SvFxAABZ8wteGtrbO2Ey9LpJ/sgA== 2024-09-15T03:28:22.474 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-15T03:28:22.474 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBBVOZm6GICNhAAYr3KCtSaePg0LZIy2pEYwg== 2024-09-15T03:28:22.474 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-15T03:28:22.474 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBEVOZmTqgMNRAAMCVlOIh9XLCH3sl52tEUmQ== 2024-09-15T03:28:22.474 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-15T03:28:22.474 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBHVOZm0xTWMhAA2WsCh/dE8rWGhr+X84va1g== 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQA/VOZmD6hVFRAAnS732ARNxpcYqAQ+ztrL9w== 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBNVOZmT6qnBxAApB9rkgUno1QCyuoWO9DkBA== 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.4] 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBQVOZmA92NBhAA9vxGRdPFe12MS6P3x3+iUA== 2024-09-15T03:28:22.475 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.5] 2024-09-15T03:28:22.476 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBTVOZmOxPrAxAAqIsCpO2gVfCA+v896t6mRw== 2024-09-15T03:28:22.476 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-09-15T03:28:22.476 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQA/VOZmB3SvFxAABZ8wteGtrbO2Ey9LpJ/sgA== 2024-09-15T03:28:22.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.613 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:28:22.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:28:22.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:28:22.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:28:22.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:28:22.909 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T03:28:22.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T03:28:22.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:28:23.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T03:28:23.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T03:28:23.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:28:23.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T03:28:23.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T03:28:23.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:28:23.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T03:28:23.376 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:23.378 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:28:23.431 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T03:28:23.431 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:28:23.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:28:23.434 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:28:23.434 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:28:23.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:28:23.436 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:28:23.436 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:28:23.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:28:23.473 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:28:23.473 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T03:28:23.474 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T03:28:23.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T03:28:23.476 INFO:tasks.ceph.mgr.x:Started 2024-09-15T03:28:23.477 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T03:28:23.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.572+0000 7fe8d1f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.572+0000 7fe8d1f9d640 10 monclient: build_initial_monmap 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.572+0000 7fe8d1f9d640 10 monclient: monmap: 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:23.573570+0000 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:23.573570+0000 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:23.574 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:23.575 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:23.575 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:23.575 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:23.575 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.573+0000 7fe8d1f9d640 1 Processor -- start 2024-09-15T03:28:23.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.573+0000 7fe8d1f9d640 1 -- start start 2024-09-15T03:28:23.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.573+0000 7fe8d1f9d640 10 monclient: init 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8cc0a0b90 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8cc09ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8cc136ea0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc09f810 con 0x7fe8cc0a0b90 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8cc06a430 con 0x7fe8cc136ea0 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0a6220 con 0x7fe8cc09ef00 2024-09-15T03:28:23.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8d1f9d640 10 monclient(hunting): authenticate will time out at 701.213140s 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:60706/0 (socket says 172.21.15.153:60706) 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/198181899 learned_addr learned my addr 172.21.15.153:0/198181899 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe8cc09ef00 auth_method 0 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 msgr2=0x7fe8cc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 msgr2=0x7fe8cc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:23.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request con 0x7fe8cc136ea0 auth_method 0 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 1 -- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 msgr2=0x7fe8cc139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 1 -- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 msgr2=0x7fe8cc139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.574+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.576+0000 7fe8cb7fe640 1 -- 172.21.15.153:0/198181899 >> v1:172.21.15.181:6789/0 conn(0x7fe8cc0a0b90 legacy=0x7fe8cc0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe8cc09ef00 auth_method 0 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 msgr2=0x7fe8cc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 msgr2=0x7fe8cc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request con 0x7fe8cc136ea0 auth_method 0 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 1 -- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 msgr2=0x7fe8cc139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 1 -- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 msgr2=0x7fe8cc139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.775+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:28:23.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.849+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919702612 0 0) 0x7fe8cc09f810 con 0x7fe8cc0a0b90 2024-09-15T03:28:23.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.849+0000 7fe8caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.849+0000 7fe8caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:23.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.849+0000 7fe8caffd640 10 monclient(hunting): my global_id is 4099 2024-09-15T03:28:23.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.849+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b80034e0 con 0x7fe8cc0a0b90 2024-09-15T03:28:23.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3834155097 0 0) 0x7fe8b80034e0 con 0x7fe8cc0a0b90 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 msgr2=0x7fe8cc09f2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8cc09f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 msgr2=0x7fe8cc139280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 --2- 172.21.15.153:0/198181899 >> v2:172.21.15.181:3300/0 conn(0x7fe8cc136ea0 0x7fe8cc139280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc09ca30 con 0x7fe8cc0a0b90 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:23.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fe8bc001770 con 0x7fe8cc0a0b90 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8d1f9d640 5 monclient: authenticate success, global_id 4099 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8d1f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: dump: 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.857 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 >> v1:172.21.15.181:6789/0 conn(0x7fe8cc0a0b90 legacy=0x7fe8cc0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8b8003ad0 0x7fe8b8003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): picked mon.a con 0x7fe8b8003ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8b8004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): picked mon.c con 0x7fe8cc09ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8004d30 0x7fe8b80051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): picked mon.b con 0x7fe8b8004d30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0a6220 con 0x7fe8b8003ad0 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800bc70 con 0x7fe8b8004d30 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800be30 con 0x7fe8cc09ef00 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8caffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8d1f9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8b8003ad0 0x7fe8b8003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8b8004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe8cc09ef00 auth_method 0 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): get_auth_request con 0x7fe8b8003ad0 auth_method 0 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8004d30 0x7fe8b80051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request con 0x7fe8b8004d30 auth_method 0 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 msgr2=0x7fe8b8004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09ef00 0x7fe8b8004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8004d30 msgr2=0x7fe8b80051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:23.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8004d30 0x7fe8b80051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 monclient: found mon.a 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 1 -- 172.21.15.153:0/198181899 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b80034e0 con 0x7fe8b8003ad0 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.853+0000 7fe8cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:23.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8cb7fe640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8b8003ad0 msgr2=0x7fe8b8003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:23.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8cb7fe640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8b8003ad0 0x7fe8b8003ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc001950 tx=0x7fe8cc09ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=833c6dbd5ba1c1ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc0070f0 con 0x7fe8b8003ad0 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: dump: 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:23.869 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8bc007290 con 0x7fe8b8003ad0 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8d1f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8d1f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 1 -- 172.21.15.153:0/198181899 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc005780 con 0x7fe8b8003ad0 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8d1f9d640 4 set_mon_vals no callback set 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.856+0000 7fe8caffd640 10 monclient: dump: 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.870 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 10 monclient: shutdown 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8b8003ad0 msgr2=0x7fe8b8003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 1 --2- 172.21.15.153:0/198181899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8b8003ad0 0x7fe8b8003ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc001950 tx=0x7fe8cc09ffd0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8cbfff640 1 -- 172.21.15.153:0/198181899 reap_dead start 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/198181899 shutdown_connections 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/198181899 >> 172.21.15.153:0/198181899 conn(0x7fe8cc0679c0 msgr2=0x7fe8cc0a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:23.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/198181899 shutdown_connections 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/198181899 wait complete. 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 10 monclient: build_initial_monmap 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.857+0000 7fe8d1f9d640 10 monclient: monmap: 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:23.872 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 Processor -- start 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 -- start start 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient: init 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8003ad0 0x7fe8cc141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b8003ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 0x7fe8cc11e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8cc141a80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc141e50 0x7fe8cc11edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8cc141e50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc1223d0 con 0x7fe8cc141a80 2024-09-15T03:28:23.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0a12a0 con 0x7fe8b8003ad0 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc09e950 con 0x7fe8cc141e50 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8d1f9d640 10 monclient(hunting): authenticate will time out at 701.497865s 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 0x7fe8cc11e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8cbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc141e50 0x7fe8cc11edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 0x7fe8cc11e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58468/0 (socket says 172.21.15.153:58468) 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/3917545643 learned_addr learned my addr 172.21.15.153:0/3917545643 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe8cc141a80 auth_method 0 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request con 0x7fe8cc141e50 auth_method 0 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.858+0000 7fe8cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cb7fe640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8003ad0 0x7fe8cc141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:23.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 10 monclient(hunting): handle_auth_done global_id 4098 payload 824 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc141e50 msgr2=0x7fe8cc11edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc141e50 0x7fe8cc11edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/3917545643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8003ad0 msgr2=0x7fe8cc141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8003ad0 0x7fe8cc141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 1 monclient: found mon.noname-a 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/3917545643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc120e20 con 0x7fe8cc141a80 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cb7fe640 10 monclient: get_auth_request con 0x7fe8b8003ad0 auth_method 0 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cbfff640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc141e50 0x7fe8cc11edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.859+0000 7fe8cb7fe640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8003ad0 0x7fe8cc141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c2ffd640 1 -- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 msgr2=0x7fe8cc11e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c2ffd640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 0x7fe8cc11e890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4013b40 tx=0x7fe8b4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4011b20 con 0x7fe8cc141a80 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: dump: 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.877 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 5 monclient: authenticate success, global_id 4098 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b400c070 con 0x7fe8cc141a80 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 10 monclient: _renew_subs 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc120760 con 0x7fe8cc141a80 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4011eb0 con 0x7fe8cc141a80 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:23.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 10 monclient: dump: 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8ca7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 10 monclient: _renew_subs 2024-09-15T03:28:23.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc122f30 con 0x7fe8cc141a80 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fe8b4005970 con 0x7fe8cc141a80 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.861+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe8b4016a60 con 0x7fe8cc141a80 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.862+0000 7fe8d1f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.862+0000 7fe8d1f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.862+0000 7fe8d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.862+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc05ed70 con 0x7fe8cc141a80 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.864+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fe8b401a030 con 0x7fe8cc141a80 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.864+0000 7fe8c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:23.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.864+0000 7fe8c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:23.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.922+0000 7fe8d1f9d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:28:23.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.922+0000 7fe8d1f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:28:23.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.922+0000 7fe8d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:23.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.922+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe8cc0723d0 con 0x7fe8cc141a80 2024-09-15T03:28:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.926+0000 7fe8c3fff640 1 -- 172.21.15.153:0/3917545643 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe8b400f040 con 0x7fe8cc141a80 2024-09-15T03:28:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.926+0000 7fe8c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:28:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.926+0000 7fe8c3fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-15T03:28:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-09-15T03:28:23.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 10 monclient: shutdown 2024-09-15T03:28:23.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:23.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 msgr2=0x7fe8cc11e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:23.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 0x7fe8cc11e890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4013b40 tx=0x7fe8b4008040 comp rx=0 tx=0).stop 2024-09-15T03:28:23.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 shutdown_connections 2024-09-15T03:28:23.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc141e50 0x7fe8cc11edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc141a80 0x7fe8cc11e890 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 --2- 172.21.15.153:0/3917545643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8b8003ad0 0x7fe8cc141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:23.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 >> 172.21.15.153:0/3917545643 conn(0x7fe8cc0679c0 msgr2=0x7fe8cc09cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:23.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 shutdown_connections 2024-09-15T03:28:23.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:23.927+0000 7fe8d1f9d640 1 -- 172.21.15.153:0/3917545643 wait complete. 2024-09-15T03:28:23.938 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T03:28:23.938 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:28:23.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T03:28:23.962 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:28:23.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T03:28:24.027 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-15T03:28:24.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T03:28:24.092 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:24.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T03:28:24.118 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:24.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T03:28:24.184 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:28:24.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T03:28:24.245 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.244+0000 7f1d1ee26ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:28:24.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 42ca15e3-1d0f-4207-acde-15c6655103b0 0 2024-09-15T03:28:24.286 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.285+0000 7f1d1ee26ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:28:24.338 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.337+0000 7f1d1ee26ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.378+0000 7fbbfd19d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.378+0000 7fbbfd19d640 10 monclient: build_initial_monmap 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.378+0000 7fbbfd19d640 10 monclient: monmap: 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:28:24.378937+0000 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:28:24.378937+0000 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:24.379 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:24.380 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:24.380 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:24.380 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:24.380 INFO:teuthology.orchestra.run.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:24.380 INFO:teuthology.orchestra.run.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:24.380 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:24.393 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.378+0000 7fbbfd19d640 1 Processor -- start 2024-09-15T03:28:24.393 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.378+0000 7fbbfd19d640 1 -- start start 2024-09-15T03:28:24.393 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.378+0000 7fbbfd19d640 10 monclient: init 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbf806f000 0x7fbbf809cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf806f000 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbf809a710 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf809e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf809d2e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf809baf0 con 0x7fbbf809a710 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf806f7e0 con 0x7fbbf806f000 2024-09-15T03:28:24.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf809bdd0 con 0x7fbbf809d2e0 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbfd19d640 10 monclient(hunting): authenticate will time out at 708.860331s 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbf806f000 0x7fbbf809cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbf806f000 0x7fbbf809cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60738/0 (socket says 172.21.15.181:60738) 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/3552493578 learned_addr learned my addr 172.21.15.181:0/3552493578 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request con 0x7fbbf806f000 auth_method 0 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf809e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): get_auth_request con 0x7fbbf809d2e0 auth_method 0 2024-09-15T03:28:24.395 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/3552493578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 msgr2=0x7fbbf809e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf809e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:24.396 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/3552493578 >> v1:172.21.15.153:6789/0 conn(0x7fbbf809a710 legacy=0x7fbbf809db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 monclient: found mon.noname-c 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/3552493578 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf809c7b0 con 0x7fbbf806f000 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.379+0000 7fbbf759e640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf809e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/3552493578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 msgr2=0x7fbbf809cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf6d9d640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf809cda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009940 tx=0x7fbbe00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dffd960b8318f9b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 1 -- 172.21.15.181:0/3552493578 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0032440 con 0x7fbbf806f000 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:24.397 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: dump: 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: _finish_auth 0 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 1 -- 172.21.15.181:0/3552493578 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe00325a0 con 0x7fbbf806f000 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbfd19d640 5 monclient: authenticate success, global_id 4108 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbfd19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbfd19d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 1 -- 172.21.15.181:0/3552493578 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0032940 con 0x7fbbf806f000 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbfd19d640 4 set_mon_vals no callback set 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.382+0000 7fbbf5d9b640 10 monclient: dump: 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:24.399 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 10 monclient: shutdown 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 -- 172.21.15.181:0/3552493578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 msgr2=0x7fbbf809cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf809cda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009940 tx=0x7fbbe00091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 -- 172.21.15.181:0/3552493578 shutdown_connections 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf809e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/3552493578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf809cda0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 -- 172.21.15.181:0/3552493578 >> 172.21.15.181:0/3552493578 conn(0x7fbbf8067d30 msgr2=0x7fbbf80a3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 -- 172.21.15.181:0/3552493578 shutdown_connections 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.383+0000 7fbbfd19d640 1 -- 172.21.15.181:0/3552493578 wait complete. 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.384+0000 7fbbfd19d640 10 monclient: build_initial_monmap 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.384+0000 7fbbfd19d640 10 monclient: monmap: 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:24.401 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.384+0000 7fbbfd19d640 1 Processor -- start 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.384+0000 7fbbfd19d640 1 -- start start 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.384+0000 7fbbfd19d640 10 monclient: init 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.384+0000 7fbbfd19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:24.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf813efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf806f000 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 0x7fbbf812fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf809a710 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf8119770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf809d2e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf811d480 con 0x7fbbf809a710 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf811cc50 con 0x7fbbf806f000 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf811d1a0 con 0x7fbbf809d2e0 2024-09-15T03:28:24.403 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbfd19d640 10 monclient(hunting): authenticate will time out at 708.866096s 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf813efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf813efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60744/0 (socket says 172.21.15.181:60744) 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/668380993 learned_addr learned my addr 172.21.15.181:0/668380993 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request con 0x7fbbf806f000 auth_method 0 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:24.404 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf659c640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 0x7fbbf812fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf8119770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf659c640 1 -- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 msgr2=0x7fbbf812fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf659c640 1 -- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 msgr2=0x7fbbf812fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): get_auth_request con 0x7fbbf809d2e0 auth_method 0 2024-09-15T03:28:24.405 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf659c640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 0x7fbbf812fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf659c640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 0x7fbbf812fac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:24.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 msgr2=0x7fbbf8119770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf8119770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 msgr2=0x7fbbf812fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 0x7fbbf812fac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 monclient: found mon.noname-b 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.385+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/668380993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0009a20 con 0x7fbbf806f000 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbf6d9d640 1 -- 172.21.15.181:0/668380993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 msgr2=0x7fbbf813efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:24.407 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbf6d9d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf813efb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0035040 tx=0x7fbbe0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0031a10 con 0x7fbbf806f000 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: dump: 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:24.408 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:24.409 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe0031b90 con 0x7fbbf806f000 2024-09-15T03:28:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 5 monclient: authenticate success, global_id 4111 2024-09-15T03:28:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe003d440 con 0x7fbbf806f000 2024-09-15T03:28:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 10 monclient: _renew_subs 2024-09-15T03:28:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe0009450 con 0x7fbbf806f000 2024-09-15T03:28:24.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:24.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:24.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbeeffd640 10 monclient: dump: 2024-09-15T03:28:24.411 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:24.411 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:24.411 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.412 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:24.412 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:24.412 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:24.412 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:24.413 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:24.413 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:24.413 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:24.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbf559a640 4 set_mon_vals no callback set 2024-09-15T03:28:24.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 10 monclient: _renew_subs 2024-09-15T03:28:24.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:24.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.387+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf813fd90 con 0x7fbbf806f000 2024-09-15T03:28:24.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.388+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fbbe0039040 con 0x7fbbf806f000 2024-09-15T03:28:24.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.388+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbbe0047440 con 0x7fbbf806f000 2024-09-15T03:28:24.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.388+0000 7fbbfd19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:24.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.388+0000 7fbbfd19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:24.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.388+0000 7fbbfd19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:24.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.388+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf805ed70 con 0x7fbbf806f000 2024-09-15T03:28:24.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.390+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fbbe0042070 con 0x7fbbf806f000 2024-09-15T03:28:24.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.390+0000 7fbbeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:24.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.390+0000 7fbbeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:24.459 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.458+0000 7fbbfd19d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "42ca15e3-1d0f-4207-acde-15c6655103b0", "id": 0}] 2024-09-15T03:28:25.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.458+0000 7fbbfd19d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "42ca15e3-1d0f-4207-acde-15c6655103b0", "id": 0}] 2024-09-15T03:28:25.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.458+0000 7fbbfd19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.458+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd new", "uuid": "42ca15e3-1d0f-4207-acde-15c6655103b0", "id": 0} v 0) v1 -- 0x7fbbf80723d0 con 0x7fbbf806f000 2024-09-15T03:28:25.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.464+0000 7fbbeeffd640 1 -- 172.21.15.181:0/668380993 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42ca15e3-1d0f-4207-acde-15c6655103b0", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbbe0037070 con 0x7fbbf806f000 2024-09-15T03:28:25.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.464+0000 7fbbeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "42ca15e3-1d0f-4207-acde-15c6655103b0", "id": 0}] 2024-09-15T03:28:25.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.464+0000 7fbbeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:25.260 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-09-15T03:28:25.261 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 10 monclient: shutdown 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 msgr2=0x7fbbf813efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf813efb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0035040 tx=0x7fbbe0032000 comp rx=0 tx=0).stop 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 shutdown_connections 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809d2e0 0x7fbbf8119770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf809a710 0x7fbbf812fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 --2- 172.21.15.181:0/668380993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806f000 0x7fbbf813efb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 >> 172.21.15.181:0/668380993 conn(0x7fbbf8067d30 msgr2=0x7fbbf80a1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:25.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 shutdown_connections 2024-09-15T03:28:25.263 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:24.465+0000 7fbbfd19d640 1 -- 172.21.15.181:0/668380993 wait complete. 2024-09-15T03:28:25.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new d77d5d67-15e9-427a-a58d-2dbfc5eded5d 1 2024-09-15T03:28:25.264 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.494+0000 7f1d1ee26ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:28:25.264 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.599+0000 7f1d1ee26ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:28:25.264 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.873+0000 7f1d1ee26ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:28:25.265 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:24.927+0000 7f1d1ee26ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:28:25.265 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:25.057+0000 7f1d1ee26ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:28:25.357 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.356+0000 7f220f2e8640 10 monclient: get_monmap_and_config 2024-09-15T03:28:25.357 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.356+0000 7f220f2e8640 10 monclient: build_initial_monmap 2024-09-15T03:28:25.357 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.356+0000 7f220f2e8640 10 monclient: monmap: 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:28:25.356976+0000 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:28:25.356976+0000 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.358 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.356+0000 7f220f2e8640 1 Processor -- start 2024-09-15T03:28:25.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 1 -- start start 2024-09-15T03:28:25.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient: init 2024-09-15T03:28:25.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:25.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:28:25.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 0x7f22080694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f22080a0b90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 0x7f220809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f220809ef00 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2208131b70 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220809f810 con 0x7f2208131b70 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220806a430 con 0x7f220809ef00 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22080a6220 con 0x7f22080a0b90 2024-09-15T03:28:25.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220f2e8640 10 monclient(hunting): authenticate will time out at 709.838544s 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d885640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f2208131b70 0x7f2208132010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:44302/0 (socket says 172.21.15.181:44302) 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d885640 1 -- 172.21.15.181:0/3292113961 learned_addr learned my addr 172.21.15.181:0/3292113961 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 1 --2- 172.21.15.181:0/3292113961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 0x7f22080694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 1 --2- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 0x7f220809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): get_auth_request con 0x7f22080a0b90 auth_method 0 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 10 monclient(hunting): get_auth_request con 0x7f220809ef00 auth_method 0 2024-09-15T03:28:25.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 1 -- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 msgr2=0x7f220809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 1 -- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 msgr2=0x7f220809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 1 --2- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 0x7f220809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220d084640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.357+0000 7f220c883640 1 --2- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 0x7f220809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 1 -- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 msgr2=0x7f220809f2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:25.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 1 --2- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 0x7f220809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 1 -- 172.21.15.181:0/3292113961 >> v1:172.21.15.181:6789/0 conn(0x7f2208131b70 legacy=0x7f2208132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:25.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 10 monclient: _finish_hunting 0 2024-09-15T03:28:25.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 1 monclient: found mon.noname-b 2024-09-15T03:28:25.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:25.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:25.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.358+0000 7f220d084640 1 -- 172.21.15.181:0/3292113961 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220809ca30 con 0x7f22080a0b90 2024-09-15T03:28:25.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220d084640 1 -- 172.21.15.181:0/3292113961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 msgr2=0x7f22080694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:25.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220d084640 1 --2- 172.21.15.181:0/3292113961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 0x7f22080694d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009850 tx=0x7f21f80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=255d684f19613d76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 1 -- 172.21.15.181:0/3292113961 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f802adf0 con 0x7f22080a0b90 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: dump: 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.372 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: _finish_auth 0 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 1 -- 172.21.15.181:0/3292113961 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f8032440 con 0x7f22080a0b90 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 5 monclient: authenticate success, global_id 4115 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:25.373 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:25.374 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 1 -- 172.21.15.181:0/3292113961 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f80327e0 con 0x7f22080a0b90 2024-09-15T03:28:25.374 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 4 set_mon_vals no callback set 2024-09-15T03:28:25.374 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.374 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:25.374 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f21fffff640 10 monclient: dump: 2024-09-15T03:28:25.374 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.375 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.375 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.375 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.375 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.375 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.375 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 10 monclient: shutdown 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 1 -- 172.21.15.181:0/3292113961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 msgr2=0x7f22080694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.376 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.361+0000 7f220f2e8640 1 --2- 172.21.15.181:0/3292113961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 0x7f22080694d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009850 tx=0x7f21f80090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 -- 172.21.15.181:0/3292113961 shutdown_connections 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 --2- 172.21.15.181:0/3292113961 >> v2:172.21.15.153:3300/0 conn(0x7f220809ef00 0x7f220809f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 --2- 172.21.15.181:0/3292113961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22080a0b90 0x7f22080694d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 -- 172.21.15.181:0/3292113961 >> 172.21.15.181:0/3292113961 conn(0x7f22080679c0 msgr2=0x7f22080a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 -- 172.21.15.181:0/3292113961 shutdown_connections 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 -- 172.21.15.181:0/3292113961 wait complete. 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 10 monclient: build_initial_monmap 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 10 monclient: monmap: 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2024-09-15T03:28:25.377 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:25.378 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.378 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:25.378 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:25.378 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:25.378 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.378 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.362+0000 7f220f2e8640 1 Processor -- start 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 -- start start 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient: init 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 0x7f220809a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f220809ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22080a0b90 0x7f2208130ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f22080a0b90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 0x7f220811b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f2208131b70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:25.379 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220811e3d0 con 0x7f22080a0b90 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2208142440 con 0x7f2208131b70 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22080a12a0 con 0x7f220809ef00 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220f2e8640 10 monclient(hunting): authenticate will time out at 709.844683s 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d885640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 0x7f220811b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 0x7f220809a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 0x7f220809a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:48406/0 (socket says 172.21.15.181:48406) 2024-09-15T03:28:25.380 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 1 -- 172.21.15.181:0/4015992463 learned_addr learned my addr 172.21.15.181:0/4015992463 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): get_auth_request con 0x7f220809ef00 auth_method 0 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d885640 1 -- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 msgr2=0x7f220811b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d885640 1 -- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 msgr2=0x7f220811b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d885640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 0x7f220811b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d885640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 0x7f220811b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22080a0b90 0x7f2208130ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): get_auth_request con 0x7f22080a0b90 auth_method 0 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220d084640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.363+0000 7f220c883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 10 monclient(hunting): handle_auth_done global_id 4118 payload 824 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 1 -- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 msgr2=0x7f220811b9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 0x7f220811b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 1 -- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22080a0b90 msgr2=0x7f2208130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22080a0b90 0x7f2208130ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 10 monclient: _finish_hunting 0 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 1 monclient: found mon.noname-c 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:25.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.364+0000 7f220d084640 1 -- 172.21.15.181:0/4015992463 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8009390 con 0x7f220809ef00 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220d084640 1 -- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 msgr2=0x7f220809a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220d084640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 0x7f220809a4f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009360 tx=0x7f21f8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8031a40 con 0x7f220809ef00 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: dump: 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.383 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f8031ba0 con 0x7f220809ef00 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 5 monclient: authenticate success, global_id 4118 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 10 monclient: _renew_subs 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220809ec70 con 0x7f220809ef00 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f803c400 con 0x7f220809ef00 2024-09-15T03:28:25.384 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 10 monclient: dump: 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21ff7fe640 4 set_mon_vals no callback set 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 10 monclient: _renew_subs 2024-09-15T03:28:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220811d090 con 0x7f220809ef00 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f21f803c830 con 0x7f220809ef00 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.366+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f21f80447f0 con 0x7f220809ef00 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.367+0000 7f220f2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.367+0000 7f220f2e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.367+0000 7f220f2e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.367+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220805ed70 con 0x7f220809ef00 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.369+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f21f80637a0 con 0x7f220809ef00 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.369+0000 7f21fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.369+0000 7f21fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:25.438 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.436+0000 7f220f2e8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d77d5d67-15e9-427a-a58d-2dbfc5eded5d", "id": 1}] 2024-09-15T03:28:25.438 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.437+0000 7f220f2e8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d77d5d67-15e9-427a-a58d-2dbfc5eded5d", "id": 1}] 2024-09-15T03:28:25.438 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.437+0000 7f220f2e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:25.438 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.437+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d77d5d67-15e9-427a-a58d-2dbfc5eded5d", "id": 1} v 0) v1 -- 0x7f220809e6f0 con 0x7f220809ef00 2024-09-15T03:28:25.441 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.440+0000 7f21fd7fa640 1 -- 172.21.15.181:0/4015992463 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d77d5d67-15e9-427a-a58d-2dbfc5eded5d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f21f8052070 con 0x7f220809ef00 2024-09-15T03:28:25.441 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.440+0000 7f21fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d77d5d67-15e9-427a-a58d-2dbfc5eded5d", "id": 1}] 2024-09-15T03:28:25.441 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.440+0000 7f21fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:25.442 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-15T03:28:25.442 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 10 monclient: shutdown 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 msgr2=0x7f220809a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 0x7f220809a4f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009360 tx=0x7f21f8032000 comp rx=0 tx=0).stop 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 shutdown_connections 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2208131b70 0x7f220811b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22080a0b90 0x7f2208130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 --2- 172.21.15.181:0/4015992463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f220809ef00 0x7f220809a4f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 >> 172.21.15.181:0/4015992463 conn(0x7f22080679c0 msgr2=0x7f220809cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 shutdown_connections 2024-09-15T03:28:25.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.441+0000 7f220f2e8640 1 -- 172.21.15.181:0/4015992463 wait complete. 2024-09-15T03:28:25.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new de4c4fc9-0d8c-444b-8f99-bef942a203db 2 2024-09-15T03:28:25.546 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.545+0000 7fcd83c34640 10 monclient: get_monmap_and_config 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.545+0000 7fcd83c34640 10 monclient: build_initial_monmap 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.545+0000 7fcd83c34640 10 monclient: monmap: 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:28:25.546125+0000 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:28:25.546125+0000 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:25.547 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.545+0000 7fcd83c34640 1 Processor -- start 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 1 -- start start 2024-09-15T03:28:25.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient: init 2024-09-15T03:28:25.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:25.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:28:25.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd7c0a0b90 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:25.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd7c09ef00 0x7fcd7c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): picked mon.noname-c con 0x7fcd7c09ef00 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd7c136ea0 0x7fcd7c139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): picked mon.noname-a con 0x7fcd7c136ea0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c09f810 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c06a430 con 0x7fcd7c136ea0 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c0a6220 con 0x7fcd7c09ef00 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd83c34640 10 monclient(hunting): authenticate will time out at 710.027588s 2024-09-15T03:28:25.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd7c09ef00 0x7fcd7c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd7c09ef00 0x7fcd7c09f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60764/0 (socket says 172.21.15.181:60764) 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 1 -- 172.21.15.181:0/1516983329 learned_addr learned my addr 172.21.15.181:0/1516983329 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): get_auth_request con 0x7fcd7c09ef00 auth_method 0 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd811cf640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 1 -- 172.21.15.181:0/1516983329 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c136ea0 msgr2=0x7fcd7c139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.546+0000 7fcd821d1640 1 --2- 172.21.15.181:0/1516983329 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c136ea0 0x7fcd7c139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 1 --2- 172.21.15.181:0/1516983329 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c136ea0 0x7fcd7c139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 1 -- 172.21.15.181:0/1516983329 >> v1:172.21.15.153:6789/0 conn(0x7fcd7c0a0b90 legacy=0x7fcd7c0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 10 monclient: _finish_hunting 0 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 1 monclient: found mon.noname-c 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd811cf640 1 -- 172.21.15.181:0/1516983329 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c09ca30 con 0x7fcd7c09ef00 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd821d1640 10 monclient: get_auth_request con 0x7fcd7c136ea0 auth_method 0 2024-09-15T03:28:25.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.547+0000 7fcd821d1640 1 --2- 172.21.15.181:0/1516983329 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c136ea0 0x7fcd7c139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd811cf640 1 -- 172.21.15.181:0/1516983329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c09ef00 msgr2=0x7fcd7c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd811cf640 1 --2- 172.21.15.181:0/1516983329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c09ef00 0x7fcd7c09f2d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0680d0 tx=0x7fcd6c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b764f2ffaddbf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 1 -- 172.21.15.181:0/1516983329 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c032440 con 0x7fcd7c09ef00 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: dump: 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.563 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: _finish_auth 0 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 1 -- 172.21.15.181:0/1516983329 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c0325a0 con 0x7fcd7c09ef00 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd83c34640 5 monclient: authenticate success, global_id 4114 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd83c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd83c34640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 1 -- 172.21.15.181:0/1516983329 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c032940 con 0x7fcd7c09ef00 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd83c34640 4 set_mon_vals no callback set 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.550+0000 7fcd809ce640 10 monclient: dump: 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.565 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 10 monclient: shutdown 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 -- 172.21.15.181:0/1516983329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c09ef00 msgr2=0x7fcd7c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 --2- 172.21.15.181:0/1516983329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c09ef00 0x7fcd7c09f2d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0680d0 tx=0x7fcd6c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 -- 172.21.15.181:0/1516983329 shutdown_connections 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 --2- 172.21.15.181:0/1516983329 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c136ea0 0x7fcd7c139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 --2- 172.21.15.181:0/1516983329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c09ef00 0x7fcd7c09f2d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 -- 172.21.15.181:0/1516983329 >> 172.21.15.181:0/1516983329 conn(0x7fcd7c0679c0 msgr2=0x7fcd7c0a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 -- 172.21.15.181:0/1516983329 shutdown_connections 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 1 -- 172.21.15.181:0/1516983329 wait complete. 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 10 monclient: build_initial_monmap 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.551+0000 7fcd83c34640 10 monclient: monmap: 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:25.567 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 Processor -- start 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 -- start start 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient: init 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c09ef00 0x7fcd7c097580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): picked mon.noname-c con 0x7fcd7c09ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 0x7fcd7c097ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): picked mon.noname-b con 0x7fcd7c0a0b90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c136ea0 0x7fcd7c098000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): picked mon.noname-a con 0x7fcd7c136ea0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c09ac80 con 0x7fcd7c136ea0 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0a12a0 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0a4ef0 con 0x7fcd7c09ef00 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd83c34640 10 monclient(hunting): authenticate will time out at 710.033504s 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 0x7fcd7c097ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 0x7fcd7c097ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60770/0 (socket says 172.21.15.181:60770) 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 -- 172.21.15.181:0/2003397130 learned_addr learned my addr 172.21.15.181:0/2003397130 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): get_auth_request con 0x7fcd7c0a0b90 auth_method 0 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd819d0640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c09ef00 0x7fcd7c097580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd821d1640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c136ea0 0x7fcd7c098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd819d0640 10 monclient(hunting): get_auth_request con 0x7fcd7c09ef00 auth_method 0 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd819d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd819d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd819d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd821d1640 10 monclient(hunting): get_auth_request con 0x7fcd7c136ea0 auth_method 0 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd821d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd821d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd821d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 -- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c09ef00 msgr2=0x7fcd7c097580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c09ef00 0x7fcd7c097580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 -- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c136ea0 msgr2=0x7fcd7c098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c136ea0 0x7fcd7c098000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 10 monclient: _finish_hunting 0 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.552+0000 7fcd811cf640 1 monclient: found mon.noname-b 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.553+0000 7fcd811cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.553+0000 7fcd811cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.553+0000 7fcd811cf640 1 -- 172.21.15.181:0/2003397130 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c009450 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.553+0000 7fcd819d0640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c09ef00 0x7fcd7c097580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd811cf640 1 -- 172.21.15.181:0/2003397130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 msgr2=0x7fcd7c097ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd811cf640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 0x7fcd7c097ac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009210 tx=0x7fcd6c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c000d10 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: dump: 2024-09-15T03:28:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c031c70 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 5 monclient: authenticate success, global_id 4117 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 10 monclient: _renew_subs 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c03b400 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c09a9a0 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 10 monclient: dump: 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd73fff640 4 set_mon_vals no callback set 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 10 monclient: _renew_subs 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.576 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c098540 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fcd6c039040 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.555+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fcd6c03bc30 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.556+0000 7fcd83c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.556+0000 7fcd83c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.556+0000 7fcd83c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.556+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c05ed70 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.558+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fcd6c049060 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.558+0000 7fcd71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.558+0000 7fcd71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:25.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.625+0000 7fcd83c34640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "de4c4fc9-0d8c-444b-8f99-bef942a203db", "id": 2}] 2024-09-15T03:28:25.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.625+0000 7fcd83c34640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "de4c4fc9-0d8c-444b-8f99-bef942a203db", "id": 2}] 2024-09-15T03:28:25.627 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.625+0000 7fcd83c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.627 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.625+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd new", "uuid": "de4c4fc9-0d8c-444b-8f99-bef942a203db", "id": 2} v 0) v1 -- 0x7fcd7c0723d0 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.634 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.633+0000 7fcd71ffb640 1 -- 172.21.15.181:0/2003397130 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "de4c4fc9-0d8c-444b-8f99-bef942a203db", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd6c03f030 con 0x7fcd7c0a0b90 2024-09-15T03:28:25.634 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.633+0000 7fcd71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "de4c4fc9-0d8c-444b-8f99-bef942a203db", "id": 2}] 2024-09-15T03:28:25.634 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.633+0000 7fcd71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:25.635 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-09-15T03:28:25.636 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 10 monclient: shutdown 2024-09-15T03:28:25.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:25.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 msgr2=0x7fcd7c097ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 0x7fcd7c097ac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009210 tx=0x7fcd6c032000 comp rx=0 tx=0).stop 2024-09-15T03:28:25.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 shutdown_connections 2024-09-15T03:28:25.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c136ea0 0x7fcd7c098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c0a0b90 0x7fcd7c097ac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.638 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 --2- 172.21.15.181:0/2003397130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c09ef00 0x7fcd7c097580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.638 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 >> 172.21.15.181:0/2003397130 conn(0x7fcd7c0679c0 msgr2=0x7fcd7c1366a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:25.638 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 shutdown_connections 2024-09-15T03:28:25.638 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.634+0000 7fcd83c34640 1 -- 172.21.15.181:0/2003397130 wait complete. 2024-09-15T03:28:25.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 8f6b762b-6dd0-4337-a983-9f53a45dde62 3 2024-09-15T03:28:25.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.736+0000 7f8dbbd00640 10 monclient: get_monmap_and_config 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.736+0000 7f8dbbd00640 10 monclient: build_initial_monmap 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.736+0000 7f8dbbd00640 10 monclient: monmap: 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:28:25.737481+0000 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:28:25.737481+0000 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:25.738 INFO:teuthology.orchestra.run.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:25.739 INFO:teuthology.orchestra.run.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:25.739 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.743 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 Processor -- start 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 -- start start 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient: init 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db40a0b90 0x7f8db40694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): picked mon.noname-b con 0x7f8db40a0b90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8db409ef00 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8db409f2d0 0x7f8db4132280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): picked mon.noname-c con 0x7f8db409f2d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db40a6220 con 0x7f8db409ef00 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db406a430 con 0x7f8db409f2d0 2024-09-15T03:28:25.744 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db409ca30 con 0x7f8db40a0b90 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.737+0000 7f8dbbd00640 10 monclient(hunting): authenticate will time out at 710.218950s 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db929b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f8db409ef00 0x7f8db4131b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:44314/0 (socket says 172.21.15.181:44314) 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8db409f2d0 0x7f8db4132280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db929b640 1 -- 172.21.15.181:0/3088349553 learned_addr learned my addr 172.21.15.181:0/3088349553 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): get_auth_request con 0x7f8db409f2d0 auth_method 0 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db40a0b90 0x7f8db40694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.745 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): get_auth_request con 0x7f8db40a0b90 auth_method 0 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-09-15T03:28:25.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 1 -- 172.21.15.181:0/3088349553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db40a0b90 msgr2=0x7f8db40694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db40a0b90 0x7f8db40694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 1 -- 172.21.15.181:0/3088349553 >> v1:172.21.15.181:6789/0 conn(0x7f8db409ef00 legacy=0x7f8db4131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 1 monclient: found mon.noname-c 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8dba29d640 1 -- 172.21.15.181:0/3088349553 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40a0f60 con 0x7f8db409f2d0 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.738+0000 7f8db9a9c640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db40a0b90 0x7f8db40694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:25.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8dba29d640 1 -- 172.21.15.181:0/3088349553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db409f2d0 msgr2=0x7f8db4132280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8dba29d640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db409f2d0 0x7f8db4132280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009e60 tx=0x7f8dac009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c565fa813eadc742 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 1 -- 172.21.15.181:0/3088349553 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac007760 con 0x7f8db409f2d0 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: dump: 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.748 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: _finish_auth 0 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 1 -- 172.21.15.181:0/3088349553 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac02d050 con 0x7f8db409f2d0 2024-09-15T03:28:25.750 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 1 -- 172.21.15.181:0/3088349553 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac007af0 con 0x7f8db409f2d0 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8db8a9a640 10 monclient: dump: 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.751 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8dbbd00640 5 monclient: authenticate success, global_id 4120 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8dbbd00640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.741+0000 7f8dbbd00640 4 set_mon_vals no callback set 2024-09-15T03:28:25.752 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 10 monclient: shutdown 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3088349553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db409f2d0 msgr2=0x7f8db4132280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db409f2d0 0x7f8db4132280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009e60 tx=0x7f8dac009780 comp rx=0 tx=0).stop 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3088349553 shutdown_connections 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db409f2d0 0x7f8db4132280 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3088349553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db40a0b90 0x7f8db40694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3088349553 >> 172.21.15.181:0/3088349553 conn(0x7f8db40679c0 msgr2=0x7f8db40a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:25.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3088349553 shutdown_connections 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.742+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3088349553 wait complete. 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient: build_initial_monmap 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient: monmap: 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:25.755 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:25.755 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.760 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 Processor -- start 2024-09-15T03:28:25.760 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 -- start start 2024-09-15T03:28:25.760 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient: init 2024-09-15T03:28:25.760 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:25.760 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:25.760 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db409ef00 0x7f8db4141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): picked mon.noname-c con 0x7f8db409ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db409f2d0 0x7f8db4133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): picked mon.noname-a con 0x7f8db409f2d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 0x7f8db411d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): picked mon.noname-b con 0x7f8db40a0b90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4134b30 con 0x7f8db409f2d0 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4141ef0 con 0x7f8db40a0b90 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4120e10 con 0x7f8db409ef00 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dbbd00640 10 monclient(hunting): authenticate will time out at 710.224878s 2024-09-15T03:28:25.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.743+0000 7f8dba29d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 0x7f8db411d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 0x7f8db411d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60786/0 (socket says 172.21.15.181:60786) 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 -- 172.21.15.181:0/3819591479 learned_addr learned my addr 172.21.15.181:0/3819591479 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): get_auth_request con 0x7f8db40a0b90 auth_method 0 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db409ef00 0x7f8db4141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): get_auth_request con 0x7f8db409ef00 auth_method 0 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db409f2d0 0x7f8db4133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): get_auth_request con 0x7f8db409f2d0 auth_method 0 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db9a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8db929b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:25.763 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 -- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db409ef00 msgr2=0x7f8db4141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db409ef00 0x7f8db4141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 -- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db409f2d0 msgr2=0x7f8db4133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db409f2d0 0x7f8db4133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 monclient: found mon.noname-b 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.744+0000 7f8dba29d640 1 -- 172.21.15.181:0/3819591479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0009390 con 0x7f8db40a0b90 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dba29d640 1 -- 172.21.15.181:0/3819591479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 msgr2=0x7f8db411d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dba29d640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 0x7f8db411d850 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0099f0 tx=0x7f8dac037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac031ac0 con 0x7f8db40a0b90 2024-09-15T03:28:25.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: dump: 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.765 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac040070 con 0x7f8db40a0b90 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dbbd00640 5 monclient: authenticate success, global_id 4123 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dbbd00640 10 monclient: _renew_subs 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dbbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dac009a20 con 0x7f8db40a0b90 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac031e70 con 0x7f8db40a0b90 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8da9ffb640 10 monclient: dump: 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:25.766 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.746+0000 7f8dabfff640 4 set_mon_vals no callback set 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 10 monclient: _renew_subs 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db41415c0 con 0x7f8db40a0b90 2024-09-15T03:28:25.767 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f8dac042030 con 0x7f8db40a0b90 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f8dac03b420 con 0x7f8db40a0b90 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.747+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db40a1310 con 0x7f8db40a0b90 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.749+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f8db40a1310 con 0x7f8db40a0b90 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.749+0000 7f8da9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:25.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.749+0000 7f8da9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:25.818 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.817+0000 7f8dbbd00640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8f6b762b-6dd0-4337-a983-9f53a45dde62", "id": 3}] 2024-09-15T03:28:25.818 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.817+0000 7f8dbbd00640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8f6b762b-6dd0-4337-a983-9f53a45dde62", "id": 3}] 2024-09-15T03:28:25.818 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.817+0000 7f8dbbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:25.818 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:25.817+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f6b762b-6dd0-4337-a983-9f53a45dde62", "id": 3} v 0) v1 -- 0x7f8db405ed70 con 0x7f8db40a0b90 2024-09-15T03:28:26.036 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.035+0000 7f1d1ee26ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:28:26.085 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.084+0000 7f1d1ee26ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:28:26.140 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.138+0000 7f8da9ffb640 1 -- 172.21.15.181:0/3819591479 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f6b762b-6dd0-4337-a983-9f53a45dde62", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8dac045640 con 0x7f8db40a0b90 2024-09-15T03:28:26.140 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.138+0000 7f8da9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8f6b762b-6dd0-4337-a983-9f53a45dde62", "id": 3}] 2024-09-15T03:28:26.140 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.138+0000 7f8da9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:26.141 INFO:teuthology.orchestra.run.smithi181.stdout:3 2024-09-15T03:28:26.143 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 10 monclient: shutdown 2024-09-15T03:28:26.143 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.143 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 msgr2=0x7f8db411d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.143 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 0x7f8db411d850 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0099f0 tx=0x7f8dac037000 comp rx=0 tx=0).stop 2024-09-15T03:28:26.144 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 shutdown_connections 2024-09-15T03:28:26.144 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db40a0b90 0x7f8db411d850 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.144 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db409f2d0 0x7f8db4133cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.144 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 --2- 172.21.15.181:0/3819591479 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db409ef00 0x7f8db4141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.144 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 >> 172.21.15.181:0/3819591479 conn(0x7f8db40679c0 msgr2=0x7f8db4134d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.145 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 shutdown_connections 2024-09-15T03:28:26.145 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.139+0000 7f8dbbd00640 1 -- 172.21.15.181:0/3819591479 wait complete. 2024-09-15T03:28:26.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new d9b93903-4d7a-471d-a4f8-16f4895771f9 4 2024-09-15T03:28:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.243+0000 7fb4e6089640 10 monclient: get_monmap_and_config 2024-09-15T03:28:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.243+0000 7fb4e6089640 10 monclient: build_initial_monmap 2024-09-15T03:28:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.243+0000 7fb4e6089640 10 monclient: monmap: 2024-09-15T03:28:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.255 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:28:26.244468+0000 2024-09-15T03:28:26.255 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:28:26.244468+0000 2024-09-15T03:28:26.255 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:26.256 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.256 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:26.256 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:26.256 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:26.256 INFO:teuthology.orchestra.run.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:26.257 INFO:teuthology.orchestra.run.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:26.257 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.257 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 1 Processor -- start 2024-09-15T03:28:26.257 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 1 -- start start 2024-09-15T03:28:26.257 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 10 monclient: init 2024-09-15T03:28:26.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:26.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:28:26.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e00a0b90 0x7fb4e00694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.258 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e00a0b90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:26.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4e009ef00 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:26.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4e009f2d0 0x7fb4e0132280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.244+0000 7fb4e6089640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e009f2d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:26.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e00a6220 con 0x7fb4e009ef00 2024-09-15T03:28:26.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e006a430 con 0x7fb4e009f2d0 2024-09-15T03:28:26.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e009ca30 con 0x7fb4e00a0b90 2024-09-15T03:28:26.261 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:26.261 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4e6089640 10 monclient(hunting): authenticate will time out at 710.726027s 2024-09-15T03:28:26.261 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4e009f2d0 0x7fb4e0132280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.261 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4e009f2d0 0x7fb4e0132280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60800/0 (socket says 172.21.15.181:60800) 2024-09-15T03:28:26.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 -- 172.21.15.181:0/3016903 learned_addr learned my addr 172.21.15.181:0/3016903 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:26.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): get_auth_request con 0x7fb4e009f2d0 auth_method 0 2024-09-15T03:28:26.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.262 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.263 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.263 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.263 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.263 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4df7fe640 1 --2- 172.21.15.181:0/3016903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e00a0b90 0x7fb4e00694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.263 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4df7fe640 10 monclient(hunting): get_auth_request con 0x7fb4e00a0b90 auth_method 0 2024-09-15T03:28:26.264 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.264 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.264 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 -- 172.21.15.181:0/3016903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e00a0b90 msgr2=0x7fb4e00694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 --2- 172.21.15.181:0/3016903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e00a0b90 0x7fb4e00694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 -- 172.21.15.181:0/3016903 >> v1:172.21.15.153:6789/0 conn(0x7fb4e009ef00 legacy=0x7fb4e0131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 monclient: found mon.noname-c 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:26.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.245+0000 7fb4dffff640 1 -- 172.21.15.181:0/3016903 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e00a0f60 con 0x7fb4e009f2d0 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.248+0000 7fb4dffff640 1 -- 172.21.15.181:0/3016903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009f2d0 msgr2=0x7fb4e0132280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.248+0000 7fb4dffff640 1 --2- 172.21.15.181:0/3016903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009f2d0 0x7fb4e0132280 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009f20 tx=0x7fb4d8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a854f1c120465ea5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 1 -- 172.21.15.181:0/3016903 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8007610 con 0x7fb4e009f2d0 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: dump: 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.266 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 1 -- 172.21.15.181:0/3016903 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d802d050 con 0x7fb4e009f2d0 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 1 -- 172.21.15.181:0/3016903 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d80079a0 con 0x7fb4e009f2d0 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4e6089640 5 monclient: authenticate success, global_id 4126 2024-09-15T03:28:26.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4e6089640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4e6089640 4 set_mon_vals no callback set 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.250+0000 7fb4deffd640 10 monclient: dump: 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.268 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 10 monclient: shutdown 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 -- 172.21.15.181:0/3016903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009f2d0 msgr2=0x7fb4e0132280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 --2- 172.21.15.181:0/3016903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009f2d0 0x7fb4e0132280 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009f20 tx=0x7fb4d8009840 comp rx=0 tx=0).stop 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 -- 172.21.15.181:0/3016903 shutdown_connections 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 --2- 172.21.15.181:0/3016903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009f2d0 0x7fb4e0132280 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 --2- 172.21.15.181:0/3016903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e00a0b90 0x7fb4e00694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 -- 172.21.15.181:0/3016903 >> 172.21.15.181:0/3016903 conn(0x7fb4e00679c0 msgr2=0x7fb4e00a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 -- 172.21.15.181:0/3016903 shutdown_connections 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.251+0000 7fb4e6089640 1 -- 172.21.15.181:0/3016903 wait complete. 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.252+0000 7fb4e6089640 10 monclient: build_initial_monmap 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.252+0000 7fb4e6089640 10 monclient: monmap: 2024-09-15T03:28:26.269 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.252+0000 7fb4e6089640 1 Processor -- start 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.252+0000 7fb4e6089640 1 -- start start 2024-09-15T03:28:26.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.252+0000 7fb4e6089640 10 monclient: init 2024-09-15T03:28:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 0x7fb4e0140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e009ef00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009f2d0 0x7fb4e0133e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e009f2d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a0b90 0x7fb4e011e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e00a0b90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00a12a0 con 0x7fb4e00a0b90 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e011fd80 con 0x7fb4e009ef00 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e0121020 con 0x7fb4e009f2d0 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4e6089640 10 monclient(hunting): authenticate will time out at 710.734707s 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 0x7fb4e0140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 0x7fb4e0140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60814/0 (socket says 172.21.15.181:60814) 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 1 -- 172.21.15.181:0/922804281 learned_addr learned my addr 172.21.15.181:0/922804281 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): get_auth_request con 0x7fb4e009ef00 auth_method 0 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.253+0000 7fb4df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4dffff640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a0b90 0x7fb4e011e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 1 -- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009f2d0 msgr2=0x7fb4e0133e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4d6ffd640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009f2d0 0x7fb4e0133e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009f2d0 0x7fb4e0133e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 1 -- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a0b90 msgr2=0x7fb4e011e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a0b90 0x7fb4e011e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 1 monclient: found mon.noname-b 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4df7fe640 1 -- 172.21.15.181:0/922804281 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d8009ae0 con 0x7fb4e009ef00 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4dffff640 10 monclient: get_auth_request con 0x7fb4e00a0b90 auth_method 0 2024-09-15T03:28:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4d6ffd640 10 monclient: get_auth_request con 0x7fb4e009f2d0 auth_method 0 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4dffff640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a0b90 0x7fb4e011e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.254+0000 7fb4d6ffd640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009f2d0 0x7fb4e0133e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4df7fe640 1 -- 172.21.15.181:0/922804281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 msgr2=0x7fb4e0140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4df7fe640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 0x7fb4e0140ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000d890 tx=0x7fb4d0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d000de20 con 0x7fb4e009ef00 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: dump: 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 5 monclient: authenticate success, global_id 4129 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 10 monclient: _renew_subs 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d000bae0 con 0x7fb4e009ef00 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e0120930 con 0x7fb4e009ef00 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:26.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d000be80 con 0x7fb4e009ef00 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 10 monclient: dump: 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.280 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4de7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 10 monclient: _renew_subs 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e0134390 con 0x7fb4e009ef00 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb4d00148d0 con 0x7fb4e009ef00 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.256+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fb4d0014b80 con 0x7fb4e009ef00 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.257+0000 7fb4e6089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.257+0000 7fb4e6089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.257+0000 7fb4e6089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.257+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e01226d0 con 0x7fb4e009ef00 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.259+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fb4d0014e00 con 0x7fb4e009ef00 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.259+0000 7fb4d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.259+0000 7fb4d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:26.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.327+0000 7fb4e6089640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d9b93903-4d7a-471d-a4f8-16f4895771f9", "id": 4}] 2024-09-15T03:28:26.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.327+0000 7fb4e6089640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d9b93903-4d7a-471d-a4f8-16f4895771f9", "id": 4}] 2024-09-15T03:28:26.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.327+0000 7fb4e6089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.327+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d9b93903-4d7a-471d-a4f8-16f4895771f9", "id": 4} v 0) v1 -- 0x7fb4e0122090 con 0x7fb4e009ef00 2024-09-15T03:28:26.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.335+0000 7fb4d7fff640 1 -- 172.21.15.181:0/922804281 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9b93903-4d7a-471d-a4f8-16f4895771f9", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb4d001e6c0 con 0x7fb4e009ef00 2024-09-15T03:28:26.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.335+0000 7fb4d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d9b93903-4d7a-471d-a4f8-16f4895771f9", "id": 4}] 2024-09-15T03:28:26.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.335+0000 7fb4d7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:26.336 INFO:teuthology.orchestra.run.smithi181.stdout:4 2024-09-15T03:28:26.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 10 monclient: shutdown 2024-09-15T03:28:26.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 msgr2=0x7fb4e0140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 0x7fb4e0140ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000d890 tx=0x7fb4d0008040 comp rx=0 tx=0).stop 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 shutdown_connections 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a0b90 0x7fb4e011e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009f2d0 0x7fb4e0133e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 --2- 172.21.15.181:0/922804281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e009ef00 0x7fb4e0140ff0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 >> 172.21.15.181:0/922804281 conn(0x7fb4e00679c0 msgr2=0x7fb4e009cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 shutdown_connections 2024-09-15T03:28:26.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.336+0000 7fb4e6089640 1 -- 172.21.15.181:0/922804281 wait complete. 2024-09-15T03:28:26.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 90de00c0-aaec-41ca-85a8-5693f2fe4e24 5 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.438+0000 7f40c002b640 10 monclient: get_monmap_and_config 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.438+0000 7f40c002b640 10 monclient: build_initial_monmap 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.439+0000 7f40c002b640 10 monclient: monmap: 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:28:26.439767+0000 2024-09-15T03:28:26.444 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:28:26.439767+0000 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:26.445 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.439+0000 7f40c002b640 1 Processor -- start 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.439+0000 7f40c002b640 1 -- start start 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.439+0000 7f40c002b640 10 monclient: init 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b806f000 0x7f40b809cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): picked mon.noname-b con 0x7f40b806f000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40b809a710 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f40b809d2e0 0x7f40b809e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): picked mon.noname-c con 0x7f40b809d2e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b809baf0 con 0x7f40b809a710 2024-09-15T03:28:26.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b806f7e0 con 0x7f40b809d2e0 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b809bdd0 con 0x7f40b806f000 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40c002b640 10 monclient(hunting): authenticate will time out at 710.921321s 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f40b809d2e0 0x7f40b809e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bd5c6640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f40b809a710 0x7f40b809db00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:44320/0 (socket says 172.21.15.181:44320) 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bd5c6640 1 -- 172.21.15.181:0/3130848121 learned_addr learned my addr 172.21.15.181:0/3130848121 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): get_auth_request con 0x7f40b809d2e0 auth_method 0 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bddc7640 1 --2- 172.21.15.181:0/3130848121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b806f000 0x7f40b809cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bddc7640 10 monclient(hunting): get_auth_request con 0x7f40b806f000 auth_method 0 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bddc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bddc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40bddc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 1 -- 172.21.15.181:0/3130848121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b806f000 msgr2=0x7f40b809cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 1 --2- 172.21.15.181:0/3130848121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b806f000 0x7f40b809cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 1 -- 172.21.15.181:0/3130848121 >> v1:172.21.15.181:6789/0 conn(0x7f40b809a710 legacy=0x7f40b809db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient: _finish_hunting 0 2024-09-15T03:28:26.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 1 monclient: found mon.noname-c 2024-09-15T03:28:26.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:26.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.440+0000 7f40be5c8640 1 -- 172.21.15.181:0/3130848121 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b809c7b0 con 0x7f40b809d2e0 2024-09-15T03:28:26.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.443+0000 7f40be5c8640 1 -- 172.21.15.181:0/3130848121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 msgr2=0x7f40b809e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:26.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.443+0000 7f40be5c8640 1 --2- 172.21.15.181:0/3130848121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b809e210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009e60 tx=0x7f40b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31ceaad4a728d550 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 1 -- 172.21.15.181:0/3130848121 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0007760 con 0x7f40b809d2e0 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: dump: 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.457 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: _finish_auth 0 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 5 monclient: authenticate success, global_id 4132 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 1 -- 172.21.15.181:0/3130848121 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b002d050 con 0x7f40b809d2e0 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 4 set_mon_vals no callback set 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 1 -- 172.21.15.181:0/3130848121 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0007b30 con 0x7f40b809d2e0 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40bcdc5640 10 monclient: dump: 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 10 monclient: shutdown 2024-09-15T03:28:26.459 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 -- 172.21.15.181:0/3130848121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 msgr2=0x7f40b809e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 --2- 172.21.15.181:0/3130848121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b809e210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009e60 tx=0x7f40b0009780 comp rx=0 tx=0).stop 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 -- 172.21.15.181:0/3130848121 shutdown_connections 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 --2- 172.21.15.181:0/3130848121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b809e210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 --2- 172.21.15.181:0/3130848121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b806f000 0x7f40b809cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 -- 172.21.15.181:0/3130848121 >> 172.21.15.181:0/3130848121 conn(0x7f40b8067d30 msgr2=0x7f40b80a3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 -- 172.21.15.181:0/3130848121 shutdown_connections 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.444+0000 7f40c002b640 1 -- 172.21.15.181:0/3130848121 wait complete. 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.445+0000 7f40c002b640 10 monclient: build_initial_monmap 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.445+0000 7f40c002b640 10 monclient: monmap: 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 0 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:26.460 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 0.000000 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:created 0.000000 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.445+0000 7f40c002b640 1 Processor -- start 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.445+0000 7f40c002b640 1 -- start start 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.445+0000 7f40c002b640 10 monclient: init 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:26.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b806f000 0x7f40b8131950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): picked mon.noname-a con 0x7f40b806f000 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b809a710 0x7f40b811b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): picked mon.noname-c con 0x7f40b809a710 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b811b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): picked mon.noname-b con 0x7f40b809d2e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b8099d80 con 0x7f40b806f000 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b813fc30 con 0x7f40b809d2e0 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b811ed50 con 0x7f40b809a710 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40c002b640 10 monclient(hunting): authenticate will time out at 710.927191s 2024-09-15T03:28:26.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b811b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b811b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60834/0 (socket says 172.21.15.181:60834) 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 -- 172.21.15.181:0/3604094839 learned_addr learned my addr 172.21.15.181:0/3604094839 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): get_auth_request con 0x7f40b809d2e0 auth_method 0 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b809a710 0x7f40b811b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.463 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b806f000 0x7f40b8131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): get_auth_request con 0x7f40b809a710 auth_method 0 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): get_auth_request con 0x7f40b806f000 auth_method 0 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bd5c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40bddc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 -- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b809a710 msgr2=0x7f40b811b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b809a710 0x7f40b811b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 -- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b806f000 msgr2=0x7f40b8131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b806f000 0x7f40b8131950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient: _finish_hunting 0 2024-09-15T03:28:26.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 monclient: found mon.noname-b 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.446+0000 7f40be5c8640 1 -- 172.21.15.181:0/3604094839 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0009a20 con 0x7f40b809d2e0 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.448+0000 7f40be5c8640 1 -- 172.21.15.181:0/3604094839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 msgr2=0x7f40b811b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.448+0000 7f40be5c8640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b811b990 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009e60 tx=0x7f40b0037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0031c30 con 0x7f40b809d2e0 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: dump: 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.466 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:26.467 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 5 monclient: authenticate success, global_id 4135 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b0040070 con 0x7f40b809d2e0 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 10 monclient: _renew_subs 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b811db10 con 0x7f40b809d2e0 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0045430 con 0x7f40b809d2e0 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:26.468 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 10 monclient: dump: 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:epoch 1 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:election_strategy: 1 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40affff640 4 set_mon_vals no callback set 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 10 monclient: _renew_subs 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b8134200 con 0x7f40b809d2e0 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f40b0042030 con 0x7f40b809d2e0 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.449+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f40b0045ca0 con 0x7f40b809d2e0 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.450+0000 7f40c002b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.450+0000 7f40c002b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.450+0000 7f40c002b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.450+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b805ed70 con 0x7f40b809d2e0 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.454+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f40b0045840 con 0x7f40b809d2e0 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.454+0000 7f40adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.454+0000 7f40adffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:26.470 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.468+0000 7f1d1ee26ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T03:28:26.522 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.521+0000 7f40c002b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "90de00c0-aaec-41ca-85a8-5693f2fe4e24", "id": 5}] 2024-09-15T03:28:26.522 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.521+0000 7f40c002b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "90de00c0-aaec-41ca-85a8-5693f2fe4e24", "id": 5}] 2024-09-15T03:28:26.522 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.521+0000 7f40c002b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:26.522 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.521+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd new", "uuid": "90de00c0-aaec-41ca-85a8-5693f2fe4e24", "id": 5} v 0) v1 -- 0x7f40b80723b0 con 0x7f40b809d2e0 2024-09-15T03:28:26.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.529+0000 7f40adffb640 1 -- 172.21.15.181:0/3604094839 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90de00c0-aaec-41ca-85a8-5693f2fe4e24", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f40b0057480 con 0x7f40b809d2e0 2024-09-15T03:28:26.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.529+0000 7f40adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "90de00c0-aaec-41ca-85a8-5693f2fe4e24", "id": 5}] 2024-09-15T03:28:26.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.529+0000 7f40adffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:26.531 INFO:teuthology.orchestra.run.smithi181.stdout:5 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 10 monclient: shutdown 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 msgr2=0x7f40b811b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b811b990 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009e60 tx=0x7f40b0037000 comp rx=0 tx=0).stop 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 shutdown_connections 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40b809d2e0 0x7f40b811b990 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b809a710 0x7f40b811b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 --2- 172.21.15.181:0/3604094839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40b806f000 0x7f40b8131950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 >> 172.21.15.181:0/3604094839 conn(0x7f40b8067d30 msgr2=0x7f40b8132b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 shutdown_connections 2024-09-15T03:28:26.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T03:28:26.530+0000 7f40c002b640 1 -- 172.21.15.181:0/3604094839 wait complete. 2024-09-15T03:28:26.541 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:28:26.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:28:26.544 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:28:26.544 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:28:26.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:28:26.546 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:28:26.546 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T03:28:26.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T03:28:26.549 INFO:tasks.ceph.osd.2:Started 2024-09-15T03:28:26.549 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:28:26.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:28:26.551 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:28:26.551 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T03:28:26.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T03:28:26.553 INFO:tasks.ceph.osd.4:Started 2024-09-15T03:28:26.553 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T03:28:26.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T03:28:26.555 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.554+0000 7f1d1ee26ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:28:26.556 INFO:tasks.ceph.osd.5:Started 2024-09-15T03:28:26.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:28:26.652 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.651+0000 7f1d1ee26ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.663+0000 7f42f5d09640 10 monclient: get_monmap_and_config 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.663+0000 7f42f5d09640 10 monclient: build_initial_monmap 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.663+0000 7f42f5d09640 10 monclient: monmap: 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:26.665214+0000 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:26.665214+0000 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:26.666 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 Processor -- start 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 -- start start 2024-09-15T03:28:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient: init 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f42f01318f0 0x7f42f0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): picked mon.noname-a con 0x7f42f01318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f42f009ea50 0x7f42f009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): picked mon.noname-c con 0x7f42f009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42f0068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f009fbf0 con 0x7f42f0068d90 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f0131d50 con 0x7f42f01318f0 2024-09-15T03:28:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f009fe30 con 0x7f42f009ea50 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42f5d09640 10 monclient(hunting): authenticate will time out at 704.304845s 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f42f01318f0 0x7f42f0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f42f01318f0 0x7f42f0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55906/0 (socket says 172.21.15.153:55906) 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 1 -- 172.21.15.153:0/3967583677 learned_addr learned my addr 172.21.15.153:0/3967583677 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 10 monclient(hunting): get_auth_request con 0x7f42f01318f0 auth_method 0 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.664+0000 7f42ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42eeffd640 1 --2- 172.21.15.153:0/3967583677 >> v2:172.21.15.181:3300/0 conn(0x7f42f009ea50 0x7f42f009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42eeffd640 10 monclient(hunting): get_auth_request con 0x7f42f009ea50 auth_method 0 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 1 -- 172.21.15.153:0/3967583677 >> v2:172.21.15.181:3300/0 conn(0x7f42f009ea50 msgr2=0x7f42f009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 1 --2- 172.21.15.153:0/3967583677 >> v2:172.21.15.181:3300/0 conn(0x7f42f009ea50 0x7f42f009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 1 -- 172.21.15.153:0/3967583677 >> v1:172.21.15.181:6789/0 conn(0x7f42f0068d90 legacy=0x7f42f00691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:26.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 1 monclient: found mon.noname-a 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42ef7fe640 1 -- 172.21.15.153:0/3967583677 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f00a1210 con 0x7f42f01318f0 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.665+0000 7f42eeffd640 1 --2- 172.21.15.153:0/3967583677 >> v2:172.21.15.181:3300/0 conn(0x7f42f009ea50 0x7f42f009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ef7fe640 1 -- 172.21.15.153:0/3967583677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f01318f0 msgr2=0x7f42f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ef7fe640 1 --2- 172.21.15.153:0/3967583677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f01318f0 0x7f42f0131f30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f42f006f600 tx=0x7f42e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9b605c6539f4c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 1 -- 172.21.15.153:0/3967583677 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0032440 con 0x7f42f01318f0 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: dump: 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 1 -- 172.21.15.153:0/3967583677 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e00325a0 con 0x7f42f01318f0 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 1 -- 172.21.15.153:0/3967583677 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0032940 con 0x7f42f01318f0 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:26.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42ee7fc640 10 monclient: dump: 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 5 monclient: authenticate success, global_id 4128 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 4 set_mon_vals no callback set 2024-09-15T03:28:26.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 10 monclient: shutdown 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 -- 172.21.15.153:0/3967583677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f01318f0 msgr2=0x7f42f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 --2- 172.21.15.153:0/3967583677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f01318f0 0x7f42f0131f30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f42f006f600 tx=0x7f42e00091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 -- 172.21.15.153:0/3967583677 shutdown_connections 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 --2- 172.21.15.153:0/3967583677 >> v2:172.21.15.181:3300/0 conn(0x7f42f009ea50 0x7f42f009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 --2- 172.21.15.153:0/3967583677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f01318f0 0x7f42f0131f30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 -- 172.21.15.153:0/3967583677 >> 172.21.15.153:0/3967583677 conn(0x7f42f0067110 msgr2=0x7f42f0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 -- 172.21.15.153:0/3967583677 shutdown_connections 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.668+0000 7f42f5d09640 1 -- 172.21.15.153:0/3967583677 wait complete. 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.669+0000 7f42f5d09640 10 monclient: build_initial_monmap 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.669+0000 7f42f5d09640 10 monclient: monmap: 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:26.677 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:26.678 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.669+0000 7f42f5d09640 1 Processor -- start 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.669+0000 7f42f5d09640 1 -- start start 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.669+0000 7f42f5d09640 10 monclient: init 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42f0068d90 0x7f42f009a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): picked mon.noname-b con 0x7f42f0068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 0x7f42f01309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): picked mon.noname-a con 0x7f42f009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42f01318f0 0x7f42f011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): picked mon.noname-c con 0x7f42f01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0141d50 con 0x7f42f009ea50 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f011ddc0 con 0x7f42f0068d90 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f00a4350 con 0x7f42f01318f0 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42f5d09640 10 monclient(hunting): authenticate will time out at 704.310337s 2024-09-15T03:28:26.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 0x7f42f01309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 0x7f42f01309c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55920/0 (socket says 172.21.15.153:55920) 2024-09-15T03:28:26.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 -- 172.21.15.153:0/1230332809 learned_addr learned my addr 172.21.15.153:0/1230332809 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:26.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): get_auth_request con 0x7f42f009ea50 auth_method 0 2024-09-15T03:28:26.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42f0068d90 0x7f42f009a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42f01318f0 0x7f42f011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): get_auth_request con 0x7f42f0068d90 auth_method 0 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): get_auth_request con 0x7f42f01318f0 auth_method 0 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:26.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient(hunting): handle_auth_done global_id 4131 payload 824 2024-09-15T03:28:26.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 -- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42f01318f0 msgr2=0x7f42f011b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42f01318f0 0x7f42f011b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 -- 172.21.15.153:0/1230332809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42f0068d90 msgr2=0x7f42f009a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42f0068d90 0x7f42f009a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 monclient: found mon.noname-a 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.670+0000 7f42eeffd640 1 -- 172.21.15.153:0/1230332809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0009450 con 0x7f42f009ea50 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42eeffd640 1 -- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 msgr2=0x7f42f01309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42eeffd640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 0x7f42f01309c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f42e400d8c0 tx=0x7f42e4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e400deb0 con 0x7f42f009ea50 2024-09-15T03:28:26.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: dump: 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:26.689 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e400bae0 con 0x7f42f009ea50 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 5 monclient: authenticate success, global_id 4131 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 10 monclient: _renew_subs 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e400bea0 con 0x7f42f009ea50 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f011cbf0 con 0x7f42f009ea50 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:26.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 10 monclient: dump: 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:26.691 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42edffb640 4 set_mon_vals no callback set 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 10 monclient: _renew_subs 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f009b350 con 0x7f42f009ea50 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f42e40148d0 con 0x7f42f009ea50 2024-09-15T03:28:26.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.673+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f42e401ec40 con 0x7f42f009ea50 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.674+0000 7f42f5d09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.674+0000 7f42f5d09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.674+0000 7f42f5d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.674+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f009bf80 con 0x7f42f009ea50 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.676+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f42f009bf80 con 0x7f42f009ea50 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.676+0000 7f42d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:26.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.676+0000 7f42d77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:26.697 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.696+0000 7f1d1ee26ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:28:26.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42f5d09640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:26.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42f5d09640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42f5d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42f011e0f0 con 0x7f42f009ea50 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42d77fe640 1 -- 172.21.15.153:0/1230332809 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f42e402d090 con 0x7f42f009ea50 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.731+0000 7f42d77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:26.734 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":8,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:28:26.522990+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:28:26.522990+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:28:26.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 10 monclient: shutdown 2024-09-15T03:28:26.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:26.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 msgr2=0x7f42f01309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:26.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 0x7f42f01309c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f42e400d8c0 tx=0x7f42e4008040 comp rx=0 tx=0).stop 2024-09-15T03:28:26.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 shutdown_connections 2024-09-15T03:28:26.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42f01318f0 0x7f42f011b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f009ea50 0x7f42f01309c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 --2- 172.21.15.153:0/1230332809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42f0068d90 0x7f42f009a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:26.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 >> 172.21.15.153:0/1230332809 conn(0x7f42f0067110 msgr2=0x7f42f00a4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:26.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 shutdown_connections 2024-09-15T03:28:26.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:26.732+0000 7f42f5d09640 1 -- 172.21.15.153:0/1230332809 wait complete. 2024-09-15T03:28:26.744 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T03:28:26.744 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T03:28:26.789 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.788+0000 7f1d1ee26ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T03:28:26.840 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.839+0000 7f1d1ee26ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:28:26.883 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.882+0000 7f1d1ee26ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:28:26.976 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:26.975+0000 7f1d1ee26ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:28:27.024 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:27.023+0000 7f1d1ee26ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:28:27.073 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:27.072+0000 7f1d1ee26ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:28:27.138 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:27.137+0000 7f1d1ee26ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:28:27.148 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-15T03:28:27.147+0000 7f415326f440 -1 Falling back to public interface 2024-09-15T03:28:27.150 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T03:28:27.149+0000 7f5813721440 -1 Falling back to public interface 2024-09-15T03:28:27.152 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-15T03:28:27.150+0000 7fa4cc63b440 -1 Falling back to public interface 2024-09-15T03:28:27.153 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-15T03:28:27.151+0000 7f3631c4f440 -1 Falling back to public interface 2024-09-15T03:28:27.153 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-15T03:28:27.151+0000 7fa35646e440 -1 Falling back to public interface 2024-09-15T03:28:27.157 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-15T03:28:27.155+0000 7f46f4e0c440 -1 Falling back to public interface 2024-09-15T03:28:27.185 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:27.184+0000 7f1d1ee26ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:28:27.227 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:27.226+0000 7f1d1ee26ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:28:27.351 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:28:27.436 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.434+0000 7fa980cef640 10 monclient: get_monmap_and_config 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.435+0000 7fa980cef640 10 monclient: build_initial_monmap 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.435+0000 7fa980cef640 10 monclient: monmap: 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:28:27.436760+0000 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:28:27.436760+0000 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:27.437 INFO:teuthology.misc.health.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:27.438 INFO:teuthology.misc.health.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:27.438 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.435+0000 7fa980cef640 1 Processor -- start 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.435+0000 7fa980cef640 1 -- start start 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.435+0000 7fa980cef640 10 monclient: init 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa97c097fa0 0x7fa97c0973f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): picked mon.noname-a con 0x7fa97c097fa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa97c098370 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa97c098770 0x7fa97c069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): picked mon.noname-c con 0x7fa97c098770 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c0a1470 con 0x7fa97c098370 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c0a1750 con 0x7fa97c097fa0 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c09fdb0 con 0x7fa97c098770 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa980cef640 10 monclient(hunting): authenticate will time out at 705.076315s 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa97c097fa0 0x7fa97c0973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa97c097fa0 0x7fa97c0973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55930/0 (socket says 172.21.15.153:55930) 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa973fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa97c098370 0x7fa97c068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53144/0 (socket says 172.21.15.153:53144) 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 -- 172.21.15.153:0/2408708185 learned_addr learned my addr 172.21.15.153:0/2408708185 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): get_auth_request con 0x7fa97c097fa0 auth_method 0 2024-09-15T03:28:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97ad9d640 1 --2- 172.21.15.153:0/2408708185 >> v2:172.21.15.181:3300/0 conn(0x7fa97c098770 0x7fa97c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97ad9d640 10 monclient(hunting): get_auth_request con 0x7fa97c098770 auth_method 0 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 -- 172.21.15.153:0/2408708185 >> v2:172.21.15.181:3300/0 conn(0x7fa97c098770 msgr2=0x7fa97c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:27.442 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 --2- 172.21.15.153:0/2408708185 >> v2:172.21.15.181:3300/0 conn(0x7fa97c098770 0x7fa97c069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.443 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 -- 172.21.15.153:0/2408708185 >> v1:172.21.15.153:6789/0 conn(0x7fa97c098370 legacy=0x7fa97c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:27.443 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:27.443 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 monclient: found mon.noname-a 2024-09-15T03:28:27.443 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:27.443 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:27.443 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.436+0000 7fa97a59c640 1 -- 172.21.15.153:0/2408708185 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c0a1190 con 0x7fa97c097fa0 2024-09-15T03:28:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa97a59c640 1 -- 172.21.15.153:0/2408708185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c097fa0 msgr2=0x7fa97c0973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa97a59c640 1 --2- 172.21.15.153:0/2408708185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c097fa0 0x7fa97c0973f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0a0160 tx=0x7fa96c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974f12b633713301 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 1 -- 172.21.15.153:0/2408708185 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c032440 con 0x7fa97c097fa0 2024-09-15T03:28:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: dump: 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: _finish_auth 0 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 1 -- 172.21.15.153:0/2408708185 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c0325a0 con 0x7fa97c097fa0 2024-09-15T03:28:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa980cef640 5 monclient: authenticate success, global_id 4134 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa980cef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa980cef640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 1 -- 172.21.15.153:0/2408708185 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c032940 con 0x7fa97c097fa0 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa980cef640 4 set_mon_vals no callback set 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.439+0000 7fa979d9b640 10 monclient: dump: 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.450 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 10 monclient: shutdown 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 -- 172.21.15.153:0/2408708185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c097fa0 msgr2=0x7fa97c0973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 --2- 172.21.15.153:0/2408708185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c097fa0 0x7fa97c0973f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0a0160 tx=0x7fa96c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 -- 172.21.15.153:0/2408708185 shutdown_connections 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 --2- 172.21.15.153:0/2408708185 >> v2:172.21.15.181:3300/0 conn(0x7fa97c098770 0x7fa97c069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 --2- 172.21.15.153:0/2408708185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c097fa0 0x7fa97c0973f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.451 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 -- 172.21.15.153:0/2408708185 >> 172.21.15.153:0/2408708185 conn(0x7fa97c0676c0 msgr2=0x7fa97c0a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 -- 172.21.15.153:0/2408708185 shutdown_connections 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 1 -- 172.21.15.153:0/2408708185 wait complete. 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 10 monclient: build_initial_monmap 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.440+0000 7fa980cef640 10 monclient: monmap: 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:27.452 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 Processor -- start 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 -- start start 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient: init 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa97c097fa0 0x7fa97c141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): picked mon.noname-b con 0x7fa97c097fa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 0x7fa97c131d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): picked mon.noname-a con 0x7fa97c098370 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:27.453 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa97c098770 0x7fa97c11b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): picked mon.noname-c con 0x7fa97c098770 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c142130 con 0x7fa97c098370 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c11f170 con 0x7fa97c097fa0 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c098cd0 con 0x7fa97c098770 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa980cef640 10 monclient(hunting): authenticate will time out at 705.081529s 2024-09-15T03:28:27.454 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 0x7fa97c131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa97c098770 0x7fa97c11b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 0x7fa97c131d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55946/0 (socket says 172.21.15.153:55946) 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 -- 172.21.15.153:0/2005583814 learned_addr learned my addr 172.21.15.153:0/2005583814 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): get_auth_request con 0x7fa97c098370 auth_method 0 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): get_auth_request con 0x7fa97c098770 auth_method 0 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:27.455 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa97c097fa0 0x7fa97c141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): get_auth_request con 0x7fa97c097fa0 auth_method 0 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:27.456 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa97a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient(hunting): handle_auth_done global_id 4137 payload 824 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 -- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa97c098770 msgr2=0x7fa97c11b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa97c098770 0x7fa97c11b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 -- 172.21.15.153:0/2005583814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa97c097fa0 msgr2=0x7fa97c141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa97c097fa0 0x7fa97c141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 monclient: found mon.noname-a 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.441+0000 7fa973fff640 1 -- 172.21.15.153:0/2005583814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c009450 con 0x7fa97c098370 2024-09-15T03:28:27.457 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa973fff640 1 -- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 msgr2=0x7fa97c131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa973fff640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 0x7fa97c131d70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c5b0 tx=0x7fa964007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964007510 con 0x7fa97c098370 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: dump: 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:27.458 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9640057d0 con 0x7fa97c098370 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 5 monclient: authenticate success, global_id 4137 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 10 monclient: _renew_subs 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c11d3d0 con 0x7fa97c098370 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964007690 con 0x7fa97c098370 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:27.459 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:27.460 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 10 monclient: dump: 2024-09-15T03:28:27.460 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:27.460 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:27.460 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.460 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:27.460 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa97959a640 4 set_mon_vals no callback set 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 10 monclient: _renew_subs 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c11dfa0 con 0x7fa97c098370 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fa96401b400 con 0x7fa97c098370 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.444+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fa964012a00 con 0x7fa97c098370 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.445+0000 7fa980cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.445+0000 7fa980cef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.445+0000 7fa980cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.445+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c09e8f0 con 0x7fa97c098370 2024-09-15T03:28:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.447+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fa964012ce0 con 0x7fa97c098370 2024-09-15T03:28:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.447+0000 7fa9727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.447+0000 7fa9727fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:27.503 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.500+0000 7fa980cef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:27.503 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.500+0000 7fa980cef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:27.503 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.500+0000 7fa980cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:27.503 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.500+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa97c0723b0 con 0x7fa97c098370 2024-09-15T03:28:27.504 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.501+0000 7fa9727fc640 1 -- 172.21.15.153:0/2005583814 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7fa96401b890 con 0x7fa97c098370 2024-09-15T03:28:27.504 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.501+0000 7fa9727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:27.504 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.501+0000 7fa9727fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:27.504 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-15T03:28:27.504 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":8,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:28:26.522990+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:28:26.522990+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 10 monclient: shutdown 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 msgr2=0x7fa97c131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 0x7fa97c131d70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c5b0 tx=0x7fa964007f00 comp rx=0 tx=0).stop 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 shutdown_connections 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa97c098770 0x7fa97c11b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.505 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c098370 0x7fa97c131d70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.506 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 --2- 172.21.15.153:0/2005583814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa97c097fa0 0x7fa97c141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:27.506 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 >> 172.21.15.153:0/2005583814 conn(0x7fa97c0676c0 msgr2=0x7fa97c0a1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:27.506 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 shutdown_connections 2024-09-15T03:28:27.506 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:27.502+0000 7fa980cef640 1 -- 172.21.15.153:0/2005583814 wait complete. 2024-09-15T03:28:27.513 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-15T03:28:27.836 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:27.835+0000 7f1d1ee26ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:28:28.858 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:28.857+0000 7f1d03aa6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T03:28:29.052 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T03:28:29.051+0000 7f1d03aa6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T03:28:30.590 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T03:28:30.589+0000 7f5813721440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:28:30.641 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-15T03:28:30.640+0000 7f415326f440 -1 osd.4 0 log_to_monitors true 2024-09-15T03:28:30.665 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-15T03:28:30.664+0000 7f3631c4f440 -1 osd.5 0 log_to_monitors true 2024-09-15T03:28:30.682 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-15T03:28:30.680+0000 7fa35646e440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:28:30.692 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-15T03:28:30.691+0000 7fa4cc63b440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:28:30.697 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-15T03:28:30.695+0000 7f46f4e0c440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:28:31.858 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T03:28:31.856+0000 7f580f9f9640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T03:28:31.859 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-15T03:28:31.855+0000 7fa4c8ff9640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T03:28:31.859 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-15T03:28:31.856+0000 7f362c3f6640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T03:28:31.859 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-15T03:28:31.855+0000 7fa352df9640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T03:28:31.860 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-15T03:28:31.856+0000 7f414fbf9640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T03:28:31.860 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-15T03:28:31.855+0000 7f46ef5f6640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T03:28:31.862 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-15T03:28:31.861+0000 7f4148a87640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:28:31.862 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-15T03:28:31.861+0000 7f3626beb640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:28:31.864 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-15T03:28:31.862+0000 7fa34bc87640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:28:31.864 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-15T03:28:31.862+0000 7f46e9deb640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:28:31.864 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-15T03:28:31.862+0000 7fa4c1e87640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:28:31.865 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T03:28:31.864+0000 7f58089eb640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:28:34.117 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:28:34.213 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.211+0000 7f1b4bf1d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:34.213 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.211+0000 7f1b4bf1d640 10 monclient: build_initial_monmap 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.211+0000 7f1b4bf1d640 10 monclient: monmap: 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:28:34.212979+0000 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:28:34.212979+0000 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:34.214 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:34.215 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.211+0000 7f1b4bf1d640 1 Processor -- start 2024-09-15T03:28:34.215 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 1 -- start start 2024-09-15T03:28:34.215 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient: init 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1b44131920 0x7f1b44131cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1b44131920 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b4409ea60 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1b44136c50 0x7f1b44139030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1b44136c50 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440a04d0 con 0x7f1b4409ea60 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.225 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b440a1600 con 0x7f1b44136c50 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4409ff20 con 0x7f1b44131920 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4bf1d640 10 monclient(hunting): authenticate will time out at 711.852642s 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1b44136c50 0x7f1b44139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1b44136c50 0x7f1b44139030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56000/0 (socket says 172.21.15.153:56000) 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 1 -- 172.21.15.153:0/3400181811 learned_addr learned my addr 172.21.15.153:0/3400181811 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): get_auth_request con 0x7f1b44136c50 auth_method 0 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.226 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b49cb9640 1 --2- 172.21.15.153:0/3400181811 >> v2:172.21.15.181:3300/0 conn(0x7f1b44131920 0x7f1b44131cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b49cb9640 10 monclient(hunting): get_auth_request con 0x7f1b44131920 auth_method 0 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b49cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b49cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.212+0000 7f1b49cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_done global_id 4173 payload 274 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 1 -- 172.21.15.153:0/3400181811 >> v2:172.21.15.181:3300/0 conn(0x7f1b44131920 msgr2=0x7f1b44131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 1 --2- 172.21.15.153:0/3400181811 >> v2:172.21.15.181:3300/0 conn(0x7f1b44131920 0x7f1b44131cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.227 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 1 -- 172.21.15.153:0/3400181811 >> v1:172.21.15.181:6789/0 conn(0x7f1b4409ea60 legacy=0x7f1b4409eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 10 monclient: _finish_hunting 0 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 1 monclient: found mon.noname-a 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.213+0000 7f1b4a4ba640 1 -- 172.21.15.153:0/3400181811 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4409fc00 con 0x7f1b44136c50 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.215+0000 7f1b4a4ba640 1 -- 172.21.15.153:0/3400181811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b44136c50 msgr2=0x7f1b44139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.215+0000 7f1b4a4ba640 1 --2- 172.21.15.153:0/3400181811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b44136c50 0x7f1b44139030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1b440a0250 tx=0x7f1b3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c81aff1ba862142 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 1 -- 172.21.15.153:0/3400181811 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c007610 con 0x7f1b44136c50 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: dump: 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:34.228 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.229 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: _finish_auth 0 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 1 -- 172.21.15.153:0/3400181811 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c02d050 con 0x7f1b44136c50 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 5 monclient: authenticate success, global_id 4173 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 1 -- 172.21.15.153:0/3400181811 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c0079a0 con 0x7f1b44136c50 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 4 set_mon_vals no callback set 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:34.230 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b48cb7640 10 monclient: dump: 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 10 monclient: shutdown 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:34.231 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/3400181811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b44136c50 msgr2=0x7f1b44139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/3400181811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b44136c50 0x7f1b44139030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1b440a0250 tx=0x7f1b3c009840 comp rx=0 tx=0).stop 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/3400181811 shutdown_connections 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/3400181811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b44136c50 0x7f1b44139030 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/3400181811 >> v2:172.21.15.181:3300/0 conn(0x7f1b44131920 0x7f1b44131cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/3400181811 >> 172.21.15.153:0/3400181811 conn(0x7f1b440a50f0 msgr2=0x7f1b440a54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/3400181811 shutdown_connections 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.216+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/3400181811 wait complete. 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.217+0000 7f1b4bf1d640 10 monclient: build_initial_monmap 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.217+0000 7f1b4bf1d640 10 monclient: monmap: 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:34.232 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.217+0000 7f1b4bf1d640 1 Processor -- start 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.217+0000 7f1b4bf1d640 1 -- start start 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.217+0000 7f1b4bf1d640 10 monclient: init 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 0x7f1b440970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.233 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1b4409ea60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b44131920 0x7f1b44097630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1b44131920 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b44136c50 0x7f1b44097b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1b44136c50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4409da00 con 0x7f1b4409ea60 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4409a790 con 0x7f1b44136c50 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440985d0 con 0x7f1b44131920 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4bf1d640 10 monclient(hunting): authenticate will time out at 711.858245s 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 0x7f1b440970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.234 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 0x7f1b440970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56004/0 (socket says 172.21.15.153:56004) 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 -- 172.21.15.153:0/4044653589 learned_addr learned my addr 172.21.15.153:0/4044653589 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): get_auth_request con 0x7f1b4409ea60 auth_method 0 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b44131920 0x7f1b44097630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): get_auth_request con 0x7f1b44131920 auth_method 0 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.235 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b44136c50 0x7f1b44097b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): get_auth_request con 0x7f1b44136c50 auth_method 0 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b494b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.236 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b4a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient(hunting): handle_auth_done global_id 4176 payload 824 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 -- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b44131920 msgr2=0x7f1b44097630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b44131920 0x7f1b44097630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 -- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b44136c50 msgr2=0x7f1b44097b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b44136c50 0x7f1b44097b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 monclient: found mon.noname-a 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.218+0000 7f1b49cb9640 1 -- 172.21.15.153:0/4044653589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c009ae0 con 0x7f1b4409ea60 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.220+0000 7f1b49cb9640 1 -- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 msgr2=0x7f1b440970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:34.237 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.220+0000 7f1b49cb9640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 0x7f1b440970f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000ea50 tx=0x7f1b400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40009600 con 0x7f1b4409ea60 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: dump: 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.238 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 5 monclient: authenticate success, global_id 4176 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4000cc20 con 0x7f1b4409ea60 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 10 monclient: _renew_subs 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4409b030 con 0x7f1b4409ea60 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40013470 con 0x7f1b4409ea60 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b35ffb640 10 monclient: dump: 2024-09-15T03:28:34.239 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b37fff640 4 set_mon_vals no callback set 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 10 monclient: _renew_subs 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.221+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4409ad50 con 0x7f1b4409ea60 2024-09-15T03:28:34.240 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 44954+0+0 (secure 0 0 0) 0x7f1b40013820 con 0x7f1b4409ea60 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b4bf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b4bf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b4bf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b440723b0 con 0x7f1b4409ea60 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b35ffb640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b280386e0 0x7f1b2803ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.222+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3603+0+0 (secure 0 0 0) 0x7f1b4004b850 con 0x7f1b4409ea60 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.223+0000 7f1b494b8640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b280386e0 0x7f1b2803ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.241 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.223+0000 7f1b494b8640 10 monclient: get_auth_request con 0x7f1b280386e0 auth_method 0 2024-09-15T03:28:34.242 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.225+0000 7f1b494b8640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b280386e0 0x7f1b2803ab90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b44136c00 tx=0x7f1b38006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:34.242 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.227+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1b400236b0 con 0x7f1b4409ea60 2024-09-15T03:28:34.242 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.227+0000 7f1b35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:34.242 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.227+0000 7f1b35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:34.384 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b4bf1d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:34.384 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b4bf1d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:34.384 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b4bf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.384 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b44067ea0 con 0x7f1b4409ea60 2024-09-15T03:28:34.385 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b35ffb640 1 -- 172.21.15.153:0/4044653589 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8140 (secure 0 0 0) 0x7f1b400232f0 con 0x7f1b4409ea60 2024-09-15T03:28:34.385 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:34.385 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.382+0000 7f1b35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:34.385 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-15T03:28:34.385 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:28:32.859989+0000","last_up_change":"2024-09-15T03:28:32.859989+0000","last_in_change":"2024-09-15T03:28:26.522990+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":31922},{"type":"v1","addr":"172.21.15.153:6801","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":31922},{"type":"v1","addr":"172.21.15.153:6803","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":31922},{"type":"v1","addr":"172.21.15.153:6807","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":31922},{"type":"v1","addr":"172.21.15.153:6805","nonce":31922}]},"public_addr":"172.21.15.153:6801/31922","cluster_addr":"172.21.15.153:6803/31922","heartbeat_back_addr":"172.21.15.153:6807/31922","heartbeat_front_addr":"172.21.15.153:6805/31922","state":["exists","up"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":31923},{"type":"v1","addr":"172.21.15.153:6809","nonce":31923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":31923},{"type":"v1","addr":"172.21.15.153:6811","nonce":31923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":31923},{"type":"v1","addr":"172.21.15.153:6815","nonce":31923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":31923},{"type":"v1","addr":"172.21.15.153:6813","nonce":31923}]},"public_addr":"172.21.15.153:6809/31923","cluster_addr":"172.21.15.153:6811/31923","heartbeat_back_addr":"172.21.15.153:6815/31923","heartbeat_front_addr":"172.21.15.153:6813/31923","state":["exists","up"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":31924},{"type":"v1","addr":"172.21.15.153:6817","nonce":31924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":31924},{"type":"v1","addr":"172.21.15.153:6819","nonce":31924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":31924},{"type":"v1","addr":"172.21.15.153:6823","nonce":31924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":31924},{"type":"v1","addr":"172.21.15.153:6821","nonce":31924}]},"public_addr":"172.21.15.153:6817/31924","cluster_addr":"172.21.15.153:6819/31924","heartbeat_back_addr":"172.21.15.153:6823/31924","heartbeat_front_addr":"172.21.15.153:6821/31924","state":["exists","up"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":31874},{"type":"v1","addr":"172.21.15.181:6809","nonce":31874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":31874},{"type":"v1","addr":"172.21.15.181:6811","nonce":31874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":31874},{"type":"v1","addr":"172.21.15.181:6815","nonce":31874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":31874},{"type":"v1","addr":"172.21.15.181:6813","nonce":31874}]},"public_addr":"172.21.15.181:6809/31874","cluster_addr":"172.21.15.181:6811/31874","heartbeat_back_addr":"172.21.15.181:6815/31874","heartbeat_front_addr":"172.21.15.181:6813/31874","state":["exists","up"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":31875},{"type":"v1","addr":"172.21.15.181:6801","nonce":31875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":31875},{"type":"v1","addr":"172.21.15.181:6803","nonce":31875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":31875},{"type":"v1","addr":"172.21.15.181:6807","nonce":31875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":31875},{"type":"v1","addr":"172.21.15.181:6805","nonce":31875}]},"public_addr":"172.21.15.181:6801/31875","cluster_addr":"172.21.15.181:6803/31875","heartbeat_back_addr":"172.21.15.181:6807/31875","heartbeat_front_addr":"172.21.15.181:6805/31875","state":["exists","up"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":31876},{"type":"v1","addr":"172.21.15.181:6817","nonce":31876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":31876},{"type":"v1","addr":"172.21.15.181:6819","nonce":31876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":31876},{"type":"v1","addr":"172.21.15.181:6823","nonce":31876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":31876},{"type":"v1","addr":"172.21.15.181:6821","nonce":31876}]},"public_addr":"172.21.15.181:6817/31876","cluster_addr":"172.21.15.181:6819/31876","heartbeat_back_addr":"172.21.15.181:6823/31876","heartbeat_front_addr":"172.21.15.181:6821/31876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b280386e0 msgr2=0x7f1b2803ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b280386e0 0x7f1b2803ab90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b44136c00 tx=0x7f1b38006f50 comp rx=0 tx=0).stop 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 10 monclient: shutdown 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 msgr2=0x7f1b440970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 0x7f1b440970f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000ea50 tx=0x7f1b400090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 shutdown_connections 2024-09-15T03:28:34.387 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b280386e0 0x7f1b2803ab90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.388 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b44136c50 0x7f1b44097b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.388 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b44131920 0x7f1b44097630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.388 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 --2- 172.21.15.153:0/4044653589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b4409ea60 0x7f1b440970f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.388 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 >> 172.21.15.153:0/4044653589 conn(0x7f1b440a50f0 msgr2=0x7f1b440a38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:34.388 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 shutdown_connections 2024-09-15T03:28:34.388 INFO:teuthology.misc.health.smithi153.stderr:2024-09-15T03:28:34.384+0000 7f1b4bf1d640 1 -- 172.21.15.153:0/4044653589 wait complete. 2024-09-15T03:28:34.395 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-15T03:28:34.396 INFO:tasks.ceph:Creating RBD pool 2024-09-15T03:28:34.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T03:28:34.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.486+0000 7f142859d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:34.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.486+0000 7f142859d640 10 monclient: build_initial_monmap 2024-09-15T03:28:34.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.486+0000 7f142859d640 10 monclient: monmap: 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:34.488334+0000 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:34.488334+0000 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:34.489 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:34.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 Processor -- start 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 -- start start 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient: init 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f142009eb00 0x7f142009eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): picked mon.noname-a con 0x7f142009eb00 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 0x7f1420068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): picked mon.noname-b con 0x7f142009d000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f142009d400 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142009cd90 con 0x7f142009d400 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142009fe10 con 0x7f142009eb00 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200a11f0 con 0x7f142009d000 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f142859d640 10 monclient(hunting): authenticate will time out at 712.127885s 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f1425b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 0x7f1420068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.487+0000 7f1426339640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f142009eb00 0x7f142009eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f142009eb00 0x7f142009eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56012/0 (socket says 172.21.15.153:56012) 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 0x7f1420068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57196/0 (socket says 172.21.15.153:57196) 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 -- 172.21.15.153:0/1956954268 learned_addr learned my addr 172.21.15.153:0/1956954268 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): get_auth_request con 0x7f142009eb00 auth_method 0 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426b3a640 1 --1- 172.21.15.153:0/1956954268 >> v1:172.21.15.153:6789/0 conn(0x7f142009d400 0x7f1420069220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53148/0 (socket says 172.21.15.153:53148) 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): get_auth_request con 0x7f142009d000 auth_method 0 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2024-09-15T03:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 -- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 msgr2=0x7f1420068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 --2- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 0x7f1420068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 -- 172.21.15.153:0/1956954268 >> v1:172.21.15.153:6789/0 conn(0x7f142009d400 legacy=0x7f1420069220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient: _finish_hunting 0 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 monclient: found mon.noname-a 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1426339640 1 -- 172.21.15.153:0/1956954268 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200a00c0 con 0x7f142009eb00 2024-09-15T03:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.488+0000 7f1425b38640 1 --2- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 0x7f1420068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1426339640 1 -- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009eb00 msgr2=0x7f142009eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1426339640 1 --2- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009eb00 0x7f142009eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f141c009910 tx=0x7f141c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac87e5a169e9bda server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 1 -- 172.21.15.153:0/1956954268 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c032440 con 0x7f142009eb00 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: dump: 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: _finish_auth 0 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 1 -- 172.21.15.153:0/1956954268 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c0325a0 con 0x7f142009eb00 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 5 monclient: authenticate success, global_id 4179 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 1 -- 172.21.15.153:0/1956954268 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c032940 con 0x7f142009eb00 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 4 set_mon_vals no callback set 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f1425337640 10 monclient: dump: 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.504 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 10 monclient: shutdown 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 1 -- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009eb00 msgr2=0x7f142009eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.491+0000 7f142859d640 1 --2- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009eb00 0x7f142009eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f141c009910 tx=0x7f141c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 -- 172.21.15.153:0/1956954268 shutdown_connections 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 --2- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d000 0x7f1420068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 --2- 172.21.15.153:0/1956954268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009eb00 0x7f142009eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 -- 172.21.15.153:0/1956954268 >> 172.21.15.153:0/1956954268 conn(0x7f1420067d30 msgr2=0x7f14200a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 -- 172.21.15.153:0/1956954268 shutdown_connections 2024-09-15T03:28:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 -- 172.21.15.153:0/1956954268 wait complete. 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 10 monclient: build_initial_monmap 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 10 monclient: monmap: 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:34.506 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 Processor -- start 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 1 -- start start 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.492+0000 7f142859d640 10 monclient: init 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 0x7f1420132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): picked mon.noname-a con 0x7f142009d000 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 0x7f1420140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): picked mon.noname-c con 0x7f142009d400 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142009eb00 0x7f1420141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): picked mon.noname-b con 0x7f142009eb00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:34.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1420120f00 con 0x7f142009d000 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201211e0 con 0x7f142009eb00 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1420120c20 con 0x7f142009d400 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f142859d640 10 monclient(hunting): authenticate will time out at 712.133418s 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 0x7f1420132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 0x7f1420140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 0x7f1420132af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56022/0 (socket says 172.21.15.153:56022) 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): get_auth_request con 0x7f142009d400 auth_method 0 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 1 -- 172.21.15.153:0/83278040 learned_addr learned my addr 172.21.15.153:0/83278040 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): get_auth_request con 0x7f142009d000 auth_method 0 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2024-09-15T03:28:34.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 -- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142009eb00 msgr2=0x7f1420141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426b3a640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142009eb00 0x7f1420141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142009eb00 0x7f1420141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 -- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 msgr2=0x7f1420132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 0x7f1420132af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient: _finish_hunting 0 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 monclient: found mon.noname-c 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1425b38640 1 -- 172.21.15.153:0/83278040 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1410009390 con 0x7f142009d400 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426b3a640 10 monclient: get_auth_request con 0x7f142009eb00 auth_method 0 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426339640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 0x7f1420132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:34.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.493+0000 7f1426b3a640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142009eb00 0x7f1420141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f1425b38640 1 -- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 msgr2=0x7f1420140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f1425b38640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 0x7f1420140ec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f141000ea80 tx=0x7f14100090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410009670 con 0x7f142009d400 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: dump: 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.511 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141000cc20 con 0x7f142009d400 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 5 monclient: authenticate success, global_id 4187 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410013470 con 0x7f142009d400 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f14167fc640 10 monclient: dump: 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:34.512 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 10 monclient: _renew_subs 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f1424b36640 4 set_mon_vals no callback set 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c009450 con 0x7f142009d400 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 10 monclient: _renew_subs 2024-09-15T03:28:34.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.496+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142011ff60 con 0x7f142009d400 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 4) v1 ==== 44954+0+0 (secure 0 0 0) 0x7f1410013860 con 0x7f142009d400 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f142859d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f142859d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f142859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142005ed70 con 0x7f142009d400 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f14167fc640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f14040386b0 0x7f140403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3603+0+0 (secure 0 0 0) 0x7f141004b610 con 0x7f142009d400 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f1426339640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f14040386b0 0x7f140403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.497+0000 7f1426339640 10 monclient: get_auth_request con 0x7f14040386b0 auth_method 0 2024-09-15T03:28:34.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.500+0000 7f1426339640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f14040386b0 0x7f140403ab60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f141c009210 tx=0x7f141c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:34.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.502+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f14100194a0 con 0x7f142009d400 2024-09-15T03:28:34.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.502+0000 7f14167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:34.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.502+0000 7f14167fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:34.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.657+0000 7f142859d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:28:34.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.657+0000 7f142859d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:28:34.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.657+0000 7f142859d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:34.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:34.657+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f1420066240 con 0x7f142009d400 2024-09-15T03:28:35.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.738+0000 7f14167fc640 1 -- 172.21.15.153:0/83278040 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f14100192c0 con 0x7f142009d400 2024-09-15T03:28:35.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.739+0000 7f14167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:28:35.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.739+0000 7f14167fc640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-15T03:28:35.741 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f14040386b0 msgr2=0x7f140403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f14040386b0 0x7f140403ab60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f141c009210 tx=0x7f141c032000 comp rx=0 tx=0).stop 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 10 monclient: shutdown 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 msgr2=0x7f1420140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 0x7f1420140ec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f141000ea80 tx=0x7f14100090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 shutdown_connections 2024-09-15T03:28:35.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f14040386b0 0x7f140403ab60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142009eb00 0x7f1420141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142009d400 0x7f1420140ec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 --2- 172.21.15.153:0/83278040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142009d000 0x7f1420132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 >> 172.21.15.153:0/83278040 conn(0x7f1420067d30 msgr2=0x7f1420133910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 shutdown_connections 2024-09-15T03:28:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.742+0000 7f142859d640 1 -- 172.21.15.153:0/83278040 wait complete. 2024-09-15T03:28:35.757 DEBUG:teuthology.orchestra.run.smithi153:> rbd --cluster ceph pool init rbd 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.818+0000 7f64ac0c7b80 10 monclient: get_monmap_and_config 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.818+0000 7f64ac0c7b80 10 monclient: build_initial_monmap 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.818+0000 7f64ac0c7b80 10 monclient: monmap: 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:35.820104+0000 2024-09-15T03:28:35.820 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:35.820104+0000 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:35.821 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:35.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 Processor -- start 2024-09-15T03:28:35.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 -- start start 2024-09-15T03:28:35.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient: init 2024-09-15T03:28:35.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient: _reopen_session rank -1 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): picked mon.noname-c-legacy con 0x55d548751bb0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 --2- >> v2:172.21.15.153:3300/0 conn(0x55d548751f80 0x55d548752350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): picked mon.noname-a con 0x55d548751f80 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 --2- >> v2:172.21.15.181:3300/0 conn(0x55d5487eb490 0x55d5487ed870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): picked mon.noname-c con 0x55d5487eb490 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55d5486730e0 con 0x55d548751bb0 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x55d5486ac260 con 0x55d548751f80 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x55d5486a7f40 con 0x55d5487eb490 2024-09-15T03:28:35.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): _renew_subs 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64ac0c7b80 10 monclient(hunting): authenticate will time out at 713.459632s 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x55d548751f80 0x55d548752350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x55d548751f80 0x55d548752350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49386/0 (socket says 172.21.15.153:49386) 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 1 -- 172.21.15.153:0/204071064 learned_addr learned my addr 172.21.15.153:0/204071064 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): get_auth_request con 0x55d548751f80 auth_method 0 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.819+0000 7f64a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a964c640 1 --2- 172.21.15.153:0/204071064 >> v2:172.21.15.181:3300/0 conn(0x55d5487eb490 0x55d5487ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:35.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 1 -- 172.21.15.153:0/204071064 >> v2:172.21.15.181:3300/0 conn(0x55d5487eb490 msgr2=0x55d5487ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 1 --2- 172.21.15.153:0/204071064 >> v2:172.21.15.181:3300/0 conn(0x55d5487eb490 0x55d5487ed870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 1 -- 172.21.15.153:0/204071064 >> v1:172.21.15.181:6789/0 conn(0x55d548751bb0 legacy=0x55d5487e6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 1 monclient: found mon.noname-a 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a3fff640 1 -- 172.21.15.153:0/204071064 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d54864c270 con 0x55d548751f80 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a964c640 10 monclient: get_auth_request con 0x55d5487eb490 auth_method 0 2024-09-15T03:28:35.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.820+0000 7f64a964c640 1 --2- 172.21.15.153:0/204071064 >> v2:172.21.15.181:3300/0 conn(0x55d5487eb490 0x55d5487ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.823+0000 7f64a3fff640 1 -- 172.21.15.153:0/204071064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751f80 msgr2=0x55d548752350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.823+0000 7f64a3fff640 1 --2- 172.21.15.153:0/204071064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751f80 0x55d548752350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x55d548735b60 tx=0x7f6490045e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77162f4bc69f525e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 1 -- 172.21.15.153:0/204071064 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649004d640 con 0x55d548751f80 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: dump: 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:35.830 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 1 -- 172.21.15.153:0/204071064 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649004d7a0 con 0x55d548751f80 2024-09-15T03:28:35.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 5 monclient: authenticate success, global_id 4191 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 10 monclient: get_monmap_and_config success 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 1 -- 172.21.15.153:0/204071064 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649004d930 con 0x55d548751f80 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 4 set_mon_vals no callback set 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64a37fe640 10 monclient: dump: 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.832 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 10 monclient: shutdown 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/204071064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751f80 msgr2=0x55d548752350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.824+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/204071064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751f80 0x55d548752350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x55d548735b60 tx=0x7f6490045e30 comp rx=0 tx=0).stop 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/204071064 shutdown_connections 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/204071064 >> v2:172.21.15.181:3300/0 conn(0x55d5487eb490 0x55d5487ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/204071064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751f80 0x55d548752350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/204071064 >> 172.21.15.153:0/204071064 conn(0x55d548735de0 msgr2=0x55d5487361e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/204071064 shutdown_connections 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/204071064 wait complete. 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 10 monclient: build_initial_monmap 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 10 monclient: monmap: 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:35.834 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.825+0000 7f64ac0c7b80 1 Processor -- start 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 -- start start 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient: init 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient: _reopen_session rank -1 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751bb0 0x55d5487ddcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): picked mon.noname-a con 0x55d548751bb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d548751f80 0x55d5487de230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): picked mon.noname-b con 0x55d548751f80 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d5487eb490 0x55d5487de770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:35.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): picked mon.noname-c con 0x55d5487eb490 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54866aba0 con 0x55d548751bb0 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54866a840 con 0x55d548751f80 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d54862d7a0 con 0x55d5487eb490 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): _renew_subs 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64ac0c7b80 10 monclient(hunting): authenticate will time out at 713.466585s 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751bb0 0x55d5487ddcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751bb0 0x55d5487ddcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49396/0 (socket says 172.21.15.153:49396) 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 1 -- 172.21.15.153:0/3287306368 learned_addr learned my addr 172.21.15.153:0/3287306368 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): get_auth_request con 0x55d548751bb0 auth_method 0 2024-09-15T03:28:35.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d5487eb490 0x55d5487de770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): get_auth_request con 0x55d5487eb490 auth_method 0 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a8e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:35.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a964c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a3fff640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d548751f80 0x55d5487de230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a3fff640 10 monclient(hunting): get_auth_request con 0x55d548751f80 auth_method 0 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.826+0000 7f64a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 1 -- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d548751f80 msgr2=0x55d5487de230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d548751f80 0x55d5487de230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 1 -- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751bb0 msgr2=0x55d5487ddcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751bb0 0x55d5487ddcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 1 monclient: found mon.noname-c 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:35.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a964c640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490005db0 con 0x55d5487eb490 2024-09-15T03:28:35.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a3fff640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d548751f80 0x55d5487de230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:35.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.827+0000 7f64a8e4b640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d548751bb0 0x55d5487ddcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:35.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a964c640 1 -- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d5487eb490 msgr2=0x55d5487de770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a964c640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d5487eb490 0x55d5487de770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f649c009480 tx=0x7f649c007040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c010830 con 0x55d5487eb490 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: dump: 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:35.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c00b050 con 0x55d5487eb490 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c010c00 con 0x55d5487eb490 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 5 monclient: authenticate success, global_id 4196 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a0ff9640 10 monclient: dump: 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 10 monclient: _renew_subs 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64a2ffd640 4 set_mon_vals no callback set 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d5486590e0 con 0x55d5487eb490 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 10 monclient: _renew_subs 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:35.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.829+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d54864c600 con 0x55d5487eb490 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.830+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 4) v1 ==== 44954+0+0 (secure 0 0 0) 0x7f649c019400 con 0x55d5487eb490 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.830+0000 7f64a0ff9640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6488036fe0 0x7f6488039490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.830+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4331+0+0 (secure 0 0 0) 0x7f649c04a650 con 0x55d5487eb490 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.831+0000 7f64ac0c7b80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.831+0000 7f64ac0c7b80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.831+0000 7f64ac0c7b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.831+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55d54864c060 con 0x55d5487eb490 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.831+0000 7f64a8e4b640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6488036fe0 0x7f6488039490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.831+0000 7f64a8e4b640 10 monclient: get_auth_request con 0x7f6488036fe0 auth_method 0 2024-09-15T03:28:35.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:35.833+0000 7f64a8e4b640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6488036fe0 0x7f6488039490 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f649800ba00 tx=0x7f64980028b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.746+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f649c04a2f0 con 0x55d5487eb490 2024-09-15T03:28:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.747+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (secure 0 0 0) 0x7f649c0047c0 con 0x55d5487eb490 2024-09-15T03:28:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.747+0000 7f64a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:28:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.747+0000 7f64a0ff9640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-15T03:28:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.747+0000 7f64ac0c7b80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.747+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55d54866a630 con 0x55d5487eb490 2024-09-15T03:28:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f649c0054e0 con 0x55d5487eb490 2024-09-15T03:28:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64a0ff9640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-15T03:28:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64a2ffd640 10 monclient: _renew_subs 2024-09-15T03:28:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64a2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:36.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64a2ffd640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=14}) v3 -- 0x7f6480002990 con 0x55d5487eb490 2024-09-15T03:28:36.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 9 ==== osd_map(14..14 src has 1..14) v4 ==== 1020+0+0 (secure 0 0 0) 0x7f649c02d030 con 0x55d5487eb490 2024-09-15T03:28:36.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x55d5487e7970 0x55d5487fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:36.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.748+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x55d5487fe090 con 0x55d5487e7970 2024-09-15T03:28:36.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.749+0000 7f64a3fff640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x55d5487e7970 0x55d5487fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:36.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.749+0000 7f64a3fff640 10 monclient: get_auth_request con 0x55d5487e7970 auth_method 0 2024-09-15T03:28:36.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.754+0000 7f64a3fff640 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x55d5487e7970 0x55d5487fdb50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:36.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.759+0000 7f64a3fff640 1 -- 172.21.15.153:0/3287306368 <== osd.5 v2:172.21.15.181:6816/31876 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f649004d910 con 0x55d5487e7970 2024-09-15T03:28:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.761+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x55d54884b130 con 0x55d5487e7970 2024-09-15T03:28:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.761+0000 7f64a3fff640 1 -- 172.21.15.153:0/3287306368 <== osd.5 v2:172.21.15.181:6816/31876 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f649004d910 con 0x55d5487e7970 2024-09-15T03:28:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.761+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x55d54884a210 con 0x55d5487e7970 2024-09-15T03:28:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.761+0000 7f64a3fff640 1 -- 172.21.15.153:0/3287306368 <== osd.5 v2:172.21.15.181:6816/31876 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f649004d910 con 0x55d5487e7970 2024-09-15T03:28:36.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.761+0000 7f64a27fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:36.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:36.761+0000 7f64a27fc640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f6484001ee0 con 0x55d5487eb490 2024-09-15T03:28:37.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.746+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 10 ==== pool_op_reply(tid 4 (0) Success v15) v1 ==== 55+0+0 (secure 0 0 0) 0x7f649c021c50 con 0x55d5487eb490 2024-09-15T03:28:37.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.746+0000 7f64a0ff9640 10 monclient: _renew_subs 2024-09-15T03:28:37.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.746+0000 7f64a0ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:37.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.746+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=15}) v3 -- 0x7f648803d6b0 con 0x55d5487eb490 2024-09-15T03:28:37.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.746+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 11 ==== osd_map(15..15 src has 1..15) v4 ==== 618+0+0 (secure 0 0 0) 0x7f649c020cb0 con 0x55d5487eb490 2024-09-15T03:28:37.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.746+0000 7f64a2ffd640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f64800053f0 con 0x55d5487e7970 2024-09-15T03:28:37.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.749+0000 7f64a3fff640 1 -- 172.21.15.153:0/3287306368 <== osd.5 v2:172.21.15.181:6816/31876 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f649004d910 con 0x55d5487e7970 2024-09-15T03:28:37.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.749+0000 7f64a27fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:37.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:37.749+0000 7f64a27fc640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v15) v4 -- 0x7f64840033b0 con 0x55d5487eb490 2024-09-15T03:28:38.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.757+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 12 ==== pool_op_reply(tid 6 (0) Success v16) v1 ==== 43+0+0 (secure 0 0 0) 0x7f649c021df0 con 0x55d5487eb490 2024-09-15T03:28:38.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.758+0000 7f64a0ff9640 10 monclient: _renew_subs 2024-09-15T03:28:38.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.758+0000 7f64a0ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:38.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.758+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=16}) v3 -- 0x7f648803d900 con 0x55d5487eb490 2024-09-15T03:28:38.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.758+0000 7f64a0ff9640 1 -- 172.21.15.153:0/3287306368 <== mon.2 v2:172.21.15.153:3301/0 13 ==== osd_map(16..16 src has 1..16) v4 ==== 646+0+0 (secure 0 0 0) 0x7f649c01da70 con 0x55d5487eb490 2024-09-15T03:28:38.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.758+0000 7f64a2ffd640 1 -- 172.21.15.153:0/3287306368 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f6480005820 con 0x55d5487e7970 2024-09-15T03:28:38.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64a3fff640 1 -- 172.21.15.153:0/3287306368 <== osd.5 v2:172.21.15.181:6816/31876 5 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f649004d910 con 0x55d5487e7970 2024-09-15T03:28:38.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x55d5487e7970 msgr2=0x55d5487fdb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:38.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x55d5487e7970 0x55d5487fdb50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6488036fe0 msgr2=0x7f6488039490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:38.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6488036fe0 0x7f6488039490 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f649800ba00 tx=0x7f64980028b0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 10 monclient: shutdown 2024-09-15T03:28:38.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:38.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d5487eb490 msgr2=0x55d5487de770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:38.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 --2- 172.21.15.153:0/3287306368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d5487eb490 0x55d5487de770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f649c009480 tx=0x7f649c007040 comp rx=0 tx=0).stop 2024-09-15T03:28:38.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64a964c640 1 -- 172.21.15.153:0/3287306368 reap_dead start 2024-09-15T03:28:38.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 shutdown_connections 2024-09-15T03:28:38.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 >> 172.21.15.153:0/3287306368 conn(0x55d548735de0 msgr2=0x55d5487e4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:38.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 shutdown_connections 2024-09-15T03:28:38.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.767+0000 7f64ac0c7b80 1 -- 172.21.15.153:0/3287306368 wait complete. 2024-09-15T03:28:38.774 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T03:28:38.775 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T03:28:38.775 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T03:28:38.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T03:28:38.776 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.877+0000 7fc390217640 10 monclient: get_monmap_and_config 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.877+0000 7fc390217640 10 monclient: build_initial_monmap 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.877+0000 7fc390217640 10 monclient: monmap: 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:38.879013+0000 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:38.879013+0000 2024-09-15T03:28:38.879 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:38.880 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:38.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.877+0000 7fc390217640 1 Processor -- start 2024-09-15T03:28:38.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 1 -- start start 2024-09-15T03:28:38.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient: init 2024-09-15T03:28:38.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:38.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:28:38.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc38806eb80 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc388068a20 0x7fc38809c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): picked mon.noname-a con 0x7fc388068a20 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 0x7fc388138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): picked mon.noname-b con 0x7fc38809ca80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388069b80 con 0x7fc38806eb80 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc388099f90 con 0x7fc388068a20 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc388069e00 con 0x7fc38809ca80 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc390217640 10 monclient(hunting): authenticate will time out at 716.518596s 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc388068a20 0x7fc38809c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:38.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 0x7fc388138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc388068a20 0x7fc38809c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49404/0 (socket says 172.21.15.153:49404) 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 1 -- 172.21.15.153:0/694827067 learned_addr learned my addr 172.21.15.153:0/694827067 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 0x7fc388138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44688/0 (socket says 172.21.15.153:44688) 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): get_auth_request con 0x7fc388068a20 auth_method 0 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): get_auth_request con 0x7fc38809ca80 auth_method 0 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38d7b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.878+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 1 -- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 msgr2=0x7fc388138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 1 --2- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 0x7fc388138ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 1 -- 172.21.15.153:0/694827067 >> v1:172.21.15.181:6789/0 conn(0x7fc38806eb80 legacy=0x7fc38806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 10 monclient: _finish_hunting 0 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 1 monclient: found mon.noname-a 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:38.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:38.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38d7b2640 1 -- 172.21.15.153:0/694827067 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38806f660 con 0x7fc388068a20 2024-09-15T03:28:38.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.879+0000 7fc38e7b4640 1 --2- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 0x7fc388138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.881+0000 7fc38d7b2640 1 -- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc388068a20 msgr2=0x7fc38809c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.881+0000 7fc38d7b2640 1 --2- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc388068a20 0x7fc38809c540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009f20 tx=0x7fc37c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe72aa48ba4c6b3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 1 -- 172.21.15.153:0/694827067 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c007610 con 0x7fc388068a20 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: dump: 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.891 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: _finish_auth 0 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 1 -- 172.21.15.153:0/694827067 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c02d050 con 0x7fc388068a20 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 5 monclient: authenticate success, global_id 4200 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:38.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 1 -- 172.21.15.153:0/694827067 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c0079e0 con 0x7fc388068a20 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 4 set_mon_vals no callback set 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc38cfb1640 10 monclient: dump: 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:38.893 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 10 monclient: shutdown 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 -- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc388068a20 msgr2=0x7fc38809c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 --2- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc388068a20 0x7fc38809c540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009f20 tx=0x7fc37c009840 comp rx=0 tx=0).stop 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 -- 172.21.15.153:0/694827067 shutdown_connections 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 --2- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc38809ca80 0x7fc388138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 --2- 172.21.15.153:0/694827067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc388068a20 0x7fc38809c540 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 -- 172.21.15.153:0/694827067 >> 172.21.15.153:0/694827067 conn(0x7fc388067cd0 msgr2=0x7fc3880a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 -- 172.21.15.153:0/694827067 shutdown_connections 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.882+0000 7fc390217640 1 -- 172.21.15.153:0/694827067 wait complete. 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.883+0000 7fc390217640 10 monclient: build_initial_monmap 2024-09-15T03:28:38.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.883+0000 7fc390217640 10 monclient: monmap: 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.883+0000 7fc390217640 1 Processor -- start 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.883+0000 7fc390217640 1 -- start start 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.883+0000 7fc390217640 10 monclient: init 2024-09-15T03:28:38.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.883+0000 7fc390217640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc388068a20 0x7fc38811e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): picked mon.noname-c con 0x7fc388068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38806eb80 0x7fc38811ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): picked mon.noname-b con 0x7fc38806eb80 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 0x7fc38811efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): picked mon.noname-a con 0x7fc38809ca80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc388120100 con 0x7fc38809ca80 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38809cf40 con 0x7fc38806eb80 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38809b710 con 0x7fc388068a20 2024-09-15T03:28:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc390217640 10 monclient(hunting): authenticate will time out at 716.524119s 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 0x7fc38811efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc388068a20 0x7fc38811e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 0x7fc38811efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49414/0 (socket says 172.21.15.153:49414) 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 -- 172.21.15.153:0/2736563577 learned_addr learned my addr 172.21.15.153:0/2736563577 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): get_auth_request con 0x7fc38809ca80 auth_method 0 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): get_auth_request con 0x7fc388068a20 auth_method 0 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:38.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38d7b2640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38806eb80 0x7fc38811ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38dfb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38d7b2640 10 monclient(hunting): get_auth_request con 0x7fc38806eb80 auth_method 0 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38d7b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38d7b2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38d7b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient(hunting): handle_auth_done global_id 4203 payload 824 2024-09-15T03:28:38.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 -- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc388068a20 msgr2=0x7fc38811e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc388068a20 0x7fc38811e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 -- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38806eb80 msgr2=0x7fc38811ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38806eb80 0x7fc38811ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient: _finish_hunting 0 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 monclient: found mon.noname-a 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38e7b4640 1 -- 172.21.15.153:0/2736563577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc380009390 con 0x7fc38809ca80 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.884+0000 7fc38d7b2640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38806eb80 0x7fc38811ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc38e7b4640 1 -- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 msgr2=0x7fc38811efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc38e7b4640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 0x7fc38811efb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc38000ea30 tx=0x7fc3800090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:38.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380009690 con 0x7fc38809ca80 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: dump: 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38000cc20 con 0x7fc38809ca80 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc390217640 5 monclient: authenticate success, global_id 4203 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc390217640 10 monclient: _renew_subs 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc390217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c009ae0 con 0x7fc38809ca80 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380013470 con 0x7fc38809ca80 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc385ffb640 10 monclient: dump: 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.887+0000 7fc390217640 10 monclient: _renew_subs 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.887+0000 7fc390217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.886+0000 7fc387fff640 4 set_mon_vals no callback set 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.887+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc388121ba0 con 0x7fc38809ca80 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc380013820 con 0x7fc38809ca80 2024-09-15T03:28:38.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc390217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc390217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc390217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38804bf60 con 0x7fc38809ca80 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc385ffb640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc374038740 0x7fc37403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc38004bb40 con 0x7fc38809ca80 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc38dfb3640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc374038740 0x7fc37403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.888+0000 7fc38dfb3640 10 monclient: get_auth_request con 0x7fc374038740 auth_method 0 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.891+0000 7fc38dfb3640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc374038740 0x7fc37403abf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc378007fd0 tx=0x7fc378007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.893+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc380022ad0 con 0x7fc38809ca80 2024-09-15T03:28:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.893+0000 7fc385ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:38.893+0000 7fc385ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:39.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.089+0000 7fc390217640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:28:39.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.089+0000 7fc390217640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:28:39.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.089+0000 7fc390217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.089+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc388121d30 con 0x7fc38809ca80 2024-09-15T03:28:39.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.091+0000 7fc385ffb640 1 -- 172.21.15.153:0/2736563577 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+87662 (secure 0 0 0) 0x7fc3800222f0 con 0x7fc38809ca80 2024-09-15T03:28:39.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.091+0000 7fc385ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:28:39.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.091+0000 7fc385ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:39.094 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:39.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc374038740 msgr2=0x7fc37403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc374038740 0x7fc37403abf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc378007fd0 tx=0x7fc378007e10 comp rx=0 tx=0).stop 2024-09-15T03:28:39.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 10 monclient: shutdown 2024-09-15T03:28:39.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 msgr2=0x7fc38811efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 0x7fc38811efb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc38000ea30 tx=0x7fc3800090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 shutdown_connections 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc374038740 0x7fc37403abf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc38809ca80 0x7fc38811efb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc38806eb80 0x7fc38811ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 --2- 172.21.15.153:0/2736563577 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc388068a20 0x7fc38811e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 >> 172.21.15.153:0/2736563577 conn(0x7fc388067cd0 msgr2=0x7fc3880a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 shutdown_connections 2024-09-15T03:28:39.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.093+0000 7fc390217640 1 -- 172.21.15.153:0/2736563577 wait complete. 2024-09-15T03:28:39.105 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":31449},{"type":"v1","addr":"172.21.15.181:6825","nonce":31449}]},"active_addr":"172.21.15.181:6825/31449","active_change":"2024-09-15T03:28:27.837938+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":613390639}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":3885058678}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":3623617885}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":14255229}]}]} 2024-09-15T03:28:39.107 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T03:28:39.107 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:28:39.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:28:39.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.197+0000 7f8791f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.197+0000 7f8791f9d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.197+0000 7f8791f9d640 10 monclient: monmap: 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.199410+0000 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.199410+0000 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.200 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.201 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.201 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.201 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.201 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.201 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.198+0000 7f8791f9d640 1 Processor -- start 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.198+0000 7f8791f9d640 1 -- start start 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.198+0000 7f8791f9d640 10 monclient: init 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f878c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f878c09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f878c137c30 0x7f878c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f878c137c30 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c131cc0 con 0x7f878c09ea50 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c09f530 con 0x7f878c1318f0 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c09fbf0 con 0x7f878c137c30 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8791f9d640 10 monclient(hunting): authenticate will time out at 716.839181s 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8783fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f878c09ea50 0x7f878c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48296/0 (socket says 172.21.15.153:48296) 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f8783fff640 1 -- 172.21.15.153:0/2013520436 learned_addr learned my addr 172.21.15.153:0/2013520436 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:39.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 1 --2- 172.21.15.153:0/2013520436 >> v2:172.21.15.181:3300/0 conn(0x7f878c137c30 0x7f878c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): get_auth_request con 0x7f878c137c30 auth_method 0 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 1 -- 172.21.15.153:0/2013520436 >> v1:172.21.15.181:6789/0 conn(0x7f878c1318f0 legacy=0x7f878c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 1 -- 172.21.15.153:0/2013520436 >> v1:172.21.15.153:6789/0 conn(0x7f878c09ea50 legacy=0x7f878c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 1 monclient: found mon.noname-c 2024-09-15T03:28:39.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:39.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.199+0000 7f878bfff640 1 -- 172.21.15.153:0/2013520436 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c09fe30 con 0x7f878c137c30 2024-09-15T03:28:39.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.202+0000 7f878bfff640 1 -- 172.21.15.153:0/2013520436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c137c30 msgr2=0x7f878c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.202+0000 7f878bfff640 1 --2- 172.21.15.153:0/2013520436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c137c30 0x7f878c13a010 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8784009e60 tx=0x7f8784009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7601b540fc7a1c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 1 -- 172.21.15.153:0/2013520436 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784007630 con 0x7f878c137c30 2024-09-15T03:28:39.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: dump: 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.210 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f8791f9d640 5 monclient: authenticate success, global_id 4213 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f8791f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 1 -- 172.21.15.153:0/2013520436 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878402d050 con 0x7f878c137c30 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f8791f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 1 -- 172.21.15.153:0/2013520436 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87840079c0 con 0x7f878c137c30 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f8791f9d640 4 set_mon_vals no callback set 2024-09-15T03:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.203+0000 7f878affd640 10 monclient: dump: 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 10 monclient: shutdown 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 -- 172.21.15.153:0/2013520436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c137c30 msgr2=0x7f878c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 --2- 172.21.15.153:0/2013520436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c137c30 0x7f878c13a010 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8784009e60 tx=0x7f8784009780 comp rx=0 tx=0).stop 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 -- 172.21.15.153:0/2013520436 shutdown_connections 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 --2- 172.21.15.153:0/2013520436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c137c30 0x7f878c13a010 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 -- 172.21.15.153:0/2013520436 >> 172.21.15.153:0/2013520436 conn(0x7f878c067110 msgr2=0x7f878c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 -- 172.21.15.153:0/2013520436 shutdown_connections 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 -- 172.21.15.153:0/2013520436 wait complete. 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 10 monclient: monmap: 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.213 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 Processor -- start 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 1 -- start start 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.204+0000 7f8791f9d640 10 monclient: init 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 0x7f878c1372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f878c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c1318f0 0x7f878c132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f878c1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f878c137c30 0x7f878c132c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f878c137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c135a90 con 0x7f878c09ea50 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c133e40 con 0x7f878c1318f0 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c09d790 con 0x7f878c137c30 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8791f9d640 10 monclient(hunting): authenticate will time out at 716.845338s 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 0x7f878c1372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 0x7f878c1372a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49430/0 (socket says 172.21.15.153:49430) 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 -- 172.21.15.153:0/4217793299 learned_addr learned my addr 172.21.15.153:0/4217793299 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): get_auth_request con 0x7f878c09ea50 auth_method 0 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f878c137c30 0x7f878c132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): get_auth_request con 0x7f878c137c30 auth_method 0 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8783fff640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c1318f0 0x7f878c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient(hunting): handle_auth_done global_id 4206 payload 824 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 -- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f878c137c30 msgr2=0x7f878c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f878c137c30 0x7f878c132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 -- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c1318f0 msgr2=0x7f878c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c1318f0 0x7f878c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f878b7fe640 1 -- 172.21.15.153:0/4217793299 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784009a20 con 0x7f878c09ea50 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8783fff640 10 monclient: get_auth_request con 0x7f878c1318f0 auth_method 0 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.205+0000 7f8783fff640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c1318f0 0x7f878c132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:39.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.207+0000 7f878b7fe640 1 -- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 msgr2=0x7f878c1372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.207+0000 7f878b7fe640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 0x7f878c1372a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f877c00c610 tx=0x7f877c007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c012840 con 0x7f878c09ea50 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: dump: 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c0129e0 con 0x7f878c09ea50 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 5 monclient: authenticate success, global_id 4206 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 10 monclient: _renew_subs 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c09bf10 con 0x7f878c09ea50 2024-09-15T03:28:39.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c012da0 con 0x7f878c09ea50 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 10 monclient: dump: 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.222 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f878a7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 10 monclient: _renew_subs 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c1367f0 con 0x7f878c09ea50 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.208+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f877c011590 con 0x7f878c09ea50 2024-09-15T03:28:39.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f8791f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f8791f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f8791f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c04bf60 con 0x7f878c09ea50 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f87837fe640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f876c038740 0x7f876c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f877c04b180 con 0x7f878c09ea50 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f8783fff640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f876c038740 0x7f876c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.209+0000 7f8783fff640 10 monclient: get_auth_request con 0x7f876c038740 auth_method 0 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.211+0000 7f8783fff640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f876c038740 0x7f876c03abf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f878c0a4fa0 tx=0x7f8774009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.213+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f877c00e470 con 0x7f878c09ea50 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.213+0000 7f87837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:39.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.213+0000 7f87837fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:39.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f8791f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:39.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f8791f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:39.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f8791f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f878c0638d0 con 0x7f878c09ea50 2024-09-15T03:28:39.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f87837fe640 1 -- 172.21.15.153:0/4217793299 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11452 (secure 0 0 0) 0x7f877c00e290 con 0x7f878c09ea50 2024-09-15T03:28:39.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f87837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:39.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.366+0000 7f87837fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:39.369 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:39.369 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":16,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:28:38.751454+0000","last_up_change":"2024-09-15T03:28:32.859989+0000","last_in_change":"2024-09-15T03:28:26.522990+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:28:33.881566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:28:34.684266+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":31922},{"type":"v1","addr":"172.21.15.153:6801","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":31922},{"type":"v1","addr":"172.21.15.153:6803","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":31922},{"type":"v1","addr":"172.21.15.153:6807","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":31922},{"type":"v1","addr":"172.21.15.153:6805","nonce":31922}]},"public_addr":"172.21.15.153:6801/31922","cluster_addr":"172.21.15.153:6803/31922","heartbeat_back_addr":"172.21.15.153:6807/31922","heartbeat_front_addr":"172.21.15.153:6805/31922","state":["exists","up"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":31923},{"type":"v1","addr":"172.21.15.153:6809","nonce":31923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":31923},{"type":"v1","addr":"172.21.15.153:6811","nonce":31923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":31923},{"type":"v1","addr":"172.21.15.153:6815","nonce":31923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":31923},{"type":"v1","addr":"172.21.15.153:6813","nonce":31923}]},"public_addr":"172.21.15.153:6809/31923","cluster_addr":"172.21.15.153:6811/31923","heartbeat_back_addr":"172.21.15.153:6815/31923","heartbeat_front_addr":"172.21.15.153:6813/31923","state":["exists","up"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":31924},{"type":"v1","addr":"172.21.15.153:6817","nonce":31924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":31924},{"type":"v1","addr":"172.21.15.153:6819","nonce":31924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":31924},{"type":"v1","addr":"172.21.15.153:6823","nonce":31924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":31924},{"type":"v1","addr":"172.21.15.153:6821","nonce":31924}]},"public_addr":"172.21.15.153:6817/31924","cluster_addr":"172.21.15.153:6819/31924","heartbeat_back_addr":"172.21.15.153:6823/31924","heartbeat_front_addr":"172.21.15.153:6821/31924","state":["exists","up"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":31874},{"type":"v1","addr":"172.21.15.181:6809","nonce":31874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":31874},{"type":"v1","addr":"172.21.15.181:6811","nonce":31874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":31874},{"type":"v1","addr":"172.21.15.181:6815","nonce":31874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":31874},{"type":"v1","addr":"172.21.15.181:6813","nonce":31874}]},"public_addr":"172.21.15.181:6809/31874","cluster_addr":"172.21.15.181:6811/31874","heartbeat_back_addr":"172.21.15.181:6815/31874","heartbeat_front_addr":"172.21.15.181:6813/31874","state":["exists","up"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":31875},{"type":"v1","addr":"172.21.15.181:6801","nonce":31875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":31875},{"type":"v1","addr":"172.21.15.181:6803","nonce":31875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":31875},{"type":"v1","addr":"172.21.15.181:6807","nonce":31875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":31875},{"type":"v1","addr":"172.21.15.181:6805","nonce":31875}]},"public_addr":"172.21.15.181:6801/31875","cluster_addr":"172.21.15.181:6803/31875","heartbeat_back_addr":"172.21.15.181:6807/31875","heartbeat_front_addr":"172.21.15.181:6805/31875","state":["exists","up"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":31876},{"type":"v1","addr":"172.21.15.181:6817","nonce":31876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":31876},{"type":"v1","addr":"172.21.15.181:6819","nonce":31876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":31876},{"type":"v1","addr":"172.21.15.181:6823","nonce":31876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":31876},{"type":"v1","addr":"172.21.15.181:6821","nonce":31876}]},"public_addr":"172.21.15.181:6817/31876","cluster_addr":"172.21.15.181:6819/31876","heartbeat_back_addr":"172.21.15.181:6823/31876","heartbeat_front_addr":"172.21.15.181:6821/31876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.685491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.637606+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.740014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.612976+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.627273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.624592+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:28:39.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f876c038740 msgr2=0x7f876c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f876c038740 0x7f876c03abf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f878c0a4fa0 tx=0x7f8774009040 comp rx=0 tx=0).stop 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 10 monclient: shutdown 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 msgr2=0x7f878c1372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 0x7f878c1372a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f877c00c610 tx=0x7f877c007ed0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 shutdown_connections 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f876c038740 0x7f876c03abf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f878c137c30 0x7f878c132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f878c1318f0 0x7f878c132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 --2- 172.21.15.153:0/4217793299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f878c09ea50 0x7f878c1372a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 >> 172.21.15.153:0/4217793299 conn(0x7f878c067110 msgr2=0x7f878c0a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 shutdown_connections 2024-09-15T03:28:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.368+0000 7f8791f9d640 1 -- 172.21.15.153:0/4217793299 wait complete. 2024-09-15T03:28:39.381 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:28:39.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.473+0000 7fbfb869f640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.473+0000 7fbfb869f640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.473+0000 7fbfb869f640 10 monclient: monmap: 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.474824+0000 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.474824+0000 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.476 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.473+0000 7fbfb869f640 1 Processor -- start 2024-09-15T03:28:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.473+0000 7fbfb869f640 1 -- start start 2024-09-15T03:28:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.473+0000 7fbfb869f640 10 monclient: init 2024-09-15T03:28:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfb01318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 0x7fbfb009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbfb0136c20 0x7fbfb0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb0136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb0132640 con 0x7fbfb01318f0 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb0131d00 con 0x7fbfb0136c20 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb009fbf0 con 0x7fbfb009ea50 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb869f640 10 monclient(hunting): authenticate will time out at 717.114459s 2024-09-15T03:28:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 0x7fbfb009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbfb0136c20 0x7fbfb0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 0x7fbfb009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44718/0 (socket says 172.21.15.153:44718) 2024-09-15T03:28:39.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3357911807 learned_addr learned my addr 172.21.15.153:0/3357911807 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): get_auth_request con 0x7fbfb009ea50 auth_method 0 2024-09-15T03:28:39.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): get_auth_request con 0x7fbfb0136c20 auth_method 0 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3357911807 >> v2:172.21.15.153:3300/0 conn(0x7fbfb0136c20 msgr2=0x7fbfb0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 --2- 172.21.15.153:0/3357911807 >> v2:172.21.15.153:3300/0 conn(0x7fbfb0136c20 0x7fbfb0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3357911807 >> v1:172.21.15.181:6789/0 conn(0x7fbfb01318f0 legacy=0x7fbfb0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 monclient: found mon.noname-b 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3357911807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb009fe30 con 0x7fbfb009ea50 2024-09-15T03:28:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.474+0000 7fbfb6c3c640 1 --2- 172.21.15.153:0/3357911807 >> v2:172.21.15.153:3300/0 conn(0x7fbfb0136c20 0x7fbfb0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:39.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3357911807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 msgr2=0x7fbfb009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5c3a640 1 --2- 172.21.15.153:0/3357911807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 0x7fbfb009ee20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009910 tx=0x7fbfa00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb7a2747936cb4d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 1 -- 172.21.15.153:0/3357911807 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0032440 con 0x7fbfb009ea50 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: dump: 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 1 -- 172.21.15.153:0/3357911807 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa00325a0 con 0x7fbfb009ea50 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb869f640 5 monclient: authenticate success, global_id 4211 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb869f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb869f640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 1 -- 172.21.15.153:0/3357911807 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0032980 con 0x7fbfb009ea50 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb869f640 4 set_mon_vals no callback set 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.477+0000 7fbfb5439640 10 monclient: dump: 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 10 monclient: shutdown 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 -- 172.21.15.153:0/3357911807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 msgr2=0x7fbfb009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3357911807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 0x7fbfb009ee20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009910 tx=0x7fbfa00091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 -- 172.21.15.153:0/3357911807 shutdown_connections 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3357911807 >> v2:172.21.15.153:3300/0 conn(0x7fbfb0136c20 0x7fbfb0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3357911807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb009ea50 0x7fbfb009ee20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 -- 172.21.15.153:0/3357911807 >> 172.21.15.153:0/3357911807 conn(0x7fbfb0067110 msgr2=0x7fbfb0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 -- 172.21.15.153:0/3357911807 shutdown_connections 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 1 -- 172.21.15.153:0/3357911807 wait complete. 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.478+0000 7fbfb869f640 10 monclient: monmap: 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.492 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 Processor -- start 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 -- start start 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient: init 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb009ea50 0x7fbfb0130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 0x7fbfb0099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb0136c20 0x7fbfb0099d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb011ca90 con 0x7fbfb01318f0 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb009b8e0 con 0x7fbfb009ea50 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb009bf10 con 0x7fbfb0136c20 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb869f640 10 monclient(hunting): authenticate will time out at 717.119875s 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb6c3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb0136c20 0x7fbfb0099d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb5c3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 0x7fbfb0099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb5c3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 0x7fbfb0099810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49444/0 (socket says 172.21.15.153:49444) 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3358503935 learned_addr learned my addr 172.21.15.153:0/3358503935 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.479+0000 7fbfb6c3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb0136c20 0x7fbfb0099d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44722/0 (socket says 172.21.15.153:44722) 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): get_auth_request con 0x7fbfb01318f0 auth_method 0 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): get_auth_request con 0x7fbfb0136c20 auth_method 0 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb6c3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient(hunting): handle_auth_done global_id 4212 payload 824 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb0136c20 msgr2=0x7fbfb0099d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb0136c20 0x7fbfb0099d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb009ea50 msgr2=0x7fbfb0130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb643b640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb009ea50 0x7fbfb0130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb009ea50 0x7fbfb0130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3358503935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8009a20 con 0x7fbfb01318f0 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb643b640 10 monclient: get_auth_request con 0x7fbfb009ea50 auth_method 0 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.480+0000 7fbfb643b640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb009ea50 0x7fbfb0130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb5c3a640 1 -- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 msgr2=0x7fbfb0099810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb5c3a640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 0x7fbfb0099810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0035040 tx=0x7fbfa0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0000c40 con 0x7fbfb01318f0 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: dump: 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 5 monclient: authenticate success, global_id 4212 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa0031c70 con 0x7fbfb01318f0 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 10 monclient: _renew_subs 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa003d440 con 0x7fbfb01318f0 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa0009450 con 0x7fbfb01318f0 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb4c38640 4 set_mon_vals no callback set 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfa67fc640 10 monclient: dump: 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.499 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 10 monclient: _renew_subs 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.482+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb0141eb0 con 0x7fbfb01318f0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbfa0039040 con 0x7fbfb01318f0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfb869f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfb869f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfb869f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb011cc20 con 0x7fbfb01318f0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfa67fc640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf94038740 0x7fbf9403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.483+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fbfa006b590 con 0x7fbfb01318f0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.484+0000 7fbfb643b640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf94038740 0x7fbf9403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.484+0000 7fbfb643b640 10 monclient: get_auth_request con 0x7fbf94038740 auth_method 0 2024-09-15T03:28:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.486+0000 7fbfb643b640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf94038740 0x7fbf9403abf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbfac0027e0 tx=0x7fbfac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.488+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbfa002f2a0 con 0x7fbfb01318f0 2024-09-15T03:28:39.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.488+0000 7fbfa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:39.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.488+0000 7fbfa67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:39.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.642+0000 7fbfb869f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:39.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.642+0000 7fbfb869f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:39.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.642+0000 7fbfb869f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.642+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb004bf60 con 0x7fbfb01318f0 2024-09-15T03:28:39.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.642+0000 7fbfa67fc640 1 -- 172.21.15.153:0/3358503935 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11452 (secure 0 0 0) 0x7fbfa0037070 con 0x7fbfb01318f0 2024-09-15T03:28:39.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.643+0000 7fbfa67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:39.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.643+0000 7fbfa67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:39.645 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":16,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:28:38.751454+0000","last_up_change":"2024-09-15T03:28:32.859989+0000","last_in_change":"2024-09-15T03:28:26.522990+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:28:33.881566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:28:34.684266+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":31922},{"type":"v1","addr":"172.21.15.153:6801","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":31922},{"type":"v1","addr":"172.21.15.153:6803","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":31922},{"type":"v1","addr":"172.21.15.153:6807","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":31922},{"type":"v1","addr":"172.21.15.153:6805","nonce":31922}]},"public_addr":"172.21.15.153:6801/31922","cluster_addr":"172.21.15.153:6803/31922","heartbeat_back_addr":"172.21.15.153:6807/31922","heartbeat_front_addr":"172.21.15.153:6805/31922","state":["exists","up"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":31923},{"type":"v1","addr":"172.21.15.153:6809","nonce":31923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":31923},{"type":"v1","addr":"172.21.15.153:6811","nonce":31923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":31923},{"type":"v1","addr":"172.21.15.153:6815","nonce":31923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":31923},{"type":"v1","addr":"172.21.15.153:6813","nonce":31923}]},"public_addr":"172.21.15.153:6809/31923","cluster_addr":"172.21.15.153:6811/31923","heartbeat_back_addr":"172.21.15.153:6815/31923","heartbeat_front_addr":"172.21.15.153:6813/31923","state":["exists","up"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":31924},{"type":"v1","addr":"172.21.15.153:6817","nonce":31924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":31924},{"type":"v1","addr":"172.21.15.153:6819","nonce":31924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":31924},{"type":"v1","addr":"172.21.15.153:6823","nonce":31924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":31924},{"type":"v1","addr":"172.21.15.153:6821","nonce":31924}]},"public_addr":"172.21.15.153:6817/31924","cluster_addr":"172.21.15.153:6819/31924","heartbeat_back_addr":"172.21.15.153:6823/31924","heartbeat_front_addr":"172.21.15.153:6821/31924","state":["exists","up"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":31874},{"type":"v1","addr":"172.21.15.181:6809","nonce":31874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":31874},{"type":"v1","addr":"172.21.15.181:6811","nonce":31874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":31874},{"type":"v1","addr":"172.21.15.181:6815","nonce":31874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":31874},{"type":"v1","addr":"172.21.15.181:6813","nonce":31874}]},"public_addr":"172.21.15.181:6809/31874","cluster_addr":"172.21.15.181:6811/31874","heartbeat_back_addr":"172.21.15.181:6815/31874","heartbeat_front_addr":"172.21.15.181:6813/31874","state":["exists","up"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":31875},{"type":"v1","addr":"172.21.15.181:6801","nonce":31875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":31875},{"type":"v1","addr":"172.21.15.181:6803","nonce":31875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":31875},{"type":"v1","addr":"172.21.15.181:6807","nonce":31875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":31875},{"type":"v1","addr":"172.21.15.181:6805","nonce":31875}]},"public_addr":"172.21.15.181:6801/31875","cluster_addr":"172.21.15.181:6803/31875","heartbeat_back_addr":"172.21.15.181:6807/31875","heartbeat_front_addr":"172.21.15.181:6805/31875","state":["exists","up"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":31876},{"type":"v1","addr":"172.21.15.181:6817","nonce":31876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":31876},{"type":"v1","addr":"172.21.15.181:6819","nonce":31876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":31876},{"type":"v1","addr":"172.21.15.181:6823","nonce":31876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":31876},{"type":"v1","addr":"172.21.15.181:6821","nonce":31876}]},"public_addr":"172.21.15.181:6817/31876","cluster_addr":"172.21.15.181:6819/31876","heartbeat_back_addr":"172.21.15.181:6823/31876","heartbeat_front_addr":"172.21.15.181:6821/31876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.685491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.637606+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.740014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.612976+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.627273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.624592+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:28:39.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.644+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf94038740 msgr2=0x7fbf9403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.644+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf94038740 0x7fbf9403abf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbfac0027e0 tx=0x7fbfac009040 comp rx=0 tx=0).stop 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 10 monclient: shutdown 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 msgr2=0x7fbfb0099810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 0x7fbfb0099810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0035040 tx=0x7fbfa0032000 comp rx=0 tx=0).stop 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 shutdown_connections 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf94038740 0x7fbf9403abf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbfb0136c20 0x7fbfb0099d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfb01318f0 0x7fbfb0099810 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 --2- 172.21.15.153:0/3358503935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb009ea50 0x7fbfb0130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 >> 172.21.15.153:0/3358503935 conn(0x7fbfb0067110 msgr2=0x7fbfb00a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 shutdown_connections 2024-09-15T03:28:39.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.645+0000 7fbfb869f640 1 -- 172.21.15.153:0/3358503935 wait complete. 2024-09-15T03:28:39.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:28:39.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:28:39.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:28:39.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:28:39.660 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:28:39.660 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:28:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.770+0000 7fc96a71d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.770+0000 7fc96a71d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.770+0000 7fc96a71d640 10 monclient: monmap: 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.772236+0000 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.772236+0000 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.774 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.774 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.774 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.774 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.774 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.774 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 1 Processor -- start 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 1 -- start start 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 10 monclient: init 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9641318f0 0x7fc964131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 10 monclient(hunting): picked mon.noname-b con 0x7fc9641318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.771+0000 7fc96a71d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc96409ea50 0x7fc96409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 10 monclient(hunting): picked mon.noname-a con 0x7fc96409ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc964068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc964131cc0 con 0x7fc964068d90 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96409fbf0 con 0x7fc96409ea50 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96409fe30 con 0x7fc9641318f0 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc96a71d640 10 monclient(hunting): authenticate will time out at 717.412080s 2024-09-15T03:28:39.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc96409ea50 0x7fc96409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc96409ea50 0x7fc96409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49460/0 (socket says 172.21.15.153:49460) 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 -- 172.21.15.153:0/109873982 learned_addr learned my addr 172.21.15.153:0/109873982 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): get_auth_request con 0x7fc96409ea50 auth_method 0 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9641318f0 0x7fc964131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): get_auth_request con 0x7fc9641318f0 auth_method 0 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 -- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9641318f0 msgr2=0x7fc964131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9641318f0 0x7fc964131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 -- 172.21.15.153:0/109873982 >> v1:172.21.15.181:6789/0 conn(0x7fc964068d90 legacy=0x7fc9640691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc9637fe640 1 -- 172.21.15.153:0/109873982 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9640a1210 con 0x7fc96409ea50 2024-09-15T03:28:39.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.772+0000 7fc963fff640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9641318f0 0x7fc964131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:39.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.774+0000 7f5c10020640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.774+0000 7f5c10020640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.774+0000 7f5c10020640 10 monclient: monmap: 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.776560+0000 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.776560+0000 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.782 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.783 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7f5c10020640 1 Processor -- start 2024-09-15T03:28:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7fc9637fe640 1 -- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 msgr2=0x7fc96409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7fc9637fe640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409ee20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc958009910 tx=0x7fc9580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f548b73e9ccdc14d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7f5c10020640 1 -- start start 2024-09-15T03:28:39.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7f5c10020640 10 monclient: init 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7fc962ffd640 1 -- 172.21.15.153:0/109873982 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958032440 con 0x7fc96409ea50 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7fc962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.775+0000 7fc962ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 10 monclient: dump: 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.785 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc96a71d640 5 monclient: authenticate success, global_id 4215 2024-09-15T03:28:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc96a71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 1 -- 172.21.15.153:0/109873982 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9580325a0 con 0x7fc96409ea50 2024-09-15T03:28:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:28:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc96a71d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc96a71d640 4 set_mon_vals no callback set 2024-09-15T03:28:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 1 -- 172.21.15.153:0/109873982 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958032940 con 0x7fc96409ea50 2024-09-15T03:28:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7fc962ffd640 10 monclient: dump: 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.789 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient(hunting): picked mon.noname-b con 0x7f5c081318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c0809ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:39.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c08136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08131cc0 con 0x7f5c0809ea50 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44738/0 (socket says 172.21.15.153:44738) 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 -- 172.21.15.153:0/1167850762 learned_addr learned my addr 172.21.15.153:0/1167850762 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0809f530 con 0x7f5c08136c20 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0809fbf0 con 0x7f5c081318f0 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c10020640 10 monclient(hunting): authenticate will time out at 717.416558s 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): get_auth_request con 0x7f5c081318f0 auth_method 0 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 -- 172.21.15.153:0/1167850762 >> v1:172.21.15.181:6789/0 conn(0x7f5c08136c20 legacy=0x7f5c08139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 -- 172.21.15.153:0/1167850762 >> v1:172.21.15.153:6789/0 conn(0x7f5c0809ea50 legacy=0x7f5c0809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 monclient: found mon.noname-b 2024-09-15T03:28:39.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.776+0000 7f5c0ddbc640 1 -- 172.21.15.153:0/1167850762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0809fe30 con 0x7f5c081318f0 2024-09-15T03:28:39.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 10 monclient: shutdown 2024-09-15T03:28:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 -- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 msgr2=0x7fc96409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409ee20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc958009910 tx=0x7fc9580091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 -- 172.21.15.153:0/109873982 shutdown_connections 2024-09-15T03:28:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409ee20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 --2- 172.21.15.153:0/109873982 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9641318f0 0x7fc964131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 -- 172.21.15.153:0/109873982 >> 172.21.15.153:0/109873982 conn(0x7fc964067110 msgr2=0x7fc964067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 -- 172.21.15.153:0/109873982 shutdown_connections 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.777+0000 7fc96a71d640 1 -- 172.21.15.153:0/109873982 wait complete. 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.778+0000 7fc96a71d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.778+0000 7fc96a71d640 10 monclient: monmap: 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.798 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.778+0000 7fc96a71d640 1 Processor -- start 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.778+0000 7fc96a71d640 1 -- start start 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.778+0000 7fc96a71d640 10 monclient: init 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 0x7fc96411bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): picked mon.noname-c con 0x7fc964068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): picked mon.noname-a con 0x7fc96409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9641318f0 0x7fc964130a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): picked mon.noname-b con 0x7fc9641318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 0x7fc96411bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7f5c0ddbc640 1 -- 172.21.15.153:0/1167850762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 msgr2=0x7f5c08131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7f5c0ddbc640 1 --2- 172.21.15.153:0/1167850762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08131f30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009940 tx=0x7f5c040091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5fc6a7fa579c566 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 0x7fc96411bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44750/0 (socket says 172.21.15.153:44750) 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 1 -- 172.21.15.153:0/3276935578 learned_addr learned my addr 172.21.15.153:0/3276935578 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc9637fe640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96411de60 con 0x7fc96409ea50 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96411c820 con 0x7fc9641318f0 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96411db80 con 0x7fc964068d90 2024-09-15T03:28:39.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc96a71d640 10 monclient(hunting): authenticate will time out at 717.419954s 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 10 monclient(hunting): get_auth_request con 0x7fc964068d90 auth_method 0 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 1 -- 172.21.15.153:0/1167850762 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04032440 con 0x7f5c081318f0 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: dump: 2024-09-15T03:28:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.804 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.804 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.804 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 1 -- 172.21.15.153:0/1167850762 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c040325a0 con 0x7f5c081318f0 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 1 -- 172.21.15.153:0/1167850762 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04032940 con 0x7f5c081318f0 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c0cdba640 10 monclient: dump: 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.807 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.807 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.807 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c10020640 5 monclient: authenticate success, global_id 4220 2024-09-15T03:28:39.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c10020640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7f5c10020640 4 set_mon_vals no callback set 2024-09-15T03:28:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.779+0000 7fc963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc9637fe640 10 monclient(hunting): get_auth_request con 0x7fc96409ea50 auth_method 0 2024-09-15T03:28:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc9637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc9637fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9641318f0 0x7fc964130a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): get_auth_request con 0x7fc9641318f0 auth_method 0 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc968cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 1 -- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9641318f0 msgr2=0x7fc964130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9641318f0 0x7fc964130a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 1 -- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 msgr2=0x7fc96409a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 1 monclient: found mon.noname-c 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc963fff640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954009390 con 0x7fc964068d90 2024-09-15T03:28:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.780+0000 7fc9637fe640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc96409ea50 0x7fc96409a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:39.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 10 monclient: shutdown 2024-09-15T03:28:39.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 msgr2=0x7f5c08131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 --2- 172.21.15.153:0/1167850762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08131f30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009940 tx=0x7f5c040091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 shutdown_connections 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 --2- 172.21.15.153:0/1167850762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08131f30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 >> 172.21.15.153:0/1167850762 conn(0x7f5c08067110 msgr2=0x7f5c08067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 shutdown_connections 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.781+0000 7f5c10020640 1 -- 172.21.15.153:0/1167850762 wait complete. 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 10 monclient: monmap: 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 1 Processor -- start 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 1 -- start start 2024-09-15T03:28:39.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 10 monclient: init 2024-09-15T03:28:39.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7fc963fff640 1 -- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 msgr2=0x7fc96411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7fc963fff640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 0x7fc96411bbf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc954009fc0 tx=0x7fc9540090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.782+0000 7f5c10020640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c0809ea50 0x7f5c08135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0809ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c0ddbc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c0809ea50 0x7f5c08135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c0ddbc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c0809ea50 0x7f5c08135c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49478/0 (socket says 172.21.15.153:49478) 2024-09-15T03:28:39.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c0ddbc640 1 -- 172.21.15.153:0/3834363036 learned_addr learned my addr 172.21.15.153:0/3834363036 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9540096f0 con 0x7fc964068d90 2024-09-15T03:28:39.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: dump: 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.819 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95400cdd0 con 0x7fc964068d90 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 5 monclient: authenticate success, global_id 4223 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 10 monclient: _renew_subs 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc958009450 con 0x7fc964068d90 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954017c80 con 0x7fc964068d90 2024-09-15T03:28:39.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc94bfff640 10 monclient: dump: 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 10 monclient: _renew_subs 2024-09-15T03:28:39.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc96a71d640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96411cd20 con 0x7fc964068d90 2024-09-15T03:28:39.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 10 monclient(hunting): picked mon.noname-c con 0x7f5c081318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c08136c20 0x7f5c08096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7fc9627fc640 4 set_mon_vals no callback set 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c0d5bb640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 10 monclient(hunting): picked mon.noname-b con 0x7f5c08136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c0e5bd640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c08136c20 0x7f5c08096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.783+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0809a200 con 0x7f5c0809ea50 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c10020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0813a480 con 0x7f5c08136c20 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c10020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0809bf10 con 0x7f5c081318f0 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c10020640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c10020640 10 monclient(hunting): authenticate will time out at 717.424314s 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): get_auth_request con 0x7f5c0809ea50 auth_method 0 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): get_auth_request con 0x7f5c081318f0 auth_method 0 2024-09-15T03:28:39.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7fc949ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc96404bf60 con 0x7fc964068d90 2024-09-15T03:28:39.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0ddbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): get_auth_request con 0x7f5c08136c20 auth_method 0 2024-09-15T03:28:39.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc95401f650 con 0x7fc964068d90 2024-09-15T03:28:39.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc94bfff640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc950038790 0x7fc95003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0d5bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc954053fe0 con 0x7fc964068d90 2024-09-15T03:28:39.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc94bfff640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7fc95003e690 0x7fc950040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc950041190 con 0x7fc95003e690 2024-09-15T03:28:39.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc954056570 con 0x7fc964068d90 2024-09-15T03:28:39.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc94bfff640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:28:39.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc9637fe640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc950038790 0x7fc95003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.784+0000 7f5c0e5bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 10 monclient(hunting): handle_auth_done global_id 4219 payload 824 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 1 -- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 msgr2=0x7f5c08096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 1 -- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c0809ea50 msgr2=0x7f5c08135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c0809ea50 0x7f5c08135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 1 monclient: found mon.noname-b 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0e5bd640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c04009450 con 0x7f5c08136c20 2024-09-15T03:28:39.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0ddbc640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c0809ea50 0x7f5c08135c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:39.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7f5c0d5bb640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c081318f0 0x7f5c08096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:39.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc9637fe640 10 monclient: get_auth_request con 0x7fc950038790 auth_method 0 2024-09-15T03:28:39.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc968cba640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7fc95003e690 0x7fc950040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.785+0000 7fc968cba640 10 monclient: get_auth_request con 0x7fc95003e690 auth_method 0 2024-09-15T03:28:39.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5c0e5bd640 1 -- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c08136c20 msgr2=0x7f5c08096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5c0e5bd640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c08136c20 0x7f5c08096fa0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5c080a0240 tx=0x7f5c000025b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000a850 con 0x7f5c08136c20 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: dump: 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.837 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0000a9b0 con 0x7f5c08136c20 2024-09-15T03:28:39.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000adb0 con 0x7f5c08136c20 2024-09-15T03:28:39.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:39.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfdffb640 10 monclient: dump: 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.840 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5bfffff640 4 set_mon_vals no callback set 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5c10020640 5 monclient: authenticate success, global_id 4219 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5c10020640 10 monclient: _renew_subs 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5c10020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.787+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c08097f20 con 0x7f5c08136c20 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.788+0000 7f5c10020640 10 monclient: _renew_subs 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.788+0000 7f5c10020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.788+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0809b830 con 0x7f5c08136c20 2024-09-15T03:28:39.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.788+0000 7fc9637fe640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc950038790 0x7fc95003ac40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc958031000 tx=0x7fc958032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.788+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5c00012630 con 0x7f5c08136c20 2024-09-15T03:28:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5c10020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5bfdffb640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bec038740 0x7f5bec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c080995c0 con 0x7f5c08136c20 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f5c0004b610 con 0x7f5c08136c20 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5bfdffb640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f5bec03e540 0x7f5bec040970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bec041040 con 0x7f5bec03e540 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5c0ddbc640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bec038740 0x7f5bec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5c0ddbc640 10 monclient: get_auth_request con 0x7f5bec038740 auth_method 0 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c00021890 con 0x7f5c08136c20 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5bfdffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5c0d5bb640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f5bec03e540 0x7f5bec040970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.789+0000 7f5c0d5bb640 10 monclient: get_auth_request con 0x7f5bec03e540 auth_method 0 2024-09-15T03:28:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.790+0000 7fc968cba640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7fc95003e690 0x7fc950040ac0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.791+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== osd.5 v2:172.21.15.181:6816/31876 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc950041190 con 0x7fc95003e690 2024-09-15T03:28:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.791+0000 7f5c0ddbc640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bec038740 0x7f5bec03abf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009210 tx=0x7f5c04032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.794+0000 7f5c0d5bb640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f5bec03e540 0x7f5bec040970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.795+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== osd.4 v2:172.21.15.181:6800/31875 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5bec041040 con 0x7f5bec03e540 2024-09-15T03:28:39.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.803+0000 7fa5da11d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.803+0000 7fa5da11d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.803+0000 7fa5da11d640 10 monclient: monmap: 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.804773+0000 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.804773+0000 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.849 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.803+0000 7fa5da11d640 1 Processor -- start 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.803+0000 7fa5da11d640 1 -- start start 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.803+0000 7fa5da11d640 10 monclient: init 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5d4131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 0x7fa5d409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5d409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5d409f220 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d409faf0 con 0x7fa5d409f220 2024-09-15T03:28:39.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 0x7fa5d409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d3fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa5d409f220 0x7fa5d4138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48320/0 (socket says 172.21.15.153:48320) 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d3fff640 1 -- 172.21.15.153:0/3050552330 learned_addr learned my addr 172.21.15.153:0/3050552330 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d409fd30 con 0x7fa5d4131ab0 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d4132590 con 0x7fa5d409e910 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5da11d640 10 monclient(hunting): authenticate will time out at 717.444813s 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): get_auth_request con 0x7fa5d409e910 auth_method 0 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.804+0000 7fa5d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 1 -- 172.21.15.153:0/3050552330 >> v1:172.21.15.181:6789/0 conn(0x7fa5d4131ab0 legacy=0x7fa5d4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 1 -- 172.21.15.153:0/3050552330 >> v1:172.21.15.153:6789/0 conn(0x7fa5d409f220 legacy=0x7fa5d4138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.805+0000 7fa5d2ffd640 1 -- 172.21.15.153:0/3050552330 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d40a1110 con 0x7fa5d409e910 2024-09-15T03:28:39.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.806+0000 7ff2f820f640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.806+0000 7ff2f820f640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.806+0000 7ff2f820f640 10 monclient: monmap: 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.807813+0000 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.807813+0000 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.854 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.855 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.806+0000 7ff2f820f640 1 Processor -- start 2024-09-15T03:28:39.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.806+0000 7ff2f820f640 1 -- start start 2024-09-15T03:28:39.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.806+0000 7ff2f820f640 10 monclient: init 2024-09-15T03:28:39.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7f33e6034640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7f33e6034640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7f33e6034640 10 monclient: monmap: 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.808869+0000 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.808869+0000 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.856 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.857 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff2f009b0a0 0x7ff2f009b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): picked mon.noname-a con 0x7ff2f009b0a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2f012fad0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009cfc0 0x7ff2f0137a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): picked mon.noname-b con 0x7ff2f009cfc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f0067c90 con 0x7ff2f012fad0 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f009dd90 con 0x7ff2f009b0a0 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7fc98371d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7fc98371d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7fc98371d640 10 monclient: monmap: 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:39.809541+0000 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:39.809541+0000 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:39.859 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00995b0 con 0x7ff2f009cfc0 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f820f640 10 monclient(hunting): authenticate will time out at 717.447867s 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f57aa640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7ff2f012fad0 0x7ff2f009c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:53570/0 (socket says 172.21.15.153:53570) 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f57aa640 1 -- 172.21.15.153:0/2652867689 learned_addr learned my addr 172.21.15.153:0/2652867689 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:39.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7ff2f5fab640 1 --2- 172.21.15.153:0/2652867689 >> v2:172.21.15.153:3300/0 conn(0x7ff2f009b0a0 0x7ff2f009b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.807+0000 7f33e6034640 1 Processor -- start 2024-09-15T03:28:39.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): get_auth_request con 0x7ff2f009b0a0 auth_method 0 2024-09-15T03:28:39.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d2ffd640 1 -- 172.21.15.153:0/3050552330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 msgr2=0x7fa5d409ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d2ffd640 1 --2- 172.21.15.153:0/3050552330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 0x7fa5d409ece0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009880 tx=0x7fa5c80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d838a7937e2417d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7f33e6034640 1 -- start start 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7f33e6034640 10 monclient: init 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009cfc0 0x7ff2f0137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): get_auth_request con 0x7ff2f009cfc0 auth_method 0 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc96411dfd0 con 0x7fc95003e690 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 1 -- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009cfc0 msgr2=0x7ff2f0137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009cfc0 0x7ff2f0137a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 1 -- 172.21.15.153:0/2652867689 >> v1:172.21.15.181:6789/0 conn(0x7ff2f012fad0 legacy=0x7ff2f009c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f5fab640 1 -- 172.21.15.153:0/2652867689 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0099890 con 0x7ff2f009b0a0 2024-09-15T03:28:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7ff2f67ac640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009cfc0 0x7ff2f0137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 1 -- 172.21.15.153:0/3050552330 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8032440 con 0x7fa5d409e910 2024-09-15T03:28:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: dump: 2024-09-15T03:28:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 1 -- 172.21.15.153:0/3050552330 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c80325a0 con 0x7fa5d409e910 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 1 -- 172.21.15.153:0/3050552330 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8032940 con 0x7fa5d409e910 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5d27fc640 10 monclient: dump: 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.867 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5da11d640 5 monclient: authenticate success, global_id 4229 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5da11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fa5da11d640 4 set_mon_vals no callback set 2024-09-15T03:28:39.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fc98371d640 1 Processor -- start 2024-09-15T03:28:39.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7fc94bfff640 1 -- 172.21.15.153:0/3276935578 <== osd.5 v2:172.21.15.181:6816/31876 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc96411dfd0 con 0x7fc95003e690 2024-09-15T03:28:39.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7f33e6034640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.808+0000 7f33e6034640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T03:28:39.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33e01318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:39.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f33e009ea50 0x7f33e009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7fc95003e690 msgr2=0x7fc950040ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 10 monclient(hunting): picked mon.noname-c con 0x7f33e009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:39.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7fc95003e690 0x7fc950040ac0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 1 -- start start 2024-09-15T03:28:39.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient: init 2024-09-15T03:28:39.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc950038790 msgr2=0x7fc95003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc950038790 0x7fc95003ac40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc958031000 tx=0x7fc958032000 comp rx=0 tx=0).stop 2024-09-15T03:28:39.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 10 monclient: shutdown 2024-09-15T03:28:39.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 msgr2=0x7fc96411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 --2- 172.21.15.153:0/3276935578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc964068d90 0x7fc96411bbf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc954009fc0 tx=0x7fc9540090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 10 monclient: shutdown 2024-09-15T03:28:39.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 msgr2=0x7fa5d409ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 --2- 172.21.15.153:0/3050552330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 0x7fa5d409ece0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009880 tx=0x7fa5c80090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33df7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f33e01318f0 0x7f33e0131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48334/0 (socket says 172.21.15.153:48334) 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33df7fe640 1 -- 172.21.15.153:0/2519990952 learned_addr learned my addr 172.21.15.153:0/2519990952 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33e0136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e0131cc0 con 0x7f33e01318f0 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e009fbf0 con 0x7f33e0136c20 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e009fe30 con 0x7f33e009ea50 2024-09-15T03:28:39.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7f33e6034640 10 monclient(hunting): authenticate will time out at 717.449370s 2024-09-15T03:28:39.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc968cba640 1 -- 172.21.15.153:0/3276935578 reap_dead start 2024-09-15T03:28:39.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 shutdown_connections 2024-09-15T03:28:39.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 >> 172.21.15.153:0/3276935578 conn(0x7fc964067110 msgr2=0x7fc96409d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 shutdown_connections 2024-09-15T03:28:39.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 --2- 172.21.15.153:0/3050552330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409e910 0x7fa5d409ece0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 >> 172.21.15.153:0/3050552330 conn(0x7fa5d4067d30 msgr2=0x7fa5d40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 shutdown_connections 2024-09-15T03:28:39.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fa5da11d640 1 -- 172.21.15.153:0/3050552330 wait complete. 2024-09-15T03:28:39.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 shutdown_connections 2024-09-15T03:28:39.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc949ffb640 1 -- 172.21.15.153:0/3276935578 wait complete. 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc97c131910 0x7fc97c131ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient(hunting): picked mon.noname-a con 0x7fc97c131910 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc97c132220 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc97c09f110 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:39.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0a0030 con 0x7fc97c132220 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c09fb40 con 0x7fc97c09f110 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97c0a1160 con 0x7fc97c131910 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.809+0000 7fc98371d640 10 monclient(hunting): authenticate will time out at 717.449974s 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7fa5da11d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7fa5da11d640 10 monclient: monmap: 2024-09-15T03:28:39.879 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.880 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 1 --2- 172.21.15.153:0/2519990952 >> v2:172.21.15.181:3300/0 conn(0x7f33e009ea50 0x7f33e009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): get_auth_request con 0x7f33e009ea50 auth_method 0 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7fc981cba640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc97c09f110 0x7fc97c138c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:53586/0 (socket says 172.21.15.153:53586) 2024-09-15T03:28:39.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7fc981cba640 1 -- 172.21.15.153:0/1294424333 learned_addr learned my addr 172.21.15.153:0/1294424333 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:39.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 1 -- 172.21.15.153:0/2519990952 >> v1:172.21.15.181:6789/0 conn(0x7f33e0136c20 legacy=0x7f33e0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 1 -- 172.21.15.153:0/2519990952 >> v1:172.21.15.153:6789/0 conn(0x7f33e01318f0 legacy=0x7f33e0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 1 monclient: found mon.noname-c 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:39.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7f33d6ffd640 1 -- 172.21.15.153:0/2519990952 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e00a1210 con 0x7f33e009ea50 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.810+0000 7fa5da11d640 1 Processor -- start 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 -- start start 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient: init 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d409e910 0x7fa5d412f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5d409e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409f220 0x7fa5d412f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5d409f220 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 0x7fa5d412fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5d4131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d40979f0 con 0x7fa5d4131ab0 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d409ac80 con 0x7fa5d409e910 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d4096a60 con 0x7fa5d409f220 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5da11d640 10 monclient(hunting): authenticate will time out at 717.451689s 2024-09-15T03:28:39.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7ff2f5fab640 1 -- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009b0a0 msgr2=0x7ff2f009b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7ff2f5fab640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009b0a0 0x7ff2f009b470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009850 tx=0x7ff2ec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fa67bddb7ba5fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fc9814b9640 1 --2- 172.21.15.153:0/1294424333 >> v2:172.21.15.153:3300/0 conn(0x7fc97c131910 0x7fc97c131ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 0x7fa5d412fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409f220 0x7fa5d412f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fc9814b9640 10 monclient(hunting): get_auth_request con 0x7fc97c131910 auth_method 0 2024-09-15T03:28:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fc9814b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409f220 0x7fa5d412f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44784/0 (socket says 172.21.15.153:44784) 2024-09-15T03:28:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 1 -- 172.21.15.153:0/247380367 learned_addr learned my addr 172.21.15.153:0/247380367 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 0x7fa5d412fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49504/0 (socket says 172.21.15.153:49504) 2024-09-15T03:28:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 10 monclient(hunting): get_auth_request con 0x7fa5d409f220 auth_method 0 2024-09-15T03:28:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fc9814b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d3fff640 10 monclient(hunting): get_auth_request con 0x7fa5d4131ab0 auth_method 0 2024-09-15T03:28:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.811+0000 7fa5d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-09-15T03:28:39.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 1 -- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409f220 msgr2=0x7fa5d412f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5d409f220 0x7fa5d412f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 1 -- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d409e910 msgr2=0x7fa5d412f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d409e910 0x7fa5d412f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fa5d3fff640 1 -- 172.21.15.153:0/247380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8009390 con 0x7fa5d4131ab0 2024-09-15T03:28:39.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 1 -- 172.21.15.153:0/1294424333 >> v1:172.21.15.181:6789/0 conn(0x7fc97c09f110 legacy=0x7fc97c138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 1 -- 172.21.15.153:0/1294424333 >> v1:172.21.15.153:6789/0 conn(0x7fc97c132220 legacy=0x7fc97c09ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:39.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.812+0000 7fc9814b9640 1 -- 172.21.15.153:0/1294424333 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c09fd80 con 0x7fc97c131910 2024-09-15T03:28:39.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.813+0000 7f33d6ffd640 1 -- 172.21.15.153:0/2519990952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e009ea50 msgr2=0x7f33e009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.813+0000 7f33d6ffd640 1 --2- 172.21.15.153:0/2519990952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e009ea50 0x7f33e009ee20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009850 tx=0x7f33c80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=263d5aa673201227 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.813+0000 7f33deffd640 1 -- 172.21.15.153:0/2519990952 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8032440 con 0x7f33e009ea50 2024-09-15T03:28:39.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.813+0000 7f33deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:39.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: dump: 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 1 -- 172.21.15.153:0/2519990952 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c80325a0 con 0x7f33e009ea50 2024-09-15T03:28:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 5 monclient: authenticate success, global_id 4222 2024-09-15T03:28:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 1 -- 172.21.15.153:0/2652867689 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec02adf0 con 0x7ff2f009b0a0 2024-09-15T03:28:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 4 set_mon_vals no callback set 2024-09-15T03:28:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 1 -- 172.21.15.153:0/2519990952 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8032940 con 0x7f33e009ea50 2024-09-15T03:28:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: dump: 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33deffd640 10 monclient: dump: 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 1 -- 172.21.15.153:0/2652867689 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec032440 con 0x7ff2f009b0a0 2024-09-15T03:28:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f820f640 5 monclient: authenticate success, global_id 4224 2024-09-15T03:28:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f820f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f820f640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f820f640 4 set_mon_vals no callback set 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 1 -- 172.21.15.153:0/2652867689 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0327e0 con 0x7ff2f009b0a0 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7ff2f4fa9640 10 monclient: dump: 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 --> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5c08098820 con 0x7f5bec03e540 2024-09-15T03:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7fa5d3fff640 1 -- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 msgr2=0x7fa5d412fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7fa5d3fff640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 0x7fa5d412fcc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00cb60 tx=0x7fa5cc0052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 10 monclient: shutdown 2024-09-15T03:28:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00a850 con 0x7fa5d4131ab0 2024-09-15T03:28:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 10 monclient: shutdown 2024-09-15T03:28:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e009ea50 msgr2=0x7f33e009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.814+0000 7f33e6034640 1 --2- 172.21.15.153:0/2519990952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e009ea50 0x7f33e009ee20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009850 tx=0x7f33c80090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 shutdown_connections 2024-09-15T03:28:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 1 --2- 172.21.15.153:0/2519990952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e009ea50 0x7f33e009ee20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 >> 172.21.15.153:0/2519990952 conn(0x7f33e0067110 msgr2=0x7f33e0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 -- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009b0a0 msgr2=0x7ff2f009b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009b0a0 0x7ff2f009b470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009850 tx=0x7ff2ec0090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: dump: 2024-09-15T03:28:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 shutdown_connections 2024-09-15T03:28:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 1 -- 172.21.15.153:0/2519990952 wait complete. 2024-09-15T03:28:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5bfdffb640 1 -- 172.21.15.153:0/3834363036 <== osd.4 v2:172.21.15.181:6800/31875 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5c08098820 con 0x7f5bec03e540 2024-09-15T03:28:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 5 monclient: authenticate success, global_id 4227 2024-09-15T03:28:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 10 monclient: _renew_subs 2024-09-15T03:28:39.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 -- 172.21.15.153:0/2652867689 shutdown_connections 2024-09-15T03:28:39.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009cfc0 0x7ff2f0137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 --2- 172.21.15.153:0/2652867689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009b0a0 0x7ff2f009b470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 -- 172.21.15.153:0/2652867689 >> 172.21.15.153:0/2652867689 conn(0x7ff2f0069520 msgr2=0x7ff2f00a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc9814b9640 1 -- 172.21.15.153:0/1294424333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c131910 msgr2=0x7fc97c131ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc9814b9640 1 --2- 172.21.15.153:0/1294424333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c131910 0x7fc97c131ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc978009850 tx=0x7fc9780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422f3a0a4c5b2f60 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc00a9d0 con 0x7fa5d4131ab0 2024-09-15T03:28:39.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d41315f0 con 0x7fa5d4131ab0 2024-09-15T03:28:39.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 -- 172.21.15.153:0/2652867689 shutdown_connections 2024-09-15T03:28:39.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 10 monclient: _renew_subs 2024-09-15T03:28:39.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7ff2f820f640 1 -- 172.21.15.153:0/2652867689 wait complete. 2024-09-15T03:28:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d409b660 con 0x7fa5d4131ab0 2024-09-15T03:28:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f5bec03e540 msgr2=0x7f5bec040970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f5bec03e540 0x7f5bec040970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc973fff640 1 -- 172.21.15.153:0/1294424333 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978032440 con 0x7fc97c131910 2024-09-15T03:28:39.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc973fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc973fff640 10 monclient: dump: 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc973fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc98371d640 5 monclient: authenticate success, global_id 4230 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fc98371d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5d1ffb640 4 set_mon_vals no callback set 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00ad90 con 0x7fa5d4131ab0 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7fa5bb7fe640 10 monclient: dump: 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f33e6034640 10 monclient: monmap: 2024-09-15T03:28:39.919 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7ff2f820f640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bec038740 msgr2=0x7f5bec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bec038740 0x7f5bec03abf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009210 tx=0x7f5c04032000 comp rx=0 tx=0).stop 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 10 monclient: shutdown 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c08136c20 msgr2=0x7f5c08096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.815+0000 7f5c10020640 1 --2- 172.21.15.153:0/3834363036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c08136c20 0x7f5c08096fa0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5c080a0240 tx=0x7f5c000025b0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f5c0e5bd640 1 -- 172.21.15.153:0/3834363036 reap_dead start 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 shutdown_connections 2024-09-15T03:28:39.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 >> 172.21.15.153:0/3834363036 conn(0x7f5c08067110 msgr2=0x7f5c080a36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7ff2f820f640 10 monclient: monmap: 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.922 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f33e6034640 1 Processor -- start 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc973fff640 1 -- 172.21.15.153:0/1294424333 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9780325a0 con 0x7fc97c131910 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc973fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc973fff640 1 -- 172.21.15.153:0/1294424333 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978032940 con 0x7fc97c131910 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc973fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc973fff640 10 monclient: dump: 2024-09-15T03:28:39.923 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.924 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f33e6034640 1 -- start start 2024-09-15T03:28:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f33e6034640 10 monclient: init 2024-09-15T03:28:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 shutdown_connections 2024-09-15T03:28:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc98371d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fc98371d640 4 set_mon_vals no callback set 2024-09-15T03:28:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7f5c10020640 1 -- 172.21.15.153:0/3834363036 wait complete. 2024-09-15T03:28:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fa5da11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.816+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5b4000f20 con 0x7fa5d4131ab0 2024-09-15T03:28:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 Processor -- start 2024-09-15T03:28:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 -- start start 2024-09-15T03:28:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient: init 2024-09-15T03:28:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa5cc007300 con 0x7fa5d4131ab0 2024-09-15T03:28:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 0x7f33e0130780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7fa5bb7fe640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5c0038740 0x7fa5c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 0x7ff2f013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fa5cc04bb10 con 0x7fa5d4131ab0 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): picked mon.noname-a con 0x7f33e009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e01318f0 0x7f33e0099660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): picked mon.noname-b con 0x7f33e01318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33e0136c20 0x7f33e0099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): picked mon.noname-c con 0x7f33e0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e009dda0 con 0x7f33e009ea50 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e011c630 con 0x7f33e01318f0 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e011b9a0 con 0x7f33e0136c20 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33e6034640 10 monclient(hunting): authenticate will time out at 717.457686s 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): picked mon.noname-c con 0x7ff2f009b0a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009cfc0 0x7ff2f012e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): picked mon.noname-a con 0x7ff2f009cfc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2f012fad0 0x7ff2f0118520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7fa5d37fe640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5c0038740 0x7fa5c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): picked mon.noname-b con 0x7ff2f012fad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f013fd20 con 0x7ff2f009cfc0 2024-09-15T03:28:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f011bd10 con 0x7ff2f012fad0 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f011ba30 con 0x7ff2f009b0a0 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f820f640 10 monclient(hunting): authenticate will time out at 717.457855s 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 0x7ff2f013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 0x7ff2f013edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44786/0 (socket says 172.21.15.153:44786) 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 1 -- 172.21.15.153:0/4005522832 learned_addr learned my addr 172.21.15.153:0/4005522832 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 10 monclient(hunting): get_auth_request con 0x7ff2f009b0a0 auth_method 0 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f5fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 0x7f33e0130780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 0x7f33e0130780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49506/0 (socket says 172.21.15.153:49506) 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 1 -- 172.21.15.153:0/2035197204 learned_addr learned my addr 172.21.15.153:0/2035197204 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 10 monclient(hunting): get_auth_request con 0x7f33e009ea50 auth_method 0 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33d6ffd640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e01318f0 0x7f33e0099660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33d6ffd640 10 monclient(hunting): get_auth_request con 0x7f33e01318f0 auth_method 0 2024-09-15T03:28:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7ff2f67ac640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2f012fad0 0x7ff2f0118520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): get_auth_request con 0x7ff2f012fad0 auth_method 0 2024-09-15T03:28:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f57aa640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009cfc0 0x7ff2f012e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7f33dffff640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33e0136c20 0x7f33e0099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): get_auth_request con 0x7f33e0136c20 auth_method 0 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f67ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f57aa640 10 monclient(hunting): get_auth_request con 0x7ff2f009cfc0 auth_method 0 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f57aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f57aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f57aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.817+0000 7fa5d37fe640 10 monclient: get_auth_request con 0x7fa5c0038740 auth_method 0 2024-09-15T03:28:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-09-15T03:28:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 1 -- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33e0136c20 msgr2=0x7f33e0099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fa5bb7fe640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7fa5c003e5c0 0x7fa5c00409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-09-15T03:28:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 1 -- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2f012fad0 msgr2=0x7ff2f0118520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2f012fad0 0x7ff2f0118520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 1 -- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009cfc0 msgr2=0x7ff2f012e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2f009cfc0 0x7ff2f012e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 1 monclient: found mon.noname-c 2024-09-15T03:28:39.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7ff2f5fab640 1 -- 172.21.15.153:0/4005522832 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e8009a20 con 0x7ff2f009b0a0 2024-09-15T03:28:39.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33e0136c20 0x7f33e0099ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 1 -- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e01318f0 msgr2=0x7f33e0099660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33e01318f0 0x7f33e0099660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7f33df7fe640 1 -- 172.21.15.153:0/2035197204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c8009390 con 0x7f33e009ea50 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 --> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5c00410c0 con 0x7fa5c003e5c0 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5cc0502a0 con 0x7fa5d4131ab0 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fa5bb7fe640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 10 monclient: shutdown 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 1 -- 172.21.15.153:0/1294424333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c131910 msgr2=0x7fc97c131ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 1 --2- 172.21.15.153:0/1294424333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c131910 0x7fc97c131ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc978009850 tx=0x7fc9780090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fa5d2ffd640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7fa5c003e5c0 0x7fa5c00409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fa5d2ffd640 10 monclient: get_auth_request con 0x7fa5c003e5c0 auth_method 0 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 1 -- 172.21.15.153:0/1294424333 shutdown_connections 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 1 --2- 172.21.15.153:0/1294424333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c131910 0x7fc97c131ce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.818+0000 7fc98371d640 1 -- 172.21.15.153:0/1294424333 >> 172.21.15.153:0/1294424333 conn(0x7fc97c067b90 msgr2=0x7fc97c0a4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.819+0000 7fc98371d640 1 -- 172.21.15.153:0/1294424333 shutdown_connections 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.819+0000 7fc98371d640 1 -- 172.21.15.153:0/1294424333 wait complete. 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.819+0000 7fc98371d640 10 monclient: build_initial_monmap 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.819+0000 7fc98371d640 10 monclient: monmap: 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:39.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:39.946 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:39.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:39.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7fc98371d640 1 Processor -- start 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7fc98371d640 1 -- start start 2024-09-15T03:28:39.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7fc98371d640 10 monclient: init 2024-09-15T03:28:39.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7fa5d37fe640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5c0038740 0x7fa5c003abf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4007fd0 tx=0x7fa5c4007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7f33df7fe640 1 -- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 msgr2=0x7f33e0130780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7f33df7fe640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 0x7f33e0130780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33d000c5e0 tx=0x7f33d0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7ff2f5fab640 1 -- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 msgr2=0x7ff2f013edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.820+0000 7ff2f5fab640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 0x7ff2f013edb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009850 tx=0x7ff2ec032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec031c70 con 0x7ff2f009b0a0 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: dump: 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.951 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.952 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:39.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 5 monclient: authenticate success, global_id 4238 2024-09-15T03:28:39.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 10 monclient: _renew_subs 2024-09-15T03:28:39.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 1 -- 172.21.15.153:0/4005522832 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ec009390 con 0x7ff2f009b0a0 2024-09-15T03:28:39.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:39.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 0x7fc97c09b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec031df0 con 0x7ff2f009b0a0 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0005cb0 con 0x7f33e009ea50 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: dump: 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.954 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0005e30 con 0x7f33e009ea50 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0007340 con 0x7f33e009ea50 2024-09-15T03:28:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): picked mon.noname-a con 0x7fc97c09f110 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc97c131910 0x7fc97c12f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): picked mon.noname-c con 0x7fc97c131910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33d7fff640 10 monclient: dump: 2024-09-15T03:28:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33e6034640 5 monclient: authenticate success, global_id 4233 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33e6034640 10 monclient: _renew_subs 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33e6034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e011c890 con 0x7f33e009ea50 2024-09-15T03:28:39.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc9814b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 0x7fc97c09b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc9814b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 0x7fc97c09b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49510/0 (socket says 172.21.15.153:49510) 2024-09-15T03:28:39.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc97c132220 0x7fc97c12f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec03c650 con 0x7ff2f009b0a0 2024-09-15T03:28:39.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc9814b9640 1 -- 172.21.15.153:0/1417176107 learned_addr learned my addr 172.21.15.153:0/1417176107 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e5ffb640 10 monclient: dump: 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.959 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 10 monclient: _renew_subs 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2f820f640 1 -- 172.21.15.153:0/4005522832 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f011aad0 con 0x7ff2f009b0a0 2024-09-15T03:28:39.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): picked mon.noname-b con 0x7fc97c132220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:39.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc980cb8640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc97c131910 0x7fc97c12f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1397a0 con 0x7fc97c09f110 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c09abf0 con 0x7fc97c132220 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c097cb0 con 0x7fc97c131910 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7f33ddffb640 4 set_mon_vals no callback set 2024-09-15T03:28:39.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc981cba640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc97c132220 0x7fc97c12f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc98371d640 10 monclient(hunting): authenticate will time out at 717.461968s 2024-09-15T03:28:39.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7ff2e7fff640 4 set_mon_vals no callback set 2024-09-15T03:28:39.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7f33e6034640 10 monclient: _renew_subs 2024-09-15T03:28:39.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7f33e6034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e01374c0 con 0x7f33e009ea50 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.821+0000 7fc9814b9640 10 monclient(hunting): get_auth_request con 0x7fc97c09f110 auth_method 0 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:39.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 1 -- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc97c131910 msgr2=0x7fc97c12f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc97c131910 0x7fc97c12f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 1 -- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc97c132220 msgr2=0x7fc97c12f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc97c132220 0x7fc97c12f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient: _finish_hunting 0 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 1 monclient: found mon.noname-a 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc9814b9640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978009390 con 0x7fc97c09f110 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc980cb8640 10 monclient: get_auth_request con 0x7fc97c131910 auth_method 0 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc981cba640 10 monclient: get_auth_request con 0x7fc97c132220 auth_method 0 2024-09-15T03:28:39.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc981cba640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc97c132220 0x7fc97c12f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:39.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7fc980cb8640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc97c131910 0x7fc97c12f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:39.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7ff2db7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:39.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f33d001b400 con 0x7f33e009ea50 2024-09-15T03:28:39.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.822+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2f004bf60 con 0x7ff2f009b0a0 2024-09-15T03:28:39.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7f33d7fff640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33c0038790 0x7f33c003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f33d004b3a0 con 0x7f33e009ea50 2024-09-15T03:28:39.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7fa5d2ffd640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7fa5c003e5c0 0x7fa5c00409f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff2ec03c9c0 con 0x7ff2f009b0a0 2024-09-15T03:28:39.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7f33d6ffd640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33c0038790 0x7f33c003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7f33d6ffd640 10 monclient: get_auth_request con 0x7f33c0038790 auth_method 0 2024-09-15T03:28:39.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7ff2e5ffb640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2d4038740 0x7ff2d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7f33e6034640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7f33e0141cc0 0x7f33e009a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7ff2ec06c380 con 0x7ff2f009b0a0 2024-09-15T03:28:39.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 --> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33e011ca20 con 0x7f33e0141cc0 2024-09-15T03:28:39.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7ff2e5ffb640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7ff2d403e5c0 0x7ff2d40409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.823+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 --> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2d40410c0 con 0x7ff2d403e5c0 2024-09-15T03:28:39.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2ec06e910 con 0x7ff2f009b0a0 2024-09-15T03:28:39.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7ff2e5ffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:28:39.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7f33dffff640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7f33e0141cc0 0x7f33e009a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== osd.1 v2:172.21.15.153:6808/31923 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa5c00410c0 con 0x7fa5c003e5c0 2024-09-15T03:28:39.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7f33dffff640 10 monclient: get_auth_request con 0x7f33e0141cc0 auth_method 0 2024-09-15T03:28:39.972 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:39.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:28:39.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7ff2f67ac640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7ff2d403e5c0 0x7ff2d40409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7ff2f67ac640 10 monclient: get_auth_request con 0x7ff2d403e5c0 auth_method 0 2024-09-15T03:28:39.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7fc9814b9640 1 -- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 msgr2=0x7fc97c09b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:39.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.824+0000 7fc9814b9640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 0x7fc97c09b500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc978009850 tx=0x7fc978032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978031c70 con 0x7fc97c09f110 2024-09-15T03:28:39.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: dump: 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: _finish_auth 0 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc978031dd0 con 0x7fc97c09f110 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 5 monclient: authenticate success, global_id 4239 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 10 monclient: _renew_subs 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c097f90 con 0x7fc97c09f110 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97803b650 con 0x7fc97c09f110 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:39.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9717fa640 10 monclient: dump: 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc9737fe640 4 set_mon_vals no callback set 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 10 monclient: _renew_subs 2024-09-15T03:28:39.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.825+0000 7fc98371d640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c0976f0 con 0x7fc97c09f110 2024-09-15T03:28:39.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.826+0000 7fc966ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:39.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.826+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc954000f20 con 0x7fc97c09f110 2024-09-15T03:28:39.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.826+0000 7f33d6ffd640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33c0038790 0x7f33c003ac40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33c8031000 tx=0x7f33c8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.826+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc978039040 con 0x7fc97c09f110 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.826+0000 7fc9717fa640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc960038740 0x7fc96003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.826+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc97806c010 con 0x7fc97c09f110 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.827+0000 7fc9717fa640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7fc96003e540 0x7fc960040970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.827+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc960041040 con 0x7fc96003e540 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.827+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc97803be40 con 0x7fc97c09f110 2024-09-15T03:28:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.827+0000 7fc9717fa640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:28:39.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.827+0000 7fc981cba640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7fc96003e540 0x7fc960040970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.827+0000 7fc981cba640 10 monclient: get_auth_request con 0x7fc96003e540 auth_method 0 2024-09-15T03:28:39.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.828+0000 7f33dffff640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7f33e0141cc0 0x7f33e009a0e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.829+0000 7ff2f67ac640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7ff2d403e5c0 0x7ff2d40409f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.829+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== osd.2 v2:172.21.15.153:6816/31924 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f33e011ca20 con 0x7f33e0141cc0 2024-09-15T03:28:39.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.829+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== osd.3 v2:172.21.15.181:6808/31874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff2d40410c0 con 0x7ff2d403e5c0 2024-09-15T03:28:39.981 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:39.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:28:39.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.832+0000 7fc981cba640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7fc96003e540 0x7fc960040970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.833+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== osd.0 v2:172.21.15.153:6800/31922 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc960041040 con 0x7fc96003e540 2024-09-15T03:28:39.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.838+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 --> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa5b4002b30 con 0x7fa5c003e5c0 2024-09-15T03:28:39.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.838+0000 7fa5bb7fe640 1 -- 172.21.15.153:0/247380367 <== osd.1 v2:172.21.15.153:6808/31923 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa5b4002b30 con 0x7fa5c003e5c0 2024-09-15T03:28:39.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.838+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7fa5c003e5c0 msgr2=0x7fa5c00409f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.838+0000 7fa5da11d640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7fa5c003e5c0 0x7fa5c00409f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5c0038740 msgr2=0x7fa5c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5c0038740 0x7fa5c003abf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4007fd0 tx=0x7fa5c4007a90 comp rx=0 tx=0).stop 2024-09-15T03:28:39.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 10 monclient: shutdown 2024-09-15T03:28:39.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 msgr2=0x7fa5d412fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 --2- 172.21.15.153:0/247380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5d4131ab0 0x7fa5d412fcc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00cb60 tx=0x7fa5cc0052b0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5d3fff640 1 -- 172.21.15.153:0/247380367 reap_dead start 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 shutdown_connections 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 >> 172.21.15.153:0/247380367 conn(0x7fa5d4067d30 msgr2=0x7fa5d40a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 shutdown_connections 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.839+0000 7fa5da11d640 1 -- 172.21.15.153:0/247380367 wait complete. 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.841+0000 7fc980cb8640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc960038740 0x7fc96003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.841+0000 7fc980cb8640 10 monclient: get_auth_request con 0x7fc960038740 auth_method 0 2024-09-15T03:28:39.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.841+0000 7ff2f57aa640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2d4038740 0x7ff2d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:39.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.841+0000 7ff2f57aa640 10 monclient: get_auth_request con 0x7ff2d4038740 auth_method 0 2024-09-15T03:28:39.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.842+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 --> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2f013f530 con 0x7ff2d403e5c0 2024-09-15T03:28:39.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.842+0000 7ff2e5ffb640 1 -- 172.21.15.153:0/4005522832 <== osd.3 v2:172.21.15.181:6808/31874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2f013f530 con 0x7ff2d403e5c0 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7ff2d403e5c0 msgr2=0x7ff2d40409f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7ff2d403e5c0 0x7ff2d40409f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2d4038740 msgr2=0x7ff2d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2d4038740 0x7ff2d403abf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0007fd0 tx=0x7ff2e0009040 comp rx=0 tx=0).stop 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 10 monclient: shutdown 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 msgr2=0x7ff2f013edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 --2- 172.21.15.153:0/4005522832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2f009b0a0 0x7ff2f013edb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009850 tx=0x7ff2ec032000 comp rx=0 tx=0).stop 2024-09-15T03:28:39.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2f67ac640 1 -- 172.21.15.153:0/4005522832 reap_dead start 2024-09-15T03:28:39.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 shutdown_connections 2024-09-15T03:28:39.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 >> 172.21.15.153:0/4005522832 conn(0x7ff2f0069520 msgr2=0x7ff2f009e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 shutdown_connections 2024-09-15T03:28:39.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7ff2db7fe640 1 -- 172.21.15.153:0/4005522832 wait complete. 2024-09-15T03:28:39.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.843+0000 7fc980cb8640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc960038740 0x7fc96003abf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0027e0 tx=0x7fc96c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:39.989 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:39.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:28:39.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.851+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 --> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc954002b40 con 0x7fc96003e540 2024-09-15T03:28:39.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.851+0000 7fc9717fa640 1 -- 172.21.15.153:0/1417176107 <== osd.0 v2:172.21.15.153:6800/31922 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc954002b40 con 0x7fc96003e540 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.851+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7fc96003e540 msgr2=0x7fc960040970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.851+0000 7fc966ffd640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7fc96003e540 0x7fc960040970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc960038740 msgr2=0x7fc96003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc960038740 0x7fc96003abf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0027e0 tx=0x7fc96c009040 comp rx=0 tx=0).stop 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 10 monclient: shutdown 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 msgr2=0x7fc97c09b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 --2- 172.21.15.153:0/1417176107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc97c09f110 0x7fc97c09b500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc978009850 tx=0x7fc978032000 comp rx=0 tx=0).stop 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc981cba640 1 -- 172.21.15.153:0/1417176107 reap_dead start 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 shutdown_connections 2024-09-15T03:28:39.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 >> 172.21.15.153:0/1417176107 conn(0x7fc97c067b90 msgr2=0x7fc97c0a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 shutdown_connections 2024-09-15T03:28:39.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.852+0000 7fc966ffd640 1 -- 172.21.15.153:0/1417176107 wait complete. 2024-09-15T03:28:39.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 --> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33e009a620 con 0x7f33e0141cc0 2024-09-15T03:28:39.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33d7fff640 1 -- 172.21.15.153:0/2035197204 <== osd.2 v2:172.21.15.153:6816/31924 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33e009a620 con 0x7f33e0141cc0 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7f33e0141cc0 msgr2=0x7f33e009a0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7f33e0141cc0 0x7f33e009a0e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33c0038790 msgr2=0x7f33c003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33c0038790 0x7f33c003ac40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33c8031000 tx=0x7f33c8032000 comp rx=0 tx=0).stop 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 10 monclient: shutdown 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 msgr2=0x7f33e0130780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 --2- 172.21.15.153:0/2035197204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33e009ea50 0x7f33e0130780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33d000c5e0 tx=0x7f33d0004960 comp rx=0 tx=0).stop 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33dffff640 1 -- 172.21.15.153:0/2035197204 reap_dead start 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 shutdown_connections 2024-09-15T03:28:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.853+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 >> 172.21.15.153:0/2035197204 conn(0x7f33e0067110 msgr2=0x7f33e00a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:39.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.854+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 shutdown_connections 2024-09-15T03:28:39.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:39.854+0000 7f33e6034640 1 -- 172.21.15.153:0/2035197204 wait complete. 2024-09-15T03:28:39.995 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:39.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:28:39.996 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:39.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:28:39.998 INFO:teuthology.orchestra.run.smithi153.stdout:47244640261 2024-09-15T03:28:39.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:28:40.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f7ca6d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:40.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f24f5554640 10 monclient: get_monmap_and_config 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f24f5554640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f24f5554640 10 monclient: monmap: 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:40.129010+0000 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:40.129010+0000 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f7ca6d5d640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f7ca6d5d640 10 monclient: monmap: 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:40.128854+0000 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:40.128854+0000 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:40.132 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f24f5554640 1 Processor -- start 2024-09-15T03:28:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f24f5554640 1 -- start start 2024-09-15T03:28:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f7ca6d5d640 1 Processor -- start 2024-09-15T03:28:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.127+0000 7f24f5554640 10 monclient: init 2024-09-15T03:28:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 1 -- start start 2024-09-15T03:28:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient: init 2024-09-15T03:28:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T03:28:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:28:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f24f01318f0 0x7f24f0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ca01318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7ca009ea50 0x7f7ca009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient(hunting): picked mon.noname-c con 0x7f24f01318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ca009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7ca0136c20 0x7f7ca0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24f009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24f0136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f009f530 con 0x7f24f0136c20 2024-09-15T03:28:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f009fbf0 con 0x7f24f009ea50 2024-09-15T03:28:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f0131d50 con 0x7f24f01318f0 2024-09-15T03:28:40.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24f5554640 10 monclient(hunting): authenticate will time out at 717.768769s 2024-09-15T03:28:40.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ca0136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:40.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24ef7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f24f0136c20 0x7f24f0139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48346/0 (socket says 172.21.15.153:48346) 2024-09-15T03:28:40.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca0131cc0 con 0x7f7ca01318f0 2024-09-15T03:28:40.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca009fbf0 con 0x7f7ca0136c20 2024-09-15T03:28:40.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca009fe30 con 0x7f7ca009ea50 2024-09-15T03:28:40.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7ca6d5d640 10 monclient(hunting): authenticate will time out at 717.768616s 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7ca009ea50 0x7f7ca009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7ca009ea50 0x7f7ca009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:39858/0 (socket says 172.21.15.153:39858) 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 1 -- 172.21.15.153:0/837722557 learned_addr learned my addr 172.21.15.153:0/837722557 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ca009ea50 auth_method 0 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f7c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.128+0000 7f24ef7fe640 1 -- 172.21.15.153:0/1424168460 learned_addr learned my addr 172.21.15.153:0/1424168460 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 1 --2- 172.21.15.153:0/837722557 >> v2:172.21.15.153:3300/0 conn(0x7f7ca0136c20 0x7f7ca0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 1 --2- 172.21.15.153:0/1424168460 >> v2:172.21.15.181:3300/0 conn(0x7f24f01318f0 0x7f24f0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): get_auth_request con 0x7f24f01318f0 auth_method 0 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): get_auth_request con 0x7f7ca0136c20 auth_method 0 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7c9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7c9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7c9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 1 -- 172.21.15.153:0/1424168460 >> v1:172.21.15.181:6789/0 conn(0x7f24f009ea50 legacy=0x7f24f009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 1 -- 172.21.15.153:0/1424168460 >> v1:172.21.15.153:6789/0 conn(0x7f24f0136c20 legacy=0x7f24f0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f24eeffd640 1 -- 172.21.15.153:0/1424168460 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f009fe30 con 0x7f24f01318f0 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 1 -- 172.21.15.153:0/837722557 >> v2:172.21.15.181:3300/0 conn(0x7f7ca009ea50 msgr2=0x7f7ca009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 1 --2- 172.21.15.153:0/837722557 >> v2:172.21.15.181:3300/0 conn(0x7f7ca009ea50 0x7f7ca009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 1 -- 172.21.15.153:0/837722557 >> v1:172.21.15.181:6789/0 conn(0x7f7ca01318f0 legacy=0x7f7ca0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 1 monclient: found mon.noname-a 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.129+0000 7f7ca52fa640 1 -- 172.21.15.153:0/837722557 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca00a1210 con 0x7f7ca0136c20 2024-09-15T03:28:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24eeffd640 1 -- 172.21.15.153:0/1424168460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f01318f0 msgr2=0x7f24f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24eeffd640 1 --2- 172.21.15.153:0/1424168460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f01318f0 0x7f24f0131f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009850 tx=0x7f24e00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=733bfad280b1f6cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7ca52fa640 1 -- 172.21.15.153:0/837722557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca0136c20 msgr2=0x7f7ca0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7ca52fa640 1 --2- 172.21.15.153:0/837722557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca0136c20 0x7f7ca0139000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7ca006f600 tx=0x7f7c98009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979870bc53c37308 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 1 -- 172.21.15.153:0/1424168460 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0032440 con 0x7f24f01318f0 2024-09-15T03:28:40.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: dump: 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.150 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 1 -- 172.21.15.153:0/1424168460 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e00325a0 con 0x7f24f01318f0 2024-09-15T03:28:40.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 1 -- 172.21.15.153:0/1424168460 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0032940 con 0x7f24f01318f0 2024-09-15T03:28:40.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 1 -- 172.21.15.153:0/837722557 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98007760 con 0x7f7ca0136c20 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24edffb640 10 monclient: dump: 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.152 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.153 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24f5554640 5 monclient: authenticate success, global_id 4234 2024-09-15T03:28:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24f5554640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f24f5554640 4 set_mon_vals no callback set 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 10 monclient: dump: 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.159 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7ca6d5d640 5 monclient: authenticate success, global_id 4242 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7ca6d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 1 -- 172.21.15.153:0/837722557 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9802d050 con 0x7f7ca0136c20 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7c9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7ca6d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.132+0000 7f7ca6d5d640 4 set_mon_vals no callback set 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7c9f7fe640 1 -- 172.21.15.153:0/837722557 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98007af0 con 0x7f7ca0136c20 2024-09-15T03:28:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7c9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:40.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 10 monclient: shutdown 2024-09-15T03:28:40.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 -- 172.21.15.153:0/1424168460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f01318f0 msgr2=0x7f24f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 --2- 172.21.15.153:0/1424168460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f01318f0 0x7f24f0131f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009850 tx=0x7f24e00090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7c9f7fe640 10 monclient: dump: 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.162 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 -- 172.21.15.153:0/1424168460 shutdown_connections 2024-09-15T03:28:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 --2- 172.21.15.153:0/1424168460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f01318f0 0x7f24f0131f30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 -- 172.21.15.153:0/1424168460 >> 172.21.15.153:0/1424168460 conn(0x7f24f0067110 msgr2=0x7f24f0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 -- 172.21.15.153:0/1424168460 shutdown_connections 2024-09-15T03:28:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f24f5554640 1 -- 172.21.15.153:0/1424168460 wait complete. 2024-09-15T03:28:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 10 monclient: shutdown 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/837722557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca0136c20 msgr2=0x7f7ca0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/837722557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca0136c20 0x7f7ca0139000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7ca006f600 tx=0x7f7c98009780 comp rx=0 tx=0).stop 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/837722557 shutdown_connections 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/837722557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca0136c20 0x7f7ca0139000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/837722557 >> v2:172.21.15.181:3300/0 conn(0x7f7ca009ea50 0x7f7ca009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/837722557 >> 172.21.15.153:0/837722557 conn(0x7f7ca0067110 msgr2=0x7f7ca0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.133+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/837722557 shutdown_connections 2024-09-15T03:28:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/837722557 wait complete. 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f24f5554640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f24f5554640 10 monclient: monmap: 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:40.166 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f7ca6d5d640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f7ca6d5d640 10 monclient: monmap: 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:40.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f24f5554640 1 Processor -- start 2024-09-15T03:28:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f24f5554640 1 -- start start 2024-09-15T03:28:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f24f5554640 10 monclient: init 2024-09-15T03:28:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.134+0000 7f7ca6d5d640 1 Processor -- start 2024-09-15T03:28:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f009ea50 0x7f24f01329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): picked mon.noname-b con 0x7f24f009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 0x7f24f0132f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f7ca6d5d640 1 -- start start 2024-09-15T03:28:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f7ca6d5d640 10 monclient: init 2024-09-15T03:28:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): picked mon.noname-a con 0x7f24f01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0136c20 0x7f24f0133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): picked mon.noname-c con 0x7f24f0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0133ef0 con 0x7f24f01318f0 2024-09-15T03:28:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0136220 con 0x7f24f009ea50 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0135f40 con 0x7f24f0136c20 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24f5554640 10 monclient(hunting): authenticate will time out at 717.775387s 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0136c20 0x7f24f0133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f009ea50 0x7f24f01329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 0x7f24f0132f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0136c20 0x7f24f0133470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44816/0 (socket says 172.21.15.153:44816) 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 1 -- 172.21.15.153:0/548542714 learned_addr learned my addr 172.21.15.153:0/548542714 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 0x7f24f0132f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49530/0 (socket says 172.21.15.153:49530) 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): get_auth_request con 0x7f24f0136c20 auth_method 0 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): get_auth_request con 0x7f24f01318f0 auth_method 0 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): get_auth_request con 0x7f24f009ea50 auth_method 0 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-09-15T03:28:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 -- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0136c20 msgr2=0x7f24f0133470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0136c20 0x7f24f0133470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 -- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f009ea50 msgr2=0x7f24f01329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f009ea50 0x7f24f01329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 monclient: found mon.noname-a 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.135+0000 7f24ee7fc640 1 -- 172.21.15.153:0/548542714 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0009390 con 0x7f24f01318f0 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca009ea50 0x7f7ca006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ca009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca01318f0 0x7f7ca0099730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ca01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 0x7f7ca0099c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ca0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca006bc00 con 0x7f7ca01318f0 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca009bf10 con 0x7f7ca009ea50 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca009db60 con 0x7f7ca0136c20 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca6d5d640 10 monclient(hunting): authenticate will time out at 717.776413s 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 0x7f7ca0099c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 0x7f7ca0099c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44832/0 (socket says 172.21.15.153:44832) 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 -- 172.21.15.153:0/2321046386 learned_addr learned my addr 172.21.15.153:0/2321046386 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): get_auth_request con 0x7f7ca0136c20 auth_method 0 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca009ea50 0x7f7ca006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): get_auth_request con 0x7f7ca009ea50 auth_method 0 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca4af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 -- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca009ea50 msgr2=0x7f7ca006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca009ea50 0x7f7ca006e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 -- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca01318f0 msgr2=0x7f7ca0099730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca01318f0 0x7f7ca0099730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.136+0000 7f7ca52fa640 1 -- 172.21.15.153:0/2321046386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98009a20 con 0x7f7ca0136c20 2024-09-15T03:28:40.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24ee7fc640 1 -- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 msgr2=0x7f24f0132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24ee7fc640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 0x7f24f0132f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c4e0 tx=0x7f24e4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e40117a0 con 0x7f24f01318f0 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: dump: 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.186 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e4011920 con 0x7f24f01318f0 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4011ce0 con 0x7f24f01318f0 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24d6ffd640 10 monclient: dump: 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.187 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24ed7fa640 4 set_mon_vals no callback set 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.138+0000 7f24f5554640 5 monclient: authenticate success, global_id 4245 2024-09-15T03:28:40.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f24f5554640 10 monclient: _renew_subs 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f24f5554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f0136500 con 0x7f24f01318f0 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f24f5554640 10 monclient: _renew_subs 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f24f5554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f0131430 con 0x7f24f01318f0 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7ca52fa640 1 -- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 msgr2=0x7f7ca0099c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7ca52fa640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 0x7f7ca0099c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009e90 tx=0x7f7c98037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98031b90 con 0x7f7ca0136c20 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: dump: 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98040070 con 0x7f7ca0136c20 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98045480 con 0x7f7ca0136c20 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9cff9640 10 monclient: dump: 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.192 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.193 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.193 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.193 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7c9effd640 4 set_mon_vals no callback set 2024-09-15T03:28:40.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7ca6d5d640 5 monclient: authenticate success, global_id 4247 2024-09-15T03:28:40.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7ca6d5d640 10 monclient: _renew_subs 2024-09-15T03:28:40.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.139+0000 7f7ca6d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca009b950 con 0x7f7ca0136c20 2024-09-15T03:28:40.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f7ca6d5d640 10 monclient: _renew_subs 2024-09-15T03:28:40.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f7ca6d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca006aca0 con 0x7f7ca0136c20 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f24e4012630 con 0x7f24f01318f0 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f24f5554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.140+0000 7f24f5554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f24d6ffd640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f24dc038790 0x7f24dc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f24f5554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f24e404b500 con 0x7f24f01318f0 2024-09-15T03:28:40.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f0134b20 con 0x7f24f01318f0 2024-09-15T03:28:40.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f7ca6d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f7ca6d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f7ca6d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca004bf60 con 0x7f7ca0136c20 2024-09-15T03:28:40.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f24eeffd640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f24dc038790 0x7f24dc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.141+0000 7f24eeffd640 10 monclient: get_auth_request con 0x7f24dc038790 auth_method 0 2024-09-15T03:28:40.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.143+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7c98042030 con 0x7f7ca0136c20 2024-09-15T03:28:40.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.144+0000 7f24eeffd640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f24dc038790 0x7f24dc03ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24e0031000 tx=0x7f24e0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.144+0000 7f7c9cff9640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7c84038790 0x7f7c8403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.144+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f7c9806d0b0 con 0x7f7ca0136c20 2024-09-15T03:28:40.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.151+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f24e401c460 con 0x7f24f01318f0 2024-09-15T03:28:40.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.151+0000 7f24d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.151+0000 7f24d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.151+0000 7f7ca4af9640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7c84038790 0x7f7c8403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.151+0000 7f7ca4af9640 10 monclient: get_auth_request con 0x7f7c84038790 auth_method 0 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.152+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7c9809ee40 con 0x7f7ca0136c20 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.152+0000 7f7c9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.152+0000 7f7c9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.154+0000 7f7ca4af9640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7c84038790 0x7f7c8403ac40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c94007fd0 tx=0x7f7c94007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f960b29f640 10 monclient: get_monmap_and_config 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f960b29f640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f960b29f640 10 monclient: monmap: 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:40.173801+0000 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:40.173801+0000 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:40.200 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7fb7e219d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7fb7e219d640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7fb7e219d640 10 monclient: monmap: 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:40.174169+0000 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:40.174169+0000 2024-09-15T03:28:40.201 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f4ae08c9640 10 monclient: get_monmap_and_config 2024-09-15T03:28:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f4ae08c9640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f4ae08c9640 10 monclient: monmap: 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:40.174149+0000 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:40.174149+0000 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7fdd9595d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7fdd9595d640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7fdd9595d640 10 monclient: monmap: 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:40.174222+0000 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:40.174222+0000 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:40.204 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:40.205 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.172+0000 7f960b29f640 1 Processor -- start 2024-09-15T03:28:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7f4ae08c9640 1 Processor -- start 2024-09-15T03:28:40.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 1 Processor -- start 2024-09-15T03:28:40.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7f960b29f640 1 -- start start 2024-09-15T03:28:40.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7f960b29f640 10 monclient: init 2024-09-15T03:28:40.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 1 -- start start 2024-09-15T03:28:40.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 10 monclient: init 2024-09-15T03:28:40.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7f4ae08c9640 1 -- start start 2024-09-15T03:28:40.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7f4ae08c9640 10 monclient: init 2024-09-15T03:28:40.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fb7e219d640 1 Processor -- start 2024-09-15T03:28:40.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T03:28:40.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd90096e10 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fdd9595d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdd900971e0 0x7fdd900975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd900971e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96041318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f960409ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:40.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4adc067820 0x7f4adc09cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4adc067820 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4adc09d060 0x7f4adc09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4adc09d060 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4adc09def0 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:40.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc09d460 con 0x7f4adc09def0 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd90133b30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc09f420 con 0x7f4adc067820 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc067bf0 con 0x7f4adc09d060 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ae08c9640 10 monclient(hunting): authenticate will time out at 717.814154s 2024-09-15T03:28:40.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9604137c30 0x7f960413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd90099620 con 0x7fdd90096e10 2024-09-15T03:28:40.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd90098510 con 0x7fdd900971e0 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90099340 con 0x7fdd90133b30 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44840/0 (socket says 172.21.15.153:44840) 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient(hunting): picked mon.noname-c con 0x7f9604137c30 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:40.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 1 -- 172.21.15.153:0/1080273984 learned_addr learned my addr 172.21.15.153:0/1080273984 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9604131cc0 con 0x7f960409ea50 2024-09-15T03:28:40.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960409f530 con 0x7f96041318f0 2024-09-15T03:28:40.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960409fbf0 con 0x7f9604137c30 2024-09-15T03:28:40.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd9595d640 10 monclient(hunting): authenticate will time out at 717.814390s 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960b29f640 10 monclient(hunting): authenticate will time out at 717.814417s 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fb7e219d640 1 -- start start 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.173+0000 7fb7e219d640 10 monclient: init 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb7dc09ebd0 0x7fb7dc09efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7dc09ebd0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7dc09cf50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7dc09d350 0x7fb7dc0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7dc09d350 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc09fea0 con 0x7fb7dc09cf50 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7dc0a1260 con 0x7fb7dc09d350 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7dc09fc40 con 0x7fb7dc09ebd0 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7e219d640 10 monclient(hunting): authenticate will time out at 717.814505s 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960903b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f96041318f0 0x7f9604131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:53626/0 (socket says 172.21.15.153:53626) 2024-09-15T03:28:40.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960903b640 1 -- 172.21.15.153:0/3171225824 learned_addr learned my addr 172.21.15.153:0/3171225824 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 1 --2- 172.21.15.153:0/1080273984 >> v2:172.21.15.181:3300/0 conn(0x7fdd900971e0 0x7fdd900975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): get_auth_request con 0x7fdd900971e0 auth_method 0 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4adc09d060 0x7f4adc09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4adc09d060 0x7f4adc09d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:39914/0 (socket says 172.21.15.153:39914) 2024-09-15T03:28:40.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 1 -- 172.21.15.153:0/3281783432 learned_addr learned my addr 172.21.15.153:0/3281783432 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): get_auth_request con 0x7f4adc09d060 auth_method 0 2024-09-15T03:28:40.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7dc09d350 0x7fb7dc0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7dc09d350 0x7fb7dc0694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49554/0 (socket says 172.21.15.153:49554) 2024-09-15T03:28:40.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 1 -- 172.21.15.153:0/2035210342 learned_addr learned my addr 172.21.15.153:0/2035210342 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 1 --2- 172.21.15.153:0/3171225824 >> v2:172.21.15.181:3300/0 conn(0x7f9604137c30 0x7f960413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): get_auth_request con 0x7f9604137c30 auth_method 0 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): get_auth_request con 0x7fb7dc09d350 auth_method 0 2024-09-15T03:28:40.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 1 --2- 172.21.15.153:0/3281783432 >> v2:172.21.15.153:3300/0 conn(0x7f4adc067820 0x7f4adc09cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): get_auth_request con 0x7f4adc067820 auth_method 0 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f960983c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ada59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 10 monclient(hunting): get_auth_request con 0x7fdd90133b30 auth_method 0 2024-09-15T03:28:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7fdd8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 1 -- 172.21.15.153:0/3281783432 >> v2:172.21.15.153:3300/0 conn(0x7f4adc067820 msgr2=0x7f4adc09cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 1 --2- 172.21.15.153:0/3281783432 >> v2:172.21.15.153:3300/0 conn(0x7f4adc067820 0x7f4adc09cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 1 -- 172.21.15.153:0/3281783432 >> v1:172.21.15.153:6789/0 conn(0x7f4adc09def0 legacy=0x7f4adc1367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.174+0000 7f4ad9d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7f4ad9d9b640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7f4ad9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2024-09-15T03:28:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7f4ad9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 1 -- 172.21.15.153:0/1080273984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 msgr2=0x7fdd90135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 1 --2- 172.21.15.153:0/1080273984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 1 -- 172.21.15.153:0/1080273984 >> v1:172.21.15.181:6789/0 conn(0x7fdd90096e10 legacy=0x7fdd9009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7f4ad9d9b640 1 -- 172.21.15.153:0/3281783432 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc09af80 con 0x7f4adc09d060 2024-09-15T03:28:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8e7fc640 1 -- 172.21.15.153:0/1080273984 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90099900 con 0x7fdd900971e0 2024-09-15T03:28:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.175+0000 7fdd8f7fe640 1 --2- 172.21.15.153:0/1080273984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7f960903b640 1 -- 172.21.15.153:0/3171225824 >> v1:172.21.15.181:6789/0 conn(0x7f96041318f0 legacy=0x7f9604131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 1 --2- 172.21.15.153:0/2035210342 >> v2:172.21.15.181:3300/0 conn(0x7fb7dc09ebd0 0x7fb7dc09efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): get_auth_request con 0x7fb7dc09ebd0 auth_method 0 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7fb7daffd640 1 -- 172.21.15.153:0/2035210342 >> v1:172.21.15.153:6789/0 conn(0x7fb7dc09cf50 legacy=0x7fb7dc068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.176+0000 7f95fbfff640 1 -- 172.21.15.153:0/3171225824 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165939732 0 0) 0x7f960409f530 con 0x7f96041318f0 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f95fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f95fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f95fbfff640 10 monclient(hunting): my global_id is 4255 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f95fbfff640 1 -- 172.21.15.153:0/3171225824 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f00034e0 con 0x7f96041318f0 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960883a640 1 -- 172.21.15.153:0/3171225824 >> v1:172.21.15.153:6789/0 conn(0x7f960409ea50 legacy=0x7f960409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 1 -- 172.21.15.153:0/3171225824 >> v1:172.21.15.181:6789/0 conn(0x7f96041318f0 legacy=0x7f9604131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 1 -- 172.21.15.153:0/3171225824 >> v1:172.21.15.153:6789/0 conn(0x7f960409ea50 legacy=0x7f960409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7f960983c640 1 -- 172.21.15.153:0/3171225824 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960409fe30 con 0x7f9604137c30 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 1 -- 172.21.15.153:0/2035210342 >> v2:172.21.15.181:3300/0 conn(0x7fb7dc09ebd0 msgr2=0x7fb7dc09efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 1 --2- 172.21.15.153:0/2035210342 >> v2:172.21.15.181:3300/0 conn(0x7fb7dc09ebd0 0x7fb7dc09efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 1 -- 172.21.15.153:0/2035210342 >> v1:172.21.15.153:6789/0 conn(0x7fb7dc09cf50 legacy=0x7fb7dc068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 1 monclient: found mon.noname-a 2024-09-15T03:28:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fb7dbfff640 1 -- 172.21.15.153:0/2035210342 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0a0080 con 0x7fb7dc09d350 2024-09-15T03:28:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fdd8e7fc640 1 -- 172.21.15.153:0/1080273984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd900971e0 msgr2=0x7fdd900975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.177+0000 7fdd8e7fc640 1 --2- 172.21.15.153:0/1080273984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd900971e0 0x7fdd900975e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdd90071470 tx=0x7fdd840090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=611921006d2835d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad9d9b640 1 -- 172.21.15.153:0/3281783432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc09d060 msgr2=0x7f4adc09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad9d9b640 1 --2- 172.21.15.153:0/3281783432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc09d060 0x7f4adc09d9b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009910 tx=0x7f4acc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62749fb4a37134e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7dbfff640 1 -- 172.21.15.153:0/2035210342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 msgr2=0x7fb7dc0694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7dbfff640 1 --2- 172.21.15.153:0/2035210342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc0694a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc076600 tx=0x7fb7d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6896d71b6baffaa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f960983c640 1 -- 172.21.15.153:0/3171225824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 msgr2=0x7f960413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f960983c640 1 --2- 172.21.15.153:0/3171225824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 0x7f960413a010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009f60 tx=0x7f95fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cabaa6739d249853 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 1 -- 172.21.15.153:0/1080273984 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8402adb0 con 0x7fdd900971e0 2024-09-15T03:28:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: dump: 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd9595d640 5 monclient: authenticate success, global_id 4243 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd9595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 1 -- 172.21.15.153:0/1080273984 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd84032440 con 0x7fdd900971e0 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd9595d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd9595d640 4 set_mon_vals no callback set 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 1 -- 172.21.15.153:0/1080273984 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd840327e0 con 0x7fdd900971e0 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fdd8dffb640 10 monclient: dump: 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 1 -- 172.21.15.153:0/3281783432 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc032440 con 0x7f4adc09d060 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: dump: 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.238 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 1 -- 172.21.15.153:0/3281783432 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc0325a0 con 0x7f4adc09d060 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 1 -- 172.21.15.153:0/3281783432 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc032940 con 0x7f4adc09d060 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ad959a640 10 monclient: dump: 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ae08c9640 5 monclient: authenticate success, global_id 4246 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ae08c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f4ae08c9640 4 set_mon_vals no callback set 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 1 -- 172.21.15.153:0/2035210342 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4007610 con 0x7fb7dc09d350 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: dump: 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 1 -- 172.21.15.153:0/2035210342 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d4007770 con 0x7fb7dc09d350 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 1 -- 172.21.15.153:0/3171225824 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc007760 con 0x7f9604137c30 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: dump: 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7e219d640 5 monclient: authenticate success, global_id 4248 2024-09-15T03:28:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7e219d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7e219d640 4 set_mon_vals no callback set 2024-09-15T03:28:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 1 -- 172.21.15.153:0/2035210342 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4007b50 con 0x7fb7dc09d350 2024-09-15T03:28:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 1 -- 172.21.15.153:0/3171225824 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc02d050 con 0x7f9604137c30 2024-09-15T03:28:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7fb7da7fc640 10 monclient: dump: 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.246 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 1 -- 172.21.15.153:0/3171225824 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc007af0 con 0x7f9604137c30 2024-09-15T03:28:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f960b29f640 5 monclient: authenticate success, global_id 4249 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f960b29f640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f960b29f640 4 set_mon_vals no callback set 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.180+0000 7f95fbfff640 10 monclient: dump: 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.248 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 10 monclient: shutdown 2024-09-15T03:28:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 1 -- 172.21.15.153:0/1080273984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd900971e0 msgr2=0x7fdd900975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 1 --2- 172.21.15.153:0/1080273984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd900971e0 0x7fdd900975e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdd90071470 tx=0x7fdd840090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 10 monclient: shutdown 2024-09-15T03:28:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3281783432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc09d060 msgr2=0x7f4adc09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3281783432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc09d060 0x7f4adc09d9b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009910 tx=0x7f4acc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 1 -- 172.21.15.153:0/1080273984 shutdown_connections 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 1 --2- 172.21.15.153:0/1080273984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 1 --2- 172.21.15.153:0/1080273984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd900971e0 0x7fdd900975e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fdd9595d640 1 -- 172.21.15.153:0/1080273984 >> 172.21.15.153:0/1080273984 conn(0x7fdd900672b0 msgr2=0x7fdd900676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3281783432 shutdown_connections 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3281783432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc09d060 0x7f4adc09d9b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3281783432 >> v2:172.21.15.153:3300/0 conn(0x7f4adc067820 0x7f4adc09cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3281783432 >> 172.21.15.153:0/3281783432 conn(0x7f4adc0a3a40 msgr2=0x7f4adc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 10 monclient: shutdown 2024-09-15T03:28:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 10 monclient: shutdown 2024-09-15T03:28:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 -- 172.21.15.153:0/2035210342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 msgr2=0x7fb7dc0694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 --2- 172.21.15.153:0/2035210342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc0694a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc076600 tx=0x7fb7d4009840 comp rx=0 tx=0).stop 2024-09-15T03:28:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 1 -- 172.21.15.153:0/3171225824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 msgr2=0x7f960413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 1 --2- 172.21.15.153:0/3171225824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 0x7f960413a010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009f60 tx=0x7f95fc009780 comp rx=0 tx=0).stop 2024-09-15T03:28:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 -- 172.21.15.153:0/2035210342 shutdown_connections 2024-09-15T03:28:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 --2- 172.21.15.153:0/2035210342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc0694a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 --2- 172.21.15.153:0/2035210342 >> v2:172.21.15.181:3300/0 conn(0x7fb7dc09ebd0 0x7fb7dc09efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 -- 172.21.15.153:0/2035210342 >> 172.21.15.153:0/2035210342 conn(0x7fb7dc067110 msgr2=0x7fb7dc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 1 -- 172.21.15.153:0/3171225824 shutdown_connections 2024-09-15T03:28:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 1 --2- 172.21.15.153:0/3171225824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 0x7f960413a010 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f960b29f640 1 -- 172.21.15.153:0/3171225824 >> 172.21.15.153:0/3171225824 conn(0x7f9604067110 msgr2=0x7f9604067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3281783432 shutdown_connections 2024-09-15T03:28:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 -- 172.21.15.153:0/2035210342 shutdown_connections 2024-09-15T03:28:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3281783432 wait complete. 2024-09-15T03:28:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.181+0000 7fb7e219d640 1 -- 172.21.15.153:0/2035210342 wait complete. 2024-09-15T03:28:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7f960b29f640 1 -- 172.21.15.153:0/3171225824 shutdown_connections 2024-09-15T03:28:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7f960b29f640 1 -- 172.21.15.153:0/3171225824 wait complete. 2024-09-15T03:28:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7fb7e219d640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7fdd9595d640 1 -- 172.21.15.153:0/1080273984 shutdown_connections 2024-09-15T03:28:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7fb7e219d640 10 monclient: monmap: 2024-09-15T03:28:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7fdd9595d640 1 -- 172.21.15.153:0/1080273984 wait complete. 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7f4ae08c9640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7f4ae08c9640 10 monclient: monmap: 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7f960b29f640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.182+0000 7f960b29f640 10 monclient: monmap: 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 Processor -- start 2024-09-15T03:28:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fb7e219d640 1 Processor -- start 2024-09-15T03:28:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fb7e219d640 1 -- start start 2024-09-15T03:28:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 Processor -- start 2024-09-15T03:28:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 -- start start 2024-09-15T03:28:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient: init 2024-09-15T03:28:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fb7e219d640 10 monclient: init 2024-09-15T03:28:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 -- start start 2024-09-15T03:28:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient: init 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fdd9595d640 10 monclient: build_initial_monmap 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fdd9595d640 10 monclient: monmap: 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:40.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:40.263 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:40.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc067820 0x7f4adc1413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4adc067820 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4adc09d060 0x7f4adc138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4adc09d060 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 0x7f4adc131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4adc09def0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc09c1f0 con 0x7f4adc09def0 2024-09-15T03:28:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0a2c90 con 0x7f4adc067820 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc135620 con 0x7f4adc09d060 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f4ae08c9640 10 monclient(hunting): authenticate will time out at 717.823901s 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 0x7f960409b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): picked mon.noname-c con 0x7f960409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96041318f0 0x7f9604096f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): picked mon.noname-a con 0x7f96041318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fb7e219d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 0x7f9604097470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): picked mon.noname-b con 0x7f9604137c30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604099a20 con 0x7f96041318f0 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960409bf10 con 0x7f9604137c30 2024-09-15T03:28:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960413a150 con 0x7f960409ea50 2024-09-15T03:28:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fdd9595d640 1 Processor -- start 2024-09-15T03:28:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fb7e219d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7f960b29f640 10 monclient(hunting): authenticate will time out at 717.823909s 2024-09-15T03:28:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.183+0000 7fb7e219d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 0x7fb7dc09a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7dc09cf50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 1 -- start start 2024-09-15T03:28:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc11bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7dc09d350 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 10 monclient: init 2024-09-15T03:28:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 0x7fb7dc11c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 0x7f960409b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7dc09ebd0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc11ddb0 con 0x7fb7dc09d350 2024-09-15T03:28:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 0x7f960409b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44844/0 (socket says 172.21.15.153:44844) 2024-09-15T03:28:40.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 -- 172.21.15.153:0/3117903943 learned_addr learned my addr 172.21.15.153:0/3117903943 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): get_auth_request con 0x7f960409ea50 auth_method 0 2024-09-15T03:28:40.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc11e370 con 0x7fb7dc09ebd0 2024-09-15T03:28:40.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc09ba40 con 0x7fb7dc09cf50 2024-09-15T03:28:40.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7e219d640 10 monclient(hunting): authenticate will time out at 717.824320s 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960883a640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96041318f0 0x7f9604096f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960883a640 10 monclient(hunting): get_auth_request con 0x7f96041318f0 auth_method 0 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960883a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960883a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960883a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 0x7fb7dc11c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 0x7fb7dc11c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:39942/0 (socket says 172.21.15.153:39942) 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 1 -- 172.21.15.153:0/1287580746 learned_addr learned my addr 172.21.15.153:0/1287580746 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): get_auth_request con 0x7fb7dc09ebd0 auth_method 0 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc11bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 0x7fb7dc09a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): get_auth_request con 0x7fb7dc09d350 auth_method 0 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): get_auth_request con 0x7fb7dc09cf50 auth_method 0 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 -- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 msgr2=0x7f9604097470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 0x7f9604097470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 -- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96041318f0 msgr2=0x7f9604096f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96041318f0 0x7f9604096f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f960903b640 1 -- 172.21.15.153:0/3117903943 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc009a20 con 0x7f960409ea50 2024-09-15T03:28:40.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:40.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:40.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4adc09d060 0x7f4adc138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 0x7fdd9006f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd90096e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:40.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd900971e0 0x7fdd9012fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4adc09d060 0x7f4adc138120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44866/0 (socket says 172.21.15.153:44866) 2024-09-15T03:28:40.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 1 -- 172.21.15.153:0/3835691473 learned_addr learned my addr 172.21.15.153:0/3835691473 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 10 monclient(hunting): get_auth_request con 0x7f4adc09d060 auth_method 0 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 1 -- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 msgr2=0x7fb7dc11c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 0x7fb7dc11c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 1 -- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 msgr2=0x7fb7dc11bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7fdd9595d640 10 monclient(hunting): picked mon.noname-a con 0x7fdd900971e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:40.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90130380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.184+0000 7f4adad9d640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 0x7f4adc131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): get_auth_request con 0x7f4adc09def0 auth_method 0 2024-09-15T03:28:40.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd90133b30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90069820 con 0x7fdd900971e0 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd901336a0 con 0x7fdd90096e10 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90069e40 con 0x7fdd90133b30 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd9595d640 10 monclient(hunting): authenticate will time out at 717.825215s 2024-09-15T03:28:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd900971e0 0x7fdd9012fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd900971e0 0x7fdd9012fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49604/0 (socket says 172.21.15.153:49604) 2024-09-15T03:28:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc11bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 1 monclient: found mon.noname-c 2024-09-15T03:28:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 1 -- 172.21.15.153:0/3092551919 learned_addr learned my addr 172.21.15.153:0/3092551919 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:40.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7db7fe640 1 -- 172.21.15.153:0/1287580746 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4009ae0 con 0x7fb7dc09cf50 2024-09-15T03:28:40.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7dbfff640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 0x7fb7dc11c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:40.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fb7daffd640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc11bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): get_auth_request con 0x7fdd900971e0 auth_method 0 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 0x7fdd9006f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): get_auth_request con 0x7fdd90096e10 auth_method 0 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4ada59c640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc067820 0x7f4adc1413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4ada59c640 10 monclient(hunting): get_auth_request con 0x7f4adc067820 auth_method 0 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4ada59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4ada59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:40.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4ada59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:40.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 1 -- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4adc09d060 msgr2=0x7f4adc138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4adc09d060 0x7f4adc138120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 1 -- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc067820 msgr2=0x7f4adc1413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc067820 0x7f4adc1413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 1 monclient: found mon.noname-a 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7f4adad9d640 1 -- 172.21.15.153:0/3835691473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc009450 con 0x7f4adc09def0 2024-09-15T03:28:40.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:40.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:40.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-09-15T03:28:40.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 -- 172.21.15.153:0/3092551919 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 msgr2=0x7fdd90130380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90130380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 -- 172.21.15.153:0/3092551919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd900971e0 msgr2=0x7fdd9012fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd900971e0 0x7fdd9012fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 monclient: found mon.noname-b 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.185+0000 7fdd8effd640 1 -- 172.21.15.153:0/3092551919 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84009390 con 0x7fdd90096e10 2024-09-15T03:28:40.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.186+0000 7f960903b640 1 -- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 msgr2=0x7f960409b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.186+0000 7f960903b640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 0x7f960409b470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c7c0 tx=0x7f95f4007be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f400ce90 con 0x7f960409ea50 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: dump: 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.292 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.293 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.293 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f4005aa0 con 0x7f960409ea50 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4005e20 con 0x7f960409ea50 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95f97fa640 10 monclient: dump: 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.294 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 5 monclient: authenticate success, global_id 4253 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f95fb7fe640 4 set_mon_vals no callback set 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 10 monclient: _renew_subs 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9604097db0 con 0x7f960409ea50 2024-09-15T03:28:40.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7db7fe640 1 -- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 msgr2=0x7fb7dc09a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7db7fe640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 0x7fb7dc09a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e960 tx=0x7fb7cc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 10 monclient: _renew_subs 2024-09-15T03:28:40.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9604130da0 con 0x7f960409ea50 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00cb50 con 0x7fb7dc09cf50 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: dump: 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.297 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.298 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.298 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.298 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.298 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.298 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc00ccb0 con 0x7fb7dc09cf50 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7e219d640 5 monclient: authenticate success, global_id 4256 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7e219d640 10 monclient: _renew_subs 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7e219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc11c600 con 0x7fb7dc09cf50 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc013680 con 0x7fb7dc09cf50 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:40.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7c37fe640 10 monclient: dump: 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fb7e219d640 10 monclient: _renew_subs 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fb7e219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc11cd90 con 0x7fb7dc09cf50 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.187+0000 7fb7d9ffb640 4 set_mon_vals no callback set 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f95f4007300 con 0x7f960409ea50 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f960b29f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f960b29f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f960b29f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960404bf60 con 0x7f960409ea50 2024-09-15T03:28:40.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f95f97fa640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95e8038790 0x7f95e803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f95f404b480 con 0x7f960409ea50 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd8effd640 1 -- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 msgr2=0x7fdd9006f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd8effd640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 0x7fdd9006f170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000b490 tx=0x7fdd80006f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8000f720 con 0x7fdd90096e10 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 10 monclient: dump: 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.302 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8000f880 con 0x7fdd90096e10 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7fdd76ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fdd8d7fa640 4 set_mon_vals no callback set 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4adad9d640 1 -- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 msgr2=0x7f4adc131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4adad9d640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 0x7f4adc131d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c810 tx=0x7f4ad40025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400a850 con 0x7f4adc09def0 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 10 monclient: dump: 2024-09-15T03:28:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad400a9b0 con 0x7f4adc09def0 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4aca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:40.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.188+0000 7f4ad8d99640 4 set_mon_vals no callback set 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400ad70 con 0x7f4adc09def0 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4aca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4aca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4aca7fc640 10 monclient: dump: 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.306 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 5 monclient: authenticate success, global_id 4266 2024-09-15T03:28:40.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 10 monclient: _renew_subs 2024-09-15T03:28:40.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fb7e219d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fb7e219d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fb7e219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4adc133830 con 0x7f4adc09def0 2024-09-15T03:28:40.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc04bf60 con 0x7fb7dc09cf50 2024-09-15T03:28:40.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 10 monclient: _renew_subs 2024-09-15T03:28:40.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc141a90 con 0x7f4adc09def0 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8000fc40 con 0x7fdd90096e10 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.189+0000 7fdd76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd76ffd640 10 monclient: dump: 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:40.309 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 5 monclient: authenticate success, global_id 4264 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 10 monclient: _renew_subs 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd90132080 con 0x7fdd90096e10 2024-09-15T03:28:40.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7f960883a640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95e8038790 0x7f95e803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 10 monclient: _renew_subs 2024-09-15T03:28:40.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9009e790 con 0x7fdd90096e10 2024-09-15T03:28:40.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.190+0000 7f960883a640 10 monclient: get_auth_request con 0x7f95e8038790 auth_method 0 2024-09-15T03:28:40.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4ae08c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4ae08c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4ae08c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc04bf60 con 0x7f4adc09def0 2024-09-15T03:28:40.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4ad4012630 con 0x7f4adc09def0 2024-09-15T03:28:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7fdd9595d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4aca7fc640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4abc038790 0x7f4abc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7fdd9595d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7fdd9595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd90131960 con 0x7fdd90096e10 2024-09-15T03:28:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f4ad404bc40 con 0x7f4adc09def0 2024-09-15T03:28:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdd80011610 con 0x7fdd90096e10 2024-09-15T03:28:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.191+0000 7fdd76ffd640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd7c038790 0x7fdd7c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.192+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fdd8004b210 con 0x7fdd90096e10 2024-09-15T03:28:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.193+0000 7f960883a640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95e8038790 0x7f95e803ac40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96041318c0 tx=0x7f9600009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.193+0000 7f4ada59c640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4abc038790 0x7f4abc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.193+0000 7f4ada59c640 10 monclient: get_auth_request con 0x7f4abc038790 auth_method 0 2024-09-15T03:28:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.195+0000 7f4ada59c640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4abc038790 0x7f4abc03ac40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0009c10 tx=0x7f4ad0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.197+0000 7fdd8e7fc640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd7c038790 0x7fdd7c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.197+0000 7fdd8e7fc640 10 monclient: get_auth_request con 0x7fdd7c038790 auth_method 0 2024-09-15T03:28:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.197+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdd8000fe60 con 0x7fdd90096e10 2024-09-15T03:28:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.197+0000 7fdd76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.197+0000 7fdd76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:40.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.198+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f95f40075b0 con 0x7f960409ea50 2024-09-15T03:28:40.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.198+0000 7f95f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.198+0000 7f95f97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:40.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.198+0000 7fb7db7fe640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 0x7fb7dc09a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e960 tx=0x7fb7cc0090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb7cc013a70 mgrmap(e 5) v1 2024-09-15T03:28:40.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.200+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4ad4052080 con 0x7f4adc09def0 2024-09-15T03:28:40.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.200+0000 7f4aca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:40.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.200+0000 7f4aca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:40.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.200+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb7cc013a70 con 0x7fb7dc09cf50 2024-09-15T03:28:40.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.201+0000 7fb7c37fe640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7c8038790 0x7fb7c803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:40.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.200+0000 7fdd8e7fc640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd7c038790 0x7fdd7c03ac40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009130 tx=0x7fdd84032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.202+0000 7fb7daffd640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7c8038790 0x7fb7c803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:40.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.202+0000 7fb7daffd640 10 monclient: get_auth_request con 0x7fb7c8038790 auth_method 0 2024-09-15T03:28:40.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.204+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fb7cc04c3d0 con 0x7fb7dc09cf50 2024-09-15T03:28:40.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.205+0000 7fb7daffd640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7c8038790 0x7fb7c803ac40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0007fd0 tx=0x7fb7d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7ca6d5d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7ca6d5d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7ca6d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7ca00638d0 con 0x7f7ca0136c20 2024-09-15T03:28:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7c9cff9640 1 -- 172.21.15.153:0/2321046386 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c9803abd0 con 0x7f7ca0136c20 2024-09-15T03:28:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7c9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.326+0000 7f7c9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:40.329 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7c84038790 msgr2=0x7f7c8403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7c84038790 0x7f7c8403ac40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c94007fd0 tx=0x7f7c94007e10 comp rx=0 tx=0).stop 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 10 monclient: shutdown 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 msgr2=0x7f7ca0099c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 0x7f7ca0099c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009e90 tx=0x7f7c98037000 comp rx=0 tx=0).stop 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 shutdown_connections 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7c84038790 0x7f7c8403ac40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca0136c20 0x7f7ca0099c70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca01318f0 0x7f7ca0099730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 --2- 172.21.15.153:0/2321046386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca009ea50 0x7f7ca006e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 >> 172.21.15.153:0/2321046386 conn(0x7f7ca0067110 msgr2=0x7f7ca00a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 shutdown_connections 2024-09-15T03:28:40.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.328+0000 7f7ca6d5d640 1 -- 172.21.15.153:0/2321046386 wait complete. 2024-09-15T03:28:40.341 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.2 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.373+0000 7f24f5554640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.373+0000 7f24f5554640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.373+0000 7f24f5554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.373+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f24f0134330 con 0x7f24f01318f0 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.374+0000 7f24d6ffd640 1 -- 172.21.15.153:0/548542714 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f24e402b090 con 0x7f24f01318f0 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.374+0000 7f24d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.374+0000 7f24d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:40.376 INFO:teuthology.orchestra.run.smithi153.stdout:47244640257 2024-09-15T03:28:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f24dc038790 msgr2=0x7f24dc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f24dc038790 0x7f24dc03ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24e0031000 tx=0x7f24e0032000 comp rx=0 tx=0).stop 2024-09-15T03:28:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 10 monclient: shutdown 2024-09-15T03:28:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 msgr2=0x7f24f0132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 0x7f24f0132f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c4e0 tx=0x7f24e4007e10 comp rx=0 tx=0).stop 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 shutdown_connections 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f24dc038790 0x7f24dc03ac40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0136c20 0x7f24f0133470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f01318f0 0x7f24f0132f30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 --2- 172.21.15.153:0/548542714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24f009ea50 0x7f24f01329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 >> 172.21.15.153:0/548542714 conn(0x7f24f0067110 msgr2=0x7f24f012bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 shutdown_connections 2024-09-15T03:28:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.377+0000 7f24f5554640 1 -- 172.21.15.153:0/548542714 wait complete. 2024-09-15T03:28:40.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.4 2024-09-15T03:28:40.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f960b29f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f960b29f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f960b29f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9604099740 con 0x7f960409ea50 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f95f97fa640 1 -- 172.21.15.153:0/3117903943 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95f4019300 con 0x7f960409ea50 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f95f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.415+0000 7f95f97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:40.418 INFO:teuthology.orchestra.run.smithi153.stdout:47244640257 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95e8038790 msgr2=0x7f95e803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95e8038790 0x7f95e803ac40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96041318c0 tx=0x7f9600009040 comp rx=0 tx=0).stop 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 10 monclient: shutdown 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 msgr2=0x7f960409b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 0x7f960409b470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c7c0 tx=0x7f95f4007be0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 shutdown_connections 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95e8038790 0x7f95e803ac40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9604137c30 0x7f9604097470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96041318f0 0x7f9604096f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 --2- 172.21.15.153:0/3117903943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f960409ea50 0x7f960409b470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.417+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 >> 172.21.15.153:0/3117903943 conn(0x7f9604067110 msgr2=0x7f960409d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.418+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 shutdown_connections 2024-09-15T03:28:40.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.418+0000 7f960b29f640 1 -- 172.21.15.153:0/3117903943 wait complete. 2024-09-15T03:28:40.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.1 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.454+0000 7fdd9595d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.454+0000 7fdd9595d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.454+0000 7fdd9595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.454+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdd9004bf60 con 0x7fdd90096e10 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.455+0000 7fdd76ffd640 1 -- 172.21.15.153:0/3092551919 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdd800172b0 con 0x7fdd90096e10 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.455+0000 7fdd76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.455+0000 7fdd76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:40.457 INFO:teuthology.orchestra.run.smithi153.stdout:47244640257 2024-09-15T03:28:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd7c038790 msgr2=0x7fdd7c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd7c038790 0x7fdd7c03ac40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009130 tx=0x7fdd84032000 comp rx=0 tx=0).stop 2024-09-15T03:28:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 10 monclient: shutdown 2024-09-15T03:28:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 msgr2=0x7fdd9006f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 0x7fdd9006f170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000b490 tx=0x7fdd80006f50 comp rx=0 tx=0).stop 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 shutdown_connections 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd7c038790 0x7fdd7c03ac40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd90133b30 0x7fdd90130380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd900971e0 0x7fdd9012fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 --2- 172.21.15.153:0/3092551919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd90096e10 0x7fdd9006f170 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 >> 172.21.15.153:0/3092551919 conn(0x7fdd900672b0 msgr2=0x7fdd90134440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 shutdown_connections 2024-09-15T03:28:40.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.457+0000 7fdd9595d640 1 -- 172.21.15.153:0/3092551919 wait complete. 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4ae08c9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4ae08c9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4ae08c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4adc06ee20 con 0x7f4adc09def0 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4aca7fc640 1 -- 172.21.15.153:0/3835691473 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ad4016020 con 0x7f4adc09def0 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4aca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:40.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.460+0000 7f4aca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:40.463 INFO:teuthology.orchestra.run.smithi153.stdout:47244640257 2024-09-15T03:28:40.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4abc038790 msgr2=0x7f4abc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4abc038790 0x7f4abc03ac40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0009c10 tx=0x7f4ad0008040 comp rx=0 tx=0).stop 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 10 monclient: shutdown 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 msgr2=0x7f4adc131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 0x7f4adc131d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c810 tx=0x7f4ad40025b0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 shutdown_connections 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4abc038790 0x7f4abc03ac40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4adc09def0 0x7f4adc131d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4adc09d060 0x7f4adc138120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 --2- 172.21.15.153:0/3835691473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc067820 0x7f4adc1413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 >> 172.21.15.153:0/3835691473 conn(0x7f4adc0a3a40 msgr2=0x7f4adc069670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 shutdown_connections 2024-09-15T03:28:40.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:40.462+0000 7f4ae08c9640 1 -- 172.21.15.153:0/3835691473 wait complete. 2024-09-15T03:28:40.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.3 2024-09-15T03:28:40.475 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.0 2024-09-15T03:28:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.027+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb7cc01add0 con 0x7fb7dc09cf50 2024-09-15T03:28:41.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.027+0000 7fb7c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.027+0000 7fb7c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.183+0000 7fb7e219d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.183+0000 7fb7e219d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.183+0000 7fb7e219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.183+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb7dc0638d0 con 0x7fb7dc09cf50 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.184+0000 7fb7c37fe640 1 -- 172.21.15.153:0/1287580746 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb7cc021020 con 0x7fb7dc09cf50 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.184+0000 7fb7c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:28:41.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.184+0000 7fb7c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:41.186 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:41.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7c8038790 msgr2=0x7fb7c803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7c8038790 0x7fb7c803ac40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0007fd0 tx=0x7fb7d0009040 comp rx=0 tx=0).stop 2024-09-15T03:28:41.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 10 monclient: shutdown 2024-09-15T03:28:41.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 msgr2=0x7fb7dc09a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 0x7fb7dc09a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e960 tx=0x7fb7cc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 shutdown_connections 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7c8038790 0x7fb7c803ac40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7dc09ebd0 0x7fb7dc11c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7dc09d350 0x7fb7dc11bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 --2- 172.21.15.153:0/1287580746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7dc09cf50 0x7fb7dc09a880 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 >> 172.21.15.153:0/1287580746 conn(0x7fb7dc067110 msgr2=0x7fb7dc09d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 shutdown_connections 2024-09-15T03:28:41.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.186+0000 7fb7e219d640 1 -- 172.21.15.153:0/1287580746 wait complete. 2024-09-15T03:28:41.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-09-15T03:28:41.198 DEBUG:teuthology.parallel:result is None 2024-09-15T03:28:41.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:28:41.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:28:41.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:28:41.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.437+0000 7fe09431d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:41.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.437+0000 7fe09431d640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.437+0000 7fe09431d640 10 monclient: monmap: 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:41.437868+0000 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:41.437868+0000 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.439 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.440 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:41.440 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:41.440 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:41.440 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:41.440 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:41.441 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.437+0000 7fe09431d640 1 Processor -- start 2024-09-15T03:28:41.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.437+0000 7fe09431d640 1 -- start start 2024-09-15T03:28:41.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.437+0000 7fe09431d640 10 monclient: init 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe08c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe08c09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 0x7fe08c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient(hunting): picked mon.noname-b con 0x7fe08c137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c132640 con 0x7fe08c09ea50 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c131cc0 con 0x7fe08c1318f0 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c09f530 con 0x7fe08c137c30 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe09431d640 10 monclient(hunting): authenticate will time out at 719.077396s 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0918b8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe08c09ea50 0x7fe08c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48380/0 (socket says 172.21.15.153:48380) 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0918b8640 1 -- 172.21.15.153:0/4062417467 learned_addr learned my addr 172.21.15.153:0/4062417467 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 1 --2- 172.21.15.153:0/4062417467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 0x7fe08c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): get_auth_request con 0x7fe08c137c30 auth_method 0 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 1 -- 172.21.15.153:0/4062417467 >> v1:172.21.15.181:6789/0 conn(0x7fe08c1318f0 legacy=0x7fe08c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 1 -- 172.21.15.153:0/4062417467 >> v1:172.21.15.153:6789/0 conn(0x7fe08c09ea50 legacy=0x7fe08c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 1 monclient: found mon.noname-b 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.438+0000 7fe0928ba640 1 -- 172.21.15.153:0/4062417467 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c09fbf0 con 0x7fe08c137c30 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0928ba640 1 -- 172.21.15.153:0/4062417467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 msgr2=0x7fe08c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0928ba640 1 --2- 172.21.15.153:0/4062417467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 0x7fe08c13a010 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe084009e90 tx=0x7fe084009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0da3acc4b2f6dce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 1 -- 172.21.15.153:0/4062417467 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084007840 con 0x7fe08c137c30 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: dump: 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.453 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe09431d640 5 monclient: authenticate success, global_id 4262 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe09431d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 1 -- 172.21.15.153:0/4062417467 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08402d050 con 0x7fe08c137c30 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe09431d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe09431d640 4 set_mon_vals no callback set 2024-09-15T03:28:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 1 -- 172.21.15.153:0/4062417467 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084007bd0 con 0x7fe08c137c30 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.441+0000 7fe0910b7640 10 monclient: dump: 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 10 monclient: shutdown 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 -- 172.21.15.153:0/4062417467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 msgr2=0x7fe08c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 --2- 172.21.15.153:0/4062417467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 0x7fe08c13a010 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe084009e90 tx=0x7fe084009780 comp rx=0 tx=0).stop 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 -- 172.21.15.153:0/4062417467 shutdown_connections 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 --2- 172.21.15.153:0/4062417467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c137c30 0x7fe08c13a010 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 -- 172.21.15.153:0/4062417467 >> 172.21.15.153:0/4062417467 conn(0x7fe08c067110 msgr2=0x7fe08c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 -- 172.21.15.153:0/4062417467 shutdown_connections 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.442+0000 7fe09431d640 1 -- 172.21.15.153:0/4062417467 wait complete. 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient: monmap: 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:41.456 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 Processor -- start 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 -- start start 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient: init 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe08c09ea50 0x7fe08c136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): picked mon.noname-b con 0x7fe08c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 0x7fe08c12fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): picked mon.noname-c con 0x7fe08c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 0x7fe08c130410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): picked mon.noname-a con 0x7fe08c137c30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1347f0 con 0x7fe08c137c30 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c13a2c0 con 0x7fe08c09ea50 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c134f20 con 0x7fe08c1318f0 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.443+0000 7fe09431d640 10 monclient(hunting): authenticate will time out at 719.082938s 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 0x7fe08c12fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 0x7fe08c130410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 0x7fe08c12fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44882/0 (socket says 172.21.15.153:44882) 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 1 -- 172.21.15.153:0/1142579680 learned_addr learned my addr 172.21.15.153:0/1142579680 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): get_auth_request con 0x7fe08c1318f0 auth_method 0 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): get_auth_request con 0x7fe08c137c30 auth_method 0 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe08c09ea50 0x7fe08c136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): get_auth_request con 0x7fe08c09ea50 auth_method 0 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0920b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 -- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 msgr2=0x7fe08c12fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 0x7fe08c12fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 -- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe08c09ea50 msgr2=0x7fe08c136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe08c09ea50 0x7fe08c136310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 monclient: found mon.noname-a 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0928ba640 1 -- 172.21.15.153:0/1142579680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084009a20 con 0x7fe08c137c30 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.444+0000 7fe0918b8640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 0x7fe08c12fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.446+0000 7fe0928ba640 1 -- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 msgr2=0x7fe08c130410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.446+0000 7fe0928ba640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 0x7fe08c130410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe084031000 tx=0x7fe084037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.446+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084007ec0 con 0x7fe08c137c30 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.446+0000 7fe0827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.446+0000 7fe0827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.446+0000 7fe0827fc640 10 monclient: dump: 2024-09-15T03:28:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe084040070 con 0x7fe08c137c30 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 5 monclient: authenticate success, global_id 4272 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 10 monclient: _renew_subs 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084045430 con 0x7fe08c137c30 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c134510 con 0x7fe08c137c30 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0908b6640 4 set_mon_vals no callback set 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe0827fc640 10 monclient: dump: 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.463 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 10 monclient: _renew_subs 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.447+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c1329d0 con 0x7fe08c137c30 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe084042030 con 0x7fe08c137c30 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe09431d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe09431d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe09431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c134320 con 0x7fe08c137c30 2024-09-15T03:28:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe0827fc640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe070038790 0x7fe07003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fe08406d3b0 con 0x7fe08c137c30 2024-09-15T03:28:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe0920b9640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe070038790 0x7fe07003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.448+0000 7fe0920b9640 10 monclient: get_auth_request con 0x7fe070038790 auth_method 0 2024-09-15T03:28:41.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.451+0000 7fe0920b9640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe070038790 0x7fe07003ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe088007fd0 tx=0x7fe088007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.453+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe0840305d0 con 0x7fe08c137c30 2024-09-15T03:28:41.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.453+0000 7fe0827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.453+0000 7fe0827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:41.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:28:41.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:28:41.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.495+0000 7ff8d6709640 10 monclient: get_monmap_and_config 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.495+0000 7ff8d6709640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.495+0000 7ff8d6709640 10 monclient: monmap: 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:41.496389+0000 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:41.496389+0000 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:41.497 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:41.498 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:41.498 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:41.498 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 1 Processor -- start 2024-09-15T03:28:41.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 1 -- start start 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 10 monclient: init 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 0x7ff8d0098500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.496+0000 7ff8d6709640 10 monclient(hunting): picked mon.noname-b con 0x7ff8d0098130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8d0131c80 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff8d0132050 0x7ff8d0067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d0132050 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d009e300 con 0x7ff8d0131c80 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d0132450 con 0x7ff8d0132050 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0098a40 con 0x7ff8d0098130 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d6709640 10 monclient(hunting): authenticate will time out at 719.136144s 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 0x7ff8d0098500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cf7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff8d0131c80 0x7ff8d0067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48390/0 (socket says 172.21.15.153:48390) 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 0x7ff8d0098500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44898/0 (socket says 172.21.15.153:44898) 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cf7fe640 1 -- 172.21.15.153:0/92688081 learned_addr learned my addr 172.21.15.153:0/92688081 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): get_auth_request con 0x7ff8d0098130 auth_method 0 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 1 --2- 172.21.15.153:0/92688081 >> v2:172.21.15.181:3300/0 conn(0x7ff8d0132050 0x7ff8d0067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): get_auth_request con 0x7ff8d0132050 auth_method 0 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8d4ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 -- 172.21.15.153:0/92688081 >> v2:172.21.15.181:3300/0 conn(0x7ff8d0132050 msgr2=0x7ff8d0067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 --2- 172.21.15.153:0/92688081 >> v2:172.21.15.181:3300/0 conn(0x7ff8d0132050 0x7ff8d0067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 -- 172.21.15.153:0/92688081 >> v1:172.21.15.153:6789/0 conn(0x7ff8d0131c80 legacy=0x7ff8d0067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 monclient: found mon.noname-b 2024-09-15T03:28:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.497+0000 7ff8cffff640 1 -- 172.21.15.153:0/92688081 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d009e020 con 0x7ff8d0098130 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.500+0000 7ff8cffff640 1 -- 172.21.15.153:0/92688081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 msgr2=0x7ff8d0098500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.500+0000 7ff8cffff640 1 --2- 172.21.15.153:0/92688081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 0x7ff8d0098500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00767f0 tx=0x7ff8c00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67d9030a114c7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 1 -- 172.21.15.153:0/92688081 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c002adb0 con 0x7ff8d0098130 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: dump: 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 1 -- 172.21.15.153:0/92688081 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c0032440 con 0x7ff8d0098130 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 1 -- 172.21.15.153:0/92688081 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c00327e0 con 0x7ff8d0098130 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:41.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8ceffd640 10 monclient: dump: 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8d6709640 5 monclient: authenticate success, global_id 4268 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8d6709640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:41.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.501+0000 7ff8d6709640 4 set_mon_vals no callback set 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 10 monclient: shutdown 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 -- 172.21.15.153:0/92688081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 msgr2=0x7ff8d0098500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 --2- 172.21.15.153:0/92688081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 0x7ff8d0098500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00767f0 tx=0x7ff8c00090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 -- 172.21.15.153:0/92688081 shutdown_connections 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 --2- 172.21.15.153:0/92688081 >> v2:172.21.15.181:3300/0 conn(0x7ff8d0132050 0x7ff8d0067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 --2- 172.21.15.153:0/92688081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0098130 0x7ff8d0098500 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 -- 172.21.15.153:0/92688081 >> 172.21.15.153:0/92688081 conn(0x7ff8d0069520 msgr2=0x7ff8d00a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 -- 172.21.15.153:0/92688081 shutdown_connections 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.502+0000 7ff8d6709640 1 -- 172.21.15.153:0/92688081 wait complete. 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.503+0000 7ff8d6709640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.503+0000 7ff8d6709640 10 monclient: monmap: 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.516 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.503+0000 7ff8d6709640 1 Processor -- start 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 1 -- start start 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 10 monclient: init 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 0x7ff8d0130ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8cffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 0x7ff8d0130ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8cffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 0x7ff8d0130ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49622/0 (socket says 172.21.15.153:49622) 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d0098130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0131c80 0x7ff8d0131000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8cffff640 1 -- 172.21.15.153:0/3761513284 learned_addr learned my addr 172.21.15.153:0/3761513284 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d0131c80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.504+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8d0132050 0x7ff8d01411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 10 monclient(hunting): picked mon.noname-b con 0x7ff8d0132050 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d011e530 con 0x7ff8d0098130 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00a04d0 con 0x7ff8d0132050 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0098c20 con 0x7ff8d0131c80 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d6709640 10 monclient(hunting): authenticate will time out at 719.144056s 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): get_auth_request con 0x7ff8d0098130 auth_method 0 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0131c80 0x7ff8d0131000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): get_auth_request con 0x7ff8d0131c80 auth_method 0 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8d0132050 0x7ff8d01411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): get_auth_request con 0x7ff8d0132050 auth_method 0 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8d4ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 1 -- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0131c80 msgr2=0x7ff8d0131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0131c80 0x7ff8d0131000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 1 -- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8d0132050 msgr2=0x7ff8d01411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8d0132050 0x7ff8d01411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 1 monclient: found mon.noname-a 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.505+0000 7ff8cffff640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c0009390 con 0x7ff8d0098130 2024-09-15T03:28:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.507+0000 7ff8cffff640 1 -- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 msgr2=0x7ff8d0130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.507+0000 7ff8cffff640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 0x7ff8d0130ac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009880 tx=0x7ff8c0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0031c70 con 0x7ff8d0098130 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: dump: 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 5 monclient: authenticate success, global_id 4275 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 10 monclient: _renew_subs 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d011c0d0 con 0x7ff8d0098130 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 10 monclient: _renew_subs 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d01416f0 con 0x7ff8d0098130 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c0031dd0 con 0x7ff8d0098130 2024-09-15T03:28:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c003c630 con 0x7ff8d0098130 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8b7fff640 10 monclient: dump: 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.508+0000 7ff8ce7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.509+0000 7ff8d6709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.509+0000 7ff8d6709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.509+0000 7ff8d6709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.509+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b0005010 con 0x7ff8d0098130 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.510+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff8c003c9e0 con 0x7ff8d0098130 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.511+0000 7ff8b7fff640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff8bc038740 0x7ff8bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.511+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7ff8c006c3a0 con 0x7ff8d0098130 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.511+0000 7ff8cf7fe640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff8bc038740 0x7ff8bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.511+0000 7ff8cf7fe640 10 monclient: get_auth_request con 0x7ff8bc038740 auth_method 0 2024-09-15T03:28:41.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.514+0000 7ff8cf7fe640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff8bc038740 0x7ff8bc03abf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00a0380 tx=0x7ff8c4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.518+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff8c0042300 con 0x7ff8d0098130 2024-09-15T03:28:41.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.518+0000 7ff8b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.518+0000 7ff8b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:41.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.558+0000 7f2590f54640 10 monclient: get_monmap_and_config 2024-09-15T03:28:41.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.558+0000 7f2590f54640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.558+0000 7f2590f54640 10 monclient: monmap: 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:41.558914+0000 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:41.558914+0000 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:41.560 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:41.561 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:41.561 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.559+0000 7f2590f54640 1 Processor -- start 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.559+0000 7f2590f54640 1 -- start start 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.559+0000 7f2590f54640 10 monclient: init 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f258c1318f0 0x7f258c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): picked mon.noname-a con 0x7f258c1318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f258c09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f258c136c20 0x7f258c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): picked mon.noname-c con 0x7f258c136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:41.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258c09f530 con 0x7f258c09ea50 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258c131d50 con 0x7f258c1318f0 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258c09fbf0 con 0x7f258c136c20 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f2590f54640 10 monclient(hunting): authenticate will time out at 719.199210s 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f258c1318f0 0x7f258c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f258c1318f0 0x7f258c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49624/0 (socket says 172.21.15.153:49624) 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 -- 172.21.15.153:0/1134742047 learned_addr learned my addr 172.21.15.153:0/1134742047 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): get_auth_request con 0x7f258c1318f0 auth_method 0 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258ad9d640 1 --2- 172.21.15.153:0/1134742047 >> v2:172.21.15.181:3300/0 conn(0x7f258c136c20 0x7f258c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258ad9d640 10 monclient(hunting): get_auth_request con 0x7f258c136c20 auth_method 0 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 -- 172.21.15.153:0/1134742047 >> v2:172.21.15.181:3300/0 conn(0x7f258c136c20 msgr2=0x7f258c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 --2- 172.21.15.153:0/1134742047 >> v2:172.21.15.181:3300/0 conn(0x7f258c136c20 0x7f258c139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 -- 172.21.15.153:0/1134742047 >> v1:172.21.15.181:6789/0 conn(0x7f258c09ea50 legacy=0x7f258c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 monclient: found mon.noname-a 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258a59c640 1 -- 172.21.15.153:0/1134742047 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c09fe30 con 0x7f258c1318f0 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.560+0000 7f258ad9d640 1 --2- 172.21.15.153:0/1134742047 >> v2:172.21.15.181:3300/0 conn(0x7f258c136c20 0x7f258c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.563+0000 7f258a59c640 1 -- 172.21.15.153:0/1134742047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c1318f0 msgr2=0x7f258c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.563+0000 7f258a59c640 1 --2- 172.21.15.153:0/1134742047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c1318f0 0x7f258c131f30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f258c09c9e0 tx=0x7f257c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788b784d36e232cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 1 -- 172.21.15.153:0/1134742047 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c032440 con 0x7f258c1318f0 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: dump: 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2590f54640 5 monclient: authenticate success, global_id 4278 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2590f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 1 -- 172.21.15.153:0/1134742047 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c0325a0 con 0x7f258c1318f0 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2590f54640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:41.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2590f54640 4 set_mon_vals no callback set 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 1 -- 172.21.15.153:0/1134742047 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c032980 con 0x7f258c1318f0 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.564+0000 7f2589d9b640 10 monclient: dump: 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.578 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 10 monclient: shutdown 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 1 -- 172.21.15.153:0/1134742047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c1318f0 msgr2=0x7f258c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 1 --2- 172.21.15.153:0/1134742047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c1318f0 0x7f258c131f30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f258c09c9e0 tx=0x7f257c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 1 -- 172.21.15.153:0/1134742047 shutdown_connections 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 1 --2- 172.21.15.153:0/1134742047 >> v2:172.21.15.181:3300/0 conn(0x7f258c136c20 0x7f258c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 1 --2- 172.21.15.153:0/1134742047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c1318f0 0x7f258c131f30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.565+0000 7f2590f54640 1 -- 172.21.15.153:0/1134742047 >> 172.21.15.153:0/1134742047 conn(0x7f258c067110 msgr2=0x7f258c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.566+0000 7f2590f54640 1 -- 172.21.15.153:0/1134742047 shutdown_connections 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.566+0000 7f2590f54640 1 -- 172.21.15.153:0/1134742047 wait complete. 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.566+0000 7f2590f54640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.566+0000 7f2590f54640 10 monclient: monmap: 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 1 Processor -- start 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 1 -- start start 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 10 monclient: init 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 0x7f258c130860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 10 monclient(hunting): picked mon.noname-c con 0x7f258c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.567+0000 7f2590f54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f258c1318f0 0x7f258c099860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): picked mon.noname-b con 0x7f258c1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c136c20 0x7f258c099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): picked mon.noname-a con 0x7f258c136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c09bf10 con 0x7f258c136c20 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c09e010 con 0x7f258c1318f0 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c09b1d0 con 0x7f258c09ea50 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2590f54640 10 monclient(hunting): authenticate will time out at 719.207054s 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 0x7f258c130860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 0x7f258c130860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44914/0 (socket says 172.21.15.153:44914) 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 -- 172.21.15.153:0/3645582647 learned_addr learned my addr 172.21.15.153:0/3645582647 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): get_auth_request con 0x7f258c09ea50 auth_method 0 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2583fff640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f258c1318f0 0x7f258c099860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2583fff640 10 monclient(hunting): get_auth_request con 0x7f258c1318f0 auth_method 0 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2583fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2583fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2583fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258ad9d640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c136c20 0x7f258c099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 -- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f258c1318f0 msgr2=0x7f258c099860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f258c1318f0 0x7f258c099860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 -- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c136c20 msgr2=0x7f258c099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c136c20 0x7f258c099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 monclient: found mon.noname-c 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258a59c640 1 -- 172.21.15.153:0/3645582647 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c009450 con 0x7f258c09ea50 2024-09-15T03:28:41.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f2583fff640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f258c1318f0 0x7f258c099860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:41.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258ad9d640 10 monclient: get_auth_request con 0x7f258c136c20 auth_method 0 2024-09-15T03:28:41.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.568+0000 7f258ad9d640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c136c20 0x7f258c099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:41.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.570+0000 7f8df3ec0640 10 monclient: get_monmap_and_config 2024-09-15T03:28:41.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.570+0000 7f8df3ec0640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.570+0000 7f8df3ec0640 10 monclient: monmap: 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:41.571206+0000 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:41.571206+0000 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:41.585 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f8df3ec0640 1 Processor -- start 2024-09-15T03:28:41.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f258a59c640 1 -- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 msgr2=0x7f258c130860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f258a59c640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 0x7f258c130860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f257c009940 tx=0x7f257c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c031a40 con 0x7f258c09ea50 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: dump: 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.587 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 5 monclient: authenticate success, global_id 4274 2024-09-15T03:28:41.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f8df3ec0640 1 -- start start 2024-09-15T03:28:41.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c031bc0 con 0x7f258c09ea50 2024-09-15T03:28:41.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f8df3ec0640 10 monclient: init 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 10 monclient: _renew_subs 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c11c970 con 0x7f258c09ea50 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c03c400 con 0x7f258c09ea50 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:41.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f25827fc640 10 monclient: dump: 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f258959a640 4 set_mon_vals no callback set 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 10 monclient: _renew_subs 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.571+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258c1374c0 con 0x7f258c09ea50 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8dec1318f0 0x7f8dec131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f8dec1318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8dec09ea50 0x7f8dec09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f8dec09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dec068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8dec1318f0 0x7f8dec131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8dec1318f0 0x7f8dec131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:40000/0 (socket says 172.21.15.153:40000) 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 1 -- 172.21.15.153:0/2102542891 learned_addr learned my addr 172.21.15.153:0/2102542891 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dec09fbf0 con 0x7f8dec068d90 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dec131d50 con 0x7f8dec09ea50 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dec09fe30 con 0x7f8dec1318f0 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df3ec0640 10 monclient(hunting): authenticate will time out at 719.211799s 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 10 monclient(hunting): get_auth_request con 0x7f8dec1318f0 auth_method 0 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df1c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f2590f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df145b640 1 --2- 172.21.15.153:0/2102542891 >> v2:172.21.15.153:3300/0 conn(0x7f8dec09ea50 0x7f8dec09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f2590f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.572+0000 7f8df145b640 10 monclient(hunting): get_auth_request con 0x7f8dec09ea50 auth_method 0 2024-09-15T03:28:41.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f2590f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df145b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df145b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df145b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c11cd60 con 0x7f258c09ea50 2024-09-15T03:28:41.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f257c03c870 con 0x7f258c09ea50 2024-09-15T03:28:41.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df145b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df145b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df145b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 1 -- 172.21.15.153:0/2102542891 >> v2:172.21.15.153:3300/0 conn(0x7f8dec09ea50 msgr2=0x7f8dec09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f25827fc640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f256c038740 0x7f256c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f257c06c2a0 con 0x7f258c09ea50 2024-09-15T03:28:41.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f2583fff640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f256c038740 0x7f256c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f2583fff640 10 monclient: get_auth_request con 0x7f256c038740 auth_method 0 2024-09-15T03:28:41.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 1 --2- 172.21.15.153:0/2102542891 >> v2:172.21.15.153:3300/0 conn(0x7f8dec09ea50 0x7f8dec09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 1 -- 172.21.15.153:0/2102542891 >> v1:172.21.15.153:6789/0 conn(0x7f8dec068d90 legacy=0x7f8dec0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 1 monclient: found mon.noname-c 2024-09-15T03:28:41.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:28:41.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.573+0000 7f8df1c5c640 1 -- 172.21.15.153:0/2102542891 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec0a1210 con 0x7f8dec1318f0 2024-09-15T03:28:41.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.576+0000 7f2583fff640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f256c038740 0x7f256c03abf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f258c09b8f0 tx=0x7f2574008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.576+0000 7f8df1c5c640 1 -- 172.21.15.153:0/2102542891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec1318f0 msgr2=0x7f8dec131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.576+0000 7f8df1c5c640 1 --2- 172.21.15.153:0/2102542891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec1318f0 0x7f8dec131f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8dec076600 tx=0x7f8de80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4e0d3da957501b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 1 -- 172.21.15.153:0/2102542891 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8032440 con 0x7f8dec1318f0 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: dump: 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.610 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 1 -- 172.21.15.153:0/2102542891 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80325a0 con 0x7f8dec1318f0 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df3ec0640 5 monclient: authenticate success, global_id 4282 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df3ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 1 -- 172.21.15.153:0/2102542891 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8032940 con 0x7f8dec1318f0 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df3ec0640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df3ec0640 4 set_mon_vals no callback set 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df0c5a640 10 monclient: dump: 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.611 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.577+0000 7f8df3ec0640 10 monclient: shutdown 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec1318f0 msgr2=0x7f8dec131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/2102542891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec1318f0 0x7f8dec131f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8dec076600 tx=0x7f8de80091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 shutdown_connections 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/2102542891 >> v2:172.21.15.153:3300/0 conn(0x7f8dec09ea50 0x7f8dec09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/2102542891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec1318f0 0x7f8dec131f30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 >> 172.21.15.153:0/2102542891 conn(0x7f8dec067110 msgr2=0x7f8dec067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 shutdown_connections 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 1 -- 172.21.15.153:0/2102542891 wait complete. 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.578+0000 7f8df3ec0640 10 monclient: monmap: 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.613 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 Processor -- start 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 -- start start 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient: init 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec068d90 0x7f8dec06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f8dec068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dec09ea50 0x7f8dec06ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f8dec09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:41.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 0x7f8dec06b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f8dec1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec09bf10 con 0x7f8dec09ea50 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec06d4b0 con 0x7f8dec068d90 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec06de00 con 0x7f8dec1318f0 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.579+0000 7f8df3ec0640 10 monclient(hunting): authenticate will time out at 719.218901s 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 0x7f8dec06b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dec09ea50 0x7f8dec06ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 0x7f8dec06b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44918/0 (socket says 172.21.15.153:44918) 2024-09-15T03:28:41.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 -- 172.21.15.153:0/630703876 learned_addr learned my addr 172.21.15.153:0/630703876 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): get_auth_request con 0x7f8dec1318f0 auth_method 0 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df1c5c640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec068d90 0x7f8dec06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df1c5c640 10 monclient(hunting): get_auth_request con 0x7f8dec068d90 auth_method 0 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df1c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df1c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df1c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): get_auth_request con 0x7f8dec09ea50 auth_method 0 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 -- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec068d90 msgr2=0x7f8dec06e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec068d90 0x7f8dec06e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 -- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dec09ea50 msgr2=0x7f8dec06ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dec09ea50 0x7f8dec06ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 monclient: found mon.noname-c 2024-09-15T03:28:41.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df245d640 1 -- 172.21.15.153:0/630703876 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8009450 con 0x7f8dec1318f0 2024-09-15T03:28:41.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.580+0000 7f8df145b640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dec09ea50 0x7f8dec06ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:41.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.581+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f257c0426c0 con 0x7f258c09ea50 2024-09-15T03:28:41.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.582+0000 7f25827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.582+0000 7f25827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:41.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8df245d640 1 -- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 msgr2=0x7f8dec06b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8df245d640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 0x7f8dec06b050 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8de400cb90 tx=0x7f8de4002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de400a8c0 con 0x7f8dec1318f0 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: dump: 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.622 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de400aa40 con 0x7f8dec1318f0 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8df3ec0640 5 monclient: authenticate success, global_id 4277 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8df3ec0640 10 monclient: _renew_subs 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8df3ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dec0a4420 con 0x7f8dec1318f0 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de400ae00 con 0x7f8dec1318f0 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.583+0000 7f8de1ffb640 10 monclient: dump: 2024-09-15T03:28:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.584+0000 7f8de3fff640 4 set_mon_vals no callback set 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.584+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8de4012630 con 0x7f8dec1318f0 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8de1ffb640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8dd0038790 0x7f8dd003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8df1c5c640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8dd0038790 0x7f8dd003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8df1c5c640 10 monclient: get_auth_request con 0x7f8dd0038790 auth_method 0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8df3ec0640 10 monclient: _renew_subs 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8df3ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dec09b570 con 0x7f8dec1318f0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.585+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8de4018ad0 con 0x7f8dec1318f0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.586+0000 7f8df3ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.586+0000 7f8df3ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.586+0000 7f8df3ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.586+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dec06d660 con 0x7f8dec1318f0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.589+0000 7f8df1c5c640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8dd0038790 0x7f8dd003ac40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8dec09c990 tx=0x7f8de8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.591+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8dec06d660 con 0x7f8dec1318f0 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.591+0000 7f8de1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.591+0000 7f8de1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.602+0000 7fbcd715d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.602+0000 7fbcd715d640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.602+0000 7fbcd715d640 10 monclient: monmap: 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:41.603524+0000 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:41.603524+0000 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.634 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.603+0000 7fbcd715d640 1 Processor -- start 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.603+0000 7fbcd715d640 1 -- start start 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.603+0000 7fbcd715d640 10 monclient: init 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbcd009ea50 0x7fbcd009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd009ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbcd0068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd0131cc0 con 0x7fbcd0068d90 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd009fbf0 con 0x7fbcd009ea50 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd009fe30 con 0x7fbcd01318f0 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd715d640 10 monclient(hunting): authenticate will time out at 719.243364s 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbcd009ea50 0x7fbcd009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbcd009ea50 0x7fbcd009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49672/0 (socket says 172.21.15.153:49672) 2024-09-15T03:28:41.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 1 -- 172.21.15.153:0/4139213814 learned_addr learned my addr 172.21.15.153:0/4139213814 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): get_auth_request con 0x7fbcd009ea50 auth_method 0 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 1 --2- 172.21.15.153:0/4139213814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): get_auth_request con 0x7fbcd01318f0 auth_method 0 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 1 -- 172.21.15.153:0/4139213814 >> v2:172.21.15.153:3300/0 conn(0x7fbcd009ea50 msgr2=0x7fbcd009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 1 --2- 172.21.15.153:0/4139213814 >> v2:172.21.15.153:3300/0 conn(0x7fbcd009ea50 0x7fbcd009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 1 -- 172.21.15.153:0/4139213814 >> v1:172.21.15.181:6789/0 conn(0x7fbcd0068d90 legacy=0x7fbcd00691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 1 monclient: found mon.noname-b 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.604+0000 7fbcd4ef9640 1 -- 172.21.15.153:0/4139213814 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd00a1210 con 0x7fbcd01318f0 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbcd4ef9640 1 -- 172.21.15.153:0/4139213814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 msgr2=0x7fbcd0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbcd4ef9640 1 --2- 172.21.15.153:0/4139213814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd0131f30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009850 tx=0x7fbcc40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a469ec41baa8a09 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbccf7fe640 1 -- 172.21.15.153:0/4139213814 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc402ade0 con 0x7fbcd01318f0 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbccf7fe640 10 monclient: dump: 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.639 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbccf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbcd715d640 5 monclient: authenticate success, global_id 4280 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.607+0000 7fbcd715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbccf7fe640 1 -- 172.21.15.153:0/4139213814 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4032440 con 0x7fbcd01318f0 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbccf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 4 set_mon_vals no callback set 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbccf7fe640 1 -- 172.21.15.153:0/4139213814 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4032820 con 0x7fbcd01318f0 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:41.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbccf7fe640 10 monclient: dump: 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 10 monclient: shutdown 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 -- 172.21.15.153:0/4139213814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 msgr2=0x7fbcd0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 --2- 172.21.15.153:0/4139213814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd0131f30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009850 tx=0x7fbcc40090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 -- 172.21.15.153:0/4139213814 shutdown_connections 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 --2- 172.21.15.153:0/4139213814 >> v2:172.21.15.153:3300/0 conn(0x7fbcd009ea50 0x7fbcd009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 --2- 172.21.15.153:0/4139213814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd0131f30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 -- 172.21.15.153:0/4139213814 >> 172.21.15.153:0/4139213814 conn(0x7fbcd0067110 msgr2=0x7fbcd0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 -- 172.21.15.153:0/4139213814 shutdown_connections 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.608+0000 7fbcd715d640 1 -- 172.21.15.153:0/4139213814 wait complete. 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.609+0000 7fbcd715d640 10 monclient: build_initial_monmap 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.609+0000 7fbcd715d640 10 monclient: monmap: 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:41.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.609+0000 7fbcd715d640 1 Processor -- start 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.609+0000 7fbcd715d640 1 -- start start 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.609+0000 7fbcd715d640 10 monclient: init 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd0068d90 0x7fbcd009ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd0068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:41.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd009ea50 0x7fbcd0130da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd01312e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd009b810 con 0x7fbcd0068d90 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd011c6f0 con 0x7fbcd009ea50 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd009bf10 con 0x7fbcd01318f0 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd715d640 10 monclient(hunting): authenticate will time out at 719.249459s 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd01312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd01312e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44932/0 (socket says 172.21.15.153:44932) 2024-09-15T03:28:41.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 1 -- 172.21.15.153:0/584899563 learned_addr learned my addr 172.21.15.153:0/584899563 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): get_auth_request con 0x7fbcd01318f0 auth_method 0 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd0068d90 0x7fbcd009ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): get_auth_request con 0x7fbcd0068d90 auth_method 0 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd009ea50 0x7fbcd0130da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): get_auth_request con 0x7fbcd009ea50 auth_method 0 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd4ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 1 -- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd009ea50 msgr2=0x7fbcd0130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.610+0000 7fbcd56fa640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd009ea50 0x7fbcd0130da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 1 -- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd0068d90 msgr2=0x7fbcd009ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd0068d90 0x7fbcd009ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 10 monclient: _finish_hunting 0 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 1 monclient: found mon.noname-c 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.611+0000 7fbcd56fa640 1 -- 172.21.15.153:0/584899563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4009390 con 0x7fbcd01318f0 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd56fa640 1 -- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 msgr2=0x7fbcd01312e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd56fa640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd01312e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800c8c0 tx=0x7fbcc8002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800aa00 con 0x7fbcd01318f0 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcccff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcccff9640 10 monclient: dump: 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcccff9640 10 monclient: _finish_auth 0 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 5 monclient: authenticate success, global_id 4283 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 10 monclient: _renew_subs 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd011dec0 con 0x7fbcd01318f0 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 10 monclient: _renew_subs 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.613+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd011e250 con 0x7fbcd01318f0 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc800ab80 con 0x7fbcd01318f0 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcccff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8002780 con 0x7fbcd01318f0 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcccff9640 10 monclient: dump: 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.649 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.614+0000 7fbcceffd640 4 set_mon_vals no callback set 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcd715d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcd715d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcd715d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb0005010 con 0x7fbcd01318f0 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbcc8012420 con 0x7fbcd01318f0 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcccff9640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcb4038740 0x7fbcb403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:41.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.615+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fbcc804bab0 con 0x7fbcd01318f0 2024-09-15T03:28:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.616+0000 7fbcd4ef9640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcb4038740 0x7fbcb403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.616+0000 7fbcd4ef9640 10 monclient: get_auth_request con 0x7fbcb4038740 auth_method 0 2024-09-15T03:28:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.618+0000 7fbcd4ef9640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcb4038740 0x7fbcb403abf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4034000 tx=0x7fbcc4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.624+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbcc8022aa0 con 0x7fbcd01318f0 2024-09-15T03:28:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.624+0000 7fbcccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.624+0000 7fbcccff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:41.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe09431d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:41.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe09431d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:41.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe09431d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe08c134320 con 0x7fe08c137c30 2024-09-15T03:28:41.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe0827fc640 1 -- 172.21.15.153:0/1142579680 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe0840500c0 con 0x7fe08c137c30 2024-09-15T03:28:41.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe0827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:41.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.695+0000 7fe0827fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:41.697 INFO:teuthology.orchestra.run.smithi153.stdout:47244640260 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe070038790 msgr2=0x7fe07003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe070038790 0x7fe07003ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe088007fd0 tx=0x7fe088007e10 comp rx=0 tx=0).stop 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 10 monclient: shutdown 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 msgr2=0x7fe08c130410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 0x7fe08c130410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe084031000 tx=0x7fe084037000 comp rx=0 tx=0).stop 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 shutdown_connections 2024-09-15T03:28:41.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe070038790 0x7fe07003ac40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe08c137c30 0x7fe08c130410 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe08c1318f0 0x7fe08c12fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 --2- 172.21.15.153:0/1142579680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe08c09ea50 0x7fe08c136310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 >> 172.21.15.153:0/1142579680 conn(0x7fe08c067110 msgr2=0x7fe08c12b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 shutdown_connections 2024-09-15T03:28:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.698+0000 7fe09431d640 1 -- 172.21.15.153:0/1142579680 wait complete. 2024-09-15T03:28:41.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8d6709640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:28:41.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8d6709640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:28:41.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8d6709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff8b00052c0 con 0x7ff8d0098130 2024-09-15T03:28:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8b7fff640 1 -- 172.21.15.153:0/3761513284 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff8c0037090 con 0x7ff8d0098130 2024-09-15T03:28:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:28:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.706+0000 7ff8b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:41.709 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:41.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.708+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff8bc038740 msgr2=0x7ff8bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.708+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff8bc038740 0x7ff8bc03abf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00a0380 tx=0x7ff8c4007f00 comp rx=0 tx=0).stop 2024-09-15T03:28:41.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.708+0000 7ff8d6709640 10 monclient: shutdown 2024-09-15T03:28:41.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.708+0000 7ff8d6709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.708+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 msgr2=0x7ff8d0130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.708+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 0x7ff8d0130ac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009880 tx=0x7ff8c0032000 comp rx=0 tx=0).stop 2024-09-15T03:28:41.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 shutdown_connections 2024-09-15T03:28:41.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff8bc038740 0x7ff8bc03abf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8d0132050 0x7ff8d01411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8d0131c80 0x7ff8d0131000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 --2- 172.21.15.153:0/3761513284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8d0098130 0x7ff8d0130ac0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 >> 172.21.15.153:0/3761513284 conn(0x7ff8d0069520 msgr2=0x7ff8d00a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 shutdown_connections 2024-09-15T03:28:41.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.709+0000 7ff8d6709640 1 -- 172.21.15.153:0/3761513284 wait complete. 2024-09-15T03:28:41.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-09-15T03:28:41.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-15T03:28:41.722 DEBUG:teuthology.parallel:result is None 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.766+0000 7f8df3ec0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.766+0000 7f8df3ec0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.767+0000 7f8df3ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.767+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f8dec06d660 con 0x7f8dec1318f0 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.767+0000 7f8de1ffb640 1 -- 172.21.15.153:0/630703876 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8dec06d660 con 0x7f8dec1318f0 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.767+0000 7f8de1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:41.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.767+0000 7f8de1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:47244640258 2024-09-15T03:28:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8dd0038790 msgr2=0x7f8dd003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8dd0038790 0x7f8dd003ac40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8dec09c990 tx=0x7f8de8032000 comp rx=0 tx=0).stop 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 10 monclient: shutdown 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 msgr2=0x7f8dec06b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 0x7f8dec06b050 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8de400cb90 tx=0x7f8de4002640 comp rx=0 tx=0).stop 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 shutdown_connections 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8dd0038790 0x7f8dd003ac40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dec1318f0 0x7f8dec06b050 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dec09ea50 0x7f8dec06ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 --2- 172.21.15.153:0/630703876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dec068d90 0x7f8dec06e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 >> 172.21.15.153:0/630703876 conn(0x7f8dec067110 msgr2=0x7f8dec0a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 shutdown_connections 2024-09-15T03:28:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.769+0000 7f8df3ec0640 1 -- 172.21.15.153:0/630703876 wait complete. 2024-09-15T03:28:41.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-15T03:28:41.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcd715d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:41.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcd715d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:41.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcd715d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbcb00052c0 con 0x7fbcd01318f0 2024-09-15T03:28:41.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcccff9640 1 -- 172.21.15.153:0/584899563 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcc8016020 con 0x7fbcd01318f0 2024-09-15T03:28:41.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:41.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.791+0000 7fbcccff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:41.793 INFO:teuthology.orchestra.run.smithi153.stdout:47244640258 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcb4038740 msgr2=0x7fbcb403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcb4038740 0x7fbcb403abf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4034000 tx=0x7fbcc4032000 comp rx=0 tx=0).stop 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 10 monclient: shutdown 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 msgr2=0x7fbcd01312e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd01312e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800c8c0 tx=0x7fbcc8002640 comp rx=0 tx=0).stop 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 shutdown_connections 2024-09-15T03:28:41.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcb4038740 0x7fbcb403abf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcd01318f0 0x7fbcd01312e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd009ea50 0x7fbcd0130da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 --2- 172.21.15.153:0/584899563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd0068d90 0x7fbcd009ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.793+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 >> 172.21.15.153:0/584899563 conn(0x7fbcd0067110 msgr2=0x7fbcd00a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.794+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 shutdown_connections 2024-09-15T03:28:41.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.794+0000 7fbcd715d640 1 -- 172.21.15.153:0/584899563 wait complete. 2024-09-15T03:28:41.805 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2024-09-15T03:28:41.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f2590f54640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:28:41.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f2590f54640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:28:41.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f2590f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:41.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f258c11cb60 con 0x7f258c09ea50 2024-09-15T03:28:41.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f25827fc640 1 -- 172.21.15.153:0/3645582647 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f257c037090 con 0x7f258c09ea50 2024-09-15T03:28:41.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f25827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:28:41.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.814+0000 7f25827fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:41.816 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:41.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.816+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f256c038740 msgr2=0x7f256c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.816+0000 7f2590f54640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f256c038740 0x7f256c03abf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f258c09b8f0 tx=0x7f2574008040 comp rx=0 tx=0).stop 2024-09-15T03:28:41.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.816+0000 7f2590f54640 10 monclient: shutdown 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.816+0000 7f2590f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.816+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 msgr2=0x7f258c130860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.816+0000 7f2590f54640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 0x7f258c130860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f257c009940 tx=0x7f257c032000 comp rx=0 tx=0).stop 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 shutdown_connections 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f256c038740 0x7f256c03abf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f258c136c20 0x7f258c099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f258c1318f0 0x7f258c099860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 --2- 172.21.15.153:0/3645582647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f258c09ea50 0x7f258c130860 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 >> 172.21.15.153:0/3645582647 conn(0x7f258c067110 msgr2=0x7f258c0a3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 shutdown_connections 2024-09-15T03:28:41.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:41.817+0000 7f2590f54640 1 -- 172.21.15.153:0/3645582647 wait complete. 2024-09-15T03:28:41.828 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-09-15T03:28:41.828 DEBUG:teuthology.parallel:result is None 2024-09-15T03:28:42.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:28:42.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:28:42.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.824+0000 7ff3a631d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.824+0000 7ff3a631d640 10 monclient: build_initial_monmap 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.824+0000 7ff3a631d640 10 monclient: monmap: 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:42.824779+0000 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:42.824779+0000 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:42.826 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.824+0000 7ff3a631d640 1 Processor -- start 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.824+0000 7ff3a631d640 1 -- start start 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.824+0000 7ff3a631d640 10 monclient: init 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3a01318f0 0x7ff3a0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a01318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:42.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3a009ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3a0136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a009f530 con 0x7ff3a009ea50 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a009fbf0 con 0x7ff3a0136c20 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a0131d50 con 0x7ff3a01318f0 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff3a631d640 10 monclient(hunting): authenticate will time out at 720.464474s 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3a01318f0 0x7ff3a0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff3a009ea50 0x7ff3a009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48408/0 (socket says 172.21.15.153:48408) 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39f7fe640 1 -- 172.21.15.153:0/514420805 learned_addr learned my addr 172.21.15.153:0/514420805 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): get_auth_request con 0x7ff3a01318f0 auth_method 0 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 1 -- 172.21.15.153:0/514420805 >> v1:172.21.15.181:6789/0 conn(0x7ff3a0136c20 legacy=0x7ff3a0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 1 -- 172.21.15.153:0/514420805 >> v1:172.21.15.153:6789/0 conn(0x7ff3a009ea50 legacy=0x7ff3a009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:42.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 1 monclient: found mon.noname-a 2024-09-15T03:28:42.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:42.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:42.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.825+0000 7ff39ffff640 1 -- 172.21.15.153:0/514420805 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a009fe30 con 0x7ff3a01318f0 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.828+0000 7ff39ffff640 1 -- 172.21.15.153:0/514420805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a01318f0 msgr2=0x7ff3a0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.828+0000 7ff39ffff640 1 --2- 172.21.15.153:0/514420805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a01318f0 0x7ff3a0131f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff390009850 tx=0x7ff3900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf69e82da66fcae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.828+0000 7ff39effd640 1 -- 172.21.15.153:0/514420805 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390032440 con 0x7ff3a01318f0 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: dump: 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: _finish_auth 0 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 1 -- 172.21.15.153:0/514420805 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3900325a0 con 0x7ff3a01318f0 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 5 monclient: authenticate success, global_id 4293 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 1 -- 172.21.15.153:0/514420805 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390032940 con 0x7ff3a01318f0 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 4 set_mon_vals no callback set 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff39effd640 10 monclient: dump: 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 10 monclient: shutdown 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 -- 172.21.15.153:0/514420805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a01318f0 msgr2=0x7ff3a0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 --2- 172.21.15.153:0/514420805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a01318f0 0x7ff3a0131f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff390009850 tx=0x7ff3900090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 -- 172.21.15.153:0/514420805 shutdown_connections 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 --2- 172.21.15.153:0/514420805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a01318f0 0x7ff3a0131f30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 -- 172.21.15.153:0/514420805 >> 172.21.15.153:0/514420805 conn(0x7ff3a0067110 msgr2=0x7ff3a0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 -- 172.21.15.153:0/514420805 shutdown_connections 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.829+0000 7ff3a631d640 1 -- 172.21.15.153:0/514420805 wait complete. 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.830+0000 7ff3a631d640 10 monclient: build_initial_monmap 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.830+0000 7ff3a631d640 10 monclient: monmap: 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.830+0000 7ff3a631d640 1 Processor -- start 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.830+0000 7ff3a631d640 1 -- start start 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient: init 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 0x7ff3a0132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a01318f0 0x7ff3a01330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a01318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:42.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 0x7ff3a01335e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0134890 con 0x7ff3a009ea50 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0134490 con 0x7ff3a01318f0 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0135ee0 con 0x7ff3a0136c20 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a631d640 10 monclient(hunting): authenticate will time out at 720.470474s 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 0x7ff3a0132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 0x7ff3a0132b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49702/0 (socket says 172.21.15.153:49702) 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 0x7ff3a01335e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 1 -- 172.21.15.153:0/1228836167 learned_addr learned my addr 172.21.15.153:0/1228836167 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): get_auth_request con 0x7ff3a009ea50 auth_method 0 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): get_auth_request con 0x7ff3a0136c20 auth_method 0 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a01318f0 0x7ff3a01330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): get_auth_request con 0x7ff3a01318f0 auth_method 0 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff39f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.831+0000 7ff3a48ba640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 1 -- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a01318f0 msgr2=0x7ff3a01330a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a01318f0 0x7ff3a01330a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 1 -- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 msgr2=0x7ff3a0132b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 0x7ff3a0132b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 10 monclient: _finish_hunting 0 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 1 monclient: found mon.noname-c 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff3a48ba640 1 -- 172.21.15.153:0/1228836167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390009390 con 0x7ff3a0136c20 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.832+0000 7ff39ffff640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 0x7ff3a0132b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a48ba640 1 -- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 msgr2=0x7ff3a01335e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a48ba640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 0x7ff3a01335e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff39800c7e0 tx=0x7ff3980025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39800a850 con 0x7ff3a0136c20 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: dump: 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.849 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39800a9d0 con 0x7ff3a0136c20 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39800ad90 con 0x7ff3a0136c20 2024-09-15T03:28:42.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff387fff640 10 monclient: dump: 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff39e7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 5 monclient: authenticate success, global_id 4286 2024-09-15T03:28:42.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 10 monclient: _renew_subs 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a01361c0 con 0x7ff3a0136c20 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 10 monclient: _renew_subs 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.834+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a0134f80 con 0x7ff3a0136c20 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.835+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff398012630 con 0x7ff3a0136c20 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.835+0000 7ff3a631d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.835+0000 7ff3a631d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.835+0000 7ff3a631d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.835+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a0135200 con 0x7ff3a0136c20 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.835+0000 7ff387fff640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff38c038790 0x7ff38c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.836+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7ff39804bbd0 con 0x7ff3a0136c20 2024-09-15T03:28:42.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.836+0000 7ff39ffff640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff38c038790 0x7ff38c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.836+0000 7ff39ffff640 10 monclient: get_auth_request con 0x7ff38c038790 auth_method 0 2024-09-15T03:28:42.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.838+0000 7ff39ffff640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff38c038790 0x7ff38c03ac40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff390009130 tx=0x7ff390032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.840+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff3a0135200 con 0x7ff3a0136c20 2024-09-15T03:28:42.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.840+0000 7ff387fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.840+0000 7ff387fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:42.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.882+0000 7fd8364ef640 10 monclient: get_monmap_and_config 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.882+0000 7fd8364ef640 10 monclient: build_initial_monmap 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.882+0000 7fd8364ef640 10 monclient: monmap: 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:42.883574+0000 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:42.883574+0000 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:42.884 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:42.885 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:42.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.883+0000 7fd8364ef640 1 Processor -- start 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.883+0000 7fd8364ef640 1 -- start start 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.883+0000 7fd8364ef640 10 monclient: init 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd8301318f0 0x7fd830131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): picked mon.noname-a con 0x7fd8301318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd83009ea50 0x7fd83009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): picked mon.noname-c con 0x7fd83009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd830068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83009fbf0 con 0x7fd830068d90 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd830131d50 con 0x7fd8301318f0 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83009fe30 con 0x7fd83009ea50 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd8364ef640 10 monclient(hunting): authenticate will time out at 720.523176s 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd8301318f0 0x7fd830131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd8301318f0 0x7fd830131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49706/0 (socket says 172.21.15.153:49706) 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd834a8c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd830068d90 0x7fd8300691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48420/0 (socket says 172.21.15.153:48420) 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 -- 172.21.15.153:0/2797998943 learned_addr learned my addr 172.21.15.153:0/2797998943 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): get_auth_request con 0x7fd8301318f0 auth_method 0 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82f7fe640 1 --2- 172.21.15.153:0/2797998943 >> v2:172.21.15.181:3300/0 conn(0x7fd83009ea50 0x7fd83009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request con 0x7fd83009ea50 auth_method 0 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 -- 172.21.15.153:0/2797998943 >> v2:172.21.15.181:3300/0 conn(0x7fd83009ea50 msgr2=0x7fd83009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 --2- 172.21.15.153:0/2797998943 >> v2:172.21.15.181:3300/0 conn(0x7fd83009ea50 0x7fd83009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 -- 172.21.15.153:0/2797998943 >> v1:172.21.15.153:6789/0 conn(0x7fd830068d90 legacy=0x7fd8300691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 monclient: found mon.noname-a 2024-09-15T03:28:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.884+0000 7fd82ffff640 1 -- 172.21.15.153:0/2797998943 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300a1210 con 0x7fd8301318f0 2024-09-15T03:28:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82ffff640 1 -- 172.21.15.153:0/2797998943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 msgr2=0x7fd830131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82ffff640 1 --2- 172.21.15.153:0/2797998943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830131f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd83006f600 tx=0x7fd8200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2eb598666db1d38 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 1 -- 172.21.15.153:0/2797998943 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820032440 con 0x7fd8301318f0 2024-09-15T03:28:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: dump: 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: _finish_auth 0 2024-09-15T03:28:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd8364ef640 5 monclient: authenticate success, global_id 4299 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd8364ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 1 -- 172.21.15.153:0/2797998943 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8200325a0 con 0x7fd8301318f0 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd8364ef640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd8364ef640 4 set_mon_vals no callback set 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 1 -- 172.21.15.153:0/2797998943 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820032940 con 0x7fd8301318f0 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.887+0000 7fd82effd640 10 monclient: dump: 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 10 monclient: shutdown 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 -- 172.21.15.153:0/2797998943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 msgr2=0x7fd830131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 --2- 172.21.15.153:0/2797998943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830131f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd83006f600 tx=0x7fd8200091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 -- 172.21.15.153:0/2797998943 shutdown_connections 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 --2- 172.21.15.153:0/2797998943 >> v2:172.21.15.181:3300/0 conn(0x7fd83009ea50 0x7fd83009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 --2- 172.21.15.153:0/2797998943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830131f30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 -- 172.21.15.153:0/2797998943 >> 172.21.15.153:0/2797998943 conn(0x7fd830067110 msgr2=0x7fd830067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 -- 172.21.15.153:0/2797998943 shutdown_connections 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 1 -- 172.21.15.153:0/2797998943 wait complete. 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 10 monclient: build_initial_monmap 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.888+0000 7fd8364ef640 10 monclient: monmap: 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:42.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 Processor -- start 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 -- start start 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient: init 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 0x7fd83009a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): picked mon.noname-c con 0x7fd830068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd83009ea50 0x7fd830130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): picked mon.noname-b con 0x7fd83009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): picked mon.noname-a con 0x7fd8301318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300a4400 con 0x7fd8301318f0 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83011dc30 con 0x7fd83009ea50 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83011df10 con 0x7fd830068d90 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd8364ef640 10 monclient(hunting): authenticate will time out at 720.528787s 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 0x7fd83009a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd834a8c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 0x7fd83009a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44948/0 (socket says 172.21.15.153:44948) 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 1 -- 172.21.15.153:0/3253907739 learned_addr learned my addr 172.21.15.153:0/3253907739 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 10 monclient(hunting): get_auth_request con 0x7fd830068d90 auth_method 0 2024-09-15T03:28:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd834a8c640 10 monclient(hunting): get_auth_request con 0x7fd8301318f0 auth_method 0 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd834a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd834a8c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.889+0000 7fd834a8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd83009ea50 0x7fd830130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd834a8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd834a8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd834a8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request con 0x7fd83009ea50 auth_method 0 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 1 -- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd83009ea50 msgr2=0x7fd830130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd83009ea50 0x7fd830130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 1 -- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 msgr2=0x7fd830140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 1 monclient: found mon.noname-c 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd82ffff640 1 -- 172.21.15.153:0/3253907739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820009450 con 0x7fd830068d90 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.890+0000 7fd834a8c640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd82ffff640 1 -- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 msgr2=0x7fd83009a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd82ffff640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 0x7fd83009a920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd820009940 tx=0x7fd820032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820031a40 con 0x7fd830068d90 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: dump: 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd820031bc0 con 0x7fd830068d90 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 5 monclient: authenticate success, global_id 4289 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 10 monclient: _renew_subs 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83011c840 con 0x7fd830068d90 2024-09-15T03:28:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82003c400 con 0x7fd830068d90 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd817fff640 10 monclient: dump: 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd82e7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 10 monclient: _renew_subs 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.892+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83011cd40 con 0x7fd830068d90 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.893+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd82003c7b0 con 0x7fd830068d90 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.893+0000 7fd8364ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd8364ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd8364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83009bf80 con 0x7fd830068d90 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd817fff640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd81c038740 0x7fd81c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd82006c310 con 0x7fd830068d90 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd82f7fe640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd81c038740 0x7fd81c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.894+0000 7fd82f7fe640 10 monclient: get_auth_request con 0x7fd81c038740 auth_method 0 2024-09-15T03:28:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.897+0000 7fd82f7fe640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd81c038740 0x7fd81c03abf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd83009b0f0 tx=0x7fd824008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.898+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd83009bf80 con 0x7fd830068d90 2024-09-15T03:28:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.898+0000 7fd817fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.898+0000 7fd817fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.924+0000 7fb93539d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.924+0000 7fb93539d640 10 monclient: build_initial_monmap 2024-09-15T03:28:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.924+0000 7fb93539d640 10 monclient: monmap: 2024-09-15T03:28:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:42.925052+0000 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:42.925052+0000 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:42.926 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.924+0000 7fb93539d640 1 Processor -- start 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.924+0000 7fb93539d640 1 -- start start 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.924+0000 7fb93539d640 10 monclient: init 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb9301318f0 0x7fb930131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9301318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 0x7fb93009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): picked mon.noname-b con 0x7fb93009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb930068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb930131cc0 con 0x7fb930068d90 2024-09-15T03:28:42.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb93009fbf0 con 0x7fb9301318f0 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93009fe30 con 0x7fb93009ea50 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb93539d640 10 monclient(hunting): authenticate will time out at 720.564456s 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 0x7fb93009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 0x7fb93009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44956/0 (socket says 172.21.15.153:44956) 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 -- 172.21.15.153:0/3829049564 learned_addr learned my addr 172.21.15.153:0/3829049564 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): get_auth_request con 0x7fb93009ea50 auth_method 0 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 1 --2- 172.21.15.153:0/3829049564 >> v2:172.21.15.181:3300/0 conn(0x7fb9301318f0 0x7fb930131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): get_auth_request con 0x7fb9301318f0 auth_method 0 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 -- 172.21.15.153:0/3829049564 >> v2:172.21.15.181:3300/0 conn(0x7fb9301318f0 msgr2=0x7fb930131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:42.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 --2- 172.21.15.153:0/3829049564 >> v2:172.21.15.181:3300/0 conn(0x7fb9301318f0 0x7fb930131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 -- 172.21.15.153:0/3829049564 >> v1:172.21.15.181:6789/0 conn(0x7fb930068d90 legacy=0x7fb9300691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:42.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:28:42.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 monclient: found mon.noname-b 2024-09-15T03:28:42.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:42.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.925+0000 7fb92e7fc640 1 -- 172.21.15.153:0/3829049564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300a1210 con 0x7fb93009ea50 2024-09-15T03:28:42.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92e7fc640 1 -- 172.21.15.153:0/3829049564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 msgr2=0x7fb93009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92e7fc640 1 --2- 172.21.15.153:0/3829049564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 0x7fb93009ee20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb924009910 tx=0x7fb9240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86170eaf8f306251 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 1 -- 172.21.15.153:0/3829049564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92402aea0 con 0x7fb93009ea50 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: dump: 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb93539d640 5 monclient: authenticate success, global_id 4292 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 1 -- 172.21.15.153:0/3829049564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb924032440 con 0x7fb93009ea50 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb93539d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb93539d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 1 -- 172.21.15.153:0/3829049564 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924032820 con 0x7fb93009ea50 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb93539d640 4 set_mon_vals no callback set 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.928+0000 7fb92dffb640 10 monclient: dump: 2024-09-15T03:28:42.939 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 10 monclient: shutdown 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 -- 172.21.15.153:0/3829049564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 msgr2=0x7fb93009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 --2- 172.21.15.153:0/3829049564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 0x7fb93009ee20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb924009910 tx=0x7fb9240091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 -- 172.21.15.153:0/3829049564 shutdown_connections 2024-09-15T03:28:42.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 --2- 172.21.15.153:0/3829049564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb93009ea50 0x7fb93009ee20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 --2- 172.21.15.153:0/3829049564 >> v2:172.21.15.181:3300/0 conn(0x7fb9301318f0 0x7fb930131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 -- 172.21.15.153:0/3829049564 >> 172.21.15.153:0/3829049564 conn(0x7fb930067110 msgr2=0x7fb930067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 -- 172.21.15.153:0/3829049564 shutdown_connections 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.929+0000 7fb93539d640 1 -- 172.21.15.153:0/3829049564 wait complete. 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.930+0000 7fb93539d640 10 monclient: build_initial_monmap 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.930+0000 7fb93539d640 10 monclient: monmap: 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:42.941 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.930+0000 7fb93539d640 1 Processor -- start 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.930+0000 7fb93539d640 1 -- start start 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.930+0000 7fb93539d640 10 monclient: init 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb930068d90 0x7fb93009aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): picked mon.noname-b con 0x7fb930068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb93009ea50 0x7fb930130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): picked mon.noname-a con 0x7fb93009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 0x7fb93011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9301318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:42.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93009bf10 con 0x7fb93009ea50 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300a44b0 con 0x7fb930068d90 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb930141830 con 0x7fb9301318f0 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb93539d640 10 monclient(hunting): authenticate will time out at 720.570168s 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 0x7fb93011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb93009ea50 0x7fb930130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 0x7fb93011b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44968/0 (socket says 172.21.15.153:44968) 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 -- 172.21.15.153:0/1040563664 learned_addr learned my addr 172.21.15.153:0/1040563664 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): get_auth_request con 0x7fb9301318f0 auth_method 0 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): get_auth_request con 0x7fb93009ea50 auth_method 0 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb930068d90 0x7fb93009aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): get_auth_request con 0x7fb930068d90 auth_method 0 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 -- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb930068d90 msgr2=0x7fb93009aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb930068d90 0x7fb93009aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 -- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb93009ea50 msgr2=0x7fb930130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb93009ea50 0x7fb930130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 monclient: found mon.noname-c 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92f7fe640 1 -- 172.21.15.153:0/1040563664 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb924009450 con 0x7fb9301318f0 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.931+0000 7fb92e7fc640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb93009ea50 0x7fb930130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.933+0000 7fb92f7fe640 1 -- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 msgr2=0x7fb93011b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.933+0000 7fb92f7fe640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 0x7fb93011b720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb92800c890 tx=0x7fb928002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92800ce80 con 0x7fb9301318f0 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: dump: 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 5 monclient: authenticate success, global_id 4295 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92800aa00 con 0x7fb9301318f0 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 10 monclient: _renew_subs 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93009b660 con 0x7fb9301318f0 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92800adc0 con 0x7fb9301318f0 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:42.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb916ffd640 10 monclient: dump: 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 10 monclient: _renew_subs 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb92d7fa640 4 set_mon_vals no callback set 2024-09-15T03:28:42.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.934+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb930141300 con 0x7fb9301318f0 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb93539d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb93539d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb93539d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93004bf60 con 0x7fb9301318f0 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb928012630 con 0x7fb9301318f0 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb916ffd640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb91c038790 0x7fb91c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fb92804bc60 con 0x7fb9301318f0 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.935+0000 7fb92effd640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb91c038790 0x7fb91c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:42.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.936+0000 7fb92effd640 10 monclient: get_auth_request con 0x7fb91c038790 auth_method 0 2024-09-15T03:28:42.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.938+0000 7fb92effd640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb91c038790 0x7fb91c03ac40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb920009d20 tx=0x7fb9200090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:42.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.940+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb928022d70 con 0x7fb9301318f0 2024-09-15T03:28:42.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.940+0000 7fb916ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:42.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:42.940+0000 7fb916ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.008+0000 7ff3a631d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.008+0000 7ff3a631d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.008+0000 7ff3a631d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.008+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff3a0135200 con 0x7ff3a0136c20 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.009+0000 7ff387fff640 1 -- 172.21.15.153:0/1228836167 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff3a0135200 con 0x7ff3a0136c20 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.009+0000 7ff387fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.009+0000 7ff387fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:43.010 INFO:teuthology.orchestra.run.smithi153.stdout:47244640261 2024-09-15T03:28:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff38c038790 msgr2=0x7ff38c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff38c038790 0x7ff38c03ac40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff390009130 tx=0x7ff390032000 comp rx=0 tx=0).stop 2024-09-15T03:28:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 10 monclient: shutdown 2024-09-15T03:28:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 msgr2=0x7ff3a01335e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 0x7ff3a01335e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff39800c7e0 tx=0x7ff3980025b0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 shutdown_connections 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff38c038790 0x7ff38c03ac40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3a0136c20 0x7ff3a01335e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a01318f0 0x7ff3a01330a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 --2- 172.21.15.153:0/1228836167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3a009ea50 0x7ff3a0132b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 >> 172.21.15.153:0/1228836167 conn(0x7ff3a0067110 msgr2=0x7ff3a009d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 shutdown_connections 2024-09-15T03:28:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.012+0000 7ff3a631d640 1 -- 172.21.15.153:0/1228836167 wait complete. 2024-09-15T03:28:43.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-15T03:28:43.027 DEBUG:teuthology.parallel:result is None 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd8364ef640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd8364ef640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd8364ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd83009bf80 con 0x7fd830068d90 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd817fff640 1 -- 172.21.15.153:0/3253907739 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd820037020 con 0x7fd830068d90 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd817fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:28:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.063+0000 7fd817fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:43.065 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:43.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd81c038740 msgr2=0x7fd81c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd81c038740 0x7fd81c03abf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd83009b0f0 tx=0x7fd824008040 comp rx=0 tx=0).stop 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 10 monclient: shutdown 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 msgr2=0x7fd83009a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 0x7fd83009a920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd820009940 tx=0x7fd820032000 comp rx=0 tx=0).stop 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 shutdown_connections 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd81c038740 0x7fd81c03abf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8301318f0 0x7fd830140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd83009ea50 0x7fd830130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 --2- 172.21.15.153:0/3253907739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd830068d90 0x7fd83009a920 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 >> 172.21.15.153:0/3253907739 conn(0x7fd830067110 msgr2=0x7fd8300a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 shutdown_connections 2024-09-15T03:28:43.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.065+0000 7fd8364ef640 1 -- 172.21.15.153:0/3253907739 wait complete. 2024-09-15T03:28:43.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-09-15T03:28:43.077 DEBUG:teuthology.parallel:result is None 2024-09-15T03:28:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb93539d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb93539d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb93539d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb93011da00 con 0x7fb9301318f0 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb916ffd640 1 -- 172.21.15.153:0/1040563664 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb928016030 con 0x7fb9301318f0 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb916ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.110+0000 7fb916ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:43.112 INFO:teuthology.orchestra.run.smithi153.stdout:47244640259 2024-09-15T03:28:43.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb91c038790 msgr2=0x7fb91c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb91c038790 0x7fb91c03ac40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb920009d20 tx=0x7fb9200090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 10 monclient: shutdown 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 msgr2=0x7fb93011b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 0x7fb93011b720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb92800c890 tx=0x7fb928002640 comp rx=0 tx=0).stop 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 shutdown_connections 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb91c038790 0x7fb91c03ac40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9301318f0 0x7fb93011b720 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb93009ea50 0x7fb930130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 --2- 172.21.15.153:0/1040563664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb930068d90 0x7fb93009aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.112+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 >> 172.21.15.153:0/1040563664 conn(0x7fb930067110 msgr2=0x7fb9301348e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.113+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 shutdown_connections 2024-09-15T03:28:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.113+0000 7fb93539d640 1 -- 172.21.15.153:0/1040563664 wait complete. 2024-09-15T03:28:43.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-15T03:28:43.124 DEBUG:teuthology.parallel:result is None 2024-09-15T03:28:43.124 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:28:43.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:28:43.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.256+0000 7f237c11d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:43.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.256+0000 7f237c11d640 10 monclient: build_initial_monmap 2024-09-15T03:28:43.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.256+0000 7f237c11d640 10 monclient: monmap: 2024-09-15T03:28:43.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:43.257295+0000 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:43.257295+0000 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:43.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 Processor -- start 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 -- start start 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient: init 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f23740974f0 0x7f237412ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): picked mon.noname-a con 0x7f23740974f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f237409a8a0 0x7f237409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): picked mon.noname-c con 0x7f237409a8a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f237412dd40 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23740978c0 con 0x7f237412dd40 2024-09-15T03:28:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237409b220 con 0x7f23740974f0 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237409ca20 con 0x7f237409a8a0 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f237c11d640 10 monclient(hunting): authenticate will time out at 720.896814s 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f23740974f0 0x7f237412ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f23740974f0 0x7f237412ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49742/0 (socket says 172.21.15.153:49742) 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 1 -- 172.21.15.153:0/220316100 learned_addr learned my addr 172.21.15.153:0/220316100 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 10 monclient(hunting): get_auth_request con 0x7f23740974f0 auth_method 0 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.257+0000 7f2379eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f23796b8640 1 --2- 172.21.15.153:0/220316100 >> v2:172.21.15.181:3300/0 conn(0x7f237409a8a0 0x7f237409ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f23796b8640 10 monclient(hunting): get_auth_request con 0x7f237409a8a0 auth_method 0 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f23796b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f23796b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f23796b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 1 -- 172.21.15.153:0/220316100 >> v2:172.21.15.181:3300/0 conn(0x7f237409a8a0 msgr2=0x7f237409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 1 --2- 172.21.15.153:0/220316100 >> v2:172.21.15.181:3300/0 conn(0x7f237409a8a0 0x7f237409ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 1 -- 172.21.15.153:0/220316100 >> v1:172.21.15.181:6789/0 conn(0x7f237412dd40 legacy=0x7f237412e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 10 monclient: _finish_hunting 0 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 1 monclient: found mon.noname-a 2024-09-15T03:28:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.258+0000 7f2379eb9640 1 -- 172.21.15.153:0/220316100 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2374098ad0 con 0x7f23740974f0 2024-09-15T03:28:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2379eb9640 1 -- 172.21.15.153:0/220316100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23740974f0 msgr2=0x7f237412ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2379eb9640 1 --2- 172.21.15.153:0/220316100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23740974f0 0x7f237412ab00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f237409d040 tx=0x7f236c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3042e0cf2c5f9da3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 1 -- 172.21.15.153:0/220316100 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c032440 con 0x7f23740974f0 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: dump: 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: _finish_auth 0 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 1 -- 172.21.15.153:0/220316100 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c0325a0 con 0x7f23740974f0 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 1 -- 172.21.15.153:0/220316100 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c032940 con 0x7f23740974f0 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f2378eb7640 10 monclient: dump: 2024-09-15T03:28:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 5 monclient: authenticate success, global_id 4308 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 4 set_mon_vals no callback set 2024-09-15T03:28:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 10 monclient: shutdown 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 1 -- 172.21.15.153:0/220316100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23740974f0 msgr2=0x7f237412ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 1 --2- 172.21.15.153:0/220316100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23740974f0 0x7f237412ab00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f237409d040 tx=0x7f236c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 1 -- 172.21.15.153:0/220316100 shutdown_connections 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 1 --2- 172.21.15.153:0/220316100 >> v2:172.21.15.181:3300/0 conn(0x7f237409a8a0 0x7f237409ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 1 --2- 172.21.15.153:0/220316100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23740974f0 0x7f237412ab00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.261+0000 7f237c11d640 1 -- 172.21.15.153:0/220316100 >> 172.21.15.153:0/220316100 conn(0x7f23740672b0 msgr2=0x7f23740676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 1 -- 172.21.15.153:0/220316100 shutdown_connections 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 1 -- 172.21.15.153:0/220316100 wait complete. 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 10 monclient: build_initial_monmap 2024-09-15T03:28:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 10 monclient: monmap: 2024-09-15T03:28:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 1 Processor -- start 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 1 -- start start 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.262+0000 7f237c11d640 10 monclient: init 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23740974f0 0x7f237413f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): picked mon.noname-b con 0x7f23740974f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 0x7f237413f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): picked mon.noname-a con 0x7f237409a8a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f237412dd40 0x7f237413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): picked mon.noname-c con 0x7f237412dd40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237412e3b0 con 0x7f237409a8a0 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237406f5d0 con 0x7f23740974f0 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237412f990 con 0x7f237412dd40 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237c11d640 10 monclient(hunting): authenticate will time out at 720.902301s 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 0x7f237413f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 0x7f237413f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49748/0 (socket says 172.21.15.153:49748) 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 -- 172.21.15.153:0/4133487824 learned_addr learned my addr 172.21.15.153:0/4133487824 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): get_auth_request con 0x7f237409a8a0 auth_method 0 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f237412dd40 0x7f237413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): get_auth_request con 0x7f237412dd40 auth_method 0 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f237a6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f2379eb9640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23740974f0 0x7f237413f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f2379eb9640 10 monclient(hunting): get_auth_request con 0x7f23740974f0 auth_method 0 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f2379eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f2379eb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f2379eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 -- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f237412dd40 msgr2=0x7f237413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f237412dd40 0x7f237413fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 -- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23740974f0 msgr2=0x7f237413f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23740974f0 0x7f237413f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient: _finish_hunting 0 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 monclient: found mon.noname-a 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.263+0000 7f23796b8640 1 -- 172.21.15.153:0/4133487824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c009450 con 0x7f237409a8a0 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.265+0000 7f23796b8640 1 -- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 msgr2=0x7f237413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.265+0000 7f23796b8640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 0x7f237413f950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f236800e190 tx=0x7f2368009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368013590 con 0x7f237409a8a0 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: dump: 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23680136f0 con 0x7f237409a8a0 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 5 monclient: authenticate success, global_id 4311 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 10 monclient: _renew_subs 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237412f310 con 0x7f237409a8a0 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368013a90 con 0x7f237409a8a0 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 10 monclient: dump: 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 10 monclient: _renew_subs 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2373fff640 4 set_mon_vals no callback set 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237406ee80 con 0x7f237409a8a0 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.266+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f236801e460 con 0x7f237409a8a0 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f237c11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f237c11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f237c11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237412fb10 con 0x7f237409a8a0 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f2371ffb640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2360038740 0x7f236003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f236804c130 con 0x7f237409a8a0 2024-09-15T03:28:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f2379eb9640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2360038740 0x7f236003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.267+0000 7f2379eb9640 10 monclient: get_auth_request con 0x7f2360038740 auth_method 0 2024-09-15T03:28:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.270+0000 7f2379eb9640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2360038740 0x7f236003abf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f236c031000 tx=0x7f236c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.271+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f237412fb10 con 0x7f237409a8a0 2024-09-15T03:28:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.271+0000 7f2371ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.271+0000 7f2371ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:43.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.425+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f237412fb10 con 0x7f2360038740 2024-09-15T03:28:43.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.427+0000 7f2371ffb640 1 -- 172.21.15.153:0/4133487824 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41016 (secure 0 0 0) 0x7f237412fb10 con 0x7f2360038740 2024-09-15T03:28:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:43.429 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2360038740 msgr2=0x7f236003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2360038740 0x7f236003abf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f236c031000 tx=0x7f236c032000 comp rx=0 tx=0).stop 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 10 monclient: shutdown 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 msgr2=0x7f237413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 0x7f237413f950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f236800e190 tx=0x7f2368009920 comp rx=0 tx=0).stop 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 shutdown_connections 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2360038740 0x7f236003abf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f237412dd40 0x7f237413fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f237409a8a0 0x7f237413f950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 --2- 172.21.15.153:0/4133487824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23740974f0 0x7f237413f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.429+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 >> 172.21.15.153:0/4133487824 conn(0x7f23740672b0 msgr2=0x7f237409b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.430+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 shutdown_connections 2024-09-15T03:28:43.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.430+0000 7f237c11d640 1 -- 172.21.15.153:0/4133487824 wait complete. 2024-09-15T03:28:43.441 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-15T03:28:41.844321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786240,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444800,"statfs":{"total":579820584960,"available":577991475200,"internally_reserved":0,"allocated":1994752,"data_stored":1344052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000308"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771558+0000","last_change":"2024-09-15T03:28:38.771829+0000","last_active":"2024-09-15T03:28:38.771558+0000","last_peered":"2024-09-15T03:28:38.771558+0000","last_clean":"2024-09-15T03:28:38.771558+0000","last_became_active":"2024-09-15T03:28:37.587284+0000","last_became_peered":"2024-09-15T03:28:37.587284+0000","last_unstale":"2024-09-15T03:28:38.771558+0000","last_undegraded":"2024-09-15T03:28:38.771558+0000","last_fullsized":"2024-09-15T03:28:38.771558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:11.288432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760004+0000","last_change":"2024-09-15T03:28:38.760264+0000","last_active":"2024-09-15T03:28:38.760004+0000","last_peered":"2024-09-15T03:28:38.760004+0000","last_clean":"2024-09-15T03:28:38.760004+0000","last_became_active":"2024-09-15T03:28:36.752311+0000","last_became_peered":"2024-09-15T03:28:36.752311+0000","last_unstale":"2024-09-15T03:28:38.760004+0000","last_undegraded":"2024-09-15T03:28:38.760004+0000","last_fullsized":"2024-09-15T03:28:38.760004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:47:34.502515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771599+0000","last_change":"2024-09-15T03:28:38.771789+0000","last_active":"2024-09-15T03:28:38.771599+0000","last_peered":"2024-09-15T03:28:38.771599+0000","last_clean":"2024-09-15T03:28:38.771599+0000","last_became_active":"2024-09-15T03:28:36.757024+0000","last_became_peered":"2024-09-15T03:28:36.757024+0000","last_unstale":"2024-09-15T03:28:38.771599+0000","last_undegraded":"2024-09-15T03:28:38.771599+0000","last_fullsized":"2024-09-15T03:28:38.771599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:33:31.289175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760078+0000","last_change":"2024-09-15T03:28:38.760299+0000","last_active":"2024-09-15T03:28:38.760078+0000","last_peered":"2024-09-15T03:28:38.760078+0000","last_clean":"2024-09-15T03:28:38.760078+0000","last_became_active":"2024-09-15T03:28:36.749668+0000","last_became_peered":"2024-09-15T03:28:36.749668+0000","last_unstale":"2024-09-15T03:28:38.760078+0000","last_undegraded":"2024-09-15T03:28:38.760078+0000","last_fullsized":"2024-09-15T03:28:38.760078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:19:51.050262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762840+0000","last_change":"2024-09-15T03:28:38.762840+0000","last_active":"2024-09-15T03:28:38.762840+0000","last_peered":"2024-09-15T03:28:38.762840+0000","last_clean":"2024-09-15T03:28:38.762840+0000","last_became_active":"2024-09-15T03:28:36.757667+0000","last_became_peered":"2024-09-15T03:28:36.757667+0000","last_unstale":"2024-09-15T03:28:38.762840+0000","last_undegraded":"2024-09-15T03:28:38.762840+0000","last_fullsized":"2024-09-15T03:28:38.762840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:01.577552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.759527+0000","last_change":"2024-09-15T03:28:38.759813+0000","last_active":"2024-09-15T03:28:38.759527+0000","last_peered":"2024-09-15T03:28:38.759527+0000","last_clean":"2024-09-15T03:28:38.759527+0000","last_became_active":"2024-09-15T03:28:36.757615+0000","last_became_peered":"2024-09-15T03:28:36.757615+0000","last_unstale":"2024-09-15T03:28:38.759527+0000","last_undegraded":"2024-09-15T03:28:38.759527+0000","last_fullsized":"2024-09-15T03:28:38.759527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:08.664245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771831+0000","last_change":"2024-09-15T03:28:38.772005+0000","last_active":"2024-09-15T03:28:38.771831+0000","last_peered":"2024-09-15T03:28:38.771831+0000","last_clean":"2024-09-15T03:28:38.771831+0000","last_became_active":"2024-09-15T03:28:36.756993+0000","last_became_peered":"2024-09-15T03:28:36.756993+0000","last_unstale":"2024-09-15T03:28:38.771831+0000","last_undegraded":"2024-09-15T03:28:38.771831+0000","last_fullsized":"2024-09-15T03:28:38.771831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.632961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762201+0000","last_change":"2024-09-15T03:28:38.762403+0000","last_active":"2024-09-15T03:28:38.762201+0000","last_peered":"2024-09-15T03:28:38.762201+0000","last_clean":"2024-09-15T03:28:38.762201+0000","last_became_active":"2024-09-15T03:28:36.757463+0000","last_became_peered":"2024-09-15T03:28:36.757463+0000","last_unstale":"2024-09-15T03:28:38.762201+0000","last_undegraded":"2024-09-15T03:28:38.762201+0000","last_fullsized":"2024-09-15T03:28:38.762201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:23.854099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":100,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-15T03:28:36.997951+0000","last_change":"2024-09-15T03:28:35.748574+0000","last_active":"2024-09-15T03:28:36.997951+0000","last_peered":"2024-09-15T03:28:36.997951+0000","last_clean":"2024-09-15T03:28:36.997951+0000","last_became_active":"2024-09-15T03:28:35.747712+0000","last_became_peered":"2024-09-15T03:28:35.747712+0000","last_unstale":"2024-09-15T03:28:36.997951+0000","last_undegraded":"2024-09-15T03:28:36.997951+0000","last_fullsized":"2024-09-15T03:28:36.997951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:54:27.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":155648,"data_stored":47234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":151552,"data_stored":47215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298152,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073688,"statfs":{"total":96636764160,"available":96331456512,"internally_reserved":0,"allocated":684032,"data_stored":576143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":159744,"data_stored":48190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073808,"statfs":{"total":96636764160,"available":96331579392,"internally_reserved":0,"allocated":688128,"data_stored":577099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:28:43.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:28:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.536+0000 7fad344d4640 10 monclient: get_monmap_and_config 2024-09-15T03:28:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.536+0000 7fad344d4640 10 monclient: build_initial_monmap 2024-09-15T03:28:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.536+0000 7fad344d4640 10 monclient: monmap: 2024-09-15T03:28:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:43.537180+0000 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:43.537180+0000 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:43.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.536+0000 7fad344d4640 1 Processor -- start 2024-09-15T03:28:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 -- start start 2024-09-15T03:28:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient: init 2024-09-15T03:28:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T03:28:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 0x7fad2c131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): picked mon.noname-b con 0x7fad2c131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fad2c09fda0 0x7fad2c132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): picked mon.noname-a con 0x7fad2c09fda0 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fad2c09e9c0 0x7fad2c09ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): picked mon.noname-c con 0x7fad2c09e9c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c0a1180 con 0x7fad2c09fda0 2024-09-15T03:28:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c0a1460 con 0x7fad2c09e9c0 2024-09-15T03:28:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c09f340 con 0x7fad2c131980 2024-09-15T03:28:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad344d4640 10 monclient(hunting): authenticate will time out at 721.176707s 2024-09-15T03:28:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 0x7fad2c131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fad2c09fda0 0x7fad2c132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fad2c09fda0 0x7fad2c132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49756/0 (socket says 172.21.15.153:49756) 2024-09-15T03:28:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 0x7fad2c131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44988/0 (socket says 172.21.15.153:44988) 2024-09-15T03:28:43.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 1 -- 172.21.15.153:0/1481343494 learned_addr learned my addr 172.21.15.153:0/1481343494 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:43.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): get_auth_request con 0x7fad2c09fda0 auth_method 0 2024-09-15T03:28:43.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): get_auth_request con 0x7fad2c131980 auth_method 0 2024-09-15T03:28:43.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad31a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.537+0000 7fad32270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-09-15T03:28:43.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 1 -- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 msgr2=0x7fad2c131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:43.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 1 --2- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 0x7fad2c131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 1 -- 172.21.15.153:0/1481343494 >> v2:172.21.15.181:3300/0 conn(0x7fad2c09e9c0 msgr2=0x7fad2c09ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad32a71640 1 --2- 172.21.15.153:0/1481343494 >> v2:172.21.15.181:3300/0 conn(0x7fad2c09e9c0 0x7fad2c09ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 1 --2- 172.21.15.153:0/1481343494 >> v2:172.21.15.181:3300/0 conn(0x7fad2c09e9c0 0x7fad2c09ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 10 monclient: _finish_hunting 0 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 1 monclient: found mon.noname-a 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:43.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad31a6f640 1 -- 172.21.15.153:0/1481343494 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c09f4b0 con 0x7fad2c09fda0 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad32a71640 10 monclient: get_auth_request con 0x7fad2c09e9c0 auth_method 0 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad32270640 1 --2- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 0x7fad2c131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.538+0000 7fad32a71640 1 --2- 172.21.15.153:0/1481343494 >> v2:172.21.15.181:3300/0 conn(0x7fad2c09e9c0 0x7fad2c09ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.540+0000 7fad31a6f640 1 -- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 msgr2=0x7fad2c132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.540+0000 7fad31a6f640 1 --2- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c132290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0a0170 tx=0x7fad200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b4953e7a947235 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.540+0000 7fad3126e640 1 -- 172.21.15.153:0/1481343494 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad20032440 con 0x7fad2c09fda0 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: dump: 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.561 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: _finish_auth 0 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 5 monclient: authenticate success, global_id 4314 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 1 -- 172.21.15.153:0/1481343494 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad200325a0 con 0x7fad2c09fda0 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:43.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:43.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 4 set_mon_vals no callback set 2024-09-15T03:28:43.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 1 -- 172.21.15.153:0/1481343494 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad20032940 con 0x7fad2c09fda0 2024-09-15T03:28:43.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:43.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad3126e640 10 monclient: dump: 2024-09-15T03:28:43.563 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.564 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.565 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.565 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 10 monclient: shutdown 2024-09-15T03:28:43.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 -- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 msgr2=0x7fad2c132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 --2- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c132290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0a0170 tx=0x7fad200091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 -- 172.21.15.153:0/1481343494 shutdown_connections 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 --2- 172.21.15.153:0/1481343494 >> v2:172.21.15.181:3300/0 conn(0x7fad2c09e9c0 0x7fad2c09ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 --2- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c132290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 --2- 172.21.15.153:0/1481343494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c131980 0x7fad2c131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 -- 172.21.15.153:0/1481343494 >> 172.21.15.153:0/1481343494 conn(0x7fad2c067bd0 msgr2=0x7fad2c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 -- 172.21.15.153:0/1481343494 shutdown_connections 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.541+0000 7fad344d4640 1 -- 172.21.15.153:0/1481343494 wait complete. 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient: build_initial_monmap 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient: monmap: 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:43.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 Processor -- start 2024-09-15T03:28:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 -- start start 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient: init 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 0x7fad2c140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): picked mon.noname-c con 0x7fad2c09e9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): picked mon.noname-a con 0x7fad2c09fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:43.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c131980 0x7fad2c1418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): picked mon.noname-b con 0x7fad2c131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:43.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c120a00 con 0x7fad2c09fda0 2024-09-15T03:28:43.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c09bf40 con 0x7fad2c131980 2024-09-15T03:28:43.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c122bc0 con 0x7fad2c09e9c0 2024-09-15T03:28:43.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:43.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.542+0000 7fad344d4640 10 monclient(hunting): authenticate will time out at 721.181952s 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 0x7fad2c140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 0x7fad2c140e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45004/0 (socket says 172.21.15.153:45004) 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 -- 172.21.15.153:0/3766101621 learned_addr learned my addr 172.21.15.153:0/3766101621 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): get_auth_request con 0x7fad2c09e9c0 auth_method 0 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c141380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49758/0 (socket says 172.21.15.153:49758) 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): get_auth_request con 0x7fad2c09fda0 auth_method 0 2024-09-15T03:28:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32a71640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c131980 0x7fad2c1418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32a71640 10 monclient(hunting): get_auth_request con 0x7fad2c131980 auth_method 0 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32a71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32a71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 -- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c131980 msgr2=0x7fad2c1418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c131980 0x7fad2c1418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 -- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 msgr2=0x7fad2c141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient: _finish_hunting 0 2024-09-15T03:28:43.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 monclient: found mon.noname-c 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad32270640 1 -- 172.21.15.153:0/3766101621 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad24009390 con 0x7fad2c09e9c0 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.543+0000 7fad31a6f640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad32270640 1 -- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 msgr2=0x7fad2c140e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad32270640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 0x7fad2c140e40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad2400ec80 tx=0x7fad240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2400eec0 con 0x7fad2c09e9c0 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: dump: 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.577 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.578 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.578 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.578 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:43.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:43.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2400cdd0 con 0x7fad2c09e9c0 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 5 monclient: authenticate success, global_id 4304 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24013680 con 0x7fad2c09e9c0 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 10 monclient: _renew_subs 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad20009450 con 0x7fad2c09e9c0 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad2a7fc640 10 monclient: dump: 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad30a6d640 4 set_mon_vals no callback set 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 10 monclient: _renew_subs 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.545+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c121930 con 0x7fad2c09e9c0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad240139f0 con 0x7fad2c09e9c0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad344d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad344d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad344d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c04bf60 con 0x7fad2c09e9c0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad2a7fc640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad18038740 0x7fad1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.546+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fad2404bd70 con 0x7fad2c09e9c0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.547+0000 7fad31a6f640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad18038740 0x7fad1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.547+0000 7fad31a6f640 10 monclient: get_auth_request con 0x7fad18038740 auth_method 0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.549+0000 7fad31a6f640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad18038740 0x7fad1803abf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0a4f60 tx=0x7fad20032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.551+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fad240236b0 con 0x7fad2c09e9c0 2024-09-15T03:28:43.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.551+0000 7fad2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.551+0000 7fad2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:43.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.703+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad2c11fe00 con 0x7fad18038740 2024-09-15T03:28:43.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.704+0000 7fad2a7fc640 1 -- 172.21.15.153:0/3766101621 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41016 (secure 0 0 0) 0x7fad2c11fe00 con 0x7fad18038740 2024-09-15T03:28:43.706 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:43.706 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.706+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad18038740 msgr2=0x7fad1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.706+0000 7fad344d4640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad18038740 0x7fad1803abf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0a4f60 tx=0x7fad20032000 comp rx=0 tx=0).stop 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.706+0000 7fad344d4640 10 monclient: shutdown 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.706+0000 7fad344d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.706+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 msgr2=0x7fad2c140e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.706+0000 7fad344d4640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 0x7fad2c140e40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad2400ec80 tx=0x7fad240090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 shutdown_connections 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad18038740 0x7fad1803abf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c131980 0x7fad2c1418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2c09fda0 0x7fad2c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 --2- 172.21.15.153:0/3766101621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2c09e9c0 0x7fad2c140e40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 >> 172.21.15.153:0/3766101621 conn(0x7fad2c067bd0 msgr2=0x7fad2c0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 shutdown_connections 2024-09-15T03:28:43.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.707+0000 7fad344d4640 1 -- 172.21.15.153:0/3766101621 wait complete. 2024-09-15T03:28:43.718 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-15T03:28:43.514451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786240,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444800,"statfs":{"total":579820584960,"available":577991475200,"internally_reserved":0,"allocated":1994752,"data_stored":1344052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.670438"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771558+0000","last_change":"2024-09-15T03:28:38.771829+0000","last_active":"2024-09-15T03:28:38.771558+0000","last_peered":"2024-09-15T03:28:38.771558+0000","last_clean":"2024-09-15T03:28:38.771558+0000","last_became_active":"2024-09-15T03:28:37.587284+0000","last_became_peered":"2024-09-15T03:28:37.587284+0000","last_unstale":"2024-09-15T03:28:38.771558+0000","last_undegraded":"2024-09-15T03:28:38.771558+0000","last_fullsized":"2024-09-15T03:28:38.771558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:11.288432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760004+0000","last_change":"2024-09-15T03:28:38.760264+0000","last_active":"2024-09-15T03:28:38.760004+0000","last_peered":"2024-09-15T03:28:38.760004+0000","last_clean":"2024-09-15T03:28:38.760004+0000","last_became_active":"2024-09-15T03:28:36.752311+0000","last_became_peered":"2024-09-15T03:28:36.752311+0000","last_unstale":"2024-09-15T03:28:38.760004+0000","last_undegraded":"2024-09-15T03:28:38.760004+0000","last_fullsized":"2024-09-15T03:28:38.760004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:47:34.502515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771599+0000","last_change":"2024-09-15T03:28:38.771789+0000","last_active":"2024-09-15T03:28:38.771599+0000","last_peered":"2024-09-15T03:28:38.771599+0000","last_clean":"2024-09-15T03:28:38.771599+0000","last_became_active":"2024-09-15T03:28:36.757024+0000","last_became_peered":"2024-09-15T03:28:36.757024+0000","last_unstale":"2024-09-15T03:28:38.771599+0000","last_undegraded":"2024-09-15T03:28:38.771599+0000","last_fullsized":"2024-09-15T03:28:38.771599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:33:31.289175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760078+0000","last_change":"2024-09-15T03:28:38.760299+0000","last_active":"2024-09-15T03:28:38.760078+0000","last_peered":"2024-09-15T03:28:38.760078+0000","last_clean":"2024-09-15T03:28:38.760078+0000","last_became_active":"2024-09-15T03:28:36.749668+0000","last_became_peered":"2024-09-15T03:28:36.749668+0000","last_unstale":"2024-09-15T03:28:38.760078+0000","last_undegraded":"2024-09-15T03:28:38.760078+0000","last_fullsized":"2024-09-15T03:28:38.760078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:19:51.050262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762840+0000","last_change":"2024-09-15T03:28:38.762840+0000","last_active":"2024-09-15T03:28:38.762840+0000","last_peered":"2024-09-15T03:28:38.762840+0000","last_clean":"2024-09-15T03:28:38.762840+0000","last_became_active":"2024-09-15T03:28:36.757667+0000","last_became_peered":"2024-09-15T03:28:36.757667+0000","last_unstale":"2024-09-15T03:28:38.762840+0000","last_undegraded":"2024-09-15T03:28:38.762840+0000","last_fullsized":"2024-09-15T03:28:38.762840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:01.577552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.759527+0000","last_change":"2024-09-15T03:28:38.759813+0000","last_active":"2024-09-15T03:28:38.759527+0000","last_peered":"2024-09-15T03:28:38.759527+0000","last_clean":"2024-09-15T03:28:38.759527+0000","last_became_active":"2024-09-15T03:28:36.757615+0000","last_became_peered":"2024-09-15T03:28:36.757615+0000","last_unstale":"2024-09-15T03:28:38.759527+0000","last_undegraded":"2024-09-15T03:28:38.759527+0000","last_fullsized":"2024-09-15T03:28:38.759527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:08.664245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771831+0000","last_change":"2024-09-15T03:28:38.772005+0000","last_active":"2024-09-15T03:28:38.771831+0000","last_peered":"2024-09-15T03:28:38.771831+0000","last_clean":"2024-09-15T03:28:38.771831+0000","last_became_active":"2024-09-15T03:28:36.756993+0000","last_became_peered":"2024-09-15T03:28:36.756993+0000","last_unstale":"2024-09-15T03:28:38.771831+0000","last_undegraded":"2024-09-15T03:28:38.771831+0000","last_fullsized":"2024-09-15T03:28:38.771831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.632961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762201+0000","last_change":"2024-09-15T03:28:38.762403+0000","last_active":"2024-09-15T03:28:38.762201+0000","last_peered":"2024-09-15T03:28:38.762201+0000","last_clean":"2024-09-15T03:28:38.762201+0000","last_became_active":"2024-09-15T03:28:36.757463+0000","last_became_peered":"2024-09-15T03:28:36.757463+0000","last_unstale":"2024-09-15T03:28:38.762201+0000","last_undegraded":"2024-09-15T03:28:38.762201+0000","last_fullsized":"2024-09-15T03:28:38.762201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:23.854099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":100,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-15T03:28:36.997951+0000","last_change":"2024-09-15T03:28:35.748574+0000","last_active":"2024-09-15T03:28:36.997951+0000","last_peered":"2024-09-15T03:28:36.997951+0000","last_clean":"2024-09-15T03:28:36.997951+0000","last_became_active":"2024-09-15T03:28:35.747712+0000","last_became_peered":"2024-09-15T03:28:35.747712+0000","last_unstale":"2024-09-15T03:28:36.997951+0000","last_undegraded":"2024-09-15T03:28:36.997951+0000","last_fullsized":"2024-09-15T03:28:36.997951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:54:27.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":155648,"data_stored":47234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":151552,"data_stored":47215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298152,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073688,"statfs":{"total":96636764160,"available":96331456512,"internally_reserved":0,"allocated":684032,"data_stored":576143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":159744,"data_stored":48190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073808,"statfs":{"total":96636764160,"available":96331579392,"internally_reserved":0,"allocated":688128,"data_stored":577099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:28:43.719 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:28:43.719 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T03:28:43.719 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T03:28:43.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.811+0000 7fcb1089f640 10 monclient: get_monmap_and_config 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.811+0000 7fcb1089f640 10 monclient: build_initial_monmap 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.811+0000 7fcb1089f640 10 monclient: monmap: 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:43.811930+0000 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:43.811930+0000 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:43.812 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.813 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:43.813 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:43.813 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:43.813 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:43.813 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:43.813 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.811+0000 7fcb1089f640 1 Processor -- start 2024-09-15T03:28:43.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.811+0000 7fcb1089f640 1 -- start start 2024-09-15T03:28:43.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.811+0000 7fcb1089f640 10 monclient: init 2024-09-15T03:28:43.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:43.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcb0c09ea50 0x7fcb0c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0c09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb0c068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c131cc0 con 0x7fcb0c068d90 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c09fbf0 con 0x7fcb0c09ea50 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c09fe30 con 0x7fcb0c1318f0 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb1089f640 10 monclient(hunting): authenticate will time out at 721.451492s 2024-09-15T03:28:43.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcb0c09ea50 0x7fcb0c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcb0c068d90 0x7fcb0c0691f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48426/0 (socket says 172.21.15.153:48426) 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45010/0 (socket says 172.21.15.153:45010) 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0ad9d640 1 -- 172.21.15.153:0/1788569626 learned_addr learned my addr 172.21.15.153:0/1788569626 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): get_auth_request con 0x7fcb0c1318f0 auth_method 0 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): get_auth_request con 0x7fcb0c09ea50 auth_method 0 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 -- 172.21.15.153:0/1788569626 >> v2:172.21.15.153:3300/0 conn(0x7fcb0c09ea50 msgr2=0x7fcb0c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 --2- 172.21.15.153:0/1788569626 >> v2:172.21.15.153:3300/0 conn(0x7fcb0c09ea50 0x7fcb0c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 -- 172.21.15.153:0/1788569626 >> v1:172.21.15.153:6789/0 conn(0x7fcb0c068d90 legacy=0x7fcb0c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 monclient: found mon.noname-b 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb0a59c640 1 -- 172.21.15.153:0/1788569626 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0a1210 con 0x7fcb0c1318f0 2024-09-15T03:28:43.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.812+0000 7fcb09d9b640 1 --2- 172.21.15.153:0/1788569626 >> v2:172.21.15.153:3300/0 conn(0x7fcb0c09ea50 0x7fcb0c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.815+0000 7fcb0a59c640 1 -- 172.21.15.153:0/1788569626 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 msgr2=0x7fcb0c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.815+0000 7fcb0a59c640 1 --2- 172.21.15.153:0/1788569626 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c131f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009850 tx=0x7fcafc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2524ff76ab6c2527 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 1 -- 172.21.15.153:0/1788569626 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc02adf0 con 0x7fcb0c1318f0 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: dump: 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.824 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: _finish_auth 0 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 1 -- 172.21.15.153:0/1788569626 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc032440 con 0x7fcb0c1318f0 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 5 monclient: authenticate success, global_id 4307 2024-09-15T03:28:43.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 1 -- 172.21.15.153:0/1788569626 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0327e0 con 0x7fcb0c1318f0 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 4 set_mon_vals no callback set 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb0959a640 10 monclient: dump: 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 10 monclient: shutdown 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 -- 172.21.15.153:0/1788569626 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 msgr2=0x7fcb0c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 --2- 172.21.15.153:0/1788569626 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c131f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009850 tx=0x7fcafc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 -- 172.21.15.153:0/1788569626 shutdown_connections 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 --2- 172.21.15.153:0/1788569626 >> v2:172.21.15.153:3300/0 conn(0x7fcb0c09ea50 0x7fcb0c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 --2- 172.21.15.153:0/1788569626 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c131f30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 -- 172.21.15.153:0/1788569626 >> 172.21.15.153:0/1788569626 conn(0x7fcb0c067110 msgr2=0x7fcb0c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 -- 172.21.15.153:0/1788569626 shutdown_connections 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.816+0000 7fcb1089f640 1 -- 172.21.15.153:0/1788569626 wait complete. 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient: build_initial_monmap 2024-09-15T03:28:43.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient: monmap: 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 Processor -- start 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 -- start start 2024-09-15T03:28:43.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient: init 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb0c068d90 0x7fcb0c09ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0c068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 0x7fcb0c130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): picked mon.noname-c con 0x7fcb0c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c09b610 con 0x7fcb0c09ea50 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c09e750 con 0x7fcb0c068d90 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:43.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c11dd30 con 0x7fcb0c1318f0 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.817+0000 7fcb1089f640 10 monclient(hunting): authenticate will time out at 721.456954s 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 0x7fcb0c130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 0x7fcb0c130c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49776/0 (socket says 172.21.15.153:49776) 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45026/0 (socket says 172.21.15.153:45026) 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 1 -- 172.21.15.153:0/23127188 learned_addr learned my addr 172.21.15.153:0/23127188 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): get_auth_request con 0x7fcb0c09ea50 auth_method 0 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): get_auth_request con 0x7fcb0c1318f0 auth_method 0 2024-09-15T03:28:43.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0a59c640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb0c068d90 0x7fcb0c09ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 -- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb0c068d90 msgr2=0x7fcb0c09ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb0c068d90 0x7fcb0c09ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 -- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 msgr2=0x7fcb0c130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:43.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 0x7fcb0c130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 monclient: found mon.noname-c 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb0ad9d640 1 -- 172.21.15.153:0/23127188 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb00009390 con 0x7fcb0c1318f0 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.818+0000 7fcb09d9b640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 0x7fcb0c130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb0ad9d640 1 -- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 msgr2=0x7fcb0c140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb0ad9d640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c140fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400cbc0 tx=0x7fcb040052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400a850 con 0x7fcb0c1318f0 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: dump: 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.832 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0400a9d0 con 0x7fcb0c1318f0 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 5 monclient: authenticate success, global_id 4310 2024-09-15T03:28:43.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 10 monclient: _renew_subs 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc009390 con 0x7fcb0c1318f0 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400ad90 con 0x7fcb0c1318f0 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcafa7fc640 10 monclient: dump: 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:43.834 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 10 monclient: _renew_subs 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c11c8d0 con 0x7fcb0c1318f0 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.820+0000 7fcb08d99640 4 set_mon_vals no callback set 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb04007300 con 0x7fcb0c1318f0 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcb1089f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcb1089f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcb1089f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c11e120 con 0x7fcb0c1318f0 2024-09-15T03:28:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcafa7fc640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaec038740 0x7fcaec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.821+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fcb0404b9e0 con 0x7fcb0c1318f0 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.822+0000 7fcb0a59c640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaec038740 0x7fcaec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.822+0000 7fcb0a59c640 10 monclient: get_auth_request con 0x7fcaec038740 auth_method 0 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.824+0000 7fcb0a59c640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaec038740 0x7fcaec03abf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c064050 tx=0x7fcafc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.826+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcb040075b0 con 0x7fcb0c1318f0 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.826+0000 7fcafa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:43.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:43.826+0000 7fcafa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.011+0000 7fcb1089f640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-15T03:28:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.012+0000 7fcb1089f640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-15T03:28:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.012+0000 7fcb1089f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.012+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcb0c11e410 con 0x7fcb0c1318f0 2024-09-15T03:28:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.012+0000 7fcafa7fc640 1 -- 172.21.15.153:0/23127188 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fcb04010af0 con 0x7fcb0c1318f0 2024-09-15T03:28:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.012+0000 7fcafa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-15T03:28:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.012+0000 7fcafa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:44.014 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:44.014 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T03:28:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaec038740 msgr2=0x7fcaec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaec038740 0x7fcaec03abf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c064050 tx=0x7fcafc032000 comp rx=0 tx=0).stop 2024-09-15T03:28:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 10 monclient: shutdown 2024-09-15T03:28:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 msgr2=0x7fcb0c140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c140fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400cbc0 tx=0x7fcb040052b0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 shutdown_connections 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaec038740 0x7fcaec03abf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb0c1318f0 0x7fcb0c140fa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c09ea50 0x7fcb0c130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 --2- 172.21.15.153:0/23127188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb0c068d90 0x7fcb0c09ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 >> 172.21.15.153:0/23127188 conn(0x7fcb0c067110 msgr2=0x7fcb0c0a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 shutdown_connections 2024-09-15T03:28:44.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.014+0000 7fcb1089f640 1 -- 172.21.15.153:0/23127188 wait complete. 2024-09-15T03:28:44.025 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T03:28:44.025 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-15T03:28:44.039 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-15T03:28:44.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:28:44.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.131+0000 7f6786489640 10 monclient: get_monmap_and_config 2024-09-15T03:28:44.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.131+0000 7f6786489640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.131+0000 7f6786489640 10 monclient: monmap: 2024-09-15T03:28:44.132 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.132 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.132 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:44.132013+0000 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:44.132013+0000 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:44.133 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.131+0000 7f6786489640 1 Processor -- start 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 1 -- start start 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient: init 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6780096c60 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6780097030 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:44.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 0x7f6780136f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient(hunting): picked mon.noname-b con 0x7f6780134b50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67800696a0 con 0x7f6780097030 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678009d2e0 con 0x7f6780096c60 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678009d6b0 con 0x7f6780134b50 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6786489640 10 monclient(hunting): authenticate will time out at 721.771560s 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 0x7f6780136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f677f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6780097030 0x7f6780097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48428/0 (socket says 172.21.15.153:48428) 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f677f7fe640 1 -- 172.21.15.153:0/1619835596 learned_addr learned my addr 172.21.15.153:0/1619835596 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): get_auth_request con 0x7f6780134b50 auth_method 0 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 1 -- 172.21.15.153:0/1619835596 >> v1:172.21.15.181:6789/0 conn(0x7f6780096c60 legacy=0x7f678009a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 1 -- 172.21.15.153:0/1619835596 >> v1:172.21.15.153:6789/0 conn(0x7f6780097030 legacy=0x7f6780097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 1 monclient: found mon.noname-b 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.132+0000 7f6784a26640 1 -- 172.21.15.153:0/1619835596 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67800992f0 con 0x7f6780134b50 2024-09-15T03:28:44.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f6784a26640 1 -- 172.21.15.153:0/1619835596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 msgr2=0x7f6780136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f6784a26640 1 --2- 172.21.15.153:0/1619835596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 0x7f6780136f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6778009e90 tx=0x7f6778009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdce2892dc5ec9f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 1 -- 172.21.15.153:0/1619835596 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778007840 con 0x7f6780134b50 2024-09-15T03:28:44.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: dump: 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.144 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 1 -- 172.21.15.153:0/1619835596 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677802d050 con 0x7f6780134b50 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f6786489640 5 monclient: authenticate success, global_id 4313 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f6786489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f6786489640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 1 -- 172.21.15.153:0/1619835596 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778007bd0 con 0x7f6780134b50 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f6786489640 4 set_mon_vals no callback set 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.135+0000 7f677effd640 10 monclient: dump: 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.145 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 10 monclient: shutdown 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 -- 172.21.15.153:0/1619835596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 msgr2=0x7f6780136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 --2- 172.21.15.153:0/1619835596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 0x7f6780136f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6778009e90 tx=0x7f6778009780 comp rx=0 tx=0).stop 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 -- 172.21.15.153:0/1619835596 shutdown_connections 2024-09-15T03:28:44.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 --2- 172.21.15.153:0/1619835596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780134b50 0x7f6780136f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 -- 172.21.15.153:0/1619835596 >> 172.21.15.153:0/1619835596 conn(0x7f67800672b0 msgr2=0x7f67800676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 -- 172.21.15.153:0/1619835596 shutdown_connections 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 1 -- 172.21.15.153:0/1619835596 wait complete. 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.136+0000 7f6786489640 10 monclient: monmap: 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:44.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 Processor -- start 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 -- start start 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient: init 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6780096c60 0x7f678012ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): picked mon.noname-b con 0x7f6780096c60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 0x7f678009b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): picked mon.noname-c con 0x7f6780097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 0x7f678009bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): picked mon.noname-a con 0x7f6780134b50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:44.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678006e500 con 0x7f6780134b50 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678012fda0 con 0x7f6780096c60 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678012fff0 con 0x7f6780097030 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.137+0000 7f6786489640 10 monclient(hunting): authenticate will time out at 721.776926s 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 0x7f678009b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 0x7f678009bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 0x7f678009b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45050/0 (socket says 172.21.15.153:45050) 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 0x7f678009bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49790/0 (socket says 172.21.15.153:49790) 2024-09-15T03:28:44.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 -- 172.21.15.153:0/3172434983 learned_addr learned my addr 172.21.15.153:0/3172434983 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): get_auth_request con 0x7f6780097030 auth_method 0 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): get_auth_request con 0x7f6780134b50 auth_method 0 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677ffff640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6780096c60 0x7f678012ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 -- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6780096c60 msgr2=0x7f678012ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6780096c60 0x7f678012ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 -- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 msgr2=0x7f678009bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 0x7f678009bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 monclient: found mon.noname-c 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f677f7fe640 1 -- 172.21.15.153:0/3172434983 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778009a20 con 0x7f6780097030 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.138+0000 7f6784a26640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 0x7f678009bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:44.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f677f7fe640 1 -- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 msgr2=0x7f678009b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f677f7fe640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 0x7f678009b810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f676800c5e0 tx=0x7f6768004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768005cb0 con 0x7f6780097030 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: dump: 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.152 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 5 monclient: authenticate success, global_id 4316 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 10 monclient: _renew_subs 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6768005e30 con 0x7f6780097030 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678009add0 con 0x7f6780097030 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768007340 con 0x7f6780097030 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:44.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6777fff640 10 monclient: dump: 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f677e7fc640 4 set_mon_vals no callback set 2024-09-15T03:28:44.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 10 monclient: _renew_subs 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.140+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678006dcd0 con 0x7f6780097030 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f676801b400 con 0x7f6780097030 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6786489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6786489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6786489640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6780130130 con 0x7f6780097030 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6777fff640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6760038740 0x7f676003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.141+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f676804b4c0 con 0x7f6780097030 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.142+0000 7f677ffff640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6760038740 0x7f676003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.142+0000 7f677ffff640 10 monclient: get_auth_request con 0x7f6760038740 auth_method 0 2024-09-15T03:28:44.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.144+0000 7f677ffff640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6760038740 0x7f676003abf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6770002750 tx=0x7f6770009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.146+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6780130130 con 0x7f6780097030 2024-09-15T03:28:44.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.146+0000 7f6777fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.146+0000 7f6777fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:44.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6786489640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:44.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6786489640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:44.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6786489640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6780130130 con 0x7f6780097030 2024-09-15T03:28:44.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6777fff640 1 -- 172.21.15.153:0/3172434983 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11409 (secure 0 0 0) 0x7f6780130130 con 0x7f6780097030 2024-09-15T03:28:44.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6777fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:28:44.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.301+0000 7f6777fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:44.303 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:44.303 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":17,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:28:43.512239+0000","last_up_change":"2024-09-15T03:28:32.859989+0000","last_in_change":"2024-09-15T03:28:26.522990+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:28:33.881566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:28:34.684266+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":31922},{"type":"v1","addr":"172.21.15.153:6801","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":31922},{"type":"v1","addr":"172.21.15.153:6803","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":31922},{"type":"v1","addr":"172.21.15.153:6807","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":31922},{"type":"v1","addr":"172.21.15.153:6805","nonce":31922}]},"public_addr":"172.21.15.153:6801/31922","cluster_addr":"172.21.15.153:6803/31922","heartbeat_back_addr":"172.21.15.153:6807/31922","heartbeat_front_addr":"172.21.15.153:6805/31922","state":["exists","up"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":31923},{"type":"v1","addr":"172.21.15.153:6809","nonce":31923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":31923},{"type":"v1","addr":"172.21.15.153:6811","nonce":31923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":31923},{"type":"v1","addr":"172.21.15.153:6815","nonce":31923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":31923},{"type":"v1","addr":"172.21.15.153:6813","nonce":31923}]},"public_addr":"172.21.15.153:6809/31923","cluster_addr":"172.21.15.153:6811/31923","heartbeat_back_addr":"172.21.15.153:6815/31923","heartbeat_front_addr":"172.21.15.153:6813/31923","state":["exists","up"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":31924},{"type":"v1","addr":"172.21.15.153:6817","nonce":31924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":31924},{"type":"v1","addr":"172.21.15.153:6819","nonce":31924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":31924},{"type":"v1","addr":"172.21.15.153:6823","nonce":31924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":31924},{"type":"v1","addr":"172.21.15.153:6821","nonce":31924}]},"public_addr":"172.21.15.153:6817/31924","cluster_addr":"172.21.15.153:6819/31924","heartbeat_back_addr":"172.21.15.153:6823/31924","heartbeat_front_addr":"172.21.15.153:6821/31924","state":["exists","up"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":31874},{"type":"v1","addr":"172.21.15.181:6809","nonce":31874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":31874},{"type":"v1","addr":"172.21.15.181:6811","nonce":31874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":31874},{"type":"v1","addr":"172.21.15.181:6815","nonce":31874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":31874},{"type":"v1","addr":"172.21.15.181:6813","nonce":31874}]},"public_addr":"172.21.15.181:6809/31874","cluster_addr":"172.21.15.181:6811/31874","heartbeat_back_addr":"172.21.15.181:6815/31874","heartbeat_front_addr":"172.21.15.181:6813/31874","state":["exists","up"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":31875},{"type":"v1","addr":"172.21.15.181:6801","nonce":31875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":31875},{"type":"v1","addr":"172.21.15.181:6803","nonce":31875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":31875},{"type":"v1","addr":"172.21.15.181:6807","nonce":31875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":31875},{"type":"v1","addr":"172.21.15.181:6805","nonce":31875}]},"public_addr":"172.21.15.181:6801/31875","cluster_addr":"172.21.15.181:6803/31875","heartbeat_back_addr":"172.21.15.181:6807/31875","heartbeat_front_addr":"172.21.15.181:6805/31875","state":["exists","up"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":31876},{"type":"v1","addr":"172.21.15.181:6817","nonce":31876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":31876},{"type":"v1","addr":"172.21.15.181:6819","nonce":31876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":31876},{"type":"v1","addr":"172.21.15.181:6823","nonce":31876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":31876},{"type":"v1","addr":"172.21.15.181:6821","nonce":31876}]},"public_addr":"172.21.15.181:6817/31876","cluster_addr":"172.21.15.181:6819/31876","heartbeat_back_addr":"172.21.15.181:6823/31876","heartbeat_front_addr":"172.21.15.181:6821/31876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.685491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.637606+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.740014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.612976+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.627273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.624592+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6760038740 msgr2=0x7f676003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6760038740 0x7f676003abf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6770002750 tx=0x7f6770009040 comp rx=0 tx=0).stop 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 10 monclient: shutdown 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 msgr2=0x7f678009b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 0x7f678009b810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f676800c5e0 tx=0x7f6768004960 comp rx=0 tx=0).stop 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 shutdown_connections 2024-09-15T03:28:44.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.303+0000 7f6786489640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6760038740 0x7f676003abf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.304+0000 7f6786489640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6780134b50 0x7f678009bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.304+0000 7f6786489640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6780097030 0x7f678009b810 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.304+0000 7f6786489640 1 --2- 172.21.15.153:0/3172434983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6780096c60 0x7f678012ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.304+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 >> 172.21.15.153:0/3172434983 conn(0x7f67800672b0 msgr2=0x7f6780136b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.304+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 shutdown_connections 2024-09-15T03:28:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.304+0000 7f6786489640 1 -- 172.21.15.153:0/3172434983 wait complete. 2024-09-15T03:28:44.315 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:28:33.881566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:28:34.684266+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-15T03:28:44.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T03:28:44.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.410+0000 7fbe02edd640 10 monclient: get_monmap_and_config 2024-09-15T03:28:44.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.410+0000 7fbe02edd640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.410+0000 7fbe02edd640 10 monclient: monmap: 2024-09-15T03:28:44.411 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.411 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.411 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:44.411056+0000 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:44.411056+0000 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:44.412 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 Processor -- start 2024-09-15T03:28:44.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 -- start start 2024-09-15T03:28:44.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient: init 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbdfc09eb00 0x7fbdfc09eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc09eb00 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbdfc09d000 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbdfc09d400 0x7fbdfc0693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc09d400 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc0a00c0 con 0x7fbdfc09d000 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdfc09fe10 con 0x7fbdfc09eb00 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdfc0a11f0 con 0x7fbdfc09d400 2024-09-15T03:28:44.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe02edd640 10 monclient(hunting): authenticate will time out at 722.050696s 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbdfc09eb00 0x7fbdfc09eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbdfc09eb00 0x7fbdfc09eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49798/0 (socket says 172.21.15.153:49798) 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 1 -- 172.21.15.153:0/3246577420 learned_addr learned my addr 172.21.15.153:0/3246577420 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): get_auth_request con 0x7fbdfc09eb00 auth_method 0 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.411+0000 7fbe00c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-09-15T03:28:44.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 1 -- 172.21.15.153:0/3246577420 >> v2:172.21.15.181:3300/0 conn(0x7fbdfc09d400 msgr2=0x7fbdfc0693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe0147a640 1 --2- 172.21.15.153:0/3246577420 >> v2:172.21.15.181:3300/0 conn(0x7fbdfc09d400 0x7fbdfc0693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 1 --2- 172.21.15.153:0/3246577420 >> v2:172.21.15.181:3300/0 conn(0x7fbdfc09d400 0x7fbdfc0693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 1 -- 172.21.15.153:0/3246577420 >> v1:172.21.15.181:6789/0 conn(0x7fbdfc09d000 legacy=0x7fbdfc068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 1 monclient: found mon.noname-a 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.412+0000 7fbe00c79640 1 -- 172.21.15.153:0/3246577420 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc0a14d0 con 0x7fbdfc09eb00 2024-09-15T03:28:44.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe00c79640 1 -- 172.21.15.153:0/3246577420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 msgr2=0x7fbdfc09eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe00c79640 1 --2- 172.21.15.153:0/3246577420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc09eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0a17e0 tx=0x7fbdf40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0f3a7fdb59ccdb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 1 -- 172.21.15.153:0/3246577420 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4032440 con 0x7fbdfc09eb00 2024-09-15T03:28:44.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:44.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: dump: 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 5 monclient: authenticate success, global_id 4329 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 1 -- 172.21.15.153:0/3246577420 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf40325a0 con 0x7fbdfc09eb00 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 4 set_mon_vals no callback set 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 1 -- 172.21.15.153:0/3246577420 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4032980 con 0x7fbdfc09eb00 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbdfb7fe640 10 monclient: dump: 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.425 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 10 monclient: shutdown 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 -- 172.21.15.153:0/3246577420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 msgr2=0x7fbdfc09eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 --2- 172.21.15.153:0/3246577420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc09eed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0a17e0 tx=0x7fbdf40091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 -- 172.21.15.153:0/3246577420 shutdown_connections 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 --2- 172.21.15.153:0/3246577420 >> v2:172.21.15.181:3300/0 conn(0x7fbdfc09d400 0x7fbdfc0693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 --2- 172.21.15.153:0/3246577420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc09eed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 -- 172.21.15.153:0/3246577420 >> 172.21.15.153:0/3246577420 conn(0x7fbdfc067d30 msgr2=0x7fbdfc0a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 -- 172.21.15.153:0/3246577420 shutdown_connections 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.415+0000 7fbe02edd640 1 -- 172.21.15.153:0/3246577420 wait complete. 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.416+0000 7fbe02edd640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.416+0000 7fbe02edd640 10 monclient: monmap: 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:44.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.416+0000 7fbe02edd640 1 Processor -- start 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.416+0000 7fbe02edd640 1 -- start start 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.416+0000 7fbe02edd640 10 monclient: init 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdfc09d000 0x7fbdfc140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc09d000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdfc09d400 0x7fbdfc132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc09d400 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc11d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc09eb00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc120fc0 con 0x7fbdfc09eb00 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc11f700 con 0x7fbdfc09d400 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc09bf10 con 0x7fbdfc09d000 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe02edd640 10 monclient(hunting): authenticate will time out at 722.056237s 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc11d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdfc09d000 0x7fbdfc140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc11d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49812/0 (socket says 172.21.15.153:49812) 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 -- 172.21.15.153:0/1232337853 learned_addr learned my addr 172.21.15.153:0/1232337853 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): get_auth_request con 0x7fbdfc09d000 auth_method 0 2024-09-15T03:28:44.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): get_auth_request con 0x7fbdfc09eb00 auth_method 0 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbdfbfff640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdfc09d400 0x7fbdfc132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbdfbfff640 10 monclient(hunting): get_auth_request con 0x7fbdfc09d400 auth_method 0 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbdfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbdfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbdfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 -- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdfc09d000 msgr2=0x7fbdfc140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdfc09d000 0x7fbdfc140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 -- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdfc09d400 msgr2=0x7fbdfc132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdfc09d400 0x7fbdfc132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 monclient: found mon.noname-a 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe0147a640 1 -- 172.21.15.153:0/1232337853 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4009450 con 0x7fbdfc09eb00 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbdfbfff640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdfc09d400 0x7fbdfc132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.417+0000 7fbe00c79640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdfc09d000 0x7fbdfc140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.419+0000 7fbe0147a640 1 -- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 msgr2=0x7fbdfc11d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.419+0000 7fbe0147a640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc11d670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c880 tx=0x7fbde4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.419+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4007830 con 0x7fbdfc09eb00 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.419+0000 7fbdf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: dump: 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4007990 con 0x7fbdfc09eb00 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 5 monclient: authenticate success, global_id 4332 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4007d50 con 0x7fbdfc09eb00 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 monclient: _renew_subs 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc11fd80 con 0x7fbdfc09eb00 2024-09-15T03:28:44.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 10 monclient: dump: 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdfaffd640 4 set_mon_vals no callback set 2024-09-15T03:28:44.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 monclient: _renew_subs 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc11ea50 con 0x7fbdfc09eb00 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbde4011600 con 0x7fbdfc09eb00 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.420+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdfc05ed70 con 0x7fbdfc09eb00 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.421+0000 7fbdf8ff9640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbddc038740 0x7fbddc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.421+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fbde404bac0 con 0x7fbdfc09eb00 2024-09-15T03:28:44.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.421+0000 7fbe00c79640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbddc038740 0x7fbddc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.421+0000 7fbe00c79640 10 monclient: get_auth_request con 0x7fbddc038740 auth_method 0 2024-09-15T03:28:44.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.423+0000 7fbe00c79640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbddc038740 0x7fbddc03abf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0a1100 tx=0x7fbdf4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.425+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbde4021ae0 con 0x7fbdfc09eb00 2024-09-15T03:28:44.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.425+0000 7fbdf8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.425+0000 7fbdf8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:44.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbe02edd640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:28:44.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbe02edd640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:28:44.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbe02edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbdfc066240 con 0x7fbdfc09eb00 2024-09-15T03:28:44.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbdf8ff9640 1 -- 172.21.15.153:0/1232337853 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbde401f020 con 0x7fbdfc09eb00 2024-09-15T03:28:44.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbdf8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:28:44.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.579+0000 7fbdf8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:44.581 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbddc038740 msgr2=0x7fbddc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbddc038740 0x7fbddc03abf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0a1100 tx=0x7fbdf4032000 comp rx=0 tx=0).stop 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 10 monclient: shutdown 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 msgr2=0x7fbdfc11d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc11d670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c880 tx=0x7fbde4008040 comp rx=0 tx=0).stop 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 shutdown_connections 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbddc038740 0x7fbddc03abf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdfc09eb00 0x7fbdfc11d670 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdfc09d400 0x7fbdfc132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 --2- 172.21.15.153:0/1232337853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdfc09d000 0x7fbdfc140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.581+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 >> 172.21.15.153:0/1232337853 conn(0x7fbdfc067d30 msgr2=0x7fbdfc1339f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.582+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 shutdown_connections 2024-09-15T03:28:44.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.582+0000 7fbe02edd640 1 -- 172.21.15.153:0/1232337853 wait complete. 2024-09-15T03:28:44.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-15T03:28:44.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.687+0000 7fd38cd5d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.687+0000 7fd38cd5d640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.687+0000 7fd38cd5d640 10 monclient: monmap: 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:44.688546+0000 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:44.688546+0000 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:44.689 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:44.690 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.688+0000 7fd38cd5d640 1 Processor -- start 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.688+0000 7fd38cd5d640 1 -- start start 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.688+0000 7fd38cd5d640 10 monclient: init 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd38809eb00 0x7fd38809eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fd38809eb00 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd38809d000 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd38809d400 0x7fd3880693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fd38809d400 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3880a00c0 con 0x7fd38809d000 2024-09-15T03:28:44.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd38809fe10 con 0x7fd38809d400 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3880a11f0 con 0x7fd38809eb00 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38cd5d640 10 monclient(hunting): authenticate will time out at 722.328086s 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd38809d400 0x7fd3880693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd385d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd38809d000 0x7fd388068ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48436/0 (socket says 172.21.15.153:48436) 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd385d9b640 1 -- 172.21.15.153:0/137096852 learned_addr learned my addr 172.21.15.153:0/137096852 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): get_auth_request con 0x7fd38809d400 auth_method 0 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd38659c640 1 --2- 172.21.15.153:0/137096852 >> v2:172.21.15.181:3300/0 conn(0x7fd38809eb00 0x7fd38809eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 1 -- 172.21.15.153:0/137096852 >> v2:172.21.15.181:3300/0 conn(0x7fd38809eb00 msgr2=0x7fd38809eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 1 --2- 172.21.15.153:0/137096852 >> v2:172.21.15.181:3300/0 conn(0x7fd38809eb00 0x7fd38809eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 1 -- 172.21.15.153:0/137096852 >> v1:172.21.15.153:6789/0 conn(0x7fd38809d000 legacy=0x7fd388068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 1 monclient: found mon.noname-a 2024-09-15T03:28:44.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:44.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.689+0000 7fd386d9d640 1 -- 172.21.15.153:0/137096852 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3880a14d0 con 0x7fd38809d400 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd386d9d640 1 -- 172.21.15.153:0/137096852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809d400 msgr2=0x7fd3880693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd386d9d640 1 --2- 172.21.15.153:0/137096852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809d400 0x7fd3880693f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3880a04a0 tx=0x7fd378009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388bb1c25458a706 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 1 -- 172.21.15.153:0/137096852 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378007840 con 0x7fd38809d400 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: dump: 2024-09-15T03:28:44.700 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 5 monclient: authenticate success, global_id 4335 2024-09-15T03:28:44.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 1 -- 172.21.15.153:0/137096852 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37802d050 con 0x7fd38809d400 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 1 -- 172.21.15.153:0/137096852 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378007bd0 con 0x7fd38809d400 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 4 set_mon_vals no callback set 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38559a640 10 monclient: dump: 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 10 monclient: shutdown 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 1 -- 172.21.15.153:0/137096852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809d400 msgr2=0x7fd3880693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.692+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/137096852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809d400 0x7fd3880693f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3880a04a0 tx=0x7fd378009780 comp rx=0 tx=0).stop 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 -- 172.21.15.153:0/137096852 shutdown_connections 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/137096852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809d400 0x7fd3880693f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/137096852 >> v2:172.21.15.181:3300/0 conn(0x7fd38809eb00 0x7fd38809eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 -- 172.21.15.153:0/137096852 >> 172.21.15.153:0/137096852 conn(0x7fd388067d30 msgr2=0x7fd3880a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 -- 172.21.15.153:0/137096852 shutdown_connections 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 -- 172.21.15.153:0/137096852 wait complete. 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 10 monclient: monmap: 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:44.704 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 Processor -- start 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 1 -- start start 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.693+0000 7fd38cd5d640 10 monclient: init 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 0x7fd388140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fd38809d000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd38809d400 0x7fd388134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fd38809d400 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 0x7fd38811e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fd38809eb00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd388121c80 con 0x7fd38809eb00 2024-09-15T03:28:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38809bf10 con 0x7fd38809d400 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3881206d0 con 0x7fd38809d000 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38cd5d640 10 monclient(hunting): authenticate will time out at 722.333368s 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 0x7fd388140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 0x7fd38811e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 0x7fd388140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45074/0 (socket says 172.21.15.153:45074) 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 0x7fd38811e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49836/0 (socket says 172.21.15.153:49836) 2024-09-15T03:28:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 -- 172.21.15.153:0/3520163150 learned_addr learned my addr 172.21.15.153:0/3520163150 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): get_auth_request con 0x7fd38809d000 auth_method 0 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): get_auth_request con 0x7fd38809eb00 auth_method 0 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd385d9b640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd38809d400 0x7fd388134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd385d9b640 10 monclient(hunting): get_auth_request con 0x7fd38809d400 auth_method 0 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd385d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd385d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd385d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 -- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd38809d400 msgr2=0x7fd388134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd38809d400 0x7fd388134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 -- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 msgr2=0x7fd38811e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 0x7fd38811e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 monclient: found mon.noname-c 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd38659c640 1 -- 172.21.15.153:0/3520163150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378009a20 con 0x7fd38809d000 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.694+0000 7fd386d9d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 0x7fd38811e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.696+0000 7fd38659c640 1 -- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 msgr2=0x7fd388140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.696+0000 7fd38659c640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 0x7fd388140cf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd38000c880 tx=0x7fd380008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38000ce10 con 0x7fd38809d000 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: dump: 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.710 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 5 monclient: authenticate success, global_id 4322 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd380007a40 con 0x7fd38809d000 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 10 monclient: _renew_subs 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3880a3500 con 0x7fd38809d000 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380007de0 con 0x7fd38809d000 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 10 monclient: dump: 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd384d99640 4 set_mon_vals no callback set 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 10 monclient: _renew_subs 2024-09-15T03:28:44.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3881201c0 con 0x7fd38809d000 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.697+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd380011600 con 0x7fd38809d000 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd38cd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd38cd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd38cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3881220f0 con 0x7fd38809d000 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd3767fc640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd368038770 0x7fd36803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fd38004b7f0 con 0x7fd38809d000 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd385d9b640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd368038770 0x7fd36803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.698+0000 7fd385d9b640 10 monclient: get_auth_request con 0x7fd368038770 auth_method 0 2024-09-15T03:28:44.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.700+0000 7fd385d9b640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd368038770 0x7fd36803ac20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd37c007fd0 tx=0x7fd37c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.702+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd380020ae0 con 0x7fd38809d000 2024-09-15T03:28:44.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.702+0000 7fd3767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.702+0000 7fd3767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd38cd5d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:28:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd38cd5d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:28:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd38cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd38805ed70 con 0x7fd38809d000 2024-09-15T03:28:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd3767fc640 1 -- 172.21.15.153:0/3520163150 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd38001e070 con 0x7fd38809d000 2024-09-15T03:28:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd3767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:28:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.857+0000 7fd3767fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:44.859 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 8 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd368038770 msgr2=0x7fd36803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd368038770 0x7fd36803ac20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd37c007fd0 tx=0x7fd37c009040 comp rx=0 tx=0).stop 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 10 monclient: shutdown 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 msgr2=0x7fd388140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 0x7fd388140cf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd38000c880 tx=0x7fd380008040 comp rx=0 tx=0).stop 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 shutdown_connections 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd368038770 0x7fd36803ac20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd38809eb00 0x7fd38811e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd38809d400 0x7fd388134b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 --2- 172.21.15.153:0/3520163150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd38809d000 0x7fd388140cf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 >> 172.21.15.153:0/3520163150 conn(0x7fd388067d30 msgr2=0x7fd388068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 shutdown_connections 2024-09-15T03:28:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.859+0000 7fd38cd5d640 1 -- 172.21.15.153:0/3520163150 wait complete. 2024-09-15T03:28:44.870 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-15T03:28:44.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.962+0000 7f786dc94640 10 monclient: get_monmap_and_config 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.962+0000 7f786dc94640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.962+0000 7f786dc94640 10 monclient: monmap: 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:44.962972+0000 2024-09-15T03:28:44.963 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:44.962972+0000 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:44.964 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 Processor -- start 2024-09-15T03:28:44.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 -- start start 2024-09-15T03:28:44.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient: init 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 0x7f7868131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): picked mon.noname-b con 0x7f78681318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f786809ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7868136c20 0x7f7868139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): picked mon.noname-c con 0x7f7868136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7868131cc0 con 0x7f786809ea50 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786809f530 con 0x7f7868136c20 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786809fbf0 con 0x7f78681318f0 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f786dc94640 10 monclient(hunting): authenticate will time out at 722.602669s 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 0x7f7868131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f785ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f786809ea50 0x7f786809ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48440/0 (socket says 172.21.15.153:48440) 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f785ffff640 1 -- 172.21.15.153:0/1595439328 learned_addr learned my addr 172.21.15.153:0/1595439328 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): get_auth_request con 0x7f78681318f0 auth_method 0 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.963+0000 7f78677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-15T03:28:44.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 1 -- 172.21.15.153:0/1595439328 >> v2:172.21.15.181:3300/0 conn(0x7f7868136c20 msgr2=0x7f7868139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f7867fff640 1 --2- 172.21.15.153:0/1595439328 >> v2:172.21.15.181:3300/0 conn(0x7f7868136c20 0x7f7868139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 1 --2- 172.21.15.153:0/1595439328 >> v2:172.21.15.181:3300/0 conn(0x7f7868136c20 0x7f7868139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 1 -- 172.21.15.153:0/1595439328 >> v1:172.21.15.153:6789/0 conn(0x7f786809ea50 legacy=0x7f786809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 1 monclient: found mon.noname-b 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.964+0000 7f78677fe640 1 -- 172.21.15.153:0/1595439328 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786809fe30 con 0x7f78681318f0 2024-09-15T03:28:44.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.966+0000 7f78677fe640 1 -- 172.21.15.153:0/1595439328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 msgr2=0x7f7868131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.966+0000 7f78677fe640 1 --2- 172.21.15.153:0/1595439328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 0x7f7868131f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7858009850 tx=0x7f78580090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7c658bc3ef1f10 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.966+0000 7f7866ffd640 1 -- 172.21.15.153:0/1595439328 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858032440 con 0x7f78681318f0 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: dump: 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.975 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 5 monclient: authenticate success, global_id 4325 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 1 -- 172.21.15.153:0/1595439328 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78580325a0 con 0x7f78681318f0 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 4 set_mon_vals no callback set 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 1 -- 172.21.15.153:0/1595439328 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858032980 con 0x7f78681318f0 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f7866ffd640 10 monclient: dump: 2024-09-15T03:28:44.976 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 10 monclient: shutdown 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 -- 172.21.15.153:0/1595439328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 msgr2=0x7f7868131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 --2- 172.21.15.153:0/1595439328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 0x7f7868131f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7858009850 tx=0x7f78580090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 -- 172.21.15.153:0/1595439328 shutdown_connections 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 --2- 172.21.15.153:0/1595439328 >> v2:172.21.15.181:3300/0 conn(0x7f7868136c20 0x7f7868139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 --2- 172.21.15.153:0/1595439328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78681318f0 0x7f7868131f30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 -- 172.21.15.153:0/1595439328 >> 172.21.15.153:0/1595439328 conn(0x7f7868067110 msgr2=0x7f7868067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 -- 172.21.15.153:0/1595439328 shutdown_connections 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.967+0000 7f786dc94640 1 -- 172.21.15.153:0/1595439328 wait complete. 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient: build_initial_monmap 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient: monmap: 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:44.978 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 Processor -- start 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 -- start start 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient: init 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 0x7f7868141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient(hunting): picked mon.noname-c con 0x7f786809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78681318f0 0x7f7868130ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient(hunting): picked mon.noname-a con 0x7f78681318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7868136c20 0x7f78680999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient(hunting): picked mon.noname-b con 0x7f7868136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786811d1a0 con 0x7f78681318f0 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786811cc50 con 0x7f7868136c20 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.968+0000 7f786dc94640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786811cec0 con 0x7f786809ea50 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f786dc94640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 0x7f7868141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f786dc94640 10 monclient(hunting): authenticate will time out at 722.608065s 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78681318f0 0x7f7868130ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 0x7f7868141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45088/0 (socket says 172.21.15.153:45088) 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 -- 172.21.15.153:0/4219497545 learned_addr learned my addr 172.21.15.153:0/4219497545 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): get_auth_request con 0x7f786809ea50 auth_method 0 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): get_auth_request con 0x7f78681318f0 auth_method 0 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7868136c20 0x7f78680999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): get_auth_request con 0x7f7868136c20 auth_method 0 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f7867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 -- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7868136c20 msgr2=0x7f78680999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7868136c20 0x7f78680999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 -- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78681318f0 msgr2=0x7f7868130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78681318f0 0x7f7868130ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 monclient: found mon.noname-c 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f78677fe640 1 -- 172.21.15.153:0/4219497545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858009390 con 0x7f786809ea50 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.969+0000 7f785ffff640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78681318f0 0x7f7868130ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f78677fe640 1 -- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 msgr2=0x7f7868141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f78677fe640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 0x7f7868141170 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7858009850 tx=0x7f7858032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858000d10 con 0x7f786809ea50 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: dump: 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7858031c70 con 0x7f786809ea50 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 5 monclient: authenticate success, global_id 4328 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 10 monclient: _renew_subs 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7868141fa0 con 0x7f786809ea50 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785803d440 con 0x7f786809ea50 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f785f7fe640 10 monclient: dump: 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 10 monclient: _renew_subs 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78681416b0 con 0x7f786809ea50 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.971+0000 7f78667fc640 4 set_mon_vals no callback set 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7858039040 con 0x7f786809ea50 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f786dc94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f786dc94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f786dc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786811b720 con 0x7f786809ea50 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f785f7fe640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7848038790 0x7f784803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.972+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f785806b9d0 con 0x7f786809ea50 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.973+0000 7f785ffff640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7848038790 0x7f784803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.973+0000 7f785ffff640 10 monclient: get_auth_request con 0x7f7848038790 auth_method 0 2024-09-15T03:28:44.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.975+0000 7f785ffff640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7848038790 0x7f784803ac40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7850007fd0 tx=0x7f7850007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:44.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.977+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f785802f2a0 con 0x7f786809ea50 2024-09-15T03:28:44.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.977+0000 7f785f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:44.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:44.977+0000 7f785f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:45.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.128+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f786811d000 con 0x7f7848038790 2024-09-15T03:28:45.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.130+0000 7f785f7fe640 1 -- 172.21.15.153:0/4219497545 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41016 (secure 0 0 0) 0x7f786811d000 con 0x7f7848038790 2024-09-15T03:28:45.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:45.132 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7848038790 msgr2=0x7f784803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7848038790 0x7f784803ac40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7850007fd0 tx=0x7f7850007f00 comp rx=0 tx=0).stop 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 10 monclient: shutdown 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 msgr2=0x7f7868141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 0x7f7868141170 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7858009850 tx=0x7f7858032000 comp rx=0 tx=0).stop 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 shutdown_connections 2024-09-15T03:28:45.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7848038790 0x7f784803ac40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7868136c20 0x7f78680999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78681318f0 0x7f7868130ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 --2- 172.21.15.153:0/4219497545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f786809ea50 0x7f7868141170 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 >> 172.21.15.153:0/4219497545 conn(0x7f7868067110 msgr2=0x7f78680a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 shutdown_connections 2024-09-15T03:28:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.132+0000 7f786dc94640 1 -- 172.21.15.153:0/4219497545 wait complete. 2024-09-15T03:28:45.143 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-15T03:28:43.844645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786240,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444800,"statfs":{"total":579820584960,"available":577991475200,"internally_reserved":0,"allocated":1994752,"data_stored":1344052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000632"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771558+0000","last_change":"2024-09-15T03:28:38.771829+0000","last_active":"2024-09-15T03:28:38.771558+0000","last_peered":"2024-09-15T03:28:38.771558+0000","last_clean":"2024-09-15T03:28:38.771558+0000","last_became_active":"2024-09-15T03:28:37.587284+0000","last_became_peered":"2024-09-15T03:28:37.587284+0000","last_unstale":"2024-09-15T03:28:38.771558+0000","last_undegraded":"2024-09-15T03:28:38.771558+0000","last_fullsized":"2024-09-15T03:28:38.771558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:11.288432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760004+0000","last_change":"2024-09-15T03:28:38.760264+0000","last_active":"2024-09-15T03:28:38.760004+0000","last_peered":"2024-09-15T03:28:38.760004+0000","last_clean":"2024-09-15T03:28:38.760004+0000","last_became_active":"2024-09-15T03:28:36.752311+0000","last_became_peered":"2024-09-15T03:28:36.752311+0000","last_unstale":"2024-09-15T03:28:38.760004+0000","last_undegraded":"2024-09-15T03:28:38.760004+0000","last_fullsized":"2024-09-15T03:28:38.760004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:47:34.502515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771599+0000","last_change":"2024-09-15T03:28:38.771789+0000","last_active":"2024-09-15T03:28:38.771599+0000","last_peered":"2024-09-15T03:28:38.771599+0000","last_clean":"2024-09-15T03:28:38.771599+0000","last_became_active":"2024-09-15T03:28:36.757024+0000","last_became_peered":"2024-09-15T03:28:36.757024+0000","last_unstale":"2024-09-15T03:28:38.771599+0000","last_undegraded":"2024-09-15T03:28:38.771599+0000","last_fullsized":"2024-09-15T03:28:38.771599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:33:31.289175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760078+0000","last_change":"2024-09-15T03:28:38.760299+0000","last_active":"2024-09-15T03:28:38.760078+0000","last_peered":"2024-09-15T03:28:38.760078+0000","last_clean":"2024-09-15T03:28:38.760078+0000","last_became_active":"2024-09-15T03:28:36.749668+0000","last_became_peered":"2024-09-15T03:28:36.749668+0000","last_unstale":"2024-09-15T03:28:38.760078+0000","last_undegraded":"2024-09-15T03:28:38.760078+0000","last_fullsized":"2024-09-15T03:28:38.760078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:19:51.050262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762840+0000","last_change":"2024-09-15T03:28:38.762840+0000","last_active":"2024-09-15T03:28:38.762840+0000","last_peered":"2024-09-15T03:28:38.762840+0000","last_clean":"2024-09-15T03:28:38.762840+0000","last_became_active":"2024-09-15T03:28:36.757667+0000","last_became_peered":"2024-09-15T03:28:36.757667+0000","last_unstale":"2024-09-15T03:28:38.762840+0000","last_undegraded":"2024-09-15T03:28:38.762840+0000","last_fullsized":"2024-09-15T03:28:38.762840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:01.577552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.759527+0000","last_change":"2024-09-15T03:28:38.759813+0000","last_active":"2024-09-15T03:28:38.759527+0000","last_peered":"2024-09-15T03:28:38.759527+0000","last_clean":"2024-09-15T03:28:38.759527+0000","last_became_active":"2024-09-15T03:28:36.757615+0000","last_became_peered":"2024-09-15T03:28:36.757615+0000","last_unstale":"2024-09-15T03:28:38.759527+0000","last_undegraded":"2024-09-15T03:28:38.759527+0000","last_fullsized":"2024-09-15T03:28:38.759527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:08.664245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771831+0000","last_change":"2024-09-15T03:28:38.772005+0000","last_active":"2024-09-15T03:28:38.771831+0000","last_peered":"2024-09-15T03:28:38.771831+0000","last_clean":"2024-09-15T03:28:38.771831+0000","last_became_active":"2024-09-15T03:28:36.756993+0000","last_became_peered":"2024-09-15T03:28:36.756993+0000","last_unstale":"2024-09-15T03:28:38.771831+0000","last_undegraded":"2024-09-15T03:28:38.771831+0000","last_fullsized":"2024-09-15T03:28:38.771831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.632961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762201+0000","last_change":"2024-09-15T03:28:38.762403+0000","last_active":"2024-09-15T03:28:38.762201+0000","last_peered":"2024-09-15T03:28:38.762201+0000","last_clean":"2024-09-15T03:28:38.762201+0000","last_became_active":"2024-09-15T03:28:36.757463+0000","last_became_peered":"2024-09-15T03:28:36.757463+0000","last_unstale":"2024-09-15T03:28:38.762201+0000","last_undegraded":"2024-09-15T03:28:38.762201+0000","last_fullsized":"2024-09-15T03:28:38.762201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:23.854099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":100,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-15T03:28:36.997951+0000","last_change":"2024-09-15T03:28:35.748574+0000","last_active":"2024-09-15T03:28:36.997951+0000","last_peered":"2024-09-15T03:28:36.997951+0000","last_clean":"2024-09-15T03:28:36.997951+0000","last_became_active":"2024-09-15T03:28:35.747712+0000","last_became_peered":"2024-09-15T03:28:35.747712+0000","last_unstale":"2024-09-15T03:28:36.997951+0000","last_undegraded":"2024-09-15T03:28:36.997951+0000","last_fullsized":"2024-09-15T03:28:36.997951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:54:27.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":155648,"data_stored":47234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":151552,"data_stored":47215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298152,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073688,"statfs":{"total":96636764160,"available":96331456512,"internally_reserved":0,"allocated":684032,"data_stored":576143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":159744,"data_stored":48190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073808,"statfs":{"total":96636764160,"available":96331579392,"internally_reserved":0,"allocated":688128,"data_stored":577099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:28:45.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:28:45.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.240+0000 7f2d3fb9d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:45.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.240+0000 7f2d3fb9d640 10 monclient: build_initial_monmap 2024-09-15T03:28:45.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.240+0000 7f2d3fb9d640 10 monclient: monmap: 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:45.240950+0000 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:45.240950+0000 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:45.246 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.240+0000 7f2d3fb9d640 1 Processor -- start 2024-09-15T03:28:45.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 1 -- start start 2024-09-15T03:28:45.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient: init 2024-09-15T03:28:45.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:45.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2d381318f0 0x7f2d38131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d381318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d3809ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d38136c20 0x7f2d38139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d38136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3809f530 con 0x7f2d3809ea50 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d38131d50 con 0x7f2d38136c20 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3809fbf0 con 0x7f2d381318f0 2024-09-15T03:28:45.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3fb9d640 10 monclient(hunting): authenticate will time out at 722.880566s 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d38136c20 0x7f2d38139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d38136c20 0x7f2d38139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46050/0 (socket says 172.21.15.153:46050) 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 learned_addr learned my addr 172.21.15.153:0/85381901 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): get_auth_request con 0x7f2d38136c20 auth_method 0 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3d939640 1 --2- 172.21.15.153:0/85381901 >> v2:172.21.15.181:3300/0 conn(0x7f2d381318f0 0x7f2d38131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3d939640 10 monclient(hunting): get_auth_request con 0x7f2d381318f0 auth_method 0 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3d939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3d939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3d939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 >> v2:172.21.15.181:3300/0 conn(0x7f2d381318f0 msgr2=0x7f2d38131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 --2- 172.21.15.153:0/85381901 >> v2:172.21.15.181:3300/0 conn(0x7f2d381318f0 0x7f2d38131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 >> v1:172.21.15.181:6789/0 conn(0x7f2d3809ea50 legacy=0x7f2d3809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient: _finish_hunting 0 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 monclient: found mon.noname-a 2024-09-15T03:28:45.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.241+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3809fe30 con 0x7f2d38136c20 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 msgr2=0x7f2d38139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3e13a640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 0x7f2d38139000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d38076600 tx=0x7f2d30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e628b3495b85ec52 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 msgr2=0x7f2d38139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 msgr2=0x7f2d38139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3e13a640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 0x7f2d38139000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d38076600 tx=0x7f2d30009840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3e13a640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 0x7f2d38139000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2d38076600 tx=0x7f2d30009840 comp rx=0 tx=0).stop 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:45.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 1 -- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 msgr2=0x7f2d38139000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 0x7f2d38139000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient: _add_conns ranks=[0,2] 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d240034c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d240073e0 0x7f2d24009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient(hunting): picked mon.noname-b con 0x7f2d240073e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 1 -- 172.21.15.153:0/85381901 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2400a180 con 0x7f2d240034c0 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 1 -- 172.21.15.153:0/85381901 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2400b380 con 0x7f2d240073e0 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:45.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3c937640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3d138640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d240073e0 0x7f2d24009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3d138640 10 monclient(hunting): get_auth_request con 0x7f2d240073e0 auth_method 0 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3d138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3d138640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.244+0000 7f2d3d138640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:28:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 10 monclient(hunting): handle_auth_done global_id 4344 payload 293 2024-09-15T03:28:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 1 -- 172.21.15.153:0/85381901 >> v1:172.21.15.153:6789/0 conn(0x7f2d240034c0 legacy=0x7f2d24005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 10 monclient: _finish_hunting 0 2024-09-15T03:28:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 1 monclient: found mon.noname-b 2024-09-15T03:28:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.245+0000 7f2d3d138640 1 -- 172.21.15.153:0/85381901 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2400b600 con 0x7f2d240073e0 2024-09-15T03:28:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3d138640 1 -- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d240073e0 msgr2=0x7f2d24009890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3d138640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d240073e0 0x7f2d24009890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2d340026c0 tx=0x7f2d34007ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78027fedf8318824 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 1 -- 172.21.15.153:0/85381901 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3400c8a0 con 0x7f2d240073e0 2024-09-15T03:28:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: dump: 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: _finish_auth 0 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3fb9d640 5 monclient: authenticate success, global_id 4344 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3fb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 1 -- 172.21.15.153:0/85381901 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3400ca00 con 0x7f2d240073e0 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 1 -- 172.21.15.153:0/85381901 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3400cdc0 con 0x7f2d240073e0 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3fb9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3fb9d640 4 set_mon_vals no callback set 2024-09-15T03:28:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.247+0000 7f2d3c937640 10 monclient: dump: 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 10 monclient: shutdown 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d240073e0 msgr2=0x7f2d24009890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d240073e0 0x7f2d24009890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2d340026c0 tx=0x7f2d34007ca0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3e13a640 1 -- 172.21.15.153:0/85381901 reap_dead start 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/85381901 shutdown_connections 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/85381901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d38136c20 0x7f2d38139000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/85381901 >> 172.21.15.153:0/85381901 conn(0x7f2d38067110 msgr2=0x7f2d38067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/85381901 shutdown_connections 2024-09-15T03:28:45.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/85381901 wait complete. 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 10 monclient: build_initial_monmap 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 10 monclient: monmap: 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:45.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.248+0000 7f2d3fb9d640 1 Processor -- start 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 -- start start 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient: init 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 0x7f2d381412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d38136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d240073e0 0x7f2d38141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d240073e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:45.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3811d930 0x7f2d380994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d3811d930 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3809af90 con 0x7f2d3811d930 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3809daa0 con 0x7f2d240073e0 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3811c980 con 0x7f2d38136c20 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3fb9d640 10 monclient(hunting): authenticate will time out at 722.888513s 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 0x7f2d381412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 0x7f2d381412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45194/0 (socket says 172.21.15.153:45194) 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 -- 172.21.15.153:0/2654837228 learned_addr learned my addr 172.21.15.153:0/2654837228 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): get_auth_request con 0x7f2d38136c20 auth_method 0 2024-09-15T03:28:45.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3811d930 0x7f2d380994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): get_auth_request con 0x7f2d3811d930 auth_method 0 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 -- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d240073e0 msgr2=0x7f2d38141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d240073e0 0x7f2d38141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 -- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3811d930 msgr2=0x7f2d380994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3811d930 0x7f2d380994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient: _finish_hunting 0 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 monclient: found mon.noname-c 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3d939640 1 -- 172.21.15.153:0/2654837228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3811c6a0 con 0x7f2d38136c20 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.249+0000 7f2d3e13a640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3811d930 0x7f2d380994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3d939640 1 -- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 msgr2=0x7f2d381412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3d939640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 0x7f2d381412c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00d8f0 tx=0x7f2d2c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00b8d0 con 0x7f2d38136c20 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: dump: 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.277 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c00ba50 con 0x7f2d38136c20 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 5 monclient: authenticate success, global_id 4331 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 10 monclient: _renew_subs 2024-09-15T03:28:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00be10 con 0x7f2d38136c20 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3811b760 con 0x7f2d38136c20 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d29ffb640 10 monclient: dump: 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d2bfff640 4 set_mon_vals no callback set 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 10 monclient: _renew_subs 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.252+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d38141ee0 con 0x7f2d38136c20 2024-09-15T03:28:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2d2c014640 con 0x7f2d38136c20 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d29ffb640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d1c038740 0x7f2d1c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f2d2c04bf80 con 0x7f2d38136c20 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d3fb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d3fb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d3fb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3811cd80 con 0x7f2d38136c20 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d3d138640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d1c038740 0x7f2d1c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.253+0000 7f2d3d138640 10 monclient: get_auth_request con 0x7f2d1c038740 auth_method 0 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.256+0000 7f2d3d138640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d1c038740 0x7f2d1c03abf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d38130fd0 tx=0x7f2d34015000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.257+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2d2c01e460 con 0x7f2d38136c20 2024-09-15T03:28:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.257+0000 7f2d29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.257+0000 7f2d29ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.412+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d3804bf60 con 0x7f2d1c038740 2024-09-15T03:28:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.414+0000 7f2d29ffb640 1 -- 172.21.15.153:0/2654837228 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41016 (secure 0 0 0) 0x7f2d3804bf60 con 0x7f2d1c038740 2024-09-15T03:28:45.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:28:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d1c038740 msgr2=0x7f2d1c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d1c038740 0x7f2d1c03abf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d38130fd0 tx=0x7f2d34015000 comp rx=0 tx=0).stop 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 10 monclient: shutdown 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 msgr2=0x7f2d381412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 0x7f2d381412c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00d8f0 tx=0x7f2d2c008040 comp rx=0 tx=0).stop 2024-09-15T03:28:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 shutdown_connections 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d1c038740 0x7f2d1c03abf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3811d930 0x7f2d380994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d240073e0 0x7f2d38141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 --2- 172.21.15.153:0/2654837228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d38136c20 0x7f2d381412c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 >> 172.21.15.153:0/2654837228 conn(0x7f2d38067110 msgr2=0x7f2d380a3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 shutdown_connections 2024-09-15T03:28:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.416+0000 7f2d3fb9d640 1 -- 172.21.15.153:0/2654837228 wait complete. 2024-09-15T03:28:45.429 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-15T03:28:43.844645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786240,"kb_used_data":1948,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444800,"statfs":{"total":579820584960,"available":577991475200,"internally_reserved":0,"allocated":1994752,"data_stored":1344052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000632"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771558+0000","last_change":"2024-09-15T03:28:38.771829+0000","last_active":"2024-09-15T03:28:38.771558+0000","last_peered":"2024-09-15T03:28:38.771558+0000","last_clean":"2024-09-15T03:28:38.771558+0000","last_became_active":"2024-09-15T03:28:37.587284+0000","last_became_peered":"2024-09-15T03:28:37.587284+0000","last_unstale":"2024-09-15T03:28:38.771558+0000","last_undegraded":"2024-09-15T03:28:38.771558+0000","last_fullsized":"2024-09-15T03:28:38.771558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:11.288432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760004+0000","last_change":"2024-09-15T03:28:38.760264+0000","last_active":"2024-09-15T03:28:38.760004+0000","last_peered":"2024-09-15T03:28:38.760004+0000","last_clean":"2024-09-15T03:28:38.760004+0000","last_became_active":"2024-09-15T03:28:36.752311+0000","last_became_peered":"2024-09-15T03:28:36.752311+0000","last_unstale":"2024-09-15T03:28:38.760004+0000","last_undegraded":"2024-09-15T03:28:38.760004+0000","last_fullsized":"2024-09-15T03:28:38.760004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:47:34.502515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771599+0000","last_change":"2024-09-15T03:28:38.771789+0000","last_active":"2024-09-15T03:28:38.771599+0000","last_peered":"2024-09-15T03:28:38.771599+0000","last_clean":"2024-09-15T03:28:38.771599+0000","last_became_active":"2024-09-15T03:28:36.757024+0000","last_became_peered":"2024-09-15T03:28:36.757024+0000","last_unstale":"2024-09-15T03:28:38.771599+0000","last_undegraded":"2024-09-15T03:28:38.771599+0000","last_fullsized":"2024-09-15T03:28:38.771599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:33:31.289175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.760078+0000","last_change":"2024-09-15T03:28:38.760299+0000","last_active":"2024-09-15T03:28:38.760078+0000","last_peered":"2024-09-15T03:28:38.760078+0000","last_clean":"2024-09-15T03:28:38.760078+0000","last_became_active":"2024-09-15T03:28:36.749668+0000","last_became_peered":"2024-09-15T03:28:36.749668+0000","last_unstale":"2024-09-15T03:28:38.760078+0000","last_undegraded":"2024-09-15T03:28:38.760078+0000","last_fullsized":"2024-09-15T03:28:38.760078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:19:51.050262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762840+0000","last_change":"2024-09-15T03:28:38.762840+0000","last_active":"2024-09-15T03:28:38.762840+0000","last_peered":"2024-09-15T03:28:38.762840+0000","last_clean":"2024-09-15T03:28:38.762840+0000","last_became_active":"2024-09-15T03:28:36.757667+0000","last_became_peered":"2024-09-15T03:28:36.757667+0000","last_unstale":"2024-09-15T03:28:38.762840+0000","last_undegraded":"2024-09-15T03:28:38.762840+0000","last_fullsized":"2024-09-15T03:28:38.762840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:01.577552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.759527+0000","last_change":"2024-09-15T03:28:38.759813+0000","last_active":"2024-09-15T03:28:38.759527+0000","last_peered":"2024-09-15T03:28:38.759527+0000","last_clean":"2024-09-15T03:28:38.759527+0000","last_became_active":"2024-09-15T03:28:36.757615+0000","last_became_peered":"2024-09-15T03:28:36.757615+0000","last_unstale":"2024-09-15T03:28:38.759527+0000","last_undegraded":"2024-09-15T03:28:38.759527+0000","last_fullsized":"2024-09-15T03:28:38.759527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:51:08.664245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.771831+0000","last_change":"2024-09-15T03:28:38.772005+0000","last_active":"2024-09-15T03:28:38.771831+0000","last_peered":"2024-09-15T03:28:38.771831+0000","last_clean":"2024-09-15T03:28:38.771831+0000","last_became_active":"2024-09-15T03:28:36.756993+0000","last_became_peered":"2024-09-15T03:28:36.756993+0000","last_unstale":"2024-09-15T03:28:38.771831+0000","last_undegraded":"2024-09-15T03:28:38.771831+0000","last_fullsized":"2024-09-15T03:28:38.771831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.632961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:28:38.762201+0000","last_change":"2024-09-15T03:28:38.762403+0000","last_active":"2024-09-15T03:28:38.762201+0000","last_peered":"2024-09-15T03:28:38.762201+0000","last_clean":"2024-09-15T03:28:38.762201+0000","last_became_active":"2024-09-15T03:28:36.757463+0000","last_became_peered":"2024-09-15T03:28:36.757463+0000","last_unstale":"2024-09-15T03:28:38.762201+0000","last_undegraded":"2024-09-15T03:28:38.762201+0000","last_fullsized":"2024-09-15T03:28:38.762201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:17:23.854099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":100,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-15T03:28:36.997951+0000","last_change":"2024-09-15T03:28:35.748574+0000","last_active":"2024-09-15T03:28:36.997951+0000","last_peered":"2024-09-15T03:28:36.997951+0000","last_clean":"2024-09-15T03:28:36.997951+0000","last_became_active":"2024-09-15T03:28:35.747712+0000","last_became_peered":"2024-09-15T03:28:35.747712+0000","last_unstale":"2024-09-15T03:28:36.997951+0000","last_undegraded":"2024-09-15T03:28:36.997951+0000","last_fullsized":"2024-09-15T03:28:36.997951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:54:27.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":155648,"data_stored":47234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":151552,"data_stored":47215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298152,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073688,"statfs":{"total":96636764160,"available":96331456512,"internally_reserved":0,"allocated":684032,"data_stored":576143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":159744,"data_stored":48190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298032,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073808,"statfs":{"total":96636764160,"available":96331579392,"internally_reserved":0,"allocated":688128,"data_stored":577099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:28:45.430 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-15T03:28:45.430 DEBUG:tasks.mon_thrash:Yielding 2024-09-15T03:28:45.431 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T03:28:45.778 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:28:45.779 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T03:28:45.779 DEBUG:teuthology.orchestra.run.smithi181:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:28:45.780 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-15T03:28:45.780 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726370925, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-15T03:28:45.780 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:28:45.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:28:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:28:45.800 INFO:teuthology.orchestra.run.smithi181.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T03:28:45.801 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:28:45.859 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T03:28:45.859 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T03:28:45.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.873+0000 7febc1e46640 10 monclient: get_monmap_and_config 2024-09-15T03:28:45.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.873+0000 7febc1e46640 10 monclient: build_initial_monmap 2024-09-15T03:28:45.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.873+0000 7febc1e46640 10 monclient: monmap: 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:45.874463+0000 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:45.874463+0000 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:45.875 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:45.876 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.874+0000 7febc1e46640 1 Processor -- start 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.874+0000 7febc1e46640 1 -- start start 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.874+0000 7febc1e46640 10 monclient: init 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.874+0000 7febc1e46640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.874+0000 7febc1e46640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.874+0000 7febc1e46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febbc09caa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7febbc09a540 0x7febbc09ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 10 monclient(hunting): picked mon.noname-a con 0x7febbc09a540 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 0x7febbc136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 10 monclient(hunting): picked mon.noname-b con 0x7febbc1346c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc09abd0 con 0x7febbc09caa0 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc09ed10 con 0x7febbc09a540 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc09d3b0 con 0x7febbc1346c0 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febc1e46640 10 monclient(hunting): authenticate will time out at 723.514082s 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 0x7febbc136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbb7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7febbc09caa0 0x7febbc06ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47930/0 (socket says 172.21.15.153:47930) 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7febbc09a540 0x7febbc09ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbb7fe640 1 -- 172.21.15.153:0/560401582 learned_addr learned my addr 172.21.15.153:0/560401582 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): get_auth_request con 0x7febbc1346c0 auth_method 0 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): get_auth_request con 0x7febbc09a540 auth_method 0 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 1 -- 172.21.15.153:0/560401582 >> v2:172.21.15.153:3300/0 conn(0x7febbc09a540 msgr2=0x7febbc09ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 1 --2- 172.21.15.153:0/560401582 >> v2:172.21.15.153:3300/0 conn(0x7febbc09a540 0x7febbc09ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 1 -- 172.21.15.153:0/560401582 >> v1:172.21.15.153:6789/0 conn(0x7febbc09caa0 legacy=0x7febbc06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 1 monclient: found mon.noname-b 2024-09-15T03:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbbfff640 1 -- 172.21.15.153:0/560401582 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc09b920 con 0x7febbc1346c0 2024-09-15T03:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.875+0000 7febbaffd640 1 --2- 172.21.15.153:0/560401582 >> v2:172.21.15.153:3300/0 conn(0x7febbc09a540 0x7febbc09ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febbbfff640 1 -- 172.21.15.153:0/560401582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 msgr2=0x7febbc136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febbbfff640 1 --2- 172.21.15.153:0/560401582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 0x7febbc136aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7febb4009850 tx=0x7febb40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3802398f566f92 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 1 -- 172.21.15.153:0/560401582 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb402ae80 con 0x7febbc1346c0 2024-09-15T03:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: dump: 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:45.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 1 -- 172.21.15.153:0/560401582 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4032440 con 0x7febbc1346c0 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 5 monclient: authenticate success, global_id 4334 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 1 -- 172.21.15.153:0/560401582 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb40327c0 con 0x7febbc1346c0 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 4 set_mon_vals no callback set 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febba7fc640 10 monclient: dump: 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 10 monclient: shutdown 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 1 -- 172.21.15.153:0/560401582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 msgr2=0x7febbc136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 1 --2- 172.21.15.153:0/560401582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 0x7febbc136aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7febb4009850 tx=0x7febb40090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 1 -- 172.21.15.153:0/560401582 shutdown_connections 2024-09-15T03:28:45.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 1 --2- 172.21.15.153:0/560401582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc1346c0 0x7febbc136aa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.878+0000 7febc1e46640 1 --2- 172.21.15.153:0/560401582 >> v2:172.21.15.153:3300/0 conn(0x7febbc09a540 0x7febbc09ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.879+0000 7febc1e46640 1 -- 172.21.15.153:0/560401582 >> 172.21.15.153:0/560401582 conn(0x7febbc067cd0 msgr2=0x7febbc0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.879+0000 7febc1e46640 1 -- 172.21.15.153:0/560401582 shutdown_connections 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.879+0000 7febc1e46640 1 -- 172.21.15.153:0/560401582 wait complete. 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.879+0000 7febc1e46640 10 monclient: build_initial_monmap 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.879+0000 7febc1e46640 10 monclient: monmap: 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.879+0000 7febc1e46640 1 Processor -- start 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 -- start start 2024-09-15T03:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient: init 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 0x7febbc132ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): picked mon.noname-a con 0x7febbc09a540 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 0x7febbc133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): picked mon.noname-c con 0x7febbc09caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febbc1346c0 0x7febbc119fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): picked mon.noname-b con 0x7febbc1346c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc11d650 con 0x7febbc09a540 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc11c0a0 con 0x7febbc1346c0 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc11d330 con 0x7febbc09caa0 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febc1e46640 10 monclient(hunting): authenticate will time out at 723.519449s 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 0x7febbc132ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 0x7febbc133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 0x7febbc132ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46070/0 (socket says 172.21.15.153:46070) 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 1 -- 172.21.15.153:0/3418013362 learned_addr learned my addr 172.21.15.153:0/3418013362 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): get_auth_request con 0x7febbc09a540 auth_method 0 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): get_auth_request con 0x7febbc09caa0 auth_method 0 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-15T03:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 -- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febbc1346c0 msgr2=0x7febbc119fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febbc1346c0 0x7febbc119fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 -- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 msgr2=0x7febbc132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 0x7febbc132ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 monclient: found mon.noname-c 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbaffd640 1 -- 172.21.15.153:0/3418013362 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb000f610 con 0x7febbc09caa0 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.880+0000 7febbb7fe640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 0x7febbc132ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febbaffd640 1 -- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 msgr2=0x7febbc133220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febbaffd640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 0x7febbc133220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7febb0009d20 tx=0x7febb0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb000f8f0 con 0x7febbc09caa0 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: dump: 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb0015040 con 0x7febbc09caa0 2024-09-15T03:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 5 monclient: authenticate success, global_id 4337 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 10 monclient: _renew_subs 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb001b670 con 0x7febbc09caa0 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb4009390 con 0x7febbc09caa0 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7feba37fe640 10 monclient: dump: 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febb9ffb640 4 set_mon_vals no callback set 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 10 monclient: _renew_subs 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.883+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc11bb20 con 0x7febbc09caa0 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7febb001ba60 con 0x7febbc09caa0 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7febc1e46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7febc1e46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7febc1e46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febbc04bf60 con 0x7febbc09caa0 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7feba37fe640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feba8038740 0x7feba803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7febb004fb00 con 0x7febbc09caa0 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7febbb7fe640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feba8038740 0x7feba803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.884+0000 7febbb7fe640 10 monclient: get_auth_request con 0x7feba8038740 auth_method 0 2024-09-15T03:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.887+0000 7febbb7fe640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feba8038740 0x7feba803abf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7febac00d7e0 tx=0x7febac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:45.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.889+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7febb00243b0 con 0x7febbc09caa0 2024-09-15T03:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.889+0000 7feba37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:45.889+0000 7feba37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:45.922 INFO:tasks.workunit:timeout=3h 2024-09-15T03:28:45.922 INFO:tasks.workunit:cleanup=True 2024-09-15T03:28:45.922 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:28:45.981 INFO:tasks.workunit.client.0.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T03:28:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7febc1e46640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:28:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7febc1e46640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7febc1e46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7febbc0638d0 con 0x7febbc09caa0 2024-09-15T03:28:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7feba37fe640 1 -- 172.21.15.153:0/3418013362 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7febb004fe60 con 0x7febbc09caa0 2024-09-15T03:28:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7feba37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.103+0000 7feba37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:46.105 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:28:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feba8038740 msgr2=0x7feba803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feba8038740 0x7feba803abf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7febac00d7e0 tx=0x7febac009040 comp rx=0 tx=0).stop 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 10 monclient: shutdown 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 msgr2=0x7febbc133220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 0x7febbc133220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7febb0009d20 tx=0x7febb0008040 comp rx=0 tx=0).stop 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 shutdown_connections 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feba8038740 0x7feba803abf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febbc1346c0 0x7febbc119fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febbc09caa0 0x7febbc133220 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 --2- 172.21.15.153:0/3418013362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc09a540 0x7febbc132ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 >> 172.21.15.153:0/3418013362 conn(0x7febbc067cd0 msgr2=0x7febbc0a1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 shutdown_connections 2024-09-15T03:28:46.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.105+0000 7febc1e46640 1 -- 172.21.15.153:0/3418013362 wait complete. 2024-09-15T03:28:46.117 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:28:46.117 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:28:46.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:28:46.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.210+0000 7fe2d3e60640 10 monclient: get_monmap_and_config 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.210+0000 7fe2d3e60640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.210+0000 7fe2d3e60640 10 monclient: monmap: 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:46.210813+0000 2024-09-15T03:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:46.210813+0000 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.210+0000 7fe2d3e60640 1 Processor -- start 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.210+0000 7fe2d3e60640 1 -- start start 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.210+0000 7fe2d3e60640 10 monclient: init 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): picked mon.noname-b con 0x7fe2cc1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2cc09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:46.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe2cc136c20 0x7fe2cc139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): picked mon.noname-a con 0x7fe2cc136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc131cc0 con 0x7fe2cc09ea50 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc09f530 con 0x7fe2cc136c20 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc09fbf0 con 0x7fe2cc1318f0 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d3e60640 10 monclient(hunting): authenticate will time out at 723.850401s 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe2cc136c20 0x7fe2cc139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d13fb640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe2cc09ea50 0x7fe2cc09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47942/0 (socket says 172.21.15.153:47942) 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): get_auth_request con 0x7fe2cc1318f0 auth_method 0 2024-09-15T03:28:46.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d13fb640 1 -- 172.21.15.153:0/1704526633 learned_addr learned my addr 172.21.15.153:0/1704526633 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): get_auth_request con 0x7fe2cc136c20 auth_method 0 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/1704526633 >> v2:172.21.15.153:3300/0 conn(0x7fe2cc136c20 msgr2=0x7fe2cc139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/1704526633 >> v2:172.21.15.153:3300/0 conn(0x7fe2cc136c20 0x7fe2cc139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/1704526633 >> v1:172.21.15.153:6789/0 conn(0x7fe2cc09ea50 legacy=0x7fe2cc09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 1 monclient: found mon.noname-b 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/1704526633 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc09fe30 con 0x7fe2cc1318f0 2024-09-15T03:28:46.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.211+0000 7fe2d23fd640 1 --2- 172.21.15.153:0/1704526633 >> v2:172.21.15.153:3300/0 conn(0x7fe2cc136c20 0x7fe2cc139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/1704526633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 msgr2=0x7fe2cc131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/1704526633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc131f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009850 tx=0x7fe2c40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=670b42fb8c89c124 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 1 -- 172.21.15.153:0/1704526633 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c402adf0 con 0x7fe2cc1318f0 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: dump: 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 1 -- 172.21.15.153:0/1704526633 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c4032440 con 0x7fe2cc1318f0 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d3e60640 5 monclient: authenticate success, global_id 4340 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d3e60640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 1 -- 172.21.15.153:0/1704526633 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40327e0 con 0x7fe2cc1318f0 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d3e60640 4 set_mon_vals no callback set 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.214+0000 7fe2d0bfa640 10 monclient: dump: 2024-09-15T03:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 10 monclient: shutdown 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 -- 172.21.15.153:0/1704526633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 msgr2=0x7fe2cc131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/1704526633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc131f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009850 tx=0x7fe2c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 -- 172.21.15.153:0/1704526633 shutdown_connections 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/1704526633 >> v2:172.21.15.153:3300/0 conn(0x7fe2cc136c20 0x7fe2cc139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/1704526633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc131f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 -- 172.21.15.153:0/1704526633 >> 172.21.15.153:0/1704526633 conn(0x7fe2cc067110 msgr2=0x7fe2cc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 -- 172.21.15.153:0/1704526633 shutdown_connections 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 1 -- 172.21.15.153:0/1704526633 wait complete. 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.215+0000 7fe2d3e60640 10 monclient: monmap: 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 Processor -- start 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 -- start start 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient: init 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2cc09ea50 0x7fe2cc1308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): picked mon.noname-b con 0x7fe2cc09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc0999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): picked mon.noname-c con 0x7fe2cc1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc136c20 0x7fe2cc099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): picked mon.noname-a con 0x7fe2cc136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc09bf10 con 0x7fe2cc136c20 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc09e060 con 0x7fe2cc09ea50 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc1374c0 con 0x7fe2cc1318f0 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d3e60640 10 monclient(hunting): authenticate will time out at 723.855768s 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc136c20 0x7fe2cc099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc0999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc136c20 0x7fe2cc099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46086/0 (socket says 172.21.15.153:46086) 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): get_auth_request con 0x7fe2cc1318f0 auth_method 0 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 1 -- 172.21.15.153:0/824419680 learned_addr learned my addr 172.21.15.153:0/824419680 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): get_auth_request con 0x7fe2cc136c20 auth_method 0 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d13fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.216+0000 7fe2d23fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2cc09ea50 0x7fe2cc1308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d1bfc640 10 monclient(hunting): get_auth_request con 0x7fe2cc09ea50 auth_method 0 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d1bfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2cc09ea50 msgr2=0x7fe2cc1308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2cc09ea50 0x7fe2cc1308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc136c20 msgr2=0x7fe2cc099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc136c20 0x7fe2cc099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 1 monclient: found mon.noname-c 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d13fb640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc009a20 con 0x7fe2cc1318f0 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.217+0000 7fe2d23fd640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc136c20 0x7fe2cc099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d13fb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 msgr2=0x7fe2cc0999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d13fb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc0999b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c4c0 tx=0x7fe2c0007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0005cb0 con 0x7fe2cc1318f0 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: dump: 2024-09-15T03:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0005e30 con 0x7fe2cc1318f0 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 5 monclient: authenticate success, global_id 4343 2024-09-15T03:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007320 con 0x7fe2cc1318f0 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2c9ffb640 10 monclient: dump: 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 10 monclient: _renew_subs 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2cbfff640 4 set_mon_vals no callback set 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c4009390 con 0x7fe2cc1318f0 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 10 monclient: _renew_subs 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.219+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc141e70 con 0x7fe2cc1318f0 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe2c001b3b0 con 0x7fe2cc1318f0 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d3e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 0x7fe2cc0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc065e00 con 0x7fe2cc11c9e0 2024-09-15T03:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 0x7fe2cc0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2c9ffb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2b8038740 0x7fe2b803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient: get_auth_request con 0x7fe2cc11c9e0 auth_method 0 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe2c004ae60 con 0x7fe2cc1318f0 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2cc11cfd0 con 0x7fe2cc11c9e0 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.220+0000 7fe2d23fd640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2b8038740 0x7fe2b803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.221+0000 7fe2d23fd640 10 monclient: get_auth_request con 0x7fe2b8038740 auth_method 0 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 msgr2=0x7fe2cc0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 0x7fe2cc0298c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4034000 tx=0x7fe2c4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4031a70 con 0x7fe2cc11c9e0 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 10 monclient: dump: 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2d23fd640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2b8038740 0x7fe2b803abf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc007230 tx=0x7fe2bc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe2c4031bd0 con 0x7fe2cc11c9e0 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 msgr2=0x7fe2cc0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 0x7fe2cc0298c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4034000 tx=0x7fe2c4032000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 msgr2=0x7fe2cc0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.223+0000 7fe2c9ffb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc11c9e0 0x7fe2cc0298c0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4034000 tx=0x7fe2c4032000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d3e60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 0x7fe2cc029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d3e60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc04bf60 con 0x7fe2cc130e10 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 0x7fe2cc029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient: get_auth_request con 0x7fe2cc130e10 auth_method 0 2024-09-15T03:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.229+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/824419680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2cc09b3e0 con 0x7fe2cc130e10 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.231+0000 7fe2d1bfc640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 msgr2=0x7fe2cc029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.231+0000 7fe2d1bfc640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 0x7fe2cc029e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc1369e0 tx=0x7fe2c403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c403c9e0 con 0x7fe2cc130e10 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 10 monclient: dump: 2024-09-15T03:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fe2c403cb40 con 0x7fe2cc130e10 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 msgr2=0x7fe2cc029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 0x7fe2cc029e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc1369e0 tx=0x7fe2c403c000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 msgr2=0x7fe2cc029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2c9ffb640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2cc130e10 0x7fe2cc029e00 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc1369e0 tx=0x7fe2c403c000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 4, 2024-09-15T03:28:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 22, 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2b8038740 msgr2=0x7fe2b803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2b8038740 0x7fe2b803abf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc007230 tx=0x7fe2bc00b040 comp rx=0 tx=0).stop 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 10 monclient: shutdown 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 msgr2=0x7fe2cc0999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc0999b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c4c0 tx=0x7fe2c0007f00 comp rx=0 tx=0).stop 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d23fd640 1 -- 172.21.15.153:0/824419680 reap_dead start 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 shutdown_connections 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 --2- 172.21.15.153:0/824419680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2cc1318f0 0x7fe2cc0999b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 >> 172.21.15.153:0/824419680 conn(0x7fe2cc067110 msgr2=0x7fe2cc0a4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 shutdown_connections 2024-09-15T03:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.232+0000 7fe2d3e60640 1 -- 172.21.15.153:0/824419680 wait complete. 2024-09-15T03:28:46.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:28:46.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:28:46.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.352+0000 7ff2406f8640 10 monclient: get_monmap_and_config 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.352+0000 7ff2406f8640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.352+0000 7ff2406f8640 10 monclient: monmap: 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:46.353480+0000 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:46.353480+0000 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 1 Processor -- start 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 1 -- start start 2024-09-15T03:28:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 10 monclient: init 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 0x7ff238096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.353+0000 7ff2406f8640 10 monclient(hunting): picked mon.noname-b con 0x7ff23809a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2380971e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff2380975b0 0x7ff23812fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 10 monclient(hunting): picked mon.noname-c con 0x7ff2380975b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23806ee10 con 0x7ff2380971e0 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23809cb20 con 0x7ff2380975b0 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23809a8e0 con 0x7ff23809a510 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff2406f8640 10 monclient(hunting): authenticate will time out at 723.993078s 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 0x7ff238096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23dc93640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff2380971e0 0x7ff23812f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47950/0 (socket says 172.21.15.153:47950) 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23dc93640 1 -- 172.21.15.153:0/1593456058 learned_addr learned my addr 172.21.15.153:0/1593456058 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): get_auth_request con 0x7ff23809a510 auth_method 0 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23ec95640 1 --2- 172.21.15.153:0/1593456058 >> v2:172.21.15.181:3300/0 conn(0x7ff2380975b0 0x7ff23812fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23ec95640 10 monclient(hunting): get_auth_request con 0x7ff2380975b0 auth_method 0 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23ec95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23ec95640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23ec95640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 1 -- 172.21.15.153:0/1593456058 >> v2:172.21.15.181:3300/0 conn(0x7ff2380975b0 msgr2=0x7ff23812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 1 --2- 172.21.15.153:0/1593456058 >> v2:172.21.15.181:3300/0 conn(0x7ff2380975b0 0x7ff23812fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 1 -- 172.21.15.153:0/1593456058 >> v1:172.21.15.153:6789/0 conn(0x7ff2380971e0 legacy=0x7ff23812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 1 monclient: found mon.noname-b 2024-09-15T03:28:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.354+0000 7ff23e494640 1 -- 172.21.15.153:0/1593456058 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23809d590 con 0x7ff23809a510 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23e494640 1 -- 172.21.15.153:0/1593456058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 msgr2=0x7ff238096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23e494640 1 --2- 172.21.15.153:0/1593456058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 0x7ff238096ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff234009850 tx=0x7ff2340090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=112d386f9af3b200 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 1 -- 172.21.15.153:0/1593456058 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234032440 con 0x7ff23809a510 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: dump: 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.361 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 1 -- 172.21.15.153:0/1593456058 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2340325a0 con 0x7ff23809a510 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 5 monclient: authenticate success, global_id 4346 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 1 -- 172.21.15.153:0/1593456058 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234032940 con 0x7ff23809a510 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 4 set_mon_vals no callback set 2024-09-15T03:28:46.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff23d492640 10 monclient: dump: 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 10 monclient: shutdown 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 1 -- 172.21.15.153:0/1593456058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 msgr2=0x7ff238096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.357+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1593456058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 0x7ff238096ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff234009850 tx=0x7ff2340090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 -- 172.21.15.153:0/1593456058 shutdown_connections 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1593456058 >> v2:172.21.15.181:3300/0 conn(0x7ff2380975b0 0x7ff23812fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1593456058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23809a510 0x7ff238096ca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 -- 172.21.15.153:0/1593456058 >> 172.21.15.153:0/1593456058 conn(0x7ff2380672b0 msgr2=0x7ff2380676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 -- 172.21.15.153:0/1593456058 shutdown_connections 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 -- 172.21.15.153:0/1593456058 wait complete. 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 10 monclient: monmap: 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.370 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 Processor -- start 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 1 -- start start 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.358+0000 7ff2406f8640 10 monclient: init 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:28:46.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 0x7ff238131be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): picked mon.noname-c con 0x7ff2380971e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2380975b0 0x7ff23813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): picked mon.noname-b con 0x7ff2380975b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff23809a510 0x7ff23813f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): picked mon.noname-a con 0x7ff23809a510 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23813f7c0 con 0x7ff23809a510 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23809c370 con 0x7ff2380975b0 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23811ede0 con 0x7ff2380971e0 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff2406f8640 10 monclient(hunting): authenticate will time out at 723.998338s 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 0x7ff238131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 0x7ff238131be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45252/0 (socket says 172.21.15.153:45252) 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 -- 172.21.15.153:0/1663521852 learned_addr learned my addr 172.21.15.153:0/1663521852 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): get_auth_request con 0x7ff2380971e0 auth_method 0 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff23809a510 0x7ff23813f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): get_auth_request con 0x7ff23809a510 auth_method 0 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23ec95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23dc93640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2380975b0 0x7ff23813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23dc93640 10 monclient(hunting): get_auth_request con 0x7ff2380975b0 auth_method 0 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23dc93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23dc93640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23dc93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2380975b0 msgr2=0x7ff23813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2380975b0 0x7ff23813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff23809a510 msgr2=0x7ff23813f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff23809a510 0x7ff23813f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 monclient: found mon.noname-c 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.359+0000 7ff23e494640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234009390 con 0x7ff2380971e0 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.361+0000 7ff23e494640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 msgr2=0x7ff238131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.361+0000 7ff23e494640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 0x7ff238131be0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff234009850 tx=0x7ff234032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234031c70 con 0x7ff2380971e0 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: dump: 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234031dd0 con 0x7ff2380971e0 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 5 monclient: authenticate success, global_id 4349 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 10 monclient: _renew_subs 2024-09-15T03:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23811fff0 con 0x7ff2380971e0 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23403c630 con 0x7ff2380971e0 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff22e7fc640 10 monclient: dump: 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.377 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff23cc91640 4 set_mon_vals no callback set 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 10 monclient: _renew_subs 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23811db10 con 0x7ff2380971e0 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 0x7ff23811c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.362+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff238120470 con 0x7ff238132120 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff23403ca20 con 0x7ff2380971e0 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 0x7ff23811c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient: get_auth_request con 0x7ff238132120 auth_method 0 2024-09-15T03:28:46.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff22e7fc640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff21c038740 0x7ff21c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23dc93640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff23804bf60 con 0x7ff238132120 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7ff23406c4b0 con 0x7ff2380971e0 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23ec95640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff21c038740 0x7ff21c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.363+0000 7ff23ec95640 10 monclient: get_auth_request con 0x7ff21c038740 auth_method 0 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff23dc93640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 msgr2=0x7ff23811c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff23dc93640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 0x7ff23811c730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff238067d60 tx=0x7ff228009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228004e60 con 0x7ff238132120 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 10 monclient: dump: 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff228007170 con 0x7ff238132120 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 msgr2=0x7ff23811c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 0x7ff23811c730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff238067d60 tx=0x7ff228009040 comp rx=0 tx=0).stop 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 msgr2=0x7ff23811c730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff22e7fc640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff238132120 0x7ff23811c730 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff238067d60 tx=0x7ff228009040 comp rx=0 tx=0).stop 2024-09-15T03:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.365+0000 7ff23ec95640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff21c038740 0x7ff21c03abf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff230007640 tx=0x7ff230009850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff2406f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 0x7ff23811d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff2406f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff238065f20 con 0x7ff23811cc70 2024-09-15T03:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 0x7ff23811d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient: get_auth_request con 0x7ff23811cc70 auth_method 0 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.371+0000 7ff23dc93640 1 -- 172.21.15.153:0/1663521852 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff23811d5b0 con 0x7ff23811cc70 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff23dc93640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 msgr2=0x7ff23811d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff23dc93640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 0x7ff23811d070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff228017000 tx=0x7ff22801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228007760 con 0x7ff23811cc70 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 10 monclient: dump: 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7ff22801c400 con 0x7ff23811cc70 2024-09-15T03:28:46.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 msgr2=0x7ff23811d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 0x7ff23811d070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff228017000 tx=0x7ff22801a000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 msgr2=0x7ff23811d070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff22e7fc640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff23811cc70 0x7ff23811d070 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff228017000 tx=0x7ff22801a000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:28:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 4, 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 22, 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:28:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:28:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:28:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:28:46.399 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff21c038740 msgr2=0x7ff21c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff21c038740 0x7ff21c03abf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff230007640 tx=0x7ff230009850 comp rx=0 tx=0).stop 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 10 monclient: shutdown 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 msgr2=0x7ff238131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 --2- 172.21.15.153:0/1663521852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2380971e0 0x7ff238131be0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff234009850 tx=0x7ff234032000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff23ec95640 1 -- 172.21.15.153:0/1663521852 reap_dead start 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff23ec95640 1 -- 172.21.15.153:0/1663521852 reap_dead start 2024-09-15T03:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 shutdown_connections 2024-09-15T03:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 >> 172.21.15.153:0/1663521852 conn(0x7ff2380672b0 msgr2=0x7ff23806cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 shutdown_connections 2024-09-15T03:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.374+0000 7ff2406f8640 1 -- 172.21.15.153:0/1663521852 wait complete. 2024-09-15T03:28:46.402 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:28:46.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:28:46.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:28:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.495+0000 7f4c75ab8640 10 monclient: get_monmap_and_config 2024-09-15T03:28:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.495+0000 7f4c75ab8640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.495+0000 7f4c75ab8640 10 monclient: monmap: 2024-09-15T03:28:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:46.496155+0000 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:46.496155+0000 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:46.497 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 1 Processor -- start 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 1 -- start start 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient: init 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c700974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:46.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c7009a8a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 0x7f4c70136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f4c70134080 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7012b210 con 0x7f4c700974f0 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c700978c0 con 0x7f4c7009a8a0 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7009ca20 con 0x7f4c70134080 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c75ab8640 10 monclient(hunting): authenticate will time out at 724.135690s 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 0x7f4c70136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4c700974f0 0x7f4c7012ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47952/0 (socket says 172.21.15.153:47952) 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6f7fe640 1 -- 172.21.15.153:0/3943228856 learned_addr learned my addr 172.21.15.153:0/3943228856 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request con 0x7f4c70134080 auth_method 0 2024-09-15T03:28:46.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.496+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 1 -- 172.21.15.153:0/3943228856 >> v1:172.21.15.181:6789/0 conn(0x7f4c7009a8a0 legacy=0x7f4c7009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 1 -- 172.21.15.153:0/3943228856 >> v1:172.21.15.153:6789/0 conn(0x7f4c700974f0 legacy=0x7f4c7012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 1 monclient: found mon.noname-b 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.497+0000 7f4c6ffff640 1 -- 172.21.15.153:0/3943228856 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c70098ad0 con 0x7f4c70134080 2024-09-15T03:28:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.499+0000 7f4c6ffff640 1 -- 172.21.15.153:0/3943228856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 msgr2=0x7f4c70136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.499+0000 7f4c6ffff640 1 --2- 172.21.15.153:0/3943228856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 0x7f4c70136460 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009e90 tx=0x7f4c68009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e655bd3874a05a3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 1 -- 172.21.15.153:0/3943228856 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68007840 con 0x7f4c70134080 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: dump: 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 1 -- 172.21.15.153:0/3943228856 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6802d050 con 0x7f4c70134080 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 5 monclient: authenticate success, global_id 4358 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 1 -- 172.21.15.153:0/3943228856 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68007bd0 con 0x7f4c70134080 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c6e7fc640 10 monclient: dump: 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 4 set_mon_vals no callback set 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 10 monclient: shutdown 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 -- 172.21.15.153:0/3943228856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 msgr2=0x7f4c70136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 --2- 172.21.15.153:0/3943228856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 0x7f4c70136460 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009e90 tx=0x7f4c68009780 comp rx=0 tx=0).stop 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 -- 172.21.15.153:0/3943228856 shutdown_connections 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 --2- 172.21.15.153:0/3943228856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70134080 0x7f4c70136460 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 -- 172.21.15.153:0/3943228856 >> 172.21.15.153:0/3943228856 conn(0x7f4c700672b0 msgr2=0x7f4c700676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 -- 172.21.15.153:0/3943228856 shutdown_connections 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.500+0000 7f4c75ab8640 1 -- 172.21.15.153:0/3943228856 wait complete. 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.501+0000 7f4c75ab8640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.501+0000 7f4c75ab8640 10 monclient: monmap: 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.501+0000 7f4c75ab8640 1 Processor -- start 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.501+0000 7f4c75ab8640 1 -- start start 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.501+0000 7f4c75ab8640 10 monclient: init 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700974f0 0x7f4c70133590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f4c700974f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 0x7f4c7013eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): picked mon.noname-c con 0x7f4c7009a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70134080 0x7f4c7013f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): picked mon.noname-a con 0x7f4c70134080 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7012c4a0 con 0x7f4c70134080 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70098300 con 0x7f4c700974f0 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7006ed00 con 0x7f4c7009a8a0 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c75ab8640 10 monclient(hunting): authenticate will time out at 724.141095s 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70134080 0x7f4c7013f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 0x7f4c7013eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70134080 0x7f4c7013f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46134/0 (socket says 172.21.15.153:46134) 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 0x7f4c7013eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45296/0 (socket says 172.21.15.153:45296) 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 1 -- 172.21.15.153:0/2013040495 learned_addr learned my addr 172.21.15.153:0/2013040495 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request con 0x7f4c70134080 auth_method 0 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): get_auth_request con 0x7f4c7009a8a0 auth_method 0 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700974f0 0x7f4c70133590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4c700974f0 auth_method 0 2024-09-15T03:28:46.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-15T03:28:46.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700974f0 msgr2=0x7f4c70133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700974f0 0x7f4c70133590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70134080 msgr2=0x7f4c7013f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70134080 0x7f4c7013f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 monclient: found mon.noname-c 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.502+0000 7f4c6effd640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68009a20 con 0x7f4c7009a8a0 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c6effd640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 msgr2=0x7f4c7013eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c6effd640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 0x7f4c7013eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400c4e0 tx=0x7f4c64007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c640117a0 con 0x7f4c7009a8a0 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: dump: 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.519 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 5 monclient: authenticate success, global_id 4361 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c64011920 con 0x7f4c7009a8a0 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 10 monclient: _renew_subs 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c70129ee0 con 0x7f4c7009a8a0 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c64011ce0 con 0x7f4c7009a8a0 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c577fe640 10 monclient: dump: 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.521 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 10 monclient: _renew_subs 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7006f3c0 con 0x7f4c7009a8a0 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.504+0000 7f4c6dffb640 4 set_mon_vals no callback set 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.505+0000 7f4c75ab8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.505+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4c64012630 con 0x7f4c7009a8a0 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.505+0000 7f4c75ab8640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 0x7f4c7006d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.505+0000 7f4c75ab8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.505+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70065f20 con 0x7f4c7013f950 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.505+0000 7f4c577fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038740 0x7f4c5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 0x7f4c7006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4c6404b520 con 0x7f4c7009a8a0 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient: get_auth_request con 0x7f4c7013f950 auth_method 0 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6ffff640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038740 0x7f4c5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6ffff640 10 monclient: get_auth_request con 0x7f4c5c038740 auth_method 0 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-09-15T03:28:46.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.506+0000 7f4c6f7fe640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c7004bf60 con 0x7f4c7013f950 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c6f7fe640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 msgr2=0x7f4c7006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c6f7fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 0x7f4c7006d490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000e710 tx=0x7f4c60007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c6ffff640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038740 0x7f4c5c03abf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c680097c0 tx=0x7f4c68037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60004b50 con 0x7f4c7013f950 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.508+0000 7f4c577fe640 10 monclient: dump: 2024-09-15T03:28:46.530 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4c60004cb0 con 0x7f4c7013f950 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:46.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 msgr2=0x7f4c7006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 0x7f4c7006d490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000e710 tx=0x7f4c60007f00 comp rx=0 tx=0).stop 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 msgr2=0x7f4c7006d490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.509+0000 7f4c577fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7013f950 0x7f4c7006d490 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000e710 tx=0x7f4c60007f00 comp rx=0 tx=0).stop 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.514+0000 7f4c75ab8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.514+0000 7f4c75ab8640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 0x7f4c70029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.514+0000 7f4c75ab8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.514+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70135b20 con 0x7f4c700298c0 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 0x7f4c70029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient: get_auth_request con 0x7f4c700298c0 auth_method 0 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.515+0000 7f4c6f7fe640 1 -- 172.21.15.153:0/2013040495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c700638d0 con 0x7f4c700298c0 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.517+0000 7f4c6f7fe640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 msgr2=0x7f4c70029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.517+0000 7f4c6f7fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 0x7f4c70029cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000e710 tx=0x7f4c60019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60015d30 con 0x7f4c700298c0 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 10 monclient: dump: 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f4c60015e90 con 0x7f4c700298c0 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 msgr2=0x7f4c70029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 0x7f4c70029cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000e710 tx=0x7f4c60019000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 msgr2=0x7f4c70029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c577fe640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c700298c0 0x7f4c70029cc0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000e710 tx=0x7f4c60019000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 4, 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:28:46.535 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 22, 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.536 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:28:46.538 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:28:46.539 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.540 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:28:46.541 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:28:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.543 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.544 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.546 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:28:46.547 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:28:46.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:28:46.548 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:28:46.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:28:46.548 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:28:46.548 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:28:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038740 msgr2=0x7f4c5c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038740 0x7f4c5c03abf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c680097c0 tx=0x7f4c68037000 comp rx=0 tx=0).stop 2024-09-15T03:28:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 10 monclient: shutdown 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 msgr2=0x7f4c7013eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 --2- 172.21.15.153:0/2013040495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009a8a0 0x7f4c7013eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400c4e0 tx=0x7f4c64007e10 comp rx=0 tx=0).stop 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c6ffff640 1 -- 172.21.15.153:0/2013040495 reap_dead start 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c6ffff640 1 -- 172.21.15.153:0/2013040495 reap_dead start 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 shutdown_connections 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 >> 172.21.15.153:0/2013040495 conn(0x7f4c700672b0 msgr2=0x7f4c7009fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.518+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 shutdown_connections 2024-09-15T03:28:46.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.519+0000 7f4c75ab8640 1 -- 172.21.15.153:0/2013040495 wait complete. 2024-09-15T03:28:46.550 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:28:46.551 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:28:46.551 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:28:46.551 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:28:46.551 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:28:46.551 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:28:46.551 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:28:46.562 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:28:46.562 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:28:46.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:28:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.692+0000 7efcbdb00640 10 monclient: get_monmap_and_config 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.692+0000 7efcbdb00640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.692+0000 7efcbdb00640 10 monclient: monmap: 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:46.693514+0000 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:46.693514+0000 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:46.695 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 1 Processor -- start 2024-09-15T03:28:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 1 -- start start 2024-09-15T03:28:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 10 monclient: init 2024-09-15T03:28:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:28:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efcb8131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 0x7efcb809ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.693+0000 7efcbdb00640 10 monclient(hunting): picked mon.noname-b con 0x7efcb809fe10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efcb809ef70 0x7efcb8138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 10 monclient(hunting): picked mon.noname-c con 0x7efcb809ef70 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80a14d0 con 0x7efcb8131980 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb81325b0 con 0x7efcb809ef70 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80a11f0 con 0x7efcb809fe10 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcbdb00640 10 monclient(hunting): authenticate will time out at 724.333054s 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 0x7efcb809ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 0x7efcb809ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45326/0 (socket says 172.21.15.153:45326) 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 -- 172.21.15.153:0/3156060665 learned_addr learned my addr 172.21.15.153:0/3156060665 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): get_auth_request con 0x7efcb809fe10 auth_method 0 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 1 --2- 172.21.15.153:0/3156060665 >> v2:172.21.15.181:3300/0 conn(0x7efcb809ef70 0x7efcb8138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): get_auth_request con 0x7efcb809ef70 auth_method 0 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 -- 172.21.15.153:0/3156060665 >> v2:172.21.15.181:3300/0 conn(0x7efcb809ef70 msgr2=0x7efcb8138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 --2- 172.21.15.153:0/3156060665 >> v2:172.21.15.181:3300/0 conn(0x7efcb809ef70 0x7efcb8138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 -- 172.21.15.153:0/3156060665 >> v1:172.21.15.181:6789/0 conn(0x7efcb8131980 legacy=0x7efcb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 monclient: found mon.noname-b 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.694+0000 7efcb6ffd640 1 -- 172.21.15.153:0/3156060665 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb809f390 con 0x7efcb809fe10 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb6ffd640 1 -- 172.21.15.153:0/3156060665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 msgr2=0x7efcb809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb6ffd640 1 --2- 172.21.15.153:0/3156060665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 0x7efcb809ea30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efcac009910 tx=0x7efcac0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d5b75a569b5d45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 1 -- 172.21.15.153:0/3156060665 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac02aea0 con 0x7efcb809fe10 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: dump: 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 5 monclient: authenticate success, global_id 4364 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 1 -- 172.21.15.153:0/3156060665 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac032440 con 0x7efcb809fe10 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 4 set_mon_vals no callback set 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 1 -- 172.21.15.153:0/3156060665 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0327e0 con 0x7efcb809fe10 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcb67fc640 10 monclient: dump: 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 10 monclient: shutdown 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 1 -- 172.21.15.153:0/3156060665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 msgr2=0x7efcb809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 1 --2- 172.21.15.153:0/3156060665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 0x7efcb809ea30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efcac009910 tx=0x7efcac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 1 -- 172.21.15.153:0/3156060665 shutdown_connections 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 1 --2- 172.21.15.153:0/3156060665 >> v2:172.21.15.181:3300/0 conn(0x7efcb809ef70 0x7efcb8138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 1 --2- 172.21.15.153:0/3156060665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb809fe10 0x7efcb809ea30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.697+0000 7efcbdb00640 1 -- 172.21.15.153:0/3156060665 >> 172.21.15.153:0/3156060665 conn(0x7efcb80a5050 msgr2=0x7efcb80a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 1 -- 172.21.15.153:0/3156060665 shutdown_connections 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 1 -- 172.21.15.153:0/3156060665 wait complete. 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 10 monclient: build_initial_monmap 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 10 monclient: monmap: 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:46.710 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 1 Processor -- start 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 1 -- start start 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.698+0000 7efcbdb00640 10 monclient: init 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcb809ef70 0x7efcb81347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): picked mon.noname-a con 0x7efcb809ef70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb809fe10 0x7efcb8134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): picked mon.noname-b con 0x7efcb809fe10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 0x7efcb8135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): picked mon.noname-c con 0x7efcb8131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80a4100 con 0x7efcb809ef70 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8136af0 con 0x7efcb809fe10 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8069eb0 con 0x7efcb8131980 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcbdb00640 10 monclient(hunting): authenticate will time out at 724.338399s 2024-09-15T03:28:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb77fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcb809ef70 msgr2=0x7efcb81347d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcb809ef70 0x7efcb81347d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 0x7efcb8135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 0x7efcb8135250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45342/0 (socket says 172.21.15.153:45342) 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 -- 172.21.15.153:0/1001145552 learned_addr learned my addr 172.21.15.153:0/1001145552 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): get_auth_request con 0x7efcb8131980 auth_method 0 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb809fe10 0x7efcb8134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): get_auth_request con 0x7efcb809fe10 auth_method 0 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 -- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb809fe10 msgr2=0x7efcb8134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb809fe10 0x7efcb8134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 -- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcb809ef70 msgr2=0x7efcb81347d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcb809ef70 0x7efcb81347d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 monclient: found mon.noname-c 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.699+0000 7efcb7fff640 1 -- 172.21.15.153:0/1001145552 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac009450 con 0x7efcb8131980 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.701+0000 7efcb7fff640 1 -- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 msgr2=0x7efcb8135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.701+0000 7efcb7fff640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 0x7efcb8135250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efcb000e900 tx=0x7efcb00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb000c940 con 0x7efcb8131980 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: dump: 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb000cac0 con 0x7efcb8131980 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 5 monclient: authenticate success, global_id 4367 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 10 monclient: _renew_subs 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb8136600 con 0x7efcb8131980 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb000ce80 con 0x7efcb8131980 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efc9f7fe640 10 monclient: dump: 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 10 monclient: _renew_subs 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcb5ffb640 4 set_mon_vals no callback set 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.702+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb8136cc0 con 0x7efcb8131980 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efcb0013680 con 0x7efcb8131980 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efcbdb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efcbdb00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efcbdb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb804bf60 con 0x7efcb8131980 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efc9f7fe640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efca40386f0 0x7efca403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7efcb004bad0 con 0x7efcb8131980 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efcb77fe640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efca40386f0 0x7efca403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:46.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.703+0000 7efcb77fe640 10 monclient: get_auth_request con 0x7efca40386f0 auth_method 0 2024-09-15T03:28:46.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.706+0000 7efcb77fe640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efca40386f0 0x7efca403aba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efca8007fd0 tx=0x7efca8007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:46.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.707+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efcb0023ad0 con 0x7efcb8131980 2024-09-15T03:28:46.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.707+0000 7efc9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:46.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.707+0000 7efc9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efcbdb00640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:28:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efcbdb00640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efcbdb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efcb809bb50 con 0x7efcb8131980 2024-09-15T03:28:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efc9f7fe640 1 -- 172.21.15.153:0/1001145552 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7efcb00232f0 con 0x7efcb8131980 2024-09-15T03:28:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efc9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.921+0000 7efc9f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:46.923 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:28:46.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efca40386f0 msgr2=0x7efca403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efca40386f0 0x7efca403aba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efca8007fd0 tx=0x7efca8007df0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 10 monclient: shutdown 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 msgr2=0x7efcb8135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 0x7efcb8135250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efcb000e900 tx=0x7efcb00090f0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 shutdown_connections 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efca40386f0 0x7efca403aba0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcb8131980 0x7efcb8135250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb809fe10 0x7efcb8134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 --2- 172.21.15.153:0/1001145552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcb809ef70 0x7efcb81347d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 >> 172.21.15.153:0/1001145552 conn(0x7efcb80a5050 msgr2=0x7efcb809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 shutdown_connections 2024-09-15T03:28:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:46.923+0000 7efcbdb00640 1 -- 172.21.15.153:0/1001145552 wait complete. 2024-09-15T03:28:49.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:28:50.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.039+0000 7f6c1a0a8640 10 monclient: get_monmap_and_config 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.039+0000 7f6c1a0a8640 10 monclient: build_initial_monmap 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.039+0000 7f6c1a0a8640 10 monclient: monmap: 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:50.040541+0000 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:50.040541+0000 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:50.041 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:50.042 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:50.042 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:50.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.040+0000 7f6c1a0a8640 1 Processor -- start 2024-09-15T03:28:50.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.040+0000 7f6c1a0a8640 1 -- start start 2024-09-15T03:28:50.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.040+0000 7f6c1a0a8640 10 monclient: init 2024-09-15T03:28:50.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 0x7f6c1409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6c14096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6c14097030 0x7f6c14097400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6c14097030 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c1412e5f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1409d2e0 con 0x7f6c1412e5f0 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1409ac00 con 0x7f6c14097030 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1409d6b0 con 0x7f6c14096c60 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:50.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c1a0a8640 10 monclient(hunting): authenticate will time out at 727.680183s 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c12ffd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f6c14097030 msgr2=0x7f6c14097400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c12ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6c14097030 0x7f6c14097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 0x7f6c1409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 0x7f6c1409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45344/0 (socket says 172.21.15.153:45344) 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 -- 172.21.15.153:0/2677437409 learned_addr learned my addr 172.21.15.153:0/2677437409 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): get_auth_request con 0x7f6c14096c60 auth_method 0 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:50.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 -- 172.21.15.153:0/2677437409 >> v2:172.21.15.153:3300/0 conn(0x7f6c14097030 msgr2=0x7f6c14097400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 --2- 172.21.15.153:0/2677437409 >> v2:172.21.15.153:3300/0 conn(0x7f6c14097030 0x7f6c14097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 -- 172.21.15.153:0/2677437409 >> v1:172.21.15.181:6789/0 conn(0x7f6c1412e5f0 legacy=0x7f6c1412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient: _finish_hunting 0 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 monclient: found mon.noname-b 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:50.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.041+0000 7f6c137fe640 1 -- 172.21.15.153:0/2677437409 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140992f0 con 0x7f6c14096c60 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c137fe640 1 -- 172.21.15.153:0/2677437409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 msgr2=0x7f6c1409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c137fe640 1 --2- 172.21.15.153:0/2677437409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 0x7f6c1409a6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400bbf0 tx=0x7f6c0400b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96abfd469054aff9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 1 -- 172.21.15.153:0/2677437409 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04032840 con 0x7f6c14096c60 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: dump: 2024-09-15T03:28:50.053 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: _finish_auth 0 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 1 -- 172.21.15.153:0/2677437409 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c040329a0 con 0x7f6c14096c60 2024-09-15T03:28:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c1a0a8640 5 monclient: authenticate success, global_id 4370 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c1a0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c1a0a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 1 -- 172.21.15.153:0/2677437409 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04032d40 con 0x7f6c14096c60 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c1a0a8640 4 set_mon_vals no callback set 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.044+0000 7f6c127fc640 10 monclient: dump: 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:50.055 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 10 monclient: shutdown 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/2677437409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 msgr2=0x7f6c1409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/2677437409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 0x7f6c1409a6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400bbf0 tx=0x7f6c0400b460 comp rx=0 tx=0).stop 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/2677437409 shutdown_connections 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/2677437409 >> v2:172.21.15.153:3300/0 conn(0x7f6c14097030 0x7f6c14097400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/2677437409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14096c60 0x7f6c1409a6c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/2677437409 >> 172.21.15.153:0/2677437409 conn(0x7f6c140672b0 msgr2=0x7f6c140676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:50.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/2677437409 shutdown_connections 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/2677437409 wait complete. 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 10 monclient: build_initial_monmap 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.045+0000 7f6c1a0a8640 10 monclient: monmap: 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:50.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 Processor -- start 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 -- start start 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient: init 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14096c60 0x7f6c14141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6c14096c60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c14097030 0x7f6c1412da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6c14097030 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 0x7f6c1412dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6c1412e5f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1409add0 con 0x7f6c14096c60 2024-09-15T03:28:50.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1411f210 con 0x7f6c14097030 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c137fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14096c60 msgr2=0x7f6c14141220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14141760 con 0x7f6c1412e5f0 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c1a0a8640 10 monclient(hunting): authenticate will time out at 727.685793s 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c137fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14096c60 0x7f6c14141220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 0x7f6c1412dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 0x7f6c1412dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45352/0 (socket says 172.21.15.153:45352) 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 1 -- 172.21.15.153:0/3903403705 learned_addr learned my addr 172.21.15.153:0/3903403705 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): get_auth_request con 0x7f6c1412e5f0 auth_method 0 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:50.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.046+0000 7f6c13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c12ffd640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c14097030 0x7f6c1412da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c12ffd640 10 monclient(hunting): get_auth_request con 0x7f6c14097030 auth_method 0 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 1 -- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c14097030 msgr2=0x7f6c1412da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c14097030 0x7f6c1412da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 1 -- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14096c60 msgr2=0x7f6c14141220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:28:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14096c60 0x7f6c14141220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 10 monclient: _finish_hunting 0 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 1 monclient: found mon.noname-c 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.047+0000 7f6c13fff640 1 -- 172.21.15.153:0/3903403705 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0400b700 con 0x7f6c1412e5f0 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c13fff640 1 -- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 msgr2=0x7f6c1412dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c13fff640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 0x7f6c1412dfa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00c8c0 tx=0x7f6c0c002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00aa00 con 0x7f6c1412e5f0 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: dump: 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:50.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c00ab80 con 0x7f6c1412e5f0 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 5 monclient: authenticate success, global_id 4373 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 10 monclient: _renew_subs 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:50.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c14141a00 con 0x7f6c1412e5f0 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c012420 con 0x7f6c1412e5f0 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6bfb7fe640 10 monclient: dump: 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:50.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 10 monclient: _renew_subs 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c11ffb640 4 set_mon_vals no callback set 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.049+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1411eea0 con 0x7f6c1412e5f0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.050+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6c0c012890 con 0x7f6c1412e5f0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.050+0000 7f6c1a0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.050+0000 7f6c1a0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.050+0000 7f6c1a0a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.050+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1411dcc0 con 0x7f6c1412e5f0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.050+0000 7f6bfb7fe640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c00038740 0x7f6c0003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.051+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f6c0c04bc80 con 0x7f6c1412e5f0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.051+0000 7f6c137fe640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c00038740 0x7f6c0003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:50.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.051+0000 7f6c137fe640 10 monclient: get_auth_request con 0x7f6c00038740 auth_method 0 2024-09-15T03:28:50.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.053+0000 7f6c137fe640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c00038740 0x7f6c0003abf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400bbc0 tx=0x7f6c0403a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:50.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.055+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6c0c0102e0 con 0x7f6c1412e5f0 2024-09-15T03:28:50.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.055+0000 7f6bfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:50.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.055+0000 7f6bfb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:50.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6c1a0a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:28:50.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6c1a0a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:50.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6c1a0a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:50.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6c1404bf60 con 0x7f6c1412e5f0 2024-09-15T03:28:50.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6bfb7fe640 1 -- 172.21.15.153:0/3903403705 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f6c0c01c030 con 0x7f6c1412e5f0 2024-09-15T03:28:50.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6bfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:50.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.269+0000 7f6bfb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:50.271 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:28:50.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c00038740 msgr2=0x7f6c0003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:50.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c00038740 0x7f6c0003abf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400bbc0 tx=0x7f6c0403a000 comp rx=0 tx=0).stop 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 10 monclient: shutdown 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 msgr2=0x7f6c1412dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 0x7f6c1412dfa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00c8c0 tx=0x7f6c0c002640 comp rx=0 tx=0).stop 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 shutdown_connections 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c00038740 0x7f6c0003abf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c1412e5f0 0x7f6c1412dfa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c14097030 0x7f6c1412da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 --2- 172.21.15.153:0/3903403705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14096c60 0x7f6c14141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:50.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 >> 172.21.15.153:0/3903403705 conn(0x7f6c140672b0 msgr2=0x7f6c1412fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:50.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 shutdown_connections 2024-09-15T03:28:50.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:50.271+0000 7f6c1a0a8640 1 -- 172.21.15.153:0/3903403705 wait complete. 2024-09-15T03:28:53.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:28:53.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.388+0000 7f9b33860640 10 monclient: get_monmap_and_config 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.388+0000 7f9b33860640 10 monclient: build_initial_monmap 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.388+0000 7f9b33860640 10 monclient: monmap: 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:53.389202+0000 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:53.389202+0000 2024-09-15T03:28:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 Processor -- start 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 -- start start 2024-09-15T03:28:53.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient: init 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 0x7f9b2c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): picked mon.noname-b con 0x7f9b2c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9b2c09ea50 0x7f9b2c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): picked mon.noname-a con 0x7f9b2c09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b2c068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c131cc0 con 0x7f9b2c068d90 2024-09-15T03:28:53.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2c09fbf0 con 0x7f9b2c09ea50 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c09fe30 con 0x7f9b2c1318f0 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b33860640 10 monclient(hunting): authenticate will time out at 731.028836s 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b30dfb640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9b2c09ea50 msgr2=0x7f9b2c09ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b30dfb640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9b2c09ea50 0x7f9b2c09ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b315fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 0x7f9b2c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.389+0000 7f9b315fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 0x7f9b2c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45356/0 (socket says 172.21.15.153:45356) 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 1 -- 172.21.15.153:0/2657607348 learned_addr learned my addr 172.21.15.153:0/2657607348 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): get_auth_request con 0x7f9b2c1318f0 auth_method 0 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 1 -- 172.21.15.153:0/2657607348 >> v2:172.21.15.153:3300/0 conn(0x7f9b2c09ea50 msgr2=0x7f9b2c09ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 1 --2- 172.21.15.153:0/2657607348 >> v2:172.21.15.153:3300/0 conn(0x7f9b2c09ea50 0x7f9b2c09ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 1 -- 172.21.15.153:0/2657607348 >> v1:172.21.15.181:6789/0 conn(0x7f9b2c068d90 legacy=0x7f9b2c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient: _finish_hunting 0 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 1 monclient: found mon.noname-b 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.390+0000 7f9b315fc640 1 -- 172.21.15.153:0/2657607348 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0a1210 con 0x7f9b2c1318f0 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b315fc640 1 -- 172.21.15.153:0/2657607348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 msgr2=0x7f9b2c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b315fc640 1 --2- 172.21.15.153:0/2657607348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 0x7f9b2c131f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009910 tx=0x7f9b280091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a893f7bff28f979c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 1 -- 172.21.15.153:0/2657607348 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28032440 con 0x7f9b2c1318f0 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: dump: 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: _finish_auth 0 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 1 -- 172.21.15.153:0/2657607348 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b280325a0 con 0x7f9b2c1318f0 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 5 monclient: authenticate success, global_id 4376 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 10 monclient: get_monmap_and_config success 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 1 -- 172.21.15.153:0/2657607348 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28032940 con 0x7f9b2c1318f0 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 4 set_mon_vals no callback set 2024-09-15T03:28:53.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b23fff640 10 monclient: dump: 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 10 monclient: shutdown 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 -- 172.21.15.153:0/2657607348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 msgr2=0x7f9b2c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 --2- 172.21.15.153:0/2657607348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 0x7f9b2c131f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009910 tx=0x7f9b280091d0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 -- 172.21.15.153:0/2657607348 shutdown_connections 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 --2- 172.21.15.153:0/2657607348 >> v2:172.21.15.153:3300/0 conn(0x7f9b2c09ea50 0x7f9b2c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 --2- 172.21.15.153:0/2657607348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c1318f0 0x7f9b2c131f30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 -- 172.21.15.153:0/2657607348 >> 172.21.15.153:0/2657607348 conn(0x7f9b2c067110 msgr2=0x7f9b2c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 -- 172.21.15.153:0/2657607348 shutdown_connections 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.393+0000 7f9b33860640 1 -- 172.21.15.153:0/2657607348 wait complete. 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.394+0000 7f9b33860640 10 monclient: build_initial_monmap 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.394+0000 7f9b33860640 10 monclient: monmap: 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.394+0000 7f9b33860640 1 Processor -- start 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.394+0000 7f9b33860640 1 -- start start 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.394+0000 7f9b33860640 10 monclient: init 2024-09-15T03:28:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b2c068d90 0x7f9b2c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): picked mon.noname-b con 0x7f9b2c068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 0x7f9b2c130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): picked mon.noname-c con 0x7f9b2c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c1318f0 0x7f9b2c140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): picked mon.noname-a con 0x7f9b2c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c09b5a0 con 0x7f9b2c1318f0 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c11dd10 con 0x7f9b2c068d90 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:53.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c11dff0 con 0x7f9b2c09ea50 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b33860640 10 monclient(hunting): authenticate will time out at 731.034502s 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b31dfd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c1318f0 msgr2=0x7f9b2c140f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b31dfd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c1318f0 0x7f9b2c140f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 0x7f9b2c130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 0x7f9b2c130bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45372/0 (socket says 172.21.15.153:45372) 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 -- 172.21.15.153:0/2023395926 learned_addr learned my addr 172.21.15.153:0/2023395926 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): get_auth_request con 0x7f9b2c09ea50 auth_method 0 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b2c068d90 0x7f9b2c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): get_auth_request con 0x7f9b2c068d90 auth_method 0 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b315fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 -- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b2c068d90 msgr2=0x7f9b2c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:53.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b2c068d90 0x7f9b2c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 -- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c1318f0 msgr2=0x7f9b2c140f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c1318f0 0x7f9b2c140f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient: _finish_hunting 0 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 monclient: found mon.noname-c 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.395+0000 7f9b30dfb640 1 -- 172.21.15.153:0/2023395926 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b28009450 con 0x7f9b2c09ea50 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b30dfb640 1 -- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 msgr2=0x7f9b2c130bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b30dfb640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 0x7f9b2c130bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00b600 tx=0x7f9b1c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c015070 con 0x7f9b2c09ea50 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:53.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: dump: 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: _finish_auth 0 2024-09-15T03:28:53.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c003d90 con 0x7f9b2c09ea50 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 5 monclient: authenticate success, global_id 4379 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c006380 con 0x7f9b2c09ea50 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 10 monclient: _renew_subs 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c11e2d0 con 0x7f9b2c09ea50 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b237fe640 4 set_mon_vals no callback set 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b217fa640 10 monclient: dump: 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 10 monclient: _renew_subs 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.398+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c11c560 con 0x7f9b2c09ea50 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9b1c010440 con 0x7f9b2c09ea50 2024-09-15T03:28:53.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b33860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b33860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b33860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c04bf60 con 0x7f9b2c09ea50 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b217fa640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b100386f0 0x7f9b1003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f9b1c04b3a0 con 0x7f9b2c09ea50 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b315fc640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b100386f0 0x7f9b1003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:53.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.399+0000 7f9b315fc640 10 monclient: get_auth_request con 0x7f9b100386f0 auth_method 0 2024-09-15T03:28:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.402+0000 7f9b315fc640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b100386f0 0x7f9b1003aba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c064050 tx=0x7f9b28032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:28:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.403+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9b1c018ab0 con 0x7f9b2c09ea50 2024-09-15T03:28:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.403+0000 7f9b217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:28:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.403+0000 7f9b217fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:28:53.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b33860640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:28:53.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b33860640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:53.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b33860640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:28:53.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9b2c0638d0 con 0x7f9b2c09ea50 2024-09-15T03:28:53.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b217fa640 1 -- 172.21.15.153:0/2023395926 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9b1c018690 con 0x7f9b2c09ea50 2024-09-15T03:28:53.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:28:53.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.619+0000 7f9b217fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:28:53.621 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b100386f0 msgr2=0x7f9b1003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b100386f0 0x7f9b1003aba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c064050 tx=0x7f9b28032000 comp rx=0 tx=0).stop 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 10 monclient: shutdown 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 msgr2=0x7f9b2c130bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 0x7f9b2c130bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00b600 tx=0x7f9b1c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 shutdown_connections 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b100386f0 0x7f9b1003aba0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c1318f0 0x7f9b2c140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b2c09ea50 0x7f9b2c130bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 --2- 172.21.15.153:0/2023395926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b2c068d90 0x7f9b2c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:53.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 >> 172.21.15.153:0/2023395926 conn(0x7f9b2c067110 msgr2=0x7f9b2c0a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:28:53.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 shutdown_connections 2024-09-15T03:28:53.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:53.621+0000 7f9b33860640 1 -- 172.21.15.153:0/2023395926 wait complete. 2024-09-15T03:28:56.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:28:56.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.737+0000 7f4ebb91d640 10 monclient: get_monmap_and_config 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.737+0000 7f4ebb91d640 10 monclient: build_initial_monmap 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.737+0000 7f4ebb91d640 10 monclient: monmap: 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:28:56.738661+0000 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:28:56.738661+0000 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:28:56.739 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:28:56.740 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:28:56.740 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.738+0000 7f4ebb91d640 1 Processor -- start 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.738+0000 7f4ebb91d640 1 -- start start 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.738+0000 7f4ebb91d640 10 monclient: init 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): picked mon.noname-c con 0x7f4eb4131980 addr v2:172.21.15.181:3300/0 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4eb409fe10 addr v1:172.21.15.153:6789/0 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 0x7f4eb4138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): picked mon.noname-a con 0x7f4eb409f140 addr v2:172.21.15.153:3300/0 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb40a14d0 con 0x7f4eb409fe10 2024-09-15T03:28:56.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb41325b0 con 0x7f4eb409f140 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb40a11f0 con 0x7f4eb4131980 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4ebb91d640 10 monclient(hunting): authenticate will time out at 734.378297s 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb8eb8640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f4eb409fe10 legacy=0x7f4eb409ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb9eba640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 msgr2=0x7f4eb4138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb9eba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 0x7f4eb4138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:36558/0 (socket says 172.21.15.153:36558) 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 learned_addr learned my addr 172.21.15.153:0/2157627246 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4eb4131980 auth_method 0 2024-09-15T03:28:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:56.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:56.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:56.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:28:56.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:56.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:56.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.739+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb8eb8640 1 -- 172.21.15.153:0/2157627246 >> v1:172.21.15.153:6789/0 conn(0x7f4eb409fe10 legacy=0x7f4eb409ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 msgr2=0x7f4eb4138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 0x7f4eb4138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4eb4131980 auth_method 0 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:56.940+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 msgr2=0x7f4eb4138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb8eb8640 1 -- 172.21.15.153:0/2157627246 >> v1:172.21.15.153:6789/0 conn(0x7f4eb409fe10 legacy=0x7f4eb409ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 0x7f4eb4138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4eb4131980 auth_method 0 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:57.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:57.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:28:57.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:57.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:57.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:57.341+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:28:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 msgr2=0x7f4eb4138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb8eb8640 1 -- 172.21.15.153:0/2157627246 >> v1:172.21.15.153:6789/0 conn(0x7f4eb409fe10 legacy=0x7f4eb409ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 0x7f4eb4138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4eb4131980 auth_method 0 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:58.142+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient(hunting): tick 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 msgr2=0x7f4eb4131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.181:3300/0 conn(0x7f4eb4131980 0x7f4eb4131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 -- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 msgr2=0x7f4eb4138cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 --2- 172.21.15.153:0/2157627246 >> v2:172.21.15.153:3300/0 conn(0x7f4eb409f140 0x7f4eb4138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 -- 172.21.15.153:0/2157627246 >> v1:172.21.15.153:6789/0 conn(0x7f4eb409fe10 legacy=0x7f4eb409ea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:28:59.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient: _add_conns ranks=[2,1] 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f4e98003630 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e98006150 addr v1:172.21.15.181:6789/0 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 -- 172.21.15.153:0/2157627246 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb4132290 con 0x7f4e98006150 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 1 -- 172.21.15.153:0/2157627246 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41325b0 con 0x7f4e98003630 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4ea6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4e98003630 auth_method 0 2024-09-15T03:28:59.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.739+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:28:59.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.741+0000 7f4eb8eb8640 1 -- 172.21.15.153:0/2157627246 >> v1:172.21.15.181:6789/0 conn(0x7f4e98006150 legacy=0x7f4e98008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:28:59.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:28:59.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4e98003630 auth_method 0 2024-09-15T03:28:59.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:28:59.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:28:59.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:28:59.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:28:59.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:28:59.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:28:59.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:28:59.940+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4e98003630 auth_method 0 2024-09-15T03:29:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:00.341+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4e98003630 auth_method 0 2024-09-15T03:29:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.142+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:01.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.286+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454562180 0 0) 0x7f4eb4132290 con 0x7f4e98006150 2024-09-15T03:29:01.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.286+0000 7f4ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.286+0000 7f4ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.286+0000 7f4ea7fff640 10 monclient(hunting): my global_id is 4357 2024-09-15T03:29:01.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.286+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea00034e0 con 0x7f4e98006150 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486539266 0 0) 0x7f4ea00034e0 con 0x7f4e98006150 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 msgr2=0x7f4e98005a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e98003630 0x7f4e98005a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb40a14d0 con 0x7f4e98006150 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f4eb0004130 con 0x7f4e98006150 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ebb91d640 5 monclient: authenticate success, global_id 4357 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ebb91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: dump: 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.292 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 >> v1:172.21.15.181:6789/0 conn(0x7f4e98006150 legacy=0x7f4e98008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 reap_dead start 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea00039d0 0x7f4ea0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): picked mon.a con 0x7f4ea00039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea0004780 0x7f4ea0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): picked mon.c con 0x7f4ea0004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4ea000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): picked mon.b con 0x7f4ea0005180 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41325b0 con 0x7f4ea00039d0 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea000c120 con 0x7f4ea0005180 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea00039d0 msgr2=0x7f4ea0003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea000c2e0 con 0x7f4ea0004780 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ea7fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea00039d0 0x7f4ea0003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4ebb91d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.287+0000 7f4eb8eb8640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea0004780 0x7f4ea0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): get_auth_request con 0x7f4ea0004780 auth_method 0 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4ea000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): get_auth_request con 0x7f4ea0005180 auth_method 0 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:01.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_done global_id 4357 payload 293 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea0004780 msgr2=0x7f4ea0004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea0004780 0x7f4ea0004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea00039d0 msgr2=0x7f4ea0003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea00039d0 0x7f4ea0003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 monclient: found mon.b 2024-09-15T03:29:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:29:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea00034e0 con 0x7f4ea0005180 2024-09-15T03:29:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb9eba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.288+0000 7f4eb8eb8640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea0004780 0x7f4ea0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:01.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4eb9eba640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 msgr2=0x7f4ea000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4ea000b9d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4071640 tx=0x7f4eac0074f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2aa0c4c197289375 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac00b040 con 0x7f4ea0005180 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: dump: 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac02a920 con 0x7f4ea0005180 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 1 -- 172.21.15.153:0/2157627246 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac02ace0 con 0x7f4ea0005180 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ebb91d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ebb91d640 4 set_mon_vals no callback set 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.291+0000 7f4ea7fff640 10 monclient: dump: 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 10 monclient: shutdown 2024-09-15T03:29:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 msgr2=0x7f4ea000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4ea000b9d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4071640 tx=0x7f4eac0074f0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2157627246 shutdown_connections 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4ea000b9d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea0004780 0x7f4ea0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2157627246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea00039d0 0x7f4ea0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2157627246 >> 172.21.15.153:0/2157627246 conn(0x7f4eb40a5050 msgr2=0x7f4eb40a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2157627246 shutdown_connections 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.292+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2157627246 wait complete. 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.293+0000 7f4ebb91d640 10 monclient: build_initial_monmap 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.293+0000 7f4ebb91d640 10 monclient: monmap: 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.293+0000 7f4ebb91d640 1 Processor -- start 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.293+0000 7f4ebb91d640 1 -- start start 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.293+0000 7f4ebb91d640 10 monclient: init 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:29:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 0x7f4eb4130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea00039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea0004780 0x7f4eb411d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea0004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4eb411d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea0005180 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb40a4a80 con 0x7f4ea0004780 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb4120d30 con 0x7f4ea0005180 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb411ef90 con 0x7f4ea00039d0 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4ebb91d640 10 monclient(hunting): authenticate will time out at 738.933468s 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb8eb8640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea0004780 msgr2=0x7f4eb411d440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb8eb8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea0004780 0x7f4eb411d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 0x7f4eb4130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 0x7f4eb4130900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34652/0 (socket says 172.21.15.153:34652) 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2365184722 learned_addr learned my addr 172.21.15.153:0/2365184722 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request con 0x7f4ea00039d0 auth_method 0 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4eb411d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): get_auth_request con 0x7f4ea0005180 auth_method 0 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb9eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 msgr2=0x7f4eb411d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4eb411d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea0004780 msgr2=0x7f4eb411d440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea0004780 0x7f4eb411d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 monclient: found mon.noname-c 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.294+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2365184722 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb4141920 con 0x7f4ea00039d0 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4eb96b9640 1 -- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 msgr2=0x7f4eb4130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4eb96b9640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 0x7f4eb4130900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8016b00 tx=0x7f4ea8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea801e070 con 0x7f4ea00039d0 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: dump: 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 5 monclient: authenticate success, global_id 4382 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea800c040 con 0x7f4ea00039d0 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 10 monclient: _renew_subs 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb4120a50 con 0x7f4ea00039d0 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8010ce0 con 0x7f4ea00039d0 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea57fa640 10 monclient: dump: 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ea77fe640 4 set_mon_vals no callback set 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 10 monclient: _renew_subs 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.297+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb4132ed0 con 0x7f4ea00039d0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4ea8022070 con 0x7f4ea00039d0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ea57fa640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e98038e00 0x7f4e9803b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4ea804c950 con 0x7f4ea00039d0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ebb91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ebb91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ebb91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb4120ef0 con 0x7f4ea00039d0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.298+0000 7f4eb8eb8640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e98038e00 0x7f4e9803b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.299+0000 7f4eb8eb8640 10 monclient: get_auth_request con 0x7f4e98038e00 auth_method 0 2024-09-15T03:29:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.301+0000 7f4eb8eb8640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e98038e00 0x7f4e9803b2b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4eb00042e0 tx=0x7f4eb0007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.303+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4ea8014350 con 0x7f4ea00039d0 2024-09-15T03:29:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.303+0000 7f4ea57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.303+0000 7f4ea57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ebb91d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ebb91d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ebb91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4eb404bf60 con 0x7f4ea00039d0 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ea57fa640 1 -- 172.21.15.153:0/2365184722 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1532 (secure 0 0 0) 0x7f4ea8029070 con 0x7f4ea00039d0 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ea57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.523+0000 7f4ea57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e98038e00 msgr2=0x7f4e9803b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e98038e00 0x7f4e9803b2b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4eb00042e0 tx=0x7f4eb0007a80 comp rx=0 tx=0).stop 2024-09-15T03:29:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 10 monclient: shutdown 2024-09-15T03:29:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 msgr2=0x7f4eb4130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 0x7f4eb4130900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8016b00 tx=0x7f4ea8008040 comp rx=0 tx=0).stop 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 shutdown_connections 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e98038e00 0x7f4e9803b2b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ea0005180 0x7f4eb411d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ea0004780 0x7f4eb411d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 --2- 172.21.15.153:0/2365184722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ea00039d0 0x7f4eb4130900 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 >> 172.21.15.153:0/2365184722 conn(0x7f4eb40a5050 msgr2=0x7f4eb409d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 shutdown_connections 2024-09-15T03:29:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.526+0000 7f4ebb91d640 1 -- 172.21.15.153:0/2365184722 wait complete. 2024-09-15T03:29:01.538 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:29:01.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:29:01.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:29:01.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.630+0000 7f9c576a8640 10 monclient: get_monmap_and_config 2024-09-15T03:29:01.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.630+0000 7f9c576a8640 10 monclient: build_initial_monmap 2024-09-15T03:29:01.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.630+0000 7f9c576a8640 10 monclient: monmap: 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:01.631369+0000 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:01.631369+0000 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:01.632 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 Processor -- start 2024-09-15T03:29:01.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 -- start start 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient: init 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9c5009c9c0 0x7f9c5009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5009c9c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9c5009e340 0x7f9c5009e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5009e340 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c5012f770 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:01.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5009bdd0 con 0x7f9c5012f770 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5009ec90 con 0x7f9c5009c9c0 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5009a880 con 0x7f9c5009e340 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c576a8640 10 monclient(hunting): authenticate will time out at 739.270892s 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c55c45640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f9c5012f770 legacy=0x7f9c5012fbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c55444640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9c5009c9c0 msgr2=0x7f9c5009d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.631+0000 7f9c55444640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9c5009c9c0 0x7f9c5009d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9c5009e340 0x7f9c5009e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9c5009e340 0x7f9c5009e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:36616/0 (socket says 172.21.15.153:36616) 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 -- 172.21.15.153:0/1336450175 learned_addr learned my addr 172.21.15.153:0/1336450175 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:01.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): get_auth_request con 0x7f9c5009e340 auth_method 0 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 -- 172.21.15.153:0/1336450175 >> v2:172.21.15.153:3300/0 conn(0x7f9c5009c9c0 msgr2=0x7f9c5009d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 --2- 172.21.15.153:0/1336450175 >> v2:172.21.15.153:3300/0 conn(0x7f9c5009c9c0 0x7f9c5009d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 -- 172.21.15.153:0/1336450175 >> v1:172.21.15.153:6789/0 conn(0x7f9c5012f770 legacy=0x7f9c5012fbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 monclient: found mon.noname-c 2024-09-15T03:29:01.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.632+0000 7f9c54c43640 1 -- 172.21.15.153:0/1336450175 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5009a440 con 0x7f9c5009e340 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c54c43640 1 -- 172.21.15.153:0/1336450175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 msgr2=0x7f9c5009e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c54c43640 1 --2- 172.21.15.153:0/1336450175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 0x7f9c5009e750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9c5009b520 tx=0x7f9c400091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c93bea68544858f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 1 -- 172.21.15.153:0/1336450175 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40032440 con 0x7f9c5009e340 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: dump: 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 1 -- 172.21.15.153:0/1336450175 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c400325a0 con 0x7f9c5009e340 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 1 -- 172.21.15.153:0/1336450175 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40032940 con 0x7f9c5009e340 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c47fff640 10 monclient: dump: 2024-09-15T03:29:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 5 monclient: authenticate success, global_id 4363 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 4 set_mon_vals no callback set 2024-09-15T03:29:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 10 monclient: shutdown 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 1 -- 172.21.15.153:0/1336450175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 msgr2=0x7f9c5009e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 1 --2- 172.21.15.153:0/1336450175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 0x7f9c5009e750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9c5009b520 tx=0x7f9c400091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 1 -- 172.21.15.153:0/1336450175 shutdown_connections 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 1 --2- 172.21.15.153:0/1336450175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 0x7f9c5009e750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 1 --2- 172.21.15.153:0/1336450175 >> v2:172.21.15.153:3300/0 conn(0x7f9c5009c9c0 0x7f9c5009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.635+0000 7f9c576a8640 1 -- 172.21.15.153:0/1336450175 >> 172.21.15.153:0/1336450175 conn(0x7f9c500a2e40 msgr2=0x7f9c500a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 1 -- 172.21.15.153:0/1336450175 shutdown_connections 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 1 -- 172.21.15.153:0/1336450175 wait complete. 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 10 monclient: build_initial_monmap 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 10 monclient: monmap: 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 1 Processor -- start 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 1 -- start start 2024-09-15T03:29:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.636+0000 7f9c576a8640 10 monclient: init 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 0x7f9c5013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5009c9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 0x7f9c5013fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5009e340 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c5012f770 0x7f9c5011bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5012f770 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5011fa70 con 0x7f9c5012f770 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50098fa0 con 0x7f9c5009e340 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5011f790 con 0x7f9c5009c9c0 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c576a8640 10 monclient(hunting): authenticate will time out at 739.276423s 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55c45640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c5012f770 msgr2=0x7f9c5011bf30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55c45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c5012f770 0x7f9c5011bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 0x7f9c5013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 0x7f9c5013f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34666/0 (socket says 172.21.15.153:34666) 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 -- 172.21.15.153:0/461749315 learned_addr learned my addr 172.21.15.153:0/461749315 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): get_auth_request con 0x7f9c5009c9c0 auth_method 0 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 0x7f9c5013fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): get_auth_request con 0x7f9c5009e340 auth_method 0 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 msgr2=0x7f9c5013fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c5009e340 0x7f9c5013fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c5012f770 msgr2=0x7f9c5011bf30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c5012f770 0x7f9c5011bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 monclient: found mon.noname-c 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.637+0000 7f9c55444640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c40009450 con 0x7f9c5009c9c0 2024-09-15T03:29:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c55444640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 msgr2=0x7f9c5013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c55444640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 0x7f9c5013f6e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00b5c0 tx=0x7f9c4c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c003cc0 con 0x7f9c5009c9c0 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: dump: 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c003e40 con 0x7f9c5009c9c0 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 5 monclient: authenticate success, global_id 4385 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c005340 con 0x7f9c5009c9c0 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 10 monclient: _renew_subs 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5011f4b0 con 0x7f9c5009c9c0 2024-09-15T03:29:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c457fa640 10 monclient: dump: 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c477fe640 4 set_mon_vals no callback set 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 10 monclient: _renew_subs 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.640+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5011e550 con 0x7f9c5009c9c0 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c576a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c576a8640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 0x7f9c5011d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50065e00 con 0x7f9c5011cc30 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 0x7f9c5011d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9c4c01a460 con 0x7f9c5009c9c0 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient: get_auth_request con 0x7f9c5011cc30 auth_method 0 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c54c43640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c5004bf60 con 0x7f9c5011cc30 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c457fa640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9c340386f0 0x7f9c3403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f9c4c04b500 con 0x7f9c5009c9c0 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c55c45640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9c340386f0 0x7f9c3403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.641+0000 7f9c55c45640 10 monclient: get_auth_request con 0x7f9c340386f0 auth_method 0 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.643+0000 7f9c54c43640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 msgr2=0x7f9c5011d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.643+0000 7f9c54c43640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 0x7f9c5011d000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c40034000 tx=0x7f9c40032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.643+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4003c930 con 0x7f9c5011cc30 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.643+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.643+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.643+0000 7f9c457fa640 10 monclient: dump: 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9c4003ca90 con 0x7f9c5011cc30 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 msgr2=0x7f9c5011d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 0x7f9c5011d000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c40034000 tx=0x7f9c40032000 comp rx=0 tx=0).stop 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 msgr2=0x7f9c5011d000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c457fa640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5011cc30 0x7f9c5011d000 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c40034000 tx=0x7f9c40032000 comp rx=0 tx=0).stop 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.644+0000 7f9c55c45640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9c340386f0 0x7f9c3403aba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9c48003cc0 tx=0x7f9c48009370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c576a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c576a8640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 0x7f9c50029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c576a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50099ef0 con 0x7f9c500298c0 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 0x7f9c50029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient: get_auth_request con 0x7f9c500298c0 auth_method 0 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.649+0000 7f9c54c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.650+0000 7f9c54c43640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-09-15T03:29:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.650+0000 7f9c54c43640 1 -- 172.21.15.153:0/461749315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9c5006ef90 con 0x7f9c500298c0 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c54c43640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 msgr2=0x7f9c50029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c54c43640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 0x7f9c50029c90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c5011d980 tx=0x7f9c4003e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c400458a0 con 0x7f9c500298c0 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 10 monclient: dump: 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4565 (secure 0 0 0) 0x7f9c40045a00 con 0x7f9c500298c0 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 msgr2=0x7f9c50029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 0x7f9c50029c90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c5011d980 tx=0x7f9c4003e040 comp rx=0 tx=0).stop 2024-09-15T03:29:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 msgr2=0x7f9c50029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.652+0000 7f9c457fa640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c500298c0 0x7f9c50029c90 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c5011d980 tx=0x7f9c4003e040 comp rx=0 tx=0).stop 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 6, 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:01.669 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:01.670 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:01.671 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:01.679 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:01.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9c340386f0 msgr2=0x7f9c3403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9c340386f0 0x7f9c3403aba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9c48003cc0 tx=0x7f9c48009370 comp rx=0 tx=0).stop 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 10 monclient: shutdown 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 msgr2=0x7f9c5013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 0x7f9c5013f6e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00b5c0 tx=0x7f9c4c009120 comp rx=0 tx=0).stop 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c55c45640 1 -- 172.21.15.153:0/461749315 reap_dead start 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 shutdown_connections 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 --2- 172.21.15.153:0/461749315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c5009c9c0 0x7f9c5013f6e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 >> 172.21.15.153:0/461749315 conn(0x7f9c500a2e40 msgr2=0x7f9c500978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 shutdown_connections 2024-09-15T03:29:01.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.653+0000 7f9c576a8640 1 -- 172.21.15.153:0/461749315 wait complete. 2024-09-15T03:29:01.682 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:29:01.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:29:01.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:29:01.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.775+0000 7f203091d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:01.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.775+0000 7f203091d640 10 monclient: build_initial_monmap 2024-09-15T03:29:01.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.775+0000 7f203091d640 10 monclient: monmap: 2024-09-15T03:29:01.777 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:01.777 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.777 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:01.776498+0000 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:01.776498+0000 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 1 Processor -- start 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 1 -- start start 2024-09-15T03:29:01.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 10 monclient: init 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 10 monclient(hunting): picked mon.noname-b con 0x7f202c09c9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f202c09e340 0x7f202c09e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.776+0000 7f203091d640 10 monclient(hunting): picked mon.noname-a con 0x7f202c09e340 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f202c12f770 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202c09bdd0 con 0x7f202c12f770 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c09ec90 con 0x7f202c09e340 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c09a880 con 0x7f202c09c9c0 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f203091d640 10 monclient(hunting): authenticate will time out at 739.416050s 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f2029d9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f202c09e340 msgr2=0x7f202c09e750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202ad9d640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f202c12f770 legacy=0x7f202c12fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f2029d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f202c09e340 0x7f202c09e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c09d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34692/0 (socket says 172.21.15.153:34692) 2024-09-15T03:29:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 -- 172.21.15.153:0/633064651 learned_addr learned my addr 172.21.15.153:0/633064651 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): get_auth_request con 0x7f202c09c9c0 auth_method 0 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient(hunting): handle_auth_done global_id 4394 payload 274 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 -- 172.21.15.153:0/633064651 >> v2:172.21.15.153:3300/0 conn(0x7f202c09e340 msgr2=0x7f202c09e750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 --2- 172.21.15.153:0/633064651 >> v2:172.21.15.153:3300/0 conn(0x7f202c09e340 0x7f202c09e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 -- 172.21.15.153:0/633064651 >> v1:172.21.15.153:6789/0 conn(0x7f202c12f770 legacy=0x7f202c12fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 monclient: found mon.noname-b 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.777+0000 7f202a59c640 1 -- 172.21.15.153:0/633064651 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c09a440 con 0x7f202c09c9c0 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202a59c640 1 -- 172.21.15.153:0/633064651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 msgr2=0x7f202c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202a59c640 1 --2- 172.21.15.153:0/633064651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c09d830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f201c009940 tx=0x7f201c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be6acf6a330644cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 1 -- 172.21.15.153:0/633064651 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c032440 con 0x7f202c09c9c0 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:01.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: dump: 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 1 -- 172.21.15.153:0/633064651 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c0325a0 con 0x7f202c09c9c0 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 5 monclient: authenticate success, global_id 4394 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 1 -- 172.21.15.153:0/633064651 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c032940 con 0x7f202c09c9c0 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 4 set_mon_vals no callback set 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f202959a640 10 monclient: dump: 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 10 monclient: shutdown 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 -- 172.21.15.153:0/633064651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 msgr2=0x7f202c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 --2- 172.21.15.153:0/633064651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c09d830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f201c009940 tx=0x7f201c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 -- 172.21.15.153:0/633064651 shutdown_connections 2024-09-15T03:29:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 --2- 172.21.15.153:0/633064651 >> v2:172.21.15.153:3300/0 conn(0x7f202c09e340 0x7f202c09e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 --2- 172.21.15.153:0/633064651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c09d830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 -- 172.21.15.153:0/633064651 >> 172.21.15.153:0/633064651 conn(0x7f202c0a2e40 msgr2=0x7f202c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 -- 172.21.15.153:0/633064651 shutdown_connections 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.780+0000 7f203091d640 1 -- 172.21.15.153:0/633064651 wait complete. 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.781+0000 7f203091d640 10 monclient: build_initial_monmap 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.781+0000 7f203091d640 10 monclient: monmap: 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.781+0000 7f203091d640 1 Processor -- start 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.781+0000 7f203091d640 1 -- start start 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.781+0000 7f203091d640 10 monclient: init 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c1417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): picked mon.noname-c con 0x7f202c09c9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c09e340 0x7f202c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): picked mon.noname-b con 0x7f202c09e340 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f202c12f770 0x7f202c068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): picked mon.noname-a con 0x7f202c12f770 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c121550 con 0x7f202c12f770 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c121b80 con 0x7f202c09e340 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c120250 con 0x7f202c09c9c0 2024-09-15T03:29:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f203091d640 10 monclient(hunting): authenticate will time out at 739.421160s 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202ad9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f202c12f770 msgr2=0x7f202c068900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f202c12f770 0x7f202c068900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c1417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c1417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34704/0 (socket says 172.21.15.153:34704) 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 -- 172.21.15.153:0/119205386 learned_addr learned my addr 172.21.15.153:0/119205386 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): get_auth_request con 0x7f202c09c9c0 auth_method 0 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f2029d9b640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c09e340 0x7f202c141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f2029d9b640 10 monclient(hunting): get_auth_request con 0x7f202c09e340 auth_method 0 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f2029d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f2029d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f2029d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c09e340 msgr2=0x7f202c141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c09e340 0x7f202c141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f202c12f770 msgr2=0x7f202c068900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f202c12f770 0x7f202c068900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 monclient: found mon.noname-c 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.782+0000 7f202a59c640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c009450 con 0x7f202c09c9c0 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f202a59c640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 msgr2=0x7f202c1417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f202a59c640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c1417d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f201c0099a0 tx=0x7f201c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c03c960 con 0x7f202c09c9c0 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: dump: 2024-09-15T03:29:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c03cac0 con 0x7f202c09c9c0 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 5 monclient: authenticate success, global_id 4397 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 10 monclient: _renew_subs 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c098e90 con 0x7f202c09c9c0 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c03ce60 con 0x7f202c09c9c0 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f201a7fc640 10 monclient: dump: 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f2028d99640 4 set_mon_vals no callback set 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 10 monclient: _renew_subs 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.784+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c120660 con 0x7f202c09c9c0 2024-09-15T03:29:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f203091d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f203091d640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 0x7f202c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c065e00 con 0x7f202c069600 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f201c039040 con 0x7f202c09c9c0 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 0x7f202c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient: get_auth_request con 0x7f202c069600 auth_method 0 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f2029d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.785+0000 7f201a7fc640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f200c0386f0 0x7f200c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.786+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f201c06bf90 con 0x7f202c09c9c0 2024-09-15T03:29:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.786+0000 7f2029d9b640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-09-15T03:29:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.786+0000 7f2029d9b640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f202c04bf60 con 0x7f202c069600 2024-09-15T03:29:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.786+0000 7f202ad9d640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f200c0386f0 0x7f200c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.786+0000 7f202ad9d640 10 monclient: get_auth_request con 0x7f200c0386f0 auth_method 0 2024-09-15T03:29:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f2029d9b640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 msgr2=0x7f202c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f2029d9b640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 0x7f202c0298c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f202000d6f0 tx=0x7f2020004ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020012820 con 0x7f202c069600 2024-09-15T03:29:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 10 monclient: dump: 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f202ad9d640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f200c0386f0 0x7f200c03aba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2024003cc0 tx=0x7f2024009370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2020012980 con 0x7f202c069600 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 msgr2=0x7f202c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 0x7f202c0298c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f202000d6f0 tx=0x7f2020004ba0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 msgr2=0x7f202c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.788+0000 7f201a7fc640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c069600 0x7f202c0298c0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f202000d6f0 tx=0x7f2020004ba0 comp rx=0 tx=0).stop 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f203091d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f203091d640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 0x7f202c029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f203091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c099ef0 con 0x7f202c0699d0 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 0x7f202c029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient: get_auth_request con 0x7f202c0699d0 auth_method 0 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.794+0000 7f2029d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.795+0000 7f2029d9b640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.795+0000 7f2029d9b640 1 -- 172.21.15.153:0/119205386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f202c06ef90 con 0x7f202c0699d0 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f2029d9b640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 msgr2=0x7f202c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f2029d9b640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 0x7f202c029e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2020017000 tx=0x7f2020018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202001e650 con 0x7f202c0699d0 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 10 monclient: dump: 2024-09-15T03:29:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f2020023070 con 0x7f202c0699d0 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 msgr2=0x7f202c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 0x7f202c029e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2020017000 tx=0x7f2020018000 comp rx=0 tx=0).stop 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 msgr2=0x7f202c029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.797+0000 7f201a7fc640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202c0699d0 0x7f202c029e00 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2020017000 tx=0x7f2020018000 comp rx=0 tx=0).stop 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:01.813 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 6, 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:01.814 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:01.815 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:01.816 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:01.817 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:01.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:01.819 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:01.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:01.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.823 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:01.827 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:01.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f200c0386f0 msgr2=0x7f200c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f200c0386f0 0x7f200c03aba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2024003cc0 tx=0x7f2024009370 comp rx=0 tx=0).stop 2024-09-15T03:29:01.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 10 monclient: shutdown 2024-09-15T03:29:01.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 msgr2=0x7f202c1417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 --2- 172.21.15.153:0/119205386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f202c09c9c0 0x7f202c1417d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f201c0099a0 tx=0x7f201c032000 comp rx=0 tx=0).stop 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f202ad9d640 1 -- 172.21.15.153:0/119205386 reap_dead start 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f202ad9d640 1 -- 172.21.15.153:0/119205386 reap_dead start 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 shutdown_connections 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 >> 172.21.15.153:0/119205386 conn(0x7f202c0a2e40 msgr2=0x7f202c0977c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 shutdown_connections 2024-09-15T03:29:01.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:01.798+0000 7f203091d640 1 -- 172.21.15.153:0/119205386 wait complete. 2024-09-15T03:29:01.829 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:29:01.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:29:21.831 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:29:21.831 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:29:21.831 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:29:21.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:29:21.835 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:29:21.835 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:29:21.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:21.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.939+0000 7f0fb0a4f640 10 monclient: get_monmap_and_config 2024-09-15T03:29:21.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.939+0000 7f0fb0a4f640 10 monclient: build_initial_monmap 2024-09-15T03:29:21.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.939+0000 7f0fb0a4f640 10 monclient: monmap: 2024-09-15T03:29:21.940 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:21.940 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:21.940 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:21.940078+0000 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:21.940078+0000 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:21.941 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.939+0000 7f0fb0a4f640 1 Processor -- start 2024-09-15T03:29:21.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.939+0000 7f0fb0a4f640 1 -- start start 2024-09-15T03:29:21.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient: init 2024-09-15T03:29:21.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:21.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 0x7f0fac131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f0fac131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fac09e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fac136bb0 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fac09faf0 con 0x7f0fac136bb0 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fac09fd30 con 0x7f0fac09e910 2024-09-15T03:29:21.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fac0a1110 con 0x7f0fac131ab0 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0fb0a4f640 10 monclient(hunting): authenticate will time out at 759.579779s 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 0x7f0fac131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0fac136bb0 0x7f0fac138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37420/0 (socket says 172.21.15.153:37420) 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faad9d640 1 -- 172.21.15.153:0/1613287383 learned_addr learned my addr 172.21.15.153:0/1613287383 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:29:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 10 monclient(hunting): get_auth_request con 0x7f0fac131ab0 auth_method 0 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 1 -- 172.21.15.153:0/1613287383 >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 msgr2=0x7f0fac131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 1 -- 172.21.15.153:0/1613287383 >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 msgr2=0x7f0fac131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 1 --2- 172.21.15.153:0/1613287383 >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 0x7f0fac131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:21.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.940+0000 7f0faa59c640 1 --2- 172.21.15.153:0/1613287383 >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 0x7f0fac131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:21.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 1 -- 172.21.15.153:0/1613287383 >> v1:172.21.15.181:6789/0 conn(0x7f0fac09e910 legacy=0x7f0fac09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:21.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 1 -- 172.21.15.153:0/1613287383 >> v1:172.21.15.153:6789/0 conn(0x7f0fac136bb0 legacy=0x7f0fac138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:21.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034902459 0 0) 0x7f0fac09fd30 con 0x7f0fac09e910 2024-09-15T03:29:21.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:21.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): my global_id is 4378 2024-09-15T03:29:21.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f940034e0 con 0x7f0fac09e910 2024-09-15T03:29:21.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2374538434 0 0) 0x7f0f940034e0 con 0x7f0fac09e910 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 msgr2=0x7f0fac131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 --2- 172.21.15.153:0/1613287383 >> v2:172.21.15.153:3300/0 conn(0x7f0fac131ab0 0x7f0fac131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 >> v1:172.21.15.153:6789/0 conn(0x7f0fac136bb0 legacy=0x7f0fac138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: _finish_hunting 0 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac0a13f0 con 0x7f0fac09e910 2024-09-15T03:29:21.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: _finish_auth 0 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f0fa0001830 con 0x7f0fac09e910 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fb0a4f640 5 monclient: authenticate success, global_id 4378 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fb0a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: dump: 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 >> v1:172.21.15.181:6789/0 conn(0x7f0fac09e910 legacy=0x7f0fac09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:21.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 0x7f0f94003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): picked mon.a con 0x7f0f94003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94004780 0x7f0f94004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): picked mon.c con 0x7f0f94004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f940052e0 0x7f0f9400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): picked mon.b con 0x7f0f940052e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac0a1110 con 0x7f0f94003a50 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9400c2b0 con 0x7f0f940052e0 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9400c470 con 0x7f0f94004780 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fb0a4f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 0x7f0f94003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94004780 0x7f0f94004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): get_auth_request con 0x7f0f94003a50 auth_method 0 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): get_auth_request con 0x7f0f94004780 auth_method 0 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f940052e0 0x7f0f9400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): get_auth_request con 0x7f0f940052e0 auth_method 0 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:21.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0fa9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.943+0000 7f0faa59c640 10 monclient(hunting): handle_auth_done global_id 4378 payload 293 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 1 -- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94004780 msgr2=0x7f0f94004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94004780 0x7f0f94004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 1 -- 172.21.15.153:0/1613287383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f940052e0 msgr2=0x7f0f9400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f940052e0 0x7f0f9400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 1 monclient: found mon.a 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faad9d640 1 -- 172.21.15.153:0/1613287383 reap_dead start 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 1 -- 172.21.15.153:0/1613287383 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f940034e0 con 0x7f0f94003a50 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 10 monclient: _finish_auth 0 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0faa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.944+0000 7f0fa9d9b640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94004780 0x7f0f94004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0faa59c640 1 -- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 msgr2=0x7f0f94003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0faa59c640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 0x7f0f94003e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c006fd0 tx=0x7f0f9c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff137dc2b505da8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c02add0 con 0x7f0f94003a50 2024-09-15T03:29:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 10 monclient: dump: 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fb0a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c02e070 con 0x7f0f94003a50 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fb0a4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fb0a4f640 4 set_mon_vals no callback set 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.946+0000 7f0fa959a640 1 -- 172.21.15.153:0/1613287383 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c034660 con 0x7f0f94003a50 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fa959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fa959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fa959a640 10 monclient: dump: 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 10 monclient: shutdown 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 msgr2=0x7f0f94003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 0x7f0f94003e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c006fd0 tx=0x7f0f9c008040 comp rx=0 tx=0).stop 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/1613287383 shutdown_connections 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/1613287383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f94003a50 0x7f0f94003e60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/1613287383 >> 172.21.15.153:0/1613287383 conn(0x7f0fac067d30 msgr2=0x7f0fac0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/1613287383 shutdown_connections 2024-09-15T03:29:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.947+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/1613287383 wait complete. 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 10 monclient: build_initial_monmap 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 10 monclient: monmap: 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 1 Processor -- start 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 1 -- start start 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 10 monclient: init 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 0x7f0fac1412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f0f94003a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.948+0000 7f0fb0a4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 0x7f0fac12fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f0fac1417e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fac130090 0x7f0fac098ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f0fac130090 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac09a6b0 con 0x7f0fac1417e0 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac06a020 con 0x7f0fac130090 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac09e120 con 0x7f0f94003a50 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fb0a4f640 10 monclient(hunting): authenticate will time out at 759.588058s 2024-09-15T03:29:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 0x7f0fac1412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 0x7f0fac12fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 0x7f0fac1412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37964/0 (socket says 172.21.15.153:37964) 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 -- 172.21.15.153:0/673636267 learned_addr learned my addr 172.21.15.153:0/673636267 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): get_auth_request con 0x7f0f94003a50 auth_method 0 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 10 monclient(hunting): get_auth_request con 0x7f0fac1417e0 auth_method 0 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:21.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fac130090 0x7f0fac098ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 msgr2=0x7f0fac12fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 msgr2=0x7f0fac12fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 0x7f0fac12fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): get_auth_request con 0x7f0fac130090 auth_method 0 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:21.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0fa9d9b640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 0x7f0fac12fb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fac130090 msgr2=0x7f0fac098ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fac130090 0x7f0fac098ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 msgr2=0x7f0fac12fb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 0x7f0fac12fb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 monclient: found mon.noname-c 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.949+0000 7f0faa59c640 1 -- 172.21.15.153:0/673636267 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac1420e0 con 0x7f0f94003a50 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0faa59c640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 msgr2=0x7f0fac1412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0faa59c640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 0x7f0fac1412a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00baf0 tx=0x7f0f9c035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c043070 con 0x7f0f94003a50 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: dump: 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.986 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 5 monclient: authenticate success, global_id 4400 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c0335c0 con 0x7f0f94003a50 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 10 monclient: _renew_subs 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:21.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac11bc60 con 0x7f0f94003a50 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c033980 con 0x7f0f94003a50 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0f9a7fc640 10 monclient: dump: 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:21.988 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:21.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fa8d99640 4 set_mon_vals no callback set 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 10 monclient: _renew_subs 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.951+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fac09b1f0 con 0x7f0f94003a50 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.952+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0f9c033d30 con 0x7f0f94003a50 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.952+0000 7f0fb0a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.952+0000 7f0fb0a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.952+0000 7f0fb0a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.952+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fac04bf60 con 0x7f0f94003a50 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.952+0000 7f0f9a7fc640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0f8c038740 0x7f0f8c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.953+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f0f9c06b9c0 con 0x7f0f94003a50 2024-09-15T03:29:21.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.953+0000 7f0fa9d9b640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0f8c038740 0x7f0f8c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:21.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.953+0000 7f0fa9d9b640 10 monclient: get_auth_request con 0x7f0f8c038740 auth_method 0 2024-09-15T03:29:21.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.955+0000 7f0fa9d9b640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0f8c038740 0x7f0f8c03abf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0fac09a2b0 tx=0x7f0fa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:21.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.957+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0f9c047370 con 0x7f0f94003a50 2024-09-15T03:29:21.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.957+0000 7f0f9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:21.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:21.957+0000 7f0f9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.175+0000 7f0fb0a4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.175+0000 7f0fb0a4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.175+0000 7f0fb0a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.175+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0fac1324b0 con 0x7f0f94003a50 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.176+0000 7f0f9a7fc640 1 -- 172.21.15.153:0/673636267 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f0f9c032920 con 0x7f0f94003a50 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.176+0000 7f0f9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.176+0000 7f0f9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:22.178 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0f8c038740 msgr2=0x7f0f8c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0f8c038740 0x7f0f8c03abf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0fac09a2b0 tx=0x7f0fa0009040 comp rx=0 tx=0).stop 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 10 monclient: shutdown 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 msgr2=0x7f0fac1412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 0x7f0fac1412a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00baf0 tx=0x7f0f9c035000 comp rx=0 tx=0).stop 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 shutdown_connections 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0f8c038740 0x7f0f8c03abf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fac130090 0x7f0fac098ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac1417e0 0x7f0fac12fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:22.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 --2- 172.21.15.153:0/673636267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f94003a50 0x7f0fac1412a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:22.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 >> 172.21.15.153:0/673636267 conn(0x7f0fac067d30 msgr2=0x7f0fac130a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:22.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.177+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 shutdown_connections 2024-09-15T03:29:22.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:22.178+0000 7f0fb0a4f640 1 -- 172.21.15.153:0/673636267 wait complete. 2024-09-15T03:29:25.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:25.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.293+0000 7f5bfc00f640 10 monclient: get_monmap_and_config 2024-09-15T03:29:25.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.293+0000 7f5bfc00f640 10 monclient: build_initial_monmap 2024-09-15T03:29:25.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.293+0000 7f5bfc00f640 10 monclient: monmap: 2024-09-15T03:29:25.296 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:25.296 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:25.296 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:25.295513+0000 2024-09-15T03:29:25.297 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:25.295513+0000 2024-09-15T03:29:25.297 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:25.297 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:25.297 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:25.297 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:25.298 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:25.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.294+0000 7f5bfc00f640 1 Processor -- start 2024-09-15T03:29:25.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.294+0000 7f5bfc00f640 1 -- start start 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.294+0000 7f5bfc00f640 10 monclient: init 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5bf406eb80 0x7f5bf406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf406eb80 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf4068a20 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bf406f490 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf4069b80 con 0x7f5bf406f490 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:25.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf4099f90 con 0x7f5bf4068a20 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf4069e00 con 0x7f5bf406eb80 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bfc00f640 10 monclient(hunting): authenticate will time out at 762.935286s 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58624/0 (socket says 172.21.15.153:58624) 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 -- 172.21.15.153:0/2376155491 learned_addr learned my addr 172.21.15.153:0/2376155491 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 10 monclient(hunting): get_auth_request con 0x7f5bf4068a20 auth_method 0 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 1 --2- 172.21.15.153:0/2376155491 >> v2:172.21.15.181:3300/0 conn(0x7f5bf406eb80 0x7f5bf406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): get_auth_request con 0x7f5bf406eb80 auth_method 0 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 -- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 msgr2=0x7f5bf409c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 -- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 msgr2=0x7f5bf409c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 --2- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf95aa640 1 --2- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:25.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 1 -- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 msgr2=0x7f5bf409c540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 1 --2- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 1 -- 172.21.15.153:0/2376155491 >> v1:172.21.15.181:6789/0 conn(0x7f5bf406f490 legacy=0x7f5bf409d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient: _finish_hunting 0 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 1 monclient: found mon.noname-c 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.295+0000 7f5bf9dab640 1 -- 172.21.15.153:0/2376155491 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf409ca80 con 0x7f5bf406eb80 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.298+0000 7f5bf9dab640 1 -- 172.21.15.153:0/2376155491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf406eb80 msgr2=0x7f5bf406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.298+0000 7f5bf9dab640 1 --2- 172.21.15.153:0/2376155491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf406eb80 0x7f5bf406ef50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5bf406aae0 tx=0x7f5bf00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fac2fd594405dbba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 1 -- 172.21.15.153:0/2376155491 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002ae00 con 0x7f5bf406eb80 2024-09-15T03:29:25.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: dump: 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:25.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: _finish_auth 0 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 1 -- 172.21.15.153:0/2376155491 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf0032440 con 0x7f5bf406eb80 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 5 monclient: authenticate success, global_id 4384 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 1 -- 172.21.15.153:0/2376155491 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf00327e0 con 0x7f5bf406eb80 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 4 set_mon_vals no callback set 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bf8da9640 10 monclient: dump: 2024-09-15T03:29:25.316 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 10 monclient: shutdown 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 -- 172.21.15.153:0/2376155491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf406eb80 msgr2=0x7f5bf406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/2376155491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf406eb80 0x7f5bf406ef50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5bf406aae0 tx=0x7f5bf00090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 -- 172.21.15.153:0/2376155491 shutdown_connections 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/2376155491 >> v2:172.21.15.153:3300/0 conn(0x7f5bf4068a20 0x7f5bf409c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/2376155491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf406eb80 0x7f5bf406ef50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 -- 172.21.15.153:0/2376155491 >> 172.21.15.153:0/2376155491 conn(0x7f5bf4067cd0 msgr2=0x7f5bf40a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 -- 172.21.15.153:0/2376155491 shutdown_connections 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.299+0000 7f5bfc00f640 1 -- 172.21.15.153:0/2376155491 wait complete. 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.300+0000 7f5bfc00f640 10 monclient: build_initial_monmap 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.300+0000 7f5bfc00f640 10 monclient: monmap: 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:25.318 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.300+0000 7f5bfc00f640 1 Processor -- start 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.300+0000 7f5bfc00f640 1 -- start start 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.300+0000 7f5bfc00f640 10 monclient: init 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4068a20 0x7f5bf4140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:25.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): picked mon.noname-b con 0x7f5bf4068a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 0x7f5bf4133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf406eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf406f490 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4121090 con 0x7f5bf406f490 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf411f280 con 0x7f5bf4068a20 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4141d90 con 0x7f5bf406eb80 2024-09-15T03:29:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfc00f640 10 monclient(hunting): authenticate will time out at 762.941082s 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 0x7f5bf4133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 0x7f5bf4133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49198/0 (socket says 172.21.15.153:49198) 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 -- 172.21.15.153:0/109293677 learned_addr learned my addr 172.21.15.153:0/109293677 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58638/0 (socket says 172.21.15.153:58638) 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): get_auth_request con 0x7f5bf406eb80 auth_method 0 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 10 monclient(hunting): get_auth_request con 0x7f5bf406f490 auth_method 0 2024-09-15T03:29:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4068a20 0x7f5bf4140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): get_auth_request con 0x7f5bf4068a20 auth_method 0 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 msgr2=0x7f5bf411d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 msgr2=0x7f5bf411d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:29:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bfa5ac640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf9dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4068a20 msgr2=0x7f5bf4140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4068a20 0x7f5bf4140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 msgr2=0x7f5bf411d940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient: _finish_hunting 0 2024-09-15T03:29:25.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 monclient: found mon.noname-c 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.301+0000 7f5bf95aa640 1 -- 172.21.15.153:0/109293677 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0009390 con 0x7f5bf406eb80 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bf95aa640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 msgr2=0x7f5bf4133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bf95aa640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 0x7f5bf4133c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e870 tx=0x7f5be4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400c6d0 con 0x7f5bf406eb80 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: dump: 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:25.324 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bfc00f640 5 monclient: authenticate success, global_id 4403 2024-09-15T03:29:25.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be400c830 con 0x7f5bf406eb80 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400cbf0 con 0x7f5bf406eb80 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bfc00f640 10 monclient: _renew_subs 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bfc00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf4120db0 con 0x7f5bf406eb80 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5bebfff640 4 set_mon_vals no callback set 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.303+0000 7f5be9ffb640 10 monclient: dump: 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:25.326 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.304+0000 7f5bfc00f640 10 monclient: _renew_subs 2024-09-15T03:29:25.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.304+0000 7f5bfc00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.304+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf4141490 con 0x7f5bf406eb80 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5be4013410 con 0x7f5bf406eb80 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5bfc00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5bfc00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5bfc00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf404bf60 con 0x7f5bf406eb80 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5be9ffb640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bd8038740 0x7f5bd803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f5be404b8e0 con 0x7f5bf406eb80 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5bf9dab640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bd8038740 0x7f5bd803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.305+0000 7f5bf9dab640 10 monclient: get_auth_request con 0x7f5bd8038740 auth_method 0 2024-09-15T03:29:25.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.308+0000 7f5bf9dab640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bd8038740 0x7f5bd803abf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0031000 tx=0x7f5bf0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:25.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.309+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5be40136c0 con 0x7f5bf406eb80 2024-09-15T03:29:25.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.309+0000 7f5be9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:25.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.309+0000 7f5be9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:25.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.527+0000 7f5bfc00f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:25.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.527+0000 7f5bfc00f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:25.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.527+0000 7f5bfc00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:25.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.527+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5bf40638d0 con 0x7f5bf406eb80 2024-09-15T03:29:25.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.528+0000 7f5be9ffb640 1 -- 172.21.15.153:0/109293677 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f5be4024350 con 0x7f5bf406eb80 2024-09-15T03:29:25.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.528+0000 7f5be9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:25.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.528+0000 7f5be9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:25.531 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:25.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bd8038740 msgr2=0x7f5bd803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:25.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bd8038740 0x7f5bd803abf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0031000 tx=0x7f5bf0032000 comp rx=0 tx=0).stop 2024-09-15T03:29:25.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 10 monclient: shutdown 2024-09-15T03:29:25.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:25.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 msgr2=0x7f5bf4133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 0x7f5bf4133c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e870 tx=0x7f5be4008040 comp rx=0 tx=0).stop 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 shutdown_connections 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5bd8038740 0x7f5bd803abf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf406f490 0x7f5bf411d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf406eb80 0x7f5bf4133c90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 --2- 172.21.15.153:0/109293677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4068a20 0x7f5bf4140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 >> 172.21.15.153:0/109293677 conn(0x7f5bf4067cd0 msgr2=0x7f5bf409d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:25.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 shutdown_connections 2024-09-15T03:29:25.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:25.530+0000 7f5bfc00f640 1 -- 172.21.15.153:0/109293677 wait complete. 2024-09-15T03:29:28.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.647+0000 7f0e94354640 10 monclient: get_monmap_and_config 2024-09-15T03:29:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.647+0000 7f0e94354640 10 monclient: build_initial_monmap 2024-09-15T03:29:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.647+0000 7f0e94354640 10 monclient: monmap: 2024-09-15T03:29:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:28.648852+0000 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:28.648852+0000 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.647+0000 7f0e94354640 1 Processor -- start 2024-09-15T03:29:28.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.647+0000 7f0e94354640 1 -- start start 2024-09-15T03:29:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.647+0000 7f0e94354640 10 monclient: init 2024-09-15T03:29:28.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): picked mon.noname-c con 0x7f0e8c06eb80 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): picked mon.noname-a con 0x7f0e8c068a20 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e8c09caf0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8c06a0e0 con 0x7f0e8c09caf0 2024-09-15T03:29:28.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c06a7a0 con 0x7f0e8c068a20 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c09cef0 con 0x7f0e8c06eb80 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e94354640 10 monclient(hunting): authenticate will time out at 766.288484s 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58660/0 (socket says 172.21.15.153:58660) 2024-09-15T03:29:28.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 learned_addr learned my addr 172.21.15.153:0/222333178 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e8c068a20 auth_method 0 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 10 monclient(hunting): get_auth_request con 0x7f0e8c06eb80 auth_method 0 2024-09-15T03:29:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:28.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:28.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.648+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:28.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.650+0000 7f0e928f1640 1 -- 172.21.15.153:0/222333178 >> v1:172.21.15.181:6789/0 conn(0x7f0e8c09caf0 legacy=0x7f0e8c09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e8c068a20 auth_method 0 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 10 monclient(hunting): get_auth_request con 0x7f0e8c06eb80 auth_method 0 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:28.849+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e8c068a20 auth_method 0 2024-09-15T03:29:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 10 monclient(hunting): get_auth_request con 0x7f0e8c06eb80 auth_method 0 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:29.250+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e8c068a20 auth_method 0 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:30.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 10 monclient(hunting): get_auth_request con 0x7f0e8c06eb80 auth_method 0 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:30.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:30.052+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:31.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient(hunting): tick 2024-09-15T03:29:31.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:29:31.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:29:31.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e8c06ef50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e8c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e8c09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e8c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 >> v1:172.21.15.181:6789/0 conn(0x7f0e8c09caf0 legacy=0x7f0e8c09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient: _add_conns ranks=[0,2] 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e700036e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:31.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0e700073c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8c069e00 con 0x7f0e700036e0 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c06a7a0 con 0x7f0e700073c0 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e83fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e700073c0 auth_method 0 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:31.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:31.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:31.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:31.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.648+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:31.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.650+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v1:172.21.15.153:6789/0 conn(0x7f0e700036e0 legacy=0x7f0e70005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e700073c0 auth_method 0 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.849+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:31.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.925+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v1:172.21.15.153:6789/0 conn(0x7f0e700036e0 legacy=0x7f0e70005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T03:29:31.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.925+0000 7f0e920f0640 1 -- 172.21.15.153:0/222333178 >> v1:172.21.15.153:6789/0 conn(0x7f0e700036e0 legacy=0x7f0e70005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:29:31.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.925+0000 7f0e920f0640 1 --1- 172.21.15.153:0/222333178 >> v1:172.21.15.153:6789/0 conn(0x7f0e700036e0 0x7f0e70005b50 :-1 s=OPENED pgs=26 cs=1 l=1).handle_message read tag failed 2024-09-15T03:29:31.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.925+0000 7f0e920f0640 1 --1- 172.21.15.153:0/222333178 >> v1:172.21.15.153:6789/0 conn(0x7f0e700036e0 0x7f0e70005b50 :-1 s=OPENED pgs=26 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:29:31.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:31.925+0000 7f0e910ee640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6789/0 2024-09-15T03:29:32.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:32.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e700073c0 auth_method 0 2024-09-15T03:29:32.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:32.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:32.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:32.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:32.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:32.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:32.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:32.250+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:33.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e700073c0 auth_method 0 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:33.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:33.052+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:34.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): tick 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e70009870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e70009870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 >> v1:172.21.15.153:6789/0 conn(0x7f0e700036e0 legacy=0x7f0e70005b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 -- 172.21.15.153:0/222333178 reap_dead start 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e70009ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0e8c06eb80 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e7000a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0e8c068a20 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:34.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e7000f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0e700073c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c06a7a0 con 0x7f0e8c068a20 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c09cef0 con 0x7f0e8c06eb80 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 1 -- 172.21.15.153:0/222333178 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e70009920 con 0x7f0e700073c0 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e83fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e7000a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e8c068a20 auth_method 0 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e7000f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): get_auth_request con 0x7f0e700073c0 auth_method 0 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e70009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): get_auth_request con 0x7f0e8c06eb80 auth_method 0 2024-09-15T03:29:34.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e920f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 msgr2=0x7f0e70009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e70009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 -- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 msgr2=0x7f0e7000a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e7000a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient: _finish_hunting 0 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 monclient: found mon.noname-b 2024-09-15T03:29:34.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:34.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:34.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e928f1640 1 -- 172.21.15.153:0/222333178 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7000b000 con 0x7f0e700073c0 2024-09-15T03:29:34.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.649+0000 7f0e918ef640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e7000a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e928f1640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e7000f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e928f1640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e7000f2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e70003ab0 tx=0x7f0e84001920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8766b5fcb3232a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 1 -- 172.21.15.153:0/222333178 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84005d10 con 0x7f0e700073c0 2024-09-15T03:29:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: dump: 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: _finish_auth 0 2024-09-15T03:29:34.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 1 -- 172.21.15.153:0/222333178 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e84005e70 con 0x7f0e700073c0 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 1 -- 172.21.15.153:0/222333178 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84007300 con 0x7f0e700073c0 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e94354640 5 monclient: authenticate success, global_id 4406 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e94354640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e94354640 4 set_mon_vals no callback set 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.652+0000 7f0e910ee640 10 monclient: dump: 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:34.664 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:34.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 10 monclient: shutdown 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 -- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 msgr2=0x7f0e7000f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e7000f2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e70003ab0 tx=0x7f0e84001920 comp rx=0 tx=0).stop 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 -- 172.21.15.153:0/222333178 shutdown_connections 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 --2- 172.21.15.153:0/222333178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e700073c0 0x7f0e7000f2f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.153:3300/0 conn(0x7f0e8c068a20 0x7f0e7000a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 --2- 172.21.15.153:0/222333178 >> v2:172.21.15.181:3300/0 conn(0x7f0e8c06eb80 0x7f0e70009ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 -- 172.21.15.153:0/222333178 >> 172.21.15.153:0/222333178 conn(0x7f0e8c067cd0 msgr2=0x7f0e8c0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:34.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 -- 172.21.15.153:0/222333178 shutdown_connections 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.653+0000 7f0e94354640 1 -- 172.21.15.153:0/222333178 wait complete. 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.654+0000 7f0e94354640 10 monclient: build_initial_monmap 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.654+0000 7f0e94354640 10 monclient: monmap: 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:34.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.654+0000 7f0e94354640 1 Processor -- start 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.654+0000 7f0e94354640 1 -- start start 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.654+0000 7f0e94354640 10 monclient: init 2024-09-15T03:29:34.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e700073c0 0x7f0e8c134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): picked mon.noname-b con 0x7f0e700073c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 0x7f0e8c11e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): picked mon.noname-a con 0x7f0e8c068a20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c06eb80 0x7f0e8c11eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): picked mon.noname-c con 0x7f0e8c06eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:34.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c120690 con 0x7f0e8c068a20 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c09c2c0 con 0x7f0e700073c0 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c09d050 con 0x7f0e8c06eb80 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e94354640 10 monclient(hunting): authenticate will time out at 772.295438s 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 0x7f0e8c11e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 0x7f0e8c11e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58704/0 (socket says 172.21.15.153:58704) 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c06eb80 0x7f0e8c11eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 -- 172.21.15.153:0/3810637770 learned_addr learned my addr 172.21.15.153:0/3810637770 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:34.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): get_auth_request con 0x7f0e8c068a20 auth_method 0 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): get_auth_request con 0x7f0e8c06eb80 auth_method 0 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e700073c0 0x7f0e8c134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e928f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): get_auth_request con 0x7f0e700073c0 auth_method 0 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e920f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient(hunting): handle_auth_done global_id 34101 payload 824 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 -- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c06eb80 msgr2=0x7f0e8c11eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c06eb80 0x7f0e8c11eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 -- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e700073c0 msgr2=0x7f0e8c134ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:34.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e700073c0 0x7f0e8c134ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient: _finish_hunting 0 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 monclient: found mon.noname-a 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.655+0000 7f0e918ef640 1 -- 172.21.15.153:0/3810637770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8800e040 con 0x7f0e8c068a20 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e918ef640 1 -- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 msgr2=0x7f0e8c11e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e918ef640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 0x7f0e8c11e5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e88019b80 tx=0x7f0e8801b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8801e050 con 0x7f0e8c068a20 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: dump: 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 5 monclient: authenticate success, global_id 34101 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8801f040 con 0x7f0e8c068a20 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 10 monclient: _renew_subs 2024-09-15T03:29:34.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c06a0e0 con 0x7f0e8c068a20 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e880168c0 con 0x7f0e8c068a20 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e827fc640 10 monclient: dump: 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 10 monclient: _renew_subs 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c141b30 con 0x7f0e8c068a20 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.658+0000 7f0e908ed640 4 set_mon_vals no callback set 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0e88023650 con 0x7f0e8c068a20 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e827fc640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0e70039640 0x7f0e7003baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f0e8804d700 con 0x7f0e8c068a20 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e94354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e94354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e94354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.659+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c04bf60 con 0x7f0e8c068a20 2024-09-15T03:29:34.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.660+0000 7f0e920f0640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0e70039640 0x7f0e7003baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.660+0000 7f0e920f0640 10 monclient: get_auth_request con 0x7f0e70039640 auth_method 0 2024-09-15T03:29:34.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.662+0000 7f0e920f0640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0e70039640 0x7f0e7003baf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c001eb0 tx=0x7f0e7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:34.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.665+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0e88027080 con 0x7f0e8c068a20 2024-09-15T03:29:34.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.665+0000 7f0e827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:34.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.665+0000 7f0e827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e94354640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e94354640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e94354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0e8c0008d0 con 0x7f0e8c068a20 2024-09-15T03:29:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e827fc640 1 -- 172.21.15.153:0/3810637770 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0e88015360 con 0x7f0e8c068a20 2024-09-15T03:29:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.880+0000 7f0e827fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:34.884 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0e70039640 msgr2=0x7f0e7003baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0e70039640 0x7f0e7003baf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c001eb0 tx=0x7f0e7c008040 comp rx=0 tx=0).stop 2024-09-15T03:29:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 10 monclient: shutdown 2024-09-15T03:29:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 msgr2=0x7f0e8c11e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 0x7f0e8c11e5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e88019b80 tx=0x7f0e8801b040 comp rx=0 tx=0).stop 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 shutdown_connections 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0e70039640 0x7f0e7003baf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c06eb80 0x7f0e8c11eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c068a20 0x7f0e8c11e5c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 --2- 172.21.15.153:0/3810637770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e700073c0 0x7f0e8c134ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 >> 172.21.15.153:0/3810637770 conn(0x7f0e8c067cd0 msgr2=0x7f0e8c06d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 shutdown_connections 2024-09-15T03:29:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.882+0000 7f0e94354640 1 -- 172.21.15.153:0/3810637770 wait complete. 2024-09-15T03:29:34.895 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:29:34.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:29:34.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:29:34.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.987+0000 7f328839d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.987+0000 7f328839d640 10 monclient: build_initial_monmap 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.988+0000 7f328839d640 10 monclient: monmap: 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:34.989723+0000 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:34.989723+0000 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.988+0000 7f328839d640 1 Processor -- start 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.988+0000 7f328839d640 1 -- start start 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.988+0000 7f328839d640 10 monclient: init 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f328009c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f328009e840 0x7f328009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient(hunting): picked mon.noname-c con 0x7f328009e840 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3280136cb0 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328009ec40 con 0x7f3280136cb0 2024-09-15T03:29:34.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328006a320 con 0x7f328009c860 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3280068a20 con 0x7f328009e840 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328839d640 10 monclient(hunting): authenticate will time out at 772.629289s 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328693a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3280136cb0 0x7f3280139090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38834/0 (socket says 172.21.15.153:38834) 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f328693a640 1 -- 172.21.15.153:0/3840338767 learned_addr learned my addr 172.21.15.153:0/3840338767 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 1 --2- 172.21.15.153:0/3840338767 >> v2:172.21.15.181:3300/0 conn(0x7f328009e840 0x7f328009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): get_auth_request con 0x7f328009e840 auth_method 0 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:34.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 1 -- 172.21.15.153:0/3840338767 >> v1:172.21.15.181:6789/0 conn(0x7f328009c860 legacy=0x7f328009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 1 -- 172.21.15.153:0/3840338767 >> v1:172.21.15.153:6789/0 conn(0x7f3280136cb0 legacy=0x7f3280139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient: _finish_hunting 0 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 1 monclient: found mon.noname-c 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.989+0000 7f3285938640 1 -- 172.21.15.153:0/3840338767 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3280068e60 con 0x7f328009e840 2024-09-15T03:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.992+0000 7f3285938640 1 -- 172.21.15.153:0/3840338767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 msgr2=0x7f328009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:34.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.992+0000 7f3285938640 1 --2- 172.21.15.153:0/3840338767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 0x7f328009aba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f327c009850 tx=0x7f327c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ca1898ff018dfd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.993+0000 7f3285137640 1 -- 172.21.15.153:0/3840338767 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c032440 con 0x7f328009e840 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.993+0000 7f3285137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.993+0000 7f3285137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.993+0000 7f3285137640 10 monclient: dump: 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.993+0000 7f3285137640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.993+0000 7f3285137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f3285137640 1 -- 172.21.15.153:0/3840338767 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c0325a0 con 0x7f328009e840 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f3285137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f3285137640 1 -- 172.21.15.153:0/3840338767 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c032940 con 0x7f328009e840 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f3285137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f3285137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f3285137640 10 monclient: dump: 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.007 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 5 monclient: authenticate success, global_id 4396 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 4 set_mon_vals no callback set 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 10 monclient: shutdown 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 -- 172.21.15.153:0/3840338767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 msgr2=0x7f328009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 --2- 172.21.15.153:0/3840338767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 0x7f328009aba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f327c009850 tx=0x7f327c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 -- 172.21.15.153:0/3840338767 shutdown_connections 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 --2- 172.21.15.153:0/3840338767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 0x7f328009aba0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 -- 172.21.15.153:0/3840338767 >> 172.21.15.153:0/3840338767 conn(0x7f32800a2e40 msgr2=0x7f32800a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 -- 172.21.15.153:0/3840338767 shutdown_connections 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.994+0000 7f328839d640 1 -- 172.21.15.153:0/3840338767 wait complete. 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient: monmap: 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 Processor -- start 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 -- start start 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient: init 2024-09-15T03:29:35.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 0x7f328006d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): picked mon.noname-c con 0x7f328009c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 0x7f328006d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): picked mon.noname-b con 0x7f328009e840 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280136cb0 0x7f328006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): picked mon.noname-a con 0x7f3280136cb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328013a2c0 con 0x7f3280136cb0 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328011ea40 con 0x7f328009e840 2024-09-15T03:29:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328006ecf0 con 0x7f328009c860 2024-09-15T03:29:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.995+0000 7f328839d640 10 monclient(hunting): authenticate will time out at 772.635922s 2024-09-15T03:29:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280136cb0 0x7f328006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 0x7f328006d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280136cb0 0x7f328006db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58710/0 (socket says 172.21.15.153:58710) 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 0x7f328006d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49266/0 (socket says 172.21.15.153:49266) 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 1 -- 172.21.15.153:0/2460686292 learned_addr learned my addr 172.21.15.153:0/2460686292 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): get_auth_request con 0x7f3280136cb0 auth_method 0 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): get_auth_request con 0x7f328009c860 auth_method 0 2024-09-15T03:29:35.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 0x7f328006d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): get_auth_request con 0x7f328009e840 auth_method 0 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 msgr2=0x7f328006d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328009e840 0x7f328006d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280136cb0 msgr2=0x7f328006db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280136cb0 0x7f328006db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 monclient: found mon.noname-c 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f3286139640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c009390 con 0x7f328009c860 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.996+0000 7f328693a640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280136cb0 0x7f328006db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f3286139640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 msgr2=0x7f328006d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f3286139640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 0x7f328006d0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f327400c4e0 tx=0x7f3274007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32740117a0 con 0x7f328009c860 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: dump: 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274011920 con 0x7f328009c860 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 5 monclient: authenticate success, global_id 4412 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274011ce0 con 0x7f328009c860 2024-09-15T03:29:35.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 10 monclient: _renew_subs 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328011e760 con 0x7f328009c860 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f3284936640 4 set_mon_vals no callback set 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f32727fc640 10 monclient: dump: 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 10 monclient: _renew_subs 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.998+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328011d8b0 con 0x7f328009c860 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f328839d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f328839d640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 0x7f328006e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f328839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3280065e00 con 0x7f328011ec00 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3274012630 con 0x7f328009c860 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 0x7f328006e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient: get_auth_request con 0x7f328011ec00 auth_method 0 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f32727fc640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3264038770 0x7f326403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 10 monclient(hunting): handle_auth_done global_id 34107 payload 824 2024-09-15T03:29:35.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f3285938640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f328011db30 con 0x7f328011ec00 2024-09-15T03:29:35.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:34.999+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f327404b530 con 0x7f328009c860 2024-09-15T03:29:35.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.000+0000 7f328693a640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3264038770 0x7f326403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.000+0000 7f328693a640 10 monclient: get_auth_request con 0x7f3264038770 auth_method 0 2024-09-15T03:29:35.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f3285938640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 msgr2=0x7f328006e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f3285938640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 0x7f328006e0a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f328009d3f0 tx=0x7f327c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c031930 con 0x7f328011ec00 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 10 monclient: dump: 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f328693a640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3264038770 0x7f326403ac20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f327800d730 tx=0x7f3278009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f327c041070 con 0x7f328011ec00 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 msgr2=0x7f328006e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 0x7f328006e0a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f328009d3f0 tx=0x7f327c032000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 msgr2=0x7f328006e0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:35.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.002+0000 7f32727fc640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328011ec00 0x7f328006e0a0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f328009d3f0 tx=0x7f327c032000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f328839d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f328839d640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 0x7f3280097e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f328839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328004bf60 con 0x7f3280097a70 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 0x7f3280097e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient: get_auth_request con 0x7f3280097a70 auth_method 0 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 10 monclient(hunting): handle_auth_done global_id 34110 payload 824 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.008+0000 7f3285938640 1 -- 172.21.15.153:0/2460686292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f32800638d0 con 0x7f3280097a70 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.010+0000 7f3285938640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 msgr2=0x7f3280097e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.010+0000 7f3285938640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 0x7f3280097e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f327c009130 tx=0x7f327c03f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c03d600 con 0x7f3280097a70 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 10 monclient: dump: 2024-09-15T03:29:35.027 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f327c03d760 con 0x7f3280097a70 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:35.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 msgr2=0x7f3280097e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 0x7f3280097e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f327c009130 tx=0x7f327c03f070 comp rx=0 tx=0).stop 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 msgr2=0x7f3280097e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f32727fc640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3280097a70 0x7f3280097e70 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f327c009130 tx=0x7f327c03f070 comp rx=0 tx=0).stop 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:29:35.029 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 14, 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.030 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.031 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:35.032 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:35.033 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.034 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.035 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:35.036 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:35.037 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:35.040 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.041 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.042 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:35.042 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.042 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.042 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:35.042 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:35.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3264038770 msgr2=0x7f326403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3264038770 0x7f326403ac20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f327800d730 tx=0x7f3278009040 comp rx=0 tx=0).stop 2024-09-15T03:29:35.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 10 monclient: shutdown 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 msgr2=0x7f328006d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 --2- 172.21.15.153:0/2460686292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328009c860 0x7f328006d0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f327400c4e0 tx=0x7f3274007e10 comp rx=0 tx=0).stop 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328693a640 1 -- 172.21.15.153:0/2460686292 reap_dead start 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328693a640 1 -- 172.21.15.153:0/2460686292 reap_dead start 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 shutdown_connections 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 >> 172.21.15.153:0/2460686292 conn(0x7f32800a2e40 msgr2=0x7f328009b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 shutdown_connections 2024-09-15T03:29:35.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.011+0000 7f328839d640 1 -- 172.21.15.153:0/2460686292 wait complete. 2024-09-15T03:29:35.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:29:35.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:29:35.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.139+0000 7fbf60edd640 10 monclient: get_monmap_and_config 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.139+0000 7fbf60edd640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.139+0000 7fbf60edd640 10 monclient: monmap: 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:35.140807+0000 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:35.140807+0000 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:35.142 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:35.142 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:35.142 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:35.142 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.139+0000 7fbf60edd640 1 Processor -- start 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.139+0000 7fbf60edd640 1 -- start start 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.139+0000 7fbf60edd640 10 monclient: init 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf5c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbf5c09ea50 0x7fbf5c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf5c136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c131cc0 con 0x7fbf5c136c20 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c09fbf0 con 0x7fbf5c1318f0 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c09fe30 con 0x7fbf5c09ea50 2024-09-15T03:29:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf60edd640 10 monclient(hunting): authenticate will time out at 772.780484s 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbf5c09ea50 0x7fbf5c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf5ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbf5c136c20 0x7fbf5c139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36362/0 (socket says 172.21.15.153:36362) 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf5ad9d640 1 -- 172.21.15.153:0/3792917748 learned_addr learned my addr 172.21.15.153:0/3792917748 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): get_auth_request con 0x7fbf5c09ea50 auth_method 0 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_done global_id 34113 payload 274 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 1 -- 172.21.15.153:0/3792917748 >> v1:172.21.15.181:6789/0 conn(0x7fbf5c1318f0 legacy=0x7fbf5c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 1 -- 172.21.15.153:0/3792917748 >> v1:172.21.15.153:6789/0 conn(0x7fbf5c136c20 legacy=0x7fbf5c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 1 monclient: found mon.noname-a 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.140+0000 7fbf59d9b640 1 -- 172.21.15.153:0/3792917748 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0a1210 con 0x7fbf5c09ea50 2024-09-15T03:29:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf59d9b640 1 -- 172.21.15.153:0/3792917748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c09ea50 msgr2=0x7fbf5c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf59d9b640 1 --2- 172.21.15.153:0/3792917748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c09ee20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009850 tx=0x7fbf440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca9a69a994eeca4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 1 -- 172.21.15.153:0/3792917748 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44032440 con 0x7fbf5c09ea50 2024-09-15T03:29:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: dump: 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 1 -- 172.21.15.153:0/3792917748 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf440325a0 con 0x7fbf5c09ea50 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf60edd640 5 monclient: authenticate success, global_id 34113 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf60edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf60edd640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 1 -- 172.21.15.153:0/3792917748 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44032940 con 0x7fbf5c09ea50 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf60edd640 4 set_mon_vals no callback set 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.143+0000 7fbf5959a640 10 monclient: dump: 2024-09-15T03:29:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 10 monclient: shutdown 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 -- 172.21.15.153:0/3792917748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c09ea50 msgr2=0x7fbf5c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 --2- 172.21.15.153:0/3792917748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c09ee20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009850 tx=0x7fbf440090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 -- 172.21.15.153:0/3792917748 shutdown_connections 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 --2- 172.21.15.153:0/3792917748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c09ee20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 -- 172.21.15.153:0/3792917748 >> 172.21.15.153:0/3792917748 conn(0x7fbf5c067110 msgr2=0x7fbf5c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 -- 172.21.15.153:0/3792917748 shutdown_connections 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.144+0000 7fbf60edd640 1 -- 172.21.15.153:0/3792917748 wait complete. 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient: monmap: 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 Processor -- start 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 -- start start 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient: init 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 0x7fbf5c099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c136c20 0x7fbf5c099e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c09b8e0 con 0x7fbf5c1318f0 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c09e000 con 0x7fbf5c09ea50 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c11bba0 con 0x7fbf5c136c20 2024-09-15T03:29:35.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf60edd640 10 monclient(hunting): authenticate will time out at 772.785864s 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 0x7fbf5c099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf5ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c136c20 0x7fbf5c099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 0x7fbf5c099920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37098/0 (socket says 172.21.15.153:37098) 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 1 -- 172.21.15.153:0/2178361376 learned_addr learned my addr 172.21.15.153:0/2178361376 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 10 monclient(hunting): get_auth_request con 0x7fbf5c1318f0 auth_method 0 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.145+0000 7fbf59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): get_auth_request con 0x7fbf5c136c20 auth_method 0 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5a59c640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5a59c640 10 monclient(hunting): get_auth_request con 0x7fbf5c09ea50 auth_method 0 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 10 monclient(hunting): handle_auth_done global_id 34116 payload 824 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c136c20 msgr2=0x7fbf5c099e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c136c20 0x7fbf5c099e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf5c09ea50 msgr2=0x7fbf5c130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 1 monclient: found mon.noname-a 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf59d9b640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf44009390 con 0x7fbf5c1318f0 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5a59c640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf5c09ea50 0x7fbf5c130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.146+0000 7fbf5ad9d640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c136c20 0x7fbf5c099e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf59d9b640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 msgr2=0x7fbf5c099920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf59d9b640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 0x7fbf5c099920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009850 tx=0x7fbf44032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44031940 con 0x7fbf5c1318f0 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: dump: 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.164 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 5 monclient: authenticate success, global_id 34116 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf44031aa0 con 0x7fbf5c1318f0 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 10 monclient: _renew_subs 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44031e40 con 0x7fbf5c1318f0 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c09bf10 con 0x7fbf5c1318f0 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf527fc640 10 monclient: dump: 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.166 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf58d99640 4 set_mon_vals no callback set 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 10 monclient: _renew_subs 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.148+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c141e20 con 0x7fbf5c1318f0 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf60edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf60edd640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 0x7fbf5c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf60edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c065e00 con 0x7fbf5c130ec0 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbf44039070 con 0x7fbf5c1318f0 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 0x7fbf5c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient: get_auth_request con 0x7fbf5c130ec0 auth_method 0 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf527fc640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf3c038790 0x7fbf3c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fbf4406b860 con 0x7fbf5c1318f0 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2024-09-15T03:29:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.149+0000 7fbf5a59c640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c04bf60 con 0x7fbf5c130ec0 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.150+0000 7fbf5ad9d640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf3c038790 0x7fbf3c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.150+0000 7fbf5ad9d640 10 monclient: get_auth_request con 0x7fbf3c038790 auth_method 0 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf5a59c640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 msgr2=0x7fbf5c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf5a59c640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 0x7fbf5c0298c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00eac0 tx=0x7fbf4c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c00cbe0 con 0x7fbf5c130ec0 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 10 monclient: dump: 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbf4c00cd40 con 0x7fbf5c130ec0 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 msgr2=0x7fbf5c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 0x7fbf5c0298c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00eac0 tx=0x7fbf4c009040 comp rx=0 tx=0).stop 2024-09-15T03:29:35.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 msgr2=0x7fbf5c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:35.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf527fc640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c130ec0 0x7fbf5c0298c0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00eac0 tx=0x7fbf4c009040 comp rx=0 tx=0).stop 2024-09-15T03:29:35.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.152+0000 7fbf5ad9d640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf3c038790 0x7fbf3c03ac40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf540056e0 tx=0x7fbf54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf60edd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:35.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf60edd640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 0x7fbf5c02a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf60edd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c131290 con 0x7fbf5c029e00 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 0x7fbf5c02a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient: get_auth_request con 0x7fbf5c029e00 auth_method 0 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.158+0000 7fbf5a59c640 1 -- 172.21.15.153:0/2178361376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf5c135eb0 con 0x7fbf5c029e00 2024-09-15T03:29:35.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.160+0000 7fbf5a59c640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 msgr2=0x7fbf5c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.160+0000 7fbf5a59c640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 0x7fbf5c02a200 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00eac0 tx=0x7fbf4c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c022430 con 0x7fbf5c029e00 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 10 monclient: dump: 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fbf4c015430 con 0x7fbf5c029e00 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 msgr2=0x7fbf5c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 0x7fbf5c02a200 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00eac0 tx=0x7fbf4c01a000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 msgr2=0x7fbf5c02a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf527fc640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf5c029e00 0x7fbf5c02a200 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00eac0 tx=0x7fbf4c01a000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:35.175 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 14, 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.176 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:35.179 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.180 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.181 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:35.182 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.183 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.184 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.186 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.187 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.188 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:35.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.189 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.189 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:35.189 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:35.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf3c038790 msgr2=0x7fbf3c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbf3c038790 0x7fbf3c03ac40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf540056e0 tx=0x7fbf54009040 comp rx=0 tx=0).stop 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 10 monclient: shutdown 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 msgr2=0x7fbf5c099920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 --2- 172.21.15.153:0/2178361376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1318f0 0x7fbf5c099920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009850 tx=0x7fbf44032000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf5ad9d640 1 -- 172.21.15.153:0/2178361376 reap_dead start 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf5ad9d640 1 -- 172.21.15.153:0/2178361376 reap_dead start 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 shutdown_connections 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 >> 172.21.15.153:0/2178361376 conn(0x7fbf5c067110 msgr2=0x7fbf5c0a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 shutdown_connections 2024-09-15T03:29:35.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.161+0000 7fbf60edd640 1 -- 172.21.15.153:0/2178361376 wait complete. 2024-09-15T03:29:35.191 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:29:35.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:29:35.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:29:35.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.284+0000 7f399a39d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:35.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.284+0000 7f399a39d640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.284+0000 7f399a39d640 10 monclient: monmap: 2024-09-15T03:29:35.286 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:35.286303+0000 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:35.286303+0000 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:35.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 Processor -- start 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 -- start start 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient: init 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39940974f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 0x7f399409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): picked mon.noname-a con 0x7f399409a8a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 0x7f3994135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): picked mon.noname-b con 0x7f3994133070 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399412b210 con 0x7f39940974f0 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39940978c0 con 0x7f399409a8a0 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399409b220 con 0x7f3994133070 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399a39d640 10 monclient(hunting): authenticate will time out at 772.925831s 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399893a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 0x7f3994135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f39937fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 0x7f399409ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399893a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 0x7f3994135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51994/0 (socket says 172.21.15.153:51994) 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f39937fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 0x7f399409ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37106/0 (socket says 172.21.15.153:37106) 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.285+0000 7f399893a640 1 -- 172.21.15.153:0/2425619548 learned_addr learned my addr 172.21.15.153:0/2425619548 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): get_auth_request con 0x7f3994133070 auth_method 0 2024-09-15T03:29:35.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): get_auth_request con 0x7f399409a8a0 auth_method 0 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient(hunting): handle_auth_done global_id 4424 payload 274 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 1 -- 172.21.15.153:0/2425619548 >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 msgr2=0x7f399409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 1 --2- 172.21.15.153:0/2425619548 >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 0x7f399409ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 1 -- 172.21.15.153:0/2425619548 >> v1:172.21.15.181:6789/0 conn(0x7f39940974f0 legacy=0x7f399412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 1 monclient: found mon.noname-b 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f399893a640 1 -- 172.21.15.153:0/2425619548 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399409ca20 con 0x7f3994133070 2024-09-15T03:29:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.286+0000 7f39937fe640 1 --2- 172.21.15.153:0/2425619548 >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 0x7f399409ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399893a640 1 -- 172.21.15.153:0/2425619548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 msgr2=0x7f3994135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399893a640 1 --2- 172.21.15.153:0/2425619548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 0x7f3994135450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f398c00c0d0 tx=0x7f398c00a950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c94c5c5a4374545 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 1 -- 172.21.15.153:0/2425619548 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c008510 con 0x7f3994133070 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: dump: 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.304 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 1 -- 172.21.15.153:0/2425619548 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c02e050 con 0x7f3994133070 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 5 monclient: authenticate success, global_id 4424 2024-09-15T03:29:35.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 1 -- 172.21.15.153:0/2425619548 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c0088a0 con 0x7f3994133070 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 4 set_mon_vals no callback set 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f3992ffd640 10 monclient: dump: 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 10 monclient: shutdown 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 1 -- 172.21.15.153:0/2425619548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 msgr2=0x7f3994135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 1 --2- 172.21.15.153:0/2425619548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 0x7f3994135450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f398c00c0d0 tx=0x7f398c00a950 comp rx=0 tx=0).stop 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 1 -- 172.21.15.153:0/2425619548 shutdown_connections 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 1 --2- 172.21.15.153:0/2425619548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3994133070 0x7f3994135450 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 1 --2- 172.21.15.153:0/2425619548 >> v2:172.21.15.153:3300/0 conn(0x7f399409a8a0 0x7f399409ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.289+0000 7f399a39d640 1 -- 172.21.15.153:0/2425619548 >> 172.21.15.153:0/2425619548 conn(0x7f39940672b0 msgr2=0x7f39940676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 1 -- 172.21.15.153:0/2425619548 shutdown_connections 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 1 -- 172.21.15.153:0/2425619548 wait complete. 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 10 monclient: monmap: 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.308 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:35.309 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:35.309 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:35.309 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.309 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.309 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:35.309 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:35.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:35.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 1 Processor -- start 2024-09-15T03:29:35.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 1 -- start start 2024-09-15T03:29:35.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.290+0000 7f399a39d640 10 monclient: init 2024-09-15T03:29:35.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:35.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940974f0 0x7f399406f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): picked mon.noname-b con 0x7f39940974f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:35.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 0x7f399412b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): picked mon.noname-c con 0x7f399409a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994133070 0x7f399412be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): picked mon.noname-a con 0x7f3994133070 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399412ea70 con 0x7f3994133070 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3994098300 con 0x7f39940974f0 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399409e570 con 0x7f399409a8a0 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399a39d640 10 monclient(hunting): authenticate will time out at 772.931385s 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994133070 0x7f399412be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 0x7f399412b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994133070 0x7f399412be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37108/0 (socket says 172.21.15.153:37108) 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): get_auth_request con 0x7f399409a8a0 auth_method 0 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 1 -- 172.21.15.153:0/3388514931 learned_addr learned my addr 172.21.15.153:0/3388514931 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): get_auth_request con 0x7f3994133070 auth_method 0 2024-09-15T03:29:35.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940974f0 0x7f399406f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): get_auth_request con 0x7f39940974f0 auth_method 0 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940974f0 msgr2=0x7f399406f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940974f0 0x7f399406f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994133070 msgr2=0x7f399412be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994133070 0x7f399412be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 monclient: found mon.noname-c 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f39937fe640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3988009390 con 0x7f399409a8a0 2024-09-15T03:29:35.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.291+0000 7f399893a640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994133070 0x7f399412be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f39937fe640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 msgr2=0x7f399412b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f39937fe640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 0x7f399412b910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f398800ea80 tx=0x7f39880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988009670 con 0x7f399409a8a0 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: dump: 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398800cc20 con 0x7f399409a8a0 2024-09-15T03:29:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 5 monclient: authenticate success, global_id 4427 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 10 monclient: _renew_subs 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c00abf0 con 0x7f399409a8a0 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988013470 con 0x7f399409a8a0 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f397bfff640 10 monclient: dump: 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 10 monclient: _renew_subs 2024-09-15T03:29:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399412e790 con 0x7f399409a8a0 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f39927fc640 4 set_mon_vals no callback set 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.294+0000 7f399a39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f399a39d640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 0x7f399412c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3994065f20 con 0x7f399412d8a0 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f39937fe640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 0x7f399412b910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f398800ea80 tx=0x7f39880090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f398804bac0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3988013860 con 0x7f399409a8a0 2024-09-15T03:29:35.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 0x7f399412c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient: get_auth_request con 0x7f399412d8a0 auth_method 0 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f397bfff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3980038740 0x7f398003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f399893a640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3980038740 0x7f398003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f399893a640 10 monclient: get_auth_request con 0x7f3980038740 auth_method 0 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.295+0000 7f3993fff640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f399404bf60 con 0x7f399412d8a0 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.297+0000 7f3993fff640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 msgr2=0x7f399412c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.297+0000 7f3993fff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 0x7f399412c390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f398400f780 tx=0x7f3984007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.297+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398400d8b0 con 0x7f399412d8a0 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.297+0000 7f397bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.297+0000 7f397bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.297+0000 7f397bfff640 10 monclient: dump: 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f399893a640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3980038740 0x7f398003abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f398c00a990 tx=0x7f398c037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f398400da10 con 0x7f399412d8a0 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 msgr2=0x7f399412c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 0x7f399412c390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f398400f780 tx=0x7f3984007e10 comp rx=0 tx=0).stop 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 msgr2=0x7f399412c390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:35.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.298+0000 7f397bfff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f399412d8a0 0x7f399412c390 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f398400f780 tx=0x7f3984007e10 comp rx=0 tx=0).stop 2024-09-15T03:29:35.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.303+0000 7f399a39d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.303+0000 7f399a39d640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 0x7f3994069aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.303+0000 7f399a39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.303+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399412a2d0 con 0x7f39940696a0 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 0x7f3994069aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient: get_auth_request con 0x7f39940696a0 auth_method 0 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.304+0000 7f3993fff640 1 -- 172.21.15.153:0/3388514931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f39940638d0 con 0x7f39940696a0 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.306+0000 7f3993fff640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 msgr2=0x7f3994069aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.306+0000 7f3993fff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 0x7f3994069aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f398400f780 tx=0x7f3984019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984014e70 con 0x7f39940696a0 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:35.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 10 monclient: dump: 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f3984007030 con 0x7f39940696a0 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:35.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 msgr2=0x7f3994069aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 0x7f3994069aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f398400f780 tx=0x7f3984019000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 msgr2=0x7f3994069aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:35.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39940696a0 0x7f3994069aa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.330 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 14, 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.334 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.334 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.334 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.334 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:35.335 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:35.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:35.339 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.340 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:35.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.342 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:35.344 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:35.345 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:35.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:35.345 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:35.345 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:35.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3980038740 msgr2=0x7f398003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3980038740 0x7f398003abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f398c00a990 tx=0x7f398c037000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 10 monclient: shutdown 2024-09-15T03:29:35.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 msgr2=0x7f399412b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 --2- 172.21.15.153:0/3388514931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f399409a8a0 0x7f399412b910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f398800ea80 tx=0x7f39880090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399893a640 1 -- 172.21.15.153:0/3388514931 reap_dead start 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f397bfff640 1 -- 172.21.15.153:0/3388514931 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (unknown 0 0 0) 0x7f398804bac0 con 0x7f399409a8a0 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399893a640 1 -- 172.21.15.153:0/3388514931 reap_dead start 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 shutdown_connections 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 >> 172.21.15.153:0/3388514931 conn(0x7f39940672b0 msgr2=0x7f3994129400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 shutdown_connections 2024-09-15T03:29:35.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.307+0000 7f399a39d640 1 -- 172.21.15.153:0/3388514931 wait complete. 2024-09-15T03:29:35.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:29:35.348 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:29:35.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.439+0000 7fe284914640 10 monclient: get_monmap_and_config 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.439+0000 7fe284914640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.439+0000 7fe284914640 10 monclient: monmap: 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:35.441085+0000 2024-09-15T03:29:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:35.441085+0000 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:35.442 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.439+0000 7fe284914640 1 Processor -- start 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.439+0000 7fe284914640 1 -- start start 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.439+0000 7fe284914640 10 monclient: init 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe280131ab0 0x7fe280131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): picked mon.noname-c con 0x7fe280131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 0x7fe28009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): picked mon.noname-b con 0x7fe28009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe28009f220 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28009faf0 con 0x7fe28009f220 2024-09-15T03:29:35.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28009fd30 con 0x7fe280131ab0 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800a1110 con 0x7fe28009e910 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe284914640 10 monclient(hunting): authenticate will time out at 773.080583s 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 0x7fe28009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 0x7fe28009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52016/0 (socket says 172.21.15.153:52016) 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3805839715 learned_addr learned my addr 172.21.15.153:0/3805839715 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): get_auth_request con 0x7fe28009e910 auth_method 0 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 1 --2- 172.21.15.153:0/3805839715 >> v2:172.21.15.181:3300/0 conn(0x7fe280131ab0 0x7fe280131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): get_auth_request con 0x7fe280131ab0 auth_method 0 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.440+0000 7fe27e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_done global_id 4430 payload 274 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3805839715 >> v2:172.21.15.181:3300/0 conn(0x7fe280131ab0 msgr2=0x7fe280131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 1 --2- 172.21.15.153:0/3805839715 >> v2:172.21.15.181:3300/0 conn(0x7fe280131ab0 0x7fe280131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3805839715 >> v1:172.21.15.181:6789/0 conn(0x7fe28009f220 legacy=0x7fe280068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:35.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 1 monclient: found mon.noname-b 2024-09-15T03:29:35.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:35.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.441+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3805839715 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800a13f0 con 0x7fe28009e910 2024-09-15T03:29:35.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3805839715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 msgr2=0x7fe28009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27dd9b640 1 --2- 172.21.15.153:0/3805839715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 0x7fe28009ece0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe274009f20 tx=0x7fe274009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63bb08eb89a2c51e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 1 -- 172.21.15.153:0/3805839715 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274007610 con 0x7fe28009e910 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: dump: 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.456 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 1 -- 172.21.15.153:0/3805839715 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27402d050 con 0x7fe28009e910 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe284914640 5 monclient: authenticate success, global_id 4430 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe284914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe284914640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe284914640 4 set_mon_vals no callback set 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 1 -- 172.21.15.153:0/3805839715 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2740079e0 con 0x7fe28009e910 2024-09-15T03:29:35.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.443+0000 7fe27d59a640 10 monclient: dump: 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 10 monclient: shutdown 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 -- 172.21.15.153:0/3805839715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 msgr2=0x7fe28009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 --2- 172.21.15.153:0/3805839715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 0x7fe28009ece0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe274009f20 tx=0x7fe274009840 comp rx=0 tx=0).stop 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 -- 172.21.15.153:0/3805839715 shutdown_connections 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 --2- 172.21.15.153:0/3805839715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28009e910 0x7fe28009ece0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 --2- 172.21.15.153:0/3805839715 >> v2:172.21.15.181:3300/0 conn(0x7fe280131ab0 0x7fe280131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 -- 172.21.15.153:0/3805839715 >> 172.21.15.153:0/3805839715 conn(0x7fe280067d30 msgr2=0x7fe2800a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 -- 172.21.15.153:0/3805839715 shutdown_connections 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.444+0000 7fe284914640 1 -- 172.21.15.153:0/3805839715 wait complete. 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient: build_initial_monmap 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient: monmap: 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:35.459 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 Processor -- start 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 -- start start 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient: init 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe28009e910 0x7fe28009ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): picked mon.noname-b con 0x7fe28009e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 0x7fe280130dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): picked mon.noname-a con 0x7fe28009f220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe280131ab0 0x7fe280131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): picked mon.noname-c con 0x7fe280131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280141ff0 con 0x7fe28009f220 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28009bde0 con 0x7fe28009e910 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28009de40 con 0x7fe280131ab0 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe284914640 10 monclient(hunting): authenticate will time out at 773.085850s 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe27dd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 0x7fe280130dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe27ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe280131ab0 0x7fe280131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe27dd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 0x7fe280130dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37114/0 (socket says 172.21.15.153:37114) 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3835386848 learned_addr learned my addr 172.21.15.153:0/3835386848 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): get_auth_request con 0x7fe28009f220 auth_method 0 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.445+0000 7fe27ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe280131ab0 0x7fe280131310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52018/0 (socket says 172.21.15.153:52018) 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): get_auth_request con 0x7fe280131ab0 auth_method 0 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe28009e910 0x7fe28009ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): get_auth_request con 0x7fe28009e910 auth_method 0 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient(hunting): handle_auth_done global_id 34125 payload 824 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe280131ab0 msgr2=0x7fe280131310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe280131ab0 0x7fe280131310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe28009e910 msgr2=0x7fe28009ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe28009e910 0x7fe28009ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 monclient: found mon.noname-a 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.446+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3835386848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274009ae0 con 0x7fe28009f220 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe27dd9b640 1 -- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 msgr2=0x7fe280130dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe27dd9b640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 0x7fe280130dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe274009f80 tx=0x7fe274036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27403b830 con 0x7fe28009f220 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: dump: 2024-09-15T03:29:35.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27403b990 con 0x7fe28009f220 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 5 monclient: authenticate success, global_id 34125 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 10 monclient: _renew_subs 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28011def0 con 0x7fe28009f220 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27403bd30 con 0x7fe28009f220 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe26e7fc640 10 monclient: dump: 2024-09-15T03:29:35.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 10 monclient: _renew_subs 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe27cd99640 4 set_mon_vals no callback set 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.448+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28011d000 con 0x7fe28009f220 2024-09-15T03:29:35.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe274038070 con 0x7fe28009f220 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe26e7fc640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe260038740 0x7fe26003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe27406c2c0 con 0x7fe28009f220 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe284914640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe284914640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe284914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28004bf60 con 0x7fe28009f220 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe27e59c640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe260038740 0x7fe26003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.449+0000 7fe27e59c640 10 monclient: get_auth_request con 0x7fe260038740 auth_method 0 2024-09-15T03:29:35.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.452+0000 7fe27e59c640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe260038740 0x7fe26003abf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe278009d20 tx=0x7fe2780090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:35.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.454+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe27403f070 con 0x7fe28009f220 2024-09-15T03:29:35.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.454+0000 7fe26e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:35.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.454+0000 7fe26e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:35.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe284914640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:29:35.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe284914640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:29:35.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe284914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:35.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe280062f40 con 0x7fe28009f220 2024-09-15T03:29:35.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe26e7fc640 1 -- 172.21.15.153:0/3835386848 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fe274031610 con 0x7fe28009f220 2024-09-15T03:29:35.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe26e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:29:35.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.671+0000 7fe26e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe260038740 msgr2=0x7fe26003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe260038740 0x7fe26003abf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe278009d20 tx=0x7fe2780090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 10 monclient: shutdown 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 msgr2=0x7fe280130dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 0x7fe280130dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe274009f80 tx=0x7fe274036000 comp rx=0 tx=0).stop 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 shutdown_connections 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe260038740 0x7fe26003abf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe280131ab0 0x7fe280131310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28009f220 0x7fe280130dd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 --2- 172.21.15.153:0/3835386848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe28009e910 0x7fe28009ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:35.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.673+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 >> 172.21.15.153:0/3835386848 conn(0x7fe280067d30 msgr2=0x7fe280134970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:35.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 shutdown_connections 2024-09-15T03:29:35.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fe284914640 1 -- 172.21.15.153:0/3835386848 wait complete. 2024-09-15T03:29:35.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:35.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=3183532445,config=3140514009,health=861809967,kv=3235807718,logm=3946845962}) 2024-09-15T03:29:35.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=3183532445,config=3140514009,health=861809967,kv=3235807718,logm=1701539498}) 2024-09-15T03:29:35.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:35.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=3183532445,config=3140514009,health=861809967,kv=3235807718,logm=3946845962}) 2024-09-15T03:29:35.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:35.674+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=1,health=12,kv=7,logm=70} crc {auth=3183532445,config=1228700967,health=2437654043,kv=1577269308,logm=3297769874}) 2024-09-15T03:29:35.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:29:36.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.154+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.154+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071191405}) 2024-09-15T03:29:36.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.154+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432973478}) 2024-09-15T03:29:36.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.154+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.154+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071191405}) 2024-09-15T03:29:36.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.154+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052259532}) 2024-09-15T03:29:36.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.166+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.166+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868131335}) 2024-09-15T03:29:36.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.166+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826489133}) 2024-09-15T03:29:36.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.166+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.166+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868131335}) 2024-09-15T03:29:36.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.166+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044176937}) 2024-09-15T03:29:36.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=43,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=309400510,mdsmap=1531151339,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4223748634,mon_config_key=3380917293,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=4158107560}) 2024-09-15T03:29:36.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=260656005,mdsmap=1531151339,mgr=2815566031,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=540857014,mon_config_key=3617648212,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=1388880852}) 2024-09-15T03:29:36.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=43,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=309400510,mdsmap=1531151339,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4223748634,mon_config_key=3380917293,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=4158107560}) 2024-09-15T03:29:36.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=18} crc {logm=1516231516,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3256762337,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=2882789221}) 2024-09-15T03:29:36.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=1123437409}) 2024-09-15T03:29:36.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=1701622636}) 2024-09-15T03:29:36.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:29:36.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=1123437409}) 2024-09-15T03:29:36.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:29:36.496+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=401917878}) 2024-09-15T03:29:36.687 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:29:36.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.784+0000 7f8bb0a17640 10 monclient: get_monmap_and_config 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.784+0000 7f8bb0a17640 10 monclient: build_initial_monmap 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.784+0000 7f8bb0a17640 10 monclient: monmap: 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:36.786566+0000 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:36.786566+0000 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:36.787 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:36.788 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.785+0000 7f8bb0a17640 1 Processor -- start 2024-09-15T03:29:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.785+0000 7f8bb0a17640 1 -- start start 2024-09-15T03:29:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.785+0000 7f8bb0a17640 10 monclient: init 2024-09-15T03:29:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8bac09c860 0x7f8bac09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): picked mon.noname-c con 0x7f8bac09c860 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bac09aba0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 0x7f8bac09df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): picked mon.noname-b con 0x7f8bac09afa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac06a320 con 0x7f8bac09aba0 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac09a9d0 con 0x7f8bac09c860 2024-09-15T03:29:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac068a20 con 0x7f8bac09afa0 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8bb0a17640 10 monclient(hunting): authenticate will time out at 774.426176s 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 0x7f8bac09df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 0x7f8bac09df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52028/0 (socket says 172.21.15.153:52028) 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 -- 172.21.15.153:0/714921933 learned_addr learned my addr 172.21.15.153:0/714921933 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): get_auth_request con 0x7f8bac09afa0 auth_method 0 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 1 --2- 172.21.15.153:0/714921933 >> v2:172.21.15.181:3300/0 conn(0x7f8bac09c860 0x7f8bac09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): get_auth_request con 0x7f8bac09c860 auth_method 0 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient(hunting): handle_auth_done global_id 4436 payload 274 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 -- 172.21.15.153:0/714921933 >> v2:172.21.15.181:3300/0 conn(0x7f8bac09c860 msgr2=0x7f8bac09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 --2- 172.21.15.153:0/714921933 >> v2:172.21.15.181:3300/0 conn(0x7f8bac09c860 0x7f8bac09cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 -- 172.21.15.153:0/714921933 >> v1:172.21.15.181:6789/0 conn(0x7f8bac09aba0 legacy=0x7f8bac09d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 monclient: found mon.noname-b 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.786+0000 7f8baad9d640 1 -- 172.21.15.153:0/714921933 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac068e60 con 0x7f8bac09afa0 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8baad9d640 1 -- 172.21.15.153:0/714921933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 msgr2=0x7f8bac09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8baad9d640 1 --2- 172.21.15.153:0/714921933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 0x7f8bac09df40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009f20 tx=0x7f8ba4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f55809eff88c7d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 1 -- 172.21.15.153:0/714921933 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4007610 con 0x7f8bac09afa0 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: dump: 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: _finish_auth 0 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 1 -- 172.21.15.153:0/714921933 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba402d050 con 0x7f8bac09afa0 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8bb0a17640 5 monclient: authenticate success, global_id 4436 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8bb0a17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8bb0a17640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 1 -- 172.21.15.153:0/714921933 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40079a0 con 0x7f8bac09afa0 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8bb0a17640 4 set_mon_vals no callback set 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.789+0000 7f8ba959a640 10 monclient: dump: 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 10 monclient: shutdown 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 -- 172.21.15.153:0/714921933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 msgr2=0x7f8bac09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/714921933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 0x7f8bac09df40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009f20 tx=0x7f8ba4009840 comp rx=0 tx=0).stop 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 -- 172.21.15.153:0/714921933 shutdown_connections 2024-09-15T03:29:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/714921933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09afa0 0x7f8bac09df40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/714921933 >> v2:172.21.15.181:3300/0 conn(0x7f8bac09c860 0x7f8bac09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 -- 172.21.15.153:0/714921933 >> 172.21.15.153:0/714921933 conn(0x7f8bac0a2e40 msgr2=0x7f8bac0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 -- 172.21.15.153:0/714921933 shutdown_connections 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 -- 172.21.15.153:0/714921933 wait complete. 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 10 monclient: build_initial_monmap 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 10 monclient: monmap: 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.790+0000 7f8bb0a17640 1 Processor -- start 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 -- start start 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient: init 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac09aba0 0x7f8bac133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): picked mon.noname-a con 0x7f8bac09aba0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bac09afa0 0x7f8bac11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): picked mon.noname-b con 0x7f8bac09afa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 0x7f8bac11ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): picked mon.noname-c con 0x7f8bac09c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac099cc0 con 0x7f8bac09aba0 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac134aa0 con 0x7f8bac09afa0 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac11fb30 con 0x7f8bac09c860 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8bb0a17640 10 monclient(hunting): authenticate will time out at 774.431487s 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 0x7f8bac11ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac09aba0 0x7f8bac133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 0x7f8bac11ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52038/0 (socket says 172.21.15.153:52038) 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 -- 172.21.15.153:0/3711524037 learned_addr learned my addr 172.21.15.153:0/3711524037 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac09aba0 0x7f8bac133970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37126/0 (socket says 172.21.15.153:37126) 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): get_auth_request con 0x7f8bac09c860 auth_method 0 2024-09-15T03:29:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): get_auth_request con 0x7f8bac09aba0 auth_method 0 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bac09afa0 0x7f8bac11d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): get_auth_request con 0x7f8bac09afa0 auth_method 0 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8ba9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 -- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bac09afa0 msgr2=0x7f8bac11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bac09afa0 0x7f8bac11d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 -- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac09aba0 msgr2=0x7f8bac133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac09aba0 0x7f8bac133970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:29:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 monclient: found mon.noname-c 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.791+0000 7f8baad9d640 1 -- 172.21.15.153:0/3711524037 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba4009ae0 con 0x7f8bac09c860 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8baad9d640 1 -- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 msgr2=0x7f8bac11ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8baad9d640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 0x7f8bac11ddf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ba402ac50 tx=0x7f8ba4036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4007170 con 0x7f8bac09c860 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: dump: 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 5 monclient: authenticate success, global_id 4439 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4032cc0 con 0x7f8bac09c860 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 10 monclient: _renew_subs 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac11f470 con 0x7f8bac09c860 2024-09-15T03:29:36.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba403b680 con 0x7f8bac09c860 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8b9a7fc640 10 monclient: dump: 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:36.813 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8ba8d99640 4 set_mon_vals no callback set 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 10 monclient: _renew_subs 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.794+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac11ef60 con 0x7f8bac09c860 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8ba4038050 con 0x7f8bac09c860 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8bb0a17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8bb0a17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8bb0a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac120ba0 con 0x7f8bac09c860 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8b9a7fc640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8b8c038740 0x7f8b8c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f8ba406c530 con 0x7f8bac09c860 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8baa59c640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8b8c038740 0x7f8b8c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.795+0000 7f8baa59c640 10 monclient: get_auth_request con 0x7f8b8c038740 auth_method 0 2024-09-15T03:29:36.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.798+0000 7f8baa59c640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8b8c038740 0x7f8b8c03abf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009d20 tx=0x7f8b9c0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:36.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.799+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8ba4032e30 con 0x7f8bac09c860 2024-09-15T03:29:36.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.799+0000 7f8b9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:36.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:36.799+0000 7f8b9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:37.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8bb0a17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:37.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8bb0a17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:37.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8bb0a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8bac04bf60 con 0x7f8bac09c860 2024-09-15T03:29:37.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8b9a7fc640 1 -- 172.21.15.153:0/3711524037 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f8ba403f070 con 0x7f8bac09c860 2024-09-15T03:29:37.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8b9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:37.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.014+0000 7f8b9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:37.017 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8b8c038740 msgr2=0x7f8b8c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8b8c038740 0x7f8b8c03abf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009d20 tx=0x7f8b9c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 10 monclient: shutdown 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 msgr2=0x7f8bac11ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 0x7f8bac11ddf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ba402ac50 tx=0x7f8ba4036000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 shutdown_connections 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8b8c038740 0x7f8b8c03abf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8bac09c860 0x7f8bac11ddf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bac09afa0 0x7f8bac11d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 --2- 172.21.15.153:0/3711524037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac09aba0 0x7f8bac133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 >> 172.21.15.153:0/3711524037 conn(0x7f8bac0a2e40 msgr2=0x7f8bac0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 shutdown_connections 2024-09-15T03:29:37.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.016+0000 7f8bb0a17640 1 -- 172.21.15.153:0/3711524037 wait complete. 2024-09-15T03:29:37.029 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:29:37.029 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:29:37.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:29:37.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:29:37.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.122+0000 7fa0ff62b640 10 monclient: get_monmap_and_config 2024-09-15T03:29:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.122+0000 7fa0ff62b640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.122+0000 7fa0ff62b640 10 monclient: monmap: 2024-09-15T03:29:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:37.123919+0000 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:37.123919+0000 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:37.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.122+0000 7fa0ff62b640 1 Processor -- start 2024-09-15T03:29:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.122+0000 7fa0ff62b640 1 -- start start 2024-09-15T03:29:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.122+0000 7fa0ff62b640 10 monclient: init 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f806eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0f8068a20 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0f8136ab0 0x7fa0f8138e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f8136ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f806a0e0 con 0x7fa0f8068a20 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f806a7a0 con 0x7fa0f8136ab0 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8069e00 con 0x7fa0f806eb80 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0ff62b640 10 monclient(hunting): authenticate will time out at 774.763479s 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0f8136ab0 0x7fa0f8138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f806ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52040/0 (socket says 172.21.15.153:52040) 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 1 -- 172.21.15.153:0/3346188506 learned_addr learned my addr 172.21.15.153:0/3346188506 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): get_auth_request con 0x7fa0f8136ab0 auth_method 0 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fcbc6640 1 --1- 172.21.15.153:0/3346188506 >> v1:172.21.15.153:6789/0 conn(0x7fa0f8068a20 0x7fa0f809c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36366/0 (socket says 172.21.15.153:36366) 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): get_auth_request con 0x7fa0f806eb80 auth_method 0 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_done global_id 34131 payload 274 2024-09-15T03:29:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 1 -- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 msgr2=0x7fa0f806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 1 --2- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f806ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 1 -- 172.21.15.153:0/3346188506 >> v1:172.21.15.153:6789/0 conn(0x7fa0f8068a20 legacy=0x7fa0f809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 1 monclient: found mon.noname-a 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fdbc8640 1 -- 172.21.15.153:0/3346188506 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f806f490 con 0x7fa0f8136ab0 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.123+0000 7fa0fd3c7640 1 --2- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0fdbc8640 1 -- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 msgr2=0x7fa0f8138e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0fdbc8640 1 --2- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8138e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009e60 tx=0x7fa0f0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ef6821197ebdc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 1 -- 172.21.15.153:0/3346188506 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0007760 con 0x7fa0f8136ab0 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: dump: 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0ff62b640 5 monclient: authenticate success, global_id 34131 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 1 -- 172.21.15.153:0/3346188506 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f002d050 con 0x7fa0f8136ab0 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0ff62b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0ff62b640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0ff62b640 4 set_mon_vals no callback set 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 1 -- 172.21.15.153:0/3346188506 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0007af0 con 0x7fa0f8136ab0 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.126+0000 7fa0effff640 10 monclient: dump: 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 10 monclient: shutdown 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 -- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 msgr2=0x7fa0f8138e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8138e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009e60 tx=0x7fa0f0009780 comp rx=0 tx=0).stop 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 -- 172.21.15.153:0/3346188506 shutdown_connections 2024-09-15T03:29:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8138e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/3346188506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 -- 172.21.15.153:0/3346188506 >> 172.21.15.153:0/3346188506 conn(0x7fa0f8067cd0 msgr2=0x7fa0f80a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 -- 172.21.15.153:0/3346188506 shutdown_connections 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.127+0000 7fa0ff62b640 1 -- 172.21.15.153:0/3346188506 wait complete. 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 10 monclient: monmap: 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 1 Processor -- start 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 1 -- start start 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 10 monclient: init 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.128+0000 7fa0ff62b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0f8068a20 0x7fa0f8097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f8068a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f80984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f806eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8098a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f8136ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f811fae0 con 0x7fa0f8136ab0 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f809d050 con 0x7fa0f8068a20 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80a3310 con 0x7fa0f806eb80 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0ff62b640 10 monclient(hunting): authenticate will time out at 774.769083s 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f80984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8098a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f80984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52042/0 (socket says 172.21.15.153:52042) 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): get_auth_request con 0x7fa0f8136ab0 auth_method 0 2024-09-15T03:29:37.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 -- 172.21.15.153:0/2531938168 learned_addr learned my addr 172.21.15.153:0/2531938168 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): get_auth_request con 0x7fa0f806eb80 auth_method 0 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0f8068a20 0x7fa0f8097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): get_auth_request con 0x7fa0f8068a20 auth_method 0 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0f8068a20 msgr2=0x7fa0f8097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0f8068a20 0x7fa0f8097f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 msgr2=0x7fa0f8098a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8098a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 monclient: found mon.noname-c 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fcbc6640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4009450 con 0x7fa0f806eb80 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.129+0000 7fa0fdbc8640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f8136ab0 0x7fa0f8098a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0fcbc6640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 msgr2=0x7fa0f80984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0fcbc6640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f80984d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800b540 tx=0x7fa0e80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8011840 con 0x7fa0f806eb80 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: dump: 2024-09-15T03:29:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e80119a0 con 0x7fa0f806eb80 2024-09-15T03:29:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 5 monclient: authenticate success, global_id 4445 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 10 monclient: _renew_subs 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f0009a20 con 0x7fa0f806eb80 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8011d60 con 0x7fa0f806eb80 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ed7fa640 10 monclient: dump: 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.155 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ef7fe640 4 set_mon_vals no callback set 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 10 monclient: _renew_subs 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.131+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f811f490 con 0x7fa0f806eb80 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ff62b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa0e8010590 con 0x7fa0f806eb80 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 0x7fa0f8098f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8065e00 con 0x7fa0f811e5a0 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 0x7fa0f8098f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient: get_auth_request con 0x7fa0f811e5a0 auth_method 0 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ed7fa640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0dc038740 0x7fa0dc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.132+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fa0e804b920 con 0x7fa0f806eb80 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.133+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_done global_id 34137 payload 824 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.133+0000 7fa0fd3c7640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0f811da30 con 0x7fa0f811e5a0 2024-09-15T03:29:37.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.133+0000 7fa0fdbc8640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0dc038740 0x7fa0dc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.133+0000 7fa0fdbc8640 10 monclient: get_auth_request con 0x7fa0dc038740 auth_method 0 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0fd3c7640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 msgr2=0x7fa0f8098f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0fd3c7640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 0x7fa0f8098f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4011bf0 tx=0x7fa0f40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400fe90 con 0x7fa0f811e5a0 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 10 monclient: dump: 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.158 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0fdbc8640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0dc038740 0x7fa0dc03abf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0031000 tx=0x7fa0f0037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa0f4014650 con 0x7fa0f811e5a0 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 msgr2=0x7fa0f8098f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 0x7fa0f8098f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4011bf0 tx=0x7fa0f40091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 msgr2=0x7fa0f8098f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:37.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.135+0000 7fa0ed7fa640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f811e5a0 0x7fa0f8098f50 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4011bf0 tx=0x7fa0f40091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0ff62b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:37.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 0x7fa0f806bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0ff62b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f811fec0 con 0x7fa0f806b840 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 0x7fa0f806bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient: get_auth_request con 0x7fa0f806b840 auth_method 0 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 10 monclient(hunting): handle_auth_done global_id 34140 payload 824 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.141+0000 7fa0fd3c7640 1 -- 172.21.15.153:0/2531938168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa0f804bf60 con 0x7fa0f806b840 2024-09-15T03:29:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.143+0000 7fa0fd3c7640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 msgr2=0x7fa0f806bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.143+0000 7fa0fd3c7640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 0x7fa0f806bc40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4011bf0 tx=0x7fa0f401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4025040 con 0x7fa0f806b840 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 10 monclient: dump: 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7fa0f401de00 con 0x7fa0f806b840 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 msgr2=0x7fa0f806bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 0x7fa0f806bc40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4011bf0 tx=0x7fa0f401d000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 msgr2=0x7fa0f806bc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ed7fa640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0f806b840 0x7fa0f806bc40 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4011bf0 tx=0x7fa0f401d000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.164 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 14, 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:37.165 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:37.166 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:37.167 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:37.168 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.169 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:37.170 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.171 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.172 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:37.173 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.174 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.175 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:37.176 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0dc038740 msgr2=0x7fa0dc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0dc038740 0x7fa0dc03abf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0031000 tx=0x7fa0f0037000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 10 monclient: shutdown 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 msgr2=0x7fa0f80984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 --2- 172.21.15.153:0/2531938168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0f806eb80 0x7fa0f80984d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800b540 tx=0x7fa0e80090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0fdbc8640 1 -- 172.21.15.153:0/2531938168 reap_dead start 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0fdbc8640 1 -- 172.21.15.153:0/2531938168 reap_dead start 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 shutdown_connections 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 >> 172.21.15.153:0/2531938168 conn(0x7fa0f8067cd0 msgr2=0x7fa0f809b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 shutdown_connections 2024-09-15T03:29:37.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.144+0000 7fa0ff62b640 1 -- 172.21.15.153:0/2531938168 wait complete. 2024-09-15T03:29:37.179 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:29:37.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:29:37.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:29:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.271+0000 7f1dea15d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.271+0000 7f1dea15d640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.271+0000 7f1dea15d640 10 monclient: monmap: 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:37.273395+0000 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:37.273395+0000 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:37.274 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 1 Processor -- start 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 1 -- start start 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient: init 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1de409c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1de409e840 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1de4137cc0 0x7f1de413a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient(hunting): picked mon.noname-a con 0x7f1de4137cc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de409ec40 con 0x7f1de409e840 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de409b2b0 con 0x7f1de409c860 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de406a320 con 0x7f1de4137cc0 2024-09-15T03:29:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.272+0000 7f1dea15d640 10 monclient(hunting): authenticate will time out at 774.912972s 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1de4137cc0 0x7f1de413a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1de409e840 0x7f1de409aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36372/0 (socket says 172.21.15.153:36372) 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de2ffd640 1 -- 172.21.15.153:0/4268658842 learned_addr learned my addr 172.21.15.153:0/4268658842 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): get_auth_request con 0x7f1de4137cc0 auth_method 0 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient(hunting): handle_auth_done global_id 34143 payload 274 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 1 -- 172.21.15.153:0/4268658842 >> v1:172.21.15.181:6789/0 conn(0x7f1de409c860 legacy=0x7f1de409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 1 -- 172.21.15.153:0/4268658842 >> v1:172.21.15.153:6789/0 conn(0x7f1de409e840 legacy=0x7f1de409aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 1 monclient: found mon.noname-a 2024-09-15T03:29:37.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.273+0000 7f1de3fff640 1 -- 172.21.15.153:0/4268658842 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4068a20 con 0x7f1de4137cc0 2024-09-15T03:29:37.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de3fff640 1 -- 172.21.15.153:0/4268658842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 msgr2=0x7f1de413a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de3fff640 1 --2- 172.21.15.153:0/4268658842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de413a0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009e60 tx=0x7f1ddc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff3a955ec175de1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 1 -- 172.21.15.153:0/4268658842 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc007630 con 0x7f1de4137cc0 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: dump: 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 1 -- 172.21.15.153:0/4268658842 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc02d050 con 0x7f1de4137cc0 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 5 monclient: authenticate success, global_id 34143 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:37.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 1 -- 172.21.15.153:0/4268658842 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0079c0 con 0x7f1de4137cc0 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 4 set_mon_vals no callback set 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1de27fc640 10 monclient: dump: 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 10 monclient: shutdown 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 -- 172.21.15.153:0/4268658842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 msgr2=0x7f1de413a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 --2- 172.21.15.153:0/4268658842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de413a0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009e60 tx=0x7f1ddc009780 comp rx=0 tx=0).stop 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 -- 172.21.15.153:0/4268658842 shutdown_connections 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 --2- 172.21.15.153:0/4268658842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de413a0a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 -- 172.21.15.153:0/4268658842 >> 172.21.15.153:0/4268658842 conn(0x7f1de40a2e40 msgr2=0x7f1de40a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 -- 172.21.15.153:0/4268658842 shutdown_connections 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.276+0000 7f1dea15d640 1 -- 172.21.15.153:0/4268658842 wait complete. 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.277+0000 7f1dea15d640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.277+0000 7f1dea15d640 10 monclient: monmap: 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.277+0000 7f1dea15d640 1 Processor -- start 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.277+0000 7f1dea15d640 1 -- start start 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.277+0000 7f1dea15d640 10 monclient: init 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de409c860 0x7f1de406efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): picked mon.noname-b con 0x7f1de409c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de409e840 0x7f1de406b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): picked mon.noname-c con 0x7f1de409e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de406bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): picked mon.noname-a con 0x7f1de4137cc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de40a26a0 con 0x7f1de4137cc0 2024-09-15T03:29:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de406ca50 con 0x7f1de409c860 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4099c40 con 0x7f1de409e840 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1dea15d640 10 monclient(hunting): authenticate will time out at 774.918321s 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de406bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de409e840 0x7f1de406b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de406bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37192/0 (socket says 172.21.15.153:37192) 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 1 -- 172.21.15.153:0/726770519 learned_addr learned my addr 172.21.15.153:0/726770519 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): get_auth_request con 0x7f1de4137cc0 auth_method 0 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): get_auth_request con 0x7f1de409e840 auth_method 0 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de409c860 0x7f1de406efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): get_auth_request con 0x7f1de409c860 auth_method 0 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient(hunting): handle_auth_done global_id 4448 payload 824 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de409c860 msgr2=0x7f1de406efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de409c860 0x7f1de406efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 msgr2=0x7f1de406bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de406bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 monclient: found mon.noname-c 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de2ffd640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc009a20 con 0x7f1de409e840 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.278+0000 7f1de3fff640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de4137cc0 0x7f1de406bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.280+0000 7f1de2ffd640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de409e840 msgr2=0x7f1de406b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.280+0000 7f1de2ffd640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de409e840 0x7f1de406b8b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c490 tx=0x7f1dd8007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8005cb0 con 0x7f1de409e840 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: dump: 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.297 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 5 monclient: authenticate success, global_id 4448 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd8005e10 con 0x7f1de409e840 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 10 monclient: _renew_subs 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de406ced0 con 0x7f1de409e840 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8007320 con 0x7f1de409e840 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dcb7fe640 10 monclient: dump: 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 10 monclient: _renew_subs 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1de1ffb640 4 set_mon_vals no callback set 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.281+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de406e0a0 con 0x7f1de409e840 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dea15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dea15d640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 0x7f1de406c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dea15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4065e00 con 0x7f1de4098cb0 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1dd801b3b0 con 0x7f1de409e840 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 0x7f1de406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient: get_auth_request con 0x7f1de4098cb0 auth_method 0 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dcb7fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dd0038740 0x7f1dd003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f1dd804adc0 con 0x7f1de409e840 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 10 monclient(hunting): handle_auth_done global_id 4451 payload 824 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de37fe640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1de404bf60 con 0x7f1de4098cb0 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de3fff640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dd0038740 0x7f1dd003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.282+0000 7f1de3fff640 10 monclient: get_auth_request con 0x7f1dd0038740 auth_method 0 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1de37fe640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 msgr2=0x7f1de406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1de37fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 0x7f1de406c330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400f780 tx=0x7f1dd4007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd400d8b0 con 0x7f1de4098cb0 2024-09-15T03:29:37.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 10 monclient: dump: 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1dd400da10 con 0x7f1de4098cb0 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 msgr2=0x7f1de406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 0x7f1de406c330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400f780 tx=0x7f1dd4007e10 comp rx=0 tx=0).stop 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 msgr2=0x7f1de406c330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1dcb7fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4098cb0 0x7f1de406c330 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400f780 tx=0x7f1dd4007e10 comp rx=0 tx=0).stop 2024-09-15T03:29:37.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.284+0000 7f1de3fff640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dd0038740 0x7f1dd003abf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0097c0 tx=0x7f1ddc037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1dea15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1dea15d640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 0x7f1de4138490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1dea15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de40008d0 con 0x7f1de4138090 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 0x7f1de4138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient: get_auth_request con 0x7f1de4138090 auth_method 0 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.290+0000 7f1de37fe640 1 -- 172.21.15.153:0/726770519 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1de40994e0 con 0x7f1de4138090 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1de37fe640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 msgr2=0x7f1de4138490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1de37fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 0x7f1de4138490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400f780 tx=0x7f1dd4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4007030 con 0x7f1de4138090 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 10 monclient: dump: 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f1dd4007190 con 0x7f1de4138090 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 msgr2=0x7f1de4138490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 0x7f1de4138490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400f780 tx=0x7f1dd4019000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 msgr2=0x7f1de4138490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:37.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dcb7fe640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de4138090 0x7f1de4138490 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 14, 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:37.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.309 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.310 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:37.311 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.312 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.313 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.318 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:37.320 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.322 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.323 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:37.324 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dd0038740 msgr2=0x7f1dd003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dd0038740 0x7f1dd003abf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0097c0 tx=0x7f1ddc037000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 10 monclient: shutdown 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de409e840 msgr2=0x7f1de406b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 1 --2- 172.21.15.153:0/726770519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de409e840 0x7f1de406b8b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c490 tx=0x7f1dd8007f00 comp rx=0 tx=0).stop 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1de3fff640 1 -- 172.21.15.153:0/726770519 reap_dead start 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1de3fff640 1 -- 172.21.15.153:0/726770519 reap_dead start 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 shutdown_connections 2024-09-15T03:29:37.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.293+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 >> 172.21.15.153:0/726770519 conn(0x7f1de40a2e40 msgr2=0x7f1de40a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.294+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 shutdown_connections 2024-09-15T03:29:37.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.294+0000 7f1dea15d640 1 -- 172.21.15.153:0/726770519 wait complete. 2024-09-15T03:29:37.327 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:29:37.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:29:37.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T03:29:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: git switch -c 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr:Or undo this operation with: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: git switch - 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:37.341 INFO:tasks.workunit.client.0.smithi181.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:29:37.345 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.421+0000 7f7ea2094640 10 monclient: get_monmap_and_config 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.421+0000 7f7ea2094640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.421+0000 7f7ea2094640 10 monclient: monmap: 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:37.422841+0000 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:37.422841+0000 2024-09-15T03:29:37.423 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:37.424 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.421+0000 7f7ea2094640 1 Processor -- start 2024-09-15T03:29:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 1 -- start start 2024-09-15T03:29:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient: init 2024-09-15T03:29:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T03:29:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e9c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e9c09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7e9c137c30 0x7f7e9c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient(hunting): picked mon.noname-a con 0x7f7e9c137c30 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c131cc0 con 0x7f7e9c09ea50 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c09f530 con 0x7f7e9c1318f0 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c09fbf0 con 0x7f7e9c137c30 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7ea2094640 10 monclient(hunting): authenticate will time out at 775.062538s 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7e9c137c30 0x7f7e9c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e93fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7e9c09ea50 0x7f7e9c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36374/0 (socket says 172.21.15.153:36374) 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e93fff640 1 -- 172.21.15.153:0/2136043455 learned_addr learned my addr 172.21.15.153:0/2136043455 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:29:37.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): get_auth_request con 0x7f7e9c137c30 auth_method 0 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_done global_id 34149 payload 274 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 1 -- 172.21.15.153:0/2136043455 >> v1:172.21.15.181:6789/0 conn(0x7f7e9c1318f0 legacy=0x7f7e9c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 1 -- 172.21.15.153:0/2136043455 >> v1:172.21.15.153:6789/0 conn(0x7f7e9c09ea50 legacy=0x7f7e9c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 1 monclient: found mon.noname-a 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.422+0000 7f7e9bfff640 1 -- 172.21.15.153:0/2136043455 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c09fe30 con 0x7f7e9c137c30 2024-09-15T03:29:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9bfff640 1 -- 172.21.15.153:0/2136043455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 msgr2=0x7f7e9c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9bfff640 1 --2- 172.21.15.153:0/2136043455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c13a010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009e60 tx=0x7f7e94009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b885dfb1d82d89eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 1 -- 172.21.15.153:0/2136043455 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94007630 con 0x7f7e9c137c30 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: dump: 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.435 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 1 -- 172.21.15.153:0/2136043455 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9402d050 con 0x7f7e9c137c30 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7ea2094640 5 monclient: authenticate success, global_id 34149 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7ea2094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7ea2094640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7ea2094640 4 set_mon_vals no callback set 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 1 -- 172.21.15.153:0/2136043455 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e940079c0 con 0x7f7e9c137c30 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.425+0000 7f7e9affd640 10 monclient: dump: 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 10 monclient: shutdown 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 -- 172.21.15.153:0/2136043455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 msgr2=0x7f7e9c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 --2- 172.21.15.153:0/2136043455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c13a010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009e60 tx=0x7f7e94009780 comp rx=0 tx=0).stop 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 -- 172.21.15.153:0/2136043455 shutdown_connections 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 --2- 172.21.15.153:0/2136043455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c13a010 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 -- 172.21.15.153:0/2136043455 >> 172.21.15.153:0/2136043455 conn(0x7f7e9c067110 msgr2=0x7f7e9c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 -- 172.21.15.153:0/2136043455 shutdown_connections 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 1 -- 172.21.15.153:0/2136043455 wait complete. 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.426+0000 7f7ea2094640 10 monclient: monmap: 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:37.438 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 Processor -- start 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 -- start start 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient: init 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:37.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e9c09ea50 0x7f7e9c136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): picked mon.noname-c con 0x7f7e9c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c1318f0 0x7f7e9c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): picked mon.noname-b con 0x7f7e9c1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): picked mon.noname-a con 0x7f7e9c137c30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c099c50 con 0x7f7e9c137c30 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c09bf10 con 0x7f7e9c1318f0 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c09d7c0 con 0x7f7e9c09ea50 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7ea2094640 10 monclient(hunting): authenticate will time out at 775.067861s 2024-09-15T03:29:37.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e9c09ea50 0x7f7e9c136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e9c09ea50 0x7f7e9c136cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52086/0 (socket says 172.21.15.153:52086) 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9bfff640 10 monclient(hunting): get_auth_request con 0x7f7e9c137c30 auth_method 0 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 1 -- 172.21.15.153:0/1511371349 learned_addr learned my addr 172.21.15.153:0/1511371349 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 10 monclient(hunting): get_auth_request con 0x7f7e9c09ea50 auth_method 0 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.427+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e93fff640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c1318f0 0x7f7e9c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e93fff640 10 monclient(hunting): get_auth_request con 0x7f7e9c1318f0 auth_method 0 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 10 monclient(hunting): handle_auth_done global_id 34152 payload 824 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e9c09ea50 msgr2=0x7f7e9c136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e9c09ea50 0x7f7e9c136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c1318f0 msgr2=0x7f7e9c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c1318f0 0x7f7e9c096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 1 monclient: found mon.noname-a 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9bfff640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e94009a20 con 0x7f7e9c137c30 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.428+0000 7f7e9b7fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e9c09ea50 0x7f7e9c136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e9bfff640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 msgr2=0x7f7e9c096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e9bfff640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c096fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e94031000 tx=0x7f7e94037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94031ac0 con 0x7f7e9c137c30 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: dump: 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e94040070 con 0x7f7e9c137c30 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94031e70 con 0x7f7e9c137c30 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 5 monclient: authenticate success, global_id 34152 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e937fe640 10 monclient: dump: 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 10 monclient: _renew_subs 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7e9a7fc640 4 set_mon_vals no callback set 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c098070 con 0x7f7e9c137c30 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 10 monclient: _renew_subs 2024-09-15T03:29:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.430+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c099620 con 0x7f7e9c137c30 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7e94042030 con 0x7f7e9c137c30 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7ea2094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7ea2094640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 0x7f7e9c0974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7ea2094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c065e00 con 0x7f7e9c098730 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e937fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e7c038740 0x7f7e7c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 0x7f7e9c0974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f7e9406cf30 con 0x7f7e9c137c30 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient: get_auth_request con 0x7f7e9c098730 auth_method 0 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e93fff640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e7c038740 0x7f7e7c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e93fff640 10 monclient: get_auth_request con 0x7f7e7c038740 auth_method 0 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.431+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2024-09-15T03:29:37.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.432+0000 7f7e9b7fe640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e9c04bf60 con 0x7f7e9c098730 2024-09-15T03:29:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e9b7fe640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 msgr2=0x7f7e9c0974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e9b7fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 0x7f7e9c0974e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00fbb0 tx=0x7f7e8c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e93fff640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e7c038740 0x7f7e7c03abf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e84007fd0 tx=0x7f7e84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00de70 con 0x7f7e9c098730 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 10 monclient: dump: 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7e8c014470 con 0x7f7e9c098730 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 msgr2=0x7f7e9c0974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 0x7f7e9c0974e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00fbb0 tx=0x7f7e8c007ed0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 msgr2=0x7f7e9c0974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.434+0000 7f7e937fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c098730 0x7f7e9c0974e0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00fbb0 tx=0x7f7e8c007ed0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7ea2094640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:37.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7ea2094640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 0x7f7e9c132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7ea2094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0638d0 con 0x7f7e9c132740 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 0x7f7e9c132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient: get_auth_request con 0x7f7e9c132740 auth_method 0 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.441+0000 7f7e9b7fe640 1 -- 172.21.15.153:0/1511371349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7e9c09aa50 con 0x7f7e9c132740 2024-09-15T03:29:37.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e9b7fe640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 msgr2=0x7f7e9c132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e9b7fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 0x7f7e9c132b40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c019000 tx=0x7f7e8c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c007020 con 0x7f7e9c132740 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 10 monclient: dump: 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.454 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f7e8c007180 con 0x7f7e9c132740 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 msgr2=0x7f7e9c132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 0x7f7e9c132b40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c019000 tx=0x7f7e8c01a000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 msgr2=0x7f7e9c132b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:37.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.443+0000 7f7e937fe640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e9c132740 0x7f7e9c132b40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 14, 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:29:37.456 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.457 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.458 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:37.459 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.461 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.462 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:37.465 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.466 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:37.467 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:37.468 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:37.469 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:37.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e7c038740 msgr2=0x7f7e7c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e7c038740 0x7f7e7c03abf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e84007fd0 tx=0x7f7e84009040 comp rx=0 tx=0).stop 2024-09-15T03:29:37.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 10 monclient: shutdown 2024-09-15T03:29:37.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 msgr2=0x7f7e9c096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 --2- 172.21.15.153:0/1511371349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e9c137c30 0x7f7e9c096fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e94031000 tx=0x7f7e94037000 comp rx=0 tx=0).stop 2024-09-15T03:29:37.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7e9bfff640 1 -- 172.21.15.153:0/1511371349 reap_dead start 2024-09-15T03:29:37.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7e9bfff640 1 -- 172.21.15.153:0/1511371349 reap_dead start 2024-09-15T03:29:37.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 shutdown_connections 2024-09-15T03:29:37.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 >> 172.21.15.153:0/1511371349 conn(0x7f7e9c067110 msgr2=0x7f7e9c0a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 shutdown_connections 2024-09-15T03:29:37.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.444+0000 7f7ea2094640 1 -- 172.21.15.153:0/1511371349 wait complete. 2024-09-15T03:29:37.471 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:29:37.471 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:29:37.472 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:29:37.472 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:29:37.472 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:29:37.472 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:29:37.472 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:29:37.483 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:29:37.483 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:29:37.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:37.520 INFO:tasks.workunit.client.0.smithi181.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T03:29:37.522 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T03:29:37.522 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T03:29:37.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.573+0000 7f0ff839d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.573+0000 7f0ff839d640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.573+0000 7f0ff839d640 10 monclient: monmap: 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:37.575335+0000 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:37.575335+0000 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:37.576 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:37.577 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:37.577 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:37.577 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 Processor -- start 2024-09-15T03:29:37.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 -- start start 2024-09-15T03:29:37.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient: init 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0ff0131980 0x7f0ff0131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff0131980 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 0x7f0ff009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff009fe10 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 0x7f0ff0068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff009ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff01325b0 con 0x7f0ff009fe10 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff00a11f0 con 0x7f0ff0131980 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff0132290 con 0x7f0ff009ef70 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff839d640 10 monclient(hunting): authenticate will time out at 775.214819s 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff6139640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f0ff0131980 msgr2=0x7f0ff0131d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff6139640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0ff0131980 0x7f0ff0131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:37.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff693a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 0x7f0ff0068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff5938640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 0x7f0ff009ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff5938640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 0x7f0ff009ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37216/0 (socket says 172.21.15.153:37216) 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.574+0000 7f0ff693a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 0x7f0ff0068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52100/0 (socket says 172.21.15.153:52100) 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 1 -- 172.21.15.153:0/1250487610 learned_addr learned my addr 172.21.15.153:0/1250487610 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): get_auth_request con 0x7f0ff009ef70 auth_method 0 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): get_auth_request con 0x7f0ff009fe10 auth_method 0 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient(hunting): handle_auth_done global_id 4460 payload 274 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 1 -- 172.21.15.153:0/1250487610 >> v2:172.21.15.181:3300/0 conn(0x7f0ff0131980 msgr2=0x7f0ff0131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 1 --2- 172.21.15.153:0/1250487610 >> v2:172.21.15.181:3300/0 conn(0x7f0ff0131980 0x7f0ff0131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 1 -- 172.21.15.153:0/1250487610 >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 msgr2=0x7f0ff009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 1 --2- 172.21.15.153:0/1250487610 >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 0x7f0ff009ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 1 monclient: found mon.noname-b 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff693a640 1 -- 172.21.15.153:0/1250487610 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff009f390 con 0x7f0ff009ef70 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.575+0000 7f0ff5938640 1 --2- 172.21.15.153:0/1250487610 >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 0x7f0ff009ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff693a640 1 -- 172.21.15.153:0/1250487610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 msgr2=0x7f0ff0068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff693a640 1 --2- 172.21.15.153:0/1250487610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 0x7f0ff0068ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0ff006f930 tx=0x7f0fe8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b49f05bb95eec3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 1 -- 172.21.15.153:0/1250487610 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8007760 con 0x7f0ff009ef70 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: dump: 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.587 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 1 -- 172.21.15.153:0/1250487610 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe802d050 con 0x7f0ff009ef70 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 5 monclient: authenticate success, global_id 4460 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 4 set_mon_vals no callback set 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 1 -- 172.21.15.153:0/1250487610 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8007b30 con 0x7f0ff009ef70 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff5137640 10 monclient: dump: 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.589 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 10 monclient: shutdown 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 -- 172.21.15.153:0/1250487610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 msgr2=0x7f0ff0068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 --2- 172.21.15.153:0/1250487610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 0x7f0ff0068ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0ff006f930 tx=0x7f0fe8009780 comp rx=0 tx=0).stop 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 -- 172.21.15.153:0/1250487610 shutdown_connections 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 --2- 172.21.15.153:0/1250487610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff009ef70 0x7f0ff0068ce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 --2- 172.21.15.153:0/1250487610 >> v2:172.21.15.153:3300/0 conn(0x7f0ff009fe10 0x7f0ff009ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 --2- 172.21.15.153:0/1250487610 >> v2:172.21.15.181:3300/0 conn(0x7f0ff0131980 0x7f0ff0131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 -- 172.21.15.153:0/1250487610 >> 172.21.15.153:0/1250487610 conn(0x7f0ff00a5050 msgr2=0x7f0ff00a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 -- 172.21.15.153:0/1250487610 shutdown_connections 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.578+0000 7f0ff839d640 1 -- 172.21.15.153:0/1250487610 wait complete. 2024-09-15T03:29:37.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient: build_initial_monmap 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient: monmap: 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 Processor -- start 2024-09-15T03:29:37.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 -- start start 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient: init 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ff009ef70 0x7f0ff009abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff009ef70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 0x7f0ff0130ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff009fe10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff0131980 0x7f0ff011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff0131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff009b6f0 con 0x7f0ff009fe10 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff0141ef0 con 0x7f0ff009ef70 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff011ddc0 con 0x7f0ff0131980 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.579+0000 7f0ff839d640 10 monclient(hunting): authenticate will time out at 775.219991s 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff6139640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ff009ef70 msgr2=0x7f0ff009abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 0x7f0ff0130ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff6139640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ff009ef70 0x7f0ff009abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff0131980 0x7f0ff011b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 0x7f0ff0130ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37226/0 (socket says 172.21.15.153:37226) 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 -- 172.21.15.153:0/3020086825 learned_addr learned my addr 172.21.15.153:0/3020086825 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): get_auth_request con 0x7f0ff009fe10 auth_method 0 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): get_auth_request con 0x7f0ff0131980 auth_method 0 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient(hunting): handle_auth_done global_id 34158 payload 824 2024-09-15T03:29:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 -- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff0131980 msgr2=0x7f0ff011b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff0131980 0x7f0ff011b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 -- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ff009ef70 msgr2=0x7f0ff009abe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ff009ef70 0x7f0ff009abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient: _finish_hunting 0 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 monclient: found mon.noname-a 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff5938640 1 -- 172.21.15.153:0/3020086825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0009450 con 0x7f0ff009fe10 2024-09-15T03:29:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.580+0000 7f0ff693a640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff0131980 0x7f0ff011b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.582+0000 7f0ff5938640 1 -- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 msgr2=0x7f0ff0130ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.582+0000 7f0ff5938640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 0x7f0ff0130ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000ec80 tx=0x7f0fe00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0009730 con 0x7f0ff009fe10 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: dump: 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe000eec0 con 0x7f0ff009fe10 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 5 monclient: authenticate success, global_id 34158 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0013470 con 0x7f0ff009fe10 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 10 monclient: _renew_subs 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8009a20 con 0x7f0ff009fe10 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff4936640 4 set_mon_vals no callback set 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 10 monclient: dump: 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 10 monclient: _renew_subs 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff011e380 con 0x7f0ff009fe10 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.583+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0fe0013820 con 0x7f0ff009fe10 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0fe67fc640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd4038740 0x7f0fd403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f0fe004bd30 con 0x7f0ff009fe10 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0ff839d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0ff839d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0ff839d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff011cec0 con 0x7f0ff009fe10 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0ff6139640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd4038740 0x7f0fd403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.584+0000 7f0ff6139640 10 monclient: get_auth_request con 0x7f0fd4038740 auth_method 0 2024-09-15T03:29:37.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.586+0000 7f0ff6139640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd4038740 0x7f0fd403abf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0038f0 tx=0x7f0fec006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:37.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.588+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0fe0023510 con 0x7f0ff009fe10 2024-09-15T03:29:37.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.588+0000 7f0fe67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:37.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.588+0000 7f0fe67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.803+0000 7f0ff839d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.803+0000 7f0ff839d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.803+0000 7f0ff839d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.803+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ff004bf60 con 0x7f0ff009fe10 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.804+0000 7f0fe67fc640 1 -- 172.21.15.153:0/3020086825 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0fe0023330 con 0x7f0ff009fe10 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.804+0000 7f0fe67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.804+0000 7f0fe67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:37.806 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:37.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd4038740 msgr2=0x7f0fd403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd4038740 0x7f0fd403abf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0038f0 tx=0x7f0fec006d80 comp rx=0 tx=0).stop 2024-09-15T03:29:37.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 10 monclient: shutdown 2024-09-15T03:29:37.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 msgr2=0x7f0ff0130ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 0x7f0ff0130ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000ec80 tx=0x7f0fe00091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 shutdown_connections 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd4038740 0x7f0fd403abf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ff0131980 0x7f0ff011b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ff009fe10 0x7f0ff0130ca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 --2- 172.21.15.153:0/3020086825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ff009ef70 0x7f0ff009abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 >> 172.21.15.153:0/3020086825 conn(0x7f0ff00a5050 msgr2=0x7f0ff009e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 shutdown_connections 2024-09-15T03:29:37.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:37.806+0000 7f0ff839d640 1 -- 172.21.15.153:0/3020086825 wait complete. 2024-09-15T03:29:38.154 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T03:29:38.219 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T03:29:38.260 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T03:29:38.262 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T03:29:38.262 INFO:tasks.workunit.client.0.smithi181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T03:29:38.302 INFO:tasks.workunit.client.0.smithi181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T03:29:38.305 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T03:29:38.305 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T03:29:38.361 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-15T03:29:38.362 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-15T03:29:38.362 DEBUG:teuthology.orchestra.run.smithi181:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-15T03:29:38.421 INFO:tasks.workunit.client.0.smithi181.stderr:+ TEST_POOL=testpool1234 2024-09-15T03:29:38.421 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-15T03:29:38.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.496+0000 7f044655d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:38.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.496+0000 7f044655d640 10 monclient: build_initial_monmap 2024-09-15T03:29:38.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.496+0000 7f044655d640 10 monclient: monmap: 2024-09-15T03:29:38.498 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:38.498281+0000 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:38.498281+0000 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:38.499 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:38.500 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 1 Processor -- start 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 1 -- start start 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient: init 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f044009cad0 0x7f044009c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient(hunting): picked mon.noname-a con 0x7f044009cad0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f044009cea0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f044009d2a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044009b710 con 0x7f044009d2a0 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044009c6e0 con 0x7f044009cea0 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:38.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044009e8f0 con 0x7f044009cad0 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f044655d640 10 monclient(hunting): authenticate will time out at 782.979873s 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.497+0000 7f043f7fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f044009cea0 legacy=0x7f044009dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f044009cad0 0x7f044009c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f0444afa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f044009d2a0 0x7f044009e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:38852/0 (socket says 172.21.15.181:38852) 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f044009cad0 0x7f044009c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:38140/0 (socket says 172.21.15.181:38140) 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f0444afa640 1 -- 172.21.15.181:0/3033176494 learned_addr learned my addr 172.21.15.181:0/3033176494 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): get_auth_request con 0x7f044009cad0 auth_method 0 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:38.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient(hunting): handle_auth_done global_id 34161 payload 274 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 1 -- 172.21.15.181:0/3033176494 >> v1:172.21.15.181:6789/0 conn(0x7f044009cea0 legacy=0x7f044009dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 1 -- 172.21.15.181:0/3033176494 >> v1:172.21.15.153:6789/0 conn(0x7f044009d2a0 legacy=0x7f044009e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 1 monclient: found mon.noname-a 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:38.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.498+0000 7f043ffff640 1 -- 172.21.15.181:0/3033176494 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044009a650 con 0x7f044009cad0 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043ffff640 1 -- 172.21.15.181:0/3033176494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009cad0 msgr2=0x7f044009c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043ffff640 1 --2- 172.21.15.181:0/3033176494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009cad0 0x7f044009c1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0430009850 tx=0x7f04300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1954c1a0b195b72d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 1 -- 172.21.15.181:0/3033176494 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430032440 con 0x7f044009cad0 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: dump: 2024-09-15T03:29:38.511 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: _finish_auth 0 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 1 -- 172.21.15.181:0/3033176494 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04300325a0 con 0x7f044009cad0 2024-09-15T03:29:38.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f044655d640 5 monclient: authenticate success, global_id 34161 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 1 -- 172.21.15.181:0/3033176494 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430032940 con 0x7f044009cad0 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f044655d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f044655d640 4 set_mon_vals no callback set 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.501+0000 7f043effd640 10 monclient: dump: 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:38.513 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 10 monclient: shutdown 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 -- 172.21.15.181:0/3033176494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009cad0 msgr2=0x7f044009c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 --2- 172.21.15.181:0/3033176494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009cad0 0x7f044009c1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0430009850 tx=0x7f04300090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 -- 172.21.15.181:0/3033176494 shutdown_connections 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 --2- 172.21.15.181:0/3033176494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009cad0 0x7f044009c1a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 -- 172.21.15.181:0/3033176494 >> 172.21.15.181:0/3033176494 conn(0x7f0440070e90 msgr2=0x7f04400a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 -- 172.21.15.181:0/3033176494 shutdown_connections 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 -- 172.21.15.181:0/3033176494 wait complete. 2024-09-15T03:29:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 10 monclient: build_initial_monmap 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 10 monclient: monmap: 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:38.515 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.502+0000 7f044655d640 1 Processor -- start 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 -- start start 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient: init 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 0x7f044013f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): picked mon.noname-c con 0x7f044009cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044009cea0 0x7f0440131dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): picked mon.noname-b con 0x7f044009cea0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:38.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 0x7f044011b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): picked mon.noname-a con 0x7f044009d2a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044011d940 con 0x7f044009d2a0 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044006eba0 con 0x7f044009cea0 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044006dde0 con 0x7f044009cad0 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f044655d640 10 monclient(hunting): authenticate will time out at 782.985538s 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043f7fe640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044009cea0 msgr2=0x7f0440131dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044009cea0 0x7f0440131dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:38.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f0444afa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 0x7f044011b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f0444afa640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 msgr2=0x7f044011b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f0444afa640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 msgr2=0x7f044011b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f0444afa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 0x7f044011b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f0444afa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 0x7f044011b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 0x7f044013f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 0x7f044013f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42282/0 (socket says 172.21.15.181:42282) 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 1 -- 172.21.15.181:0/1099641830 learned_addr learned my addr 172.21.15.181:0/1099641830 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): get_auth_request con 0x7f044009cad0 auth_method 0 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:38.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.503+0000 7f043ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 1 -- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044009cea0 msgr2=0x7f0440131dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044009cea0 0x7f0440131dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 1 -- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 msgr2=0x7f044011b880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 0x7f044011b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 1 monclient: found mon.noname-c 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.504+0000 7f043ffff640 1 -- 172.21.15.181:0/1099641830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0430009390 con 0x7f044009cad0 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f043ffff640 1 -- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 msgr2=0x7f044013f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f043ffff640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 0x7f044013f060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0430009850 tx=0x7f0430032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430031a70 con 0x7f044009cad0 2024-09-15T03:29:38.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: dump: 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: _finish_auth 0 2024-09-15T03:29:38.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0430031bd0 con 0x7f044009cad0 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f044655d640 5 monclient: authenticate success, global_id 4466 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f044655d640 10 monclient: _renew_subs 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f044655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044013feb0 con 0x7f044009cad0 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043003b400 con 0x7f044009cad0 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f0427fff640 10 monclient: dump: 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f043e7fc640 4 set_mon_vals no callback set 2024-09-15T03:29:38.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f044655d640 10 monclient: _renew_subs 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.506+0000 7f044655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.507+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044011d3c0 con 0x7f044009cad0 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.507+0000 7f044655d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.507+0000 7f044655d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.507+0000 7f044655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.507+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0430039040 con 0x7f044009cad0 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.507+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044011f360 con 0x7f044009cad0 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.508+0000 7f0427fff640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f042c038740 0x7f042c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:38.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.508+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f043006bc90 con 0x7f044009cad0 2024-09-15T03:29:38.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.508+0000 7f043f7fe640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f042c038740 0x7f042c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:38.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.508+0000 7f043f7fe640 10 monclient: get_auth_request con 0x7f042c038740 auth_method 0 2024-09-15T03:29:38.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.510+0000 7f043f7fe640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f042c038740 0x7f042c03abf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0434003940 tx=0x7f0434006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:38.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.513+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f043004b500 con 0x7f044009cad0 2024-09-15T03:29:38.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.513+0000 7f0427fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:38.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.513+0000 7f0427fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:38.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.667+0000 7f044655d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T03:29:38.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.667+0000 7f044655d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T03:29:38.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.667+0000 7f044655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:38.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:38.667+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f044011f360 con 0x7f044009cad0 2024-09-15T03:29:40.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.924+0000 7f48c8f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.924+0000 7f48c8f5d640 10 monclient: build_initial_monmap 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.924+0000 7f48c8f5d640 10 monclient: monmap: 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:40.925039+0000 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:40.925039+0000 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:40.926 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.924+0000 7f48c8f5d640 1 Processor -- start 2024-09-15T03:29:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.924+0000 7f48c8f5d640 1 -- start start 2024-09-15T03:29:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.924+0000 7f48c8f5d640 10 monclient: init 2024-09-15T03:29:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 0x7f48c4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f48c41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f48c409ea50 0x7f48c409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f48c409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48c4068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c4131cc0 con 0x7f48c4068d90 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c409fbf0 con 0x7f48c409ea50 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c409fe30 con 0x7f48c41318f0 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c8f5d640 10 monclient(hunting): authenticate will time out at 778.564730s 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 0x7f48c4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c2d9d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f48c4068d90 legacy=0x7f48c40691f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c1d9b640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f48c409ea50 msgr2=0x7f48c409ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 0x7f48c4131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52122/0 (socket says 172.21.15.153:52122) 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 1 -- 172.21.15.153:0/922488105 learned_addr learned my addr 172.21.15.153:0/922488105 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): get_auth_request con 0x7f48c41318f0 auth_method 0 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c1d9b640 1 --2- 172.21.15.153:0/922488105 >> v2:172.21.15.181:3300/0 conn(0x7f48c409ea50 0x7f48c409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.925+0000 7f48c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 1 -- 172.21.15.153:0/922488105 >> v2:172.21.15.181:3300/0 conn(0x7f48c409ea50 msgr2=0x7f48c409ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 1 --2- 172.21.15.153:0/922488105 >> v2:172.21.15.181:3300/0 conn(0x7f48c409ea50 0x7f48c409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 1 -- 172.21.15.153:0/922488105 >> v1:172.21.15.181:6789/0 conn(0x7f48c4068d90 legacy=0x7f48c40691f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 1 monclient: found mon.noname-b 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.926+0000 7f48c259c640 1 -- 172.21.15.153:0/922488105 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c40a1210 con 0x7f48c41318f0 2024-09-15T03:29:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.928+0000 7f48c259c640 1 -- 172.21.15.153:0/922488105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 msgr2=0x7f48c4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.928+0000 7f48c259c640 1 --2- 172.21.15.153:0/922488105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 0x7f48c4131f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009910 tx=0x7f48b80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4233fe373e1eeb67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 1 -- 172.21.15.153:0/922488105 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8032440 con 0x7f48c41318f0 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: dump: 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: _finish_auth 0 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 5 monclient: authenticate success, global_id 4469 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 1 -- 172.21.15.153:0/922488105 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b80325a0 con 0x7f48c41318f0 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 4 set_mon_vals no callback set 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 1 -- 172.21.15.153:0/922488105 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8032940 con 0x7f48c41318f0 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:40.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c159a640 10 monclient: dump: 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 10 monclient: shutdown 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:40.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 -- 172.21.15.153:0/922488105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 msgr2=0x7f48c4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/922488105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 0x7f48c4131f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009910 tx=0x7f48b80091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 -- 172.21.15.153:0/922488105 shutdown_connections 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/922488105 >> v2:172.21.15.181:3300/0 conn(0x7f48c409ea50 0x7f48c409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/922488105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c41318f0 0x7f48c4131f30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 -- 172.21.15.153:0/922488105 >> 172.21.15.153:0/922488105 conn(0x7f48c4067110 msgr2=0x7f48c4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 -- 172.21.15.153:0/922488105 shutdown_connections 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.929+0000 7f48c8f5d640 1 -- 172.21.15.153:0/922488105 wait complete. 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 10 monclient: build_initial_monmap 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 10 monclient: monmap: 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:40.940 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 1 Processor -- start 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 1 -- start start 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 10 monclient: init 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 0x7f48c409ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:40.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f48c4068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.930+0000 7f48c8f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c409ea50 0x7f48c4130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f48c409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c41318f0 0x7f48c4140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f48c41318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c411df10 con 0x7f48c409ea50 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c411dc30 con 0x7f48c41318f0 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c411e1f0 con 0x7f48c4068d90 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c8f5d640 10 monclient(hunting): authenticate will time out at 778.570041s 2024-09-15T03:29:40.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 0x7f48c409ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 0x7f48c409ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52134/0 (socket says 172.21.15.153:52134) 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 -- 172.21.15.153:0/2622369875 learned_addr learned my addr 172.21.15.153:0/2622369875 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): get_auth_request con 0x7f48c4068d90 auth_method 0 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c409ea50 0x7f48c4130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): get_auth_request con 0x7f48c409ea50 auth_method 0 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c2d9d640 1 -- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c41318f0 msgr2=0x7f48c4140f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:40.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c2d9d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c41318f0 0x7f48c4140f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 -- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c41318f0 msgr2=0x7f48c4140f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c41318f0 0x7f48c4140f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 -- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c409ea50 msgr2=0x7f48c4130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c409ea50 0x7f48c4130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:40.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 monclient: found mon.noname-c 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c259c640 1 -- 172.21.15.153:0/2622369875 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b8009450 con 0x7f48c4068d90 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.931+0000 7f48c1d9b640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c409ea50 0x7f48c4130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c259c640 1 -- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 msgr2=0x7f48c409ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c259c640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 0x7f48c409ab20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009910 tx=0x7f48b8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b803c960 con 0x7f48c4068d90 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: dump: 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b803cac0 con 0x7f48c4068d90 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 5 monclient: authenticate success, global_id 4472 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 10 monclient: _renew_subs 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c411c890 con 0x7f48c4068d90 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b803ce80 con 0x7f48c4068d90 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48b27fc640 10 monclient: dump: 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:40.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c0d99640 4 set_mon_vals no callback set 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 10 monclient: _renew_subs 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.933+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c411cee0 con 0x7f48c4068d90 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.934+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f48b8039040 con 0x7f48c4068d90 2024-09-15T03:29:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.934+0000 7f48c8f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.934+0000 7f48c8f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.934+0000 7f48c8f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.934+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c404bf60 con 0x7f48c4068d90 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.935+0000 7f48b27fc640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48a4038740 0x7f48a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.935+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f48b806c010 con 0x7f48c4068d90 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.935+0000 7f48c1d9b640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48a4038740 0x7f48a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.935+0000 7f48c1d9b640 10 monclient: get_auth_request con 0x7f48a4038740 auth_method 0 2024-09-15T03:29:40.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.937+0000 7f48c1d9b640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48a4038740 0x7f48a403abf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f48c409b2c0 tx=0x7f48b40090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:40.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.939+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f48b809de40 con 0x7f48c4068d90 2024-09-15T03:29:40.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.939+0000 7f48b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:40.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:40.939+0000 7f48b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:41.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.152+0000 7f48c8f5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.152+0000 7f48c8f5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.152+0000 7f48c8f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.152+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f48c411e330 con 0x7f48c4068d90 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.153+0000 7f48b27fc640 1 -- 172.21.15.153:0/2622369875 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f48c411e330 con 0x7f48c4068d90 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.153+0000 7f48b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.153+0000 7f48b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:41.154 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.154+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48a4038740 msgr2=0x7f48a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48a4038740 0x7f48a403abf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f48c409b2c0 tx=0x7f48b40090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 10 monclient: shutdown 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 msgr2=0x7f48c409ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 0x7f48c409ab20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009910 tx=0x7f48b8032000 comp rx=0 tx=0).stop 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 shutdown_connections 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48a4038740 0x7f48a403abf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c41318f0 0x7f48c4140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:41.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c409ea50 0x7f48c4130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:41.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 --2- 172.21.15.153:0/2622369875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c4068d90 0x7f48c409ab20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:41.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 >> 172.21.15.153:0/2622369875 conn(0x7f48c4067110 msgr2=0x7f48c40a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:41.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 shutdown_connections 2024-09-15T03:29:41.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:41.155+0000 7f48c8f5d640 1 -- 172.21.15.153:0/2622369875 wait complete. 2024-09-15T03:29:41.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:41.503+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:41.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:41.503+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:41.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:41.503+0000 7f0426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:42.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:42.504+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:42.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:42.504+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:43.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:43.503+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:43.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:43.503+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:44.167 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.269+0000 7f4a5ae6f640 10 monclient: get_monmap_and_config 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.269+0000 7f4a5ae6f640 10 monclient: build_initial_monmap 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.269+0000 7f4a5ae6f640 10 monclient: monmap: 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:44.269773+0000 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:44.269773+0000 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:44.270 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:44.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:44.271 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:44.271 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:44.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:44.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.269+0000 7f4a5ae6f640 1 Processor -- start 2024-09-15T03:29:44.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.269+0000 7f4a5ae6f640 1 -- start start 2024-09-15T03:29:44.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.269+0000 7f4a5ae6f640 10 monclient: init 2024-09-15T03:29:44.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:44.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a54131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 0x7f4a5409e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5409fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4a5409ef00 0x7f4a54138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5409ef00 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a540a1180 con 0x7f4a54131980 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a540a1460 con 0x7f4a5409ef00 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54132460 con 0x7f4a5409fda0 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5ae6f640 10 monclient(hunting): authenticate will time out at 781.909587s 2024-09-15T03:29:44.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a58c0b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4a54131980 0x7f4a54131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36386/0 (socket says 172.21.15.153:36386) 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a58c0b640 1 -- 172.21.15.153:0/711731766 learned_addr learned my addr 172.21.15.153:0/711731766 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 1 --2- 172.21.15.153:0/711731766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 0x7f4a5409e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): get_auth_request con 0x7f4a5409fda0 auth_method 0 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 1 --2- 172.21.15.153:0/711731766 >> v2:172.21.15.153:3300/0 conn(0x7f4a5409ef00 0x7f4a54138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): get_auth_request con 0x7f4a5409ef00 auth_method 0 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:44.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a5940c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 1 -- 172.21.15.153:0/711731766 >> v2:172.21.15.153:3300/0 conn(0x7f4a5409ef00 msgr2=0x7f4a54138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 1 --2- 172.21.15.153:0/711731766 >> v2:172.21.15.153:3300/0 conn(0x7f4a5409ef00 0x7f4a54138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 1 -- 172.21.15.153:0/711731766 >> v1:172.21.15.153:6789/0 conn(0x7f4a54131980 legacy=0x7f4a54131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 1 monclient: found mon.noname-b 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:44.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.270+0000 7f4a4bfff640 1 -- 172.21.15.153:0/711731766 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a541325d0 con 0x7f4a5409fda0 2024-09-15T03:29:44.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.273+0000 7f4a4bfff640 1 -- 172.21.15.153:0/711731766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 msgr2=0x7f4a5409e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:44.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.273+0000 7f4a4bfff640 1 --2- 172.21.15.153:0/711731766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 0x7f4a5409e9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009850 tx=0x7f4a440090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c51b405febe3f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:44.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 1 -- 172.21.15.153:0/711731766 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4402adf0 con 0x7f4a5409fda0 2024-09-15T03:29:44.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:44.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:44.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: dump: 2024-09-15T03:29:44.285 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:44.285 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:44.285 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 5 monclient: authenticate success, global_id 4475 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 1 -- 172.21.15.153:0/711731766 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a44032440 con 0x7f4a5409fda0 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:44.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 4 set_mon_vals no callback set 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 1 -- 172.21.15.153:0/711731766 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a440327e0 con 0x7f4a5409fda0 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a4b7fe640 10 monclient: dump: 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:44.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 10 monclient: shutdown 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/711731766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 msgr2=0x7f4a5409e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/711731766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 0x7f4a5409e9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009850 tx=0x7f4a440090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/711731766 shutdown_connections 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/711731766 >> v2:172.21.15.153:3300/0 conn(0x7f4a5409ef00 0x7f4a54138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/711731766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409fda0 0x7f4a5409e9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/711731766 >> 172.21.15.153:0/711731766 conn(0x7f4a54067bd0 msgr2=0x7f4a540a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/711731766 shutdown_connections 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.274+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/711731766 wait complete. 2024-09-15T03:29:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.275+0000 7f4a5ae6f640 10 monclient: build_initial_monmap 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.275+0000 7f4a5ae6f640 10 monclient: monmap: 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.275+0000 7f4a5ae6f640 1 Processor -- start 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.275+0000 7f4a5ae6f640 1 -- start start 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.275+0000 7f4a5ae6f640 10 monclient: init 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 0x7f4a54134030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5409ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a5409fda0 0x7f4a54134570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5409fda0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54131980 0x7f4a54134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): picked mon.noname-a con 0x7f4a54131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:44.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5409bf10 con 0x7f4a54131980 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54137c10 con 0x7f4a5409fda0 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a541378a0 con 0x7f4a5409ef00 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5ae6f640 10 monclient(hunting): authenticate will time out at 781.915401s 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 0x7f4a54134030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 0x7f4a54134030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52150/0 (socket says 172.21.15.153:52150) 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 -- 172.21.15.153:0/3980703154 learned_addr learned my addr 172.21.15.153:0/3980703154 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): get_auth_request con 0x7f4a5409ef00 auth_method 0 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:44.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54131980 0x7f4a54134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): get_auth_request con 0x7f4a54131980 auth_method 0 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a4bfff640 1 -- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a5409fda0 msgr2=0x7f4a54134570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a4bfff640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a5409fda0 0x7f4a54134570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a5940c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient(hunting): handle_auth_done global_id 4478 payload 824 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 -- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a5409fda0 msgr2=0x7f4a54134570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a5409fda0 0x7f4a54134570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 -- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54131980 msgr2=0x7f4a54134ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54131980 0x7f4a54134ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient: _finish_hunting 0 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 monclient: found mon.noname-c 2024-09-15T03:29:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.276+0000 7f4a58c0b640 1 -- 172.21.15.153:0/3980703154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a44009390 con 0x7f4a5409ef00 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.278+0000 7f4a58c0b640 1 -- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 msgr2=0x7f4a54134030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.278+0000 7f4a58c0b640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 0x7f4a54134030 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c610 tx=0x7f4a50004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50005cb0 con 0x7f4a5409ef00 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: dump: 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:44.294 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: _finish_auth 0 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 5 monclient: authenticate success, global_id 4478 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50005e30 con 0x7f4a5409ef00 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 10 monclient: _renew_subs 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a541375c0 con 0x7f4a5409ef00 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:44.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50007340 con 0x7f4a5409ef00 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a48ff9640 10 monclient: dump: 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:44.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a4affd640 4 set_mon_vals no callback set 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 10 monclient: _renew_subs 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.279+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a54138330 con 0x7f4a5409ef00 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4a5001b400 con 0x7f4a5409ef00 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a5ae6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a5ae6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a5ae6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a541385b0 con 0x7f4a5409ef00 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a48ff9640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4a400386f0 0x7f4a4003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4a5004b500 con 0x7f4a5409ef00 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a4bfff640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4a400386f0 0x7f4a4003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.280+0000 7f4a4bfff640 10 monclient: get_auth_request con 0x7f4a400386f0 auth_method 0 2024-09-15T03:29:44.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.283+0000 7f4a4bfff640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4a400386f0 0x7f4a4003aba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a540a4fb0 tx=0x7f4a44032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:44.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.285+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4a50007730 con 0x7f4a5409ef00 2024-09-15T03:29:44.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.285+0000 7f4a48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:44.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.285+0000 7f4a48ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a5ae6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a5ae6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a5ae6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4a540a3f20 con 0x7f4a5409ef00 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a48ff9640 1 -- 172.21.15.153:0/3980703154 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4a5001a360 con 0x7f4a5409ef00 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.502+0000 7f4a48ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:44.504 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:44.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4a400386f0 msgr2=0x7f4a4003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:44.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4a400386f0 0x7f4a4003aba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a540a4fb0 tx=0x7f4a44032000 comp rx=0 tx=0).stop 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 10 monclient: shutdown 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 msgr2=0x7f4a54134030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 0x7f4a54134030 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c610 tx=0x7f4a50004960 comp rx=0 tx=0).stop 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 shutdown_connections 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4a400386f0 0x7f4a4003aba0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a54131980 0x7f4a54134ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a5409fda0 0x7f4a54134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 --2- 172.21.15.153:0/3980703154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a5409ef00 0x7f4a54134030 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 >> 172.21.15.153:0/3980703154 conn(0x7f4a54067bd0 msgr2=0x7f4a5409d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 shutdown_connections 2024-09-15T03:29:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:44.504+0000 7f4a5ae6f640 1 -- 172.21.15.153:0/3980703154 wait complete. 2024-09-15T03:29:44.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:44.504+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:44.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:44.504+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:45.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:45.504+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:45.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:45.504+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:46.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:46.504+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:46.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:46.504+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:47.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:47.505+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:47.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:47.505+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:47.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:29:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.620+0000 7f5ff6c17640 10 monclient: get_monmap_and_config 2024-09-15T03:29:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.620+0000 7f5ff6c17640 10 monclient: build_initial_monmap 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.620+0000 7f5ff6c17640 10 monclient: monmap: 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:47.621499+0000 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:47.621499+0000 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:47.622 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.621+0000 7f5ff6c17640 1 Processor -- start 2024-09-15T03:29:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.621+0000 7f5ff6c17640 1 -- start start 2024-09-15T03:29:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.621+0000 7f5ff6c17640 10 monclient: init 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.621+0000 7f5ff6c17640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.621+0000 7f5ff6c17640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 0x7f5ff009cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): picked mon.noname-c con 0x7f5ff012f700 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): picked mon.noname-b con 0x7f5ff012fad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): picked mon.noname-a con 0x7f5ff012c560 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff0097eb0 con 0x7f5ff012c560 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff009ac10 con 0x7f5ff012f700 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff009c1e0 con 0x7f5ff012fad0 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff6c17640 10 monclient(hunting): authenticate will time out at 785.261067s 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff49b3640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 msgr2=0x7f5ff009cf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff49b3640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 0x7f5ff009cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35142/0 (socket says 172.21.15.153:35142) 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 learned_addr learned my addr 172.21.15.153:0/3414195184 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request con 0x7f5ff012c560 auth_method 0 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:47.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 10 monclient(hunting): get_auth_request con 0x7f5ff012fad0 auth_method 0 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:47.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:47.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.622+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request con 0x7f5ff012c560 auth_method 0 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 10 monclient(hunting): get_auth_request con 0x7f5ff012fad0 auth_method 0 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 msgr2=0x7f5ff009cf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 0x7f5ff009cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:29:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:47.823+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:29:48.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:48.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 msgr2=0x7f5ff009cf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:48.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 0x7f5ff009cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 10 monclient(hunting): get_auth_request con 0x7f5ff012fad0 auth_method 0 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request con 0x7f5ff012c560 auth_method 0 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:48.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:29:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:48.224+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:29:48.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:48.504+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:48.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:48.504+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 msgr2=0x7f5ff009cf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 0x7f5ff009cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 10 monclient(hunting): get_auth_request con 0x7f5ff012fad0 auth_method 0 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request con 0x7f5ff012c560 auth_method 0 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:49.025+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:29:49.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:49.505+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:49.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:49.505+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:50.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:50.505+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:50.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:50.505+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:50.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 10 monclient(hunting): tick 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 msgr2=0x7f5ff012fea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff012fad0 0x7f5ff012fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 msgr2=0x7f5ff009cf10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.181:3300/0 conn(0x7f5ff012f700 0x7f5ff009cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 -- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 msgr2=0x7f5ff012c9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 --2- 172.21.15.153:0/3414195184 >> v2:172.21.15.153:3300/0 conn(0x7f5ff012c560 0x7f5ff012c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 10 monclient: _add_conns ranks=[0,1] 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fd4003670 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:50.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fd4007370 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:50.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 -- 172.21.15.153:0/3414195184 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff009c4c0 con 0x7f5fd4003670 2024-09-15T03:29:50.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 1 -- 172.21.15.153:0/3414195184 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd400c0e0 con 0x7f5fd4007370 2024-09-15T03:29:50.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5fee7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:50.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.622+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> v1:172.21.15.181:6789/0 conn(0x7f5fd4007370 legacy=0x7f5fd4009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:50.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.624+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> v1:172.21.15.153:6789/0 conn(0x7f5fd4003670 legacy=0x7f5fd4005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:50.823+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> v1:172.21.15.181:6789/0 conn(0x7f5fd4007370 legacy=0x7f5fd4009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:51.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.224+0000 7f5feffff640 1 -- 172.21.15.153:0/3414195184 >> v1:172.21.15.181:6789/0 conn(0x7f5fd4007370 legacy=0x7f5fd4009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:51.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:51.505+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:51.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:51.505+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:51.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:51.505+0000 7f0426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:51.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.944+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883959701 0 0) 0x7f5ff009c4c0 con 0x7f5fd4003670 2024-09-15T03:29:51.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.944+0000 7f5fef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:51.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.944+0000 7f5fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:51.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.944+0000 7f5fef7fe640 10 monclient(hunting): my global_id is 34173 2024-09-15T03:29:51.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.944+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe00034e0 con 0x7f5fd4003670 2024-09-15T03:29:51.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074050532 0 0) 0x7f5fe00034e0 con 0x7f5fd4003670 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 >> v1:172.21.15.181:6789/0 conn(0x7f5fd4007370 legacy=0x7f5fd4009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd400d340 con 0x7f5fd4003670 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f5fe4004280 con 0x7f5fd4003670 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff6c17640 5 monclient: authenticate success, global_id 34173 2024-09-15T03:29:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff6c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: dump: 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 >> v1:172.21.15.153:6789/0 conn(0x7f5fd4003670 legacy=0x7f5fd4005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:29:51.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 reap_dead start 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 0x7f5fe0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): picked mon.a con 0x7f5fe0003a00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0004780 0x7f5fe0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): picked mon.c con 0x7f5fe0004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0005250 0x7f5fe000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): picked mon.b con 0x7f5fe0005250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe000c1c0 con 0x7f5fe0003a00 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe000c330 con 0x7f5fe0005250 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:51.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe000c4f0 con 0x7f5fe0004780 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5fef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff6c17640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 0x7f5fe0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5feffff640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0004780 0x7f5fe0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff49b3640 10 monclient(hunting): get_auth_request con 0x7f5fe0003a00 auth_method 0 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff49b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff49b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff49b3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff51b4640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0005250 msgr2=0x7f5fe000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5feffff640 10 monclient(hunting): get_auth_request con 0x7f5fe0004780 auth_method 0 2024-09-15T03:29:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5feffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5feffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.947+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0005250 0x7f5fe000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5feffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5feffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5feffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_done global_id 34173 payload 293 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0004780 msgr2=0x7f5fe0004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0004780 0x7f5fe0004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0005250 msgr2=0x7f5fe000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0005250 0x7f5fe000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient: _finish_hunting 0 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 1 monclient: found mon.a 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00034e0 con 0x7f5fe0003a00 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 10 monclient: _finish_auth 0 2024-09-15T03:29:51.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.948+0000 7f5ff49b3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.952+0000 7f5ff49b3640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 msgr2=0x7f5fe0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.952+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 0x7f5fe0003dd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40038f0 tx=0x7f5fe4007b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f735710dce5a24c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:51.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4007150 con 0x7f5fe0003a00 2024-09-15T03:29:51.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:51.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:51.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: dump: 2024-09-15T03:29:51.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:51.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:51.983 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:51.984 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:51.984 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:51.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5ff6c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:51.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe4007310 con 0x7f5fe0003a00 2024-09-15T03:29:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 1 -- 172.21.15.153:0/3414195184 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40074c0 con 0x7f5fe0003a00 2024-09-15T03:29:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5fef7fe640 10 monclient: dump: 2024-09-15T03:29:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:51.986 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:51.987 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:51.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5ff6c17640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:51.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.967+0000 7f5ff6c17640 4 set_mon_vals no callback set 2024-09-15T03:29:51.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 10 monclient: shutdown 2024-09-15T03:29:51.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:51.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 -- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 msgr2=0x7f5fe0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 0x7f5fe0003dd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40038f0 tx=0x7f5fe4007b60 comp rx=0 tx=0).stop 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 -- 172.21.15.153:0/3414195184 shutdown_connections 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0005250 0x7f5fe000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0004780 0x7f5fe0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/3414195184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0003a00 0x7f5fe0003dd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 -- 172.21.15.153:0/3414195184 >> 172.21.15.153:0/3414195184 conn(0x7f5ff0067d00 msgr2=0x7f5ff00a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:51.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 -- 172.21.15.153:0/3414195184 shutdown_connections 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.968+0000 7f5ff6c17640 1 -- 172.21.15.153:0/3414195184 wait complete. 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.969+0000 7f5ff6c17640 10 monclient: build_initial_monmap 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.969+0000 7f5ff6c17640 10 monclient: monmap: 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:51.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:51.990 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:51.990 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.969+0000 7f5ff6c17640 1 Processor -- start 2024-09-15T03:29:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 -- start start 2024-09-15T03:29:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient: init 2024-09-15T03:29:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 0x7f5ff006d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe0003a00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0004780 0x7f5ff012b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe0004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 0x7f5ff012bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): picked mon.noname-a con 0x7f5fe0005250 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff0118610 con 0x7f5fe0005250 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff0119ce0 con 0x7f5fe0004780 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff0118aa0 con 0x7f5fe0003a00 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff6c17640 10 monclient(hunting): authenticate will time out at 789.609725s 2024-09-15T03:29:51.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5feffff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0004780 msgr2=0x7f5ff012b8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff49b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 0x7f5ff006d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 0x7f5ff012bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5feffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0004780 0x7f5ff012b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 0x7f5ff012bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35180/0 (socket says 172.21.15.153:35180) 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff49b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 0x7f5ff006d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40816/0 (socket says 172.21.15.153:40816) 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 1 -- 172.21.15.153:0/44301446 learned_addr learned my addr 172.21.15.153:0/44301446 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request con 0x7f5fe0005250 auth_method 0 2024-09-15T03:29:51.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff49b3640 10 monclient(hunting): get_auth_request con 0x7f5fe0003a00 auth_method 0 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff49b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff49b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff49b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:51.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.970+0000 7f5ff51b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff51b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 1 -- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0004780 msgr2=0x7f5ff012b8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0004780 0x7f5ff012b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 1 -- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 msgr2=0x7f5ff012bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:51.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 0x7f5ff012bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 10 monclient: _finish_hunting 0 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 1 monclient: found mon.noname-c 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff49b3640 1 -- 172.21.15.153:0/44301446 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff0119a00 con 0x7f5fe0003a00 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.971+0000 7f5ff51b4640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 0x7f5ff012bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5ff49b3640 1 -- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 msgr2=0x7f5ff006d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5ff49b3640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 0x7f5ff006d660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4033000 tx=0x7f5fe4034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4004520 con 0x7f5fe0003a00 2024-09-15T03:29:51.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5fecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5fecff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5fecff9640 10 monclient: dump: 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:51.998 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5fecff9640 10 monclient: _finish_auth 0 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.973+0000 7f5fecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 5 monclient: authenticate success, global_id 4481 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe40046c0 con 0x7f5fe0003a00 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 10 monclient: _renew_subs 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff0119fc0 con 0x7f5fe0003a00 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:51.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4004a80 con 0x7f5fe0003a00 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 10 monclient: dump: 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.000 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 10 monclient: _renew_subs 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff0117a70 con 0x7f5fe0003a00 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5feeffd640 4 set_mon_vals no callback set 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.974+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5fe4039430 con 0x7f5fe0003a00 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5fecff9640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5fd4038f50 0x7f5fd403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f5fe406f250 con 0x7f5fe0003a00 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5ff6c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5ff6c17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5ff6c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:52.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff00a0100 con 0x7f5fe0003a00 2024-09-15T03:29:52.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5feffff640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5fd4038f50 0x7f5fd403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.975+0000 7f5feffff640 10 monclient: get_auth_request con 0x7f5fd4038f50 auth_method 0 2024-09-15T03:29:52.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.978+0000 7f5feffff640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5fd4038f50 0x7f5fd403b400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc006d30 tx=0x7f5fdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.980+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5ff00a0100 con 0x7f5fe0003a00 2024-09-15T03:29:52.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.980+0000 7f5fecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:51.980+0000 7f5fecff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:52.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5ff6c17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:29:52.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5ff6c17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:52.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5ff6c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:52.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5ff00a0100 con 0x7f5fe0003a00 2024-09-15T03:29:52.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5fecff9640 1 -- 172.21.15.153:0/44301446 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f5fe4052610 con 0x7f5fe0003a00 2024-09-15T03:29:52.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5fecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:29:52.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.196+0000 7f5fecff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:52.198 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:29:52.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5fd4038f50 msgr2=0x7f5fd403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5fd4038f50 0x7f5fd403b400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc006d30 tx=0x7f5fdc008040 comp rx=0 tx=0).stop 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 10 monclient: shutdown 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 msgr2=0x7f5ff006d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 0x7f5ff006d660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4033000 tx=0x7f5fe4034000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 shutdown_connections 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5fd4038f50 0x7f5fd403b400 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fe0005250 0x7f5ff012bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe0004780 0x7f5ff012b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 --2- 172.21.15.153:0/44301446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fe0003a00 0x7f5ff006d660 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 >> 172.21.15.153:0/44301446 conn(0x7f5ff0067d00 msgr2=0x7f5ff009d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 shutdown_connections 2024-09-15T03:29:52.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.198+0000 7f5ff6c17640 1 -- 172.21.15.153:0/44301446 wait complete. 2024-09-15T03:29:52.210 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:29:52.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:29:52.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:29:52.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.303+0000 7fbaed39d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.303+0000 7fbaed39d640 10 monclient: build_initial_monmap 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.303+0000 7fbaed39d640 10 monclient: monmap: 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:52.304492+0000 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:52.304492+0000 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:52.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 Processor -- start 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 -- start start 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient: init 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): picked mon.noname-b con 0x7fbae8131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbae809fda0 0x7fbae8132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): picked mon.noname-a con 0x7fbae809fda0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbae809e9c0 0x7fbae809ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): picked mon.noname-c con 0x7fbae809e9c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae80a1180 con 0x7fbae809fda0 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae80a1460 con 0x7fbae809e9c0 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae809f340 con 0x7fbae8131980 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbaed39d640 10 monclient(hunting): authenticate will time out at 789.943813s 2024-09-15T03:29:52.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae67fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbae809fda0 0x7fbae8132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae8131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40832/0 (socket says 172.21.15.153:40832) 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae77fe640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fbae809e9c0 msgr2=0x7fbae809ee00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae67fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbae809fda0 0x7fbae8132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35194/0 (socket says 172.21.15.153:35194) 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 1 -- 172.21.15.153:0/2585900034 learned_addr learned my addr 172.21.15.153:0/2585900034 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 10 monclient(hunting): get_auth_request con 0x7fbae8131980 auth_method 0 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae67fc640 10 monclient(hunting): get_auth_request con 0x7fbae809fda0 auth_method 0 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.304+0000 7fbae77fe640 1 --2- 172.21.15.153:0/2585900034 >> v2:172.21.15.181:3300/0 conn(0x7fbae809e9c0 0x7fbae809ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient(hunting): handle_auth_done global_id 34179 payload 274 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 1 -- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 msgr2=0x7fbae8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 1 --2- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 1 -- 172.21.15.153:0/2585900034 >> v2:172.21.15.181:3300/0 conn(0x7fbae809e9c0 msgr2=0x7fbae809ee00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 1 --2- 172.21.15.153:0/2585900034 >> v2:172.21.15.181:3300/0 conn(0x7fbae809e9c0 0x7fbae809ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient: _finish_hunting 0 2024-09-15T03:29:52.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 1 monclient: found mon.noname-a 2024-09-15T03:29:52.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:52.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae67fc640 1 -- 172.21.15.153:0/2585900034 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae809f4b0 con 0x7fbae809fda0 2024-09-15T03:29:52.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.305+0000 7fbae6ffd640 1 --2- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae8131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae67fc640 1 -- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 msgr2=0x7fbae8132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae67fc640 1 --2- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae8132290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbae80a0170 tx=0x7fbadc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9bbcc0009ea5c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 1 -- 172.21.15.153:0/2585900034 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc02ae10 con 0x7fbae809fda0 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: dump: 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 5 monclient: authenticate success, global_id 34179 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 1 -- 172.21.15.153:0/2585900034 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc032440 con 0x7fbae809fda0 2024-09-15T03:29:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 4 set_mon_vals no callback set 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 1 -- 172.21.15.153:0/2585900034 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc032820 con 0x7fbae809fda0 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbae5ffb640 10 monclient: dump: 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 10 monclient: shutdown 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 -- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 msgr2=0x7fbae8132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 --2- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae8132290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbae80a0170 tx=0x7fbadc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 -- 172.21.15.153:0/2585900034 shutdown_connections 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 --2- 172.21.15.153:0/2585900034 >> v2:172.21.15.181:3300/0 conn(0x7fbae809e9c0 0x7fbae809ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 --2- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae8132290 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 --2- 172.21.15.153:0/2585900034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae8131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 -- 172.21.15.153:0/2585900034 >> 172.21.15.153:0/2585900034 conn(0x7fbae8067bd0 msgr2=0x7fbae80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 -- 172.21.15.153:0/2585900034 shutdown_connections 2024-09-15T03:29:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.308+0000 7fbaed39d640 1 -- 172.21.15.153:0/2585900034 wait complete. 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.309+0000 7fbaed39d640 10 monclient: build_initial_monmap 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.309+0000 7fbaed39d640 10 monclient: monmap: 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.309+0000 7fbaed39d640 1 Processor -- start 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.309+0000 7fbaed39d640 1 -- start start 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.309+0000 7fbaed39d640 10 monclient: init 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae809e9c0 0x7fbae809a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): picked mon.noname-b con 0x7fbae809e9c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae81309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): picked mon.noname-a con 0x7fbae809fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae811b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): picked mon.noname-c con 0x7fbae8131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae811dae0 con 0x7fbae809fda0 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae811ddc0 con 0x7fbae809e9c0 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae811e0a0 con 0x7fbae8131980 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbaed39d640 10 monclient(hunting): authenticate will time out at 789.949224s 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae81309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae811b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae6ffd640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae809e9c0 msgr2=0x7fbae809a440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae81309c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35202/0 (socket says 172.21.15.153:35202) 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 -- 172.21.15.153:0/92468144 learned_addr learned my addr 172.21.15.153:0/92468144 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): get_auth_request con 0x7fbae809fda0 auth_method 0 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae6ffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae809e9c0 0x7fbae809a440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): get_auth_request con 0x7fbae8131980 auth_method 0 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient(hunting): handle_auth_done global_id 34182 payload 824 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 msgr2=0x7fbae811b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbae8131980 0x7fbae811b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae809e9c0 msgr2=0x7fbae809a440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae809e9c0 0x7fbae809a440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient: _finish_hunting 0 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 monclient: found mon.noname-a 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.310+0000 7fbae67fc640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009450 con 0x7fbae809fda0 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.312+0000 7fbae67fc640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 msgr2=0x7fbae81309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.312+0000 7fbae67fc640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae81309c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009880 tx=0x7fbadc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.312+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc031c70 con 0x7fbae809fda0 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.312+0000 7fbaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.312+0000 7fbaceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.312+0000 7fbaceffd640 10 monclient: dump: 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 10 monclient: _finish_auth 0 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc031dd0 con 0x7fbae809fda0 2024-09-15T03:29:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 5 monclient: authenticate success, global_id 34182 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 10 monclient: _renew_subs 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc009390 con 0x7fbae809fda0 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc03c650 con 0x7fbae809fda0 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 10 monclient: dump: 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbae57fa640 4 set_mon_vals no callback set 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 10 monclient: _renew_subs 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae8141eb0 con 0x7fbae809fda0 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 0x7fbae811bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae811e260 con 0x7fbae8130f00 2024-09-15T03:29:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.313+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbadc03ca80 con 0x7fbae809fda0 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 0x7fbae811bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient: get_auth_request con 0x7fbae8130f00 auth_method 0 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbaceffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbad40386f0 0x7fbad403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fbadc06c0d0 con 0x7fbae809fda0 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_done global_id 34185 payload 824 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae6ffd640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbae804bf60 con 0x7fbae8130f00 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae77fe640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbad40386f0 0x7fbad403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.314+0000 7fbae77fe640 10 monclient: get_auth_request con 0x7fbad40386f0 auth_method 0 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbae6ffd640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 msgr2=0x7fbae811bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbae6ffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 0x7fbae811bbf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006c70 tx=0x7fbad80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad800cd80 con 0x7fbae8130f00 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 10 monclient: dump: 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.332 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbad8013470 con 0x7fbae8130f00 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 msgr2=0x7fbae811bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 0x7fbae811bbf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006c70 tx=0x7fbad80091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbae77fe640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbad40386f0 0x7fbad403aba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbae0003c70 tx=0x7fbae00096a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 msgr2=0x7fbae811bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.316+0000 7fbaceffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae8130f00 0x7fbae811bbf0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006c70 tx=0x7fbad80091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbaed39d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbaed39d640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 0x7fbae8096da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbaed39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae8065e00 con 0x7fbae811c130 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 0x7fbae8096da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient: get_auth_request con 0x7fbae811c130 auth_method 0 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 10 monclient(hunting): handle_auth_done global_id 34188 payload 824 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.322+0000 7fbae6ffd640 1 -- 172.21.15.153:0/92468144 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbae811c530 con 0x7fbae811c130 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbae6ffd640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 msgr2=0x7fbae8096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbae6ffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 0x7fbae8096da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbad8019000 tx=0x7fbad8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8023070 con 0x7fbae811c130 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 10 monclient: dump: 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fbad8006250 con 0x7fbae811c130 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 msgr2=0x7fbae8096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 0x7fbae8096da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbad8019000 tx=0x7fbad8017000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 msgr2=0x7fbae8096da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaceffd640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae811c130 0x7fbae8096da0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbad8019000 tx=0x7fbad8017000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:52.338 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:52.339 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:52.340 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:52.342 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:52.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.344 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:52.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:52.346 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:52.348 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.349 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.351 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:52.351 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbad40386f0 msgr2=0x7fbad403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbad40386f0 0x7fbad403aba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbae0003c70 tx=0x7fbae00096a0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 10 monclient: shutdown 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 msgr2=0x7fbae81309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 --2- 172.21.15.153:0/92468144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbae809fda0 0x7fbae81309c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009880 tx=0x7fbadc032000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbae77fe640 1 -- 172.21.15.153:0/92468144 reap_dead start 2024-09-15T03:29:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbae77fe640 1 -- 172.21.15.153:0/92468144 reap_dead start 2024-09-15T03:29:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 shutdown_connections 2024-09-15T03:29:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 >> 172.21.15.153:0/92468144 conn(0x7fbae8067bd0 msgr2=0x7fbae80a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 shutdown_connections 2024-09-15T03:29:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.325+0000 7fbaed39d640 1 -- 172.21.15.153:0/92468144 wait complete. 2024-09-15T03:29:52.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:29:52.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:29:52.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.447+0000 7fa4b1d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.447+0000 7fa4b1d5d640 10 monclient: build_initial_monmap 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.447+0000 7fa4b1d5d640 10 monclient: monmap: 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:29:52.447833+0000 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:29:52.447833+0000 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:52.448 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:52.449 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.447+0000 7fa4b1d5d640 1 Processor -- start 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.447+0000 7fa4b1d5d640 1 -- start start 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.447+0000 7fa4b1d5d640 10 monclient: init 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa4ac067820 0x7fa4ac09cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4ac067820 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4ac09d060 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa4ac09e0c0 0x7fa4ac1377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa4ac09e0c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac09d460 con 0x7fa4ac09d060 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac09f420 con 0x7fa4ac09e0c0 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac067bf0 con 0x7fa4ac067820 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4b1d5d640 10 monclient(hunting): authenticate will time out at 790.087393s 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4ab7fe640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fa4ac067820 msgr2=0x7fa4ac09cb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4aaffd640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fa4ac09d060 legacy=0x7fa4ac09d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4ab7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa4ac067820 0x7fa4ac09cb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa4ac09e0c0 0x7fa4ac1377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa4ac09e0c0 0x7fa4ac1377d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35230/0 (socket says 172.21.15.153:35230) 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 -- 172.21.15.153:0/2200098206 learned_addr learned my addr 172.21.15.153:0/2200098206 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): get_auth_request con 0x7fa4ac09e0c0 auth_method 0 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient(hunting): handle_auth_done global_id 34191 payload 274 2024-09-15T03:29:52.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 -- 172.21.15.153:0/2200098206 >> v2:172.21.15.181:3300/0 conn(0x7fa4ac067820 msgr2=0x7fa4ac09cb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 --2- 172.21.15.153:0/2200098206 >> v2:172.21.15.181:3300/0 conn(0x7fa4ac067820 0x7fa4ac09cb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 -- 172.21.15.153:0/2200098206 >> v1:172.21.15.181:6789/0 conn(0x7fa4ac09d060 legacy=0x7fa4ac09d9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 monclient: found mon.noname-a 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.448+0000 7fa4abfff640 1 -- 172.21.15.153:0/2200098206 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac09af80 con 0x7fa4ac09e0c0 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.451+0000 7fa4abfff640 1 -- 172.21.15.153:0/2200098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09e0c0 msgr2=0x7fa4ac1377d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.451+0000 7fa4abfff640 1 --2- 172.21.15.153:0/2200098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09e0c0 0x7fa4ac1377d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009f20 tx=0x7fa4a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655d0d5deabea764 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 1 -- 172.21.15.153:0/2200098206 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4007610 con 0x7fa4ac09e0c0 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: dump: 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 1 -- 172.21.15.153:0/2200098206 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a402d050 con 0x7fa4ac09e0c0 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 5 monclient: authenticate success, global_id 34191 2024-09-15T03:29:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 4 set_mon_vals no callback set 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 1 -- 172.21.15.153:0/2200098206 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40079a0 con 0x7fa4ac09e0c0 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4aa7fc640 10 monclient: dump: 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 10 monclient: shutdown 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2200098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09e0c0 msgr2=0x7fa4ac1377d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2200098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09e0c0 0x7fa4ac1377d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009f20 tx=0x7fa4a4009840 comp rx=0 tx=0).stop 2024-09-15T03:29:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2200098206 shutdown_connections 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2200098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09e0c0 0x7fa4ac1377d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2200098206 >> v2:172.21.15.181:3300/0 conn(0x7fa4ac067820 0x7fa4ac09cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2200098206 >> 172.21.15.153:0/2200098206 conn(0x7fa4ac0a3a40 msgr2=0x7fa4ac067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2200098206 shutdown_connections 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.452+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2200098206 wait complete. 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.453+0000 7fa4b1d5d640 10 monclient: build_initial_monmap 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.453+0000 7fa4b1d5d640 10 monclient: monmap: 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:29:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.453+0000 7fa4b1d5d640 1 Processor -- start 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.453+0000 7fa4b1d5d640 1 -- start start 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.453+0000 7fa4b1d5d640 10 monclient: init 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4ac067820 0x7fa4ac13ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4ac067820 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 0x7fa4ac12f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa4ac09d060 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac09e0c0 0x7fa4ac119620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4ac09e0c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac098d00 con 0x7fa4ac09d060 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac11cda0 con 0x7fa4ac067820 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac13fbc0 con 0x7fa4ac09e0c0 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4b1d5d640 10 monclient(hunting): authenticate will time out at 790.093133s 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 0x7fa4ac12f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac09e0c0 0x7fa4ac119620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4ab7fe640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4ac067820 msgr2=0x7fa4ac13ed50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 0x7fa4ac12f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35236/0 (socket says 172.21.15.153:35236) 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 -- 172.21.15.153:0/2535222796 learned_addr learned my addr 172.21.15.153:0/2535222796 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): get_auth_request con 0x7fa4ac09d060 auth_method 0 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): get_auth_request con 0x7fa4ac09e0c0 auth_method 0 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4ab7fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4ac067820 0x7fa4ac13ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient(hunting): handle_auth_done global_id 34194 payload 824 2024-09-15T03:29:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac09e0c0 msgr2=0x7fa4ac119620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac09e0c0 0x7fa4ac119620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4ac067820 msgr2=0x7fa4ac13ed50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4ac067820 0x7fa4ac13ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:29:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 monclient: found mon.noname-a 2024-09-15T03:29:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.454+0000 7fa4aaffd640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4009ae0 con 0x7fa4ac09d060 2024-09-15T03:29:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4aaffd640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 msgr2=0x7fa4ac12f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4aaffd640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 0x7fa4ac12f9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000b5c0 tx=0x7fa4a0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0003cc0 con 0x7fa4ac09d060 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: dump: 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a0003e20 con 0x7fa4ac09d060 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 5 monclient: authenticate success, global_id 34194 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 10 monclient: _renew_subs 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0005340 con 0x7fa4ac09d060 2024-09-15T03:29:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac11cac0 con 0x7fa4ac09d060 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4937fe640 10 monclient: dump: 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4a9ffb640 4 set_mon_vals no callback set 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 10 monclient: _renew_subs 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.456+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac13f290 con 0x7fa4ac09d060 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4b1d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 0x7fa4ac119b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac065e00 con 0x7fa4ac12fee0 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa4a001a400 con 0x7fa4ac09d060 2024-09-15T03:29:52.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 0x7fa4ac119b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient: get_auth_request con 0x7fa4ac12fee0 auth_method 0 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.457+0000 7fa4937fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4980386f0 0x7fa49803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.458+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fa4a004b320 con 0x7fa4ac09d060 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.458+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.458+0000 7fa4ab7fe640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4ac11cf60 con 0x7fa4ac12fee0 2024-09-15T03:29:52.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.458+0000 7fa4abfff640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4980386f0 0x7fa49803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.458+0000 7fa4abfff640 10 monclient: get_auth_request con 0x7fa4980386f0 auth_method 0 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4ab7fe640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 msgr2=0x7fa4ac119b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4ab7fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 0x7fa4ac119b60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00b480 tx=0x7fa49c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0050f0 con 0x7fa4ac12fee0 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 10 monclient: dump: 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.481 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa49c005250 con 0x7fa4ac12fee0 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 msgr2=0x7fa4ac119b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 0x7fa4ac119b60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00b480 tx=0x7fa49c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 msgr2=0x7fa4ac119b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:52.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4937fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac12fee0 0x7fa4ac119b60 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00b480 tx=0x7fa49c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:29:52.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.460+0000 7fa4abfff640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4980386f0 0x7fa49803aba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac09bfc0 tx=0x7fa4a4036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4b1d5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 0x7fa4ac097410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4b1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1302b0 con 0x7fa4ac11a0a0 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 0x7fa4ac097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient: get_auth_request con 0x7fa4ac11a0a0 auth_method 0 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:52.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 10 monclient(hunting): handle_auth_done global_id 4496 payload 824 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.466+0000 7fa4ab7fe640 1 -- 172.21.15.153:0/2535222796 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa4ac11a4a0 con 0x7fa4ac11a0a0 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.468+0000 7fa4ab7fe640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 msgr2=0x7fa4ac097410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.468+0000 7fa4ab7fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 0x7fa4ac097410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007fd0 tx=0x7fa49c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.468+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0059e0 con 0x7fa4ac11a0a0 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 10 monclient: dump: 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:52.486 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fa49c003cb0 con 0x7fa4ac11a0a0 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 msgr2=0x7fa4ac097410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 0x7fa4ac097410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007fd0 tx=0x7fa49c015000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 msgr2=0x7fa4ac097410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:29:52.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4937fe640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ac11a0a0 0x7fa4ac097410 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007fd0 tx=0x7fa49c015000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.488 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:29:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:29:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:29:52.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:29:52.492 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:29:52.493 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.494 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:52.495 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.496 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:29:52.497 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:29:52.498 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:52.498 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:52.498 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.500 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.500 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:29:52.500 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:29:52.501 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.502 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.502 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:29:52.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:29:52.504 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.504 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.504 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.505 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:29:52.507 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:29:52.508 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4980386f0 msgr2=0x7fa49803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4980386f0 0x7fa49803aba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac09bfc0 tx=0x7fa4a4036000 comp rx=0 tx=0).stop 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 10 monclient: shutdown 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 msgr2=0x7fa4ac12f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 --2- 172.21.15.153:0/2535222796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ac09d060 0x7fa4ac12f9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000b5c0 tx=0x7fa4a0009120 comp rx=0 tx=0).stop 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4abfff640 1 -- 172.21.15.153:0/2535222796 reap_dead start 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4abfff640 1 -- 172.21.15.153:0/2535222796 reap_dead start 2024-09-15T03:29:52.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 shutdown_connections 2024-09-15T03:29:52.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 >> 172.21.15.153:0/2535222796 conn(0x7fa4ac0a3a40 msgr2=0x7fa4ac132920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:52.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 shutdown_connections 2024-09-15T03:29:52.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:29:52.469+0000 7fa4b1d5d640 1 -- 172.21.15.153:0/2535222796 wait complete. 2024-09-15T03:29:52.511 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:29:52.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:29:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:52.505+0000 7f0426ffd640 10 monclient: tick 2024-09-15T03:29:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:52.505+0000 7f0426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:52.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:52.956+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v18) v1 ==== 152+0+0 (secure 0 0 0) 0x7f043003f070 con 0x7f044009cad0 2024-09-15T03:29:52.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:52.957+0000 7f0427fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T03:29:52.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:52.957+0000 7f0427fff640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-15T03:29:53.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.053+0000 7f044655d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T03:29:53.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.053+0000 7f044655d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T03:29:53.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.053+0000 7f044655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:53.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.053+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f044011f360 con 0x7f044009cad0 2024-09-15T03:29:53.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.054+0000 7f0427fff640 1 -- 172.21.15.181:0/1099641830 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v18) v1 ==== 159+0+0 (secure 0 0 0) 0x7f0430037030 con 0x7f044009cad0 2024-09-15T03:29:53.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.054+0000 7f0427fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-15T03:29:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.054+0000 7f0427fff640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-15T03:29:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool1234' already exists 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f042c038740 msgr2=0x7f042c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f042c038740 0x7f042c03abf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0434003940 tx=0x7f0434006d80 comp rx=0 tx=0).stop 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 10 monclient: shutdown 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 msgr2=0x7f044013f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 0x7f044013f060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0430009850 tx=0x7f0430032000 comp rx=0 tx=0).stop 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 shutdown_connections 2024-09-15T03:29:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f042c038740 0x7f042c03abf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044009d2a0 0x7f044011b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044009cea0 0x7f0440131dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.056+0000 7f044655d640 1 --2- 172.21.15.181:0/1099641830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f044009cad0 0x7f044013f060 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.057+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 >> 172.21.15.181:0/1099641830 conn(0x7f0440070e90 msgr2=0x7f04400a0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:53.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.057+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 shutdown_connections 2024-09-15T03:29:53.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.057+0000 7f044655d640 1 -- 172.21.15.181:0/1099641830 wait complete. 2024-09-15T03:29:53.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-15T03:29:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.142+0000 7fba16b1d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.142+0000 7fba16b1d640 10 monclient: build_initial_monmap 2024-09-15T03:29:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.142+0000 7fba16b1d640 10 monclient: monmap: 2024-09-15T03:29:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:53.142818+0000 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:53.142818+0000 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:53.144 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:53.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.142+0000 7fba16b1d640 1 Processor -- start 2024-09-15T03:29:53.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.142+0000 7fba16b1d640 1 -- start start 2024-09-15T03:29:53.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.142+0000 7fba16b1d640 10 monclient: init 2024-09-15T03:29:53.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:53.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 0x7fba10131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fba10131ac0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fba1009ff50 0x7fba101323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fba1009ff50 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba10098150 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100a1330 con 0x7fba10098150 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba10098c30 con 0x7fba1009ff50 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba10139e10 con 0x7fba10131ac0 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba16b1d640 10 monclient(hunting): authenticate will time out at 797.624253s 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 0x7fba10131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:53.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fba1009ff50 0x7fba101323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 0x7fba10131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38944/0 (socket says 172.21.15.181:38944) 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fba1009ff50 0x7fba101323d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34642/0 (socket says 172.21.15.181:34642) 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 1 -- 172.21.15.181:0/3617467002 learned_addr learned my addr 172.21.15.181:0/3617467002 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): get_auth_request con 0x7fba10131ac0 auth_method 0 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): get_auth_request con 0x7fba1009ff50 auth_method 0 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:53.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient(hunting): handle_auth_done global_id 34197 payload 274 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 -- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 msgr2=0x7fba10131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 --2- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 0x7fba10131e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 -- 172.21.15.181:0/3617467002 >> v1:172.21.15.153:6789/0 conn(0x7fba10098150 legacy=0x7fba10098520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:53.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 monclient: found mon.noname-a 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba0ffff640 1 -- 172.21.15.181:0/3617467002 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba10139f50 con 0x7fba1009ff50 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.143+0000 7fba148b9640 1 --2- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 0x7fba10131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.146+0000 7fba0ffff640 1 -- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1009ff50 msgr2=0x7fba101323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.146+0000 7fba0ffff640 1 --2- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1009ff50 0x7fba101323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba04009850 tx=0x7fba040090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9cf4f9e2bc1166 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 1 -- 172.21.15.181:0/3617467002 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0402adf0 con 0x7fba1009ff50 2024-09-15T03:29:53.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: dump: 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:53.154 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:53.154 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:53.154 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:53.154 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:53.154 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:53.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 1 -- 172.21.15.181:0/3617467002 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba04032440 con 0x7fba1009ff50 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba16b1d640 5 monclient: authenticate success, global_id 34197 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba16b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba16b1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:53.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba16b1d640 4 set_mon_vals no callback set 2024-09-15T03:29:53.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 1 -- 172.21.15.181:0/3617467002 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba040327e0 con 0x7fba1009ff50 2024-09-15T03:29:53.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:53.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:53.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.147+0000 7fba0f7fe640 10 monclient: dump: 2024-09-15T03:29:53.156 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:53.156 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:53.158 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:53.158 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:53.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 10 monclient: shutdown 2024-09-15T03:29:53.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:53.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 -- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1009ff50 msgr2=0x7fba101323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:53.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 --2- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1009ff50 0x7fba101323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba04009850 tx=0x7fba040090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 -- 172.21.15.181:0/3617467002 shutdown_connections 2024-09-15T03:29:53.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 --2- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1009ff50 0x7fba101323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 --2- 172.21.15.181:0/3617467002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba10131ac0 0x7fba10131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 -- 172.21.15.181:0/3617467002 >> 172.21.15.181:0/3617467002 conn(0x7fba100a5190 msgr2=0x7fba100a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 -- 172.21.15.181:0/3617467002 shutdown_connections 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 -- 172.21.15.181:0/3617467002 wait complete. 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 10 monclient: build_initial_monmap 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 10 monclient: monmap: 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:53.171 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.148+0000 7fba16b1d640 1 Processor -- start 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 -- start start 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient: init 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10098150 0x7fba10137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fba10098150 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 0x7fba10133950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fba1009ff50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:53.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba10131ac0 0x7fba10133e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fba10131ac0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba101383d0 con 0x7fba10098150 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10135980 con 0x7fba10131ac0 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba10069810 con 0x7fba1009ff50 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba16b1d640 10 monclient(hunting): authenticate will time out at 797.630534s 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba150ba640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba10131ac0 msgr2=0x7fba10133e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:53.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba150ba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba10131ac0 0x7fba10133e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 0x7fba10133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10098150 0x7fba10137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 0x7fba10133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38960/0 (socket says 172.21.15.181:38960) 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 -- 172.21.15.181:0/664205833 learned_addr learned my addr 172.21.15.181:0/664205833 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): get_auth_request con 0x7fba1009ff50 auth_method 0 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): get_auth_request con 0x7fba10098150 auth_method 0 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:53.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba148b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 -- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba10131ac0 msgr2=0x7fba10133e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba10131ac0 0x7fba10133e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 -- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10098150 msgr2=0x7fba10137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10098150 0x7fba10137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:29:53.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.149+0000 7fba0ffff640 1 monclient: found mon.noname-c 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.150+0000 7fba0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.150+0000 7fba0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.150+0000 7fba0ffff640 1 -- 172.21.15.181:0/664205833 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08009ae0 con 0x7fba1009ff50 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.150+0000 7fba148b9640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10098150 0x7fba10137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0ffff640 1 -- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 msgr2=0x7fba10133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0ffff640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 0x7fba10133950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba04009360 tx=0x7fba04032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04031a10 con 0x7fba1009ff50 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:53.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: dump: 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:53.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: _finish_auth 0 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba16b1d640 5 monclient: authenticate success, global_id 4502 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba04031b70 con 0x7fba1009ff50 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0403c400 con 0x7fba1009ff50 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0cff9640 10 monclient: dump: 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba0effd640 4 set_mon_vals no callback set 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba16b1d640 10 monclient: _renew_subs 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba16b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.152+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba04009390 con 0x7fba1009ff50 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 10 monclient: _renew_subs 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba10136f30 con 0x7fba1009ff50 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.153+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1004bf60 con 0x7fba1009ff50 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.154+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fba0403c830 con 0x7fba1009ff50 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.154+0000 7fba0cff9640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb9fc038740 0x7fb9fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.154+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4693+0+0 (secure 0 0 0) 0x7fba0406c0a0 con 0x7fba1009ff50 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.154+0000 7fba148b9640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb9fc038740 0x7fb9fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.154+0000 7fba148b9640 10 monclient: get_auth_request con 0x7fb9fc038740 auth_method 0 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.156+0000 7fba148b9640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb9fc038740 0x7fb9fc03abf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba080072d0 tx=0x7fba0800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.158+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fba0409de40 con 0x7fba1009ff50 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.158+0000 7fba0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.158+0000 7fba0cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.313+0000 7fba16b1d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T03:29:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.313+0000 7fba16b1d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T03:29:53.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.313+0000 7fba16b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:53.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.313+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7fba10072640 con 0x7fba1009ff50 2024-09-15T03:29:53.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.962+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v19) v1 ==== 143+0+0 (secure 0 0 0) 0x7fba040370b0 con 0x7fba1009ff50 2024-09-15T03:29:53.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.962+0000 7fba0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T03:29:53.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:53.962+0000 7fba0cff9640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-15T03:29:54.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.055+0000 7fba16b1d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T03:29:54.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.055+0000 7fba16b1d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T03:29:54.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.055+0000 7fba16b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:54.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.055+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7fba10072640 con 0x7fba1009ff50 2024-09-15T03:29:54.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.966+0000 7fba0cff9640 1 -- 172.21.15.181:0/664205833 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fba04038d20 con 0x7fba1009ff50 2024-09-15T03:29:54.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.966+0000 7fba0cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-15T03:29:54.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.966+0000 7fba0cff9640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-15T03:29:54.967 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 3 pg_num_min to 2 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb9fc038740 msgr2=0x7fb9fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb9fc038740 0x7fb9fc03abf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba080072d0 tx=0x7fba0800b040 comp rx=0 tx=0).stop 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 10 monclient: shutdown 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 msgr2=0x7fba10133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 0x7fba10133950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba04009360 tx=0x7fba04032000 comp rx=0 tx=0).stop 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 shutdown_connections 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb9fc038740 0x7fb9fc03abf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba10131ac0 0x7fba10133e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1009ff50 0x7fba10133950 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:54.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 --2- 172.21.15.181:0/664205833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10098150 0x7fba10137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:54.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.968+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 >> 172.21.15.181:0/664205833 conn(0x7fba100a5190 msgr2=0x7fba100a3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:54.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.969+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 shutdown_connections 2024-09-15T03:29:54.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:54.969+0000 7fba16b1d640 1 -- 172.21.15.181:0/664205833 wait complete. 2024-09-15T03:29:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-15T03:29:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 2 2024-09-15T03:29:55.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.059+0000 7fbc9c79d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:55.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.059+0000 7fbc9c79d640 10 monclient: build_initial_monmap 2024-09-15T03:29:55.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.059+0000 7fbc9c79d640 10 monclient: monmap: 2024-09-15T03:29:55.060 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:55.059966+0000 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:55.059966+0000 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.059+0000 7fbc9c79d640 1 Processor -- start 2024-09-15T03:29:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.059+0000 7fbc9c79d640 1 -- start start 2024-09-15T03:29:55.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.059+0000 7fbc9c79d640 10 monclient: init 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc9409ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc9409f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 0x7fbc9413a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc94069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc94067420 con 0x7fbc9409f020 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9409fdc0 con 0x7fbc9409ec50 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9406a170 con 0x7fbc94069d10 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9c79d640 10 monclient(hunting): authenticate will time out at 799.541539s 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9a539640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fbc9409ec50 legacy=0x7fbc94131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc99d38640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbc9409f020 0x7fbc94136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:44878/0 (socket says 172.21.15.181:44878) 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc99d38640 1 -- 172.21.15.181:0/2114791537 learned_addr learned my addr 172.21.15.181:0/2114791537 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9ad3a640 1 --2- 172.21.15.181:0/2114791537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 0x7fbc9413a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9ad3a640 10 monclient(hunting): get_auth_request con 0x7fbc94069d10 auth_method 0 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9ad3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9ad3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.060+0000 7fbc9ad3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_done global_id 4505 payload 274 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/2114791537 >> v1:172.21.15.181:6789/0 conn(0x7fbc9409ec50 legacy=0x7fbc94131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/2114791537 >> v1:172.21.15.153:6789/0 conn(0x7fbc9409f020 legacy=0x7fbc94136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 10 monclient: _finish_hunting 0 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 1 monclient: found mon.noname-b 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.061+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/2114791537 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc940a0510 con 0x7fbc94069d10 2024-09-15T03:29:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/2114791537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 msgr2=0x7fbc9413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9ad3a640 1 --2- 172.21.15.181:0/2114791537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 0x7fbc9413a490 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009e90 tx=0x7fbc8c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12e80ab4348dcab9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 1 -- 172.21.15.181:0/2114791537 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c007840 con 0x7fbc94069d10 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: dump: 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: _finish_auth 0 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 1 -- 172.21.15.181:0/2114791537 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8c02d050 con 0x7fbc94069d10 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 1 -- 172.21.15.181:0/2114791537 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c007bd0 con 0x7fbc94069d10 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc99537640 10 monclient: dump: 2024-09-15T03:29:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 5 monclient: authenticate success, global_id 4505 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 4 set_mon_vals no callback set 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 10 monclient: shutdown 2024-09-15T03:29:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 1 -- 172.21.15.181:0/2114791537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 msgr2=0x7fbc9413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/2114791537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 0x7fbc9413a490 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009e90 tx=0x7fbc8c009780 comp rx=0 tx=0).stop 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 1 -- 172.21.15.181:0/2114791537 shutdown_connections 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.064+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/2114791537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc94069d10 0x7fbc9413a490 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 1 -- 172.21.15.181:0/2114791537 >> 172.21.15.181:0/2114791537 conn(0x7fbc940a5d40 msgr2=0x7fbc940a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 1 -- 172.21.15.181:0/2114791537 shutdown_connections 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 1 -- 172.21.15.181:0/2114791537 wait complete. 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 10 monclient: build_initial_monmap 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 10 monclient: monmap: 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 1 Processor -- start 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 1 -- start start 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.065+0000 7fbc9c79d640 10 monclient: init 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:29:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc94069d10 0x7fbc9409a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc94069d10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc9409ec50 0x7fbc94130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9409ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 0x7fbc9411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9411e330 con 0x7fbc94069d10 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9411da60 con 0x7fbc9409ec50 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9411dd40 con 0x7fbc9409f020 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:55.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9c79d640 10 monclient(hunting): authenticate will time out at 799.547267s 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc99d38640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc9409ec50 msgr2=0x7fbc94130940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc99d38640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc9409ec50 0x7fbc94130940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc94069d10 0x7fbc9409a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 0x7fbc9411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc94069d10 0x7fbc9409a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34662/0 (socket says 172.21.15.181:34662) 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 0x7fbc9411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38978/0 (socket says 172.21.15.181:38978) 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 1 -- 172.21.15.181:0/3056725150 learned_addr learned my addr 172.21.15.181:0/3056725150 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): get_auth_request con 0x7fbc94069d10 auth_method 0 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): get_auth_request con 0x7fbc9409f020 auth_method 0 2024-09-15T03:29:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9a539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc9409ec50 msgr2=0x7fbc94130940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc9409ec50 0x7fbc94130940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc94069d10 msgr2=0x7fbc9409a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc94069d10 0x7fbc9409a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient: _finish_hunting 0 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 monclient: found mon.noname-c 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.066+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/3056725150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c009a20 con 0x7fbc9409f020 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9ad3a640 1 -- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 msgr2=0x7fbc9411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9ad3a640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 0x7fbc9411b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc94071a00 tx=0x7fbc8c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c031c00 con 0x7fbc9409f020 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: dump: 2024-09-15T03:29:55.081 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8c040070 con 0x7fbc9409f020 2024-09-15T03:29:55.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c045430 con 0x7fbc9409f020 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc8a7fc640 10 monclient: dump: 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.083 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 5 monclient: authenticate success, global_id 4508 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 10 monclient: _renew_subs 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc98d36640 4 set_mon_vals no callback set 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9411e020 con 0x7fbc9409f020 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 10 monclient: _renew_subs 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.069+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9409b240 con 0x7fbc9409f020 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbc8c042030 con 0x7fbc9409f020 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc9c79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc9c79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc9c79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc940008d0 con 0x7fbc9409f020 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc8a7fc640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc780386f0 0x7fbc7803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4709+0+0 (secure 0 0 0) 0x7fbc8c06d4a0 con 0x7fbc9409f020 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc9a539640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc780386f0 0x7fbc7803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.070+0000 7fbc9a539640 10 monclient: get_auth_request con 0x7fbc780386f0 auth_method 0 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.072+0000 7fbc9a539640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc780386f0 0x7fbc7803aba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc90003850 tx=0x7fbc90009120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.075+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbc8c045860 con 0x7fbc9409f020 2024-09-15T03:29:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.075+0000 7fbc8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:55.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.075+0000 7fbc8a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:55.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.228+0000 7fbc9c79d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:29:55.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.228+0000 7fbc9c79d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:29:55.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.228+0000 7fbc9c79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.228+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fbc940008d0 con 0x7fbc9409f020 2024-09-15T03:29:55.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.229+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v20) v1 ==== 105+0+14 (secure 0 0 0) 0x7fbc8c0500c0 con 0x7fbc9409f020 2024-09-15T03:29:55.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.229+0000 7fbc8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:29:55.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.229+0000 7fbc8a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc9c79d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:29:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc9c79d640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:29:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc9c79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fbc940008d0 con 0x7fbc9409f020 2024-09-15T03:29:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc8a7fc640 1 -- 172.21.15.181:0/3056725150 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v20) v1 ==== 105+0+14 (secure 0 0 0) 0x7fbc8c044bc0 con 0x7fbc9409f020 2024-09-15T03:29:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc8a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:29:55.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.321+0000 7fbc8a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:29:55.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc780386f0 msgr2=0x7fbc7803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc780386f0 0x7fbc7803aba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc90003850 tx=0x7fbc90009120 comp rx=0 tx=0).stop 2024-09-15T03:29:55.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 10 monclient: shutdown 2024-09-15T03:29:55.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:55.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 msgr2=0x7fbc9411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 0x7fbc9411b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc94071a00 tx=0x7fbc8c037000 comp rx=0 tx=0).stop 2024-09-15T03:29:55.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 shutdown_connections 2024-09-15T03:29:55.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc780386f0 0x7fbc7803aba0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc9409f020 0x7fbc9411b760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc9409ec50 0x7fbc94130940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 --2- 172.21.15.181:0/3056725150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc94069d10 0x7fbc9409a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.323+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 >> 172.21.15.181:0/3056725150 conn(0x7fbc940a5d40 msgr2=0x7fbc94134ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:55.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.324+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 shutdown_connections 2024-09-15T03:29:55.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.324+0000 7fbc9c79d640 1 -- 172.21.15.181:0/3056725150 wait complete. 2024-09-15T03:29:55.333 INFO:tasks.workunit.client.0.smithi181.stdout:pg_num_min: 2 2024-09-15T03:29:55.334 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-15T03:29:55.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.408+0000 7f710fc94640 10 monclient: get_monmap_and_config 2024-09-15T03:29:55.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.408+0000 7f710fc94640 10 monclient: build_initial_monmap 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.408+0000 7f710fc94640 10 monclient: monmap: 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:55.409481+0000 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:55.409481+0000 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:55.410 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:55.411 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:55.411 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:55.411 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 1 Processor -- start 2024-09-15T03:29:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 1 -- start start 2024-09-15T03:29:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient: init 2024-09-15T03:29:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:29:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7108098050 0x7f7108132000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient(hunting): picked mon.noname-c con 0x7f7108098050 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f710809fe90 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7108136cf0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71080a1230 con 0x7f710809fe90 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71080a1510 con 0x7f7108136cf0 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7108098b30 con 0x7f7108098050 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710fc94640 10 monclient(hunting): authenticate will time out at 799.890963s 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.409+0000 7f710da30640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f7108098050 msgr2=0x7f7108132000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.410+0000 7f710da30640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7108098050 0x7f7108132000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.410+0000 7f710e231640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f7108136cf0 legacy=0x7f71081390d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.410+0000 7f710d22f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f710809fe90 0x7f7108098420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:44880/0 (socket says 172.21.15.181:44880) 2024-09-15T03:29:55.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.410+0000 7f710d22f640 1 -- 172.21.15.181:0/3782235929 learned_addr learned my addr 172.21.15.181:0/3782235929 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.412+0000 7f710d22f640 1 -- 172.21.15.181:0/3782235929 >> v1:172.21.15.153:6789/0 conn(0x7f710809fe90 legacy=0x7f7108098420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.412+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577732477 0 0) 0x7f71080a1230 con 0x7f710809fe90 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.412+0000 7f710ca2e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.412+0000 7f710ca2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.412+0000 7f710ca2e640 10 monclient(hunting): my global_id is 34206 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.412+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f40034e0 con 0x7f710809fe90 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852446468 0 0) 0x7f70f40034e0 con 0x7f710809fe90 2024-09-15T03:29:55.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 >> v2:172.21.15.181:3300/0 conn(0x7f7108098050 msgr2=0x7f7108132000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 --2- 172.21.15.181:0/3782235929 >> v2:172.21.15.181:3300/0 conn(0x7f7108098050 0x7f7108132000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 >> v1:172.21.15.181:6789/0 conn(0x7f7108136cf0 legacy=0x7f71081390d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: _finish_hunting 0 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7108132540 con 0x7f710809fe90 2024-09-15T03:29:55.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: _finish_auth 0 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f70f8001720 con 0x7f710809fe90 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710fc94640 5 monclient: authenticate success, global_id 34206 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710fc94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: dump: 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.421 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 >> v1:172.21.15.153:6789/0 conn(0x7f710809fe90 legacy=0x7f7108098420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f40039d0 0x7f70f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): picked mon.b con 0x7f70f40039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 0x7f70f4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): picked mon.c con 0x7f70f4004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f4005330 0x7f70f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): picked mon.a con 0x7f70f4005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108098b30 con 0x7f70f4005330 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f400c290 con 0x7f70f40039d0 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f400c450 con 0x7f70f4004780 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710ca2e640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:29:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710fc94640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710da30640 1 -- 172.21.15.181:0/3782235929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f40039d0 msgr2=0x7f70f4003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710da30640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f40039d0 0x7f70f4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f4005330 0x7f70f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 0x7f70f4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): get_auth_request con 0x7f70f4005330 auth_method 0 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): get_auth_request con 0x7f70f4004780 auth_method 0 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:29:55.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient(hunting): handle_auth_done global_id 34206 payload 293 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 -- 172.21.15.181:0/3782235929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f40039d0 msgr2=0x7f70f4003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f40039d0 0x7f70f4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 -- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f4005330 msgr2=0x7f70f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f4005330 0x7f70f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient: _finish_hunting 0 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 monclient: found mon.c 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:55.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 1 -- 172.21.15.181:0/3782235929 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f40034e0 con 0x7f70f4004780 2024-09-15T03:29:55.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 10 monclient: _finish_auth 0 2024-09-15T03:29:55.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710d22f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:55.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f4005330 0x7f70f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:55.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.413+0000 7f710e231640 1 -- 172.21.15.181:0/3782235929 reap_dead start 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.416+0000 7f710d22f640 1 -- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 msgr2=0x7f70f4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.416+0000 7f710d22f640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 0x7f70f4004bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70f8001490 tx=0x7f70f8007a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc30251f88eca7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.416+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8005890 con 0x7f70f4004780 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.416+0000 7f710ca2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.416+0000 7f710ca2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.416+0000 7f710ca2e640 10 monclient: dump: 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f8005a10 con 0x7f70f4004780 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710ca2e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:55.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710ca2e640 1 -- 172.21.15.181:0/3782235929 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8005dd0 con 0x7f70f4004780 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 4 set_mon_vals no callback set 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710ca2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710ca2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710ca2e640 10 monclient: dump: 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.437 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 10 monclient: shutdown 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 -- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 msgr2=0x7f70f4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 0x7f70f4004bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70f8001490 tx=0x7f70f8007a90 comp rx=0 tx=0).stop 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 -- 172.21.15.181:0/3782235929 shutdown_connections 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 --2- 172.21.15.181:0/3782235929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70f4004780 0x7f70f4004bf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 -- 172.21.15.181:0/3782235929 >> 172.21.15.181:0/3782235929 conn(0x7f71080a5190 msgr2=0x7f71080a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:55.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 -- 172.21.15.181:0/3782235929 shutdown_connections 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.417+0000 7f710fc94640 1 -- 172.21.15.181:0/3782235929 wait complete. 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.418+0000 7f710fc94640 10 monclient: build_initial_monmap 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.418+0000 7f710fc94640 10 monclient: monmap: 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:55.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.418+0000 7f710fc94640 1 Processor -- start 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.418+0000 7f710fc94640 1 -- start start 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.418+0000 7f710fc94640 10 monclient: init 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f4004780 0x7f7108134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): picked mon.noname-b con 0x7f70f4004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 0x7f7108134ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): picked mon.noname-c con 0x7f7108134ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71081353e0 0x7f7108147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710da30640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f4004780 msgr2=0x7f7108134590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): picked mon.noname-a con 0x7f71081353e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108098de0 con 0x7f71081353e0 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71080a3040 con 0x7f70f4004780 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710da30640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f4004780 0x7f7108134590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710811ca70 con 0x7f7108134ad0 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f70fdffb640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:29:55.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710fc94640 10 monclient(hunting): authenticate will time out at 799.900109s 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 0x7f7108134ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71081353e0 0x7f7108147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 0x7f7108134ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:39004/0 (socket says 172.21.15.181:39004) 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 -- 172.21.15.181:0/3661101514 learned_addr learned my addr 172.21.15.181:0/3661101514 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): get_auth_request con 0x7f7108134ad0 auth_method 0 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): get_auth_request con 0x7f71081353e0 auth_method 0 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:55.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710e231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 -- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f4004780 msgr2=0x7f7108134590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f4004780 0x7f7108134590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 -- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71081353e0 msgr2=0x7f7108147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71081353e0 0x7f7108147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:55.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient: _finish_hunting 0 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 monclient: found mon.noname-c 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.419+0000 7f710d22f640 1 -- 172.21.15.181:0/3661101514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710811d940 con 0x7f7108134ad0 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.421+0000 7f710d22f640 1 -- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 msgr2=0x7f7108134ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.421+0000 7f710d22f640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 0x7f7108134ea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70f8001490 tx=0x7f70f8034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8007510 con 0x7f7108134ad0 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: dump: 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.447 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: _finish_auth 0 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f8007670 con 0x7f7108134ad0 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 5 monclient: authenticate success, global_id 4511 2024-09-15T03:29:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 10 monclient: _renew_subs 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7108135f70 con 0x7f7108134ad0 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f80396d0 con 0x7f7108134ad0 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fdffb640 10 monclient: dump: 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:55.449 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 10 monclient: _renew_subs 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f70fffff640 4 set_mon_vals no callback set 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.422+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710811b8a0 con 0x7f7108134ad0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f710fc94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f70f8039b00 con 0x7f7108134ad0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f710fc94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f710fc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710804bf60 con 0x7f7108134ad0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f70fdffb640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f70ec038740 0x7f70ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f70f806c960 con 0x7f7108134ad0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f710da30640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f70ec038740 0x7f70ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.423+0000 7f710da30640 10 monclient: get_auth_request con 0x7f70ec038740 auth_method 0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.426+0000 7f710da30640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f70ec038740 0x7f70ec03abf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7104006fd0 tx=0x7f7104006e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.428+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f70f802f420 con 0x7f7108134ad0 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.428+0000 7f70fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.428+0000 7f70fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:55.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.581+0000 7f710fc94640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T03:29:55.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.581+0000 7f710fc94640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T03:29:55.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.581+0000 7f710fc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:55.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.581+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f710811c480 con 0x7f7108134ad0 2024-09-15T03:29:55.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.970+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v21) v1 ==== 145+0+0 (secure 0 0 0) 0x7f70f802e380 con 0x7f7108134ad0 2024-09-15T03:29:55.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.970+0000 7f70fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T03:29:55.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:55.970+0000 7f70fdffb640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-15T03:29:56.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.060+0000 7f710fc94640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T03:29:56.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.060+0000 7f710fc94640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T03:29:56.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.060+0000 7f710fc94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:56.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.060+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f710811c480 con 0x7f7108134ad0 2024-09-15T03:29:56.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.971+0000 7f70fdffb640 1 -- 172.21.15.181:0/3661101514 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v22) v1 ==== 145+0+0 (secure 0 0 0) 0x7f70f802e960 con 0x7f7108134ad0 2024-09-15T03:29:56.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.971+0000 7f70fdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-15T03:29:56.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.971+0000 7f70fdffb640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-15T03:29:56.972 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 3 pg_num_max to 33 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f70ec038740 msgr2=0x7f70ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f70ec038740 0x7f70ec03abf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7104006fd0 tx=0x7f7104006e50 comp rx=0 tx=0).stop 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 10 monclient: shutdown 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 msgr2=0x7f7108134ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 0x7f7108134ea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70f8001490 tx=0x7f70f8034000 comp rx=0 tx=0).stop 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 shutdown_connections 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f70ec038740 0x7f70ec03abf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71081353e0 0x7f7108147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7108134ad0 0x7f7108134ea0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 --2- 172.21.15.181:0/3661101514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f4004780 0x7f7108134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:56.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 >> 172.21.15.181:0/3661101514 conn(0x7f71080a5190 msgr2=0x7f710809e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:56.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 shutdown_connections 2024-09-15T03:29:56.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:56.973+0000 7f710fc94640 1 -- 172.21.15.181:0/3661101514 wait complete. 2024-09-15T03:29:56.986 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-15T03:29:56.986 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 33 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.063+0000 7f8abba57640 10 monclient: get_monmap_and_config 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.063+0000 7f8abba57640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.063+0000 7f8abba57640 10 monclient: monmap: 2024-09-15T03:29:57.064 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:57.063946+0000 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:57.063946+0000 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.063+0000 7f8abba57640 1 Processor -- start 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.063+0000 7f8abba57640 1 -- start start 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient: init 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ab409ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ab409f020 0x7f8ab4069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab409f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 0x7f8ab4139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab4136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab4067420 con 0x7f8ab409ec50 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab409fdc0 con 0x7f8ab409f020 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40a0510 con 0x7f8ab4136d60 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8abba57640 10 monclient(hunting): authenticate will time out at 801.545509s 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab8ff2640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f8ab409f020 msgr2=0x7f8ab4069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab8ff2640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ab409f020 0x7f8ab4069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab9ff4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 0x7f8ab4139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab97f3640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8ab409ec50 0x7f8ab4131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:44882/0 (socket says 172.21.15.181:44882) 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab97f3640 1 -- 172.21.15.181:0/1195948274 learned_addr learned my addr 172.21.15.181:0/1195948274 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab9ff4640 10 monclient(hunting): get_auth_request con 0x7f8ab4136d60 auth_method 0 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab9ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab9ff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.064+0000 7f8ab9ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_done global_id 4514 payload 274 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/1195948274 >> v2:172.21.15.181:3300/0 conn(0x7f8ab409f020 msgr2=0x7f8ab4069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 1 --2- 172.21.15.181:0/1195948274 >> v2:172.21.15.181:3300/0 conn(0x7f8ab409f020 0x7f8ab4069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/1195948274 >> v1:172.21.15.153:6789/0 conn(0x7f8ab409ec50 legacy=0x7f8ab4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 1 monclient: found mon.noname-b 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:57.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.065+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/1195948274 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab40a0200 con 0x7f8ab4136d60 2024-09-15T03:29:57.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/1195948274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 msgr2=0x7f8ab4139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab9ff4640 1 --2- 172.21.15.181:0/1195948274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 0x7f8ab4139140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009e60 tx=0x7f8aac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3b0c4cd45d185f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 1 -- 172.21.15.181:0/1195948274 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac007840 con 0x7f8ab4136d60 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: dump: 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.077 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 1 -- 172.21.15.181:0/1195948274 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac02d050 con 0x7f8ab4136d60 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 1 -- 172.21.15.181:0/1195948274 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac007bd0 con 0x7f8ab4136d60 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8ab3fff640 10 monclient: dump: 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 5 monclient: authenticate success, global_id 4514 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 4 set_mon_vals no callback set 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 10 monclient: shutdown 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 -- 172.21.15.181:0/1195948274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 msgr2=0x7f8ab4139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 --2- 172.21.15.181:0/1195948274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 0x7f8ab4139140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009e60 tx=0x7f8aac009780 comp rx=0 tx=0).stop 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 -- 172.21.15.181:0/1195948274 shutdown_connections 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 --2- 172.21.15.181:0/1195948274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab4136d60 0x7f8ab4139140 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 --2- 172.21.15.181:0/1195948274 >> v2:172.21.15.181:3300/0 conn(0x7f8ab409f020 0x7f8ab4069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 -- 172.21.15.181:0/1195948274 >> 172.21.15.181:0/1195948274 conn(0x7f8ab40a5d40 msgr2=0x7f8ab40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 -- 172.21.15.181:0/1195948274 shutdown_connections 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.068+0000 7f8abba57640 1 -- 172.21.15.181:0/1195948274 wait complete. 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.069+0000 7f8abba57640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.069+0000 7f8abba57640 10 monclient: monmap: 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:57.080 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.069+0000 7f8abba57640 1 Processor -- start 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.069+0000 7f8abba57640 1 -- start start 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.069+0000 7f8abba57640 10 monclient: init 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab409ec50 0x7f8ab40970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab409ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab409f020 0x7f8ab4097620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab409f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 0x7f8ab4097b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab4136d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4131560 con 0x7f8ab4136d60 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab409c000 con 0x7f8ab409f020 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40a4d70 con 0x7f8ab409ec50 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8abba57640 10 monclient(hunting): authenticate will time out at 801.551409s 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab8ff2640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab409f020 msgr2=0x7f8ab4097620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab8ff2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab409f020 0x7f8ab4097620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 0x7f8ab4097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab409ec50 0x7f8ab40970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 0x7f8ab4097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34692/0 (socket says 172.21.15.181:34692) 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/2895475095 learned_addr learned my addr 172.21.15.181:0/2895475095 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): get_auth_request con 0x7f8ab4136d60 auth_method 0 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): get_auth_request con 0x7f8ab409ec50 auth_method 0 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.070+0000 7f8ab97f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 10 monclient(hunting): handle_auth_done global_id 34212 payload 824 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab409ec50 msgr2=0x7f8ab40970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab409ec50 0x7f8ab40970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab409f020 msgr2=0x7f8ab4097620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:57.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab409f020 0x7f8ab4097620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 1 monclient: found mon.noname-a 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/2895475095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac009a20 con 0x7f8ab4136d60 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.071+0000 7f8ab97f3640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab409ec50 0x7f8ab40970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab9ff4640 1 -- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 msgr2=0x7f8ab4097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab9ff4640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 0x7f8ab4097b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009e60 tx=0x7f8aac037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac031cd0 con 0x7f8ab4136d60 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: dump: 2024-09-15T03:29:57.085 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac040070 con 0x7f8ab4136d60 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 5 monclient: authenticate success, global_id 34212 2024-09-15T03:29:57.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 10 monclient: _renew_subs 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac0456b0 con 0x7f8ab4136d60 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab4099180 con 0x7f8ab4136d60 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab17fa640 10 monclient: dump: 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.087 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8ab37fe640 4 set_mon_vals no callback set 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 10 monclient: _renew_subs 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.073+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab4098c70 con 0x7f8ab4136d60 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8aac042030 con 0x7f8ab4136d60 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8abba57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8abba57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8abba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab40008d0 con 0x7f8ab4136d60 2024-09-15T03:29:57.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8ab17fa640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8aa00386f0 0x7f8aa003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.074+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f8aac06ce90 con 0x7f8ab4136d60 2024-09-15T03:29:57.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.075+0000 7f8ab97f3640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8aa00386f0 0x7f8aa003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.075+0000 7f8ab97f3640 10 monclient: get_auth_request con 0x7f8aa00386f0 auth_method 0 2024-09-15T03:29:57.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.077+0000 7f8ab97f3640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8aa00386f0 0x7f8aa003aba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4007fd0 tx=0x7f8aa4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.080+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8aac043590 con 0x7f8ab4136d60 2024-09-15T03:29:57.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.080+0000 7f8ab17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.080+0000 7f8ab17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:57.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.233+0000 7f8abba57640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:29:57.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.233+0000 7f8abba57640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:29:57.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.233+0000 7f8abba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.233+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f8ab4136af0 con 0x7f8ab4136d60 2024-09-15T03:29:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.234+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (secure 0 0 0) 0x7f8aac04a070 con 0x7f8ab4136d60 2024-09-15T03:29:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.234+0000 7f8ab17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:29:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.234+0000 7f8ab17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:29:57.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.323+0000 7f8abba57640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:29:57.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.323+0000 7f8abba57640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:29:57.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.323+0000 7f8abba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.323+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f8ab4072640 con 0x7f8ab4136d60 2024-09-15T03:29:57.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.324+0000 7f8ab17fa640 1 -- 172.21.15.181:0/2895475095 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (secure 0 0 0) 0x7f8aac030b00 con 0x7f8ab4136d60 2024-09-15T03:29:57.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.324+0000 7f8ab17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:29:57.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.324+0000 7f8ab17fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:29:57.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8aa00386f0 msgr2=0x7f8aa003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8aa00386f0 0x7f8aa003aba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4007fd0 tx=0x7f8aa4007f00 comp rx=0 tx=0).stop 2024-09-15T03:29:57.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 10 monclient: shutdown 2024-09-15T03:29:57.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:57.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 msgr2=0x7f8ab4097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 0x7f8ab4097b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009e60 tx=0x7f8aac037000 comp rx=0 tx=0).stop 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 shutdown_connections 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8aa00386f0 0x7f8aa003aba0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab4136d60 0x7f8ab4097b60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab409f020 0x7f8ab4097620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 --2- 172.21.15.181:0/2895475095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab409ec50 0x7f8ab40970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 >> 172.21.15.181:0/2895475095 conn(0x7f8ab40a5d40 msgr2=0x7f8ab40a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 shutdown_connections 2024-09-15T03:29:57.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.326+0000 7f8abba57640 1 -- 172.21.15.181:0/2895475095 wait complete. 2024-09-15T03:29:57.336 INFO:tasks.workunit.client.0.smithi181.stdout:pg_num_max: 33 2024-09-15T03:29:57.336 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-15T03:29:57.336 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:29:57.337 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-15T03:29:57.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.410+0000 7fa26e880640 10 monclient: get_monmap_and_config 2024-09-15T03:29:57.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.410+0000 7fa26e880640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.410+0000 7fa26e880640 10 monclient: monmap: 2024-09-15T03:29:57.411 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:57.411349+0000 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:57.411349+0000 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:57.412 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 1 Processor -- start 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 1 -- start start 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient: init 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa268098050 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa26809fe90 0x7fa268098420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient(hunting): picked mon.noname-a con 0x7fa26809fe90 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:57.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa268098960 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa268132710 con 0x7fa268098050 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2680a1230 con 0x7fa268098960 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2680a1510 con 0x7fa26809fe90 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26e880640 10 monclient(hunting): authenticate will time out at 801.892845s 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.411+0000 7fa26ce1d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fa268098960 legacy=0x7fa268138d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa26809fe90 0x7fa268098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa26809fe90 0x7fa268098420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34694/0 (socket says 172.21.15.181:34694) 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa267fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa268098050 0x7fa268132000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:44898/0 (socket says 172.21.15.181:44898) 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 -- 172.21.15.181:0/2423419596 learned_addr learned my addr 172.21.15.181:0/2423419596 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): get_auth_request con 0x7fa26809fe90 auth_method 0 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient(hunting): handle_auth_done global_id 34215 payload 274 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 -- 172.21.15.181:0/2423419596 >> v1:172.21.15.181:6789/0 conn(0x7fa268098960 legacy=0x7fa268138d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 -- 172.21.15.181:0/2423419596 >> v1:172.21.15.153:6789/0 conn(0x7fa268098050 legacy=0x7fa268132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 monclient: found mon.noname-a 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.412+0000 7fa2677fe640 1 -- 172.21.15.181:0/2423419596 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa268131bf0 con 0x7fa26809fe90 2024-09-15T03:29:57.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.415+0000 7fa2677fe640 1 -- 172.21.15.181:0/2423419596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 msgr2=0x7fa268098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.415+0000 7fa2677fe640 1 --2- 172.21.15.181:0/2423419596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268098420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009910 tx=0x7fa25c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbc6b96f986203f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 1 -- 172.21.15.181:0/2423419596 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c02ae50 con 0x7fa26809fe90 2024-09-15T03:29:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: dump: 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 1 -- 172.21.15.181:0/2423419596 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c02f4b0 con 0x7fa26809fe90 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 5 monclient: authenticate success, global_id 34215 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 1 -- 172.21.15.181:0/2423419596 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c02f850 con 0x7fa26809fe90 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 4 set_mon_vals no callback set 2024-09-15T03:29:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa266ffd640 10 monclient: dump: 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 10 monclient: shutdown 2024-09-15T03:29:57.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:57.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 1 -- 172.21.15.181:0/2423419596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 msgr2=0x7fa268098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.417+0000 7fa26e880640 1 --2- 172.21.15.181:0/2423419596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268098420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009910 tx=0x7fa25c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 1 -- 172.21.15.181:0/2423419596 shutdown_connections 2024-09-15T03:29:57.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 1 --2- 172.21.15.181:0/2423419596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268098420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 1 -- 172.21.15.181:0/2423419596 >> 172.21.15.181:0/2423419596 conn(0x7fa2680a5190 msgr2=0x7fa2680a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:57.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 1 -- 172.21.15.181:0/2423419596 shutdown_connections 2024-09-15T03:29:57.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 1 -- 172.21.15.181:0/2423419596 wait complete. 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 10 monclient: monmap: 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.418+0000 7fa26e880640 1 Processor -- start 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 -- start start 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient: init 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa268098050 0x7fa268138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): picked mon.noname-b con 0x7fa268098050 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa268098960 0x7fa268134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): picked mon.noname-c con 0x7fa268098960 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): picked mon.noname-a con 0x7fa26809fe90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa268098d70 con 0x7fa26809fe90 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26813a1b0 con 0x7fa268098050 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa268135690 con 0x7fa268098960 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26e880640 10 monclient(hunting): authenticate will time out at 801.900479s 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa267fff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa268098050 msgr2=0x7fa268138420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa267fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa268098050 0x7fa268138420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa268098960 0x7fa268134220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268134760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34700/0 (socket says 172.21.15.181:34700) 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 1 -- 172.21.15.181:0/2265234041 learned_addr learned my addr 172.21.15.181:0/2265234041 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): get_auth_request con 0x7fa26809fe90 auth_method 0 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): get_auth_request con 0x7fa268098960 auth_method 0 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa26ce1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.419+0000 7fa2677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 10 monclient(hunting): handle_auth_done global_id 34218 payload 824 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 1 -- 172.21.15.181:0/2265234041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa268098960 msgr2=0x7fa268134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa268098960 0x7fa268134220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 1 -- 172.21.15.181:0/2265234041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa268098050 msgr2=0x7fa268138420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa268098050 0x7fa268138420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 1 monclient: found mon.noname-a 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa26ce1d640 1 -- 172.21.15.181:0/2265234041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c009450 con 0x7fa26809fe90 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.420+0000 7fa2677fe640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa268098960 0x7fa268134220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26ce1d640 1 -- 172.21.15.181:0/2265234041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 msgr2=0x7fa268134760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26ce1d640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa26809fe90 0x7fa268134760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa26000beb0 tx=0x7fa2600096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260016430 con 0x7fa26809fe90 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: dump: 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2600165d0 con 0x7fa26809fe90 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600167c0 con 0x7fa26809fe90 2024-09-15T03:29:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 5 monclient: authenticate success, global_id 34218 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 10 monclient: _renew_subs 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 1 -- 172.21.15.181:0/2265234041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa268137760 con 0x7fa26809fe90 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa24ffff640 10 monclient: dump: 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.439 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.439 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.439 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.439 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa2667fc640 4 set_mon_vals no callback set 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 10 monclient: _renew_subs 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.422+0000 7fa26e880640 1 -- 172.21.15.181:0/2265234041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2681369a0 con 0x7fa26809fe90 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa260012020 con 0x7fa26809fe90 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa26e880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa26e880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa26e880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa26e880640 1 -- 172.21.15.181:0/2265234041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2681379a0 con 0x7fa26809fe90 2024-09-15T03:29:57.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa24ffff640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2540386f0 0x7fa25403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fa26004bf50 con 0x7fa26809fe90 2024-09-15T03:29:57.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa267fff640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2540386f0 0x7fa25403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.423+0000 7fa267fff640 10 monclient: get_auth_request con 0x7fa2540386f0 auth_method 0 2024-09-15T03:29:57.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.426+0000 7fa267fff640 1 --2- 172.21.15.181:0/2265234041 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2540386f0 0x7fa25403aba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2580048d0 tx=0x7fa258007ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.428+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa2600138e0 con 0x7fa26809fe90 2024-09-15T03:29:57.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.428+0000 7fa24ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.428+0000 7fa24ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:57.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.581+0000 7fa26e880640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-15T03:29:57.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.581+0000 7fa26e880640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-15T03:29:57.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.581+0000 7fa26e880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.581+0000 7fa26e880640 1 -- 172.21.15.181:0/2265234041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) v1 -- 0x7fa26804bf60 con 0x7fa26809fe90 2024-09-15T03:29:57.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.582+0000 7fa24ffff640 1 -- 172.21.15.181:0/2265234041 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v22) v1 ==== 150+0+0 (secure 0 0 0) 0x7fa26001d300 con 0x7fa26809fe90 2024-09-15T03:29:57.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.582+0000 7fa24ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-15T03:29:57.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.582+0000 7fa24ffff640 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-15T03:29:57.584 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-15T03:29:57.587 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:29:57.587 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-15T03:29:57.587 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:29:57.587 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-15T03:29:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.661+0000 7f548d71d640 10 monclient: get_monmap_and_config 2024-09-15T03:29:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.661+0000 7f548d71d640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.661+0000 7f548d71d640 10 monclient: monmap: 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:57.662191+0000 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:57.662191+0000 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:57.664 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.661+0000 7f548d71d640 1 Processor -- start 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 -- start start 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient: init 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f5488131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): picked mon.noname-b con 0x7f5488131bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f548809ebe0 0x7f5488068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): picked mon.noname-c con 0x7f548809ebe0 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:57.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f548809f000 0x7f5488069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): picked mon.noname-a con 0x7f548809f000 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:57.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54880a06d0 con 0x7f548809f000 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5488071780 con 0x7f548809ebe0 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54880a0010 con 0x7f5488131bd0 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f548d71d640 10 monclient(hunting): authenticate will time out at 802.143619s 2024-09-15T03:29:57.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f54867fc640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f548809ebe0 msgr2=0x7f5488068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f54867fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f548809ebe0 0x7f5488068e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.662+0000 7f54877fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f548809f000 0x7f5488069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f548809f000 0x7f5488069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34702/0 (socket says 172.21.15.181:34702) 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f5488131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 -- 172.21.15.181:0/635011493 learned_addr learned my addr 172.21.15.181:0/635011493 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): get_auth_request con 0x7f548809f000 auth_method 0 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): get_auth_request con 0x7f5488131bd0 auth_method 0 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient(hunting): handle_auth_done global_id 34221 payload 274 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 -- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 msgr2=0x7f5488131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 --2- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f5488131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 -- 172.21.15.181:0/635011493 >> v2:172.21.15.181:3300/0 conn(0x7f548809ebe0 msgr2=0x7f5488068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 --2- 172.21.15.181:0/635011493 >> v2:172.21.15.181:3300/0 conn(0x7f548809ebe0 0x7f5488068e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 monclient: found mon.noname-a 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f54877fe640 1 -- 172.21.15.181:0/635011493 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54880a1910 con 0x7f548809f000 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.663+0000 7f5486ffd640 1 --2- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f5488131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f54877fe640 1 -- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 msgr2=0x7f5488069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f54877fe640 1 --2- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488069340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f54880a2b90 tx=0x7f5480009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3819b07d5ceffb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 1 -- 172.21.15.181:0/635011493 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480007610 con 0x7f548809f000 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: dump: 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 5 monclient: authenticate success, global_id 34221 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 1 -- 172.21.15.181:0/635011493 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548002d050 con 0x7f548809f000 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 1 -- 172.21.15.181:0/635011493 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54800079a0 con 0x7f548809f000 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 4 set_mon_vals no callback set 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f5485ffb640 10 monclient: dump: 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.683 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 10 monclient: shutdown 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 1 -- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 msgr2=0x7f5488069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.666+0000 7f548d71d640 1 --2- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488069340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f54880a2b90 tx=0x7f5480009840 comp rx=0 tx=0).stop 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 -- 172.21.15.181:0/635011493 shutdown_connections 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 --2- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488069340 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 --2- 172.21.15.181:0/635011493 >> v2:172.21.15.181:3300/0 conn(0x7f548809ebe0 0x7f5488068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 --2- 172.21.15.181:0/635011493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f5488131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 -- 172.21.15.181:0/635011493 >> 172.21.15.181:0/635011493 conn(0x7f54880713b0 msgr2=0x7f54880a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:57.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 -- 172.21.15.181:0/635011493 shutdown_connections 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 -- 172.21.15.181:0/635011493 wait complete. 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 10 monclient: monmap: 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.667+0000 7f548d71d640 1 Processor -- start 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 -- start start 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient: init 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548809ebe0 0x7f5488140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): picked mon.noname-b con 0x7f548809ebe0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): picked mon.noname-a con 0x7f548809f000 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f548811d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f5486ffd640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548809ebe0 msgr2=0x7f5488140d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): picked mon.noname-c con 0x7f5488131bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5488120fb0 con 0x7f548809f000 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548809bec0 con 0x7f548809ebe0 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f5486ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548809ebe0 0x7f5488140d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548809e2c0 con 0x7f5488131bd0 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f548d71d640 10 monclient(hunting): authenticate will time out at 802.149523s 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f548811d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488130a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34710/0 (socket says 172.21.15.181:34710) 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 1 -- 172.21.15.181:0/4133074840 learned_addr learned my addr 172.21.15.181:0/4133074840 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): get_auth_request con 0x7f548809f000 auth_method 0 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): get_auth_request con 0x7f5488131bd0 auth_method 0 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.668+0000 7f54877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 10 monclient(hunting): handle_auth_done global_id 34224 payload 824 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 1 -- 172.21.15.181:0/4133074840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 msgr2=0x7f548811d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f548811d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 1 -- 172.21.15.181:0/4133074840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548809ebe0 msgr2=0x7f5488140d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548809ebe0 0x7f5488140d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 1 monclient: found mon.noname-a 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54867fc640 1 -- 172.21.15.181:0/4133074840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5478009390 con 0x7f548809f000 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.669+0000 7f54877fe640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5488131bd0 0x7f548811d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f54867fc640 1 -- 172.21.15.181:0/4133074840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 msgr2=0x7f5488130a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f54867fc640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f548809f000 0x7f5488130a80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f547c00b630 tx=0x7f547c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c015070 con 0x7f548809f000 2024-09-15T03:29:57.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: dump: 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c003d90 con 0x7f548809f000 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c006380 con 0x7f548809f000 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 5 monclient: authenticate success, global_id 34224 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 10 monclient: _renew_subs 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 1 -- 172.21.15.181:0/4133074840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5480009ae0 con 0x7f548809f000 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f546effd640 10 monclient: dump: 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.691 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f54857fa640 4 set_mon_vals no callback set 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 10 monclient: _renew_subs 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.671+0000 7f548d71d640 1 -- 172.21.15.181:0/4133074840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5488141b80 con 0x7f548809f000 2024-09-15T03:29:57.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f547c010440 con 0x7f548809f000 2024-09-15T03:29:57.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f548d71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f548d71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f548d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f548d71d640 1 -- 172.21.15.181:0/4133074840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5488121290 con 0x7f548809f000 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f546effd640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54740386f0 0x7f547403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f547c04b640 con 0x7f548809f000 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f5486ffd640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54740386f0 0x7f547403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.672+0000 7f5486ffd640 10 monclient: get_auth_request con 0x7f54740386f0 auth_method 0 2024-09-15T03:29:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.675+0000 7f5486ffd640 1 --2- 172.21.15.181:0/4133074840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54740386f0 0x7f547403aba0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5478009610 tx=0x7f54780090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.677+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f547c01a770 con 0x7f548809f000 2024-09-15T03:29:57.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.677+0000 7f546effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.677+0000 7f546effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:57.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.830+0000 7f548d71d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-15T03:29:57.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.830+0000 7f548d71d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-15T03:29:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.830+0000 7f548d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.830+0000 7f548d71d640 1 -- 172.21.15.181:0/4133074840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) v1 -- 0x7f548804bf60 con 0x7f548809f000 2024-09-15T03:29:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.831+0000 7f546effd640 1 -- 172.21.15.181:0/4133074840 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v22) v1 ==== 150+0+0 (secure 0 0 0) 0x7f547c018780 con 0x7f548809f000 2024-09-15T03:29:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.831+0000 7f546effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-15T03:29:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.831+0000 7f546effd640 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-15T03:29:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-15T03:29:57.835 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:29:57.835 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-15T03:29:57.836 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:29:57.836 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-15T03:29:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.910+0000 7ff021d54640 10 monclient: get_monmap_and_config 2024-09-15T03:29:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.910+0000 7ff021d54640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.910+0000 7ff021d54640 10 monclient: monmap: 2024-09-15T03:29:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:57.911319+0000 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:57.911319+0000 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:57.912 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 Processor -- start 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 -- start start 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient: init 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 0x7ff01c132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): picked mon.noname-b con 0x7ff01c131c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff01c09d210 0x7ff01c1325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): picked mon.noname-c con 0x7ff01c09d210 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 0x7ff01c068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): picked mon.noname-a con 0x7ff01c0a0ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c0a1800 con 0x7ff01c0a0ab0 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c09fea0 con 0x7ff01c09d210 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0a1520 con 0x7ff01c131c90 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff021d54640 10 monclient(hunting): authenticate will time out at 802.392731s 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff01affd640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7ff01c09d210 msgr2=0x7ff01c1325a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.911+0000 7ff01affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff01c09d210 0x7ff01c1325a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 0x7ff01c132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 0x7ff01c068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 0x7ff01c068e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34722/0 (socket says 172.21.15.181:34722) 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 0x7ff01c132060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:39052/0 (socket says 172.21.15.181:39052) 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 1 -- 172.21.15.181:0/1945831939 learned_addr learned my addr 172.21.15.181:0/1945831939 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): get_auth_request con 0x7ff01c0a0ab0 auth_method 0 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): get_auth_request con 0x7ff01c131c90 auth_method 0 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 -- 172.21.15.181:0/1945831939 >> v2:172.21.15.181:3300/0 conn(0x7ff01c09d210 msgr2=0x7ff01c1325a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1945831939 >> v2:172.21.15.181:3300/0 conn(0x7ff01c09d210 0x7ff01c1325a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 -- 172.21.15.181:0/1945831939 >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 msgr2=0x7ff01c068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1945831939 >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 0x7ff01c068e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:29:57.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:57.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01b7fe640 1 -- 172.21.15.181:0/1945831939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c069340 con 0x7ff01c131c90 2024-09-15T03:29:57.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.912+0000 7ff01bfff640 1 --2- 172.21.15.181:0/1945831939 >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 0x7ff01c068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01b7fe640 1 -- 172.21.15.181:0/1945831939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 msgr2=0x7ff01c132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1945831939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 0x7ff01c132060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0a51b0 tx=0x7ff00c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa0cdbb35c282f03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 1 -- 172.21.15.181:0/1945831939 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c02ae10 con 0x7ff01c131c90 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: dump: 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.925 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 1 -- 172.21.15.181:0/1945831939 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c032440 con 0x7ff01c131c90 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 1 -- 172.21.15.181:0/1945831939 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c0327e0 con 0x7ff01c131c90 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:57.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff01a7fc640 10 monclient: dump: 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 5 monclient: authenticate success, global_id 4529 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 4 set_mon_vals no callback set 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 10 monclient: shutdown 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 -- 172.21.15.181:0/1945831939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 msgr2=0x7ff01c132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 --2- 172.21.15.181:0/1945831939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 0x7ff01c132060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0a51b0 tx=0x7ff00c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 -- 172.21.15.181:0/1945831939 shutdown_connections 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 --2- 172.21.15.181:0/1945831939 >> v2:172.21.15.153:3300/0 conn(0x7ff01c0a0ab0 0x7ff01c068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 --2- 172.21.15.181:0/1945831939 >> v2:172.21.15.181:3300/0 conn(0x7ff01c09d210 0x7ff01c1325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 --2- 172.21.15.181:0/1945831939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c131c90 0x7ff01c132060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 -- 172.21.15.181:0/1945831939 >> 172.21.15.181:0/1945831939 conn(0x7ff01c071370 msgr2=0x7ff01c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.915+0000 7ff021d54640 1 -- 172.21.15.181:0/1945831939 shutdown_connections 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.916+0000 7ff021d54640 1 -- 172.21.15.181:0/1945831939 wait complete. 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.916+0000 7ff021d54640 10 monclient: build_initial_monmap 2024-09-15T03:29:57.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.916+0000 7ff021d54640 10 monclient: monmap: 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.916+0000 7ff021d54640 1 Processor -- start 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.916+0000 7ff021d54640 1 -- start start 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.916+0000 7ff021d54640 10 monclient: init 2024-09-15T03:29:57.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c0a0ab0 0x7ff01c1411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): picked mon.noname-c con 0x7ff01c0a0ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff01c131c90 0x7ff01c130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): picked mon.noname-a con 0x7ff01c131c90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff01c131130 0x7ff01c11e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): picked mon.noname-b con 0x7ff01c131130 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c122030 con 0x7ff01c131c90 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0a0fb0 con 0x7ff01c131130 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c09f130 con 0x7ff01c0a0ab0 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff021d54640 10 monclient(hunting): authenticate will time out at 802.398352s 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01bfff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff01c131130 msgr2=0x7ff01c11e6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff01c131130 0x7ff01c11e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff01c131c90 0x7ff01c130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff01c131c90 0x7ff01c130bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34732/0 (socket says 172.21.15.181:34732) 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 -- 172.21.15.181:0/1442785408 learned_addr learned my addr 172.21.15.181:0/1442785408 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): get_auth_request con 0x7ff01c131c90 auth_method 0 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c0a0ab0 0x7ff01c1411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): get_auth_request con 0x7ff01c0a0ab0 auth_method 0 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient(hunting): handle_auth_done global_id 34230 payload 824 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 -- 172.21.15.181:0/1442785408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c0a0ab0 msgr2=0x7ff01c1411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff01c0a0ab0 0x7ff01c1411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 -- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff01c131130 msgr2=0x7ff01c11e6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff01c131130 0x7ff01c11e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient: _finish_hunting 0 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 monclient: found mon.noname-a 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.917+0000 7ff01affd640 1 -- 172.21.15.181:0/1442785408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff014009ae0 con 0x7ff01c131c90 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff01affd640 1 -- 172.21.15.181:0/1442785408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff01c131c90 msgr2=0x7ff01c130bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff01affd640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff01c131c90 0x7ff01c130bf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff01000b600 tx=0x7ff0100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010015070 con 0x7ff01c131c90 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:57.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: dump: 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: _finish_auth 0 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:57.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 5 monclient: authenticate success, global_id 34230 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff010003d90 con 0x7ff01c131c90 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 10 monclient: _renew_subs 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 1 -- 172.21.15.181:0/1442785408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c009390 con 0x7ff01c131c90 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010006380 con 0x7ff01c131c90 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff0037fe640 10 monclient: dump: 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.935 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 10 monclient: _renew_subs 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff021d54640 1 -- 172.21.15.181:0/1442785408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c1420b0 con 0x7ff01c131c90 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.920+0000 7ff019ffb640 4 set_mon_vals no callback set 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff010010440 con 0x7ff01c131c90 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff021d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff021d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff021d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff021d54640 1 -- 172.21.15.181:0/1442785408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c09ae70 con 0x7ff01c131c90 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff0037fe640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0080386f0 0x7ff00803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7ff01004b6c0 con 0x7ff01c131c90 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0080386f0 0x7ff00803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff01b7fe640 10 monclient: get_auth_request con 0x7ff0080386f0 auth_method 0 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff01b7fe640 1 -- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0080386f0 msgr2=0x7ff00803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0080386f0 0x7ff00803aba0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0a3150 tx=0x7ff00c032000 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-15T03:29:57.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.921+0000 7ff01b7fe640 1 --2- 172.21.15.181:0/1442785408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0080386f0 0x7ff00803aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:57.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.925+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff01c09ae70 con 0x7ff01c131c90 2024-09-15T03:29:57.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.925+0000 7ff0037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:57.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:57.925+0000 7ff0037fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:58.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.079+0000 7ff021d54640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-15T03:29:58.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.080+0000 7ff021d54640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-15T03:29:58.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.080+0000 7ff021d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.080+0000 7ff021d54640 1 -- 172.21.15.181:0/1442785408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) v1 -- 0x7ff01c122250 con 0x7ff01c131c90 2024-09-15T03:29:58.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.080+0000 7ff0037fe640 1 -- 172.21.15.181:0/1442785408 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v22) v1 ==== 146+0+0 (secure 0 0 0) 0x7ff01c122250 con 0x7ff01c131c90 2024-09-15T03:29:58.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.080+0000 7ff0037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-15T03:29:58.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.080+0000 7ff0037fe640 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-15T03:29:58.082 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-15T03:29:58.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:29:58.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-15T03:29:58.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:29:58.086 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.159+0000 7f0e03d09640 10 monclient: get_monmap_and_config 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.159+0000 7f0e03d09640 10 monclient: build_initial_monmap 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.159+0000 7f0e03d09640 10 monclient: monmap: 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:58.160682+0000 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:58.160682+0000 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:58.161 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:58.162 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:58.162 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.160+0000 7f0e03d09640 1 Processor -- start 2024-09-15T03:29:58.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.160+0000 7f0e03d09640 1 -- start start 2024-09-15T03:29:58.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.160+0000 7f0e03d09640 10 monclient: init 2024-09-15T03:29:58.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient(hunting): picked mon.noname-b con 0x7f0dfc131c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0dfc09d210 addr v1:172.21.15.153:6789/0 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dfc0a0ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc0a1800 con 0x7f0dfc09d210 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc09fea0 con 0x7f0dfc0a0ab0 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0a1520 con 0x7f0dfc131c90 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e03d09640 10 monclient(hunting): authenticate will time out at 802.642195s 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e022a6640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f0dfc0a0ab0 legacy=0x7f0dfc069510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e012a4640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0dfc09d210 0x7f0dfc068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:44910/0 (socket says 172.21.15.181:44910) 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e012a4640 1 -- 172.21.15.181:0/1837591674 learned_addr learned my addr 172.21.15.181:0/1837591674 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 1 --2- 172.21.15.181:0/1837591674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): get_auth_request con 0x7f0dfc131c90 auth_method 0 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_done global_id 4535 payload 274 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 1 -- 172.21.15.181:0/1837591674 >> v1:172.21.15.181:6789/0 conn(0x7f0dfc0a0ab0 legacy=0x7f0dfc069510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 1 -- 172.21.15.181:0/1837591674 >> v1:172.21.15.153:6789/0 conn(0x7f0dfc09d210 legacy=0x7f0dfc068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient: _finish_hunting 0 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 1 monclient: found mon.noname-b 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:29:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.161+0000 7f0e01aa5640 1 -- 172.21.15.181:0/1837591674 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc1325a0 con 0x7f0dfc131c90 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.164+0000 7f0e01aa5640 1 -- 172.21.15.181:0/1837591674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 msgr2=0x7f0dfc132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.164+0000 7f0e01aa5640 1 --2- 172.21.15.181:0/1837591674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc132060 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009e90 tx=0x7f0df8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8fe71028d63acc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 1 -- 172.21.15.181:0/1837591674 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8007840 con 0x7f0dfc131c90 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: dump: 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: _finish_auth 0 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 1 -- 172.21.15.181:0/1837591674 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df802d050 con 0x7f0dfc131c90 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 5 monclient: authenticate success, global_id 4535 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 1 -- 172.21.15.181:0/1837591674 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8007bd0 con 0x7f0dfc131c90 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 4 set_mon_vals no callback set 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e00aa3640 10 monclient: dump: 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.175 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 10 monclient: shutdown 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 -- 172.21.15.181:0/1837591674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 msgr2=0x7f0dfc132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 --2- 172.21.15.181:0/1837591674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc132060 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009e90 tx=0x7f0df8009780 comp rx=0 tx=0).stop 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 -- 172.21.15.181:0/1837591674 shutdown_connections 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 --2- 172.21.15.181:0/1837591674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc132060 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 -- 172.21.15.181:0/1837591674 >> 172.21.15.181:0/1837591674 conn(0x7f0dfc071370 msgr2=0x7f0dfc0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 -- 172.21.15.181:0/1837591674 shutdown_connections 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.165+0000 7f0e03d09640 1 -- 172.21.15.181:0/1837591674 wait complete. 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient: build_initial_monmap 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient: monmap: 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:58.176 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 Processor -- start 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 -- start start 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient: init 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:29:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc0a0ab0 0x7f0dfc09aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): picked mon.noname-a con 0x7f0dfc0a0ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): picked mon.noname-c con 0x7f0dfc131c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dfc11ce90 0x7f0dfc1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): picked mon.noname-b con 0x7f0dfc11ce90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc11ca90 con 0x7f0dfc0a0ab0 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc11e060 con 0x7f0dfc11ce90 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc11dd80 con 0x7f0dfc131c90 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e03d09640 10 monclient(hunting): authenticate will time out at 802.647943s 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e022a6640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dfc11ce90 msgr2=0x7f0dfc1411a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.166+0000 7f0e022a6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dfc11ce90 0x7f0dfc1411a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc0a0ab0 0x7f0dfc09aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc130c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:39072/0 (socket says 172.21.15.181:39072) 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc0a0ab0 0x7f0dfc09aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:34742/0 (socket says 172.21.15.181:34742) 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 -- 172.21.15.181:0/2973126306 learned_addr learned my addr 172.21.15.181:0/2973126306 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): get_auth_request con 0x7f0dfc131c90 auth_method 0 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:58.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): get_auth_request con 0x7f0dfc0a0ab0 auth_method 0 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:58.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e01aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 -- 172.21.15.181:0/2973126306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dfc11ce90 msgr2=0x7f0dfc1411a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 --2- 172.21.15.181:0/2973126306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dfc11ce90 0x7f0dfc1411a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 -- 172.21.15.181:0/2973126306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc0a0ab0 msgr2=0x7f0dfc09aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 --2- 172.21.15.181:0/2973126306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc0a0ab0 0x7f0dfc09aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient: _finish_hunting 0 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 monclient: found mon.noname-c 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.167+0000 7f0e012a4640 1 -- 172.21.15.181:0/2973126306 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8009a20 con 0x7f0dfc131c90 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.169+0000 7f0e012a4640 1 -- 172.21.15.181:0/2973126306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 msgr2=0x7f0dfc130c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.169+0000 7f0e012a4640 1 --2- 172.21.15.181:0/2973126306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dfc131c90 0x7f0dfc130c60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00c610 tx=0x7f0dec004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:58.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec005cb0 con 0x7f0dfc131c90 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: dump: 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec005e30 con 0x7f0dfc131c90 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 5 monclient: authenticate success, global_id 4538 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 10 monclient: _renew_subs 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 1 -- 172.21.15.181:0/2973126306 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc11e340 con 0x7f0dfc131c90 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec007340 con 0x7f0dfc131c90 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 10 monclient: dump: 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.183 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 10 monclient: _renew_subs 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df3fff640 4 set_mon_vals no callback set 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0e03d09640 1 -- 172.21.15.181:0/2973126306 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc11c000 con 0x7f0dfc131c90 2024-09-15T03:29:58.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.170+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0dec01b400 con 0x7f0dfc131c90 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0df1ffb640 1 --2- 172.21.15.181:0/2973126306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0de00386f0 0x7f0de003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0e03d09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0e03d09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f0dec04b670 con 0x7f0dfc131c90 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0e03d09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0e03d09640 1 -- 172.21.15.181:0/2973126306 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc09b690 con 0x7f0dfc131c90 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0e01aa5640 1 --2- 172.21.15.181:0/2973126306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0de00386f0 0x7f0de003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.171+0000 7f0e01aa5640 10 monclient: get_auth_request con 0x7f0de00386f0 auth_method 0 2024-09-15T03:29:58.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.173+0000 7f0e01aa5640 1 --2- 172.21.15.181:0/2973126306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0de00386f0 0x7f0de003aba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0a51b0 tx=0x7f0df8037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:58.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.176+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0dec012760 con 0x7f0dfc131c90 2024-09-15T03:29:58.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.176+0000 7f0df1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:58.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.176+0000 7f0df1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:58.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.328+0000 7f0e03d09640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-15T03:29:58.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.328+0000 7f0e03d09640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-15T03:29:58.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.328+0000 7f0e03d09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:29:58.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.328+0000 7f0e03d09640 1 -- 172.21.15.181:0/2973126306 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) v1 -- 0x7f0dfc0008d0 con 0x7f0dfc131c90 2024-09-15T03:29:58.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.329+0000 7f0df1ffb640 1 -- 172.21.15.181:0/2973126306 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v22) v1 ==== 149+0+0 (secure 0 0 0) 0x7f0dec01a360 con 0x7f0dfc131c90 2024-09-15T03:29:58.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.329+0000 7f0df1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-15T03:29:58.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.329+0000 7f0df1ffb640 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-15T03:29:58.330 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-15T03:29:58.334 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:29:58.334 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-15T03:29:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.408+0000 7fe09b4af640 10 monclient: get_monmap_and_config 2024-09-15T03:29:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.408+0000 7fe09b4af640 10 monclient: build_initial_monmap 2024-09-15T03:29:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.408+0000 7fe09b4af640 10 monclient: monmap: 2024-09-15T03:29:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:29:58.409334+0000 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:29:58.409334+0000 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:29:58.410 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 Processor -- start 2024-09-15T03:29:58.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 -- start start 2024-09-15T03:29:58.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient: init 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe094131ac0 0x7fe094131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): picked mon.noname-a con 0x7fe094131ac0 addr v2:172.21.15.153:3300/0 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe09409ff50 addr v1:172.21.15.181:6789/0 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe0941323d0 0x7fe094098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): picked mon.noname-c con 0x7fe0941323d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0940a1610 con 0x7fe09409ff50 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0941327a0 con 0x7fe094131ac0 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940a1330 con 0x7fe0941323d0 2024-09-15T03:29:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe09b4af640 10 monclient(hunting): authenticate will time out at 802.890790s 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe098a4a640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fe09409ff50 legacy=0x7fe094098150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe099a4c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fe0941323d0 msgr2=0x7fe094098860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.409+0000 7fe099a4c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe0941323d0 0x7fe094098860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe094131ac0 0x7fe094131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe094131ac0 0x7fe094131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35902/0 (socket says 172.21.15.181:35902) 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 -- 172.21.15.181:0/3107050249 learned_addr learned my addr 172.21.15.181:0/3107050249 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): get_auth_request con 0x7fe094131ac0 auth_method 0 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient(hunting): handle_auth_done global_id 34236 payload 274 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 -- 172.21.15.181:0/3107050249 >> v2:172.21.15.181:3300/0 conn(0x7fe0941323d0 msgr2=0x7fe094098860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 --2- 172.21.15.181:0/3107050249 >> v2:172.21.15.181:3300/0 conn(0x7fe0941323d0 0x7fe094098860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 -- 172.21.15.181:0/3107050249 >> v1:172.21.15.181:6789/0 conn(0x7fe09409ff50 legacy=0x7fe094098150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient: _finish_hunting 0 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 monclient: found mon.noname-a 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.410+0000 7fe09924b640 1 -- 172.21.15.181:0/3107050249 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe094139e10 con 0x7fe094131ac0 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09924b640 1 -- 172.21.15.181:0/3107050249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe094131ac0 msgr2=0x7fe094131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09924b640 1 --2- 172.21.15.181:0/3107050249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe094131ac0 0x7fe094131e90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe09409feb0 tx=0x7fe0900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457a83d1c6df9cec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 1 -- 172.21.15.181:0/3107050249 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090032440 con 0x7fe094131ac0 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: dump: 2024-09-15T03:29:58.422 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: _finish_auth 0 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 1 -- 172.21.15.181:0/3107050249 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0900325a0 con 0x7fe094131ac0 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 5 monclient: authenticate success, global_id 34236 2024-09-15T03:29:58.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 10 monclient: get_monmap_and_config success 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 4 set_mon_vals no callback set 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 1 -- 172.21.15.181:0/3107050249 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090032940 con 0x7fe094131ac0 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe08bfff640 10 monclient: dump: 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 10 monclient: shutdown 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 1 -- 172.21.15.181:0/3107050249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe094131ac0 msgr2=0x7fe094131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3107050249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe094131ac0 0x7fe094131e90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe09409feb0 tx=0x7fe0900091d0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 1 -- 172.21.15.181:0/3107050249 shutdown_connections 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3107050249 >> v2:172.21.15.181:3300/0 conn(0x7fe0941323d0 0x7fe094098860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3107050249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe094131ac0 0x7fe094131e90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.413+0000 7fe09b4af640 1 -- 172.21.15.181:0/3107050249 >> 172.21.15.181:0/3107050249 conn(0x7fe0940a5190 msgr2=0x7fe0940a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:29:58.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 1 -- 172.21.15.181:0/3107050249 shutdown_connections 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 1 -- 172.21.15.181:0/3107050249 wait complete. 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 10 monclient: build_initial_monmap 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 10 monclient: monmap: 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.426 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 1 Processor -- start 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 1 -- start start 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.414+0000 7fe09b4af640 10 monclient: init 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient: _reopen_session rank -1 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe09409ff50 0x7fe09412cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): picked mon.noname-c con 0x7fe09409ff50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe094131ac0 0x7fe09412d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): picked mon.noname-b con 0x7fe094131ac0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:29:58.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 0x7fe09412e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): picked mon.noname-a con 0x7fe0941323d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940984e0 con 0x7fe0941323d0 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094130300 con 0x7fe094131ac0 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe094132eb0 con 0x7fe09409ff50 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): _renew_subs 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09b4af640 10 monclient(hunting): authenticate will time out at 802.896536s 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe098a4a640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe094131ac0 msgr2=0x7fe09412d070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe098a4a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe094131ac0 0x7fe09412d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:29:58.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 0x7fe09412e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe09409ff50 0x7fe09412cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 0x7fe09412e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35914/0 (socket says 172.21.15.181:35914) 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 1 -- 172.21.15.181:0/3584180136 learned_addr learned my addr 172.21.15.181:0/3584180136 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): get_auth_request con 0x7fe0941323d0 auth_method 0 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): get_auth_request con 0x7fe09409ff50 auth_method 0 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:58.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe099a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.415+0000 7fe09924b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 10 monclient(hunting): handle_auth_done global_id 34239 payload 824 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 1 -- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe09409ff50 msgr2=0x7fe09412cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe09409ff50 0x7fe09412cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 1 -- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe094131ac0 msgr2=0x7fe09412d070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe094131ac0 0x7fe09412d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 10 monclient: _finish_hunting 0 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 1 monclient: found mon.noname-a 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:29:58.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe099a4c640 1 -- 172.21.15.181:0/3584180136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009450 con 0x7fe0941323d0 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.416+0000 7fe09924b640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe09409ff50 0x7fe09412cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe099a4c640 1 -- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 msgr2=0x7fe09412e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe099a4c640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 0x7fe09412e850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00bf80 tx=0x7fe08c009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c016430 con 0x7fe0941323d0 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: dump: 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: _finish_auth 0 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c0165d0 con 0x7fe0941323d0 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0167c0 con 0x7fe0941323d0 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 5 monclient: authenticate success, global_id 34239 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 10 monclient: _renew_subs 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe094133190 con 0x7fe0941323d0 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe0897fa640 10 monclient: dump: 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:29:58.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe08b7fe640 4 set_mon_vals no callback set 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 10 monclient: _renew_subs 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.418+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe094132b40 con 0x7fe0941323d0 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe09b4af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe08c012020 con 0x7fe0941323d0 2024-09-15T03:29:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe09b4af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:58.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe09b4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0941333b0 con 0x7fe0941323d0 2024-09-15T03:29:58.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe0897fa640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0780386f0 0x7fe07803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:29:58.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fe08c04c0f0 con 0x7fe0941323d0 2024-09-15T03:29:58.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe09924b640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0780386f0 0x7fe07803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:29:58.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.419+0000 7fe09924b640 10 monclient: get_auth_request con 0x7fe0780386f0 auth_method 0 2024-09-15T03:29:58.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.422+0000 7fe09924b640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0780386f0 0x7fe07803aba0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe094096d60 tx=0x7fe090032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:29:58.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.422+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:29:58.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.424+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe08c0078f0 con 0x7fe0941323d0 2024-09-15T03:29:58.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.424+0000 7fe0897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:29:58.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.424+0000 7fe0897fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:29:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.577+0000 7fe09b4af640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T03:29:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.577+0000 7fe09b4af640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T03:29:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.577+0000 7fe09b4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.577+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7fe09404bf60 con 0x7fe0941323d0 2024-09-15T03:29:58.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.991+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v23) v1 ==== 143+0+0 (secure 0 0 0) 0x7fe08c01d300 con 0x7fe0941323d0 2024-09-15T03:29:58.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.991+0000 7fe0897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T03:29:58.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:58.991+0000 7fe0897fa640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-15T03:29:59.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:59.084+0000 7fe09b4af640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T03:29:59.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:59.084+0000 7fe09b4af640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T03:29:59.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:59.084+0000 7fe09b4af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:29:59.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:29:59.084+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7fe094138ec0 con 0x7fe0941323d0 2024-09-15T03:30:00.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.002+0000 7fe0897fa640 1 -- 172.21.15.181:0/3584180136 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v24) v1 ==== 143+0+0 (secure 0 0 0) 0x7fe08c0072f0 con 0x7fe0941323d0 2024-09-15T03:30:00.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.002+0000 7fe0897fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-15T03:30:00.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.002+0000 7fe0897fa640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-15T03:30:00.003 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 3 pg_num_min to 0 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.004+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0780386f0 msgr2=0x7fe07803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.004+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0780386f0 0x7fe07803aba0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe094096d60 tx=0x7fe090032000 comp rx=0 tx=0).stop 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.004+0000 7fe09b4af640 10 monclient: shutdown 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.004+0000 7fe09b4af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.004+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 msgr2=0x7fe09412e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.004+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 0x7fe09412e850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00bf80 tx=0x7fe08c009730 comp rx=0 tx=0).stop 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 shutdown_connections 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0780386f0 0x7fe07803aba0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0941323d0 0x7fe09412e850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe094131ac0 0x7fe09412d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 --2- 172.21.15.181:0/3584180136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe09409ff50 0x7fe09412cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 >> 172.21.15.181:0/3584180136 conn(0x7fe0940a5190 msgr2=0x7fe0940697d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 shutdown_connections 2024-09-15T03:30:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.005+0000 7fe09b4af640 1 -- 172.21.15.181:0/3584180136 wait complete. 2024-09-15T03:30:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-15T03:30:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-15T03:30:00.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.094+0000 7fcebe79d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.094+0000 7fcebe79d640 10 monclient: build_initial_monmap 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.094+0000 7fcebe79d640 10 monclient: monmap: 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:00.095673+0000 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:00.095673+0000 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:00.096 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:00.097 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:00.097 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:00.097 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.095+0000 7fcebe79d640 1 Processor -- start 2024-09-15T03:30:00.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.095+0000 7fcebe79d640 1 -- start start 2024-09-15T03:30:00.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.095+0000 7fcebe79d640 10 monclient: init 2024-09-15T03:30:00.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:00.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:30:00.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fceb8131c90 0x7fceb8132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient(hunting): picked mon.noname-c con 0x7fceb8131c90 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fceb809d210 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fceb80a0ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80a1800 con 0x7fceb809d210 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb809fea0 con 0x7fceb80a0ab0 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb80a1520 con 0x7fceb8131c90 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fceb7fff640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fceb8131c90 msgr2=0x7fceb8132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebe79d640 10 monclient(hunting): authenticate will time out at 804.577366s 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fceb7fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fceb8131c90 0x7fceb8132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fcebcd3a640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fceb80a0ab0 legacy=0x7fceb8069510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fceb77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fceb809d210 0x7fceb8068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:49936/0 (socket says 172.21.15.181:49936) 2024-09-15T03:30:00.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.096+0000 7fceb77fe640 1 -- 172.21.15.181:0/2290445840 learned_addr learned my addr 172.21.15.181:0/2290445840 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb77fe640 1 -- 172.21.15.181:0/2290445840 >> v1:172.21.15.153:6789/0 conn(0x7fceb809d210 legacy=0x7fceb8068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159599972 0 0) 0x7fceb80a1800 con 0x7fceb809d210 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): my global_id is 34242 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea00034e0 con 0x7fceb809d210 2024-09-15T03:30:00.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650152418 0 0) 0x7fcea00034e0 con 0x7fceb809d210 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 >> v2:172.21.15.181:3300/0 conn(0x7fceb8131c90 msgr2=0x7fceb8132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 --2- 172.21.15.181:0/2290445840 >> v2:172.21.15.181:3300/0 conn(0x7fceb8131c90 0x7fceb8132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 >> v1:172.21.15.181:6789/0 conn(0x7fceb80a0ab0 legacy=0x7fceb8069510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb81325a0 con 0x7fceb809d210 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fceac0017d0 con 0x7fceb809d210 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fcebe79d640 5 monclient: authenticate success, global_id 34242 2024-09-15T03:30:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fcebe79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: dump: 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 >> v1:172.21.15.153:6789/0 conn(0x7fceb809d210 legacy=0x7fceb8068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 0x7fcea0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): picked mon.c con 0x7fcea00039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea0004780 0x7fcea0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): picked mon.a con 0x7fcea0004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea0005280 0x7fcea000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): picked mon.b con 0x7fcea0005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80a1520 con 0x7fcea0004780 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea000c240 con 0x7fcea0005280 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea000c400 con 0x7fcea00039d0 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fcebe79d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2290445840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea0005280 msgr2=0x7fcea000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fcebcd3a640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea0005280 0x7fcea000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb77fe640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea0004780 0x7fcea0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb77fe640 10 monclient(hunting): get_auth_request con 0x7fcea0004780 auth_method 0 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb7fff640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 0x7fcea0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb7fff640 10 monclient(hunting): get_auth_request con 0x7fcea00039d0 auth_method 0 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.099+0000 7fceb7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient(hunting): handle_auth_done global_id 34242 payload 293 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 1 -- 172.21.15.181:0/2290445840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea0005280 msgr2=0x7fcea000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea0005280 0x7fcea000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 1 -- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea0004780 msgr2=0x7fcea0004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea0004780 0x7fcea0004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 1 monclient: found mon.c 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 1 -- 172.21.15.181:0/2290445840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00034e0 con 0x7fcea00039d0 2024-09-15T03:30:00.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2290445840 reap_dead start 2024-09-15T03:30:00.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:00.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:00.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.100+0000 7fceb77fe640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcea0004780 0x7fcea0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:00.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb7fff640 1 -- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 msgr2=0x7fcea0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:00.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb7fff640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 0x7fcea0003dc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcea80099c0 tx=0x7fcea8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1935335702a6b7a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:00.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8003b80 con 0x7fcea00039d0 2024-09-15T03:30:00.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:00.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: dump: 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea8003d20 con 0x7fcea00039d0 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 1 -- 172.21.15.181:0/2290445840 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea802ab80 con 0x7fcea00039d0 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 4 set_mon_vals no callback set 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fceb6ffd640 10 monclient: dump: 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 10 monclient: shutdown 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 -- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 msgr2=0x7fcea0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 0x7fcea0003dc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcea80099c0 tx=0x7fcea8009060 comp rx=0 tx=0).stop 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 -- 172.21.15.181:0/2290445840 shutdown_connections 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 --2- 172.21.15.181:0/2290445840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcea00039d0 0x7fcea0003dc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 -- 172.21.15.181:0/2290445840 >> 172.21.15.181:0/2290445840 conn(0x7fceb8071370 msgr2=0x7fceb80a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 -- 172.21.15.181:0/2290445840 shutdown_connections 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.103+0000 7fcebe79d640 1 -- 172.21.15.181:0/2290445840 wait complete. 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.104+0000 7fcebe79d640 10 monclient: build_initial_monmap 2024-09-15T03:30:00.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.104+0000 7fcebe79d640 10 monclient: monmap: 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.104+0000 7fcebe79d640 1 Processor -- start 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.104+0000 7fcebe79d640 1 -- start start 2024-09-15T03:30:00.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.104+0000 7fcebe79d640 10 monclient: init 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea00039d0 0x7fceb8140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): picked mon.noname-b con 0x7fcea00039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81413f0 0x7fceb8130cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): picked mon.noname-a con 0x7fceb81413f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fceb811de90 0x7fceb811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): picked mon.noname-c con 0x7fceb811de90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb811f0d0 con 0x7fceb81413f0 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb811edf0 con 0x7fcea00039d0 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb811f3b0 con 0x7fceb811de90 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb7fff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea00039d0 msgr2=0x7fceb8140eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebe79d640 10 monclient(hunting): authenticate will time out at 804.586447s 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea00039d0 0x7fceb8140eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fceb811de90 0x7fceb811b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81413f0 0x7fceb8130cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fceb811de90 0x7fceb811b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:56114/0 (socket says 172.21.15.181:56114) 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2941712426 learned_addr learned my addr 172.21.15.181:0/2941712426 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): get_auth_request con 0x7fceb811de90 auth_method 0 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): get_auth_request con 0x7fceb81413f0 auth_method 0 2024-09-15T03:30:00.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fceb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2941712426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea00039d0 msgr2=0x7fceb8140eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 --2- 172.21.15.181:0/2941712426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcea00039d0 0x7fceb8140eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2941712426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81413f0 msgr2=0x7fceb8130cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 --2- 172.21.15.181:0/2941712426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81413f0 0x7fceb8130cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 monclient: found mon.noname-c 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.105+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2941712426 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8141c60 con 0x7fceb811de90 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebcd3a640 1 -- 172.21.15.181:0/2941712426 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fceb811de90 msgr2=0x7fceb811b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebcd3a640 1 --2- 172.21.15.181:0/2941712426 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fceb811de90 0x7fceb811b900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fceb80a61b0 tx=0x7fceb0009740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0006570 con 0x7fceb811de90 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: dump: 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.124 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: _finish_auth 0 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 5 monclient: authenticate success, global_id 4544 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb000d040 con 0x7fceb811de90 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0006940 con 0x7fceb811de90 2024-09-15T03:30:00.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 10 monclient: _renew_subs 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 1 -- 172.21.15.181:0/2941712426 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb8131200 con 0x7fceb811de90 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fce9ffff640 10 monclient: dump: 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.126 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fceb67fc640 4 set_mon_vals no callback set 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 10 monclient: _renew_subs 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.108+0000 7fcebe79d640 1 -- 172.21.15.181:0/2941712426 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb811be40 con 0x7fceb811de90 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fceb001b3b0 con 0x7fceb811de90 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fcebe79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fcebe79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fcebe79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fcebe79d640 1 -- 172.21.15.181:0/2941712426 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb811f570 con 0x7fceb811de90 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fce9ffff640 1 --2- 172.21.15.181:0/2941712426 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcea4038740 0x7fcea403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4709+0+0 (secure 0 0 0) 0x7fceb004b410 con 0x7fceb811de90 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fceb7fff640 1 --2- 172.21.15.181:0/2941712426 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcea4038740 0x7fcea403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.109+0000 7fceb7fff640 10 monclient: get_auth_request con 0x7fcea4038740 auth_method 0 2024-09-15T03:30:00.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.112+0000 7fceb7fff640 1 --2- 172.21.15.181:0/2941712426 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcea4038740 0x7fcea403abf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcea8003940 tx=0x7fcea8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:00.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.114+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fceb000da60 con 0x7fceb811de90 2024-09-15T03:30:00.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.114+0000 7fce9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:00.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.114+0000 7fce9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.266+0000 7fcebe79d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.266+0000 7fcebe79d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.266+0000 7fcebe79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.266+0000 7fcebe79d640 1 -- 172.21.15.181:0/2941712426 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fceb80008d0 con 0x7fceb811de90 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.267+0000 7fce9ffff640 1 -- 172.21.15.181:0/2941712426 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v24) v1 ==== 158+0+0 (secure 0 0 0) 0x7fceb0017070 con 0x7fceb811de90 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.267+0000 7fce9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.267+0000 7fce9ffff640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-15T03:30:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-15T03:30:00.272 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:00.272 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.345+0000 7f2c6eb5d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.345+0000 7f2c6eb5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.345+0000 7f2c6eb5d640 10 monclient: monmap: 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:00.346585+0000 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:00.346585+0000 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:00.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 1 Processor -- start 2024-09-15T03:30:00.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 1 -- start start 2024-09-15T03:30:00.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 10 monclient: init 2024-09-15T03:30:00.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c68131ac0 0x7f2c68131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c68131ac0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2c6809ff50 0x7f2c681323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6809ff50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c68098150 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.346+0000 7f2c6eb5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c680a1330 con 0x7f2c68098150 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6eb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6eb5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c68098c30 con 0x7f2c68131ac0 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6eb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6eb5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c68139e10 con 0x7f2c6809ff50 2024-09-15T03:30:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6eb5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6eb5d640 10 monclient(hunting): authenticate will time out at 804.828021s 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c5ffff640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f2c6809ff50 msgr2=0x7f2c681323d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6d0fa640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f2c68098150 legacy=0x7f2c68098520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c5ffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2c6809ff50 0x7f2c681323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c68131ac0 0x7f2c68131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c68131ac0 0x7f2c68131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35930/0 (socket says 172.21.15.181:35930) 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 -- 172.21.15.181:0/3163984062 learned_addr learned my addr 172.21.15.181:0/3163984062 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): get_auth_request con 0x7f2c68131ac0 auth_method 0 2024-09-15T03:30:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_done global_id 34248 payload 274 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 -- 172.21.15.181:0/3163984062 >> v2:172.21.15.181:3300/0 conn(0x7f2c6809ff50 msgr2=0x7f2c681323d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 --2- 172.21.15.181:0/3163984062 >> v2:172.21.15.181:3300/0 conn(0x7f2c6809ff50 0x7f2c681323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 -- 172.21.15.181:0/3163984062 >> v1:172.21.15.181:6789/0 conn(0x7f2c68098150 legacy=0x7f2c68098520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient: _finish_hunting 0 2024-09-15T03:30:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 monclient: found mon.noname-a 2024-09-15T03:30:00.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:00.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:00.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.347+0000 7f2c6c8f9640 1 -- 172.21.15.181:0/3163984062 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c68139f50 con 0x7f2c68131ac0 2024-09-15T03:30:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.350+0000 7f2c6c8f9640 1 -- 172.21.15.181:0/3163984062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 msgr2=0x7f2c68131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.350+0000 7f2c6c8f9640 1 --2- 172.21.15.181:0/3163984062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68131e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c68132af0 tx=0x7f2c640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fe74aa67416062 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.350+0000 7f2c5f7fe640 1 -- 172.21.15.181:0/3163984062 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64032440 con 0x7f2c68131ac0 2024-09-15T03:30:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.350+0000 7f2c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.350+0000 7f2c5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.350+0000 7f2c5f7fe640 10 monclient: dump: 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 1 -- 172.21.15.181:0/3163984062 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c640325a0 con 0x7f2c68131ac0 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 1 -- 172.21.15.181:0/3163984062 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64032940 con 0x7f2c68131ac0 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c5f7fe640 10 monclient: dump: 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 5 monclient: authenticate success, global_id 34248 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 4 set_mon_vals no callback set 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 10 monclient: shutdown 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/3163984062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 msgr2=0x7f2c68131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/3163984062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68131e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c68132af0 tx=0x7f2c640091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/3163984062 shutdown_connections 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/3163984062 >> v2:172.21.15.181:3300/0 conn(0x7f2c6809ff50 0x7f2c681323d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/3163984062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68131e90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/3163984062 >> 172.21.15.181:0/3163984062 conn(0x7f2c680a5190 msgr2=0x7f2c680a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/3163984062 shutdown_connections 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.351+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/3163984062 wait complete. 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient: monmap: 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 1 Processor -- start 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 1 -- start start 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient: init 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c68098150 0x7f2c68138170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c68098150 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c6809ff50 0x7f2c681412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.352+0000 7f2c6eb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6809ff50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c68131ac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68135750 con 0x7f2c68131ac0 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68069780 con 0x7f2c6809ff50 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68098e70 con 0x7f2c68098150 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6eb5d640 10 monclient(hunting): authenticate will time out at 804.834026s 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c5ffff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c6809ff50 msgr2=0x7f2c681412c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c5ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c6809ff50 0x7f2c681412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c68098150 0x7f2c68138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68141800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35932/0 (socket says 172.21.15.181:35932) 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 -- 172.21.15.181:0/115594869 learned_addr learned my addr 172.21.15.181:0/115594869 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): get_auth_request con 0x7f2c68131ac0 auth_method 0 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): get_auth_request con 0x7f2c68098150 auth_method 0 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6c8f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient(hunting): handle_auth_done global_id 34251 payload 824 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 -- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c68098150 msgr2=0x7f2c68138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c68098150 0x7f2c68138170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 -- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c6809ff50 msgr2=0x7f2c681412c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c6809ff50 0x7f2c681412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient: _finish_hunting 0 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 monclient: found mon.noname-a 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.353+0000 7f2c6d0fa640 1 -- 172.21.15.181:0/115594869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64009450 con 0x7f2c68131ac0 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.355+0000 7f2c6d0fa640 1 -- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 msgr2=0x7f2c68141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.355+0000 7f2c6d0fa640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68141800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000bbb0 tx=0x7f2c600097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.355+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6000ccb0 con 0x7f2c68131ac0 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.355+0000 7f2c5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.355+0000 7f2c5cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.355+0000 7f2c5cff9640 10 monclient: dump: 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 5 monclient: authenticate success, global_id 34251 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6000ce10 con 0x7f2c68131ac0 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 monclient: _renew_subs 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c68137180 con 0x7f2c68131ac0 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60006150 con 0x7f2c68131ac0 2024-09-15T03:30:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 10 monclient: dump: 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:00.371 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 monclient: _renew_subs 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c681386b0 con 0x7f2c68131ac0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5effd640 4 set_mon_vals no callback set 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2c60012020 con 0x7f2c68131ac0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.356+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6804bf60 con 0x7f2c68131ac0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.357+0000 7f2c5cff9640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c540386f0 0x7f2c5403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.357+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f2c6004b970 con 0x7f2c68131ac0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.357+0000 7f2c6c8f9640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c540386f0 0x7f2c5403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.357+0000 7f2c6c8f9640 10 monclient: get_auth_request con 0x7f2c540386f0 auth_method 0 2024-09-15T03:30:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.359+0000 7f2c6c8f9640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c540386f0 0x7f2c5403aba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2c6809feb0 tx=0x7f2c64032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.361+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2c6001d8e0 con 0x7f2c68131ac0 2024-09-15T03:30:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.361+0000 7f2c5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.361+0000 7f2c5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:00.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.515+0000 7f2c6eb5d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T03:30:00.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.515+0000 7f2c6eb5d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T03:30:00.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.515+0000 7f2c6eb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:00.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.515+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f2c6809c100 con 0x7f2c68131ac0 2024-09-15T03:30:00.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.998+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v25) v1 ==== 143+0+0 (secure 0 0 0) 0x7f2c6001d300 con 0x7f2c68131ac0 2024-09-15T03:30:00.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.998+0000 7f2c5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T03:30:00.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:00.998+0000 7f2c5cff9640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-15T03:30:01.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:01.089+0000 7f2c6eb5d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T03:30:01.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:01.089+0000 7f2c6eb5d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T03:30:01.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:01.089+0000 7f2c6eb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:01.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:01.089+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f2c68137370 con 0x7f2c68131ac0 2024-09-15T03:30:02.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.008+0000 7f2c5cff9640 1 -- 172.21.15.181:0/115594869 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v26) v1 ==== 143+0+0 (secure 0 0 0) 0x7f2c6001d8e0 con 0x7f2c68131ac0 2024-09-15T03:30:02.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.008+0000 7f2c5cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-15T03:30:02.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.008+0000 7f2c5cff9640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-15T03:30:02.009 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 3 pg_num_max to 0 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c540386f0 msgr2=0x7f2c5403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c540386f0 0x7f2c5403aba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2c6809feb0 tx=0x7f2c64032000 comp rx=0 tx=0).stop 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 10 monclient: shutdown 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 msgr2=0x7f2c68141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68141800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000bbb0 tx=0x7f2c600097b0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 shutdown_connections 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c540386f0 0x7f2c5403aba0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c68131ac0 0x7f2c68141800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c6809ff50 0x7f2c681412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 --2- 172.21.15.181:0/115594869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c68098150 0x7f2c68138170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 >> 172.21.15.181:0/115594869 conn(0x7f2c680a5190 msgr2=0x7f2c680a3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 shutdown_connections 2024-09-15T03:30:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.010+0000 7f2c6eb5d640 1 -- 172.21.15.181:0/115594869 wait complete. 2024-09-15T03:30:02.022 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-15T03:30:02.022 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:02.022 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.098+0000 7f796d80f640 10 monclient: get_monmap_and_config 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.098+0000 7f796d80f640 10 monclient: build_initial_monmap 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.098+0000 7f796d80f640 10 monclient: monmap: 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:02.098912+0000 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:02.098912+0000 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.099 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:02.100 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:02.100 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:02.100 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:02.100 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:02.100 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.098+0000 7f796d80f640 1 Processor -- start 2024-09-15T03:30:02.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.098+0000 7f796d80f640 1 -- start start 2024-09-15T03:30:02.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.098+0000 7f796d80f640 10 monclient: init 2024-09-15T03:30:02.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:02.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): picked mon.noname-a con 0x7f796809ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f79681321a0 0x7f7968068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): picked mon.noname-c con 0x7f79681321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f796809f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7968067420 con 0x7f796809f020 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796809fdc0 con 0x7f796809ec50 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79680a0510 con 0x7f79681321a0 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f796d80f640 10 monclient(hunting): authenticate will time out at 806.580248s 2024-09-15T03:30:02.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f79667fc640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f79681321a0 msgr2=0x7f7968068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f79667fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f79681321a0 0x7f7968068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35942/0 (socket says 172.21.15.181:35942) 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 -- 172.21.15.181:0/2727577586 learned_addr learned my addr 172.21.15.181:0/2727577586 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): get_auth_request con 0x7f796809ec50 auth_method 0 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f79677fe640 1 --1- 172.21.15.181:0/2727577586 >> v1:172.21.15.153:6789/0 conn(0x7f796809f020 0x7f7968069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:49940/0 (socket says 172.21.15.181:49940) 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient(hunting): handle_auth_done global_id 34254 payload 274 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 -- 172.21.15.181:0/2727577586 >> v2:172.21.15.181:3300/0 conn(0x7f79681321a0 msgr2=0x7f7968068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2727577586 >> v2:172.21.15.181:3300/0 conn(0x7f79681321a0 0x7f7968068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 -- 172.21.15.181:0/2727577586 >> v1:172.21.15.153:6789/0 conn(0x7f796809f020 legacy=0x7f7968069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 monclient: found mon.noname-a 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.099+0000 7f7966ffd640 1 -- 172.21.15.181:0/2727577586 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680a0200 con 0x7f796809ec50 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.102+0000 7f7966ffd640 1 -- 172.21.15.181:0/2727577586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 msgr2=0x7f7968131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.102+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2727577586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 0x7f7968131c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79680a2b00 tx=0x7f7958009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69818f7fcc2e50 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 1 -- 172.21.15.181:0/2727577586 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795802aea0 con 0x7f796809ec50 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: dump: 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.111 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 5 monclient: authenticate success, global_id 34254 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 1 -- 172.21.15.181:0/2727577586 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795802d050 con 0x7f796809ec50 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:02.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 4 set_mon_vals no callback set 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 1 -- 172.21.15.181:0/2727577586 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958007840 con 0x7f796809ec50 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f7965ffb640 10 monclient: dump: 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.113 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 10 monclient: shutdown 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 -- 172.21.15.181:0/2727577586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 msgr2=0x7f7968131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 --2- 172.21.15.181:0/2727577586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 0x7f7968131c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79680a2b00 tx=0x7f7958009840 comp rx=0 tx=0).stop 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 -- 172.21.15.181:0/2727577586 shutdown_connections 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 --2- 172.21.15.181:0/2727577586 >> v2:172.21.15.181:3300/0 conn(0x7f79681321a0 0x7f7968068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 --2- 172.21.15.181:0/2727577586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 -- 172.21.15.181:0/2727577586 >> 172.21.15.181:0/2727577586 conn(0x7f79680a5d40 msgr2=0x7f79680a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 -- 172.21.15.181:0/2727577586 shutdown_connections 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.103+0000 7f796d80f640 1 -- 172.21.15.181:0/2727577586 wait complete. 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.104+0000 7f796d80f640 10 monclient: build_initial_monmap 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.104+0000 7f796d80f640 10 monclient: monmap: 2024-09-15T03:30:02.114 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.104+0000 7f796d80f640 1 Processor -- start 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.104+0000 7f796d80f640 1 -- start start 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.104+0000 7f796d80f640 10 monclient: init 2024-09-15T03:30:02.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f796809ec50 0x7f796809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): picked mon.noname-b con 0x7f796809ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): picked mon.noname-c con 0x7f796809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79681321a0 0x7f796811b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): picked mon.noname-a con 0x7f79681321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796809c000 con 0x7f79681321a0 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796809dd80 con 0x7f796809ec50 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796811e030 con 0x7f796809f020 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f796d80f640 10 monclient(hunting): authenticate will time out at 806.586156s 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f7966ffd640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f796809ec50 msgr2=0x7f796809ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f7966ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f796809ec50 0x7f796809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79681321a0 0x7f796811b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79681321a0 0x7f796811b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35948/0 (socket says 172.21.15.181:35948) 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 -- 172.21.15.181:0/2941671233 learned_addr learned my addr 172.21.15.181:0/2941671233 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): get_auth_request con 0x7f796809f020 auth_method 0 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): get_auth_request con 0x7f79681321a0 auth_method 0 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient(hunting): handle_auth_done global_id 34257 payload 824 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 -- 172.21.15.181:0/2941671233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 msgr2=0x7f7968130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 -- 172.21.15.181:0/2941671233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f796809ec50 msgr2=0x7f796809ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f796809ec50 0x7f796809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79677fe640 1 -- 172.21.15.181:0/2941671233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958009ae0 con 0x7f79681321a0 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.105+0000 7f79667fc640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f79677fe640 1 -- 172.21.15.181:0/2941671233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79681321a0 msgr2=0x7f796811b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f79677fe640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79681321a0 0x7f796811b760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7960007fd0 tx=0x7f7960007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79600074f0 con 0x7f79681321a0 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: dump: 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: _finish_auth 0 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7960007670 con 0x7f79681321a0 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960005a00 con 0x7f79681321a0 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f794effd640 10 monclient: dump: 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.121 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 5 monclient: authenticate success, global_id 34257 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 10 monclient: _renew_subs 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f79657fa640 4 set_mon_vals no callback set 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 1 -- 172.21.15.181:0/2941671233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7968141a40 con 0x7f79681321a0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 10 monclient: _renew_subs 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.108+0000 7f796d80f640 1 -- 172.21.15.181:0/2941671233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7968141560 con 0x7f79681321a0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f796001b3b0 con 0x7f79681321a0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f794effd640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79540386f0 0x7f795403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f796d80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f796d80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f796d80f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f796004ade0 con 0x7f79681321a0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f796d80f640 1 -- 172.21.15.181:0/2941671233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796809b6b0 con 0x7f79681321a0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79540386f0 0x7f795403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.109+0000 7f7966ffd640 10 monclient: get_auth_request con 0x7f79540386f0 auth_method 0 2024-09-15T03:30:02.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.112+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2941671233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79540386f0 0x7f795403aba0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7968132a50 tx=0x7f7958036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:02.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.114+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7960017020 con 0x7f79681321a0 2024-09-15T03:30:02.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.114+0000 7f794effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:02.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.114+0000 7f794effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:02.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.266+0000 7f796d80f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:30:02.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.266+0000 7f796d80f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:30:02.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.266+0000 7f796d80f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.266+0000 7f796d80f640 1 -- 172.21.15.181:0/2941671233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f79681419e0 con 0x7f79681321a0 2024-09-15T03:30:02.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.267+0000 7f794effd640 1 -- 172.21.15.181:0/2941671233 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v26) v1 ==== 158+0+0 (secure 0 0 0) 0x7f796001a360 con 0x7f79681321a0 2024-09-15T03:30:02.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.267+0000 7f794effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-15T03:30:02.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.267+0000 7f794effd640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-15T03:30:02.268 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-15T03:30:02.271 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:02.272 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-15T03:30:02.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.345+0000 7f48e66e8640 10 monclient: get_monmap_and_config 2024-09-15T03:30:02.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.345+0000 7f48e66e8640 10 monclient: build_initial_monmap 2024-09-15T03:30:02.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.345+0000 7f48e66e8640 10 monclient: monmap: 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:02.346218+0000 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:02.346218+0000 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:02.348 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:02.348 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:02.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 Processor -- start 2024-09-15T03:30:02.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 -- start start 2024-09-15T03:30:02.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient: init 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f48e0131ac0 0x7f48e0131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): picked mon.noname-a con 0x7f48e0131ac0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48e009ff50 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e01323d0 0x7f48e0098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): picked mon.noname-b con 0x7f48e01323d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e00a1610 con 0x7f48e009ff50 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e01327d0 con 0x7f48e0131ac0 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e00a1330 con 0x7f48e01323d0 2024-09-15T03:30:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48e66e8640 10 monclient(hunting): authenticate will time out at 806.827711s 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.346+0000 7f48df7fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f48e009ff50 legacy=0x7f48e0098150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f48e0131ac0 0x7f48e0131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e01323d0 0x7f48e0098860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f48e0131ac0 0x7f48e0131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35954/0 (socket says 172.21.15.181:35954) 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 -- 172.21.15.181:0/906730349 learned_addr learned my addr 172.21.15.181:0/906730349 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): get_auth_request con 0x7f48e0131ac0 auth_method 0 2024-09-15T03:30:02.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): get_auth_request con 0x7f48e01323d0 auth_method 0 2024-09-15T03:30:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient(hunting): handle_auth_done global_id 34260 payload 274 2024-09-15T03:30:02.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 -- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e01323d0 msgr2=0x7f48e0098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:02.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 --2- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e01323d0 0x7f48e0098860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 -- 172.21.15.181:0/906730349 >> v1:172.21.15.181:6789/0 conn(0x7f48e009ff50 legacy=0x7f48e0098150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:02.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:02.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 monclient: found mon.noname-a 2024-09-15T03:30:02.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:02.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48dffff640 1 -- 172.21.15.181:0/906730349 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e0139e10 con 0x7f48e0131ac0 2024-09-15T03:30:02.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.347+0000 7f48e4c85640 1 --2- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e01323d0 0x7f48e0098860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:02.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48dffff640 1 -- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 msgr2=0x7f48e0131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:02.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48dffff640 1 --2- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0131e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009910 tx=0x7f48d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c5bbf11a575de5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:02.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 1 -- 172.21.15.181:0/906730349 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d002aea0 con 0x7f48e0131ac0 2024-09-15T03:30:02.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:02.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: dump: 2024-09-15T03:30:02.365 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.365 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.367 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:02.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 1 -- 172.21.15.181:0/906730349 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0032440 con 0x7f48e0131ac0 2024-09-15T03:30:02.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:02.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 1 -- 172.21.15.181:0/906730349 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00327e0 con 0x7f48e0131ac0 2024-09-15T03:30:02.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48deffd640 10 monclient: dump: 2024-09-15T03:30:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.370 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48e66e8640 5 monclient: authenticate success, global_id 34260 2024-09-15T03:30:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48e66e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.350+0000 7f48e66e8640 4 set_mon_vals no callback set 2024-09-15T03:30:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 10 monclient: shutdown 2024-09-15T03:30:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 -- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 msgr2=0x7f48e0131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 --2- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0131e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009910 tx=0x7f48d00091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 -- 172.21.15.181:0/906730349 shutdown_connections 2024-09-15T03:30:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 --2- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e01323d0 0x7f48e0098860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 --2- 172.21.15.181:0/906730349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0131e90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 -- 172.21.15.181:0/906730349 >> 172.21.15.181:0/906730349 conn(0x7f48e00a5190 msgr2=0x7f48e00a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 -- 172.21.15.181:0/906730349 shutdown_connections 2024-09-15T03:30:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 1 -- 172.21.15.181:0/906730349 wait complete. 2024-09-15T03:30:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 10 monclient: build_initial_monmap 2024-09-15T03:30:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.351+0000 7f48e66e8640 10 monclient: monmap: 2024-09-15T03:30:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:02.375 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:02.375 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 Processor -- start 2024-09-15T03:30:02.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 -- start start 2024-09-15T03:30:02.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient: init 2024-09-15T03:30:02.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e009ff50 0x7f48e01411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): picked mon.noname-c con 0x7f48e009ff50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0137fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): picked mon.noname-a con 0x7f48e0131ac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e01323d0 0x7f48e012eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): picked mon.noname-b con 0x7f48e01323d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0098470 con 0x7f48e0131ac0 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0130b90 con 0x7f48e01323d0 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0098e70 con 0x7f48e009ff50 2024-09-15T03:30:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e66e8640 10 monclient(hunting): authenticate will time out at 806.833727s 2024-09-15T03:30:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e4c85640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e01323d0 msgr2=0x7f48e012eaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.352+0000 7f48e4c85640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e01323d0 0x7f48e012eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0137fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e009ff50 0x7f48e01411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0137fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35958/0 (socket says 172.21.15.181:35958) 2024-09-15T03:30:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 -- 172.21.15.181:0/1454470793 learned_addr learned my addr 172.21.15.181:0/1454470793 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): get_auth_request con 0x7f48e0131ac0 auth_method 0 2024-09-15T03:30:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): get_auth_request con 0x7f48e009ff50 auth_method 0 2024-09-15T03:30:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient(hunting): handle_auth_done global_id 34263 payload 824 2024-09-15T03:30:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 -- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e009ff50 msgr2=0x7f48e01411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e009ff50 0x7f48e01411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 -- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e01323d0 msgr2=0x7f48e012eaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e01323d0 0x7f48e012eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48df7fe640 1 -- 172.21.15.181:0/1454470793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8009a20 con 0x7f48e0131ac0 2024-09-15T03:30:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.353+0000 7f48dffff640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e009ff50 0x7f48e01411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.355+0000 7f48df7fe640 1 -- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 msgr2=0x7f48e0137fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.355+0000 7f48df7fe640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0137fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f48d400b5c0 tx=0x7f48d4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4003d90 con 0x7f48e0131ac0 2024-09-15T03:30:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: dump: 2024-09-15T03:30:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.386 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 5 monclient: authenticate success, global_id 34263 2024-09-15T03:30:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d4006150 con 0x7f48e0131ac0 2024-09-15T03:30:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4006590 con 0x7f48e0131ac0 2024-09-15T03:30:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 monclient: _renew_subs 2024-09-15T03:30:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d0009450 con 0x7f48e0131ac0 2024-09-15T03:30:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 10 monclient: dump: 2024-09-15T03:30:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48de7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 monclient: _renew_subs 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e0130680 con 0x7f48e0131ac0 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f48d40066f0 con 0x7f48e0131ac0 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.356+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e004bf60 con 0x7f48e0131ac0 2024-09-15T03:30:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.357+0000 7f48c7fff640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48cc0386f0 0x7f48cc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.357+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f48d404b1f0 con 0x7f48e0131ac0 2024-09-15T03:30:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.357+0000 7f48dffff640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48cc0386f0 0x7f48cc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.357+0000 7f48dffff640 10 monclient: get_auth_request con 0x7f48cc0386f0 auth_method 0 2024-09-15T03:30:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.359+0000 7f48dffff640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48cc0386f0 0x7f48cc03aba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f48d0031000 tx=0x7f48d0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.361+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f48d400e8c0 con 0x7f48e0131ac0 2024-09-15T03:30:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.361+0000 7f48c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.361+0000 7f48c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.514+0000 7f48e66e8640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.514+0000 7f48e66e8640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.514+0000 7f48e66e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:02.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:02.514+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f48e00a3370 con 0x7f48e0131ac0 2024-09-15T03:30:03.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.009+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v27) v1 ==== 176+0+0 (secure 0 0 0) 0x7f48d4017070 con 0x7f48e0131ac0 2024-09-15T03:30:03.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.009+0000 7f48c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:03.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.009+0000 7f48c7fff640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-15T03:30:03.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.102+0000 7f48e66e8640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:03.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.102+0000 7f48e66e8640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:03.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.102+0000 7f48e66e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.102+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f48e0138e80 con 0x7f48e0131ac0 2024-09-15T03:30:03.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.103+0000 7f48c7fff640 1 -- 172.21.15.181:0/1454470793 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v27) v1 ==== 183+0+0 (secure 0 0 0) 0x7f48d400e2c0 con 0x7f48e0131ac0 2024-09-15T03:30:03.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.103+0000 7f48c7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:03.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.103+0000 7f48c7fff640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-15T03:30:03.104 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool1234' does not exist 2024-09-15T03:30:03.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48cc0386f0 msgr2=0x7f48cc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48cc0386f0 0x7f48cc03aba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f48d0031000 tx=0x7f48d0032000 comp rx=0 tx=0).stop 2024-09-15T03:30:03.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 10 monclient: shutdown 2024-09-15T03:30:03.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:03.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 msgr2=0x7f48e0137fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0137fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f48d400b5c0 tx=0x7f48d4009120 comp rx=0 tx=0).stop 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 shutdown_connections 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f48cc0386f0 0x7f48cc03aba0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e01323d0 0x7f48e012eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0131ac0 0x7f48e0137fc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 --2- 172.21.15.181:0/1454470793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48e009ff50 0x7f48e01411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 >> 172.21.15.181:0/1454470793 conn(0x7f48e00a5190 msgr2=0x7f48e0098680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 shutdown_connections 2024-09-15T03:30:03.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.105+0000 7f48e66e8640 1 -- 172.21.15.181:0/1454470793 wait complete. 2024-09-15T03:30:03.115 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-15T03:30:03.115 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:03.116 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.194+0000 7f091c39d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.194+0000 7f091c39d640 10 monclient: build_initial_monmap 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.194+0000 7f091c39d640 10 monclient: monmap: 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:03.194887+0000 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:03.194887+0000 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.195 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:03.196 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:03.196 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:03.196 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:03.196 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:03.196 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.194+0000 7f091c39d640 1 Processor -- start 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.194+0000 7f091c39d640 1 -- start start 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.194+0000 7f091c39d640 10 monclient: init 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f091409ec50 0x7f0914131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f091409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f09141321a0 0x7f0914068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f09141321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f091409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0914067420 con 0x7f091409ec50 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091409fdc0 con 0x7f09141321a0 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140a0510 con 0x7f091409f020 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091c39d640 10 monclient(hunting): authenticate will time out at 807.676546s 2024-09-15T03:30:03.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f0919938640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f09141321a0 msgr2=0x7f0914068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f0919938640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f09141321a0 0x7f0914068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f091409ec50 0x7f0914131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f091409ec50 0x7f0914131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35970/0 (socket says 172.21.15.181:35970) 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 1 -- 172.21.15.181:0/3802497351 learned_addr learned my addr 172.21.15.181:0/3802497351 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): get_auth_request con 0x7f091409ec50 auth_method 0 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): get_auth_request con 0x7f091409f020 auth_method 0 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.195+0000 7f091a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 10 monclient(hunting): handle_auth_done global_id 34266 payload 274 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 1 -- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 msgr2=0x7f0914069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 1 --2- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 1 -- 172.21.15.181:0/3802497351 >> v2:172.21.15.181:3300/0 conn(0x7f09141321a0 msgr2=0x7f0914068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 1 --2- 172.21.15.181:0/3802497351 >> v2:172.21.15.181:3300/0 conn(0x7f09141321a0 0x7f0914068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 10 monclient: _finish_hunting 0 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 1 monclient: found mon.noname-a 2024-09-15T03:30:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:03.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a139640 1 -- 172.21.15.181:0/3802497351 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09140a0200 con 0x7f091409ec50 2024-09-15T03:30:03.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.196+0000 7f091a93a640 1 --2- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:03.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.198+0000 7f091a139640 1 -- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f091409ec50 msgr2=0x7f0914131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:03.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.198+0000 7f091a139640 1 --2- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f091409ec50 0x7f0914131c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09140a2b00 tx=0x7f09100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8adf16dc4df096 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:03.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 1 -- 172.21.15.181:0/3802497351 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910032440 con 0x7f091409ec50 2024-09-15T03:30:03.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: dump: 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: _finish_auth 0 2024-09-15T03:30:03.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 1 -- 172.21.15.181:0/3802497351 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09100325a0 con 0x7f091409ec50 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f091c39d640 5 monclient: authenticate success, global_id 34266 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f091c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f091c39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f091c39d640 4 set_mon_vals no callback set 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 1 -- 172.21.15.181:0/3802497351 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910032940 con 0x7f091409ec50 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.199+0000 7f0919137640 10 monclient: dump: 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.211 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 10 monclient: shutdown 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 -- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f091409ec50 msgr2=0x7f0914131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 --2- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f091409ec50 0x7f0914131c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09140a2b00 tx=0x7f09100091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 -- 172.21.15.181:0/3802497351 shutdown_connections 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 --2- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 --2- 172.21.15.181:0/3802497351 >> v2:172.21.15.181:3300/0 conn(0x7f09141321a0 0x7f0914068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 --2- 172.21.15.181:0/3802497351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f091409ec50 0x7f0914131c60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 -- 172.21.15.181:0/3802497351 >> 172.21.15.181:0/3802497351 conn(0x7f09140a5d40 msgr2=0x7f09140a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 -- 172.21.15.181:0/3802497351 shutdown_connections 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 -- 172.21.15.181:0/3802497351 wait complete. 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 10 monclient: build_initial_monmap 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 10 monclient: monmap: 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:03.213 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.200+0000 7f091c39d640 1 Processor -- start 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 -- start start 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient: init 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f091409ec50 0x7f091409a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f091409ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f091409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a139640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f091409ec50 msgr2=0x7f091409a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09141321a0 0x7f0914140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f09141321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091409b570 con 0x7f09141321a0 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a139640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f091409ec50 0x7f091409a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091409c000 con 0x7f091409ec50 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140a40a0 con 0x7f091409f020 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:03.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091c39d640 10 monclient(hunting): authenticate will time out at 807.682536s 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09141321a0 0x7f0914140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f0919938640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09141321a0 0x7f0914140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:35984/0 (socket says 172.21.15.181:35984) 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 1 -- 172.21.15.181:0/1581528961 learned_addr learned my addr 172.21.15.181:0/1581528961 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 10 monclient(hunting): get_auth_request con 0x7f09141321a0 auth_method 0 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f0919938640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:56166/0 (socket says 172.21.15.181:56166) 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f091a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f0919938640 10 monclient(hunting): get_auth_request con 0x7f091409f020 auth_method 0 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f0919938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f0919938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.201+0000 7f0919938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f0919938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f0919938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f0919938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 10 monclient(hunting): handle_auth_done global_id 34269 payload 824 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 1 -- 172.21.15.181:0/1581528961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 msgr2=0x7f0914130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 1 -- 172.21.15.181:0/1581528961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f091409ec50 msgr2=0x7f091409a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f091409ec50 0x7f091409a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 1 monclient: found mon.noname-a 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f091a93a640 1 -- 172.21.15.181:0/1581528961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c009a20 con 0x7f09141321a0 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.202+0000 7f0919938640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f091409f020 0x7f0914130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091a93a640 1 -- 172.21.15.181:0/1581528961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09141321a0 msgr2=0x7f0914140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091a93a640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09141321a0 0x7f0914140ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f090c00c0f0 tx=0x7f090c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 1 -- 172.21.15.181:0/1581528961 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c00ced0 con 0x7f09141321a0 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: dump: 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 1 -- 172.21.15.181:0/1581528961 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c00f040 con 0x7f09141321a0 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 5 monclient: authenticate success, global_id 34269 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 10 monclient: _renew_subs 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 1 -- 172.21.15.181:0/1581528961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0910009450 con 0x7f09141321a0 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 1 -- 172.21.15.181:0/1581528961 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c0148e0 con 0x7f09141321a0 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:03.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f090a7fc640 10 monclient: dump: 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f0918936640 4 set_mon_vals no callback set 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 10 monclient: _renew_subs 2024-09-15T03:30:03.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.204+0000 7f091c39d640 1 -- 172.21.15.181:0/1581528961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091411cdf0 con 0x7f09141321a0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f091c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f091c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f091c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f091c39d640 1 -- 172.21.15.181:0/1581528961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09140008d0 con 0x7f09141321a0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f090a7fc640 1 -- 172.21.15.181:0/1581528961 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f090c003020 con 0x7f09141321a0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f090a7fc640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f08f80386f0 0x7f08f803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.205+0000 7f090a7fc640 1 -- 172.21.15.181:0/1581528961 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f090c04c4a0 con 0x7f09141321a0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.206+0000 7f091a139640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f08f80386f0 0x7f08f803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.206+0000 7f091a139640 10 monclient: get_auth_request con 0x7f08f80386f0 auth_method 0 2024-09-15T03:30:03.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.208+0000 7f091a139640 1 --2- 172.21.15.181:0/1581528961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f08f80386f0 0x7f08f803aba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09140719d0 tx=0x7f0910032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:03.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.210+0000 7f090a7fc640 1 -- 172.21.15.181:0/1581528961 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f090c012db0 con 0x7f09141321a0 2024-09-15T03:30:03.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.210+0000 7f090a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:03.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.210+0000 7f090a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:03.366 INFO:tasks.workunit.client.0.smithi181.stderr:foo-rule not valid: foo-rule doesn't represent a float 2024-09-15T03:30:03.366 INFO:tasks.workunit.client.0.smithi181.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-15T03:30:03.366 INFO:tasks.workunit.client.0.smithi181.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] : create pool 2024-09-15T03:30:03.366 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: invalid command 2024-09-15T03:30:03.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:03.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.444+0000 7f8ff92e8640 10 monclient: get_monmap_and_config 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.444+0000 7f8ff92e8640 10 monclient: build_initial_monmap 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.444+0000 7f8ff92e8640 10 monclient: monmap: 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:03.444984+0000 2024-09-15T03:30:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:03.444984+0000 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:03.446 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.444+0000 7f8ff92e8640 1 Processor -- start 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.444+0000 7f8ff92e8640 1 -- start start 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.444+0000 7f8ff92e8640 10 monclient: init 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ff409ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ff409f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ff4069d10 0x7f8ff413a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8ff4069d10 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff406a120 con 0x7f8ff409ec50 2024-09-15T03:30:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff4067420 con 0x7f8ff409f020 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff409fdc0 con 0x7f8ff4069d10 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff92e8640 10 monclient(hunting): authenticate will time out at 807.926401s 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff27fc640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f8ff409f020 legacy=0x7f8ff4136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff37fe640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f8ff4069d10 msgr2=0x7f8ff413a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff37fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ff4069d10 0x7f8ff413a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8ff409ec50 0x7f8ff4131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:49948/0 (socket says 172.21.15.181:49948) 2024-09-15T03:30:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.445+0000 7f8ff2ffd640 1 -- 172.21.15.181:0/3594672802 learned_addr learned my addr 172.21.15.181:0/3594672802 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 1 -- 172.21.15.181:0/3594672802 >> v1:172.21.15.153:6789/0 conn(0x7f8ff409ec50 legacy=0x7f8ff4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416873580 0 0) 0x7f8fe4001450 con 0x7f8ff409ec50 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): my global_id is 34272 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc0034e0 con 0x7f8ff409ec50 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2071010246 0 0) 0x7f8fdc0034e0 con 0x7f8ff409ec50 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 >> v2:172.21.15.181:3300/0 conn(0x7f8ff4069d10 msgr2=0x7f8ff413a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 --2- 172.21.15.181:0/3594672802 >> v2:172.21.15.181:3300/0 conn(0x7f8ff4069d10 0x7f8ff413a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 >> v1:172.21.15.181:6789/0 conn(0x7f8ff409f020 legacy=0x7f8ff4136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40a0510 con 0x7f8ff409ec50 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f8fe4001970 con 0x7f8ff409ec50 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:30:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: dump: 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 >> v1:172.21.15.153:6789/0 conn(0x7f8ff409ec50 legacy=0x7f8ff4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8fdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): picked mon.c con 0x7f8fdc0039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc004780 0x7f8fdc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): picked mon.a con 0x7f8fdc004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fdc005280 0x7f8fdc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): picked mon.b con 0x7f8fdc005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff409fdc0 con 0x7f8fdc004780 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc00c240 con 0x7f8fdc005280 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc00c400 con 0x7f8fdc0039d0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff37fe640 1 -- 172.21.15.181:0/3594672802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fdc005280 msgr2=0x7f8fdc00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff37fe640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fdc005280 0x7f8fdc00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8fdc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc004780 0x7f8fdc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): get_auth_request con 0x7f8fdc0039d0 auth_method 0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): get_auth_request con 0x7f8fdc004780 auth_method 0 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.448+0000 7f8ff27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_done global_id 34272 payload 293 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 1 -- 172.21.15.181:0/3594672802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fdc005280 msgr2=0x7f8fdc00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fdc005280 0x7f8fdc00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 1 -- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc004780 msgr2=0x7f8fdc004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc004780 0x7f8fdc004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 1 monclient: found mon.c 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff37fe640 1 -- 172.21.15.181:0/3594672802 reap_dead start 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 1 -- 172.21.15.181:0/3594672802 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0034e0 con 0x7f8fdc0039d0 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff92e8640 5 monclient: authenticate success, global_id 34272 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff92e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.449+0000 7f8ff27fc640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc004780 0x7f8fdc004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff2ffd640 1 -- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 msgr2=0x7f8fdc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8fdc003dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d660 tx=0x7f8ff409fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bab43cecb349b036 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4008220 con 0x7f8fdc0039d0 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff1ffb640 10 monclient: dump: 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.462 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8fdc003dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d660 tx=0x7f8ff409fca0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f8fe4008550 mon_map magic: 0 v1 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff92e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe40083a0 con 0x7f8fdc0039d0 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff92e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.452+0000 7f8ff92e8640 4 set_mon_vals no callback set 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 10 monclient: shutdown 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 -- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 msgr2=0x7f8fdc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8fdc003dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d660 tx=0x7f8ff409fca0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 -- 172.21.15.181:0/3594672802 shutdown_connections 2024-09-15T03:30:03.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff1ffb640 1 -- 172.21.15.181:0/3594672802 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f8fe4008550 con 0x7f8fdc0039d0 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff1ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/3594672802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8fdc003dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 -- 172.21.15.181:0/3594672802 >> 172.21.15.181:0/3594672802 conn(0x7f8ff40a5d40 msgr2=0x7f8ff40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 -- 172.21.15.181:0/3594672802 shutdown_connections 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 1 -- 172.21.15.181:0/3594672802 wait complete. 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.453+0000 7f8ff92e8640 10 monclient: build_initial_monmap 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient: monmap: 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:03.465 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 Processor -- start 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 -- start start 2024-09-15T03:30:03.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient: init 2024-09-15T03:30:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8ff409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fdc0039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ff411bc70 0x7f8ff4130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff411bc70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 0x7f8ff4140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): picked mon.noname-a con 0x7f8ff4131090 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff409b640 con 0x7f8ff4131090 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff409dd10 con 0x7f8ff411bc70 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff411dc70 con 0x7f8fdc0039d0 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff92e8640 10 monclient(hunting): authenticate will time out at 807.935840s 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff27fc640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ff411bc70 msgr2=0x7f8ff4130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.454+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ff411bc70 0x7f8ff4130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 0x7f8ff4140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8ff409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 0x7f8ff4140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36000/0 (socket says 172.21.15.181:36000) 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8ff409aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:56172/0 (socket says 172.21.15.181:56172) 2024-09-15T03:30:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 -- 172.21.15.181:0/2371839388 learned_addr learned my addr 172.21.15.181:0/2371839388 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): get_auth_request con 0x7f8ff4131090 auth_method 0 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): get_auth_request con 0x7f8fdc0039d0 auth_method 0 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient(hunting): handle_auth_done global_id 34275 payload 824 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 -- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 msgr2=0x7f8ff409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8ff409aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 -- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ff411bc70 msgr2=0x7f8ff4130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ff411bc70 0x7f8ff4130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff37fe640 1 -- 172.21.15.181:0/2371839388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff411df50 con 0x7f8ff4131090 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.455+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8ff409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.457+0000 7f8ff37fe640 1 -- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 msgr2=0x7f8ff4140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.457+0000 7f8ff37fe640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 0x7f8ff4140f90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40a1b50 tx=0x7f8fec0096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00ca80 con 0x7f8ff4131090 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: dump: 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec00cbe0 con 0x7f8ff4131090 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 5 monclient: authenticate success, global_id 34275 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 monclient: _renew_subs 2024-09-15T03:30:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff411e230 con 0x7f8ff4131090 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00cd90 con 0x7f8ff4131090 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 10 monclient: dump: 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 monclient: _renew_subs 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff411c880 con 0x7f8ff4131090 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff17fa640 4 set_mon_vals no callback set 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8fec012020 con 0x7f8ff4131090 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.458+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff40008d0 con 0x7f8ff4131090 2024-09-15T03:30:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.459+0000 7f8fdaffd640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fe0038720 0x7f8fe003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.459+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8fec04b330 con 0x7f8ff4131090 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.459+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fe0038720 0x7f8fe003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.459+0000 7f8ff2ffd640 10 monclient: get_auth_request con 0x7f8fe0038720 auth_method 0 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.461+0000 7f8ff2ffd640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fe0038720 0x7f8fe003abd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8fe402f000 tx=0x7f8fe4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.463+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8fec0138e0 con 0x7f8ff4131090 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.463+0000 7f8fdaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.463+0000 7f8fdaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:03.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.620+0000 7f8ff92e8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T03:30:03.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.620+0000 7f8ff92e8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T03:30:03.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.620+0000 7f8ff92e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:03.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:03.620+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f8ff40a0200 con 0x7f8ff4131090 2024-09-15T03:30:04.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.029+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v28) v1 ==== 154+0+0 (secure 0 0 0) 0x7f8fec01d300 con 0x7f8ff4131090 2024-09-15T03:30:04.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.029+0000 7f8fdaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T03:30:04.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.029+0000 7f8fdaffd640 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-15T03:30:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.130+0000 7f8ff92e8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T03:30:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.130+0000 7f8ff92e8640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T03:30:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.130+0000 7f8ff92e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.130+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f8ff4064900 con 0x7f8ff4131090 2024-09-15T03:30:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.131+0000 7f8fdaffd640 1 -- 172.21.15.181:0/2371839388 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8fec0132e0 con 0x7f8ff4131090 2024-09-15T03:30:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.131+0000 7f8fdaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-15T03:30:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.131+0000 7f8fdaffd640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-15T03:30:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' already exists 2024-09-15T03:30:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fe0038720 msgr2=0x7f8fe003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fe0038720 0x7f8fe003abd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8fe402f000 tx=0x7f8fe4030000 comp rx=0 tx=0).stop 2024-09-15T03:30:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 10 monclient: shutdown 2024-09-15T03:30:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 msgr2=0x7f8ff4140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 0x7f8ff4140f90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40a1b50 tx=0x7f8fec0096e0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 shutdown_connections 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fe0038720 0x7f8fe003abd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ff4131090 0x7f8ff4140f90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ff411bc70 0x7f8ff4130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 --2- 172.21.15.181:0/2371839388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0039d0 0x7f8ff409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 >> 172.21.15.181:0/2371839388 conn(0x7f8ff40a5d40 msgr2=0x7f8ff409df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 shutdown_connections 2024-09-15T03:30:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.133+0000 7f8ff92e8640 1 -- 172.21.15.181:0/2371839388 wait complete. 2024-09-15T03:30:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.219+0000 7f1453cc0640 10 monclient: get_monmap_and_config 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.219+0000 7f1453cc0640 10 monclient: build_initial_monmap 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.219+0000 7f1453cc0640 10 monclient: monmap: 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:04.219991+0000 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:04.219991+0000 2024-09-15T03:30:04.220 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:04.221 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.219+0000 7f1453cc0640 1 Processor -- start 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 1 -- start start 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient: init 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f144c09ec50 0x7f144c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f144c09ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f144c1321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f144c09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c067420 con 0x7f144c1321a0 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c09fdc0 con 0x7f144c09f020 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c0a0510 con 0x7f144c09ec50 2024-09-15T03:30:04.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:04.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f145225d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f144c09f020 legacy=0x7f144c069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:04.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1451a5c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f144c09ec50 msgr2=0x7f144c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:04.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1453cc0640 10 monclient(hunting): authenticate will time out at 808.701681s 2024-09-15T03:30:04.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.220+0000 7f1451a5c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f144c09ec50 0x7f144c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:04.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.221+0000 7f145125b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f144c1321a0 0x7f144c068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:49958/0 (socket says 172.21.15.181:49958) 2024-09-15T03:30:04.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.221+0000 7f145125b640 1 -- 172.21.15.181:0/1532785069 learned_addr learned my addr 172.21.15.181:0/1532785069 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:04.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f145125b640 1 -- 172.21.15.181:0/1532785069 >> v1:172.21.15.153:6789/0 conn(0x7f144c1321a0 legacy=0x7f144c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:04.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277555105 0 0) 0x7f144c067420 con 0x7f144c1321a0 2024-09-15T03:30:04.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:04.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:04.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): my global_id is 34278 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14380034e0 con 0x7f144c1321a0 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2965428129 0 0) 0x7f14380034e0 con 0x7f144c1321a0 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 >> v2:172.21.15.181:3300/0 conn(0x7f144c09ec50 msgr2=0x7f144c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 --2- 172.21.15.181:0/1532785069 >> v2:172.21.15.181:3300/0 conn(0x7f144c09ec50 0x7f144c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 >> v1:172.21.15.181:6789/0 conn(0x7f144c09f020 legacy=0x7f144c069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0a0200 con 0x7f144c1321a0 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: _finish_auth 0 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:04.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f143c001720 con 0x7f144c1321a0 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: dump: 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:04.229 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 >> v1:172.21.15.153:6789/0 conn(0x7f144c1321a0 legacy=0x7f144c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f1438003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): picked mon.a con 0x7f14380039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1438004780 0x7f1438004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): picked mon.b con 0x7f1438004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1438005330 0x7f143800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): picked mon.c con 0x7f1438005330 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0a0510 con 0x7f14380039d0 2024-09-15T03:30:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143800c290 con 0x7f1438004780 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143800c450 con 0x7f1438005330 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f1450a5a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f145125b640 1 -- 172.21.15.181:0/1532785069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1438004780 msgr2=0x7f1438004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.223+0000 7f145125b640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1438004780 0x7f1438004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f1438003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): get_auth_request con 0x7f14380039d0 auth_method 0 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1438005330 0x7f143800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:04.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): get_auth_request con 0x7f1438005330 auth_method 0 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient(hunting): handle_auth_done global_id 34278 payload 293 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 -- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1438005330 msgr2=0x7f143800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:04.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1438005330 0x7f143800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 -- 172.21.15.181:0/1532785069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1438004780 msgr2=0x7f1438004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1438004780 0x7f1438004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 monclient: found mon.a 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 1 -- 172.21.15.181:0/1532785069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14380034e0 con 0x7f14380039d0 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 10 monclient: _finish_auth 0 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1451a5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1453cc0640 5 monclient: authenticate success, global_id 34278 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f1453cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1438005330 0x7f143800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:04.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.224+0000 7f145225d640 1 -- 172.21.15.181:0/1532785069 reap_dead start 2024-09-15T03:30:04.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1451a5c640 1 -- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 msgr2=0x7f1438003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:04.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1451a5c640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f1438003dc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f14480099c0 tx=0x7f1448009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee6f261bcccb547 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:04.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448003b80 con 0x7f14380039d0 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: dump: 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:04.241 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448003d20 con 0x7f14380039d0 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 1 -- 172.21.15.181:0/1532785069 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144802ab80 con 0x7f14380039d0 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 4 set_mon_vals no callback set 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1450a5a640 10 monclient: dump: 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 10 monclient: shutdown 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 1 -- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 msgr2=0x7f1438003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f1438003dc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f14480099c0 tx=0x7f1448009060 comp rx=0 tx=0).stop 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 1 -- 172.21.15.181:0/1532785069 shutdown_connections 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 1 --2- 172.21.15.181:0/1532785069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f1438003dc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 1 -- 172.21.15.181:0/1532785069 >> 172.21.15.181:0/1532785069 conn(0x7f144c0a5d40 msgr2=0x7f144c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:04.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.227+0000 7f1453cc0640 1 -- 172.21.15.181:0/1532785069 shutdown_connections 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.228+0000 7f1453cc0640 1 -- 172.21.15.181:0/1532785069 wait complete. 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.228+0000 7f1453cc0640 10 monclient: build_initial_monmap 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.228+0000 7f1453cc0640 10 monclient: monmap: 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:04.244 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.228+0000 7f1453cc0640 1 Processor -- start 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.228+0000 7f1453cc0640 1 -- start start 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.228+0000 7f1453cc0640 10 monclient: init 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f144c140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f14380039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f144c141260 0x7f144c130a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f144c141260 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f144c130fd0 0x7f144c11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f144c130fd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c141630 con 0x7f14380039d0 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c1313c0 con 0x7f144c141260 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c09c000 con 0x7f144c130fd0 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1453cc0640 10 monclient(hunting): authenticate will time out at 808.710216s 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145125b640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f144c141260 msgr2=0x7f144c130a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1441ffb640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145125b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f144c141260 0x7f144c130a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f144c140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f144c140d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36018/0 (socket says 172.21.15.181:36018) 2024-09-15T03:30:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 -- 172.21.15.181:0/3019943111 learned_addr learned my addr 172.21.15.181:0/3019943111 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): get_auth_request con 0x7f14380039d0 auth_method 0 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f144c130fd0 0x7f144c11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): get_auth_request con 0x7f144c130fd0 auth_method 0 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f145225d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient(hunting): handle_auth_done global_id 34281 payload 824 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 -- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f144c130fd0 msgr2=0x7f144c11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f144c130fd0 0x7f144c11b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 -- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f144c141260 msgr2=0x7f144c130a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f144c141260 0x7f144c130a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 monclient: found mon.noname-a 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.229+0000 7f1451a5c640 1 -- 172.21.15.181:0/3019943111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c11eea0 con 0x7f14380039d0 2024-09-15T03:30:04.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1451a5c640 1 -- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 msgr2=0x7f144c140d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1451a5c640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f144c140d20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1448009a20 tx=0x7f1448030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448005610 con 0x7f14380039d0 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: dump: 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:04.249 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448005770 con 0x7f14380039d0 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 5 monclient: authenticate success, global_id 34281 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 10 monclient: _renew_subs 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c141bd0 con 0x7f14380039d0 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448033680 con 0x7f14380039d0 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1441ffb640 10 monclient: dump: 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 10 monclient: _renew_subs 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1443fff640 4 set_mon_vals no callback set 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.232+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c11dc60 con 0x7f14380039d0 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1453cc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1453cc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1453cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c0008d0 con 0x7f14380039d0 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1448030070 con 0x7f14380039d0 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1441ffb640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1430038740 0x7f143003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4684+0+0 (secure 0 0 0) 0x7f144803a070 con 0x7f14380039d0 2024-09-15T03:30:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.233+0000 7f145125b640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1430038740 0x7f143003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:04.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.234+0000 7f145125b640 10 monclient: get_auth_request con 0x7f1430038740 auth_method 0 2024-09-15T03:30:04.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.236+0000 7f145125b640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1430038740 0x7f143003abf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f143c001ed0 tx=0x7f144c068b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:04.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.238+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1448030320 con 0x7f14380039d0 2024-09-15T03:30:04.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.238+0000 7f1441ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:04.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.238+0000 7f1441ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:04.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.393+0000 7f1453cc0640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T03:30:04.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.393+0000 7f1453cc0640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T03:30:04.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.393+0000 7f1453cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:04.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:04.393+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f144c064900 con 0x7f14380039d0 2024-09-15T03:30:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.032+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v30) v1 ==== 188+0+0 (secure 0 0 0) 0x7f144803de00 con 0x7f14380039d0 2024-09-15T03:30:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.032+0000 7f1441ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T03:30:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.032+0000 7f1441ffb640 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-15T03:30:06.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.141+0000 7f1453cc0640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T03:30:06.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.141+0000 7f1453cc0640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T03:30:06.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.141+0000 7f1453cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.141+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f144c11bca0 con 0x7f14380039d0 2024-09-15T03:30:06.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.142+0000 7f1441ffb640 1 -- 172.21.15.181:0/3019943111 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v30) v1 ==== 195+0+0 (secure 0 0 0) 0x7f144806c030 con 0x7f14380039d0 2024-09-15T03:30:06.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.142+0000 7f1441ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-15T03:30:06.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.142+0000 7f1441ffb640 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-15T03:30:06.144 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'fooo' already exists 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1430038740 msgr2=0x7f143003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1430038740 0x7f143003abf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f143c001ed0 tx=0x7f144c068b50 comp rx=0 tx=0).stop 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 10 monclient: shutdown 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 msgr2=0x7f144c140d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f144c140d20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1448009a20 tx=0x7f1448030000 comp rx=0 tx=0).stop 2024-09-15T03:30:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 shutdown_connections 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1430038740 0x7f143003abf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f144c130fd0 0x7f144c11b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f144c141260 0x7f144c130a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 --2- 172.21.15.181:0/3019943111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14380039d0 0x7f144c140d20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 >> 172.21.15.181:0/3019943111 conn(0x7f144c0a5d40 msgr2=0x7f144c069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 shutdown_connections 2024-09-15T03:30:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.145+0000 7f1453cc0640 1 -- 172.21.15.181:0/3019943111 wait complete. 2024-09-15T03:30:06.155 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create foooo 123 2024-09-15T03:30:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.228+0000 7f45b3634640 10 monclient: get_monmap_and_config 2024-09-15T03:30:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.228+0000 7f45b3634640 10 monclient: build_initial_monmap 2024-09-15T03:30:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.228+0000 7f45b3634640 10 monclient: monmap: 2024-09-15T03:30:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:06.229374+0000 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:06.229374+0000 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:06.230 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:06.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 Processor -- start 2024-09-15T03:30:06.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 -- start start 2024-09-15T03:30:06.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient: init 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45ac131c90 0x7f45ac132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac131c90 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac09d210 0x7f45ac1325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac09d210 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f45ac0a0ab0 0x7f45ac068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac0a0ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac0a1800 con 0x7f45ac131c90 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac09fea0 con 0x7f45ac0a0ab0 2024-09-15T03:30:06.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0a1520 con 0x7f45ac09d210 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b3634640 10 monclient(hunting): authenticate will time out at 810.710935s 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.229+0000 7f45b1bd1640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f45ac0a0ab0 msgr2=0x7f45ac068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b1bd1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f45ac0a0ab0 0x7f45ac068e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45ac131c90 0x7f45ac132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45ac131c90 0x7f45ac132060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36034/0 (socket says 172.21.15.181:36034) 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 -- 172.21.15.181:0/3407961754 learned_addr learned my addr 172.21.15.181:0/3407961754 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac09d210 0x7f45ac1325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): get_auth_request con 0x7f45ac131c90 auth_method 0 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:06.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): get_auth_request con 0x7f45ac09d210 auth_method 0 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient(hunting): handle_auth_done global_id 34284 payload 274 2024-09-15T03:30:06.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 -- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac09d210 msgr2=0x7f45ac1325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 --2- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac09d210 0x7f45ac1325a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 -- 172.21.15.181:0/3407961754 >> v2:172.21.15.181:3300/0 conn(0x7f45ac0a0ab0 msgr2=0x7f45ac068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 --2- 172.21.15.181:0/3407961754 >> v2:172.21.15.181:3300/0 conn(0x7f45ac0a0ab0 0x7f45ac068e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient: _finish_hunting 0 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 monclient: found mon.noname-a 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.230+0000 7f45b13d0640 1 -- 172.21.15.181:0/3407961754 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac069340 con 0x7f45ac131c90 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45b13d0640 1 -- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 msgr2=0x7f45ac132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45b13d0640 1 --2- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac132060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0a3150 tx=0x7f45a80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816b5b8f58e7c6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 1 -- 172.21.15.181:0/3407961754 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8032440 con 0x7f45ac131c90 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: dump: 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:06.243 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45b3634640 5 monclient: authenticate success, global_id 34284 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 1 -- 172.21.15.181:0/3407961754 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a80325a0 con 0x7f45ac131c90 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45b3634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45b3634640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45b3634640 4 set_mon_vals no callback set 2024-09-15T03:30:06.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 1 -- 172.21.15.181:0/3407961754 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8032980 con 0x7f45ac131c90 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.233+0000 7f45a3fff640 10 monclient: dump: 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:06.245 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 10 monclient: shutdown 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 -- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 msgr2=0x7f45ac132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 --2- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac132060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0a3150 tx=0x7f45a80091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 -- 172.21.15.181:0/3407961754 shutdown_connections 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 --2- 172.21.15.181:0/3407961754 >> v2:172.21.15.181:3300/0 conn(0x7f45ac0a0ab0 0x7f45ac068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 --2- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac09d210 0x7f45ac1325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 --2- 172.21.15.181:0/3407961754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac132060 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 -- 172.21.15.181:0/3407961754 >> 172.21.15.181:0/3407961754 conn(0x7f45ac071370 msgr2=0x7f45ac0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 -- 172.21.15.181:0/3407961754 shutdown_connections 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 -- 172.21.15.181:0/3407961754 wait complete. 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 10 monclient: build_initial_monmap 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 10 monclient: monmap: 2024-09-15T03:30:06.246 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.234+0000 7f45b3634640 1 Processor -- start 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 -- start start 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient: init 2024-09-15T03:30:06.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45ac0a0ab0 0x7f45ac141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac0a0ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac131c90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac141bd0 0x7f45ac11e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac141bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac120cc0 con 0x7f45ac131c90 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0a0fb0 con 0x7f45ac0a0ab0 2024-09-15T03:30:06.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0a4b60 con 0x7f45ac141bd0 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b3634640 10 monclient(hunting): authenticate will time out at 810.716624s 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b13d0640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45ac0a0ab0 msgr2=0x7f45ac141150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b13d0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45ac0a0ab0 0x7f45ac141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36036/0 (socket says 172.21.15.181:36036) 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac141bd0 0x7f45ac11e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 1 -- 172.21.15.181:0/246269116 learned_addr learned my addr 172.21.15.181:0/246269116 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): get_auth_request con 0x7f45ac131c90 auth_method 0 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 10 monclient(hunting): get_auth_request con 0x7f45ac141bd0 auth_method 0 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.235+0000 7f45b1bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b1bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 10 monclient(hunting): handle_auth_done global_id 34287 payload 824 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 1 -- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac141bd0 msgr2=0x7f45ac11e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac141bd0 0x7f45ac11e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 1 -- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45ac0a0ab0 msgr2=0x7f45ac141150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45ac0a0ab0 0x7f45ac141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 10 monclient: _finish_hunting 0 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 1 monclient: found mon.noname-a 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.236+0000 7f45b0bcf640 1 -- 172.21.15.181:0/246269116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a8009450 con 0x7f45ac131c90 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b0bcf640 1 -- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 msgr2=0x7f45ac141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b0bcf640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac141690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f459c00ea20 tx=0x7f459c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c00cc20 con 0x7f45ac131c90 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: dump: 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: _finish_auth 0 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c00cd80 con 0x7f45ac131c90 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c013680 con 0x7f45ac131c90 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a17fa640 10 monclient: dump: 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 5 monclient: authenticate success, global_id 34287 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 10 monclient: _renew_subs 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac122eb0 con 0x7f45ac131c90 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45a37fe640 4 set_mon_vals no callback set 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 10 monclient: _renew_subs 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.238+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac120050 con 0x7f45ac131c90 2024-09-15T03:30:06.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f459c0139f0 con 0x7f45ac131c90 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45b3634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45b3634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45b3634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac0008d0 con 0x7f45ac131c90 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45a17fa640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45900386f0 0x7f459003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f459c04c070 con 0x7f45ac131c90 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45b13d0640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45900386f0 0x7f459003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.239+0000 7f45b13d0640 10 monclient: get_auth_request con 0x7f45900386f0 auth_method 0 2024-09-15T03:30:06.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.242+0000 7f45b13d0640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45900386f0 0x7f459003aba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f45a8034000 tx=0x7f45a8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:06.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.244+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f459c0234d0 con 0x7f45ac131c90 2024-09-15T03:30:06.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.244+0000 7f45a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:06.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.244+0000 7f45a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:06.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.398+0000 7f45b3634640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T03:30:06.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.398+0000 7f45b3634640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T03:30:06.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.398+0000 7f45b3634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:06.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:06.398+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f45ac0695e0 con 0x7f45ac131c90 2024-09-15T03:30:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.034+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f459c0232f0 con 0x7f45ac131c90 2024-09-15T03:30:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.034+0000 7f45a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T03:30:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.034+0000 7f45a17fa640 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-15T03:30:07.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.164+0000 7f45b3634640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T03:30:07.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.164+0000 7f45b3634640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T03:30:07.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.164+0000 7f45b3634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.164+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f45ac120950 con 0x7f45ac131c90 2024-09-15T03:30:07.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.165+0000 7f45a17fa640 1 -- 172.21.15.181:0/246269116 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f459c04a900 con 0x7f45ac131c90 2024-09-15T03:30:07.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.165+0000 7f45a17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-15T03:30:07.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.165+0000 7f45a17fa640 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-15T03:30:07.167 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foooo' already exists 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45900386f0 msgr2=0x7f459003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45900386f0 0x7f459003aba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f45a8034000 tx=0x7f45a8032000 comp rx=0 tx=0).stop 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 10 monclient: shutdown 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 msgr2=0x7f45ac141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac141690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f459c00ea20 tx=0x7f459c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 shutdown_connections 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45900386f0 0x7f459003aba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45ac141bd0 0x7f45ac11e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ac131c90 0x7f45ac141690 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 --2- 172.21.15.181:0/246269116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45ac0a0ab0 0x7f45ac141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 >> 172.21.15.181:0/246269116 conn(0x7f45ac071370 msgr2=0x7f45ac0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 shutdown_connections 2024-09-15T03:30:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.167+0000 7f45b3634640 1 -- 172.21.15.181:0/246269116 wait complete. 2024-09-15T03:30:07.178 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create foo 123 2024-09-15T03:30:07.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.251+0000 7fe80191d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:07.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.251+0000 7fe80191d640 10 monclient: build_initial_monmap 2024-09-15T03:30:07.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.251+0000 7fe80191d640 10 monclient: monmap: 2024-09-15T03:30:07.252 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:07.252 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:07.252107+0000 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:07.252107+0000 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:07.253 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.251+0000 7fe80191d640 1 Processor -- start 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 -- start start 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient: init 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe7fc09ec50 0x7fe7fc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc09ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 0x7fe7fc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc1321a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 0x7fe7fc069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7fc09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc067420 con 0x7fe7fc1321a0 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc09fdc0 con 0x7fe7fc09ec50 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0a0510 con 0x7fe7fc09f020 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe80191d640 10 monclient(hunting): authenticate will time out at 811.733495s 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7faffd640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fe7fc09ec50 msgr2=0x7fe7fc131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:07.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7faffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe7fc09ec50 0x7fe7fc131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 0x7fe7fc068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 0x7fe7fc069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 0x7fe7fc069240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:56204/0 (socket says 172.21.15.181:56204) 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 1 -- 172.21.15.181:0/4229890125 learned_addr learned my addr 172.21.15.181:0/4229890125 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): get_auth_request con 0x7fe7fc09f020 auth_method 0 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 0x7fe7fc068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36044/0 (socket says 172.21.15.181:36044) 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): get_auth_request con 0x7fe7fc1321a0 auth_method 0 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.252+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_done global_id 4577 payload 274 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 1 -- 172.21.15.181:0/4229890125 >> v2:172.21.15.181:3300/0 conn(0x7fe7fc09ec50 msgr2=0x7fe7fc131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:07.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 1 --2- 172.21.15.181:0/4229890125 >> v2:172.21.15.181:3300/0 conn(0x7fe7fc09ec50 0x7fe7fc131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 1 -- 172.21.15.181:0/4229890125 >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 msgr2=0x7fe7fc068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 1 --2- 172.21.15.181:0/4229890125 >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 0x7fe7fc068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7fb7fe640 1 -- 172.21.15.181:0/4229890125 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0a0200 con 0x7fe7fc09f020 2024-09-15T03:30:07.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.253+0000 7fe7f27fc640 1 --2- 172.21.15.181:0/4229890125 >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 0x7fe7fc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.255+0000 7fe7fb7fe640 1 -- 172.21.15.181:0/4229890125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 msgr2=0x7fe7fc069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.255+0000 7fe7fb7fe640 1 --2- 172.21.15.181:0/4229890125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 0x7fe7fc069240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0719d0 tx=0x7fe7f4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=553a5a8c7f357022 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 1 -- 172.21.15.181:0/4229890125 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4007610 con 0x7fe7fc09f020 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: dump: 2024-09-15T03:30:07.266 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 1 -- 172.21.15.181:0/4229890125 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f402d050 con 0x7fe7fc09f020 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 5 monclient: authenticate success, global_id 4577 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 1 -- 172.21.15.181:0/4229890125 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40079a0 con 0x7fe7fc09f020 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 4 set_mon_vals no callback set 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe7fa7fc640 10 monclient: dump: 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.268 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.269 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 10 monclient: shutdown 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 -- 172.21.15.181:0/4229890125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 msgr2=0x7fe7fc069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 --2- 172.21.15.181:0/4229890125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 0x7fe7fc069240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0719d0 tx=0x7fe7f4009840 comp rx=0 tx=0).stop 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 -- 172.21.15.181:0/4229890125 shutdown_connections 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 --2- 172.21.15.181:0/4229890125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc09f020 0x7fe7fc069240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 --2- 172.21.15.181:0/4229890125 >> v2:172.21.15.153:3300/0 conn(0x7fe7fc1321a0 0x7fe7fc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 --2- 172.21.15.181:0/4229890125 >> v2:172.21.15.181:3300/0 conn(0x7fe7fc09ec50 0x7fe7fc131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 -- 172.21.15.181:0/4229890125 >> 172.21.15.181:0/4229890125 conn(0x7fe7fc0a5d40 msgr2=0x7fe7fc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 -- 172.21.15.181:0/4229890125 shutdown_connections 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.256+0000 7fe80191d640 1 -- 172.21.15.181:0/4229890125 wait complete. 2024-09-15T03:30:07.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.257+0000 7fe80191d640 10 monclient: build_initial_monmap 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.257+0000 7fe80191d640 10 monclient: monmap: 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.257+0000 7fe80191d640 1 Processor -- start 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.257+0000 7fe80191d640 1 -- start start 2024-09-15T03:30:07.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.257+0000 7fe80191d640 10 monclient: init 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7fc09ec50 0x7fe7fc140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7fc09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 0x7fe7fc130e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc09f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc1321a0 0x7fe7fc131380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7faffd640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7fc09ec50 msgr2=0x7fe7fc140f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc09c000 con 0x7fe7fc09f020 2024-09-15T03:30:07.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0a4780 con 0x7fe7fc09ec50 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7faffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7fc09ec50 0x7fe7fc140f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc141db0 con 0x7fe7fc1321a0 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe80191d640 10 monclient(hunting): authenticate will time out at 811.739259s 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 0x7fe7fc130e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc1321a0 0x7fe7fc131380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 0x7fe7fc130e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36050/0 (socket says 172.21.15.181:36050) 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 -- 172.21.15.181:0/410442632 learned_addr learned my addr 172.21.15.181:0/410442632 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): get_auth_request con 0x7fe7fc09f020 auth_method 0 2024-09-15T03:30:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): get_auth_request con 0x7fe7fc1321a0 auth_method 0 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient(hunting): handle_auth_done global_id 34293 payload 824 2024-09-15T03:30:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 -- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc1321a0 msgr2=0x7fe7fc131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc1321a0 0x7fe7fc131380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 -- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7fc09ec50 msgr2=0x7fe7fc140f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7fc09ec50 0x7fe7fc140f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 monclient: found mon.noname-a 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7f27fc640 1 -- 172.21.15.181:0/410442632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4009390 con 0x7fe7fc09f020 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.258+0000 7fe7fb7fe640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc1321a0 0x7fe7fc131380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f27fc640 1 -- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 msgr2=0x7fe7fc130e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f27fc640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 0x7fe7fc130e40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400ec80 tx=0x7fe7e40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e400eec0 con 0x7fe7fc09f020 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: dump: 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.276 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e400cdd0 con 0x7fe7fc09f020 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 5 monclient: authenticate success, global_id 34293 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 10 monclient: _renew_subs 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f4009ae0 con 0x7fe7fc09f020 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4013680 con 0x7fe7fc09f020 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f37fe640 10 monclient: dump: 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.277 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe7f9ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 10 monclient: _renew_subs 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.261+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc11f460 con 0x7fe7fc09f020 2024-09-15T03:30:07.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe80191d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe80191d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe80191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe7e40139f0 con 0x7fe7fc09f020 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc121400 con 0x7fe7fc09f020 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe7f37fe640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe7dc0386f0 0x7fe7dc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fe7e404c150 con 0x7fe7fc09f020 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe7faffd640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe7dc0386f0 0x7fe7dc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.262+0000 7fe7faffd640 10 monclient: get_auth_request con 0x7fe7dc0386f0 auth_method 0 2024-09-15T03:30:07.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.264+0000 7fe7faffd640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe7dc0386f0 0x7fe7dc03aba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec003940 tx=0x7fe7ec006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:07.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.267+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe7e40234d0 con 0x7fe7fc09f020 2024-09-15T03:30:07.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.267+0000 7fe7f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:07.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.267+0000 7fe7f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe80191d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T03:30:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe80191d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T03:30:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe80191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7fe7fc0008d0 con 0x7fe7fc09f020 2024-09-15T03:30:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe7e40232f0 con 0x7fe7fc09f020 2024-09-15T03:30:07.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe7f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T03:30:07.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.423+0000 7fe7f37fe640 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-15T03:30:07.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.516+0000 7fe80191d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T03:30:07.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.516+0000 7fe80191d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T03:30:07.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.516+0000 7fe80191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.516+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7fe7fc0a13e0 con 0x7fe7fc09f020 2024-09-15T03:30:07.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.517+0000 7fe7f37fe640 1 -- 172.21.15.181:0/410442632 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe7e404a890 con 0x7fe7fc09f020 2024-09-15T03:30:07.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.517+0000 7fe7f37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-15T03:30:07.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.517+0000 7fe7f37fe640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-15T03:30:07.518 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' already exists 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe7dc0386f0 msgr2=0x7fe7dc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe7dc0386f0 0x7fe7dc03aba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec003940 tx=0x7fe7ec006d80 comp rx=0 tx=0).stop 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 10 monclient: shutdown 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 msgr2=0x7fe7fc130e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 0x7fe7fc130e40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400ec80 tx=0x7fe7e40090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 shutdown_connections 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe7dc0386f0 0x7fe7dc03aba0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7fc1321a0 0x7fe7fc131380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7fc09f020 0x7fe7fc130e40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 --2- 172.21.15.181:0/410442632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7fc09ec50 0x7fe7fc140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.518+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 >> 172.21.15.181:0/410442632 conn(0x7fe7fc0a5d40 msgr2=0x7fe7fc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:07.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.519+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 shutdown_connections 2024-09-15T03:30:07.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.519+0000 7fe80191d640 1 -- 172.21.15.181:0/410442632 wait complete. 2024-09-15T03:30:07.528 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.603+0000 7efe82699640 10 monclient: get_monmap_and_config 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.603+0000 7efe82699640 10 monclient: build_initial_monmap 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.603+0000 7efe82699640 10 monclient: monmap: 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:07.604735+0000 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:07.604735+0000 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:07.605 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:07.606 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:07.606 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:07.606 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.604+0000 7efe82699640 1 Processor -- start 2024-09-15T03:30:07.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.604+0000 7efe82699640 1 -- start start 2024-09-15T03:30:07.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.604+0000 7efe82699640 10 monclient: init 2024-09-15T03:30:07.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe7c131c90 0x7efe7c132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): picked mon.noname-a con 0x7efe7c131c90 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c09d210 0x7efe7c1325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): picked mon.noname-b con 0x7efe7c09d210 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe7c0a0ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c0a1800 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c09fea0 con 0x7efe7c131c90 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c0a1520 con 0x7efe7c09d210 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe82699640 10 monclient(hunting): authenticate will time out at 812.086342s 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe80c36640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7efe7c0a0ab0 legacy=0x7efe7c068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe7c131c90 0x7efe7c132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c09d210 0x7efe7c1325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe7c131c90 0x7efe7c132060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36062/0 (socket says 172.21.15.181:36062) 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 -- 172.21.15.181:0/992444505 learned_addr learned my addr 172.21.15.181:0/992444505 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): get_auth_request con 0x7efe7c131c90 auth_method 0 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): get_auth_request con 0x7efe7c09d210 auth_method 0 2024-09-15T03:30:07.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient(hunting): handle_auth_done global_id 34296 payload 274 2024-09-15T03:30:07.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 -- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c09d210 msgr2=0x7efe7c1325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 --2- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c09d210 0x7efe7c1325a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 -- 172.21.15.181:0/992444505 >> v1:172.21.15.181:6789/0 conn(0x7efe7c0a0ab0 legacy=0x7efe7c068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7bfff640 1 -- 172.21.15.181:0/992444505 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c069510 con 0x7efe7c131c90 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.605+0000 7efe7b7fe640 1 --2- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c09d210 0x7efe7c1325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.608+0000 7efe7bfff640 1 -- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c131c90 msgr2=0x7efe7c132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:07.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.608+0000 7efe7bfff640 1 --2- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c131c90 0x7efe7c132060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009910 tx=0x7efe6c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e36a49bd23602e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:07.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 1 -- 172.21.15.181:0/992444505 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c02aea0 con 0x7efe7c131c90 2024-09-15T03:30:07.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:07.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: dump: 2024-09-15T03:30:07.619 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.619 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: _finish_auth 0 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 5 monclient: authenticate success, global_id 34296 2024-09-15T03:30:07.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 1 -- 172.21.15.181:0/992444505 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c032440 con 0x7efe7c131c90 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 4 set_mon_vals no callback set 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 1 -- 172.21.15.181:0/992444505 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0327e0 con 0x7efe7c131c90 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe7affd640 10 monclient: dump: 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.621 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 10 monclient: shutdown 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 -- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c131c90 msgr2=0x7efe7c132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 --2- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c131c90 0x7efe7c132060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009910 tx=0x7efe6c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 -- 172.21.15.181:0/992444505 shutdown_connections 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 --2- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c09d210 0x7efe7c1325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 --2- 172.21.15.181:0/992444505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c131c90 0x7efe7c132060 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 -- 172.21.15.181:0/992444505 >> 172.21.15.181:0/992444505 conn(0x7efe7c071370 msgr2=0x7efe7c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 -- 172.21.15.181:0/992444505 shutdown_connections 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.609+0000 7efe82699640 1 -- 172.21.15.181:0/992444505 wait complete. 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 10 monclient: build_initial_monmap 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 10 monclient: monmap: 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:07.623 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 1 Processor -- start 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 1 -- start start 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 10 monclient: init 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.610+0000 7efe82699640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 0x7efe7c131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): picked mon.noname-a con 0x7efe7c0a0ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c131c90 0x7efe7c11ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): picked mon.noname-c con 0x7efe7c131c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe7c1214a0 0x7efe7c11f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): picked mon.noname-b con 0x7efe7c1214a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c1229e0 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c0a0fb0 con 0x7efe7c1214a0 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c09f180 con 0x7efe7c131c90 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe82699640 10 monclient(hunting): authenticate will time out at 812.092061s 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe80c36640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe7c1214a0 msgr2=0x7efe7c11f2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe80c36640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe7c1214a0 0x7efe7c11f2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 0x7efe7c131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 0x7efe7c131410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:36066/0 (socket says 172.21.15.181:36066) 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 -- 172.21.15.181:0/870078866 learned_addr learned my addr 172.21.15.181:0/870078866 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): get_auth_request con 0x7efe7c0a0ab0 auth_method 0 2024-09-15T03:30:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c131c90 0x7efe7c11ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): get_auth_request con 0x7efe7c131c90 auth_method 0 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient(hunting): handle_auth_done global_id 34299 payload 824 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 -- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c131c90 msgr2=0x7efe7c11ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c131c90 0x7efe7c11ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 -- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe7c1214a0 msgr2=0x7efe7c11f2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe7c1214a0 0x7efe7c11f2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.611+0000 7efe7bfff640 1 -- 172.21.15.181:0/870078866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70009390 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe7bfff640 1 -- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 msgr2=0x7efe7c131410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe7bfff640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 0x7efe7c131410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009970 tx=0x7efe6c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c02aea0 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: dump: 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c000d10 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c03d440 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe63fff640 10 monclient: dump: 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe7a7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.613+0000 7efe82699640 5 monclient: authenticate success, global_id 34299 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 monclient: _renew_subs 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c009450 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 monclient: _renew_subs 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c120de0 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efe6c039040 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.614+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c1224b0 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.615+0000 7efe63fff640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe680386f0 0x7efe6803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.615+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5470+0+0 (secure 0 0 0) 0x7efe6c06bb40 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.615+0000 7efe7b7fe640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe680386f0 0x7efe6803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.615+0000 7efe7b7fe640 10 monclient: get_auth_request con 0x7efe680386f0 auth_method 0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.617+0000 7efe7b7fe640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe680386f0 0x7efe6803aba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe7c120d00 tx=0x7efe700090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.619+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efe6c02f4d0 con 0x7efe7c0a0ab0 2024-09-15T03:30:07.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.619+0000 7efe63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:07.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.619+0000 7efe63fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:07.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.773+0000 7efe82699640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T03:30:07.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.773+0000 7efe82699640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T03:30:07.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.773+0000 7efe82699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:07.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:07.773+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efe7c0008d0 con 0x7efe7c0a0ab0 2024-09-15T03:30:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.039+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v32) v1 ==== 152+0+0 (secure 0 0 0) 0x7efe6c0420c0 con 0x7efe7c0a0ab0 2024-09-15T03:30:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.039+0000 7efe63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T03:30:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.039+0000 7efe63fff640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-15T03:30:08.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.167+0000 7efe82699640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T03:30:08.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.167+0000 7efe82699640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T03:30:08.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.167+0000 7efe82699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:08.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:08.167+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efe7c0008d0 con 0x7efe7c0a0ab0 2024-09-15T03:30:09.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.163+0000 7efe63fff640 1 -- 172.21.15.181:0/870078866 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v33) v1 ==== 152+0+0 (secure 0 0 0) 0x7efe6c037030 con 0x7efe7c0a0ab0 2024-09-15T03:30:09.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.164+0000 7efe63fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-15T03:30:09.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.164+0000 7efe63fff640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-15T03:30:09.165 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 4 size to 1 2024-09-15T03:30:09.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.169+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe680386f0 msgr2=0x7efe6803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:09.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.169+0000 7efe82699640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe680386f0 0x7efe6803aba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe7c120d00 tx=0x7efe700090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.169+0000 7efe82699640 10 monclient: shutdown 2024-09-15T03:30:09.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.169+0000 7efe82699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.169+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 msgr2=0x7efe7c131410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.169+0000 7efe82699640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 0x7efe7c131410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009970 tx=0x7efe6c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.170+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 shutdown_connections 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.170+0000 7efe82699640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe680386f0 0x7efe6803aba0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe7c120d00 tx=0x7efe700090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.170+0000 7efe82699640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe7c1214a0 0x7efe7c11f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.170+0000 7efe82699640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe7c131c90 0x7efe7c11ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.170+0000 7efe82699640 1 --2- 172.21.15.181:0/870078866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe7c0a0ab0 0x7efe7c131410 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009970 tx=0x7efe6c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:09.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.170+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 >> 172.21.15.181:0/870078866 conn(0x7efe7c071370 msgr2=0x7efe7c069d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:09.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.172+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 shutdown_connections 2024-09-15T03:30:09.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.173+0000 7efe82699640 1 -- 172.21.15.181:0/870078866 wait complete. 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_config_value foo min_size 1 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:+ local pool_name config_opt expected_val val 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:+ pool_name=foo 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:+ config_opt=min_size 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_val=1 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:++ get_config_value_or_die foo min_size 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:++ local pool_name config_opt raw val 2024-09-15T03:30:09.195 INFO:tasks.workunit.client.0.smithi181.stderr:++ pool_name=foo 2024-09-15T03:30:09.196 INFO:tasks.workunit.client.0.smithi181.stderr:++ config_opt=min_size 2024-09-15T03:30:09.196 INFO:tasks.workunit.client.0.smithi181.stderr:+++ ceph osd pool get foo min_size 2024-09-15T03:30:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:++ raw='min_size: 1' 2024-09-15T03:30:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:++ [[ 0 -ne 0 ]] 2024-09-15T03:30:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:+++ echo min_size: 1 2024-09-15T03:30:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:+++ sed -e 's/[{} "]//g' 2024-09-15T03:30:09.562 INFO:tasks.workunit.client.0.smithi181.stderr:++ raw=min_size:1 2024-09-15T03:30:09.562 INFO:tasks.workunit.client.0.smithi181.stderr:+++ echo min_size:1 2024-09-15T03:30:09.563 INFO:tasks.workunit.client.0.smithi181.stderr:+++ cut -f2 -d: 2024-09-15T03:30:09.564 INFO:tasks.workunit.client.0.smithi181.stderr:++ val=1 2024-09-15T03:30:09.564 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo 1 2024-09-15T03:30:09.564 INFO:tasks.workunit.client.0.smithi181.stderr:++ return 0 2024-09-15T03:30:09.564 INFO:tasks.workunit.client.0.smithi181.stderr:+ val=1 2024-09-15T03:30:09.564 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 1 != \1 ]] 2024-09-15T03:30:09.564 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set foo size 4 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.638+0000 7f7666d1d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.638+0000 7f7666d1d640 10 monclient: build_initial_monmap 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.638+0000 7f7666d1d640 10 monclient: monmap: 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:09.638865+0000 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:09.638865+0000 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:09.640 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:09.640 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:09.640 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:09.640 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:09.640 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:09.640 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:09.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.638+0000 7f7666d1d640 1 Processor -- start 2024-09-15T03:30:09.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.638+0000 7f7666d1d640 1 -- start start 2024-09-15T03:30:09.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.638+0000 7f7666d1d640 10 monclient: init 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f766009ec50 0x7f7660131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f766009ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76601321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 0x7f7660069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f766009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7660067420 con 0x7f76601321a0 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766009fdc0 con 0x7f766009ec50 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600a0510 con 0x7f766009f020 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:09.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7666d1d640 10 monclient(hunting): authenticate will time out at 814.120272s 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 0x7f7660069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f766009ec50 0x7f7660131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7657fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f76601321a0 0x7f7660068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:39582/0 (socket says 172.21.15.181:39582) 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7657fff640 1 -- 172.21.15.181:0/4239924280 learned_addr learned my addr 172.21.15.181:0/4239924280 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): get_auth_request con 0x7f766009ec50 auth_method 0 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 0x7f7660069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59954/0 (socket says 172.21.15.181:59954) 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): get_auth_request con 0x7f766009f020 auth_method 0 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:09.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient(hunting): handle_auth_done global_id 34308 payload 274 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 1 -- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 msgr2=0x7f7660069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 1 --2- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 0x7f7660069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 1 -- 172.21.15.181:0/4239924280 >> v1:172.21.15.153:6789/0 conn(0x7f76601321a0 legacy=0x7f7660068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient: _finish_hunting 0 2024-09-15T03:30:09.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 1 monclient: found mon.noname-a 2024-09-15T03:30:09.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:09.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:09.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f7664ab9640 1 -- 172.21.15.181:0/4239924280 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600a0200 con 0x7f766009ec50 2024-09-15T03:30:09.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.639+0000 7f76652ba640 1 --2- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 0x7f7660069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.642+0000 7f7664ab9640 1 -- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009ec50 msgr2=0x7f7660131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.642+0000 7f7664ab9640 1 --2- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009ec50 0x7f7660131c60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f765c009850 tx=0x7f765c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3387e6cf71e3f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 1 -- 172.21.15.181:0/4239924280 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c02adf0 con 0x7f766009ec50 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: dump: 2024-09-15T03:30:09.651 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:09.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 1 -- 172.21.15.181:0/4239924280 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c032440 con 0x7f766009ec50 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 1 -- 172.21.15.181:0/4239924280 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0327e0 con 0x7f766009ec50 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 5 monclient: authenticate success, global_id 34308 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 4 set_mon_vals no callback set 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f76577fe640 10 monclient: dump: 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 10 monclient: shutdown 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 1 -- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009ec50 msgr2=0x7f7660131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 1 --2- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009ec50 0x7f7660131c60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f765c009850 tx=0x7f765c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 1 -- 172.21.15.181:0/4239924280 shutdown_connections 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 1 --2- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009f020 0x7f7660069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 1 --2- 172.21.15.181:0/4239924280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009ec50 0x7f7660131c60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.643+0000 7f7666d1d640 1 -- 172.21.15.181:0/4239924280 >> 172.21.15.181:0/4239924280 conn(0x7f76600a5d40 msgr2=0x7f76600a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 1 -- 172.21.15.181:0/4239924280 shutdown_connections 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 1 -- 172.21.15.181:0/4239924280 wait complete. 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 10 monclient: build_initial_monmap 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 10 monclient: monmap: 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:09.655 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 1 Processor -- start 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 1 -- start start 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.644+0000 7f7666d1d640 10 monclient: init 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 0x7f766009aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f766009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 0x7f7660130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f766009f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76601321a0 0x7f7660141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f76601321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766011dee0 con 0x7f766009f020 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766009de10 con 0x7f76601321a0 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766011dc00 con 0x7f766009ec50 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7666d1d640 10 monclient(hunting): authenticate will time out at 814.126415s 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f76652ba640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76601321a0 msgr2=0x7f7660141030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f76652ba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76601321a0 0x7f7660141030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 0x7f7660130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 0x7f766009aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 0x7f7660130ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:47178/0 (socket says 172.21.15.181:47178) 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 1 -- 172.21.15.181:0/870006427 learned_addr learned my addr 172.21.15.181:0/870006427 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): get_auth_request con 0x7f766009f020 auth_method 0 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): get_auth_request con 0x7f766009ec50 auth_method 0 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient(hunting): handle_auth_done global_id 4595 payload 824 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 -- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76601321a0 msgr2=0x7f7660141030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76601321a0 0x7f7660141030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 -- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 msgr2=0x7f7660130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 0x7f7660130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient: _finish_hunting 0 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 monclient: found mon.noname-c 2024-09-15T03:30:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7664ab9640 1 -- 172.21.15.181:0/870006427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658009a20 con 0x7f766009ec50 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.645+0000 7f7657fff640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 0x7f7660130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7664ab9640 1 -- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 msgr2=0x7f766009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7664ab9640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 0x7f766009aae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f765c034000 tx=0x7f765c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c031a90 con 0x7f766009ec50 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: dump: 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c031c10 con 0x7f766009ec50 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c03c400 con 0x7f766009ec50 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:09.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7654ff9640 10 monclient: dump: 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 5 monclient: authenticate success, global_id 4595 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 10 monclient: _renew_subs 2024-09-15T03:30:09.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c009390 con 0x7f766009ec50 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7656ffd640 4 set_mon_vals no callback set 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 10 monclient: _renew_subs 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.648+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766011bc20 con 0x7f766009ec50 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.649+0000 7f7666d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.649+0000 7f7666d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.649+0000 7f7666d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.649+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f765c03c830 con 0x7f766009ec50 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.649+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766011be10 con 0x7f766009ec50 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.650+0000 7f7654ff9640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f764c038720 0x7f764c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.650+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f765c06c3c0 con 0x7f766009ec50 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.650+0000 7f7657fff640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f764c038720 0x7f764c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.650+0000 7f7657fff640 10 monclient: get_auth_request con 0x7f764c038720 auth_method 0 2024-09-15T03:30:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.652+0000 7f7657fff640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f764c038720 0x7f764c03abd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f766009b260 tx=0x7f7650004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.654+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f765c03b3a0 con 0x7f766009ec50 2024-09-15T03:30:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.654+0000 7f7654ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.654+0000 7f7654ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:09.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.808+0000 7f7666d1d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T03:30:09.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.808+0000 7f7666d1d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T03:30:09.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.808+0000 7f7666d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:09.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:09.808+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f766009c070 con 0x7f766009ec50 2024-09-15T03:30:10.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.182+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f766009c070 con 0x7f766009ec50 2024-09-15T03:30:10.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.182+0000 7f7654ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T03:30:10.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.182+0000 7f7654ff9640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-15T03:30:10.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.290+0000 7f7666d1d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T03:30:10.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.290+0000 7f7666d1d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T03:30:10.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.290+0000 7f7666d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:10.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:10.290+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f766009c070 con 0x7f766009ec50 2024-09-15T03:30:11.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.185+0000 7f7654ff9640 1 -- 172.21.15.181:0/870006427 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f766009c070 con 0x7f766009ec50 2024-09-15T03:30:11.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.185+0000 7f7654ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-15T03:30:11.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.185+0000 7f7654ff9640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-15T03:30:11.187 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 4 size to 4 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.189+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f764c038720 msgr2=0x7f764c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.189+0000 7f7666d1d640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f764c038720 0x7f764c03abd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f766009b260 tx=0x7f7650004960 comp rx=0 tx=0).stop 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 10 monclient: shutdown 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 msgr2=0x7f766009aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 0x7f766009aae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f765c034000 tx=0x7f765c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 shutdown_connections 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f764c038720 0x7f764c03abd0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f766009b260 tx=0x7f7650004960 comp rx=0 tx=0).stop 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76601321a0 0x7f7660141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f766009f020 0x7f7660130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 --2- 172.21.15.181:0/870006427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f766009ec50 0x7f766009aae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 >> 172.21.15.181:0/870006427 conn(0x7f76600a5d40 msgr2=0x7f766009e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:11.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.190+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 shutdown_connections 2024-09-15T03:30:11.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.192+0000 7f7666d1d640 1 -- 172.21.15.181:0/870006427 wait complete. 2024-09-15T03:30:11.216 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_config_value foo min_size 2 2024-09-15T03:30:11.216 INFO:tasks.workunit.client.0.smithi181.stderr:+ local pool_name config_opt expected_val val 2024-09-15T03:30:11.216 INFO:tasks.workunit.client.0.smithi181.stderr:+ pool_name=foo 2024-09-15T03:30:11.216 INFO:tasks.workunit.client.0.smithi181.stderr:+ config_opt=min_size 2024-09-15T03:30:11.216 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_val=2 2024-09-15T03:30:11.218 INFO:tasks.workunit.client.0.smithi181.stderr:++ get_config_value_or_die foo min_size 2024-09-15T03:30:11.218 INFO:tasks.workunit.client.0.smithi181.stderr:++ local pool_name config_opt raw val 2024-09-15T03:30:11.218 INFO:tasks.workunit.client.0.smithi181.stderr:++ pool_name=foo 2024-09-15T03:30:11.218 INFO:tasks.workunit.client.0.smithi181.stderr:++ config_opt=min_size 2024-09-15T03:30:11.219 INFO:tasks.workunit.client.0.smithi181.stderr:+++ ceph osd pool get foo min_size 2024-09-15T03:30:11.586 INFO:tasks.workunit.client.0.smithi181.stderr:++ raw='min_size: 2' 2024-09-15T03:30:11.587 INFO:tasks.workunit.client.0.smithi181.stderr:++ [[ 0 -ne 0 ]] 2024-09-15T03:30:11.587 INFO:tasks.workunit.client.0.smithi181.stderr:+++ echo min_size: 2 2024-09-15T03:30:11.587 INFO:tasks.workunit.client.0.smithi181.stderr:+++ sed -e 's/[{} "]//g' 2024-09-15T03:30:11.588 INFO:tasks.workunit.client.0.smithi181.stderr:++ raw=min_size:2 2024-09-15T03:30:11.588 INFO:tasks.workunit.client.0.smithi181.stderr:+++ echo min_size:2 2024-09-15T03:30:11.588 INFO:tasks.workunit.client.0.smithi181.stderr:+++ cut -f2 -d: 2024-09-15T03:30:11.590 INFO:tasks.workunit.client.0.smithi181.stderr:++ val=2 2024-09-15T03:30:11.590 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo 2 2024-09-15T03:30:11.590 INFO:tasks.workunit.client.0.smithi181.stderr:++ return 0 2024-09-15T03:30:11.590 INFO:tasks.workunit.client.0.smithi181.stderr:+ val=2 2024-09-15T03:30:11.591 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 2 != \2 ]] 2024-09-15T03:30:11.591 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set foo size 10 2024-09-15T03:30:11.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.663+0000 7f677b79d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.663+0000 7f677b79d640 10 monclient: build_initial_monmap 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.663+0000 7f677b79d640 10 monclient: monmap: 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:11.664318+0000 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:11.664318+0000 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:11.665 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:11.666 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:11.666 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:11.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 Processor -- start 2024-09-15T03:30:11.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 -- start start 2024-09-15T03:30:11.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient: init 2024-09-15T03:30:11.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:11.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:30:11.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f677409ec50 0x7f6774131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:11.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): picked mon.noname-c con 0x7f677409ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:11.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67741321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:11.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f677409f020 0x7f6774069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:11.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): picked mon.noname-a con 0x7f677409f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:11.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6774067420 con 0x7f67741321a0 2024-09-15T03:30:11.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:11.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677409fdc0 con 0x7f677409f020 2024-09-15T03:30:11.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:11.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67740a0510 con 0x7f677409ec50 2024-09-15T03:30:11.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f677b79d640 10 monclient(hunting): authenticate will time out at 816.145820s 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f6778d38640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f67741321a0 legacy=0x7f6774068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f6779539640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f677409ec50 msgr2=0x7f6774131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.664+0000 7f6779539640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f677409ec50 0x7f6774131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f677409f020 0x7f6774069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f677409f020 0x7f6774069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:47202/0 (socket says 172.21.15.181:47202) 2024-09-15T03:30:11.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 -- 172.21.15.181:0/1277912500 learned_addr learned my addr 172.21.15.181:0/1277912500 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:11.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): get_auth_request con 0x7f677409f020 auth_method 0 2024-09-15T03:30:11.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:11.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:11.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:11.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient(hunting): handle_auth_done global_id 34320 payload 274 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 -- 172.21.15.181:0/1277912500 >> v2:172.21.15.181:3300/0 conn(0x7f677409ec50 msgr2=0x7f6774131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 --2- 172.21.15.181:0/1277912500 >> v2:172.21.15.181:3300/0 conn(0x7f677409ec50 0x7f6774131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 -- 172.21.15.181:0/1277912500 >> v1:172.21.15.181:6789/0 conn(0x7f67741321a0 legacy=0x7f6774068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 monclient: found mon.noname-a 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:11.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.665+0000 7f6779d3a640 1 -- 172.21.15.181:0/1277912500 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740a0200 con 0x7f677409f020 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f6779d3a640 1 -- 172.21.15.181:0/1277912500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677409f020 msgr2=0x7f6774069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f6779d3a640 1 --2- 172.21.15.181:0/1277912500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677409f020 0x7f6774069410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f67740719d0 tx=0x7f676c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17b5d5cee735db8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 1 -- 172.21.15.181:0/1277912500 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c02aea0 con 0x7f677409f020 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: dump: 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:11.686 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: _finish_auth 0 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 1 -- 172.21.15.181:0/1277912500 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c02d050 con 0x7f677409f020 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 1 -- 172.21.15.181:0/1277912500 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c007840 con 0x7f677409f020 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 5 monclient: authenticate success, global_id 34320 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 4 set_mon_vals no callback set 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f676bfff640 10 monclient: dump: 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:11.688 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 10 monclient: shutdown 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 1 -- 172.21.15.181:0/1277912500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677409f020 msgr2=0x7f6774069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.668+0000 7f677b79d640 1 --2- 172.21.15.181:0/1277912500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677409f020 0x7f6774069410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f67740719d0 tx=0x7f676c009840 comp rx=0 tx=0).stop 2024-09-15T03:30:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 -- 172.21.15.181:0/1277912500 shutdown_connections 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 --2- 172.21.15.181:0/1277912500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f677409f020 0x7f6774069410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 --2- 172.21.15.181:0/1277912500 >> v2:172.21.15.181:3300/0 conn(0x7f677409ec50 0x7f6774131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 -- 172.21.15.181:0/1277912500 >> 172.21.15.181:0/1277912500 conn(0x7f67740a5d40 msgr2=0x7f67740a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 -- 172.21.15.181:0/1277912500 shutdown_connections 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 -- 172.21.15.181:0/1277912500 wait complete. 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 10 monclient: build_initial_monmap 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 10 monclient: monmap: 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:11.690 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 Processor -- start 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 1 -- start start 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.669+0000 7f677b79d640 10 monclient: init 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677409ec50 0x7f677409a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): picked mon.noname-b con 0x7f677409ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f677409f020 0x7f6774130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): picked mon.noname-c con 0x7f677409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 0x7f6774130ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): picked mon.noname-a con 0x7f67741321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67741349e0 con 0x7f67741321a0 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677411dd80 con 0x7f677409ec50 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6774141d00 con 0x7f677409f020 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:11.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f677b79d640 10 monclient(hunting): authenticate will time out at 816.151514s 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779539640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677409ec50 msgr2=0x7f677409a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779539640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677409ec50 0x7f677409a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 0x7f6774130ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f677409f020 0x7f6774130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 0x7f6774130ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:47206/0 (socket says 172.21.15.181:47206) 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 1 -- 172.21.15.181:0/1040613806 learned_addr learned my addr 172.21.15.181:0/1040613806 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): get_auth_request con 0x7f67741321a0 auth_method 0 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): get_auth_request con 0x7f677409f020 auth_method 0 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:11.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6779d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.670+0000 7f6778d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 10 monclient(hunting): handle_auth_done global_id 34323 payload 824 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 1 -- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f677409f020 msgr2=0x7f6774130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f677409f020 0x7f6774130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 1 -- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677409ec50 msgr2=0x7f677409a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677409ec50 0x7f677409a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 1 monclient: found mon.noname-a 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.671+0000 7f6779d3a640 1 -- 172.21.15.181:0/1040613806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c009ae0 con 0x7f67741321a0 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f6779d3a640 1 -- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 msgr2=0x7f6774130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f6779d3a640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 0x7f6774130ec0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f676c009880 tx=0x7f676c036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c032ce0 con 0x7f67741321a0 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: dump: 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: _finish_auth 0 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c032e40 con 0x7f67741321a0 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 5 monclient: authenticate success, global_id 34323 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 10 monclient: _renew_subs 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677411daa0 con 0x7f67741321a0 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c03b620 con 0x7f67741321a0 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f67697fa640 10 monclient: dump: 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f676b7fe640 4 set_mon_vals no callback set 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 10 monclient: _renew_subs 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.673+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677411cbb0 con 0x7f67741321a0 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f677b79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f677b79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f677b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67740008d0 con 0x7f67741321a0 2024-09-15T03:30:11.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f676c003020 con 0x7f67741321a0 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f67697fa640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6758038720 0x7f675803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f676c06c980 con 0x7f67741321a0 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f6779539640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6758038720 0x7f675803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.674+0000 7f6779539640 10 monclient: get_auth_request con 0x7f6758038720 auth_method 0 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.677+0000 7f6779539640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6758038720 0x7f675803abd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6770006fd0 tx=0x7f6770006d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.679+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f676c039900 con 0x7f67741321a0 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.679+0000 7f67697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.679+0000 7f67697fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:11.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.832+0000 7f677b79d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T03:30:11.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.832+0000 7f677b79d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T03:30:11.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.832+0000 7f677b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:11.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:11.832+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f67740a13c0 con 0x7f67741321a0 2024-09-15T03:30:12.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.195+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v36) v1 ==== 124+0+0 (secure 0 0 0) 0x7f676c03f070 con 0x7f67741321a0 2024-09-15T03:30:12.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.195+0000 7f67697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T03:30:12.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.195+0000 7f67697fa640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-15T03:30:12.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.322+0000 7f677b79d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T03:30:12.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.322+0000 7f677b79d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T03:30:12.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.322+0000 7f677b79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:12.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:12.322+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f6774064900 con 0x7f67741321a0 2024-09-15T03:30:12.512 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:30:12.512 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:30:12.512 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:30:12.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:30:12.515 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:30:12.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:30:12.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:12.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.617+0000 7fc7d079d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:12.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.617+0000 7fc7d079d640 10 monclient: build_initial_monmap 2024-09-15T03:30:12.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.617+0000 7fc7d079d640 10 monclient: monmap: 2024-09-15T03:30:12.619 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:12.619 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:12.619 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:12.618445+0000 2024-09-15T03:30:12.619 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:12.618445+0000 2024-09-15T03:30:12.619 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:12.620 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 1 Processor -- start 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 1 -- start start 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 10 monclient: init 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 0x7fc7c809d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.618+0000 7fc7d079d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7c809c9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7c809e340 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7c809e750 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c809bdd0 con 0x7fc7c809e340 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c809a880 con 0x7fc7c809e750 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8130590 con 0x7fc7c809c9c0 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7d079d640 10 monclient(hunting): authenticate will time out at 810.258103s 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7cdd38640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc7c809e340 0x7fc7c812f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:51720/0 (socket says 172.21.15.153:51720) 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 0x7fc7c809d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7cdd38640 1 -- 172.21.15.153:0/2943407197 learned_addr learned my addr 172.21.15.153:0/2943407197 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): get_auth_request con 0x7fc7c809c9c0 auth_method 0 2024-09-15T03:30:12.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient(hunting): handle_auth_done global_id 4604 payload 274 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 1 -- 172.21.15.153:0/2943407197 >> v1:172.21.15.181:6789/0 conn(0x7fc7c809e750 legacy=0x7fc7c812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 1 -- 172.21.15.153:0/2943407197 >> v1:172.21.15.153:6789/0 conn(0x7fc7c809e340 legacy=0x7fc7c812f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient: _finish_hunting 0 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 1 monclient: found mon.noname-b 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:12.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.619+0000 7fc7ce539640 1 -- 172.21.15.153:0/2943407197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c809a440 con 0x7fc7c809c9c0 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7ce539640 1 -- 172.21.15.153:0/2943407197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 msgr2=0x7fc7c809d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7ce539640 1 --2- 172.21.15.153:0/2943407197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 0x7fc7c809d830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40098b0 tx=0x7fc7c40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5ea2eafebac15a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 1 -- 172.21.15.153:0/2943407197 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4032440 con 0x7fc7c809c9c0 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: dump: 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:12.630 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: _finish_auth 0 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 1 -- 172.21.15.153:0/2943407197 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c40325a0 con 0x7fc7c809c9c0 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 5 monclient: authenticate success, global_id 4604 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:12.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 1 -- 172.21.15.153:0/2943407197 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4032940 con 0x7fc7c809c9c0 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 4 set_mon_vals no callback set 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7cd537640 10 monclient: dump: 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:12.632 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 10 monclient: shutdown 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 1 -- 172.21.15.153:0/2943407197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 msgr2=0x7fc7c809d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.622+0000 7fc7d079d640 1 --2- 172.21.15.153:0/2943407197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 0x7fc7c809d830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40098b0 tx=0x7fc7c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 -- 172.21.15.153:0/2943407197 shutdown_connections 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 --2- 172.21.15.153:0/2943407197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809c9c0 0x7fc7c809d830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 -- 172.21.15.153:0/2943407197 >> 172.21.15.153:0/2943407197 conn(0x7fc7c80a2e40 msgr2=0x7fc7c80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:12.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 -- 172.21.15.153:0/2943407197 shutdown_connections 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 -- 172.21.15.153:0/2943407197 wait complete. 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 10 monclient: build_initial_monmap 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 10 monclient: monmap: 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:12.634 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 Processor -- start 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 1 -- start start 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.623+0000 7fc7d079d640 10 monclient: init 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c809c9c0 0x7fc7c8068bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7c809c9c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 0x7fc7c80690f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7c809e340 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 0x7fc7c8069630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7c809e750 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:12.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8098d60 con 0x7fc7c809c9c0 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8121570 con 0x7fc7c809e340 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8121290 con 0x7fc7c809e750 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7d079d640 10 monclient(hunting): authenticate will time out at 810.263420s 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c809c9c0 0x7fc7c8068bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 0x7fc7c8069630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 0x7fc7c8069630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:42210/0 (socket says 172.21.15.153:42210) 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): get_auth_request con 0x7fc7c809c9c0 auth_method 0 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 -- 172.21.15.153:0/849377167 learned_addr learned my addr 172.21.15.153:0/849377167 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): get_auth_request con 0x7fc7c809e750 auth_method 0 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 0x7fc7c80690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ce539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:12.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 10 monclient(hunting): get_auth_request con 0x7fc7c809e340 auth_method 0 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 msgr2=0x7fc7c80690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient(hunting): handle_auth_done global_id 4607 payload 824 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 msgr2=0x7fc7c80690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 0x7fc7c80690f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 msgr2=0x7fc7c80690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7cdd38640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 0x7fc7c80690f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 0x7fc7c80690f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c809c9c0 msgr2=0x7fc7c8068bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c809c9c0 0x7fc7c8068bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:12.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 monclient: found mon.noname-c 2024-09-15T03:30:12.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:12.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:12.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.624+0000 7fc7ced3a640 1 -- 172.21.15.153:0/849377167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c4009390 con 0x7fc7c809e750 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7ced3a640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 msgr2=0x7fc7c8069630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7ced3a640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 0x7fc7c8069630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000c890 tx=0x7fc7c0002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000ce80 con 0x7fc7c809e750 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: dump: 2024-09-15T03:30:12.641 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c000aa00 con 0x7fc7c809e750 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 5 monclient: authenticate success, global_id 4607 2024-09-15T03:30:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000ae00 con 0x7fc7c809e750 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 10 monclient: _renew_subs 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c812f290 con 0x7fc7c809e750 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7ccd36640 4 set_mon_vals no callback set 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7be7fc640 10 monclient: dump: 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:12.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 10 monclient: _renew_subs 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:12.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.627+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c8069d10 con 0x7fc7c809e750 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc7c0012630 con 0x7fc7c809e750 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7d079d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7d079d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7d079d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c8099d30 con 0x7fc7c809e750 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7be7fc640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7ac038740 0x7fc7ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fc7c004bac0 con 0x7fc7c809e750 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7ce539640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7ac038740 0x7fc7ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.628+0000 7fc7ce539640 10 monclient: get_auth_request con 0x7fc7ac038740 auth_method 0 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.630+0000 7fc7ce539640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7ac038740 0x7fc7ac03abf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7c809ef00 tx=0x7fc7c4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.632+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc7c8099d30 con 0x7fc7c809e750 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.632+0000 7fc7be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.632+0000 7fc7be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:12.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.848+0000 7fc7d079d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.848+0000 7fc7d079d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.848+0000 7fc7d079d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.848+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc7c8099d30 con 0x7fc7c809e750 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.849+0000 7fc7be7fc640 1 -- 172.21.15.153:0/849377167 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fc7c8099d30 con 0x7fc7c809e750 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.849+0000 7fc7be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.849+0000 7fc7be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:12.850 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7ac038740 msgr2=0x7fc7ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7ac038740 0x7fc7ac03abf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7c809ef00 tx=0x7fc7c4032000 comp rx=0 tx=0).stop 2024-09-15T03:30:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 10 monclient: shutdown 2024-09-15T03:30:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 msgr2=0x7fc7c8069630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 0x7fc7c8069630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000c890 tx=0x7fc7c0002640 comp rx=0 tx=0).stop 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 shutdown_connections 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7ac038740 0x7fc7ac03abf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7c809e750 0x7fc7c8069630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c809e340 0x7fc7c80690f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 --2- 172.21.15.153:0/849377167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c809c9c0 0x7fc7c8068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 >> 172.21.15.153:0/849377167 conn(0x7fc7c80a2e40 msgr2=0x7fc7c812fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 shutdown_connections 2024-09-15T03:30:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:12.851+0000 7fc7d079d640 1 -- 172.21.15.153:0/849377167 wait complete. 2024-09-15T03:30:13.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.197+0000 7f67697fa640 1 -- 172.21.15.181:0/1040613806 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v37) v1 ==== 124+0+0 (secure 0 0 0) 0x7f676c039300 con 0x7f67741321a0 2024-09-15T03:30:13.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.197+0000 7f67697fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-15T03:30:13.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.197+0000 7f67697fa640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-15T03:30:13.199 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 4 size to 10 2024-09-15T03:30:13.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6758038720 msgr2=0x7f675803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6758038720 0x7f675803abd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6770006fd0 tx=0x7f6770006d30 comp rx=0 tx=0).stop 2024-09-15T03:30:13.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 10 monclient: shutdown 2024-09-15T03:30:13.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 msgr2=0x7f6774130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 0x7f6774130ec0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f676c009880 tx=0x7f676c036000 comp rx=0 tx=0).stop 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 shutdown_connections 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6758038720 0x7f675803abd0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6770006fd0 tx=0x7f6770006d30 comp rx=0 tx=0).stop 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67741321a0 0x7f6774130ec0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f677409f020 0x7f6774130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 --2- 172.21.15.181:0/1040613806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677409ec50 0x7f677409a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.205+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 >> 172.21.15.181:0/1040613806 conn(0x7f67740a5d40 msgr2=0x7f6774134bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.208+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 shutdown_connections 2024-09-15T03:30:13.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.208+0000 7f677b79d640 1 -- 172.21.15.181:0/1040613806 wait complete. 2024-09-15T03:30:13.226 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_config_value foo min_size 5 2024-09-15T03:30:13.226 INFO:tasks.workunit.client.0.smithi181.stderr:+ local pool_name config_opt expected_val val 2024-09-15T03:30:13.226 INFO:tasks.workunit.client.0.smithi181.stderr:+ pool_name=foo 2024-09-15T03:30:13.226 INFO:tasks.workunit.client.0.smithi181.stderr:+ config_opt=min_size 2024-09-15T03:30:13.228 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_val=5 2024-09-15T03:30:13.229 INFO:tasks.workunit.client.0.smithi181.stderr:++ get_config_value_or_die foo min_size 2024-09-15T03:30:13.229 INFO:tasks.workunit.client.0.smithi181.stderr:++ local pool_name config_opt raw val 2024-09-15T03:30:13.229 INFO:tasks.workunit.client.0.smithi181.stderr:++ pool_name=foo 2024-09-15T03:30:13.229 INFO:tasks.workunit.client.0.smithi181.stderr:++ config_opt=min_size 2024-09-15T03:30:13.229 INFO:tasks.workunit.client.0.smithi181.stderr:+++ ceph osd pool get foo min_size 2024-09-15T03:30:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:++ raw='min_size: 5' 2024-09-15T03:30:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:++ [[ 0 -ne 0 ]] 2024-09-15T03:30:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:+++ echo min_size: 5 2024-09-15T03:30:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:+++ sed -e 's/[{} "]//g' 2024-09-15T03:30:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:++ raw=min_size:5 2024-09-15T03:30:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:+++ echo min_size:5 2024-09-15T03:30:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:+++ cut -f2 -d: 2024-09-15T03:30:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:++ val=5 2024-09-15T03:30:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:++ echo 5 2024-09-15T03:30:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:++ return 0 2024-09-15T03:30:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ val=5 2024-09-15T03:30:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 5 != \5 ]] 2024-09-15T03:30:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-15T03:30:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set foo size 0 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.687+0000 7fee7a75d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.687+0000 7fee7a75d640 10 monclient: build_initial_monmap 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.687+0000 7fee7a75d640 10 monclient: monmap: 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:13.687815+0000 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:13.687815+0000 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:13.688 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:13.689 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:13.689 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:13.689 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:13.689 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.687+0000 7fee7a75d640 1 Processor -- start 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.687+0000 7fee7a75d640 1 -- start start 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.687+0000 7fee7a75d640 10 monclient: init 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee74098c70 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee74099c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 0x7fee7413a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient(hunting): picked mon.noname-c con 0x7fee741383e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7409f680 con 0x7fee74098c70 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7409f420 con 0x7fee74099c70 2024-09-15T03:30:13.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee74068a20 con 0x7fee741383e0 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee7a75d640 10 monclient(hunting): authenticate will time out at 818.169461s 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 0x7fee7413a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee6bfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fee74099c70 0x7fee74136cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51048/0 (socket says 172.21.15.181:51048) 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee6bfff640 1 -- 172.21.15.181:0/1296653056 learned_addr learned my addr 172.21.15.181:0/1296653056 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 10 monclient(hunting): get_auth_request con 0x7fee741383e0 auth_method 0 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 msgr2=0x7fee7413a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:30:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 msgr2=0x7fee7413a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:13.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 1 --2- 172.21.15.181:0/1296653056 >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 0x7fee7413a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:13.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.688+0000 7fee78cfa640 1 --2- 172.21.15.181:0/1296653056 >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 0x7fee7413a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.690+0000 7fee6bfff640 1 -- 172.21.15.181:0/1296653056 >> v1:172.21.15.181:6789/0 conn(0x7fee74099c70 legacy=0x7fee74136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.690+0000 7fee73fff640 1 -- 172.21.15.181:0/1296653056 >> v1:172.21.15.153:6789/0 conn(0x7fee74098c70 legacy=0x7fee74099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069327674 0 0) 0x7fee7409f680 con 0x7fee74098c70 2024-09-15T03:30:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): my global_id is 34335 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee600034e0 con 0x7fee74098c70 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1933670204 0 0) 0x7fee600034e0 con 0x7fee74098c70 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 msgr2=0x7fee7413a7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 --2- 172.21.15.181:0/1296653056 >> v2:172.21.15.181:3300/0 conn(0x7fee741383e0 0x7fee7413a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 >> v1:172.21.15.181:6789/0 conn(0x7fee74099c70 legacy=0x7fee74136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74096a30 con 0x7fee74098c70 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:13.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fee64001830 con 0x7fee74098c70 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee7a75d640 5 monclient: authenticate success, global_id 34335 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee7a75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: dump: 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.705 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 >> v1:172.21.15.153:6789/0 conn(0x7fee74098c70 legacy=0x7fee74099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60003a50 0x7fee60003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): picked mon.a con 0x7fee60003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee60004780 0x7fee60004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): picked mon.c con 0x7fee60004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee6000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): picked mon.b con 0x7fee60005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee74068a20 con 0x7fee60003a50 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6000c2d0 con 0x7fee60005330 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6000c490 con 0x7fee60004780 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee737fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee7a75d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee6000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): get_auth_request con 0x7fee60005330 auth_method 0 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60003a50 0x7fee60003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): get_auth_request con 0x7fee60003a50 auth_method 0 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee60004780 0x7fee60004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): get_auth_request con 0x7fee60004780 auth_method 0 2024-09-15T03:30:13.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee73fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient(hunting): handle_auth_done global_id 34335 payload 293 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee60004780 msgr2=0x7fee60004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee60004780 0x7fee60004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60003a50 msgr2=0x7fee60003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:13.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee60003a50 0x7fee60003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:30:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 monclient: found mon.b 2024-09-15T03:30:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600034e0 con 0x7fee60005330 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 10 monclient: _finish_auth 0 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.691+0000 7fee78cfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.692+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 reap_dead start 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee78cfa640 1 -- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 msgr2=0x7fee6000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee78cfa640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee6000b9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009590 tx=0x7fee6c009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a756cba1494cf860 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c02e610 con 0x7fee60005330 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: dump: 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee7a75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6c02e790 con 0x7fee60005330 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 1 -- 172.21.15.181:0/1296653056 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c02eb50 con 0x7fee60005330 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee7a75d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee7a75d640 4 set_mon_vals no callback set 2024-09-15T03:30:13.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.694+0000 7fee737fe640 10 monclient: dump: 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.713 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 10 monclient: shutdown 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 -- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 msgr2=0x7fee6000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee6000b9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009590 tx=0x7fee6c009720 comp rx=0 tx=0).stop 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 -- 172.21.15.181:0/1296653056 shutdown_connections 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 --2- 172.21.15.181:0/1296653056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee6000b9f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 -- 172.21.15.181:0/1296653056 >> 172.21.15.181:0/1296653056 conn(0x7fee74070e90 msgr2=0x7fee740a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 -- 172.21.15.181:0/1296653056 shutdown_connections 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 1 -- 172.21.15.181:0/1296653056 wait complete. 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 10 monclient: build_initial_monmap 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.695+0000 7fee7a75d640 10 monclient: monmap: 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 Processor -- start 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 -- start start 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient: init 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee74137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): picked mon.noname-b con 0x7fee60005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee74120d30 0x7fee7411e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): picked mon.noname-a con 0x7fee74120d30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee7411eb60 0x7fee7411efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): picked mon.noname-c con 0x7fee7411eb60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7406eba0 con 0x7fee74120d30 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee740977e0 con 0x7fee60005330 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee741222e0 con 0x7fee7411eb60 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee7a75d640 10 monclient(hunting): authenticate will time out at 818.177869s 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee74137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee74137d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43126/0 (socket says 172.21.15.181:43126) 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 1 -- 172.21.15.181:0/3527980240 learned_addr learned my addr 172.21.15.181:0/3527980240 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 10 monclient(hunting): get_auth_request con 0x7fee60005330 auth_method 0 2024-09-15T03:30:13.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 1 -- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 msgr2=0x7fee74137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 1 -- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 msgr2=0x7fee74137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.696+0000 7fee73fff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee74137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee74120d30 0x7fee7411e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee73fff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee74137d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee7411eb60 0x7fee7411efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): get_auth_request con 0x7fee74120d30 auth_method 0 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): get_auth_request con 0x7fee7411eb60 auth_method 0 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient(hunting): handle_auth_done global_id 34338 payload 824 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 -- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee7411eb60 msgr2=0x7fee7411efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee7411eb60 0x7fee7411efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 -- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 msgr2=0x7fee74137d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee60005330 0x7fee74137d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee6bfff640 1 -- 172.21.15.181:0/3527980240 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74122950 con 0x7fee74120d30 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.697+0000 7fee78cfa640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee7411eb60 0x7fee7411efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee6bfff640 1 -- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee74120d30 msgr2=0x7fee7411e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee6bfff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee74120d30 0x7fee7411e620 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0078c0 tx=0x7fee5c007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c018030 con 0x7fee74120d30 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: dump: 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:13.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 5 monclient: authenticate success, global_id 34338 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c00f800 con 0x7fee74120d30 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00fba0 con 0x7fee74120d30 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 10 monclient: _renew_subs 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 1 -- 172.21.15.181:0/3527980240 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee74120160 con 0x7fee74120d30 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee70ff9640 10 monclient: dump: 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.722 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee72ffd640 4 set_mon_vals no callback set 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 10 monclient: _renew_subs 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.699+0000 7fee7a75d640 1 -- 172.21.15.181:0/3527980240 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee74137270 con 0x7fee74120d30 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.700+0000 7fee7a75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.700+0000 7fee7a75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.700+0000 7fee7a75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.700+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fee5c01b070 con 0x7fee74120d30 2024-09-15T03:30:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.700+0000 7fee7a75d640 1 -- 172.21.15.181:0/3527980240 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7409a070 con 0x7fee74120d30 2024-09-15T03:30:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.700+0000 7fee70ff9640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee54038740 0x7fee5403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.701+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fee5c019070 con 0x7fee74120d30 2024-09-15T03:30:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.701+0000 7fee73fff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee54038740 0x7fee5403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.701+0000 7fee73fff640 10 monclient: get_auth_request con 0x7fee54038740 auth_method 0 2024-09-15T03:30:13.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.703+0000 7fee73fff640 1 --2- 172.21.15.181:0/3527980240 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee54038740 0x7fee5403abf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fee64007fd0 tx=0x7fee64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:13.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.705+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fee5c01bb00 con 0x7fee74120d30 2024-09-15T03:30:13.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.705+0000 7fee70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:13.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.705+0000 7fee70ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:13.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee7a75d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-15T03:30:13.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee7a75d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-15T03:30:13.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee7a75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:13.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee7a75d640 1 -- 172.21.15.181:0/3527980240 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) v1 -- 0x7fee740008d0 con 0x7fee74120d30 2024-09-15T03:30:13.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee70ff9640 1 -- 172.21.15.181:0/3527980240 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v37) v1 ==== 136+0+0 (secure 0 0 0) 0x7fee5c01b320 con 0x7fee74120d30 2024-09-15T03:30:13.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-15T03:30:13.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.859+0000 7fee70ff9640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-15T03:30:13.862 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-15T03:30:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-15T03:30:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set foo size 20 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.939+0000 7f3dbbe34640 10 monclient: get_monmap_and_config 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.939+0000 7f3dbbe34640 10 monclient: build_initial_monmap 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.939+0000 7f3dbbe34640 10 monclient: monmap: 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:13.940096+0000 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:13.940096+0000 2024-09-15T03:30:13.940 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:13.941 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.939+0000 7f3dbbe34640 1 Processor -- start 2024-09-15T03:30:13.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 1 -- start start 2024-09-15T03:30:13.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient: init 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient(hunting): picked mon.noname-b con 0x7f3db409ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3db41321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3db409f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db4067420 con 0x7f3db41321a0 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db409fdc0 con 0x7f3db409f020 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40a0510 con 0x7f3db409ec50 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dbbe34640 10 monclient(hunting): authenticate will time out at 818.421690s 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dba3d1640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3db409f020 0x7f3db4069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51060/0 (socket says 172.21.15.181:51060) 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.940+0000 7f3dba3d1640 1 -- 172.21.15.181:0/3545567190 learned_addr learned my addr 172.21.15.181:0/3545567190 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 1 --2- 172.21.15.181:0/3545567190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db4131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): get_auth_request con 0x7f3db409ec50 auth_method 0 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:13.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_done global_id 4616 payload 274 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 1 -- 172.21.15.181:0/3545567190 >> v1:172.21.15.181:6789/0 conn(0x7f3db409f020 legacy=0x7f3db4069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 1 -- 172.21.15.181:0/3545567190 >> v1:172.21.15.153:6789/0 conn(0x7f3db41321a0 legacy=0x7f3db4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient: _finish_hunting 0 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 1 monclient: found mon.noname-b 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:13.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.941+0000 7f3db9bd0640 1 -- 172.21.15.181:0/3545567190 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db40a0200 con 0x7f3db409ec50 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db9bd0640 1 -- 172.21.15.181:0/3545567190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 msgr2=0x7f3db4131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db9bd0640 1 --2- 172.21.15.181:0/3545567190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db4131c60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009880 tx=0x7f3db00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=987ad06c6c2a2cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 1 -- 172.21.15.181:0/3545567190 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0032440 con 0x7f3db409ec50 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: dump: 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.950 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: _finish_auth 0 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3dbbe34640 5 monclient: authenticate success, global_id 4616 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 1 -- 172.21.15.181:0/3545567190 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db00325a0 con 0x7f3db409ec50 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3dbbe34640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3dbbe34640 4 set_mon_vals no callback set 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 1 -- 172.21.15.181:0/3545567190 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0032940 con 0x7f3db409ec50 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.944+0000 7f3db8bce640 10 monclient: dump: 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 10 monclient: shutdown 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 -- 172.21.15.181:0/3545567190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 msgr2=0x7f3db4131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 --2- 172.21.15.181:0/3545567190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db4131c60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009880 tx=0x7f3db00090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 -- 172.21.15.181:0/3545567190 shutdown_connections 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 --2- 172.21.15.181:0/3545567190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db4131c60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 -- 172.21.15.181:0/3545567190 >> 172.21.15.181:0/3545567190 conn(0x7f3db40a5d40 msgr2=0x7f3db40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 -- 172.21.15.181:0/3545567190 shutdown_connections 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 1 -- 172.21.15.181:0/3545567190 wait complete. 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 10 monclient: build_initial_monmap 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.945+0000 7f3dbbe34640 10 monclient: monmap: 2024-09-15T03:30:13.953 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 Processor -- start 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 -- start start 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient: init 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db409a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): picked mon.noname-c con 0x7f3db409ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db409f020 0x7f3db4130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): picked mon.noname-a con 0x7f3db409f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 0x7f3db411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): picked mon.noname-b con 0x7f3db41321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db409de10 con 0x7f3db409f020 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db411e250 con 0x7f3db41321a0 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db411dee0 con 0x7f3db409ec50 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dbbe34640 10 monclient(hunting): authenticate will time out at 818.427524s 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 0x7f3db411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 0x7f3db411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43128/0 (socket says 172.21.15.181:43128) 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 -- 172.21.15.181:0/1497381564 learned_addr learned my addr 172.21.15.181:0/1497381564 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 10 monclient(hunting): get_auth_request con 0x7f3db41321a0 auth_method 0 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 -- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 msgr2=0x7f3db411b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 -- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 msgr2=0x7f3db411b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db409f020 0x7f3db4130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db409a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 0x7f3db411b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): get_auth_request con 0x7f3db409f020 auth_method 0 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): get_auth_request con 0x7f3db409ec50 auth_method 0 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3dba3d1640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 0x7f3db411b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.946+0000 7f3db93cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 10 monclient(hunting): handle_auth_done global_id 4619 payload 824 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 1 -- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 msgr2=0x7f3db411b760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41321a0 0x7f3db411b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 1 -- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db409f020 msgr2=0x7f3db4130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db409f020 0x7f3db4130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 10 monclient: _finish_hunting 0 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 1 monclient: found mon.noname-c 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db9bd0640 1 -- 172.21.15.181:0/1497381564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db0009390 con 0x7f3db409ec50 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.947+0000 7f3db93cf640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db409f020 0x7f3db4130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3db9bd0640 1 -- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 msgr2=0x7f3db409a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:13.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3db9bd0640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3db409ec50 0x7f3db409a7e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3db409f500 tx=0x7f3db0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0000d10 con 0x7f3db409ec50 2024-09-15T03:30:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: dump: 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db0031c70 con 0x7f3db409ec50 2024-09-15T03:30:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3dbbe34640 5 monclient: authenticate success, global_id 4619 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3dbbe34640 10 monclient: _renew_subs 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3dbbe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3dbbe34640 1 -- 172.21.15.181:0/1497381564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db4141f80 con 0x7f3db409ec50 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db003e450 con 0x7f3db409ec50 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3da9ffb640 10 monclient: dump: 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.949+0000 7f3dabfff640 4 set_mon_vals no callback set 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.950+0000 7f3dbbe34640 10 monclient: _renew_subs 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.950+0000 7f3dbbe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.950+0000 7f3dbbe34640 1 -- 172.21.15.181:0/1497381564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db4141620 con 0x7f3db409ec50 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3dbbe34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3dbbe34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3dbbe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3dbbe34640 1 -- 172.21.15.181:0/1497381564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db409b3f0 con 0x7f3db409ec50 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3db003e8a0 con 0x7f3db409ec50 2024-09-15T03:30:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3da9ffb640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d98038790 0x7f3d9803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f3db006bf40 con 0x7f3db409ec50 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3db93cf640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d98038790 0x7f3d9803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.951+0000 7f3db93cf640 10 monclient: get_auth_request con 0x7f3d98038790 auth_method 0 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.954+0000 7f3db93cf640 1 --2- 172.21.15.181:0/1497381564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d98038790 0x7f3d9803ac40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3db411db30 tx=0x7f3da4007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.956+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3db00456d0 con 0x7f3db409ec50 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.956+0000 7f3da9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:13.956+0000 7f3da9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:14.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.111+0000 7f3dbbe34640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-15T03:30:14.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.111+0000 7f3dbbe34640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-15T03:30:14.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.111+0000 7f3dbbe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.111+0000 7f3dbbe34640 1 -- 172.21.15.181:0/1497381564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) v1 -- 0x7f3db409b3f0 con 0x7f3db409ec50 2024-09-15T03:30:14.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.112+0000 7f3da9ffb640 1 -- 172.21.15.181:0/1497381564 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v37) v1 ==== 137+0+0 (secure 0 0 0) 0x7f3db0037020 con 0x7f3db409ec50 2024-09-15T03:30:14.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.112+0000 7f3da9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-15T03:30:14.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.112+0000 7f3da9ffb640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-15T03:30:14.113 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-15T03:30:14.117 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:14.117 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool delete foo 2024-09-15T03:30:14.117 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:14.118 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo 2024-09-15T03:30:14.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.188+0000 7f886d04f640 10 monclient: get_monmap_and_config 2024-09-15T03:30:14.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.188+0000 7f886d04f640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.188+0000 7f886d04f640 10 monclient: monmap: 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:14.189349+0000 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:14.189349+0000 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:14.190 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 1 Processor -- start 2024-09-15T03:30:14.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 1 -- start start 2024-09-15T03:30:14.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient: init 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88680a1800 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88680983d0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f8868137200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient(hunting): picked mon.noname-b con 0x7f886812ec70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88680a1bd0 con 0x7f88680983d0 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886809cbf0 con 0x7f88680a1800 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886812f600 con 0x7f886812ec70 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f886d04f640 10 monclient(hunting): authenticate will time out at 818.670667s 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f8866d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f88680a1800 0x7f886809aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51074/0 (socket says 172.21.15.181:51074) 2024-09-15T03:30:14.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.189+0000 7f8866d9d640 1 -- 172.21.15.181:0/2358135311 learned_addr learned my addr 172.21.15.181:0/2358135311 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 1 --2- 172.21.15.181:0/2358135311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f8868137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): get_auth_request con 0x7f886812ec70 auth_method 0 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient(hunting): handle_auth_done global_id 4622 payload 274 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 1 -- 172.21.15.181:0/2358135311 >> v1:172.21.15.181:6789/0 conn(0x7f88680a1800 legacy=0x7f886809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 1 -- 172.21.15.181:0/2358135311 >> v1:172.21.15.153:6789/0 conn(0x7f88680983d0 legacy=0x7f8868133ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 1 monclient: found mon.noname-b 2024-09-15T03:30:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.190+0000 7f886759e640 1 -- 172.21.15.181:0/2358135311 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868098cc0 con 0x7f886812ec70 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886759e640 1 -- 172.21.15.181:0/2358135311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 msgr2=0x7f8868137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886759e640 1 --2- 172.21.15.181:0/2358135311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f8868137200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8860009e90 tx=0x7f8860009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e444f64dc5fc972f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 1 -- 172.21.15.181:0/2358135311 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860007840 con 0x7f886812ec70 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: dump: 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.201 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 1 -- 172.21.15.181:0/2358135311 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886002d050 con 0x7f886812ec70 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 5 monclient: authenticate success, global_id 4622 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:14.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 1 -- 172.21.15.181:0/2358135311 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860007bd0 con 0x7f886812ec70 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 4 set_mon_vals no callback set 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f8865d9b640 10 monclient: dump: 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.203 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 10 monclient: shutdown 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 -- 172.21.15.181:0/2358135311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 msgr2=0x7f8868137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 --2- 172.21.15.181:0/2358135311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f8868137200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8860009e90 tx=0x7f8860009780 comp rx=0 tx=0).stop 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 -- 172.21.15.181:0/2358135311 shutdown_connections 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 --2- 172.21.15.181:0/2358135311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f8868137200 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 -- 172.21.15.181:0/2358135311 >> 172.21.15.181:0/2358135311 conn(0x7f8868071100 msgr2=0x7f88680a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 -- 172.21.15.181:0/2358135311 shutdown_connections 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.193+0000 7f886d04f640 1 -- 172.21.15.181:0/2358135311 wait complete. 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.194+0000 7f886d04f640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.194+0000 7f886d04f640 10 monclient: monmap: 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:14.205 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.194+0000 7f886d04f640 1 Processor -- start 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.194+0000 7f886d04f640 1 -- start start 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.194+0000 7f886d04f640 10 monclient: init 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 0x7f886809b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): picked mon.noname-a con 0x7f88680a1800 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f886812d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): picked mon.noname-c con 0x7f886812ec70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 0x7f8868119760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): picked mon.noname-b con 0x7f8868133140 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886811bfa0 con 0x7f88680a1800 2024-09-15T03:30:14.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886811bc30 con 0x7f8868133140 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886809c100 con 0x7f886812ec70 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886d04f640 10 monclient(hunting): authenticate will time out at 818.676372s 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 0x7f8868119760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 0x7f8868119760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43138/0 (socket says 172.21.15.181:43138) 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 -- 172.21.15.181:0/340167056 learned_addr learned my addr 172.21.15.181:0/340167056 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 10 monclient(hunting): get_auth_request con 0x7f8868133140 auth_method 0 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 msgr2=0x7f8868119760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 msgr2=0x7f8868119760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 0x7f8868119760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886759e640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 0x7f8868119760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f8866d9d640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 msgr2=0x7f886809b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f8866d9d640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 msgr2=0x7f886809b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f8866d9d640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 0x7f886809b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f8866d9d640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 0x7f886809b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f886812d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): get_auth_request con 0x7f886812ec70 auth_method 0 2024-09-15T03:30:14.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.195+0000 7f886659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 10 monclient(hunting): handle_auth_done global_id 4625 payload 824 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 msgr2=0x7f8868119760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8868133140 0x7f8868119760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 msgr2=0x7f886809b8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88680a1800 0x7f886809b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 1 monclient: found mon.noname-c 2024-09-15T03:30:14.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.196+0000 7f886659c640 1 -- 172.21.15.181:0/340167056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8860009a20 con 0x7f886812ec70 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886659c640 1 -- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 msgr2=0x7f886812d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886659c640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f886812ec70 0x7f886812d820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f885c00c510 tx=0x7f885c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0117a0 con 0x7f886812ec70 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: dump: 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.210 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c011920 con 0x7f886812ec70 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 5 monclient: authenticate success, global_id 4625 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 10 monclient: _renew_subs 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 1 -- 172.21.15.181:0/340167056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886813fb70 con 0x7f886812ec70 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c011ce0 con 0x7f886812ec70 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f884effd640 10 monclient: dump: 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 10 monclient: _renew_subs 2024-09-15T03:30:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886d04f640 1 -- 172.21.15.181:0/340167056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886811aa60 con 0x7f886812ec70 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.198+0000 7f886559a640 4 set_mon_vals no callback set 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.199+0000 7f886d04f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.199+0000 7f886d04f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.199+0000 7f886d04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.199+0000 7f886d04f640 1 -- 172.21.15.181:0/340167056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88680008d0 con 0x7f886812ec70 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.199+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f885c012630 con 0x7f886812ec70 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.200+0000 7f884effd640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8854038790 0x7f885403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.200+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f885c04bab0 con 0x7f886812ec70 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.200+0000 7f8866d9d640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8854038790 0x7f885403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.200+0000 7f8866d9d640 10 monclient: get_auth_request con 0x7f8854038790 auth_method 0 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.202+0000 7f8866d9d640 1 --2- 172.21.15.181:0/340167056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8854038790 0x7f885403ac40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88580047e0 tx=0x7f8858007b40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.204+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f885c01c460 con 0x7f886812ec70 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.204+0000 7f884effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.204+0000 7f884effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:14.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.358+0000 7f886d04f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-15T03:30:14.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.358+0000 7f886d04f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-15T03:30:14.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.358+0000 7f886d04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:14.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.358+0000 7f886d04f640 1 -- 172.21.15.181:0/340167056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) v1 -- 0x7f8868098fa0 con 0x7f886812ec70 2024-09-15T03:30:14.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.360+0000 7f884effd640 1 -- 172.21.15.181:0/340167056 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v37) v1 ==== 274+0+0 (secure 0 0 0) 0x7f885c0102e0 con 0x7f886812ec70 2024-09-15T03:30:14.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.360+0000 7f884effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-15T03:30:14.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.360+0000 7f884effd640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T03:30:14.361 INFO:tasks.workunit.client.0.smithi181.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T03:30:14.365 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:14.365 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-15T03:30:14.365 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:14.365 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo foo 2024-09-15T03:30:14.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.437+0000 7efcad95d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:14.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.437+0000 7efcad95d640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.437+0000 7efcad95d640 10 monclient: monmap: 2024-09-15T03:30:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:14.438033+0000 2024-09-15T03:30:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:14.438033+0000 2024-09-15T03:30:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:14.441 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.437+0000 7efcad95d640 1 Processor -- start 2024-09-15T03:30:14.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.437+0000 7efcad95d640 1 -- start start 2024-09-15T03:30:14.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.437+0000 7efcad95d640 10 monclient: init 2024-09-15T03:30:14.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 0x7efca8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): picked mon.noname-c con 0x7efca809ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efca81321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efca809f020 0x7efca8069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): picked mon.noname-a con 0x7efca809f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8067420 con 0x7efca81321a0 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca809fdc0 con 0x7efca809f020 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca80a0510 con 0x7efca809ec50 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efcad95d640 10 monclient(hunting): authenticate will time out at 818.919519s 2024-09-15T03:30:14.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 0x7efca8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca67fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7efca81321a0 0x7efca8068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51084/0 (socket says 172.21.15.181:51084) 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca67fc640 1 -- 172.21.15.181:0/253397310 learned_addr learned my addr 172.21.15.181:0/253397310 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 10 monclient(hunting): get_auth_request con 0x7efca809ec50 auth_method 0 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 1 -- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 msgr2=0x7efca8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 1 -- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 msgr2=0x7efca8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 1 --2- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 0x7efca8131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca6ffd640 1 --2- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 0x7efca8131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 1 --2- 172.21.15.181:0/253397310 >> v2:172.21.15.153:3300/0 conn(0x7efca809f020 0x7efca8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): get_auth_request con 0x7efca809f020 auth_method 0 2024-09-15T03:30:14.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.438+0000 7efca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 10 monclient(hunting): handle_auth_done global_id 34344 payload 274 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 1 -- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 msgr2=0x7efca8131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 1 --2- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 0x7efca8131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 1 -- 172.21.15.181:0/253397310 >> v1:172.21.15.181:6789/0 conn(0x7efca81321a0 legacy=0x7efca8068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.439+0000 7efca77fe640 1 -- 172.21.15.181:0/253397310 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca80a0200 con 0x7efca809f020 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca77fe640 1 -- 172.21.15.181:0/253397310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809f020 msgr2=0x7efca8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca77fe640 1 --2- 172.21.15.181:0/253397310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809f020 0x7efca8069410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efca80719d0 tx=0x7efca0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878f819d1ed124fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 1 -- 172.21.15.181:0/253397310 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0007760 con 0x7efca809f020 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: dump: 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.459 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 1 -- 172.21.15.181:0/253397310 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca002d050 con 0x7efca809f020 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 1 -- 172.21.15.181:0/253397310 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0007af0 con 0x7efca809f020 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 5 monclient: authenticate success, global_id 34344 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 4 set_mon_vals no callback set 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efca5ffb640 10 monclient: dump: 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.461 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 10 monclient: shutdown 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 -- 172.21.15.181:0/253397310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809f020 msgr2=0x7efca8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 --2- 172.21.15.181:0/253397310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809f020 0x7efca8069410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efca80719d0 tx=0x7efca0009780 comp rx=0 tx=0).stop 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 -- 172.21.15.181:0/253397310 shutdown_connections 2024-09-15T03:30:14.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 --2- 172.21.15.181:0/253397310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809f020 0x7efca8069410 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 --2- 172.21.15.181:0/253397310 >> v2:172.21.15.181:3300/0 conn(0x7efca809ec50 0x7efca8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 -- 172.21.15.181:0/253397310 >> 172.21.15.181:0/253397310 conn(0x7efca80a5d40 msgr2=0x7efca80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 -- 172.21.15.181:0/253397310 shutdown_connections 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.442+0000 7efcad95d640 1 -- 172.21.15.181:0/253397310 wait complete. 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.443+0000 7efcad95d640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.443+0000 7efcad95d640 10 monclient: monmap: 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.463 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.443+0000 7efcad95d640 1 Processor -- start 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.443+0000 7efcad95d640 1 -- start start 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.443+0000 7efcad95d640 10 monclient: init 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809ec50 0x7efca809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): picked mon.noname-a con 0x7efca809ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efca809f020 0x7efca81309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): picked mon.noname-c con 0x7efca809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 0x7efca811b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): picked mon.noname-b con 0x7efca81321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca8141bf0 con 0x7efca809ec50 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca809c000 con 0x7efca81321a0 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca809e430 con 0x7efca809f020 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efcad95d640 10 monclient(hunting): authenticate will time out at 818.925450s 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 0x7efca811b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 0x7efca811b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43156/0 (socket says 172.21.15.181:43156) 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 -- 172.21.15.181:0/521914830 learned_addr learned my addr 172.21.15.181:0/521914830 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:14.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 10 monclient(hunting): get_auth_request con 0x7efca81321a0 auth_method 0 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 -- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 msgr2=0x7efca811b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 -- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 msgr2=0x7efca811b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809ec50 0x7efca809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 0x7efca811b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): get_auth_request con 0x7efca809ec50 auth_method 0 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efca809f020 0x7efca81309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca77fe640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 0x7efca811b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): get_auth_request con 0x7efca809f020 auth_method 0 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient(hunting): handle_auth_done global_id 34347 payload 824 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 -- 172.21.15.181:0/521914830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efca809f020 msgr2=0x7efca81309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efca809f020 0x7efca81309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 -- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 msgr2=0x7efca811b760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca81321a0 0x7efca811b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.444+0000 7efca6ffd640 1 -- 172.21.15.181:0/521914830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0009a20 con 0x7efca809ec50 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efca6ffd640 1 -- 172.21.15.181:0/521914830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809ec50 msgr2=0x7efca809aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efca6ffd640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efca809ec50 0x7efca809aa00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc98006ea0 tx=0x7efc98008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800b850 con 0x7efca809ec50 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:14.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: dump: 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800b9b0 con 0x7efca809ec50 2024-09-15T03:30:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800bd70 con 0x7efca809ec50 2024-09-15T03:30:14.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 5 monclient: authenticate success, global_id 34347 2024-09-15T03:30:14.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efc8effd640 10 monclient: dump: 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 10 monclient: _renew_subs 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efca57fa640 4 set_mon_vals no callback set 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 1 -- 172.21.15.181:0/521914830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca811daf0 con 0x7efca809ec50 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 10 monclient: _renew_subs 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.447+0000 7efcad95d640 1 -- 172.21.15.181:0/521914830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca811cb90 con 0x7efca809ec50 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efcad95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efcad95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efcad95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efcad95d640 1 -- 172.21.15.181:0/521914830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca811e120 con 0x7efca809ec50 2024-09-15T03:30:14.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efc98014640 con 0x7efca809ec50 2024-09-15T03:30:14.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efc8effd640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efc94038740 0x7efc9403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.448+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7efc9804c0b0 con 0x7efca809ec50 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.449+0000 7efca67fc640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efc94038740 0x7efc9403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.449+0000 7efca67fc640 10 monclient: get_auth_request con 0x7efc94038740 auth_method 0 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.451+0000 7efca67fc640 1 --2- 172.21.15.181:0/521914830 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efc94038740 0x7efc9403abf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efca809bb60 tx=0x7efc9c007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.453+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efc980148f0 con 0x7efca809ec50 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.453+0000 7efc8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.453+0000 7efc8effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.607+0000 7efcad95d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-15T03:30:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.607+0000 7efcad95d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-15T03:30:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.607+0000 7efcad95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.607+0000 7efcad95d640 1 -- 172.21.15.181:0/521914830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) v1 -- 0x7efca80008d0 con 0x7efca809ec50 2024-09-15T03:30:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.608+0000 7efc8effd640 1 -- 172.21.15.181:0/521914830 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v37) v1 ==== 290+0+0 (secure 0 0 0) 0x7efc9801b300 con 0x7efca809ec50 2024-09-15T03:30:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.608+0000 7efc8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-15T03:30:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.608+0000 7efc8effd640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T03:30:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-15T03:30:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-15T03:30:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo foo --force 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.687+0000 7f7bd1c34640 10 monclient: get_monmap_and_config 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.687+0000 7f7bd1c34640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.687+0000 7f7bd1c34640 10 monclient: monmap: 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:14.687899+0000 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:14.687899+0000 2024-09-15T03:30:14.688 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:14.689 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.687+0000 7f7bd1c34640 1 Processor -- start 2024-09-15T03:30:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 -- start start 2024-09-15T03:30:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient: init 2024-09-15T03:30:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:30:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): picked mon.noname-c con 0x7f7bcc09ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): picked mon.noname-b con 0x7f7bcc1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7bcc09f020 0x7f7bcc069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): picked mon.noname-a con 0x7f7bcc09f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc067420 con 0x7f7bcc09f020 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc09fdc0 con 0x7f7bcc09ec50 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc0a0510 con 0x7f7bcc1321a0 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bd1c34640 10 monclient(hunting): authenticate will time out at 819.169496s 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43166/0 (socket says 172.21.15.181:43166) 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/1512859113 learned_addr learned my addr 172.21.15.181:0/1512859113 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 10 monclient(hunting): get_auth_request con 0x7f7bcc09ec50 auth_method 0 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 msgr2=0x7f7bcc131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 msgr2=0x7f7bcc131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 --2- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 1 --2- 172.21.15.181:0/1512859113 >> v2:172.21.15.153:3300/0 conn(0x7f7bcc09f020 0x7f7bcc069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcb7fe640 1 --2- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): get_auth_request con 0x7f7bcc09f020 auth_method 0 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 1 --2- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): get_auth_request con 0x7f7bcc1321a0 auth_method 0 2024-09-15T03:30:14.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.688+0000 7f7bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_done global_id 34350 payload 274 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 1 -- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 msgr2=0x7f7bcc068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 1 --2- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 1 -- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 msgr2=0x7f7bcc131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 1 --2- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.689+0000 7f7bcbfff640 1 -- 172.21.15.181:0/1512859113 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0a0200 con 0x7f7bcc09f020 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.691+0000 7f7bcbfff640 1 -- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09f020 msgr2=0x7f7bcc069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.691+0000 7f7bcbfff640 1 --2- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09f020 0x7f7bcc069240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0719d0 tx=0x7f7bc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7262e9f2ae50752c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 1 -- 172.21.15.181:0/1512859113 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4007760 con 0x7f7bcc09f020 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: dump: 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.707 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 1 -- 172.21.15.181:0/1512859113 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc402d050 con 0x7f7bcc09f020 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 1 -- 172.21.15.181:0/1512859113 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4007af0 con 0x7f7bcc09f020 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 5 monclient: authenticate success, global_id 34350 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 4 set_mon_vals no callback set 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bcaffd640 10 monclient: dump: 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 10 monclient: shutdown 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 -- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09f020 msgr2=0x7f7bcc069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 --2- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09f020 0x7f7bcc069240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0719d0 tx=0x7f7bc4009780 comp rx=0 tx=0).stop 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 -- 172.21.15.181:0/1512859113 shutdown_connections 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 --2- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09f020 0x7f7bcc069240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 --2- 172.21.15.181:0/1512859113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 --2- 172.21.15.181:0/1512859113 >> v2:172.21.15.181:3300/0 conn(0x7f7bcc09ec50 0x7f7bcc131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 -- 172.21.15.181:0/1512859113 >> 172.21.15.181:0/1512859113 conn(0x7f7bcc0a5d40 msgr2=0x7f7bcc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:14.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 -- 172.21.15.181:0/1512859113 shutdown_connections 2024-09-15T03:30:14.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.692+0000 7f7bd1c34640 1 -- 172.21.15.181:0/1512859113 wait complete. 2024-09-15T03:30:14.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.693+0000 7f7bd1c34640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.693+0000 7f7bd1c34640 10 monclient: monmap: 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:14.713 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.693+0000 7f7bd1c34640 1 Processor -- start 2024-09-15T03:30:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.693+0000 7f7bd1c34640 1 -- start start 2024-09-15T03:30:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.693+0000 7f7bd1c34640 10 monclient: init 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09ec50 0x7f7bcc09a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): picked mon.noname-a con 0x7f7bcc09ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 0x7f7bcc130910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): picked mon.noname-b con 0x7f7bcc09f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:14.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): picked mon.noname-c con 0x7f7bcc1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:14.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc11e080 con 0x7f7bcc09ec50 2024-09-15T03:30:14.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc09c000 con 0x7f7bcc09f020 2024-09-15T03:30:14.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc09dff0 con 0x7f7bcc1321a0 2024-09-15T03:30:14.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bd1c34640 10 monclient(hunting): authenticate will time out at 819.175129s 2024-09-15T03:30:14.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 0x7f7bcc130910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 0x7f7bcc130910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43170/0 (socket says 172.21.15.181:43170) 2024-09-15T03:30:14.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 -- 172.21.15.181:0/2153596553 learned_addr learned my addr 172.21.15.181:0/2153596553 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:14.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 10 monclient(hunting): get_auth_request con 0x7f7bcc09f020 auth_method 0 2024-09-15T03:30:14.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 -- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 msgr2=0x7f7bcc130910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 -- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 msgr2=0x7f7bcc130910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 0x7f7bcc130910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09ec50 0x7f7bcc09a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bc3fff640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 0x7f7bcc130910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): get_auth_request con 0x7f7bcc09ec50 auth_method 0 2024-09-15T03:30:14.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): get_auth_request con 0x7f7bcc1321a0 auth_method 0 2024-09-15T03:30:14.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient(hunting): handle_auth_done global_id 34353 payload 824 2024-09-15T03:30:14.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/2153596553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 msgr2=0x7f7bcc11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bcc1321a0 0x7f7bcc11b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 msgr2=0x7f7bcc130910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:14.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bcc09f020 0x7f7bcc130910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:14.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.694+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/2153596553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc4009a20 con 0x7f7bcc09ec50 2024-09-15T03:30:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bcb7fe640 1 -- 172.21.15.181:0/2153596553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09ec50 msgr2=0x7f7bcc09a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bcb7fe640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bcc09ec50 0x7f7bcc09a860 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00e870 tx=0x7f7bbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 1 -- 172.21.15.181:0/2153596553 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00c6d0 con 0x7f7bcc09ec50 2024-09-15T03:30:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: dump: 2024-09-15T03:30:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.725 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.725 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.725 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.725 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.725 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 1 -- 172.21.15.181:0/2153596553 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc00c830 con 0x7f7bcc09ec50 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 5 monclient: authenticate success, global_id 34353 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 1 -- 172.21.15.181:0/2153596553 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00cbf0 con 0x7f7bcc09ec50 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 10 monclient: _renew_subs 2024-09-15T03:30:14.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 1 -- 172.21.15.181:0/2153596553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc09b210 con 0x7f7bcc09ec50 2024-09-15T03:30:14.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bca7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:14.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:14.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bc37fe640 10 monclient: dump: 2024-09-15T03:30:14.727 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.728 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.728 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.728 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.728 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.728 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.728 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.729 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.729 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.729 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 10 monclient: _renew_subs 2024-09-15T03:30:14.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.697+0000 7f7bd1c34640 1 -- 172.21.15.181:0/2153596553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc11cad0 con 0x7f7bcc09ec50 2024-09-15T03:30:14.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bc37fe640 1 -- 172.21.15.181:0/2153596553 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7bbc013410 con 0x7f7bcc09ec50 2024-09-15T03:30:14.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bd1c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bd1c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bd1c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bc37fe640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7bac038740 0x7f7bac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bd1c34640 1 -- 172.21.15.181:0/2153596553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc0008d0 con 0x7f7bcc09ec50 2024-09-15T03:30:14.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bc37fe640 1 -- 172.21.15.181:0/2153596553 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f7bbc04bdd0 con 0x7f7bcc09ec50 2024-09-15T03:30:14.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bc3fff640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7bac038740 0x7f7bac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.698+0000 7f7bc3fff640 10 monclient: get_auth_request con 0x7f7bac038740 auth_method 0 2024-09-15T03:30:14.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.700+0000 7f7bc3fff640 1 --2- 172.21.15.181:0/2153596553 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7bac038740 0x7f7bac03abf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc09b4b0 tx=0x7f7bb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.703+0000 7f7bc37fe640 1 -- 172.21.15.181:0/2153596553 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7bbc01a910 con 0x7f7bcc09ec50 2024-09-15T03:30:14.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.703+0000 7f7bc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.703+0000 7f7bc37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-15T03:30:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:Invalid command: unused arguments: ['--force'] 2024-09-15T03:30:14.861 INFO:tasks.workunit.client.0.smithi181.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-15T03:30:14.861 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: invalid command 2024-09-15T03:30:14.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:14.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-15T03:30:14.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:14.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-15T03:30:14.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.936+0000 7f4e91217640 10 monclient: get_monmap_and_config 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.936+0000 7f4e91217640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.936+0000 7f4e91217640 10 monclient: monmap: 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:14.937587+0000 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:14.937587+0000 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:14.938 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 Processor -- start 2024-09-15T03:30:14.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 -- start start 2024-09-15T03:30:14.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient: init 2024-09-15T03:30:14.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:30:14.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 0x7f4e8c09c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8c09cad0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4e8c09cea0 0x7f4e8c09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8c09cea0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e8c09d2a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c09b710 con 0x7f4e8c09d2a0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c09c6e0 con 0x7f4e8c09cea0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c09a650 con 0x7f4e8c09cad0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.937+0000 7f4e91217640 10 monclient(hunting): authenticate will time out at 819.418880s 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 0x7f4e8c09c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8b59e640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4e8c09d2a0 0x7f4e8c09e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51088/0 (socket says 172.21.15.181:51088) 2024-09-15T03:30:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8b59e640 1 -- 172.21.15.181:0/4269898846 learned_addr learned my addr 172.21.15.181:0/4269898846 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 10 monclient(hunting): get_auth_request con 0x7f4e8c09cad0 auth_method 0 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 1 -- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 msgr2=0x7f4e8c09c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 1 -- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 msgr2=0x7f4e8c09c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 1 --2- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 0x7f4e8c09c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8ad9d640 1 --2- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 0x7f4e8c09c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 1 --2- 172.21.15.181:0/4269898846 >> v2:172.21.15.153:3300/0 conn(0x7f4e8c09cea0 0x7f4e8c09dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): get_auth_request con 0x7f4e8c09cea0 auth_method 0 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient(hunting): handle_auth_done global_id 34356 payload 274 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 1 -- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 msgr2=0x7f4e8c09c1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 1 --2- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 0x7f4e8c09c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 1 -- 172.21.15.181:0/4269898846 >> v1:172.21.15.181:6789/0 conn(0x7f4e8c09d2a0 legacy=0x7f4e8c09e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 1 monclient: found mon.noname-a 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.938+0000 7f4e8259c640 1 -- 172.21.15.181:0/4269898846 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c09f660 con 0x7f4e8c09cea0 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8259c640 1 -- 172.21.15.181:0/4269898846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 msgr2=0x7f4e8c09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8259c640 1 --2- 172.21.15.181:0/4269898846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 0x7f4e8c09dad0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009850 tx=0x7f4e740090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d498cc39a4b5c5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 1 -- 172.21.15.181:0/4269898846 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7402ae50 con 0x7f4e8c09cea0 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: dump: 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 1 -- 172.21.15.181:0/4269898846 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74032440 con 0x7f4e8c09cea0 2024-09-15T03:30:14.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e91217640 5 monclient: authenticate success, global_id 34356 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e91217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e91217640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 1 -- 172.21.15.181:0/4269898846 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e740327e0 con 0x7f4e8c09cea0 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e91217640 4 set_mon_vals no callback set 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.941+0000 7f4e8a59c640 10 monclient: dump: 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.953 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 10 monclient: shutdown 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 -- 172.21.15.181:0/4269898846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 msgr2=0x7f4e8c09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 --2- 172.21.15.181:0/4269898846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 0x7f4e8c09dad0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009850 tx=0x7f4e740090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 -- 172.21.15.181:0/4269898846 shutdown_connections 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 --2- 172.21.15.181:0/4269898846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 0x7f4e8c09dad0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 --2- 172.21.15.181:0/4269898846 >> v2:172.21.15.181:3300/0 conn(0x7f4e8c09cad0 0x7f4e8c09c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 -- 172.21.15.181:0/4269898846 >> 172.21.15.181:0/4269898846 conn(0x7f4e8c070e90 msgr2=0x7f4e8c0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 -- 172.21.15.181:0/4269898846 shutdown_connections 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.942+0000 7f4e91217640 1 -- 172.21.15.181:0/4269898846 wait complete. 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient: build_initial_monmap 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient: monmap: 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:14.955 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 1 Processor -- start 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 1 -- start start 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient: init 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e8c09cad0 0x7f4e8c130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8c09cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 0x7f4e8c11a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.943+0000 7f4e91217640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8c09cea0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 0x7f4e8c11aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8c09d2a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0a29d0 con 0x7f4e8c09cea0 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c13da30 con 0x7f4e8c09d2a0 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c11db20 con 0x7f4e8c09cad0 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e91217640 10 monclient(hunting): authenticate will time out at 819.425032s 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 0x7f4e8c11aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 0x7f4e8c11aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43186/0 (socket says 172.21.15.181:43186) 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 -- 172.21.15.181:0/2635366143 learned_addr learned my addr 172.21.15.181:0/2635366143 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 10 monclient(hunting): get_auth_request con 0x7f4e8c09d2a0 auth_method 0 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 0x7f4e8c11a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): get_auth_request con 0x7f4e8c09cea0 auth_method 0 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e8c09cad0 0x7f4e8c130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): get_auth_request con 0x7f4e8c09cad0 auth_method 0 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 -- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 msgr2=0x7f4e8c11aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 -- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 msgr2=0x7f4e8c11aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 0x7f4e8c11aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:14.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8b59e640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 0x7f4e8c11aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient(hunting): handle_auth_done global_id 34359 payload 824 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 -- 172.21.15.181:0/2635366143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e8c09cad0 msgr2=0x7f4e8c130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e8c09cad0 0x7f4e8c130a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 -- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 msgr2=0x7f4e8c11aa50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e8c09d2a0 0x7f4e8c11aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 monclient: found mon.noname-a 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.944+0000 7f4e8259c640 1 -- 172.21.15.181:0/2635366143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e74009390 con 0x7f4e8c09cea0 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.946+0000 7f4e8259c640 1 -- 172.21.15.181:0/2635366143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 msgr2=0x7f4e8c11a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.946+0000 7f4e8259c640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8c09cea0 0x7f4e8c11a510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4e74034000 tx=0x7f4e74032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 1 -- 172.21.15.181:0/2635366143 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74031c70 con 0x7f4e8c09cea0 2024-09-15T03:30:14.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: dump: 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.962 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 5 monclient: authenticate success, global_id 34359 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 10 monclient: _renew_subs 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 1 -- 172.21.15.181:0/2635366143 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74031e10 con 0x7f4e8c09cea0 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 1 -- 172.21.15.181:0/2635366143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c11de00 con 0x7f4e8c09cea0 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 1 -- 172.21.15.181:0/2635366143 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7403c650 con 0x7f4e8c09cea0 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e837fe640 10 monclient: dump: 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:14.964 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 10 monclient: _renew_subs 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e89d9b640 4 set_mon_vals no callback set 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.947+0000 7f4e91217640 1 -- 172.21.15.181:0/2635366143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c11cc30 con 0x7f4e8c09cea0 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e91217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e91217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e91217640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:14.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e91217640 1 -- 172.21.15.181:0/2635366143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c0008d0 con 0x7f4e8c09cea0 2024-09-15T03:30:14.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e837fe640 1 -- 172.21.15.181:0/2635366143 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4e7403ca00 con 0x7f4e8c09cea0 2024-09-15T03:30:14.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e837fe640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e6c038740 0x7f4e6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:14.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e837fe640 1 -- 172.21.15.181:0/2635366143 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f4e7406cc80 con 0x7f4e8c09cea0 2024-09-15T03:30:14.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e8ad9d640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e6c038740 0x7f4e6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:14.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.948+0000 7f4e8ad9d640 10 monclient: get_auth_request con 0x7f4e6c038740 auth_method 0 2024-09-15T03:30:14.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.950+0000 7f4e8ad9d640 1 --2- 172.21.15.181:0/2635366143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4e6c038740 0x7f4e6c03abf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c006fd0 tx=0x7f4e7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:14.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.952+0000 7f4e837fe640 1 -- 172.21.15.181:0/2635366143 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4e7403ccb0 con 0x7f4e8c09cea0 2024-09-15T03:30:14.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.952+0000 7f4e837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:14.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:14.952+0000 7f4e837fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:15.117 INFO:tasks.workunit.client.0.smithi181.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-15T03:30:15.117 INFO:tasks.workunit.client.0.smithi181.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-15T03:30:15.117 INFO:tasks.workunit.client.0.smithi181.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-15T03:30:15.117 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: invalid command 2024-09-15T03:30:15.119 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:15.119 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-15T03:30:15.119 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:15.119 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-15T03:30:15.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.193+0000 7f796d2c0640 10 monclient: get_monmap_and_config 2024-09-15T03:30:15.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.193+0000 7f796d2c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:15.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.193+0000 7f796d2c0640 10 monclient: monmap: 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:15.194276+0000 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:15.194276+0000 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:15.195 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:15.196 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:15.196 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:15.196 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 Processor -- start 2024-09-15T03:30:15.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 -- start start 2024-09-15T03:30:15.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient: init 2024-09-15T03:30:15.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:15.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f796809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79681321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 0x7f7968069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): picked mon.noname-c con 0x7f796809f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7968067420 con 0x7f79681321a0 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796809fdc0 con 0x7f796809f020 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a0510 con 0x7f796809ec50 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f796d2c0640 10 monclient(hunting): authenticate will time out at 819.675624s 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 0x7f7968069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f795e7fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f79681321a0 0x7f7968068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51096/0 (socket says 172.21.15.181:51096) 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 10 monclient(hunting): get_auth_request con 0x7f796809f020 auth_method 0 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f795e7fc640 1 -- 172.21.15.181:0/995047877 learned_addr learned my addr 172.21.15.181:0/995047877 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 1 -- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 msgr2=0x7f7968069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 1 -- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 msgr2=0x7f7968069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 1 --2- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 0x7f7968069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f79677fe640 1 --2- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 0x7f7968069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.194+0000 7f7966ffd640 1 --2- 172.21.15.181:0/995047877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): get_auth_request con 0x7f796809ec50 auth_method 0 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient(hunting): handle_auth_done global_id 4640 payload 274 2024-09-15T03:30:15.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 1 -- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 msgr2=0x7f7968069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 1 --2- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 0x7f7968069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 1 -- 172.21.15.181:0/995047877 >> v1:172.21.15.181:6789/0 conn(0x7f79681321a0 legacy=0x7f7968068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 1 monclient: found mon.noname-b 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.195+0000 7f7966ffd640 1 -- 172.21.15.181:0/995047877 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680a0200 con 0x7f796809ec50 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f7966ffd640 1 -- 172.21.15.181:0/995047877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 msgr2=0x7f7968131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f7966ffd640 1 --2- 172.21.15.181:0/995047877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 0x7f7968131c60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7958009850 tx=0x7f79580090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4abef87dd45d600b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 1 -- 172.21.15.181:0/995047877 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795802adf0 con 0x7f796809ec50 2024-09-15T03:30:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: dump: 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.215 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 1 -- 172.21.15.181:0/995047877 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7958032440 con 0x7f796809ec50 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 5 monclient: authenticate success, global_id 4640 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 1 -- 172.21.15.181:0/995047877 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79580327e0 con 0x7f796809ec50 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 4 set_mon_vals no callback set 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f79667fc640 10 monclient: dump: 2024-09-15T03:30:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 10 monclient: shutdown 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 1 -- 172.21.15.181:0/995047877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 msgr2=0x7f7968131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.198+0000 7f796d2c0640 1 --2- 172.21.15.181:0/995047877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 0x7f7968131c60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7958009850 tx=0x7f79580090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 1 -- 172.21.15.181:0/995047877 shutdown_connections 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 1 --2- 172.21.15.181:0/995047877 >> v2:172.21.15.181:3300/0 conn(0x7f796809f020 0x7f7968069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 1 --2- 172.21.15.181:0/995047877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809ec50 0x7f7968131c60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 1 -- 172.21.15.181:0/995047877 >> 172.21.15.181:0/995047877 conn(0x7f79680a5d40 msgr2=0x7f79680a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 1 -- 172.21.15.181:0/995047877 shutdown_connections 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 1 -- 172.21.15.181:0/995047877 wait complete. 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.199+0000 7f796d2c0640 10 monclient: monmap: 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 Processor -- start 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 -- start start 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient: init 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 0x7f796809ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): picked mon.noname-a con 0x7f796809ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): picked mon.noname-c con 0x7f796809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 0x7f7968141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f79681321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796809c000 con 0x7f796809ec50 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796811cd10 con 0x7f79681321a0 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a4270 con 0x7f796809f020 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f796d2c0640 10 monclient(hunting): authenticate will time out at 819.681562s 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 0x7f7968141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 0x7f7968141470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43204/0 (socket says 172.21.15.181:43204) 2024-09-15T03:30:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 -- 172.21.15.181:0/2456754741 learned_addr learned my addr 172.21.15.181:0/2456754741 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 10 monclient(hunting): get_auth_request con 0x7f79681321a0 auth_method 0 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 0x7f796809ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 -- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 msgr2=0x7f7968141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 -- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 msgr2=0x7f7968141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): get_auth_request con 0x7f796809f020 auth_method 0 2024-09-15T03:30:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 0x7f7968141470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): get_auth_request con 0x7f796809ec50 auth_method 0 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f79677fe640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 0x7f7968141470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f795e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.200+0000 7f7966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 10 monclient(hunting): handle_auth_done global_id 34362 payload 824 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 1 -- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 msgr2=0x7f7968130fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 1 -- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 msgr2=0x7f7968141470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79681321a0 0x7f7968141470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:15.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 1 monclient: found mon.noname-a 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f7966ffd640 1 -- 172.21.15.181:0/2456754741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958009390 con 0x7f796809ec50 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.201+0000 7f795e7fc640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f796809f020 0x7f7968130fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f7966ffd640 1 -- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 msgr2=0x7f796809ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f7966ffd640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f796809ec50 0x7f796809ae70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7968071a00 tx=0x7f7958032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 1 -- 172.21.15.181:0/2456754741 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958031bb0 con 0x7f796809ec50 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: dump: 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:15.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 1 -- 172.21.15.181:0/2456754741 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7958031d50 con 0x7f796809ec50 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 5 monclient: authenticate success, global_id 34362 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 10 monclient: _renew_subs 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 1 -- 172.21.15.181:0/2456754741 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795803d670 con 0x7f796809ec50 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 1 -- 172.21.15.181:0/2456754741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796811cad0 con 0x7f796809ec50 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f795f7fe640 10 monclient: dump: 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f7965ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 10 monclient: _renew_subs 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.203+0000 7f796d2c0640 1 -- 172.21.15.181:0/2456754741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796811bc30 con 0x7f796809ec50 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f796d2c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f796d2c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f796d2c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f796d2c0640 1 -- 172.21.15.181:0/2456754741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79680008d0 con 0x7f796809ec50 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f795f7fe640 1 -- 172.21.15.181:0/2456754741 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7958039040 con 0x7f796809ec50 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f795f7fe640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7948038790 0x7f794803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.204+0000 7f795f7fe640 1 -- 172.21.15.181:0/2456754741 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f795806bc00 con 0x7f796809ec50 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.205+0000 7f795e7fc640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7948038790 0x7f794803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.205+0000 7f795e7fc640 10 monclient: get_auth_request con 0x7f7948038790 auth_method 0 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.207+0000 7f795e7fc640 1 --2- 172.21.15.181:0/2456754741 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7948038790 0x7f794803ac40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f796811e020 tx=0x7f7950007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.209+0000 7f795f7fe640 1 -- 172.21.15.181:0/2456754741 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f795803daa0 con 0x7f796809ec50 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.209+0000 7f795f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.209+0000 7f795f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:15.365 INFO:tasks.workunit.client.0.smithi181.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-15T03:30:15.365 INFO:tasks.workunit.client.0.smithi181.stderr:Invalid command: unused arguments: ['foo'] 2024-09-15T03:30:15.365 INFO:tasks.workunit.client.0.smithi181.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-15T03:30:15.365 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: invalid command 2024-09-15T03:30:15.367 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:15.367 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-15T03:30:15.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.441+0000 7f23740c0640 10 monclient: get_monmap_and_config 2024-09-15T03:30:15.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.441+0000 7f23740c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:15.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.441+0000 7f23740c0640 10 monclient: monmap: 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:15.442402+0000 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:15.442402+0000 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:15.443 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 Processor -- start 2024-09-15T03:30:15.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 -- start start 2024-09-15T03:30:15.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient: init 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f236c131c90 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 0x7f236c069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): picked mon.noname-b con 0x7f236c09d210 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 0x7f236c138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): picked mon.noname-c con 0x7f236c0a0ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236c0a1800 con 0x7f236c131c90 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236c09fea0 con 0x7f236c0a0ab0 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0a1520 con 0x7f236c09d210 2024-09-15T03:30:15.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f23740c0640 10 monclient(hunting): authenticate will time out at 819.923852s 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f237265d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 0x7f236c138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.442+0000 7f2371e5c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f236c131c90 0x7f236c132060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51098/0 (socket says 172.21.15.181:51098) 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f2371e5c640 1 -- 172.21.15.181:0/1183030182 learned_addr learned my addr 172.21.15.181:0/1183030182 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 10 monclient(hunting): get_auth_request con 0x7f236c0a0ab0 auth_method 0 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 1 -- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 msgr2=0x7f236c138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 1 -- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 msgr2=0x7f236c138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 1 --2- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 0x7f236c138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237265d640 1 --2- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 0x7f236c138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:15.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 1 --2- 172.21.15.181:0/1183030182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 0x7f236c069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): get_auth_request con 0x7f236c09d210 auth_method 0 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient(hunting): handle_auth_done global_id 4646 payload 274 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 1 -- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 msgr2=0x7f236c138fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 1 --2- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 0x7f236c138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 1 -- 172.21.15.181:0/1183030182 >> v1:172.21.15.181:6789/0 conn(0x7f236c131c90 legacy=0x7f236c132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 1 monclient: found mon.noname-b 2024-09-15T03:30:15.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:15.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:15.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.443+0000 7f237165b640 1 -- 172.21.15.181:0/1183030182 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c132770 con 0x7f236c09d210 2024-09-15T03:30:15.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f237165b640 1 -- 172.21.15.181:0/1183030182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 msgr2=0x7f236c069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f237165b640 1 --2- 172.21.15.181:0/1183030182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 0x7f236c069e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f235c009850 tx=0x7f235c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b0815c0d3b45b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 1 -- 172.21.15.181:0/1183030182 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c02adf0 con 0x7f236c09d210 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: dump: 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.456 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: _finish_auth 0 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 1 -- 172.21.15.181:0/1183030182 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c032440 con 0x7f236c09d210 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 1 -- 172.21.15.181:0/1183030182 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c0327e0 con 0x7f236c09d210 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f2370e5a640 10 monclient: dump: 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.457 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.446+0000 7f23740c0640 5 monclient: authenticate success, global_id 4646 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 4 set_mon_vals no callback set 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 10 monclient: shutdown 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 -- 172.21.15.181:0/1183030182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 msgr2=0x7f236c069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 --2- 172.21.15.181:0/1183030182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 0x7f236c069e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f235c009850 tx=0x7f235c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 -- 172.21.15.181:0/1183030182 shutdown_connections 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 --2- 172.21.15.181:0/1183030182 >> v2:172.21.15.181:3300/0 conn(0x7f236c0a0ab0 0x7f236c138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 --2- 172.21.15.181:0/1183030182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c09d210 0x7f236c069e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 -- 172.21.15.181:0/1183030182 >> 172.21.15.181:0/1183030182 conn(0x7f236c071370 msgr2=0x7f236c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 -- 172.21.15.181:0/1183030182 shutdown_connections 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.447+0000 7f23740c0640 1 -- 172.21.15.181:0/1183030182 wait complete. 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient: monmap: 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:15.459 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 Processor -- start 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 -- start start 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient: init 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c0a0ab0 0x7f236c131440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): picked mon.noname-c con 0x7f236c0a0ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 0x7f236c096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): picked mon.noname-a con 0x7f236c131c90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): picked mon.noname-b con 0x7f236c09b860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0a0fb0 con 0x7f236c131c90 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c099ee0 con 0x7f236c09b860 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c099c00 con 0x7f236c0a0ab0 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.448+0000 7f23740c0640 10 monclient(hunting): authenticate will time out at 819.929913s 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43222/0 (socket says 172.21.15.181:43222) 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 -- 172.21.15.181:0/3370997076 learned_addr learned my addr 172.21.15.181:0/3370997076 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:15.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 10 monclient(hunting): get_auth_request con 0x7f236c09b860 auth_method 0 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 0x7f236c096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 msgr2=0x7f236c096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 msgr2=0x7f236c096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c0a0ab0 0x7f236c131440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): get_auth_request con 0x7f236c131c90 auth_method 0 2024-09-15T03:30:15.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237265d640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): get_auth_request con 0x7f236c0a0ab0 auth_method 0 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f2371e5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient(hunting): handle_auth_done global_id 34365 payload 824 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c0a0ab0 msgr2=0x7f236c131440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c0a0ab0 0x7f236c131440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 msgr2=0x7f236c096f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 monclient: found mon.noname-a 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.449+0000 7f237165b640 1 -- 172.21.15.181:0/3370997076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c009390 con 0x7f236c131c90 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.451+0000 7f237165b640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 msgr2=0x7f236c096a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.451+0000 7f237165b640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 0x7f236c096a30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f235c031000 tx=0x7f235c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c031b10 con 0x7f236c131c90 2024-09-15T03:30:15.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: dump: 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c031c70 con 0x7f236c131c90 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 5 monclient: authenticate success, global_id 34365 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 10 monclient: _renew_subs 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c099920 con 0x7f236c131c90 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c03d440 con 0x7f236c131c90 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2361ffb640 10 monclient: dump: 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f2363fff640 4 set_mon_vals no callback set 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 10 monclient: _renew_subs 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.452+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c098b60 con 0x7f236c131c90 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f23740c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f23740c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f23740c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c0008d0 con 0x7f236c131c90 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f235c039040 con 0x7f236c131c90 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f2361ffb640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2350038790 0x7f235003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.453+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f235c06b540 con 0x7f236c131c90 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.454+0000 7f2371e5c640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2350038790 0x7f235003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.454+0000 7f2371e5c640 10 monclient: get_auth_request con 0x7f2350038790 auth_method 0 2024-09-15T03:30:15.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.456+0000 7f2371e5c640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2350038790 0x7f235003ac40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2368006df0 tx=0x7f2368005330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.458+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f235c043500 con 0x7f236c131c90 2024-09-15T03:30:15.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.458+0000 7f2361ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.458+0000 7f2361ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:15.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.611+0000 7f23740c0640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:15.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.611+0000 7f23740c0640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:15.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.611+0000 7f23740c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:15.611+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f236c1394f0 con 0x7f236c131c90 2024-09-15T03:30:15.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.966+0000 7fdfc039d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.966+0000 7fdfc039d640 10 monclient: build_initial_monmap 2024-09-15T03:30:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.966+0000 7fdfc039d640 10 monclient: monmap: 2024-09-15T03:30:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:15.967191+0000 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:15.967191+0000 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:15.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 Processor -- start 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 -- start start 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient: init 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfb8131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdfb809fe10 0x7fdfb809ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb809fe10 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb809ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:15.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80a14d0 con 0x7fdfb8131980 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb81325b0 con 0x7fdfb809fe10 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80a11f0 con 0x7fdfb809ef70 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfc039d640 10 monclient(hunting): authenticate will time out at 813.606850s 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe139640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdfb8131980 0x7fdfb8131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44836/0 (socket says 172.21.15.153:44836) 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe139640 1 -- 172.21.15.153:0/1608610659 learned_addr learned my addr 172.21.15.153:0/1608610659 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbd938640 1 --2- 172.21.15.153:0/1608610659 >> v2:172.21.15.153:3300/0 conn(0x7fdfb809fe10 0x7fdfb809ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe93a640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe93a640 10 monclient(hunting): get_auth_request con 0x7fdfb809ef70 auth_method 0 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.967+0000 7fdfbe93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): get_auth_request con 0x7fdfb809fe10 auth_method 0 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbe93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbe93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbe93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient(hunting): handle_auth_done global_id 34368 payload 274 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 1 -- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 msgr2=0x7fdfb8138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:15.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 1 -- 172.21.15.153:0/1608610659 >> v1:172.21.15.153:6789/0 conn(0x7fdfb8131980 legacy=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient: _finish_hunting 0 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 1 monclient: found mon.noname-a 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbd938640 1 -- 172.21.15.153:0/1608610659 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb809f390 con 0x7fdfb809fe10 2024-09-15T03:30:15.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.968+0000 7fdfbe93a640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd938640 1 -- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb809fe10 msgr2=0x7fdfb809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd938640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb809fe10 0x7fdfb809ea30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009850 tx=0x7fdfac0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3dd444617ec9f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 1 -- 172.21.15.153:0/1608610659 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac02adf0 con 0x7fdfb809fe10 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: dump: 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.980 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: _finish_auth 0 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 1 -- 172.21.15.153:0/1608610659 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac032440 con 0x7fdfb809fe10 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 5 monclient: authenticate success, global_id 34368 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 1 -- 172.21.15.153:0/1608610659 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0327e0 con 0x7fdfb809fe10 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 4 set_mon_vals no callback set 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfbd137640 10 monclient: dump: 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 10 monclient: shutdown 2024-09-15T03:30:15.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 -- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb809fe10 msgr2=0x7fdfb809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb809fe10 0x7fdfb809ea30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009850 tx=0x7fdfac0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 -- 172.21.15.153:0/1608610659 shutdown_connections 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 --2- 172.21.15.153:0/1608610659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb809fe10 0x7fdfb809ea30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 -- 172.21.15.153:0/1608610659 >> 172.21.15.153:0/1608610659 conn(0x7fdfb80a5050 msgr2=0x7fdfb80a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 -- 172.21.15.153:0/1608610659 shutdown_connections 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.971+0000 7fdfc039d640 1 -- 172.21.15.153:0/1608610659 wait complete. 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.972+0000 7fdfc039d640 10 monclient: build_initial_monmap 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.972+0000 7fdfc039d640 10 monclient: monmap: 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:15.983 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.972+0000 7fdfc039d640 1 Processor -- start 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.972+0000 7fdfc039d640 1 -- start start 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.972+0000 7fdfc039d640 10 monclient: init 2024-09-15T03:30:15.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb809ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfb809fe10 0x7fdfb8141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb809fe10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb8131980 0x7fdfb81421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb8131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb8069eb0 con 0x7fdfb8131980 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80a4190 con 0x7fdfb809fe10 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb8138090 con 0x7fdfb809ef70 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfc039d640 10 monclient(hunting): authenticate will time out at 813.612230s 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb8131980 0x7fdfb81421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8141760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52308/0 (socket says 172.21.15.153:52308) 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 -- 172.21.15.153:0/3306108235 learned_addr learned my addr 172.21.15.153:0/3306108235 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): get_auth_request con 0x7fdfb809ef70 auth_method 0 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): get_auth_request con 0x7fdfb8131980 auth_method 0 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbd938640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfb809fe10 0x7fdfb8141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbd938640 10 monclient(hunting): get_auth_request con 0x7fdfb809fe10 auth_method 0 2024-09-15T03:30:15.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbd938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbd938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbd938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient(hunting): handle_auth_done global_id 4655 payload 824 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 -- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfb809fe10 msgr2=0x7fdfb8141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfb809fe10 0x7fdfb8141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 -- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb8131980 msgr2=0x7fdfb81421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb8131980 0x7fdfb81421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient: _finish_hunting 0 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 monclient: found mon.noname-c 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:15.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe139640 1 -- 172.21.15.153:0/3306108235 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0009a20 con 0x7fdfb809ef70 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.973+0000 7fdfbe93a640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb8131980 0x7fdfb81421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.975+0000 7fdfbe139640 1 -- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 msgr2=0x7fdfb8141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.975+0000 7fdfbe139640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8141760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c5e0 tx=0x7fdfb4004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.975+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4005cb0 con 0x7fdfb809ef70 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: dump: 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 5 monclient: authenticate success, global_id 4655 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 10 monclient: _renew_subs 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb4005e30 con 0x7fdfb809ef70 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac009390 con 0x7fdfb809ef70 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:15.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4007320 con 0x7fdfb809ef70 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 10 monclient: dump: 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:15.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfbc936640 4 set_mon_vals no callback set 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 10 monclient: _renew_subs 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb8136c80 con 0x7fdfb809ef70 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.976+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdfb401b3b0 con 0x7fdfb809ef70 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfc039d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfaa7fc640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf9c038740 0x7fdf9c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfc039d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfc039d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb804bf60 con 0x7fdfb809ef70 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fdfb404b290 con 0x7fdfb809ef70 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfbd938640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf9c038740 0x7fdf9c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:15.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.977+0000 7fdfbd938640 10 monclient: get_auth_request con 0x7fdf9c038740 auth_method 0 2024-09-15T03:30:15.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.980+0000 7fdfbd938640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf9c038740 0x7fdf9c03abf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdfb806f930 tx=0x7fdfac032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:15.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.981+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdfb4017080 con 0x7fdfb809ef70 2024-09-15T03:30:15.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.981+0000 7fdfaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:15.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:15.981+0000 7fdfaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.200+0000 7fdfc039d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.200+0000 7fdfc039d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.200+0000 7fdfc039d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.200+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdfb8067f00 con 0x7fdfb809ef70 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.201+0000 7fdfaa7fc640 1 -- 172.21.15.153:0/3306108235 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fdfb4018810 con 0x7fdfb809ef70 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.201+0000 7fdfaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.201+0000 7fdfaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:16.203 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf9c038740 msgr2=0x7fdf9c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf9c038740 0x7fdf9c03abf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdfb806f930 tx=0x7fdfac032000 comp rx=0 tx=0).stop 2024-09-15T03:30:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 10 monclient: shutdown 2024-09-15T03:30:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 msgr2=0x7fdfb8141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8141760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c5e0 tx=0x7fdfb4004960 comp rx=0 tx=0).stop 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 shutdown_connections 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf9c038740 0x7fdf9c03abf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfb8131980 0x7fdfb81421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfb809fe10 0x7fdfb8141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 --2- 172.21.15.153:0/3306108235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfb809ef70 0x7fdfb8141760 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 >> 172.21.15.153:0/3306108235 conn(0x7fdfb80a5050 msgr2=0x7fdfb809d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 shutdown_connections 2024-09-15T03:30:16.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:16.203+0000 7fdfc039d640 1 -- 172.21.15.153:0/3306108235 wait complete. 2024-09-15T03:30:16.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.714+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v38) v1 ==== 155+0+0 (secure 0 0 0) 0x7f235c037020 con 0x7f236c131c90 2024-09-15T03:30:16.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.714+0000 7f2361ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:16.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.714+0000 7f2361ffb640 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-15T03:30:16.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.838+0000 7f23740c0640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.838+0000 7f23740c0640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.838+0000 7f23740c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.838+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f236c099d40 con 0x7f236c131c90 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.839+0000 7f2361ffb640 1 -- 172.21.15.181:0/3370997076 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v38) v1 ==== 162+0+0 (secure 0 0 0) 0x7f236c099d40 con 0x7f236c131c90 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.839+0000 7f2361ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.839+0000 7f2361ffb640 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-15T03:30:16.840 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foooo' does not exist 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2350038790 msgr2=0x7f235003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2350038790 0x7f235003ac40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2368006df0 tx=0x7f2368005330 comp rx=0 tx=0).stop 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 10 monclient: shutdown 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 msgr2=0x7f236c096a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 0x7f236c096a30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f235c031000 tx=0x7f235c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 shutdown_connections 2024-09-15T03:30:16.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2350038790 0x7f235003ac40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f236c09b860 0x7f236c096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f236c131c90 0x7f236c096a30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 --2- 172.21.15.181:0/3370997076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f236c0a0ab0 0x7f236c131440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 >> 172.21.15.181:0/3370997076 conn(0x7f236c071370 msgr2=0x7f236c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 shutdown_connections 2024-09-15T03:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.842+0000 7f23740c0640 1 -- 172.21.15.181:0/3370997076 wait complete. 2024-09-15T03:30:16.852 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-15T03:30:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.927+0000 7f7af1d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.927+0000 7f7af1d9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.927+0000 7f7af1d9d640 10 monclient: monmap: 2024-09-15T03:30:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:16.927911+0000 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:16.927911+0000 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:16.929 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.927+0000 7f7af1d9d640 1 Processor -- start 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.927+0000 7f7af1d9d640 1 -- start start 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.927+0000 7f7af1d9d640 10 monclient: init 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 0x7f7aec131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7aec09ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7aec1321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7aec09f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec067420 con 0x7f7aec09f020 2024-09-15T03:30:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec09fdc0 con 0x7f7aec1321a0 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0a0510 con 0x7f7aec09ec50 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7af1d9d640 10 monclient(hunting): authenticate will time out at 821.409423s 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7ae2ffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f7aec1321a0 0x7f7aec068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51100/0 (socket says 172.21.15.181:51100) 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7ae2ffd640 1 -- 172.21.15.181:0/2696512987 learned_addr learned my addr 172.21.15.181:0/2696512987 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 1 --2- 172.21.15.181:0/2696512987 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 0x7f7aec131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): get_auth_request con 0x7f7aec09ec50 auth_method 0 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.928+0000 7f7aeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 10 monclient(hunting): handle_auth_done global_id 4658 payload 274 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/2696512987 >> v1:172.21.15.181:6789/0 conn(0x7f7aec1321a0 legacy=0x7f7aec068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/2696512987 >> v1:172.21.15.153:6789/0 conn(0x7f7aec09f020 legacy=0x7f7aec069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:16.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:16.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:16.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.929+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/2696512987 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0a0200 con 0x7f7aec09ec50 2024-09-15T03:30:16.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.931+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/2696512987 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 msgr2=0x7f7aec131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:16.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.931+0000 7f7aeb7fe640 1 --2- 172.21.15.181:0/2696512987 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 0x7f7aec131c60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009880 tx=0x7f7adc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c35e925ede4003b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 1 -- 172.21.15.181:0/2696512987 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc032440 con 0x7f7aec09ec50 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: dump: 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 5 monclient: authenticate success, global_id 4658 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 1 -- 172.21.15.181:0/2696512987 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc0325a0 con 0x7f7aec09ec50 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 4 set_mon_vals no callback set 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 1 -- 172.21.15.181:0/2696512987 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc032940 con 0x7f7aec09ec50 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:16.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7aeaffd640 10 monclient: dump: 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 10 monclient: shutdown 2024-09-15T03:30:16.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 -- 172.21.15.181:0/2696512987 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 msgr2=0x7f7aec131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/2696512987 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 0x7f7aec131c60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009880 tx=0x7f7adc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 -- 172.21.15.181:0/2696512987 shutdown_connections 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/2696512987 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09ec50 0x7f7aec131c60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 -- 172.21.15.181:0/2696512987 >> 172.21.15.181:0/2696512987 conn(0x7f7aec0a5d40 msgr2=0x7f7aec0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 -- 172.21.15.181:0/2696512987 shutdown_connections 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.932+0000 7f7af1d9d640 1 -- 172.21.15.181:0/2696512987 wait complete. 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.933+0000 7f7af1d9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.933+0000 7f7af1d9d640 10 monclient: monmap: 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.933+0000 7f7af1d9d640 1 Processor -- start 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.933+0000 7f7af1d9d640 1 -- start start 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.933+0000 7f7af1d9d640 10 monclient: init 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:16.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7aec09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09f020 0x7f7aec130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7aec09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 0x7f7aec1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7aec1321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec09de10 con 0x7f7aec1321a0 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec09b440 con 0x7f7aec09ec50 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec11dc70 con 0x7f7aec09f020 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7af1d9d640 10 monclient(hunting): authenticate will time out at 821.415215s 2024-09-15T03:30:16.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43236/0 (socket says 172.21.15.181:43236) 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/3794070888 learned_addr learned my addr 172.21.15.181:0/3794070888 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 10 monclient(hunting): get_auth_request con 0x7f7aec09ec50 auth_method 0 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 0x7f7aec1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 msgr2=0x7f7aec09a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09f020 0x7f7aec130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 msgr2=0x7f7aec09a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:16.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): get_auth_request con 0x7f7aec1321a0 auth_method 0 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): get_auth_request con 0x7f7aec09f020 auth_method 0 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aeb7fe640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:16.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7ae2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient(hunting): handle_auth_done global_id 34374 payload 824 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09f020 msgr2=0x7f7aec130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09f020 0x7f7aec130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 msgr2=0x7f7aec09a960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:16.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.934+0000 7f7aebfff640 1 -- 172.21.15.181:0/3794070888 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc009390 con 0x7f7aec1321a0 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.936+0000 7f7aebfff640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 msgr2=0x7f7aec1411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.936+0000 7f7aebfff640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 0x7f7aec1411a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c7e0 tx=0x7f7ae40025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400a850 con 0x7f7aec1321a0 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: dump: 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:16.951 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 5 monclient: authenticate success, global_id 34374 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae400a9d0 con 0x7f7aec1321a0 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 10 monclient: _renew_subs 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec11df50 con 0x7f7aec1321a0 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400ad90 con 0x7f7aec1321a0 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:16.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7ae3fff640 10 monclient: dump: 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7aea7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 10 monclient: _renew_subs 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.937+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec11c750 con 0x7f7aec1321a0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7ae4012630 con 0x7f7aec1321a0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7af1d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7af1d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7af1d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aec11e190 con 0x7f7aec1321a0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7ae3fff640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7acc038790 0x7f7acc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f7ae404be90 con 0x7f7aec1321a0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7aeb7fe640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7acc038790 0x7f7acc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.938+0000 7f7aeb7fe640 10 monclient: get_auth_request con 0x7f7acc038790 auth_method 0 2024-09-15T03:30:16.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.940+0000 7f7aeb7fe640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7acc038790 0x7f7acc03ac40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7adc035040 tx=0x7f7adc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:16.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.943+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7aec11e190 con 0x7f7aec1321a0 2024-09-15T03:30:16.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.943+0000 7f7ae3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:16.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:16.943+0000 7f7ae3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:17.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.098+0000 7f7af1d9d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:17.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.098+0000 7f7af1d9d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:17.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.098+0000 7f7af1d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:17.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.098+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7aec11e190 con 0x7f7aec1321a0 2024-09-15T03:30:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.718+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v39) v1 ==== 152+0+0 (secure 0 0 0) 0x7f7aec11e190 con 0x7f7aec1321a0 2024-09-15T03:30:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.718+0000 7f7ae3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.718+0000 7f7ae3fff640 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-15T03:30:17.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.823+0000 7f7af1d9d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:17.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.823+0000 7f7af1d9d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.823+0000 7f7af1d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.823+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7aec11e190 con 0x7f7aec1321a0 2024-09-15T03:30:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.824+0000 7f7ae3fff640 1 -- 172.21.15.181:0/3794070888 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v39) v1 ==== 159+0+0 (secure 0 0 0) 0x7f7aec11e190 con 0x7f7aec1321a0 2024-09-15T03:30:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.824+0000 7f7ae3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.824+0000 7f7ae3fff640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-15T03:30:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'fooo' does not exist 2024-09-15T03:30:17.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7acc038790 msgr2=0x7f7acc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7acc038790 0x7f7acc03ac40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7adc035040 tx=0x7f7adc032000 comp rx=0 tx=0).stop 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 10 monclient: shutdown 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 msgr2=0x7f7aec1411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 0x7f7aec1411a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c7e0 tx=0x7f7ae40025b0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 shutdown_connections 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7acc038790 0x7f7acc03ac40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7aec1321a0 0x7f7aec1411a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7aec09f020 0x7f7aec130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 --2- 172.21.15.181:0/3794070888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7aec09ec50 0x7f7aec09a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 >> 172.21.15.181:0/3794070888 conn(0x7f7aec0a5d40 msgr2=0x7f7aec09e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:17.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 shutdown_connections 2024-09-15T03:30:17.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.826+0000 7f7af1d9d640 1 -- 172.21.15.181:0/3794070888 wait complete. 2024-09-15T03:30:17.836 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.910+0000 7fdb47c68640 10 monclient: get_monmap_and_config 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.910+0000 7fdb47c68640 10 monclient: build_initial_monmap 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.911+0000 7fdb47c68640 10 monclient: monmap: 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:17.911776+0000 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:17.911776+0000 2024-09-15T03:30:17.912 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:17.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.911+0000 7fdb47c68640 1 Processor -- start 2024-09-15T03:30:17.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.911+0000 7fdb47c68640 1 -- start start 2024-09-15T03:30:17.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.911+0000 7fdb47c68640 10 monclient: init 2024-09-15T03:30:17.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:17.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T03:30:17.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): picked mon.noname-b con 0x7fdb40098c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 0x7fdb4009dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): picked mon.noname-c con 0x7fdb40099580 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb40099c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4009f680 con 0x7fdb40099c70 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb4009f420 con 0x7fdb40099580 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40068a20 con 0x7fdb40098c70 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb47c68640 10 monclient(hunting): authenticate will time out at 822.393227s 2024-09-15T03:30:17.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb46205640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fdb40099c70 0x7fdb4009e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:51108/0 (socket says 172.21.15.181:51108) 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb46205640 1 -- 172.21.15.181:0/1783676564 learned_addr learned my addr 172.21.15.181:0/1783676564 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 1 --2- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 0x7fdb4009dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 10 monclient(hunting): get_auth_request con 0x7fdb40099580 auth_method 0 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 1 -- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 msgr2=0x7fdb4009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 1 --2- 172.21.15.181:0/1783676564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 1 -- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 msgr2=0x7fdb4009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 1 --2- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 0x7fdb4009dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:17.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45203640 1 --2- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 0x7fdb4009dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): get_auth_request con 0x7fdb40098c70 auth_method 0 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient(hunting): handle_auth_done global_id 4664 payload 274 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 1 -- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 msgr2=0x7fdb4009dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 1 --2- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 0x7fdb4009dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 1 -- 172.21.15.181:0/1783676564 >> v1:172.21.15.181:6789/0 conn(0x7fdb40099c70 legacy=0x7fdb4009e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient: _finish_hunting 0 2024-09-15T03:30:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 1 monclient: found mon.noname-b 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.912+0000 7fdb45a04640 1 -- 172.21.15.181:0/1783676564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb40096a30 con 0x7fdb40098c70 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.915+0000 7fdb45a04640 1 -- 172.21.15.181:0/1783676564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 msgr2=0x7fdb40099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.915+0000 7fdb45a04640 1 --2- 172.21.15.181:0/1783676564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40099040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009850 tx=0x7fdb380090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d3ebff5705f734c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 1 -- 172.21.15.181:0/1783676564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3802adf0 con 0x7fdb40098c70 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: dump: 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: _finish_auth 0 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 1 -- 172.21.15.181:0/1783676564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38032440 con 0x7fdb40098c70 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 5 monclient: authenticate success, global_id 4664 2024-09-15T03:30:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 4 set_mon_vals no callback set 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 1 -- 172.21.15.181:0/1783676564 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380327e0 con 0x7fdb40098c70 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb44a02640 10 monclient: dump: 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:17.924 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 10 monclient: shutdown 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 -- 172.21.15.181:0/1783676564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 msgr2=0x7fdb40099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 --2- 172.21.15.181:0/1783676564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40099040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009850 tx=0x7fdb380090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 -- 172.21.15.181:0/1783676564 shutdown_connections 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 --2- 172.21.15.181:0/1783676564 >> v2:172.21.15.181:3300/0 conn(0x7fdb40099580 0x7fdb4009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 --2- 172.21.15.181:0/1783676564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40099040 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 -- 172.21.15.181:0/1783676564 >> 172.21.15.181:0/1783676564 conn(0x7fdb40070e90 msgr2=0x7fdb400a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 -- 172.21.15.181:0/1783676564 shutdown_connections 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.916+0000 7fdb47c68640 1 -- 172.21.15.181:0/1783676564 wait complete. 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.917+0000 7fdb47c68640 10 monclient: build_initial_monmap 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.917+0000 7fdb47c68640 10 monclient: monmap: 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:17.926 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.917+0000 7fdb47c68640 1 Processor -- start 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.917+0000 7fdb47c68640 1 -- start start 2024-09-15T03:30:17.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.917+0000 7fdb47c68640 10 monclient: init 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40133ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): picked mon.noname-c con 0x7fdb40098c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 0x7fdb40141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): picked mon.noname-a con 0x7fdb40099580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): picked mon.noname-b con 0x7fdb40099c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:17.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4011fe00 con 0x7fdb40099580 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401222e0 con 0x7fdb40099c70 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40122000 con 0x7fdb40098c70 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb47c68640 10 monclient(hunting): authenticate will time out at 822.399232s 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43244/0 (socket says 172.21.15.181:43244) 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 -- 172.21.15.181:0/3315097463 learned_addr learned my addr 172.21.15.181:0/3315097463 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 10 monclient(hunting): get_auth_request con 0x7fdb40099c70 auth_method 0 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:17.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 msgr2=0x7fdb40141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 msgr2=0x7fdb40141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb46205640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 0x7fdb40141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): get_auth_request con 0x7fdb40099580 auth_method 0 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40133ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): get_auth_request con 0x7fdb40098c70 auth_method 0 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient(hunting): handle_auth_done global_id 34377 payload 824 2024-09-15T03:30:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 msgr2=0x7fdb40133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40133ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 msgr2=0x7fdb40141540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient: _finish_hunting 0 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 monclient: found mon.noname-a 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45203640 1 -- 172.21.15.181:0/3315097463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb38009390 con 0x7fdb40099580 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.918+0000 7fdb45a04640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40133ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb45203640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 msgr2=0x7fdb40141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb45203640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 0x7fdb40141000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdb30006ea0 tx=0x7fdb30008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000b8d0 con 0x7fdb40099580 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: dump: 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3000ba30 con 0x7fdb40099580 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000bdf0 con 0x7fdb40099580 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb35ffb640 10 monclient: dump: 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:17.938 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 5 monclient: authenticate success, global_id 34377 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 10 monclient: _renew_subs 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:17.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb37fff640 4 set_mon_vals no callback set 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb40121130 con 0x7fdb40099580 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 10 monclient: _renew_subs 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.921+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb401345b0 con 0x7fdb40099580 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdb30014640 con 0x7fdb40099580 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb47c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb35ffb640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdb24038740 0x7fdb2403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb47c68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb47c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4776+0+0 (secure 0 0 0) 0x7fdb3004bab0 con 0x7fdb40099580 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb40121350 con 0x7fdb40099580 2024-09-15T03:30:17.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb45a04640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdb24038740 0x7fdb2403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:17.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.922+0000 7fdb45a04640 10 monclient: get_auth_request con 0x7fdb24038740 auth_method 0 2024-09-15T03:30:17.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.924+0000 7fdb45a04640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdb24038740 0x7fdb2403abf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdb4006abc0 tx=0x7fdb38032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:17.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.927+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdb30011d80 con 0x7fdb40099580 2024-09-15T03:30:17.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.927+0000 7fdb35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:17.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:17.927+0000 7fdb35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:18.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:18.104+0000 7fdb47c68640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:18.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:18.104+0000 7fdb47c68640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:18.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:18.104+0000 7fdb47c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:18.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:18.104+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb400008d0 con 0x7fdb40099580 2024-09-15T03:30:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.168+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v40) v1 ==== 149+0+0 (secure 0 0 0) 0x7fdb300117b0 con 0x7fdb40099580 2024-09-15T03:30:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.168+0000 7fdb35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.168+0000 7fdb35ffb640 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-15T03:30:19.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:19.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.318+0000 7fdb47c68640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.318+0000 7fdb47c68640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.318+0000 7fdb47c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:19.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.318+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb401224a0 con 0x7fdb40099580 2024-09-15T03:30:19.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.319+0000 7fdb35ffb640 1 -- 172.21.15.181:0/3315097463 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (secure 0 0 0) 0x7fdb30011990 con 0x7fdb40099580 2024-09-15T03:30:19.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.319+0000 7fdb35ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.319+0000 7fdb35ffb640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-15T03:30:19.320 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' does not exist 2024-09-15T03:30:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdb24038740 msgr2=0x7fdb2403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdb24038740 0x7fdb2403abf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdb4006abc0 tx=0x7fdb38032000 comp rx=0 tx=0).stop 2024-09-15T03:30:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 10 monclient: shutdown 2024-09-15T03:30:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 msgr2=0x7fdb40141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 0x7fdb40141000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdb30006ea0 tx=0x7fdb30008040 comp rx=0 tx=0).stop 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 shutdown_connections 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdb24038740 0x7fdb2403abf0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb40099c70 0x7fdb40141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb40099580 0x7fdb40141000 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 --2- 172.21.15.181:0/3315097463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb40098c70 0x7fdb40133ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 >> 172.21.15.181:0/3315097463 conn(0x7fdb40070e90 msgr2=0x7fdb4006d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 shutdown_connections 2024-09-15T03:30:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.321+0000 7fdb47c68640 1 -- 172.21.15.181:0/3315097463 wait complete. 2024-09-15T03:30:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T03:30:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.337+0000 7f755f6d4640 10 monclient: get_monmap_and_config 2024-09-15T03:30:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.337+0000 7f755f6d4640 10 monclient: build_initial_monmap 2024-09-15T03:30:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.337+0000 7f755f6d4640 10 monclient: monmap: 2024-09-15T03:30:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:19.338219+0000 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:19.338219+0000 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:19.339 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 Processor -- start 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 -- start start 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient: init 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 0x7f755809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f755809c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f755809aba0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f755809af70 0x7f755809df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f755809af70 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755806a320 con 0x7f755809aba0 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755809a9d0 con 0x7f755809af70 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7558068a20 con 0x7f755809c860 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755f6d4640 10 monclient(hunting): authenticate will time out at 816.977734s 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 0x7f755809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f755809af70 0x7f755809df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755cc6f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f755809aba0 0x7f755809d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44844/0 (socket says 172.21.15.153:44844) 2024-09-15T03:30:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 0x7f755809cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52338/0 (socket says 172.21.15.153:52338) 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755cc6f640 1 -- 172.21.15.153:0/4091654815 learned_addr learned my addr 172.21.15.153:0/4091654815 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f755809af70 0x7f755809df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35840/0 (socket says 172.21.15.153:35840) 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): get_auth_request con 0x7f755809af70 auth_method 0 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 10 monclient(hunting): get_auth_request con 0x7f755809c860 auth_method 0 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755dc71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.338+0000 7f755d470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755d470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755d470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 10 monclient(hunting): handle_auth_done global_id 34380 payload 274 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 1 -- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 msgr2=0x7f755809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 1 --2- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 0x7f755809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 1 -- 172.21.15.153:0/4091654815 >> v1:172.21.15.153:6789/0 conn(0x7f755809aba0 legacy=0x7f755809d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 10 monclient: _finish_hunting 0 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 1 monclient: found mon.noname-a 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755dc71640 1 -- 172.21.15.153:0/4091654815 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558068e60 con 0x7f755809af70 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.339+0000 7f755d470640 1 --2- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 0x7f755809cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.341+0000 7f755dc71640 1 -- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809af70 msgr2=0x7f755809df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.341+0000 7f755dc71640 1 --2- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809af70 0x7f755809df40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7550009e60 tx=0x7f7550009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2518357e1c601d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 1 -- 172.21.15.153:0/4091654815 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550007760 con 0x7f755809af70 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: dump: 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: _finish_auth 0 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 1 -- 172.21.15.153:0/4091654815 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755002d050 con 0x7f755809af70 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 5 monclient: authenticate success, global_id 34380 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 1 -- 172.21.15.153:0/4091654815 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550007af0 con 0x7f755809af70 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 4 set_mon_vals no callback set 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f754ffff640 10 monclient: dump: 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.350 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 10 monclient: shutdown 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 -- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809af70 msgr2=0x7f755809df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 --2- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809af70 0x7f755809df40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7550009e60 tx=0x7f7550009780 comp rx=0 tx=0).stop 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 -- 172.21.15.153:0/4091654815 shutdown_connections 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 --2- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809af70 0x7f755809df40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 --2- 172.21.15.153:0/4091654815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809c860 0x7f755809cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 -- 172.21.15.153:0/4091654815 >> 172.21.15.153:0/4091654815 conn(0x7f75580a2e40 msgr2=0x7f75580a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 -- 172.21.15.153:0/4091654815 shutdown_connections 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.342+0000 7f755f6d4640 1 -- 172.21.15.153:0/4091654815 wait complete. 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 10 monclient: build_initial_monmap 2024-09-15T03:30:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 10 monclient: monmap: 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 1 Processor -- start 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 1 -- start start 2024-09-15T03:30:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 10 monclient: init 2024-09-15T03:30:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.343+0000 7f755f6d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f755809aba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 0x7f7558132070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f755809af70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809c860 0x7f755806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f755809c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75580a1200 con 0x7f755809c860 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755811f210 con 0x7f755809af70 2024-09-15T03:30:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755811ef30 con 0x7f755809aba0 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755f6d4640 10 monclient(hunting): authenticate will time out at 816.983066s 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809c860 0x7f755806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52354/0 (socket says 172.21.15.153:52354) 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 -- 172.21.15.153:0/3507804200 learned_addr learned my addr 172.21.15.153:0/3507804200 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): get_auth_request con 0x7f755809aba0 auth_method 0 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): get_auth_request con 0x7f755809c860 auth_method 0 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 0x7f7558132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 10 monclient(hunting): get_auth_request con 0x7f755809af70 auth_method 0 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient(hunting): handle_auth_done global_id 4673 payload 824 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 msgr2=0x7f7558132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 msgr2=0x7f7558132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 msgr2=0x7f7558132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 0x7f7558132070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755cc6f640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 0x7f7558132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 0x7f7558132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809c860 msgr2=0x7f755806d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809c860 0x7f755806d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient: _finish_hunting 0 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 monclient: found mon.noname-c 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755d470640 1 -- 172.21.15.153:0/3507804200 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554009390 con 0x7f755809aba0 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.344+0000 7f755dc71640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809c860 0x7f755806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755d470640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 msgr2=0x7f7558131b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755d470640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f755400ec80 tx=0x7f75540090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755400eec0 con 0x7f755809aba0 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: dump: 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.361 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755400cdd0 con 0x7f755809aba0 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 5 monclient: authenticate success, global_id 4673 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554013680 con 0x7f755809aba0 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 10 monclient: _renew_subs 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7550009a20 con 0x7f755809aba0 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754d7fa640 10 monclient: dump: 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f754f7fe640 4 set_mon_vals no callback set 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 10 monclient: _renew_subs 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.346+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7558141b20 con 0x7f755809aba0 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f755d470640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f755400ec80 tx=0x7f75540090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f755404bee0 osd_map(40..40 src has 1..40) v4 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f75540139f0 con 0x7f755809aba0 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f755f6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f755f6d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f755f6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755804bf60 con 0x7f755809aba0 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.347+0000 7f754d7fa640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f753c038740 0x7f753c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.348+0000 7f755cc6f640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f753c038740 0x7f753c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.348+0000 7f755cc6f640 10 monclient: get_auth_request con 0x7f753c038740 auth_method 0 2024-09-15T03:30:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.350+0000 7f755cc6f640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f753c038740 0x7f753c03abf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7548007d10 tx=0x7f7548009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:19.352+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f755404bee0 con 0x7f755809aba0 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.405+0000 7fd67419d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.405+0000 7fd67419d640 10 monclient: build_initial_monmap 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.405+0000 7fd67419d640 10 monclient: monmap: 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:19.405991+0000 2024-09-15T03:30:19.406 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:19.405991+0000 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:19.407 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:19.408 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.405+0000 7fd67419d640 1 Processor -- start 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.405+0000 7fd67419d640 1 -- start start 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.405+0000 7fd67419d640 10 monclient: init 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd66c09ec50 0x7fd66c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): picked mon.noname-a con 0x7fd66c09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 0x7fd66c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): picked mon.noname-c con 0x7fd66c1321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:19.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd66c09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c067420 con 0x7fd66c09f020 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c09fdc0 con 0x7fd66c09ec50 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0a0510 con 0x7fd66c1321a0 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67419d640 10 monclient(hunting): authenticate will time out at 823.887444s 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 0x7fd66c068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67273a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd66c09f020 0x7fd66c069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39600/0 (socket says 172.21.15.181:39600) 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd67273a640 1 -- 172.21.15.181:0/1388192567 learned_addr learned my addr 172.21.15.181:0/1388192567 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 10 monclient(hunting): get_auth_request con 0x7fd66c1321a0 auth_method 0 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 1 --2- 172.21.15.181:0/1388192567 >> v2:172.21.15.153:3300/0 conn(0x7fd66c09ec50 0x7fd66c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): get_auth_request con 0x7fd66c09ec50 auth_method 0 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 1 -- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 msgr2=0x7fd66c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 1 -- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 msgr2=0x7fd66c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 1 --2- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 0x7fd66c068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671738640 1 --2- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 0x7fd66c068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient(hunting): handle_auth_done global_id 34386 payload 274 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 1 -- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 msgr2=0x7fd66c068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 1 --2- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 0x7fd66c068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 1 -- 172.21.15.181:0/1388192567 >> v1:172.21.15.181:6789/0 conn(0x7fd66c09f020 legacy=0x7fd66c069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient: _finish_hunting 0 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 1 monclient: found mon.noname-a 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.406+0000 7fd671f39640 1 -- 172.21.15.181:0/1388192567 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0a0200 con 0x7fd66c09ec50 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd671f39640 1 -- 172.21.15.181:0/1388192567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 msgr2=0x7fd66c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd671f39640 1 --2- 172.21.15.181:0/1388192567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c131c60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0a2b00 tx=0x7fd6640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b28a440c5cd2bb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 1 -- 172.21.15.181:0/1388192567 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66402ae00 con 0x7fd66c09ec50 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: dump: 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: _finish_auth 0 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 1 -- 172.21.15.181:0/1388192567 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664032440 con 0x7fd66c09ec50 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd67419d640 5 monclient: authenticate success, global_id 34386 2024-09-15T03:30:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd67419d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd67419d640 4 set_mon_vals no callback set 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 1 -- 172.21.15.181:0/1388192567 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664032820 con 0x7fd66c09ec50 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.409+0000 7fd670f37640 10 monclient: dump: 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 10 monclient: shutdown 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 -- 172.21.15.181:0/1388192567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 msgr2=0x7fd66c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 --2- 172.21.15.181:0/1388192567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c131c60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0a2b00 tx=0x7fd6640090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 -- 172.21.15.181:0/1388192567 shutdown_connections 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 --2- 172.21.15.181:0/1388192567 >> v2:172.21.15.181:3300/0 conn(0x7fd66c1321a0 0x7fd66c068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 --2- 172.21.15.181:0/1388192567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c131c60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 -- 172.21.15.181:0/1388192567 >> 172.21.15.181:0/1388192567 conn(0x7fd66c0a5d40 msgr2=0x7fd66c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 -- 172.21.15.181:0/1388192567 shutdown_connections 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.410+0000 7fd67419d640 1 -- 172.21.15.181:0/1388192567 wait complete. 2024-09-15T03:30:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient: build_initial_monmap 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient: monmap: 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 Processor -- start 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 -- start start 2024-09-15T03:30:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient: init 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): picked mon.noname-a con 0x7fd66c09ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 0x7fd66c130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): picked mon.noname-c con 0x7fd66c09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): picked mon.noname-b con 0x7fd66c1321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c09dd10 con 0x7fd66c09ec50 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c11ce10 con 0x7fd66c1321a0 2024-09-15T03:30:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c11dd70 con 0x7fd66c09f020 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.411+0000 7fd67419d640 10 monclient(hunting): authenticate will time out at 823.892926s 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46580/0 (socket says 172.21.15.181:46580) 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 -- 172.21.15.181:0/3414433039 learned_addr learned my addr 172.21.15.181:0/3414433039 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 10 monclient(hunting): get_auth_request con 0x7fd66c1321a0 auth_method 0 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 0x7fd66c130c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): get_auth_request con 0x7fd66c09ec50 auth_method 0 2024-09-15T03:30:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): get_auth_request con 0x7fd66c09f020 auth_method 0 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 msgr2=0x7fd66c1411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 msgr2=0x7fd66c1411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd67273a640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient(hunting): handle_auth_done global_id 4676 payload 824 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 msgr2=0x7fd66c1411b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 msgr2=0x7fd66c09ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c09ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient: _finish_hunting 0 2024-09-15T03:30:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 monclient: found mon.noname-c 2024-09-15T03:30:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671738640 1 -- 172.21.15.181:0/3414433039 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd664009390 con 0x7fd66c09f020 2024-09-15T03:30:19.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.412+0000 7fd671f39640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c09ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd671738640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 msgr2=0x7fd66c130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd671738640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 0x7fd66c130c50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd660007fb0 tx=0x7fd6600098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66000ee30 con 0x7fd66c09f020 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: dump: 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd67419d640 5 monclient: authenticate success, global_id 4676 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd660013590 con 0x7fd66c09f020 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd67419d640 10 monclient: _renew_subs 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd67419d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c11e050 con 0x7fd66c09f020 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660013950 con 0x7fd66c09f020 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd669ffb640 10 monclient: dump: 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 10 monclient: _renew_subs 2024-09-15T03:30:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c11bc30 con 0x7fd66c09f020 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.414+0000 7fd66bfff640 4 set_mon_vals no callback set 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd660013d40 con 0x7fd66c09f020 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd669ffb640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd658038740 0x7fd65803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fd66004c050 con 0x7fd66c09f020 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c0008d0 con 0x7fd66c09f020 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.415+0000 7fd671f39640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd658038740 0x7fd65803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.416+0000 7fd671f39640 10 monclient: get_auth_request con 0x7fd658038740 auth_method 0 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.418+0000 7fd671f39640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd658038740 0x7fd65803abf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd664034000 tx=0x7fd664032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.420+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd660014420 con 0x7fd66c09f020 2024-09-15T03:30:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.420+0000 7fd669ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.420+0000 7fd669ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:19.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.574+0000 7fd67419d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.574+0000 7fd67419d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.574+0000 7fd67419d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.574+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd66c0a13e0 con 0x7fd66c09f020 2024-09-15T03:30:19.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.575+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (secure 0 0 0) 0x7fd66001b310 con 0x7fd66c09f020 2024-09-15T03:30:19.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.575+0000 7fd669ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.575+0000 7fd669ffb640 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-15T03:30:19.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.666+0000 7fd67419d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.666+0000 7fd67419d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.666+0000 7fd67419d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.666+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd66c068fc0 con 0x7fd66c09f020 2024-09-15T03:30:19.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.668+0000 7fd669ffb640 1 -- 172.21.15.181:0/3414433039 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (secure 0 0 0) 0x7fd6600122e0 con 0x7fd66c09f020 2024-09-15T03:30:19.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.668+0000 7fd669ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.668+0000 7fd669ffb640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-15T03:30:19.669 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' does not exist 2024-09-15T03:30:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd658038740 msgr2=0x7fd65803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd658038740 0x7fd65803abf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd664034000 tx=0x7fd664032000 comp rx=0 tx=0).stop 2024-09-15T03:30:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 10 monclient: shutdown 2024-09-15T03:30:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 msgr2=0x7fd66c130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 0x7fd66c130c50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd660007fb0 tx=0x7fd6600098b0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 shutdown_connections 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd658038740 0x7fd65803abf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c1321a0 0x7fd66c1411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd66c09f020 0x7fd66c130c50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 --2- 172.21.15.181:0/3414433039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66c09ec50 0x7fd66c09ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 >> 172.21.15.181:0/3414433039 conn(0x7fd66c0a5d40 msgr2=0x7fd66c09df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 shutdown_connections 2024-09-15T03:30:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.670+0000 7fd67419d640 1 -- 172.21.15.181:0/3414433039 wait complete. 2024-09-15T03:30:19.680 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.754+0000 7fb4a7b1d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.754+0000 7fb4a7b1d640 10 monclient: build_initial_monmap 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.754+0000 7fb4a7b1d640 10 monclient: monmap: 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:19.754883+0000 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:19.754883+0000 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.755 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:19.756 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:19.756 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:19.756 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:19.756 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:19.756 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.758 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.754+0000 7fb4a7b1d640 1 Processor -- start 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.754+0000 7fb4a7b1d640 1 -- start start 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.754+0000 7fb4a7b1d640 10 monclient: init 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4a009ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 0x7fb4a0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 0x7fb4a0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a0136d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a0067420 con 0x7fb4a009ec50 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a009fdc0 con 0x7fb4a0136d60 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00a0510 con 0x7fb4a009f020 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a7b1d640 10 monclient(hunting): authenticate will time out at 824.236515s 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 0x7fb4a0139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a58b9640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fb4a009ec50 0x7fb4a0131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39616/0 (socket says 172.21.15.181:39616) 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a58b9640 1 -- 172.21.15.181:0/2998756122 learned_addr learned my addr 172.21.15.181:0/2998756122 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 10 monclient(hunting): get_auth_request con 0x7fb4a0136d60 auth_method 0 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2998756122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 0x7fb4a0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): get_auth_request con 0x7fb4a009f020 auth_method 0 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 1 -- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 msgr2=0x7fb4a0139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 1 -- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 msgr2=0x7fb4a0139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 1 --2- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 0x7fb4a0139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a60ba640 1 --2- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 0x7fb4a0139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_done global_id 4679 payload 274 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 msgr2=0x7fb4a0139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 0x7fb4a0139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2998756122 >> v1:172.21.15.181:6789/0 conn(0x7fb4a009ec50 legacy=0x7fb4a0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient: _finish_hunting 0 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 1 monclient: found mon.noname-b 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.755+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2998756122 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00a0200 con 0x7fb4a009f020 2024-09-15T03:30:19.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2998756122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 msgr2=0x7fb4a0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:19.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2998756122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 0x7fb4a0069d10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb490009850 tx=0x7fb4900090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbfba6dd9d25aac8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a48b7640 1 -- 172.21.15.181:0/2998756122 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49002adf0 con 0x7fb4a009f020 2024-09-15T03:30:19.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a48b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a48b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a48b7640 10 monclient: dump: 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a48b7640 10 monclient: _finish_auth 0 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a48b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a7b1d640 5 monclient: authenticate success, global_id 4679 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.758+0000 7fb4a7b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a48b7640 1 -- 172.21.15.181:0/2998756122 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490032440 con 0x7fb4a009f020 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a48b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a48b7640 1 -- 172.21.15.181:0/2998756122 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4900327e0 con 0x7fb4a009f020 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a48b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a48b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a48b7640 10 monclient: dump: 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 4 set_mon_vals no callback set 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 10 monclient: shutdown 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2998756122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 msgr2=0x7fb4a0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2998756122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 0x7fb4a0069d10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb490009850 tx=0x7fb4900090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2998756122 shutdown_connections 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2998756122 >> v2:172.21.15.181:3300/0 conn(0x7fb4a0136d60 0x7fb4a0139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2998756122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009f020 0x7fb4a0069d10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2998756122 >> 172.21.15.181:0/2998756122 conn(0x7fb4a00a5d40 msgr2=0x7fb4a00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2998756122 shutdown_connections 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.759+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2998756122 wait complete. 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient: build_initial_monmap 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient: monmap: 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.772 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 Processor -- start 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 -- start start 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient: init 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 0x7fb4a00975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a0136d60 0x7fb4a0098050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a0136d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00994b0 con 0x7fb4a0136d60 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a009c000 con 0x7fb4a009f020 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a009d960 con 0x7fb4a009ec50 2024-09-15T03:30:19.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:19.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.760+0000 7fb4a7b1d640 10 monclient(hunting): authenticate will time out at 824.241998s 2024-09-15T03:30:19.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 0x7fb4a00975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46588/0 (socket says 172.21.15.181:46588) 2024-09-15T03:30:19.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2825648992 learned_addr learned my addr 172.21.15.181:0/2825648992 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:19.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 10 monclient(hunting): get_auth_request con 0x7fb4a009f020 auth_method 0 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a0136d60 0x7fb4a0098050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): get_auth_request con 0x7fb4a0136d60 auth_method 0 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): get_auth_request con 0x7fb4a009ec50 auth_method 0 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 msgr2=0x7fb4a0097b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 msgr2=0x7fb4a0097b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:19.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient(hunting): handle_auth_done global_id 4682 payload 824 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 msgr2=0x7fb4a0097b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a0136d60 msgr2=0x7fb4a0098050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a0136d60 0x7fb4a0098050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:19.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient: _finish_hunting 0 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 monclient: found mon.noname-c 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a58b9640 1 -- 172.21.15.181:0/2825648992 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb490009390 con 0x7fb4a009ec50 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.761+0000 7fb4a60ba640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a0136d60 0x7fb4a0098050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb4a58b9640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 msgr2=0x7fb4a00975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb4a58b9640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 0x7fb4a00975d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00c4c0 tx=0x7fb49c007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c005cb0 con 0x7fb4a009ec50 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: dump: 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.780 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb4a7b1d640 5 monclient: authenticate success, global_id 4682 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c005e30 con 0x7fb4a009ec50 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb4a7b1d640 10 monclient: _renew_subs 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a0131800 con 0x7fb4a009ec50 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c007320 con 0x7fb4a009ec50 2024-09-15T03:30:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb495ffb640 10 monclient: dump: 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:19.782 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.763+0000 7fb497fff640 4 set_mon_vals no callback set 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.764+0000 7fb4a7b1d640 10 monclient: _renew_subs 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.764+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.764+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a0098f30 con 0x7fb4a009ec50 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.764+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb49c01b3b0 con 0x7fb4a009ec50 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.764+0000 7fb495ffb640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb484038740 0x7fb48403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fb49c04ae90 con 0x7fb4a009ec50 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb484038740 0x7fb48403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:19.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb4a7b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb4a7b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a009a9f0 con 0x7fb4a009ec50 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.765+0000 7fb4a50b8640 10 monclient: get_auth_request con 0x7fb484038740 auth_method 0 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.767+0000 7fb4a50b8640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb484038740 0x7fb48403abf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb490000e10 tx=0x7fb490032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.769+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb49c017020 con 0x7fb4a009ec50 2024-09-15T03:30:19.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.769+0000 7fb495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:19.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.769+0000 7fb495ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.925+0000 7fb4a7b1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.925+0000 7fb4a7b1d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.925+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.925+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4a00008d0 con 0x7fb4a009ec50 2024-09-15T03:30:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.927+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7fb49c018680 con 0x7fb4a009ec50 2024-09-15T03:30:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.927+0000 7fb495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:19.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:19.927+0000 7fb495ffb640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-15T03:30:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.019+0000 7fb4a7b1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.019+0000 7fb4a7b1d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.019+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.019+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4a0136af0 con 0x7fb4a009ec50 2024-09-15T03:30:20.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.020+0000 7fb495ffb640 1 -- 172.21.15.181:0/2825648992 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7fb49c00e290 con 0x7fb4a009ec50 2024-09-15T03:30:20.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.020+0000 7fb495ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.020+0000 7fb495ffb640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-15T03:30:20.022 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'fooo' does not exist 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb484038740 msgr2=0x7fb48403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb484038740 0x7fb48403abf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb490000e10 tx=0x7fb490032000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 10 monclient: shutdown 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 msgr2=0x7fb4a00975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 0x7fb4a00975d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00c4c0 tx=0x7fb49c007f00 comp rx=0 tx=0).stop 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 shutdown_connections 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb484038740 0x7fb48403abf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a0136d60 0x7fb4a0098050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a009f020 0x7fb4a0097b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 --2- 172.21.15.181:0/2825648992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4a009ec50 0x7fb4a00975d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 >> 172.21.15.181:0/2825648992 conn(0x7fb4a00a5d40 msgr2=0x7fb4a00a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 shutdown_connections 2024-09-15T03:30:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.022+0000 7fb4a7b1d640 1 -- 172.21.15.181:0/2825648992 wait complete. 2024-09-15T03:30:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-15T03:30:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.107+0000 7feacf06f640 10 monclient: get_monmap_and_config 2024-09-15T03:30:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.107+0000 7feacf06f640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.107+0000 7feacf06f640 10 monclient: monmap: 2024-09-15T03:30:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:20.108065+0000 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:20.108065+0000 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:20.109 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.107+0000 7feacf06f640 1 Processor -- start 2024-09-15T03:30:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 1 -- start start 2024-09-15T03:30:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient: init 2024-09-15T03:30:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feac809ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feac809f020 0x7feac8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): picked mon.noname-a con 0x7feac809f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): picked mon.noname-b con 0x7feac8136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac8067420 con 0x7feac809ec50 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac809fdc0 con 0x7feac809f020 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac80a0510 con 0x7feac8136d60 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacf06f640 10 monclient(hunting): authenticate will time out at 824.589540s 2024-09-15T03:30:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feac809f020 0x7feac8069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59806/0 (socket says 172.21.15.181:59806) 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 1 -- 172.21.15.181:0/3191253837 learned_addr learned my addr 172.21.15.181:0/3191253837 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacce0b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7feac809ec50 0x7feac8131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:34544/0 (socket says 172.21.15.181:34544) 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): get_auth_request con 0x7feac8136d60 auth_method 0 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): get_auth_request con 0x7feac809f020 auth_method 0 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feacd60c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.108+0000 7feac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 10 monclient(hunting): handle_auth_done global_id 4685 payload 274 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 1 -- 172.21.15.181:0/3191253837 >> v2:172.21.15.153:3300/0 conn(0x7feac809f020 msgr2=0x7feac8069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 1 --2- 172.21.15.181:0/3191253837 >> v2:172.21.15.153:3300/0 conn(0x7feac809f020 0x7feac8069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 1 -- 172.21.15.181:0/3191253837 >> v1:172.21.15.153:6789/0 conn(0x7feac809ec50 legacy=0x7feac8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 1 monclient: found mon.noname-b 2024-09-15T03:30:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.109+0000 7feacd60c640 1 -- 172.21.15.181:0/3191253837 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac80a0200 con 0x7feac8136d60 2024-09-15T03:30:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.111+0000 7feacd60c640 1 -- 172.21.15.181:0/3191253837 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 msgr2=0x7feac8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.111+0000 7feacd60c640 1 --2- 172.21.15.181:0/3191253837 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8139140 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feac0009f20 tx=0x7feac0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b90ed93097716488 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 1 -- 172.21.15.181:0/3191253837 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac0007610 con 0x7feac8136d60 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: dump: 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 1 -- 172.21.15.181:0/3191253837 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac002d050 con 0x7feac8136d60 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 5 monclient: authenticate success, global_id 4685 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 1 -- 172.21.15.181:0/3191253837 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00079a0 con 0x7feac8136d60 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 4 set_mon_vals no callback set 2024-09-15T03:30:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feac77fe640 10 monclient: dump: 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 10 monclient: shutdown 2024-09-15T03:30:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 -- 172.21.15.181:0/3191253837 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 msgr2=0x7feac8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 --2- 172.21.15.181:0/3191253837 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8139140 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feac0009f20 tx=0x7feac0009840 comp rx=0 tx=0).stop 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 -- 172.21.15.181:0/3191253837 shutdown_connections 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 --2- 172.21.15.181:0/3191253837 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8139140 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 --2- 172.21.15.181:0/3191253837 >> v2:172.21.15.153:3300/0 conn(0x7feac809f020 0x7feac8069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 -- 172.21.15.181:0/3191253837 >> 172.21.15.181:0/3191253837 conn(0x7feac80a5d40 msgr2=0x7feac80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 -- 172.21.15.181:0/3191253837 shutdown_connections 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.112+0000 7feacf06f640 1 -- 172.21.15.181:0/3191253837 wait complete. 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.113+0000 7feacf06f640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.113+0000 7feacf06f640 10 monclient: monmap: 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:20.125 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.113+0000 7feacf06f640 1 Processor -- start 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.113+0000 7feacf06f640 1 -- start start 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.113+0000 7feacf06f640 10 monclient: init 2024-09-15T03:30:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 0x7feac8131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): picked mon.noname-a con 0x7feac809ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): picked mon.noname-b con 0x7feac809f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8097530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): picked mon.noname-c con 0x7feac8136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac809c000 con 0x7feac809ec50 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac80a4ce0 con 0x7feac809f020 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac8098c50 con 0x7feac8136d60 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacf06f640 10 monclient(hunting): authenticate will time out at 824.595208s 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46598/0 (socket says 172.21.15.181:46598) 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 -- 172.21.15.181:0/860845686 learned_addr learned my addr 172.21.15.181:0/860845686 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 10 monclient(hunting): get_auth_request con 0x7feac809f020 auth_method 0 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8097530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 0x7feac8131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): get_auth_request con 0x7feac8136d60 auth_method 0 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): get_auth_request con 0x7feac809ec50 auth_method 0 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 msgr2=0x7feac8096ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 msgr2=0x7feac8096ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feac7fff640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacd60c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient(hunting): handle_auth_done global_id 34398 payload 824 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 msgr2=0x7feac8097530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8097530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 msgr2=0x7feac8096ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 monclient: found mon.noname-a 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.114+0000 7feacce0b640 1 -- 172.21.15.181:0/860845686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac0009ae0 con 0x7feac809ec50 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.116+0000 7feacce0b640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 msgr2=0x7feac8131410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacce0b640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 0x7feac8131410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c490 tx=0x7feabc007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0074f0 con 0x7feac809ec50 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: dump: 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc007670 con 0x7feac809ec50 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 5 monclient: authenticate success, global_id 34398 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 10 monclient: _renew_subs 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac80984c0 con 0x7feac809ec50 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc005a20 con 0x7feac809ec50 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac4ff9640 10 monclient: dump: 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feac6ffd640 4 set_mon_vals no callback set 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 10 monclient: _renew_subs 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.117+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac809b4b0 con 0x7feac809ec50 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feacf06f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feacf06f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7feabc01b3b0 con 0x7feac809ec50 2024-09-15T03:30:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feacf06f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac80008d0 con 0x7feac809ec50 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feac4ff9640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feaac038770 0x7feaac03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7feabc04ae70 con 0x7feac809ec50 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feac7fff640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feaac038770 0x7feaac03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.118+0000 7feac7fff640 10 monclient: get_auth_request con 0x7feaac038770 auth_method 0 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.121+0000 7feac7fff640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feaac038770 0x7feaac03ac20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7feac8098850 tx=0x7feab4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.123+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7feabc0187f0 con 0x7feac809ec50 2024-09-15T03:30:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.123+0000 7feac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.123+0000 7feac4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:20.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.174+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7554023ad0 con 0x7f755809aba0 2024-09-15T03:30:20.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.174+0000 7f754d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.174+0000 7f754d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:20.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.279+0000 7feacf06f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.279+0000 7feacf06f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.279+0000 7feacf06f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.279+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feac8136af0 con 0x7feac809ec50 2024-09-15T03:30:20.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.280+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7feabc017020 con 0x7feac809ec50 2024-09-15T03:30:20.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.280+0000 7feac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.280+0000 7feac4ff9640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-15T03:30:20.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.370+0000 7feacf06f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.370+0000 7feacf06f640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.370+0000 7feacf06f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.370+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feac809a770 con 0x7feac809ec50 2024-09-15T03:30:20.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.371+0000 7feac4ff9640 1 -- 172.21.15.181:0/860845686 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (secure 0 0 0) 0x7feabc018260 con 0x7feac809ec50 2024-09-15T03:30:20.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.371+0000 7feac4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.371+0000 7feac4ff9640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-15T03:30:20.373 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'fooo' does not exist 2024-09-15T03:30:20.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feaac038770 msgr2=0x7feaac03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feaac038770 0x7feaac03ac20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7feac8098850 tx=0x7feab4008040 comp rx=0 tx=0).stop 2024-09-15T03:30:20.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 10 monclient: shutdown 2024-09-15T03:30:20.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 msgr2=0x7feac8131410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 0x7feac8131410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c490 tx=0x7feabc007e40 comp rx=0 tx=0).stop 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 shutdown_connections 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feaac038770 0x7feaac03ac20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feac8136d60 0x7feac8097530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac809f020 0x7feac8096ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 --2- 172.21.15.181:0/860845686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac809ec50 0x7feac8131410 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 >> 172.21.15.181:0/860845686 conn(0x7feac80a5d40 msgr2=0x7feac80a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 shutdown_connections 2024-09-15T03:30:20.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.373+0000 7feacf06f640 1 -- 172.21.15.181:0/860845686 wait complete. 2024-09-15T03:30:20.383 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-15T03:30:20.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f755f6d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:20.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f755f6d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:20.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f755f6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f755806f140 con 0x7f755809aba0 2024-09-15T03:30:20.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f754d7fa640 1 -- 172.21.15.153:0/3507804200 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f755404d620 con 0x7f755809aba0 2024-09-15T03:30:20.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f754d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:20.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.388+0000 7f754d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:20.391 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:20.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f753c038740 msgr2=0x7f753c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f753c038740 0x7f753c03abf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7548007d10 tx=0x7f7548009040 comp rx=0 tx=0).stop 2024-09-15T03:30:20.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 10 monclient: shutdown 2024-09-15T03:30:20.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 msgr2=0x7f7558131b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f755400ec80 tx=0x7f75540090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 shutdown_connections 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f753c038740 0x7f753c03abf0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f755809c860 0x7f755806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f755809af70 0x7f7558132070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 --2- 172.21.15.153:0/3507804200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f755809aba0 0x7f7558131b30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 >> 172.21.15.153:0/3507804200 conn(0x7f75580a2e40 msgr2=0x7f7558098b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 shutdown_connections 2024-09-15T03:30:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.390+0000 7f755f6d4640 1 -- 172.21.15.153:0/3507804200 wait complete. 2024-09-15T03:30:20.402 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:30:20.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:30:20.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.457+0000 7f57b920f640 10 monclient: get_monmap_and_config 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.457+0000 7f57b920f640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.457+0000 7f57b920f640 10 monclient: monmap: 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:20.458017+0000 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:20.458017+0000 2024-09-15T03:30:20.458 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:20.459 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.457+0000 7f57b920f640 1 Processor -- start 2024-09-15T03:30:20.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.457+0000 7f57b920f640 1 -- start start 2024-09-15T03:30:20.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.457+0000 7f57b920f640 10 monclient: init 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57b40a0ad0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b409f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient(hunting): picked mon.noname-b con 0x7f57b409ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57b4136e00 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b406a5b0 con 0x7f57b4136e00 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b40a0460 con 0x7f57b40a0ad0 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b409f7a0 con 0x7f57b409ee60 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b920f640 10 monclient(hunting): authenticate will time out at 824.939354s 2024-09-15T03:30:20.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b2d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f57b40a0ad0 0x7f57b40691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39620/0 (socket says 172.21.15.181:39620) 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57b2d9d640 1 -- 172.21.15.181:0/1939588056 learned_addr learned my addr 172.21.15.181:0/1939588056 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 1 --2- 172.21.15.181:0/1939588056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b409f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): get_auth_request con 0x7f57b409ee60 auth_method 0 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient(hunting): handle_auth_done global_id 4691 payload 274 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 1 -- 172.21.15.181:0/1939588056 >> v1:172.21.15.181:6789/0 conn(0x7f57b40a0ad0 legacy=0x7f57b40691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 1 -- 172.21.15.181:0/1939588056 >> v1:172.21.15.153:6789/0 conn(0x7f57b4136e00 legacy=0x7f57b41391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 1 monclient: found mon.noname-b 2024-09-15T03:30:20.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.458+0000 7f57aa59c640 1 -- 172.21.15.181:0/1939588056 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40a0ea0 con 0x7f57b409ee60 2024-09-15T03:30:20.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57aa59c640 1 -- 172.21.15.181:0/1939588056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 msgr2=0x7f57b409f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57aa59c640 1 --2- 172.21.15.181:0/1939588056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b409f260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f579c009880 tx=0x7f579c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61b122ba61fae437 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 1 -- 172.21.15.181:0/1939588056 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c032440 con 0x7f57b409ee60 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: dump: 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.473 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 1 -- 172.21.15.181:0/1939588056 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c0325a0 con 0x7f57b409ee60 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 1 -- 172.21.15.181:0/1939588056 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c032940 con 0x7f57b409ee60 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b259c640 10 monclient: dump: 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.475 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b920f640 5 monclient: authenticate success, global_id 4691 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b920f640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.461+0000 7f57b920f640 4 set_mon_vals no callback set 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 10 monclient: shutdown 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 -- 172.21.15.181:0/1939588056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 msgr2=0x7f57b409f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 --2- 172.21.15.181:0/1939588056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b409f260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f579c009880 tx=0x7f579c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 -- 172.21.15.181:0/1939588056 shutdown_connections 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 --2- 172.21.15.181:0/1939588056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b409f260 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 -- 172.21.15.181:0/1939588056 >> 172.21.15.181:0/1939588056 conn(0x7f57b4071370 msgr2=0x7f57b40a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 -- 172.21.15.181:0/1939588056 shutdown_connections 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 1 -- 172.21.15.181:0/1939588056 wait complete. 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.462+0000 7f57b920f640 10 monclient: monmap: 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:20.477 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 Processor -- start 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 -- start start 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient: init 2024-09-15T03:30:20.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b4134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): picked mon.noname-c con 0x7f57b409ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): picked mon.noname-b con 0x7f57b40a0ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b4136e00 0x7f57b411da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): picked mon.noname-a con 0x7f57b4136e00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4142120 con 0x7f57b4136e00 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4121220 con 0x7f57b40a0ad0 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40a11e0 con 0x7f57b409ee60 2024-09-15T03:30:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.463+0000 7f57b920f640 10 monclient(hunting): authenticate will time out at 824.944898s 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46602/0 (socket says 172.21.15.181:46602) 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 -- 172.21.15.181:0/971350153 learned_addr learned my addr 172.21.15.181:0/971350153 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 10 monclient(hunting): get_auth_request con 0x7f57b40a0ad0 auth_method 0 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b4134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): get_auth_request con 0x7f57b409ee60 auth_method 0 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b4136e00 0x7f57b411da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): get_auth_request con 0x7f57b4136e00 auth_method 0 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 msgr2=0x7f57b411d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 msgr2=0x7f57b411d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57aa59c640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient(hunting): handle_auth_done global_id 4694 payload 824 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 msgr2=0x7f57b411d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b4136e00 msgr2=0x7f57b411da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b4136e00 0x7f57b411da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 monclient: found mon.noname-c 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.464+0000 7f57b2d9d640 1 -- 172.21.15.181:0/971350153 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c009390 con 0x7f57b409ee60 2024-09-15T03:30:20.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b2d9d640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 msgr2=0x7f57b4134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b2d9d640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b4134ae0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c440 tx=0x7f57a4007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4005cb0 con 0x7f57b409ee60 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: dump: 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.483 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a4005e30 con 0x7f57b409ee60 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4007320 con 0x7f57b409ee60 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57ab7fe640 10 monclient: dump: 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.484 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b1d9b640 4 set_mon_vals no callback set 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 5 monclient: authenticate success, global_id 4694 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 10 monclient: _renew_subs 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b409e9a0 con 0x7f57b409ee60 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 10 monclient: _renew_subs 2024-09-15T03:30:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.466+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b411fd00 con 0x7f57b409ee60 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f57a401b3b0 con 0x7f57b409ee60 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57b920f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57b920f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57b920f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57ab7fe640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5794038740 0x7f579403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b4120e10 con 0x7f57b409ee60 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f57a404ade0 con 0x7f57b409ee60 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57aa59c640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5794038740 0x7f579403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.467+0000 7f57aa59c640 10 monclient: get_auth_request con 0x7f5794038740 auth_method 0 2024-09-15T03:30:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.470+0000 7f57aa59c640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5794038740 0x7f579403abf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f57b411ed30 tx=0x7f579c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.472+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f57b4120e10 con 0x7f57b409ee60 2024-09-15T03:30:20.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.472+0000 7f57ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.472+0000 7f57ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:20.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.494+0000 7fb3ae22b640 10 monclient: get_monmap_and_config 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.494+0000 7fb3ae22b640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.494+0000 7fb3ae22b640 10 monclient: monmap: 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:20.496504+0000 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:20.496504+0000 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:20.497 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 1 Processor -- start 2024-09-15T03:30:20.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 1 -- start start 2024-09-15T03:30:20.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 10 monclient: init 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 0x7fb3a8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a8131980 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb3a809fe10 0x7fb3a809ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.495+0000 7fb3ae22b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a809fe10 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 0x7fb3a8068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a809ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a81325b0 con 0x7fb3a809fe10 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a80a11f0 con 0x7fb3a8131980 2024-09-15T03:30:20.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8132290 con 0x7fb3a809ef70 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3ae22b640 10 monclient(hunting): authenticate will time out at 818.136044s 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb3a809fe10 0x7fb3a809ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 0x7fb3a8068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 0x7fb3a8068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52358/0 (socket says 172.21.15.153:52358) 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 1 -- 172.21.15.153:0/1762973085 learned_addr learned my addr 172.21.15.153:0/1762973085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb3a809fe10 0x7fb3a809ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35854/0 (socket says 172.21.15.153:35854) 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): get_auth_request con 0x7fb3a809ef70 auth_method 0 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): get_auth_request con 0x7fb3a809fe10 auth_method 0 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 1 --2- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 0x7fb3a8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 10 monclient(hunting): get_auth_request con 0x7fb3a8131980 auth_method 0 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 1 -- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 msgr2=0x7fb3a8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 1 -- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 msgr2=0x7fb3a8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 1 --2- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 0x7fb3a8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a77fe640 1 --2- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 0x7fb3a8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_done global_id 34404 payload 274 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 msgr2=0x7fb3a8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 0x7fb3a8068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 msgr2=0x7fb3a8131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 0x7fb3a8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/1762973085 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a809f390 con 0x7fb3a809fe10 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.496+0000 7fb3a7fff640 1 --2- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 0x7fb3a8068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:20.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809fe10 msgr2=0x7fb3a809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809fe10 0x7fb3a809ea30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb3a806f930 tx=0x7fb39c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ee8ab613f306f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 1 -- 172.21.15.153:0/1762973085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c02adf0 con 0x7fb3a809fe10 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: dump: 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.513 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 1 -- 172.21.15.153:0/1762973085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c032440 con 0x7fb3a809fe10 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 5 monclient: authenticate success, global_id 34404 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 4 set_mon_vals no callback set 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 1 -- 172.21.15.153:0/1762973085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c032820 con 0x7fb3a809fe10 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:20.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3a67fc640 10 monclient: dump: 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 10 monclient: shutdown 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 1 -- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809fe10 msgr2=0x7fb3a809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809fe10 0x7fb3a809ea30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb3a806f930 tx=0x7fb39c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 1 -- 172.21.15.153:0/1762973085 shutdown_connections 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809ef70 0x7fb3a8068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/1762973085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809fe10 0x7fb3a809ea30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.499+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/1762973085 >> v2:172.21.15.181:3300/0 conn(0x7fb3a8131980 0x7fb3a8131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.500+0000 7fb3ae22b640 1 -- 172.21.15.153:0/1762973085 >> 172.21.15.153:0/1762973085 conn(0x7fb3a80a5050 msgr2=0x7fb3a80a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.500+0000 7fb3ae22b640 1 -- 172.21.15.153:0/1762973085 shutdown_connections 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.500+0000 7fb3ae22b640 1 -- 172.21.15.153:0/1762973085 wait complete. 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.500+0000 7fb3ae22b640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.500+0000 7fb3ae22b640 10 monclient: monmap: 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:20.516 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.500+0000 7fb3ae22b640 1 Processor -- start 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 -- start start 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient: init 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 0x7fb3a809ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a809ef70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:20.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809fe10 0x7fb3a8130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a809fe10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 0x7fb3a81411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a8131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80a2860 con 0x7fb3a809ef70 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a811dc30 con 0x7fb3a8131980 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a811df10 con 0x7fb3a809fe10 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3ae22b640 10 monclient(hunting): authenticate will time out at 818.141472s 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 0x7fb3a809ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809fe10 0x7fb3a8130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 0x7fb3a809ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35868/0 (socket says 172.21.15.153:35868) 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809fe10 0x7fb3a8130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52370/0 (socket says 172.21.15.153:52370) 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 -- 172.21.15.153:0/861969992 learned_addr learned my addr 172.21.15.153:0/861969992 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): get_auth_request con 0x7fb3a809ef70 auth_method 0 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): get_auth_request con 0x7fb3a809fe10 auth_method 0 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 0x7fb3a81411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 10 monclient(hunting): get_auth_request con 0x7fb3a8131980 auth_method 0 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 msgr2=0x7fb3a81411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 msgr2=0x7fb3a81411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient(hunting): handle_auth_done global_id 34407 payload 824 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 0x7fb3a81411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809fe10 msgr2=0x7fb3a8130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a7fff640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 0x7fb3a81411d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809fe10 0x7fb3a8130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 msgr2=0x7fb3a81411d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a8131980 0x7fb3a81411d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a77fe640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a0009450 con 0x7fb3a809ef70 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.501+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3a809fe10 0x7fb3a8130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3a77fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 msgr2=0x7fb3a809ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3a77fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 0x7fb3a809ab30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb39800e0e0 tx=0x7fb398009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398013590 con 0x7fb3a809ef70 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: dump: 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.522 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 5 monclient: authenticate success, global_id 34407 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3980136f0 con 0x7fb3a809ef70 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 10 monclient: _renew_subs 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c009390 con 0x7fb3a809ef70 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398013ab0 con 0x7fb3a809ef70 2024-09-15T03:30:20.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb38f7fe640 10 monclient: dump: 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 10 monclient: _renew_subs 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3a5ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a811e2a0 con 0x7fb3a809ef70 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.504+0000 7fb3ae22b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb39801e460 con 0x7fb3a809ef70 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 0x7fb3a8141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8065e00 con 0x7fb3a811bbc0 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 0x7fb3a8141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient: get_auth_request con 0x7fb3a811bbc0 auth_method 0 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb38f7fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb394038740 0x7fb39403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fb39804c1d0 con 0x7fb3a809ef70 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_done global_id 34410 payload 824 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3a811e0d0 con 0x7fb3a811bbc0 2024-09-15T03:30:20.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a7fff640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb394038740 0x7fb39403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.505+0000 7fb3a7fff640 10 monclient: get_auth_request con 0x7fb394038740 auth_method 0 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 msgr2=0x7fb3a8141710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 0x7fb3a8141710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb3a809c9c0 tx=0x7fb39c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c046440 con 0x7fb3a811bbc0 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb38f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb38f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb38f7fe640 10 monclient: dump: 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.507+0000 7fb3a7fff640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb394038740 0x7fb39403abf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006cb0 tx=0x7fb3a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb39c0465a0 con 0x7fb3a811bbc0 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 msgr2=0x7fb3a8141710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 0x7fb3a8141710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb3a809c9c0 tx=0x7fb39c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 msgr2=0x7fb3a8141710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.508+0000 7fb38f7fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a811bbc0 0x7fb3a8141710 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb3a809c9c0 tx=0x7fb39c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3ae22b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 0x7fb3a8141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3ae22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a811dda0 con 0x7fb3a8131050 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 0x7fb3a8141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3a6ffd640 10 monclient: get_auth_request con 0x7fb3a8131050 auth_method 0 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.513+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.514+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.514+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.514+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.514+0000 7fb3a6ffd640 10 monclient(hunting): handle_auth_done global_id 34413 payload 824 2024-09-15T03:30:20.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.514+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/861969992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb3a804bf60 con 0x7fb3a8131050 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb3a6ffd640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 msgr2=0x7fb3a8141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb3a6ffd640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 0x7fb3a8141c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb3a809b6e0 tx=0x7fb39c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c04c070 con 0x7fb3a8131050 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 10 monclient: dump: 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.533 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fb39c03cc60 con 0x7fb3a8131050 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 msgr2=0x7fb3a8141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 0x7fb3a8141c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb3a809b6e0 tx=0x7fb39c03d000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 msgr2=0x7fb3a8141c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:20.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.516+0000 7fb38f7fe640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8131050 0x7fb3a8141c50 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb3a809b6e0 tx=0x7fb39c03d000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.535 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:20.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.538 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:20.539 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:20.540 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:20.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:20.543 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:20.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:20.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.546 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:20.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:30:20.549 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.550 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.550 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:20.550 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:20.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb394038740 msgr2=0x7fb39403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb394038740 0x7fb39403abf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006cb0 tx=0x7fb3a0008040 comp rx=0 tx=0).stop 2024-09-15T03:30:20.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 10 monclient: shutdown 2024-09-15T03:30:20.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 msgr2=0x7fb3a809ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 --2- 172.21.15.153:0/861969992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a809ef70 0x7fb3a809ab30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb39800e0e0 tx=0x7fb398009930 comp rx=0 tx=0).stop 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3a7fff640 1 -- 172.21.15.153:0/861969992 reap_dead start 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3a7fff640 1 -- 172.21.15.153:0/861969992 reap_dead start 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 shutdown_connections 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 >> 172.21.15.153:0/861969992 conn(0x7fb3a80a5050 msgr2=0x7fb3a80a4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 shutdown_connections 2024-09-15T03:30:20.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.517+0000 7fb3ae22b640 1 -- 172.21.15.153:0/861969992 wait complete. 2024-09-15T03:30:20.552 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:30:20.552 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:30:20.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:30:20.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.626+0000 7f57b920f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.626+0000 7f57b920f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.626+0000 7f57b920f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.626+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57b4120de0 con 0x7f57b409ee60 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.645+0000 7fc5c8b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.645+0000 7fc5c8b5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.645+0000 7fc5c8b5d640 10 monclient: monmap: 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:20.646798+0000 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:20.646798+0000 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:20.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.645+0000 7fc5c8b5d640 1 Processor -- start 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.645+0000 7fc5c8b5d640 1 -- start start 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.645+0000 7fc5c8b5d640 10 monclient: init 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5c41318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 0x7fc5c409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc5c4136c20 0x7fc5c4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c4136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c4132640 con 0x7fc5c41318f0 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c4131d00 con 0x7fc5c4136c20 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c409fbf0 con 0x7fc5c409ea50 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c8b5d640 10 monclient(hunting): authenticate will time out at 818.286416s 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc5c41318f0 0x7fc5c4131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44860/0 (socket says 172.21.15.153:44860) 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c259c640 1 -- 172.21.15.153:0/1986786475 learned_addr learned my addr 172.21.15.153:0/1986786475 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/1986786475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 0x7fc5c409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): get_auth_request con 0x7fc5c409ea50 auth_method 0 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_done global_id 4703 payload 274 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/1986786475 >> v2:172.21.15.181:3300/0 conn(0x7fc5c4136c20 msgr2=0x7fc5c4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/1986786475 >> v2:172.21.15.181:3300/0 conn(0x7fc5c4136c20 0x7fc5c4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/1986786475 >> v1:172.21.15.153:6789/0 conn(0x7fc5c41318f0 legacy=0x7fc5c4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 1 monclient: found mon.noname-b 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.646+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/1986786475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c409fe30 con 0x7fc5c409ea50 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/1986786475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 msgr2=0x7fc5c409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/1986786475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 0x7fc5c409ee20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009850 tx=0x7fc5b80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8098c2495795d38d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 1 -- 172.21.15.153:0/1986786475 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8032440 con 0x7fc5c409ea50 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: dump: 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.660 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 1 -- 172.21.15.153:0/1986786475 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b80325a0 con 0x7fc5c409ea50 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c8b5d640 5 monclient: authenticate success, global_id 4703 2024-09-15T03:30:20.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 1 -- 172.21.15.153:0/1986786475 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8032940 con 0x7fc5c409ea50 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c8b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c8b5d640 4 set_mon_vals no callback set 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.649+0000 7fc5c159a640 10 monclient: dump: 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.662 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 10 monclient: shutdown 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/1986786475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 msgr2=0x7fc5c409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/1986786475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 0x7fc5c409ee20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009850 tx=0x7fc5b80090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/1986786475 shutdown_connections 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/1986786475 >> v2:172.21.15.181:3300/0 conn(0x7fc5c4136c20 0x7fc5c4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/1986786475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c409ea50 0x7fc5c409ee20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/1986786475 >> 172.21.15.153:0/1986786475 conn(0x7fc5c4067110 msgr2=0x7fc5c4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/1986786475 shutdown_connections 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/1986786475 wait complete. 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.650+0000 7fc5c8b5d640 10 monclient: monmap: 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:20.664 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 Processor -- start 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 -- start start 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient: init 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 0x7fc5c4097540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 0x7fc5c4097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c41318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4136c20 0x7fc5c4097fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c4136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4130bd0 con 0x7fc5c409ea50 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4099390 con 0x7fc5c41318f0 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c409da80 con 0x7fc5c4136c20 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c8b5d640 10 monclient(hunting): authenticate will time out at 818.291723s 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 0x7fc5c4097540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 0x7fc5c4097540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35896/0 (socket says 172.21.15.153:35896) 2024-09-15T03:30:20.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4136c20 0x7fc5c4097fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 1 -- 172.21.15.153:0/3749636540 learned_addr learned my addr 172.21.15.153:0/3749636540 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): get_auth_request con 0x7fc5c409ea50 auth_method 0 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): get_auth_request con 0x7fc5c4136c20 auth_method 0 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 0x7fc5c4097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c1d9b640 10 monclient(hunting): get_auth_request con 0x7fc5c41318f0 auth_method 0 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.651+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 msgr2=0x7fc5c4097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 msgr2=0x7fc5c4097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:20.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 0x7fc5c4097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 10 monclient(hunting): handle_auth_done global_id 34416 payload 824 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 0x7fc5c4097a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4136c20 msgr2=0x7fc5c4097fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4136c20 0x7fc5c4097fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 msgr2=0x7fc5c4097a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c41318f0 0x7fc5c4097a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 1 monclient: found mon.noname-a 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c259c640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8009390 con 0x7fc5c409ea50 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.652+0000 7fc5c2d9d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4136c20 0x7fc5c4097fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:20.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c259c640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 msgr2=0x7fc5c4097540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c259c640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 0x7fc5c4097540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400c610 tx=0x7fc5b4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4005cb0 con 0x7fc5c409ea50 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: dump: 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4005e30 con 0x7fc5c409ea50 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 5 monclient: authenticate success, global_id 34416 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4007340 con 0x7fc5c409ea50 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 10 monclient: _renew_subs 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c409bf70 con 0x7fc5c409ea50 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5b27fc640 10 monclient: dump: 2024-09-15T03:30:20.671 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c0d99640 4 set_mon_vals no callback set 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 10 monclient: _renew_subs 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:20.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.654+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c4131640 con 0x7fc5c409ea50 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c8b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 0x7fc5c402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4065e00 con 0x7fc5c4098500 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc5b401b400 con 0x7fc5c409ea50 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 0x7fc5c402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient: get_auth_request con 0x7fc5c4098500 auth_method 0 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5b27fc640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5a4038740 0x7fc5a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_done global_id 4709 payload 824 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5c409ad40 con 0x7fc5c4098500 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fc5b404b550 con 0x7fc5c409ea50 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.655+0000 7fc5c2d9d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5a4038740 0x7fc5a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.656+0000 7fc5c2d9d640 10 monclient: get_auth_request con 0x7fc5a4038740 auth_method 0 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.657+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 msgr2=0x7fc5c402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.657+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 0x7fc5c402aa40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8034000 tx=0x7fc5b8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.657+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b803ca10 con 0x7fc5c4098500 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.657+0000 7fc5b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.657+0000 7fc5b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.657+0000 7fc5b27fc640 10 monclient: dump: 2024-09-15T03:30:20.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc5b803cb70 con 0x7fc5c4098500 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 msgr2=0x7fc5c402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 0x7fc5c402aa40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8034000 tx=0x7fc5b8032000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 msgr2=0x7fc5c402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:20.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5b27fc640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c4098500 0x7fc5c402aa40 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8034000 tx=0x7fc5b8032000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.658+0000 7fc5c2d9d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5a4038740 0x7fc5a403abf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5c409ac80 tx=0x7fc5bc009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.663+0000 7fc5c8b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.663+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 0x7fc5c405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.663+0000 7fc5c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.663+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c404bf60 con 0x7fc5c405e310 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 0x7fc5c405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient: get_auth_request con 0x7fc5c405e310 auth_method 0 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 10 monclient(hunting): handle_auth_done global_id 4712 payload 824 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.664+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/3749636540 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc5c4133ea0 con 0x7fc5c405e310 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5c1d9b640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 msgr2=0x7fc5c405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5c1d9b640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 0x7fc5c405e710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40686d0 tx=0x7fc5b803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8046e20 con 0x7fc5c405e310 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 10 monclient: dump: 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.679 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fc5b803b5e0 con 0x7fc5c405e310 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 msgr2=0x7fc5c405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 0x7fc5c405e710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40686d0 tx=0x7fc5b803d000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 msgr2=0x7fc5c405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.666+0000 7fc5b27fc640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5c405e310 0x7fc5c405e710 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40686d0 tx=0x7fc5b803d000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:20.681 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:20.682 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:20.683 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:20.684 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:20.685 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.686 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:20.687 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:20.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:20.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.691 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:30:20.692 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:20.693 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:20.694 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:20.694 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:20.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:20.694 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:20.694 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:20.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5a4038740 msgr2=0x7fc5a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5a4038740 0x7fc5a403abf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5c409ac80 tx=0x7fc5bc009780 comp rx=0 tx=0).stop 2024-09-15T03:30:20.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 10 monclient: shutdown 2024-09-15T03:30:20.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 msgr2=0x7fc5c4097540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c2d9d640 1 -- 172.21.15.153:0/3749636540 reap_dead start 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 0x7fc5c4097540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400c610 tx=0x7fc5b4004960 comp rx=0 tx=0).stop 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 shutdown_connections 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 --2- 172.21.15.153:0/3749636540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c409ea50 0x7fc5c4097540 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 >> 172.21.15.153:0/3749636540 conn(0x7fc5c4067110 msgr2=0x7fc5c40a3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 shutdown_connections 2024-09-15T03:30:20.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.667+0000 7fc5c8b5d640 1 -- 172.21.15.153:0/3749636540 wait complete. 2024-09-15T03:30:20.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:30:20.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:30:20.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:30:20.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.763+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v40) v1 ==== 162+0+0 (secure 0 0 0) 0x7f57a4017020 con 0x7f57b409ee60 2024-09-15T03:30:20.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.763+0000 7f57ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.763+0000 7f57ab7fe640 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.789+0000 7f1d9d99d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.789+0000 7f1d9d99d640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.789+0000 7f1d9d99d640 10 monclient: monmap: 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:20.791127+0000 2024-09-15T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:20.791127+0000 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:20.792 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.789+0000 7f1d9d99d640 1 Processor -- start 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 -- start start 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient: init 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 0x7f1d98131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d98131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 0x7f1d98068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d9809f220 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9809faf0 con 0x7f1d9809f220 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9809fd30 con 0x7f1d98131ab0 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980a1110 con 0x7f1d9809e910 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d9d99d640 10 monclient(hunting): authenticate will time out at 818.430578s 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 0x7f1d98068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 0x7f1d98068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35898/0 (socket says 172.21.15.153:35898) 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52400/0 (socket says 172.21.15.153:52400) 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 1 -- 172.21.15.153:0/3498549654 learned_addr learned my addr 172.21.15.153:0/3498549654 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): get_auth_request con 0x7f1d9809f220 auth_method 0 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): get_auth_request con 0x7f1d9809e910 auth_method 0 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 0x7f1d98131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 10 monclient(hunting): get_auth_request con 0x7f1d98131ab0 auth_method 0 2024-09-15T03:30:20.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 1 -- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 msgr2=0x7f1d98131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 1 -- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 msgr2=0x7f1d98131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 0x7f1d98131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d96ffd640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 0x7f1d98131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 10 monclient(hunting): handle_auth_done global_id 4715 payload 274 2024-09-15T03:30:20.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 1 -- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 msgr2=0x7f1d98131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 0x7f1d98131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.790+0000 7f1d967fc640 1 -- 172.21.15.153:0/3498549654 >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 msgr2=0x7f1d98068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d967fc640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 0x7f1d98068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d967fc640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d967fc640 1 monclient: found mon.noname-b 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d967fc640 1 -- 172.21.15.153:0/3498549654 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980a13f0 con 0x7f1d9809e910 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.791+0000 7f1d977fe640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 0x7f1d98068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d967fc640 1 -- 172.21.15.153:0/3498549654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 msgr2=0x7f1d9809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d967fc640 1 --2- 172.21.15.153:0/3498549654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809ece0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d98076760 tx=0x7f1d800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abaf57e0daaa4756 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 1 -- 172.21.15.153:0/3498549654 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8002ae90 con 0x7f1d9809e910 2024-09-15T03:30:20.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: dump: 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 1 -- 172.21.15.153:0/3498549654 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d80032440 con 0x7f1d9809e910 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d9d99d640 5 monclient: authenticate success, global_id 4715 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d9d99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d9d99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 1 -- 172.21.15.153:0/3498549654 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d800327e0 con 0x7f1d9809e910 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d9d99d640 4 set_mon_vals no callback set 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.793+0000 7f1d95ffb640 10 monclient: dump: 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 10 monclient: shutdown 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 -- 172.21.15.153:0/3498549654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 msgr2=0x7f1d9809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/3498549654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809ece0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d98076760 tx=0x7f1d800090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 -- 172.21.15.153:0/3498549654 shutdown_connections 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.153:3300/0 conn(0x7f1d9809f220 0x7f1d98068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/3498549654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809ece0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/3498549654 >> v2:172.21.15.181:3300/0 conn(0x7f1d98131ab0 0x7f1d98131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 -- 172.21.15.153:0/3498549654 >> 172.21.15.153:0/3498549654 conn(0x7f1d98067d30 msgr2=0x7f1d980a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 -- 172.21.15.153:0/3498549654 shutdown_connections 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.794+0000 7f1d9d99d640 1 -- 172.21.15.153:0/3498549654 wait complete. 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 10 monclient: build_initial_monmap 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 10 monclient: monmap: 2024-09-15T03:30:20.811 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 1 Processor -- start 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 1 -- start start 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 10 monclient: init 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:20.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.795+0000 7f1d9d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d9809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 0x7f1d9811bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9809f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d98131ab0 0x7f1d98130ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d98131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9811cf40 con 0x7f1d98131ab0 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9811c920 con 0x7f1d9809f220 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9811ded0 con 0x7f1d9809e910 2024-09-15T03:30:20.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d9d99d640 10 monclient(hunting): authenticate will time out at 818.436062s 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d98131ab0 0x7f1d98130ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52402/0 (socket says 172.21.15.153:52402) 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 -- 172.21.15.153:0/2034778361 learned_addr learned my addr 172.21.15.153:0/2034778361 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): get_auth_request con 0x7f1d9809e910 auth_method 0 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): get_auth_request con 0x7f1d98131ab0 auth_method 0 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 0x7f1d9811bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 10 monclient(hunting): get_auth_request con 0x7f1d9809f220 auth_method 0 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 msgr2=0x7f1d9811bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 msgr2=0x7f1d9811bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 0x7f1d9811bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 0x7f1d9811bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient(hunting): handle_auth_done global_id 4718 payload 824 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 msgr2=0x7f1d9811bb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d9809f220 0x7f1d9811bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d98131ab0 msgr2=0x7f1d98130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d98131ab0 0x7f1d98130ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 monclient: found mon.noname-c 2024-09-15T03:30:20.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d96ffd640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d88009450 con 0x7f1d9809e910 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.796+0000 7f1d977fe640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d98131ab0 0x7f1d98130ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d96ffd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 msgr2=0x7f1d9809aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d96ffd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809aa00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000e110 tx=0x7f1d90009930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9000ee90 con 0x7f1d9809e910 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: dump: 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: _finish_auth 0 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 5 monclient: authenticate success, global_id 4718 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d90013590 con 0x7f1d9809e910 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 10 monclient: _renew_subs 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d80009390 con 0x7f1d9809e910 2024-09-15T03:30:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90013950 con 0x7f1d9809e910 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d8effd640 10 monclient: dump: 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:20.819 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d957fa640 4 set_mon_vals no callback set 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 10 monclient: _renew_subs 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.798+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9809b530 con 0x7f1d9809e910 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1d90013d00 con 0x7f1d9809e910 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d9d99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d98063740 con 0x7f1d98131010 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d8effd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d78038740 0x7f1d7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f1d9004c170 con 0x7f1d9809e910 2024-09-15T03:30:20.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.799+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 10 monclient: get_auth_request con 0x7f1d98131010 auth_method 0 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d977fe640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d78038740 0x7f1d7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d977fe640 10 monclient: get_auth_request con 0x7f1d78038740 auth_method 0 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.800+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:20.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:20.802+0000 7f1d977fe640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d78038740 0x7f1d7803abf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1d9809b960 tx=0x7f1d880091d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:20.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.857+0000 7f57b920f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.857+0000 7f57b920f640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.857+0000 7f57b920f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:20.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.857+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57b4120fd0 con 0x7f57b409ee60 2024-09-15T03:30:20.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.859+0000 7f57ab7fe640 1 -- 172.21.15.181:0/971350153 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v40) v1 ==== 162+0+0 (secure 0 0 0) 0x7f57a404f070 con 0x7f57b409ee60 2024-09-15T03:30:20.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.859+0000 7f57ab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:20.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.859+0000 7f57ab7fe640 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-15T03:30:20.860 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'fuggg' does not exist 2024-09-15T03:30:20.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5794038740 msgr2=0x7f579403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5794038740 0x7f579403abf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f57b411ed30 tx=0x7f579c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:20.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 10 monclient: shutdown 2024-09-15T03:30:20.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:20.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 msgr2=0x7f57b4134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b4134ae0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c440 tx=0x7f57a4007f00 comp rx=0 tx=0).stop 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 shutdown_connections 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5794038740 0x7f579403abf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b4136e00 0x7f57b411da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b40a0ad0 0x7f57b411d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 --2- 172.21.15.181:0/971350153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57b409ee60 0x7f57b4134ae0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 >> 172.21.15.181:0/971350153 conn(0x7f57b4071370 msgr2=0x7f57b4068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 shutdown_connections 2024-09-15T03:30:20.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:20.861+0000 7f57b920f640 1 -- 172.21.15.181:0/971350153 wait complete. 2024-09-15T03:30:20.871 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo OK 2024-09-15T03:30:20.872 INFO:tasks.workunit.client.0.smithi181.stdout:OK 2024-09-15T03:30:20.872 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:30:20.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:30:20.898 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-15T03:30:20.899 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-15T03:30:20.899 DEBUG:teuthology.orchestra.run.smithi181:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-15T03:30:20.960 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush dump 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 10 monclient: get_auth_request con 0x7f1d98131010 auth_method 0 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:21.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:21.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:21.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.001+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:30:21.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.031+0000 7f8fa9f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:21.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.031+0000 7f8fa9f9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.031+0000 7f8fa9f9d640 10 monclient: monmap: 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:21.032217+0000 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:21.032217+0000 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.033 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:21.034 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:21.034 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:21.034 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:21.034 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:21.034 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 Processor -- start 2024-09-15T03:30:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 -- start start 2024-09-15T03:30:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient: init 2024-09-15T03:30:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T03:30:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 0x7f8fa409b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa40972c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa409bcb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fa4099790 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa409c5c0 con 0x7f8fa4099790 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa409d6a0 con 0x7f8fa409bcb0 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa409c880 con 0x7f8fa40972c0 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa9f9d640 10 monclient(hunting): authenticate will time out at 825.513766s 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46612/0 (socket says 172.21.15.181:46612) 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 1 -- 172.21.15.181:0/1343044422 learned_addr learned my addr 172.21.15.181:0/1343044422 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 10 monclient(hunting): get_auth_request con 0x7f8fa409bcb0 auth_method 0 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 1 -- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 msgr2=0x7f8fa409c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 1 -- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 msgr2=0x7f8fa409c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.032+0000 7f8fa2ffd640 1 --2- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa2ffd640 1 --2- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/1343044422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 0x7f8fa409b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): get_auth_request con 0x7f8fa40972c0 auth_method 0 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_done global_id 4721 payload 274 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 1 -- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 msgr2=0x7f8fa409c080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 1 -- 172.21.15.181:0/1343044422 >> v1:172.21.15.181:6789/0 conn(0x7f8fa4099790 legacy=0x7f8fa4099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.033+0000 7f8fa37fe640 1 -- 172.21.15.181:0/1343044422 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4097690 con 0x7f8fa40972c0 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa37fe640 1 -- 172.21.15.181:0/1343044422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 msgr2=0x7f8fa409b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/1343044422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 0x7f8fa409b770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4068eb0 tx=0x7f8f940090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4be566da541cabd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 1 -- 172.21.15.181:0/1343044422 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9402adb0 con 0x7f8fa40972c0 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: dump: 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 5 monclient: authenticate success, global_id 4721 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 1 -- 172.21.15.181:0/1343044422 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94032440 con 0x7f8fa40972c0 2024-09-15T03:30:21.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 1 -- 172.21.15.181:0/1343044422 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f940327e0 con 0x7f8fa40972c0 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 4 set_mon_vals no callback set 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa27fc640 10 monclient: dump: 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 10 monclient: shutdown 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/1343044422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 msgr2=0x7f8fa409b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.036+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/1343044422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 0x7f8fa409b770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4068eb0 tx=0x7f8f940090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/1343044422 shutdown_connections 2024-09-15T03:30:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/1343044422 >> v2:172.21.15.181:3300/0 conn(0x7f8fa409bcb0 0x7f8fa409c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/1343044422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa40972c0 0x7f8fa409b770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/1343044422 >> 172.21.15.181:0/1343044422 conn(0x7f8fa40714d0 msgr2=0x7f8fa40a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/1343044422 shutdown_connections 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/1343044422 wait complete. 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.037+0000 7f8fa9f9d640 10 monclient: monmap: 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 Processor -- start 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 -- start start 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient: init 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 0x7f8fa4140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa40972c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa4099790 0x7f8fa412fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa4099790 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa409bcb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4130760 con 0x7f8fa40972c0 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4121260 con 0x7f8fa409bcb0 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4120c10 con 0x7f8fa4099790 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa9f9d640 10 monclient(hunting): authenticate will time out at 825.519598s 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46618/0 (socket says 172.21.15.181:46618) 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 -- 172.21.15.181:0/2686206767 learned_addr learned my addr 172.21.15.181:0/2686206767 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 10 monclient(hunting): get_auth_request con 0x7f8fa409bcb0 auth_method 0 2024-09-15T03:30:21.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 msgr2=0x7f8fa411d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 msgr2=0x7f8fa411d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 0x7f8fa4140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): get_auth_request con 0x7f8fa40972c0 auth_method 0 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa3fff640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:21.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa2ffd640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa4099790 0x7f8fa412fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa2ffd640 10 monclient(hunting): get_auth_request con 0x7f8fa4099790 auth_method 0 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.038+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 10 monclient(hunting): handle_auth_done global_id 34425 payload 824 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa4099790 msgr2=0x7f8fa412fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa4099790 0x7f8fa412fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 msgr2=0x7f8fa411d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 1 monclient: found mon.noname-a 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.039+0000 7f8fa37fe640 1 -- 172.21.15.181:0/2686206767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94009390 con 0x7f8fa40972c0 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa37fe640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 msgr2=0x7f8fa4140f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa37fe640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 0x7f8fa4140f70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f94031000 tx=0x7f8f94032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9403b8f0 con 0x7f8fa40972c0 2024-09-15T03:30:21.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:21.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: dump: 2024-09-15T03:30:21.067 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa9f9d640 5 monclient: authenticate success, global_id 34425 2024-09-15T03:30:21.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9403ba90 con 0x7f8fa40972c0 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9403be50 con 0x7f8fa40972c0 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa9f9d640 10 monclient: _renew_subs 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa9f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa4120ef0 con 0x7f8fa40972c0 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8f8b7fe640 10 monclient: dump: 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.069 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.041+0000 7f8fa1ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 10 monclient: _renew_subs 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa411fd20 con 0x7f8fa40972c0 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8f94039040 con 0x7f8fa40972c0 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.042+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa40636e0 con 0x7f8fa40972c0 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.043+0000 7f8f8b7fe640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f90038790 0x7f8f9003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.043+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f8f94041030 con 0x7f8fa40972c0 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.043+0000 7f8fa2ffd640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f90038790 0x7f8f9003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.043+0000 7f8fa2ffd640 10 monclient: get_auth_request con 0x7f8f90038790 auth_method 0 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.045+0000 7f8fa2ffd640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f90038790 0x7f8f9003ac40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fa411ed50 tx=0x7f8f98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.047+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8f94044940 con 0x7f8fa40972c0 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.047+0000 7f8f8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.047+0000 7f8f8b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:21.205 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.204+0000 7f8fa9f9d640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-15T03:30:21.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.204+0000 7f8fa9f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-15T03:30:21.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.204+0000 7f8fa9f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.204+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f8fa4064b20 con 0x7f8fa40972c0 2024-09-15T03:30:21.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.205+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v40) v1 ==== 62+0+7238 (secure 0 0 0) 0x7f8f94037020 con 0x7f8fa40972c0 2024-09-15T03:30:21.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.205+0000 7f8f8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-15T03:30:21.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.205+0000 7f8f8b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:21.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.303+0000 7f8fa9f9d640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-15T03:30:21.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.303+0000 7f8fa9f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-15T03:30:21.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.303+0000 7f8fa9f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.303+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f8fa4120d50 con 0x7f8fa40972c0 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.304+0000 7f8f8b7fe640 1 -- 172.21.15.181:0/2686206767 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v40) v1 ==== 62+0+7238 (secure 0 0 0) 0x7f8fa4120d50 con 0x7f8fa40972c0 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.304+0000 7f8f8b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.304+0000 7f8f8b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stdout: "devices": [ 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 0, 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd.0", 2024-09-15T03:30:21.306 INFO:tasks.workunit.client.0.smithi181.stdout: "class": "ssd" 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 1, 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd.1", 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "class": "ssd" 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd.2", 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "class": "ssd" 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-15T03:30:21.307 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd.3", 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "class": "ssd" 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 4, 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd.4", 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "class": "ssd" 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd.5", 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: "class": "ssd" 2024-09-15T03:30:21.308 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "types": [ 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 0, 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "osd" 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 1, 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "host" 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 2, 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "chassis" 2024-09-15T03:30:21.309 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 3, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "rack" 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 4, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "row" 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 5, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "pdu" 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.310 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 6, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "pod" 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 7, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "room" 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 8, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "datacenter" 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 9, 2024-09-15T03:30:21.311 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "zone" 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 10, 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "region" 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 11, 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "root" 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: "buckets": [ 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.312 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -1, 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "default", 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 11, 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "type_name": "root", 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 34560, 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "alg": "straw2", 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": "rjenkins1", 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "items": [ 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -3, 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 0 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.313 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -5, 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 1 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -2, 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "default~ssd", 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 11, 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "type_name": "root", 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 34560, 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "alg": "straw2", 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": "rjenkins1", 2024-09-15T03:30:21.314 INFO:tasks.workunit.client.0.smithi181.stdout: "items": [ 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -4, 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 0 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -6, 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 1 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.315 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -3, 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "smithi153", 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 1, 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "type_name": "host", 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "alg": "straw2", 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": "rjenkins1", 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "items": [ 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 0, 2024-09-15T03:30:21.316 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 0 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 1 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 1, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 2 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.317 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -4, 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "smithi153~ssd", 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 1, 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "type_name": "host", 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "alg": "straw2", 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": "rjenkins1", 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "items": [ 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 0, 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.318 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 0 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 1 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 1, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 2 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.319 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -5, 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "smithi181", 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 1, 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "type_name": "host", 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "alg": "straw2", 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": "rjenkins1", 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "items": [ 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 0 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.320 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 4, 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 1 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 2 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.321 INFO:tasks.workunit.client.0.smithi181.stdout: "id": -6, 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "smithi181~ssd", 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "type_id": 1, 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "type_name": "host", 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 17280, 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "alg": "straw2", 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": "rjenkins1", 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "items": [ 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 0 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.322 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 4, 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 1 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "weight": 5760, 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "pos": 2 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "rules": [ 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.323 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_id": 0, 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_name": "replicated_rule", 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 1, 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "steps": [ 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "take", 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "item": -1, 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "item_name": "default" 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "choose_firstn", 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 0, 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: "type": "osd" 2024-09-15T03:30:21.324 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "emit" 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_id": 1, 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_name": "erasure-code", 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 3, 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "steps": [ 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "set_chooseleaf_tries", 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 5 2024-09-15T03:30:21.325 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "set_choose_tries", 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 100 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "take", 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "item": -1, 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "item_name": "default" 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "choose_indep", 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 0, 2024-09-15T03:30:21.326 INFO:tasks.workunit.client.0.smithi181.stdout: "type": "osd" 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "emit" 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "tunables": { 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "choose_local_tries": 0, 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "choose_total_tries": 50, 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:30:21.327 INFO:tasks.workunit.client.0.smithi181.stdout: "chooseleaf_stable": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "straw_calc_version": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "profile": "jewel", 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "optimal_tunables": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "legacy_tunables": 0, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "minimum_required_version": "jewel", 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "require_feature_tunables": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "require_feature_tunables2": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "has_v2_rules": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "require_feature_tunables3": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "has_v3_rules": 0, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "has_v4_buckets": 1, 2024-09-15T03:30:21.328 INFO:tasks.workunit.client.0.smithi181.stdout: "require_feature_tunables5": 1, 2024-09-15T03:30:21.329 INFO:tasks.workunit.client.0.smithi181.stdout: "has_v5_rules": 0 2024-09-15T03:30:21.329 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.329 INFO:tasks.workunit.client.0.smithi181.stdout: "choose_args": {} 2024-09-15T03:30:21.329 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-15T03:30:21.329 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f90038790 msgr2=0x7f8f9003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f90038790 0x7f8f9003ac40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fa411ed50 tx=0x7f8f98008040 comp rx=0 tx=0).stop 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 10 monclient: shutdown 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 msgr2=0x7f8fa4140f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 0x7f8fa4140f70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f94031000 tx=0x7f8f94032000 comp rx=0 tx=0).stop 2024-09-15T03:30:21.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 shutdown_connections 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f90038790 0x7f8f9003ac40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa409bcb0 0x7f8fa411d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fa4099790 0x7f8fa412fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 --2- 172.21.15.181:0/2686206767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fa40972c0 0x7f8fa4140f70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 >> 172.21.15.181:0/2686206767 conn(0x7f8fa40714d0 msgr2=0x7f8fa409d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 shutdown_connections 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.306+0000 7f8fa9f9d640 1 -- 172.21.15.181:0/2686206767 wait complete. 2024-09-15T03:30:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule dump 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.388+0000 7ff518820640 10 monclient: get_monmap_and_config 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.388+0000 7ff518820640 10 monclient: build_initial_monmap 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.388+0000 7ff518820640 10 monclient: monmap: 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:21.388927+0000 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:21.388927+0000 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:21.392 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.388+0000 7ff518820640 1 Processor -- start 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.388+0000 7ff518820640 1 -- start start 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.388+0000 7ff518820640 10 monclient: init 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff51409ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): picked mon.noname-b con 0x7ff51409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff514136d60 0x7ff514139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): picked mon.noname-a con 0x7ff514136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff514067420 con 0x7ff51409ec50 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51409f3f0 con 0x7ff514136d60 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140a0480 con 0x7ff51409f020 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff518820640 10 monclient(hunting): authenticate will time out at 825.870461s 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff514136d60 0x7ff514139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff51259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff51409ec50 0x7ff514131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:34562/0 (socket says 172.21.15.181:34562) 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff514136d60 0x7ff514139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:53438/0 (socket says 172.21.15.181:53438) 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff51259c640 1 -- 172.21.15.181:0/551024566 learned_addr learned my addr 172.21.15.181:0/551024566 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): get_auth_request con 0x7ff514136d60 auth_method 0 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): get_auth_request con 0x7ff51409f020 auth_method 0 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff512d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.389+0000 7ff511d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 10 monclient(hunting): handle_auth_done global_id 34428 payload 274 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 1 -- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 msgr2=0x7ff514069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 1 --2- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 1 -- 172.21.15.181:0/551024566 >> v1:172.21.15.153:6789/0 conn(0x7ff51409ec50 legacy=0x7ff514131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 1 monclient: found mon.noname-a 2024-09-15T03:30:21.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:21.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.390+0000 7ff512d9d640 1 -- 172.21.15.181:0/551024566 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140a0200 con 0x7ff514136d60 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff512d9d640 1 -- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514136d60 msgr2=0x7ff514139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff512d9d640 1 --2- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514136d60 0x7ff514139140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009e60 tx=0x7ff50c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea132664f6328a6f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 1 -- 172.21.15.181:0/551024566 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c007760 con 0x7ff514136d60 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: dump: 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.404 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 1 -- 172.21.15.181:0/551024566 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c02d050 con 0x7ff514136d60 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 1 -- 172.21.15.181:0/551024566 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c007af0 con 0x7ff514136d60 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 5 monclient: authenticate success, global_id 34428 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 4 set_mon_vals no callback set 2024-09-15T03:30:21.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff51159a640 10 monclient: dump: 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 10 monclient: shutdown 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 -- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514136d60 msgr2=0x7ff514139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 --2- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514136d60 0x7ff514139140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009e60 tx=0x7ff50c009780 comp rx=0 tx=0).stop 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 -- 172.21.15.181:0/551024566 shutdown_connections 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 --2- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514136d60 0x7ff514139140 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 --2- 172.21.15.181:0/551024566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 -- 172.21.15.181:0/551024566 >> 172.21.15.181:0/551024566 conn(0x7ff5140a5cf0 msgr2=0x7ff5140a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 -- 172.21.15.181:0/551024566 shutdown_connections 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.393+0000 7ff518820640 1 -- 172.21.15.181:0/551024566 wait complete. 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.394+0000 7ff518820640 10 monclient: build_initial_monmap 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.394+0000 7ff518820640 10 monclient: monmap: 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:21.407 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.394+0000 7ff518820640 1 Processor -- start 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.394+0000 7ff518820640 1 -- start start 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.394+0000 7ff518820640 10 monclient: init 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51409ec50 0x7ff514099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): picked mon.noname-a con 0x7ff51409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514099f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): picked mon.noname-c con 0x7ff51409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): picked mon.noname-b con 0x7ff514136d60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51409c020 con 0x7ff51409ec50 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140697d0 con 0x7ff514136d60 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51409b960 con 0x7ff51409f020 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff518820640 10 monclient(hunting): authenticate will time out at 825.876149s 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46632/0 (socket says 172.21.15.181:46632) 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 -- 172.21.15.181:0/2043935737 learned_addr learned my addr 172.21.15.181:0/2043935737 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 10 monclient(hunting): get_auth_request con 0x7ff514136d60 auth_method 0 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 msgr2=0x7ff51409a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 msgr2=0x7ff51409a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51409ec50 0x7ff514099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff512d9d640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): get_auth_request con 0x7ff51409f020 auth_method 0 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): get_auth_request con 0x7ff51409ec50 auth_method 0 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff51259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient(hunting): handle_auth_done global_id 4730 payload 824 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 msgr2=0x7ff51409a4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51409ec50 msgr2=0x7ff514099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51409ec50 0x7ff514099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 monclient: found mon.noname-c 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.395+0000 7ff511d9b640 1 -- 172.21.15.181:0/2043935737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c009a20 con 0x7ff51409f020 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff511d9b640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 msgr2=0x7ff514099f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff511d9b640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514099f90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff50800e960 tx=0x7ff5080090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508009540 con 0x7ff51409f020 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: dump: 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.413 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50800cad0 con 0x7ff51409f020 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800ce50 con 0x7ff51409f020 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff5027fc640 10 monclient: dump: 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.414 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 5 monclient: authenticate success, global_id 4730 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 10 monclient: _renew_subs 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51411c870 con 0x7ff51409f020 2024-09-15T03:30:21.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff510d99640 4 set_mon_vals no callback set 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 10 monclient: _renew_subs 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.398+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51411db00 con 0x7ff51409f020 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff518820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff518820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff518820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff508013680 con 0x7ff51409f020 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51411ba20 con 0x7ff51409f020 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff5027fc640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4f4038740 0x7ff4f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7ff50804bc60 con 0x7ff51409f020 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff51259c640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4f4038740 0x7ff4f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.399+0000 7ff51259c640 10 monclient: get_auth_request con 0x7ff4f4038740 auth_method 0 2024-09-15T03:30:21.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.401+0000 7ff51259c640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4f4038740 0x7ff4f403abf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff504009a40 tx=0x7ff504006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.403+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff508023ad0 con 0x7ff51409f020 2024-09-15T03:30:21.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.403+0000 7ff5027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.403+0000 7ff5027fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 10 monclient: get_auth_request con 0x7f1d98131010 auth_method 0 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:30:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:21.402+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:30:21.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.558+0000 7ff518820640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-15T03:30:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.558+0000 7ff518820640 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-15T03:30:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.558+0000 7ff518820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.558+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7ff5140008d0 con 0x7ff51409f020 2024-09-15T03:30:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.559+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v40) v1 ==== 67+0+1058 (secure 0 0 0) 0x7ff508021070 con 0x7ff51409f020 2024-09-15T03:30:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.559+0000 7ff5027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-15T03:30:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.559+0000 7ff5027fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:21.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff518820640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-15T03:30:21.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff518820640 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff518820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7ff514072640 con 0x7ff51409f020 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff5027fc640 1 -- 172.21.15.181:0/2043935737 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v40) v1 ==== 67+0+1058 (secure 0 0 0) 0x7ff5080232f0 con 0x7ff51409f020 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff5027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.652+0000 7ff5027fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stdout:[ 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_id": 0, 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_name": "replicated_rule", 2024-09-15T03:30:21.654 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 1, 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "steps": [ 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "take", 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "item": -1, 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "item_name": "default" 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "choose_firstn", 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 0, 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "type": "osd" 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "emit" 2024-09-15T03:30:21.655 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_id": 1, 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "rule_name": "erasure-code", 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 3, 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "steps": [ 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "set_chooseleaf_tries", 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 5 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.656 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "set_choose_tries", 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 100 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "take", 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "item": -1, 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "item_name": "default" 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "choose_indep", 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "num": 0, 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "type": "osd" 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:30:21.657 INFO:tasks.workunit.client.0.smithi181.stdout: "op": "emit" 2024-09-15T03:30:21.658 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.658 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:30:21.658 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:30:21.658 INFO:tasks.workunit.client.0.smithi181.stdout:] 2024-09-15T03:30:21.658 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4f4038740 msgr2=0x7ff4f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4f4038740 0x7ff4f403abf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff504009a40 tx=0x7ff504006f50 comp rx=0 tx=0).stop 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 10 monclient: shutdown 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 msgr2=0x7ff514099f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514099f90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff50800e960 tx=0x7ff5080090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 shutdown_connections 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4f4038740 0x7ff4f403abf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff514136d60 0x7ff51409a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51409f020 0x7ff514099f90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 --2- 172.21.15.181:0/2043935737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51409ec50 0x7ff514099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 >> 172.21.15.181:0/2043935737 conn(0x7ff5140a5cf0 msgr2=0x7ff51409d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:21.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 shutdown_connections 2024-09-15T03:30:21.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.654+0000 7ff518820640 1 -- 172.21.15.181:0/2043935737 wait complete. 2024-09-15T03:30:21.664 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.736+0000 7ff08659d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.736+0000 7ff08659d640 10 monclient: build_initial_monmap 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.736+0000 7ff08659d640 10 monclient: monmap: 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:21.736955+0000 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:21.736955+0000 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:21.737 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.738 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:21.738 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:21.738 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:21.738 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:21.738 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:21.738 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.736+0000 7ff08659d640 1 Processor -- start 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.736+0000 7ff08659d640 1 -- start start 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient: init 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 0x7ff08006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient(hunting): picked mon.noname-c con 0x7ff08006e1a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff08006e570 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff08009e170 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:21.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08009f7a0 con 0x7ff08009e170 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800696e0 con 0x7ff08006e570 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff080069c60 con 0x7ff08006e1a0 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff08659d640 10 monclient(hunting): authenticate will time out at 826.218466s 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 0x7ff08006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 0x7ff08006a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46642/0 (socket says 172.21.15.181:46642) 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 -- 172.21.15.181:0/3244681854 learned_addr learned my addr 172.21.15.181:0/3244681854 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 10 monclient(hunting): get_auth_request con 0x7ff08006e1a0 auth_method 0 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07f7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7ff08006e570 0x7ff08009da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39640/0 (socket says 172.21.15.181:39640) 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 -- 172.21.15.181:0/3244681854 >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 msgr2=0x7ff08006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:21.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 -- 172.21.15.181:0/3244681854 >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 msgr2=0x7ff08006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:21.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 --2- 172.21.15.181:0/3244681854 >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 0x7ff08006a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:21.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.737+0000 7ff07ffff640 1 --2- 172.21.15.181:0/3244681854 >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 0x7ff08006a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:21.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.739+0000 7ff07f7fe640 1 -- 172.21.15.181:0/3244681854 >> v1:172.21.15.181:6789/0 conn(0x7ff08006e570 legacy=0x7ff08009da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff084b3a640 1 -- 172.21.15.181:0/3244681854 >> v1:172.21.15.153:6789/0 conn(0x7ff08009e170 legacy=0x7ff08009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443638603 0 0) 0x7ff08009f7a0 con 0x7ff08009e170 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 10 monclient(hunting): my global_id is 34434 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0680034e0 con 0x7ff08009e170 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703873144 0 0) 0x7ff0680034e0 con 0x7ff08009e170 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 msgr2=0x7ff08006a640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 --2- 172.21.15.181:0/3244681854 >> v2:172.21.15.181:3300/0 conn(0x7ff08006e1a0 0x7ff08006a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 >> v1:172.21.15.181:6789/0 conn(0x7ff08006e570 legacy=0x7ff08009da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.740+0000 7ff07effd640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080069980 con 0x7ff08009e170 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7ff078001bb0 con 0x7ff08009e170 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff08659d640 5 monclient: authenticate success, global_id 34434 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff08659d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: dump: 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.748 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 >> v1:172.21.15.153:6789/0 conn(0x7ff08009e170 legacy=0x7ff08009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff068003a50 0x7ff068003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): picked mon.a con 0x7ff068003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff068004780 0x7ff068004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): picked mon.c con 0x7ff068004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff06800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): picked mon.b con 0x7ff0680052e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080069c60 con 0x7ff068003a50 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06800c2b0 con 0x7ff0680052e0 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06800c470 con 0x7ff068004780 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff08659d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff06800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): get_auth_request con 0x7ff0680052e0 auth_method 0 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff068004780 0x7ff068004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff068003a50 0x7ff068003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): get_auth_request con 0x7ff068004780 auth_method 0 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): get_auth_request con 0x7ff068003a50 auth_method 0 2024-09-15T03:30:21.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff07ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient(hunting): handle_auth_done global_id 34434 payload 293 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 -- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff068004780 msgr2=0x7ff068004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff068004780 0x7ff068004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 -- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff068003a50 msgr2=0x7ff068003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:21.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff068003a50 0x7ff068003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 monclient: found mon.b 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 1 -- 172.21.15.181:0/3244681854 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680034e0 con 0x7ff0680052e0 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.741+0000 7ff084b3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.742+0000 7ff084b3a640 1 -- 172.21.15.181:0/3244681854 reap_dead start 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff084b3a640 1 -- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 msgr2=0x7ff06800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff084b3a640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff06800b9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff07800a760 tx=0x7ff078007db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69ca7d0a3285f7a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078005b00 con 0x7ff0680052e0 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:21.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: dump: 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078005c60 con 0x7ff0680052e0 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 4 set_mon_vals no callback set 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 1 -- 172.21.15.181:0/3244681854 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07802b680 con 0x7ff0680052e0 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff07effd640 10 monclient: dump: 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.755 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 10 monclient: shutdown 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 1 -- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 msgr2=0x7ff06800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff06800b9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff07800a760 tx=0x7ff078007db0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 1 -- 172.21.15.181:0/3244681854 shutdown_connections 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 1 --2- 172.21.15.181:0/3244681854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff06800b9d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.744+0000 7ff08659d640 1 -- 172.21.15.181:0/3244681854 >> 172.21.15.181:0/3244681854 conn(0x7ff080071340 msgr2=0x7ff0800a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 1 -- 172.21.15.181:0/3244681854 shutdown_connections 2024-09-15T03:30:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 1 -- 172.21.15.181:0/3244681854 wait complete. 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 10 monclient: build_initial_monmap 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 10 monclient: monmap: 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 1 Processor -- start 2024-09-15T03:30:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 1 -- start start 2024-09-15T03:30:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.745+0000 7ff08659d640 10 monclient: init 2024-09-15T03:30:21.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff08013d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0680052e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0801210d0 0x7ff08013cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0801210d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff080140dc0 0x7ff080141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): picked mon.noname-a con 0x7ff080140dc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080122660 con 0x7ff080140dc0 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080122280 con 0x7ff0680052e0 2024-09-15T03:30:21.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff080120570 con 0x7ff0801210d0 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff08659d640 10 monclient(hunting): authenticate will time out at 826.227418s 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff08013d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff08013d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46662/0 (socket says 172.21.15.181:46662) 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 -- 172.21.15.181:0/2482479380 learned_addr learned my addr 172.21.15.181:0/2482479380 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): get_auth_request con 0x7ff0680052e0 auth_method 0 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:21.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07f7fe640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0801210d0 0x7ff08013cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff084b3a640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff080140dc0 0x7ff080141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient(hunting): handle_auth_done global_id 44098 payload 824 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 -- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0801210d0 msgr2=0x7ff08013cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0801210d0 0x7ff08013cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 -- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff080140dc0 msgr2=0x7ff080141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff080140dc0 0x7ff080141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 monclient: found mon.noname-b 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:21.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07ffff640 1 -- 172.21.15.181:0/2482479380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08011fd60 con 0x7ff0680052e0 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07f7fe640 10 monclient: get_auth_request con 0x7ff0801210d0 auth_method 0 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff084b3a640 10 monclient: get_auth_request con 0x7ff080140dc0 auth_method 0 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff07f7fe640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0801210d0 0x7ff08013cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.746+0000 7ff084b3a640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff080140dc0 0x7ff080141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.748+0000 7ff07ffff640 1 -- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 msgr2=0x7ff08013d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.748+0000 7ff07ffff640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff08013d850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff070011760 tx=0x7ff070008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000c070 con 0x7ff0680052e0 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: dump: 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.766 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070015880 con 0x7ff0680052e0 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070015c40 con 0x7ff0680052e0 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 5 monclient: authenticate success, global_id 44098 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 10 monclient: dump: 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 10 monclient: _renew_subs 2024-09-15T03:30:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff080121fa0 con 0x7ff0680052e0 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff07e7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 10 monclient: _renew_subs 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08013e400 con 0x7ff0680052e0 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.749+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff07000f6a0 con 0x7ff0680052e0 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff067fff640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff06c038790 0x7ff06c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7ff07004bca0 con 0x7ff0680052e0 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff08659d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff08659d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff08659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0800008d0 con 0x7ff0680052e0 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff07f7fe640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff06c038790 0x7ff06c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.750+0000 7ff07f7fe640 10 monclient: get_auth_request con 0x7ff06c038790 auth_method 0 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.752+0000 7ff07f7fe640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff06c038790 0x7ff06c03ac40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff074007b20 tx=0x7ff074009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.754+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff07000f950 con 0x7ff0680052e0 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.754+0000 7ff067fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:21.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.754+0000 7ff067fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:21.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff08659d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:21.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff08659d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:21.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff08659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:21.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ff080064a20 con 0x7ff0680052e0 2024-09-15T03:30:21.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v40) v1 ==== 65+0+29 (secure 0 0 0) 0x7ff070040070 con 0x7ff0680052e0 2024-09-15T03:30:21.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff067fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:21.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:21.912+0000 7ff067fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:22.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff08659d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:22.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff08659d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:22.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff08659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ff08013d440 con 0x7ff0680052e0 2024-09-15T03:30:22.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff067fff640 1 -- 172.21.15.181:0/2482479380 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v40) v1 ==== 65+0+29 (secure 0 0 0) 0x7ff070013340 con 0x7ff0680052e0 2024-09-15T03:30:22.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff067fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:22.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.006+0000 7ff067fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:22.008 INFO:tasks.workunit.client.0.smithi181.stdout:replicated_rule 2024-09-15T03:30:22.008 INFO:tasks.workunit.client.0.smithi181.stdout:erasure-code 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff06c038790 msgr2=0x7ff06c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff06c038790 0x7ff06c03ac40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff074007b20 tx=0x7ff074009040 comp rx=0 tx=0).stop 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 10 monclient: shutdown 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 msgr2=0x7ff08013d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff08013d850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff070011760 tx=0x7ff070008040 comp rx=0 tx=0).stop 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 shutdown_connections 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff06c038790 0x7ff06c03ac40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff080140dc0 0x7ff080141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0801210d0 0x7ff08013cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 --2- 172.21.15.181:0/2482479380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0680052e0 0x7ff08013d850 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 >> 172.21.15.181:0/2482479380 conn(0x7ff080071340 msgr2=0x7ff0800a2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 shutdown_connections 2024-09-15T03:30:22.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.008+0000 7ff08659d640 1 -- 172.21.15.181:0/2482479380 wait complete. 2024-09-15T03:30:22.018 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule list 2024-09-15T03:30:22.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.090+0000 7f8f2875d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:22.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.090+0000 7f8f2875d640 10 monclient: build_initial_monmap 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.090+0000 7f8f2875d640 10 monclient: monmap: 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:22.090948+0000 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:22.090948+0000 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:22.093 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.090+0000 7f8f2875d640 1 Processor -- start 2024-09-15T03:30:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.090+0000 7f8f2875d640 1 -- start start 2024-09-15T03:30:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.090+0000 7f8f2875d640 10 monclient: init 2024-09-15T03:30:22.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8f2009ec50 0x7f8f20131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f2009ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f20068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f201321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f2009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f20067420 con 0x7f8f2009f020 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f200a0480 con 0x7f8f2009ec50 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f200a0200 con 0x7f8f201321a0 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f2875d640 10 monclient(hunting): authenticate will time out at 826.572406s 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8f2009ec50 0x7f8f20131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f26cfa640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f8f2009f020 0x7f8f20069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39650/0 (socket says 172.21.15.181:39650) 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f26cfa640 1 -- 172.21.15.181:0/1864823960 learned_addr learned my addr 172.21.15.181:0/1864823960 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): get_auth_request con 0x7f8f2009ec50 auth_method 0 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient(hunting): handle_auth_done global_id 44101 payload 274 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 1 -- 172.21.15.181:0/1864823960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 msgr2=0x7f8f20068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f25cf8640 1 --2- 172.21.15.181:0/1864823960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f20068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 1 --2- 172.21.15.181:0/1864823960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f20068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 1 -- 172.21.15.181:0/1864823960 >> v1:172.21.15.181:6789/0 conn(0x7f8f2009f020 legacy=0x7f8f20069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient: _finish_hunting 0 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 1 monclient: found mon.noname-c 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.091+0000 7f8f264f9640 1 -- 172.21.15.181:0/1864823960 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2009fd80 con 0x7f8f2009ec50 2024-09-15T03:30:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f264f9640 1 -- 172.21.15.181:0/1864823960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009ec50 msgr2=0x7f8f20131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f264f9640 1 --2- 172.21.15.181:0/1864823960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009ec50 0x7f8f20131c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009850 tx=0x7f8f1c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=996b379ab674ccb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 1 -- 172.21.15.181:0/1864823960 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c032440 con 0x7f8f2009ec50 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: dump: 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.106 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: _finish_auth 0 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f2875d640 5 monclient: authenticate success, global_id 44101 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 1 -- 172.21.15.181:0/1864823960 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c0325a0 con 0x7f8f2009ec50 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f2875d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f2875d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f2875d640 4 set_mon_vals no callback set 2024-09-15T03:30:22.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 1 -- 172.21.15.181:0/1864823960 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c032940 con 0x7f8f2009ec50 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.094+0000 7f8f254f7640 10 monclient: dump: 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.108 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 10 monclient: shutdown 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 -- 172.21.15.181:0/1864823960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009ec50 msgr2=0x7f8f20131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1864823960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009ec50 0x7f8f20131c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009850 tx=0x7f8f1c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 -- 172.21.15.181:0/1864823960 shutdown_connections 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1864823960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f20068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1864823960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009ec50 0x7f8f20131c60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 -- 172.21.15.181:0/1864823960 >> 172.21.15.181:0/1864823960 conn(0x7f8f200a5cf0 msgr2=0x7f8f200a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 -- 172.21.15.181:0/1864823960 shutdown_connections 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.095+0000 7f8f2875d640 1 -- 172.21.15.181:0/1864823960 wait complete. 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient: build_initial_monmap 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient: monmap: 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:22.110 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 Processor -- start 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 -- start start 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient: init 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2009ec50 0x7f8f20140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f2009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 0x7f8f20130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f2009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:22.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f2011b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f201321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2011d360 con 0x7f8f2009ec50 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2011d800 con 0x7f8f2009f020 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f20141df0 con 0x7f8f201321a0 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f2875d640 10 monclient(hunting): authenticate will time out at 826.577852s 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 0x7f8f20130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 0x7f8f20130cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46678/0 (socket says 172.21.15.181:46678) 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 1 -- 172.21.15.181:0/1334891119 learned_addr learned my addr 172.21.15.181:0/1334891119 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 10 monclient(hunting): get_auth_request con 0x7f8f2009f020 auth_method 0 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.096+0000 7f8f25cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f26cfa640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f2011b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2009ec50 0x7f8f20140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): get_auth_request con 0x7f8f2009ec50 auth_method 0 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f26cfa640 10 monclient(hunting): get_auth_request con 0x7f8f201321a0 auth_method 0 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f26cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f26cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f26cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f264f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 10 monclient(hunting): handle_auth_done global_id 44104 payload 824 2024-09-15T03:30:22.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 1 -- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 msgr2=0x7f8f2011b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f2011b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 1 -- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2009ec50 msgr2=0x7f8f20140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2009ec50 0x7f8f20140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 10 monclient: _finish_hunting 0 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 1 monclient: found mon.noname-b 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f25cf8640 1 -- 172.21.15.181:0/1334891119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c009390 con 0x7f8f2009f020 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.097+0000 7f8f26cfa640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f2011b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f25cf8640 1 -- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 msgr2=0x7f8f20130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f25cf8640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 0x7f8f20130cf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c880 tx=0x7f8f10008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1000ce10 con 0x7f8f2009f020 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: dump: 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10007a40 con 0x7f8f2009f020 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 5 monclient: authenticate success, global_id 44104 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 10 monclient: _renew_subs 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2011ee20 con 0x7f8f2009f020 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10007de0 con 0x7f8f2009f020 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f167fc640 10 monclient: dump: 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 10 monclient: _renew_subs 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2011dec0 con 0x7f8f2009f020 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.099+0000 7f8f24cf6640 4 set_mon_vals no callback set 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8f10011600 con 0x7f8f2009f020 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f2875d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f2875d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f2875d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f200008d0 con 0x7f8f2009f020 2024-09-15T03:30:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f167fc640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f04038740 0x7f8f0403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f8f1004bbd0 con 0x7f8f2009f020 2024-09-15T03:30:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f264f9640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f04038740 0x7f8f0403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.100+0000 7f8f264f9640 10 monclient: get_auth_request con 0x7f8f04038740 auth_method 0 2024-09-15T03:30:22.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.103+0000 7f8f264f9640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f04038740 0x7f8f0403abf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f2009bf50 tx=0x7f8f1c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.105+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8f10010b30 con 0x7f8f2009f020 2024-09-15T03:30:22.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.105+0000 7f8f167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.105+0000 7f8f167fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:22.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient: get_auth_request con 0x7f1d98131010 auth_method 0 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 10 monclient(hunting): handle_auth_done global_id 44107 payload 824 2024-09-15T03:30:22.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.203+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d9811e2f0 con 0x7f1d98131010 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d80034000 tx=0x7f1d80032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d80000c40 con 0x7f1d98131010 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 10 monclient: dump: 2024-09-15T03:30:22.208 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1d8002ae90 con 0x7f1d98131010 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d80034000 tx=0x7f1d80032000 comp rx=0 tx=0).stop 2024-09-15T03:30:22.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 msgr2=0x7f1d98140e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:22.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.206+0000 7f1d8effd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98131010 0x7f1d98140e10 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d80034000 tx=0x7f1d80032000 comp rx=0 tx=0).stop 2024-09-15T03:30:22.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d9d99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:22.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 0x7f1d98141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d9d99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d981313e0 con 0x7f1d98141350 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 0x7f1d98141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient: get_auth_request con 0x7f1d98141350 auth_method 0 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.212+0000 7f1d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.213+0000 7f1d967fc640 10 monclient(hunting): handle_auth_done global_id 44110 payload 824 2024-09-15T03:30:22.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.213+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1d9804bf60 con 0x7f1d98141350 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d967fc640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 msgr2=0x7f1d98141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d967fc640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 0x7f1d98141720 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d98068700 tx=0x7f1d80040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8003b990 con 0x7f1d98141350 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 10 monclient: dump: 2024-09-15T03:30:22.218 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f1d80050070 con 0x7f1d98141350 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:22.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 msgr2=0x7f1d98141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 0x7f1d98141720 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d98068700 tx=0x7f1d80040040 comp rx=0 tx=0).stop 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 msgr2=0x7f1d98141720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.215+0000 7f1d8effd640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d98141350 0x7f1d98141720 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d98068700 tx=0x7f1d80040040 comp rx=0 tx=0).stop 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:22.220 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:22.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:22.222 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:22.223 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:22.224 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.225 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:22.226 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:22.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:22.228 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:22.230 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:22.231 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:30:22.232 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:22.232 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:22.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:22.232 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:22.232 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:22.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d78038740 msgr2=0x7f1d7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d78038740 0x7f1d7803abf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1d9809b960 tx=0x7f1d880091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 10 monclient: shutdown 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 msgr2=0x7f1d9809aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 --2- 172.21.15.153:0/2034778361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d9809e910 0x7f1d9809aa00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000e110 tx=0x7f1d90009930 comp rx=0 tx=0).stop 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d977fe640 1 -- 172.21.15.153:0/2034778361 reap_dead start 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d977fe640 1 -- 172.21.15.153:0/2034778361 reap_dead start 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 shutdown_connections 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 >> 172.21.15.153:0/2034778361 conn(0x7f1d98067d30 msgr2=0x7f1d98134970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 shutdown_connections 2024-09-15T03:30:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.216+0000 7f1d9d99d640 1 -- 172.21.15.153:0/2034778361 wait complete. 2024-09-15T03:30:22.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:30:22.235 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:30:22.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:30:22.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f2875d640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-15T03:30:22.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f2875d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-15T03:30:22.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f2875d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f8f200a13e0 con 0x7f8f2009f020 2024-09-15T03:30:22.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v40) v1 ==== 67+0+29 (secure 0 0 0) 0x7f8f10010350 con 0x7f8f2009f020 2024-09-15T03:30:22.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-15T03:30:22.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.266+0000 7f8f167fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.329+0000 7fdccbf5d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.329+0000 7fdccbf5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.329+0000 7fdccbf5d640 10 monclient: monmap: 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:22.330998+0000 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:22.330998+0000 2024-09-15T03:30:22.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:22.332 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.329+0000 7fdccbf5d640 1 Processor -- start 2024-09-15T03:30:22.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 -- start start 2024-09-15T03:30:22.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient: init 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdcc4131ab0 0x7fdcc4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc4131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 0x7fdcc409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 0x7fdcc4068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc409f220 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc409faf0 con 0x7fdcc409f220 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc409fd30 con 0x7fdcc4131ab0 2024-09-15T03:30:22.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc40a1110 con 0x7fdcc409e910 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdccbf5d640 10 monclient(hunting): authenticate will time out at 819.970499s 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 0x7fdcc4068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 0x7fdcc409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 0x7fdcc4068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35908/0 (socket says 172.21.15.153:35908) 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 0x7fdcc409ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52414/0 (socket says 172.21.15.153:52414) 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 1 -- 172.21.15.153:0/937942191 learned_addr learned my addr 172.21.15.153:0/937942191 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): get_auth_request con 0x7fdcc409f220 auth_method 0 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): get_auth_request con 0x7fdcc409e910 auth_method 0 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 1 --2- 172.21.15.153:0/937942191 >> v2:172.21.15.181:3300/0 conn(0x7fdcc4131ab0 0x7fdcc4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): get_auth_request con 0x7fdcc4131ab0 auth_method 0 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc9cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_done global_id 4736 payload 274 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 -- 172.21.15.153:0/937942191 >> v2:172.21.15.181:3300/0 conn(0x7fdcc4131ab0 msgr2=0x7fdcc4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 --2- 172.21.15.153:0/937942191 >> v2:172.21.15.181:3300/0 conn(0x7fdcc4131ab0 0x7fdcc4131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 -- 172.21.15.153:0/937942191 >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 msgr2=0x7fdcc4068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 --2- 172.21.15.153:0/937942191 >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 0x7fdcc4068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient: _finish_hunting 0 2024-09-15T03:30:22.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 monclient: found mon.noname-b 2024-09-15T03:30:22.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:22.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:22.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcc94f8640 1 -- 172.21.15.153:0/937942191 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc40a13f0 con 0x7fdcc409e910 2024-09-15T03:30:22.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.330+0000 7fdcca4fa640 1 --2- 172.21.15.153:0/937942191 >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 0x7fdcc4068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:22.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc94f8640 1 -- 172.21.15.153:0/937942191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 msgr2=0x7fdcc409ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc94f8640 1 --2- 172.21.15.153:0/937942191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 0x7fdcc409ece0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4076760 tx=0x7fdcc00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297cb647cf53f505 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 1 -- 172.21.15.153:0/937942191 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc002ae90 con 0x7fdcc409e910 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: dump: 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: _finish_auth 0 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdccbf5d640 5 monclient: authenticate success, global_id 4736 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 1 -- 172.21.15.153:0/937942191 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0032440 con 0x7fdcc409e910 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdccbf5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdccbf5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdccbf5d640 4 set_mon_vals no callback set 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 1 -- 172.21.15.153:0/937942191 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00327e0 con 0x7fdcc409e910 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:22.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.333+0000 7fdcc8cf7640 10 monclient: dump: 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 10 monclient: shutdown 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 -- 172.21.15.153:0/937942191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 msgr2=0x7fdcc409ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/937942191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 0x7fdcc409ece0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4076760 tx=0x7fdcc00090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 -- 172.21.15.153:0/937942191 shutdown_connections 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/937942191 >> v2:172.21.15.153:3300/0 conn(0x7fdcc409f220 0x7fdcc4068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/937942191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc409e910 0x7fdcc409ece0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/937942191 >> v2:172.21.15.181:3300/0 conn(0x7fdcc4131ab0 0x7fdcc4131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 -- 172.21.15.153:0/937942191 >> 172.21.15.153:0/937942191 conn(0x7fdcc4067d30 msgr2=0x7fdcc40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 -- 172.21.15.153:0/937942191 shutdown_connections 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.334+0000 7fdccbf5d640 1 -- 172.21.15.153:0/937942191 wait complete. 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient: monmap: 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:22.348 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 Processor -- start 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 -- start start 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient: init 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcc409e910 0x7fdcc409ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc409e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 0x7fdcc4130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc409f220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 0x7fdcc411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc4131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc411e000 con 0x7fdcc409f220 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc409be50 con 0x7fdcc409e910 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc409dd00 con 0x7fdcc4131ab0 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdccbf5d640 10 monclient(hunting): authenticate will time out at 819.975888s 2024-09-15T03:30:22.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdcc94f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 0x7fdcc4130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdcca4fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 0x7fdcc411b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdcca4fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 0x7fdcc411b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52420/0 (socket says 172.21.15.153:52420) 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 -- 172.21.15.153:0/2403489152 learned_addr learned my addr 172.21.15.153:0/2403489152 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): get_auth_request con 0x7fdcc4131ab0 auth_method 0 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.335+0000 7fdcc94f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 0x7fdcc4130c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35910/0 (socket says 172.21.15.153:35910) 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): get_auth_request con 0x7fdcc409f220 auth_method 0 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcc409e910 0x7fdcc409ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): get_auth_request con 0x7fdcc409e910 auth_method 0 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc9cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient(hunting): handle_auth_done global_id 4739 payload 824 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 -- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcc409e910 msgr2=0x7fdcc409ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcc409e910 0x7fdcc409ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 -- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 msgr2=0x7fdcc4130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 0x7fdcc4130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient: _finish_hunting 0 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 monclient: found mon.noname-c 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcca4fa640 1 -- 172.21.15.153:0/2403489152 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc009ae0 con 0x7fdcc4131ab0 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.336+0000 7fdcc94f8640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 0x7fdcc4130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcca4fa640 1 -- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 msgr2=0x7fdcc411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcca4fa640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 0x7fdcc411b6b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009fd0 tx=0x7fdcbc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc018070 con 0x7fdcc4131ab0 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: dump: 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.354 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc00f040 con 0x7fdcc4131ab0 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 5 monclient: authenticate success, global_id 4739 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 10 monclient: _renew_subs 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc0146b0 con 0x7fdcc4131ab0 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc0009390 con 0x7fdcc4131ab0 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb5ffb640 10 monclient: dump: 2024-09-15T03:30:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdcb7fff640 4 set_mon_vals no callback set 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 10 monclient: _renew_subs 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.338+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc4141be0 con 0x7fdcc4131ab0 2024-09-15T03:30:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdcbc003020 con 0x7fdcc4131ab0 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdccbf5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdccbf5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdccbf5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc411ce20 con 0x7fdcc4131ab0 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdcb5ffb640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdca8038740 0x7fdca803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.339+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fdcbc04bbf0 con 0x7fdcc4131ab0 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.340+0000 7fdcc9cf9640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdca8038740 0x7fdca803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.340+0000 7fdcc9cf9640 10 monclient: get_auth_request con 0x7fdca8038740 auth_method 0 2024-09-15T03:30:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.342+0000 7fdcc9cf9640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdca8038740 0x7fdca803abf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009c90 tx=0x7fdcb80090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.344+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdcbc0092e0 con 0x7fdcc4131ab0 2024-09-15T03:30:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.344+0000 7fdcb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.344+0000 7fdcb5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f2875d640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-15T03:30:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f2875d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-15T03:30:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f2875d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f8f20072640 con 0x7f8f2009f020 2024-09-15T03:30:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f167fc640 1 -- 172.21.15.181:0/1334891119 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v40) v1 ==== 67+0+29 (secure 0 0 0) 0x7f8f10010530 con 0x7f8f2009f020 2024-09-15T03:30:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f167fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-15T03:30:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.363+0000 7f8f167fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:22.365 INFO:tasks.workunit.client.0.smithi181.stdout:replicated_rule 2024-09-15T03:30:22.365 INFO:tasks.workunit.client.0.smithi181.stdout:erasure-code 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f04038740 msgr2=0x7f8f0403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f04038740 0x7f8f0403abf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f2009bf50 tx=0x7f8f1c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 10 monclient: shutdown 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 msgr2=0x7f8f20130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 0x7f8f20130cf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c880 tx=0x7f8f10008040 comp rx=0 tx=0).stop 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 shutdown_connections 2024-09-15T03:30:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8f04038740 0x7f8f0403abf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f201321a0 0x7f8f2011b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f2009f020 0x7f8f20130cf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 --2- 172.21.15.181:0/1334891119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f2009ec50 0x7f8f20140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.365+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 >> 172.21.15.181:0/1334891119 conn(0x7f8f200a5cf0 msgr2=0x7f8f2009d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.366+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 shutdown_connections 2024-09-15T03:30:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.366+0000 7f8f2875d640 1 -- 172.21.15.181:0/1334891119 wait complete. 2024-09-15T03:30:22.375 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-15T03:30:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.451+0000 7ff27d434640 10 monclient: get_monmap_and_config 2024-09-15T03:30:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.451+0000 7ff27d434640 10 monclient: build_initial_monmap 2024-09-15T03:30:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.451+0000 7ff27d434640 10 monclient: monmap: 2024-09-15T03:30:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:22.452165+0000 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:22.452165+0000 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:22.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.451+0000 7ff27d434640 1 Processor -- start 2024-09-15T03:30:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.451+0000 7ff27d434640 1 -- start start 2024-09-15T03:30:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.451+0000 7ff27d434640 10 monclient: init 2024-09-15T03:30:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff27809ec50 0x7ff278131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): picked mon.noname-c con 0x7ff27809ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2781321a0 0x7ff278068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): picked mon.noname-b con 0x7ff2781321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff27809f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff278067420 con 0x7ff27809f020 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27809fdc0 con 0x7ff27809ec50 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780a0510 con 0x7ff2781321a0 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff27d434640 10 monclient(hunting): authenticate will time out at 826.933493s 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff27809ec50 0x7ff278131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff27809ec50 0x7ff278131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46690/0 (socket says 172.21.15.181:46690) 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2777fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7ff27809f020 0x7ff278069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39652/0 (socket says 172.21.15.181:39652) 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 -- 172.21.15.181:0/3489468649 learned_addr learned my addr 172.21.15.181:0/3489468649 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): get_auth_request con 0x7ff27809ec50 auth_method 0 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 1 --2- 172.21.15.181:0/3489468649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2781321a0 0x7ff278068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): get_auth_request con 0x7ff2781321a0 auth_method 0 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff2767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient(hunting): handle_auth_done global_id 44119 payload 274 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 -- 172.21.15.181:0/3489468649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2781321a0 msgr2=0x7ff278068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 --2- 172.21.15.181:0/3489468649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2781321a0 0x7ff278068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 -- 172.21.15.181:0/3489468649 >> v1:172.21.15.181:6789/0 conn(0x7ff27809f020 legacy=0x7ff278069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 monclient: found mon.noname-c 2024-09-15T03:30:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:22.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.452+0000 7ff276ffd640 1 -- 172.21.15.181:0/3489468649 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780a0200 con 0x7ff27809ec50 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.455+0000 7ff276ffd640 1 -- 172.21.15.181:0/3489468649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff27809ec50 msgr2=0x7ff278131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.455+0000 7ff276ffd640 1 --2- 172.21.15.181:0/3489468649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff27809ec50 0x7ff278131c60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff268009910 tx=0x7ff2680091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1f134dbbdefcb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 1 -- 172.21.15.181:0/3489468649 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26802aea0 con 0x7ff27809ec50 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: dump: 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 1 -- 172.21.15.181:0/3489468649 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff268032440 con 0x7ff27809ec50 2024-09-15T03:30:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 5 monclient: authenticate success, global_id 44119 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 1 -- 172.21.15.181:0/3489468649 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2680327e0 con 0x7ff27809ec50 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 4 set_mon_vals no callback set 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff275ffb640 10 monclient: dump: 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 10 monclient: shutdown 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 -- 172.21.15.181:0/3489468649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff27809ec50 msgr2=0x7ff278131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 --2- 172.21.15.181:0/3489468649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff27809ec50 0x7ff278131c60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff268009910 tx=0x7ff2680091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 -- 172.21.15.181:0/3489468649 shutdown_connections 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 --2- 172.21.15.181:0/3489468649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2781321a0 0x7ff278068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 --2- 172.21.15.181:0/3489468649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff27809ec50 0x7ff278131c60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 -- 172.21.15.181:0/3489468649 >> 172.21.15.181:0/3489468649 conn(0x7ff2780a5d40 msgr2=0x7ff2780a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 -- 172.21.15.181:0/3489468649 shutdown_connections 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.456+0000 7ff27d434640 1 -- 172.21.15.181:0/3489468649 wait complete. 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.457+0000 7ff27d434640 10 monclient: build_initial_monmap 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.457+0000 7ff27d434640 10 monclient: monmap: 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:22.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.457+0000 7ff27d434640 1 Processor -- start 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.457+0000 7ff27d434640 1 -- start start 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.457+0000 7ff27d434640 10 monclient: init 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff27809ec50 0x7ff27809aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): picked mon.noname-a con 0x7ff27809ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff27809f020 0x7ff278130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): picked mon.noname-c con 0x7ff27809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 0x7ff2781410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): picked mon.noname-b con 0x7ff2781321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27811dc50 con 0x7ff27809ec50 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27811df30 con 0x7ff2781321a0 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27811ccf0 con 0x7ff27809f020 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff27d434640 10 monclient(hunting): authenticate will time out at 826.939346s 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 0x7ff2781410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 0x7ff2781410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46692/0 (socket says 172.21.15.181:46692) 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 -- 172.21.15.181:0/3329236061 learned_addr learned my addr 172.21.15.181:0/3329236061 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): get_auth_request con 0x7ff2781321a0 auth_method 0 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff27809f020 0x7ff278130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): get_auth_request con 0x7ff27809f020 auth_method 0 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff27809ec50 0x7ff27809aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): get_auth_request con 0x7ff27809ec50 auth_method 0 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff276ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient(hunting): handle_auth_done global_id 44122 payload 824 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 -- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff27809f020 msgr2=0x7ff278130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff27809f020 0x7ff278130b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 -- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff27809ec50 msgr2=0x7ff27809aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff27809ec50 0x7ff27809aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.458+0000 7ff2777fe640 1 -- 172.21.15.181:0/3329236061 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268009450 con 0x7ff2781321a0 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff2777fe640 1 -- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 msgr2=0x7ff2781410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff2777fe640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 0x7ff2781410e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff270007fd0 tx=0x7ff270004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000ce40 con 0x7ff2781321a0 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: dump: 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: _finish_auth 0 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 5 monclient: authenticate success, global_id 44122 2024-09-15T03:30:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000a640 con 0x7ff2781321a0 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 10 monclient: _renew_subs 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000aa00 con 0x7ff2781321a0 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27809b940 con 0x7ff2781321a0 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff25effd640 10 monclient: dump: 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff2757fa640 4 set_mon_vals no callback set 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 10 monclient: _renew_subs 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.461+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27811bc20 con 0x7ff2781321a0 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff2700070f0 con 0x7ff2781321a0 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff27d434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff27d434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff27d434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780008d0 con 0x7ff2781321a0 2024-09-15T03:30:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff25effd640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff264038790 0x7ff26403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4434+0+0 (secure 0 0 0) 0x7ff27004b720 con 0x7ff2781321a0 2024-09-15T03:30:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff276ffd640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff264038790 0x7ff26403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.462+0000 7ff276ffd640 10 monclient: get_auth_request con 0x7ff264038790 auth_method 0 2024-09-15T03:30:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.464+0000 7ff276ffd640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff264038790 0x7ff26403ac40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff268031000 tx=0x7ff268032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.466+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff27000ae30 con 0x7ff2781321a0 2024-09-15T03:30:22.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.466+0000 7ff25effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:22.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.466+0000 7ff25effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.557+0000 7fdccbf5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:30:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.557+0000 7fdccbf5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:30:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.557+0000 7fdccbf5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.557+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fdcc404bf60 con 0x7fdcc4131ab0 2024-09-15T03:30:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fdcb5ffb640 1 -- 172.21.15.153:0/2403489152 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fdcbc012760 con 0x7fdcc4131ab0 2024-09-15T03:30:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fdcb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:30:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fdcb5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:22.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=10,kv=8,logm=69} crc {auth=2297201413,config=3140514009,health=2439085623,kv=3235807718,logm=981886922}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=2297201413,config=3140514009,health=2609900051,kv=3235807718,logm=3114641932}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=10,kv=8,logm=69} crc {auth=2297201413,config=3140514009,health=2439085623,kv=3235807718,logm=981886922}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.558+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=11,kv=8,logm=69} crc {auth=2682161071,config=3140514009,health=344943315,kv=3235807718,logm=2966668237}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.559+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.559+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055982789}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.559+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4262776877}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.559+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.559+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055982789}) 2024-09-15T03:30:22.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.559+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156023757}) 2024-09-15T03:30:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdca8038740 msgr2=0x7fdca803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdca8038740 0x7fdca803abf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009c90 tx=0x7fdcb80090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 10 monclient: shutdown 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 msgr2=0x7fdcc411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 0x7fdcc411b6b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009fd0 tx=0x7fdcbc00b040 comp rx=0 tx=0).stop 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 shutdown_connections 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdca8038740 0x7fdca803abf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc4131ab0 0x7fdcc411b6b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc409f220 0x7fdcc4130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 --2- 172.21.15.153:0/2403489152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcc409e910 0x7fdcc409ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 >> 172.21.15.153:0/2403489152 conn(0x7fdcc4067d30 msgr2=0x7fdcc40a3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 shutdown_connections 2024-09-15T03:30:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:22.561+0000 7fdccbf5d640 1 -- 172.21.15.153:0/2403489152 wait complete. 2024-09-15T03:30:22.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.566+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.566+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944655488}) 2024-09-15T03:30:22.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.566+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1255828520}) 2024-09-15T03:30:22.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.566+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.566+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944655488}) 2024-09-15T03:30:22.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.566+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=197606107}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.567+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.567+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346768325}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.567+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113007509}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.567+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.567+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346768325}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.567+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984696454}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.568+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.568+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712666069}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.568+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2470828161}) 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.568+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.568+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712666069}) 2024-09-15T03:30:22.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.568+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2682998079}) 2024-09-15T03:30:22.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.569+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.569+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=87,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1258431334,mdsmap=1531151339,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977773,mgrstat=3368944740}) 2024-09-15T03:30:22.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.569+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=4} crc {logm=3902592304,mdsmap=1531151339,mgr=2938112866,mgr_metadata=2280977772,mgrstat=601415842}) 2024-09-15T03:30:22.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.569+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.569+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=87,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1258431334,mdsmap=1531151339,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977773,mgrstat=3368944740}) 2024-09-15T03:30:22.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.569+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=193528893,mdsmap=1531151339,mgr=3735254825,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1367564362}) 2024-09-15T03:30:22.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.570+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.570+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=75} crc {mgrstat=588893230,mon_config_key=4117641549,monmap=244636553,osd_metadata=3163829895,osd_pg_creating=1744246674,osd_snap=454132730,osdmap=3749435361}) 2024-09-15T03:30:22.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.570+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2478655059,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1744246674,osd_snap=454132731,osdmap=4250305790}) 2024-09-15T03:30:22.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.570+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:30:22.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.570+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=75} crc {mgrstat=588893230,mon_config_key=4117641549,monmap=244636553,osd_metadata=3163829895,osd_pg_creating=1744246674,osd_snap=454132730,osdmap=3749435361}) 2024-09-15T03:30:22.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:30:22.570+0000 7fb6abbf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=3117271040,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1744246674,osd_snap=454132730,osdmap=2534302211}) 2024-09-15T03:30:22.574 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:30:22.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.626+0000 7ff27d434640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:22.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.626+0000 7ff27d434640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:22.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.626+0000 7ff27d434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:22.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:22.626+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7ff278064900 con 0x7ff2781321a0 2024-09-15T03:30:23.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.176+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v41) v1 ==== 126+0+0 (secure 0 0 0) 0x7ff270010770 con 0x7ff2781321a0 2024-09-15T03:30:23.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.176+0000 7ff25effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.176+0000 7ff25effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:23.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.274+0000 7ff27d434640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.274+0000 7ff27d434640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.274+0000 7ff27d434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.274+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7ff27811c720 con 0x7ff2781321a0 2024-09-15T03:30:23.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.275+0000 7ff25effd640 1 -- 172.21.15.181:0/3329236061 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (secure 0 0 0) 0x7ff270010950 con 0x7ff2781321a0 2024-09-15T03:30:23.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.275+0000 7ff25effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.275+0000 7ff25effd640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-15T03:30:23.276 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo already exists 2024-09-15T03:30:23.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff264038790 msgr2=0x7ff26403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff264038790 0x7ff26403ac40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff268031000 tx=0x7ff268032000 comp rx=0 tx=0).stop 2024-09-15T03:30:23.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 10 monclient: shutdown 2024-09-15T03:30:23.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:23.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 msgr2=0x7ff2781410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 0x7ff2781410e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff270007fd0 tx=0x7ff270004ef0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 shutdown_connections 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff264038790 0x7ff26403ac40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2781321a0 0x7ff2781410e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff27809f020 0x7ff278130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 --2- 172.21.15.181:0/3329236061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff27809ec50 0x7ff27809aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 >> 172.21.15.181:0/3329236061 conn(0x7ff2780a5d40 msgr2=0x7ff2780a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 shutdown_connections 2024-09-15T03:30:23.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.277+0000 7ff27d434640 1 -- 172.21.15.181:0/3329236061 wait complete. 2024-09-15T03:30:23.286 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.361+0000 7f6e1a620640 10 monclient: get_monmap_and_config 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.361+0000 7f6e1a620640 10 monclient: build_initial_monmap 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.361+0000 7f6e1a620640 10 monclient: monmap: 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:23.362008+0000 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:23.362008+0000 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.363 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:23.363 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:23.363 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:23.363 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:23.363 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:23.363 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 Processor -- start 2024-09-15T03:30:23.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 -- start start 2024-09-15T03:30:23.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient: init 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e141321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1409f320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e14070920 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6e14071400 0x7f6e14138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): picked mon.noname-a con 0x7f6e14071400 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e14096b60 con 0x7f6e14070920 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1409e4d0 con 0x7f6e14071400 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1409e7b0 con 0x7f6e1409f320 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e1a620640 10 monclient(hunting): authenticate will time out at 827.843695s 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e0bfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f6e14070920 0x7f6e14070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39662/0 (socket says 172.21.15.181:39662) 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e0bfff640 1 -- 172.21.15.181:0/4155178325 learned_addr learned my addr 172.21.15.181:0/4155178325 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e18bbd640 1 --2- 172.21.15.181:0/4155178325 >> v2:172.21.15.153:3300/0 conn(0x7f6e14071400 0x7f6e14138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e13fff640 1 --2- 172.21.15.181:0/4155178325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e141321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e13fff640 10 monclient(hunting): get_auth_request con 0x7f6e1409f320 auth_method 0 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.362+0000 7f6e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): get_auth_request con 0x7f6e14071400 auth_method 0 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e18bbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 10 monclient(hunting): handle_auth_done global_id 4748 payload 274 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 1 -- 172.21.15.181:0/4155178325 >> v2:172.21.15.153:3300/0 conn(0x7f6e14071400 msgr2=0x7f6e14138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 1 --2- 172.21.15.181:0/4155178325 >> v2:172.21.15.153:3300/0 conn(0x7f6e14071400 0x7f6e14138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 1 -- 172.21.15.181:0/4155178325 >> v1:172.21.15.181:6789/0 conn(0x7f6e14070920 legacy=0x7f6e14070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 1 monclient: found mon.noname-b 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.363+0000 7f6e13fff640 1 -- 172.21.15.181:0/4155178325 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1409e1f0 con 0x7f6e1409f320 2024-09-15T03:30:23.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e13fff640 1 -- 172.21.15.181:0/4155178325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 msgr2=0x7f6e141321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:23.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e13fff640 1 --2- 172.21.15.181:0/4155178325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e141321b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009850 tx=0x7f6e040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511a2d8fd6e6494 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 1 -- 172.21.15.181:0/4155178325 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0402adf0 con 0x7f6e1409f320 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: dump: 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.374 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 1 -- 172.21.15.181:0/4155178325 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e04032440 con 0x7f6e1409f320 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e1a620640 5 monclient: authenticate success, global_id 4748 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e1a620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e1a620640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e1a620640 4 set_mon_vals no callback set 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 1 -- 172.21.15.181:0/4155178325 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e040327e0 con 0x7f6e1409f320 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.366+0000 7f6e137fe640 10 monclient: dump: 2024-09-15T03:30:23.375 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 10 monclient: shutdown 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 -- 172.21.15.181:0/4155178325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 msgr2=0x7f6e141321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 --2- 172.21.15.181:0/4155178325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e141321b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009850 tx=0x7f6e040090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 -- 172.21.15.181:0/4155178325 shutdown_connections 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 --2- 172.21.15.181:0/4155178325 >> v2:172.21.15.153:3300/0 conn(0x7f6e14071400 0x7f6e14138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 --2- 172.21.15.181:0/4155178325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e141321b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 -- 172.21.15.181:0/4155178325 >> 172.21.15.181:0/4155178325 conn(0x7f6e140a2b70 msgr2=0x7f6e1409ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 -- 172.21.15.181:0/4155178325 shutdown_connections 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 -- 172.21.15.181:0/4155178325 wait complete. 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 10 monclient: build_initial_monmap 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 10 monclient: monmap: 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:23.377 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 Processor -- start 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 1 -- start start 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.367+0000 7f6e1a620640 10 monclient: init 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e14070920 0x7f6e141367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): picked mon.noname-a con 0x7f6e14070920 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 0x7f6e14136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): picked mon.noname-b con 0x7f6e14071400 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e14137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1409f320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141189b0 con 0x7f6e14070920 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141387e0 con 0x7f6e14071400 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14097230 con 0x7f6e1409f320 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e1a620640 10 monclient(hunting): authenticate will time out at 827.849335s 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 0x7f6e14136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 0x7f6e14136d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46700/0 (socket says 172.21.15.181:46700) 2024-09-15T03:30:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 -- 172.21.15.181:0/2184038556 learned_addr learned my addr 172.21.15.181:0/2184038556 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): get_auth_request con 0x7f6e14071400 auth_method 0 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e14070920 0x7f6e141367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e14137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): get_auth_request con 0x7f6e14070920 auth_method 0 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): get_auth_request con 0x7f6e1409f320 auth_method 0 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e18bbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient(hunting): handle_auth_done global_id 44125 payload 824 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 -- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 msgr2=0x7f6e14137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e14137240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 -- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e14070920 msgr2=0x7f6e141367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e14070920 0x7f6e141367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 monclient: found mon.noname-b 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.368+0000 7f6e0bfff640 1 -- 172.21.15.181:0/2184038556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04009390 con 0x7f6e14071400 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e0bfff640 1 -- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 msgr2=0x7f6e14136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e0bfff640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 0x7f6e14136d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00c610 tx=0x7f6dfc004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc005cb0 con 0x7f6e14071400 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: dump: 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.382 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc005e30 con 0x7f6e14071400 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 5 monclient: authenticate success, global_id 44125 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 10 monclient: _renew_subs 2024-09-15T03:30:23.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1409d5b0 con 0x7f6e14071400 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc007340 con 0x7f6e14071400 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 10 monclient: dump: 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e12ffd640 4 set_mon_vals no callback set 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 10 monclient: _renew_subs 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14138260 con 0x7f6e14071400 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.371+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6dfc01b400 con 0x7f6e14071400 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e1a620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e1a620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e1a620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e10ff9640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6df4038740 0x7f6df403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e140008d0 con 0x7f6e14071400 2024-09-15T03:30:23.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f6dfc04b560 con 0x7f6e14071400 2024-09-15T03:30:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e13fff640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6df4038740 0x7f6df403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.372+0000 7f6e13fff640 10 monclient: get_auth_request con 0x7f6df4038740 auth_method 0 2024-09-15T03:30:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.374+0000 7f6e13fff640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6df4038740 0x7f6df403abf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e140719b0 tx=0x7f6e04032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.376+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6dfc012760 con 0x7f6e14071400 2024-09-15T03:30:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.376+0000 7f6e10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.376+0000 7f6e10ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:23.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.532+0000 7f6e1a620640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.532+0000 7f6e1a620640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.532+0000 7f6e1a620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.532+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f6e14064b20 con 0x7f6e14071400 2024-09-15T03:30:23.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.534+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (secure 0 0 0) 0x7f6dfc01a360 con 0x7f6e14071400 2024-09-15T03:30:23.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.534+0000 7f6e10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.534+0000 7f6e10ff9640 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-15T03:30:23.575 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:30:23.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.627+0000 7f6e1a620640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.627+0000 7f6e1a620640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.627+0000 7f6e1a620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.627+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f6e141309a0 con 0x7f6e14071400 2024-09-15T03:30:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.628+0000 7f6e10ff9640 1 -- 172.21.15.181:0/2184038556 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (secure 0 0 0) 0x7f6dfc0410c0 con 0x7f6e14071400 2024-09-15T03:30:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.628+0000 7f6e10ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-15T03:30:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.628+0000 7f6e10ff9640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-15T03:30:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo already exists 2024-09-15T03:30:23.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6df4038740 msgr2=0x7f6df403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6df4038740 0x7f6df403abf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e140719b0 tx=0x7f6e04032000 comp rx=0 tx=0).stop 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 10 monclient: shutdown 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 msgr2=0x7f6e14136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 0x7f6e14136d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00c610 tx=0x7f6dfc004960 comp rx=0 tx=0).stop 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 shutdown_connections 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6df4038740 0x7f6df403abf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e1409f320 0x7f6e14137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e14071400 0x7f6e14136d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 --2- 172.21.15.181:0/2184038556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e14070920 0x7f6e141367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 >> 172.21.15.181:0/2184038556 conn(0x7f6e140a2b70 msgr2=0x7f6e1409b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 shutdown_connections 2024-09-15T03:30:23.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.630+0000 7f6e1a620640 1 -- 172.21.15.181:0/2184038556 wait complete. 2024-09-15T03:30:23.640 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-15T03:30:23.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.666+0000 7f80abf9d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.666+0000 7f80abf9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.666+0000 7f80abf9d640 10 monclient: monmap: 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:23.668530+0000 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:23.668530+0000 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:23.669 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:23.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:23.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 1 Processor -- start 2024-09-15T03:30:23.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 1 -- start start 2024-09-15T03:30:23.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 10 monclient: init 2024-09-15T03:30:23.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:23.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T03:30:23.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f80a4096c60 0x7f80a409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f80a4096c60 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 0x7f80a4097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.667+0000 7f80abf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f80a4097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80a412e5f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a409d2e0 con 0x7f80a412e5f0 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a409ac00 con 0x7f80a4096c60 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a409d6b0 con 0x7f80a4097030 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80abf9d640 10 monclient(hunting): authenticate will time out at 821.308058s 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 0x7f80a4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f80a4096c60 0x7f80a409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 0x7f80a4097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52440/0 (socket says 172.21.15.153:52440) 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f80a4096c60 0x7f80a409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35922/0 (socket says 172.21.15.153:35922) 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 -- 172.21.15.153:0/3678334124 learned_addr learned my addr 172.21.15.153:0/3678334124 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80aa53a640 1 --1- 172.21.15.153:0/3678334124 >> v1:172.21.15.153:6789/0 conn(0x7f80a412e5f0 0x7f80a412ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44876/0 (socket says 172.21.15.153:44876) 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): get_auth_request con 0x7f80a4097030 auth_method 0 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): get_auth_request con 0x7f80a4096c60 auth_method 0 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient(hunting): handle_auth_done global_id 4754 payload 274 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 -- 172.21.15.153:0/3678334124 >> v2:172.21.15.153:3300/0 conn(0x7f80a4096c60 msgr2=0x7f80a409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 --2- 172.21.15.153:0/3678334124 >> v2:172.21.15.153:3300/0 conn(0x7f80a4096c60 0x7f80a409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 -- 172.21.15.153:0/3678334124 >> v1:172.21.15.153:6789/0 conn(0x7f80a412e5f0 legacy=0x7f80a412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient: _finish_hunting 0 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 monclient: found mon.noname-b 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.668+0000 7f80a9538640 1 -- 172.21.15.153:0/3678334124 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a40992f0 con 0x7f80a4097030 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a9538640 1 -- 172.21.15.153:0/3678334124 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 msgr2=0x7f80a4097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a9538640 1 --2- 172.21.15.153:0/3678334124 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 0x7f80a4097430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8094009910 tx=0x7f80940091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aabe03e816ef9564 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 1 -- 172.21.15.153:0/3678334124 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809402aea0 con 0x7f80a4097030 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: dump: 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: _finish_auth 0 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 1 -- 172.21.15.153:0/3678334124 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8094032440 con 0x7f80a4097030 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 5 monclient: authenticate success, global_id 4754 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 4 set_mon_vals no callback set 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 1 -- 172.21.15.153:0/3678334124 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094032820 con 0x7f80a4097030 2024-09-15T03:30:23.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80a8d37640 10 monclient: dump: 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:23.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 10 monclient: shutdown 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 1 -- 172.21.15.153:0/3678334124 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 msgr2=0x7f80a4097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 1 --2- 172.21.15.153:0/3678334124 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 0x7f80a4097430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8094009910 tx=0x7f80940091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 1 -- 172.21.15.153:0/3678334124 shutdown_connections 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 1 --2- 172.21.15.153:0/3678334124 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4097030 0x7f80a4097430 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 1 --2- 172.21.15.153:0/3678334124 >> v2:172.21.15.153:3300/0 conn(0x7f80a4096c60 0x7f80a409a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.671+0000 7f80abf9d640 1 -- 172.21.15.153:0/3678334124 >> 172.21.15.153:0/3678334124 conn(0x7f80a40672b0 msgr2=0x7f80a40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 1 -- 172.21.15.153:0/3678334124 shutdown_connections 2024-09-15T03:30:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 1 -- 172.21.15.153:0/3678334124 wait complete. 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 10 monclient: monmap: 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 1 Processor -- start 2024-09-15T03:30:23.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 1 -- start start 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.672+0000 7f80abf9d640 10 monclient: init 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 0x7f80a413f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f80a4096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80a4097030 0x7f80a4136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f80a4097030 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a412e5f0 0x7f80a4137480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f80a412e5f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a412e080 con 0x7f80a412e5f0 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a409add0 con 0x7f80a4097030 2024-09-15T03:30:23.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a409ed20 con 0x7f80a4096c60 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80abf9d640 10 monclient(hunting): authenticate will time out at 821.313393s 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 0x7f80a413f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 0x7f80a413f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52456/0 (socket says 172.21.15.153:52456) 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 -- 172.21.15.153:0/1849332873 learned_addr learned my addr 172.21.15.153:0/1849332873 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): get_auth_request con 0x7f80a4096c60 auth_method 0 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a412e5f0 0x7f80a4137480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): get_auth_request con 0x7f80a412e5f0 auth_method 0 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9538640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80a4097030 0x7f80a4136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9538640 10 monclient(hunting): get_auth_request con 0x7f80a4097030 auth_method 0 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9538640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient(hunting): handle_auth_done global_id 4757 payload 824 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 -- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80a4097030 msgr2=0x7f80a4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80a4097030 0x7f80a4136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 -- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a412e5f0 msgr2=0x7f80a4137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a412e5f0 0x7f80a4137480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient: _finish_hunting 0 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 monclient: found mon.noname-c 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80a9d39640 1 -- 172.21.15.153:0/1849332873 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8094009450 con 0x7f80a4096c60 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.673+0000 7f80aa53a640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a412e5f0 0x7f80a4137480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.675+0000 7f80a9d39640 1 -- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 msgr2=0x7f80a413f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.675+0000 7f80a9d39640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 0x7f80a413f150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80a000ea20 tx=0x7f80a00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0009600 con 0x7f80a4096c60 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: dump: 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.690 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 5 monclient: authenticate success, global_id 4757 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a000cc20 con 0x7f80a4096c60 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 10 monclient: _renew_subs 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a412ceb0 con 0x7f80a4096c60 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0013470 con 0x7f80a4096c60 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f8099ffb640 10 monclient: dump: 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 10 monclient: _renew_subs 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f809bfff640 4 set_mon_vals no callback set 2024-09-15T03:30:23.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.676+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a413f690 con 0x7f80a4096c60 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f80a0013860 con 0x7f80a4096c60 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f80abf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f80abf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f80abf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f8099ffb640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8088038740 0x7f808803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a404bf60 con 0x7f80a4096c60 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f80a004bd30 con 0x7f80a4096c60 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f80a9538640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8088038740 0x7f808803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.677+0000 7f80a9538640 10 monclient: get_auth_request con 0x7f8088038740 auth_method 0 2024-09-15T03:30:23.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.680+0000 7f80a9538640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8088038740 0x7f808803abf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8094034000 tx=0x7f8094032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.681+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f80a00236b0 con 0x7f80a4096c60 2024-09-15T03:30:23.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.681+0000 7f8099ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.681+0000 7f8099ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:23.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.717+0000 7f1ff22c0640 10 monclient: get_monmap_and_config 2024-09-15T03:30:23.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.717+0000 7f1ff22c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:23.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.717+0000 7f1ff22c0640 10 monclient: monmap: 2024-09-15T03:30:23.718 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:23.718 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.718 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:23.718043+0000 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:23.718043+0000 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:23.719 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.717+0000 7f1ff22c0640 1 Processor -- start 2024-09-15T03:30:23.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 1 -- start start 2024-09-15T03:30:23.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient: init 2024-09-15T03:30:23.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:23.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1fec09ec50 0x7f1fec131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1fec09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fec1321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1fec09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec067420 con 0x7f1fec1321a0 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec09fdc0 con 0x7f1fec09ec50 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0a0510 con 0x7f1fec09f020 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff22c0640 10 monclient(hunting): authenticate will time out at 828.199569s 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1fec09ec50 0x7f1fec131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1feb7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1fec1321a0 0x7f1fec068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:34588/0 (socket says 172.21.15.181:34588) 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1feb7fe640 1 -- 172.21.15.181:0/812162575 learned_addr learned my addr 172.21.15.181:0/812162575 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): get_auth_request con 0x7f1fec09ec50 auth_method 0 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 1 --2- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): get_auth_request con 0x7f1fec09f020 auth_method 0 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1febfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.718+0000 7f1ff085d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 10 monclient(hunting): handle_auth_done global_id 34461 payload 274 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 1 -- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 msgr2=0x7f1fec069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 1 --2- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 1 -- 172.21.15.181:0/812162575 >> v1:172.21.15.153:6789/0 conn(0x7f1fec1321a0 legacy=0x7f1fec068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.719+0000 7f1febfff640 1 -- 172.21.15.181:0/812162575 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0a0200 con 0x7f1fec09ec50 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.721+0000 7f1febfff640 1 -- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 msgr2=0x7f1fec131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:23.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.721+0000 7f1febfff640 1 --2- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec131c60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009850 tx=0x7f1fdc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a9b44d92526cbb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 1 -- 172.21.15.181:0/812162575 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc02adf0 con 0x7f1fec09ec50 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: dump: 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 1 -- 172.21.15.181:0/812162575 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc032440 con 0x7f1fec09ec50 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 5 monclient: authenticate success, global_id 34461 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 1 -- 172.21.15.181:0/812162575 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0327e0 con 0x7f1fec09ec50 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 4 set_mon_vals no callback set 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1feaffd640 10 monclient: dump: 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 10 monclient: shutdown 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:23.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 -- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 msgr2=0x7f1fec131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec131c60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009850 tx=0x7f1fdc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 -- 172.21.15.181:0/812162575 shutdown_connections 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/812162575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec131c60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 -- 172.21.15.181:0/812162575 >> 172.21.15.181:0/812162575 conn(0x7f1fec0a5d40 msgr2=0x7f1fec0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 -- 172.21.15.181:0/812162575 shutdown_connections 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.722+0000 7f1ff22c0640 1 -- 172.21.15.181:0/812162575 wait complete. 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.723+0000 7f1ff22c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.723+0000 7f1ff22c0640 10 monclient: monmap: 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.723+0000 7f1ff22c0640 1 Processor -- start 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.723+0000 7f1ff22c0640 1 -- start start 2024-09-15T03:30:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.723+0000 7f1ff22c0640 10 monclient: init 2024-09-15T03:30:23.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:23.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:23.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1fec09ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1fec09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fec1321a0 0x7f1fec11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1fec1321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec11dc70 con 0x7f1fec09ec50 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec11e1c0 con 0x7f1fec1321a0 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec141dd0 con 0x7f1fec09f020 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff22c0640 10 monclient(hunting): authenticate will time out at 828.205201s 2024-09-15T03:30:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fec1321a0 0x7f1fec11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec09aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:53548/0 (socket says 172.21.15.181:53548) 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fec1321a0 0x7f1fec11b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46702/0 (socket says 172.21.15.181:46702) 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec130ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59958/0 (socket says 172.21.15.181:59958) 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 -- 172.21.15.181:0/1283334499 learned_addr learned my addr 172.21.15.181:0/1283334499 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): get_auth_request con 0x7f1fec09ec50 auth_method 0 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): get_auth_request con 0x7f1fec09f020 auth_method 0 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): get_auth_request con 0x7f1fec1321a0 auth_method 0 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1ff085d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient(hunting): handle_auth_done global_id 34464 payload 824 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 -- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 msgr2=0x7f1fec130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 -- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fec1321a0 msgr2=0x7f1fec11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fec1321a0 0x7f1fec11b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 monclient: found mon.noname-a 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1febfff640 1 -- 172.21.15.181:0/1283334499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc009390 con 0x7f1fec09ec50 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.724+0000 7f1feb7fe640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.726+0000 7f1febfff640 1 -- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 msgr2=0x7f1fec09aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.726+0000 7f1febfff640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec09aa60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098b0 tx=0x7f1fdc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc031a10 con 0x7f1fec09ec50 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: dump: 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.741 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc031b70 con 0x7f1fec09ec50 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 5 monclient: authenticate success, global_id 34464 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc03c400 con 0x7f1fec09ec50 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 10 monclient: dump: 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.742 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 10 monclient: _renew_subs 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fea7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:23.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec11dee0 con 0x7f1fec09ec50 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 10 monclient: _renew_subs 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec11cca0 con 0x7f1fec09ec50 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.727+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1fdc03c830 con 0x7f1fec09ec50 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1ff22c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1ff22c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1ff22c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec11ce90 con 0x7f1fec09ec50 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1fe3fff640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fcc038790 0x7f1fcc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f1fdc06c260 con 0x7f1fec09ec50 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1feb7fe640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fcc038790 0x7f1fcc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.728+0000 7f1feb7fe640 10 monclient: get_auth_request con 0x7f1fcc038790 auth_method 0 2024-09-15T03:30:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.730+0000 7f1feb7fe640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fcc038790 0x7f1fcc03ac40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0630f0 tx=0x7f1fd4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:23.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.732+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1fdc0426c0 con 0x7f1fec09ec50 2024-09-15T03:30:23.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.732+0000 7f1fe3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:23.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.732+0000 7f1fe3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:23.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.890+0000 7f1ff22c0640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T03:30:23.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.890+0000 7f1ff22c0640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T03:30:23.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.890+0000 7f1ff22c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:23.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:23.890+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f1fec0008d0 con 0x7f1fec09ec50 2024-09-15T03:30:23.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f80abf9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:23.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f80abf9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:23.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f80abf9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:23.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f80a412c200 con 0x7f80a4096c60 2024-09-15T03:30:23.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f8099ffb640 1 -- 172.21.15.153:0/1849332873 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f80a00232f0 con 0x7f80a4096c60 2024-09-15T03:30:23.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f8099ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:23.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.898+0000 7f8099ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:23.901 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8088038740 msgr2=0x7f808803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8088038740 0x7f808803abf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8094034000 tx=0x7f8094032000 comp rx=0 tx=0).stop 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 10 monclient: shutdown 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 msgr2=0x7f80a413f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 0x7f80a413f150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80a000ea20 tx=0x7f80a00090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 shutdown_connections 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8088038740 0x7f808803abf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a412e5f0 0x7f80a4137480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80a4097030 0x7f80a4136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 --2- 172.21.15.153:0/1849332873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80a4096c60 0x7f80a413f150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:23.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.900+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 >> 172.21.15.153:0/1849332873 conn(0x7f80a40672b0 msgr2=0x7f80a409d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:23.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.901+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 shutdown_connections 2024-09-15T03:30:23.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:23.901+0000 7f80abf9d640 1 -- 172.21.15.153:0/1849332873 wait complete. 2024-09-15T03:30:23.913 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:30:23.913 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:30:23.913 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:30:23.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:30:24.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.005+0000 7f087c31d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.005+0000 7f087c31d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.005+0000 7f087c31d640 10 monclient: monmap: 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:24.007657+0000 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:24.007657+0000 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:24.009 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:24.009 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.006+0000 7f087c31d640 1 Processor -- start 2024-09-15T03:30:24.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.006+0000 7f087c31d640 1 -- start start 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.006+0000 7f087c31d640 10 monclient: init 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0874131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f087409fda0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 0x7f0874139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient(hunting): picked mon.noname-b con 0x7f087409f0d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087409f510 con 0x7f087409fda0 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08740a1180 con 0x7f0874131980 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08740a1460 con 0x7f087409f0d0 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087c31d640 10 monclient(hunting): authenticate will time out at 821.647137s 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 0x7f0874139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f08798b8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f087409fda0 0x7f087409e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44884/0 (socket says 172.21.15.153:44884) 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 0x7f0874139cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52460/0 (socket says 172.21.15.153:52460) 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f08798b8640 1 -- 172.21.15.153:0/3067108971 learned_addr learned my addr 172.21.15.153:0/3067108971 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): get_auth_request con 0x7f087409f0d0 auth_method 0 2024-09-15T03:30:24.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient(hunting): handle_auth_done global_id 4766 payload 274 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 1 -- 172.21.15.153:0/3067108971 >> v1:172.21.15.181:6789/0 conn(0x7f0874131980 legacy=0x7f0874131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 1 -- 172.21.15.153:0/3067108971 >> v1:172.21.15.153:6789/0 conn(0x7f087409fda0 legacy=0x7f087409e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 1 monclient: found mon.noname-b 2024-09-15T03:30:24.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.007+0000 7f087a8ba640 1 -- 172.21.15.153:0/3067108971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874132460 con 0x7f087409f0d0 2024-09-15T03:30:24.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087a8ba640 1 -- 172.21.15.153:0/3067108971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 msgr2=0x7f0874139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087a8ba640 1 --2- 172.21.15.153:0/3067108971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 0x7f0874139cd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f086c009e90 tx=0x7f086c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12216c9bd83d8d58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 1 -- 172.21.15.153:0/3067108971 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c007840 con 0x7f087409f0d0 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: dump: 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.024 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 1 -- 172.21.15.153:0/3067108971 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c02d050 con 0x7f087409f0d0 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087c31d640 5 monclient: authenticate success, global_id 4766 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087c31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 1 -- 172.21.15.153:0/3067108971 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c007bd0 con 0x7f087409f0d0 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087c31d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087c31d640 4 set_mon_vals no callback set 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f08790b7640 10 monclient: dump: 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.010+0000 7f087c31d640 10 monclient: shutdown 2024-09-15T03:30:24.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 -- 172.21.15.153:0/3067108971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 msgr2=0x7f0874139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 --2- 172.21.15.153:0/3067108971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 0x7f0874139cd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f086c009e90 tx=0x7f086c009780 comp rx=0 tx=0).stop 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 -- 172.21.15.153:0/3067108971 shutdown_connections 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 --2- 172.21.15.153:0/3067108971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f087409f0d0 0x7f0874139cd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 -- 172.21.15.153:0/3067108971 >> 172.21.15.153:0/3067108971 conn(0x7f0874067bd0 msgr2=0x7f08740a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 -- 172.21.15.153:0/3067108971 shutdown_connections 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 -- 172.21.15.153:0/3067108971 wait complete. 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 10 monclient: monmap: 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:24.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 Processor -- start 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 1 -- start start 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.011+0000 7f087c31d640 10 monclient: init 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f087409f0d0 0x7f0874138410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): picked mon.noname-a con 0x7f087409f0d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f087409fda0 0x7f0874133d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): picked mon.noname-b con 0x7f087409fda0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 0x7f08741342b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): picked mon.noname-c con 0x7f0874131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874135d90 con 0x7f087409f0d0 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874138000 con 0x7f087409fda0 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0874137340 con 0x7f0874131980 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087c31d640 10 monclient(hunting): authenticate will time out at 821.652386s 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 0x7f08741342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f087409f0d0 0x7f0874138410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 0x7f08741342b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52466/0 (socket says 172.21.15.153:52466) 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 learned_addr learned my addr 172.21.15.153:0/3711970971 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): get_auth_request con 0x7f0874131980 auth_method 0 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f087409f0d0 0x7f0874138410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35942/0 (socket says 172.21.15.153:35942) 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): get_auth_request con 0x7f087409f0d0 auth_method 0 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f08798b8640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f087409fda0 0x7f0874133d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f08798b8640 10 monclient(hunting): get_auth_request con 0x7f087409fda0 auth_method 0 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f08798b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f08798b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f08798b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient(hunting): handle_auth_done global_id 4769 payload 824 2024-09-15T03:30:24.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f087409fda0 msgr2=0x7f0874133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f087409fda0 0x7f0874133d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f087409f0d0 msgr2=0x7f0874138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f087409f0d0 0x7f0874138410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 monclient: found mon.noname-c 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c009a20 con 0x7f0874131980 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f08798b8640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f087409fda0 0x7f0874133d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.012+0000 7f087a0b9640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f087409f0d0 0x7f0874138410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 msgr2=0x7f08741342b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087a8ba640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 0x7f08741342b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f086c009de0 tx=0x7f086c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c007ec0 con 0x7f0874131980 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: dump: 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c040070 con 0x7f0874131980 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 5 monclient: authenticate success, global_id 4769 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 10 monclient: _renew_subs 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c045480 con 0x7f0874131980 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087409bf10 con 0x7f0874131980 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f086a7fc640 10 monclient: dump: 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.034 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f08788b6640 4 set_mon_vals no callback set 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 10 monclient: _renew_subs 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0874135890 con 0x7f0874131980 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.015+0000 7f087c31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087c31d640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 0x7f08741347f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087c31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874065e00 con 0x7f08741363e0 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f086c042030 con 0x7f0874131980 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 0x7f08741347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient: get_auth_request con 0x7f08741363e0 auth_method 0 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f086a7fc640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0858038790 0x7f085803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 10 monclient(hunting): handle_auth_done global_id 34470 payload 824 2024-09-15T03:30:24.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f087a0b9640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f087404bf60 con 0x7f08741363e0 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f086c06cfb0 con 0x7f0874131980 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f08798b8640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0858038790 0x7f085803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.016+0000 7f08798b8640 10 monclient: get_auth_request con 0x7f0858038790 auth_method 0 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f087a0b9640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 msgr2=0x7f08741347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f087a0b9640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 0x7f08741347f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f087000fa50 tx=0x7f0870007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000dda0 con 0x7f08741363e0 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 10 monclient: dump: 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.037 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0870014470 con 0x7f08741363e0 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 msgr2=0x7f08741347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 0x7f08741347f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f087000fa50 tx=0x7f0870007e10 comp rx=0 tx=0).stop 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 msgr2=0x7f08741347f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.018+0000 7f086a7fc640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08741363e0 0x7f08741347f0 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f087000fa50 tx=0x7f0870007e10 comp rx=0 tx=0).stop 2024-09-15T03:30:24.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.019+0000 7f08798b8640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0858038790 0x7f085803ac40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0864007fd0 tx=0x7f0864009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087c31d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:24.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087c31d640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 0x7f0874098e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087c31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087409f510 con 0x7f0874098a70 2024-09-15T03:30:24.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 0x7f0874098e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient: get_auth_request con 0x7f0874098a70 auth_method 0 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 10 monclient(hunting): handle_auth_done global_id 34473 payload 824 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.024+0000 7f087a0b9640 1 -- 172.21.15.153:0/3711970971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08740638d0 con 0x7f0874098a70 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087a0b9640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 msgr2=0x7f0874098e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087a0b9640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 0x7f0874098e70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0870019000 tx=0x7f087001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087001ecb0 con 0x7f0874098a70 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 10 monclient: dump: 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f087001ee10 con 0x7f0874098a70 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 msgr2=0x7f0874098e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 0x7f0874098e70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0870019000 tx=0x7f087001a000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 msgr2=0x7f0874098e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f086a7fc640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0874098a70 0x7f0874098e70 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0870019000 tx=0x7f087001a000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:30:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:24.059 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0858038790 msgr2=0x7f085803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0858038790 0x7f085803ac40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0864007fd0 tx=0x7f0864009040 comp rx=0 tx=0).stop 2024-09-15T03:30:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 10 monclient: shutdown 2024-09-15T03:30:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 msgr2=0x7f08741342b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 --2- 172.21.15.153:0/3711970971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0874131980 0x7f08741342b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f086c009de0 tx=0x7f086c037000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 reap_dead start 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087a8ba640 1 -- 172.21.15.153:0/3711970971 reap_dead start 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 shutdown_connections 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 >> 172.21.15.153:0/3711970971 conn(0x7f0874067bd0 msgr2=0x7f087409e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 shutdown_connections 2024-09-15T03:30:24.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.027+0000 7f087c31d640 1 -- 172.21.15.153:0/3711970971 wait complete. 2024-09-15T03:30:24.061 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:30:24.061 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:30:24.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:30:24.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.154+0000 7f02b9d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:24.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.154+0000 7f02b9d9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.154+0000 7f02b9d9d640 10 monclient: monmap: 2024-09-15T03:30:24.156 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:24.156187+0000 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:24.156187+0000 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:24.157 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 Processor -- start 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 -- start start 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient: init 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f02b4096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02b409aaa0 0x7f02b40971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f02b409aaa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f02b4097720 0x7f02b412e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f02b4097720 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b4099620 con 0x7f02b409aaa0 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b4099340 con 0x7f02b4097720 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4099900 con 0x7f02b4096e10 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b9d9d640 10 monclient(hunting): authenticate will time out at 821.795776s 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52482/0 (socket says 172.21.15.153:52482) 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 1 -- 172.21.15.153:0/1847486965 learned_addr learned my addr 172.21.15.153:0/1847486965 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 10 monclient(hunting): get_auth_request con 0x7f02b4096e10 auth_method 0 2024-09-15T03:30:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/1847486965 >> v2:172.21.15.153:3300/0 conn(0x7f02b409aaa0 0x7f02b40971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b2ffd640 10 monclient(hunting): get_auth_request con 0x7f02b409aaa0 auth_method 0 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.155+0000 7f02b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b3fff640 1 --2- 172.21.15.153:0/1847486965 >> v2:172.21.15.181:3300/0 conn(0x7f02b4097720 0x7f02b412e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b3fff640 10 monclient(hunting): get_auth_request con 0x7f02b4097720 auth_method 0 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_done global_id 34476 payload 274 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 1 -- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 msgr2=0x7f02b409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 1 -- 172.21.15.153:0/1847486965 >> v2:172.21.15.181:3300/0 conn(0x7f02b4097720 msgr2=0x7f02b412e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/1847486965 >> v2:172.21.15.181:3300/0 conn(0x7f02b4097720 0x7f02b412e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 1 monclient: found mon.noname-a 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b2ffd640 1 -- 172.21.15.153:0/1847486965 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b4097b10 con 0x7f02b409aaa0 2024-09-15T03:30:24.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b3fff640 1 --2- 172.21.15.153:0/1847486965 >> v2:172.21.15.181:3300/0 conn(0x7f02b4097720 0x7f02b412e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:24.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.156+0000 7f02b37fe640 1 --2- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.158+0000 7f02b2ffd640 1 -- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b409aaa0 msgr2=0x7f02b40971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.158+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b409aaa0 0x7f02b40971e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02b409aff0 tx=0x7f02a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a35c17d158e4843 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 1 -- 172.21.15.153:0/1847486965 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a802adf0 con 0x7f02b409aaa0 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: dump: 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 5 monclient: authenticate success, global_id 34476 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 1 -- 172.21.15.153:0/1847486965 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a8032440 con 0x7f02b409aaa0 2024-09-15T03:30:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 1 -- 172.21.15.153:0/1847486965 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8032820 con 0x7f02b409aaa0 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 4 set_mon_vals no callback set 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b27fc640 10 monclient: dump: 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.173 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 10 monclient: shutdown 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 -- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b409aaa0 msgr2=0x7f02b40971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b409aaa0 0x7f02b40971e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02b409aff0 tx=0x7f02a80090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 -- 172.21.15.153:0/1847486965 shutdown_connections 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/1847486965 >> v2:172.21.15.181:3300/0 conn(0x7f02b4097720 0x7f02b412e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b409aaa0 0x7f02b40971e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/1847486965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 -- 172.21.15.153:0/1847486965 >> 172.21.15.153:0/1847486965 conn(0x7f02b40672b0 msgr2=0x7f02b40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 -- 172.21.15.153:0/1847486965 shutdown_connections 2024-09-15T03:30:24.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.159+0000 7f02b9d9d640 1 -- 172.21.15.153:0/1847486965 wait complete. 2024-09-15T03:30:24.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.160+0000 7f02b9d9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.160+0000 7f02b9d9d640 10 monclient: monmap: 2024-09-15T03:30:24.175 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.160+0000 7f02b9d9d640 1 Processor -- start 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.160+0000 7f02b9d9d640 1 -- start start 2024-09-15T03:30:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.160+0000 7f02b9d9d640 10 monclient: init 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b412d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f02b4096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4097720 0x7f02b4118450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f02b4097720 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b409aaa0 0x7f02b4118990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f02b409aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b411b820 con 0x7f02b4097720 2024-09-15T03:30:24.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b411a270 con 0x7f02b409aaa0 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b413d820 con 0x7f02b4096e10 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b9d9d640 10 monclient(hunting): authenticate will time out at 821.801137s 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b412d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4097720 0x7f02b4118450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4097720 0x7f02b4118450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35964/0 (socket says 172.21.15.153:35964) 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b412d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52490/0 (socket says 172.21.15.153:52490) 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 1 -- 172.21.15.153:0/323651181 learned_addr learned my addr 172.21.15.153:0/323651181 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): get_auth_request con 0x7f02b4097720 auth_method 0 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): get_auth_request con 0x7f02b4096e10 auth_method 0 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b3fff640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b409aaa0 0x7f02b4118990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b3fff640 10 monclient(hunting): get_auth_request con 0x7f02b409aaa0 auth_method 0 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient(hunting): handle_auth_done global_id 4775 payload 824 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b409aaa0 msgr2=0x7f02b4118990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02b409aaa0 0x7f02b4118990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4097720 msgr2=0x7f02b4118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4097720 0x7f02b4118450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 monclient: found mon.noname-c 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b37fe640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a400b700 con 0x7f02b4096e10 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.161+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b4097720 0x7f02b4118450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b37fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 msgr2=0x7f02b412d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b37fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b412d6f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02a4010f40 tx=0x7f02a400b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a400b920 con 0x7f02b4096e10 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: dump: 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b9d9d640 5 monclient: authenticate success, global_id 4775 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a400eea0 con 0x7f02b4096e10 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b9d9d640 10 monclient: _renew_subs 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b9d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a8009390 con 0x7f02b4096e10 2024-09-15T03:30:24.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4011a80 con 0x7f02b4096e10 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f029b7fe640 10 monclient: dump: 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.184 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 10 monclient: _renew_subs 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b411a940 con 0x7f02b4096e10 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.163+0000 7f02b1ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 0x7f02b40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4065f20 con 0x7f02b4119690 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 0x7f02b40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b2ffd640 10 monclient: get_auth_request con 0x7f02b4119690 auth_method 0 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f02a4019440 con 0x7f02b4096e10 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.164+0000 7f02b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_done global_id 4778 payload 824 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b2ffd640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b404bf60 con 0x7f02b4119690 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f029b7fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02a0038740 0x7f02a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f02a404e6f0 con 0x7f02b4096e10 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b3fff640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02a0038740 0x7f02a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.165+0000 7f02b3fff640 10 monclient: get_auth_request con 0x7f02a0038740 auth_method 0 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f02b2ffd640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 msgr2=0x7f02b40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 0x7f02b40298c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009360 tx=0x7f02a8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8031e70 con 0x7f02b4119690 2024-09-15T03:30:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 10 monclient: dump: 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f02a8047410 con 0x7f02b4119690 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 msgr2=0x7f02b40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 0x7f02b40298c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009360 tx=0x7f02a8032000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 msgr2=0x7f02b40298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f029b7fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4119690 0x7f02b40298c0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009360 tx=0x7f02a8032000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.167+0000 7f02b3fff640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02a0038740 0x7f02a003abf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0076a0 tx=0x7f02ac0098b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.188+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v42) v1 ==== 126+0+0 (secure 0 0 0) 0x7f1fdc042300 con 0x7f1fec09ec50 2024-09-15T03:30:24.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.188+0000 7f1fe3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T03:30:24.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.188+0000 7f1fe3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:24.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b9d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:24.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 0x7f02b4029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b9d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b406cd80 con 0x7f02b4130bc0 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 0x7f02b4029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient: get_auth_request con 0x7f02b4130bc0 auth_method 0 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 10 monclient(hunting): handle_auth_done global_id 4781 payload 824 2024-09-15T03:30:24.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.173+0000 7f02b2ffd640 1 -- 172.21.15.153:0/323651181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02b40638d0 con 0x7f02b4130bc0 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b2ffd640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 msgr2=0x7f02b4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b2ffd640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 0x7f02b4029e00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02b4119e30 tx=0x7f02a803e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a803dc10 con 0x7f02b4130bc0 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 10 monclient: dump: 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.194 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f02a803dd70 con 0x7f02b4130bc0 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 msgr2=0x7f02b4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 0x7f02b4029e00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02b4119e30 tx=0x7f02a803e000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 msgr2=0x7f02b4029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f029b7fe640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4130bc0 0x7f02b4029e00 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02b4119e30 tx=0x7f02a803e000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:24.208 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:24.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02a0038740 msgr2=0x7f02a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02a0038740 0x7f02a003abf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0076a0 tx=0x7f02ac0098b0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 10 monclient: shutdown 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 msgr2=0x7f02b412d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 --2- 172.21.15.153:0/323651181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b4096e10 0x7f02b412d6f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02a4010f40 tx=0x7f02a400b460 comp rx=0 tx=0).stop 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b3fff640 1 -- 172.21.15.153:0/323651181 reap_dead start 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b3fff640 1 -- 172.21.15.153:0/323651181 reap_dead start 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 shutdown_connections 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 >> 172.21.15.153:0/323651181 conn(0x7f02b40672b0 msgr2=0x7f02b412f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.176+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 shutdown_connections 2024-09-15T03:30:24.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.177+0000 7f02b9d9d640 1 -- 172.21.15.153:0/323651181 wait complete. 2024-09-15T03:30:24.210 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:30:24.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:30:24.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:30:24.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1ff22c0640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T03:30:24.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1ff22c0640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T03:30:24.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1ff22c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:24.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f1fec0008d0 con 0x7f1fec09ec50 2024-09-15T03:30:24.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1fe3fff640 1 -- 172.21.15.181:0/1283334499 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v42) v1 ==== 149+0+0 (secure 0 0 0) 0x7f1fdc037020 con 0x7f1fec09ec50 2024-09-15T03:30:24.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1fe3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-15T03:30:24.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.284+0000 7f1fe3fff640 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-15T03:30:24.286 INFO:tasks.workunit.client.0.smithi181.stderr:rule bar already exists 2024-09-15T03:30:24.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fcc038790 msgr2=0x7f1fcc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fcc038790 0x7f1fcc03ac40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0630f0 tx=0x7f1fd4007f00 comp rx=0 tx=0).stop 2024-09-15T03:30:24.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 10 monclient: shutdown 2024-09-15T03:30:24.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 msgr2=0x7f1fec09aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec09aa60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098b0 tx=0x7f1fdc032000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.286+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 shutdown_connections 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fcc038790 0x7f1fcc03ac40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fec1321a0 0x7f1fec11b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fec09f020 0x7f1fec130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 --2- 172.21.15.181:0/1283334499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fec09ec50 0x7f1fec09aa60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 >> 172.21.15.181:0/1283334499 conn(0x7f1fec0a5d40 msgr2=0x7f1fec134b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 shutdown_connections 2024-09-15T03:30:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.287+0000 7f1ff22c0640 1 -- 172.21.15.181:0/1283334499 wait complete. 2024-09-15T03:30:24.296 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm-device-class all 2024-09-15T03:30:24.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.303+0000 7f30a559d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:24.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.303+0000 7f30a559d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.303+0000 7f30a559d640 10 monclient: monmap: 2024-09-15T03:30:24.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:24.305460+0000 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:24.305460+0000 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:24.306 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 Processor -- start 2024-09-15T03:30:24.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 -- start start 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient: init 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30a01318f0 0x7f30a0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): picked mon.noname-a con 0x7f30a01318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f30a009ea50 0x7f30a009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): picked mon.noname-c con 0x7f30a009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30a0068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a009fbf0 con 0x7f30a0068d90 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a0131d50 con 0x7f30a01318f0 2024-09-15T03:30:24.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a009fe30 con 0x7f30a009ea50 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f30a559d640 10 monclient(hunting): authenticate will time out at 821.944880s 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f309effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30a01318f0 0x7f30a0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.304+0000 7f309f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f30a0068d90 0x7f30a00691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44892/0 (socket says 172.21.15.153:44892) 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309f7fe640 1 -- 172.21.15.153:0/2830423647 learned_addr learned my addr 172.21.15.153:0/2830423647 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): get_auth_request con 0x7f30a01318f0 auth_method 0 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 1 --2- 172.21.15.153:0/2830423647 >> v2:172.21.15.181:3300/0 conn(0x7f30a009ea50 0x7f30a009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): get_auth_request con 0x7f30a009ea50 auth_method 0 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient(hunting): handle_auth_done global_id 34482 payload 274 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 1 -- 172.21.15.153:0/2830423647 >> v2:172.21.15.181:3300/0 conn(0x7f30a009ea50 msgr2=0x7f30a009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 1 --2- 172.21.15.153:0/2830423647 >> v2:172.21.15.181:3300/0 conn(0x7f30a009ea50 0x7f30a009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 1 -- 172.21.15.153:0/2830423647 >> v1:172.21.15.153:6789/0 conn(0x7f30a0068d90 legacy=0x7f30a00691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 1 monclient: found mon.noname-a 2024-09-15T03:30:24.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:24.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.305+0000 7f309effd640 1 -- 172.21.15.153:0/2830423647 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00a1210 con 0x7f30a01318f0 2024-09-15T03:30:24.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309effd640 1 -- 172.21.15.153:0/2830423647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a01318f0 msgr2=0x7f30a0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309effd640 1 --2- 172.21.15.153:0/2830423647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a01318f0 0x7f30a0131f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f30a006f600 tx=0x7f30900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ce3d1562e00f18 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 1 -- 172.21.15.153:0/2830423647 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309002ae00 con 0x7f30a01318f0 2024-09-15T03:30:24.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: dump: 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.320 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 1 -- 172.21.15.153:0/2830423647 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3090032440 con 0x7f30a01318f0 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 5 monclient: authenticate success, global_id 34482 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 4 set_mon_vals no callback set 2024-09-15T03:30:24.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 1 -- 172.21.15.153:0/2830423647 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30900327e0 con 0x7f30a01318f0 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f309dffb640 10 monclient: dump: 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.322 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 10 monclient: shutdown 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 -- 172.21.15.153:0/2830423647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a01318f0 msgr2=0x7f30a0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 --2- 172.21.15.153:0/2830423647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a01318f0 0x7f30a0131f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f30a006f600 tx=0x7f30900090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 -- 172.21.15.153:0/2830423647 shutdown_connections 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 --2- 172.21.15.153:0/2830423647 >> v2:172.21.15.181:3300/0 conn(0x7f30a009ea50 0x7f30a009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 --2- 172.21.15.153:0/2830423647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a01318f0 0x7f30a0131f30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 -- 172.21.15.153:0/2830423647 >> 172.21.15.153:0/2830423647 conn(0x7f30a0067110 msgr2=0x7f30a0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 -- 172.21.15.153:0/2830423647 shutdown_connections 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.308+0000 7f30a559d640 1 -- 172.21.15.153:0/2830423647 wait complete. 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.309+0000 7f30a559d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.309+0000 7f30a559d640 10 monclient: monmap: 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:24.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.309+0000 7f30a559d640 1 Processor -- start 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.309+0000 7f30a559d640 1 -- start start 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.309+0000 7f30a559d640 10 monclient: init 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:24.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30a0068d90 0x7f30a009a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): picked mon.noname-c con 0x7f30a0068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 0x7f30a0130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): picked mon.noname-a con 0x7f30a009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01318f0 0x7f30a0140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): picked mon.noname-b con 0x7f30a01318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00a4420 con 0x7f30a009ea50 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a011e260 con 0x7f30a01318f0 2024-09-15T03:30:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a011dfb0 con 0x7f30a0068d90 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f30a559d640 10 monclient(hunting): authenticate will time out at 821.950236s 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 0x7f30a0130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30a0068d90 0x7f30a009a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 0x7f30a0130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35972/0 (socket says 172.21.15.153:35972) 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 -- 172.21.15.153:0/3163372322 learned_addr learned my addr 172.21.15.153:0/3163372322 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): get_auth_request con 0x7f30a009ea50 auth_method 0 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): get_auth_request con 0x7f30a0068d90 auth_method 0 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01318f0 0x7f30a0140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): get_auth_request con 0x7f30a01318f0 auth_method 0 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient(hunting): handle_auth_done global_id 34485 payload 824 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30a0068d90 msgr2=0x7f30a009a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30a0068d90 0x7f30a009a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01318f0 msgr2=0x7f30a0140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01318f0 0x7f30a0140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 monclient: found mon.noname-a 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.310+0000 7f309e7fc640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3090009390 con 0x7f30a009ea50 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f309e7fc640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 msgr2=0x7f30a0130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f309e7fc640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 0x7f30a0130b80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f309400ea20 tx=0x7f30940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094009600 con 0x7f30a009ea50 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 10 monclient: dump: 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f30a559d640 5 monclient: authenticate success, global_id 34485 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309400cc20 con 0x7f30a009ea50 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f30a559d640 10 monclient: _renew_subs 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f30a559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:24.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a011c7f0 con 0x7f30a009ea50 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094013470 con 0x7f30a009ea50 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f3086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f3086ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f3086ffd640 10 monclient: dump: 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 10 monclient: _renew_subs 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a011bdd0 con 0x7f30a009ea50 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.312+0000 7f309d7fa640 4 set_mon_vals no callback set 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 0x7f30a0141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0065e00 con 0x7f30a01310c0 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3094013860 con 0x7f30a009ea50 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f309effd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 0x7f30a0141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f309effd640 10 monclient: get_auth_request con 0x7f30a01310c0 auth_method 0 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f309effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f309effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.313+0000 7f309effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f3086ffd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f308c038770 0x7f308c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f309404bd20 con 0x7f30a009ea50 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309effd640 10 monclient(hunting): handle_auth_done global_id 44149 payload 824 2024-09-15T03:30:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309effd640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a004bf60 con 0x7f30a01310c0 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309f7fe640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f308c038770 0x7f308c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.314+0000 7f309f7fe640 10 monclient: get_auth_request con 0x7f308c038770 auth_method 0 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f309effd640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 msgr2=0x7f30a0141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f309effd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 0x7f30a0141330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3090034000 tx=0x7f3090032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090031c20 con 0x7f30a01310c0 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 10 monclient: dump: 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f309f7fe640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f308c038770 0x7f308c03ac20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f30a009b700 tx=0x7f30980052b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3090031d80 con 0x7f30a01310c0 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 msgr2=0x7f30a0141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 0x7f30a0141330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3090034000 tx=0x7f3090032000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 msgr2=0x7f30a0141330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:24.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.316+0000 7f3086ffd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a01310c0 0x7f30a0141330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f30a559d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f30a559d640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 0x7f30a0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f30a559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a011c980 con 0x7f30a0141870 2024-09-15T03:30:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 0x7f30a0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient: get_auth_request con 0x7f30a0141870 auth_method 0 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.322+0000 7f309effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.323+0000 7f309effd640 10 monclient(hunting): handle_auth_done global_id 44152 payload 824 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.323+0000 7f309effd640 1 -- 172.21.15.153:0/3163372322 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f30a011e0f0 con 0x7f30a0141870 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f309effd640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 msgr2=0x7f30a0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f309effd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 0x7f30a0096a60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30a009b590 tx=0x7f309003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30900469c0 con 0x7f30a0141870 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 10 monclient: dump: 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f3090031a10 con 0x7f30a0141870 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:24.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 msgr2=0x7f30a0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 0x7f30a0096a60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30a009b590 tx=0x7f309003d000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 msgr2=0x7f30a0096a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.325+0000 7f3086ffd640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0141870 0x7f30a0096a60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:30:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 20, 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:24.353 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:24.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f308c038770 msgr2=0x7f308c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f308c038770 0x7f308c03ac20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f30a009b700 tx=0x7f30980052b0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 10 monclient: shutdown 2024-09-15T03:30:24.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 msgr2=0x7f30a0130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 0x7f30a0130b80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f309400ea20 tx=0x7f30940090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f309f7fe640 1 -- 172.21.15.153:0/3163372322 reap_dead start 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 shutdown_connections 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 --2- 172.21.15.153:0/3163372322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a009ea50 0x7f30a0130b80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 >> 172.21.15.153:0/3163372322 conn(0x7f30a0067110 msgr2=0x7f30a01349b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 shutdown_connections 2024-09-15T03:30:24.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.326+0000 7f30a559d640 1 -- 172.21.15.153:0/3163372322 wait complete. 2024-09-15T03:30:24.356 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:30:24.357 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:30:24.357 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:30:24.357 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:30:24.357 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:30:24.357 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:30:24.358 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:30:24.369 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:30:24.369 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:30:24.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.371+0000 7f6ae1b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.371+0000 7f6ae1b5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.371+0000 7f6ae1b5d640 10 monclient: monmap: 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:24.372347+0000 2024-09-15T03:30:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:24.372347+0000 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:24.373 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 Processor -- start 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 -- start start 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient: init 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6adc09ec50 0x7f6adc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6adc09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 0x7f6adc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6adc1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6adc09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc067420 con 0x7f6adc09f020 2024-09-15T03:30:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc0a0480 con 0x7f6adc09ec50 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0a0200 con 0x7f6adc1321a0 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6ae1b5d640 10 monclient(hunting): authenticate will time out at 828.853806s 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adb7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f6adc09ec50 msgr2=0x7f6adc131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adb7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6adc09ec50 0x7f6adc131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adbfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f6adc09f020 0x7f6adc069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:39666/0 (socket says 172.21.15.181:39666) 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adbfff640 1 -- 172.21.15.181:0/91052716 learned_addr learned my addr 172.21.15.181:0/91052716 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adaffd640 1 --2- 172.21.15.181:0/91052716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 0x7f6adc068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adaffd640 10 monclient(hunting): get_auth_request con 0x7f6adc1321a0 auth_method 0 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.372+0000 7f6adaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 10 monclient(hunting): handle_auth_done global_id 4787 payload 274 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 1 -- 172.21.15.181:0/91052716 >> v2:172.21.15.153:3300/0 conn(0x7f6adc09ec50 msgr2=0x7f6adc131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 1 --2- 172.21.15.181:0/91052716 >> v2:172.21.15.153:3300/0 conn(0x7f6adc09ec50 0x7f6adc131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 1 -- 172.21.15.181:0/91052716 >> v1:172.21.15.181:6789/0 conn(0x7f6adc09f020 legacy=0x7f6adc069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 1 monclient: found mon.noname-b 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.373+0000 7f6adaffd640 1 -- 172.21.15.181:0/91052716 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc09fd80 con 0x7f6adc1321a0 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6adaffd640 1 -- 172.21.15.181:0/91052716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 msgr2=0x7f6adc068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6adaffd640 1 --2- 172.21.15.181:0/91052716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 0x7f6adc068d00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009850 tx=0x7f6ad00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2df46f955e133c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 1 -- 172.21.15.181:0/91052716 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0032440 con 0x7f6adc1321a0 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: dump: 2024-09-15T03:30:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 5 monclient: authenticate success, global_id 4787 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 1 -- 172.21.15.181:0/91052716 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad00325a0 con 0x7f6adc1321a0 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 1 -- 172.21.15.181:0/91052716 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0032940 con 0x7f6adc1321a0 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 4 set_mon_vals no callback set 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ada7fc640 10 monclient: dump: 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.388 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 10 monclient: shutdown 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/91052716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 msgr2=0x7f6adc068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.376+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/91052716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 0x7f6adc068d00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009850 tx=0x7f6ad00090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/91052716 shutdown_connections 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/91052716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc1321a0 0x7f6adc068d00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/91052716 >> v2:172.21.15.153:3300/0 conn(0x7f6adc09ec50 0x7f6adc131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/91052716 >> 172.21.15.181:0/91052716 conn(0x7f6adc0a5cf0 msgr2=0x7f6adc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/91052716 shutdown_connections 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/91052716 wait complete. 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 10 monclient: monmap: 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:24.390 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.377+0000 7f6ae1b5d640 1 Processor -- start 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 -- start start 2024-09-15T03:30:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient: init 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc09ec50 0x7f6adc09a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6adc09ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc09f020 0x7f6adc130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6adc09f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 0x7f6adc11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6adc1321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:24.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc09b510 con 0x7f6adc09f020 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc134ad0 con 0x7f6adc1321a0 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc11dee0 con 0x7f6adc09ec50 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6ae1b5d640 10 monclient(hunting): authenticate will time out at 828.859521s 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adaffd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc09f020 msgr2=0x7f6adc130ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:24.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc09f020 0x7f6adc130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 0x7f6adc11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 0x7f6adc11b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46762/0 (socket says 172.21.15.181:46762) 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 -- 172.21.15.181:0/4073047076 learned_addr learned my addr 172.21.15.181:0/4073047076 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): get_auth_request con 0x7f6adc1321a0 auth_method 0 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc09ec50 0x7f6adc09a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): get_auth_request con 0x7f6adc09ec50 auth_method 0 2024-09-15T03:30:24.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient(hunting): handle_auth_done global_id 44155 payload 824 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 -- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc09ec50 msgr2=0x7f6adc09a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc09ec50 0x7f6adc09a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 -- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc09f020 msgr2=0x7f6adc130ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc09f020 0x7f6adc130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 monclient: found mon.noname-b 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:24.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.378+0000 7f6adbfff640 1 -- 172.21.15.181:0/4073047076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0009390 con 0x7f6adc1321a0 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6adbfff640 1 -- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 msgr2=0x7f6adc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6adbfff640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 0x7f6adc11b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400ccf0 tx=0x7f6ad4005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400aa60 con 0x7f6adc1321a0 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: dump: 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 5 monclient: authenticate success, global_id 44155 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad400abe0 con 0x7f6adc1321a0 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad40070f0 con 0x7f6adc1321a0 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 10 monclient: _renew_subs 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc141e80 con 0x7f6adc1321a0 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ac37fe640 10 monclient: dump: 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ad9ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 10 monclient: _renew_subs 2024-09-15T03:30:24.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.381+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc1414f0 con 0x7f6adc1321a0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6ad40074a0 con 0x7f6adc1321a0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ac37fe640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ac80386f0 0x7f6ac803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ae1b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ae1b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ae1b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc11e100 con 0x7f6adc1321a0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6ad404bb40 con 0x7f6adc1321a0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6adb7fe640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ac80386f0 0x7f6ac803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.382+0000 7f6adb7fe640 10 monclient: get_auth_request con 0x7f6ac80386f0 auth_method 0 2024-09-15T03:30:24.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.384+0000 7f6adb7fe640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ac80386f0 0x7f6ac803aba0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0038f0 tx=0x7f6acc0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.387+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6ad4007750 con 0x7f6adc1321a0 2024-09-15T03:30:24.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.387+0000 7f6ac37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.387+0000 7f6ac37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.498+0000 7fd93939d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.498+0000 7fd93939d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.498+0000 7fd93939d640 10 monclient: monmap: 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:24.500684+0000 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:24.500684+0000 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.501 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.502 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:24.502 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:24.502 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:24.502 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:24.502 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:24.502 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.499+0000 7fd93939d640 1 Processor -- start 2024-09-15T03:30:24.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.499+0000 7fd93939d640 1 -- start start 2024-09-15T03:30:24.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.499+0000 7fd93939d640 10 monclient: init 2024-09-15T03:30:24.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9341318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd93409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): picked mon.noname-b con 0x7fd93409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd934136c20 0x7fd934139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): picked mon.noname-c con 0x7fd934136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd934132640 con 0x7fd9341318f0 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd932ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fd9341318f0 legacy=0x7fd934131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd934131d00 con 0x7fd934136c20 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93409fbf0 con 0x7fd93409ea50 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd93939d640 10 monclient(hunting): authenticate will time out at 822.140162s 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd93409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd93409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52518/0 (socket says 172.21.15.153:52518) 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 -- 172.21.15.153:0/348317436 learned_addr learned my addr 172.21.15.153:0/348317436 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): get_auth_request con 0x7fd93409ea50 auth_method 0 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9337fe640 1 --2- 172.21.15.153:0/348317436 >> v2:172.21.15.181:3300/0 conn(0x7fd934136c20 0x7fd934139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9337fe640 10 monclient(hunting): get_auth_request con 0x7fd934136c20 auth_method 0 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient(hunting): handle_auth_done global_id 4793 payload 274 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 -- 172.21.15.153:0/348317436 >> v2:172.21.15.181:3300/0 conn(0x7fd934136c20 msgr2=0x7fd934139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 --2- 172.21.15.153:0/348317436 >> v2:172.21.15.181:3300/0 conn(0x7fd934136c20 0x7fd934139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 -- 172.21.15.153:0/348317436 >> v1:172.21.15.153:6789/0 conn(0x7fd9341318f0 legacy=0x7fd934131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 monclient: found mon.noname-b 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9327fc640 1 -- 172.21.15.153:0/348317436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93409fe30 con 0x7fd93409ea50 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.500+0000 7fd9337fe640 1 --2- 172.21.15.153:0/348317436 >> v2:172.21.15.181:3300/0 conn(0x7fd934136c20 0x7fd934139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd9327fc640 1 -- 172.21.15.153:0/348317436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 msgr2=0x7fd93409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd9327fc640 1 --2- 172.21.15.153:0/348317436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd93409ee20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd928009910 tx=0x7fd9280091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3acef3fee6aec455 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 1 -- 172.21.15.153:0/348317436 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92802aea0 con 0x7fd93409ea50 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: dump: 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.510 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 1 -- 172.21.15.153:0/348317436 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd928032440 con 0x7fd93409ea50 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd93939d640 5 monclient: authenticate success, global_id 4793 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd93939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd93939d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd93939d640 4 set_mon_vals no callback set 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 1 -- 172.21.15.153:0/348317436 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280327e0 con 0x7fd93409ea50 2024-09-15T03:30:24.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.503+0000 7fd931ffb640 10 monclient: dump: 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 10 monclient: shutdown 2024-09-15T03:30:24.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 -- 172.21.15.153:0/348317436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 msgr2=0x7fd93409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 --2- 172.21.15.153:0/348317436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd93409ee20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd928009910 tx=0x7fd9280091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 -- 172.21.15.153:0/348317436 shutdown_connections 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 --2- 172.21.15.153:0/348317436 >> v2:172.21.15.181:3300/0 conn(0x7fd934136c20 0x7fd934139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 --2- 172.21.15.153:0/348317436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd93409ee20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 -- 172.21.15.153:0/348317436 >> 172.21.15.153:0/348317436 conn(0x7fd934067110 msgr2=0x7fd934067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 -- 172.21.15.153:0/348317436 shutdown_connections 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 1 -- 172.21.15.153:0/348317436 wait complete. 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 10 monclient: build_initial_monmap 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.504+0000 7fd93939d640 10 monclient: monmap: 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:24.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 Processor -- start 2024-09-15T03:30:24.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 -- start start 2024-09-15T03:30:24.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient: init 2024-09-15T03:30:24.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:24.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:24.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd934130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): picked mon.noname-c con 0x7fd93409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9341318f0 0x7fd934130e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9341318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd934136c20 0x7fd934099930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): picked mon.noname-b con 0x7fd934136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93409b480 con 0x7fd9341318f0 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93411cfd0 con 0x7fd934136c20 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93411ca80 con 0x7fd93409ea50 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd9327fc640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9341318f0 msgr2=0x7fd934130e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd9327fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9341318f0 0x7fd934130e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd934130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd93939d640 10 monclient(hunting): authenticate will time out at 822.145859s 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd934130900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52528/0 (socket says 172.21.15.153:52528) 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 1 -- 172.21.15.153:0/2264898575 learned_addr learned my addr 172.21.15.153:0/2264898575 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): get_auth_request con 0x7fd93409ea50 auth_method 0 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd932ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd9337fe640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd934136c20 0x7fd934099930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.505+0000 7fd9337fe640 10 monclient(hunting): get_auth_request con 0x7fd934136c20 auth_method 0 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd9337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd9337fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd9337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd9337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd9337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 10 monclient(hunting): handle_auth_done global_id 4796 payload 824 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 1 -- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd934136c20 msgr2=0x7fd934099930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd934136c20 0x7fd934099930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 1 -- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9341318f0 msgr2=0x7fd934130e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9341318f0 0x7fd934130e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 1 monclient: found mon.noname-c 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.506+0000 7fd932ffd640 1 -- 172.21.15.153:0/2264898575 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd928009450 con 0x7fd93409ea50 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.508+0000 7fd932ffd640 1 -- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 msgr2=0x7fd934130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.508+0000 7fd932ffd640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd934130900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd92400b5f0 tx=0x7fd924009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.508+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd924015070 con 0x7fd93409ea50 2024-09-15T03:30:24.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: dump: 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.523 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: _finish_auth 0 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd924003d90 con 0x7fd93409ea50 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9240063a0 con 0x7fd93409ea50 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 5 monclient: authenticate success, global_id 4796 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 10 monclient: _renew_subs 2024-09-15T03:30:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934141e20 con 0x7fd93409ea50 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd9317fa640 4 set_mon_vals no callback set 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd91affd640 10 monclient: dump: 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:24.525 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 10 monclient: _renew_subs 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.510+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93409b750 con 0x7fd93409ea50 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd924010440 con 0x7fd93409ea50 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd93939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd93939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd93939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93404bf60 con 0x7fd93409ea50 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd91affd640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd920038740 0x7fd92003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd92404b4a0 con 0x7fd93409ea50 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd9327fc640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd920038740 0x7fd92003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.511+0000 7fd9327fc640 10 monclient: get_auth_request con 0x7fd920038740 auth_method 0 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.514+0000 7fd9327fc640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd920038740 0x7fd92003abf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd934076600 tx=0x7fd928032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.516+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd9240106f0 con 0x7fd93409ea50 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.516+0000 7fd91affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:24.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.516+0000 7fd91affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:24.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.544+0000 7f6ae1b5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T03:30:24.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.544+0000 7f6ae1b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T03:30:24.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.544+0000 7f6ae1b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:24.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:24.544+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f6adc0008d0 con 0x7f6adc1321a0 2024-09-15T03:30:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd93939d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd93939d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd93939d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd93411cd50 con 0x7fd93409ea50 2024-09-15T03:30:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd91affd640 1 -- 172.21.15.153:0/2264898575 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd9240197e0 con 0x7fd93409ea50 2024-09-15T03:30:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd91affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.731+0000 7fd91affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:24.734 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd920038740 msgr2=0x7fd92003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd920038740 0x7fd92003abf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd934076600 tx=0x7fd928032000 comp rx=0 tx=0).stop 2024-09-15T03:30:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 10 monclient: shutdown 2024-09-15T03:30:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 msgr2=0x7fd934130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd934130900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd92400b5f0 tx=0x7fd924009120 comp rx=0 tx=0).stop 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 shutdown_connections 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd920038740 0x7fd92003abf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd934136c20 0x7fd934099930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9341318f0 0x7fd934130e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 --2- 172.21.15.153:0/2264898575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd93409ea50 0x7fd934130900 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 >> 172.21.15.153:0/2264898575 conn(0x7fd934067110 msgr2=0x7fd9340a3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 shutdown_connections 2024-09-15T03:30:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:24.733+0000 7fd93939d640 1 -- 172.21.15.153:0/2264898575 wait complete. 2024-09-15T03:30:27.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:27.378+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:27.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:27.378+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:27.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:27.378+0000 7f6ac27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:27.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.848+0000 7f8d31d54640 10 monclient: get_monmap_and_config 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.848+0000 7f8d31d54640 10 monclient: build_initial_monmap 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.848+0000 7f8d31d54640 10 monclient: monmap: 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:27.850678+0000 2024-09-15T03:30:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:27.850678+0000 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:27.852 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.849+0000 7f8d31d54640 1 Processor -- start 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.849+0000 7f8d31d54640 1 -- start start 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.849+0000 7f8d31d54640 10 monclient: init 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d2c131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d2c09fda0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8d2c09f0d0 0x7f8d2c139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient(hunting): picked mon.noname-a con 0x7f8d2c09f0d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c09f510 con 0x7f8d2c131980 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c0a1180 con 0x7f8d2c09fda0 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c0a1460 con 0x7f8d2c09f0d0 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d31d54640 10 monclient(hunting): authenticate will time out at 825.490301s 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d2b7fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f8d2c131980 legacy=0x7f8d2c131d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d2bfff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f8d2c09f0d0 msgr2=0x7f8d2c139cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d2bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8d2c09f0d0 0x7f8d2c139cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d2affd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f8d2c09fda0 0x7f8d2c09e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:43976/0 (socket says 172.21.15.153:43976) 2024-09-15T03:30:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.850+0000 7f8d2affd640 1 -- 172.21.15.153:0/1793903088 learned_addr learned my addr 172.21.15.153:0/1793903088 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.852+0000 7f8d2affd640 1 -- 172.21.15.153:0/1793903088 >> v1:172.21.15.181:6789/0 conn(0x7f8d2c09fda0 legacy=0x7f8d2c09e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975578468 0 0) 0x7f8d2c0a1180 con 0x7f8d2c09fda0 2024-09-15T03:30:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): my global_id is 44164 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d140034e0 con 0x7f8d2c09fda0 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857688507 0 0) 0x7f8d140034e0 con 0x7f8d2c09fda0 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 >> v2:172.21.15.153:3300/0 conn(0x7f8d2c09f0d0 msgr2=0x7f8d2c139cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 --2- 172.21.15.153:0/1793903088 >> v2:172.21.15.153:3300/0 conn(0x7f8d2c09f0d0 0x7f8d2c139cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 >> v1:172.21.15.153:6789/0 conn(0x7f8d2c131980 legacy=0x7f8d2c131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:30:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c132460 con 0x7f8d2c09fda0 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f8d200017e0 con 0x7f8d2c09fda0 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: dump: 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 >> v1:172.21.15.181:6789/0 conn(0x7f8d2c09fda0 legacy=0x7f8d2c09e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 0x7f8d14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): picked mon.b con 0x7f8d140039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d14004780 0x7f8d14004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): picked mon.a con 0x7f8d14004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d14005280 0x7f8d1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): picked mon.c con 0x7f8d14005280 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0a1460 con 0x7f8d14004780 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400c240 con 0x7f8d140039d0 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400c400 con 0x7f8d14005280 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2affd640 1 -- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d14004780 msgr2=0x7f8d14004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2affd640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d14004780 0x7f8d14004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 0x7f8d14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d14005280 0x7f8d1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): get_auth_request con 0x7f8d140039d0 auth_method 0 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): get_auth_request con 0x7f8d14005280 auth_method 0 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.853+0000 7f8d2b7fe640 10 monclient(hunting): handle_auth_done global_id 44164 payload 293 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 1 -- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d14005280 msgr2=0x7f8d1400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d14005280 0x7f8d1400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 1 -- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d14004780 msgr2=0x7f8d14004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d14004780 0x7f8d14004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 1 monclient: found mon.b 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 1 -- 172.21.15.153:0/1793903088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140034e0 con 0x7f8d140039d0 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d31d54640 5 monclient: authenticate success, global_id 44164 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d31d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2bfff640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d14005280 0x7f8d1400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:27.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.854+0000 7f8d2bfff640 1 -- 172.21.15.153:0/1793903088 reap_dead start 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.856+0000 7f8d2b7fe640 1 -- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 msgr2=0x7f8d14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.856+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 0x7f8d14003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009f90 tx=0x7f8d1c0095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44d5ded3d6d9f0d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c032430 con 0x7f8d140039d0 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 10 monclient: dump: 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:27.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 0x7f8d14003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009f90 tx=0x7f8d1c0095d0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f8d1c032820 mon_map magic: 0 v1 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c02d070 con 0x7f8d140039d0 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 4 set_mon_vals no callback set 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 10 monclient: shutdown 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 1 -- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 msgr2=0x7f8d14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:27.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 0x7f8d14003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009f90 tx=0x7f8d1c0095d0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 1 -- 172.21.15.153:0/1793903088 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f8d1c032820 con 0x7f8d140039d0 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d2a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 1 -- 172.21.15.153:0/1793903088 shutdown_connections 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 1 --2- 172.21.15.153:0/1793903088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d140039d0 0x7f8d14003dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.857+0000 7f8d31d54640 1 -- 172.21.15.153:0/1793903088 >> 172.21.15.153:0/1793903088 conn(0x7f8d2c067bd0 msgr2=0x7f8d2c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.858+0000 7f8d31d54640 1 -- 172.21.15.153:0/1793903088 shutdown_connections 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.858+0000 7f8d31d54640 1 -- 172.21.15.153:0/1793903088 wait complete. 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.858+0000 7f8d31d54640 10 monclient: build_initial_monmap 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.858+0000 7f8d31d54640 10 monclient: monmap: 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:27.874 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.858+0000 7f8d31d54640 1 Processor -- start 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 -- start start 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient: init 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d140039d0 0x7f8d2c09a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): picked mon.noname-a con 0x7f8d140039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 0x7f8d2c130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): picked mon.noname-c con 0x7f8d2c130860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d2c11b6b0 0x7f8d2c11bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): picked mon.noname-b con 0x7f8d2c11b6b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c1311b0 con 0x7f8d140039d0 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c09bf10 con 0x7f8d2c11b6b0 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c134670 con 0x7f8d2c130860 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d31d54640 10 monclient(hunting): authenticate will time out at 825.499465s 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2b7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d140039d0 msgr2=0x7f8d2c09a800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d140039d0 0x7f8d2c09a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:27.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 0x7f8d2c130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 0x7f8d2c130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45952/0 (socket says 172.21.15.153:45952) 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 -- 172.21.15.153:0/391520791 learned_addr learned my addr 172.21.15.153:0/391520791 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): get_auth_request con 0x7f8d2c130860 auth_method 0 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d2c11b6b0 0x7f8d2c11bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): get_auth_request con 0x7f8d2c11b6b0 auth_method 0 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:27.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient(hunting): handle_auth_done global_id 4799 payload 824 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 -- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d2c11b6b0 msgr2=0x7f8d2c11bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d2c11b6b0 0x7f8d2c11bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 -- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d140039d0 msgr2=0x7f8d2c09a800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d140039d0 0x7f8d2c09a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:27.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 monclient: found mon.noname-c 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.859+0000 7f8d2affd640 1 -- 172.21.15.153:0/391520791 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c11ca90 con 0x7f8d2c130860 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d2affd640 1 -- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 msgr2=0x7f8d2c130c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d2affd640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 0x7f8d2c130c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c860 tx=0x7f8d200011a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000ce50 con 0x7f8d2c130860 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: dump: 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:27.879 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d20007570 con 0x7f8d2c130860 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 5 monclient: authenticate success, global_id 4799 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 10 monclient: _renew_subs 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c141920 con 0x7f8d2c130860 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d200047a0 con 0x7f8d2c130860 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d137fe640 10 monclient: dump: 2024-09-15T03:30:27.881 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:27.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 10 monclient: _renew_subs 2024-09-15T03:30:27.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:27.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c11da30 con 0x7f8d2c130860 2024-09-15T03:30:27.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.862+0000 7f8d29ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d31d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d31d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d31d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8d20004b90 con 0x7f8d2c130860 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c09b440 con 0x7f8d2c130860 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d137fe640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d180386f0 0x7f8d1803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8d2004b1c0 con 0x7f8d2c130860 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d180386f0 0x7f8d1803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:27.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.863+0000 7f8d2b7fe640 10 monclient: get_auth_request con 0x7f8d180386f0 auth_method 0 2024-09-15T03:30:27.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.866+0000 7f8d2b7fe640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d180386f0 0x7f8d1803aba0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c003cc0 tx=0x7f8d1c035000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:27.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.867+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8d20019490 con 0x7f8d2c130860 2024-09-15T03:30:27.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.867+0000 7f8d137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:27.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:27.867+0000 7f8d137fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.084+0000 7f8d31d54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.084+0000 7f8d31d54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.084+0000 7f8d31d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.084+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8d2c04bf60 con 0x7f8d2c130860 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.085+0000 7f8d137fe640 1 -- 172.21.15.153:0/391520791 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f8d20018020 con 0x7f8d2c130860 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.085+0000 7f8d137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.085+0000 7f8d137fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:28.087 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d180386f0 msgr2=0x7f8d1803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d180386f0 0x7f8d1803aba0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c003cc0 tx=0x7f8d1c035000 comp rx=0 tx=0).stop 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 10 monclient: shutdown 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 msgr2=0x7f8d2c130c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 0x7f8d2c130c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c860 tx=0x7f8d200011a0 comp rx=0 tx=0).stop 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 shutdown_connections 2024-09-15T03:30:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d180386f0 0x7f8d1803aba0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d2c11b6b0 0x7f8d2c11bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d2c130860 0x7f8d2c130c70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 --2- 172.21.15.153:0/391520791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d140039d0 0x7f8d2c09a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 >> 172.21.15.153:0/391520791 conn(0x7f8d2c067bd0 msgr2=0x7f8d2c134880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 shutdown_connections 2024-09-15T03:30:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:28.087+0000 7f8d31d54640 1 -- 172.21.15.153:0/391520791 wait complete. 2024-09-15T03:30:28.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:28.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:28.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:28.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:29.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:29.378+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:29.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:29.378+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:30.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:30.378+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:30.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:30.378+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:31.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.201+0000 7fa5c439d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.201+0000 7fa5c439d640 10 monclient: build_initial_monmap 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.201+0000 7fa5c439d640 10 monclient: monmap: 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:31.203648+0000 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:31.203648+0000 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:31.205 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.202+0000 7fa5c439d640 1 Processor -- start 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.202+0000 7fa5c439d640 1 -- start start 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.202+0000 7fa5c439d640 10 monclient: init 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5bc0a1ef0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5bc09c6a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 0x7fa5bc13a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc09b2e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc06f010 con 0x7fa5bc0a1ef0 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc09dcc0 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c2139640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa5bc0a1ef0 legacy=0x7fa5bc09abd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0a22c0 con 0x7fa5bc09b2e0 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c439d640 10 monclient(hunting): authenticate will time out at 828.843347s 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 0x7fa5bc13a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 0x7fa5bc13a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45954/0 (socket says 172.21.15.153:45954) 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 -- 172.21.15.153:0/3550551076 learned_addr learned my addr 172.21.15.153:0/3550551076 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): get_auth_request con 0x7fa5bc09b2e0 auth_method 0 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient(hunting): handle_auth_done global_id 4802 payload 274 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 -- 172.21.15.153:0/3550551076 >> v1:172.21.15.181:6789/0 conn(0x7fa5bc09c6a0 legacy=0x7fa5bc136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 -- 172.21.15.153:0/3550551076 >> v1:172.21.15.153:6789/0 conn(0x7fa5bc0a1ef0 legacy=0x7fa5bc09abd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 monclient: found mon.noname-b 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.203+0000 7fa5c293a640 1 -- 172.21.15.153:0/3550551076 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc09b710 con 0x7fa5bc09b2e0 2024-09-15T03:30:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c293a640 1 -- 172.21.15.153:0/3550551076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 msgr2=0x7fa5bc13a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c293a640 1 --2- 172.21.15.153:0/3550551076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 0x7fa5bc13a430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009f50 tx=0x7fa5b4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41c12417574d7db1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 1 -- 172.21.15.153:0/3550551076 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4007610 con 0x7fa5bc09b2e0 2024-09-15T03:30:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: dump: 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: _finish_auth 0 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:31.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c439d640 5 monclient: authenticate success, global_id 4802 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c439d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 1 -- 172.21.15.153:0/3550551076 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b402d050 con 0x7fa5bc09b2e0 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c439d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c439d640 4 set_mon_vals no callback set 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 1 -- 172.21.15.153:0/3550551076 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40079a0 con 0x7fa5bc09b2e0 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.206+0000 7fa5c1137640 10 monclient: dump: 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 10 monclient: shutdown 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 -- 172.21.15.153:0/3550551076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 msgr2=0x7fa5bc13a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 --2- 172.21.15.153:0/3550551076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 0x7fa5bc13a430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009f50 tx=0x7fa5b4009840 comp rx=0 tx=0).stop 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 -- 172.21.15.153:0/3550551076 shutdown_connections 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 --2- 172.21.15.153:0/3550551076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09b2e0 0x7fa5bc13a430 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 -- 172.21.15.153:0/3550551076 >> 172.21.15.153:0/3550551076 conn(0x7fa5bc0678a0 msgr2=0x7fa5bc0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 -- 172.21.15.153:0/3550551076 shutdown_connections 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 1 -- 172.21.15.153:0/3550551076 wait complete. 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 10 monclient: build_initial_monmap 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.207+0000 7fa5c439d640 10 monclient: monmap: 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:31.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 Processor -- start 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 -- start start 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient: init 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5bc09b2e0 0x7fa5bc097740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc09b2e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 0x7fa5bc12eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc09c6a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:31.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc0a1ef0 0x7fa5bc126630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5bc0a1ef0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc098360 con 0x7fa5bc0a1ef0 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc134990 con 0x7fa5bc09b2e0 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc129060 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c439d640 10 monclient(hunting): authenticate will time out at 828.848651s 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c293a640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc0a1ef0 msgr2=0x7fa5bc126630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c293a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc0a1ef0 0x7fa5bc126630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 0x7fa5bc12eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 0x7fa5bc12eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45970/0 (socket says 172.21.15.153:45970) 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 1 -- 172.21.15.153:0/691391428 learned_addr learned my addr 172.21.15.153:0/691391428 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): get_auth_request con 0x7fa5bc09c6a0 auth_method 0 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c2139640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5bc09b2e0 0x7fa5bc097740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c2139640 10 monclient(hunting): get_auth_request con 0x7fa5bc09b2e0 auth_method 0 2024-09-15T03:30:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c2139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c2139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c2139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.208+0000 7fa5c1938640 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 1 -- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5bc09b2e0 msgr2=0x7fa5bc097740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5bc09b2e0 0x7fa5bc097740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 1 -- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc0a1ef0 msgr2=0x7fa5bc126630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc0a1ef0 0x7fa5bc126630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 10 monclient: _finish_hunting 0 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 1 monclient: found mon.noname-c 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:31.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.209+0000 7fa5c1938640 1 -- 172.21.15.153:0/691391428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4009ae0 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c1938640 1 -- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 msgr2=0x7fa5bc12eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c1938640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 0x7fa5bc12eb90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c640 tx=0x7fa5b8007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8012840 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: dump: 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b80129a0 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 5 monclient: authenticate success, global_id 4805 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 10 monclient: _renew_subs 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:31.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc128d80 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8012d60 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5ae7fc640 10 monclient: dump: 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:31.227 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c0936640 4 set_mon_vals no callback set 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 10 monclient: _renew_subs 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.211+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc141650 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa5b8011590 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5c439d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5c439d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5c439d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc04bf60 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5ae7fc640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5a00386f0 0x7fa5a003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa5b804bb40 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.212+0000 7fa5c2139640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5a00386f0 0x7fa5a003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.213+0000 7fa5c2139640 10 monclient: get_auth_request con 0x7fa5a00386f0 auth_method 0 2024-09-15T03:30:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.215+0000 7fa5c2139640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5a00386f0 0x7fa5a003aba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0003940 tx=0x7fa5b0006ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.217+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa5b8010aa0 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.217+0000 7fa5ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:31.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.217+0000 7fa5ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:31.378+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:31.378+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:31.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.430+0000 7fa5c439d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.430+0000 7fa5c439d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.430+0000 7fa5c439d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.430+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa5bc0638d0 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.431+0000 7fa5ae7fc640 1 -- 172.21.15.153:0/691391428 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa5b8016070 con 0x7fa5bc09c6a0 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.431+0000 7fa5ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.431+0000 7fa5ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:31.433 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5a00386f0 msgr2=0x7fa5a003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5a00386f0 0x7fa5a003aba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0003940 tx=0x7fa5b0006ef0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 10 monclient: shutdown 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 msgr2=0x7fa5bc12eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 0x7fa5bc12eb90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c640 tx=0x7fa5b8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 shutdown_connections 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa5a00386f0 0x7fa5a003aba0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5bc0a1ef0 0x7fa5bc126630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5bc09c6a0 0x7fa5bc12eb90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 --2- 172.21.15.153:0/691391428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5bc09b2e0 0x7fa5bc097740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 >> 172.21.15.153:0/691391428 conn(0x7fa5bc0678a0 msgr2=0x7fa5bc134ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 shutdown_connections 2024-09-15T03:30:31.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:31.433+0000 7fa5c439d640 1 -- 172.21.15.153:0/691391428 wait complete. 2024-09-15T03:30:32.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:32.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:32.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:32.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:33.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:33.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:33.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:33.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:34.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:34.378+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:34.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:34.378+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:34.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:30:34.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.549+0000 7fa94bac0640 10 monclient: get_monmap_and_config 2024-09-15T03:30:34.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.549+0000 7fa94bac0640 10 monclient: build_initial_monmap 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.549+0000 7fa94bac0640 10 monclient: monmap: 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:34.551472+0000 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:34.551472+0000 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:34.552 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.550+0000 7fa94bac0640 1 Processor -- start 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.550+0000 7fa94bac0640 1 -- start start 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.550+0000 7fa94bac0640 10 monclient: init 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa944098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient(hunting): picked mon.noname-a con 0x7fa9440981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa944131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa944132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa944098ad0 con 0x7fa944131da0 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa944067f30 con 0x7fa944132170 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa944131bd0 con 0x7fa9440981c0 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94bac0640 10 monclient(hunting): authenticate will time out at 832.191126s 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94905b640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa944131da0 legacy=0x7fa944067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94985c640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa944098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94985c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa944098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94a05d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa944132170 0x7fa944067820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:43990/0 (socket says 172.21.15.153:43990) 2024-09-15T03:30:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.551+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 learned_addr learned my addr 172.21.15.153:0/595721669 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:30:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.553+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.181:6789/0 conn(0x7fa944132170 legacy=0x7fa944067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:34.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.752+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa944098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:34.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.752+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.153:6789/0 conn(0x7fa944131da0 legacy=0x7fa944067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:34.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:34.752+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa944098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:30:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:35.153+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa944098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:35.153+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.153:6789/0 conn(0x7fa944131da0 legacy=0x7fa944067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:35.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:35.153+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa944098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:30:35.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:35.378+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:35.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:35.378+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:35.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:35.954+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.153:6789/0 conn(0x7fa944131da0 legacy=0x7fa944067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:35.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:35.954+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa944098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:35.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:35.954+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa944098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:30:36.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:36.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:36.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:36.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:37.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:37.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:37.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:37.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:37.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:37.379+0000 7f6ac27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:37.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): tick 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 monclient(hunting): continuing hunt 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa944098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa944098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.181:6789/0 conn(0x7fa944132170 legacy=0x7fa944067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.153:6789/0 conn(0x7fa944131da0 legacy=0x7fa944067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient: _add_conns ranks=[2,4] 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fa928003700 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fa928006270 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:37.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa944131bd0 con 0x7fa928006270 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa928008dc0 con 0x7fa928003700 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa93b7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 10 monclient(hunting): get_auth_request con 0x7fa928003700 auth_method 0 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:37.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 10 monclient(hunting): get_auth_request con 0x7fa928006270 auth_method 0 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:37.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.552+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:37.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:37.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 10 monclient(hunting): get_auth_request con 0x7fa928003700 auth_method 0 2024-09-15T03:30:37.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 10 monclient(hunting): get_auth_request con 0x7fa928006270 auth_method 0 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:37.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:37.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:37.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:37.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:37.753+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 10 monclient(hunting): get_auth_request con 0x7fa928003700 auth_method 0 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:38.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 10 monclient(hunting): get_auth_request con 0x7fa928006270 auth_method 0 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:38.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.154+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:30:38.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:38.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:38.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:38.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:38.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 10 monclient(hunting): get_auth_request con 0x7fa928006270 auth_method 0 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 10 monclient(hunting): get_auth_request con 0x7fa928003700 auth_method 0 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:38.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:38.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:30:38.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:30:38.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94905b640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:30:38.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:38.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:38.955+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:30:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.252+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v43) v1 ==== 131+0+0 (secure 0 0 0) 0x7f6ad4010af0 con 0x7f6adc1321a0 2024-09-15T03:30:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.252+0000 7f6ac37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T03:30:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.252+0000 7f6ac37fe640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-15T03:30:39.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.353+0000 7f6ae1b5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T03:30:39.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.353+0000 7f6ae1b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T03:30:39.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.353+0000 7f6ae1b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:39.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.353+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f6adc0a13e0 con 0x7f6adc1321a0 2024-09-15T03:30:39.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.379+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:39.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:39.379+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.380+0000 7f6ac27fc640 10 monclient: tick 2024-09-15T03:30:40.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.380+0000 7f6ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): tick 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 monclient(hunting): continuing hunt 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 msgr2=0x7fa928005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003700 0x7fa928005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 msgr2=0x7fa928008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928006270 0x7fa928008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 reap_dead start 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa92800b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fa9440981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa928003b30 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928007e10 0x7fa928008260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fa928007e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94409ca60 con 0x7fa928003b30 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa944131bd0 con 0x7fa9440981c0 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 1 -- 172.21.15.153:0/595721669 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa928008dc0 con 0x7fa928007e10 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa93b7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa94985c640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa92800b1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.552+0000 7fa94985c640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa92800b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.181:3300/0 conn(0x7fa928007e10 0x7fa928008260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): get_auth_request con 0x7fa928007e10 auth_method 0 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient(hunting): handle_auth_done global_id 44173 payload 274 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 msgr2=0x7fa92800b1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa92800b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 >> v1:172.21.15.181:6789/0 conn(0x7fa928003b30 legacy=0x7fa9280066f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 1 monclient: found mon.noname-c 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.553+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944098ad0 con 0x7fa928007e10 2024-09-15T03:30:40.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.556+0000 7fa94a05d640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa928007e10 msgr2=0x7fa928008260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.556+0000 7fa94a05d640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa928007e10 0x7fa928008260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9440662e0 tx=0x7fa93c001c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b923eaf5e5a09835 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.556+0000 7fa94885a640 1 -- 172.21.15.153:0/595721669 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c02aa90 con 0x7fa928007e10 2024-09-15T03:30:40.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.556+0000 7fa94885a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.556+0000 7fa94885a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 10 monclient: dump: 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 10 monclient: _finish_auth 0 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 1 -- 172.21.15.153:0/595721669 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c02abf0 con 0x7fa928007e10 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 1 -- 172.21.15.153:0/595721669 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0070f0 con 0x7fa928007e10 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94885a640 10 monclient: dump: 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 5 monclient: authenticate success, global_id 44173 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:40.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 4 set_mon_vals no callback set 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 10 monclient: shutdown 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 1 -- 172.21.15.153:0/595721669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa928007e10 msgr2=0x7fa928008260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa928007e10 0x7fa928008260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9440662e0 tx=0x7fa93c001c70 comp rx=0 tx=0).stop 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 1 -- 172.21.15.153:0/595721669 shutdown_connections 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 1 --2- 172.21.15.153:0/595721669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa928007e10 0x7fa928008260 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 1 --2- 172.21.15.153:0/595721669 >> v2:172.21.15.153:3300/0 conn(0x7fa9440981c0 0x7fa92800b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.557+0000 7fa94bac0640 1 -- 172.21.15.153:0/595721669 >> 172.21.15.153:0/595721669 conn(0x7fa9440a1e30 msgr2=0x7fa9440a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.558+0000 7fa94bac0640 1 -- 172.21.15.153:0/595721669 shutdown_connections 2024-09-15T03:30:40.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.558+0000 7fa94bac0640 1 -- 172.21.15.153:0/595721669 wait complete. 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.558+0000 7fa94bac0640 10 monclient: build_initial_monmap 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.558+0000 7fa94bac0640 10 monclient: monmap: 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:40.569 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 Processor -- start 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 -- start start 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient: init 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 0x7fa944141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): picked mon.noname-c con 0x7fa928003b30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928007e10 0x7fa944130a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): picked mon.noname-a con 0x7fa928007e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9440981c0 0x7fa94411a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): picked mon.noname-b con 0x7fa9440981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94409b930 con 0x7fa928007e10 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa944098cb0 con 0x7fa9440981c0 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa944141ec0 con 0x7fa928003b30 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94bac0640 10 monclient(hunting): authenticate will time out at 838.198693s 2024-09-15T03:30:40.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94905b640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928007e10 msgr2=0x7fa944130a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94905b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928007e10 0x7fa944130a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 0x7fa944141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 0x7fa944141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36864/0 (socket says 172.21.15.153:36864) 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 1 -- 172.21.15.153:0/1705450378 learned_addr learned my addr 172.21.15.153:0/1705450378 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): get_auth_request con 0x7fa928003b30 auth_method 0 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.559+0000 7fa94985c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94a05d640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9440981c0 0x7fa94411a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94a05d640 10 monclient(hunting): get_auth_request con 0x7fa9440981c0 auth_method 0 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94a05d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94a05d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94a05d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 1 -- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9440981c0 msgr2=0x7fa94411a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9440981c0 0x7fa94411a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 1 -- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928007e10 msgr2=0x7fa944130a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928007e10 0x7fa944130a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 1 monclient: found mon.noname-c 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.560+0000 7fa94985c640 1 -- 172.21.15.153:0/1705450378 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944067f30 con 0x7fa928003b30 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa94985c640 1 -- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 msgr2=0x7fa944141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa94985c640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 0x7fa944141070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa94000acb0 tx=0x7fa940005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940016440 con 0x7fa928003b30 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: dump: 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa94bac0640 5 monclient: authenticate success, global_id 4808 2024-09-15T03:30:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9400165c0 con 0x7fa928003b30 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa94bac0640 10 monclient: _renew_subs 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa94bac0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94411cc20 con 0x7fa928003b30 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940016980 con 0x7fa928003b30 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa939ffb640 10 monclient: dump: 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.563+0000 7fa94bac0640 10 monclient: _renew_subs 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.563+0000 7fa94bac0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.562+0000 7fa93b7fe640 4 set_mon_vals no callback set 2024-09-15T03:30:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.563+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94411c050 con 0x7fa928003b30 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa9400153c0 con 0x7fa928003b30 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa94bac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa94bac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa94bac0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa939ffb640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa928039fb0 0x7fa92803c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94404bf60 con 0x7fa928003b30 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4248+0+0 (secure 0 0 0) 0x7fa94004c500 con 0x7fa928003b30 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa94905b640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa928039fb0 0x7fa92803c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.564+0000 7fa94905b640 10 monclient: get_auth_request con 0x7fa928039fb0 auth_method 0 2024-09-15T03:30:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.567+0000 7fa94905b640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa928039fb0 0x7fa92803c460 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9440a2af0 tx=0x7fa934003a00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.569+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa940015670 con 0x7fa928003b30 2024-09-15T03:30:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.569+0000 7fa939ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.569+0000 7fa939ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:40.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.719+0000 7f6ac37fe640 1 -- 172.21.15.181:0/4073047076 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v44) v1 ==== 282+0+0 (secure 0 0 0) 0x7f6ad40320a0 con 0x7f6adc1321a0 2024-09-15T03:30:40.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.719+0000 7f6ac37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-15T03:30:40.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.719+0000 7f6ac37fe640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-15T03:30:40.722 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-15T03:30:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ac80386f0 msgr2=0x7f6ac803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ac80386f0 0x7f6ac803aba0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0038f0 tx=0x7f6acc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 10 monclient: shutdown 2024-09-15T03:30:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 msgr2=0x7f6adc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 0x7f6adc11b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400ccf0 tx=0x7f6ad4005370 comp rx=0 tx=0).stop 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 shutdown_connections 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ac80386f0 0x7f6ac803aba0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6adc1321a0 0x7f6adc11b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6adc09f020 0x7f6adc130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 --2- 172.21.15.181:0/4073047076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6adc09ec50 0x7f6adc09a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 >> 172.21.15.181:0/4073047076 conn(0x7f6adc0a5cf0 msgr2=0x7f6adc134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 shutdown_connections 2024-09-15T03:30:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.722+0000 7f6ae1b5d640 1 -- 172.21.15.181:0/4073047076 wait complete. 2024-09-15T03:30:40.736 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-15T03:30:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa94bac0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa94bac0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa94bac0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa944098ee0 con 0x7fa928003b30 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa939ffb640 1 -- 172.21.15.153:0/1705450378 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fa940009240 con 0x7fa928003b30 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa939ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.789+0000 7fa939ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:40.791 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:30:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.791+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa928039fb0 msgr2=0x7fa92803c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.791+0000 7fa94bac0640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa928039fb0 0x7fa92803c460 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9440a2af0 tx=0x7fa934003a00 comp rx=0 tx=0).stop 2024-09-15T03:30:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.791+0000 7fa94bac0640 10 monclient: shutdown 2024-09-15T03:30:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.791+0000 7fa94bac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.791+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 msgr2=0x7fa944141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.791+0000 7fa94bac0640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 0x7fa944141070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa94000acb0 tx=0x7fa940005040 comp rx=0 tx=0).stop 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 shutdown_connections 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa928039fb0 0x7fa92803c460 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9440981c0 0x7fa94411a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928007e10 0x7fa944130a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 --2- 172.21.15.153:0/1705450378 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa928003b30 0x7fa944141070 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 >> 172.21.15.153:0/1705450378 conn(0x7fa9440a1e30 msgr2=0x7fa944096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 shutdown_connections 2024-09-15T03:30:40.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.792+0000 7fa94bac0640 1 -- 172.21.15.153:0/1705450378 wait complete. 2024-09-15T03:30:40.803 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:30:40.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:30:40.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:30:40.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.812+0000 7ff68060f640 10 monclient: get_monmap_and_config 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.812+0000 7ff68060f640 10 monclient: build_initial_monmap 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.812+0000 7ff68060f640 10 monclient: monmap: 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:40.814659+0000 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:40.814659+0000 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:40.815 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:40.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.813+0000 7ff68060f640 1 Processor -- start 2024-09-15T03:30:40.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.813+0000 7ff68060f640 1 -- start start 2024-09-15T03:30:40.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.813+0000 7ff68060f640 10 monclient: init 2024-09-15T03:30:40.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6780a0ad0 0x7ff6780691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6780a0ad0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): picked mon.noname-b con 0x7ff67809ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff678131ad0 0x7ff678131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): picked mon.noname-a con 0x7ff678131ad0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67809f7a0 con 0x7ff678131ad0 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67806a5b0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6780a0460 con 0x7ff67809ee60 2024-09-15T03:30:40.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff68060f640 10 monclient(hunting): authenticate will time out at 845.296272s 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67ebac640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7ff678131ad0 msgr2=0x7ff678131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67ebac640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff678131ad0 0x7ff678131f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6780a0ad0 0x7ff6780691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6780a0ad0 0x7ff6780691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34334/0 (socket says 172.21.15.181:34334) 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 -- 172.21.15.181:0/1165621703 learned_addr learned my addr 172.21.15.181:0/1165621703 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): get_auth_request con 0x7ff6780a0ad0 auth_method 0 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 1 --2- 172.21.15.181:0/1165621703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): get_auth_request con 0x7ff67809ee60 auth_method 0 2024-09-15T03:30:40.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_done global_id 44179 payload 274 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 -- 172.21.15.181:0/1165621703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 msgr2=0x7ff67809f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 --2- 172.21.15.181:0/1165621703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 -- 172.21.15.181:0/1165621703 >> v2:172.21.15.153:3300/0 conn(0x7ff678131ad0 msgr2=0x7ff678131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 --2- 172.21.15.181:0/1165621703 >> v2:172.21.15.153:3300/0 conn(0x7ff678131ad0 0x7ff678131f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient: _finish_hunting 0 2024-09-15T03:30:40.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 monclient: found mon.noname-c 2024-09-15T03:30:40.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:40.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.814+0000 7ff67e3ab640 1 -- 172.21.15.181:0/1165621703 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6780a0ea0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67e3ab640 1 -- 172.21.15.181:0/1165621703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 msgr2=0x7ff6780691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67e3ab640 1 --2- 172.21.15.181:0/1165621703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff6780691d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff67806abc0 tx=0x7ff6740091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b0c2179c6a3db22 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 1 -- 172.21.15.181:0/1165621703 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674032440 con 0x7ff6780a0ad0 2024-09-15T03:30:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: dump: 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: _finish_auth 0 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 1 -- 172.21.15.181:0/1165621703 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6740325a0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 5 monclient: authenticate success, global_id 44179 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 1 -- 172.21.15.181:0/1165621703 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674032940 con 0x7ff6780a0ad0 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 4 set_mon_vals no callback set 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff67d3a9640 10 monclient: dump: 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 10 monclient: shutdown 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 -- 172.21.15.181:0/1165621703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 msgr2=0x7ff6780691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 --2- 172.21.15.181:0/1165621703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff6780691d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff67806abc0 tx=0x7ff6740091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 -- 172.21.15.181:0/1165621703 shutdown_connections 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 --2- 172.21.15.181:0/1165621703 >> v2:172.21.15.153:3300/0 conn(0x7ff678131ad0 0x7ff678131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 --2- 172.21.15.181:0/1165621703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 --2- 172.21.15.181:0/1165621703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff6780691d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 -- 172.21.15.181:0/1165621703 >> 172.21.15.181:0/1165621703 conn(0x7ff678071370 msgr2=0x7ff6780a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 -- 172.21.15.181:0/1165621703 shutdown_connections 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.818+0000 7ff68060f640 1 -- 172.21.15.181:0/1165621703 wait complete. 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient: build_initial_monmap 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient: monmap: 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:40.831 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 Processor -- start 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 -- start start 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient: init 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): picked mon.noname-c con 0x7ff67809ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff678130b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6780a0ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff678131ad0 0x7ff67811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): picked mon.noname-a con 0x7ff678131ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff678134b70 con 0x7ff678131ad0 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67811df20 con 0x7ff6780a0ad0 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff678136470 con 0x7ff67809ee60 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff68060f640 10 monclient(hunting): authenticate will time out at 845.301847s 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67ebac640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff678131ad0 msgr2=0x7ff67811b900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff678130b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67ebac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff678131ad0 0x7ff67811b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff678130b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34336/0 (socket says 172.21.15.181:34336) 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 1 -- 172.21.15.181:0/399479985 learned_addr learned my addr 172.21.15.181:0/399479985 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 10 monclient(hunting): get_auth_request con 0x7ff6780a0ad0 auth_method 0 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.819+0000 7ff67dbaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): get_auth_request con 0x7ff67809ee60 auth_method 0 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67e3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 10 monclient(hunting): handle_auth_done global_id 44182 payload 824 2024-09-15T03:30:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 1 -- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 msgr2=0x7ff67809a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 1 -- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff678131ad0 msgr2=0x7ff67811b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff678131ad0 0x7ff67811b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 10 monclient: _finish_hunting 0 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 1 monclient: found mon.noname-b 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.820+0000 7ff67dbaa640 1 -- 172.21.15.181:0/399479985 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff674009450 con 0x7ff6780a0ad0 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff67dbaa640 1 -- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 msgr2=0x7ff678130b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff67dbaa640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff678130b20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff66800ea20 tx=0x7ff6680090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668009600 con 0x7ff6780a0ad0 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:40.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: dump: 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 5 monclient: authenticate success, global_id 44182 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66800cc20 con 0x7ff6780a0ad0 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 10 monclient: _renew_subs 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668013470 con 0x7ff6780a0ad0 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6781318e0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff67cba8640 4 set_mon_vals no callback set 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff66e7fc640 10 monclient: dump: 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 10 monclient: _renew_subs 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.822+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67811dc40 con 0x7ff6780a0ad0 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff668013860 con 0x7ff6780a0ad0 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff68060f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff68060f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff68060f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6780008d0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff66e7fc640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff65c038720 0x7ff65c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4248+0+0 (secure 0 0 0) 0x7ff66804bbf0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff67e3ab640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff65c038720 0x7ff65c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.823+0000 7ff67e3ab640 10 monclient: get_auth_request con 0x7ff65c038720 auth_method 0 2024-09-15T03:30:40.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.826+0000 7ff67e3ab640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff65c038720 0x7ff65c03abd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff6780a00e0 tx=0x7ff674032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.828+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff668023ad0 con 0x7ff6780a0ad0 2024-09-15T03:30:40.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.828+0000 7ff66e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.828+0000 7ff66e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:40.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.897+0000 7f7a70b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:40.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.897+0000 7f7a70b5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:40.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.897+0000 7f7a70b5d640 10 monclient: monmap: 2024-09-15T03:30:40.910 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:40.910 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:40.897924+0000 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:40.897924+0000 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.897+0000 7f7a70b5d640 1 Processor -- start 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.897+0000 7f7a70b5d640 1 -- start start 2024-09-15T03:30:40.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.897+0000 7f7a70b5d640 10 monclient: init 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 0x7f7a6c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a6c0981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7a6c131da0 0x7f7a6c067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a6c131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a6c132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c098ad0 con 0x7f7a6c132170 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6c131bd0 con 0x7f7a6c131da0 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c09ca60 con 0x7f7a6c0981c0 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a70b5d640 10 monclient(hunting): authenticate will time out at 838.537449s 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6ad9d640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f7a6c132170 legacy=0x7f7a6c067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 0x7f7a6c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 0x7f7a6c098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36874/0 (socket says 172.21.15.153:36874) 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 -- 172.21.15.153:0/1997761387 learned_addr learned my addr 172.21.15.153:0/1997761387 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): get_auth_request con 0x7f7a6c0981c0 auth_method 0 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 1 --2- 172.21.15.153:0/1997761387 >> v2:172.21.15.181:3300/0 conn(0x7f7a6c131da0 0x7f7a6c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): get_auth_request con 0x7f7a6c131da0 auth_method 0 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_done global_id 4817 payload 274 2024-09-15T03:30:40.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 -- 172.21.15.153:0/1997761387 >> v2:172.21.15.181:3300/0 conn(0x7f7a6c131da0 msgr2=0x7f7a6c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/1997761387 >> v2:172.21.15.181:3300/0 conn(0x7f7a6c131da0 0x7f7a6c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 -- 172.21.15.153:0/1997761387 >> v1:172.21.15.153:6789/0 conn(0x7f7a6c132170 legacy=0x7f7a6c067650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 monclient: found mon.noname-b 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.898+0000 7f7a6a59c640 1 -- 172.21.15.153:0/1997761387 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c09cca0 con 0x7f7a6c0981c0 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.901+0000 7f7a6a59c640 1 -- 172.21.15.153:0/1997761387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 msgr2=0x7f7a6c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.901+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/1997761387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 0x7f7a6c098590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009f20 tx=0x7f7a64009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d55b3deb5d33655 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 1 -- 172.21.15.153:0/1997761387 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64007610 con 0x7f7a6c0981c0 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: dump: 2024-09-15T03:30:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: _finish_auth 0 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 5 monclient: authenticate success, global_id 4817 2024-09-15T03:30:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 1 -- 172.21.15.153:0/1997761387 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6402d050 con 0x7f7a6c0981c0 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 1 -- 172.21.15.153:0/1997761387 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a640079a0 con 0x7f7a6c0981c0 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 4 set_mon_vals no callback set 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a6959a640 10 monclient: dump: 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 10 monclient: shutdown 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 -- 172.21.15.153:0/1997761387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 msgr2=0x7f7a6c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/1997761387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 0x7f7a6c098590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009f20 tx=0x7f7a64009840 comp rx=0 tx=0).stop 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 -- 172.21.15.153:0/1997761387 shutdown_connections 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/1997761387 >> v2:172.21.15.181:3300/0 conn(0x7f7a6c131da0 0x7f7a6c067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/1997761387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c0981c0 0x7f7a6c098590 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 -- 172.21.15.153:0/1997761387 >> 172.21.15.153:0/1997761387 conn(0x7f7a6c0a1e30 msgr2=0x7f7a6c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 -- 172.21.15.153:0/1997761387 shutdown_connections 2024-09-15T03:30:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.902+0000 7f7a70b5d640 1 -- 172.21.15.153:0/1997761387 wait complete. 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.903+0000 7f7a70b5d640 10 monclient: build_initial_monmap 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.903+0000 7f7a70b5d640 10 monclient: monmap: 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.903+0000 7f7a70b5d640 1 Processor -- start 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.903+0000 7f7a70b5d640 1 -- start start 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.903+0000 7f7a70b5d640 10 monclient: init 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a6c0981c0 0x7f7a6c11dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a6c0981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 0x7f7a6c11e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a6c131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a6c132170 0x7f7a6c11ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a6c132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c098cb0 con 0x7f7a6c132170 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0a0b90 con 0x7f7a6c0981c0 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c122560 con 0x7f7a6c131da0 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a70b5d640 10 monclient(hunting): authenticate will time out at 838.543090s 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6ad9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a6c132170 msgr2=0x7f7a6c11ea60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:40.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a6c132170 0x7f7a6c11ea60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 0x7f7a6c11e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a6c0981c0 0x7f7a6c11dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 0x7f7a6c11e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36888/0 (socket says 172.21.15.153:36888) 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 -- 172.21.15.153:0/3586431431 learned_addr learned my addr 172.21.15.153:0/3586431431 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): get_auth_request con 0x7f7a6c0981c0 auth_method 0 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): get_auth_request con 0x7f7a6c131da0 auth_method 0 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient(hunting): handle_auth_done global_id 4820 payload 824 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a6c0981c0 msgr2=0x7f7a6c11dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a6c0981c0 0x7f7a6c11dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a6c132170 msgr2=0x7f7a6c11ea60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a6c132170 0x7f7a6c11ea60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 monclient: found mon.noname-c 2024-09-15T03:30:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a69d9b640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64009ae0 con 0x7f7a6c131da0 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.904+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a6c0981c0 0x7f7a6c11dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a69d9b640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 msgr2=0x7f7a6c11e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a69d9b640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 0x7f7a6c11e520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400ea50 tx=0x7f7a540090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54009620 con 0x7f7a6c131da0 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: dump: 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.924 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5400cc20 con 0x7f7a6c131da0 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a70b5d640 5 monclient: authenticate success, global_id 4820 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a70b5d640 10 monclient: _renew_subs 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a70b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c11f8a0 con 0x7f7a6c131da0 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54013470 con 0x7f7a6c131da0 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a627fc640 10 monclient: dump: 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.926 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.906+0000 7f7a68d99640 4 set_mon_vals no callback set 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 10 monclient: _renew_subs 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c120470 con 0x7f7a6c131da0 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7a54013860 con 0x7f7a6c131da0 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 0x7f7a6c11a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c065f90 con 0x7f7a6c11efa0 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.907+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 0x7f7a6c11a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient: get_auth_request con 0x7f7a6c11efa0 auth_method 0 2024-09-15T03:30:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a627fc640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a4c038740 0x7f7a4c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4248+0+0 (secure 0 0 0) 0x7f7a5404bbf0 con 0x7f7a6c131da0 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6ad9d640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a4c038740 0x7f7a4c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6a59c640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a6c122090 con 0x7f7a6c11efa0 2024-09-15T03:30:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.908+0000 7f7a6ad9d640 10 monclient: get_auth_request con 0x7f7a4c038740 auth_method 0 2024-09-15T03:30:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a6a59c640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 msgr2=0x7f7a6c11a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 0x7f7a6c11a4a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a64033000 tx=0x7f7a64036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a6ad9d640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a4c038740 0x7f7a4c03abf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c003940 tx=0x7f7a5c006cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64031850 con 0x7f7a6c11efa0 2024-09-15T03:30:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a627fc640 10 monclient: dump: 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.933 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.910+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7a64041070 con 0x7f7a6c11efa0 2024-09-15T03:30:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.911+0000 7f7a627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.911+0000 7f7a627fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.911+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 msgr2=0x7f7a6c11a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.911+0000 7f7a627fc640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 0x7f7a6c11a4a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a64033000 tx=0x7f7a64036000 comp rx=0 tx=0).stop 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.911+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 msgr2=0x7f7a6c11a4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.911+0000 7f7a627fc640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11efa0 0x7f7a6c11a4a0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a64033000 tx=0x7f7a64036000 comp rx=0 tx=0).stop 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a70b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 0x7f7a6c11ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a70b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c04bf60 con 0x7f7a6c11a9e0 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 0x7f7a6c11ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient: get_auth_request con 0x7f7a6c11a9e0 auth_method 0 2024-09-15T03:30:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.916+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.917+0000 7f7a6a59c640 10 monclient(hunting): handle_auth_done global_id 4826 payload 824 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.917+0000 7f7a6a59c640 1 -- 172.21.15.153:0/3586431431 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a6c098ee0 con 0x7f7a6c11a9e0 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a6a59c640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 msgr2=0x7f7a6c11ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a6a59c640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 0x7f7a6c11ade0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c121e90 tx=0x7f7a6403f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6403d610 con 0x7f7a6c11a9e0 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:40.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 10 monclient: dump: 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f7a6403d770 con 0x7f7a6c11a9e0 2024-09-15T03:30:40.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 msgr2=0x7f7a6c11ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 0x7f7a6c11ade0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c121e90 tx=0x7f7a6403f070 comp rx=0 tx=0).stop 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 msgr2=0x7f7a6c11ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a627fc640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c11a9e0 0x7f7a6c11ade0 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c121e90 tx=0x7f7a6403f070 comp rx=0 tx=0).stop 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:40.937 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 24, 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:40.938 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:40.939 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:40.940 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:40.941 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:40.942 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:40.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:40.946 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.947 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.948 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:40.949 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:40.950 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:40.951 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:40.951 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a4c038740 msgr2=0x7f7a4c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a4c038740 0x7f7a4c03abf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c003940 tx=0x7f7a5c006cb0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 10 monclient: shutdown 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 msgr2=0x7f7a6c11e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 1 --2- 172.21.15.153:0/3586431431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a6c131da0 0x7f7a6c11e520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400ea50 tx=0x7f7a540090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a6ad9d640 1 -- 172.21.15.153:0/3586431431 reap_dead start 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a6ad9d640 1 -- 172.21.15.153:0/3586431431 reap_dead start 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 shutdown_connections 2024-09-15T03:30:40.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.919+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 >> 172.21.15.153:0/3586431431 conn(0x7f7a6c0a1e30 msgr2=0x7f7a6c096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:40.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.920+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 shutdown_connections 2024-09-15T03:30:40.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:40.920+0000 7f7a70b5d640 1 -- 172.21.15.153:0/3586431431 wait complete. 2024-09-15T03:30:40.954 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:30:40.954 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:30:40.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:30:40.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.989+0000 7ff68060f640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T03:30:40.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.989+0000 7ff68060f640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T03:30:40.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.989+0000 7ff68060f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:40.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:40.989+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7ff678141e10 con 0x7ff6780a0ad0 2024-09-15T03:30:41.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.048+0000 7fe5d8e78640 10 monclient: get_monmap_and_config 2024-09-15T03:30:41.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.048+0000 7fe5d8e78640 10 monclient: build_initial_monmap 2024-09-15T03:30:41.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.048+0000 7fe5d8e78640 10 monclient: monmap: 2024-09-15T03:30:41.050 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:41.050 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.050 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:30:41.048956+0000 2024-09-15T03:30:41.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:30:41.048956+0000 2024-09-15T03:30:41.051 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:41.051 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.051 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:41.051 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:41.051 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:41.051 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:41.052 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:41.052 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.048+0000 7fe5d8e78640 1 Processor -- start 2024-09-15T03:30:41.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.048+0000 7fe5d8e78640 1 -- start start 2024-09-15T03:30:41.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.048+0000 7fe5d8e78640 10 monclient: init 2024-09-15T03:30:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T03:30:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5d4131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe5d409fda0 0x7fe5d409e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d409fda0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 0x7fe5d4138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d409ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40a1180 con 0x7fe5d4131980 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d40a1460 con 0x7fe5d409fda0 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d4132460 con 0x7fe5d409ef00 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d8e78640 10 monclient(hunting): authenticate will time out at 838.688473s 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d259c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fe5d4131980 legacy=0x7fe5d4131d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d1d9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fe5d409fda0 msgr2=0x7fe5d409e9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d1d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe5d409fda0 0x7fe5d409e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 0x7fe5d4138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 0x7fe5d4138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36912/0 (socket says 172.21.15.153:36912) 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/2631352505 learned_addr learned my addr 172.21.15.153:0/2631352505 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): get_auth_request con 0x7fe5d409ef00 auth_method 0 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_done global_id 4829 payload 274 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/2631352505 >> v2:172.21.15.153:3300/0 conn(0x7fe5d409fda0 msgr2=0x7fe5d409e9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 --2- 172.21.15.153:0/2631352505 >> v2:172.21.15.153:3300/0 conn(0x7fe5d409fda0 0x7fe5d409e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/2631352505 >> v1:172.21.15.153:6789/0 conn(0x7fe5d4131980 legacy=0x7fe5d4131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 monclient: found mon.noname-b 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.049+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/2631352505 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d41325d0 con 0x7fe5d409ef00 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.052+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/2631352505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 msgr2=0x7fe5d4138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.052+0000 7fe5d2d9d640 1 --2- 172.21.15.153:0/2631352505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 0x7fe5d4138cc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009f20 tx=0x7fe5cc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfecd824cd14cf77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.052+0000 7fe5d159a640 1 -- 172.21.15.153:0/2631352505 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc007610 con 0x7fe5d409ef00 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.052+0000 7fe5d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: dump: 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:41.077 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: _finish_auth 0 2024-09-15T03:30:41.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 1 -- 172.21.15.153:0/2631352505 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc02d050 con 0x7fe5d409ef00 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 5 monclient: authenticate success, global_id 4829 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 1 -- 172.21.15.153:0/2631352505 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0079e0 con 0x7fe5d409ef00 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 4 set_mon_vals no callback set 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:41.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d159a640 10 monclient: dump: 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 10 monclient: shutdown 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 -- 172.21.15.153:0/2631352505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 msgr2=0x7fe5d4138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/2631352505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 0x7fe5d4138cc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009f20 tx=0x7fe5cc009840 comp rx=0 tx=0).stop 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 -- 172.21.15.153:0/2631352505 shutdown_connections 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/2631352505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d409ef00 0x7fe5d4138cc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/2631352505 >> v2:172.21.15.153:3300/0 conn(0x7fe5d409fda0 0x7fe5d409e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 -- 172.21.15.153:0/2631352505 >> 172.21.15.153:0/2631352505 conn(0x7fe5d4067bd0 msgr2=0x7fe5d40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 -- 172.21.15.153:0/2631352505 shutdown_connections 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.053+0000 7fe5d8e78640 1 -- 172.21.15.153:0/2631352505 wait complete. 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient: build_initial_monmap 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient: monmap: 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:30:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 Processor -- start 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 -- start start 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient: init 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:41.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d409ef00 0x7fe5d4131360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d409ef00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5d409fda0 0x7fe5d4133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d409fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 0x7fe5d41342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d4131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d409bf10 con 0x7fe5d409fda0 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40a4160 con 0x7fe5d409ef00 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d4135220 con 0x7fe5d4131980 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d8e78640 10 monclient(hunting): authenticate will time out at 838.693972s 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d1d9b640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5d409fda0 msgr2=0x7fe5d4133db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.054+0000 7fe5d1d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5d409fda0 0x7fe5d4133db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 0x7fe5d41342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 0x7fe5d41342f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36924/0 (socket says 172.21.15.153:36924) 2024-09-15T03:30:41.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/3609719703 learned_addr learned my addr 172.21.15.153:0/3609719703 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:30:41.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): get_auth_request con 0x7fe5d4131980 auth_method 0 2024-09-15T03:30:41.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:41.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:41.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:41.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:41.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:41.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:41.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d259c640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d409ef00 0x7fe5d4131360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:41.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d259c640 10 monclient(hunting): get_auth_request con 0x7fe5d409ef00 auth_method 0 2024-09-15T03:30:41.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:41.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:41.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient(hunting): handle_auth_done global_id 4832 payload 824 2024-09-15T03:30:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d409ef00 msgr2=0x7fe5d4131360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d409ef00 0x7fe5d4131360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5d409fda0 msgr2=0x7fe5d4133db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5d409fda0 0x7fe5d4133db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 monclient: found mon.noname-c 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.055+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc009ae0 con 0x7fe5d4131980 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 msgr2=0x7fe5d41342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d2d9d640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 0x7fe5d41342f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009f20 tx=0x7fe5cc036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc031cd0 con 0x7fe5d4131980 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: dump: 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.089 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:41.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 5 monclient: authenticate success, global_id 4832 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc031e30 con 0x7fe5d4131980 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc03c630 con 0x7fe5d4131980 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 10 monclient: _renew_subs 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d4135dc0 con 0x7fe5d4131980 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d0d99640 4 set_mon_vals no callback set 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:41.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5ca7fc640 10 monclient: dump: 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.092 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 10 monclient: _renew_subs 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.057+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d41358b0 con 0x7fe5d4131980 2024-09-15T03:30:41.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d8e78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 0x7fe5d4134830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d4065e00 con 0x7fe5d4136480 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe5cc003020 con 0x7fe5d4131980 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 0x7fe5d4134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient: get_auth_request con 0x7fe5d4136480 auth_method 0 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5ca7fc640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5b40386f0 0x7fe5b403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.058+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4248+0+0 (secure 0 0 0) 0x7fe5cc06c680 con 0x7fe5d4131980 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.059+0000 7fe5d1d9b640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5b40386f0 0x7fe5b403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.059+0000 7fe5d259c640 10 monclient(hunting): handle_auth_done global_id 44194 payload 824 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.059+0000 7fe5d259c640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5d404bf60 con 0x7fe5d4136480 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.059+0000 7fe5d1d9b640 10 monclient: get_auth_request con 0x7fe5b40386f0 auth_method 0 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5d259c640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 msgr2=0x7fe5d4134830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5d259c640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 0x7fe5d4134830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00d6f0 tx=0x7fe5bc004ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc012820 con 0x7fe5d4136480 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 10 monclient: dump: 2024-09-15T03:30:41.095 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5d1d9b640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5b40386f0 0x7fe5b403aba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4135540 tx=0x7fe5c4006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe5bc012980 con 0x7fe5d4136480 2024-09-15T03:30:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 msgr2=0x7fe5d4134830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 0x7fe5d4134830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00d6f0 tx=0x7fe5bc004ba0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 msgr2=0x7fe5d4134830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.061+0000 7fe5ca7fc640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4136480 0x7fe5d4134830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d8e78640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 0x7fe5d4098e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d8e78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40638d0 con 0x7fe5d4098a70 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 0x7fe5d4098e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient: get_auth_request con 0x7fe5d4098a70 auth_method 0 2024-09-15T03:30:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 10 monclient(hunting): handle_auth_done global_id 44197 payload 824 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.067+0000 7fe5d259c640 1 -- 172.21.15.153:0/3609719703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe5d4137510 con 0x7fe5d4098a70 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d259c640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 msgr2=0x7fe5d4098e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d259c640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 0x7fe5d4098e70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc017000 tx=0x7fe5bc018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00ba10 con 0x7fe5d4098a70 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 10 monclient: dump: 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fe5bc023070 con 0x7fe5d4098a70 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:30:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 msgr2=0x7fe5d4098e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 0x7fe5d4098e70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc017000 tx=0x7fe5bc018000 comp rx=0 tx=0).stop 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 msgr2=0x7fe5d4098e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5ca7fc640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5d4098a70 0x7fe5d4098e70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:41.100 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 24, 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:41.101 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:41.102 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:30:41.103 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:30:41.104 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:30:41.105 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:30:41.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:41.107 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:41.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.109 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:30:41.110 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:30:41.111 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.112 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:41.113 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:30:41.114 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:30:41.115 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:30:41.115 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:30:41.115 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:30:41.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:30:41.115 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:30:41.115 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:30:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5b40386f0 msgr2=0x7fe5b403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5b40386f0 0x7fe5b403aba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4135540 tx=0x7fe5c4006d80 comp rx=0 tx=0).stop 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 10 monclient: shutdown 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 msgr2=0x7fe5d41342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 0x7fe5d41342f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009f20 tx=0x7fe5cc036000 comp rx=0 tx=0).stop 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d2d9d640 1 -- 172.21.15.153:0/3609719703 reap_dead start 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 shutdown_connections 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 --2- 172.21.15.153:0/3609719703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5d4131980 0x7fe5d41342f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 >> 172.21.15.153:0/3609719703 conn(0x7fe5d4067bd0 msgr2=0x7fe5d409e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 shutdown_connections 2024-09-15T03:30:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:30:41.070+0000 7fe5d8e78640 1 -- 172.21.15.153:0/3609719703 wait complete. 2024-09-15T03:30:41.117 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:30:41.118 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:30:41.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.718+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v45) v1 ==== 135+0+0 (secure 0 0 0) 0x7ff6680232f0 con 0x7ff6780a0ad0 2024-09-15T03:30:41.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.718+0000 7ff66e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T03:30:41.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.718+0000 7ff66e7fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-15T03:30:41.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.817+0000 7ff68060f640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T03:30:41.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.817+0000 7ff68060f640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T03:30:41.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.817+0000 7ff68060f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:41.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:41.817+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7ff67809b2a0 con 0x7ff6780a0ad0 2024-09-15T03:30:42.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.596+0000 7ff66e7fc640 1 -- 172.21.15.181:0/399479985 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v46) v1 ==== 237+0+0 (secure 0 0 0) 0x7ff6680234d0 con 0x7ff6780a0ad0 2024-09-15T03:30:42.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.596+0000 7ff66e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-15T03:30:42.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.596+0000 7ff66e7fc640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-15T03:30:42.598 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-15T03:30:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff65c038720 msgr2=0x7ff65c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff65c038720 0x7ff65c03abd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff6780a00e0 tx=0x7ff674032000 comp rx=0 tx=0).stop 2024-09-15T03:30:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 10 monclient: shutdown 2024-09-15T03:30:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 msgr2=0x7ff678130b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff678130b20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff66800ea20 tx=0x7ff6680090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 shutdown_connections 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff65c038720 0x7ff65c03abd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff678131ad0 0x7ff67811b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6780a0ad0 0x7ff678130b20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 --2- 172.21.15.181:0/399479985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff67809ee60 0x7ff67809a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 >> 172.21.15.181:0/399479985 conn(0x7ff678071370 msgr2=0x7ff6780a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 shutdown_connections 2024-09-15T03:30:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.599+0000 7ff68060f640 1 -- 172.21.15.181:0/399479985 wait complete. 2024-09-15T03:30:42.611 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.686+0000 7f731a99d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.686+0000 7f731a99d640 10 monclient: build_initial_monmap 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.686+0000 7f731a99d640 10 monclient: monmap: 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:42.687688+0000 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:42.687688+0000 2024-09-15T03:30:42.696 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.687+0000 7f731a99d640 1 Processor -- start 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.687+0000 7f731a99d640 1 -- start start 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.687+0000 7f731a99d640 10 monclient: init 2024-09-15T03:30:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f731409cad0 0x7f731409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): picked mon.noname-a con 0x7f731409cad0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731409dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): picked mon.noname-b con 0x7f731409cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f731409d270 0x7f731409e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): picked mon.noname-c con 0x7f731409d270 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:42.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731409b710 con 0x7f731409cad0 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731409c6e0 con 0x7f731409d270 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731409a650 con 0x7f731409cea0 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f731a99d640 10 monclient(hunting): authenticate will time out at 847.169183s 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f731409d270 0x7f731409e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7313fff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f731409cad0 msgr2=0x7f731409c1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7313fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f731409cad0 0x7f731409c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f731409d270 0x7f731409e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34338/0 (socket says 172.21.15.181:34338) 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 -- 172.21.15.181:0/133481069 learned_addr learned my addr 172.21.15.181:0/133481069 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): get_auth_request con 0x7f731409d270 auth_method 0 2024-09-15T03:30:42.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f73137fe640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731409dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f73137fe640 10 monclient(hunting): get_auth_request con 0x7f731409cea0 auth_method 0 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f73137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f73137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f73137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient(hunting): handle_auth_done global_id 44200 payload 274 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 -- 172.21.15.181:0/133481069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 msgr2=0x7f731409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:42.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731409dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 -- 172.21.15.181:0/133481069 >> v2:172.21.15.153:3300/0 conn(0x7f731409cad0 msgr2=0x7f731409c1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 --2- 172.21.15.181:0/133481069 >> v2:172.21.15.153:3300/0 conn(0x7f731409cad0 0x7f731409c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 monclient: found mon.noname-c 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f7318f3a640 1 -- 172.21.15.181:0/133481069 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731409f660 con 0x7f731409d270 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.688+0000 7f73137fe640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731409dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7318f3a640 1 -- 172.21.15.181:0/133481069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409d270 msgr2=0x7f731409e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7318f3a640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409d270 0x7f731409e010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f731406edb0 tx=0x7f730c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10e9d7c88d3f47a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 1 -- 172.21.15.181:0/133481069 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c007610 con 0x7f731409d270 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: dump: 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:42.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f731a99d640 5 monclient: authenticate success, global_id 44200 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 1 -- 172.21.15.181:0/133481069 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c02d050 con 0x7f731409d270 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f731a99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f731a99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f731a99d640 4 set_mon_vals no callback set 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 1 -- 172.21.15.181:0/133481069 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0079e0 con 0x7f731409d270 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.691+0000 7f7312ffd640 10 monclient: dump: 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.703 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 10 monclient: shutdown 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 -- 172.21.15.181:0/133481069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409d270 msgr2=0x7f731409e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409d270 0x7f731409e010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f731406edb0 tx=0x7f730c009840 comp rx=0 tx=0).stop 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 -- 172.21.15.181:0/133481069 shutdown_connections 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409d270 0x7f731409e010 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 --2- 172.21.15.181:0/133481069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731409dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 --2- 172.21.15.181:0/133481069 >> v2:172.21.15.153:3300/0 conn(0x7f731409cad0 0x7f731409c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 -- 172.21.15.181:0/133481069 >> 172.21.15.181:0/133481069 conn(0x7f7314070e90 msgr2=0x7f73140a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 -- 172.21.15.181:0/133481069 shutdown_connections 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 -- 172.21.15.181:0/133481069 wait complete. 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 10 monclient: build_initial_monmap 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 10 monclient: monmap: 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:42.705 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.692+0000 7f731a99d640 1 Processor -- start 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 -- start start 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient: init 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 0x7f731413b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): picked mon.noname-b con 0x7f731409cad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): picked mon.noname-c con 0x7f731409cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f731409d270 0x7f731413e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): picked mon.noname-a con 0x7f731409d270 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:42.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73141200b0 con 0x7f731409d270 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731411e0c0 con 0x7f731409cad0 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731411d860 con 0x7f731409cea0 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f731a99d640 10 monclient(hunting): authenticate will time out at 847.174538s 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 0x7f731413b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7318f3a640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f731409d270 msgr2=0x7f731413e910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7318f3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f731409d270 0x7f731413e910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 0x7f731413b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34348/0 (socket says 172.21.15.181:34348) 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 -- 172.21.15.181:0/3468294606 learned_addr learned my addr 172.21.15.181:0/3468294606 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:42.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): get_auth_request con 0x7f731409cad0 auth_method 0 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): get_auth_request con 0x7f731409cea0 auth_method 0 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:42.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f73137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient(hunting): handle_auth_done global_id 44203 payload 824 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 -- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 msgr2=0x7f731413aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731413aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 -- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f731409d270 msgr2=0x7f731413e910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f731409d270 0x7f731413e910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 monclient: found mon.noname-b 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:42.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.693+0000 7f7313fff640 1 -- 172.21.15.181:0/3468294606 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c009ae0 con 0x7f731409cad0 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f7313fff640 1 -- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 msgr2=0x7f731413b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f7313fff640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 0x7f731413b3a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f730400b600 tx=0x7f73040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304015070 con 0x7f731409cad0 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: dump: 2024-09-15T03:30:42.714 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: _finish_auth 0 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7304003d90 con 0x7f731409cad0 2024-09-15T03:30:42.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 5 monclient: authenticate success, global_id 44203 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 10 monclient: _renew_subs 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731411faf0 con 0x7f731409cad0 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304006380 con 0x7f731409cad0 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f72fbfff640 10 monclient: dump: 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:42.716 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 10 monclient: _renew_subs 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731413bdb0 con 0x7f731409cad0 2024-09-15T03:30:42.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.696+0000 7f73127fc640 4 set_mon_vals no callback set 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7304010440 con 0x7f731409cad0 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f731a99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f731a99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f731a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f72fbfff640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7300038740 0x7f730003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73141201f0 con 0x7f731409cad0 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4472+0+0 (secure 0 0 0) 0x7f730404b400 con 0x7f731409cad0 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f73137fe640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7300038740 0x7f730003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:42.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.697+0000 7f73137fe640 10 monclient: get_auth_request con 0x7f7300038740 auth_method 0 2024-09-15T03:30:42.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.699+0000 7f73137fe640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7300038740 0x7f730003abf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7308006e70 tx=0x7f7308008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:42.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.701+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f73141201f0 con 0x7f731409cad0 2024-09-15T03:30:42.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.701+0000 7f72fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:42.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.701+0000 7f72fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:42.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.860+0000 7f731a99d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T03:30:42.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.860+0000 7f731a99d640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T03:30:42.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.860+0000 7f731a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:42.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:42.860+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f73141201f0 con 0x7f731409cad0 2024-09-15T03:30:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.732+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7f73141201f0 con 0x7f731409cad0 2024-09-15T03:30:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.732+0000 7f72fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T03:30:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.732+0000 7f72fbfff640 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-15T03:30:43.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.831+0000 7f731a99d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T03:30:43.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.831+0000 7f731a99d640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T03:30:43.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.831+0000 7f731a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:43.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:43.831+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f73141201f0 con 0x7f731409cad0 2024-09-15T03:30:44.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.737+0000 7f72fbfff640 1 -- 172.21.15.181:0/3468294606 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v48) v1 ==== 237+0+0 (secure 0 0 0) 0x7f73141201f0 con 0x7f731409cad0 2024-09-15T03:30:44.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.737+0000 7f72fbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-15T03:30:44.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.737+0000 7f72fbfff640 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-15T03:30:44.739 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-15T03:30:44.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7300038740 msgr2=0x7f730003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7300038740 0x7f730003abf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7308006e70 tx=0x7f7308008040 comp rx=0 tx=0).stop 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 10 monclient: shutdown 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 msgr2=0x7f731413b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 0x7f731413b3a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f730400b600 tx=0x7f73040091a0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 shutdown_connections 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7300038740 0x7f730003abf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f731409d270 0x7f731413e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f731409cea0 0x7f731413aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 --2- 172.21.15.181:0/3468294606 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f731409cad0 0x7f731413b3a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 >> 172.21.15.181:0/3468294606 conn(0x7f7314070e90 msgr2=0x7f73140a1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 shutdown_connections 2024-09-15T03:30:44.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.740+0000 7f731a99d640 1 -- 172.21.15.181:0/3468294606 wait complete. 2024-09-15T03:30:44.752 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.834+0000 7f59dce17640 10 monclient: get_monmap_and_config 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.834+0000 7f59dce17640 10 monclient: build_initial_monmap 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.834+0000 7f59dce17640 10 monclient: monmap: 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:44.834889+0000 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:44.834889+0000 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.835 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:44.836 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:44.836 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:44.836 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:44.836 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:44.836 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.834+0000 7f59dce17640 1 Processor -- start 2024-09-15T03:30:44.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.834+0000 7f59dce17640 1 -- start start 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.834+0000 7f59dce17640 10 monclient: init 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59d809ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f59d809f020 0x7f59d8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient(hunting): picked mon.noname-a con 0x7f59d809f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59d8136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d8067420 con 0x7f59d809ec50 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d809fdc0 con 0x7f59d8136d60 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80a0510 con 0x7f59d809f020 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59dce17640 10 monclient(hunting): authenticate will time out at 849.316473s 2024-09-15T03:30:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59d659c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f59d809ec50 legacy=0x7f59d8131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59d5d9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f59d809f020 msgr2=0x7f59d8069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59d5d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f59d809f020 0x7f59d8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59d6d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f59d8136d60 0x7f59d8139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35922/0 (socket says 172.21.15.181:35922) 2024-09-15T03:30:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.835+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 learned_addr learned my addr 172.21.15.181:0/1192088654 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:44.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.837+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 >> v1:172.21.15.181:6789/0 conn(0x7f59d8136d60 legacy=0x7f59d8139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:44.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.837+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503779332 0 0) 0x7f59d809fdc0 con 0x7f59d8136d60 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.837+0000 7f59d559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.837+0000 7f59d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.837+0000 7f59d559a640 10 monclient(hunting): my global_id is 44206 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.837+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c00034e0 con 0x7f59d8136d60 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727432646 0 0) 0x7f59c00034e0 con 0x7f59d8136d60 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 >> v2:172.21.15.153:3300/0 conn(0x7f59d809f020 msgr2=0x7f59d8069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 --2- 172.21.15.181:0/1192088654 >> v2:172.21.15.153:3300/0 conn(0x7f59d809f020 0x7f59d8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 >> v1:172.21.15.153:6789/0 conn(0x7f59d809ec50 legacy=0x7f59d8131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:30:44.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80a0200 con 0x7f59d8136d60 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: _finish_auth 0 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59dce17640 5 monclient: authenticate success, global_id 44206 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f59d0001aa0 con 0x7f59d8136d60 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59dce17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:30:44.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: dump: 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:44.849 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 >> v1:172.21.15.181:6789/0 conn(0x7f59d8136d60 legacy=0x7f59d8139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59c0003a50 0x7f59c0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): picked mon.c con 0x7f59c0003a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59c0004780 0x7f59c0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): picked mon.a con 0x7f59c0004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:44.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59c000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): picked mon.b con 0x7f59c00099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80a0510 con 0x7f59c0004780 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c000c360 con 0x7f59c00099c0 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c000c520 con 0x7f59c0003a50 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d559a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59dce17640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d5d9b640 1 -- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59c0004780 msgr2=0x7f59c0004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d5d9b640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59c0004780 0x7f59c0004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59c000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:44.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): get_auth_request con 0x7f59c00099c0 auth_method 0 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d659c640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59c0003a50 0x7f59c0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d659c640 10 monclient(hunting): get_auth_request con 0x7f59c0003a50 auth_method 0 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient(hunting): handle_auth_done global_id 44206 payload 293 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59c0003a50 msgr2=0x7f59c0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:44.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59c0003a50 0x7f59c0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59c0004780 msgr2=0x7f59c0004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59c0004780 0x7f59c0004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 monclient: found mon.b 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c00034e0 con 0x7f59c00099c0 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 10 monclient: _finish_auth 0 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.838+0000 7f59d6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.839+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 reap_dead start 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 msgr2=0x7f59c000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d6d9d640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59c000bda0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f59d0001c80 tx=0x7f59d809fca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c3aeb4eef72229 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:44.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d002b6b0 con 0x7f59c00099c0 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: dump: 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59dce17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d000d040 con 0x7f59c00099c0 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59dce17640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59dce17640 4 set_mon_vals no callback set 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 1 -- 172.21.15.181:0/1192088654 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d002ba80 con 0x7f59c00099c0 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.841+0000 7f59d559a640 10 monclient: dump: 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:44.855 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 10 monclient: shutdown 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 -- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 msgr2=0x7f59c000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59c000bda0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f59d0001c80 tx=0x7f59d809fca0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 -- 172.21.15.181:0/1192088654 shutdown_connections 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 --2- 172.21.15.181:0/1192088654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59c000bda0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 -- 172.21.15.181:0/1192088654 >> 172.21.15.181:0/1192088654 conn(0x7f59d80a5d40 msgr2=0x7f59d80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 -- 172.21.15.181:0/1192088654 shutdown_connections 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 1 -- 172.21.15.181:0/1192088654 wait complete. 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 10 monclient: build_initial_monmap 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.842+0000 7f59dce17640 10 monclient: monmap: 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:44.857 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 Processor -- start 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 -- start start 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient: init 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): picked mon.noname-b con 0x7f59c00099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59d8099cd0 0x7f59d809a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): picked mon.noname-c con 0x7f59d8099cd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d809a5e0 0x7f59d8147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): picked mon.noname-a con 0x7f59d809a5e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d811ca50 con 0x7f59d809a5e0 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d809b6b0 con 0x7f59c00099c0 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d809c000 con 0x7f59d8099cd0 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59dce17640 10 monclient(hunting): authenticate will time out at 849.324767s 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34372/0 (socket says 172.21.15.181:34372) 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 1 -- 172.21.15.181:0/1959018290 learned_addr learned my addr 172.21.15.181:0/1959018290 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): get_auth_request con 0x7f59c00099c0 auth_method 0 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d6d9d640 1 -- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d809a5e0 msgr2=0x7f59d8147200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d6d9d640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d809a5e0 0x7f59d8147200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.843+0000 7f59d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 10 monclient(hunting): handle_auth_done global_id 44209 payload 824 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 1 -- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59d8099cd0 msgr2=0x7f59d809a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59d8099cd0 0x7f59d809a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 1 -- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d809a5e0 msgr2=0x7f59d8147200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d809a5e0 0x7f59d8147200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 1 monclient: found mon.noname-b 2024-09-15T03:30:44.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:44.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:44.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.844+0000 7f59d659c640 1 -- 172.21.15.181:0/1959018290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d809d8c0 con 0x7f59c00099c0 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59d659c640 1 -- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 msgr2=0x7f59d8099790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59d659c640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59c800d8f0 tx=0x7f59c8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c800de80 con 0x7f59c00099c0 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: dump: 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 5 monclient: authenticate success, global_id 44209 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c8003d90 con 0x7f59c00099c0 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 10 monclient: _renew_subs 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d811d960 con 0x7f59c00099c0 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8005320 con 0x7f59c00099c0 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59c67fc640 10 monclient: dump: 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:44.865 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 10 monclient: _renew_subs 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59d4d99640 4 set_mon_vals no callback set 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.846+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d811b810 con 0x7f59c00099c0 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59d659c640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59c800d8f0 tx=0x7f59c8008040 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f59c804ba10 osd_map(48..48 src has 1..48) v4 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f59c8005690 con 0x7f59c00099c0 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59dce17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59dce17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59dce17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d80008d0 con 0x7f59c00099c0 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59c67fc640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f59b80386f0 0x7f59b803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59d5d9b640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f59b80386f0 0x7f59b803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:44.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.847+0000 7f59d5d9b640 10 monclient: get_auth_request con 0x7f59b80386f0 auth_method 0 2024-09-15T03:30:44.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.850+0000 7f59d5d9b640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f59b80386f0 0x7f59b803aba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f59cc006fd0 tx=0x7f59cc006d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:44.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:44.851+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4704+0+0 (secure 0 0 0) 0x7f59c804ba10 con 0x7f59c00099c0 2024-09-15T03:30:45.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.675+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f59c8019d80 con 0x7f59c00099c0 2024-09-15T03:30:45.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.675+0000 7f59c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:45.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.675+0000 7f59c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:45.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.838+0000 7f59dce17640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T03:30:45.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.838+0000 7f59dce17640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T03:30:45.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.838+0000 7f59dce17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:45.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:45.838+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f59d80a13e0 con 0x7f59c00099c0 2024-09-15T03:30:46.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.742+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v49) v1 ==== 150+0+0 (secure 0 0 0) 0x7f59c8016070 con 0x7f59c00099c0 2024-09-15T03:30:46.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.742+0000 7f59c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T03:30:46.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.742+0000 7f59c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.840+0000 7f59dce17640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.840+0000 7f59dce17640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.840+0000 7f59dce17640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.840+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f59d8136160 con 0x7f59c00099c0 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.841+0000 7f59c67fc640 1 -- 172.21.15.181:0/1959018290 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v49) v1 ==== 177+0+0 (secure 0 0 0) 0x7f59c8019780 con 0x7f59c00099c0 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.841+0000 7f59c67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.841+0000 7f59c67fc640 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-15T03:30:46.842 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo-ssd already exists 2024-09-15T03:30:46.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f59b80386f0 msgr2=0x7f59b803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:46.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f59b80386f0 0x7f59b803aba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f59cc006fd0 tx=0x7f59cc006d30 comp rx=0 tx=0).stop 2024-09-15T03:30:46.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 10 monclient: shutdown 2024-09-15T03:30:46.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:46.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 msgr2=0x7f59d8099790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59c800d8f0 tx=0x7f59c8008040 comp rx=0 tx=0).stop 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 shutdown_connections 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f59b80386f0 0x7f59b803aba0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d809a5e0 0x7f59d8147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59d8099cd0 0x7f59d809a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 --2- 172.21.15.181:0/1959018290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59c00099c0 0x7f59d8099790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 >> 172.21.15.181:0/1959018290 conn(0x7f59d80a5d40 msgr2=0x7f59d80a35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 shutdown_connections 2024-09-15T03:30:46.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.843+0000 7f59dce17640 1 -- 172.21.15.181:0/1959018290 wait complete. 2024-09-15T03:30:46.854 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-15T03:30:46.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.931+0000 7f838cf14640 10 monclient: get_monmap_and_config 2024-09-15T03:30:46.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.931+0000 7f838cf14640 10 monclient: build_initial_monmap 2024-09-15T03:30:46.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.931+0000 7f838cf14640 10 monclient: monmap: 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:46.932328+0000 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:46.932328+0000 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:46.933 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 Processor -- start 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 -- start start 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient: init 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f83880a0ad0 0x7f83880691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): picked mon.noname-c con 0x7f83880a0ad0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f838809ee60 0x7f838809f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): picked mon.noname-a con 0x7f838809ee60 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8388131ad0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:46.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838806a5b0 con 0x7f8388131ad0 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838809f7a0 con 0x7f838809ee60 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83880a0ea0 con 0x7f83880a0ad0 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838cf14640 10 monclient(hunting): authenticate will time out at 851.413782s 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f83880a0ad0 0x7f83880691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f8386d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f8388131ad0 0x7f8388131ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35938/0 (socket says 172.21.15.181:35938) 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f8386d9d640 1 -- 172.21.15.181:0/2881290462 learned_addr learned my addr 172.21.15.181:0/2881290462 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 10 monclient(hunting): get_auth_request con 0x7f83880a0ad0 auth_method 0 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:46.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f837ffff640 1 -- 172.21.15.181:0/2881290462 >> v2:172.21.15.153:3300/0 conn(0x7f838809ee60 msgr2=0x7f838809f260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f837ffff640 1 --2- 172.21.15.181:0/2881290462 >> v2:172.21.15.153:3300/0 conn(0x7f838809ee60 0x7f838809f260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.932+0000 7f838659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 10 monclient(hunting): handle_auth_done global_id 44212 payload 274 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 1 -- 172.21.15.181:0/2881290462 >> v2:172.21.15.153:3300/0 conn(0x7f838809ee60 msgr2=0x7f838809f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 1 --2- 172.21.15.181:0/2881290462 >> v2:172.21.15.153:3300/0 conn(0x7f838809ee60 0x7f838809f260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 1 -- 172.21.15.181:0/2881290462 >> v1:172.21.15.181:6789/0 conn(0x7f8388131ad0 legacy=0x7f8388131ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:46.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 1 monclient: found mon.noname-c 2024-09-15T03:30:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.933+0000 7f838659c640 1 -- 172.21.15.181:0/2881290462 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880a0460 con 0x7f83880a0ad0 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.935+0000 7f838659c640 1 -- 172.21.15.181:0/2881290462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 msgr2=0x7f83880691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.935+0000 7f838659c640 1 --2- 172.21.15.181:0/2881290462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f83880691d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f838806abc0 tx=0x7f83780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15aea3852228ab56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 1 -- 172.21.15.181:0/2881290462 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378032440 con 0x7f83880a0ad0 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: dump: 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: _finish_auth 0 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 1 -- 172.21.15.181:0/2881290462 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83780325a0 con 0x7f83880a0ad0 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 5 monclient: authenticate success, global_id 44212 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 4 set_mon_vals no callback set 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 1 -- 172.21.15.181:0/2881290462 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378032940 con 0x7f83880a0ad0 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f8385d9b640 10 monclient: dump: 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 10 monclient: shutdown 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 -- 172.21.15.181:0/2881290462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 msgr2=0x7f83880691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 --2- 172.21.15.181:0/2881290462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f83880691d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f838806abc0 tx=0x7f83780090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 -- 172.21.15.181:0/2881290462 shutdown_connections 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 --2- 172.21.15.181:0/2881290462 >> v2:172.21.15.153:3300/0 conn(0x7f838809ee60 0x7f838809f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 --2- 172.21.15.181:0/2881290462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f83880691d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 -- 172.21.15.181:0/2881290462 >> 172.21.15.181:0/2881290462 conn(0x7f8388071370 msgr2=0x7f83880a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 -- 172.21.15.181:0/2881290462 shutdown_connections 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.936+0000 7f838cf14640 1 -- 172.21.15.181:0/2881290462 wait complete. 2024-09-15T03:30:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient: build_initial_monmap 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient: monmap: 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:46.949 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 Processor -- start 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 -- start start 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient: init 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838809ee60 0x7f8388130ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): picked mon.noname-a con 0x7f838809ee60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f8388140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): picked mon.noname-b con 0x7f83880a0ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8388131ad0 0x7f8388141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): picked mon.noname-c con 0x7f8388131ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:46.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838811f0b0 con 0x7f838809ee60 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838811cd70 con 0x7f83880a0ad0 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838811edd0 con 0x7f8388131ad0 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838cf14640 10 monclient(hunting): authenticate will time out at 851.418805s 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f8388140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f8388140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34394/0 (socket says 172.21.15.181:34394) 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 1 -- 172.21.15.181:0/2932980557 learned_addr learned my addr 172.21.15.181:0/2932980557 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 10 monclient(hunting): get_auth_request con 0x7f83880a0ad0 auth_method 0 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:46.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838659c640 1 -- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838809ee60 msgr2=0x7f8388130ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f838659c640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838809ee60 0x7f8388130ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.937+0000 7f837ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f8386d9d640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8388131ad0 0x7f8388141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f8386d9d640 10 monclient(hunting): get_auth_request con 0x7f8388131ad0 auth_method 0 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f8386d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f8386d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f8386d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 10 monclient(hunting): handle_auth_done global_id 44215 payload 824 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 1 -- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8388131ad0 msgr2=0x7f8388141460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8388131ad0 0x7f8388141460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 1 -- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838809ee60 msgr2=0x7f8388130ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838809ee60 0x7f8388130ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 1 monclient: found mon.noname-b 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.938+0000 7f837ffff640 1 -- 172.21.15.181:0/2932980557 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8378009390 con 0x7f83880a0ad0 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837ffff640 1 -- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 msgr2=0x7f8388140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837ffff640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f8388140f20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f837000b600 tx=0x7f83700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370015070 con 0x7f83880a0ad0 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: dump: 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:46.953 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370003d90 con 0x7f83880a0ad0 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 5 monclient: authenticate success, global_id 44215 2024-09-15T03:30:46.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 10 monclient: _renew_subs 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838811f390 con 0x7f83880a0ad0 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370006380 con 0x7f83880a0ad0 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f837e7fc640 10 monclient: dump: 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:46.955 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 10 monclient: _renew_subs 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838559a640 4 set_mon_vals no callback set 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.940+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83881311e0 con 0x7f83880a0ad0 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8370010440 con 0x7f83880a0ad0 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f838cf14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f838cf14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f838cf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83880008d0 con 0x7f83880a0ad0 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f837e7fc640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8368038770 0x7f836803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:46.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4767+0+0 (secure 0 0 0) 0x7f837004b4e0 con 0x7f83880a0ad0 2024-09-15T03:30:46.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f838659c640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8368038770 0x7f836803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:46.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.941+0000 7f838659c640 10 monclient: get_auth_request con 0x7f8368038770 auth_method 0 2024-09-15T03:30:46.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.944+0000 7f838659c640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8368038770 0x7f836803ac20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f83880675c0 tx=0x7f8378032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:46.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.945+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8370018af0 con 0x7f83880a0ad0 2024-09-15T03:30:46.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.945+0000 7f837e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:46.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:46.945+0000 7f837e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.102+0000 7f838cf14640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T03:30:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.102+0000 7f838cf14640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T03:30:47.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.102+0000 7f838cf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.102+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f83881314c0 con 0x7f83880a0ad0 2024-09-15T03:30:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.743+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v50) v1 ==== 150+0+0 (secure 0 0 0) 0x7f837004b030 con 0x7f83880a0ad0 2024-09-15T03:30:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.743+0000 7f837e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T03:30:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.743+0000 7f837e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.839+0000 7f838cf14640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.839+0000 7f838cf14640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.839+0000 7f838cf14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.839+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f8388064a20 con 0x7f83880a0ad0 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.840+0000 7f837e7fc640 1 -- 172.21.15.181:0/2932980557 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v50) v1 ==== 177+0+0 (secure 0 0 0) 0x7f837004b020 con 0x7f83880a0ad0 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.840+0000 7f837e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.840+0000 7f837e7fc640 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-15T03:30:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo-hdd already exists 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8368038770 msgr2=0x7f836803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8368038770 0x7f836803ac20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f83880675c0 tx=0x7f8378032000 comp rx=0 tx=0).stop 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 10 monclient: shutdown 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 msgr2=0x7f8388140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f8388140f20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f837000b600 tx=0x7f83700091a0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 shutdown_connections 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8368038770 0x7f836803ac20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8388131ad0 0x7f8388141460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83880a0ad0 0x7f8388140f20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 --2- 172.21.15.181:0/2932980557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838809ee60 0x7f8388130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 >> 172.21.15.181:0/2932980557 conn(0x7f8388071370 msgr2=0x7f83880a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:47.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 shutdown_connections 2024-09-15T03:30:47.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.841+0000 7f838cf14640 1 -- 172.21.15.181:0/2932980557 wait complete. 2024-09-15T03:30:47.852 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-15T03:30:47.852 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo-ssd 2024-09-15T03:30:47.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.925+0000 7fc2d219d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:47.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.925+0000 7fc2d219d640 10 monclient: build_initial_monmap 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.925+0000 7fc2d219d640 10 monclient: monmap: 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:47.926325+0000 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:47.926325+0000 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:47.928 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 Processor -- start 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 -- start start 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient: init 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2cc09cad0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc09c1a0 0x7fc2cc1348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2cc09c1a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc2cc134e10 0x7fc2cc1371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2cc134e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:47.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc09d5b0 con 0x7fc2cc09cad0 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2cc09b710 con 0x7fc2cc134e10 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc09a650 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2d219d640 10 monclient(hunting): authenticate will time out at 852.407880s 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.926+0000 7fc2cb7fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fc2cc09cad0 legacy=0x7fc2cc09cea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc2cc134e10 0x7fc2cc1371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc2cc134e10 0x7fc2cc1371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34410/0 (socket says 172.21.15.181:34410) 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 -- 172.21.15.181:0/4055042424 learned_addr learned my addr 172.21.15.181:0/4055042424 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:47.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): get_auth_request con 0x7fc2cc134e10 auth_method 0 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2caffd640 1 --2- 172.21.15.181:0/4055042424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc09c1a0 0x7fc2cc1348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2caffd640 10 monclient(hunting): get_auth_request con 0x7fc2cc09c1a0 auth_method 0 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient(hunting): handle_auth_done global_id 44218 payload 274 2024-09-15T03:30:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 -- 172.21.15.181:0/4055042424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc09c1a0 msgr2=0x7fc2cc1348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 --2- 172.21.15.181:0/4055042424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc09c1a0 0x7fc2cc1348d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 -- 172.21.15.181:0/4055042424 >> v1:172.21.15.153:6789/0 conn(0x7fc2cc09cad0 legacy=0x7fc2cc09cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 monclient: found mon.noname-c 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.927+0000 7fc2cbfff640 1 -- 172.21.15.181:0/4055042424 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc09f660 con 0x7fc2cc134e10 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2cbfff640 1 -- 172.21.15.181:0/4055042424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc134e10 msgr2=0x7fc2cc1371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2cbfff640 1 --2- 172.21.15.181:0/4055042424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc134e10 0x7fc2cc1371f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009f20 tx=0x7fc2c4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77f854b64123a38f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 1 -- 172.21.15.181:0/4055042424 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4007610 con 0x7fc2cc134e10 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: dump: 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 5 monclient: authenticate success, global_id 44218 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 1 -- 172.21.15.181:0/4055042424 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c402d050 con 0x7fc2cc134e10 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 4 set_mon_vals no callback set 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 1 -- 172.21.15.181:0/4055042424 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40079a0 con 0x7fc2cc134e10 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2ca7fc640 10 monclient: dump: 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 10 monclient: shutdown 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 -- 172.21.15.181:0/4055042424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc134e10 msgr2=0x7fc2cc1371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 --2- 172.21.15.181:0/4055042424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc134e10 0x7fc2cc1371f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009f20 tx=0x7fc2c4009840 comp rx=0 tx=0).stop 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 -- 172.21.15.181:0/4055042424 shutdown_connections 2024-09-15T03:30:47.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 --2- 172.21.15.181:0/4055042424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc134e10 0x7fc2cc1371f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 --2- 172.21.15.181:0/4055042424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc09c1a0 0x7fc2cc1348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 -- 172.21.15.181:0/4055042424 >> 172.21.15.181:0/4055042424 conn(0x7fc2cc070e90 msgr2=0x7fc2cc0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 -- 172.21.15.181:0/4055042424 shutdown_connections 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.930+0000 7fc2d219d640 1 -- 172.21.15.181:0/4055042424 wait complete. 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.931+0000 7fc2d219d640 10 monclient: build_initial_monmap 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.931+0000 7fc2d219d640 10 monclient: monmap: 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:47.940 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.931+0000 7fc2d219d640 1 Processor -- start 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.931+0000 7fc2d219d640 1 -- start start 2024-09-15T03:30:47.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.931+0000 7fc2d219d640 10 monclient: init 2024-09-15T03:30:47.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 0x7fc2cc11c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2cc09c1a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2cc09cad0 0x7fc2cc11c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2cc09cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc134e10 0x7fc2cc11cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2cc134e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc11e2e0 con 0x7fc2cc09cad0 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0a2090 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc120550 con 0x7fc2cc134e10 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2d219d640 10 monclient(hunting): authenticate will time out at 852.413296s 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 0x7fc2cc11c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 0x7fc2cc11c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34426/0 (socket says 172.21.15.181:34426) 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 -- 172.21.15.181:0/2649532409 learned_addr learned my addr 172.21.15.181:0/2649532409 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): get_auth_request con 0x7fc2cc09c1a0 auth_method 0 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2caffd640 1 -- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2cc09cad0 msgr2=0x7fc2cc11c810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2caffd640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2cc09cad0 0x7fc2cc11c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:47.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cbfff640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc134e10 0x7fc2cc11cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cbfff640 10 monclient(hunting): get_auth_request con 0x7fc2cc134e10 auth_method 0 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient(hunting): handle_auth_done global_id 44221 payload 824 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 -- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc134e10 msgr2=0x7fc2cc11cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc134e10 0x7fc2cc11cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 -- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2cc09cad0 msgr2=0x7fc2cc11c810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2cc09cad0 0x7fc2cc11c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cb7fe640 1 -- 172.21.15.181:0/2649532409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4009ae0 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.932+0000 7fc2cbfff640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc134e10 0x7fc2cc11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2cb7fe640 1 -- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 msgr2=0x7fc2cc11c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2cb7fe640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 0x7fc2cc11c2d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00b600 tx=0x7fc2bc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc015070 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 10 monclient: dump: 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:47.949 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2d219d640 5 monclient: authenticate success, global_id 44221 2024-09-15T03:30:47.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.934+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc003d90 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 monclient: _renew_subs 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc06eba0 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2b37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc006380 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2b37fe640 10 monclient: dump: 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:47.951 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 monclient: _renew_subs 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc11fde0 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2c9ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc2bc010440 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:47.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.935+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc0008d0 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.936+0000 7fc2b37fe640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2b80386f0 0x7fc2b803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.936+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fc2bc04b560 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.936+0000 7fc2caffd640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2b80386f0 0x7fc2b803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.936+0000 7fc2caffd640 10 monclient: get_auth_request con 0x7fc2b80386f0 auth_method 0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.938+0000 7fc2caffd640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2b80386f0 0x7fc2b803aba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0006f40 tx=0x7fc2c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.938+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.940+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc2bc018ab0 con 0x7fc2cc09c1a0 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.940+0000 7fc2b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:47.940+0000 7fc2b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2d219d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2d219d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2d219d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fc2cc064a20 con 0x7fc2cc09c1a0 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fc2bc018690 con 0x7fc2cc09c1a0 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.097+0000 7fc2b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:48.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2d219d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2d219d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2d219d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fc2cc11e9f0 con 0x7fc2cc09c1a0 2024-09-15T03:30:48.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2b37fe640 1 -- 172.21.15.181:0/2649532409 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fc2bc0280a0 con 0x7fc2cc09c1a0 2024-09-15T03:30:48.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2b37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.191+0000 7fc2b37fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:48.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2b80386f0 msgr2=0x7fc2b803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2b80386f0 0x7fc2b803aba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0006f40 tx=0x7fc2c0008040 comp rx=0 tx=0).stop 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 10 monclient: shutdown 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 msgr2=0x7fc2cc11c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 0x7fc2cc11c2d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00b600 tx=0x7fc2bc0091a0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 shutdown_connections 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2b80386f0 0x7fc2b803aba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2cc134e10 0x7fc2cc11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2cc09cad0 0x7fc2cc11c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 --2- 172.21.15.181:0/2649532409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2cc09c1a0 0x7fc2cc11c2d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 >> 172.21.15.181:0/2649532409 conn(0x7fc2cc070e90 msgr2=0x7fc2cc098a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 shutdown_connections 2024-09-15T03:30:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.193+0000 7fc2d219d640 1 -- 172.21.15.181:0/2649532409 wait complete. 2024-09-15T03:30:48.203 INFO:tasks.workunit.client.0.smithi181.stdout:foo-ssd 2024-09-15T03:30:48.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-15T03:30:48.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false grep foo-hdd 2024-09-15T03:30:48.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:48.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo-hdd 2024-09-15T03:30:48.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.276+0000 7fbaf631d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.276+0000 7fbaf631d640 10 monclient: build_initial_monmap 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.276+0000 7fbaf631d640 10 monclient: monmap: 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:48.277485+0000 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:48.277485+0000 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:48.278 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:48.279 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:48.279 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:48.279 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:48.279 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 Processor -- start 2024-09-15T03:30:48.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 -- start start 2024-09-15T03:30:48.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient: init 2024-09-15T03:30:48.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:48.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T03:30:48.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbaf0098c70 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbaf0099c70 0x7fbaf0136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf0099c70 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbaf0137200 0x7fbaf01395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf0137200 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf009f680 con 0x7fbaf0098c70 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf009f420 con 0x7fbaf0099c70 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf0068a20 con 0x7fbaf0137200 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.277+0000 7fbaf631d640 10 monclient(hunting): authenticate will time out at 852.758955s 2024-09-15T03:30:48.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaeffff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fbaf0098c70 legacy=0x7fbaf0099040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaef7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fbaf0099c70 msgr2=0x7fbaf0136cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaef7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbaf0099c70 0x7fbaf0136cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbaf0137200 0x7fbaf01395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbaf0137200 0x7fbaf01395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49896/0 (socket says 172.21.15.181:49896) 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 -- 172.21.15.181:0/1693532212 learned_addr learned my addr 172.21.15.181:0/1693532212 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): get_auth_request con 0x7fbaf0137200 auth_method 0 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:48.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_done global_id 44224 payload 274 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 -- 172.21.15.181:0/1693532212 >> v2:172.21.15.153:3300/0 conn(0x7fbaf0099c70 msgr2=0x7fbaf0136cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 --2- 172.21.15.181:0/1693532212 >> v2:172.21.15.153:3300/0 conn(0x7fbaf0099c70 0x7fbaf0136cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 -- 172.21.15.181:0/1693532212 >> v1:172.21.15.153:6789/0 conn(0x7fbaf0098c70 legacy=0x7fbaf0099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient: _finish_hunting 0 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 monclient: found mon.noname-c 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.278+0000 7fbaf48ba640 1 -- 172.21.15.181:0/1693532212 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf0096a30 con 0x7fbaf0137200 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf48ba640 1 -- 172.21.15.181:0/1693532212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 msgr2=0x7fbaf01395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf48ba640 1 --2- 172.21.15.181:0/1693532212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf01395e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0096c20 tx=0x7fbae8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e88e3ad00140a0a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 1 -- 172.21.15.181:0/1693532212 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8007610 con 0x7fbaf0137200 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: dump: 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 1 -- 172.21.15.181:0/1693532212 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae802d050 con 0x7fbaf0137200 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 5 monclient: authenticate success, global_id 44224 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 4 set_mon_vals no callback set 2024-09-15T03:30:48.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 1 -- 172.21.15.181:0/1693532212 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae80079a0 con 0x7fbaf0137200 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaeeffd640 10 monclient: dump: 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 10 monclient: shutdown 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 -- 172.21.15.181:0/1693532212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 msgr2=0x7fbaf01395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 --2- 172.21.15.181:0/1693532212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf01395e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0096c20 tx=0x7fbae8009840 comp rx=0 tx=0).stop 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 -- 172.21.15.181:0/1693532212 shutdown_connections 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 --2- 172.21.15.181:0/1693532212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf01395e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 --2- 172.21.15.181:0/1693532212 >> v2:172.21.15.153:3300/0 conn(0x7fbaf0099c70 0x7fbaf0136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 -- 172.21.15.181:0/1693532212 >> 172.21.15.181:0/1693532212 conn(0x7fbaf0070e90 msgr2=0x7fbaf00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 -- 172.21.15.181:0/1693532212 shutdown_connections 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.281+0000 7fbaf631d640 1 -- 172.21.15.181:0/1693532212 wait complete. 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.282+0000 7fbaf631d640 10 monclient: build_initial_monmap 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.282+0000 7fbaf631d640 10 monclient: monmap: 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:48.293 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.282+0000 7fbaf631d640 1 Processor -- start 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.282+0000 7fbaf631d640 1 -- start start 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.282+0000 7fbaf631d640 10 monclient: init 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:48.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbaf0098c70 0x7fbaf011e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf0098c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf0099c70 0x7fbaf011ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf0099c70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf011f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): picked mon.noname-b con 0x7fbaf0137200 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0136d00 con 0x7fbaf0099c70 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf006eba0 con 0x7fbaf0137200 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00a1af0 con 0x7fbaf0098c70 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:48.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf631d640 10 monclient(hunting): authenticate will time out at 852.764227s 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf011f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf011f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49906/0 (socket says 172.21.15.181:49906) 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 -- 172.21.15.181:0/3020843906 learned_addr learned my addr 172.21.15.181:0/3020843906 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): get_auth_request con 0x7fbaf0137200 auth_method 0 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaef7fe640 1 -- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf0099c70 msgr2=0x7fbaf011ed50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaef7fe640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf0099c70 0x7fbaf011ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:48.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient(hunting): handle_auth_done global_id 44227 payload 824 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaeffff640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbaf0098c70 0x7fbaf011e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 -- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbaf0098c70 msgr2=0x7fbaf011e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbaf0098c70 0x7fbaf011e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 -- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf0099c70 msgr2=0x7fbaf011ed50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf0099c70 0x7fbaf011ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient: _finish_hunting 0 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 monclient: found mon.noname-b 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.283+0000 7fbaf48ba640 1 -- 172.21.15.181:0/3020843906 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8009ae0 con 0x7fbaf0137200 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbaf48ba640 1 -- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 msgr2=0x7fbaf011f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbaf48ba640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf011f290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009880 tx=0x7fbae8036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8031cb0 con 0x7fbaf0137200 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 10 monclient: dump: 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.298 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae8031e10 con 0x7fbaf0137200 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.285+0000 7fbaf631d640 5 monclient: authenticate success, global_id 44227 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 monclient: _renew_subs 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf01201a0 con 0x7fbaf0137200 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbad7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:48.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae803c650 con 0x7fbaf0137200 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbad7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbad7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbad7fff640 10 monclient: dump: 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.300 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 monclient: _renew_subs 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf0120d30 con 0x7fbaf0137200 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaee7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbae803ca80 con 0x7fbaf0137200 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.286+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf0122130 con 0x7fbaf0137200 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.287+0000 7fbad7fff640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbadc0386f0 0x7fbadc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.287+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fbae806ce50 con 0x7fbaf0137200 2024-09-15T03:30:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.287+0000 7fbaeffff640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbadc0386f0 0x7fbadc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.287+0000 7fbaeffff640 10 monclient: get_auth_request con 0x7fbadc0386f0 auth_method 0 2024-09-15T03:30:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.289+0000 7fbaeffff640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbadc0386f0 0x7fbadc03aba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbae0006fd0 tx=0x7fbae0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.291+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbae803cd30 con 0x7fbaf0137200 2024-09-15T03:30:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.291+0000 7fbad7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.291+0000 7fbad7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:48.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbaf631d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbaf631d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbaf631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fbaf009a070 con 0x7fbaf0137200 2024-09-15T03:30:48.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fbae8040030 con 0x7fbaf0137200 2024-09-15T03:30:48.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbad7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.449+0000 7fbad7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:48.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbaf631d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbaf631d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbaf631d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fbaf00008d0 con 0x7fbaf0137200 2024-09-15T03:30:48.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbad7fff640 1 -- 172.21.15.181:0/3020843906 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fbae8040070 con 0x7fbaf0137200 2024-09-15T03:30:48.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbad7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-15T03:30:48.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.543+0000 7fbad7fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:48.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbadc0386f0 msgr2=0x7fbadc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbadc0386f0 0x7fbadc03aba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbae0006fd0 tx=0x7fbae0008040 comp rx=0 tx=0).stop 2024-09-15T03:30:48.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 10 monclient: shutdown 2024-09-15T03:30:48.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:48.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 msgr2=0x7fbaf011f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf011f290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009880 tx=0x7fbae8036000 comp rx=0 tx=0).stop 2024-09-15T03:30:48.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 shutdown_connections 2024-09-15T03:30:48.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbadc0386f0 0x7fbadc03aba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf0137200 0x7fbaf011f290 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf0099c70 0x7fbaf011ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 --2- 172.21.15.181:0/3020843906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbaf0098c70 0x7fbaf011e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.545+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 >> 172.21.15.181:0/3020843906 conn(0x7fbaf0070e90 msgr2=0x7fbaf0097230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:48.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.546+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 shutdown_connections 2024-09-15T03:30:48.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.546+0000 7fbaf631d640 1 -- 172.21.15.181:0/3020843906 wait complete. 2024-09-15T03:30:48.556 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:48.556 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-15T03:30:48.556 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo-hdd 2024-09-15T03:30:48.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.630+0000 7fa7e895d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.630+0000 7fa7e895d640 10 monclient: build_initial_monmap 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.630+0000 7fa7e895d640 10 monclient: monmap: 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:48.631229+0000 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:48.631229+0000 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:48.633 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.630+0000 7fa7e895d640 1 Processor -- start 2024-09-15T03:30:48.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 1 -- start start 2024-09-15T03:30:48.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient: init 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa7e40a19a0 0x7fa7e409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e40a19a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7e409be20 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e409c1f0 0x7fa7e4135700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e409c1f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40a1d70 con 0x7fa7e409be20 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e40988e0 con 0x7fa7e40a19a0 2024-09-15T03:30:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e4098600 con 0x7fa7e409c1f0 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e895d640 10 monclient(hunting): authenticate will time out at 853.112633s 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa7e40a19a0 0x7fa7e409b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa7e40a19a0 0x7fa7e409b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49920/0 (socket says 172.21.15.181:49920) 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 1 -- 172.21.15.181:0/3885688782 learned_addr learned my addr 172.21.15.181:0/3885688782 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): get_auth_request con 0x7fa7e40a19a0 auth_method 0 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e1d9b640 1 --1- 172.21.15.181:0/3885688782 >> v1:172.21.15.181:6789/0 conn(0x7fa7e409be20 0x7fa7e4099250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40846/0 (socket says 172.21.15.181:40846) 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.631+0000 7fa7e259c640 10 monclient(hunting): handle_auth_done global_id 44230 payload 274 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 1 -- 172.21.15.181:0/3885688782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e409c1f0 msgr2=0x7fa7e4135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 1 --2- 172.21.15.181:0/3885688782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e409c1f0 0x7fa7e4135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 1 -- 172.21.15.181:0/3885688782 >> v1:172.21.15.181:6789/0 conn(0x7fa7e409be20 legacy=0x7fa7e4099250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 1 monclient: found mon.noname-c 2024-09-15T03:30:48.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:48.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.632+0000 7fa7e259c640 1 -- 172.21.15.181:0/3885688782 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4097a70 con 0x7fa7e40a19a0 2024-09-15T03:30:48.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e259c640 1 -- 172.21.15.181:0/3885688782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e40a19a0 msgr2=0x7fa7e409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:48.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e259c640 1 --2- 172.21.15.181:0/3885688782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e40a19a0 0x7fa7e409b8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4098a20 tx=0x7fa7d40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3c11dc56811c33a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 1 -- 172.21.15.181:0/3885688782 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4032440 con 0x7fa7e40a19a0 2024-09-15T03:30:48.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: dump: 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.645 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: _finish_auth 0 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 1 -- 172.21.15.181:0/3885688782 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d40325a0 con 0x7fa7e40a19a0 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 1 -- 172.21.15.181:0/3885688782 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4032980 con 0x7fa7e40a19a0 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.634+0000 7fa7e159a640 10 monclient: dump: 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.646 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 5 monclient: authenticate success, global_id 44230 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 4 set_mon_vals no callback set 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 10 monclient: shutdown 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 -- 172.21.15.181:0/3885688782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e40a19a0 msgr2=0x7fa7e409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 --2- 172.21.15.181:0/3885688782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e40a19a0 0x7fa7e409b8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4098a20 tx=0x7fa7d40091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 -- 172.21.15.181:0/3885688782 shutdown_connections 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 --2- 172.21.15.181:0/3885688782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e409c1f0 0x7fa7e4135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 --2- 172.21.15.181:0/3885688782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e40a19a0 0x7fa7e409b8e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 -- 172.21.15.181:0/3885688782 >> 172.21.15.181:0/3885688782 conn(0x7fa7e40711a0 msgr2=0x7fa7e40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 -- 172.21.15.181:0/3885688782 shutdown_connections 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.635+0000 7fa7e895d640 1 -- 172.21.15.181:0/3885688782 wait complete. 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient: build_initial_monmap 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient: monmap: 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 Processor -- start 2024-09-15T03:30:48.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 -- start start 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient: init 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 0x7fa7e413f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e409be20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e409c1f0 0x7fa7e41335d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e409c1f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e40a19a0 0x7fa7e4133b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e40a19a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e4136c70 con 0x7fa7e409c1f0 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e41356c0 con 0x7fa7e409be20 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e409cc10 con 0x7fa7e40a19a0 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e895d640 10 monclient(hunting): authenticate will time out at 853.117810s 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 0x7fa7e413f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 0x7fa7e413f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49930/0 (socket says 172.21.15.181:49930) 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 1 -- 172.21.15.181:0/2081248896 learned_addr learned my addr 172.21.15.181:0/2081248896 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): get_auth_request con 0x7fa7e409be20 auth_method 0 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e1d9b640 1 -- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e409c1f0 msgr2=0x7fa7e41335d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e1d9b640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e409c1f0 0x7fa7e41335d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.636+0000 7fa7e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e2d9d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e40a19a0 0x7fa7e4133b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e2d9d640 10 monclient(hunting): get_auth_request con 0x7fa7e40a19a0 auth_method 0 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 10 monclient(hunting): handle_auth_done global_id 44233 payload 824 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 1 -- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e40a19a0 msgr2=0x7fa7e4133b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e40a19a0 0x7fa7e4133b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 1 -- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e409c1f0 msgr2=0x7fa7e41335d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e409c1f0 0x7fa7e41335d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 1 monclient: found mon.noname-b 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e259c640 1 -- 172.21.15.181:0/2081248896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4009450 con 0x7fa7e409be20 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.637+0000 7fa7e2d9d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e40a19a0 0x7fa7e4133b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e259c640 1 -- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 msgr2=0x7fa7e413f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:48.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e259c640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 0x7fa7e413f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4035040 tx=0x7fa7d4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40318c0 con 0x7fa7e409be20 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: dump: 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.654 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d4031a20 con 0x7fa7e409be20 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 5 monclient: authenticate success, global_id 44233 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 10 monclient: _renew_subs 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4031de0 con 0x7fa7e409be20 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e4130700 con 0x7fa7e409be20 2024-09-15T03:30:48.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7d27fc640 10 monclient: dump: 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.656 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e0d99640 4 set_mon_vals no callback set 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 10 monclient: _renew_subs 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.639+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e412fcf0 con 0x7fa7e409be20 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.640+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa7d403c610 con 0x7fa7e409be20 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.640+0000 7fa7e895d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.640+0000 7fa7e895d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.640+0000 7fa7e895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.640+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e40008d0 con 0x7fa7e409be20 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.641+0000 7fa7d27fc640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa7c40386f0 0x7fa7c403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.641+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fa7d406c0b0 con 0x7fa7e409be20 2024-09-15T03:30:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.641+0000 7fa7e1d9b640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa7c40386f0 0x7fa7c403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.641+0000 7fa7e1d9b640 10 monclient: get_auth_request con 0x7fa7c40386f0 auth_method 0 2024-09-15T03:30:48.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.643+0000 7fa7e1d9b640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa7c40386f0 0x7fa7c403aba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8004880 tx=0x7fa7d8007ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.643+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:48.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.645+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa7d404aa90 con 0x7fa7e409be20 2024-09-15T03:30:48.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.645+0000 7fa7d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:48.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.645+0000 7fa7d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:48.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.803+0000 7fa7e895d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:48.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.803+0000 7fa7e895d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:48.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.803+0000 7fa7e895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.803+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fa7e4136eb0 con 0x7fa7e409be20 2024-09-15T03:30:48.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.804+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fa7d40370a0 con 0x7fa7e409be20 2024-09-15T03:30:48.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.804+0000 7fa7d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:48.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.804+0000 7fa7d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:48.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.898+0000 7fa7e895d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:48.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.898+0000 7fa7e895d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:48.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.898+0000 7fa7e895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.898+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fa7e4136eb0 con 0x7fa7e409be20 2024-09-15T03:30:48.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.899+0000 7fa7d27fc640 1 -- 172.21.15.181:0/2081248896 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7fa7d404a2b0 con 0x7fa7e409be20 2024-09-15T03:30:48.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.899+0000 7fa7d27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:48.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.899+0000 7fa7d27fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:48.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa7c40386f0 msgr2=0x7fa7c403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa7c40386f0 0x7fa7c403aba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8004880 tx=0x7fa7d8007ac0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 10 monclient: shutdown 2024-09-15T03:30:48.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:48.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 msgr2=0x7fa7e413f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 0x7fa7e413f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4035040 tx=0x7fa7d4032000 comp rx=0 tx=0).stop 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 shutdown_connections 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa7c40386f0 0x7fa7c403aba0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7e40a19a0 0x7fa7e4133b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e409c1f0 0x7fa7e41335d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 --2- 172.21.15.181:0/2081248896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7e409be20 0x7fa7e413f6b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 >> 172.21.15.181:0/2081248896 conn(0x7fa7e40711a0 msgr2=0x7fa7e4099250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 shutdown_connections 2024-09-15T03:30:48.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.901+0000 7fa7e895d640 1 -- 172.21.15.181:0/2081248896 wait complete. 2024-09-15T03:30:48.911 INFO:tasks.workunit.client.0.smithi181.stdout:foo-hdd 2024-09-15T03:30:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-15T03:30:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false grep foo-ssd 2024-09-15T03:30:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo-ssd 2024-09-15T03:30:48.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.984+0000 7f3d5a680640 10 monclient: get_monmap_and_config 2024-09-15T03:30:48.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.984+0000 7f3d5a680640 10 monclient: build_initial_monmap 2024-09-15T03:30:48.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.984+0000 7f3d5a680640 10 monclient: monmap: 2024-09-15T03:30:48.985 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:48.985 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:48.985308+0000 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:48.985308+0000 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:48.986 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 Processor -- start 2024-09-15T03:30:48.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 -- start start 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient: init 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d54098c70 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54099c70 0x7f3d54136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): picked mon.noname-b con 0x7f3d54099c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3d54137200 0x7f3d541395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): picked mon.noname-c con 0x7f3d54137200 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5409f680 con 0x7f3d54098c70 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5409f420 con 0x7f3d54137200 2024-09-15T03:30:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d54068a20 con 0x7f3d54099c70 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d5a680640 10 monclient(hunting): authenticate will time out at 853.466829s 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d53fff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f3d54098c70 legacy=0x7f3d54099040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d58c1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3d54137200 0x7f3d541395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d58c1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3d54137200 0x7f3d541395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49932/0 (socket says 172.21.15.181:49932) 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d58c1d640 1 -- 172.21.15.181:0/1337952495 learned_addr learned my addr 172.21.15.181:0/1337952495 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.985+0000 7f3d58c1d640 10 monclient(hunting): get_auth_request con 0x7f3d54137200 auth_method 0 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:48.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d537fe640 1 --2- 172.21.15.181:0/1337952495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54099c70 0x7f3d54136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d537fe640 10 monclient(hunting): get_auth_request con 0x7f3d54099c70 auth_method 0 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient(hunting): handle_auth_done global_id 44236 payload 274 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 1 -- 172.21.15.181:0/1337952495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54099c70 msgr2=0x7f3d54136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 1 --2- 172.21.15.181:0/1337952495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54099c70 0x7f3d54136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 1 -- 172.21.15.181:0/1337952495 >> v1:172.21.15.153:6789/0 conn(0x7f3d54098c70 legacy=0x7f3d54099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient: _finish_hunting 0 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 1 monclient: found mon.noname-c 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.986+0000 7f3d58c1d640 1 -- 172.21.15.181:0/1337952495 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d54096a30 con 0x7f3d54137200 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.988+0000 7f3d58c1d640 1 -- 172.21.15.181:0/1337952495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54137200 msgr2=0x7f3d541395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d58c1d640 1 --2- 172.21.15.181:0/1337952495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54137200 0x7f3d541395e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009f20 tx=0x7f3d4c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f305623c80d72667 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 1 -- 172.21.15.181:0/1337952495 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c007610 con 0x7f3d54137200 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: dump: 2024-09-15T03:30:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 5 monclient: authenticate success, global_id 44236 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 1 -- 172.21.15.181:0/1337952495 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c02d050 con 0x7f3d54137200 2024-09-15T03:30:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 1 -- 172.21.15.181:0/1337952495 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c0079a0 con 0x7f3d54137200 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 4 set_mon_vals no callback set 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d52ffd640 10 monclient: dump: 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:48.996 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:48.996 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 10 monclient: shutdown 2024-09-15T03:30:49.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:49.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 -- 172.21.15.181:0/1337952495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54137200 msgr2=0x7f3d541395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 --2- 172.21.15.181:0/1337952495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54137200 0x7f3d541395e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009f20 tx=0x7f3d4c009840 comp rx=0 tx=0).stop 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 -- 172.21.15.181:0/1337952495 shutdown_connections 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 --2- 172.21.15.181:0/1337952495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54137200 0x7f3d541395e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 --2- 172.21.15.181:0/1337952495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54099c70 0x7f3d54136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 -- 172.21.15.181:0/1337952495 >> 172.21.15.181:0/1337952495 conn(0x7f3d54070e90 msgr2=0x7f3d540a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 -- 172.21.15.181:0/1337952495 shutdown_connections 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.989+0000 7f3d5a680640 1 -- 172.21.15.181:0/1337952495 wait complete. 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.990+0000 7f3d5a680640 10 monclient: build_initial_monmap 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.990+0000 7f3d5a680640 10 monclient: monmap: 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:49.008 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.990+0000 7f3d5a680640 1 Processor -- start 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.990+0000 7f3d5a680640 1 -- start start 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.990+0000 7f3d5a680640 10 monclient: init 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54098c70 0x7f3d5411e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): picked mon.noname-c con 0x7f3d54098c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 0x7f3d5411eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): picked mon.noname-b con 0x7f3d54099c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:49.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54137200 0x7f3d5411f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): picked mon.noname-a con 0x7f3d54137200 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d54120670 con 0x7f3d54137200 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540a1b30 con 0x7f3d54099c70 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d541361b0 con 0x7f3d54098c70 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d5a680640 10 monclient(hunting): authenticate will time out at 853.472136s 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d58c1d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54137200 msgr2=0x7f3d5411f050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d58c1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54137200 0x7f3d5411f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 0x7f3d5411eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 0x7f3d5411eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49946/0 (socket says 172.21.15.181:49946) 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 -- 172.21.15.181:0/3815834334 learned_addr learned my addr 172.21.15.181:0/3815834334 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): get_auth_request con 0x7f3d54099c70 auth_method 0 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d53fff640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54098c70 0x7f3d5411e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d53fff640 10 monclient(hunting): get_auth_request con 0x7f3d54098c70 auth_method 0 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient(hunting): handle_auth_done global_id 44239 payload 824 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 -- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54098c70 msgr2=0x7f3d5411e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54098c70 0x7f3d5411e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 -- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54137200 msgr2=0x7f3d5411f050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54137200 0x7f3d5411f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:49.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.991+0000 7f3d537fe640 1 -- 172.21.15.181:0/3815834334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c009ae0 con 0x7f3d54099c70 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d537fe640 1 -- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 msgr2=0x7f3d5411eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d537fe640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 0x7f3d5411eb10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800d890 tx=0x7f3d48008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4800de20 con 0x7f3d54099c70 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: dump: 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: _finish_auth 0 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4800bae0 con 0x7f3d54099c70 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 5 monclient: authenticate success, global_id 44239 2024-09-15T03:30:49.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 10 monclient: _renew_subs 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5406eba0 con 0x7f3d54099c70 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4800be80 con 0x7f3d54099c70 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:49.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d3bfff640 10 monclient: dump: 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 10 monclient: _renew_subs 2024-09-15T03:30:49.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d527fc640 4 set_mon_vals no callback set 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.993+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d54120120 con 0x7f3d54099c70 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.994+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3d48014640 con 0x7f3d54099c70 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.994+0000 7f3d5a680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.994+0000 7f3d5a680640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.994+0000 7f3d5a680640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.994+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5409a070 con 0x7f3d54099c70 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.995+0000 7f3d3bfff640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038740 0x7f3d4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.995+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f3d4804c1c0 con 0x7f3d54099c70 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.995+0000 7f3d53fff640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038740 0x7f3d4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.995+0000 7f3d53fff640 10 monclient: get_auth_request con 0x7f3d40038740 auth_method 0 2024-09-15T03:30:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.997+0000 7f3d53fff640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038740 0x7f3d4003abf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3d440038f0 tx=0x7f3d44006ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.999+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3d48011570 con 0x7f3d54099c70 2024-09-15T03:30:49.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.999+0000 7f3d3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:48.999+0000 7f3d3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:49.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d5a680640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:49.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d5a680640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:49.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d5a680640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f3d540008d0 con 0x7f3d54099c70 2024-09-15T03:30:49.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7f3d4801dbb0 con 0x7f3d54099c70 2024-09-15T03:30:49.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:49.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.157+0000 7f3d3bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:49.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.251+0000 7f3d5a680640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:49.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.251+0000 7f3d5a680640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:49.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.251+0000 7f3d5a680640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.251+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f3d54121e00 con 0x7f3d54099c70 2024-09-15T03:30:49.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.252+0000 7f3d3bfff640 1 -- 172.21.15.181:0/3815834334 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (secure 0 0 0) 0x7f3d48011340 con 0x7f3d54099c70 2024-09-15T03:30:49.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.252+0000 7f3d3bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-15T03:30:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.252+0000 7f3d3bfff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:49.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038740 msgr2=0x7f3d4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038740 0x7f3d4003abf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3d440038f0 tx=0x7f3d44006ef0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 10 monclient: shutdown 2024-09-15T03:30:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 msgr2=0x7f3d5411eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 0x7f3d5411eb10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800d890 tx=0x7f3d48008040 comp rx=0 tx=0).stop 2024-09-15T03:30:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 shutdown_connections 2024-09-15T03:30:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038740 0x7f3d4003abf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d54137200 0x7f3d5411f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d54099c70 0x7f3d5411eb10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 --2- 172.21.15.181:0/3815834334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d54098c70 0x7f3d5411e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 >> 172.21.15.181:0/3815834334 conn(0x7f3d54070e90 msgr2=0x7f3d540972d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:49.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 shutdown_connections 2024-09-15T03:30:49.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.254+0000 7f3d5a680640 1 -- 172.21.15.181:0/3815834334 wait complete. 2024-09-15T03:30:49.264 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:30:49.264 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-15T03:30:49.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.340+0000 7f4ce7678640 10 monclient: get_monmap_and_config 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.340+0000 7f4ce7678640 10 monclient: build_initial_monmap 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.340+0000 7f4ce7678640 10 monclient: monmap: 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:49.341621+0000 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:49.341621+0000 2024-09-15T03:30:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:49.343 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.341+0000 7f4ce7678640 1 Processor -- start 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.341+0000 7f4ce7678640 1 -- start start 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.341+0000 7f4ce7678640 10 monclient: init 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4ce00a0ad0 0x7f4ce00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce00a0ad0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce009ee60 0x7f4ce009f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce009ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ce0131ad0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce006a5b0 con 0x7f4ce0131ad0 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce009f7a0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce00a0ea0 con 0x7f4ce009ee60 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce7678640 10 monclient(hunting): authenticate will time out at 853.823147s 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5c15640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f4ce0131ad0 legacy=0x7f4ce0131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4ce00a0ad0 0x7f4ce00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4ce00a0ad0 0x7f4ce00691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49948/0 (socket says 172.21.15.181:49948) 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 -- 172.21.15.181:0/25233841 learned_addr learned my addr 172.21.15.181:0/25233841 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): get_auth_request con 0x7f4ce00a0ad0 auth_method 0 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 1 --2- 172.21.15.181:0/25233841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce009ee60 0x7f4ce009f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): get_auth_request con 0x7f4ce009ee60 auth_method 0 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient(hunting): handle_auth_done global_id 44242 payload 274 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 -- 172.21.15.181:0/25233841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce009ee60 msgr2=0x7f4ce009f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 --2- 172.21.15.181:0/25233841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce009ee60 0x7f4ce009f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 -- 172.21.15.181:0/25233841 >> v1:172.21.15.153:6789/0 conn(0x7f4ce0131ad0 legacy=0x7f4ce0131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient: _finish_hunting 0 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 monclient: found mon.noname-c 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.342+0000 7f4ce5414640 1 -- 172.21.15.181:0/25233841 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce00a0460 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4ce5414640 1 -- 172.21.15.181:0/25233841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 msgr2=0x7f4ce00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:49.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4ce5414640 1 --2- 172.21.15.181:0/25233841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce00691d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009910 tx=0x7f4cdc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=294f922128c7ce36 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 1 -- 172.21.15.181:0/25233841 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc02aea0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: dump: 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:49.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 1 -- 172.21.15.181:0/25233841 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc032440 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4ce7678640 5 monclient: authenticate success, global_id 44242 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4ce7678640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4ce7678640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4ce7678640 4 set_mon_vals no callback set 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 1 -- 172.21.15.181:0/25233841 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0327e0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.345+0000 7f4cd7fff640 10 monclient: dump: 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.359 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 10 monclient: shutdown 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 -- 172.21.15.181:0/25233841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 msgr2=0x7f4ce00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 --2- 172.21.15.181:0/25233841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce00691d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009910 tx=0x7f4cdc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 -- 172.21.15.181:0/25233841 shutdown_connections 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 --2- 172.21.15.181:0/25233841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce009ee60 0x7f4ce009f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 --2- 172.21.15.181:0/25233841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce00691d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 -- 172.21.15.181:0/25233841 >> 172.21.15.181:0/25233841 conn(0x7f4ce0071370 msgr2=0x7f4ce00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 -- 172.21.15.181:0/25233841 shutdown_connections 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 -- 172.21.15.181:0/25233841 wait complete. 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 10 monclient: build_initial_monmap 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 10 monclient: monmap: 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:49.361 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.346+0000 7f4ce7678640 1 Processor -- start 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 -- start start 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient: init 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce009ee60 0x7f4ce009a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce009ee60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce0130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce00a0ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce0131ad0 0x7f4ce0141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce0131ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:49.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce011de10 con 0x7f4ce009ee60 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce011e0f0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce011e3d0 con 0x7f4ce0131ad0 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce7678640 10 monclient(hunting): authenticate will time out at 853.828604s 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce0130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5414640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce009ee60 msgr2=0x7f4ce009a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5414640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce009ee60 0x7f4ce009a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce0130cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49950/0 (socket says 172.21.15.181:49950) 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 -- 172.21.15.181:0/1629028152 learned_addr learned my addr 172.21.15.181:0/1629028152 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:49.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): get_auth_request con 0x7f4ce00a0ad0 auth_method 0 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce0131ad0 0x7f4ce0141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): get_auth_request con 0x7f4ce0131ad0 auth_method 0 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce5c15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 10 monclient(hunting): handle_auth_done global_id 44245 payload 824 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce0131ad0 msgr2=0x7f4ce0141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce0131ad0 0x7f4ce0141130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce009ee60 msgr2=0x7f4ce009a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.347+0000 7f4ce4c13640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce009ee60 0x7f4ce009a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.348+0000 7f4ce4c13640 10 monclient: _finish_hunting 0 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.348+0000 7f4ce4c13640 1 monclient: found mon.noname-b 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.348+0000 7f4ce4c13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.348+0000 7f4ce4c13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.348+0000 7f4ce4c13640 1 -- 172.21.15.181:0/1629028152 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc009450 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce4c13640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 msgr2=0x7f4ce0130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:49.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce4c13640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce0130cf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000ea20 tx=0x7f4cd00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0009600 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: dump: 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.366 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: _finish_auth 0 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd000cc20 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 5 monclient: authenticate success, global_id 44245 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 10 monclient: _renew_subs 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce011e6b0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0013470 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd57fa640 10 monclient: dump: 2024-09-15T03:30:49.367 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4cd77fe640 4 set_mon_vals no callback set 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 10 monclient: _renew_subs 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.350+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce009b6b0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4cd0013860 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4cd57fa640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce7678640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce7678640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce7678640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f4cd004be30 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce00008d0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce5414640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce5414640 10 monclient: get_auth_request con 0x7f4cc40386f0 auth_method 0 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce5414640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 msgr2=0x7f4cc403aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce5414640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 msgr2=0x7f4cc403aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce5414640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:49.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.351+0000 7f4ce5414640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:49.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.355+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4cd00236b0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.355+0000 7f4cd57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.355+0000 7f4cd57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:49.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.520+0000 7f4ce7678640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T03:30:49.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.520+0000 7f4ce7678640 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T03:30:49.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.520+0000 7f4ce7678640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.520+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f4ce0064a20 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.552+0000 7f4ce5414640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.552+0000 7f4ce5414640 10 monclient: get_auth_request con 0x7f4cc40386f0 auth_method 0 2024-09-15T03:30:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.554+0000 7f4ce5414640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00a00e0 tx=0x7f4cdc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.760+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v51) v1 ==== 151+0+0 (secure 0 0 0) 0x7f4cd002a0d0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.760+0000 7f4cd57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T03:30:49.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.760+0000 7f4cd57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:49.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4ce7678640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T03:30:49.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4ce7678640 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T03:30:49.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4ce7678640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f4ce011be90 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4cd57fa640 1 -- 172.21.15.181:0/1629028152 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v51) v1 ==== 151+0+0 (secure 0 0 0) 0x7f4cd00232f0 con 0x7f4ce00a0ad0 2024-09-15T03:30:49.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4cd57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-15T03:30:49.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.862+0000 7f4cd57fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 msgr2=0x7f4cc403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00a00e0 tx=0x7f4cdc032000 comp rx=0 tx=0).stop 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 10 monclient: shutdown 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 msgr2=0x7f4ce0130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce0130cf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000ea20 tx=0x7f4cd00090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 shutdown_connections 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc40386f0 0x7f4cc403aba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ce0131ad0 0x7f4ce0141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ce00a0ad0 0x7f4ce0130cf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 --2- 172.21.15.181:0/1629028152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce009ee60 0x7f4ce009a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 >> 172.21.15.181:0/1629028152 conn(0x7f4ce0071370 msgr2=0x7f4ce00a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 shutdown_connections 2024-09-15T03:30:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.864+0000 7f4ce7678640 1 -- 172.21.15.181:0/1629028152 wait complete. 2024-09-15T03:30:49.875 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-15T03:30:49.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.950+0000 7fec9c8c0640 10 monclient: get_monmap_and_config 2024-09-15T03:30:49.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.950+0000 7fec9c8c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:49.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.950+0000 7fec9c8c0640 10 monclient: monmap: 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:49.950934+0000 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:49.950934+0000 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.950+0000 7fec9c8c0640 1 Processor -- start 2024-09-15T03:30:49.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.950+0000 7fec9c8c0640 1 -- start start 2024-09-15T03:30:49.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.950+0000 7fec9c8c0640 10 monclient: init 2024-09-15T03:30:49.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:49.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:30:49.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec9809ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:49.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fec9809f020 0x7fec98069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fec9809f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:49.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fec98136d60 0x7fec98139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fec98136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec98067420 con 0x7fec9809ec50 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9809fdc0 con 0x7fec98136d60 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec980a0510 con 0x7fec9809f020 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9c8c0640 10 monclient(hunting): authenticate will time out at 854.432459s 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec9659c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fec9809ec50 legacy=0x7fec98131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec96d9d640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fec98136d60 msgr2=0x7fec98139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fec9809f020 0x7fec98069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec96d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fec98136d60 0x7fec98139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fec9809f020 0x7fec98069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49952/0 (socket says 172.21.15.181:49952) 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 -- 172.21.15.181:0/960130491 learned_addr learned my addr 172.21.15.181:0/960130491 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): get_auth_request con 0x7fec9809f020 auth_method 0 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient(hunting): handle_auth_done global_id 44248 payload 274 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 -- 172.21.15.181:0/960130491 >> v2:172.21.15.153:3300/0 conn(0x7fec98136d60 msgr2=0x7fec98139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:49.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 --2- 172.21.15.181:0/960130491 >> v2:172.21.15.153:3300/0 conn(0x7fec98136d60 0x7fec98139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 -- 172.21.15.181:0/960130491 >> v1:172.21.15.153:6789/0 conn(0x7fec9809ec50 legacy=0x7fec98131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:49.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:49.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 monclient: found mon.noname-c 2024-09-15T03:30:49.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:49.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.951+0000 7fec95d9b640 1 -- 172.21.15.181:0/960130491 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec980a0200 con 0x7fec9809f020 2024-09-15T03:30:49.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec95d9b640 1 -- 172.21.15.181:0/960130491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809f020 msgr2=0x7fec98069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec95d9b640 1 --2- 172.21.15.181:0/960130491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809f020 0x7fec98069d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec980a2b00 tx=0x7fec8c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8a807213c3aacba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 1 -- 172.21.15.181:0/960130491 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c032440 con 0x7fec9809f020 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: dump: 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: _finish_auth 0 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9c8c0640 5 monclient: authenticate success, global_id 44248 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 1 -- 172.21.15.181:0/960130491 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c0325a0 con 0x7fec9809f020 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9c8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9c8c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9c8c0640 4 set_mon_vals no callback set 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 1 -- 172.21.15.181:0/960130491 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c032940 con 0x7fec9809f020 2024-09-15T03:30:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.954+0000 7fec9559a640 10 monclient: dump: 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.965 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 10 monclient: shutdown 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 -- 172.21.15.181:0/960130491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809f020 msgr2=0x7fec98069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/960130491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809f020 0x7fec98069d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec980a2b00 tx=0x7fec8c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 -- 172.21.15.181:0/960130491 shutdown_connections 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/960130491 >> v2:172.21.15.153:3300/0 conn(0x7fec98136d60 0x7fec98139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/960130491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809f020 0x7fec98069d10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 -- 172.21.15.181:0/960130491 >> 172.21.15.181:0/960130491 conn(0x7fec980a5d40 msgr2=0x7fec980a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 -- 172.21.15.181:0/960130491 shutdown_connections 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 1 -- 172.21.15.181:0/960130491 wait complete. 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 10 monclient: build_initial_monmap 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.955+0000 7fec9c8c0640 10 monclient: monmap: 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 Processor -- start 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 -- start start 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient: init 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): picked mon.noname-b con 0x7fec9809ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec9809f020 0x7fec98098f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fec9809f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec98136d60 0x7fec98099440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fec98136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9809af50 con 0x7fec9809f020 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9809c000 con 0x7fec9809ec50 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec980a4450 con 0x7fec98136d60 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9c8c0640 10 monclient(hunting): authenticate will time out at 854.437790s 2024-09-15T03:30:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49962/0 (socket says 172.21.15.181:49962) 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 1 -- 172.21.15.181:0/695327887 learned_addr learned my addr 172.21.15.181:0/695327887 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): get_auth_request con 0x7fec9809ec50 auth_method 0 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec95d9b640 1 -- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec9809f020 msgr2=0x7fec98098f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec95d9b640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec9809f020 0x7fec98098f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.956+0000 7fec9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 10 monclient(hunting): handle_auth_done global_id 44251 payload 824 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 1 -- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec98136d60 msgr2=0x7fec98099440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:49.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec98136d60 0x7fec98099440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 1 -- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec9809f020 msgr2=0x7fec98098f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec9809f020 0x7fec98098f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 1 monclient: found mon.noname-b 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.957+0000 7fec9659c640 1 -- 172.21.15.181:0/695327887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c009450 con 0x7fec9809ec50 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9659c640 1 -- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 msgr2=0x7fec980989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9659c640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec8800bb80 tx=0x7fec880097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8800cc80 con 0x7fec9809ec50 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: dump: 2024-09-15T03:30:49.970 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8800cde0 con 0x7fec9809ec50 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 5 monclient: authenticate success, global_id 44251 2024-09-15T03:30:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88005340 con 0x7fec9809ec50 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 10 monclient: _renew_subs 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9809aa60 con 0x7fec9809ec50 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec867fc640 10 monclient: dump: 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec94d99640 4 set_mon_vals no callback set 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 10 monclient: _renew_subs 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.959+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9809b120 con 0x7fec9809ec50 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fec88012070 con 0x7fec9809ec50 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec9c8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec9c8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec9c8c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:49.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec867fc640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec780386f0 0x7fec7803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9811c280 con 0x7fec9809ec50 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fec8804bef0 con 0x7fec9809ec50 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec95d9b640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec780386f0 0x7fec7803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.960+0000 7fec95d9b640 10 monclient: get_auth_request con 0x7fec780386f0 auth_method 0 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.963+0000 7fec95d9b640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec780386f0 0x7fec7803aba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fec980719d0 tx=0x7fec8c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.964+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fec880143d0 con 0x7fec9809ec50 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.964+0000 7fec867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:49.964+0000 7fec867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:50.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:50.118+0000 7fec9c8c0640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T03:30:50.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:50.118+0000 7fec9c8c0640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T03:30:50.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:50.118+0000 7fec9c8c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:50.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:50.118+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7fec980008d0 con 0x7fec9809ec50 2024-09-15T03:30:51.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.770+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v53) v1 ==== 177+0+0 (secure 0 0 0) 0x7fec8801a030 con 0x7fec9809ec50 2024-09-15T03:30:51.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.770+0000 7fec867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T03:30:51.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.770+0000 7fec867fc640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-15T03:30:51.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.867+0000 7fec9c8c0640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T03:30:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.867+0000 7fec9c8c0640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T03:30:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.867+0000 7fec9c8c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.867+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7fec980a13e0 con 0x7fec9809ec50 2024-09-15T03:30:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:51.867+0000 7fec9659c640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec8800bb80 tx=0x7fec880097f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fec88049020 mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v53) v1 2024-09-15T03:30:52.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.695+0000 7fec867fc640 1 -- 172.21.15.181:0/695327887 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v53) v1 ==== 184+0+0 (secure 0 0 0) 0x7fec88049020 con 0x7fec9809ec50 2024-09-15T03:30:52.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.695+0000 7fec867fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-15T03:30:52.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.695+0000 7fec867fc640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-15T03:30:52.697 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ec-foo' already exists 2024-09-15T03:30:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec780386f0 msgr2=0x7fec7803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec780386f0 0x7fec7803aba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fec980719d0 tx=0x7fec8c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 10 monclient: shutdown 2024-09-15T03:30:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 msgr2=0x7fec980989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec8800bb80 tx=0x7fec880097f0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 shutdown_connections 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec780386f0 0x7fec7803aba0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec98136d60 0x7fec98099440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec9809f020 0x7fec98098f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 --2- 172.21.15.181:0/695327887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec9809ec50 0x7fec980989c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 >> 172.21.15.181:0/695327887 conn(0x7fec980a5d40 msgr2=0x7fec9809dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 shutdown_connections 2024-09-15T03:30:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.699+0000 7fec9c8c0640 1 -- 172.21.15.181:0/695327887 wait complete. 2024-09-15T03:30:52.713 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-15T03:30:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.792+0000 7fc483457640 10 monclient: get_monmap_and_config 2024-09-15T03:30:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.792+0000 7fc483457640 10 monclient: build_initial_monmap 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.792+0000 7fc483457640 10 monclient: monmap: 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:52.793648+0000 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:52.793648+0000 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.793+0000 7fc483457640 1 Processor -- start 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.793+0000 7fc483457640 1 -- start start 2024-09-15T03:30:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.793+0000 7fc483457640 10 monclient: init 2024-09-15T03:30:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:30:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc47c09ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc47c09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 0x7fc47c13a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient(hunting): picked mon.noname-b con 0x7fc47c069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c067420 con 0x7fc47c09ec50 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c09fdc0 con 0x7fc47c09f020 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c06a170 con 0x7fc47c069d10 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc483457640 10 monclient(hunting): authenticate will time out at 857.275289s 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4811f3640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fc47c09ec50 legacy=0x7fc47c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4809f2640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc47c09f020 0x7fc47c136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40848/0 (socket says 172.21.15.181:40848) 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4809f2640 1 -- 172.21.15.181:0/698340073 learned_addr learned my addr 172.21.15.181:0/698340073 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 1 --2- 172.21.15.181:0/698340073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 0x7fc47c13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): get_auth_request con 0x7fc47c069d10 auth_method 0 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:52.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient(hunting): handle_auth_done global_id 4865 payload 274 2024-09-15T03:30:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 1 -- 172.21.15.181:0/698340073 >> v1:172.21.15.181:6789/0 conn(0x7fc47c09f020 legacy=0x7fc47c136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 1 -- 172.21.15.181:0/698340073 >> v1:172.21.15.153:6789/0 conn(0x7fc47c09ec50 legacy=0x7fc47c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient: _finish_hunting 0 2024-09-15T03:30:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 1 monclient: found mon.noname-b 2024-09-15T03:30:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.794+0000 7fc4819f4640 1 -- 172.21.15.181:0/698340073 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c0a0510 con 0x7fc47c069d10 2024-09-15T03:30:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc4819f4640 1 -- 172.21.15.181:0/698340073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 msgr2=0x7fc47c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc4819f4640 1 --2- 172.21.15.181:0/698340073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 0x7fc47c13a490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc474009880 tx=0x7fc4740090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef23d24af4faf21d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 1 -- 172.21.15.181:0/698340073 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc474032440 con 0x7fc47c069d10 2024-09-15T03:30:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: dump: 2024-09-15T03:30:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 1 -- 172.21.15.181:0/698340073 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4740325a0 con 0x7fc47c069d10 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc483457640 5 monclient: authenticate success, global_id 4865 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc483457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc483457640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc483457640 4 set_mon_vals no callback set 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 1 -- 172.21.15.181:0/698340073 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc474032940 con 0x7fc47c069d10 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.797+0000 7fc473fff640 10 monclient: dump: 2024-09-15T03:30:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 10 monclient: shutdown 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 -- 172.21.15.181:0/698340073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 msgr2=0x7fc47c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 --2- 172.21.15.181:0/698340073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 0x7fc47c13a490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc474009880 tx=0x7fc4740090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 -- 172.21.15.181:0/698340073 shutdown_connections 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 --2- 172.21.15.181:0/698340073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c069d10 0x7fc47c13a490 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 -- 172.21.15.181:0/698340073 >> 172.21.15.181:0/698340073 conn(0x7fc47c0a5d40 msgr2=0x7fc47c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 -- 172.21.15.181:0/698340073 shutdown_connections 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.798+0000 7fc483457640 1 -- 172.21.15.181:0/698340073 wait complete. 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.799+0000 7fc483457640 10 monclient: build_initial_monmap 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.799+0000 7fc483457640 10 monclient: monmap: 2024-09-15T03:30:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.799+0000 7fc483457640 1 Processor -- start 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.799+0000 7fc483457640 1 -- start start 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.799+0000 7fc483457640 10 monclient: init 2024-09-15T03:30:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc47c069d10 0x7fc47c09b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): picked mon.noname-a con 0x7fc47c069d10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 0x7fc47c1300c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): picked mon.noname-b con 0x7fc47c09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c09f020 0x7fc47c130600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): picked mon.noname-c con 0x7fc47c09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c098830 con 0x7fc47c069d10 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c0993b0 con 0x7fc47c09ec50 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c096ac0 con 0x7fc47c09f020 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc483457640 10 monclient(hunting): authenticate will time out at 857.281297s 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4811f3640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc47c069d10 msgr2=0x7fc47c09b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 0x7fc47c1300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4811f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc47c069d10 0x7fc47c09b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 0x7fc47c1300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49968/0 (socket says 172.21.15.181:49968) 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 -- 172.21.15.181:0/177900590 learned_addr learned my addr 172.21.15.181:0/177900590 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): get_auth_request con 0x7fc47c09ec50 auth_method 0 2024-09-15T03:30:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4819f4640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c09f020 0x7fc47c130600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4819f4640 10 monclient(hunting): get_auth_request con 0x7fc47c09f020 auth_method 0 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4819f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4819f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4819f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient(hunting): handle_auth_done global_id 44254 payload 824 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 -- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c09f020 msgr2=0x7fc47c130600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c09f020 0x7fc47c130600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 -- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc47c069d10 msgr2=0x7fc47c09b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc47c069d10 0x7fc47c09b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient: _finish_hunting 0 2024-09-15T03:30:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 monclient: found mon.noname-b 2024-09-15T03:30:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.800+0000 7fc4809f2640 1 -- 172.21.15.181:0/177900590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474009390 con 0x7fc47c09ec50 2024-09-15T03:30:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.802+0000 7fc4809f2640 1 -- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 msgr2=0x7fc47c1300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.802+0000 7fc4809f2640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 0x7fc47c1300c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00c5e0 tx=0x7fc46c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c005cb0 con 0x7fc47c09ec50 2024-09-15T03:30:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:52.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: dump: 2024-09-15T03:30:52.824 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:52.824 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:52.824 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.824 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.824 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:52.825 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: _finish_auth 0 2024-09-15T03:30:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 5 monclient: authenticate success, global_id 44254 2024-09-15T03:30:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c005e30 con 0x7fc47c09ec50 2024-09-15T03:30:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 10 monclient: _renew_subs 2024-09-15T03:30:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47c098550 con 0x7fc47c09ec50 2024-09-15T03:30:52.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:52.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c007340 con 0x7fc47c09ec50 2024-09-15T03:30:52.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:52.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4717fa640 10 monclient: dump: 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:52.829 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:52.829 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:52.829 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:52.829 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:52.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc4737fe640 4 set_mon_vals no callback set 2024-09-15T03:30:52.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 10 monclient: _renew_subs 2024-09-15T03:30:52.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:52.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.803+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c0975f0 con 0x7fc47c09ec50 2024-09-15T03:30:52.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc46c01b400 con 0x7fc47c09ec50 2024-09-15T03:30:52.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc483457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:52.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc483457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:52.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc483457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:52.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c0008d0 con 0x7fc47c09ec50 2024-09-15T03:30:52.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc4717fa640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4600386f0 0x7fc46003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:52.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5496+0+0 (secure 0 0 0) 0x7fc46c04b9f0 con 0x7fc47c09ec50 2024-09-15T03:30:52.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc4811f3640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4600386f0 0x7fc46003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:52.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.804+0000 7fc4811f3640 10 monclient: get_auth_request con 0x7fc4600386f0 auth_method 0 2024-09-15T03:30:52.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.806+0000 7fc4811f3640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4600386f0 0x7fc46003aba0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc478003c70 tx=0x7fc478009310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:52.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.809+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc46c01ab40 con 0x7fc47c09ec50 2024-09-15T03:30:52.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.809+0000 7fc4717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:52.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.809+0000 7fc4717fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:52.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.962+0000 7fc483457640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:52.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.962+0000 7fc483457640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:52.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.962+0000 7fc483457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:52.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:52.962+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc47c0a13e0 con 0x7fc47c09ec50 2024-09-15T03:30:53.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.790+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v55) v1 ==== 154+0+0 (secure 0 0 0) 0x7fc46c01a360 con 0x7fc47c09ec50 2024-09-15T03:30:53.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.790+0000 7fc4717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:53.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.790+0000 7fc4717fa640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.887+0000 7fc483457640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.888+0000 7fc483457640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.888+0000 7fc483457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.888+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc47c072640 con 0x7fc47c09ec50 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.888+0000 7fc4717fa640 1 -- 172.21.15.181:0/177900590 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc46c01a540 con 0x7fc47c09ec50 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.888+0000 7fc4717fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-15T03:30:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.888+0000 7fc4717fa640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-15T03:30:53.890 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ec-foo' does not exist 2024-09-15T03:30:53.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4600386f0 msgr2=0x7fc46003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4600386f0 0x7fc46003aba0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc478003c70 tx=0x7fc478009310 comp rx=0 tx=0).stop 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 10 monclient: shutdown 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 msgr2=0x7fc47c1300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 0x7fc47c1300c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00c5e0 tx=0x7fc46c004960 comp rx=0 tx=0).stop 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 shutdown_connections 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4600386f0 0x7fc46003aba0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc47c09f020 0x7fc47c130600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc47c09ec50 0x7fc47c1300c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 --2- 172.21.15.181:0/177900590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc47c069d10 0x7fc47c09b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 >> 172.21.15.181:0/177900590 conn(0x7fc47c0a5d40 msgr2=0x7fc47c068c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 shutdown_connections 2024-09-15T03:30:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.890+0000 7fc483457640 1 -- 172.21.15.181:0/177900590 wait complete. 2024-09-15T03:30:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls 2024-09-15T03:30:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.973+0000 7f3e11c34640 10 monclient: get_monmap_and_config 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.973+0000 7f3e11c34640 10 monclient: build_initial_monmap 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.973+0000 7f3e11c34640 10 monclient: monmap: 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:53.973971+0000 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:53.973971+0000 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:53.975 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:53.975 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:53.975 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:53.975 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:53.975 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:53.975 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:53.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.973+0000 7f3e11c34640 1 Processor -- start 2024-09-15T03:30:53.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.973+0000 7f3e11c34640 1 -- start start 2024-09-15T03:30:53.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.973+0000 7f3e11c34640 10 monclient: init 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3e0c09ec50 0x7f3e0c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): picked mon.noname-a con 0x7f3e0c09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3e0c1321a0 0x7f3e0c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): picked mon.noname-c con 0x7f3e0c1321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e0c09f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0c067420 con 0x7f3e0c09f020 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c0a0480 con 0x7f3e0c09ec50 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c0a0200 con 0x7f3e0c1321a0 2024-09-15T03:30:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e11c34640 10 monclient(hunting): authenticate will time out at 858.455430s 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0bfff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f3e0c09f020 legacy=0x7f3e0c069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0b7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f3e0c09ec50 msgr2=0x7f3e0c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0b7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3e0c09ec50 0x7f3e0c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3e0c1321a0 0x7f3e0c068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3e0c1321a0 0x7f3e0c068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49978/0 (socket says 172.21.15.181:49978) 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 -- 172.21.15.181:0/92698869 learned_addr learned my addr 172.21.15.181:0/92698869 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): get_auth_request con 0x7f3e0c1321a0 auth_method 0 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:53.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient(hunting): handle_auth_done global_id 44257 payload 274 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 -- 172.21.15.181:0/92698869 >> v2:172.21.15.153:3300/0 conn(0x7f3e0c09ec50 msgr2=0x7f3e0c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 --2- 172.21.15.181:0/92698869 >> v2:172.21.15.153:3300/0 conn(0x7f3e0c09ec50 0x7f3e0c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 -- 172.21.15.181:0/92698869 >> v1:172.21.15.153:6789/0 conn(0x7f3e0c09f020 legacy=0x7f3e0c069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 monclient: found mon.noname-c 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.974+0000 7f3e0affd640 1 -- 172.21.15.181:0/92698869 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c09fd80 con 0x7f3e0c1321a0 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0affd640 1 -- 172.21.15.181:0/92698869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c1321a0 msgr2=0x7f3e0c068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0affd640 1 --2- 172.21.15.181:0/92698869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c1321a0 0x7f3e0c068d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c1325a0 tx=0x7f3e000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62d2ccdb6fc4f5ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 1 -- 172.21.15.181:0/92698869 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00032440 con 0x7f3e0c1321a0 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: dump: 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 1 -- 172.21.15.181:0/92698869 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e000325a0 con 0x7f3e0c1321a0 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e11c34640 5 monclient: authenticate success, global_id 44257 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e11c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e11c34640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:53.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 1 -- 172.21.15.181:0/92698869 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00032940 con 0x7f3e0c1321a0 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e11c34640 4 set_mon_vals no callback set 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.977+0000 7f3e0a7fc640 10 monclient: dump: 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:53.988 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 10 monclient: shutdown 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 -- 172.21.15.181:0/92698869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c1321a0 msgr2=0x7f3e0c068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 --2- 172.21.15.181:0/92698869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c1321a0 0x7f3e0c068d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c1325a0 tx=0x7f3e000091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 -- 172.21.15.181:0/92698869 shutdown_connections 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 --2- 172.21.15.181:0/92698869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c1321a0 0x7f3e0c068d00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 --2- 172.21.15.181:0/92698869 >> v2:172.21.15.153:3300/0 conn(0x7f3e0c09ec50 0x7f3e0c131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 -- 172.21.15.181:0/92698869 >> 172.21.15.181:0/92698869 conn(0x7f3e0c0a5cf0 msgr2=0x7f3e0c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 -- 172.21.15.181:0/92698869 shutdown_connections 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 1 -- 172.21.15.181:0/92698869 wait complete. 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 10 monclient: build_initial_monmap 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.978+0000 7f3e11c34640 10 monclient: monmap: 2024-09-15T03:30:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 Processor -- start 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 -- start start 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient: init 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:53.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e0c09ec50 0x7f3e0c11de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): picked mon.noname-a con 0x7f3e0c09ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 0x7f3e0c11e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): picked mon.noname-b con 0x7f3e0c09f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0c1321a0 0x7f3e0c11e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): picked mon.noname-c con 0x7f3e0c1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c1205d0 con 0x7f3e0c09ec50 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c09c020 con 0x7f3e0c09f020 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c121a80 con 0x7f3e0c1321a0 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:53.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e11c34640 10 monclient(hunting): authenticate will time out at 858.460770s 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 0x7f3e0c11e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 0x7f3e0c11e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49992/0 (socket says 172.21.15.181:49992) 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0b7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e0c09ec50 msgr2=0x7f3e0c11de20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 1 -- 172.21.15.181:0/524476163 learned_addr learned my addr 172.21.15.181:0/524476163 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): get_auth_request con 0x7f3e0c09f020 auth_method 0 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0b7fe640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e0c09ec50 0x7f3e0c11de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.979+0000 7f3e0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:53.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0bfff640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0c1321a0 0x7f3e0c11e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0bfff640 10 monclient(hunting): get_auth_request con 0x7f3e0c1321a0 auth_method 0 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 10 monclient(hunting): handle_auth_done global_id 44260 payload 824 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 1 -- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0c1321a0 msgr2=0x7f3e0c11e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0c1321a0 0x7f3e0c11e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 1 -- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e0c09ec50 msgr2=0x7f3e0c11de20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e0c09ec50 0x7f3e0c11de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 1 monclient: found mon.noname-b 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:53.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.980+0000 7f3e0affd640 1 -- 172.21.15.181:0/524476163 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00009450 con 0x7f3e0c09f020 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e0affd640 1 -- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 msgr2=0x7f3e0c11e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e0affd640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 0x7f3e0c11e360 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009210 tx=0x7f3e00032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0003c960 con 0x7f3e0c09f020 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: dump: 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:53.994 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 5 monclient: authenticate success, global_id 44260 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 10 monclient: _renew_subs 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0003cac0 con 0x7f3e0c09f020 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c1217a0 con 0x7f3e0c09f020 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0003ce40 con 0x7f3e0c09f020 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:53.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3df37fe640 10 monclient: dump: 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 10 monclient: _renew_subs 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:53.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c131550 con 0x7f3e0c09f020 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.982+0000 7f3e09ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3e00039040 con 0x7f3e0c09f020 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3e11c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3e11c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3e11c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c1215d0 con 0x7f3e0c09f020 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3df37fe640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3df80386f0 0x7f3df803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.983+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f3e0006bdb0 con 0x7f3e0c09f020 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.984+0000 7f3e0b7fe640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3df80386f0 0x7f3df803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:53.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.984+0000 7f3e0b7fe640 10 monclient: get_auth_request con 0x7f3df80386f0 auth_method 0 2024-09-15T03:30:53.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.986+0000 7f3e0b7fe640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3df80386f0 0x7f3df803aba0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0038f0 tx=0x7f3dfc006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:53.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.988+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3e00042310 con 0x7f3e0c09f020 2024-09-15T03:30:53.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.988+0000 7f3df37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:53.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:53.988+0000 7f3df37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:54.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.143+0000 7f3e11c34640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.143+0000 7f3e11c34640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.143+0000 7f3e11c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.143+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f3e0c0008d0 con 0x7f3e0c09f020 2024-09-15T03:30:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.144+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v55) v1 ==== 65+0+60 (secure 0 0 0) 0x7f3e00040020 con 0x7f3e0c09f020 2024-09-15T03:30:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.144+0000 7f3df37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.144+0000 7f3df37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3e11c34640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3e11c34640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3e11c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f3e0c0008d0 con 0x7f3e0c09f020 2024-09-15T03:30:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3df37fe640 1 -- 172.21.15.181:0/524476163 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v55) v1 ==== 65+0+60 (secure 0 0 0) 0x7f3e00037070 con 0x7f3e0c09f020 2024-09-15T03:30:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3df37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.237+0000 7f3df37fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.239+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3df80386f0 msgr2=0x7f3df803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.239+0000 7f3e11c34640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3df80386f0 0x7f3df803aba0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0038f0 tx=0x7f3dfc006d80 comp rx=0 tx=0).stop 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 10 monclient: shutdown 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 msgr2=0x7f3e0c11e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 0x7f3e0c11e360 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009210 tx=0x7f3e00032000 comp rx=0 tx=0).stop 2024-09-15T03:30:54.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 shutdown_connections 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3df80386f0 0x7f3df803aba0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0c1321a0 0x7f3e0c11e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e0c09f020 0x7f3e0c11e360 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 --2- 172.21.15.181:0/524476163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e0c09ec50 0x7f3e0c11de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 >> 172.21.15.181:0/524476163 conn(0x7f3e0c0a5cf0 msgr2=0x7f3e0c09e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 shutdown_connections 2024-09-15T03:30:54.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.240+0000 7f3e11c34640 1 -- 172.21.15.181:0/524476163 wait complete. 2024-09-15T03:30:54.249 INFO:tasks.workunit.client.0.smithi181.stdout:foo 2024-09-15T03:30:54.249 INFO:tasks.workunit.client.0.smithi181.stdout:foo-ssd 2024-09-15T03:30:54.250 INFO:tasks.workunit.client.0.smithi181.stdout:foo-hdd 2024-09-15T03:30:54.250 INFO:tasks.workunit.client.0.smithi181.stdout:ec-foo 2024-09-15T03:30:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-15T03:30:54.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.324+0000 7f115e75d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:54.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.324+0000 7f115e75d640 10 monclient: build_initial_monmap 2024-09-15T03:30:54.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.324+0000 7f115e75d640 10 monclient: monmap: 2024-09-15T03:30:54.325 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:54.325 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:54.325304+0000 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:54.325304+0000 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:54.326 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 1 Processor -- start 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 1 -- start start 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient: init 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 0x7f115809c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient(hunting): picked mon.noname-b con 0x7f115809cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f115809cea0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f115809d270 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115809b710 con 0x7f115809d270 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115809c6e0 con 0x7f115809cea0 2024-09-15T03:30:54.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115809e8f0 con 0x7f115809cad0 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115e75d640 10 monclient(hunting): authenticate will time out at 858.806875s 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f115ccfa640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f115809d270 legacy=0x7f115809e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f11577fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f115809cea0 0x7f115809dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40858/0 (socket says 172.21.15.181:40858) 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.325+0000 7f11577fe640 1 -- 172.21.15.181:0/2895474500 learned_addr learned my addr 172.21.15.181:0/2895474500 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 1 --2- 172.21.15.181:0/2895474500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 0x7f115809c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): get_auth_request con 0x7f115809cad0 auth_method 0 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:54.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient(hunting): handle_auth_done global_id 4874 payload 274 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 1 -- 172.21.15.181:0/2895474500 >> v1:172.21.15.181:6789/0 conn(0x7f115809cea0 legacy=0x7f115809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 1 -- 172.21.15.181:0/2895474500 >> v1:172.21.15.153:6789/0 conn(0x7f115809d270 legacy=0x7f115809e1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 1 monclient: found mon.noname-b 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:54.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.326+0000 7f1157fff640 1 -- 172.21.15.181:0/2895474500 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115809a650 con 0x7f115809cad0 2024-09-15T03:30:54.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1157fff640 1 -- 172.21.15.181:0/2895474500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 msgr2=0x7f115809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1157fff640 1 --2- 172.21.15.181:0/2895474500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 0x7f115809c1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f115809c870 tx=0x7f11480090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e1fac7e2a9390c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 1 -- 172.21.15.181:0/2895474500 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148032440 con 0x7f115809cad0 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: dump: 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.339 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 1 -- 172.21.15.181:0/2895474500 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11480325a0 con 0x7f115809cad0 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 1 -- 172.21.15.181:0/2895474500 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148032940 con 0x7f115809cad0 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f1156ffd640 10 monclient: dump: 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f115e75d640 5 monclient: authenticate success, global_id 4874 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f115e75d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.329+0000 7f115e75d640 4 set_mon_vals no callback set 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 10 monclient: shutdown 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:54.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 -- 172.21.15.181:0/2895474500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 msgr2=0x7f115809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 --2- 172.21.15.181:0/2895474500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 0x7f115809c1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f115809c870 tx=0x7f11480090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 -- 172.21.15.181:0/2895474500 shutdown_connections 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 --2- 172.21.15.181:0/2895474500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cad0 0x7f115809c1a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 -- 172.21.15.181:0/2895474500 >> 172.21.15.181:0/2895474500 conn(0x7f1158070e90 msgr2=0x7f11580a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 -- 172.21.15.181:0/2895474500 shutdown_connections 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 1 -- 172.21.15.181:0/2895474500 wait complete. 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 10 monclient: build_initial_monmap 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.330+0000 7f115e75d640 10 monclient: monmap: 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:54.342 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 Processor -- start 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 -- start start 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient: init 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f1158141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): picked mon.noname-a con 0x7f115809cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1158134f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): picked mon.noname-c con 0x7f115809cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): picked mon.noname-b con 0x7f115809d270 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158122600 con 0x7f115809cad0 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11581205f0 con 0x7f115809d270 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1158122040 con 0x7f115809cea0 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115e75d640 10 monclient(hunting): authenticate will time out at 858.812768s 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f1157fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 msgr2=0x7f1158141180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f1157fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f1158141180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49996/0 (socket says 172.21.15.181:49996) 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 learned_addr learned my addr 172.21.15.181:0/201520583 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): get_auth_request con 0x7f115809d270 auth_method 0 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.331+0000 7f115ccfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f11577fe640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1158134f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f11577fe640 10 monclient(hunting): get_auth_request con 0x7f115809cea0 auth_method 0 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f11577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:54.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f11577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f11577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 10 monclient(hunting): handle_auth_done global_id 44263 payload 824 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 msgr2=0x7f1158134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1158134f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 msgr2=0x7f1158141180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f1158141180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 10 monclient: _finish_hunting 0 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 1 monclient: found mon.noname-b 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148009390 con 0x7f115809d270 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.332+0000 7f11577fe640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1158134f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:54.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 msgr2=0x7f115811ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f115000bb10 tx=0x7f11500097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115000cbc0 con 0x7f115809d270 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: dump: 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.347 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: _finish_auth 0 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115000cd40 con 0x7f115809d270 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 5 monclient: authenticate success, global_id 44263 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 10 monclient: _renew_subs 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1158122320 con 0x7f115809d270 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150006150 con 0x7f115809d270 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:54.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: dump: 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 10 monclient: _renew_subs 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1158141860 con 0x7f115809d270 2024-09-15T03:30:54.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f11567fc640 4 set_mon_vals no callback set 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 msgr2=0x7f115811ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 msgr2=0x7f115811ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f115000bb10 tx=0x7f11500097d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f115000bb10 tx=0x7f11500097d0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 msgr2=0x7f115811ea30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115809d270 0x7f115811ea30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1144005750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): picked mon.c con 0x7f115809cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f11440080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): picked mon.a con 0x7f115809cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 0x7f114400af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): picked mon.b con 0x7f1144008af0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114400ba20 con 0x7f115809cad0 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114400bb60 con 0x7f1144008af0 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114400bca0 con 0x7f115809cea0 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.334+0000 7f113ffff640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 0x7f114400af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f11577fe640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 msgr2=0x7f11440080e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): get_auth_request con 0x7f1144008af0 auth_method 0 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:54.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f11577fe640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f11440080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f1157fff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1144005750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f1157fff640 10 monclient(hunting): get_auth_request con 0x7f115809cea0 auth_method 0 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f1157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f1157fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f1157fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient(hunting): handle_auth_done global_id 44263 payload 293 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 msgr2=0x7f1144005750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1144005750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 msgr2=0x7f11440080e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f11440080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient: _finish_hunting 0 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 monclient: found mon.b 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f114400bf90 con 0x7f1144008af0 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 10 monclient: _finish_auth 0 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115ccfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f1157fff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1144005750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115e75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115e75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115e75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.335+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11580a2ac0 con 0x7f1144008af0 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f115ccfa640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 msgr2=0x7f114400af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f115ccfa640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 0x7f114400af70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f115000bae0 tx=0x7f1150015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150006930 con 0x7f1144008af0 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f113ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f113ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f113ffff640 10 monclient: dump: 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150006a90 con 0x7f1144008af0 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f113ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:54.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.337+0000 7f1155ffb640 4 set_mon_vals no callback set 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.338+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1150012020 con 0x7f1144008af0 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.338+0000 7f113ffff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1144040fe0 0x7f1144043490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.338+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 4 ==== osd_map(55..55 src has 1..55) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f115004cee0 con 0x7f1144008af0 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.338+0000 7f1157fff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1144040fe0 0x7f1144043490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.338+0000 7f1157fff640 10 monclient: get_auth_request con 0x7f1144040fe0 auth_method 0 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.341+0000 7f1157fff640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1144040fe0 0x7f1144043490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f115806edb0 tx=0x7f1148032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.342+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1150029920 con 0x7f1144008af0 2024-09-15T03:30:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.342+0000 7f113ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.342+0000 7f113ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:54.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.499+0000 7f115e75d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:54.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.499+0000 7f115e75d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.499+0000 7f115e75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.499+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1158122b20 con 0x7f1144008af0 2024-09-15T03:30:54.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.794+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v56) v1 ==== 110+0+0 (secure 0 0 0) 0x7f1150014340 con 0x7f1144008af0 2024-09-15T03:30:54.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.794+0000 7f113ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:54.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.794+0000 7f113ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.890+0000 7f115e75d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.890+0000 7f115e75d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.890+0000 7f115e75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.890+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f11580008d0 con 0x7f1144008af0 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.891+0000 7f113ffff640 1 -- 172.21.15.181:0/201520583 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v56) v1 ==== 139+0+0 (secure 0 0 0) 0x7f1150029320 con 0x7f1144008af0 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.891+0000 7f113ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.891+0000 7f113ffff640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-15T03:30:54.892 INFO:tasks.workunit.client.0.smithi181.stderr:already renamed to 'foo-asdf' 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1144040fe0 msgr2=0x7f1144043490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1144040fe0 0x7f1144043490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f115806edb0 tx=0x7f1148032000 comp rx=0 tx=0).stop 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 10 monclient: shutdown 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 msgr2=0x7f114400af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 0x7f114400af70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f115000bae0 tx=0x7f1150015000 comp rx=0 tx=0).stop 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 shutdown_connections 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1144040fe0 0x7f1144043490 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1144008af0 0x7f114400af70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115809cad0 0x7f11440080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 --2- 172.21.15.181:0/201520583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115809cea0 0x7f1144005750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.892+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 >> 172.21.15.181:0/201520583 conn(0x7f1158070e90 msgr2=0x7f1158069660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:54.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.893+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 shutdown_connections 2024-09-15T03:30:54.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.893+0000 7f115e75d640 1 -- 172.21.15.181:0/201520583 wait complete. 2024-09-15T03:30:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-15T03:30:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.976+0000 7f119bf1d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.976+0000 7f119bf1d640 10 monclient: build_initial_monmap 2024-09-15T03:30:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.976+0000 7f119bf1d640 10 monclient: monmap: 2024-09-15T03:30:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:54.976910+0000 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:54.976910+0000 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.976+0000 7f119bf1d640 1 Processor -- start 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.976+0000 7f119bf1d640 1 -- start start 2024-09-15T03:30:54.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.976+0000 7f119bf1d640 10 monclient: init 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f119409ec50 0x7f1194131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f119409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11941321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 0x7f1194069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f119409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1194067420 con 0x7f11941321a0 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f119409fdc0 con 0x7f119409ec50 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11940a0510 con 0x7f119409f020 2024-09-15T03:30:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119bf1d640 10 monclient(hunting): authenticate will time out at 859.458403s 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f1199cb9640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f119409ec50 msgr2=0x7f1194131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f1199cb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f119409ec50 0x7f1194131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f11994b8640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f11941321a0 0x7f1194068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40868/0 (socket says 172.21.15.181:40868) 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f11994b8640 1 -- 172.21.15.181:0/3990374830 learned_addr learned my addr 172.21.15.181:0/3990374830 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 1 --2- 172.21.15.181:0/3990374830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 0x7f1194069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): get_auth_request con 0x7f119409f020 auth_method 0 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient(hunting): handle_auth_done global_id 4880 payload 274 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 1 -- 172.21.15.181:0/3990374830 >> v2:172.21.15.153:3300/0 conn(0x7f119409ec50 msgr2=0x7f1194131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 1 --2- 172.21.15.181:0/3990374830 >> v2:172.21.15.153:3300/0 conn(0x7f119409ec50 0x7f1194131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 1 -- 172.21.15.181:0/3990374830 >> v1:172.21.15.181:6789/0 conn(0x7f11941321a0 legacy=0x7f1194068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient: _finish_hunting 0 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 1 monclient: found mon.noname-b 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.977+0000 7f119a4ba640 1 -- 172.21.15.181:0/3990374830 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940a0200 con 0x7f119409f020 2024-09-15T03:30:54.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.980+0000 7f119a4ba640 1 -- 172.21.15.181:0/3990374830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 msgr2=0x7f1194069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.980+0000 7f119a4ba640 1 --2- 172.21.15.181:0/3990374830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 0x7f1194069410 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1190009e60 tx=0x7f1190009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9719ff747b3edbb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 1 -- 172.21.15.181:0/3990374830 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190007840 con 0x7f119409f020 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: dump: 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.992 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: _finish_auth 0 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 1 -- 172.21.15.181:0/3990374830 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119002d050 con 0x7f119409f020 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 5 monclient: authenticate success, global_id 4880 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:54.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 4 set_mon_vals no callback set 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 1 -- 172.21.15.181:0/3990374830 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190007bd0 con 0x7f119409f020 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f1198cb7640 10 monclient: dump: 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:54.995 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 10 monclient: shutdown 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 -- 172.21.15.181:0/3990374830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 msgr2=0x7f1194069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 --2- 172.21.15.181:0/3990374830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 0x7f1194069410 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1190009e60 tx=0x7f1190009780 comp rx=0 tx=0).stop 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 -- 172.21.15.181:0/3990374830 shutdown_connections 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 --2- 172.21.15.181:0/3990374830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119409f020 0x7f1194069410 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 --2- 172.21.15.181:0/3990374830 >> v2:172.21.15.153:3300/0 conn(0x7f119409ec50 0x7f1194131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 -- 172.21.15.181:0/3990374830 >> 172.21.15.181:0/3990374830 conn(0x7f11940a5d40 msgr2=0x7f11940a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 -- 172.21.15.181:0/3990374830 shutdown_connections 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.981+0000 7f119bf1d640 1 -- 172.21.15.181:0/3990374830 wait complete. 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient: build_initial_monmap 2024-09-15T03:30:54.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient: monmap: 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 Processor -- start 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 -- start start 2024-09-15T03:30:54.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient: init 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 0x7f119409a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f119409ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119409f020 0x7f1194130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f119409f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11941321a0 0x7f119411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f11941321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194141d20 con 0x7f119409f020 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.982+0000 7f119bf1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119411dc00 con 0x7f119409ec50 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119bf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119bf1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119409c000 con 0x7f11941321a0 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119bf1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119bf1d640 10 monclient(hunting): authenticate will time out at 859.464013s 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 0x7f119409a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f11994b8640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119409f020 msgr2=0x7f1194130ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f11994b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119409f020 0x7f1194130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 0x7f119409a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50012/0 (socket says 172.21.15.181:50012) 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 -- 172.21.15.181:0/1743464002 learned_addr learned my addr 172.21.15.181:0/1743464002 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): get_auth_request con 0x7f119409ec50 auth_method 0 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119a4ba640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11941321a0 0x7f119411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119a4ba640 10 monclient(hunting): get_auth_request con 0x7f11941321a0 auth_method 0 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119a4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f119a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient(hunting): handle_auth_done global_id 44266 payload 824 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 -- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11941321a0 msgr2=0x7f119411b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11941321a0 0x7f119411b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 -- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119409f020 msgr2=0x7f1194130ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119409f020 0x7f1194130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 monclient: found mon.noname-b 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.983+0000 7f1199cb9640 1 -- 172.21.15.181:0/1743464002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1190009a20 con 0x7f119409ec50 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1199cb9640 1 -- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 msgr2=0x7f119409a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1199cb9640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 0x7f119409a9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f118c00b630 tx=0x7f118c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c015070 con 0x7f119409ec50 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: dump: 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.002 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: _finish_auth 0 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f119bf1d640 5 monclient: authenticate success, global_id 44266 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c003d90 con 0x7f119409ec50 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f119bf1d640 10 monclient: _renew_subs 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c006380 con 0x7f119409ec50 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f119bf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119409e380 con 0x7f119409ec50 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f1189ffb640 10 monclient: dump: 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.985+0000 7f118bfff640 4 set_mon_vals no callback set 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 10 monclient: _renew_subs 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119411cd10 con 0x7f119409ec50 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f118c010440 con 0x7f119409ec50 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f1189ffb640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f11780386f0 0x7f117803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f118c04b640 con 0x7f119409ec50 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.987+0000 7f11994b8640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f11780386f0 0x7f117803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.986+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11940008d0 con 0x7f119409ec50 2024-09-15T03:30:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.987+0000 7f11994b8640 10 monclient: get_auth_request con 0x7f11780386f0 auth_method 0 2024-09-15T03:30:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.989+0000 7f11994b8640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f11780386f0 0x7f117803aba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f119409b8b0 tx=0x7f1184007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.991+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f118c018a70 con 0x7f119409ec50 2024-09-15T03:30:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.991+0000 7f1189ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:54.991+0000 7f1189ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:55.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.149+0000 7f119bf1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.149+0000 7f119bf1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.149+0000 7f119bf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.149+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f11940a13e0 con 0x7f119409ec50 2024-09-15T03:30:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.150+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v56) v1 ==== 139+0+0 (secure 0 0 0) 0x7f118c018a70 con 0x7f119409ec50 2024-09-15T03:30:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.150+0000 7f1189ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.150+0000 7f1189ffb640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-15T03:30:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f119bf1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f119bf1d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f119bf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1194064900 con 0x7f119409ec50 2024-09-15T03:30:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f1189ffb640 1 -- 172.21.15.181:0/1743464002 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v56) v1 ==== 139+0+0 (secure 0 0 0) 0x7f118c018a70 con 0x7f119409ec50 2024-09-15T03:30:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f1189ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-15T03:30:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.245+0000 7f1189ffb640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-15T03:30:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:already renamed to 'foo-asdf' 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f11780386f0 msgr2=0x7f117803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f11780386f0 0x7f117803aba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f119409b8b0 tx=0x7f1184007b80 comp rx=0 tx=0).stop 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 10 monclient: shutdown 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 msgr2=0x7f119409a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 0x7f119409a9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f118c00b630 tx=0x7f118c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 shutdown_connections 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f11780386f0 0x7f117803aba0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11941321a0 0x7f119411b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119409f020 0x7f1194130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 --2- 172.21.15.181:0/1743464002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f119409ec50 0x7f119409a9f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 >> 172.21.15.181:0/1743464002 conn(0x7f11940a5d40 msgr2=0x7f1194134a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 shutdown_connections 2024-09-15T03:30:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.247+0000 7f119bf1d640 1 -- 172.21.15.181:0/1743464002 wait complete. 2024-09-15T03:30:55.257 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-15T03:30:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.334+0000 7f3f50e68640 10 monclient: get_monmap_and_config 2024-09-15T03:30:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.334+0000 7f3f50e68640 10 monclient: build_initial_monmap 2024-09-15T03:30:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.334+0000 7f3f50e68640 10 monclient: monmap: 2024-09-15T03:30:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:55.334858+0000 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:55.334858+0000 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:55.336 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.334+0000 7f3f50e68640 1 Processor -- start 2024-09-15T03:30:55.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.334+0000 7f3f50e68640 1 -- start start 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.334+0000 7f3f50e68640 10 monclient: init 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 0x7f3f4c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4c09ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f4c1321a0 0x7f3f4c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4c1321a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:55.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f4c09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c067420 con 0x7f3f4c09f020 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c09fdc0 con 0x7f3f4c1321a0 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0a0510 con 0x7f3f4c09ec50 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f50e68640 10 monclient(hunting): authenticate will time out at 859.816265s 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f41d9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f3f4c1321a0 msgr2=0x7f3f4c068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4ad9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3f4c09f020 0x7f3f4c069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40882/0 (socket says 172.21.15.181:40882) 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4ad9d640 1 -- 172.21.15.181:0/2530507928 learned_addr learned my addr 172.21.15.181:0/2530507928 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f41d9b640 1 --2- 172.21.15.181:0/2530507928 >> v2:172.21.15.153:3300/0 conn(0x7f3f4c1321a0 0x7f3f4c068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:55.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 1 --2- 172.21.15.181:0/2530507928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 0x7f3f4c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): get_auth_request con 0x7f3f4c09ec50 auth_method 0 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient(hunting): handle_auth_done global_id 4886 payload 274 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 1 -- 172.21.15.181:0/2530507928 >> v2:172.21.15.153:3300/0 conn(0x7f3f4c1321a0 msgr2=0x7f3f4c068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:55.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 1 --2- 172.21.15.181:0/2530507928 >> v2:172.21.15.153:3300/0 conn(0x7f3f4c1321a0 0x7f3f4c068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 1 -- 172.21.15.181:0/2530507928 >> v1:172.21.15.181:6789/0 conn(0x7f3f4c09f020 legacy=0x7f3f4c069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:55.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:30:55.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 1 monclient: found mon.noname-b 2024-09-15T03:30:55.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:55.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:55.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.335+0000 7f3f4a59c640 1 -- 172.21.15.181:0/2530507928 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0a0200 con 0x7f3f4c09ec50 2024-09-15T03:30:55.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.338+0000 7f3f4a59c640 1 -- 172.21.15.181:0/2530507928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 msgr2=0x7f3f4c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:55.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.338+0000 7f3f4a59c640 1 --2- 172.21.15.181:0/2530507928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 0x7f3f4c131c60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009850 tx=0x7f3f3c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd8a644e9b583b73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:55.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 1 -- 172.21.15.181:0/2530507928 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c032440 con 0x7f3f4c09ec50 2024-09-15T03:30:55.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: dump: 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: _finish_auth 0 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 1 -- 172.21.15.181:0/2530507928 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c0325a0 con 0x7f3f4c09ec50 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 1 -- 172.21.15.181:0/2530507928 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c032940 con 0x7f3f4c09ec50 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 5 monclient: authenticate success, global_id 4886 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 4 set_mon_vals no callback set 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f49d9b640 10 monclient: dump: 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 10 monclient: shutdown 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 -- 172.21.15.181:0/2530507928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 msgr2=0x7f3f4c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 --2- 172.21.15.181:0/2530507928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 0x7f3f4c131c60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009850 tx=0x7f3f3c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 -- 172.21.15.181:0/2530507928 shutdown_connections 2024-09-15T03:30:55.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 --2- 172.21.15.181:0/2530507928 >> v2:172.21.15.153:3300/0 conn(0x7f3f4c1321a0 0x7f3f4c068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 --2- 172.21.15.181:0/2530507928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c09ec50 0x7f3f4c131c60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 -- 172.21.15.181:0/2530507928 >> 172.21.15.181:0/2530507928 conn(0x7f3f4c0a5d40 msgr2=0x7f3f4c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 -- 172.21.15.181:0/2530507928 shutdown_connections 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.339+0000 7f3f50e68640 1 -- 172.21.15.181:0/2530507928 wait complete. 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.340+0000 7f3f50e68640 10 monclient: build_initial_monmap 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.340+0000 7f3f50e68640 10 monclient: monmap: 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:55.353 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.340+0000 7f3f50e68640 1 Processor -- start 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.340+0000 7f3f50e68640 1 -- start start 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.340+0000 7f3f50e68640 10 monclient: init 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4c09ec50 0x7f3f4c099cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4c09ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 0x7f3f4c09a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4c09f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c1321a0 0x7f3f4c09a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4c1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c130c40 con 0x7f3f4c09ec50 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c1315b0 con 0x7f3f4c09f020 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c09c000 con 0x7f3f4c1321a0 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f50e68640 10 monclient(hunting): authenticate will time out at 859.822145s 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4a59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4c09ec50 msgr2=0x7f3f4c099cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4c09ec50 0x7f3f4c099cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 0x7f3f4c09a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 0x7f3f4c09a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50018/0 (socket says 172.21.15.181:50018) 2024-09-15T03:30:55.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 -- 172.21.15.181:0/4121979148 learned_addr learned my addr 172.21.15.181:0/4121979148 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): get_auth_request con 0x7f3f4c09f020 auth_method 0 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c1321a0 0x7f3f4c09a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): get_auth_request con 0x7f3f4c1321a0 auth_method 0 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:55.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient(hunting): handle_auth_done global_id 44269 payload 824 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 -- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c1321a0 msgr2=0x7f3f4c09a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c1321a0 0x7f3f4c09a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 -- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4c09ec50 msgr2=0x7f3f4c099cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4c09ec50 0x7f3f4c099cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 monclient: found mon.noname-b 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.341+0000 7f3f41d9b640 1 -- 172.21.15.181:0/4121979148 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c009390 con 0x7f3f4c09f020 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f41d9b640 1 -- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 msgr2=0x7f3f4c09a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f41d9b640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 0x7f3f4c09a1f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400b600 tx=0x7f3f340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34015070 con 0x7f3f4c09f020 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: dump: 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.358 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f34003d90 con 0x7f3f4c09f020 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34006380 con 0x7f3f4c09f020 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f42ffd640 10 monclient: dump: 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.359 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f50e68640 5 monclient: authenticate success, global_id 44269 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f50e68640 10 monclient: _renew_subs 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f50e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c1255d0 con 0x7f3f4c09f020 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.343+0000 7f3f4959a640 4 set_mon_vals no callback set 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 10 monclient: _renew_subs 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c124670 con 0x7f3f4c09f020 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f34010440 con 0x7f3f4c09f020 2024-09-15T03:30:55.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f42ffd640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f2c0386f0 0x7f3f2c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.345+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f3f3404b710 con 0x7f3f4c09f020 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.344+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c09de70 con 0x7f3f4c09f020 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.345+0000 7f3f4a59c640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f2c0386f0 0x7f3f2c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.345+0000 7f3f4a59c640 10 monclient: get_auth_request con 0x7f3f2c0386f0 auth_method 0 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.347+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:55.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.347+0000 7f3f4a59c640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f2c0386f0 0x7f3f2c03aba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0a0150 tx=0x7f3f3c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:55.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.349+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f34018ab0 con 0x7f3f4c09f020 2024-09-15T03:30:55.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.349+0000 7f3f42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:55.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.349+0000 7f3f42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:55.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.507+0000 7f3f50e68640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T03:30:55.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.507+0000 7f3f50e68640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T03:30:55.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.507+0000 7f3f50e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.507+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7f3f4c0008d0 con 0x7f3f4c09f020 2024-09-15T03:30:55.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.799+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v57) v1 ==== 110+0+0 (secure 0 0 0) 0x7f3f34018690 con 0x7f3f4c09f020 2024-09-15T03:30:55.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.799+0000 7f3f42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T03:30:55.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.799+0000 7f3f42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f50e68640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T03:30:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f50e68640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T03:30:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f50e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7f3f4c0a13e0 con 0x7f3f4c09f020 2024-09-15T03:30:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f42ffd640 1 -- 172.21.15.181:0/4121979148 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v57) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3f34018ab0 con 0x7f3f4c09f020 2024-09-15T03:30:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f42ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-15T03:30:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.895+0000 7f3f42ffd640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-15T03:30:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:already renamed to 'bar-asdf' 2024-09-15T03:30:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f2c0386f0 msgr2=0x7f3f2c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f2c0386f0 0x7f3f2c03aba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0a0150 tx=0x7f3f3c032000 comp rx=0 tx=0).stop 2024-09-15T03:30:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 10 monclient: shutdown 2024-09-15T03:30:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 msgr2=0x7f3f4c09a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 0x7f3f4c09a1f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400b600 tx=0x7f3f340091a0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 shutdown_connections 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f2c0386f0 0x7f3f2c03aba0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f4c1321a0 0x7f3f4c09a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c09f020 0x7f3f4c09a1f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 --2- 172.21.15.181:0/4121979148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4c09ec50 0x7f3f4c099cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 >> 172.21.15.181:0/4121979148 conn(0x7f3f4c0a5d40 msgr2=0x7f3f4c0a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 shutdown_connections 2024-09-15T03:30:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.897+0000 7f3f50e68640 1 -- 172.21.15.181:0/4121979148 wait complete. 2024-09-15T03:30:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls 2024-09-15T03:30:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo-asdf 2024-09-15T03:30:55.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.979+0000 7ffba5eaf640 10 monclient: get_monmap_and_config 2024-09-15T03:30:55.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.979+0000 7ffba5eaf640 10 monclient: build_initial_monmap 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.979+0000 7ffba5eaf640 10 monclient: monmap: 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:55.980327+0000 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:55.980327+0000 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:55.981 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:55.982 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:55.982 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:55.982 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:55.982 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 Processor -- start 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 -- start start 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient: init 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): picked mon.noname-b con 0x7ffba0131c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ffba009d210 0x7ffba01325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): picked mon.noname-c con 0x7ffba009d210 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffba00a0ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:55.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba00a1800 con 0x7ffba00a0ab0 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba009fea0 con 0x7ffba009d210 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00a1520 con 0x7ffba0131c90 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffba5eaf640 10 monclient(hunting): authenticate will time out at 860.461755s 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9ffff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7ffba00a0ab0 legacy=0x7ffba0068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ffba009d210 0x7ffba01325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ffba009d210 0x7ffba01325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50032/0 (socket says 172.21.15.181:50032) 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 1 -- 172.21.15.181:0/2161623873 learned_addr learned my addr 172.21.15.181:0/2161623873 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 10 monclient(hunting): get_auth_request con 0x7ffba009d210 auth_method 0 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.980+0000 7ffb9f7fe640 1 --2- 172.21.15.181:0/2161623873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): get_auth_request con 0x7ffba0131c90 auth_method 0 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:55.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 10 monclient(hunting): handle_auth_done global_id 44272 payload 274 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 1 -- 172.21.15.181:0/2161623873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 msgr2=0x7ffba0132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 1 --2- 172.21.15.181:0/2161623873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0132060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 1 -- 172.21.15.181:0/2161623873 >> v1:172.21.15.153:6789/0 conn(0x7ffba00a0ab0 legacy=0x7ffba0068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 10 monclient: _finish_hunting 0 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 1 monclient: found mon.noname-c 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.981+0000 7ffb9effd640 1 -- 172.21.15.181:0/2161623873 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0069510 con 0x7ffba009d210 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.983+0000 7ffb9effd640 1 -- 172.21.15.181:0/2161623873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba009d210 msgr2=0x7ffba01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.983+0000 7ffb9effd640 1 --2- 172.21.15.181:0/2161623873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba009d210 0x7ffba01325a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009910 tx=0x7ffb940091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73764817e716dd9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 1 -- 172.21.15.181:0/2161623873 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9402aea0 con 0x7ffba009d210 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:55.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: dump: 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:55.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 1 -- 172.21.15.181:0/2161623873 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94032440 con 0x7ffba009d210 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 5 monclient: authenticate success, global_id 44272 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 1 -- 172.21.15.181:0/2161623873 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb940327e0 con 0x7ffba009d210 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 4 set_mon_vals no callback set 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffb9e7fc640 10 monclient: dump: 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.995 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 10 monclient: shutdown 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 -- 172.21.15.181:0/2161623873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba009d210 msgr2=0x7ffba01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/2161623873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba009d210 0x7ffba01325a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009910 tx=0x7ffb940091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 -- 172.21.15.181:0/2161623873 shutdown_connections 2024-09-15T03:30:55.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/2161623873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba009d210 0x7ffba01325a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/2161623873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 -- 172.21.15.181:0/2161623873 >> 172.21.15.181:0/2161623873 conn(0x7ffba0071370 msgr2=0x7ffba00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 -- 172.21.15.181:0/2161623873 shutdown_connections 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.984+0000 7ffba5eaf640 1 -- 172.21.15.181:0/2161623873 wait complete. 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.985+0000 7ffba5eaf640 10 monclient: build_initial_monmap 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.985+0000 7ffba5eaf640 10 monclient: monmap: 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:55.997 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.985+0000 7ffba5eaf640 1 Processor -- start 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.985+0000 7ffba5eaf640 1 -- start start 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.985+0000 7ffba5eaf640 10 monclient: init 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba00a0ab0 0x7ffba01412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): picked mon.noname-a con 0x7ffba00a0ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): picked mon.noname-c con 0x7ffba0131c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 0x7ffba011e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): picked mon.noname-b con 0x7ffba0141d70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00a0fb0 con 0x7ffba00a0ab0 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba009f120 con 0x7ffba0141d70 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba0121ec0 con 0x7ffba0131c90 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffba5eaf640 10 monclient(hunting): authenticate will time out at 860.467194s 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9f7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba00a0ab0 msgr2=0x7ffba01412f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 0x7ffba011e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba00a0ab0 0x7ffba01412f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 0x7ffba011e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50038/0 (socket says 172.21.15.181:50038) 2024-09-15T03:30:55.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 -- 172.21.15.181:0/605513339 learned_addr learned my addr 172.21.15.181:0/605513339 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): get_auth_request con 0x7ffba0141d70 auth_method 0 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9effd640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9effd640 10 monclient(hunting): get_auth_request con 0x7ffba0131c90 auth_method 0 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient(hunting): handle_auth_done global_id 44275 payload 824 2024-09-15T03:30:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 -- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 msgr2=0x7ffba0141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 -- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba00a0ab0 msgr2=0x7ffba01412f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba00a0ab0 0x7ffba01412f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 monclient: found mon.noname-b 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9ffff640 1 -- 172.21.15.181:0/605513339 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94009450 con 0x7ffba0141d70 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.986+0000 7ffb9effd640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.988+0000 7ffb9ffff640 1 -- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 msgr2=0x7ffba011e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.988+0000 7ffb9ffff640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 0x7ffba011e620 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800bb40 tx=0x7ffb980097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.988+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9800cbf0 con 0x7ffba0141d70 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: dump: 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.006 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffba5eaf640 5 monclient: authenticate success, global_id 44275 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9800cd70 con 0x7ffba0141d70 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffba5eaf640 10 monclient: _renew_subs 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffba5eaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba0121be0 con 0x7ffba0141d70 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98006150 con 0x7ffba0141d70 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb877fe640 10 monclient: dump: 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.008 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 10 monclient: _renew_subs 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba0122e60 con 0x7ffba0141d70 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.990+0000 7ffb9dffb640 4 set_mon_vals no callback set 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffb98012020 con 0x7ffba0141d70 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.991+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba00008d0 con 0x7ffba0141d70 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.992+0000 7ffb877fe640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffb8c038740 0x7ffb8c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.992+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5151+0+0 (secure 0 0 0) 0x7ffb9804bb80 con 0x7ffba0141d70 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.992+0000 7ffb9f7fe640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffb8c038740 0x7ffb8c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.992+0000 7ffb9f7fe640 10 monclient: get_auth_request con 0x7ffb8c038740 auth_method 0 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.994+0000 7ffb9f7fe640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffb8c038740 0x7ffb8c03abf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009430 tx=0x7ffb900090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.996+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffb9801d900 con 0x7ffba0141d70 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.996+0000 7ffb877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:56.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:55.996+0000 7ffb877fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:56.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffba5eaf640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffba5eaf640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffba5eaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ffba01200e0 con 0x7ffba0141d70 2024-09-15T03:30:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (secure 0 0 0) 0x7ffb9801d300 con 0x7ffba0141d70 2024-09-15T03:30:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffb877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.152+0000 7ffb877fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:56.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffba5eaf640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffba5eaf640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffba5eaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ffba01230e0 con 0x7ffba0141d70 2024-09-15T03:30:56.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffb877fe640 1 -- 172.21.15.181:0/605513339 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (secure 0 0 0) 0x7ffb9801d4e0 con 0x7ffba0141d70 2024-09-15T03:30:56.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffb877fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.246+0000 7ffb877fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffb8c038740 msgr2=0x7ffb8c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffb8c038740 0x7ffb8c03abf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009430 tx=0x7ffb900090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 10 monclient: shutdown 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 msgr2=0x7ffba011e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 0x7ffba011e620 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800bb40 tx=0x7ffb980097d0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 shutdown_connections 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffb8c038740 0x7ffb8c03abf0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0141d70 0x7ffba011e620 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffba0131c90 0x7ffba0141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 --2- 172.21.15.181:0/605513339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffba00a0ab0 0x7ffba01412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 >> 172.21.15.181:0/605513339 conn(0x7ffba0071370 msgr2=0x7ffba00a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 shutdown_connections 2024-09-15T03:30:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.248+0000 7ffba5eaf640 1 -- 172.21.15.181:0/605513339 wait complete. 2024-09-15T03:30:56.258 INFO:tasks.workunit.client.0.smithi181.stdout:foo-asdf 2024-09-15T03:30:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls 2024-09-15T03:30:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep bar-asdf 2024-09-15T03:30:56.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.330+0000 7fa4346d4640 10 monclient: get_monmap_and_config 2024-09-15T03:30:56.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.330+0000 7fa4346d4640 10 monclient: build_initial_monmap 2024-09-15T03:30:56.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.330+0000 7fa4346d4640 10 monclient: monmap: 2024-09-15T03:30:56.331 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:56.331 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.331 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:56.331273+0000 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:56.331273+0000 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:56.332 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 1 Processor -- start 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 1 -- start start 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient: init 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa42c09c250 0x7fa42c09da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c09c250 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:56.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa42c09dfa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa42c09e3a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c09b790 con 0x7fa42c09dfa0 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c09b4b0 con 0x7fa42c09e3a0 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c09b230 con 0x7fa42c09c250 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa4346d4640 10 monclient(hunting): authenticate will time out at 860.812735s 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa432470640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa42c09c250 msgr2=0x7fa42c09da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa432c71640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa42c09e3a0 0x7fa42c09cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40884/0 (socket says 172.21.15.181:40884) 2024-09-15T03:30:56.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa431c6f640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa42c09dfa0 legacy=0x7fa42c09c620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:30:56.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 learned_addr learned my addr 172.21.15.181:0/4088357321 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:30:56.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.331+0000 7fa432470640 1 --2- 172.21.15.181:0/4088357321 >> v2:172.21.15.153:3300/0 conn(0x7fa42c09c250 0x7fa42c09da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:56.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 >> v1:172.21.15.181:6789/0 conn(0x7fa42c09e3a0 legacy=0x7fa42c09cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836503112 0 0) 0x7fa42c09b4b0 con 0x7fa42c09e3a0 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): my global_id is 44278 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4180034e0 con 0x7fa42c09e3a0 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948781416 0 0) 0x7fa4180034e0 con 0x7fa42c09e3a0 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 >> v2:172.21.15.153:3300/0 conn(0x7fa42c09c250 msgr2=0x7fa42c09da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 --2- 172.21.15.181:0/4088357321 >> v2:172.21.15.153:3300/0 conn(0x7fa42c09c250 0x7fa42c09da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 >> v1:172.21.15.153:6789/0 conn(0x7fa42c09dfa0 legacy=0x7fa42c09c620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: _finish_hunting 0 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:30:56.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c09ba70 con 0x7fa42c09e3a0 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: _finish_auth 0 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa4346d4640 5 monclient: authenticate success, global_id 44278 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fa424001b60 con 0x7fa42c09e3a0 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa4346d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: dump: 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.347 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 >> v1:172.21.15.181:6789/0 conn(0x7fa42c09e3a0 legacy=0x7fa42c09cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4180039d0 0x7fa418003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): picked mon.a con 0x7fa4180039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa418004780 0x7fa418004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): picked mon.c con 0x7fa418004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa41800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): picked mon.b con 0x7fa418005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c09b230 con 0x7fa4180039d0 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41800c240 con 0x7fa418005280 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41800c400 con 0x7fa418004780 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa43146e640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:30:56.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa4346d4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa41800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): get_auth_request con 0x7fa418005280 auth_method 0 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432470640 1 -- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4180039d0 msgr2=0x7fa418003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432470640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4180039d0 0x7fa418003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient(hunting): handle_auth_done global_id 44278 payload 293 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa418004780 msgr2=0x7fa418004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa418004780 0x7fa418004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4180039d0 msgr2=0x7fa418003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4180039d0 0x7fa418003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient: _finish_hunting 0 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 monclient: found mon.b 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4180034e0 con 0x7fa418005280 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 10 monclient: _finish_auth 0 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.334+0000 7fa432c71640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.335+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 reap_dead start 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa432c71640 1 -- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 msgr2=0x7fa41800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa432c71640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa41800b9d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa42400abd0 tx=0x7fa42400a370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7a122890943e6be server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42402b7b0 con 0x7fa418005280 2024-09-15T03:30:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: dump: 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42400d040 con 0x7fa418005280 2024-09-15T03:30:56.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa4346d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa4346d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa4346d4640 4 set_mon_vals no callback set 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 1 -- 172.21.15.181:0/4088357321 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42402bb80 con 0x7fa418005280 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.337+0000 7fa43146e640 10 monclient: dump: 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.353 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 10 monclient: shutdown 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 -- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 msgr2=0x7fa41800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa41800b9d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa42400abd0 tx=0x7fa42400a370 comp rx=0 tx=0).stop 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 -- 172.21.15.181:0/4088357321 shutdown_connections 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 --2- 172.21.15.181:0/4088357321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa41800b9d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 -- 172.21.15.181:0/4088357321 >> 172.21.15.181:0/4088357321 conn(0x7fa42c071340 msgr2=0x7fa42c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 -- 172.21.15.181:0/4088357321 shutdown_connections 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 1 -- 172.21.15.181:0/4088357321 wait complete. 2024-09-15T03:30:56.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 10 monclient: build_initial_monmap 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.338+0000 7fa4346d4640 10 monclient: monmap: 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 Processor -- start 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 -- start start 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient: init 2024-09-15T03:30:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa42c132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): picked mon.noname-b con 0x7fa418005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c132e80 0x7fa42c11c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c132e80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c11c9e0 0x7fa42c11ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c11c9e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c11e520 con 0x7fa42c132e80 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c06e970 con 0x7fa418005280 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:56.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c0a2a10 con 0x7fa42c11c9e0 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4346d4640 10 monclient(hunting): authenticate will time out at 860.820833s 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa42c132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa42c132940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50058/0 (socket says 172.21.15.181:50058) 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 1 -- 172.21.15.181:0/1536232535 learned_addr learned my addr 172.21.15.181:0/1536232535 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 10 monclient(hunting): get_auth_request con 0x7fa418005280 auth_method 0 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa431c6f640 1 -- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c132e80 msgr2=0x7fa42c11c4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa431c6f640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c132e80 0x7fa42c11c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:56.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.339+0000 7fa432470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432c71640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c11c9e0 0x7fa42c11ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432c71640 10 monclient(hunting): get_auth_request con 0x7fa42c11c9e0 auth_method 0 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432c71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 10 monclient(hunting): handle_auth_done global_id 44281 payload 824 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 1 -- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c11c9e0 msgr2=0x7fa42c11ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c11c9e0 0x7fa42c11ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 1 -- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c132e80 msgr2=0x7fa42c11c4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c132e80 0x7fa42c11c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 10 monclient: _finish_hunting 0 2024-09-15T03:30:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 1 monclient: found mon.noname-b 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432470640 1 -- 172.21.15.181:0/1536232535 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c11eb70 con 0x7fa418005280 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.340+0000 7fa432c71640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c11c9e0 0x7fa42c11ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa432470640 1 -- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 msgr2=0x7fa42c132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa432470640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa42c132940 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa42800b840 tx=0x7fa4280091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428003e20 con 0x7fa418005280 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: dump: 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: _finish_auth 0 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa428006150 con 0x7fa418005280 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428006300 con 0x7fa418005280 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4227fc640 10 monclient: dump: 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 5 monclient: authenticate success, global_id 44281 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 10 monclient: _renew_subs 2024-09-15T03:30:56.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa430c6d640 4 set_mon_vals no callback set 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c11fa60 con 0x7fa418005280 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 10 monclient: _renew_subs 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.342+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c133f50 con 0x7fa418005280 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4346d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4346d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4346d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa428012020 con 0x7fa418005280 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c0008d0 con 0x7fa418005280 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4227fc640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4100386f0 0x7fa41003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.343+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5151+0+0 (secure 0 0 0) 0x7fa42804afb0 con 0x7fa418005280 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.344+0000 7fa431c6f640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4100386f0 0x7fa41003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:56.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.344+0000 7fa431c6f640 10 monclient: get_auth_request con 0x7fa4100386f0 auth_method 0 2024-09-15T03:30:56.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.346+0000 7fa431c6f640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4100386f0 0x7fa41003aba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa42c11dc30 tx=0x7fa41c007ce0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:56.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.348+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa428014570 con 0x7fa418005280 2024-09-15T03:30:56.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.348+0000 7fa4227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:56.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.348+0000 7fa4227fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:56.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.506+0000 7fa4346d4640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.506+0000 7fa4346d4640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.506+0000 7fa4346d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.506+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fa42c099ed0 con 0x7fa418005280 2024-09-15T03:30:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.507+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (secure 0 0 0) 0x7fa428013ad0 con 0x7fa418005280 2024-09-15T03:30:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.507+0000 7fa4227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.507+0000 7fa4227fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.601+0000 7fa4346d4640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.601+0000 7fa4346d4640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.601+0000 7fa4346d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.601+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fa42c072640 con 0x7fa418005280 2024-09-15T03:30:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.602+0000 7fa4227fc640 1 -- 172.21.15.181:0/1536232535 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (secure 0 0 0) 0x7fa428024080 con 0x7fa418005280 2024-09-15T03:30:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.602+0000 7fa4227fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:30:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.602+0000 7fa4227fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:30:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4100386f0 msgr2=0x7fa41003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4100386f0 0x7fa41003aba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa42c11dc30 tx=0x7fa41c007ce0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 10 monclient: shutdown 2024-09-15T03:30:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 msgr2=0x7fa42c132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa42c132940 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa42800b840 tx=0x7fa4280091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 shutdown_connections 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4100386f0 0x7fa41003aba0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c11c9e0 0x7fa42c11ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c132e80 0x7fa42c11c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 --2- 172.21.15.181:0/1536232535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa418005280 0x7fa42c132940 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 >> 172.21.15.181:0/1536232535 conn(0x7fa42c071340 msgr2=0x7fa42c096a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 shutdown_connections 2024-09-15T03:30:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:56.604+0000 7fa4346d4640 1 -- 172.21.15.181:0/1536232535 wait complete. 2024-09-15T03:30:56.613 INFO:tasks.workunit.client.0.smithi181.stdout:bar-asdf 2024-09-15T03:30:56.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm foo 2024-09-15T03:30:56.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'does not exist' 2024-09-15T03:30:56.972 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:56.865+0000 7f51b67fc640 1 -- 172.21.15.181:0/2100040752 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v57) v1 ==== 103+0+0 (secure 0 0 0) 0x7f51c0017020 con 0x7f51c81321a0 2024-09-15T03:30:56.972 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:56.865+0000 7f51b67fc640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-15T03:30:56.972 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:56.960+0000 7f51b67fc640 1 -- 172.21.15.181:0/2100040752 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v57) v1 ==== 103+0+0 (secure 0 0 0) 0x7f51c0010300 con 0x7f51c81321a0 2024-09-15T03:30:56.972 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:56.960+0000 7f51b67fc640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-15T03:30:56.972 INFO:tasks.workunit.client.0.smithi181.stdout:rule foo does not exist 2024-09-15T03:30:56.972 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm bar 2024-09-15T03:30:56.973 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'does not exist' 2024-09-15T03:30:57.949 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:57.840+0000 7f36d37fe640 1 -- 172.21.15.181:0/1260325656 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v57) v1 ==== 103+0+0 (secure 0 0 0) 0x7f36c403f070 con 0x7f36dc09f020 2024-09-15T03:30:57.949 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:57.840+0000 7f36d37fe640 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-15T03:30:57.949 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:57.937+0000 7f36d37fe640 1 -- 172.21.15.181:0/1260325656 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v57) v1 ==== 103+0+0 (secure 0 0 0) 0x7f36c4037030 con 0x7f36dc09f020 2024-09-15T03:30:57.949 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-15T03:30:57.937+0000 7f36d37fe640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-15T03:30:57.949 INFO:tasks.workunit.client.0.smithi181.stdout:rule bar does not exist 2024-09-15T03:30:57.950 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.024+0000 7f6a0671d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.024+0000 7f6a0671d640 10 monclient: build_initial_monmap 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.024+0000 7f6a0671d640 10 monclient: monmap: 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:58.025034+0000 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:58.025034+0000 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:58.025 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.026 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:58.026 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:58.026 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:58.026 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:58.026 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:58.026 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.024+0000 7f6a0671d640 1 Processor -- start 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 1 -- start start 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient: init 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6a00098050 0x7f6a00132000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a00098050 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 0x7f6a00098420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0009fe90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a00098960 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000a1230 con 0x7f6a00098960 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0006a230 con 0x7f6a00098050 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a000a1510 con 0x7f6a0009fe90 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f6a0671d640 10 monclient(hunting): authenticate will time out at 862.506562s 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6a00098050 0x7f6a00132000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6a00098050 0x7f6a00132000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50092/0 (socket says 172.21.15.181:50092) 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 1 -- 172.21.15.181:0/2003024233 learned_addr learned my addr 172.21.15.181:0/2003024233 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): get_auth_request con 0x7f6a00098050 auth_method 0 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:58.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 1 --2- 172.21.15.181:0/2003024233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 0x7f6a00098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): get_auth_request con 0x7f6a0009fe90 auth_method 0 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.025+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_done global_id 4901 payload 274 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 1 -- 172.21.15.181:0/2003024233 >> v2:172.21.15.181:3300/0 conn(0x7f6a00098050 msgr2=0x7f6a00132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 1 --2- 172.21.15.181:0/2003024233 >> v2:172.21.15.181:3300/0 conn(0x7f6a00098050 0x7f6a00132000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 1 -- 172.21.15.181:0/2003024233 >> v1:172.21.15.181:6789/0 conn(0x7f6a00098960 legacy=0x7f6a00068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:30:58.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 1 monclient: found mon.noname-b 2024-09-15T03:30:58.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:58.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:30:58.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.026+0000 7f69ff7fe640 1 -- 172.21.15.181:0/2003024233 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00132540 con 0x7f6a0009fe90 2024-09-15T03:30:58.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69ff7fe640 1 -- 172.21.15.181:0/2003024233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 msgr2=0x7f6a00098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:58.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69ff7fe640 1 --2- 172.21.15.181:0/2003024233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 0x7f6a00098420 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009850 tx=0x7f69f40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c579113408eb1ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:58.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 1 -- 172.21.15.181:0/2003024233 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f402adf0 con 0x7f6a0009fe90 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: dump: 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:58.038 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: _finish_auth 0 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 1 -- 172.21.15.181:0/2003024233 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f4032440 con 0x7f6a0009fe90 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 5 monclient: authenticate success, global_id 4901 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 1 -- 172.21.15.181:0/2003024233 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f40327e0 con 0x7f6a0009fe90 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 4 set_mon_vals no callback set 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f69feffd640 10 monclient: dump: 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:58.039 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 10 monclient: shutdown 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 -- 172.21.15.181:0/2003024233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 msgr2=0x7f6a00098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 --2- 172.21.15.181:0/2003024233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 0x7f6a00098420 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009850 tx=0x7f69f40090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 -- 172.21.15.181:0/2003024233 shutdown_connections 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 --2- 172.21.15.181:0/2003024233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0009fe90 0x7f6a00098420 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 --2- 172.21.15.181:0/2003024233 >> v2:172.21.15.181:3300/0 conn(0x7f6a00098050 0x7f6a00132000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 -- 172.21.15.181:0/2003024233 >> 172.21.15.181:0/2003024233 conn(0x7f6a000a5190 msgr2=0x7f6a000a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 -- 172.21.15.181:0/2003024233 shutdown_connections 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.029+0000 7f6a0671d640 1 -- 172.21.15.181:0/2003024233 wait complete. 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.030+0000 7f6a0671d640 10 monclient: build_initial_monmap 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.030+0000 7f6a0671d640 10 monclient: monmap: 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.041 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.030+0000 7f6a0671d640 1 Processor -- start 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.030+0000 7f6a0671d640 1 -- start start 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.030+0000 7f6a0671d640 10 monclient: init 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 0x7f6a0009e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a00098050 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a00098960 0x7f6a00130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a00098960 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a0009fe90 0x7f6a00140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0009fe90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0009b380 con 0x7f6a0009fe90 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0011e3c0 con 0x7f6a00098050 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0009f1b0 con 0x7f6a00098960 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a0671d640 10 monclient(hunting): authenticate will time out at 862.512287s 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 0x7f6a0009e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 0x7f6a0009e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50096/0 (socket says 172.21.15.181:50096) 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 -- 172.21.15.181:0/921729333 learned_addr learned my addr 172.21.15.181:0/921729333 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): get_auth_request con 0x7f6a00098050 auth_method 0 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:58.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a04cba640 1 -- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a0009fe90 msgr2=0x7f6a00140f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f6a04cba640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a0009fe90 0x7f6a00140f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a00098960 0x7f6a00130c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): get_auth_request con 0x7f6a00098960 auth_method 0 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:58.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient(hunting): handle_auth_done global_id 44302 payload 824 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 -- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a00098960 msgr2=0x7f6a00130c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a00098960 0x7f6a00130c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 -- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a0009fe90 msgr2=0x7f6a00140f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a0009fe90 0x7f6a00140f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient: _finish_hunting 0 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 monclient: found mon.noname-b 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.031+0000 7f69fffff640 1 -- 172.21.15.181:0/921729333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f4009390 con 0x7f6a00098050 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.033+0000 7f69fffff640 1 -- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 msgr2=0x7f6a0009e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.033+0000 7f69fffff640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 0x7f6a0009e5f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f69f000ea20 tx=0x7f69f00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00095a0 con 0x7f6a00098050 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: dump: 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: _finish_auth 0 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f000cb50 con 0x7f6a00098050 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 5 monclient: authenticate success, global_id 44302 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 10 monclient: _renew_subs 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0011ddd0 con 0x7f6a00098050 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0013470 con 0x7f6a00098050 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69e7fff640 10 monclient: dump: 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f69fe7fc640 4 set_mon_vals no callback set 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 10 monclient: _renew_subs 2024-09-15T03:30:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.034+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0011c6e0 con 0x7f6a00098050 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f69f00137e0 con 0x7f6a00098050 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f6a0671d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f6a0671d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f6a0671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0004bf60 con 0x7f6a00098050 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f69e7fff640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69ec038740 0x7f69ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f69f004c240 con 0x7f6a00098050 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f69ff7fe640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69ec038740 0x7f69ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.035+0000 7f69ff7fe640 10 monclient: get_auth_request con 0x7f69ec038740 auth_method 0 2024-09-15T03:30:58.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.037+0000 7f69ff7fe640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69ec038740 0x7f69ec03abf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f69f4031000 tx=0x7f69f4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:58.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.039+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f69f001b770 con 0x7f6a00098050 2024-09-15T03:30:58.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.039+0000 7f69e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:58.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.039+0000 7f69e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:58.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.196+0000 7f6a0671d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:58.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.196+0000 7f6a0671d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:58.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.196+0000 7f6a0671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:58.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.196+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f6a00131bf0 con 0x7f6a00098050 2024-09-15T03:30:58.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.809+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v58) v1 ==== 110+0+0 (secure 0 0 0) 0x7f69f001b3b0 con 0x7f6a00098050 2024-09-15T03:30:58.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.809+0000 7f69e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:58.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.809+0000 7f69e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.906+0000 7f6a0671d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.907+0000 7f6a0671d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.907+0000 7f6a0671d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.907+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f6a00064a00 con 0x7f6a00098050 2024-09-15T03:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.907+0000 7f69e7fff640 1 -- 172.21.15.181:0/921729333 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v58) v1 ==== 134+0+0 (secure 0 0 0) 0x7f69f001b590 con 0x7f6a00098050 2024-09-15T03:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.907+0000 7f69e7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:58.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.907+0000 7f69e7fff640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-15T03:30:58.909 INFO:tasks.workunit.client.0.smithi181.stderr:already renamed to 'foo' 2024-09-15T03:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69ec038740 msgr2=0x7f69ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69ec038740 0x7f69ec03abf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f69f4031000 tx=0x7f69f4032000 comp rx=0 tx=0).stop 2024-09-15T03:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 10 monclient: shutdown 2024-09-15T03:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 msgr2=0x7f6a0009e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 0x7f6a0009e5f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f69f000ea20 tx=0x7f69f00091d0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 shutdown_connections 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69ec038740 0x7f69ec03abf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a0009fe90 0x7f6a00140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a00098960 0x7f6a00130c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 --2- 172.21.15.181:0/921729333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a00098050 0x7f6a0009e5f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 >> 172.21.15.181:0/921729333 conn(0x7f6a000a5190 msgr2=0x7f6a000a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 shutdown_connections 2024-09-15T03:30:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.909+0000 7f6a0671d640 1 -- 172.21.15.181:0/921729333 wait complete. 2024-09-15T03:30:58.919 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-15T03:30:58.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.992+0000 7fed96d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.992+0000 7fed96d9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.992+0000 7fed96d9d640 10 monclient: monmap: 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:58.993601+0000 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:58.993601+0000 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:58.994 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:58.995 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:58.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 1 Processor -- start 2024-09-15T03:30:58.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 1 -- start start 2024-09-15T03:30:58.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 10 monclient: init 2024-09-15T03:30:58.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fed900972c0 0x7fed90099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.993+0000 7fed96d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fed900972c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed90097cb0 0x7fed9009b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fed90097cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fed9009bf10 0x7fed9009c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fed9009bf10 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed900989d0 con 0x7fed900972c0 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed90097690 con 0x7fed9009bf10 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:58.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9009d8d0 con 0x7fed90097cb0 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed96d9d640 10 monclient(hunting): authenticate will time out at 863.475043s 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fed9009bf10 0x7fed9009c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed94b39640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fed900972c0 msgr2=0x7fed90099040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fed9009bf10 0x7fed9009c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48106/0 (socket says 172.21.15.181:48106) 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 -- 172.21.15.181:0/3121336387 learned_addr learned my addr 172.21.15.181:0/3121336387 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): get_auth_request con 0x7fed9009bf10 auth_method 0 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed94b39640 1 --2- 172.21.15.181:0/3121336387 >> v2:172.21.15.153:3300/0 conn(0x7fed900972c0 0x7fed90099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed8f7fe640 1 -- 172.21.15.181:0/3121336387 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:30:58.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed8ffff640 1 --2- 172.21.15.181:0/3121336387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed90097cb0 0x7fed9009b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed8ffff640 10 monclient(hunting): get_auth_request con 0x7fed90097cb0 auth_method 0 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient(hunting): handle_auth_done global_id 44305 payload 274 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 -- 172.21.15.181:0/3121336387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed90097cb0 msgr2=0x7fed9009b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 --2- 172.21.15.181:0/3121336387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed90097cb0 0x7fed9009b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 -- 172.21.15.181:0/3121336387 >> v2:172.21.15.153:3300/0 conn(0x7fed900972c0 msgr2=0x7fed90099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 --2- 172.21.15.181:0/3121336387 >> v2:172.21.15.153:3300/0 conn(0x7fed900972c0 0x7fed90099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:59.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 monclient: found mon.noname-c 2024-09-15T03:30:59.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:59.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.994+0000 7fed9533a640 1 -- 172.21.15.181:0/3121336387 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed90097a70 con 0x7fed9009bf10 2024-09-15T03:30:59.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed9533a640 1 -- 172.21.15.181:0/3121336387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 msgr2=0x7fed9009c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:59.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed9533a640 1 --2- 172.21.15.181:0/3121336387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9009c350 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fed9009ab70 tx=0x7fed8800a950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1ad8647a7e22a9c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:59.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 1 -- 172.21.15.181:0/3121336387 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88008510 con 0x7fed9009bf10 2024-09-15T03:30:59.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: dump: 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 1 -- 172.21.15.181:0/3121336387 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8802e050 con 0x7fed9009bf10 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 5 monclient: authenticate success, global_id 44305 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 1 -- 172.21.15.181:0/3121336387 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed880088a0 con 0x7fed9009bf10 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 4 set_mon_vals no callback set 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed8f7fe640 10 monclient: dump: 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.009 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 10 monclient: shutdown 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 -- 172.21.15.181:0/3121336387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 msgr2=0x7fed9009c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 --2- 172.21.15.181:0/3121336387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9009c350 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fed9009ab70 tx=0x7fed8800a950 comp rx=0 tx=0).stop 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 -- 172.21.15.181:0/3121336387 shutdown_connections 2024-09-15T03:30:59.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 --2- 172.21.15.181:0/3121336387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9009c350 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 --2- 172.21.15.181:0/3121336387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed90097cb0 0x7fed9009b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 --2- 172.21.15.181:0/3121336387 >> v2:172.21.15.153:3300/0 conn(0x7fed900972c0 0x7fed90099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.997+0000 7fed96d9d640 1 -- 172.21.15.181:0/3121336387 >> 172.21.15.181:0/3121336387 conn(0x7fed900a0fe0 msgr2=0x7fed900a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 1 -- 172.21.15.181:0/3121336387 shutdown_connections 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 1 -- 172.21.15.181:0/3121336387 wait complete. 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 10 monclient: monmap: 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.011 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 1 Processor -- start 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 1 -- start start 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.998+0000 7fed96d9d640 10 monclient: init 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed900972c0 0x7fed901410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fed900972c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed90097cb0 0x7fed90131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fed90097cb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:59.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9011b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fed9009bf10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9009cc90 con 0x7fed90097cb0 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9009f110 con 0x7fed9009bf10 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9011eda0 con 0x7fed900972c0 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed96d9d640 10 monclient(hunting): authenticate will time out at 863.480359s 2024-09-15T03:30:59.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9011b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed8ffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed90097cb0 msgr2=0x7fed90131c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9011b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48110/0 (socket says 172.21.15.181:48110) 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 -- 172.21.15.181:0/2338563883 learned_addr learned my addr 172.21.15.181:0/2338563883 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): get_auth_request con 0x7fed9009bf10 auth_method 0 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed8ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed90097cb0 0x7fed90131c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed900972c0 0x7fed901410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:59.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): get_auth_request con 0x7fed900972c0 auth_method 0 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed94b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient(hunting): handle_auth_done global_id 44308 payload 824 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 -- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed900972c0 msgr2=0x7fed901410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed900972c0 0x7fed901410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 -- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed90097cb0 msgr2=0x7fed90131c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed90097cb0 0x7fed90131c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:59.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 monclient: found mon.noname-b 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:58.999+0000 7fed9533a640 1 -- 172.21.15.181:0/2338563883 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8800abf0 con 0x7fed9009bf10 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.001+0000 7fed9533a640 1 -- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 msgr2=0x7fed9011b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.001+0000 7fed9533a640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9011b7c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fed8800c100 tx=0x7fed88035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed880338b0 con 0x7fed9009bf10 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: dump: 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.016 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed88033a30 con 0x7fed9009bf10 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 5 monclient: authenticate success, global_id 44308 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:59.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88033df0 con 0x7fed9009bf10 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8cff9640 10 monclient: dump: 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 10 monclient: _renew_subs 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9011f080 con 0x7fed9009bf10 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed8effd640 4 set_mon_vals no callback set 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 10 monclient: _renew_subs 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.002+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9011deb0 con 0x7fed9009bf10 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fed88039050 con 0x7fed9009bf10 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed96d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed96d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed96d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9004bf60 con 0x7fed9009bf10 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed8cff9640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed7c0386f0 0x7fed7c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fed8806c870 con 0x7fed9009bf10 2024-09-15T03:30:59.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed94b39640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed7c0386f0 0x7fed7c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.003+0000 7fed94b39640 10 monclient: get_auth_request con 0x7fed7c0386f0 auth_method 0 2024-09-15T03:30:59.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.005+0000 7fed94b39640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed7c0386f0 0x7fed7c03aba0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fed800038f0 tx=0x7fed800090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:59.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.007+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fed8803c620 con 0x7fed9009bf10 2024-09-15T03:30:59.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.007+0000 7fed8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:59.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.007+0000 7fed8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:59.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.163+0000 7fed96d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:59.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.163+0000 7fed96d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:59.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.163+0000 7fed96d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.163+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7fed9006ec40 con 0x7fed9009bf10 2024-09-15T03:30:59.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.163+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v58) v1 ==== 134+0+0 (secure 0 0 0) 0x7fed88040070 con 0x7fed9009bf10 2024-09-15T03:30:59.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.164+0000 7fed8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:59.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.164+0000 7fed8cff9640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.258+0000 7fed96d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.258+0000 7fed96d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.258+0000 7fed96d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.258+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7fed90064b40 con 0x7fed9009bf10 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.259+0000 7fed8cff9640 1 -- 172.21.15.181:0/2338563883 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v58) v1 ==== 134+0+0 (secure 0 0 0) 0x7fed8804a030 con 0x7fed9009bf10 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.259+0000 7fed8cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.259+0000 7fed8cff9640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-15T03:30:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:already renamed to 'foo' 2024-09-15T03:30:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed7c0386f0 msgr2=0x7fed7c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed7c0386f0 0x7fed7c03aba0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fed800038f0 tx=0x7fed800090f0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 10 monclient: shutdown 2024-09-15T03:30:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 msgr2=0x7fed9011b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9011b7c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fed8800c100 tx=0x7fed88035000 comp rx=0 tx=0).stop 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 shutdown_connections 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed7c0386f0 0x7fed7c03aba0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed9009bf10 0x7fed9011b7c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed90097cb0 0x7fed90131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 --2- 172.21.15.181:0/2338563883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed900972c0 0x7fed901410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 >> 172.21.15.181:0/2338563883 conn(0x7fed900a0fe0 msgr2=0x7fed9006da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 shutdown_connections 2024-09-15T03:30:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.261+0000 7fed96d9d640 1 -- 172.21.15.181:0/2338563883 wait complete. 2024-09-15T03:30:59.270 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-15T03:30:59.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.344+0000 7f68feb9d640 10 monclient: get_monmap_and_config 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.344+0000 7f68feb9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.344+0000 7f68feb9d640 10 monclient: monmap: 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:30:59.345531+0000 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:30:59.345531+0000 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:30:59.346 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:30:59.347 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:30:59.347 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:30:59.347 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 Processor -- start 2024-09-15T03:30:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 -- start start 2024-09-15T03:30:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient: init 2024-09-15T03:30:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:30:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68f8099e10 addr v1:172.21.15.181:6789/0 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f68f809dfa0 0x7f68f809d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f68f809dfa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f68f81348d0 0x7f68f8136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f68f81348d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f809f620 con 0x7f68f8099e10 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f809b7c0 con 0x7f68f809dfa0 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f809f860 con 0x7f68f81348d0 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.345+0000 7f68feb9d640 10 monclient(hunting): authenticate will time out at 863.826920s 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f68f81348d0 0x7f68f8136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68f7fff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f68f809dfa0 msgr2=0x7f68f809d1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68f7fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f68f809dfa0 0x7f68f809d1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f68f81348d0 0x7f68f8136cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48126/0 (socket says 172.21.15.181:48126) 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 1 -- 172.21.15.181:0/1047839550 learned_addr learned my addr 172.21.15.181:0/1047839550 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 10 monclient(hunting): get_auth_request con 0x7f68f81348d0 auth_method 0 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.346+0000 7f68fd13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fc939640 1 -- 172.21.15.181:0/1047839550 >> v1:172.21.15.181:6789/0 conn(0x7f68f8099e10 legacy=0x7f68f809e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 10 monclient(hunting): handle_auth_done global_id 44311 payload 274 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 1 -- 172.21.15.181:0/1047839550 >> v2:172.21.15.153:3300/0 conn(0x7f68f809dfa0 msgr2=0x7f68f809d1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 1 --2- 172.21.15.181:0/1047839550 >> v2:172.21.15.153:3300/0 conn(0x7f68f809dfa0 0x7f68f809d1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 1 -- 172.21.15.181:0/1047839550 >> v1:172.21.15.181:6789/0 conn(0x7f68f8099e10 legacy=0x7f68f809e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 10 monclient: _finish_hunting 0 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 1 monclient: found mon.noname-c 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.348+0000 7f68fd13a640 1 -- 172.21.15.181:0/1047839550 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f809b4e0 con 0x7f68f81348d0 2024-09-15T03:30:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68fd13a640 1 -- 172.21.15.181:0/1047839550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f81348d0 msgr2=0x7f68f8136cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68fd13a640 1 --2- 172.21.15.181:0/1047839550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f81348d0 0x7f68f8136cb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f68f80648a0 tx=0x7f68f0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c8bdbba5218889 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 1 -- 172.21.15.181:0/1047839550 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f002e050 con 0x7f68f81348d0 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: dump: 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: _finish_auth 0 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 1 -- 172.21.15.181:0/1047839550 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f0007610 con 0x7f68f81348d0 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68feb9d640 5 monclient: authenticate success, global_id 44311 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68feb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68feb9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:30:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 1 -- 172.21.15.181:0/1047839550 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f00079f0 con 0x7f68f81348d0 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68feb9d640 4 set_mon_vals no callback set 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.351+0000 7f68f77fe640 10 monclient: dump: 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.360 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 10 monclient: shutdown 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 -- 172.21.15.181:0/1047839550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f81348d0 msgr2=0x7f68f8136cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 --2- 172.21.15.181:0/1047839550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f81348d0 0x7f68f8136cb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f68f80648a0 tx=0x7f68f0009840 comp rx=0 tx=0).stop 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 -- 172.21.15.181:0/1047839550 shutdown_connections 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 --2- 172.21.15.181:0/1047839550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f81348d0 0x7f68f8136cb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 --2- 172.21.15.181:0/1047839550 >> v2:172.21.15.153:3300/0 conn(0x7f68f809dfa0 0x7f68f809d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 -- 172.21.15.181:0/1047839550 >> 172.21.15.181:0/1047839550 conn(0x7f68f80a3b30 msgr2=0x7f68f809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 -- 172.21.15.181:0/1047839550 shutdown_connections 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 -- 172.21.15.181:0/1047839550 wait complete. 2024-09-15T03:30:59.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 10 monclient: build_initial_monmap 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 10 monclient: monmap: 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.352+0000 7f68feb9d640 1 Processor -- start 2024-09-15T03:30:59.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 -- start start 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient: init 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 0x7f68f8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f68f8099e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f81348d0 0x7f68f8069240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f68f81348d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f8069780 0x7f68f8147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f68f8069780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80a1a90 con 0x7f68f81348d0 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f806eb80 con 0x7f68f8099e10 2024-09-15T03:30:59.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f811e200 con 0x7f68f8069780 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68feb9d640 10 monclient(hunting): authenticate will time out at 863.834622s 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 0x7f68f8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 0x7f68f8068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48140/0 (socket says 172.21.15.181:48140) 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 -- 172.21.15.181:0/542919733 learned_addr learned my addr 172.21.15.181:0/542919733 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): get_auth_request con 0x7f68f8099e10 auth_method 0 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68f7fff640 1 -- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f81348d0 msgr2=0x7f68f8069240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68f7fff640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f81348d0 0x7f68f8069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:30:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fd13a640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f8069780 0x7f68f8147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient(hunting): handle_auth_done global_id 44317 payload 824 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 -- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f8069780 msgr2=0x7f68f8147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f8069780 0x7f68f8147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 -- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f81348d0 msgr2=0x7f68f8069240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f81348d0 0x7f68f8069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 10 monclient: _finish_hunting 0 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 1 monclient: found mon.noname-b 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.353+0000 7f68fc939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.354+0000 7f68fc939640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.354+0000 7f68fc939640 1 -- 172.21.15.181:0/542919733 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f0009ae0 con 0x7f68f8099e10 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68fc939640 1 -- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 msgr2=0x7f68f8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68fc939640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 0x7f68f8068d00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c540 tx=0x7f68ec007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0074f0 con 0x7f68f8099e10 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: dump: 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec005780 con 0x7f68f8099e10 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 5 monclient: authenticate success, global_id 44317 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 10 monclient: _renew_subs 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f811f550 con 0x7f68f8099e10 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec005b40 con 0x7f68f8099e10 2024-09-15T03:30:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f4ff9640 10 monclient: dump: 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68f6ffd640 4 set_mon_vals no callback set 2024-09-15T03:30:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 10 monclient: _renew_subs 2024-09-15T03:30:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.356+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f8134630 con 0x7f68f8099e10 2024-09-15T03:30:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f68ec01b3b0 con 0x7f68f8099e10 2024-09-15T03:30:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68f4ff9640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68dc038740 0x7f68dc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:30:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f68ec04b1b0 con 0x7f68f8099e10 2024-09-15T03:30:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68feb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68feb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68feb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f804bf60 con 0x7f68f8099e10 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68f7fff640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68dc038740 0x7f68dc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.357+0000 7f68f7fff640 10 monclient: get_auth_request con 0x7f68dc038740 auth_method 0 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.360+0000 7f68f7fff640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68dc038740 0x7f68dc03abf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f68f806a2b0 tx=0x7f68e4006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.362+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f68ec018820 con 0x7f68f8099e10 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.362+0000 7f68f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:30:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.362+0000 7f68f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:30:59.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.518+0000 7f68feb9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T03:30:59.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.519+0000 7f68feb9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T03:30:59.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.519+0000 7f68feb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.519+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f68f811f790 con 0x7f68f8099e10 2024-09-15T03:30:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.819+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v59) v1 ==== 110+0+0 (secure 0 0 0) 0x7f68ec017030 con 0x7f68f8099e10 2024-09-15T03:30:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.819+0000 7f68f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T03:30:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.819+0000 7f68f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.915+0000 7f68feb9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.915+0000 7f68feb9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.915+0000 7f68feb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.915+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f68f8072640 con 0x7f68f8099e10 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.916+0000 7f68f4ff9640 1 -- 172.21.15.181:0/542919733 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v59) v1 ==== 134+0+0 (secure 0 0 0) 0x7f68ec017020 con 0x7f68f8099e10 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.916+0000 7f68f4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.916+0000 7f68f4ff9640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-15T03:30:59.917 INFO:tasks.workunit.client.0.smithi181.stderr:already renamed to 'bar' 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.917+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68dc038740 msgr2=0x7f68dc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.917+0000 7f68feb9d640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68dc038740 0x7f68dc03abf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f68f806a2b0 tx=0x7f68e4006d80 comp rx=0 tx=0).stop 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.917+0000 7f68feb9d640 10 monclient: shutdown 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.917+0000 7f68feb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.917+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 msgr2=0x7f68f8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.917+0000 7f68feb9d640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 0x7f68f8068d00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c540 tx=0x7f68ec007d30 comp rx=0 tx=0).stop 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 shutdown_connections 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68dc038740 0x7f68dc03abf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f8069780 0x7f68f8147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f81348d0 0x7f68f8069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 --2- 172.21.15.181:0/542919733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f8099e10 0x7f68f8068d00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:30:59.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 >> 172.21.15.181:0/542919733 conn(0x7f68f80a3b30 msgr2=0x7f68f80a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:30:59.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 shutdown_connections 2024-09-15T03:30:59.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:30:59.918+0000 7f68feb9d640 1 -- 172.21.15.181:0/542919733 wait complete. 2024-09-15T03:30:59.928 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls 2024-09-15T03:30:59.928 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false grep foo-asdf 2024-09-15T03:30:59.928 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:30:59.928 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo-asdf 2024-09-15T03:31:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.000+0000 7f099ea0f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.000+0000 7f099ea0f640 10 monclient: build_initial_monmap 2024-09-15T03:31:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.000+0000 7f099ea0f640 10 monclient: monmap: 2024-09-15T03:31:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:00.001145+0000 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:00.001145+0000 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:00.002 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 Processor -- start 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 -- start start 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient: init 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 0x7f099809b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): picked mon.noname-b con 0x7f09980677e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f099809c560 0x7f099809c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): picked mon.noname-a con 0x7f099809c560 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f099809ce70 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09980680e0 con 0x7f099809ce70 2024-09-15T03:31:00.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099806a050 con 0x7f099809c560 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099809dca0 con 0x7f09980677e0 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099ea0f640 10 monclient(hunting): authenticate will time out at 864.482593s 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f099cfac640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f099809ce70 legacy=0x7f0998131db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f09977fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f099809c560 msgr2=0x7f099809c930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f09977fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f099809c560 0x7f099809c930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.001+0000 7f0997fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 0x7f099809b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 0x7f099809b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:46788/0 (socket says 172.21.15.181:46788) 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 -- 172.21.15.181:0/2924280862 learned_addr learned my addr 172.21.15.181:0/2924280862 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): get_auth_request con 0x7f09980677e0 auth_method 0 2024-09-15T03:31:00.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient(hunting): handle_auth_done global_id 4913 payload 274 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 -- 172.21.15.181:0/2924280862 >> v2:172.21.15.153:3300/0 conn(0x7f099809c560 msgr2=0x7f099809c930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 --2- 172.21.15.181:0/2924280862 >> v2:172.21.15.153:3300/0 conn(0x7f099809c560 0x7f099809c930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 -- 172.21.15.181:0/2924280862 >> v1:172.21.15.153:6789/0 conn(0x7f099809ce70 legacy=0x7f0998131db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:00.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 monclient: found mon.noname-b 2024-09-15T03:31:00.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:00.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:00.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.002+0000 7f0997fff640 1 -- 172.21.15.181:0/2924280862 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099809df80 con 0x7f09980677e0 2024-09-15T03:31:00.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0997fff640 1 -- 172.21.15.181:0/2924280862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 msgr2=0x7f099809b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:00.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0997fff640 1 --2- 172.21.15.181:0/2924280862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 0x7f099809b2b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0988009910 tx=0x7f09880091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=319886f4e32717fe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 1 -- 172.21.15.181:0/2924280862 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988032440 con 0x7f09980677e0 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: dump: 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.014 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 1 -- 172.21.15.181:0/2924280862 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09880325a0 con 0x7f09980677e0 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 5 monclient: authenticate success, global_id 4913 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 1 -- 172.21.15.181:0/2924280862 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988032940 con 0x7f09980677e0 2024-09-15T03:31:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 4 set_mon_vals no callback set 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f0996ffd640 10 monclient: dump: 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.016 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 10 monclient: shutdown 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 1 -- 172.21.15.181:0/2924280862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 msgr2=0x7f099809b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.005+0000 7f099ea0f640 1 --2- 172.21.15.181:0/2924280862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 0x7f099809b2b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0988009910 tx=0x7f09880091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 -- 172.21.15.181:0/2924280862 shutdown_connections 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 --2- 172.21.15.181:0/2924280862 >> v2:172.21.15.153:3300/0 conn(0x7f099809c560 0x7f099809c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 --2- 172.21.15.181:0/2924280862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09980677e0 0x7f099809b2b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 -- 172.21.15.181:0/2924280862 >> 172.21.15.181:0/2924280862 conn(0x7f09980a16a0 msgr2=0x7f09980a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 -- 172.21.15.181:0/2924280862 shutdown_connections 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 -- 172.21.15.181:0/2924280862 wait complete. 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 10 monclient: build_initial_monmap 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 10 monclient: monmap: 2024-09-15T03:31:00.017 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.006+0000 7f099ea0f640 1 Processor -- start 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 -- start start 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient: init 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 0x7f09981410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): picked mon.noname-b con 0x7f09980677e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f099809c560 0x7f0998094790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): picked mon.noname-c con 0x7f099809c560 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f099809ce70 0x7f099808d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): picked mon.noname-a con 0x7f099809ce70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998091240 con 0x7f099809ce70 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998090c80 con 0x7f09980677e0 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09981415f0 con 0x7f099809c560 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099ea0f640 10 monclient(hunting): authenticate will time out at 864.488574s 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 0x7f09981410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099cfac640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f099809ce70 msgr2=0x7f099808d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f099cfac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f099809ce70 0x7f099808d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 0x7f09981410b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48142/0 (socket says 172.21.15.181:48142) 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 -- 172.21.15.181:0/1258854204 learned_addr learned my addr 172.21.15.181:0/1258854204 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): get_auth_request con 0x7f09980677e0 auth_method 0 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f09977fe640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f099809c560 0x7f0998094790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f09977fe640 10 monclient(hunting): get_auth_request con 0x7f099809c560 auth_method 0 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f09977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f09977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f09977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient(hunting): handle_auth_done global_id 44320 payload 824 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 -- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f099809c560 msgr2=0x7f0998094790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f099809c560 0x7f0998094790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 -- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f099809ce70 msgr2=0x7f099808d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f099809ce70 0x7f099808d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 monclient: found mon.noname-b 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.007+0000 7f0997fff640 1 -- 172.21.15.181:0/1258854204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0988009450 con 0x7f09980677e0 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f0997fff640 1 -- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 msgr2=0x7f09981410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f0997fff640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 0x7f09981410b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0988035040 tx=0x7f0988032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098803c960 con 0x7f09980677e0 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: dump: 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: _finish_auth 0 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 5 monclient: authenticate success, global_id 44320 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 10 monclient: _renew_subs 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098803cac0 con 0x7f09980677e0 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09981418f0 con 0x7f09980677e0 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098803ce80 con 0x7f09980677e0 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f097ffff640 10 monclient: dump: 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.024 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 10 monclient: _renew_subs 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099808ef60 con 0x7f09980677e0 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.010+0000 7f09967fc640 4 set_mon_vals no callback set 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0988039040 con 0x7f09980677e0 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f099ea0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f099ea0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f097ffff640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09840386f0 0x7f098403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f099ea0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09980008d0 con 0x7f09980677e0 2024-09-15T03:31:00.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f098806bd40 con 0x7f09980677e0 2024-09-15T03:31:00.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f09977fe640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09840386f0 0x7f098403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.011+0000 7f09977fe640 10 monclient: get_auth_request con 0x7f09840386f0 auth_method 0 2024-09-15T03:31:00.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.014+0000 7f09977fe640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09840386f0 0x7f098403aba0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f098c006fd0 tx=0x7f098c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.015+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0988043360 con 0x7f09980677e0 2024-09-15T03:31:00.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.015+0000 7f097ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.015+0000 7f097ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:00.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.170+0000 7f099ea0f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.170+0000 7f099ea0f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.170+0000 7f099ea0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.170+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f0998064a20 con 0x7f09980677e0 2024-09-15T03:31:00.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.171+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (secure 0 0 0) 0x7f0988040020 con 0x7f09980677e0 2024-09-15T03:31:00.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.171+0000 7f097ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.171+0000 7f097ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.265+0000 7f099ea0f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.265+0000 7f099ea0f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.265+0000 7f099ea0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.265+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f0998064a20 con 0x7f09980677e0 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.266+0000 7f097ffff640 1 -- 172.21.15.181:0/1258854204 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (secure 0 0 0) 0x7f0988037070 con 0x7f09980677e0 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.266+0000 7f097ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.266+0000 7f097ffff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:00.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09840386f0 msgr2=0x7f098403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09840386f0 0x7f098403aba0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f098c006fd0 tx=0x7f098c008040 comp rx=0 tx=0).stop 2024-09-15T03:31:00.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 10 monclient: shutdown 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 msgr2=0x7f09981410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 0x7f09981410b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0988035040 tx=0x7f0988032000 comp rx=0 tx=0).stop 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 shutdown_connections 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09840386f0 0x7f098403aba0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f099809ce70 0x7f099808d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f099809c560 0x7f0998094790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 --2- 172.21.15.181:0/1258854204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09980677e0 0x7f09981410b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 >> 172.21.15.181:0/1258854204 conn(0x7f09980a16a0 msgr2=0x7f0998097580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 shutdown_connections 2024-09-15T03:31:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.268+0000 7f099ea0f640 1 -- 172.21.15.181:0/1258854204 wait complete. 2024-09-15T03:31:00.278 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:31:00.278 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule ls 2024-09-15T03:31:00.279 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false grep bar-asdf 2024-09-15T03:31:00.279 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:31:00.279 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep bar-asdf 2024-09-15T03:31:00.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.350+0000 7fcfaae80640 10 monclient: get_monmap_and_config 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.350+0000 7fcfaae80640 10 monclient: build_initial_monmap 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.350+0000 7fcfaae80640 10 monclient: monmap: 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:00.351523+0000 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:00.351523+0000 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:00.353 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 1 Processor -- start 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 1 -- start start 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 10 monclient: init 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4131c00 0x7fcfa4131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa4131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcfa40a0b00 0x7fcfa4132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa40a0b00 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.351+0000 7fcfaae80640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcfa4068d00 0x7fcfa4069100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa4068d00 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:00.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa409d230 con 0x7fcfa4068d00 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa40a1b70 con 0x7fcfa40a0b00 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40a08c0 con 0x7fcfa4131c00 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfaae80640 10 monclient(hunting): authenticate will time out at 864.833034s 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa941d640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fcfa4068d00 msgr2=0x7fcfa4069100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa941d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcfa4068d00 0x7fcfa4069100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:00.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcfa40a0b00 0x7fcfa4132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcfa40a0b00 0x7fcfa4132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48154/0 (socket says 172.21.15.181:48154) 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 -- 172.21.15.181:0/1335459979 learned_addr learned my addr 172.21.15.181:0/1335459979 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): get_auth_request con 0x7fcfa40a0b00 auth_method 0 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa8c1c640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4131c00 0x7fcfa4131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_done global_id 44323 payload 274 2024-09-15T03:31:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 -- 172.21.15.181:0/1335459979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4131c00 msgr2=0x7fcfa4131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4131c00 0x7fcfa4131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 -- 172.21.15.181:0/1335459979 >> v2:172.21.15.153:3300/0 conn(0x7fcfa4068d00 msgr2=0x7fcfa4069100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 --2- 172.21.15.181:0/1335459979 >> v2:172.21.15.153:3300/0 conn(0x7fcfa4068d00 0x7fcfa4069100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 monclient: found mon.noname-c 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa3fff640 1 -- 172.21.15.181:0/1335459979 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa40a0200 con 0x7fcfa40a0b00 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa8c1c640 10 monclient: get_auth_request con 0x7fcfa4131c00 auth_method 0 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.352+0000 7fcfa8c1c640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4131c00 0x7fcfa4131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa3fff640 1 -- 172.21.15.181:0/1335459979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 msgr2=0x7fcfa4132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa3fff640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4132510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcfa406aba0 tx=0x7fcf900091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60437df95cfa7ff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 1 -- 172.21.15.181:0/1335459979 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90032440 con 0x7fcfa40a0b00 2024-09-15T03:31:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: dump: 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 1 -- 172.21.15.181:0/1335459979 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf900325a0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 1 -- 172.21.15.181:0/1335459979 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90032940 con 0x7fcfa40a0b00 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfa37fe640 10 monclient: dump: 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.367 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 5 monclient: authenticate success, global_id 44323 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 4 set_mon_vals no callback set 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 10 monclient: shutdown 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 -- 172.21.15.181:0/1335459979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 msgr2=0x7fcfa4132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4132510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcfa406aba0 tx=0x7fcf900091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 -- 172.21.15.181:0/1335459979 shutdown_connections 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 --2- 172.21.15.181:0/1335459979 >> v2:172.21.15.153:3300/0 conn(0x7fcfa4068d00 0x7fcfa4069100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4132510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 --2- 172.21.15.181:0/1335459979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4131c00 0x7fcfa4131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 -- 172.21.15.181:0/1335459979 >> 172.21.15.181:0/1335459979 conn(0x7fcfa4071370 msgr2=0x7fcfa40a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.355+0000 7fcfaae80640 1 -- 172.21.15.181:0/1335459979 shutdown_connections 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.356+0000 7fcfaae80640 1 -- 172.21.15.181:0/1335459979 wait complete. 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.356+0000 7fcfaae80640 10 monclient: build_initial_monmap 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.356+0000 7fcfaae80640 10 monclient: monmap: 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:00.369 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.356+0000 7fcfaae80640 1 Processor -- start 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.356+0000 7fcfaae80640 1 -- start start 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.356+0000 7fcfaae80640 10 monclient: init 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4068d00 0x7fcfa4130ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa4068d00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa40a0b00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfa4131c00 0x7fcfa41413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa4131c00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:00.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa411d330 con 0x7fcfa4131c00 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa411f0b0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa411edd0 con 0x7fcfa4068d00 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfaae80640 10 monclient(hunting): authenticate will time out at 864.838457s 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa941d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfa4131c00 msgr2=0x7fcfa41413f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa941d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfa4131c00 0x7fcfa41413f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48170/0 (socket says 172.21.15.181:48170) 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 -- 172.21.15.181:0/3004777953 learned_addr learned my addr 172.21.15.181:0/3004777953 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): get_auth_request con 0x7fcfa40a0b00 auth_method 0 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa8c1c640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4068d00 0x7fcfa4130ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa8c1c640 10 monclient(hunting): get_auth_request con 0x7fcfa4068d00 auth_method 0 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa8c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa8c1c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa8c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient(hunting): handle_auth_done global_id 44326 payload 824 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 -- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4068d00 msgr2=0x7fcfa4130ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4068d00 0x7fcfa4130ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 -- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfa4131c00 msgr2=0x7fcfa41413f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfa4131c00 0x7fcfa41413f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 monclient: found mon.noname-b 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa3fff640 1 -- 172.21.15.181:0/3004777953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009450 con 0x7fcfa40a0b00 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.357+0000 7fcfa8c1c640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4068d00 0x7fcfa4130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa3fff640 1 -- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 msgr2=0x7fcfa4140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa3fff640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4140eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf90035040 tx=0x7fcf90032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90031b10 con 0x7fcfa40a0b00 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: dump: 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 5 monclient: authenticate success, global_id 44326 2024-09-15T03:31:00.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90031cb0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9003b650 con 0x7fcfa40a0b00 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 10 monclient: _renew_subs 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa411f390 con 0x7fcfa40a0b00 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa2ffd640 4 set_mon_vals no callback set 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfa0ff9640 10 monclient: dump: 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.375 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 10 monclient: _renew_subs 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.360+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa41311e0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcf90039040 con 0x7fcfa40a0b00 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfaae80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfaae80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfaae80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfa0ff9640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcf880386f0 0x7fcf8803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa40008d0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fcf9006bc10 con 0x7fcfa40a0b00 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfa8c1c640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcf880386f0 0x7fcf8803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.361+0000 7fcfa8c1c640 10 monclient: get_auth_request con 0x7fcf880386f0 auth_method 0 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.363+0000 7fcfa8c1c640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcf880386f0 0x7fcf8803aba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ab50 tx=0x7fcf98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.365+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcf9003ba40 con 0x7fcfa40a0b00 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.365+0000 7fcfa0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.365+0000 7fcfa0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.521+0000 7fcfaae80640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.521+0000 7fcfaae80640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.521+0000 7fcfaae80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.521+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fcfa4064a20 con 0x7fcfa40a0b00 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.522+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (secure 0 0 0) 0x7fcf90037030 con 0x7fcfa40a0b00 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.522+0000 7fcfa0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.522+0000 7fcfa0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfaae80640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfaae80640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfaae80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fcfa411f1f0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfa0ff9640 1 -- 172.21.15.181:0/3004777953 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (secure 0 0 0) 0x7fcfa411f1f0 con 0x7fcfa40a0b00 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfa0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-15T03:31:00.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.615+0000 7fcfa0ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:00.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcf880386f0 msgr2=0x7fcf8803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcf880386f0 0x7fcf8803aba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ab50 tx=0x7fcf98008040 comp rx=0 tx=0).stop 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 10 monclient: shutdown 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 msgr2=0x7fcfa4140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4140eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf90035040 tx=0x7fcf90032000 comp rx=0 tx=0).stop 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 shutdown_connections 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcf880386f0 0x7fcf8803aba0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfa4131c00 0x7fcfa41413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfa40a0b00 0x7fcfa4140eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 --2- 172.21.15.181:0/3004777953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfa4068d00 0x7fcfa4130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 >> 172.21.15.181:0/3004777953 conn(0x7fcfa4071370 msgr2=0x7fcfa409ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:00.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 shutdown_connections 2024-09-15T03:31:00.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.617+0000 7fcfaae80640 1 -- 172.21.15.181:0/3004777953 wait complete. 2024-09-15T03:31:00.627 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:31:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm foo 2024-09-15T03:31:00.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.701+0000 7fe23099d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:00.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.701+0000 7fe23099d640 10 monclient: build_initial_monmap 2024-09-15T03:31:00.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.701+0000 7fe23099d640 10 monclient: monmap: 2024-09-15T03:31:00.707 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:00.707 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.707 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:00.702680+0000 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:00.702680+0000 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 1 Processor -- start 2024-09-15T03:31:00.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 1 -- start start 2024-09-15T03:31:00.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 10 monclient: init 2024-09-15T03:31:00.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:00.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:31:00.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe22c098c70 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:00.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.702+0000 7fe23099d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe22c099c70 0x7fe22c136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 10 monclient(hunting): picked mon.noname-a con 0x7fe22c099c70 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:00.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 0x7fe22c1395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 10 monclient(hunting): picked mon.noname-b con 0x7fe22c137200 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:00.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c09f680 con 0x7fe22c098c70 2024-09-15T03:31:00.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c09f420 con 0x7fe22c099c70 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c068a20 con 0x7fe22c137200 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe23099d640 10 monclient(hunting): authenticate will time out at 865.184040s 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22a59c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fe22c098c70 legacy=0x7fe22c099040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe221d9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fe22c099c70 msgr2=0x7fe22c136cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe221d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe22c099c70 0x7fe22c136cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 0x7fe22c1395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 0x7fe22c1395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:46828/0 (socket says 172.21.15.181:46828) 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 -- 172.21.15.181:0/2860971070 learned_addr learned my addr 172.21.15.181:0/2860971070 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): get_auth_request con 0x7fe22c137200 auth_method 0 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_done global_id 4922 payload 274 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 -- 172.21.15.181:0/2860971070 >> v2:172.21.15.153:3300/0 conn(0x7fe22c099c70 msgr2=0x7fe22c136cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 --2- 172.21.15.181:0/2860971070 >> v2:172.21.15.153:3300/0 conn(0x7fe22c099c70 0x7fe22c136cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 -- 172.21.15.181:0/2860971070 >> v1:172.21.15.153:6789/0 conn(0x7fe22c098c70 legacy=0x7fe22c099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:31:00.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 monclient: found mon.noname-b 2024-09-15T03:31:00.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:00.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:00.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.703+0000 7fe22ad9d640 1 -- 172.21.15.181:0/2860971070 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c096a30 con 0x7fe22c137200 2024-09-15T03:31:00.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.706+0000 7fe22ad9d640 1 -- 172.21.15.181:0/2860971070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 msgr2=0x7fe22c1395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:00.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.706+0000 7fe22ad9d640 1 --2- 172.21.15.181:0/2860971070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 0x7fe22c1395e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe224009f50 tx=0x7fe224009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2355fb6c368398c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 1 -- 172.21.15.181:0/2860971070 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22402aea0 con 0x7fe22c137200 2024-09-15T03:31:00.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: dump: 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: _finish_auth 0 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 1 -- 172.21.15.181:0/2860971070 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22402d050 con 0x7fe22c137200 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe23099d640 5 monclient: authenticate success, global_id 4922 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe23099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe23099d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 1 -- 172.21.15.181:0/2860971070 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224007840 con 0x7fe22c137200 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe23099d640 4 set_mon_vals no callback set 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe229d9b640 10 monclient: dump: 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.707+0000 7fe23099d640 10 monclient: shutdown 2024-09-15T03:31:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 -- 172.21.15.181:0/2860971070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 msgr2=0x7fe22c1395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 --2- 172.21.15.181:0/2860971070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 0x7fe22c1395e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe224009f50 tx=0x7fe224009840 comp rx=0 tx=0).stop 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 -- 172.21.15.181:0/2860971070 shutdown_connections 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 --2- 172.21.15.181:0/2860971070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c137200 0x7fe22c1395e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 --2- 172.21.15.181:0/2860971070 >> v2:172.21.15.153:3300/0 conn(0x7fe22c099c70 0x7fe22c136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 -- 172.21.15.181:0/2860971070 >> 172.21.15.181:0/2860971070 conn(0x7fe22c070e90 msgr2=0x7fe22c0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 -- 172.21.15.181:0/2860971070 shutdown_connections 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 -- 172.21.15.181:0/2860971070 wait complete. 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 10 monclient: build_initial_monmap 2024-09-15T03:31:00.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 10 monclient: monmap: 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.708+0000 7fe23099d640 1 Processor -- start 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 -- start start 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient: init 2024-09-15T03:31:00.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:00.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:00.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c098c70 0x7fe22c11ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): picked mon.noname-a con 0x7fe22c098c70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:00.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c099c70 0x7fe22c11ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): picked mon.noname-c con 0x7fe22c099c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:00.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 0x7fe22c11f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): picked mon.noname-b con 0x7fe22c137200 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c136520 con 0x7fe22c098c70 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c120980 con 0x7fe22c137200 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c06eba0 con 0x7fe22c099c70 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe23099d640 10 monclient(hunting): authenticate will time out at 865.190460s 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22a59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c098c70 msgr2=0x7fe22c11ea50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 0x7fe22c11f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c098c70 0x7fe22c11ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 0x7fe22c11f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48180/0 (socket says 172.21.15.181:48180) 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 -- 172.21.15.181:0/3911204423 learned_addr learned my addr 172.21.15.181:0/3911204423 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): get_auth_request con 0x7fe22c137200 auth_method 0 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe221d9b640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c099c70 0x7fe22c11ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe221d9b640 10 monclient(hunting): get_auth_request con 0x7fe22c099c70 auth_method 0 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe221d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:00.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe221d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe221d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient(hunting): handle_auth_done global_id 44329 payload 824 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 -- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c099c70 msgr2=0x7fe22c11ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c099c70 0x7fe22c11ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 -- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c098c70 msgr2=0x7fe22c11ea50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c098c70 0x7fe22c11ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 monclient: found mon.noname-b 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.709+0000 7fe22ad9d640 1 -- 172.21.15.181:0/3911204423 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe224009ae0 con 0x7fe22c137200 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe22ad9d640 1 -- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 msgr2=0x7fe22c11f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:00.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe22ad9d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 0x7fe22c11f4d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe224009ab0 tx=0x7fe224036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224031820 con 0x7fe22c137200 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: dump: 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.729 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe224031980 con 0x7fe22c137200 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 5 monclient: authenticate success, global_id 44329 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 10 monclient: _renew_subs 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224031d40 con 0x7fe22c137200 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c0a1bd0 con 0x7fe22c137200 2024-09-15T03:31:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe222ffd640 10 monclient: dump: 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe22959a640 4 set_mon_vals no callback set 2024-09-15T03:31:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 10 monclient: _renew_subs 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.712+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c121ec0 con 0x7fe22c137200 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe224038070 con 0x7fe22c137200 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe23099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe23099d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe23099d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c09a070 con 0x7fe22c137200 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe222ffd640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe20c038720 0x7fe20c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fe22406cc30 con 0x7fe22c137200 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe22a59c640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe20c038720 0x7fe20c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.713+0000 7fe22a59c640 10 monclient: get_auth_request con 0x7fe20c038720 auth_method 0 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.715+0000 7fe22a59c640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe20c038720 0x7fe20c03abd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0038f0 tx=0x7fe21c006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:00.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.717+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe22403b5b0 con 0x7fe22c137200 2024-09-15T03:31:00.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.717+0000 7fe222ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:00.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.717+0000 7fe222ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:00.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.878+0000 7fe23099d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:00.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.878+0000 7fe23099d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:00.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.878+0000 7fe23099d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:00.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:00.878+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fe22c0008d0 con 0x7fe22c137200 2024-09-15T03:31:01.118 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:31:01.119 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:31:01.119 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:31:01.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:31:01.122 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:31:01.122 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:31:01.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:01.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.226+0000 7f567dea8640 10 monclient: get_monmap_and_config 2024-09-15T03:31:01.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.226+0000 7f567dea8640 10 monclient: build_initial_monmap 2024-09-15T03:31:01.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.226+0000 7f567dea8640 10 monclient: monmap: 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:01.227556+0000 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:01.227556+0000 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:01.228 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:01.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.227+0000 7f567dea8640 1 Processor -- start 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.227+0000 7f567dea8640 1 -- start start 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.227+0000 7f567dea8640 10 monclient: init 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f56781318f0 0x7f5678131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): picked mon.noname-c con 0x7f56781318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 0x7f567809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): picked mon.noname-a con 0x7f567809ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5678068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567809fbf0 con 0x7f5678068d90 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5678131d50 con 0x7f567809ea50 2024-09-15T03:31:01.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567809fe30 con 0x7f56781318f0 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f567dea8640 10 monclient(hunting): authenticate will time out at 858.867199s 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 0x7f567809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5677fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5678068d90 0x7f56780691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45052/0 (socket says 172.21.15.153:45052) 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5677fff640 1 -- 172.21.15.153:0/902887295 learned_addr learned my addr 172.21.15.153:0/902887295 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 1 --2- 172.21.15.153:0/902887295 >> v2:172.21.15.181:3300/0 conn(0x7f56781318f0 0x7f5678131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 10 monclient(hunting): get_auth_request con 0x7f567809ea50 auth_method 0 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:01.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): get_auth_request con 0x7f56781318f0 auth_method 0 2024-09-15T03:31:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 1 -- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 msgr2=0x7f567809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 1 -- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 msgr2=0x7f567809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 1 --2- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 0x7f567809ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f5676ffd640 1 --2- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 0x7f567809ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient(hunting): handle_auth_done global_id 44332 payload 274 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 1 -- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 msgr2=0x7f567809ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 1 --2- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 0x7f567809ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 1 -- 172.21.15.153:0/902887295 >> v1:172.21.15.153:6789/0 conn(0x7f5678068d90 legacy=0x7f56780691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:01.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:01.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:01.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:01.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.228+0000 7f56777fe640 1 -- 172.21.15.153:0/902887295 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56780a1210 con 0x7f56781318f0 2024-09-15T03:31:01.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56777fe640 1 -- 172.21.15.153:0/902887295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56781318f0 msgr2=0x7f5678131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:01.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56777fe640 1 --2- 172.21.15.153:0/902887295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56781318f0 0x7f5678131f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5678076600 tx=0x7f56680090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a563ecd2f11524d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56767fc640 1 -- 172.21.15.153:0/902887295 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566802ae00 con 0x7f56781318f0 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56767fc640 10 monclient: dump: 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:01.242 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56767fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f56767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f567dea8640 5 monclient: authenticate success, global_id 44332 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.231+0000 7f567dea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f56767fc640 1 -- 172.21.15.153:0/902887295 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668032440 con 0x7f56781318f0 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f56767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f56767fc640 1 -- 172.21.15.153:0/902887295 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680327e0 con 0x7f56781318f0 2024-09-15T03:31:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 4 set_mon_vals no callback set 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f56767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f56767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f56767fc640 10 monclient: dump: 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:01.244 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 10 monclient: shutdown 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 -- 172.21.15.153:0/902887295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56781318f0 msgr2=0x7f5678131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 --2- 172.21.15.153:0/902887295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56781318f0 0x7f5678131f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5678076600 tx=0x7f56680090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 -- 172.21.15.153:0/902887295 shutdown_connections 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 --2- 172.21.15.153:0/902887295 >> v2:172.21.15.153:3300/0 conn(0x7f567809ea50 0x7f567809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 --2- 172.21.15.153:0/902887295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56781318f0 0x7f5678131f30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 -- 172.21.15.153:0/902887295 >> 172.21.15.153:0/902887295 conn(0x7f5678067110 msgr2=0x7f5678067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 -- 172.21.15.153:0/902887295 shutdown_connections 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.232+0000 7f567dea8640 1 -- 172.21.15.153:0/902887295 wait complete. 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.233+0000 7f567dea8640 10 monclient: build_initial_monmap 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.233+0000 7f567dea8640 10 monclient: monmap: 2024-09-15T03:31:01.245 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.233+0000 7f567dea8640 1 Processor -- start 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.233+0000 7f567dea8640 1 -- start start 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.233+0000 7f567dea8640 10 monclient: init 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:01.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 0x7f567809aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): picked mon.noname-c con 0x7f5678068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f567809ea50 0x7f5678130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): picked mon.noname-b con 0x7f567809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): picked mon.noname-a con 0x7f56781318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567811dd90 con 0x7f56781318f0 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567809bf10 con 0x7f567809ea50 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567811c7e0 con 0x7f5678068d90 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f567dea8640 10 monclient(hunting): authenticate will time out at 858.873259s 2024-09-15T03:31:01.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 0x7f567809aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33368/0 (socket says 172.21.15.153:33368) 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 0x7f567809aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46878/0 (socket says 172.21.15.153:46878) 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 -- 172.21.15.153:0/2003744779 learned_addr learned my addr 172.21.15.153:0/2003744779 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 10 monclient(hunting): get_auth_request con 0x7f56781318f0 auth_method 0 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): get_auth_request con 0x7f5678068d90 auth_method 0 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 msgr2=0x7f567811b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 msgr2=0x7f567811b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f567809ea50 0x7f5678130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5677fff640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): get_auth_request con 0x7f567809ea50 auth_method 0 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:01.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f5676ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient(hunting): handle_auth_done global_id 4928 payload 824 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f567809ea50 msgr2=0x7f5678130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f567809ea50 0x7f5678130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 msgr2=0x7f567811b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:01.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.234+0000 7f56777fe640 1 -- 172.21.15.153:0/2003744779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668009390 con 0x7f5678068d90 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.236+0000 7f56777fe640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 msgr2=0x7f567809aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.236+0000 7f56777fe640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 0x7f567809aa20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5668031000 tx=0x7f5668032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668031d40 con 0x7f5678068d90 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: dump: 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.251 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 5 monclient: authenticate success, global_id 4928 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 10 monclient: _renew_subs 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566803d440 con 0x7f5678068d90 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56780a43e0 con 0x7f5678068d90 2024-09-15T03:31:01.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566803d800 con 0x7f5678068d90 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f565f7fe640 10 monclient: dump: 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:01.253 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 10 monclient: _renew_subs 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567809b5f0 con 0x7f5678068d90 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.237+0000 7f5675ffb640 4 set_mon_vals no callback set 2024-09-15T03:31:01.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5668039040 con 0x7f5678068d90 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f567dea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f567dea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f567dea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567804bf60 con 0x7f5678068d90 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f565f7fe640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5664038740 0x7f566403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f566806b9e0 con 0x7f5678068d90 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f5676ffd640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5664038740 0x7f566403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.238+0000 7f5676ffd640 10 monclient: get_auth_request con 0x7f5664038740 auth_method 0 2024-09-15T03:31:01.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.240+0000 7f5676ffd640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5664038740 0x7f566403abf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f567809b290 tx=0x7f566c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:01.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.242+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5668043530 con 0x7f5678068d90 2024-09-15T03:31:01.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.242+0000 7f565f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:01.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.242+0000 7f565f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:01.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f567dea8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:01.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f567dea8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:01.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f567dea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:01.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f567811da00 con 0x7f5678068d90 2024-09-15T03:31:01.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f565f7fe640 1 -- 172.21.15.153:0/2003744779 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f5668037020 con 0x7f5678068d90 2024-09-15T03:31:01.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f565f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:01.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.458+0000 7f565f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:01.460 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:01.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5664038740 msgr2=0x7f566403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5664038740 0x7f566403abf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f567809b290 tx=0x7f566c008040 comp rx=0 tx=0).stop 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 10 monclient: shutdown 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 msgr2=0x7f567809aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 0x7f567809aa20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5668031000 tx=0x7f5668032000 comp rx=0 tx=0).stop 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 shutdown_connections 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5664038740 0x7f566403abf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56781318f0 0x7f567811b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f567809ea50 0x7f5678130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 --2- 172.21.15.153:0/2003744779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5678068d90 0x7f567809aa20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 >> 172.21.15.153:0/2003744779 conn(0x7f5678067110 msgr2=0x7f56780a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:01.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 shutdown_connections 2024-09-15T03:31:01.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:01.460+0000 7f567dea8640 1 -- 172.21.15.153:0/2003744779 wait complete. 2024-09-15T03:31:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.825+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v60) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe22404a070 con 0x7fe22c137200 2024-09-15T03:31:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.825+0000 7fe222ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.825+0000 7fe222ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.925+0000 7fe23099d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.925+0000 7fe23099d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.925+0000 7fe23099d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.925+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fe22c0a1550 con 0x7fe22c137200 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.926+0000 7fe222ffd640 1 -- 172.21.15.181:0/3911204423 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7fe224040030 con 0x7fe22c137200 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.926+0000 7fe222ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.926+0000 7fe222ffd640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-15T03:31:01.927 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo does not exist 2024-09-15T03:31:01.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe20c038720 msgr2=0x7fe20c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:01.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe20c038720 0x7fe20c03abd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0038f0 tx=0x7fe21c006d80 comp rx=0 tx=0).stop 2024-09-15T03:31:01.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 10 monclient: shutdown 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 msgr2=0x7fe22c11f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 0x7fe22c11f4d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe224009ab0 tx=0x7fe224036000 comp rx=0 tx=0).stop 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 shutdown_connections 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe20c038720 0x7fe20c03abd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe22c137200 0x7fe22c11f4d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe22c099c70 0x7fe22c11ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 --2- 172.21.15.181:0/3911204423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe22c098c70 0x7fe22c11ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 >> 172.21.15.181:0/3911204423 conn(0x7fe22c070e90 msgr2=0x7fe22c097360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 shutdown_connections 2024-09-15T03:31:01.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:01.928+0000 7fe23099d640 1 -- 172.21.15.181:0/3911204423 wait complete. 2024-09-15T03:31:01.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm foo 2024-09-15T03:31:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.012+0000 7f185979d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.012+0000 7f185979d640 10 monclient: build_initial_monmap 2024-09-15T03:31:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.012+0000 7f185979d640 10 monclient: monmap: 2024-09-15T03:31:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:02.013 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:02.012922+0000 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:02.012922+0000 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:02.014 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.012+0000 7f185979d640 1 Processor -- start 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.012+0000 7f185979d640 1 -- start start 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.012+0000 7f185979d640 10 monclient: init 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f18540a0ad0 0x7f18540691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): picked mon.noname-a con 0x7f18540a0ad0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f185409ee60 0x7f185409f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): picked mon.noname-c con 0x7f185409ee60 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1854131ad0 0x7f1854131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): picked mon.noname-b con 0x7f1854131ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185409f7a0 con 0x7f18540a0ad0 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185406a5b0 con 0x7f185409ee60 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18540a0460 con 0x7f1854131ad0 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f185979d640 10 monclient(hunting): authenticate will time out at 866.494203s 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f185409ee60 0x7f185409f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f185409ee60 0x7f185409f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48196/0 (socket says 172.21.15.181:48196) 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 -- 172.21.15.181:0/425590341 learned_addr learned my addr 172.21.15.181:0/425590341 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): get_auth_request con 0x7f185409ee60 auth_method 0 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f1852ffd640 1 --2- 172.21.15.181:0/425590341 >> v2:172.21.15.153:3300/0 conn(0x7f18540a0ad0 0x7f18540691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f1852ffd640 10 monclient(hunting): get_auth_request con 0x7f18540a0ad0 auth_method 0 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f1852ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f1852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f1852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18537fe640 1 --2- 172.21.15.181:0/425590341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1854131ad0 0x7f1854131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18537fe640 10 monclient(hunting): get_auth_request con 0x7f1854131ad0 auth_method 0 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient(hunting): handle_auth_done global_id 44338 payload 274 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 -- 172.21.15.181:0/425590341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1854131ad0 msgr2=0x7f1854131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 --2- 172.21.15.181:0/425590341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1854131ad0 0x7f1854131f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 -- 172.21.15.181:0/425590341 >> v2:172.21.15.153:3300/0 conn(0x7f18540a0ad0 msgr2=0x7f18540691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 --2- 172.21.15.181:0/425590341 >> v2:172.21.15.153:3300/0 conn(0x7f18540a0ad0 0x7f18540691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 monclient: found mon.noname-c 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.013+0000 7f18527fc640 1 -- 172.21.15.181:0/425590341 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18540a0ea0 con 0x7f185409ee60 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f18527fc640 1 -- 172.21.15.181:0/425590341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185409ee60 msgr2=0x7f185409f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f18527fc640 1 --2- 172.21.15.181:0/425590341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185409ee60 0x7f185409f260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f18540a0830 tx=0x7f1848009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef681cf67aee4a76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 1 -- 172.21.15.181:0/425590341 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848007610 con 0x7f185409ee60 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: dump: 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 1 -- 172.21.15.181:0/425590341 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184802d050 con 0x7f185409ee60 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f185979d640 5 monclient: authenticate success, global_id 44338 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f185979d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f185979d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 1 -- 172.21.15.181:0/425590341 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18480079a0 con 0x7f185409ee60 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f185979d640 4 set_mon_vals no callback set 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.016+0000 7f1851ffb640 10 monclient: dump: 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.029 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 10 monclient: shutdown 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 -- 172.21.15.181:0/425590341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185409ee60 msgr2=0x7f185409f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 --2- 172.21.15.181:0/425590341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185409ee60 0x7f185409f260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f18540a0830 tx=0x7f1848009840 comp rx=0 tx=0).stop 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 -- 172.21.15.181:0/425590341 shutdown_connections 2024-09-15T03:31:02.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 --2- 172.21.15.181:0/425590341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1854131ad0 0x7f1854131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 --2- 172.21.15.181:0/425590341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185409ee60 0x7f185409f260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 --2- 172.21.15.181:0/425590341 >> v2:172.21.15.153:3300/0 conn(0x7f18540a0ad0 0x7f18540691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 -- 172.21.15.181:0/425590341 >> 172.21.15.181:0/425590341 conn(0x7f1854071370 msgr2=0x7f18540a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 -- 172.21.15.181:0/425590341 shutdown_connections 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 1 -- 172.21.15.181:0/425590341 wait complete. 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 10 monclient: build_initial_monmap 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.017+0000 7f185979d640 10 monclient: monmap: 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:02.031 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 Processor -- start 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 -- start start 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient: init 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f185409ee60 0x7f1854135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): picked mon.noname-c con 0x7f185409ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 0x7f1854130b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): picked mon.noname-b con 0x7f18540a0ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:02.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 0x7f185411b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): picked mon.noname-a con 0x7f1854131ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540a1210 con 0x7f1854131ad0 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185411dd20 con 0x7f18540a0ad0 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1854141c30 con 0x7f185409ee60 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f185979d640 10 monclient(hunting): authenticate will time out at 866.499536s 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 0x7f1854130b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 0x7f1854130b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48208/0 (socket says 172.21.15.181:48208) 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 -- 172.21.15.181:0/2530462844 learned_addr learned my addr 172.21.15.181:0/2530462844 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): get_auth_request con 0x7f18540a0ad0 auth_method 0 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f185409ee60 0x7f1854135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 0x7f185411b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 10 monclient(hunting): get_auth_request con 0x7f1854131ad0 auth_method 0 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): get_auth_request con 0x7f185409ee60 auth_method 0 2024-09-15T03:31:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 msgr2=0x7f185411b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 msgr2=0x7f185411b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 0x7f185411b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18537fe640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 0x7f185411b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f1852ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient(hunting): handle_auth_done global_id 44341 payload 824 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f185409ee60 msgr2=0x7f1854135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f185409ee60 0x7f1854135030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 msgr2=0x7f185411b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 0x7f185411b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 monclient: found mon.noname-b 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.018+0000 7f18527fc640 1 -- 172.21.15.181:0/2530462844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848009ae0 con 0x7f18540a0ad0 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f18527fc640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 msgr2=0x7f1854130b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f18527fc640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 0x7f1854130b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1848009f50 tx=0x7f1848036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848032930 con 0x7f18540a0ad0 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: dump: 2024-09-15T03:31:02.036 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: _finish_auth 0 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848032ab0 con 0x7f18540a0ad0 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 5 monclient: authenticate success, global_id 44341 2024-09-15T03:31:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 10 monclient: _renew_subs 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185411ec50 con 0x7f18540a0ad0 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848032e70 con 0x7f18540a0ad0 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f183affd640 10 monclient: dump: 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 10 monclient: _renew_subs 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1854131060 con 0x7f18540a0ad0 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.021+0000 7f18517fa640 4 set_mon_vals no callback set 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1848038050 con 0x7f18540a0ad0 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f185979d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f185979d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f185979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185411f3b0 con 0x7f18540a0ad0 2024-09-15T03:31:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f183affd640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1840038740 0x7f184003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5086+0+0 (secure 0 0 0) 0x7f184806c9d0 con 0x7f18540a0ad0 2024-09-15T03:31:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f1852ffd640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1840038740 0x7f184003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.022+0000 7f1852ffd640 10 monclient: get_auth_request con 0x7f1840038740 auth_method 0 2024-09-15T03:31:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.024+0000 7f1852ffd640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1840038740 0x7f184003abf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f184c007ba0 tx=0x7f184c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.026+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f184803b620 con 0x7f18540a0ad0 2024-09-15T03:31:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.026+0000 7f183affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.026+0000 7f183affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:02.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.184+0000 7f185979d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.184+0000 7f185979d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.184+0000 7f185979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.184+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f185411f040 con 0x7f18540a0ad0 2024-09-15T03:31:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.185+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f184803f030 con 0x7f18540a0ad0 2024-09-15T03:31:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.185+0000 7f183affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.185+0000 7f183affd640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.279+0000 7f185979d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.279+0000 7f185979d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.279+0000 7f185979d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.279+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f18540008d0 con 0x7f18540a0ad0 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.280+0000 7f183affd640 1 -- 172.21.15.181:0/2530462844 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f184803f020 con 0x7f18540a0ad0 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.280+0000 7f183affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.280+0000 7f183affd640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-15T03:31:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo does not exist 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1840038740 msgr2=0x7f184003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1840038740 0x7f184003abf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f184c007ba0 tx=0x7f184c008040 comp rx=0 tx=0).stop 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 10 monclient: shutdown 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 msgr2=0x7f1854130b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 0x7f1854130b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1848009f50 tx=0x7f1848036000 comp rx=0 tx=0).stop 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 shutdown_connections 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1840038740 0x7f184003abf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1854131ad0 0x7f185411b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18540a0ad0 0x7f1854130b20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 --2- 172.21.15.181:0/2530462844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f185409ee60 0x7f1854135030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.281+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 >> 172.21.15.181:0/2530462844 conn(0x7f1854071370 msgr2=0x7f185409e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:02.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.282+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 shutdown_connections 2024-09-15T03:31:02.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.282+0000 7f185979d640 1 -- 172.21.15.181:0/2530462844 wait complete. 2024-09-15T03:31:02.292 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm bar 2024-09-15T03:31:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.365+0000 7fe92c39d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.365+0000 7fe92c39d640 10 monclient: build_initial_monmap 2024-09-15T03:31:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.365+0000 7fe92c39d640 10 monclient: monmap: 2024-09-15T03:31:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:02.366324+0000 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:02.366324+0000 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:02.367 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 Processor -- start 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 -- start start 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient: init 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe92409ec50 0x7fe924131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe92409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9241321a0 0x7fe924068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe9241321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe92409f020 0x7fe924069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe92409f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe924067420 con 0x7fe92409ec50 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92409fdc0 con 0x7fe92409f020 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240a0510 con 0x7fe9241321a0 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92c39d640 10 monclient(hunting): authenticate will time out at 866.847866s 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.366+0000 7fe92a93a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe92409f020 0x7fe924069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe92409f020 0x7fe924069240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48222/0 (socket says 172.21.15.181:48222) 2024-09-15T03:31:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 -- 172.21.15.181:0/2798572480 learned_addr learned my addr 172.21.15.181:0/2798572480 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): get_auth_request con 0x7fe92409f020 auth_method 0 2024-09-15T03:31:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a139640 1 --2- 172.21.15.181:0/2798572480 >> v2:172.21.15.153:3300/0 conn(0x7fe92409ec50 0x7fe924131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe929938640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9241321a0 0x7fe924068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a139640 10 monclient(hunting): get_auth_request con 0x7fe92409ec50 auth_method 0 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient(hunting): handle_auth_done global_id 44344 payload 274 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 -- 172.21.15.181:0/2798572480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9241321a0 msgr2=0x7fe924068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9241321a0 0x7fe924068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 -- 172.21.15.181:0/2798572480 >> v2:172.21.15.153:3300/0 conn(0x7fe92409ec50 msgr2=0x7fe924131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:02.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 --2- 172.21.15.181:0/2798572480 >> v2:172.21.15.153:3300/0 conn(0x7fe92409ec50 0x7fe924131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient: _finish_hunting 0 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 monclient: found mon.noname-c 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe92a93a640 1 -- 172.21.15.181:0/2798572480 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240a0200 con 0x7fe92409f020 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe929938640 10 monclient: get_auth_request con 0x7fe9241321a0 auth_method 0 2024-09-15T03:31:02.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.367+0000 7fe929938640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9241321a0 0x7fe924068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:02.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92a93a640 1 -- 172.21.15.181:0/2798572480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe92409f020 msgr2=0x7fe924069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92a93a640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe92409f020 0x7fe924069240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe9240a2b00 tx=0x7fe91c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b489d026b56fe0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 1 -- 172.21.15.181:0/2798572480 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c007610 con 0x7fe92409f020 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: dump: 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.380 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: _finish_auth 0 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 1 -- 172.21.15.181:0/2798572480 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c02d050 con 0x7fe92409f020 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 5 monclient: authenticate success, global_id 44344 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:02.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 1 -- 172.21.15.181:0/2798572480 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0079e0 con 0x7fe92409f020 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 4 set_mon_vals no callback set 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe929137640 10 monclient: dump: 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 10 monclient: shutdown 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 -- 172.21.15.181:0/2798572480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe92409f020 msgr2=0x7fe924069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe92409f020 0x7fe924069240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe9240a2b00 tx=0x7fe91c009840 comp rx=0 tx=0).stop 2024-09-15T03:31:02.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 -- 172.21.15.181:0/2798572480 shutdown_connections 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe92409f020 0x7fe924069240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2798572480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9241321a0 0x7fe924068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2798572480 >> v2:172.21.15.153:3300/0 conn(0x7fe92409ec50 0x7fe924131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 -- 172.21.15.181:0/2798572480 >> 172.21.15.181:0/2798572480 conn(0x7fe9240a5d40 msgr2=0x7fe9240a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 -- 172.21.15.181:0/2798572480 shutdown_connections 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.370+0000 7fe92c39d640 1 -- 172.21.15.181:0/2798572480 wait complete. 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.371+0000 7fe92c39d640 10 monclient: build_initial_monmap 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.371+0000 7fe92c39d640 10 monclient: monmap: 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.371+0000 7fe92c39d640 1 Processor -- start 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.371+0000 7fe92c39d640 1 -- start start 2024-09-15T03:31:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.371+0000 7fe92c39d640 10 monclient: init 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe92409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe92409f020 0x7fe924130cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe92409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 0x7fe92411d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe9241321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe924120e10 con 0x7fe92409ec50 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92409e010 con 0x7fe9241321a0 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe924141c50 con 0x7fe92409f020 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92c39d640 10 monclient(hunting): authenticate will time out at 866.853188s 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 0x7fe92411d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 0x7fe92411d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48238/0 (socket says 172.21.15.181:48238) 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:40472/0 (socket says 172.21.15.181:40472) 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe92409f020 0x7fe924130cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 -- 172.21.15.181:0/2988675779 learned_addr learned my addr 172.21.15.181:0/2988675779 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): get_auth_request con 0x7fe9241321a0 auth_method 0 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 10 monclient(hunting): get_auth_request con 0x7fe92409ec50 auth_method 0 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): get_auth_request con 0x7fe92409f020 auth_method 0 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 msgr2=0x7fe924140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 msgr2=0x7fe924140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a139640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe929938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient(hunting): handle_auth_done global_id 44347 payload 824 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe92409f020 msgr2=0x7fe924130cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe92409f020 0x7fe924130cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 msgr2=0x7fe924140d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient: _finish_hunting 0 2024-09-15T03:31:02.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 monclient: found mon.noname-b 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.372+0000 7fe92a93a640 1 -- 172.21.15.181:0/2988675779 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c009ae0 con 0x7fe9241321a0 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe92a93a640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 msgr2=0x7fe92411d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe92a93a640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 0x7fe92411d7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009f50 tx=0x7fe91c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c032cc0 con 0x7fe9241321a0 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 10 monclient: dump: 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.390 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c032e20 con 0x7fe9241321a0 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe92c39d640 5 monclient: authenticate success, global_id 44347 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c03b660 con 0x7fe9241321a0 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe92c39d640 10 monclient: _renew_subs 2024-09-15T03:31:02.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe92c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92409c000 con 0x7fe9241321a0 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe91a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe91a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe91a7fc640 10 monclient: dump: 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.392 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.374+0000 7fe928936640 4 set_mon_vals no callback set 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe92c39d640 10 monclient: _renew_subs 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe92c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92411fa40 con 0x7fe9241321a0 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.375+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe91c038050 con 0x7fe9241321a0 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe92c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe92c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe92c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe924121280 con 0x7fe9241321a0 2024-09-15T03:31:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe91a7fc640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe908038740 0x7fe90803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5086+0+0 (secure 0 0 0) 0x7fe91c06c980 con 0x7fe9241321a0 2024-09-15T03:31:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe92a139640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe908038740 0x7fe90803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.376+0000 7fe92a139640 10 monclient: get_auth_request con 0x7fe908038740 auth_method 0 2024-09-15T03:31:02.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.378+0000 7fe92a139640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe908038740 0x7fe90803abf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe920006fd0 tx=0x7fe920008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.380+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe91c03ba50 con 0x7fe9241321a0 2024-09-15T03:31:02.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.380+0000 7fe91a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.380+0000 7fe91a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:02.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.539+0000 7fe92c39d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T03:31:02.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.539+0000 7fe92c39d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T03:31:02.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.539+0000 7fe92c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.539+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7fe92411f3a0 con 0x7fe9241321a0 2024-09-15T03:31:02.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.600+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v61) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe91c03f030 con 0x7fe9241321a0 2024-09-15T03:31:02.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.600+0000 7fe91a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T03:31:02.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.600+0000 7fe91a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.696+0000 7fe92c39d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.696+0000 7fe92c39d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.696+0000 7fe92c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.696+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7fe9240008d0 con 0x7fe9241321a0 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.697+0000 7fe91a7fc640 1 -- 172.21.15.181:0/2988675779 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v61) v1 ==== 103+0+0 (secure 0 0 0) 0x7fe91c03f030 con 0x7fe9241321a0 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.697+0000 7fe91a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.697+0000 7fe91a7fc640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-15T03:31:02.698 INFO:tasks.workunit.client.0.smithi181.stderr:rule bar does not exist 2024-09-15T03:31:02.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe908038740 msgr2=0x7fe90803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe908038740 0x7fe90803abf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe920006fd0 tx=0x7fe920008040 comp rx=0 tx=0).stop 2024-09-15T03:31:02.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 10 monclient: shutdown 2024-09-15T03:31:02.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:02.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 msgr2=0x7fe92411d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 0x7fe92411d7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009f50 tx=0x7fe91c036000 comp rx=0 tx=0).stop 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 shutdown_connections 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe908038740 0x7fe90803abf0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9241321a0 0x7fe92411d7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe92409f020 0x7fe924130cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 --2- 172.21.15.181:0/2988675779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92409ec50 0x7fe924140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 >> 172.21.15.181:0/2988675779 conn(0x7fe9240a5d40 msgr2=0x7fe9240a3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 shutdown_connections 2024-09-15T03:31:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.699+0000 7fe92c39d640 1 -- 172.21.15.181:0/2988675779 wait complete. 2024-09-15T03:31:02.709 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create pinning_pool 1 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.781+0000 7f1f3c60f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.782+0000 7f1f3c60f640 10 monclient: build_initial_monmap 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.782+0000 7f1f3c60f640 10 monclient: monmap: 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:02.782801+0000 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:02.782801+0000 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:02.783 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.784 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:02.784 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:02.784 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:02.784 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:02.784 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:02.784 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.782+0000 7f1f3c60f640 1 Processor -- start 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.782+0000 7f1f3c60f640 1 -- start start 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.782+0000 7f1f3c60f640 10 monclient: init 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 0x7f1f3412e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f3409e560 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f3409aeb0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f34136dd0 0x7f1f341391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f34136dd0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f34064c50 con 0x7f1f3409aeb0 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f34066090 con 0x7f1f34136dd0 2024-09-15T03:31:02.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3412f0c0 con 0x7f1f3409e560 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3c60f640 10 monclient(hunting): authenticate will time out at 867.264289s 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f34136dd0 0x7f1f341391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f34136dd0 0x7f1f341391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48244/0 (socket says 172.21.15.181:48244) 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 1 -- 172.21.15.181:0/4251684013 learned_addr learned my addr 172.21.15.181:0/4251684013 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 10 monclient(hunting): get_auth_request con 0x7f1f34136dd0 auth_method 0 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3abac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 1 --2- 172.21.15.181:0/4251684013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 0x7f1f3412e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): get_auth_request con 0x7f1f3409e560 auth_method 0 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_done global_id 4940 payload 274 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 1 -- 172.21.15.181:0/4251684013 >> v2:172.21.15.181:3300/0 conn(0x7f1f34136dd0 msgr2=0x7f1f341391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 1 --2- 172.21.15.181:0/4251684013 >> v2:172.21.15.181:3300/0 conn(0x7f1f34136dd0 0x7f1f341391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 1 -- 172.21.15.181:0/4251684013 >> v1:172.21.15.181:6789/0 conn(0x7f1f3409aeb0 legacy=0x7f1f3409b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient: _finish_hunting 0 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 1 monclient: found mon.noname-b 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:02.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.783+0000 7f1f3a3ab640 1 -- 172.21.15.181:0/4251684013 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3406ed70 con 0x7f1f3409e560 2024-09-15T03:31:02.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.786+0000 7f1f3a3ab640 1 -- 172.21.15.181:0/4251684013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 msgr2=0x7f1f3412e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.786+0000 7f1f3a3ab640 1 --2- 172.21.15.181:0/4251684013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 0x7f1f3412e7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009850 tx=0x7f1f240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f29b32b98be12cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 1 -- 172.21.15.181:0/4251684013 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2402ae80 con 0x7f1f3409e560 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: dump: 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: _finish_auth 0 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 1 -- 172.21.15.181:0/4251684013 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f24032440 con 0x7f1f3409e560 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 1 -- 172.21.15.181:0/4251684013 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f240327c0 con 0x7f1f3409e560 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 5 monclient: authenticate success, global_id 4940 2024-09-15T03:31:02.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 4 set_mon_vals no callback set 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f393a9640 10 monclient: dump: 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 10 monclient: shutdown 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 -- 172.21.15.181:0/4251684013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 msgr2=0x7f1f3412e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/4251684013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 0x7f1f3412e7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009850 tx=0x7f1f240090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 -- 172.21.15.181:0/4251684013 shutdown_connections 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/4251684013 >> v2:172.21.15.181:3300/0 conn(0x7f1f34136dd0 0x7f1f341391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/4251684013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409e560 0x7f1f3412e7c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 -- 172.21.15.181:0/4251684013 >> 172.21.15.181:0/4251684013 conn(0x7f1f340a1db0 msgr2=0x7f1f3409de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 -- 172.21.15.181:0/4251684013 shutdown_connections 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.787+0000 7f1f3c60f640 1 -- 172.21.15.181:0/4251684013 wait complete. 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.788+0000 7f1f3c60f640 10 monclient: build_initial_monmap 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.788+0000 7f1f3c60f640 10 monclient: monmap: 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.788+0000 7f1f3c60f640 1 Processor -- start 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.788+0000 7f1f3c60f640 1 -- start start 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.788+0000 7f1f3c60f640 10 monclient: init 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409aeb0 0x7f1f34135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f3409aeb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 0x7f1f3412da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f3409e560 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 0x7f1f341413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): picked mon.noname-a con 0x7f1f34136dd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34128ee0 con 0x7f1f34136dd0 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34127d30 con 0x7f1f3409e560 2024-09-15T03:31:02.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34128c00 con 0x7f1f3409aeb0 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3c60f640 10 monclient(hunting): authenticate will time out at 867.270175s 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 0x7f1f3412da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 0x7f1f3412da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48256/0 (socket says 172.21.15.181:48256) 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 -- 172.21.15.181:0/3059416211 learned_addr learned my addr 172.21.15.181:0/3059416211 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): get_auth_request con 0x7f1f3409e560 auth_method 0 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409aeb0 0x7f1f34135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 0x7f1f341413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 10 monclient(hunting): get_auth_request con 0x7f1f34136dd0 auth_method 0 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): get_auth_request con 0x7f1f3409aeb0 auth_method 0 2024-09-15T03:31:02.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 msgr2=0x7f1f341413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 msgr2=0x7f1f341413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 0x7f1f341413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3a3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f3abac640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 0x7f1f341413b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient(hunting): handle_auth_done global_id 44353 payload 824 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409aeb0 msgr2=0x7f1f34135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409aeb0 0x7f1f34135e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 msgr2=0x7f1f341413b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:02.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 0x7f1f341413b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient: _finish_hunting 0 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 monclient: found mon.noname-b 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.789+0000 7f1f39baa640 1 -- 172.21.15.181:0/3059416211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f24009390 con 0x7f1f3409e560 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f39baa640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 msgr2=0x7f1f3412da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f39baa640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 0x7f1f3412da00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000b4f0 tx=0x7f1f300090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000f720 con 0x7f1f3409e560 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: dump: 2024-09-15T03:31:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3000f8a0 con 0x7f1f3409e560 2024-09-15T03:31:02.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f3c60f640 5 monclient: authenticate success, global_id 44353 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000fc60 con 0x7f1f3409e560 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f3c60f640 10 monclient: _renew_subs 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f3c60f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f341291c0 con 0x7f1f3409e560 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f2a7fc640 10 monclient: dump: 2024-09-15T03:31:02.808 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.791+0000 7f1f38ba8640 4 set_mon_vals no callback set 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 10 monclient: _renew_subs 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f34127620 con 0x7f1f3409e560 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1f30011610 con 0x7f1f3409e560 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.792+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f34136890 con 0x7f1f3409e560 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.793+0000 7f1f2a7fc640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f18038740 0x7f1f1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.793+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f1f3004b6d0 con 0x7f1f3409e560 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.793+0000 7f1f3a3ab640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f18038740 0x7f1f1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:02.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.793+0000 7f1f3a3ab640 10 monclient: get_auth_request con 0x7f1f18038740 auth_method 0 2024-09-15T03:31:02.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.795+0000 7f1f3a3ab640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f18038740 0x7f1f1803abf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1f34069760 tx=0x7f1f24032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:02.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.797+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1f30017d20 con 0x7f1f3409e560 2024-09-15T03:31:02.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.797+0000 7f1f2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:02.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.797+0000 7f1f2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:02.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.955+0000 7f1f3c60f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T03:31:02.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.955+0000 7f1f3c60f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T03:31:02.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.955+0000 7f1f3c60f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:02.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:02.955+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7f1f340008d0 con 0x7f1f3409e560 2024-09-15T03:31:03.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.840+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v62) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1f30017720 con 0x7f1f3409e560 2024-09-15T03:31:03.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.840+0000 7f1f2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T03:31:03.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.840+0000 7f1f2a7fc640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-15T03:31:03.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.939+0000 7f1f3c60f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T03:31:03.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.939+0000 7f1f3c60f640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T03:31:03.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.939+0000 7f1f3c60f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:03.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.939+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7f1f3409e340 con 0x7f1f3409e560 2024-09-15T03:31:03.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.940+0000 7f1f2a7fc640 1 -- 172.21.15.181:0/3059416211 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v62) v1 ==== 134+0+0 (secure 0 0 0) 0x7f1f30017900 con 0x7f1f3409e560 2024-09-15T03:31:03.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.940+0000 7f1f2a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-15T03:31:03.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.940+0000 7f1f2a7fc640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-15T03:31:03.942 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'pinning_pool' already exists 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f18038740 msgr2=0x7f1f1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f18038740 0x7f1f1803abf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1f34069760 tx=0x7f1f24032000 comp rx=0 tx=0).stop 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 10 monclient: shutdown 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 msgr2=0x7f1f3412da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 0x7f1f3412da00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000b4f0 tx=0x7f1f300090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:03.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 shutdown_connections 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f18038740 0x7f1f1803abf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f34136dd0 0x7f1f341413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f3409e560 0x7f1f3412da00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 --2- 172.21.15.181:0/3059416211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3409aeb0 0x7f1f34135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 >> 172.21.15.181:0/3059416211 conn(0x7f1f340a1db0 msgr2=0x7f1f3409d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 shutdown_connections 2024-09-15T03:31:03.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:03.942+0000 7f1f3c60f640 1 -- 172.21.15.181:0/3059416211 wait complete. 2024-09-15T03:31:03.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-15T03:31:03.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:31:03.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.026+0000 7fd4b7f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.026+0000 7fd4b7f9d640 10 monclient: build_initial_monmap 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.026+0000 7fd4b7f9d640 10 monclient: monmap: 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:04.027705+0000 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:04.027705+0000 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:04.028 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:04.029 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:04.029 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:04.029 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:04.029 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.027+0000 7fd4b7f9d640 1 Processor -- start 2024-09-15T03:31:04.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.027+0000 7fd4b7f9d640 1 -- start start 2024-09-15T03:31:04.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.027+0000 7fd4b7f9d640 10 monclient: init 2024-09-15T03:31:04.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:04.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:31:04.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd4b0098c70 0x7fd4b0099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4b0098c70 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd4b0099580 0x7fd4b009dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4b0099580 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099c70 0x7fd4b009e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b0099c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b009f680 con 0x7fd4b0099580 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b009f420 con 0x7fd4b0098c70 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0068a20 con 0x7fd4b0099c70 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b7f9d640 10 monclient(hunting): authenticate will time out at 868.509237s 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd4b0098c70 0x7fd4b0099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd4b0098c70 0x7fd4b0099040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48266/0 (socket says 172.21.15.181:48266) 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 -- 172.21.15.181:0/4159553790 learned_addr learned my addr 172.21.15.181:0/4159553790 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): get_auth_request con 0x7fd4b0098c70 auth_method 0 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b653a640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099c70 0x7fd4b009e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5538640 1 --2- 172.21.15.181:0/4159553790 >> v2:172.21.15.153:3300/0 conn(0x7fd4b0099580 0x7fd4b009dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_done global_id 44356 payload 274 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 -- 172.21.15.181:0/4159553790 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099c70 msgr2=0x7fd4b009e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099c70 0x7fd4b009e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 -- 172.21.15.181:0/4159553790 >> v2:172.21.15.153:3300/0 conn(0x7fd4b0099580 msgr2=0x7fd4b009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 --2- 172.21.15.181:0/4159553790 >> v2:172.21.15.153:3300/0 conn(0x7fd4b0099580 0x7fd4b009dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient: _finish_hunting 0 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 monclient: found mon.noname-c 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:04.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5d39640 1 -- 172.21.15.181:0/4159553790 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b0096a30 con 0x7fd4b0098c70 2024-09-15T03:31:04.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b653a640 10 monclient: get_auth_request con 0x7fd4b0099c70 auth_method 0 2024-09-15T03:31:04.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5538640 10 monclient: get_auth_request con 0x7fd4b0099580 auth_method 0 2024-09-15T03:31:04.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b653a640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099c70 0x7fd4b009e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:04.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.028+0000 7fd4b5538640 1 --2- 172.21.15.181:0/4159553790 >> v2:172.21.15.153:3300/0 conn(0x7fd4b0099580 0x7fd4b009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:04.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b5d39640 1 -- 172.21.15.181:0/4159553790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 msgr2=0x7fd4b0099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b5d39640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0099040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009910 tx=0x7fd4ac0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ee291e4fe28152e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 1 -- 172.21.15.181:0/4159553790 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac032440 con 0x7fd4b0098c70 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: dump: 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: _finish_auth 0 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 1 -- 172.21.15.181:0/4159553790 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac0325a0 con 0x7fd4b0098c70 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b7f9d640 5 monclient: authenticate success, global_id 44356 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b7f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b7f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 1 -- 172.21.15.181:0/4159553790 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac032940 con 0x7fd4b0098c70 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b7f9d640 4 set_mon_vals no callback set 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.031+0000 7fd4b4d37640 10 monclient: dump: 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 10 monclient: shutdown 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/4159553790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 msgr2=0x7fd4b0099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0099040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009910 tx=0x7fd4ac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/4159553790 shutdown_connections 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099c70 0x7fd4b009e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 --2- 172.21.15.181:0/4159553790 >> v2:172.21.15.153:3300/0 conn(0x7fd4b0099580 0x7fd4b009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 --2- 172.21.15.181:0/4159553790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0099040 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/4159553790 >> 172.21.15.181:0/4159553790 conn(0x7fd4b0070e90 msgr2=0x7fd4b00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/4159553790 shutdown_connections 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/4159553790 wait complete. 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 10 monclient: build_initial_monmap 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 10 monclient: monmap: 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:04.049 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 Processor -- start 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 1 -- start start 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.032+0000 7fd4b7f9d640 10 monclient: init 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b0098c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099580 0x7fd4b011e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4b0099580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4b0099c70 0x7fd4b011edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4b0099c70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b006eba0 con 0x7fd4b0099c70 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00983d0 con 0x7fd4b0098c70 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b01363b0 con 0x7fd4b0099580 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b7f9d640 10 monclient(hunting): authenticate will time out at 868.514447s 2024-09-15T03:31:04.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0134d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48282/0 (socket says 172.21.15.181:48282) 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 -- 172.21.15.181:0/1316960916 learned_addr learned my addr 172.21.15.181:0/1316960916 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): get_auth_request con 0x7fd4b0098c70 auth_method 0 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099580 0x7fd4b011e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): get_auth_request con 0x7fd4b0099580 auth_method 0 2024-09-15T03:31:04.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b653a640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4b0099c70 0x7fd4b011edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b653a640 10 monclient(hunting): get_auth_request con 0x7fd4b0099c70 auth_method 0 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b653a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b653a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b653a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient(hunting): handle_auth_done global_id 44359 payload 824 2024-09-15T03:31:04.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 -- 172.21.15.181:0/1316960916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099580 msgr2=0x7fd4b011e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4b0099580 0x7fd4b011e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 -- 172.21.15.181:0/1316960916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4b0099c70 msgr2=0x7fd4b011edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4b0099c70 0x7fd4b011edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient: _finish_hunting 0 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 monclient: found mon.noname-b 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.033+0000 7fd4b5d39640 1 -- 172.21.15.181:0/1316960916 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac009450 con 0x7fd4b0098c70 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b5d39640 1 -- 172.21.15.181:0/1316960916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 msgr2=0x7fd4b0134d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b5d39640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4b0098c70 0x7fd4b0134d30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac035040 tx=0x7fd4ac032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac031a90 con 0x7fd4b0098c70 2024-09-15T03:31:04.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: dump: 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac031c10 con 0x7fd4b0098c70 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 5 monclient: authenticate success, global_id 44359 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac03d440 con 0x7fd4b0098c70 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 10 monclient: _renew_subs 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/1316960916 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b0120830 con 0x7fd4b0098c70 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a7fff640 4 set_mon_vals no callback set 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:04.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4a5ffb640 10 monclient: dump: 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 10 monclient: _renew_subs 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.036+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/1316960916 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b0122410 con 0x7fd4b0098c70 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd4ac039040 con 0x7fd4b0098c70 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4b7f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4a5ffb640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd494038740 0x7fd49403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4b7f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4b7f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd4ac06ba10 con 0x7fd4b0098c70 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/1316960916 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b0141db0 con 0x7fd4b0098c70 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4b5538640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd494038740 0x7fd49403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.037+0000 7fd4b5538640 10 monclient: get_auth_request con 0x7fd494038740 auth_method 0 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.039+0000 7fd4b5538640 1 --2- 172.21.15.181:0/1316960916 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd494038740 0x7fd49403abf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006fd0 tx=0x7fd4a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.042+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd4ac037020 con 0x7fd4b0098c70 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.042+0000 7fd4a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.042+0000 7fd4a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.200+0000 7fd4b7f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.200+0000 7fd4b7f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.200+0000 7fd4b7f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.200+0000 7fd4b7f9d640 1 -- 172.21.15.181:0/1316960916 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) v1 -- 0x7fd4b00008d0 con 0x7fd4b0098c70 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.201+0000 7fd4a5ffb640 1 -- 172.21.15.181:0/1316960916 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v62) v1 ==== 132+0+0 (secure 0 0 0) 0x7fd4ac044b30 con 0x7fd4b0098c70 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.201+0000 7fd4a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.201+0000 7fd4a5ffb640 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-15T03:31:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-15T03:31:04.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:31:04.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-15T03:31:04.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.279+0000 7fceef99d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.279+0000 7fceef99d640 10 monclient: build_initial_monmap 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.279+0000 7fceef99d640 10 monclient: monmap: 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:04.280049+0000 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:04.280049+0000 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:04.281 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:04.282 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:04.282 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.279+0000 7fceef99d640 1 Processor -- start 2024-09-15T03:31:04.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.279+0000 7fceef99d640 1 -- start start 2024-09-15T03:31:04.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.279+0000 7fceef99d640 10 monclient: init 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee81321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): picked mon.noname-b con 0x7fcee809f320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcee8070920 0x7fcee8070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): picked mon.noname-c con 0x7fcee8070920 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcee8071230 0x7fcee812e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): picked mon.noname-a con 0x7fcee8071230 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee8096b60 con 0x7fcee8071230 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee809e4d0 con 0x7fcee8070920 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee809e7b0 con 0x7fcee809f320 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceef99d640 10 monclient(hunting): authenticate will time out at 868.761469s 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcee8070920 0x7fcee8070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcee8070920 0x7fcee8070cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48288/0 (socket says 172.21.15.181:48288) 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 -- 172.21.15.181:0/3954519816 learned_addr learned my addr 172.21.15.181:0/3954519816 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): get_auth_request con 0x7fcee8070920 auth_method 0 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceed739640 1 --2- 172.21.15.181:0/3954519816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee81321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceedf3a640 1 --2- 172.21.15.181:0/3954519816 >> v2:172.21.15.153:3300/0 conn(0x7fcee8071230 0x7fcee812e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient(hunting): handle_auth_done global_id 44362 payload 274 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 -- 172.21.15.181:0/3954519816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 msgr2=0x7fcee81321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 --2- 172.21.15.181:0/3954519816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee81321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 -- 172.21.15.181:0/3954519816 >> v2:172.21.15.153:3300/0 conn(0x7fcee8071230 msgr2=0x7fcee812e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 --2- 172.21.15.181:0/3954519816 >> v2:172.21.15.153:3300/0 conn(0x7fcee8071230 0x7fcee812e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient: _finish_hunting 0 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 monclient: found mon.noname-c 2024-09-15T03:31:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:04.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceecf38640 1 -- 172.21.15.181:0/3954519816 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee809e1f0 con 0x7fcee8070920 2024-09-15T03:31:04.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceedf3a640 10 monclient: get_auth_request con 0x7fcee8071230 auth_method 0 2024-09-15T03:31:04.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.280+0000 7fceedf3a640 1 --2- 172.21.15.181:0/3954519816 >> v2:172.21.15.153:3300/0 conn(0x7fcee8071230 0x7fcee812e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fceecf38640 1 -- 172.21.15.181:0/3954519816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 msgr2=0x7fcee8070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fceecf38640 1 --2- 172.21.15.181:0/3954519816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8070cf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcee80719b0 tx=0x7fced80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e6405fbf7fb1643 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 1 -- 172.21.15.181:0/3954519816 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8032440 con 0x7fcee8070920 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: dump: 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: _finish_auth 0 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 1 -- 172.21.15.181:0/3954519816 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced80325a0 con 0x7fcee8070920 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fceef99d640 5 monclient: authenticate success, global_id 44362 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fceef99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:04.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fceef99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 1 -- 172.21.15.181:0/3954519816 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8032940 con 0x7fcee8070920 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fceef99d640 4 set_mon_vals no callback set 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.283+0000 7fcedffff640 10 monclient: dump: 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.294 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 10 monclient: shutdown 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 -- 172.21.15.181:0/3954519816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 msgr2=0x7fcee8070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 --2- 172.21.15.181:0/3954519816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8070cf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcee80719b0 tx=0x7fced80091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 -- 172.21.15.181:0/3954519816 shutdown_connections 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 --2- 172.21.15.181:0/3954519816 >> v2:172.21.15.153:3300/0 conn(0x7fcee8071230 0x7fcee812e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 --2- 172.21.15.181:0/3954519816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8070cf0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 --2- 172.21.15.181:0/3954519816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee81321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 -- 172.21.15.181:0/3954519816 >> 172.21.15.181:0/3954519816 conn(0x7fcee80a2b70 msgr2=0x7fcee809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 -- 172.21.15.181:0/3954519816 shutdown_connections 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 1 -- 172.21.15.181:0/3954519816 wait complete. 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 10 monclient: build_initial_monmap 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.284+0000 7fceef99d640 10 monclient: monmap: 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:04.296 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 Processor -- start 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 -- start start 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient: init 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): picked mon.noname-b con 0x7fcee8070920 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcee8071230 0x7fcee811b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): picked mon.noname-a con 0x7fcee8071230 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee811bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): picked mon.noname-c con 0x7fcee809f320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee811ef40 con 0x7fcee8071230 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee811ec60 con 0x7fcee8070920 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee811f220 con 0x7fcee809f320 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceef99d640 10 monclient(hunting): authenticate will time out at 868.766917s 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.285+0000 7fceed739640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8130b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48290/0 (socket says 172.21.15.181:48290) 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 -- 172.21.15.181:0/1561194804 learned_addr learned my addr 172.21.15.181:0/1561194804 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): get_auth_request con 0x7fcee8070920 auth_method 0 2024-09-15T03:31:04.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceedf3a640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee811bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceecf38640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcee8071230 0x7fcee811b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient(hunting): handle_auth_done global_id 44365 payload 824 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 -- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 msgr2=0x7fcee811bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee811bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 -- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcee8071230 msgr2=0x7fcee811b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcee8071230 0x7fcee811b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient: _finish_hunting 0 2024-09-15T03:31:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 monclient: found mon.noname-b 2024-09-15T03:31:04.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:04.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceed739640 1 -- 172.21.15.181:0/1561194804 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8009450 con 0x7fcee8070920 2024-09-15T03:31:04.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceedf3a640 10 monclient: get_auth_request con 0x7fcee809f320 auth_method 0 2024-09-15T03:31:04.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceedf3a640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee811bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:04.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceecf38640 10 monclient: get_auth_request con 0x7fcee8071230 auth_method 0 2024-09-15T03:31:04.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.286+0000 7fceecf38640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcee8071230 0x7fcee811b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceed739640 1 -- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 msgr2=0x7fcee8130b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceed739640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8130b30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcee400c880 tx=0x7fcee4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400ce10 con 0x7fcee8070920 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: dump: 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.302 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee4007a40 con 0x7fcee8070920 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 5 monclient: authenticate success, global_id 44365 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 10 monclient: _renew_subs 2024-09-15T03:31:04.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee8141a90 con 0x7fcee8070920 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4007e20 con 0x7fcee8070920 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedd7fa640 10 monclient: dump: 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.304 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fcedf7fe640 4 set_mon_vals no callback set 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 10 monclient: _renew_subs 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.288+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee811dea0 con 0x7fcee8070920 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcee4011600 con 0x7fcee8070920 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fceef99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fceef99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fceef99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee80008d0 con 0x7fcee8070920 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fcedd7fa640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcecc038740 0x7fcecc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fcee404bfa0 con 0x7fcee8070920 2024-09-15T03:31:04.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fceecf38640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcecc038740 0x7fcecc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.289+0000 7fceecf38640 10 monclient: get_auth_request con 0x7fcecc038740 auth_method 0 2024-09-15T03:31:04.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.292+0000 7fceecf38640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcecc038740 0x7fcecc03abf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcee809c590 tx=0x7fced8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.294+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcee4017af0 con 0x7fcee8070920 2024-09-15T03:31:04.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.294+0000 7fcedd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.294+0000 7fcedd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.448+0000 7fceef99d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:31:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.448+0000 7fceef99d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:31:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.448+0000 7fceef99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.448+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcee80716c0 con 0x7fcee8070920 2024-09-15T03:31:04.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:04.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.577+0000 7f1cd951d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:04.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.577+0000 7f1cd951d640 10 monclient: build_initial_monmap 2024-09-15T03:31:04.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.577+0000 7f1cd951d640 10 monclient: monmap: 2024-09-15T03:31:04.578 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:04.578 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.578 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:04.578065+0000 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:04.578065+0000 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:04.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:04.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.577+0000 7f1cd951d640 1 Processor -- start 2024-09-15T03:31:04.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.577+0000 7f1cd951d640 1 -- start start 2024-09-15T03:31:04.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient: init 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd406eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cd4068a20 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cd4136ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd4069ba0 con 0x7f1cd4068a20 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd406a3c0 con 0x7f1cd4136ab0 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4099ef0 con 0x7f1cd406eb80 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd951d640 10 monclient(hunting): authenticate will time out at 862.217492s 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46946/0 (socket says 172.21.15.153:46946) 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd27fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1cd4068a20 0x7f1cd409c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45064/0 (socket says 172.21.15.153:45064) 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/366249311 learned_addr learned my addr 172.21.15.153:0/366249311 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1cd406eb80 auth_method 0 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient(hunting): handle_auth_done global_id 4949 payload 274 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/366249311 >> v1:172.21.15.181:6789/0 conn(0x7f1cd4136ab0 legacy=0x7f1cd4138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/366249311 >> v1:172.21.15.153:6789/0 conn(0x7f1cd4068a20 legacy=0x7f1cd409c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:04.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.578+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/366249311 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4069e00 con 0x7f1cd406eb80 2024-09-15T03:31:04.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.581+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/366249311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 msgr2=0x7f1cd406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:04.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.581+0000 7f1cd2ffd640 1 --2- 172.21.15.153:0/366249311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406ef50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009f50 tx=0x7f1cc4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb583a6e4d6e159 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 1 -- 172.21.15.153:0/366249311 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4007610 con 0x7f1cd406eb80 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: dump: 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 1 -- 172.21.15.153:0/366249311 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc402d050 con 0x7f1cd406eb80 2024-09-15T03:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 5 monclient: authenticate success, global_id 4949 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 1 -- 172.21.15.153:0/366249311 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc40079a0 con 0x7f1cd406eb80 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd1ffb640 10 monclient: dump: 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:04.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 4 set_mon_vals no callback set 2024-09-15T03:31:04.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 10 monclient: shutdown 2024-09-15T03:31:04.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:04.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 -- 172.21.15.153:0/366249311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 msgr2=0x7f1cd406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 --2- 172.21.15.153:0/366249311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406ef50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009f50 tx=0x7f1cc4009840 comp rx=0 tx=0).stop 2024-09-15T03:31:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 -- 172.21.15.153:0/366249311 shutdown_connections 2024-09-15T03:31:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 --2- 172.21.15.153:0/366249311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406ef50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 -- 172.21.15.153:0/366249311 >> 172.21.15.153:0/366249311 conn(0x7f1cd4067cd0 msgr2=0x7f1cd40a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 -- 172.21.15.153:0/366249311 shutdown_connections 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.582+0000 7f1cd951d640 1 -- 172.21.15.153:0/366249311 wait complete. 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.583+0000 7f1cd951d640 10 monclient: build_initial_monmap 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.583+0000 7f1cd951d640 10 monclient: monmap: 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.583+0000 7f1cd951d640 1 Processor -- start 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.583+0000 7f1cd951d640 1 -- start start 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.583+0000 7f1cd951d640 10 monclient: init 2024-09-15T03:31:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd4068a20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): picked mon.noname-c con 0x7f1cd406eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cd4136ab0 0x7f1cd406cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd4136ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd411d940 con 0x7f1cd4068a20 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd409cf40 con 0x7f1cd4136ab0 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40a2100 con 0x7f1cd406eb80 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd951d640 10 monclient(hunting): authenticate will time out at 862.223233s 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33374/0 (socket says 172.21.15.153:33374) 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46948/0 (socket says 172.21.15.153:46948) 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/4002560172 learned_addr learned my addr 172.21.15.153:0/4002560172 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1cd4068a20 auth_method 0 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): get_auth_request con 0x7f1cd406eb80 auth_method 0 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 msgr2=0x7f1cd406c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 msgr2=0x7f1cd406c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:04.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd37fe640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cd4136ab0 0x7f1cd406cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd2ffd640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd37fe640 10 monclient(hunting): get_auth_request con 0x7f1cd4136ab0 auth_method 0 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient(hunting): handle_auth_done global_id 4952 payload 824 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cd4136ab0 msgr2=0x7f1cd406cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cd4136ab0 0x7f1cd406cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 msgr2=0x7f1cd406c290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 monclient: found mon.noname-c 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd27fc640 1 -- 172.21.15.153:0/4002560172 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc4009ae0 con 0x7f1cd406eb80 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.584+0000 7f1cd37fe640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cd4136ab0 0x7f1cd406cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.586+0000 7f1cd27fc640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 msgr2=0x7f1cd406c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.586+0000 7f1cd27fc640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406c7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1cc800c490 tx=0x7f1cc8007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc80074f0 con 0x7f1cd406eb80 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:04.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: dump: 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:04.608 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc8007670 con 0x7f1cd406eb80 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 5 monclient: authenticate success, global_id 4952 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 10 monclient: _renew_subs 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd411ec90 con 0x7f1cd406eb80 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8005a20 con 0x7f1cd406eb80 2024-09-15T03:31:04.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 10 monclient: dump: 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 10 monclient: _renew_subs 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd406dbb0 con 0x7f1cd406eb80 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cd17fa640 4 set_mon_vals no callback set 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.587+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1cc801b3b0 con 0x7f1cd406eb80 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cd951d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cd951d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cd951d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd411e8d0 con 0x7f1cd406eb80 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cbaffd640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1cc0038790 0x7f1cc003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f1cc804b320 con 0x7f1cd406eb80 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cd2ffd640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1cc0038790 0x7f1cc003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.588+0000 7f1cd2ffd640 10 monclient: get_auth_request con 0x7f1cc0038790 auth_method 0 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.590+0000 7f1cd2ffd640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1cc0038790 0x7f1cc003ac40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009ab0 tx=0x7f1cc4036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.592+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1cc8018620 con 0x7f1cd406eb80 2024-09-15T03:31:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.592+0000 7f1cbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.592+0000 7f1cbaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.808+0000 7f1cd951d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.808+0000 7f1cd951d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.808+0000 7f1cd951d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.808+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1cd404bf60 con 0x7f1cd406eb80 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.809+0000 7f1cbaffd640 1 -- 172.21.15.153:0/4002560172 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f1cc8017610 con 0x7f1cd406eb80 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.809+0000 7f1cbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.809+0000 7f1cbaffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:04.810 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1cc0038790 msgr2=0x7f1cc003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1cc0038790 0x7f1cc003ac40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009ab0 tx=0x7f1cc4036000 comp rx=0 tx=0).stop 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 10 monclient: shutdown 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 msgr2=0x7f1cd406c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406c7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1cc800c490 tx=0x7f1cc8007e40 comp rx=0 tx=0).stop 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 shutdown_connections 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1cc0038790 0x7f1cc003ac40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cd4136ab0 0x7f1cd406cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd406eb80 0x7f1cd406c7d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 --2- 172.21.15.153:0/4002560172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd4068a20 0x7f1cd406c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 >> 172.21.15.153:0/4002560172 conn(0x7f1cd4067cd0 msgr2=0x7f1cd409b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:04.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 shutdown_connections 2024-09-15T03:31:04.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:04.811+0000 7f1cd951d640 1 -- 172.21.15.153:0/4002560172 wait complete. 2024-09-15T03:31:04.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.850+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v63) v1 ==== 172+0+0 (secure 0 0 0) 0x7fcee401e030 con 0x7fcee8070920 2024-09-15T03:31:04.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.850+0000 7fcedd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:31:04.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.850+0000 7fcedd7fa640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-15T03:31:04.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fceef99d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:31:04.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fceef99d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:31:04.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fceef99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:04.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcee8064b20 con 0x7fcee8070920 2024-09-15T03:31:04.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fcedd7fa640 1 -- 172.21.15.181:0/1561194804 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v63) v1 ==== 179+0+0 (secure 0 0 0) 0x7fcee401e030 con 0x7fcee8070920 2024-09-15T03:31:04.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fcedd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:31:04.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.944+0000 7fcedd7fa640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-15T03:31:04.946 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'pinning_pool' does not exist 2024-09-15T03:31:04.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcecc038740 msgr2=0x7fcecc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcecc038740 0x7fcecc03abf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcee809c590 tx=0x7fced8032000 comp rx=0 tx=0).stop 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 10 monclient: shutdown 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 msgr2=0x7fcee8130b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8130b30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcee400c880 tx=0x7fcee4008040 comp rx=0 tx=0).stop 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 shutdown_connections 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcecc038740 0x7fcecc03abf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcee809f320 0x7fcee811bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcee8071230 0x7fcee811b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 --2- 172.21.15.181:0/1561194804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcee8070920 0x7fcee8130b30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 >> 172.21.15.181:0/1561194804 conn(0x7fcee80a2b70 msgr2=0x7fcee809b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:04.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 shutdown_connections 2024-09-15T03:31:04.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:04.946+0000 7fceef99d640 1 -- 172.21.15.181:0/1561194804 wait complete. 2024-09-15T03:31:04.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-15T03:31:04.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:31:04.957 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket foo osd 2024-09-15T03:31:05.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.028+0000 7f8733ec0640 10 monclient: get_monmap_and_config 2024-09-15T03:31:05.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.028+0000 7f8733ec0640 10 monclient: build_initial_monmap 2024-09-15T03:31:05.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.028+0000 7f8733ec0640 10 monclient: monmap: 2024-09-15T03:31:05.031 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:05.031 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.031 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:05.029749+0000 2024-09-15T03:31:05.031 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:05.029749+0000 2024-09-15T03:31:05.031 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:05.032 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.029+0000 7f8733ec0640 1 Processor -- start 2024-09-15T03:31:05.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.029+0000 7f8733ec0640 1 -- start start 2024-09-15T03:31:05.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.029+0000 7f8733ec0640 10 monclient: init 2024-09-15T03:31:05.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:05.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:31:05.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f872c09f180 0x7f872c130040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f872c09f180 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:05.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f872c09cee0 0x7f872c09bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f872c09cee0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:05.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f872c09c080 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:05.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c09e610 con 0x7f872c09c080 2024-09-15T03:31:05.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c09f550 con 0x7f872c09f180 2024-09-15T03:31:05.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c09e2a0 con 0x7f872c09cee0 2024-09-15T03:31:05.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:05.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8733ec0640 10 monclient(hunting): authenticate will time out at 869.511282s 2024-09-15T03:31:05.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f872c09cee0 0x7f872c09bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f872c09cee0 0x7f872c09bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48294/0 (socket says 172.21.15.181:48294) 2024-09-15T03:31:05.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 -- 172.21.15.181:0/3559797729 learned_addr learned my addr 172.21.15.181:0/3559797729 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:05.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): get_auth_request con 0x7f872c09cee0 auth_method 0 2024-09-15T03:31:05.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:05.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:05.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:05.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8731c5c640 1 --2- 172.21.15.181:0/3559797729 >> v2:172.21.15.153:3300/0 conn(0x7f872c09f180 0x7f872c130040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8731c5c640 10 monclient(hunting): get_auth_request con 0x7f872c09f180 auth_method 0 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8731c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8731c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f8731c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient(hunting): handle_auth_done global_id 44371 payload 274 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 -- 172.21.15.181:0/3559797729 >> v2:172.21.15.153:3300/0 conn(0x7f872c09f180 msgr2=0x7f872c130040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 --2- 172.21.15.181:0/3559797729 >> v2:172.21.15.153:3300/0 conn(0x7f872c09f180 0x7f872c130040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 -- 172.21.15.181:0/3559797729 >> v1:172.21.15.153:6789/0 conn(0x7f872c09c080 legacy=0x7f872c070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient: _finish_hunting 0 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 monclient: found mon.noname-c 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.030+0000 7f873145b640 1 -- 172.21.15.181:0/3559797729 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c130580 con 0x7f872c09cee0 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f873145b640 1 -- 172.21.15.181:0/3559797729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09cee0 msgr2=0x7f872c09bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f873145b640 1 --2- 172.21.15.181:0/3559797729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09cee0 0x7f872c09bb40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f872c0a3050 tx=0x7f871c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a7b4f44814dd19 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 1 -- 172.21.15.181:0/3559797729 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c032440 con 0x7f872c09cee0 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: dump: 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.051 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: _finish_auth 0 2024-09-15T03:31:05.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:05.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 1 -- 172.21.15.181:0/3559797729 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c0325a0 con 0x7f872c09cee0 2024-09-15T03:31:05.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8733ec0640 5 monclient: authenticate success, global_id 44371 2024-09-15T03:31:05.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8733ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:05.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:05.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8733ec0640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:05.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 1 -- 172.21.15.181:0/3559797729 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c032940 con 0x7f872c09cee0 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8733ec0640 4 set_mon_vals no callback set 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.033+0000 7f8730c5a640 10 monclient: dump: 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.054 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 10 monclient: shutdown 2024-09-15T03:31:05.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 -- 172.21.15.181:0/3559797729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09cee0 msgr2=0x7f872c09bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 --2- 172.21.15.181:0/3559797729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09cee0 0x7f872c09bb40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f872c0a3050 tx=0x7f871c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 -- 172.21.15.181:0/3559797729 shutdown_connections 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 --2- 172.21.15.181:0/3559797729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09cee0 0x7f872c09bb40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 --2- 172.21.15.181:0/3559797729 >> v2:172.21.15.153:3300/0 conn(0x7f872c09f180 0x7f872c130040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 -- 172.21.15.181:0/3559797729 >> 172.21.15.181:0/3559797729 conn(0x7f872c0a2b70 msgr2=0x7f872c09ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:05.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 -- 172.21.15.181:0/3559797729 shutdown_connections 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 -- 172.21.15.181:0/3559797729 wait complete. 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 10 monclient: build_initial_monmap 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 10 monclient: monmap: 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:05.057 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.034+0000 7f8733ec0640 1 Processor -- start 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 -- start start 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient: init 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09c080 0x7f872c11d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f872c09c080 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f872c09cee0 0x7f872c11d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f872c09cee0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f872c09f180 0x7f872c11da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f872c09f180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c12dc40 con 0x7f872c09f180 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c11f0b0 con 0x7f872c09c080 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c09b380 con 0x7f872c09cee0 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8733ec0640 10 monclient(hunting): authenticate will time out at 869.516467s 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09c080 0x7f872c11d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09c080 0x7f872c11d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48298/0 (socket says 172.21.15.181:48298) 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 -- 172.21.15.181:0/487783995 learned_addr learned my addr 172.21.15.181:0/487783995 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): get_auth_request con 0x7f872c09c080 auth_method 0 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient(hunting): handle_auth_done global_id 44374 payload 824 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f873245d640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f872c09f180 0x7f872c11da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f873145b640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f872c09cee0 0x7f872c11d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 -- 172.21.15.181:0/487783995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f872c09cee0 msgr2=0x7f872c11d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f872c09cee0 0x7f872c11d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 -- 172.21.15.181:0/487783995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f872c09f180 msgr2=0x7f872c11da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f872c09f180 0x7f872c11da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient: _finish_hunting 0 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 monclient: found mon.noname-b 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.035+0000 7f8731c5c640 1 -- 172.21.15.181:0/487783995 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c009450 con 0x7f872c09c080 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8731c5c640 1 -- 172.21.15.181:0/487783995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09c080 msgr2=0x7f872c11d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8731c5c640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872c09c080 0x7f872c11d010 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f872800d8c0 tx=0x7f8728008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800de50 con 0x7f872c09c080 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: dump: 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872800bae0 con 0x7f872c09c080 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 5 monclient: authenticate success, global_id 44374 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 10 monclient: _renew_subs 2024-09-15T03:31:05.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 1 -- 172.21.15.181:0/487783995 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c06ec70 con 0x7f872c09c080 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800be80 con 0x7f872c09c080 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8721ffb640 10 monclient: dump: 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.063 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.064 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.064 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.064 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.064 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.064 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.064 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 10 monclient: _renew_subs 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8733ec0640 1 -- 172.21.15.181:0/487783995 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c11eb60 con 0x7f872c09c080 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.038+0000 7f8723fff640 4 set_mon_vals no callback set 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8728014640 con 0x7f872c09c080 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8721ffb640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8710038790 0x7f871003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8733ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:05.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f872804c290 con 0x7f872c09c080 2024-09-15T03:31:05.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8733ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:05.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8733ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f8733ec0640 1 -- 172.21.15.181:0/487783995 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872c120a90 con 0x7f872c09c080 2024-09-15T03:31:05.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f873145b640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8710038790 0x7f871003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.039+0000 7f873145b640 10 monclient: get_auth_request con 0x7f8710038790 auth_method 0 2024-09-15T03:31:05.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.041+0000 7f873145b640 1 --2- 172.21.15.181:0/487783995 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8710038790 0x7f871003ac40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f872c09e580 tx=0x7f871c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:05.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.043+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f872801e460 con 0x7f872c09c080 2024-09-15T03:31:05.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.043+0000 7f8721ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:05.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.043+0000 7f8721ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.200+0000 7f8733ec0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.200+0000 7f8733ec0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.200+0000 7f8733ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.200+0000 7f8733ec0640 1 -- 172.21.15.181:0/487783995 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) v1 -- 0x7f872c0008d0 con 0x7f872c09c080 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.201+0000 7f8721ffb640 1 -- 172.21.15.181:0/487783995 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v63) v1 ==== 136+0+0 (secure 0 0 0) 0x7f872801dbb0 con 0x7f872c09c080 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.201+0000 7f8721ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.201+0000 7f8721ffb640 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-15T03:31:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-15T03:31:05.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:31:05.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket foo root 2024-09-15T03:31:05.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.278+0000 7f1dcc29a640 10 monclient: get_monmap_and_config 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.278+0000 7f1dcc29a640 10 monclient: build_initial_monmap 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.278+0000 7f1dcc29a640 10 monclient: monmap: 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:05.279588+0000 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:05.279588+0000 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:05.280 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:05.281 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.279+0000 7f1dcc29a640 1 Processor -- start 2024-09-15T03:31:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.279+0000 7f1dcc29a640 1 -- start start 2024-09-15T03:31:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.279+0000 7f1dcc29a640 10 monclient: init 2024-09-15T03:31:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T03:31:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc409cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1dc409cea0 0x7f1dc409dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc409cea0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1dc409d270 0x7f1dc409e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc409d270 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc409b710 con 0x7f1dc409cea0 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc409c6e0 con 0x7f1dc409d270 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc409a650 con 0x7f1dc409cad0 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dcc29a640 10 monclient(hunting): authenticate will time out at 869.761150s 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1dc409d270 0x7f1dc409e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1dc409d270 0x7f1dc409e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48306/0 (socket says 172.21.15.181:48306) 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 -- 172.21.15.181:0/3374584529 learned_addr learned my addr 172.21.15.181:0/3374584529 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): get_auth_request con 0x7f1dc409d270 auth_method 0 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca036640 1 --2- 172.21.15.181:0/3374584529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc409c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dc9835640 1 --2- 172.21.15.181:0/3374584529 >> v2:172.21.15.153:3300/0 conn(0x7f1dc409cea0 0x7f1dc409dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca036640 10 monclient(hunting): get_auth_request con 0x7f1dc409cad0 auth_method 0 2024-09-15T03:31:05.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca036640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca036640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dc9835640 10 monclient(hunting): get_auth_request con 0x7f1dc409cea0 auth_method 0 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dc9835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dc9835640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dc9835640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient(hunting): handle_auth_done global_id 44377 payload 274 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 -- 172.21.15.181:0/3374584529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 msgr2=0x7f1dc409c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 --2- 172.21.15.181:0/3374584529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc409c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 -- 172.21.15.181:0/3374584529 >> v2:172.21.15.153:3300/0 conn(0x7f1dc409cea0 msgr2=0x7f1dc409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 --2- 172.21.15.181:0/3374584529 >> v2:172.21.15.153:3300/0 conn(0x7f1dc409cea0 0x7f1dc409dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient: _finish_hunting 0 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 monclient: found mon.noname-c 2024-09-15T03:31:05.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:05.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.280+0000 7f1dca837640 1 -- 172.21.15.181:0/3374584529 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc409f660 con 0x7f1dc409d270 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dca837640 1 -- 172.21.15.181:0/3374584529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 msgr2=0x7f1dc409e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dca837640 1 --2- 172.21.15.181:0/3374584529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc409e010 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1dc406edb0 tx=0x7f1db40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee21dd912a2f0550 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 1 -- 172.21.15.181:0/3374584529 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4032440 con 0x7f1dc409d270 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: dump: 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: _finish_auth 0 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:05.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 1 -- 172.21.15.181:0/3374584529 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db40325a0 con 0x7f1dc409d270 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 5 monclient: authenticate success, global_id 44377 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 1 -- 172.21.15.181:0/3374584529 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4032980 con 0x7f1dc409d270 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 4 set_mon_vals no callback set 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dc9034640 10 monclient: dump: 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.295 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 10 monclient: shutdown 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 1 -- 172.21.15.181:0/3374584529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 msgr2=0x7f1dc409e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.283+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/3374584529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc409e010 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1dc406edb0 tx=0x7f1db40091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 -- 172.21.15.181:0/3374584529 shutdown_connections 2024-09-15T03:31:05.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/3374584529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc409e010 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/3374584529 >> v2:172.21.15.153:3300/0 conn(0x7f1dc409cea0 0x7f1dc409dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/3374584529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc409c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 -- 172.21.15.181:0/3374584529 >> 172.21.15.181:0/3374584529 conn(0x7f1dc4070e90 msgr2=0x7f1dc40a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 -- 172.21.15.181:0/3374584529 shutdown_connections 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 -- 172.21.15.181:0/3374584529 wait complete. 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 10 monclient: build_initial_monmap 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 10 monclient: monmap: 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.297 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 Processor -- start 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 1 -- start start 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.284+0000 7f1dcc29a640 10 monclient: init 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc41329e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc409cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 0x7f1dc411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc409cea0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc411c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc409d270 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc406eba0 con 0x7f1dc409cea0 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40a26b0 con 0x7f1dc409d270 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc411fb00 con 0x7f1dc409cad0 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dcc29a640 10 monclient(hunting): authenticate will time out at 869.766419s 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc411c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc411c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48308/0 (socket says 172.21.15.181:48308) 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 -- 172.21.15.181:0/1617841658 learned_addr learned my addr 172.21.15.181:0/1617841658 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): get_auth_request con 0x7f1dc409d270 auth_method 0 2024-09-15T03:31:05.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc41329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 0x7f1dc411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): get_auth_request con 0x7f1dc409cad0 auth_method 0 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 10 monclient(hunting): get_auth_request con 0x7f1dc409cea0 auth_method 0 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:05.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 msgr2=0x7f1dc411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 msgr2=0x7f1dc411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 0x7f1dc411c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dc9835640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 0x7f1dc411c4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient(hunting): handle_auth_done global_id 44380 payload 824 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 msgr2=0x7f1dc41329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc41329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 msgr2=0x7f1dc411c4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 0x7f1dc411c4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient: _finish_hunting 0 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 monclient: found mon.noname-b 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.285+0000 7f1dca837640 1 -- 172.21.15.181:0/1617841658 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db4009450 con 0x7f1dc409d270 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dca837640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 msgr2=0x7f1dc411c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dca837640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc411c9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1db4035040 tx=0x7f1db4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4031c70 con 0x7f1dc409d270 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: dump: 2024-09-15T03:31:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4031dd0 con 0x7f1dc409d270 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 5 monclient: authenticate success, global_id 44380 2024-09-15T03:31:05.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 10 monclient: _renew_subs 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc411e0a0 con 0x7f1dc409d270 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db403d690 con 0x7f1dc409d270 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc27fc640 10 monclient: dump: 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:05.304 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 10 monclient: _renew_subs 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc411ec10 con 0x7f1dc409d270 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.288+0000 7f1dc8833640 4 set_mon_vals no callback set 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dcc29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dcc29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dcc29a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1db4039040 con 0x7f1dc409d270 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc411ffa0 con 0x7f1dc409d270 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dc27fc640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1db0038740 0x7f1db003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f1db406bbf0 con 0x7f1dc409d270 2024-09-15T03:31:05.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dca036640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1db0038740 0x7f1db003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:05.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.289+0000 7f1dca036640 10 monclient: get_auth_request con 0x7f1db0038740 auth_method 0 2024-09-15T03:31:05.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.292+0000 7f1dca036640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1db0038740 0x7f1db003abf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc0075a0 tx=0x7f1dbc0097e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:05.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.293+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1db402f4d0 con 0x7f1dc409d270 2024-09-15T03:31:05.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.293+0000 7f1dc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:05.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.293+0000 7f1dc27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:05.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.452+0000 7f1dcc29a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T03:31:05.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.452+0000 7f1dcc29a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T03:31:05.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.452+0000 7f1dcc29a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.452+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f1dc40008d0 con 0x7f1dc409d270 2024-09-15T03:31:05.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.853+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1db4042020 con 0x7f1dc409d270 2024-09-15T03:31:05.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.853+0000 7f1dc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T03:31:05.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.853+0000 7f1dc27fc640 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-15T03:31:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dcc29a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T03:31:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dcc29a640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T03:31:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dcc29a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f1dc409b430 con 0x7f1dc409d270 2024-09-15T03:31:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dc27fc640 1 -- 172.21.15.181:0/1617841658 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v64) v1 ==== 126+0+0 (secure 0 0 0) 0x7f1db4042030 con 0x7f1dc409d270 2024-09-15T03:31:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dc27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-15T03:31:05.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.950+0000 7f1dc27fc640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-15T03:31:05.952 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'foo' already exists 2024-09-15T03:31:05.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1db0038740 msgr2=0x7f1db003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1db0038740 0x7f1db003abf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc0075a0 tx=0x7f1dbc0097e0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 10 monclient: shutdown 2024-09-15T03:31:05.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 msgr2=0x7f1dc411c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc411c9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1db4035040 tx=0x7f1db4032000 comp rx=0 tx=0).stop 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 shutdown_connections 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1db0038740 0x7f1db003abf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dc409d270 0x7f1dc411c9f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc409cea0 0x7f1dc411c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 --2- 172.21.15.181:0/1617841658 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1dc409cad0 0x7f1dc41329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 >> 172.21.15.181:0/1617841658 conn(0x7f1dc4070e90 msgr2=0x7f1dc40a1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 shutdown_connections 2024-09-15T03:31:05.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:05.952+0000 7f1dcc29a640 1 -- 172.21.15.181:0/1617841658 wait complete. 2024-09-15T03:31:05.962 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 2024-09-15T03:31:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.033+0000 7f1274657640 10 monclient: get_monmap_and_config 2024-09-15T03:31:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.033+0000 7f1274657640 10 monclient: build_initial_monmap 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.033+0000 7f1274657640 10 monclient: monmap: 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:06.034043+0000 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:06.034043+0000 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:06.035 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.033+0000 7f1274657640 1 Processor -- start 2024-09-15T03:31:06.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.033+0000 7f1274657640 1 -- start start 2024-09-15T03:31:06.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.033+0000 7f1274657640 10 monclient: init 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c09eb10 0x7f126c0a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): picked mon.noname-b con 0x7f126c09eb10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f126c09fde0 0x7f126c0a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): picked mon.noname-c con 0x7f126c09fde0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f126c0a06f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c09c6c0 con 0x7f126c0a06f0 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126c09e030 con 0x7f126c09fde0 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c09c9e0 con 0x7f126c09eb10 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1274657640 10 monclient(hunting): authenticate will time out at 870.515570s 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f126c09fde0 0x7f126c0a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f126c09fde0 0x7f126c0a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48312/0 (socket says 172.21.15.181:48312) 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 -- 172.21.15.181:0/1633665443 learned_addr learned my addr 172.21.15.181:0/1633665443 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): get_auth_request con 0x7f126c09fde0 auth_method 0 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:06.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f12723f3640 1 --2- 172.21.15.181:0/1633665443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c09eb10 0x7f126c0a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f12723f3640 10 monclient(hunting): get_auth_request con 0x7f126c09eb10 auth_method 0 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f12723f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f12723f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f12723f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient(hunting): handle_auth_done global_id 44383 payload 274 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 -- 172.21.15.181:0/1633665443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c09eb10 msgr2=0x7f126c0a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 --2- 172.21.15.181:0/1633665443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c09eb10 0x7f126c0a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 -- 172.21.15.181:0/1633665443 >> v1:172.21.15.153:6789/0 conn(0x7f126c0a06f0 legacy=0x7f126c136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient: _finish_hunting 0 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 monclient: found mon.noname-c 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:06.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.034+0000 7f1271bf2640 1 -- 172.21.15.181:0/1633665443 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c09eee0 con 0x7f126c09fde0 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.037+0000 7f1271bf2640 1 -- 172.21.15.181:0/1633665443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09fde0 msgr2=0x7f126c0a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.037+0000 7f1271bf2640 1 --2- 172.21.15.181:0/1633665443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09fde0 0x7f126c0a01b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f126c066400 tx=0x7f12680091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cd6c7359e0851f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.037+0000 7f12713f1640 1 -- 172.21.15.181:0/1633665443 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268032440 con 0x7f126c09fde0 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.037+0000 7f12713f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: dump: 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:06.046 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: _finish_auth 0 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 1 -- 172.21.15.181:0/1633665443 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12680325a0 con 0x7f126c09fde0 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 5 monclient: authenticate success, global_id 44383 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 1 -- 172.21.15.181:0/1633665443 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268032940 con 0x7f126c09fde0 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 4 set_mon_vals no callback set 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:06.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f12713f1640 10 monclient: dump: 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:06.048 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 10 monclient: shutdown 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 -- 172.21.15.181:0/1633665443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09fde0 msgr2=0x7f126c0a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 --2- 172.21.15.181:0/1633665443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09fde0 0x7f126c0a01b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f126c066400 tx=0x7f12680091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:06.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 -- 172.21.15.181:0/1633665443 shutdown_connections 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 --2- 172.21.15.181:0/1633665443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09fde0 0x7f126c0a01b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 --2- 172.21.15.181:0/1633665443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c09eb10 0x7f126c0a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 -- 172.21.15.181:0/1633665443 >> 172.21.15.181:0/1633665443 conn(0x7f126c0a52a0 msgr2=0x7f126c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 -- 172.21.15.181:0/1633665443 shutdown_connections 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.038+0000 7f1274657640 1 -- 172.21.15.181:0/1633665443 wait complete. 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient: build_initial_monmap 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient: monmap: 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:06.050 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 Processor -- start 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 -- start start 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient: init 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:06.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 0x7f126c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): picked mon.noname-b con 0x7f126c09eb10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 0x7f126c124440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): picked mon.noname-a con 0x7f126c09fde0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c0a06f0 0x7f126c1382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): picked mon.noname-c con 0x7f126c0a06f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c1256b0 con 0x7f126c09fde0 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c06e880 con 0x7f126c09eb10 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c124980 con 0x7f126c0a06f0 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.039+0000 7f1274657640 10 monclient(hunting): authenticate will time out at 870.520989s 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 0x7f126c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 0x7f126c06dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48316/0 (socket says 172.21.15.181:48316) 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 -- 172.21.15.181:0/2562950743 learned_addr learned my addr 172.21.15.181:0/2562950743 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): get_auth_request con 0x7f126c09eb10 auth_method 0 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:06.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 0x7f126c124440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c0a06f0 0x7f126c1382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 10 monclient(hunting): get_auth_request con 0x7f126c09fde0 auth_method 0 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:06.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): get_auth_request con 0x7f126c0a06f0 auth_method 0 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 msgr2=0x7f126c124440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 msgr2=0x7f126c124440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 0x7f126c124440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:06.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1271bf2640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 0x7f126c124440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f1272bf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient(hunting): handle_auth_done global_id 44386 payload 824 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c0a06f0 msgr2=0x7f126c1382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c0a06f0 0x7f126c1382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 msgr2=0x7f126c124440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 0x7f126c124440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient: _finish_hunting 0 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 monclient: found mon.noname-b 2024-09-15T03:31:06.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:06.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.040+0000 7f12723f3640 1 -- 172.21.15.181:0/2562950743 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268009450 con 0x7f126c09eb10 2024-09-15T03:31:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f12723f3640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 msgr2=0x7f126c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f12723f3640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 0x7f126c06dc50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f126000d8c0 tx=0x7f1260008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126000de50 con 0x7f126c09eb10 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: dump: 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 5 monclient: authenticate success, global_id 44386 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260009ac0 con 0x7f126c09eb10 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 10 monclient: _renew_subs 2024-09-15T03:31:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c126440 con 0x7f126c09eb10 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260009e80 con 0x7f126c09eb10 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f125e7fc640 10 monclient: dump: 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 10 monclient: _renew_subs 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c1387e0 con 0x7f126c09eb10 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.042+0000 7f1270bf0640 4 set_mon_vals no callback set 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.043+0000 7f1274657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:06.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.043+0000 7f1274657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.043+0000 7f1274657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.043+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c1265d0 con 0x7f126c09eb10 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.043+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1260014620 con 0x7f126c09eb10 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.044+0000 7f125e7fc640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1250038790 0x7f125003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.044+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5058+0+0 (secure 0 0 0) 0x7f126004c320 con 0x7f126c09eb10 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.044+0000 7f1271bf2640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1250038790 0x7f125003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.044+0000 7f1271bf2640 10 monclient: get_auth_request con 0x7f1250038790 auth_method 0 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.046+0000 7f1271bf2640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1250038790 0x7f125003ac40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f126c0a1a00 tx=0x7f1268032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.048+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f126c1265d0 con 0x7f126c09eb10 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.048+0000 7f125e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:06.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.048+0000 7f125e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:06.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.213+0000 7f1274657640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:31:06.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.213+0000 7f1274657640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T03:31:06.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.213+0000 7f1274657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.213+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f126c1265d0 con 0x7f126c09eb10 2024-09-15T03:31:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.858+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v65) v1 ==== 58+0+2 (secure 0 0 0) 0x7f126c1265d0 con 0x7f126c09eb10 2024-09-15T03:31:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.858+0000 7f125e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T03:31:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.858+0000 7f125e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:06.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.961+0000 7f1274657640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:31:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.961+0000 7f1274657640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T03:31:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.961+0000 7f1274657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:06.961+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f126c1265d0 con 0x7f126c09eb10 2024-09-15T03:31:07.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.539+0000 7f125e7fc640 1 -- 172.21.15.181:0/2562950743 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v66) v1 ==== 58+0+2 (secure 0 0 0) 0x7f126c1265d0 con 0x7f126c09eb10 2024-09-15T03:31:07.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.539+0000 7f125e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T03:31:07.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.539+0000 7f125e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:07.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1250038790 msgr2=0x7f125003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1250038790 0x7f125003ac40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f126c0a1a00 tx=0x7f1268032000 comp rx=0 tx=0).stop 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 10 monclient: shutdown 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 msgr2=0x7f126c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 0x7f126c06dc50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f126000d8c0 tx=0x7f1260008040 comp rx=0 tx=0).stop 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 shutdown_connections 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1250038790 0x7f125003ac40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f126c0a06f0 0x7f126c1382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f126c09fde0 0x7f126c124440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 --2- 172.21.15.181:0/2562950743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f126c09eb10 0x7f126c06dc50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 >> 172.21.15.181:0/2562950743 conn(0x7f126c0a52a0 msgr2=0x7f126c09a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 shutdown_connections 2024-09-15T03:31:07.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.542+0000 7f1274657640 1 -- 172.21.15.181:0/2562950743 wait complete. 2024-09-15T03:31:07.552 INFO:tasks.workunit.client.0.smithi181.stderr:+ o1=7 2024-09-15T03:31:07.553 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 2024-09-15T03:31:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.624+0000 7f4c7580f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.624+0000 7f4c7580f640 10 monclient: build_initial_monmap 2024-09-15T03:31:07.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.624+0000 7f4c7580f640 10 monclient: monmap: 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:07.625005+0000 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:07.625005+0000 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.624+0000 7f4c7580f640 1 Processor -- start 2024-09-15T03:31:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.624+0000 7f4c7580f640 1 -- start start 2024-09-15T03:31:07.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.624+0000 7f4c7580f640 10 monclient: init 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 0x7f4c7009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): picked mon.noname-b con 0x7f4c70131b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4c7009f240 0x7f4c70131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): picked mon.noname-c con 0x7f4c7009f240 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c70132450 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7006a230 con 0x7f4c70132450 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c70132890 con 0x7f4c7009f240 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700a0300 con 0x7f4c70131b40 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c7580f640 10 monclient(hunting): authenticate will time out at 872.106392s 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4c7009f240 0x7f4c70131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4c7009f240 0x7f4c70131f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48324/0 (socket says 172.21.15.181:48324) 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 1 -- 172.21.15.181:0/616816061 learned_addr learned my addr 172.21.15.181:0/616816061 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request con 0x7f4c7009f240 auth_method 0 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6f7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4c70132450 0x7f4c70068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:54282/0 (socket says 172.21.15.181:54282) 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 1 --2- 172.21.15.181:0/616816061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 0x7f4c7009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): get_auth_request con 0x7f4c70131b40 auth_method 0 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient(hunting): handle_auth_done global_id 4967 payload 274 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 1 -- 172.21.15.181:0/616816061 >> v2:172.21.15.181:3300/0 conn(0x7f4c7009f240 msgr2=0x7f4c70131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 1 --2- 172.21.15.181:0/616816061 >> v2:172.21.15.181:3300/0 conn(0x7f4c7009f240 0x7f4c70131f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 1 -- 172.21.15.181:0/616816061 >> v1:172.21.15.181:6789/0 conn(0x7f4c70132450 legacy=0x7f4c70068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 1 monclient: found mon.noname-b 2024-09-15T03:31:07.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:07.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:07.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.625+0000 7f4c6effd640 1 -- 172.21.15.181:0/616816061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c700a00c0 con 0x7f4c70131b40 2024-09-15T03:31:07.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.628+0000 7f4c6effd640 1 -- 172.21.15.181:0/616816061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 msgr2=0x7f4c7009ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:07.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.628+0000 7f4c6effd640 1 --2- 172.21.15.181:0/616816061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 0x7f4c7009ed00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009850 tx=0x7f4c600090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0435912ca2537e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:07.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 1 -- 172.21.15.181:0/616816061 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6002adf0 con 0x7f4c70131b40 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: dump: 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 1 -- 172.21.15.181:0/616816061 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c60032440 con 0x7f4c70131b40 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 5 monclient: authenticate success, global_id 4967 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 1 -- 172.21.15.181:0/616816061 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c600327e0 con 0x7f4c70131b40 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 4 set_mon_vals no callback set 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c6dffb640 10 monclient: dump: 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.639 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 10 monclient: shutdown 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 -- 172.21.15.181:0/616816061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 msgr2=0x7f4c7009ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 --2- 172.21.15.181:0/616816061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 0x7f4c7009ed00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009850 tx=0x7f4c600090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 -- 172.21.15.181:0/616816061 shutdown_connections 2024-09-15T03:31:07.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 --2- 172.21.15.181:0/616816061 >> v2:172.21.15.181:3300/0 conn(0x7f4c7009f240 0x7f4c70131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 --2- 172.21.15.181:0/616816061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c70131b40 0x7f4c7009ed00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 -- 172.21.15.181:0/616816061 >> 172.21.15.181:0/616816061 conn(0x7f4c700a5100 msgr2=0x7f4c700a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 -- 172.21.15.181:0/616816061 shutdown_connections 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.629+0000 7f4c7580f640 1 -- 172.21.15.181:0/616816061 wait complete. 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.630+0000 7f4c7580f640 10 monclient: build_initial_monmap 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.630+0000 7f4c7580f640 10 monclient: monmap: 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:07.641 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.630+0000 7f4c7580f640 1 Processor -- start 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.630+0000 7f4c7580f640 1 -- start start 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.630+0000 7f4c7580f640 10 monclient: init 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009f240 0x7f4c701411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): picked mon.noname-c con 0x7f4c7009f240 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 0x7f4c7012ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): picked mon.noname-b con 0x7f4c70131b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70132450 0x7f4c70098b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): picked mon.noname-a con 0x7f4c70132450 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:07.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700a0a30 con 0x7f4c70132450 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c701315e0 con 0x7f4c70131b40 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7011c140 con 0x7f4c7009f240 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c7580f640 10 monclient(hunting): authenticate will time out at 872.112075s 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 0x7f4c7012ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 0x7f4c7012ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48326/0 (socket says 172.21.15.181:48326) 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 -- 172.21.15.181:0/1318174056 learned_addr learned my addr 172.21.15.181:0/1318174056 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request con 0x7f4c70131b40 auth_method 0 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009f240 0x7f4c701411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): get_auth_request con 0x7f4c7009f240 auth_method 0 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70132450 0x7f4c70098b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4c70132450 auth_method 0 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_done global_id 44392 payload 824 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 -- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009f240 msgr2=0x7f4c701411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009f240 0x7f4c701411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 -- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70132450 msgr2=0x7f4c70098b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70132450 0x7f4c70098b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:07.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 monclient: found mon.noname-b 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6e7fc640 1 -- 172.21.15.181:0/1318174056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60009390 con 0x7f4c70131b40 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.631+0000 7f4c6effd640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009f240 0x7f4c701411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c6e7fc640 1 -- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 msgr2=0x7f4c7012ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c6e7fc640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 0x7f4c7012ed10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400d890 tx=0x7f4c64008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6400de20 con 0x7f4c70131b40 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: dump: 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.646 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:07.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c64009ac0 con 0x7f4c70131b40 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c7580f640 5 monclient: authenticate success, global_id 44392 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c64009e80 con 0x7f4c70131b40 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c56ffd640 10 monclient: dump: 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.648 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c7580f640 10 monclient: _renew_subs 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c7580f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7011be30 con 0x7f4c70131b40 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.633+0000 7f4c6d7fa640 4 set_mon_vals no callback set 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 10 monclient: _renew_subs 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7009b1f0 con 0x7f4c70131b40 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4c64014620 con 0x7f4c70131b40 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:07.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c56ffd640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038770 0x7f4c5c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c700636e0 con 0x7f4c70131b40 2024-09-15T03:31:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f4c6404c0e0 con 0x7f4c70131b40 2024-09-15T03:31:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.634+0000 7f4c6effd640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038770 0x7f4c5c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.635+0000 7f4c6effd640 10 monclient: get_auth_request con 0x7f4c5c038770 auth_method 0 2024-09-15T03:31:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.637+0000 7f4c6effd640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038770 0x7f4c5c03ac20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4c60031000 tx=0x7f4c60032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:07.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.639+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4c64018080 con 0x7f4c70131b40 2024-09-15T03:31:07.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.639+0000 7f4c56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:07.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.639+0000 7f4c56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.807+0000 7f4c7580f640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:31:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.807+0000 7f4c7580f640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T03:31:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.807+0000 7f4c7580f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:07.807+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f4c70064a70 con 0x7f4c70131b40 2024-09-15T03:31:07.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:07.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.927+0000 7f0f22700640 10 monclient: get_monmap_and_config 2024-09-15T03:31:07.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.927+0000 7f0f22700640 10 monclient: build_initial_monmap 2024-09-15T03:31:07.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.927+0000 7f0f22700640 10 monclient: monmap: 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:07.928452+0000 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:07.928452+0000 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:07.929 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:07.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.928+0000 7f0f22700640 1 Processor -- start 2024-09-15T03:31:07.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.928+0000 7f0f22700640 1 -- start start 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.928+0000 7f0f22700640 10 monclient: init 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f1c131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 0x7f0f1c09ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient(hunting): picked mon.noname-b con 0x7f0f1c09fe10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f1c09ef70 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c0a14d0 con 0x7f0f1c131980 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c1325b0 con 0x7f0f1c09ef70 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0a11f0 con 0x7f0f1c09fe10 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f22700640 10 monclient(hunting): authenticate will time out at 865.568310s 2024-09-15T03:31:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f1bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0f1c131980 0x7f0f1c131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45458/0 (socket says 172.21.15.153:45458) 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f1bfff640 1 -- 172.21.15.153:0/3596678628 learned_addr learned my addr 172.21.15.153:0/3596678628 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 0x7f0f1c09ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): get_auth_request con 0x7f0f1c09fe10 auth_method 0 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient(hunting): handle_auth_done global_id 4973 payload 274 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 1 -- 172.21.15.153:0/3596678628 >> v1:172.21.15.181:6789/0 conn(0x7f0f1c09ef70 legacy=0x7f0f1c138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 1 -- 172.21.15.153:0/3596678628 >> v1:172.21.15.153:6789/0 conn(0x7f0f1c131980 legacy=0x7f0f1c131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 1 monclient: found mon.noname-b 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:07.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.929+0000 7f0f13fff640 1 -- 172.21.15.153:0/3596678628 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c09f390 con 0x7f0f1c09fe10 2024-09-15T03:31:07.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f13fff640 1 -- 172.21.15.153:0/3596678628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 msgr2=0x7f0f1c09ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:07.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f13fff640 1 --2- 172.21.15.153:0/3596678628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 0x7f0f1c09ea30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009880 tx=0x7f0f040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d5a39284755b5f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:07.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 1 -- 172.21.15.153:0/3596678628 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04032440 con 0x7f0f1c09fe10 2024-09-15T03:31:07.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: dump: 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.941 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 1 -- 172.21.15.153:0/3596678628 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f040325a0 con 0x7f0f1c09fe10 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 1 -- 172.21.15.153:0/3596678628 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04032940 con 0x7f0f1c09fe10 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f22700640 5 monclient: authenticate success, global_id 4973 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f22700640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f22700640 4 set_mon_vals no callback set 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.932+0000 7f0f1b7fe640 10 monclient: dump: 2024-09-15T03:31:07.942 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 10 monclient: shutdown 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 -- 172.21.15.153:0/3596678628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 msgr2=0x7f0f1c09ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 --2- 172.21.15.153:0/3596678628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 0x7f0f1c09ea30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009880 tx=0x7f0f040090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 -- 172.21.15.153:0/3596678628 shutdown_connections 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 --2- 172.21.15.153:0/3596678628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09fe10 0x7f0f1c09ea30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 -- 172.21.15.153:0/3596678628 >> 172.21.15.153:0/3596678628 conn(0x7f0f1c0a5050 msgr2=0x7f0f1c0a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 -- 172.21.15.153:0/3596678628 shutdown_connections 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 1 -- 172.21.15.153:0/3596678628 wait complete. 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 10 monclient: build_initial_monmap 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.933+0000 7f0f22700640 10 monclient: monmap: 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:07.944 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 Processor -- start 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 -- start start 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient: init 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 0x7f0f1c1343b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): picked mon.noname-c con 0x7f0f1c09ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f1c09fe10 0x7f0f1c1348f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1c09fe10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c131980 0x7f0f1c134e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): picked mon.noname-b con 0x7f0f1c131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1358b0 con 0x7f0f1c09fe10 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c135ff0 con 0x7f0f1c131980 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c137d00 con 0x7f0f1c09ef70 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:07.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f22700640 10 monclient(hunting): authenticate will time out at 865.573823s 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 0x7f0f1c1343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 0x7f0f1c1343b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56444/0 (socket says 172.21.15.153:56444) 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f1c09fe10 0x7f0f1c1348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 1 -- 172.21.15.153:0/446354957 learned_addr learned my addr 172.21.15.153:0/446354957 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 10 monclient(hunting): get_auth_request con 0x7f0f1c09ef70 auth_method 0 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f13fff640 10 monclient(hunting): get_auth_request con 0x7f0f1c09fe10 auth_method 0 2024-09-15T03:31:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.934+0000 7f0f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c131980 0x7f0f1c134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): get_auth_request con 0x7f0f1c131980 auth_method 0 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:07.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f20c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 10 monclient(hunting): handle_auth_done global_id 4976 payload 824 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 1 -- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c131980 msgr2=0x7f0f1c134e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c131980 0x7f0f1c134e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 1 -- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f1c09fe10 msgr2=0x7f0f1c1348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f1c09fe10 0x7f0f1c1348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 1 monclient: found mon.noname-c 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f1bfff640 1 -- 172.21.15.153:0/446354957 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04009390 con 0x7f0f1c09ef70 2024-09-15T03:31:07.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.935+0000 7f0f13fff640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f1c09fe10 0x7f0f1c1348f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f1bfff640 1 -- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 msgr2=0x7f0f1c1343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f1bfff640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 0x7f0f1c1343b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c640 tx=0x7f0f0c004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c005cb0 con 0x7f0f1c09ef70 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: dump: 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.950 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 5 monclient: authenticate success, global_id 4976 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c005e30 con 0x7f0f1c09ef70 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 10 monclient: _renew_subs 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c007340 con 0x7f0f1c09ef70 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c137a20 con 0x7f0f1c09ef70 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f1affd640 4 set_mon_vals no callback set 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f18ff9640 10 monclient: dump: 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 10 monclient: _renew_subs 2024-09-15T03:31:07.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.937+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c137610 con 0x7f0f1c09ef70 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0f0c01b400 con 0x7f0f1c09ef70 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f22700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f22700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f22700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c138100 con 0x7f0f1c09ef70 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f18ff9640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0efc038790 0x7f0efc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f0f0c04b840 con 0x7f0f1c09ef70 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.938+0000 7f0f13fff640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0efc038790 0x7f0efc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:07.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.939+0000 7f0f13fff640 10 monclient: get_auth_request con 0x7f0efc038790 auth_method 0 2024-09-15T03:31:07.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.941+0000 7f0f13fff640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0efc038790 0x7f0efc03ac40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c06f930 tx=0x7f0f04032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:07.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.943+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0f0c007730 con 0x7f0f1c09ef70 2024-09-15T03:31:07.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.943+0000 7f0f18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:07.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:07.943+0000 7f0f18ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:08.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f22700640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f22700640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f22700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0f1c067f00 con 0x7f0f1c09ef70 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f18ff9640 1 -- 172.21.15.153:0/446354957 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0f0c01a360 con 0x7f0f1c09ef70 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.161+0000 7f0f18ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:08.163 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0efc038790 msgr2=0x7f0efc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0efc038790 0x7f0efc03ac40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c06f930 tx=0x7f0f04032000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 10 monclient: shutdown 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 msgr2=0x7f0f1c1343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 0x7f0f1c1343b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c640 tx=0x7f0f0c004960 comp rx=0 tx=0).stop 2024-09-15T03:31:08.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 shutdown_connections 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0efc038790 0x7f0efc03ac40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c131980 0x7f0f1c134e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f1c09fe10 0x7f0f1c1348f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 --2- 172.21.15.153:0/446354957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f1c09ef70 0x7f0f1c1343b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 >> 172.21.15.153:0/446354957 conn(0x7f0f1c0a5050 msgr2=0x7f0f1c09dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 shutdown_connections 2024-09-15T03:31:08.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.163+0000 7f0f22700640 1 -- 172.21.15.153:0/446354957 wait complete. 2024-09-15T03:31:08.175 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:31:08.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:31:08.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:31:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.267+0000 7f07e5509640 10 monclient: get_monmap_and_config 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.267+0000 7f07e5509640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.267+0000 7f07e5509640 10 monclient: monmap: 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:08.268706+0000 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:08.268706+0000 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:08.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.268+0000 7f07e5509640 1 Processor -- start 2024-09-15T03:31:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.268+0000 7f07e5509640 1 -- start start 2024-09-15T03:31:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.268+0000 7f07e5509640 10 monclient: init 2024-09-15T03:31:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:31:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07e01318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07e009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f07e0137c30 0x7f07e013a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient(hunting): picked mon.noname-c con 0x7f07e0137c30 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e0131cc0 con 0x7f07e01318f0 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e009f530 con 0x7f07e009ea50 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e009fbf0 con 0x7f07e0137c30 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07e5509640 10 monclient(hunting): authenticate will time out at 865.908102s 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07deffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f07e01318f0 0x7f07e0131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45472/0 (socket says 172.21.15.153:45472) 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07deffd640 1 -- 172.21.15.153:0/3547304633 learned_addr learned my addr 172.21.15.153:0/3547304633 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 1 --2- 172.21.15.153:0/3547304633 >> v2:172.21.15.181:3300/0 conn(0x7f07e0137c30 0x7f07e013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): get_auth_request con 0x7f07e0137c30 auth_method 0 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.269+0000 7f07df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07deffd640 1 -- 172.21.15.153:0/3547304633 >> v1:172.21.15.153:6789/0 conn(0x7f07e01318f0 legacy=0x7f07e0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07d67fc640 1 -- 172.21.15.153:0/3547304633 >> v1:172.21.15.181:6789/0 conn(0x7f07e009ea50 legacy=0x7f07e009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07de7fc640 1 -- 172.21.15.153:0/3547304633 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260371296 0 0) 0x7f07e0131cc0 con 0x7f07e01318f0 2024-09-15T03:31:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07de7fc640 10 monclient(hunting): my global_id is 54105 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07de7fc640 1 -- 172.21.15.153:0/3547304633 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc0034e0 con 0x7f07e01318f0 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 10 monclient(hunting): handle_auth_done global_id 44398 payload 274 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 1 -- 172.21.15.153:0/3547304633 >> v1:172.21.15.181:6789/0 conn(0x7f07e009ea50 legacy=0x7f07e009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 1 -- 172.21.15.153:0/3547304633 >> v1:172.21.15.153:6789/0 conn(0x7f07e01318f0 legacy=0x7f07e0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:08.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.271+0000 7f07df7fe640 1 -- 172.21.15.153:0/3547304633 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e009fe30 con 0x7f07e0137c30 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07df7fe640 1 -- 172.21.15.153:0/3547304633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0137c30 msgr2=0x7f07e013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07df7fe640 1 --2- 172.21.15.153:0/3547304633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0137c30 0x7f07e013a010 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009f60 tx=0x7f07d8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eaaf868fa8b431d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 1 -- 172.21.15.153:0/3547304633 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8007760 con 0x7f07e0137c30 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: dump: 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.283 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 1 -- 172.21.15.153:0/3547304633 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d802d050 con 0x7f07e0137c30 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07e5509640 5 monclient: authenticate success, global_id 44398 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07e5509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 1 -- 172.21.15.153:0/3547304633 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8007af0 con 0x7f07e0137c30 2024-09-15T03:31:08.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07e5509640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:08.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07e5509640 4 set_mon_vals no callback set 2024-09-15T03:31:08.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.274+0000 7f07de7fc640 10 monclient: dump: 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 10 monclient: shutdown 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 -- 172.21.15.153:0/3547304633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0137c30 msgr2=0x7f07e013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 --2- 172.21.15.153:0/3547304633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0137c30 0x7f07e013a010 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009f60 tx=0x7f07d8009780 comp rx=0 tx=0).stop 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 -- 172.21.15.153:0/3547304633 shutdown_connections 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 --2- 172.21.15.153:0/3547304633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0137c30 0x7f07e013a010 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 -- 172.21.15.153:0/3547304633 >> 172.21.15.153:0/3547304633 conn(0x7f07e0067110 msgr2=0x7f07e0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 -- 172.21.15.153:0/3547304633 shutdown_connections 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 1 -- 172.21.15.153:0/3547304633 wait complete. 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.275+0000 7f07e5509640 10 monclient: monmap: 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 Processor -- start 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 -- start start 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient: init 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 0x7f07e0132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): picked mon.noname-c con 0x7f07e009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e01318f0 0x7f07e0130480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): picked mon.noname-b con 0x7f07e01318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:08.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0137c30 0x7f07e0136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): picked mon.noname-a con 0x7f07e0137c30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0134750 con 0x7f07e0137c30 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0132740 con 0x7f07e01318f0 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e01311c0 con 0x7f07e009ea50 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.276+0000 7f07e5509640 10 monclient(hunting): authenticate will time out at 865.915981s 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 0x7f07e0132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 0x7f07e0132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56454/0 (socket says 172.21.15.153:56454) 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0137c30 0x7f07e0136e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 -- 172.21.15.153:0/750941345 learned_addr learned my addr 172.21.15.153:0/750941345 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): get_auth_request con 0x7f07e009ea50 auth_method 0 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): get_auth_request con 0x7f07e0137c30 auth_method 0 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e01318f0 0x7f07e0130480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): get_auth_request con 0x7f07e01318f0 auth_method 0 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient(hunting): handle_auth_done global_id 4979 payload 824 2024-09-15T03:31:08.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e01318f0 msgr2=0x7f07e0130480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e01318f0 0x7f07e0130480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0137c30 msgr2=0x7f07e0136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0137c30 0x7f07e0136e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 monclient: found mon.noname-c 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07deffd640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8009a20 con 0x7f07e009ea50 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.277+0000 7f07df7fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0137c30 0x7f07e0136e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07deffd640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 msgr2=0x7f07e0132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07deffd640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 0x7f07e0132b20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f07d000c790 tx=0x7f07d0007be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d000ce60 con 0x7f07e009ea50 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: dump: 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0005aa0 con 0x7f07e009ea50 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 5 monclient: authenticate success, global_id 4979 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 10 monclient: _renew_subs 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e0133060 con 0x7f07e009ea50 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0005e20 con 0x7f07e009ea50 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07d77fe640 10 monclient: dump: 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.297 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 10 monclient: _renew_subs 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07ddffb640 4 set_mon_vals no callback set 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.279+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e01373d0 con 0x7f07e009ea50 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07e5509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07e5509640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 0x7f07e0140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07e5509640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0137650 con 0x7f07e0140b50 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f07d0007300 con 0x7f07e009ea50 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 0x7f07e0140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient: get_auth_request con 0x7f07e0140b50 auth_method 0 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d67fc640 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-09-15T03:31:08.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.280+0000 7f07d77fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07c0038740 0x7f07c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.281+0000 7f07d67fc640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07e0130be0 con 0x7f07e0140b50 2024-09-15T03:31:08.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.281+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f07d004b7b0 con 0x7f07e009ea50 2024-09-15T03:31:08.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.281+0000 7f07df7fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07c0038740 0x7f07c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.281+0000 7f07df7fe640 10 monclient: get_auth_request con 0x7f07c0038740 auth_method 0 2024-09-15T03:31:08.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d67fc640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 msgr2=0x7f07e0140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d67fc640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 0x7f07e0140f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07e006aa40 tx=0x7f07c8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c80074f0 con 0x7f07e0140b50 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 10 monclient: dump: 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07df7fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07c0038740 0x7f07c003abf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f07e0076600 tx=0x7f07d8037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f07c8005780 con 0x7f07e0140b50 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 msgr2=0x7f07e0140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 0x7f07e0140f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07e006aa40 tx=0x7f07c8009040 comp rx=0 tx=0).stop 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 msgr2=0x7f07e0140f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.283+0000 7f07d77fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0140b50 0x7f07e0140f20 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07e006aa40 tx=0x7f07c8009040 comp rx=0 tx=0).stop 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07e5509640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07e5509640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 0x7f07e012cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07e5509640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0065e00 con 0x7f07e0141460 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 0x7f07e012cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient: get_auth_request con 0x7f07e0141460 auth_method 0 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.289+0000 7f07d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.290+0000 7f07d67fc640 10 monclient(hunting): handle_auth_done global_id 54114 payload 824 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.290+0000 7f07d67fc640 1 -- 172.21.15.153:0/750941345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f07e004bf60 con 0x7f07e0141460 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d67fc640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 msgr2=0x7f07e012cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d67fc640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 0x7f07e012cc50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07e006aa40 tx=0x7f07c8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8021620 con 0x7f07e0141460 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:08.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 10 monclient: dump: 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.306 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f07c8021780 con 0x7f07e0141460 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 msgr2=0x7f07e012cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 0x7f07e012cc50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07e006aa40 tx=0x7f07c8018000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 msgr2=0x7f07e012cc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07d77fe640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07e0141460 0x7f07e012cc50 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07e006aa40 tx=0x7f07c8018000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:08.307 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:08.308 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.309 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.310 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:08.311 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.312 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.314 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.315 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:08.316 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:08.319 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07c0038740 msgr2=0x7f07c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07e5509640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07c0038740 0x7f07c003abf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f07e0076600 tx=0x7f07d8037000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07e5509640 10 monclient: shutdown 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07e5509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 msgr2=0x7f07e0132b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07e5509640 1 --2- 172.21.15.153:0/750941345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07e009ea50 0x7f07e0132b20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f07d000c790 tx=0x7f07d0007be0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07df7fe640 1 -- 172.21.15.153:0/750941345 reap_dead start 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.292+0000 7f07df7fe640 1 -- 172.21.15.153:0/750941345 reap_dead start 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.293+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 shutdown_connections 2024-09-15T03:31:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.293+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 >> 172.21.15.153:0/750941345 conn(0x7f07e0067110 msgr2=0x7f07e009d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.293+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 shutdown_connections 2024-09-15T03:31:08.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.293+0000 7f07e5509640 1 -- 172.21.15.153:0/750941345 wait complete. 2024-09-15T03:31:08.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:31:08.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:31:08.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:31:08.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.416+0000 7f5c62ec0640 10 monclient: get_monmap_and_config 2024-09-15T03:31:08.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.416+0000 7f5c62ec0640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.416+0000 7f5c62ec0640 10 monclient: monmap: 2024-09-15T03:31:08.417 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.417 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.417 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:08.417201+0000 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:08.417201+0000 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:08.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 Processor -- start 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 -- start start 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient: init 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 0x7f5c5c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5c06eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5c5c068a20 0x7f5c5c09c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5c068a20 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c5c06f490 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c069b80 con 0x7f5c5c06f490 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c099f90 con 0x7f5c5c068a20 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c069e00 con 0x7f5c5c06eb80 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c62ec0640 10 monclient(hunting): authenticate will time out at 866.056721s 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 0x7f5c5c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 0x7f5c5c06ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56462/0 (socket says 172.21.15.153:56462) 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 1 -- 172.21.15.153:0/251034017 learned_addr learned my addr 172.21.15.153:0/251034017 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): get_auth_request con 0x7f5c5c06eb80 auth_method 0 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c6145d640 1 --1- 172.21.15.153:0/251034017 >> v1:172.21.15.153:6789/0 conn(0x7f5c5c06f490 0x7f5c5c09d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45480/0 (socket says 172.21.15.153:45480) 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c5bfff640 1 --2- 172.21.15.153:0/251034017 >> v2:172.21.15.181:3300/0 conn(0x7f5c5c068a20 0x7f5c5c09c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c5bfff640 10 monclient(hunting): get_auth_request con 0x7f5c5c068a20 auth_method 0 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.417+0000 7f5c5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_done global_id 4982 payload 274 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 1 -- 172.21.15.153:0/251034017 >> v2:172.21.15.181:3300/0 conn(0x7f5c5c068a20 msgr2=0x7f5c5c09c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/251034017 >> v2:172.21.15.181:3300/0 conn(0x7f5c5c068a20 0x7f5c5c09c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 1 -- 172.21.15.153:0/251034017 >> v1:172.21.15.153:6789/0 conn(0x7f5c5c06f490 legacy=0x7f5c5c09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 1 monclient: found mon.noname-b 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c60c5c640 1 -- 172.21.15.153:0/251034017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c09ca80 con 0x7f5c5c06eb80 2024-09-15T03:31:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.418+0000 7f5c5bfff640 1 --2- 172.21.15.153:0/251034017 >> v2:172.21.15.181:3300/0 conn(0x7f5c5c068a20 0x7f5c5c09c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.420+0000 7f5c60c5c640 1 -- 172.21.15.153:0/251034017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 msgr2=0x7f5c5c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.420+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/251034017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 0x7f5c5c06ef50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009910 tx=0x7f5c500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82717a56c669f1d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 1 -- 172.21.15.153:0/251034017 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5002aea0 con 0x7f5c5c06eb80 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: dump: 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 5 monclient: authenticate success, global_id 4982 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 1 -- 172.21.15.153:0/251034017 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c50032440 con 0x7f5c5c06eb80 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 4 set_mon_vals no callback set 2024-09-15T03:31:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 1 -- 172.21.15.153:0/251034017 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c500327e0 con 0x7f5c5c06eb80 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c5b7fe640 10 monclient: dump: 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 10 monclient: shutdown 2024-09-15T03:31:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 -- 172.21.15.153:0/251034017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 msgr2=0x7f5c5c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/251034017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 0x7f5c5c06ef50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009910 tx=0x7f5c500091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 -- 172.21.15.153:0/251034017 shutdown_connections 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/251034017 >> v2:172.21.15.181:3300/0 conn(0x7f5c5c068a20 0x7f5c5c09c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/251034017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c06eb80 0x7f5c5c06ef50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 -- 172.21.15.153:0/251034017 >> 172.21.15.153:0/251034017 conn(0x7f5c5c067cd0 msgr2=0x7f5c5c0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 -- 172.21.15.153:0/251034017 shutdown_connections 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.421+0000 7f5c62ec0640 1 -- 172.21.15.153:0/251034017 wait complete. 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.422+0000 7f5c62ec0640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.422+0000 7f5c62ec0640 10 monclient: monmap: 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.422+0000 7f5c62ec0640 1 Processor -- start 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.422+0000 7f5c62ec0640 1 -- start start 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.422+0000 7f5c62ec0640 10 monclient: init 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c068a20 0x7f5c5c141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5c068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c5c06eb80 0x7f5c5c133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5c06eb80 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 0x7f5c5c11d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f5c5c06f490 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c09cf40 con 0x7f5c5c06f490 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0a1f90 con 0x7f5c5c06eb80 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c141f80 con 0x7f5c5c068a20 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c62ec0640 10 monclient(hunting): authenticate will time out at 866.062178s 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c068a20 0x7f5c5c141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c068a20 0x7f5c5c141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56466/0 (socket says 172.21.15.153:56466) 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 0x7f5c5c11d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 1 -- 172.21.15.153:0/1639746615 learned_addr learned my addr 172.21.15.153:0/1639746615 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): get_auth_request con 0x7f5c5c068a20 auth_method 0 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): get_auth_request con 0x7f5c5c06f490 auth_method 0 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c5c06eb80 0x7f5c5c133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): get_auth_request con 0x7f5c5c06eb80 auth_method 0 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient(hunting): handle_auth_done global_id 54117 payload 824 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c068a20 msgr2=0x7f5c5c141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c068a20 0x7f5c5c141130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c5c06eb80 msgr2=0x7f5c5c133e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c5c06eb80 0x7f5c5c133e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 monclient: found mon.noname-a 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c6145d640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c50009450 con 0x7f5c5c06f490 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.423+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c068a20 0x7f5c5c141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c6145d640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 msgr2=0x7f5c5c11d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c6145d640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 0x7f5c5c11d940 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400c760 tx=0x7f5c54004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c540070f0 con 0x7f5c5c06f490 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: dump: 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54007270 con 0x7f5c5c06f490 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c62ec0640 5 monclient: authenticate success, global_id 54117 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54007630 con 0x7f5c5c06f490 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c62ec0640 10 monclient: _renew_subs 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c62ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c11ffe0 con 0x7f5c5c06f490 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c58ff9640 10 monclient: dump: 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.441 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.425+0000 7f5c5affd640 4 set_mon_vals no callback set 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 10 monclient: _renew_subs 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c11f410 con 0x7f5c5c06f490 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 0x7f5c5c11de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c1210e0 con 0x7f5c5c134360 2024-09-15T03:31:08.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5c5401b400 con 0x7f5c5c06f490 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 0x7f5c5c11de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient: get_auth_request con 0x7f5c5c134360 auth_method 0 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.426+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.427+0000 7f5c58ff9640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5c40038740 0x7f5c4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.427+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_done global_id 4988 payload 824 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.427+0000 7f5c60c5c640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c5c04bf60 con 0x7f5c5c134360 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.427+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f5c5404b9c0 con 0x7f5c5c06f490 2024-09-15T03:31:08.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.427+0000 7f5c5bfff640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5c40038740 0x7f5c4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.427+0000 7f5c5bfff640 10 monclient: get_auth_request con 0x7f5c40038740 auth_method 0 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c60c5c640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 msgr2=0x7f5c5c11de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 0x7f5c5c11de80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009910 tx=0x7f5c50032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5002aea0 con 0x7f5c5c134360 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 10 monclient: dump: 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.444 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c5bfff640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5c40038740 0x7f5c4003abf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c48006e70 tx=0x7f5c48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5c5003d600 con 0x7f5c5c134360 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 msgr2=0x7f5c5c11de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 0x7f5c5c11de80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009910 tx=0x7f5c50032000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 msgr2=0x7f5c5c11de80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:08.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.429+0000 7f5c58ff9640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c134360 0x7f5c5c11de80 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009910 tx=0x7f5c50032000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c62ec0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:08.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 0x7f5c5c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c62ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c065e00 con 0x7f5c5c11e3c0 2024-09-15T03:31:08.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 0x7f5c5c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient: get_auth_request con 0x7f5c5c11e3c0 auth_method 0 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 10 monclient(hunting): handle_auth_done global_id 4991 payload 824 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.435+0000 7f5c60c5c640 1 -- 172.21.15.153:0/1639746615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c5c11e7c0 con 0x7f5c5c11e3c0 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c60c5c640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 msgr2=0x7f5c5c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c60c5c640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 0x7f5c5c096a60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5c50042000 tx=0x7f5c5003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50046d00 con 0x7f5c5c11e3c0 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 10 monclient: dump: 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.449 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f5c50046e60 con 0x7f5c5c11e3c0 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 msgr2=0x7f5c5c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 0x7f5c5c096a60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5c50042000 tx=0x7f5c5003e000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 msgr2=0x7f5c5c096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c58ff9640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c5c11e3c0 0x7f5c5c096a60 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5c50042000 tx=0x7f5c5003e000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.450 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:08.451 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.452 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:08.453 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:08.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.456 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.457 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.458 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.461 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.462 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.463 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:08.463 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:08.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5c40038740 msgr2=0x7f5c4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5c40038740 0x7f5c4003abf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c48006e70 tx=0x7f5c48008040 comp rx=0 tx=0).stop 2024-09-15T03:31:08.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 10 monclient: shutdown 2024-09-15T03:31:08.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 msgr2=0x7f5c5c11d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 0x7f5c5c11d940 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400c760 tx=0x7f5c54004e20 comp rx=0 tx=0).stop 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c6145d640 1 -- 172.21.15.153:0/1639746615 reap_dead start 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 shutdown_connections 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 --2- 172.21.15.153:0/1639746615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c5c06f490 0x7f5c5c11d940 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.438+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 >> 172.21.15.153:0/1639746615 conn(0x7f5c5c067cd0 msgr2=0x7f5c5c134b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.439+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 shutdown_connections 2024-09-15T03:31:08.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.439+0000 7f5c62ec0640 1 -- 172.21.15.153:0/1639746615 wait complete. 2024-09-15T03:31:08.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:31:08.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:31:08.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:31:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.546+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v67) v1 ==== 58+0+2 (secure 0 0 0) 0x7f4c64011520 con 0x7f4c70131b40 2024-09-15T03:31:08.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.546+0000 7f4c56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T03:31:08.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.546+0000 7f4c56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:08.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.562+0000 7f11562dd640 10 monclient: get_monmap_and_config 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.562+0000 7f11562dd640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.562+0000 7f11562dd640 10 monclient: monmap: 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:08.563433+0000 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:08.563433+0000 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:08.576 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.563+0000 7f11562dd640 1 Processor -- start 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.563+0000 7f11562dd640 1 -- start start 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.563+0000 7f11562dd640 10 monclient: init 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.563+0000 7f11562dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.563+0000 7f11562dd640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.563+0000 7f11562dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f1150131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): picked mon.noname-b con 0x7f1150131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f115009fda0 0x7f1150132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): picked mon.noname-c con 0x7f115009fda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f115009e9c0 0x7f115009ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): picked mon.noname-a con 0x7f115009e9c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11500a1180 con 0x7f115009e9c0 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11500a1460 con 0x7f115009fda0 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115009f340 con 0x7f1150131980 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f11562dd640 10 monclient(hunting): authenticate will time out at 866.203080s 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f1150131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f1150131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56490/0 (socket says 172.21.15.153:56490) 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f115009e9c0 0x7f115009ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 1 -- 172.21.15.153:0/1450355206 learned_addr learned my addr 172.21.15.153:0/1450355206 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): get_auth_request con 0x7f1150131980 auth_method 0 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): get_auth_request con 0x7f115009e9c0 auth_method 0 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114f7fe640 1 --2- 172.21.15.153:0/1450355206 >> v2:172.21.15.181:3300/0 conn(0x7f115009fda0 0x7f1150132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114f7fe640 10 monclient(hunting): get_auth_request con 0x7f115009fda0 auth_method 0 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient(hunting): handle_auth_done global_id 54120 payload 274 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 -- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 msgr2=0x7f1150131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 --2- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f1150131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 -- 172.21.15.153:0/1450355206 >> v2:172.21.15.181:3300/0 conn(0x7f115009fda0 msgr2=0x7f1150132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 --2- 172.21.15.153:0/1450355206 >> v2:172.21.15.181:3300/0 conn(0x7f115009fda0 0x7f1150132290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 monclient: found mon.noname-a 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f115487a640 1 -- 172.21.15.153:0/1450355206 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115009f4b0 con 0x7f115009e9c0 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114ffff640 1 --2- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f1150131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.564+0000 7f114f7fe640 1 --2- 172.21.15.153:0/1450355206 >> v2:172.21.15.181:3300/0 conn(0x7f115009fda0 0x7f1150132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f115487a640 1 -- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009e9c0 msgr2=0x7f115009ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f115487a640 1 --2- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009e9c0 0x7f115009ee00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11500a4f60 tx=0x7f1148009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92d7c372db9ce37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 1 -- 172.21.15.153:0/1450355206 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148007760 con 0x7f115009e9c0 2024-09-15T03:31:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: dump: 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 1 -- 172.21.15.153:0/1450355206 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114802d050 con 0x7f115009e9c0 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 5 monclient: authenticate success, global_id 54120 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 4 set_mon_vals no callback set 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 1 -- 172.21.15.153:0/1450355206 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148007af0 con 0x7f115009e9c0 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f114effd640 10 monclient: dump: 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.583 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 10 monclient: shutdown 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 1 -- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009e9c0 msgr2=0x7f115009ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.567+0000 7f11562dd640 1 --2- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009e9c0 0x7f115009ee00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11500a4f60 tx=0x7f1148009780 comp rx=0 tx=0).stop 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 -- 172.21.15.153:0/1450355206 shutdown_connections 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 --2- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009e9c0 0x7f115009ee00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 --2- 172.21.15.153:0/1450355206 >> v2:172.21.15.181:3300/0 conn(0x7f115009fda0 0x7f1150132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 --2- 172.21.15.153:0/1450355206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f1150131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 -- 172.21.15.153:0/1450355206 >> 172.21.15.153:0/1450355206 conn(0x7f1150067bd0 msgr2=0x7f11500a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 -- 172.21.15.153:0/1450355206 shutdown_connections 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 -- 172.21.15.153:0/1450355206 wait complete. 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 10 monclient: monmap: 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:08.585 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.568+0000 7f11562dd640 1 Processor -- start 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 -- start start 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient: init 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009e9c0 0x7f1150140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): picked mon.noname-b con 0x7f115009e9c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:08.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009fda0 0x7f1150141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): picked mon.noname-a con 0x7f115009fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f11501418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): picked mon.noname-c con 0x7f1150131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150120850 con 0x7f115009fda0 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500a4950 con 0x7f115009e9c0 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1150121bf0 con 0x7f1150131980 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f11562dd640 10 monclient(hunting): authenticate will time out at 866.208495s 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009fda0 0x7f1150141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f11501418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009fda0 0x7f1150141380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40058/0 (socket says 172.21.15.153:40058) 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 1 -- 172.21.15.153:0/3235265703 learned_addr learned my addr 172.21.15.153:0/3235265703 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): get_auth_request con 0x7f1150131980 auth_method 0 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): get_auth_request con 0x7f115009fda0 auth_method 0 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009e9c0 0x7f1150140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): get_auth_request con 0x7f115009e9c0 auth_method 0 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient(hunting): handle_auth_done global_id 4997 payload 824 2024-09-15T03:31:08.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009e9c0 msgr2=0x7f1150140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009e9c0 0x7f1150140e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009fda0 msgr2=0x7f1150141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009fda0 0x7f1150141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 monclient: found mon.noname-c 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f115487a640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1140009450 con 0x7f1150131980 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.569+0000 7f114f7fe640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115009fda0 0x7f1150141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f115487a640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 msgr2=0x7f11501418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f115487a640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f11501418c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1148031000 tx=0x7f1148037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148031b70 con 0x7f1150131980 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: dump: 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1148040070 con 0x7f1150131980 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 5 monclient: authenticate success, global_id 4997 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 10 monclient: _renew_subs 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1148009a20 con 0x7f1150131980 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148045430 con 0x7f1150131980 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f1137fff640 10 monclient: dump: 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.592 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 10 monclient: _renew_subs 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150121630 con 0x7f1150131980 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.572+0000 7f114e7fc640 4 set_mon_vals no callback set 2024-09-15T03:31:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f11562dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f115487a640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f11501418c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1148031000 tx=0x7f1148037000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f1148042030 mgrmap(e 5) v1 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f11562dd640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 0x7f1150099e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150065e00 con 0x7f1150099a80 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1148042030 con 0x7f1150131980 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 0x7f1150099e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient: get_auth_request con 0x7f1150099a80 auth_method 0 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f1137fff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f113c038740 0x7f113c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114f7fe640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f113c038740 0x7f113c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 10 monclient(hunting): handle_auth_done global_id 44419 payload 824 2024-09-15T03:31:08.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114ffff640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1150121e10 con 0x7f1150099a80 2024-09-15T03:31:08.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.573+0000 7f114f7fe640 10 monclient: get_auth_request con 0x7f113c038740 auth_method 0 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.575+0000 7f114ffff640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 msgr2=0x7f1150099e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.575+0000 7f114ffff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 0x7f1150099e50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1140011bf0 tx=0x7f11400091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114000fe90 con 0x7f1150099a80 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 10 monclient: dump: 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f114f7fe640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f113c038740 0x7f113c03abf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1144010370 tx=0x7f1144008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1140014650 con 0x7f1150099a80 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 msgr2=0x7f1150099e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 0x7f1150099e50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1140011bf0 tx=0x7f11400091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 msgr2=0x7f1150099e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.576+0000 7f1137fff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1150099a80 0x7f1150099e50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f11562dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f11562dd640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 0x7f115011df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f11562dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150121820 con 0x7f115009a390 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 0x7f115011df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient: get_auth_request con 0x7f115009a390 auth_method 0 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 10 monclient(hunting): handle_auth_done global_id 44422 payload 824 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.582+0000 7f114ffff640 1 -- 172.21.15.153:0/3235265703 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1150121a30 con 0x7f115009a390 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.584+0000 7f114ffff640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 msgr2=0x7f115011df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.584+0000 7f114ffff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 0x7f115011df90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1140011bf0 tx=0x7f114001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140025040 con 0x7f115009a390 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 10 monclient: dump: 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f1140014440 con 0x7f115009a390 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 msgr2=0x7f115011df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 0x7f115011df90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1140011bf0 tx=0x7f114001d000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 msgr2=0x7f115011df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f115009a390 0x7f115011df90 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1140011bf0 tx=0x7f114001d000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.602 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:08.615 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f113c038740 msgr2=0x7f113c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f113c038740 0x7f113c03abf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1144010370 tx=0x7f1144008040 comp rx=0 tx=0).stop 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 10 monclient: shutdown 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 msgr2=0x7f11501418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f115487a640 1 -- 172.21.15.153:0/3235265703 reap_dead start 2024-09-15T03:31:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f11501418c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1148031000 tx=0x7f1148037000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f1137fff640 1 -- 172.21.15.153:0/3235265703 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5403+0+0 (unknown 0 0 0) 0x7f114806d590 con 0x7f1150131980 2024-09-15T03:31:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 shutdown_connections 2024-09-15T03:31:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 --2- 172.21.15.153:0/3235265703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1150131980 0x7f11501418c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 >> 172.21.15.153:0/3235265703 conn(0x7f1150067bd0 msgr2=0x7f115009c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 shutdown_connections 2024-09-15T03:31:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.585+0000 7f11562dd640 1 -- 172.21.15.153:0/3235265703 wait complete. 2024-09-15T03:31:08.618 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:31:08.618 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:31:08.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:31:08.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.655+0000 7f4c7580f640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:31:08.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.655+0000 7f4c7580f640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T03:31:08.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.655+0000 7f4c7580f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:08.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:08.655+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f4c70141730 con 0x7f4c70131b40 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.711+0000 7f363e500640 10 monclient: get_monmap_and_config 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.711+0000 7f363e500640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.711+0000 7f363e500640 10 monclient: monmap: 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:08.711838+0000 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:08.711838+0000 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:08.712 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:08.713 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:08.713 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:08.713 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:08.713 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.711+0000 7f363e500640 1 Processor -- start 2024-09-15T03:31:08.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.711+0000 7f363e500640 1 -- start start 2024-09-15T03:31:08.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.711+0000 7f363e500640 10 monclient: init 2024-09-15T03:31:08.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 0x7f3638131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): picked mon.noname-a con 0x7f36381318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 0x7f363809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): picked mon.noname-b con 0x7f363809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3638068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3638131cc0 con 0x7f3638068d90 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363809fbf0 con 0x7f36381318f0 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363809fe30 con 0x7f363809ea50 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f363e500640 10 monclient(hunting): authenticate will time out at 866.351448s 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 0x7f3638131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 0x7f363809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 0x7f363809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56514/0 (socket says 172.21.15.153:56514) 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 0x7f3638131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40060/0 (socket says 172.21.15.153:40060) 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 -- 172.21.15.153:0/1607724359 learned_addr learned my addr 172.21.15.153:0/1607724359 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): get_auth_request con 0x7f363809ea50 auth_method 0 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): get_auth_request con 0x7f36381318f0 auth_method 0 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient(hunting): handle_auth_done global_id 5000 payload 274 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 -- 172.21.15.153:0/1607724359 >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 msgr2=0x7f3638131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 --2- 172.21.15.153:0/1607724359 >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 0x7f3638131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 -- 172.21.15.153:0/1607724359 >> v1:172.21.15.181:6789/0 conn(0x7f3638068d90 legacy=0x7f36380691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:08.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f36377fe640 1 -- 172.21.15.153:0/1607724359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36380a1210 con 0x7f363809ea50 2024-09-15T03:31:08.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.712+0000 7f3637fff640 1 --2- 172.21.15.153:0/1607724359 >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 0x7f3638131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:08.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.715+0000 7f36377fe640 1 -- 172.21.15.153:0/1607724359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 msgr2=0x7f363809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.715+0000 7f36377fe640 1 --2- 172.21.15.153:0/1607724359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 0x7f363809ee20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f362c009910 tx=0x7f362c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b69b996a687ea0c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 1 -- 172.21.15.153:0/1607724359 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c032440 con 0x7f363809ea50 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: dump: 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.726 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 5 monclient: authenticate success, global_id 5000 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 1 -- 172.21.15.153:0/1607724359 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c0325a0 con 0x7f363809ea50 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 1 -- 172.21.15.153:0/1607724359 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c032940 con 0x7f363809ea50 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 4 set_mon_vals no callback set 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f3636ffd640 10 monclient: dump: 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 10 monclient: shutdown 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 -- 172.21.15.153:0/1607724359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 msgr2=0x7f363809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 --2- 172.21.15.153:0/1607724359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 0x7f363809ee20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f362c009910 tx=0x7f362c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 -- 172.21.15.153:0/1607724359 shutdown_connections 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 --2- 172.21.15.153:0/1607724359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f363809ea50 0x7f363809ee20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 --2- 172.21.15.153:0/1607724359 >> v2:172.21.15.153:3300/0 conn(0x7f36381318f0 0x7f3638131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 -- 172.21.15.153:0/1607724359 >> 172.21.15.153:0/1607724359 conn(0x7f3638067110 msgr2=0x7f3638067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 -- 172.21.15.153:0/1607724359 shutdown_connections 2024-09-15T03:31:08.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.716+0000 7f363e500640 1 -- 172.21.15.153:0/1607724359 wait complete. 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.717+0000 7f363e500640 10 monclient: build_initial_monmap 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.717+0000 7f363e500640 10 monclient: monmap: 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.717+0000 7f363e500640 1 Processor -- start 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.717+0000 7f363e500640 1 -- start start 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.717+0000 7f363e500640 10 monclient: init 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 0x7f363809a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): picked mon.noname-a con 0x7f3638068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f363809ea50 0x7f3638130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): picked mon.noname-b con 0x7f363809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36381318f0 0x7f363811b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): picked mon.noname-c con 0x7f36381318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363809bf10 con 0x7f3638068d90 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363809e790 con 0x7f363809ea50 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363811c950 con 0x7f36381318f0 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363e500640 10 monclient(hunting): authenticate will time out at 866.357221s 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 0x7f363809a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 0x7f363809a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40064/0 (socket says 172.21.15.153:40064) 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 -- 172.21.15.153:0/2182234878 learned_addr learned my addr 172.21.15.153:0/2182234878 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): get_auth_request con 0x7f3638068d90 auth_method 0 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36381318f0 0x7f363811b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): get_auth_request con 0x7f36381318f0 auth_method 0 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f363809ea50 0x7f3638130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): get_auth_request con 0x7f363809ea50 auth_method 0 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f363ca9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f36377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient(hunting): handle_auth_done global_id 54129 payload 824 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 -- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36381318f0 msgr2=0x7f363811b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36381318f0 0x7f363811b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 -- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f363809ea50 msgr2=0x7f3638130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f363809ea50 0x7f3638130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 monclient: found mon.noname-a 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.718+0000 7f3637fff640 1 -- 172.21.15.153:0/2182234878 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3628009390 con 0x7f3638068d90 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f3637fff640 1 -- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 msgr2=0x7f363809a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f3637fff640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 0x7f363809a9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f362800ec80 tx=0x7f36280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362800eec0 con 0x7f3638068d90 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: dump: 2024-09-15T03:31:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: _finish_auth 0 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362800cdd0 con 0x7f3638068d90 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f363e500640 5 monclient: authenticate success, global_id 54129 2024-09-15T03:31:08.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628013680 con 0x7f3638068d90 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f363e500640 10 monclient: _renew_subs 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f363e500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c009450 con 0x7f3638068d90 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f361ffff640 10 monclient: dump: 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:08.737 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.720+0000 7f36367fc640 4 set_mon_vals no callback set 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 10 monclient: _renew_subs 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36381413a0 con 0x7f3638068d90 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f36280139f0 con 0x7f3638068d90 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.721+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363811e1a0 con 0x7f3638068d90 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.722+0000 7f361ffff640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3624038740 0x7f362403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.722+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5403+0+0 (secure 0 0 0) 0x7f362804c190 con 0x7f3638068d90 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.722+0000 7f36377fe640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3624038740 0x7f362403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.722+0000 7f36377fe640 10 monclient: get_auth_request con 0x7f3624038740 auth_method 0 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.724+0000 7f36377fe640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3624038740 0x7f362403abf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3638064050 tx=0x7f362c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:08.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.726+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f36280236b0 con 0x7f3638068d90 2024-09-15T03:31:08.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.726+0000 7f361ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:08.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.726+0000 7f361ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:08.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.941+0000 7f363e500640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:31:08.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.941+0000 7f363e500640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:31:08.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.941+0000 7f363e500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:08.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.941+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f363811de40 con 0x7f3638068d90 2024-09-15T03:31:08.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.942+0000 7f361ffff640 1 -- 172.21.15.153:0/2182234878 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f36280232f0 con 0x7f3638068d90 2024-09-15T03:31:08.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.942+0000 7f361ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:31:08.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.942+0000 7f361ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:08.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.943+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.943+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=411857536,config=3140514009,health=2157093694,kv=3235807718,logm=400213307}) 2024-09-15T03:31:08.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.943+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=4019180528,config=3140514009,health=1886805112,kv=3235807718,logm=949375685}) 2024-09-15T03:31:08.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.943+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.943+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=411857536,config=3140514009,health=2157093694,kv=3235807718,logm=400213307}) 2024-09-15T03:31:08.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.943+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=213386942,config=3140514009,health=1886805112,kv=3235807718,logm=82640810}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.944+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.944+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436413780}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.944+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224735388}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.944+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.944+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436413780}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.944+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000359824}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.945+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.945+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294705826}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.945+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=864635928}) 2024-09-15T03:31:08.946 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.945+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.947 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.945+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294705826}) 2024-09-15T03:31:08.947 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.945+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713278338}) 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3624038740 msgr2=0x7f362403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3624038740 0x7f362403abf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3638064050 tx=0x7f362c032000 comp rx=0 tx=0).stop 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 10 monclient: shutdown 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 msgr2=0x7f363809a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 0x7f363809a9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f362800ec80 tx=0x7f36280090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 shutdown_connections 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3624038740 0x7f362403abf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36381318f0 0x7f363811b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f363809ea50 0x7f3638130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 --2- 172.21.15.153:0/2182234878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3638068d90 0x7f363809a9a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 >> 172.21.15.153:0/2182234878 conn(0x7f3638067110 msgr2=0x7f36380a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 shutdown_connections 2024-09-15T03:31:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:08.945+0000 7f363e500640 1 -- 172.21.15.153:0/2182234878 wait complete. 2024-09-15T03:31:08.949 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.946+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.949 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.946+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705778046}) 2024-09-15T03:31:08.949 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.946+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3485777370}) 2024-09-15T03:31:08.949 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.946+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.950 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.946+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705778046}) 2024-09-15T03:31:08.950 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.946+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830525635}) 2024-09-15T03:31:08.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.955+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.955+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805532304}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.955+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012638730}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.955+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.955+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805532304}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.955+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2261860437}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.956+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.956+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760337157}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.956+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281340212}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.956+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.956+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760337157}) 2024-09-15T03:31:08.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.956+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3029064152}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.957+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.957+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275271108}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.957+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4142371569}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.957+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.957+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275271108}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.957+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79374880}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=997330111}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269510069}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=997330111}) 2024-09-15T03:31:08.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3263280010}) 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=14} crc {logm=1534936582,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3525685874,mon_config_key=4117641549,monmap=244636553,osd_metadata=205594370,osd_pg_creating=1590106205,osd_snap=454132730,osdmap=2006157754}) 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=1127210488,mdsmap=1531151339,mgr=2572398688,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=948757977,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1590106205,osd_snap=454132730,osdmap=769660661}) 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=14} crc {logm=1534936582,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3525685874,mon_config_key=4117641549,monmap=244636553,osd_metadata=205594370,osd_pg_creating=1590106205,osd_snap=454132730,osdmap=2006157754}) 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.958+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=13} crc {logm=1181875733,mdsmap=1531151339,mgr=2938112866,mgr_metadata=2280977772,mgrstat=948757977,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1590106205,osd_snap=454132730,osdmap=1191637307}) 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.960+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.960+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=485981617}) 2024-09-15T03:31:08.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.960+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1569393338}) 2024-09-15T03:31:08.963 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.960+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:31:08.963 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.960+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=485981617}) 2024-09-15T03:31:08.963 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:31:08.960+0000 7fb75cdf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3963577873}) 2024-09-15T03:31:09.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.556+0000 7f4c56ffd640 1 -- 172.21.15.181:0/1318174056 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v68) v1 ==== 58+0+2 (secure 0 0 0) 0x7f4c6403c030 con 0x7f4c70131b40 2024-09-15T03:31:09.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.556+0000 7f4c56ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T03:31:09.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.556+0000 7f4c56ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038770 msgr2=0x7f4c5c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038770 0x7f4c5c03ac20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4c60031000 tx=0x7f4c60032000 comp rx=0 tx=0).stop 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 10 monclient: shutdown 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 msgr2=0x7f4c7012ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 0x7f4c7012ed10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400d890 tx=0x7f4c64008040 comp rx=0 tx=0).stop 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 shutdown_connections 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c5c038770 0x7f4c5c03ac20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c70132450 0x7f4c70098b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c70131b40 0x7f4c7012ed10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 --2- 172.21.15.181:0/1318174056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7009f240 0x7f4c701411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.558+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 >> 172.21.15.181:0/1318174056 conn(0x7f4c700a5100 msgr2=0x7f4c700a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.559+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 shutdown_connections 2024-09-15T03:31:09.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.559+0000 7f4c7580f640 1 -- 172.21.15.181:0/1318174056 wait complete. 2024-09-15T03:31:09.571 INFO:tasks.workunit.client.0.smithi181.stderr:+ o2=9 2024-09-15T03:31:09.571 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-15T03:31:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.652+0000 7fd121e6f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.652+0000 7fd121e6f640 10 monclient: build_initial_monmap 2024-09-15T03:31:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.652+0000 7fd121e6f640 10 monclient: monmap: 2024-09-15T03:31:09.653 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:09.653232+0000 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:09.653232+0000 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:09.654 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 1 Processor -- start 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 1 -- start start 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient: init 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd11c098c70 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd11c099c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd11c1383e0 0x7fd11c13a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:09.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient(hunting): picked mon.noname-c con 0x7fd11c1383e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c09f680 con 0x7fd11c098c70 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c09f420 con 0x7fd11c099c70 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c068a20 con 0x7fd11c1383e0 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd121e6f640 10 monclient(hunting): authenticate will time out at 874.134783s 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd11c1383e0 0x7fd11c13a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd11c1383e0 0x7fd11c13a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:56420/0 (socket says 172.21.15.181:56420) 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 1 -- 172.21.15.181:0/2723015820 learned_addr learned my addr 172.21.15.181:0/2723015820 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 10 monclient(hunting): get_auth_request con 0x7fd11c1383e0 auth_method 0 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:09.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.653+0000 7fd11affd640 1 --1- 172.21.15.181:0/2723015820 >> v1:172.21.15.181:6789/0 conn(0x7fd11c099c70 0x7fd11c136cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:59062/0 (socket says 172.21.15.181:59062) 2024-09-15T03:31:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.654+0000 7fd11bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.654+0000 7fd11bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:09.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.654+0000 7fd11bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11affd640 1 -- 172.21.15.181:0/2723015820 >> v1:172.21.15.181:6789/0 conn(0x7fd11c099c70 legacy=0x7fd11c136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 10 monclient(hunting): handle_auth_done global_id 44428 payload 274 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 1 -- 172.21.15.181:0/2723015820 >> v1:172.21.15.181:6789/0 conn(0x7fd11c099c70 legacy=0x7fd11c136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 1 -- 172.21.15.181:0/2723015820 >> v1:172.21.15.153:6789/0 conn(0x7fd11c098c70 legacy=0x7fd11c099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11b7fe640 1 -- 172.21.15.181:0/2723015820 >> v1:172.21.15.153:6789/0 conn(0x7fd11c098c70 legacy=0x7fd11c099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 1 monclient: found mon.noname-c 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:09.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.656+0000 7fd11bfff640 1 -- 172.21.15.181:0/2723015820 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c096a30 con 0x7fd11c1383e0 2024-09-15T03:31:09.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11bfff640 1 -- 172.21.15.181:0/2723015820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 msgr2=0x7fd11c13a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11bfff640 1 --2- 172.21.15.181:0/2723015820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c13a7c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd11c096c20 tx=0x7fd114009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e229f74f59b2a3f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 1 -- 172.21.15.181:0/2723015820 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114007610 con 0x7fd11c1383e0 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: dump: 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 1 -- 172.21.15.181:0/2723015820 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd114007770 con 0x7fd11c1383e0 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 5 monclient: authenticate success, global_id 44428 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 1 -- 172.21.15.181:0/2723015820 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114007b10 con 0x7fd11c1383e0 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 4 set_mon_vals no callback set 2024-09-15T03:31:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd11a7fc640 10 monclient: dump: 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 10 monclient: shutdown 2024-09-15T03:31:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/2723015820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 msgr2=0x7fd11c13a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 --2- 172.21.15.181:0/2723015820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c13a7c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd11c096c20 tx=0x7fd114009840 comp rx=0 tx=0).stop 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/2723015820 shutdown_connections 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 --2- 172.21.15.181:0/2723015820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c13a7c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/2723015820 >> 172.21.15.181:0/2723015820 conn(0x7fd11c070e90 msgr2=0x7fd11c0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/2723015820 shutdown_connections 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/2723015820 wait complete. 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.660+0000 7fd121e6f640 10 monclient: build_initial_monmap 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.660+0000 7fd121e6f640 10 monclient: monmap: 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.660+0000 7fd121e6f640 1 Processor -- start 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.660+0000 7fd121e6f640 1 -- start start 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.660+0000 7fd121e6f640 10 monclient: init 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd11c098c70 0x7fd11c137da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): picked mon.noname-a con 0x7fd11c098c70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:09.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11c099c70 0x7fd11c11e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): picked mon.noname-c con 0x7fd11c099c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c11ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): picked mon.noname-b con 0x7fd11c1383e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c122000 con 0x7fd11c098c70 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c06ec10 con 0x7fd11c1383e0 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c0978b0 con 0x7fd11c099c70 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd121e6f640 10 monclient(hunting): authenticate will time out at 874.142288s 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c11ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c11ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:56430/0 (socket says 172.21.15.181:56430) 2024-09-15T03:31:09.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 -- 172.21.15.181:0/4275320657 learned_addr learned my addr 172.21.15.181:0/4275320657 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): get_auth_request con 0x7fd11c1383e0 auth_method 0 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd11c098c70 0x7fd11c137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11c099c70 0x7fd11c11e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): get_auth_request con 0x7fd11c098c70 auth_method 0 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:09.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): get_auth_request con 0x7fd11c099c70 auth_method 0 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient(hunting): handle_auth_done global_id 44431 payload 824 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 -- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11c099c70 msgr2=0x7fd11c11e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11c099c70 0x7fd11c11e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 -- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd11c098c70 msgr2=0x7fd11c137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd11c098c70 0x7fd11c137da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 monclient: found mon.noname-b 2024-09-15T03:31:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.661+0000 7fd11bfff640 1 -- 172.21.15.181:0/4275320657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd114009ae0 con 0x7fd11c1383e0 2024-09-15T03:31:09.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.663+0000 7fd11bfff640 1 -- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 msgr2=0x7fd11c11ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:09.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.663+0000 7fd11bfff640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c11ec40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd11400b0d0 tx=0x7fd114037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:09.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114032850 con 0x7fd11c1383e0 2024-09-15T03:31:09.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:09.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: dump: 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:09.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1140329b0 con 0x7fd11c1383e0 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 5 monclient: authenticate success, global_id 44431 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 10 monclient: _renew_subs 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114032d70 con 0x7fd11c1383e0 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c121d20 con 0x7fd11c1383e0 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 10 monclient: dump: 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:09.676 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd119ffb640 4 set_mon_vals no callback set 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 10 monclient: _renew_subs 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c137240 con 0x7fd11c1383e0 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd114003020 con 0x7fd11c1383e0 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.664+0000 7fd121e6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd121e6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd121e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c0008d0 con 0x7fd11c1383e0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd1137fe640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0fc038790 0x7fd0fc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5518+0+0 (secure 0 0 0) 0x7fd11406c570 con 0x7fd11c1383e0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd11b7fe640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0fc038790 0x7fd0fc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.665+0000 7fd11b7fe640 10 monclient: get_auth_request con 0x7fd0fc038790 auth_method 0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.667+0000 7fd11b7fe640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0fc038790 0x7fd0fc03ac40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd10c007fd0 tx=0x7fd10c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.669+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd114030560 con 0x7fd11c1383e0 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.669+0000 7fd1137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:09.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.669+0000 7fd1137fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:09.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.826+0000 7fd121e6f640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:09.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.826+0000 7fd121e6f640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:09.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.826+0000 7fd121e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:09.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:09.826+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7fd11c069890 con 0x7fd11c1383e0 2024-09-15T03:31:09.957 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:31:09.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.058+0000 7f58fed5d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.058+0000 7f58fed5d640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.058+0000 7f58fed5d640 10 monclient: monmap: 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:10.058919+0000 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:10.058919+0000 2024-09-15T03:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.058+0000 7f58fed5d640 1 Processor -- start 2024-09-15T03:31:10.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 1 -- start start 2024-09-15T03:31:10.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient: init 2024-09-15T03:31:10.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T03:31:10.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f58f81318f0 0x7f58f8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): picked mon.noname-a con 0x7f58f81318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:10.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58f809ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:10.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f8136c20 0x7f58f8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): picked mon.noname-b con 0x7f58f8136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f8131cc0 con 0x7f58f809ea50 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f809f530 con 0x7f58f81318f0 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f809fbf0 con 0x7f58f8136c20 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fed5d640 10 monclient(hunting): authenticate will time out at 867.698530s 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f58f81318f0 0x7f58f8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58f7fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f58f809ea50 0x7f58f809ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45494/0 (socket says 172.21.15.153:45494) 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58f7fff640 1 -- 172.21.15.153:0/2709513443 learned_addr learned my addr 172.21.15.153:0/2709513443 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): get_auth_request con 0x7f58f81318f0 auth_method 0 2024-09-15T03:31:10.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f8136c20 0x7f58f8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): get_auth_request con 0x7f58f8136c20 auth_method 0 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_done global_id 54138 payload 274 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 1 -- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f8136c20 msgr2=0x7f58f8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f8136c20 0x7f58f8139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 1 -- 172.21.15.153:0/2709513443 >> v1:172.21.15.153:6789/0 conn(0x7f58f809ea50 legacy=0x7f58f809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fcaf9640 1 -- 172.21.15.153:0/2709513443 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f809fe30 con 0x7f58f81318f0 2024-09-15T03:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.059+0000 7f58fd2fa640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f8136c20 0x7f58f8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58fcaf9640 1 -- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f81318f0 msgr2=0x7f58f8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58fcaf9640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f81318f0 0x7f58f8131f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009850 tx=0x7f58f00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2d6467b583bfc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 1 -- 172.21.15.153:0/2709513443 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f002adf0 con 0x7f58f81318f0 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: dump: 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58fed5d640 5 monclient: authenticate success, global_id 54138 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 1 -- 172.21.15.153:0/2709513443 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0032440 con 0x7f58f81318f0 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58fed5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58fed5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58fed5d640 4 set_mon_vals no callback set 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 1 -- 172.21.15.153:0/2709513443 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f00327e0 con 0x7f58f81318f0 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.062+0000 7f58f77fe640 10 monclient: dump: 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 10 monclient: shutdown 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 -- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f81318f0 msgr2=0x7f58f8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f81318f0 0x7f58f8131f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009850 tx=0x7f58f00090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 -- 172.21.15.153:0/2709513443 shutdown_connections 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f8136c20 0x7f58f8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 --2- 172.21.15.153:0/2709513443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f81318f0 0x7f58f8131f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 -- 172.21.15.153:0/2709513443 >> 172.21.15.153:0/2709513443 conn(0x7f58f8067110 msgr2=0x7f58f8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 -- 172.21.15.153:0/2709513443 shutdown_connections 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.063+0000 7f58fed5d640 1 -- 172.21.15.153:0/2709513443 wait complete. 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 10 monclient: monmap: 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:10.077 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 1 Processor -- start 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 1 -- start start 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 10 monclient: init 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 0x7f58f8130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.064+0000 7f58fed5d640 10 monclient(hunting): picked mon.noname-c con 0x7f58f809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58f81318f0 0x7f58f80997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): picked mon.noname-b con 0x7f58f81318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f8136c20 0x7f58f8099cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): picked mon.noname-a con 0x7f58f8136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:10.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f8141f20 con 0x7f58f8136c20 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f809b2e0 con 0x7f58f81318f0 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f811cd60 con 0x7f58f809ea50 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fed5d640 10 monclient(hunting): authenticate will time out at 867.704059s 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 0x7f58f8130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 0x7f58f8130970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56552/0 (socket says 172.21.15.153:56552) 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 -- 172.21.15.153:0/3770356305 learned_addr learned my addr 172.21.15.153:0/3770356305 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): get_auth_request con 0x7f58f809ea50 auth_method 0 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f8136c20 0x7f58f8099cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): get_auth_request con 0x7f58f8136c20 auth_method 0 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58f7fff640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58f81318f0 0x7f58f80997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient(hunting): handle_auth_done global_id 5012 payload 824 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 -- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58f81318f0 msgr2=0x7f58f80997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58f81318f0 0x7f58f80997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 -- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f8136c20 msgr2=0x7f58f8099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f8136c20 0x7f58f8099cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 monclient: found mon.noname-c 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fcaf9640 1 -- 172.21.15.153:0/3770356305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e0009390 con 0x7f58f809ea50 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58f7fff640 10 monclient: get_auth_request con 0x7f58f81318f0 auth_method 0 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58f7fff640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58f81318f0 0x7f58f80997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.065+0000 7f58fd2fa640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f8136c20 0x7f58f8099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fcaf9640 1 -- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 msgr2=0x7f58f8130970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fcaf9640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 0x7f58f8130970 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f58f00098b0 tx=0x7f58f0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0031a10 con 0x7f58f809ea50 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: dump: 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0031b70 con 0x7f58f809ea50 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 5 monclient: authenticate success, global_id 5012 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 10 monclient: _renew_subs 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f0009390 con 0x7f58f809ea50 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f003d420 con 0x7f58f809ea50 2024-09-15T03:31:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f4ff9640 10 monclient: dump: 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 10 monclient: _renew_subs 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58f6ffd640 4 set_mon_vals no callback set 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.067+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f811b720 con 0x7f58f809ea50 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.068+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f58f0039040 con 0x7f58f809ea50 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.068+0000 7f58fed5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.068+0000 7f58fed5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.068+0000 7f58fed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.068+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f804bf60 con 0x7f58f809ea50 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.069+0000 7f58f4ff9640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58d8038740 0x7f58d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.069+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5518+0+0 (secure 0 0 0) 0x7f58f006bbc0 con 0x7f58f809ea50 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.069+0000 7f58f7fff640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58d8038740 0x7f58d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.069+0000 7f58f7fff640 10 monclient: get_auth_request con 0x7f58d8038740 auth_method 0 2024-09-15T03:31:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.071+0000 7f58f7fff640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58d8038740 0x7f58d803abf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f58f80a4ef0 tx=0x7f58e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.073+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f58f00394d0 con 0x7f58f809ea50 2024-09-15T03:31:10.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.073+0000 7f58f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.073+0000 7f58f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:10.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58fed5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58fed5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58fed5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f58f80638d0 con 0x7f58f809ea50 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58f4ff9640 1 -- 172.21.15.153:0/3770356305 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f58f0037080 con 0x7f58f809ea50 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.287+0000 7f58f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:10.289 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58d8038740 msgr2=0x7f58d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58d8038740 0x7f58d803abf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f58f80a4ef0 tx=0x7f58e8009040 comp rx=0 tx=0).stop 2024-09-15T03:31:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 10 monclient: shutdown 2024-09-15T03:31:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 msgr2=0x7f58f8130970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 0x7f58f8130970 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f58f00098b0 tx=0x7f58f0032000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 shutdown_connections 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58d8038740 0x7f58d803abf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f8136c20 0x7f58f8099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58f81318f0 0x7f58f80997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 --2- 172.21.15.153:0/3770356305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f809ea50 0x7f58f8130970 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.289+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 >> 172.21.15.153:0/3770356305 conn(0x7f58f8067110 msgr2=0x7f58f80a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.290+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 shutdown_connections 2024-09-15T03:31:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.290+0000 7f58fed5d640 1 -- 172.21.15.153:0/3770356305 wait complete. 2024-09-15T03:31:10.301 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:31:10.301 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:31:10.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:31:10.301 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:31:10.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.394+0000 7efdc344f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.394+0000 7efdc344f640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.394+0000 7efdc344f640 10 monclient: monmap: 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:10.395711+0000 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:10.395711+0000 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:10.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:10.397 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:10.397 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:10.397 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.395+0000 7efdc344f640 1 Processor -- start 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.395+0000 7efdc344f640 1 -- start start 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.395+0000 7efdc344f640 10 monclient: init 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdbc131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efdbc09fe10 0x7efdbc09ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): picked mon.noname-a con 0x7efdbc09fe10 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efdbc09ef70 0x7efdbc138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): picked mon.noname-c con 0x7efdbc09ef70 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc0a14d0 con 0x7efdbc131980 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdbc1325b0 con 0x7efdbc09fe10 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdbc0a11f0 con 0x7efdbc09ef70 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc344f640 10 monclient(hunting): authenticate will time out at 868.035333s 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efdbc09fe10 0x7efdbc09ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efdbc09fe10 0x7efdbc09ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40096/0 (socket says 172.21.15.153:40096) 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 -- 172.21.15.153:0/2028390938 learned_addr learned my addr 172.21.15.153:0/2028390938 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): get_auth_request con 0x7efdbc09fe10 auth_method 0 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc19ec640 1 --2- 172.21.15.153:0/2028390938 >> v2:172.21.15.181:3300/0 conn(0x7efdbc09ef70 0x7efdbc138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient(hunting): handle_auth_done global_id 54144 payload 274 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 -- 172.21.15.153:0/2028390938 >> v2:172.21.15.181:3300/0 conn(0x7efdbc09ef70 msgr2=0x7efdbc138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 --2- 172.21.15.153:0/2028390938 >> v2:172.21.15.181:3300/0 conn(0x7efdbc09ef70 0x7efdbc138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 -- 172.21.15.153:0/2028390938 >> v1:172.21.15.181:6789/0 conn(0x7efdbc131980 legacy=0x7efdbc131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.396+0000 7efdc09ea640 1 -- 172.21.15.153:0/2028390938 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc09f390 con 0x7efdbc09fe10 2024-09-15T03:31:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdc09ea640 1 -- 172.21.15.153:0/2028390938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc09fe10 msgr2=0x7efdbc09ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdc09ea640 1 --2- 172.21.15.153:0/2028390938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc09fe10 0x7efdbc09ea30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdbc06a9f0 tx=0x7efdb00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ba0be5e7df7ef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 1 -- 172.21.15.153:0/2028390938 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0032440 con 0x7efdbc09fe10 2024-09-15T03:31:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: dump: 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 1 -- 172.21.15.153:0/2028390938 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb00325a0 con 0x7efdbc09fe10 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdc344f640 5 monclient: authenticate success, global_id 54144 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdc344f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 1 -- 172.21.15.153:0/2028390938 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0032940 con 0x7efdbc09fe10 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdbbfff640 10 monclient: dump: 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdc344f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.399+0000 7efdc344f640 4 set_mon_vals no callback set 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 10 monclient: shutdown 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 -- 172.21.15.153:0/2028390938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc09fe10 msgr2=0x7efdbc09ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 --2- 172.21.15.153:0/2028390938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc09fe10 0x7efdbc09ea30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdbc06a9f0 tx=0x7efdb00091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 -- 172.21.15.153:0/2028390938 shutdown_connections 2024-09-15T03:31:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 --2- 172.21.15.153:0/2028390938 >> v2:172.21.15.181:3300/0 conn(0x7efdbc09ef70 0x7efdbc138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 --2- 172.21.15.153:0/2028390938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc09fe10 0x7efdbc09ea30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 -- 172.21.15.153:0/2028390938 >> 172.21.15.153:0/2028390938 conn(0x7efdbc0a5050 msgr2=0x7efdbc0a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 -- 172.21.15.153:0/2028390938 shutdown_connections 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 1 -- 172.21.15.153:0/2028390938 wait complete. 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.400+0000 7efdc344f640 10 monclient: monmap: 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 Processor -- start 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 -- start start 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient: init 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 0x7efdbc1347f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): picked mon.noname-c con 0x7efdbc09ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc09fe10 0x7efdbc134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): picked mon.noname-b con 0x7efdbc09fe10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc131980 0x7efdbc135270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): picked mon.noname-a con 0x7efdbc131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc069eb0 con 0x7efdbc131980 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0a4290 con 0x7efdbc09fe10 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc137b60 con 0x7efdbc09ef70 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.401+0000 7efdc344f640 10 monclient(hunting): authenticate will time out at 868.040937s 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 0x7efdbc1347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc131980 0x7efdbc135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 0x7efdbc1347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56562/0 (socket says 172.21.15.153:56562) 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 -- 172.21.15.153:0/4151766908 learned_addr learned my addr 172.21.15.153:0/4151766908 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): get_auth_request con 0x7efdbc09ef70 auth_method 0 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): get_auth_request con 0x7efdbc131980 auth_method 0 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc09fe10 0x7efdbc134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): get_auth_request con 0x7efdbc09fe10 auth_method 0 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient(hunting): handle_auth_done global_id 5015 payload 824 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc09fe10 msgr2=0x7efdbc134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc09fe10 0x7efdbc134d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc131980 msgr2=0x7efdbc135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc131980 0x7efdbc135270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 monclient: found mon.noname-c 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc11eb640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0009450 con 0x7efdbc09ef70 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.402+0000 7efdc19ec640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc131980 0x7efdbc135270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc11eb640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 msgr2=0x7efdbc1347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc11eb640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 0x7efdbc1347f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efdb400b610 tx=0x7efdb40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400f7a0 con 0x7efdbc09ef70 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: dump: 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb400f920 con 0x7efdbc09ef70 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 5 monclient: authenticate success, global_id 5015 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 10 monclient: _renew_subs 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc138120 con 0x7efdbc09ef70 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400fce0 con 0x7efdbc09ef70 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdb97fa640 10 monclient: dump: 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 10 monclient: _renew_subs 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdbb7fe640 4 set_mon_vals no callback set 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.404+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc136c00 con 0x7efdbc09ef70 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efdb4010630 con 0x7efdbc09ef70 2024-09-15T03:31:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc344f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc344f640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 0x7efdbc098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc065e00 con 0x7efdbc1357b0 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 0x7efdbc098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient: get_auth_request con 0x7efdbc1357b0 auth_method 0 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdb97fa640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efda8038740 0x7efda803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5518+0+0 (secure 0 0 0) 0x7efdb404ba40 con 0x7efdbc09ef70 2024-09-15T03:31:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.405+0000 7efdc09ea640 10 monclient(hunting): handle_auth_done global_id 54150 payload 824 2024-09-15T03:31:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.406+0000 7efdc09ea640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdbc136df0 con 0x7efdbc1357b0 2024-09-15T03:31:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.406+0000 7efdc19ec640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efda8038740 0x7efda803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.406+0000 7efdc19ec640 10 monclient: get_auth_request con 0x7efda8038740 auth_method 0 2024-09-15T03:31:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdc09ea640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 msgr2=0x7efdbc098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdc09ea640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 0x7efdbc098a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdb0034000 tx=0x7efdb0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb003baa0 con 0x7efdbc1357b0 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 10 monclient: dump: 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efdb0040070 con 0x7efdbc1357b0 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 msgr2=0x7efdbc098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 0x7efdbc098a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdb0034000 tx=0x7efdb0032000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 msgr2=0x7efdbc098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdb97fa640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc1357b0 0x7efdbc098a70 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdb0034000 tx=0x7efdb0032000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.408+0000 7efdc19ec640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efda8038740 0x7efda803abf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efdac00d8f0 tx=0x7efdac007710 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc344f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc344f640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 0x7efdbc099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc344f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc137da0 con 0x7efdbc098fb0 2024-09-15T03:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 0x7efdbc099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient: get_auth_request con 0x7efdbc098fb0 auth_method 0 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 10 monclient(hunting): handle_auth_done global_id 54153 payload 824 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.414+0000 7efdc09ea640 1 -- 172.21.15.153:0/4151766908 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efdbc067f00 con 0x7efdbc098fb0 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.416+0000 7efdc09ea640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 msgr2=0x7efdbc099380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.416+0000 7efdc09ea640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 0x7efdbc099380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efdb0003080 tx=0x7efdb003e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0046910 con 0x7efdbc098fb0 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 10 monclient: dump: 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7efdb00503b0 con 0x7efdbc098fb0 2024-09-15T03:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 msgr2=0x7efdbc099380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 0x7efdbc099380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efdb0003080 tx=0x7efdb003e070 comp rx=0 tx=0).stop 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 msgr2=0x7efdbc099380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdb97fa640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdbc098fb0 0x7efdbc099380 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efdb0003080 tx=0x7efdb003e070 comp rx=0 tx=0).stop 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:31:10.429 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.430 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.431 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:10.432 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.433 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.435 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:10.437 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:10.440 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.441 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.441 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:10.441 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efda8038740 msgr2=0x7efda803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efda8038740 0x7efda803abf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efdac00d8f0 tx=0x7efdac007710 comp rx=0 tx=0).stop 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 10 monclient: shutdown 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 msgr2=0x7efdbc1347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 --2- 172.21.15.153:0/4151766908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdbc09ef70 0x7efdbc1347f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efdb400b610 tx=0x7efdb40090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc19ec640 1 -- 172.21.15.153:0/4151766908 reap_dead start 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc19ec640 1 -- 172.21.15.153:0/4151766908 reap_dead start 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 shutdown_connections 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 >> 172.21.15.153:0/4151766908 conn(0x7efdbc0a5050 msgr2=0x7efdbc09e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 shutdown_connections 2024-09-15T03:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.417+0000 7efdc344f640 1 -- 172.21.15.153:0/4151766908 wait complete. 2024-09-15T03:31:10.443 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:31:10.444 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:31:10.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:31:10.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.536+0000 7f1ec879d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.536+0000 7f1ec879d640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.536+0000 7f1ec879d640 10 monclient: monmap: 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:10.537658+0000 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:10.537658+0000 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:10.553 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.537+0000 7f1ec879d640 1 Processor -- start 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.537+0000 7f1ec879d640 1 -- start start 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.537+0000 7f1ec879d640 10 monclient: init 2024-09-15T03:31:10.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1ec0131980 0x7f1ec0131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec0131980 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1ec009fda0 0x7f1ec0132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec009fda0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ec009e9c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec009f510 con 0x7f1ec009e9c0 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00a1180 con 0x7f1ec009fda0 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00a1460 con 0x7f1ec0131980 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec879d640 10 monclient(hunting): authenticate will time out at 868.177196s 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1ec009fda0 0x7f1ec0132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1ec009fda0 0x7f1ec0132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40118/0 (socket says 172.21.15.153:40118) 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 -- 172.21.15.153:0/404403243 learned_addr learned my addr 172.21.15.153:0/404403243 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): get_auth_request con 0x7f1ec009fda0 auth_method 0 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 1 --2- 172.21.15.153:0/404403243 >> v2:172.21.15.181:3300/0 conn(0x7f1ec0131980 0x7f1ec0131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): get_auth_request con 0x7f1ec0131980 auth_method 0 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_done global_id 54156 payload 274 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 -- 172.21.15.153:0/404403243 >> v2:172.21.15.181:3300/0 conn(0x7f1ec0131980 msgr2=0x7f1ec0131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/404403243 >> v2:172.21.15.181:3300/0 conn(0x7f1ec0131980 0x7f1ec0131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 -- 172.21.15.153:0/404403243 >> v1:172.21.15.181:6789/0 conn(0x7f1ec009e9c0 legacy=0x7f1ec009ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.538+0000 7f1ec5d38640 1 -- 172.21.15.153:0/404403243 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0139cd0 con 0x7f1ec009fda0 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5d38640 1 -- 172.21.15.153:0/404403243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009fda0 msgr2=0x7f1ec0132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/404403243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009fda0 0x7f1ec0132290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00a4fb0 tx=0x7f1eb40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf568bd1f8c0c3e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 1 -- 172.21.15.153:0/404403243 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4032440 con 0x7f1ec009fda0 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: dump: 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 1 -- 172.21.15.153:0/404403243 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb40325a0 con 0x7f1ec009fda0 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec879d640 5 monclient: authenticate success, global_id 54156 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec879d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec879d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 1 -- 172.21.15.153:0/404403243 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4032940 con 0x7f1ec009fda0 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec879d640 4 set_mon_vals no callback set 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.541+0000 7f1ec5537640 10 monclient: dump: 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.562 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 10 monclient: shutdown 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 -- 172.21.15.153:0/404403243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009fda0 msgr2=0x7f1ec0132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 --2- 172.21.15.153:0/404403243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009fda0 0x7f1ec0132290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00a4fb0 tx=0x7f1eb40091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 -- 172.21.15.153:0/404403243 shutdown_connections 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 --2- 172.21.15.153:0/404403243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009fda0 0x7f1ec0132290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 --2- 172.21.15.153:0/404403243 >> v2:172.21.15.181:3300/0 conn(0x7f1ec0131980 0x7f1ec0131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 -- 172.21.15.153:0/404403243 >> 172.21.15.153:0/404403243 conn(0x7f1ec0067bd0 msgr2=0x7f1ec00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 -- 172.21.15.153:0/404403243 shutdown_connections 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 1 -- 172.21.15.153:0/404403243 wait complete. 2024-09-15T03:31:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.542+0000 7f1ec879d640 10 monclient: monmap: 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:10.564 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 Processor -- start 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 -- start start 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient: init 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 0x7f1ec01374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec009e9c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec009fda0 0x7f1ec0130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec009fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ec0131980 0x7f1ec01311a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec0131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec009e490 con 0x7f1ec009e9c0 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0136bc0 con 0x7f1ec0131980 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0136570 con 0x7f1ec009fda0 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec879d640 10 monclient(hunting): authenticate will time out at 868.182585s 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 0x7f1ec01374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 0x7f1ec01374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40122/0 (socket says 172.21.15.153:40122) 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 -- 172.21.15.153:0/2984253836 learned_addr learned my addr 172.21.15.153:0/2984253836 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): get_auth_request con 0x7f1ec009e9c0 auth_method 0 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec009fda0 0x7f1ec0130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): get_auth_request con 0x7f1ec009fda0 auth_method 0 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6d3a640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ec0131980 0x7f1ec01311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6d3a640 10 monclient(hunting): get_auth_request con 0x7f1ec0131980 auth_method 0 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient(hunting): handle_auth_done global_id 54159 payload 824 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec009fda0 msgr2=0x7f1ec0130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec009fda0 0x7f1ec0130c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ec0131980 msgr2=0x7f1ec01311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ec0131980 0x7f1ec01311a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.543+0000 7f1ec6539640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4009450 con 0x7f1ec009e9c0 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.544+0000 7f1ec6d3a640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ec0131980 0x7f1ec01311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec6539640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 msgr2=0x7f1ec01374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec6539640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 0x7f1ec01374d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800ea20 tx=0x7f1eb80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8009600 con 0x7f1ec009e9c0 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: dump: 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb800cc20 con 0x7f1ec009e9c0 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 5 monclient: authenticate success, global_id 54159 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 10 monclient: _renew_subs 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec0136850 con 0x7f1ec009e9c0 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8013470 con 0x7f1ec009e9c0 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ebe7fc640 10 monclient: dump: 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 10 monclient: _renew_subs 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec4d36640 4 set_mon_vals no callback set 2024-09-15T03:31:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.546+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec0135610 con 0x7f1ec009e9c0 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec879d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec879d640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 0x7f1ec0141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0065e00 con 0x7f1ec01411c0 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1eb8013860 con 0x7f1ec009e9c0 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 0x7f1ec0141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient: get_auth_request con 0x7f1ec01411c0 auth_method 0 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ebe7fc640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1eac038740 0x7f1eac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5518+0+0 (secure 0 0 0) 0x7f1eb804c130 con 0x7f1ec009e9c0 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_done global_id 5021 payload 824 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec5d38640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ec004bf60 con 0x7f1ec01411c0 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec6d3a640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1eac038740 0x7f1eac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.547+0000 7f1ec6d3a640 10 monclient: get_auth_request con 0x7f1eac038740 auth_method 0 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.549+0000 7f1ec5d38640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 msgr2=0x7f1ec0141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.549+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 0x7f1ec0141590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4034000 tx=0x7f1eb4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.549+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4031b00 con 0x7f1ec01411c0 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.549+0000 7f1ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.549+0000 7f1ebe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.549+0000 7f1ebe7fc640 10 monclient: dump: 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ec6d3a640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1eac038740 0x7f1eac03abf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1ec009c0a0 tx=0x7f1eb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1eb4031c60 con 0x7f1ec01411c0 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 msgr2=0x7f1ec0141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 0x7f1ec0141590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4034000 tx=0x7f1eb4032000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 msgr2=0x7f1ec0141590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.550+0000 7f1ebe7fc640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec01411c0 0x7f1ec0141590 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4034000 tx=0x7f1eb4032000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.557+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v69) v1 ==== 203+0+0 (secure 0 0 0) 0x7fd114008040 con 0x7fd11c1383e0 2024-09-15T03:31:10.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.557+0000 7fd1137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:10.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.557+0000 7fd1137fe640 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-15T03:31:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec879d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec879d640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 0x7f1ec0098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec879d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00638d0 con 0x7f1ec0141ad0 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 0x7f1ec0098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient: get_auth_request con 0x7f1ec0141ad0 auth_method 0 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 10 monclient(hunting): handle_auth_done global_id 5024 payload 824 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.556+0000 7f1ec5d38640 1 -- 172.21.15.153:0/2984253836 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1ec0141ed0 con 0x7f1ec0141ad0 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.559+0000 7f1ec5d38640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 msgr2=0x7f1ec0098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.559+0000 7f1ec5d38640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 0x7f1ec0098a70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0134a00 tx=0x7f1eb403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4055070 con 0x7f1ec0141ad0 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 10 monclient: dump: 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f1eb403ce40 con 0x7f1ec0141ad0 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 msgr2=0x7f1ec0098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 0x7f1ec0098a70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0134a00 tx=0x7f1eb403d000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 msgr2=0x7f1ec0098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.560+0000 7f1ebe7fc640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec0141ad0 0x7f1ec0098a70 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0134a00 tx=0x7f1eb403d000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:10.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.584 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:10.586 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.587 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.588 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:10.596 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:10.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1eac038740 msgr2=0x7f1eac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1eac038740 0x7f1eac03abf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1ec009c0a0 tx=0x7f1eb0009040 comp rx=0 tx=0).stop 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 10 monclient: shutdown 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 msgr2=0x7f1ec01374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 --2- 172.21.15.153:0/2984253836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec009e9c0 0x7f1ec01374d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800ea20 tx=0x7f1eb80090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec6d3a640 1 -- 172.21.15.153:0/2984253836 reap_dead start 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec6d3a640 1 -- 172.21.15.153:0/2984253836 reap_dead start 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 shutdown_connections 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 >> 172.21.15.153:0/2984253836 conn(0x7f1ec0067bd0 msgr2=0x7f1ec009d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 shutdown_connections 2024-09-15T03:31:10.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.561+0000 7f1ec879d640 1 -- 172.21.15.153:0/2984253836 wait complete. 2024-09-15T03:31:10.598 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:31:10.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:31:10.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:31:10.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.655+0000 7fd121e6f640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:10.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.655+0000 7fd121e6f640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:10.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.655+0000 7fd121e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:10.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.655+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7fd11c064a20 con 0x7fd11c1383e0 2024-09-15T03:31:10.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.656+0000 7fd1137fe640 1 -- 172.21.15.181:0/4275320657 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v69) v1 ==== 201+0+0 (secure 0 0 0) 0x7fd114040020 con 0x7fd11c1383e0 2024-09-15T03:31:10.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.656+0000 7fd1137fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:10.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.656+0000 7fd1137fe640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T03:31:10.658 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0fc038790 msgr2=0x7fd0fc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0fc038790 0x7fd0fc03ac40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd10c007fd0 tx=0x7fd10c009040 comp rx=0 tx=0).stop 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 10 monclient: shutdown 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 msgr2=0x7fd11c11ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c11ec40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd11400b0d0 tx=0x7fd114037000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 shutdown_connections 2024-09-15T03:31:10.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0fc038790 0x7fd0fc03ac40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11c1383e0 0x7fd11c11ec40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11c099c70 0x7fd11c11e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 --2- 172.21.15.181:0/4275320657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd11c098c70 0x7fd11c137da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.658+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 >> 172.21.15.181:0/4275320657 conn(0x7fd11c070e90 msgr2=0x7fd11c0a15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 shutdown_connections 2024-09-15T03:31:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.659+0000 7fd121e6f640 1 -- 172.21.15.181:0/4275320657 wait complete. 2024-09-15T03:31:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-15T03:31:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.692+0000 7fc642234640 10 monclient: get_monmap_and_config 2024-09-15T03:31:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.692+0000 7fc642234640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.692+0000 7fc642234640 10 monclient: monmap: 2024-09-15T03:31:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:10.693258+0000 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:10.693258+0000 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:10.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 Processor -- start 2024-09-15T03:31:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 -- start start 2024-09-15T03:31:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient: init 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc63c096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc63c0971e0 0x7fc63c0975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): picked mon.noname-c con 0x7fc63c0971e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc63c133b30 0x7fc63c135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): picked mon.noname-a con 0x7fc63c133b30 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63c099620 con 0x7fc63c096e10 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c099340 con 0x7fc63c133b30 2024-09-15T03:31:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c099900 con 0x7fc63c0971e0 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc642234640 10 monclient(hunting): authenticate will time out at 868.332804s 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc63c133b30 0x7fc63c135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc63c096e10 0x7fc63c09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45504/0 (socket says 172.21.15.153:45504) 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc63c133b30 0x7fc63c135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40124/0 (socket says 172.21.15.153:40124) 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63b7fe640 1 -- 172.21.15.153:0/2367328293 learned_addr learned my addr 172.21.15.153:0/2367328293 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63bfff640 10 monclient(hunting): get_auth_request con 0x7fc63c133b30 auth_method 0 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.693+0000 7fc63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63affd640 1 --2- 172.21.15.153:0/2367328293 >> v2:172.21.15.181:3300/0 conn(0x7fc63c0971e0 0x7fc63c0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63affd640 10 monclient(hunting): get_auth_request con 0x7fc63c0971e0 auth_method 0 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 10 monclient(hunting): handle_auth_done global_id 54162 payload 274 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 1 -- 172.21.15.153:0/2367328293 >> v2:172.21.15.181:3300/0 conn(0x7fc63c0971e0 msgr2=0x7fc63c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2367328293 >> v2:172.21.15.181:3300/0 conn(0x7fc63c0971e0 0x7fc63c0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 1 -- 172.21.15.153:0/2367328293 >> v1:172.21.15.153:6789/0 conn(0x7fc63c096e10 legacy=0x7fc63c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.694+0000 7fc63bfff640 1 -- 172.21.15.153:0/2367328293 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c09ac70 con 0x7fc63c133b30 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.696+0000 7fc63bfff640 1 -- 172.21.15.153:0/2367328293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c133b30 msgr2=0x7fc63c135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.696+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2367328293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c133b30 0x7fc63c135f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc63c09b020 tx=0x7fc634009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd381261a3cd889 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 1 -- 172.21.15.153:0/2367328293 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634007760 con 0x7fc63c133b30 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: dump: 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 1 -- 172.21.15.153:0/2367328293 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63402d050 con 0x7fc63c133b30 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 5 monclient: authenticate success, global_id 54162 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 1 -- 172.21.15.153:0/2367328293 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634007af0 con 0x7fc63c133b30 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 4 set_mon_vals no callback set 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc63a7fc640 10 monclient: dump: 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 10 monclient: shutdown 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 -- 172.21.15.153:0/2367328293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c133b30 msgr2=0x7fc63c135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 --2- 172.21.15.153:0/2367328293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c133b30 0x7fc63c135f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc63c09b020 tx=0x7fc634009780 comp rx=0 tx=0).stop 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 -- 172.21.15.153:0/2367328293 shutdown_connections 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 --2- 172.21.15.153:0/2367328293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c133b30 0x7fc63c135f10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 --2- 172.21.15.153:0/2367328293 >> v2:172.21.15.181:3300/0 conn(0x7fc63c0971e0 0x7fc63c0975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 -- 172.21.15.153:0/2367328293 >> 172.21.15.153:0/2367328293 conn(0x7fc63c0672b0 msgr2=0x7fc63c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 -- 172.21.15.153:0/2367328293 shutdown_connections 2024-09-15T03:31:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.697+0000 7fc642234640 1 -- 172.21.15.153:0/2367328293 wait complete. 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.698+0000 7fc642234640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.698+0000 7fc642234640 10 monclient: monmap: 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.698+0000 7fc642234640 1 Processor -- start 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.698+0000 7fc642234640 1 -- start start 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.698+0000 7fc642234640 10 monclient: init 2024-09-15T03:31:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 0x7fc63c117f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): picked mon.noname-a con 0x7fc63c096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 0x7fc63c118480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): picked mon.noname-c con 0x7fc63c0971e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c133b30 0x7fc63c1189c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): picked mon.noname-b con 0x7fc63c133b30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c119de0 con 0x7fc63c096e10 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c1332f0 con 0x7fc63c133b30 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c09b050 con 0x7fc63c0971e0 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc642234640 10 monclient(hunting): authenticate will time out at 868.338328s 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 0x7fc63c117f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 0x7fc63c117f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40130/0 (socket says 172.21.15.153:40130) 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 -- 172.21.15.153:0/2710176932 learned_addr learned my addr 172.21.15.153:0/2710176932 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): get_auth_request con 0x7fc63c096e10 auth_method 0 2024-09-15T03:31:10.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63affd640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 0x7fc63c118480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63affd640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 msgr2=0x7fc63c118480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63affd640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 msgr2=0x7fc63c118480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63affd640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 0x7fc63c118480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63affd640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 0x7fc63c118480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c133b30 0x7fc63c1189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): get_auth_request con 0x7fc63c133b30 auth_method 0 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient(hunting): handle_auth_done global_id 54165 payload 824 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 msgr2=0x7fc63c118480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc63c0971e0 0x7fc63c118480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c133b30 msgr2=0x7fc63c1189c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c133b30 0x7fc63c1189c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.699+0000 7fc63b7fe640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634009a20 con 0x7fc63c096e10 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.701+0000 7fc63b7fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 msgr2=0x7fc63c117f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.701+0000 7fc63b7fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 0x7fc63c117f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00c5e0 tx=0x7fc62c004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c005cb0 con 0x7fc63c096e10 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: dump: 2024-09-15T03:31:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c005e30 con 0x7fc63c096e10 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 5 monclient: authenticate success, global_id 54165 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c007320 con 0x7fc63c096e10 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 10 monclient: _renew_subs 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c12f3f0 con 0x7fc63c096e10 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 10 monclient: dump: 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.719 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc639ffb640 4 set_mon_vals no callback set 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 10 monclient: _renew_subs 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c119860 con 0x7fc63c096e10 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.702+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc62c01b3b0 con 0x7fc63c096e10 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc6237fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc628038740 0x7fc62803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7fc62c04b3e0 con 0x7fc63c096e10 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc642234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc642234640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 0x7fc63c06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c065f20 con 0x7fc63c118f00 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63affd640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc628038740 0x7fc62803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63affd640 10 monclient: get_auth_request con 0x7fc628038740 auth_method 0 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 0x7fc63c06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient: get_auth_request con 0x7fc63c118f00 auth_method 0 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 10 monclient(hunting): handle_auth_done global_id 44449 payload 824 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.703+0000 7fc63bfff640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc63c04bf60 con 0x7fc63c118f00 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.705+0000 7fc63affd640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc628038740 0x7fc62803abf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc630006bf0 tx=0x7fc630008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc63bfff640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 msgr2=0x7fc63c06c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 0x7fc63c06c6c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc634034000 tx=0x7fc634037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634041070 con 0x7fc63c118f00 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 10 monclient: dump: 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc63403ce00 con 0x7fc63c118f00 2024-09-15T03:31:10.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:10.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 msgr2=0x7fc63c06c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 0x7fc63c06c6c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc634034000 tx=0x7fc634037000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 msgr2=0x7fc63c06c6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:10.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.706+0000 7fc6237fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c118f00 0x7fc63c06c6c0 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc634034000 tx=0x7fc634037000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc642234640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc642234640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 0x7fc63c06d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc642234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c0638d0 con 0x7fc63c06cc00 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 0x7fc63c06d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient: get_auth_request con 0x7fc63c06cc00 auth_method 0 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 10 monclient(hunting): handle_auth_done global_id 44452 payload 824 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.712+0000 7fc63bfff640 1 -- 172.21.15.153:0/2710176932 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc63c11b3b0 con 0x7fc63c06cc00 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc63bfff640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 msgr2=0x7fc63c06d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc63bfff640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 0x7fc63c06d000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097c0 tx=0x7fc634041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634055070 con 0x7fc63c06cc00 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 10 monclient: dump: 2024-09-15T03:31:10.727 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fc634031bd0 con 0x7fc63c06cc00 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:10.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 msgr2=0x7fc63c06d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 0x7fc63c06d000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097c0 tx=0x7fc634041000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 msgr2=0x7fc63c06d000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc6237fe640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc63c06cc00 0x7fc63c06d000 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097c0 tx=0x7fc634041000 comp rx=0 tx=0).stop 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:31:10.729 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.730 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.731 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:10.732 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:10.733 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.734 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.735 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:10.736 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:10.738 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.740 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:10.741 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:10.742 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc628038740 msgr2=0x7fc62803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc642234640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc628038740 0x7fc62803abf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc630006bf0 tx=0x7fc630008040 comp rx=0 tx=0).stop 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc642234640 10 monclient: shutdown 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc642234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 msgr2=0x7fc63c117f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc63bfff640 1 -- 172.21.15.153:0/2710176932 reap_dead start 2024-09-15T03:31:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.715+0000 7fc642234640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 0x7fc63c117f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00c5e0 tx=0x7fc62c004960 comp rx=0 tx=0).stop 2024-09-15T03:31:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.716+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 shutdown_connections 2024-09-15T03:31:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.716+0000 7fc642234640 1 --2- 172.21.15.153:0/2710176932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc63c096e10 0x7fc63c117f40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.716+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 >> 172.21.15.153:0/2710176932 conn(0x7fc63c0672b0 msgr2=0x7fc63c1319e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.716+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 shutdown_connections 2024-09-15T03:31:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.716+0000 7fc642234640 1 -- 172.21.15.153:0/2710176932 wait complete. 2024-09-15T03:31:10.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:31:10.744 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:31:10.745 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:31:10.745 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:31:10.745 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:31:10.748 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:31:10.748 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.745+0000 7f3fe9cc9640 10 monclient: get_monmap_and_config 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.745+0000 7f3fe9cc9640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.745+0000 7f3fe9cc9640 10 monclient: monmap: 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:10.746464+0000 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:10.746464+0000 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.752 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 1 Processor -- start 2024-09-15T03:31:10.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 1 -- start start 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 10 monclient: init 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3fe4096a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3fe4099f00 0x7f3fe4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe4099f00 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.746+0000 7f3fe9cc9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe409d4e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe409cb80 con 0x7f3fe4096a30 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe4098c50 con 0x7f3fe409d4e0 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe4098f30 con 0x7f3fe4099f00 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe9cc9640 10 monclient(hunting): authenticate will time out at 875.228043s 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe37fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3fe4096a30 0x7f3fe4096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:59066/0 (socket says 172.21.15.181:59066) 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe37fe640 1 -- 172.21.15.181:0/706260196 learned_addr learned my addr 172.21.15.181:0/706260196 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.181:3300/0 conn(0x7f3fe4099f00 0x7f3fe4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe2ffd640 10 monclient(hunting): get_auth_request con 0x7f3fe4099f00 auth_method 0 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): get_auth_request con 0x7f3fe409d4e0 auth_method 0 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_done global_id 54168 payload 274 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.181:3300/0 conn(0x7f3fe4099f00 msgr2=0x7f3fe4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.181:3300/0 conn(0x7f3fe4099f00 0x7f3fe4133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v1:172.21.15.181:6789/0 conn(0x7f3fe4096a30 legacy=0x7f3fe4096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 1 monclient: found mon.noname-a 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.747+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40a1ba0 con 0x7f3fe409d4e0 2024-09-15T03:31:10.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.751+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.751+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.751+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f3fe409d190 tx=0x7f3fdc009780 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.751+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:10.758 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:31:10.758 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:31:10.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:10.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.889+0000 7f9e75709640 10 monclient: get_monmap_and_config 2024-09-15T03:31:10.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.889+0000 7f9e75709640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.889+0000 7f9e75709640 10 monclient: monmap: 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:10.890430+0000 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:10.890430+0000 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:10.891 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:10.892 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:10.892 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:10.892 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 Processor -- start 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 -- start start 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient: init 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e70131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9e7009e910 0x7f9e7009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): picked mon.noname-a con 0x7f9e7009e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:10.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 0x7f9e70138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7009f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7009faf0 con 0x7f9e70131ab0 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e70132590 con 0x7f9e7009e910 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7009fd30 con 0x7f9e7009f250 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e75709640 10 monclient(hunting): authenticate will time out at 868.529769s 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6e7fc640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9e7009e910 msgr2=0x7f9e7009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:10.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6e7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9e7009e910 0x7f9e7009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 0x7f9e70138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 0x7f9e70138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56600/0 (socket says 172.21.15.153:56600) 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 1 -- 172.21.15.153:0/1573752447 learned_addr learned my addr 172.21.15.153:0/1573752447 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 10 monclient(hunting): get_auth_request con 0x7f9e7009f250 auth_method 0 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.890+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_done global_id 5027 payload 274 2024-09-15T03:31:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 1 -- 172.21.15.153:0/1573752447 >> v2:172.21.15.153:3300/0 conn(0x7f9e7009e910 msgr2=0x7f9e7009ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 1 --2- 172.21.15.153:0/1573752447 >> v2:172.21.15.153:3300/0 conn(0x7f9e7009e910 0x7f9e7009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 1 -- 172.21.15.153:0/1573752447 >> v1:172.21.15.181:6789/0 conn(0x7f9e70131ab0 legacy=0x7f9e70131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.891+0000 7f9e6f7fe640 1 -- 172.21.15.153:0/1573752447 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700a1110 con 0x7f9e7009f250 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.893+0000 7f9e6f7fe640 1 -- 172.21.15.153:0/1573752447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 msgr2=0x7f9e70138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.893+0000 7f9e6f7fe640 1 --2- 172.21.15.153:0/1573752447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 0x7f9e70138bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009f20 tx=0x7f9e68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d678cbd9cf3ee780 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 1 -- 172.21.15.153:0/1573752447 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68007610 con 0x7f9e7009f250 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:10.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: dump: 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 1 -- 172.21.15.153:0/1573752447 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6802d050 con 0x7f9e7009f250 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 5 monclient: authenticate success, global_id 5027 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 1 -- 172.21.15.153:0/1573752447 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e680079a0 con 0x7f9e7009f250 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 4 set_mon_vals no callback set 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e6dffb640 10 monclient: dump: 2024-09-15T03:31:10.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 10 monclient: shutdown 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:10.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 -- 172.21.15.153:0/1573752447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 msgr2=0x7f9e70138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 --2- 172.21.15.153:0/1573752447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 0x7f9e70138bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009f20 tx=0x7f9e68009840 comp rx=0 tx=0).stop 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 -- 172.21.15.153:0/1573752447 shutdown_connections 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 --2- 172.21.15.153:0/1573752447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009f250 0x7f9e70138bc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 --2- 172.21.15.153:0/1573752447 >> v2:172.21.15.153:3300/0 conn(0x7f9e7009e910 0x7f9e7009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 -- 172.21.15.153:0/1573752447 >> 172.21.15.153:0/1573752447 conn(0x7f9e70067d30 msgr2=0x7f9e700a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 -- 172.21.15.153:0/1573752447 shutdown_connections 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.894+0000 7f9e75709640 1 -- 172.21.15.153:0/1573752447 wait complete. 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.895+0000 7f9e75709640 10 monclient: build_initial_monmap 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.895+0000 7f9e75709640 10 monclient: monmap: 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:10.906 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.895+0000 7f9e75709640 1 Processor -- start 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.895+0000 7f9e75709640 1 -- start start 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.895+0000 7f9e75709640 10 monclient: init 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 0x7f9e701310d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e7009f250 0x7f9e7009a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7009f250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e70131ab0 0x7f9e70096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): picked mon.noname-a con 0x7f9e70131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70097bf0 con 0x7f9e70131ab0 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70097d80 con 0x7f9e7009f250 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70099dc0 con 0x7f9e7009e910 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e75709640 10 monclient(hunting): authenticate will time out at 868.535149s 2024-09-15T03:31:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6f7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e70131ab0 msgr2=0x7f9e70096a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e70131ab0 0x7f9e70096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 0x7f9e701310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 0x7f9e701310d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56604/0 (socket says 172.21.15.153:56604) 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 -- 172.21.15.153:0/1033840941 learned_addr learned my addr 172.21.15.153:0/1033840941 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): get_auth_request con 0x7f9e7009e910 auth_method 0 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e7009f250 0x7f9e7009a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): get_auth_request con 0x7f9e7009f250 auth_method 0 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:10.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient(hunting): handle_auth_done global_id 5030 payload 824 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 -- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e7009f250 msgr2=0x7f9e7009a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e7009f250 0x7f9e7009a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 -- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e70131ab0 msgr2=0x7f9e70096a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e70131ab0 0x7f9e70096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 monclient: found mon.noname-c 2024-09-15T03:31:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.896+0000 7f9e6effd640 1 -- 172.21.15.153:0/1033840941 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68009ae0 con 0x7f9e7009e910 2024-09-15T03:31:10.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.898+0000 7f9e6effd640 1 -- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 msgr2=0x7f9e701310d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.898+0000 7f9e6effd640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 0x7f9e701310d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c5e0 tx=0x7f9e60007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60012840 con 0x7f9e7009e910 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: dump: 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e600129a0 con 0x7f9e7009e910 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60012da0 con 0x7f9e7009e910 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 10 monclient: dump: 2024-09-15T03:31:10.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 5 monclient: authenticate success, global_id 5030 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 10 monclient: _renew_subs 2024-09-15T03:31:10.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e70099ae0 con 0x7f9e7009e910 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e6d7fa640 4 set_mon_vals no callback set 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 10 monclient: _renew_subs 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7009a9d0 con 0x7f9e7009e910 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.899+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9e60011590 con 0x7f9e7009e910 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e75709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e75709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e75709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7004bf60 con 0x7f9e7009e910 2024-09-15T03:31:10.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e56ffd640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e5c0386f0 0x7f9e5c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f9e6004bfe0 con 0x7f9e7009e910 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e6e7fc640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e5c0386f0 0x7f9e5c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.900+0000 7f9e6e7fc640 10 monclient: get_auth_request con 0x7f9e5c0386f0 auth_method 0 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.902+0000 7f9e6e7fc640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e5c0386f0 0x7f9e5c03aba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9e7009e4e0 tx=0x7f9e640090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.902+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.905+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9e60010aa0 con 0x7f9e7009e910 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.905+0000 7f9e56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:10.905+0000 7f9e56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.952+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:10.952+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:31:11.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e75709640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:11.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e75709640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:11.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e75709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:11.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e7009a1e0 con 0x7f9e7009e910 2024-09-15T03:31:11.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e56ffd640 1 -- 172.21.15.153:0/1033840941 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9e60016070 con 0x7f9e7009e910 2024-09-15T03:31:11.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:11.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.120+0000 7f9e56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:11.122 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e5c0386f0 msgr2=0x7f9e5c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e5c0386f0 0x7f9e5c03aba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9e7009e4e0 tx=0x7f9e640090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 10 monclient: shutdown 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 msgr2=0x7f9e701310d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 0x7f9e701310d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c5e0 tx=0x7f9e60007ed0 comp rx=0 tx=0).stop 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 shutdown_connections 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e5c0386f0 0x7f9e5c03aba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:11.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e70131ab0 0x7f9e70096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:11.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e7009f250 0x7f9e7009a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:11.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 --2- 172.21.15.153:0/1033840941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e7009e910 0x7f9e701310d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:11.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.122+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 >> 172.21.15.153:0/1033840941 conn(0x7f9e70067d30 msgr2=0x7f9e7009d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:11.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.123+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 shutdown_connections 2024-09-15T03:31:11.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:11.123+0000 7f9e75709640 1 -- 172.21.15.153:0/1033840941 wait complete. 2024-09-15T03:31:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:11.353+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:11.353+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:31:12.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:12.154+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:12.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:12.154+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:31:13.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:13.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:13.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:13.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:13.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.747+0000 7f3fe17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:13.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.755+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:13.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:13.755+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-15T03:31:14.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:14.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.239+0000 7f7e2655d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.239+0000 7f7e2655d640 10 monclient: build_initial_monmap 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.239+0000 7f7e2655d640 10 monclient: monmap: 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:14.240691+0000 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:14.240691+0000 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:14.241 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:14.242 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:14.242 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:14.242 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.240+0000 7f7e2655d640 1 Processor -- start 2024-09-15T03:31:14.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.240+0000 7f7e2655d640 1 -- start start 2024-09-15T03:31:14.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.240+0000 7f7e2655d640 10 monclient: init 2024-09-15T03:31:14.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:14.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e20131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e2009fe10 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7e2009f140 0x7f7e20139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2009f140 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e200a14d0 con 0x7f7e2009fe10 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e201325b0 con 0x7f7e20131980 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2009f560 con 0x7f7e2009f140 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e2655d640 10 monclient(hunting): authenticate will time out at 871.880288s 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e24afa640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f7e2009f140 msgr2=0x7f7e20139cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e1f7fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f7e2009fe10 legacy=0x7f7e2009ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e24afa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7e2009f140 0x7f7e20139cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e1ffff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f7e20131980 0x7f7e20131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:43410/0 (socket says 172.21.15.153:43410) 2024-09-15T03:31:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.241+0000 7f7e1ffff640 1 -- 172.21.15.153:0/3288261573 learned_addr learned my addr 172.21.15.153:0/3288261573 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.243+0000 7f7e1ffff640 1 -- 172.21.15.153:0/3288261573 >> v1:172.21.15.181:6789/0 conn(0x7f7e20131980 legacy=0x7f7e20131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776579628 0 0) 0x7f7e201325b0 con 0x7f7e20131980 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): my global_id is 44461 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e080034e0 con 0x7f7e20131980 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1612266376 0 0) 0x7f7e080034e0 con 0x7f7e20131980 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 >> v2:172.21.15.153:3300/0 conn(0x7f7e2009f140 msgr2=0x7f7e20139cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 --2- 172.21.15.153:0/3288261573 >> v2:172.21.15.153:3300/0 conn(0x7f7e2009f140 0x7f7e20139cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 >> v1:172.21.15.153:6789/0 conn(0x7f7e2009fe10 legacy=0x7f7e2009ea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e200a11f0 con 0x7f7e20131980 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: _finish_auth 0 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f7e10001720 con 0x7f7e20131980 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:14.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: dump: 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:31:14.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 >> v1:172.21.15.181:6789/0 conn(0x7f7e20131980 legacy=0x7f7e20131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e08003a50 0x7f7e08003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): picked mon.b con 0x7f7e08003a50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e08004780 0x7f7e08004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): picked mon.a con 0x7f7e08004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 0x7f7e0800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): picked mon.c con 0x7f7e080099c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0800c330 con 0x7f7e08004780 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0800c4a0 con 0x7f7e08003a50 2024-09-15T03:31:14.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0800c660 con 0x7f7e080099c0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1f7fe640 1 -- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e08004780 msgr2=0x7f7e08004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1f7fe640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e08004780 0x7f7e08004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 0x7f7e0800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e08003a50 0x7f7e08003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): get_auth_request con 0x7f7e080099c0 auth_method 0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 10 monclient(hunting): get_auth_request con 0x7f7e08003a50 auth_method 0 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:14.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e24afa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.244+0000 7f7e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 10 monclient(hunting): handle_auth_done global_id 44461 payload 293 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 -- 172.21.15.153:0/3288261573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e08003a50 msgr2=0x7f7e08003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e08003a50 0x7f7e08003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 -- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e08004780 msgr2=0x7f7e08004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e08004780 0x7f7e08004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 10 monclient: _finish_hunting 0 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 monclient: found mon.c 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 -- 172.21.15.153:0/3288261573 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e080034e0 con 0x7f7e080099c0 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 10 monclient: _finish_auth 0 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e2655d640 5 monclient: authenticate success, global_id 44461 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e2655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.245+0000 7f7e24afa640 1 -- 172.21.15.153:0/3288261573 reap_dead start 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.247+0000 7f7e24afa640 1 -- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 msgr2=0x7f7e0800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.247+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 0x7f7e0800bda0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009f90 tx=0x7f7e180095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8408f998425340ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18032430 con 0x7f7e080099c0 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: dump: 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.261 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1802d070 con 0x7f7e080099c0 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 1 -- 172.21.15.153:0/3288261573 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18032820 con 0x7f7e080099c0 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 4 set_mon_vals no callback set 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:14.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e1effd640 10 monclient: dump: 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 10 monclient: shutdown 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:14.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 -- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 msgr2=0x7f7e0800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 0x7f7e0800bda0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009f90 tx=0x7f7e180095d0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 -- 172.21.15.153:0/3288261573 shutdown_connections 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3288261573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e080099c0 0x7f7e0800bda0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 -- 172.21.15.153:0/3288261573 >> 172.21.15.153:0/3288261573 conn(0x7f7e200a5050 msgr2=0x7f7e200a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 -- 172.21.15.153:0/3288261573 shutdown_connections 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.248+0000 7f7e2655d640 1 -- 172.21.15.153:0/3288261573 wait complete. 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.249+0000 7f7e2655d640 10 monclient: build_initial_monmap 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.249+0000 7f7e2655d640 10 monclient: monmap: 2024-09-15T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.249+0000 7f7e2655d640 1 Processor -- start 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.249+0000 7f7e2655d640 1 -- start start 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.249+0000 7f7e2655d640 10 monclient: init 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:31:14.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e080099c0 0x7f7e2009a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e080099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e20130940 0x7f7e20130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e20130940 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 0x7f7e2011bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2011b6b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20131250 con 0x7f7e080099c0 2024-09-15T03:31:14.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20069eb0 con 0x7f7e20130940 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2009df70 con 0x7f7e2011b6b0 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1ffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e080099c0 msgr2=0x7f7e2009a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e2655d640 10 monclient(hunting): authenticate will time out at 871.889222s 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e080099c0 0x7f7e2009a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 0x7f7e2011bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 0x7f7e2011bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56628/0 (socket says 172.21.15.153:56628) 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 -- 172.21.15.153:0/3901980216 learned_addr learned my addr 172.21.15.153:0/3901980216 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): get_auth_request con 0x7f7e2011b6b0 auth_method 0 2024-09-15T03:31:14.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e20130940 0x7f7e20130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): get_auth_request con 0x7f7e20130940 auth_method 0 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:14.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient(hunting): handle_auth_done global_id 5033 payload 824 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 -- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e20130940 msgr2=0x7f7e20130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e20130940 0x7f7e20130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 -- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e080099c0 msgr2=0x7f7e2009a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e080099c0 0x7f7e2009a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient: _finish_hunting 0 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 monclient: found mon.noname-c 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.250+0000 7f7e24afa640 1 -- 172.21.15.153:0/3901980216 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2011dd40 con 0x7f7e2011b6b0 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.252+0000 7f7e24afa640 1 -- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 msgr2=0x7f7e2011bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.252+0000 7f7e24afa640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 0x7f7e2011bae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7e1802ad80 tx=0x7f7e18035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.252+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18009360 con 0x7f7e2011b6b0 2024-09-15T03:31:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.252+0000 7f7e07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.252+0000 7f7e07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.252+0000 7f7e07fff640 10 monclient: dump: 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:14.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 5 monclient: authenticate success, global_id 5033 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e180052e0 con 0x7f7e2011b6b0 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18005680 con 0x7f7e2011b6b0 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 10 monclient: _renew_subs 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2011da60 con 0x7f7e2011b6b0 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 10 monclient: dump: 2024-09-15T03:31:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e1e7fc640 4 set_mon_vals no callback set 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 10 monclient: _renew_subs 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e20136110 con 0x7f7e2011b6b0 2024-09-15T03:31:14.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.253+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7e18039070 con 0x7f7e2011b6b0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e2655d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e2655d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e2655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2004bf60 con 0x7f7e2011b6b0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e07fff640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e0c0386f0 0x7f7e0c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f7e18006e40 con 0x7f7e2011b6b0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e1ffff640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e0c0386f0 0x7f7e0c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.254+0000 7f7e1ffff640 10 monclient: get_auth_request con 0x7f7e0c0386f0 auth_method 0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.257+0000 7f7e1ffff640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e0c0386f0 0x7f7e0c03aba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7e10001900 tx=0x7f7e10007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.258+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7e180475f0 con 0x7f7e2011b6b0 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.258+0000 7f7e07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.258+0000 7f7e07fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.471+0000 7f7e2655d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.471+0000 7f7e2655d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.471+0000 7f7e2655d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.471+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7e20067f00 con 0x7f7e2011b6b0 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.472+0000 7f7e07fff640 1 -- 172.21.15.153:0/3901980216 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7e18030b30 con 0x7f7e2011b6b0 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.472+0000 7f7e07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.472+0000 7f7e07fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:14.473 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:14.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e0c0386f0 msgr2=0x7f7e0c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:14.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e0c0386f0 0x7f7e0c03aba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7e10001900 tx=0x7f7e10007a90 comp rx=0 tx=0).stop 2024-09-15T03:31:14.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 10 monclient: shutdown 2024-09-15T03:31:14.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 msgr2=0x7f7e2011bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 0x7f7e2011bae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7e1802ad80 tx=0x7f7e18035000 comp rx=0 tx=0).stop 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 shutdown_connections 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e0c0386f0 0x7f7e0c03aba0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e2011b6b0 0x7f7e2011bae0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e20130940 0x7f7e20130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 --2- 172.21.15.153:0/3901980216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e080099c0 0x7f7e2009a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 >> 172.21.15.153:0/3901980216 conn(0x7f7e200a5050 msgr2=0x7f7e200a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 shutdown_connections 2024-09-15T03:31:14.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:14.474+0000 7f7e2655d640 1 -- 172.21.15.153:0/3901980216 wait complete. 2024-09-15T03:31:14.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:14.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:14.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:14.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:14.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:15.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:15.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:15.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:15.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:15.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:15.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:15.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:15.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:16.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:16.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:16.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:16.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:16.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:16.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:16.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:16.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:16.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:16.959+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:16.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:16.959+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-09-15T03:31:17.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.589+0000 7f263a51d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.589+0000 7f263a51d640 10 monclient: build_initial_monmap 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.589+0000 7f263a51d640 10 monclient: monmap: 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:17.591090+0000 2024-09-15T03:31:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:17.591090+0000 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:17.592 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 1 Processor -- start 2024-09-15T03:31:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 1 -- start start 2024-09-15T03:31:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient: init 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f263409c9c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f263409e340 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f26341381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient(hunting): picked mon.noname-b con 0x7f263409e750 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263409bdd0 con 0x7f263409e340 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263409df40 con 0x7f263409c9c0 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263409a880 con 0x7f263409e750 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f263a51d640 10 monclient(hunting): authenticate will time out at 875.230729s 2024-09-15T03:31:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f26337fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f263409e340 legacy=0x7f2634134aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f26341381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f26341381d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38514/0 (socket says 172.21.15.153:38514) 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2633fff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f263409c9c0 0x7f263409d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:37854/0 (socket says 172.21.15.153:37854) 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 1 -- 172.21.15.153:0/3657826109 learned_addr learned my addr 172.21.15.153:0/3657826109 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): get_auth_request con 0x7f263409e750 auth_method 0 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.590+0000 7f2638aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 10 monclient(hunting): handle_auth_done global_id 5036 payload 274 2024-09-15T03:31:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 1 -- 172.21.15.153:0/3657826109 >> v1:172.21.15.181:6789/0 conn(0x7f263409c9c0 legacy=0x7f263409d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 1 -- 172.21.15.153:0/3657826109 >> v1:172.21.15.153:6789/0 conn(0x7f263409e340 legacy=0x7f2634134aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 10 monclient: _finish_hunting 0 2024-09-15T03:31:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 1 monclient: found mon.noname-b 2024-09-15T03:31:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.591+0000 7f2638aba640 1 -- 172.21.15.153:0/3657826109 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263409a440 con 0x7f263409e750 2024-09-15T03:31:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.593+0000 7f2638aba640 1 -- 172.21.15.153:0/3657826109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 msgr2=0x7f26341381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.593+0000 7f2638aba640 1 --2- 172.21.15.153:0/3657826109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f26341381d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f262c009f50 tx=0x7f262c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90e5a616e66227d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 1 -- 172.21.15.153:0/3657826109 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c007610 con 0x7f263409e750 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: dump: 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 1 -- 172.21.15.153:0/3657826109 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c02d050 con 0x7f263409e750 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 5 monclient: authenticate success, global_id 5036 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 1 -- 172.21.15.153:0/3657826109 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c0079a0 con 0x7f263409e750 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 4 set_mon_vals no callback set 2024-09-15T03:31:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f2632ffd640 10 monclient: dump: 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 10 monclient: shutdown 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 -- 172.21.15.153:0/3657826109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 msgr2=0x7f26341381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 --2- 172.21.15.153:0/3657826109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f26341381d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f262c009f50 tx=0x7f262c009840 comp rx=0 tx=0).stop 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 -- 172.21.15.153:0/3657826109 shutdown_connections 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 --2- 172.21.15.153:0/3657826109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f26341381d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 -- 172.21.15.153:0/3657826109 >> 172.21.15.153:0/3657826109 conn(0x7f26340a2e40 msgr2=0x7f26340a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 -- 172.21.15.153:0/3657826109 shutdown_connections 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.594+0000 7f263a51d640 1 -- 172.21.15.153:0/3657826109 wait complete. 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.595+0000 7f263a51d640 10 monclient: build_initial_monmap 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.595+0000 7f263a51d640 10 monclient: monmap: 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.595+0000 7f263a51d640 1 Processor -- start 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.595+0000 7f263a51d640 1 -- start start 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.595+0000 7f263a51d640 10 monclient: init 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:17.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263409c9c0 0x7f263413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): picked mon.noname-b con 0x7f263409c9c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f263409e340 0x7f2634131be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): picked mon.noname-a con 0x7f263409e340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f2634134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): picked mon.noname-c con 0x7f263409e750 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634119cb0 con 0x7f263409e340 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634119f90 con 0x7f263409c9c0 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263411a270 con 0x7f263409e750 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:17.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f263a51d640 10 monclient(hunting): authenticate will time out at 875.236264s 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f26337fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f263409e340 msgr2=0x7f2634131be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f26337fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f263409e340 0x7f2634131be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f2634134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f2634134a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38528/0 (socket says 172.21.15.153:38528) 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 -- 172.21.15.153:0/1292631483 learned_addr learned my addr 172.21.15.153:0/1292631483 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): get_auth_request con 0x7f263409e750 auth_method 0 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263409c9c0 0x7f263413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): get_auth_request con 0x7f263409c9c0 auth_method 0 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient(hunting): handle_auth_done global_id 5039 payload 824 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 -- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263409c9c0 msgr2=0x7f263413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263409c9c0 0x7f263413eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 -- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f263409e340 msgr2=0x7f2634131be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f263409e340 0x7f2634131be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient: _finish_hunting 0 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 monclient: found mon.noname-c 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.596+0000 7f2638aba640 1 -- 172.21.15.153:0/1292631483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c009ae0 con 0x7f263409e750 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.598+0000 7f2638aba640 1 -- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 msgr2=0x7f2634134a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.598+0000 7f2638aba640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f2634134a60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f262c009ea0 tx=0x7f262c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c031cd0 con 0x7f263409e750 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: dump: 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:17.611 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: _finish_auth 0 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c031e30 con 0x7f263409e750 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 5 monclient: authenticate success, global_id 5039 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 10 monclient: _renew_subs 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2634136510 con 0x7f263409e750 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c03c670 con 0x7f263409e750 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 10 monclient: dump: 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 10 monclient: _renew_subs 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263413fbd0 con 0x7f263409e750 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f26327fc640 4 set_mon_vals no callback set 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.599+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f262c003030 con 0x7f263409e750 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f263a51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f263a51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f263a51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f262bfff640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26140386f0 0x7f261403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263404bf60 con 0x7f263409e750 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f262c06cb60 con 0x7f263409e750 2024-09-15T03:31:17.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f2633fff640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26140386f0 0x7f261403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:17.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.600+0000 7f2633fff640 10 monclient: get_auth_request con 0x7f26140386f0 auth_method 0 2024-09-15T03:31:17.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.602+0000 7f2633fff640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26140386f0 0x7f261403aba0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2624009a40 tx=0x7f2624006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:17.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.604+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f262c03caa0 con 0x7f263409e750 2024-09-15T03:31:17.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.604+0000 7f262bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:17.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.604+0000 7f262bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:17.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:17.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:17.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:17.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:17.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:17.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:17.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:17.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.816+0000 7f263a51d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.816+0000 7f263a51d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.816+0000 7f263a51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.816+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f263411a3b0 con 0x7f263409e750 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.817+0000 7f262bfff640 1 -- 172.21.15.153:0/1292631483 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f262c02e4a0 con 0x7f263409e750 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.817+0000 7f262bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.817+0000 7f262bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:17.819 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:17.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26140386f0 msgr2=0x7f261403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:17.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26140386f0 0x7f261403aba0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2624009a40 tx=0x7f2624006f50 comp rx=0 tx=0).stop 2024-09-15T03:31:17.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 10 monclient: shutdown 2024-09-15T03:31:17.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:17.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 msgr2=0x7f2634134a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:17.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f2634134a60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f262c009ea0 tx=0x7f262c036000 comp rx=0 tx=0).stop 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 shutdown_connections 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26140386f0 0x7f261403aba0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f263409e750 0x7f2634134a60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f263409e340 0x7f2634131be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 --2- 172.21.15.153:0/1292631483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f263409c9c0 0x7f263413eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 >> 172.21.15.153:0/1292631483 conn(0x7f26340a2e40 msgr2=0x7f2634137aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 shutdown_connections 2024-09-15T03:31:17.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:17.819+0000 7f263a51d640 1 -- 172.21.15.153:0/1292631483 wait complete. 2024-09-15T03:31:18.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:18.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:18.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:18.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:18.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:19.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:19.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:19.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:19.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:19.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:19.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:19.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:19.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:20.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:20.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:20.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:20.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:20.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:20.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.934+0000 7f4a120d4640 10 monclient: get_monmap_and_config 2024-09-15T03:31:20.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.934+0000 7f4a120d4640 10 monclient: build_initial_monmap 2024-09-15T03:31:20.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.934+0000 7f4a120d4640 10 monclient: monmap: 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:20.936439+0000 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:20.936439+0000 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:20.937 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:20.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.935+0000 7f4a120d4640 1 Processor -- start 2024-09-15T03:31:20.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.935+0000 7f4a120d4640 1 -- start start 2024-09-15T03:31:20.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.935+0000 7f4a120d4640 10 monclient: init 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a0c09c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0c09e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0c136cb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0c09ec40 con 0x7f4a0c09c860 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a0c09a9d0 con 0x7f4a0c136cb0 2024-09-15T03:31:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c06a320 con 0x7f4a0c09e840 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a120d4640 10 monclient(hunting): authenticate will time out at 878.576112s 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38530/0 (socket says 172.21.15.153:38530) 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 learned_addr learned my addr 172.21.15.153:0/1411635444 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 10 monclient(hunting): get_auth_request con 0x7f4a0c09e840 auth_method 0 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c136cb0 auth_method 0 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.936+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:20.938+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.181:6789/0 conn(0x7f4a0c09c860 legacy=0x7f4a0c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 10 monclient(hunting): get_auth_request con 0x7f4a0c09e840 auth_method 0 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:21.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c136cb0 auth_method 0 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.137+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:31:21.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 10 monclient(hunting): get_auth_request con 0x7f4a0c09e840 auth_method 0 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c136cb0 auth_method 0 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:21.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:21.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:21.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:21.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:21.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:21.538+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:31:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:21.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:21.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:21.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:21.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:21.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 10 monclient(hunting): get_auth_request con 0x7f4a0c09e840 auth_method 0 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:22.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c136cb0 auth_method 0 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:22.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:22.339+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:31:22.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:22.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:22.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:22.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:22.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:22.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:22.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:22.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.360+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:23.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.360+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-09-15T03:31:23.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:23.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:23.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:23.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:23.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:23.748+0000 7f3fe17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:23.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient(hunting): tick 2024-09-15T03:31:23.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 monclient(hunting): continuing hunt 2024-09-15T03:31:23.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 msgr2=0x7f4a0c09aba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c09e840 0x7f4a0c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 msgr2=0x7f4a0c139090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f4a0c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.181:6789/0 conn(0x7f4a0c09c860 legacy=0x7f4a0c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient: _add_conns ranks=[0,3] 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49f8003720 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 0x7f49f80098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient(hunting): picked mon.noname-a con 0x7f49f8007420 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0c068a20 con 0x7f49f8003720 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a0c09a9d0 con 0x7f49f8007420 2024-09-15T03:31:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a097fa640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 msgr2=0x7f49f80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 0x7f49f80098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:23.936+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:24.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:24.137+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 msgr2=0x7f49f80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:24.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:24.137+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:24.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:24.137+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 0x7f49f80098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:31:24.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:24.538+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:24.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:24.538+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 msgr2=0x7f49f80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:24.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:24.538+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 0x7f49f80098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:31:24.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:24.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:24.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:24.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:24.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:24.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:24.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:24.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:25.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:25.339+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:25.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:25.339+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 msgr2=0x7f49f80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:25.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:25.339+0000 7f4a0affd640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 0x7f49f80098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:31:25.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:25.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:25.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:25.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:25.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:25.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:25.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:25.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:26.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:26.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:26.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:26.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:26.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:26.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:26.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:26.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): tick 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 monclient(hunting): continuing hunt 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 msgr2=0x7f49f80098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.153:3300/0 conn(0x7f49f8007420 0x7f49f80098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T03:31:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f49f8009fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0c136cb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 reap_dead start 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49f8003720 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49f800a520 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f800b040 con 0x7f49f8003720 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0c068a20 con 0x7f49f800a520 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 1 -- 172.21.15.153:0/1411635444 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a0c09a9d0 con 0x7f4a0c136cb0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a097fa640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0affd640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0b7fe640 1 --2- 172.21.15.153:0/1411635444 >> v2:172.21.15.181:3300/0 conn(0x7f4a0c136cb0 0x7f49f8009fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0b7fe640 10 monclient(hunting): get_auth_request con 0x7f4a0c136cb0 auth_method 0 2024-09-15T03:31:26.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:26.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:26.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.937+0000 7f4a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.181:6789/0 conn(0x7f49f800a520 legacy=0x7f49f800d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_done global_id 44470 payload 274 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.181:6789/0 conn(0x7f49f800a520 legacy=0x7f49f800d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> v1:172.21.15.153:6789/0 conn(0x7f49f8003720 legacy=0x7f49f8005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.939+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f80099d0 con 0x7f4a0c136cb0 2024-09-15T03:31:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.942+0000 7f4a0b7fe640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a0c136cb0 msgr2=0x7f49f8009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.942+0000 7f4a0b7fe640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a0c136cb0 0x7f49f8009fb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0014b0 tx=0x7f49fc001c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84faf7e0ddc128 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 1 -- 172.21.15.153:0/1411635444 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0072a0 con 0x7f4a0c136cb0 2024-09-15T03:31:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: dump: 2024-09-15T03:31:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 1 -- 172.21.15.153:0/1411635444 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc02d070 con 0x7f4a0c136cb0 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 1 -- 172.21.15.153:0/1411635444 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc007650 con 0x7f4a0c136cb0 2024-09-15T03:31:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 5 monclient: authenticate success, global_id 44470 2024-09-15T03:31:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 4 set_mon_vals no callback set 2024-09-15T03:31:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a0a7fc640 10 monclient: dump: 2024-09-15T03:31:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 10 monclient: shutdown 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 1 -- 172.21.15.153:0/1411635444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a0c136cb0 msgr2=0x7f49f8009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.943+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a0c136cb0 0x7f49f8009fb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0014b0 tx=0x7f49fc001c80 comp rx=0 tx=0).stop 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 1 -- 172.21.15.153:0/1411635444 shutdown_connections 2024-09-15T03:31:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1411635444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a0c136cb0 0x7f49f8009fb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 1 -- 172.21.15.153:0/1411635444 >> 172.21.15.153:0/1411635444 conn(0x7f4a0c0a2e40 msgr2=0x7f4a0c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 1 -- 172.21.15.153:0/1411635444 shutdown_connections 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 1 -- 172.21.15.153:0/1411635444 wait complete. 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 10 monclient: build_initial_monmap 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.944+0000 7f4a120d4640 10 monclient: monmap: 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:26.962 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.945+0000 7f4a120d4640 1 Processor -- start 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.945+0000 7f4a120d4640 1 -- start start 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.945+0000 7f4a120d4640 10 monclient: init 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8003720 0x7f4a0c141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): picked mon.noname-b con 0x7f49f8003720 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f800a520 0x7f4a0c141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): picked mon.noname-a con 0x7f49f800a520 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 0x7f4a0c141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0c136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c122170 con 0x7f49f800a520 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c11fff0 con 0x7f49f8003720 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c0a2780 con 0x7f4a0c136cb0 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a120d4640 10 monclient(hunting): authenticate will time out at 884.586132s 2024-09-15T03:31:26.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0affd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f800a520 msgr2=0x7f4a0c141660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:26.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f800a520 0x7f4a0c141660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:26.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 0x7f4a0c141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:26.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 0x7f4a0c141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47246/0 (socket says 172.21.15.153:47246) 2024-09-15T03:31:26.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1433330305 learned_addr learned my addr 172.21.15.153:0/1433330305 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:26.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c136cb0 auth_method 0 2024-09-15T03:31:26.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8003720 0x7f4a0c141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): get_auth_request con 0x7f49f8003720 auth_method 0 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:26.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_done global_id 5042 payload 824 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8003720 msgr2=0x7f4a0c141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8003720 0x7f4a0c141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f800a520 msgr2=0x7f4a0c141660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f800a520 0x7f4a0c141660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 monclient: found mon.noname-c 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:26.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.946+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1433330305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c09ec40 con 0x7f4a0c136cb0 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a0bfff640 1 -- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 msgr2=0x7f4a0c141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a0bfff640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 0x7f4a0c141ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009590 tx=0x7f4a0400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04009c90 con 0x7f4a0c136cb0 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: dump: 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:26.968 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0400f030 con 0x7f4a0c136cb0 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 5 monclient: authenticate success, global_id 5042 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 10 monclient: _renew_subs 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0c099c90 con 0x7f4a0c136cb0 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400a050 con 0x7f4a0c136cb0 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f49f37fe640 10 monclient: dump: 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 10 monclient: _renew_subs 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a09ffb640 4 set_mon_vals no callback set 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.949+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0c120bc0 con 0x7f4a0c136cb0 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4a04003040 con 0x7f4a0c136cb0 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f4a120d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f4a120d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f4a120d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0c04bf60 con 0x7f4a0c136cb0 2024-09-15T03:31:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f49f37fe640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49f80398c0 0x7f49f803bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.950+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f4a0404be40 con 0x7f4a0c136cb0 2024-09-15T03:31:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.951+0000 7f4a0b7fe640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49f80398c0 0x7f49f803bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.951+0000 7f4a0b7fe640 10 monclient: get_auth_request con 0x7f49f80398c0 auth_method 0 2024-09-15T03:31:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.953+0000 7f4a0b7fe640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49f80398c0 0x7f49f803bd70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f49f8003690 tx=0x7f49fc033000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.955+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4a040132b0 con 0x7f4a0c136cb0 2024-09-15T03:31:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.955+0000 7f49f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:26.955+0000 7f49f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f4a120d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f4a120d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f4a120d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4a0c121c40 con 0x7f4a0c136cb0 2024-09-15T03:31:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f49f37fe640 1 -- 172.21.15.153:0/1433330305 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f4a0401a070 con 0x7f4a0c136cb0 2024-09-15T03:31:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f49f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.176+0000 7f49f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:27.180 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49f80398c0 msgr2=0x7f49f803bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49f80398c0 0x7f49f803bd70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f49f8003690 tx=0x7f49fc033000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 10 monclient: shutdown 2024-09-15T03:31:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 msgr2=0x7f4a0c141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 0x7f4a0c141ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009590 tx=0x7f4a0400b040 comp rx=0 tx=0).stop 2024-09-15T03:31:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 shutdown_connections 2024-09-15T03:31:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49f80398c0 0x7f49f803bd70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a0c136cb0 0x7f4a0c141ba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49f800a520 0x7f4a0c141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 --2- 172.21.15.153:0/1433330305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8003720 0x7f4a0c141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 >> 172.21.15.153:0/1433330305 conn(0x7f4a0c0a2e40 msgr2=0x7f4a0c098a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 shutdown_connections 2024-09-15T03:31:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.178+0000 7f4a120d4640 1 -- 172.21.15.153:0/1433330305 wait complete. 2024-09-15T03:31:27.192 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:31:27.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:31:27.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:31:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.284+0000 7f2dea154640 10 monclient: get_monmap_and_config 2024-09-15T03:31:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.284+0000 7f2dea154640 10 monclient: build_initial_monmap 2024-09-15T03:31:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.284+0000 7f2dea154640 10 monclient: monmap: 2024-09-15T03:31:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:27.286439+0000 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:27.286439+0000 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:27.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 Processor -- start 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 -- start start 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient: init 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2de4096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 0x7f2de40975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40971e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2de4133b30 0x7f2de4135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): picked mon.noname-c con 0x7f2de4133b30 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de4099620 con 0x7f2de4096e10 2024-09-15T03:31:27.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de4099340 con 0x7f2de4133b30 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4099900 con 0x7f2de40971e0 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2dea154640 10 monclient(hunting): authenticate will time out at 884.925964s 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.285+0000 7f2de37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f2de4096e10 legacy=0x7f2de409a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 0x7f2de40975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 0x7f2de40975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47258/0 (socket says 172.21.15.153:47258) 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 -- 172.21.15.153:0/4036463627 learned_addr learned my addr 172.21.15.153:0/4036463627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request con 0x7f2de40971e0 auth_method 0 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 1 --2- 172.21.15.153:0/4036463627 >> v2:172.21.15.181:3300/0 conn(0x7f2de4133b30 0x7f2de4135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): get_auth_request con 0x7f2de4133b30 auth_method 0 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_done global_id 5045 payload 274 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 -- 172.21.15.153:0/4036463627 >> v2:172.21.15.181:3300/0 conn(0x7f2de4133b30 msgr2=0x7f2de4135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:27.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/4036463627 >> v2:172.21.15.181:3300/0 conn(0x7f2de4133b30 0x7f2de4135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 -- 172.21.15.153:0/4036463627 >> v1:172.21.15.153:6789/0 conn(0x7f2de4096e10 legacy=0x7f2de409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:27.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:27.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:27.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:27.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.286+0000 7f2de2ffd640 1 -- 172.21.15.153:0/4036463627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de409ac70 con 0x7f2de40971e0 2024-09-15T03:31:27.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de2ffd640 1 -- 172.21.15.153:0/4036463627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 msgr2=0x7f2de40975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/4036463627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 0x7f2de40975b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2de409aff0 tx=0x7f2dd80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ffcf40db0031f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 1 -- 172.21.15.153:0/4036463627 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd802ae90 con 0x7f2de40971e0 2024-09-15T03:31:27.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:27.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: dump: 2024-09-15T03:31:27.301 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.302 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.302 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.302 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 1 -- 172.21.15.153:0/4036463627 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd8032440 con 0x7f2de40971e0 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 5 monclient: authenticate success, global_id 5045 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 1 -- 172.21.15.153:0/4036463627 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd80327e0 con 0x7f2de40971e0 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2de27fc640 10 monclient: dump: 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.304 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 4 set_mon_vals no callback set 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 10 monclient: shutdown 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 1 -- 172.21.15.153:0/4036463627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 msgr2=0x7f2de40975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 1 --2- 172.21.15.153:0/4036463627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 0x7f2de40975b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2de409aff0 tx=0x7f2dd80091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 1 -- 172.21.15.153:0/4036463627 shutdown_connections 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 1 --2- 172.21.15.153:0/4036463627 >> v2:172.21.15.181:3300/0 conn(0x7f2de4133b30 0x7f2de4135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 1 --2- 172.21.15.153:0/4036463627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de40971e0 0x7f2de40975b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.289+0000 7f2dea154640 1 -- 172.21.15.153:0/4036463627 >> 172.21.15.153:0/4036463627 conn(0x7f2de40672b0 msgr2=0x7f2de40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.290+0000 7f2dea154640 1 -- 172.21.15.153:0/4036463627 shutdown_connections 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.290+0000 7f2dea154640 1 -- 172.21.15.153:0/4036463627 wait complete. 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.290+0000 7f2dea154640 10 monclient: build_initial_monmap 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.290+0000 7f2dea154640 10 monclient: monmap: 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.290+0000 7f2dea154640 1 Processor -- start 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 -- start start 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient: init 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 0x7f2de4141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): picked mon.noname-c con 0x7f2de4096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2de40971e0 0x7f2de409b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40971e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2de4133b30 0x7f2de412e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): picked mon.noname-a con 0x7f2de4133b30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de411e3b0 con 0x7f2de4133b30 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de409b050 con 0x7f2de40971e0 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de412f790 con 0x7f2de4096e10 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2dea154640 10 monclient(hunting): authenticate will time out at 884.931556s 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de3fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2de4133b30 msgr2=0x7f2de412e920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2de4133b30 0x7f2de412e920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 0x7f2de4141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 0x7f2de4141580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47270/0 (socket says 172.21.15.153:47270) 2024-09-15T03:31:27.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 -- 172.21.15.153:0/3741182825 learned_addr learned my addr 172.21.15.153:0/3741182825 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): get_auth_request con 0x7f2de4096e10 auth_method 0 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2de40971e0 0x7f2de409b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request con 0x7f2de40971e0 auth_method 0 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient(hunting): handle_auth_done global_id 5048 payload 824 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2de40971e0 msgr2=0x7f2de409b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2de40971e0 0x7f2de409b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2de4133b30 msgr2=0x7f2de412e920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2de4133b30 0x7f2de412e920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.291+0000 7f2de37fe640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd8009450 con 0x7f2de4096e10 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2de37fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 msgr2=0x7f2de4141580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2de37fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 0x7f2de4141580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400b630 tx=0x7f2dd40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4015070 con 0x7f2de4096e10 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: dump: 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4003d90 con 0x7f2de4096e10 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 5 monclient: authenticate success, global_id 5048 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4006380 con 0x7f2de4096e10 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 10 monclient: _renew_subs 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de411daa0 con 0x7f2de4096e10 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dcb7fe640 10 monclient: dump: 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.313 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2de1ffb640 4 set_mon_vals no callback set 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 10 monclient: _renew_subs 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.294+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de412ee60 con 0x7f2de4096e10 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dea154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2dd4010440 con 0x7f2de4096e10 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dea154640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 0x7f2de4117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4065f20 con 0x7f2de412f0b0 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 0x7f2de4117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de412f0b0 auth_method 0 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dcb7fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2dd00386f0 0x7f2dd003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f2dd404b990 con 0x7f2de4096e10 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_done global_id 5051 payload 824 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.295+0000 7f2de2ffd640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2de411de80 con 0x7f2de412f0b0 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.296+0000 7f2de3fff640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2dd00386f0 0x7f2dd003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.296+0000 7f2de3fff640 10 monclient: get_auth_request con 0x7f2dd00386f0 auth_method 0 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2de2ffd640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 msgr2=0x7f2de4117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 0x7f2de4117480 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8000c90 con 0x7f2de412f0b0 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 10 monclient: dump: 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.316 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2dd802ae90 con 0x7f2de412f0b0 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 msgr2=0x7f2de4117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 0x7f2de4117480 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd8032000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 msgr2=0x7f2de4117480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2dcb7fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de412f0b0 0x7f2de4117480 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd8032000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.298+0000 7f2de3fff640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2dd00386f0 0x7f2dd003aba0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009b10 tx=0x7f2ddc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2dea154640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:27.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2dea154640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 0x7f2de4117d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2dea154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de411e160 con 0x7f2de41179c0 2024-09-15T03:31:27.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 0x7f2de4117d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41179c0 auth_method 0 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_done global_id 5054 payload 824 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.304+0000 7f2de2ffd640 1 -- 172.21.15.153:0/3741182825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2de41182d0 con 0x7f2de41179c0 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2de2ffd640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 msgr2=0x7f2de4117d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2de2ffd640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 0x7f2de4117d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009210 tx=0x7f2dd803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8046840 con 0x7f2de41179c0 2024-09-15T03:31:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 10 monclient: dump: 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f2dd80469a0 con 0x7f2de41179c0 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 msgr2=0x7f2de4117d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 0x7f2de4117d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009210 tx=0x7f2dd803c000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 msgr2=0x7f2de4117d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.306+0000 7f2dcb7fe640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de41179c0 0x7f2de4117d90 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009210 tx=0x7f2dd803c000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:27.322 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:27.323 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:27.324 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:27.325 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:27.326 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.327 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.328 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:27.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:27.330 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:27.331 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:27.332 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:31:27.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:31:27.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:27.335 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2dd00386f0 msgr2=0x7f2dd003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2dd00386f0 0x7f2dd003aba0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009b10 tx=0x7f2ddc00b040 comp rx=0 tx=0).stop 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 10 monclient: shutdown 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 msgr2=0x7f2de4141580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 --2- 172.21.15.153:0/3741182825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2de4096e10 0x7f2de4141580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400b630 tx=0x7f2dd40091a0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2de3fff640 1 -- 172.21.15.153:0/3741182825 reap_dead start 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2de3fff640 1 -- 172.21.15.153:0/3741182825 reap_dead start 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 shutdown_connections 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 >> 172.21.15.153:0/3741182825 conn(0x7f2de40672b0 msgr2=0x7f2de409fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 shutdown_connections 2024-09-15T03:31:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.307+0000 7f2dea154640 1 -- 172.21.15.153:0/3741182825 wait complete. 2024-09-15T03:31:27.338 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:31:27.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:31:27.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.429+0000 7fa8fb35d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.429+0000 7fa8fb35d640 10 monclient: build_initial_monmap 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.429+0000 7fa8fb35d640 10 monclient: monmap: 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:27.431042+0000 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:27.431042+0000 2024-09-15T03:31:27.431 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:27.432 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.429+0000 7fa8fb35d640 1 Processor -- start 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 1 -- start start 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient: init 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa8f40981c0 0x7fa8f4098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f40981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8f4131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa8f4132170 0x7fa8f4067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f4132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f4098ad0 con 0x7fa8f4131da0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f4131bd0 con 0x7fa8f40981c0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f409ca60 con 0x7fa8f4132170 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8fb35d640 10 monclient(hunting): authenticate will time out at 885.070573s 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8f90f9640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa8f40981c0 msgr2=0x7fa8f4098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8f88f8640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa8f4131da0 legacy=0x7fa8f4067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8f90f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa8f40981c0 0x7fa8f4098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:27.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.430+0000 7fa8f98fa640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa8f4132170 0x7fa8f4067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa8f4132170 0x7fa8f4067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:57848/0 (socket says 172.21.15.153:57848) 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 -- 172.21.15.153:0/1850008653 learned_addr learned my addr 172.21.15.153:0/1850008653 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): get_auth_request con 0x7fa8f4132170 auth_method 0 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient(hunting): handle_auth_done global_id 44485 payload 274 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 -- 172.21.15.153:0/1850008653 >> v2:172.21.15.153:3300/0 conn(0x7fa8f40981c0 msgr2=0x7fa8f4098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 --2- 172.21.15.153:0/1850008653 >> v2:172.21.15.153:3300/0 conn(0x7fa8f40981c0 0x7fa8f4098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 -- 172.21.15.153:0/1850008653 >> v1:172.21.15.153:6789/0 conn(0x7fa8f4131da0 legacy=0x7fa8f4067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient: _finish_hunting 0 2024-09-15T03:31:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 monclient: found mon.noname-c 2024-09-15T03:31:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.431+0000 7fa8f98fa640 1 -- 172.21.15.153:0/1850008653 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f409cca0 con 0x7fa8f4132170 2024-09-15T03:31:27.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8f98fa640 1 -- 172.21.15.153:0/1850008653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 msgr2=0x7fa8f4067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8f98fa640 1 --2- 172.21.15.153:0/1850008653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 0x7fa8f4067820 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa8f406f820 tx=0x7fa8ec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83038ce930244848 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 1 -- 172.21.15.153:0/1850008653 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec02aea0 con 0x7fa8f4132170 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: dump: 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 1 -- 172.21.15.153:0/1850008653 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ec02d050 con 0x7fa8f4132170 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 1 -- 172.21.15.153:0/1850008653 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec007840 con 0x7fa8f4132170 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8fb35d640 5 monclient: authenticate success, global_id 44485 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8fb35d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8fb35d640 4 set_mon_vals no callback set 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.434+0000 7fa8e7fff640 10 monclient: dump: 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 10 monclient: shutdown 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 -- 172.21.15.153:0/1850008653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 msgr2=0x7fa8f4067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/1850008653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 0x7fa8f4067820 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa8f406f820 tx=0x7fa8ec009840 comp rx=0 tx=0).stop 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 -- 172.21.15.153:0/1850008653 shutdown_connections 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/1850008653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 0x7fa8f4067820 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/1850008653 >> v2:172.21.15.153:3300/0 conn(0x7fa8f40981c0 0x7fa8f4098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 -- 172.21.15.153:0/1850008653 >> 172.21.15.153:0/1850008653 conn(0x7fa8f40a1e30 msgr2=0x7fa8f40a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 -- 172.21.15.153:0/1850008653 shutdown_connections 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 1 -- 172.21.15.153:0/1850008653 wait complete. 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 10 monclient: build_initial_monmap 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.435+0000 7fa8fb35d640 10 monclient: monmap: 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 Processor -- start 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 -- start start 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient: init 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f40981c0 0x7fa8f4130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f40981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 0x7fa8f4140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f4131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 0x7fa8f4141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f4132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f40a0b30 con 0x7fa8f40981c0 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4141940 con 0x7fa8f4132170 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f90f9640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f40981c0 msgr2=0x7fa8f4130980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f411bfe0 con 0x7fa8f4131da0 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8fb35d640 10 monclient(hunting): authenticate will time out at 885.076757s 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f90f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f40981c0 0x7fa8f4130980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 0x7fa8f4140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 0x7fa8f4140ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47300/0 (socket says 172.21.15.153:47300) 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 1 -- 172.21.15.153:0/4307595 learned_addr learned my addr 172.21.15.153:0/4307595 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): get_auth_request con 0x7fa8f4131da0 auth_method 0 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.436+0000 7fa8f88f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f98fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 0x7fa8f4141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f98fa640 10 monclient(hunting): get_auth_request con 0x7fa8f4132170 auth_method 0 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f98fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f98fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f98fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 10 monclient(hunting): handle_auth_done global_id 5057 payload 824 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 msgr2=0x7fa8f4141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f4132170 0x7fa8f4141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f40981c0 msgr2=0x7fa8f4130980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f40981c0 0x7fa8f4130980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 10 monclient: _finish_hunting 0 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 1 monclient: found mon.noname-c 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.437+0000 7fa8f88f8640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec009ae0 con 0x7fa8f4131da0 2024-09-15T03:31:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8f88f8640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 msgr2=0x7fa8f4140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8f88f8640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 0x7fa8f4140ec0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000b570 tx=0x7fa8f0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0003d90 con 0x7fa8f4131da0 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: dump: 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 5 monclient: authenticate success, global_id 5057 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f0006150 con 0x7fa8f4131da0 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 10 monclient: _renew_subs 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f411daa0 con 0x7fa8f4131da0 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0006510 con 0x7fa8f4131da0 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e57fa640 10 monclient: dump: 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8e77fe640 4 set_mon_vals no callback set 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 10 monclient: _renew_subs 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.439+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4130ec0 con 0x7fa8f4131da0 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8fb35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 0x7fa8f411a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4065f90 con 0x7fa8f411a450 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa8f00103c0 con 0x7fa8f4131da0 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 0x7fa8f411a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8e57fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8d80386f0 0x7fa8d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient: get_auth_request con 0x7fa8f411a450 auth_method 0 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7fa8f004b620 con 0x7fa8f4131da0 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.440+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.441+0000 7fa8f98fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8d80386f0 0x7fa8d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.441+0000 7fa8f98fa640 10 monclient: get_auth_request con 0x7fa8d80386f0 auth_method 0 2024-09-15T03:31:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.441+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_done global_id 44491 payload 824 2024-09-15T03:31:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.441+0000 7fa8f90f9640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8f404bf60 con 0x7fa8f411a450 2024-09-15T03:31:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8f90f9640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 msgr2=0x7fa8f411a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8f90f9640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 0x7fa8f411a820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800b480 tx=0x7fa8e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e80050f0 con 0x7fa8f411a450 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8f98fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8d80386f0 0x7fa8d803aba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec030000 tx=0x7fa8ec036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 10 monclient: dump: 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa8e8005250 con 0x7fa8f411a450 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 msgr2=0x7fa8f411a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 0x7fa8f411a820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800b480 tx=0x7fa8e80091a0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 msgr2=0x7fa8f411a820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.443+0000 7fa8e57fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411a450 0x7fa8f411a820 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800b480 tx=0x7fa8e80091a0 comp rx=0 tx=0).stop 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8fb35d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 0x7fa8f409a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8fb35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f411d710 con 0x7fa8f411ad60 2024-09-15T03:31:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 0x7fa8f409a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 10 monclient: get_auth_request con 0x7fa8f411ad60 auth_method 0 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.449+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.450+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.450+0000 7fa8f90f9640 10 monclient(hunting): handle_auth_done global_id 44494 payload 824 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.450+0000 7fa8f90f9640 1 -- 172.21.15.153:0/4307595 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa8f4098ee0 con 0x7fa8f411ad60 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8f90f9640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 msgr2=0x7fa8f409a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8f90f9640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 0x7fa8f409a320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800b480 tx=0x7fa8e8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e80057f0 con 0x7fa8f411ad60 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 10 monclient: dump: 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fa8e801b860 con 0x7fa8f411ad60 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:31:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 msgr2=0x7fa8f409a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.452+0000 7fa8e57fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 0x7fa8f409a320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800b480 tx=0x7fa8e8015000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8e57fa640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 msgr2=0x7fa8f409a320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8e57fa640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8f411ad60 0x7fa8f409a320 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800b480 tx=0x7fa8e8015000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-15T03:31:27.464 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:27.465 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:27.466 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:31:27.467 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:31:27.468 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:27.469 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:27.471 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:31:27.472 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.474 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:31:27.475 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:31:27.476 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:31:27.476 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:31:27.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:31:27.476 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:31:27.476 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:31:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8d80386f0 msgr2=0x7fa8d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8d80386f0 0x7fa8d803aba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec030000 tx=0x7fa8ec036000 comp rx=0 tx=0).stop 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 10 monclient: shutdown 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 msgr2=0x7fa8f4140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 --2- 172.21.15.153:0/4307595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8f4131da0 0x7fa8f4140ec0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000b570 tx=0x7fa8f0009120 comp rx=0 tx=0).stop 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8f98fa640 1 -- 172.21.15.153:0/4307595 reap_dead start 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8f98fa640 1 -- 172.21.15.153:0/4307595 reap_dead start 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 shutdown_connections 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 >> 172.21.15.153:0/4307595 conn(0x7fa8f40a1e30 msgr2=0x7fa8f41349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 shutdown_connections 2024-09-15T03:31:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:27.453+0000 7fa8fb35d640 1 -- 172.21.15.153:0/4307595 wait complete. 2024-09-15T03:31:27.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:31:27.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:31:27.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:27.747+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:27.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:27.747+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:27.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:27.747+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:27.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:27.747+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:28.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:28.748+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:28.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:28.748+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:28.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:28.748+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:28.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:28.748+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:29.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:29.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:29.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:29.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:29.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:29.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:29.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:29.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:30.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:30.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:30.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:30.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:30.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:30.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:30.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:30.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:31.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:31.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:31.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:31.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:31.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:31.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:31.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:31.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:32.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:32.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:32.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:32.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:32.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:32.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:32.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:32.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:33.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:33.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:33.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:33.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:33.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:33.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:33.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:33.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:33.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:33.749+0000 7f3fe17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:34.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:34.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:34.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:34.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:35.749+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:35.749+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:35.749+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:35.749+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:36.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:36.163+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:31:36.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:36.163+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-15T03:31:36.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:36.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:36.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:36.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:36.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:36.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:36.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:36.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:37.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:37.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:37.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:37.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:37.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:37.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:37.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:37.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:38.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:38.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:38.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:38.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:39.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:39.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:39.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:39.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:39.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:39.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:40.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:40.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:40.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:40.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:40.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:40.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:40.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:40.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:41.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:41.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:41.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:41.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:41.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:41.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:41.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:41.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:42.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:42.750+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:42.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:42.750+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:42.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:42.750+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:42.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:42.750+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:43.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:43.751+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:43.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:43.751+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:43.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:43.751+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:43.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:43.751+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:43.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:43.751+0000 7f3fe17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:44.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:44.751+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:44.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:44.751+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:44.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:44.751+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:44.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:44.751+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:45.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:45.751+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:45.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:45.751+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:45.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:45.751+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:45.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:45.751+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:46.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:46.751+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:46.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:46.751+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:46.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:46.751+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:46.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:46.751+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:47.480 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:31:47.480 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:31:47.480 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:31:47.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:31:47.483 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:31:47.483 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:31:47.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.588+0000 7f8ee26a8640 10 monclient: get_monmap_and_config 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.588+0000 7f8ee26a8640 10 monclient: build_initial_monmap 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.588+0000 7f8ee26a8640 10 monclient: monmap: 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:47.588854+0000 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:47.588854+0000 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:47.589 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:47.590 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:47.590 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:47.590 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:47.590 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:47.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.588+0000 7f8ee26a8640 1 Processor -- start 2024-09-15T03:31:47.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.588+0000 7f8ee26a8640 1 -- start start 2024-09-15T03:31:47.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.588+0000 7f8ee26a8640 10 monclient: init 2024-09-15T03:31:47.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:47.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:31:47.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8edc131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 0x7f8edc09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8edc09e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8edc09f220 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc09faf0 con 0x7f8edc09f220 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc09fd30 con 0x7f8edc131ab0 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc132590 con 0x7f8edc09e910 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee26a8640 10 monclient(hunting): authenticate will time out at 905.228537s 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 0x7f8edc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee0c45640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8edc09f220 0x7f8edc138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:42400/0 (socket says 172.21.15.153:42400) 2024-09-15T03:31:47.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8ee0c45640 1 -- 172.21.15.153:0/4029754231 learned_addr learned my addr 172.21.15.153:0/4029754231 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): get_auth_request con 0x7f8edc09e910 auth_method 0 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_done global_id 5060 payload 274 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 1 -- 172.21.15.153:0/4029754231 >> v1:172.21.15.181:6789/0 conn(0x7f8edc131ab0 legacy=0x7f8edc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 1 -- 172.21.15.153:0/4029754231 >> v1:172.21.15.153:6789/0 conn(0x7f8edc09f220 legacy=0x7f8edc138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:47.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:47.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:47.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.589+0000 7f8edb7fe640 1 -- 172.21.15.153:0/4029754231 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0a1110 con 0x7f8edc09e910 2024-09-15T03:31:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edb7fe640 1 -- 172.21.15.153:0/4029754231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 msgr2=0x7f8edc09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edb7fe640 1 --2- 172.21.15.153:0/4029754231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 0x7f8edc09ece0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009880 tx=0x7f8ed00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69e9493a7de9cf4b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 1 -- 172.21.15.153:0/4029754231 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0032440 con 0x7f8edc09e910 2024-09-15T03:31:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 10 monclient: dump: 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:47.602 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 1 -- 172.21.15.153:0/4029754231 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed00325a0 con 0x7f8edc09e910 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8ee26a8640 5 monclient: authenticate success, global_id 5060 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8ee26a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 1 -- 172.21.15.153:0/4029754231 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0032940 con 0x7f8edc09e910 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.592+0000 7f8edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8edaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8edaffd640 10 monclient: dump: 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 4 set_mon_vals no callback set 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 10 monclient: shutdown 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 -- 172.21.15.153:0/4029754231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 msgr2=0x7f8edc09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/4029754231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 0x7f8edc09ece0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009880 tx=0x7f8ed00090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 -- 172.21.15.153:0/4029754231 shutdown_connections 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/4029754231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc09e910 0x7f8edc09ece0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 -- 172.21.15.153:0/4029754231 >> 172.21.15.153:0/4029754231 conn(0x7f8edc067d30 msgr2=0x7f8edc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 -- 172.21.15.153:0/4029754231 shutdown_connections 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.593+0000 7f8ee26a8640 1 -- 172.21.15.153:0/4029754231 wait complete. 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.594+0000 7f8ee26a8640 10 monclient: build_initial_monmap 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.594+0000 7f8ee26a8640 10 monclient: monmap: 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.594+0000 7f8ee26a8640 1 Processor -- start 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.594+0000 7f8ee26a8640 1 -- start start 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.594+0000 7f8ee26a8640 10 monclient: init 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8edc09e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8edc09f220 0x7f8edc12f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:47.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8edc09f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 0x7f8edc12fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8edc131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0979f0 con 0x7f8edc09e910 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc09ac80 con 0x7f8edc09f220 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc096a60 con 0x7f8edc131ab0 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee26a8640 10 monclient(hunting): authenticate will time out at 905.234131s 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:47.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 0x7f8edc12fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40444/0 (socket says 172.21.15.153:40444) 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 -- 172.21.15.153:0/1605187166 learned_addr learned my addr 172.21.15.153:0/1605187166 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 10 monclient(hunting): get_auth_request con 0x7f8edc09e910 auth_method 0 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): get_auth_request con 0x7f8edc131ab0 auth_method 0 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 msgr2=0x7f8edc12f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:47.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 msgr2=0x7f8edc12f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edbfff640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8edc09f220 0x7f8edc12f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): get_auth_request con 0x7f8edc09f220 auth_method 0 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:47.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8edb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient(hunting): handle_auth_done global_id 5063 payload 824 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8edc09f220 msgr2=0x7f8edc12f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8edc09f220 0x7f8edc12f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 msgr2=0x7f8edc12f240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient: _finish_hunting 0 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 monclient: found mon.noname-c 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:47.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.595+0000 7f8ee0c45640 1 -- 172.21.15.153:0/1605187166 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009390 con 0x7f8edc131ab0 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ee0c45640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 msgr2=0x7f8edc12fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ee0c45640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 0x7f8edc12fcc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400cb60 tx=0x7f8ed40052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400a850 con 0x7f8edc131ab0 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 10 monclient: dump: 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed400a9d0 con 0x7f8edc131ab0 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ee26a8640 5 monclient: authenticate success, global_id 5063 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ee26a8640 10 monclient: _renew_subs 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ee26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc1315f0 con 0x7f8edc131ab0 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.597+0000 7f8ec3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400ad90 con 0x7f8edc131ab0 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ec3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ec3fff640 10 monclient: dump: 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 10 monclient: _renew_subs 2024-09-15T03:31:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc09b660 con 0x7f8edc131ab0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8eda7fc640 4 set_mon_vals no callback set 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8ed4007300 con 0x7f8edc131ab0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.598+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8edc04bf60 con 0x7f8edc131ab0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.599+0000 7f8ec3fff640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec8038740 0x7f8ec803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.599+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f8ed404bfc0 con 0x7f8edc131ab0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.599+0000 7f8edbfff640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec8038740 0x7f8ec803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.599+0000 7f8edbfff640 10 monclient: get_auth_request con 0x7f8ec8038740 auth_method 0 2024-09-15T03:31:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.601+0000 7f8edbfff640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec8038740 0x7f8ec803abf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc007ce0 tx=0x7f8ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.603+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8ed401b400 con 0x7f8edc131ab0 2024-09-15T03:31:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.603+0000 7f8ec3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.603+0000 7f8ec3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:47.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:47.751+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:47.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:47.751+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:47.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:47.751+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:47.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:47.751+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ee26a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ee26a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ee26a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8edc066280 con 0x7f8edc131ab0 2024-09-15T03:31:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ec3fff640 1 -- 172.21.15.153:0/1605187166 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f8ed4010770 con 0x7f8edc131ab0 2024-09-15T03:31:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ec3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.821+0000 7f8ec3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:47.823 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec8038740 msgr2=0x7f8ec803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec8038740 0x7f8ec803abf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc007ce0 tx=0x7f8ecc009040 comp rx=0 tx=0).stop 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 10 monclient: shutdown 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 msgr2=0x7f8edc12fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 0x7f8edc12fcc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400cb60 tx=0x7f8ed40052b0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 shutdown_connections 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec8038740 0x7f8ec803abf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc131ab0 0x7f8edc12fcc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8edc09f220 0x7f8edc12f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 --2- 172.21.15.153:0/1605187166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc09e910 0x7f8edc12f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 >> 172.21.15.153:0/1605187166 conn(0x7f8edc067d30 msgr2=0x7f8edc0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 shutdown_connections 2024-09-15T03:31:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:47.823+0000 7f8ee26a8640 1 -- 172.21.15.153:0/1605187166 wait complete. 2024-09-15T03:31:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:48.752+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:48.752+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:48.752+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:48.752+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:49.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:49.752+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:49.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:49.752+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:49.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:49.752+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:49.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:49.752+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:50.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:50.752+0000 7f3fe17fa640 10 monclient: tick 2024-09-15T03:31:50.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:50.752+0000 7f3fe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:50.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:50.752+0000 7f3fe17fa640 10 monclient: renew subs? -- yes 2024-09-15T03:31:50.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:50.752+0000 7f3fe17fa640 10 monclient: _renew_subs - empty 2024-09-15T03:31:50.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.940+0000 7f931395d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.940+0000 7f931395d640 10 monclient: build_initial_monmap 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.940+0000 7f931395d640 10 monclient: monmap: 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:50.940931+0000 2024-09-15T03:31:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:50.940931+0000 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:50.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.940+0000 7f931395d640 1 Processor -- start 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 -- start start 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient: init 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f930c1318f0 0x7f930c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): picked mon.noname-c con 0x7f930c1318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 0x7f930c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): picked mon.noname-b con 0x7f930c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 0x7f930c0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): picked mon.noname-a con 0x7f930c068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c131cc0 con 0x7f930c068d90 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c09fbf0 con 0x7f930c1318f0 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c09fe30 con 0x7f930c09ea50 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f931395d640 10 monclient(hunting): authenticate will time out at 908.580483s 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 0x7f930c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 0x7f930c0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 0x7f930c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43912/0 (socket says 172.21.15.153:43912) 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 -- 172.21.15.153:0/3265690952 learned_addr learned my addr 172.21.15.153:0/3265690952 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): get_auth_request con 0x7f930c09ea50 auth_method 0 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 10 monclient(hunting): get_auth_request con 0x7f930c068d90 auth_method 0 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 1 -- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 msgr2=0x7f930c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 1 -- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 msgr2=0x7f930c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 0x7f930c0691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9311efa640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 0x7f930c0691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f93116f9640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.181:3300/0 conn(0x7f930c1318f0 0x7f930c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f93116f9640 10 monclient(hunting): get_auth_request con 0x7f930c1318f0 auth_method 0 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f93116f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f93116f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f93116f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient(hunting): handle_auth_done global_id 5066 payload 274 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 -- 172.21.15.153:0/3265690952 >> v2:172.21.15.181:3300/0 conn(0x7f930c1318f0 msgr2=0x7f930c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.181:3300/0 conn(0x7f930c1318f0 0x7f930c131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 -- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 msgr2=0x7f930c0691f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 0x7f930c0691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient: _finish_hunting 0 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 monclient: found mon.noname-b 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.941+0000 7f9310ef8640 1 -- 172.21.15.153:0/3265690952 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0a1210 con 0x7f930c09ea50 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f9310ef8640 1 -- 172.21.15.153:0/3265690952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 msgr2=0x7f930c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f9310ef8640 1 --2- 172.21.15.153:0/3265690952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 0x7f930c09ee20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f930c076600 tx=0x7f9300009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f3b7cbc782942a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 1 -- 172.21.15.153:0/3265690952 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300007610 con 0x7f930c09ea50 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: dump: 2024-09-15T03:31:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: _finish_auth 0 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:50.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 1 -- 172.21.15.153:0/3265690952 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930002d050 con 0x7f930c09ea50 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f931395d640 5 monclient: authenticate success, global_id 5066 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f931395d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f931395d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 1 -- 172.21.15.153:0/3265690952 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93000079a0 con 0x7f930c09ea50 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f931395d640 4 set_mon_vals no callback set 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.944+0000 7f930bfff640 10 monclient: dump: 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.953 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 10 monclient: shutdown 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 -- 172.21.15.153:0/3265690952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 msgr2=0x7f930c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 --2- 172.21.15.153:0/3265690952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 0x7f930c09ee20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f930c076600 tx=0x7f9300009840 comp rx=0 tx=0).stop 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 -- 172.21.15.153:0/3265690952 shutdown_connections 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.153:3300/0 conn(0x7f930c068d90 0x7f930c0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 --2- 172.21.15.153:0/3265690952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c09ea50 0x7f930c09ee20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 --2- 172.21.15.153:0/3265690952 >> v2:172.21.15.181:3300/0 conn(0x7f930c1318f0 0x7f930c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 -- 172.21.15.153:0/3265690952 >> 172.21.15.153:0/3265690952 conn(0x7f930c067110 msgr2=0x7f930c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 -- 172.21.15.153:0/3265690952 shutdown_connections 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.945+0000 7f931395d640 1 -- 172.21.15.153:0/3265690952 wait complete. 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.946+0000 7f931395d640 10 monclient: build_initial_monmap 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.946+0000 7f931395d640 10 monclient: monmap: 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:50.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.946+0000 7f931395d640 1 Processor -- start 2024-09-15T03:31:50.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.946+0000 7f931395d640 1 -- start start 2024-09-15T03:31:50.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.946+0000 7f931395d640 10 monclient: init 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 0x7f930c09a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): picked mon.noname-c con 0x7f930c068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c09ea50 0x7f930c11bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): picked mon.noname-b con 0x7f930c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 0x7f930c130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): picked mon.noname-a con 0x7f930c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0a4420 con 0x7f930c1318f0 2024-09-15T03:31:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c11e260 con 0x7f930c09ea50 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c11df80 con 0x7f930c068d90 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f931395d640 10 monclient(hunting): authenticate will time out at 908.586104s 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 0x7f930c09a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 0x7f930c09a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43926/0 (socket says 172.21.15.153:43926) 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 -- 172.21.15.153:0/3620931310 learned_addr learned my addr 172.21.15.153:0/3620931310 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): get_auth_request con 0x7f930c068d90 auth_method 0 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:50.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 0x7f930c130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 10 monclient(hunting): get_auth_request con 0x7f930c1318f0 auth_method 0 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c09ea50 0x7f930c11bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): get_auth_request con 0x7f930c09ea50 auth_method 0 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 msgr2=0x7f930c130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 msgr2=0x7f930c130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 0x7f930c130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9311efa640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 0x7f930c130b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f9310ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient(hunting): handle_auth_done global_id 5069 payload 824 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c09ea50 msgr2=0x7f930c11bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c09ea50 0x7f930c11bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 msgr2=0x7f930c130b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 0x7f930c130b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient: _finish_hunting 0 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 monclient: found mon.noname-c 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.947+0000 7f93116f9640 1 -- 172.21.15.153:0/3620931310 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9300009ae0 con 0x7f930c068d90 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93116f9640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 msgr2=0x7f930c09a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93116f9640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 0x7f930c09a900 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00d890 tx=0x7f92fc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc00de20 con 0x7f930c068d90 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: dump: 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc00bae0 con 0x7f930c068d90 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 5 monclient: authenticate success, global_id 5069 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 10 monclient: _renew_subs 2024-09-15T03:31:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c11c740 con 0x7f930c068d90 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc00be80 con 0x7f930c068d90 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f93097fa640 10 monclient: dump: 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:50.967 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 10 monclient: _renew_subs 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c1310c0 con 0x7f930c068d90 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.949+0000 7f930b7fe640 4 set_mon_vals no callback set 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.950+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f92fc014640 con 0x7f930c068d90 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.950+0000 7f931395d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.950+0000 7f931395d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.950+0000 7f931395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.950+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c04bf60 con 0x7f930c068d90 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.950+0000 7f93097fa640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92f8038740 0x7f92f803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.951+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f92fc04c550 con 0x7f930c068d90 2024-09-15T03:31:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.951+0000 7f9310ef8640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92f8038740 0x7f92f803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.951+0000 7f9310ef8640 10 monclient: get_auth_request con 0x7f92f8038740 auth_method 0 2024-09-15T03:31:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.953+0000 7f9310ef8640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92f8038740 0x7f92f803abf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f930c064050 tx=0x7f9300036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.955+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f92fc011570 con 0x7f930c068d90 2024-09-15T03:31:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.955+0000 7f93097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:50.955+0000 7f93097fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f931395d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f931395d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f931395d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f930c0638d0 con 0x7f930c068d90 2024-09-15T03:31:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f93097fa640 1 -- 172.21.15.153:0/3620931310 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f92fc01dbb0 con 0x7f930c068d90 2024-09-15T03:31:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f93097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.169+0000 7f93097fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:51.171 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:51.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe3fff640 10 monclient: get_auth_request con 0x7f3fe409d4e0 auth_method 0 2024-09-15T03:31:51.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe3fff640 0 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:31:51.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient: ms_handle_reset current mon v2:172.21.15.153:3300/0 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 1 -- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 msgr2=0x7f3fe41360e0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 1 --2- 172.21.15.181:0/706260196 >> v2:172.21.15.153:3300/0 conn(0x7f3fe409d4e0 0x7f3fe41360e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient: _add_conns ranks=[0,2] 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fcc0034c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 1 --2- 172.21.15.181:0/706260196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fcc0073e0 0x7f3fcc009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3fcc0073e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 1 -- 172.21.15.181:0/706260196 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc00a130 con 0x7f3fcc0034c0 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 1 -- 172.21.15.181:0/706260196 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc00b2f0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.169+0000 7f3fe27fc640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.153:3300/0 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/706260196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fcc0073e0 0x7f3fcc009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): get_auth_request con 0x7f3fcc0073e0 auth_method 0 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient(hunting): handle_auth_done global_id 54168 payload 293 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 1 -- 172.21.15.181:0/706260196 >> v1:172.21.15.153:6789/0 conn(0x7f3fcc0034c0 legacy=0x7f3fcc005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:51.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.170+0000 7f3fe2ffd640 1 -- 172.21.15.181:0/706260196 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc00b520 con 0x7f3fcc0073e0 2024-09-15T03:31:51.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92f8038740 msgr2=0x7f92f803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92f8038740 0x7f92f803abf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f930c064050 tx=0x7f9300036000 comp rx=0 tx=0).stop 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 10 monclient: shutdown 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 msgr2=0x7f930c09a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 0x7f930c09a900 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00d890 tx=0x7f92fc008040 comp rx=0 tx=0).stop 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 shutdown_connections 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92f8038740 0x7f92f803abf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930c1318f0 0x7f930c130b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c09ea50 0x7f930c11bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 --2- 172.21.15.153:0/3620931310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f930c068d90 0x7f930c09a900 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 >> 172.21.15.153:0/3620931310 conn(0x7f930c067110 msgr2=0x7f930c134960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 shutdown_connections 2024-09-15T03:31:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:51.172+0000 7f931395d640 1 -- 172.21.15.153:0/3620931310 wait complete. 2024-09-15T03:31:51.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe2ffd640 1 -- 172.21.15.181:0/706260196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fcc0073e0 msgr2=0x7f3fcc009890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:51.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/706260196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fcc0073e0 0x7f3fcc009890 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800ba50 tx=0x7f3fd8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf56211581d6f5e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 1 -- 172.21.15.181:0/706260196 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd800d9f0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: dump: 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.186 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 1 -- 172.21.15.181:0/706260196 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd800db50 con 0x7f3fcc0073e0 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 1 -- 172.21.15.181:0/706260196 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8014430 con 0x7f3fcc0073e0 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe27fc640 10 monclient: dump: 2024-09-15T03:31:51.187 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe9cc9640 5 monclient: authenticate success, global_id 54168 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe9cc9640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:51.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.173+0000 7f3fe9cc9640 4 set_mon_vals no callback set 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 10 monclient: shutdown 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/706260196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fcc0073e0 msgr2=0x7f3fcc009890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/706260196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fcc0073e0 0x7f3fcc009890 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800ba50 tx=0x7f3fd8008040 comp rx=0 tx=0).stop 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe3fff640 1 -- 172.21.15.181:0/706260196 reap_dead start 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/706260196 shutdown_connections 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/706260196 >> 172.21.15.181:0/706260196 conn(0x7f3fe40711a0 msgr2=0x7f3fe40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/706260196 shutdown_connections 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.174+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/706260196 wait complete. 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.175+0000 7f3fe9cc9640 10 monclient: build_initial_monmap 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.175+0000 7f3fe9cc9640 10 monclient: monmap: 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.175+0000 7f3fe9cc9640 1 Processor -- start 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.175+0000 7f3fe9cc9640 1 -- start start 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.175+0000 7f3fe9cc9640 10 monclient: init 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 0x7f3fe413db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f3fcc0073e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 0x7f3fe413cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe4121150 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe413e590 0x7f3fe413d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe413e590 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe411fe30 con 0x7f3fe4121150 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4122040 con 0x7f3fcc0073e0 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4122320 con 0x7f3fe413e590 2024-09-15T03:31:51.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe9cc9640 10 monclient(hunting): authenticate will time out at 915.657350s 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 0x7f3fe413db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 0x7f3fe413db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46724/0 (socket says 172.21.15.181:46724) 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 -- 172.21.15.181:0/898669058 learned_addr learned my addr 172.21.15.181:0/898669058 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): get_auth_request con 0x7f3fcc0073e0 auth_method 0 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 0x7f3fe413cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe413e590 0x7f3fe413d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 10 monclient(hunting): get_auth_request con 0x7f3fe4121150 auth_method 0 2024-09-15T03:31:51.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): get_auth_request con 0x7f3fe413e590 auth_method 0 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 msgr2=0x7f3fe413cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 msgr2=0x7f3fe413cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:51.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:51.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 0x7f3fe413cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:51.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:51.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:51.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 0x7f3fe413cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:51.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:51.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient(hunting): handle_auth_done global_id 44506 payload 824 2024-09-15T03:31:51.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe413e590 msgr2=0x7f3fe413d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe413e590 0x7f3fe413d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 msgr2=0x7f3fe413cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:51.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 0x7f3fe413cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:51.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:51.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:51.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.176+0000 7f3fe37fe640 1 -- 172.21.15.181:0/898669058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4122600 con 0x7f3fcc0073e0 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe37fe640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 msgr2=0x7f3fe413db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe37fe640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 0x7f3fe413db20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400c510 tx=0x7f3fd4004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4005cb0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: dump: 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 5 monclient: authenticate success, global_id 44506 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 10 monclient: _renew_subs 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd4005e30 con 0x7f3fcc0073e0 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4141050 con 0x7f3fcc0073e0 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4007320 con 0x7f3fcc0073e0 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fcb7fe640 10 monclient: dump: 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.199 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe17fa640 4 set_mon_vals no callback set 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 10 monclient: _renew_subs 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.179+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe41414f0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3fd401b3b0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fe9cc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fe9cc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fe9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4141ed0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fcb7fe640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3fd0038770 0x7f3fd003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f3fd404b3a0 con 0x7f3fcc0073e0 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3fd0038770 0x7f3fd003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.180+0000 7f3fe2ffd640 10 monclient: get_auth_request con 0x7f3fd0038770 auth_method 0 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.183+0000 7f3fe2ffd640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3fd0038770 0x7f3fd003ac20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3fe409c7a0 tx=0x7f3fd801b000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.185+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3fd4018600 con 0x7f3fcc0073e0 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.185+0000 7f3fcb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:51.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.185+0000 7f3fcb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:51.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.346+0000 7f3fe9cc9640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:51.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.346+0000 7f3fe9cc9640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:51.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.347+0000 7f3fe9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.347+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f3fe4122180 con 0x7f3fcc0073e0 2024-09-15T03:31:51.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.347+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v69) v1 ==== 201+0+0 (secure 0 0 0) 0x7f3fe4122180 con 0x7f3fcc0073e0 2024-09-15T03:31:51.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.347+0000 7f3fcb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:51.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.347+0000 7f3fcb7fe640 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.442+0000 7f3fe9cc9640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.442+0000 7f3fe9cc9640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.442+0000 7f3fe9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.442+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f3fe4122180 con 0x7f3fcc0073e0 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.442+0000 7f3fcb7fe640 1 -- 172.21.15.181:0/898669058 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v69) v1 ==== 201+0+0 (secure 0 0 0) 0x7f3fe4122180 con 0x7f3fcc0073e0 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.443+0000 7f3fcb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.443+0000 7f3fcb7fe640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T03:31:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.444+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3fd0038770 msgr2=0x7f3fd003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.444+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3fd0038770 0x7f3fd003ac20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3fe409c7a0 tx=0x7f3fd801b000 comp rx=0 tx=0).stop 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.444+0000 7f3fe9cc9640 10 monclient: shutdown 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.444+0000 7f3fe9cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.444+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 msgr2=0x7f3fe413db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.444+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 0x7f3fe413db20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400c510 tx=0x7f3fd4004900 comp rx=0 tx=0).stop 2024-09-15T03:31:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 shutdown_connections 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3fd0038770 0x7f3fd003ac20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe413e590 0x7f3fe413d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe4121150 0x7f3fe413cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 --2- 172.21.15.181:0/898669058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0073e0 0x7f3fe413db20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 >> 172.21.15.181:0/898669058 conn(0x7f3fe40711a0 msgr2=0x7f3fe406dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 shutdown_connections 2024-09-15T03:31:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.445+0000 7f3fe9cc9640 1 -- 172.21.15.181:0/898669058 wait complete. 2024-09-15T03:31:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-15T03:31:51.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.529+0000 7fcca779d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:51.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.529+0000 7fcca779d640 10 monclient: build_initial_monmap 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.529+0000 7fcca779d640 10 monclient: monmap: 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:51.530037+0000 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:51.530037+0000 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:51.531 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.529+0000 7fcca779d640 1 Processor -- start 2024-09-15T03:31:51.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 1 -- start start 2024-09-15T03:31:51.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient: init 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcca009ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 0x7fcca0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): picked mon.noname-b con 0x7fcca009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcca0136d60 0x7fcca0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): picked mon.noname-c con 0x7fcca0136d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca0067420 con 0x7fcca009ec50 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca009fdc0 con 0x7fcca0136d60 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca00a0510 con 0x7fcca009f020 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca779d640 10 monclient(hunting): authenticate will time out at 916.011498s 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcca0136d60 0x7fcca0139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcca0136d60 0x7fcca0139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46738/0 (socket says 172.21.15.181:46738) 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 1 -- 172.21.15.181:0/829119021 learned_addr learned my addr 172.21.15.181:0/829119021 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 10 monclient(hunting): get_auth_request con 0x7fcca0136d60 auth_method 0 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca5539640 1 --1- 172.21.15.181:0/829119021 >> v1:172.21.15.181:6789/0 conn(0x7fcca009ec50 0x7fcca0131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50762/0 (socket says 172.21.15.181:50762) 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 1 --2- 172.21.15.181:0/829119021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 0x7fcca0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): get_auth_request con 0x7fcca009f020 auth_method 0 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.530+0000 7fcca4d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 10 monclient(hunting): handle_auth_done global_id 5075 payload 274 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 1 -- 172.21.15.181:0/829119021 >> v2:172.21.15.181:3300/0 conn(0x7fcca0136d60 msgr2=0x7fcca0139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:51.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 1 --2- 172.21.15.181:0/829119021 >> v2:172.21.15.181:3300/0 conn(0x7fcca0136d60 0x7fcca0139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 1 -- 172.21.15.181:0/829119021 >> v1:172.21.15.181:6789/0 conn(0x7fcca009ec50 legacy=0x7fcca0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:51.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 10 monclient: _finish_hunting 0 2024-09-15T03:31:51.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 1 monclient: found mon.noname-b 2024-09-15T03:31:51.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:51.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:51.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.531+0000 7fcca4d38640 1 -- 172.21.15.181:0/829119021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca00a0200 con 0x7fcca009f020 2024-09-15T03:31:51.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.533+0000 7fcca4d38640 1 -- 172.21.15.181:0/829119021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 msgr2=0x7fcca0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:51.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.533+0000 7fcca4d38640 1 --2- 172.21.15.181:0/829119021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 0x7fcca0069d10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009850 tx=0x7fcc900090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d4a623e47c723bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:51.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 1 -- 172.21.15.181:0/829119021 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9002adf0 con 0x7fcca009f020 2024-09-15T03:31:51.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: dump: 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.546 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.547 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.547 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.547 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.547 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:51.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:51.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 5 monclient: authenticate success, global_id 5075 2024-09-15T03:31:51.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:51.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 1 -- 172.21.15.181:0/829119021 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc90032440 con 0x7fcca009f020 2024-09-15T03:31:51.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:51.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 4 set_mon_vals no callback set 2024-09-15T03:31:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 1 -- 172.21.15.181:0/829119021 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc90032820 con 0x7fcca009f020 2024-09-15T03:31:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcc97fff640 10 monclient: dump: 2024-09-15T03:31:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.550 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.550 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.550 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.550 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.550 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.550 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.551 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.551 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.551 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 10 monclient: shutdown 2024-09-15T03:31:51.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:51.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 -- 172.21.15.181:0/829119021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 msgr2=0x7fcca0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 --2- 172.21.15.181:0/829119021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 0x7fcca0069d10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009850 tx=0x7fcc900090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 -- 172.21.15.181:0/829119021 shutdown_connections 2024-09-15T03:31:51.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 --2- 172.21.15.181:0/829119021 >> v2:172.21.15.181:3300/0 conn(0x7fcca0136d60 0x7fcca0139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 --2- 172.21.15.181:0/829119021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca009f020 0x7fcca0069d10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 -- 172.21.15.181:0/829119021 >> 172.21.15.181:0/829119021 conn(0x7fcca00a5d40 msgr2=0x7fcca00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 -- 172.21.15.181:0/829119021 shutdown_connections 2024-09-15T03:31:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.534+0000 7fcca779d640 1 -- 172.21.15.181:0/829119021 wait complete. 2024-09-15T03:31:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.535+0000 7fcca779d640 10 monclient: build_initial_monmap 2024-09-15T03:31:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.535+0000 7fcca779d640 10 monclient: monmap: 2024-09-15T03:31:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.535+0000 7fcca779d640 1 Processor -- start 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.535+0000 7fcca779d640 1 -- start start 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.535+0000 7fcca779d640 10 monclient: init 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:51.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 0x7fcca00970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): picked mon.noname-b con 0x7fcca009ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:51.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca009f020 0x7fcca0097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): picked mon.noname-a con 0x7fcca009f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:51.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca0136d60 0x7fcca0097b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): picked mon.noname-c con 0x7fcca0136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca009c000 con 0x7fcca009f020 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca009a840 con 0x7fcca009ec50 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca009d8f0 con 0x7fcca0136d60 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:51.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca779d640 10 monclient(hunting): authenticate will time out at 916.017311s 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 0x7fcca00970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 0x7fcca00970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46754/0 (socket says 172.21.15.181:46754) 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 -- 172.21.15.181:0/3198730320 learned_addr learned my addr 172.21.15.181:0/3198730320 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): get_auth_request con 0x7fcca009ec50 auth_method 0 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:51.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:51.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:51.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:51.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:51.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient(hunting): handle_auth_done global_id 44512 payload 824 2024-09-15T03:31:51.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5d3a640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca0136d60 0x7fcca0097b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 -- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca0136d60 msgr2=0x7fcca0097b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca4d38640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca009f020 0x7fcca0097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca0136d60 0x7fcca0097b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 -- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca009f020 msgr2=0x7fcca0097600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:51.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca009f020 0x7fcca0097600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:51.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient: _finish_hunting 0 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 monclient: found mon.noname-b 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.536+0000 7fcca5539640 1 -- 172.21.15.181:0/3198730320 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc90009390 con 0x7fcca009ec50 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.538+0000 7fcca5539640 1 -- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 msgr2=0x7fcca00970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.538+0000 7fcca5539640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 0x7fcca00970c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00c5b0 tx=0x7fcc9c007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c0075f0 con 0x7fcca009ec50 2024-09-15T03:31:51.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: dump: 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.562 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 5 monclient: authenticate success, global_id 44512 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 10 monclient: _renew_subs 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca0098c80 con 0x7fcca009ec50 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c0057d0 con 0x7fcca009ec50 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c005b90 con 0x7fcca009ec50 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 10 monclient: dump: 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:51.564 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc977fe640 4 set_mon_vals no callback set 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 10 monclient: _renew_subs 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca009ad70 con 0x7fcca009ec50 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.539+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcc9c01b400 con 0x7fcca009ec50 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcca779d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:51.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcca779d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcca779d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcc957fa640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc84038740 0x7fcc8403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca0099af0 con 0x7fcca009ec50 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5800+0+0 (secure 0 0 0) 0x7fcc9c04bb30 con 0x7fcca009ec50 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcca4d38640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc84038740 0x7fcc8403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.540+0000 7fcca4d38640 10 monclient: get_auth_request con 0x7fcc84038740 auth_method 0 2024-09-15T03:31:51.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.542+0000 7fcca4d38640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc84038740 0x7fcc8403abf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcc90034000 tx=0x7fcc90032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:51.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.544+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcc9c04de60 con 0x7fcca009ec50 2024-09-15T03:31:51.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.544+0000 7fcc957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:51.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.544+0000 7fcc957fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.702+0000 7fcca779d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.702+0000 7fcca779d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.702+0000 7fcca779d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:51.702+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fcca00008d0 con 0x7fcca009ec50 2024-09-15T03:31:52.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.686+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v70) v1 ==== 203+0+0 (secure 0 0 0) 0x7fcc9c01b6b0 con 0x7fcca009ec50 2024-09-15T03:31:52.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.686+0000 7fcc957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:52.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.686+0000 7fcc957fa640 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-15T03:31:52.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.783+0000 7fcca779d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:52.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.783+0000 7fcca779d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:52.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.783+0000 7fcca779d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:52.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.783+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fcca0099870 con 0x7fcca009ec50 2024-09-15T03:31:52.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.784+0000 7fcc957fa640 1 -- 172.21.15.181:0/3198730320 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v70) v1 ==== 201+0+0 (secure 0 0 0) 0x7fcc9c01a360 con 0x7fcca009ec50 2024-09-15T03:31:52.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.784+0000 7fcc957fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:52.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.784+0000 7fcc957fa640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T03:31:52.786 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T03:31:52.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc84038740 msgr2=0x7fcc8403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:52.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc84038740 0x7fcc8403abf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcc90034000 tx=0x7fcc90032000 comp rx=0 tx=0).stop 2024-09-15T03:31:52.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 10 monclient: shutdown 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 msgr2=0x7fcca00970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 0x7fcca00970c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00c5b0 tx=0x7fcc9c007f00 comp rx=0 tx=0).stop 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 shutdown_connections 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc84038740 0x7fcc8403abf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca0136d60 0x7fcca0097b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca009f020 0x7fcca0097600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 --2- 172.21.15.181:0/3198730320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcca009ec50 0x7fcca00970c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 >> 172.21.15.181:0/3198730320 conn(0x7fcca00a5d40 msgr2=0x7fcca009e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:52.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 shutdown_connections 2024-09-15T03:31:52.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.786+0000 7fcca779d640 1 -- 172.21.15.181:0/3198730320 wait complete. 2024-09-15T03:31:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-15T03:31:52.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.870+0000 7f3bb199d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:52.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.870+0000 7f3bb199d640 10 monclient: build_initial_monmap 2024-09-15T03:31:52.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.870+0000 7f3bb199d640 10 monclient: monmap: 2024-09-15T03:31:52.871 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:52.871 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:52.871 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:52.871174+0000 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:52.871174+0000 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:52.872 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.870+0000 7f3bb199d640 1 Processor -- start 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 1 -- start start 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient: init 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 0x7f3bac131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bac09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bac1321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bac09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac067420 con 0x7f3bac1321a0 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac09fdc0 con 0x7f3bac09f020 2024-09-15T03:31:52.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bac0a0510 con 0x7f3bac09ec50 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bb199d640 10 monclient(hunting): authenticate will time out at 917.352578s 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bab7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3bac09f020 0x7f3bac069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50766/0 (socket says 172.21.15.181:50766) 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3bab7fe640 1 -- 172.21.15.181:0/3923103040 learned_addr learned my addr 172.21.15.181:0/3923103040 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3baaffd640 1 --2- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 0x7f3bac131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3baaffd640 10 monclient(hunting): get_auth_request con 0x7f3bac09ec50 auth_method 0 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3baaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3baaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.871+0000 7f3baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.872+0000 7f3baaffd640 1 -- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 msgr2=0x7f3bac131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.872+0000 7f3baaffd640 1 -- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 msgr2=0x7f3bac131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:52.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.872+0000 7f3baaffd640 1 --2- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 0x7f3bac131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:52.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.872+0000 7f3baaffd640 1 --2- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 0x7f3bac131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:52.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.873+0000 7f3bab7fe640 1 -- 172.21.15.181:0/3923103040 >> v1:172.21.15.181:6789/0 conn(0x7f3bac09f020 legacy=0x7f3bac069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:52.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560718212 0 0) 0x7f3bac09fdc0 con 0x7f3bac09f020 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): my global_id is 44515 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 -- 172.21.15.181:0/3923103040 >> v1:172.21.15.153:6789/0 conn(0x7f3bac1321a0 legacy=0x7f3bac068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b940034e0 con 0x7f3bac09f020 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899021487 0 0) 0x7f3b940034e0 con 0x7f3bac09f020 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 msgr2=0x7f3bac131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 --2- 172.21.15.181:0/3923103040 >> v2:172.21.15.153:3300/0 conn(0x7f3bac09ec50 0x7f3bac131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 >> v1:172.21.15.153:6789/0 conn(0x7f3bac1321a0 legacy=0x7f3bac068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:31:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac0a0200 con 0x7f3bac09f020 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bb199d640 5 monclient: authenticate success, global_id 44515 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f3ba4001aa0 con 0x7f3bac09f020 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bb199d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: dump: 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 >> v1:172.21.15.181:6789/0 conn(0x7f3bac09f020 legacy=0x7f3bac069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b94003ad0 0x7f3b94003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): picked mon.c con 0x7f3b94003ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:52.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3b94004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): picked mon.b con 0x7f3b94004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b94005330 0x7f3b9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): picked mon.a con 0x7f3b94005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0a0510 con 0x7f3b94005330 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9400c2f0 con 0x7f3b94004780 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9400c4b0 con 0x7f3b94003ad0 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3ba9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bb199d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3b94004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): get_auth_request con 0x7f3b94004780 auth_method 0 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b94003ad0 0x7f3b94003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): get_auth_request con 0x7f3b94003ad0 auth_method 0 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b94005330 0x7f3b9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 10 monclient(hunting): get_auth_request con 0x7f3b94005330 auth_method 0 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient(hunting): handle_auth_done global_id 44515 payload 293 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 -- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b94003ad0 msgr2=0x7f3b94003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b94003ad0 0x7f3b94003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 -- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b94005330 msgr2=0x7f3b9400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b94005330 0x7f3b9400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 monclient: found mon.b 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 1 -- 172.21.15.181:0/3923103040 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940034e0 con 0x7f3b94004780 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3baa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b94005330 0x7f3b9400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.874+0000 7f3bab7fe640 1 -- 172.21.15.181:0/3923103040 reap_dead start 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3baa7fc640 1 -- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 msgr2=0x7f3b94004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3b94004bf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0001b00 tx=0x7f3ba0007c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bcfd037019f0523 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba002ac50 con 0x7f3b94004780 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: dump: 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3bb199d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba002adb0 con 0x7f3b94004780 2024-09-15T03:31:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 1 -- 172.21.15.181:0/3923103040 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0007340 con 0x7f3b94004780 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3bb199d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3bb199d640 4 set_mon_vals no callback set 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.877+0000 7f3ba9ffb640 10 monclient: dump: 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:52.891 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 10 monclient: shutdown 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 -- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 msgr2=0x7f3b94004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3b94004bf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0001b00 tx=0x7f3ba0007c40 comp rx=0 tx=0).stop 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 -- 172.21.15.181:0/3923103040 shutdown_connections 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 --2- 172.21.15.181:0/3923103040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3b94004bf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 -- 172.21.15.181:0/3923103040 >> 172.21.15.181:0/3923103040 conn(0x7f3bac0a5d40 msgr2=0x7f3bac0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 -- 172.21.15.181:0/3923103040 shutdown_connections 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.878+0000 7f3bb199d640 1 -- 172.21.15.181:0/3923103040 wait complete. 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient: build_initial_monmap 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient: monmap: 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:52.893 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 Processor -- start 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 -- start start 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient: init 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3bac09a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b94004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bac1362e0 0x7f3bac130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): picked mon.noname-c con 0x7f3bac1362e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 0x7f3bac11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bac130ec0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac09c000 con 0x7f3bac130ec0 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1349e0 con 0x7f3b94004780 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac11dd80 con 0x7f3bac1362e0 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3bb199d640 10 monclient(hunting): authenticate will time out at 917.360845s 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3bac09a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3bac09a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46768/0 (socket says 172.21.15.181:46768) 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 1 -- 172.21.15.181:0/1119038786 learned_addr learned my addr 172.21.15.181:0/1119038786 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 10 monclient(hunting): get_auth_request con 0x7f3b94004780 auth_method 0 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.879+0000 7f3baaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 0x7f3bac11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 10 monclient(hunting): get_auth_request con 0x7f3bac130ec0 auth_method 0 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bac1362e0 0x7f3bac130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baa7fc640 10 monclient(hunting): get_auth_request con 0x7f3bac1362e0 auth_method 0 2024-09-15T03:31:52.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 10 monclient(hunting): handle_auth_done global_id 44518 payload 824 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 msgr2=0x7f3bac11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 msgr2=0x7f3bac11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bac1362e0 msgr2=0x7f3bac130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 0x7f3bac11b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bac1362e0 0x7f3bac130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 msgr2=0x7f3bac11b760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3bab7fe640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 0x7f3bac11b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 0x7f3bac11b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:52.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:52.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:52.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.880+0000 7f3baaffd640 1 -- 172.21.15.181:0/1119038786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac11daa0 con 0x7f3b94004780 2024-09-15T03:31:52.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3baaffd640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 msgr2=0x7f3bac09a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:52.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3baaffd640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3bac09a6a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c010a80 tx=0x7f3b9c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:52.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c00eca0 con 0x7f3b94004780 2024-09-15T03:31:52.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: dump: 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:52.901 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c00ee20 con 0x7f3b94004780 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 5 monclient: authenticate success, global_id 44518 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 10 monclient: _renew_subs 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0166d0 con 0x7f3b94004780 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac09b360 con 0x7f3b94004780 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3b92ffd640 10 monclient: dump: 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:52.902 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3ba97fa640 4 set_mon_vals no callback set 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 10 monclient: _renew_subs 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.882+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac11bca0 con 0x7f3b94004780 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3b9c016a40 con 0x7f3b94004780 2024-09-15T03:31:52.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3bb199d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3bb199d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3bb199d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bac141d00 con 0x7f3b94004780 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3b92ffd640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b98038740 0x7f3b9803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6044+0+0 (secure 0 0 0) 0x7f3b9c04cb60 con 0x7f3b94004780 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b98038740 0x7f3b9803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.883+0000 7f3baa7fc640 10 monclient: get_auth_request con 0x7f3b98038740 auth_method 0 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.886+0000 7f3baa7fc640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b98038740 0x7f3b9803abf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ba002f000 tx=0x7f3ba0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.887+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3b9c014430 con 0x7f3b94004780 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.887+0000 7f3b92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:52.887+0000 7f3b92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.045+0000 7f3bb199d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.045+0000 7f3bb199d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.045+0000 7f3bb199d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.045+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f3bac09b360 con 0x7f3b94004780 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.046+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v70) v1 ==== 201+0+0 (secure 0 0 0) 0x7f3bac09b360 con 0x7f3b94004780 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.046+0000 7f3b92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:53.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.046+0000 7f3b92ffd640 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.141+0000 7f3bb199d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.141+0000 7f3bb199d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.141+0000 7f3bb199d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.141+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f3bac09b300 con 0x7f3b94004780 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.142+0000 7f3b92ffd640 1 -- 172.21.15.181:0/1119038786 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v70) v1 ==== 201+0+0 (secure 0 0 0) 0x7f3b9c027b40 con 0x7f3b94004780 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.142+0000 7f3b92ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-15T03:31:53.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.142+0000 7f3b92ffd640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T03:31:53.144 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.143+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b98038740 msgr2=0x7f3b9803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.143+0000 7f3bb199d640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b98038740 0x7f3b9803abf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ba002f000 tx=0x7f3ba0030000 comp rx=0 tx=0).stop 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 10 monclient: shutdown 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 msgr2=0x7f3bac09a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3bac09a6a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c010a80 tx=0x7f3b9c008040 comp rx=0 tx=0).stop 2024-09-15T03:31:53.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 shutdown_connections 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b98038740 0x7f3b9803abf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bac130ec0 0x7f3bac11b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bac1362e0 0x7f3bac130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 --2- 172.21.15.181:0/1119038786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b94004780 0x7f3bac09a6a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 >> 172.21.15.181:0/1119038786 conn(0x7f3bac0a5d40 msgr2=0x7f3bac069570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 shutdown_connections 2024-09-15T03:31:53.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.144+0000 7f3bb199d640 1 -- 172.21.15.181:0/1119038786 wait complete. 2024-09-15T03:31:53.153 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket bar root 2024-09-15T03:31:53.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.226+0000 7f657d31d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.226+0000 7f657d31d640 10 monclient: build_initial_monmap 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.226+0000 7f657d31d640 10 monclient: monmap: 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:53.227594+0000 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:53.227594+0000 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.228 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:53.229 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:53.229 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:53.229 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:53.229 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:53.229 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 1 Processor -- start 2024-09-15T03:31:53.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 1 -- start start 2024-09-15T03:31:53.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient: init 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 0x7f657809c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient(hunting): picked mon.noname-b con 0x7f657809cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f657809cea0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f657809d270 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657809b710 con 0x7f657809d270 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657809c6e0 con 0x7f657809cea0 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657809e8f0 con 0x7f657809cad0 2024-09-15T03:31:53.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.227+0000 7f657d31d640 10 monclient(hunting): authenticate will time out at 917.708946s 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f65767fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f657809cea0 0x7f657809dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50778/0 (socket says 172.21.15.181:50778) 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f65767fc640 1 -- 172.21.15.181:0/2691889513 learned_addr learned my addr 172.21.15.181:0/2691889513 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 1 --2- 172.21.15.181:0/2691889513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 0x7f657809c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): get_auth_request con 0x7f657809cad0 auth_method 0 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:53.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient(hunting): handle_auth_done global_id 5081 payload 274 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 1 -- 172.21.15.181:0/2691889513 >> v1:172.21.15.181:6789/0 conn(0x7f657809cea0 legacy=0x7f657809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 1 -- 172.21.15.181:0/2691889513 >> v1:172.21.15.153:6789/0 conn(0x7f657809d270 legacy=0x7f657809e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:53.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.228+0000 7f6576ffd640 1 -- 172.21.15.181:0/2691889513 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657809a650 con 0x7f657809cad0 2024-09-15T03:31:53.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6576ffd640 1 -- 172.21.15.181:0/2691889513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 msgr2=0x7f657809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:53.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6576ffd640 1 --2- 172.21.15.181:0/2691889513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 0x7f657809c1a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f657809c870 tx=0x7f65700090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b51cfaba2ed2486 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 1 -- 172.21.15.181:0/2691889513 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570032440 con 0x7f657809cad0 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: dump: 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.241 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 1 -- 172.21.15.181:0/2691889513 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65700325a0 con 0x7f657809cad0 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f657d31d640 5 monclient: authenticate success, global_id 5081 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f657d31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f657d31d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 1 -- 172.21.15.181:0/2691889513 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570032940 con 0x7f657809cad0 2024-09-15T03:31:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f657d31d640 4 set_mon_vals no callback set 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.231+0000 7f6575ffb640 10 monclient: dump: 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 10 monclient: shutdown 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 -- 172.21.15.181:0/2691889513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 msgr2=0x7f657809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 --2- 172.21.15.181:0/2691889513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 0x7f657809c1a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f657809c870 tx=0x7f65700090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 -- 172.21.15.181:0/2691889513 shutdown_connections 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 --2- 172.21.15.181:0/2691889513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cad0 0x7f657809c1a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 -- 172.21.15.181:0/2691889513 >> 172.21.15.181:0/2691889513 conn(0x7f6578070e90 msgr2=0x7f65780a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 -- 172.21.15.181:0/2691889513 shutdown_connections 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.232+0000 7f657d31d640 1 -- 172.21.15.181:0/2691889513 wait complete. 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient: build_initial_monmap 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient: monmap: 2024-09-15T03:31:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 1 Processor -- start 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 1 -- start start 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient: init 2024-09-15T03:31:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 0x7f657813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient(hunting): picked mon.noname-a con 0x7f657809cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cea0 0x7f657812fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.233+0000 7f657d31d640 10 monclient(hunting): picked mon.noname-c con 0x7f657809cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 0x7f6578130010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 10 monclient(hunting): picked mon.noname-b con 0x7f657809d270 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657811ced0 con 0x7f657809cad0 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657806eba0 con 0x7f657809d270 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65780a2be0 con 0x7f657809cea0 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f657d31d640 10 monclient(hunting): authenticate will time out at 917.715038s 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 0x7f6578130010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 0x7f6578130010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46782/0 (socket says 172.21.15.181:46782) 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 -- 172.21.15.181:0/537223427 learned_addr learned my addr 172.21.15.181:0/537223427 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): get_auth_request con 0x7f657809d270 auth_method 0 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 0x7f657813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65767fc640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cea0 0x7f657812fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 10 monclient(hunting): get_auth_request con 0x7f657809cad0 auth_method 0 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65767fc640 10 monclient(hunting): get_auth_request con 0x7f657809cea0 auth_method 0 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient(hunting): handle_auth_done global_id 44521 payload 824 2024-09-15T03:31:53.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 msgr2=0x7f657813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cea0 msgr2=0x7f657812fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cea0 0x7f657812fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 msgr2=0x7f657813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 msgr2=0x7f657813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 0x7f657813ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f6576ffd640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 0x7f657813ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 0x7f657813ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.234+0000 7f65777fe640 1 -- 172.21.15.181:0/537223427 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570009390 con 0x7f657809d270 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f65777fe640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 msgr2=0x7f6578130010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f65777fe640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 0x7f6578130010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f656800c890 tx=0x7f6568002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800ce80 con 0x7f657809d270 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: dump: 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.250 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: _finish_auth 0 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 5 monclient: authenticate success, global_id 44521 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 10 monclient: _renew_subs 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657813f730 con 0x7f657809d270 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656800aa00 con 0x7f657809d270 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800adc0 con 0x7f657809d270 2024-09-15T03:31:53.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f655effd640 10 monclient: dump: 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 10 monclient: _renew_subs 2024-09-15T03:31:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f65757fa640 4 set_mon_vals no callback set 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.236+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657811b0c0 con 0x7f657809d270 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6568012630 con 0x7f657809d270 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f657d31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f657d31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f657d31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65780008d0 con 0x7f657809d270 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f655effd640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6564038740 0x7f656403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.237+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6044+0+0 (secure 0 0 0) 0x7f656804c3c0 con 0x7f657809d270 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.238+0000 7f6576ffd640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6564038740 0x7f656403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.238+0000 7f6576ffd640 10 monclient: get_auth_request con 0x7f6564038740 auth_method 0 2024-09-15T03:31:53.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.240+0000 7f6576ffd640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6564038740 0x7f656403abf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6570031000 tx=0x7f6570032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:53.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.242+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6568022d70 con 0x7f657809d270 2024-09-15T03:31:53.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.242+0000 7f655effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:53.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.242+0000 7f655effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:53.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.399+0000 7f657d31d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:53.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.399+0000 7f657d31d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:53.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.399+0000 7f657d31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.399+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f657809f660 con 0x7f657809d270 2024-09-15T03:31:53.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.687+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65680160a0 con 0x7f657809d270 2024-09-15T03:31:53.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.687+0000 7f655effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:53.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.687+0000 7f655effd640 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-15T03:31:53.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f657d31d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:53.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f657d31d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:53.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f657d31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f657813f410 con 0x7f657809d270 2024-09-15T03:31:53.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f655effd640 1 -- 172.21.15.181:0/537223427 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v71) v1 ==== 126+0+0 (secure 0 0 0) 0x7f6568022770 con 0x7f657809d270 2024-09-15T03:31:53.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f655effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:53.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.785+0000 7f655effd640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-15T03:31:53.787 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'bar' already exists 2024-09-15T03:31:53.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6564038740 msgr2=0x7f656403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6564038740 0x7f656403abf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6570031000 tx=0x7f6570032000 comp rx=0 tx=0).stop 2024-09-15T03:31:53.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 10 monclient: shutdown 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 msgr2=0x7f6578130010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 0x7f6578130010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f656800c890 tx=0x7f6568002640 comp rx=0 tx=0).stop 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 shutdown_connections 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6564038740 0x7f656403abf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f657809d270 0x7f6578130010 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f657809cea0 0x7f657812fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 --2- 172.21.15.181:0/537223427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f657809cad0 0x7f657813ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 >> 172.21.15.181:0/537223427 conn(0x7f6578070e90 msgr2=0x7f6578132a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 shutdown_connections 2024-09-15T03:31:53.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.787+0000 7f657d31d640 1 -- 172.21.15.181:0/537223427 wait complete. 2024-09-15T03:31:53.797 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket bar root 2024-09-15T03:31:53.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.870+0000 7f2425f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.870+0000 7f2425f5d640 10 monclient: build_initial_monmap 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.870+0000 7f2425f5d640 10 monclient: monmap: 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:53.871521+0000 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:53.871521+0000 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:53.873 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 1 Processor -- start 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 1 -- start start 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient: init 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2420098c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 0x7f2420136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2420099c70 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2420137200 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242009f680 con 0x7f2420137200 2024-09-15T03:31:53.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242009f420 con 0x7f2420098c70 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2420068a20 con 0x7f2420099c70 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.871+0000 7f2425f5d640 10 monclient(hunting): authenticate will time out at 918.352980s 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241f7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f2420098c70 0x7f2420099040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50782/0 (socket says 172.21.15.181:50782) 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241f7fe640 1 -- 172.21.15.181:0/1464694867 learned_addr learned my addr 172.21.15.181:0/1464694867 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 0x7f2420136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 10 monclient(hunting): get_auth_request con 0x7f2420099c70 auth_method 0 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 1 -- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 msgr2=0x7f2420136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:53.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 1 -- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 msgr2=0x7f2420136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 0x7f2420136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.872+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 0x7f2420136cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241f7fe640 1 -- 172.21.15.181:0/1464694867 >> v1:172.21.15.181:6789/0 conn(0x7f2420098c70 legacy=0x7f2420099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241ffff640 1 -- 172.21.15.181:0/1464694867 >> v1:172.21.15.153:6789/0 conn(0x7f2420137200 legacy=0x7f24201395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945376309 0 0) 0x7f242009f420 con 0x7f2420098c70 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): my global_id is 44524 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24080034e0 con 0x7f2420098c70 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076473111 0 0) 0x7f24080034e0 con 0x7f2420098c70 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 msgr2=0x7f2420136cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:53.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 --2- 172.21.15.181:0/1464694867 >> v2:172.21.15.153:3300/0 conn(0x7f2420099c70 0x7f2420136cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 >> v1:172.21.15.153:6789/0 conn(0x7f2420137200 legacy=0x7f24201395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2420096a30 con 0x7f2420098c70 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f2410001b60 con 0x7f2420098c70 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: dump: 2024-09-15T03:31:53.881 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 >> v1:172.21.15.181:6789/0 conn(0x7f2420098c70 legacy=0x7f2420099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:53.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408003a50 0x7f2408003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): picked mon.c con 0x7f2408003a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 0x7f2408004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): picked mon.b con 0x7f2408004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2408005330 0x7f240800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): picked mon.a con 0x7f2408005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2420068a20 con 0x7f2408005330 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240800c2d0 con 0x7f2408004780 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240800c490 con 0x7f2408003a50 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:53.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 0x7f2408004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): get_auth_request con 0x7f2408004780 auth_method 0 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241f7fe640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408003a50 0x7f2408003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241ffff640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2408005330 0x7f240800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241f7fe640 10 monclient(hunting): get_auth_request con 0x7f2408003a50 auth_method 0 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.874+0000 7f241ffff640 10 monclient(hunting): get_auth_request con 0x7f2408005330 auth_method 0 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 10 monclient(hunting): handle_auth_done global_id 44524 payload 293 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 1 -- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408003a50 msgr2=0x7f2408003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408003a50 0x7f2408003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 1 -- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2408005330 msgr2=0x7f240800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2408005330 0x7f240800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 1 monclient: found mon.b 2024-09-15T03:31:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 1 -- 172.21.15.181:0/1464694867 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24080034e0 con 0x7f2408004780 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 10 monclient: _finish_auth 0 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f2425f5d640 5 monclient: authenticate success, global_id 44524 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f2425f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241ffff640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2408005330 0x7f240800b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241f7fe640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408003a50 0x7f2408003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.875+0000 7f241ffff640 1 -- 172.21.15.181:0/1464694867 reap_dead start 2024-09-15T03:31:53.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.877+0000 7f241effd640 1 -- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 msgr2=0x7f2408004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:53.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.877+0000 7f241effd640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 0x7f2408004bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2414006fd0 tx=0x7f2414008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bea62fd026a7f9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:53.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.877+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241402add0 con 0x7f2408004780 2024-09-15T03:31:53.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.877+0000 7f241e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 10 monclient: dump: 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.892 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241402e070 con 0x7f2408004780 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 1 -- 172.21.15.181:0/1464694867 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414034660 con 0x7f2408004780 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 4 set_mon_vals no callback set 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f241e7fc640 10 monclient: dump: 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.893 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 10 monclient: shutdown 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 -- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 msgr2=0x7f2408004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 0x7f2408004bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2414006fd0 tx=0x7f2414008040 comp rx=0 tx=0).stop 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 -- 172.21.15.181:0/1464694867 shutdown_connections 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 --2- 172.21.15.181:0/1464694867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2408004780 0x7f2408004bf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 -- 172.21.15.181:0/1464694867 >> 172.21.15.181:0/1464694867 conn(0x7f2420070e90 msgr2=0x7f24200a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 -- 172.21.15.181:0/1464694867 shutdown_connections 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.878+0000 7f2425f5d640 1 -- 172.21.15.181:0/1464694867 wait complete. 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.879+0000 7f2425f5d640 10 monclient: build_initial_monmap 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.879+0000 7f2425f5d640 10 monclient: monmap: 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:53.895 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.879+0000 7f2425f5d640 1 Processor -- start 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.879+0000 7f2425f5d640 1 -- start start 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.879+0000 7f2425f5d640 10 monclient: init 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408004780 0x7f242011e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2408004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f242011ed00 0x7f242011f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f242011ed00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 0x7f24201476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f242011f630 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201202f0 con 0x7f242011ed00 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242006eba0 con 0x7f242011f630 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2420097b50 con 0x7f2408004780 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f2425f5d640 10 monclient(hunting): authenticate will time out at 918.361188s 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 0x7f24201476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 0x7f24201476a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46796/0 (socket says 172.21.15.181:46796) 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 -- 172.21.15.181:0/780211763 learned_addr learned my addr 172.21.15.181:0/780211763 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): get_auth_request con 0x7f242011f630 auth_method 0 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241effd640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f242011ed00 0x7f242011f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241f7fe640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408004780 0x7f242011e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241effd640 10 monclient(hunting): get_auth_request con 0x7f242011ed00 auth_method 0 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241f7fe640 10 monclient(hunting): get_auth_request con 0x7f2408004780 auth_method 0 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient(hunting): handle_auth_done global_id 44527 payload 824 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 -- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408004780 msgr2=0x7f242011e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408004780 0x7f242011e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 -- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f242011ed00 msgr2=0x7f242011f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f242011ed00 0x7f242011f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 monclient: found mon.noname-b 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.880+0000 7f241ffff640 1 -- 172.21.15.181:0/780211763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2420120ed0 con 0x7f242011f630 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f241ffff640 1 -- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 msgr2=0x7f24201476a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f241ffff640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 0x7f24201476a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f242006abc0 tx=0x7f2418009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24180074f0 con 0x7f242011f630 2024-09-15T03:31:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: dump: 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 5 monclient: authenticate success, global_id 44527 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418005780 con 0x7f242011f630 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 10 monclient: _renew_subs 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2420121d90 con 0x7f242011f630 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418005b20 con 0x7f242011f630 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f24077fe640 10 monclient: dump: 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 10 monclient: _renew_subs 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2420136e00 con 0x7f242011f630 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.882+0000 7f241dffb640 4 set_mon_vals no callback set 2024-09-15T03:31:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.883+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f241800e730 con 0x7f242011f630 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.883+0000 7f2425f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.883+0000 7f2425f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.883+0000 7f2425f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.883+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242009a070 con 0x7f242011f630 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.884+0000 7f24077fe640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f240c038770 0x7f240c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.884+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f241804aaa0 con 0x7f242011f630 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.884+0000 7f241f7fe640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f240c038770 0x7f240c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.884+0000 7f241f7fe640 10 monclient: get_auth_request con 0x7f240c038770 auth_method 0 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.886+0000 7f241f7fe640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f240c038770 0x7f240c03ac20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2410001d10 tx=0x7f2410009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.888+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2418011aa0 con 0x7f242011f630 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.888+0000 7f24077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:53.888+0000 7f24077fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f2425f5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f2425f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f2425f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f242011fc70 con 0x7f242011f630 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v71) v1 ==== 126+0+0 (secure 0 0 0) 0x7f24180112e0 con 0x7f242011f630 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f24077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.048+0000 7f24077fe640 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.143+0000 7f2425f5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.143+0000 7f2425f5d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.143+0000 7f2425f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.143+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f2420122520 con 0x7f242011f630 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.144+0000 7f24077fe640 1 -- 172.21.15.181:0/780211763 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v71) v1 ==== 126+0+0 (secure 0 0 0) 0x7f24180114c0 con 0x7f242011f630 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.144+0000 7f24077fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.144+0000 7f24077fe640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-15T03:31:54.145 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'bar' already exists 2024-09-15T03:31:54.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f240c038770 msgr2=0x7f240c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f240c038770 0x7f240c03ac20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2410001d10 tx=0x7f2410009040 comp rx=0 tx=0).stop 2024-09-15T03:31:54.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 10 monclient: shutdown 2024-09-15T03:31:54.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:54.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 msgr2=0x7f24201476a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 0x7f24201476a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f242006abc0 tx=0x7f2418009040 comp rx=0 tx=0).stop 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 shutdown_connections 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f240c038770 0x7f240c03ac20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f242011f630 0x7f24201476a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f242011ed00 0x7f242011f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 --2- 172.21.15.181:0/780211763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2408004780 0x7f242011e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 >> 172.21.15.181:0/780211763 conn(0x7f2420070e90 msgr2=0x7f24200a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 shutdown_connections 2024-09-15T03:31:54.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.146+0000 7f2425f5d640 1 -- 172.21.15.181:0/780211763 wait complete. 2024-09-15T03:31:54.155 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush link host1 root=bar 2024-09-15T03:31:54.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:54.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.229+0000 7f29b9500640 10 monclient: get_monmap_and_config 2024-09-15T03:31:54.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.229+0000 7f29b9500640 10 monclient: build_initial_monmap 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.229+0000 7f29b9500640 10 monclient: monmap: 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:54.229945+0000 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:54.229945+0000 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:54.231 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:54.232 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:54.232 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:54.232 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.229+0000 7f29b9500640 1 Processor -- start 2024-09-15T03:31:54.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.229+0000 7f29b9500640 1 -- start start 2024-09-15T03:31:54.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.229+0000 7f29b9500640 10 monclient: init 2024-09-15T03:31:54.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:54.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f29b409ec50 0x7f29b4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient(hunting): picked mon.noname-c con 0x7f29b409ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29b41321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29b409f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b4067420 con 0x7f29b409f020 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b409fdc0 con 0x7f29b41321a0 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b40a0510 con 0x7f29b409ec50 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b9500640 10 monclient(hunting): authenticate will time out at 918.711315s 2024-09-15T03:31:54.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f29b409ec50 0x7f29b4131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f29b409ec50 0x7f29b4131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46804/0 (socket says 172.21.15.181:46804) 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 1 -- 172.21.15.181:0/3404925599 learned_addr learned my addr 172.21.15.181:0/3404925599 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 10 monclient(hunting): get_auth_request con 0x7f29b409ec50 auth_method 0 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.230+0000 7f29b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b27fc640 1 -- 172.21.15.181:0/3404925599 >> v1:172.21.15.181:6789/0 conn(0x7f29b41321a0 legacy=0x7f29b4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b37fe640 1 -- 172.21.15.181:0/3404925599 >> v1:172.21.15.153:6789/0 conn(0x7f29b409f020 legacy=0x7f29b4069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 10 monclient(hunting): handle_auth_done global_id 44530 payload 274 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 1 -- 172.21.15.181:0/3404925599 >> v1:172.21.15.181:6789/0 conn(0x7f29b41321a0 legacy=0x7f29b4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 1 -- 172.21.15.181:0/3404925599 >> v1:172.21.15.153:6789/0 conn(0x7f29b409f020 legacy=0x7f29b4069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:54.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 1 monclient: found mon.noname-c 2024-09-15T03:31:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.232+0000 7f29b2ffd640 1 -- 172.21.15.181:0/3404925599 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b40a0200 con 0x7f29b409ec50 2024-09-15T03:31:54.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b2ffd640 1 -- 172.21.15.181:0/3404925599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b409ec50 msgr2=0x7f29b4131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b2ffd640 1 --2- 172.21.15.181:0/3404925599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b409ec50 0x7f29b4131c60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009a10 tx=0x7f29a40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9c8bd7ecd013d69 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 1 -- 172.21.15.181:0/3404925599 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a402f4b0 con 0x7f29b409ec50 2024-09-15T03:31:54.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:54.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: dump: 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:54.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 1 -- 172.21.15.181:0/3404925599 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a402f610 con 0x7f29b409ec50 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 1 -- 172.21.15.181:0/3404925599 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a402f9b0 con 0x7f29b409ec50 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b9500640 5 monclient: authenticate success, global_id 44530 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b9500640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b9500640 4 set_mon_vals no callback set 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.235+0000 7f29b1ffb640 10 monclient: dump: 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.248 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 10 monclient: shutdown 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 -- 172.21.15.181:0/3404925599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b409ec50 msgr2=0x7f29b4131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 --2- 172.21.15.181:0/3404925599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b409ec50 0x7f29b4131c60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009a10 tx=0x7f29a40091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 -- 172.21.15.181:0/3404925599 shutdown_connections 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 --2- 172.21.15.181:0/3404925599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b409ec50 0x7f29b4131c60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 -- 172.21.15.181:0/3404925599 >> 172.21.15.181:0/3404925599 conn(0x7f29b40a5d40 msgr2=0x7f29b40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 -- 172.21.15.181:0/3404925599 shutdown_connections 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 1 -- 172.21.15.181:0/3404925599 wait complete. 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.236+0000 7f29b9500640 10 monclient: build_initial_monmap 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient: monmap: 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:54.249 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 Processor -- start 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 -- start start 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient: init 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b409ec50 0x7f29b409a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): picked mon.noname-c con 0x7f29b409ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 0x7f29b41309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): picked mon.noname-a con 0x7f29b409f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 0x7f29b411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): picked mon.noname-b con 0x7f29b41321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b409b350 con 0x7f29b409f020 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b409c000 con 0x7f29b41321a0 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b40a4190 con 0x7f29b409ec50 2024-09-15T03:31:54.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b9500640 10 monclient(hunting): authenticate will time out at 918.718814s 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 0x7f29b411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 0x7f29b411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46818/0 (socket says 172.21.15.181:46818) 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 1 -- 172.21.15.181:0/2609724436 learned_addr learned my addr 172.21.15.181:0/2609724436 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): get_auth_request con 0x7f29b41321a0 auth_method 0 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.237+0000 7f29b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 0x7f29b41309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b2ffd640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b409ec50 0x7f29b409a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 10 monclient(hunting): get_auth_request con 0x7f29b409f020 auth_method 0 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b2ffd640 10 monclient(hunting): get_auth_request con 0x7f29b409ec50 auth_method 0 2024-09-15T03:31:54.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 10 monclient(hunting): handle_auth_done global_id 44536 payload 824 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b409ec50 msgr2=0x7f29b409a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b409ec50 0x7f29b409a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 msgr2=0x7f29b41309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 msgr2=0x7f29b41309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 msgr2=0x7f29b41309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 0x7f29b41309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b27fc640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 0x7f29b41309f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 0x7f29b41309f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:54.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b37fe640 1 -- 172.21.15.181:0/2609724436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4009450 con 0x7f29b41321a0 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.238+0000 7f29b2ffd640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b409ec50 0x7f29b409a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b37fe640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 msgr2=0x7f29b411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b37fe640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 0x7f29b411b760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c870 tx=0x7f29ac0018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac001950 con 0x7f29b41321a0 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: dump: 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.255 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: _finish_auth 0 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac00a640 con 0x7f29b41321a0 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 5 monclient: authenticate success, global_id 44536 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 10 monclient: _renew_subs 2024-09-15T03:31:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b411db10 con 0x7f29b41321a0 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac00aa00 con 0x7f29b41321a0 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f299affd640 10 monclient: dump: 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.257 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 10 monclient: _renew_subs 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b17fa640 4 set_mon_vals no callback set 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.240+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b4136450 con 0x7f29b41321a0 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f29ac0070f0 con 0x7f29b41321a0 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f29b9500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f29b9500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f29b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b40008d0 con 0x7f29b41321a0 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f299affd640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a0038740 0x7f29a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f29ac04bd00 con 0x7f29b41321a0 2024-09-15T03:31:54.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f29b2ffd640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a0038740 0x7f29a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.241+0000 7f29b2ffd640 10 monclient: get_auth_request con 0x7f29a0038740 auth_method 0 2024-09-15T03:31:54.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.244+0000 7f29b2ffd640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a0038740 0x7f29a003abf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29b4132a50 tx=0x7f29a4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.246+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f29ac00ae30 con 0x7f29b41321a0 2024-09-15T03:31:54.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.246+0000 7f299affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.246+0000 7f299affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.290+0000 7f9d16aaf640 10 monclient: get_monmap_and_config 2024-09-15T03:31:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.290+0000 7f9d16aaf640 10 monclient: build_initial_monmap 2024-09-15T03:31:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.290+0000 7f9d16aaf640 10 monclient: monmap: 2024-09-15T03:31:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:54.291516+0000 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:54.291516+0000 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:54.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.291+0000 7f9d16aaf640 1 Processor -- start 2024-09-15T03:31:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.291+0000 7f9d16aaf640 1 -- start start 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.291+0000 7f9d16aaf640 10 monclient: init 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d1006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9d1006eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9d10068a20 0x7f9d1009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9d10068a20 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d1009caf0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1006a0e0 con 0x7f9d1009caf0 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d1006a7a0 con 0x7f9d10068a20 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1009cef0 con 0x7f9d1006eb80 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d16aaf640 10 monclient(hunting): authenticate will time out at 911.931132s 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d1006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d1006ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43940/0 (socket says 172.21.15.153:43940) 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 -- 172.21.15.153:0/3782117202 learned_addr learned my addr 172.21.15.153:0/3782117202 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): get_auth_request con 0x7f9d1006eb80 auth_method 0 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 1 --2- 172.21.15.153:0/3782117202 >> v2:172.21.15.181:3300/0 conn(0x7f9d10068a20 0x7f9d1009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): get_auth_request con 0x7f9d10068a20 auth_method 0 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient(hunting): handle_auth_done global_id 5093 payload 274 2024-09-15T03:31:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 -- 172.21.15.153:0/3782117202 >> v2:172.21.15.181:3300/0 conn(0x7f9d10068a20 msgr2=0x7f9d1009c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 --2- 172.21.15.153:0/3782117202 >> v2:172.21.15.181:3300/0 conn(0x7f9d10068a20 0x7f9d1009c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 -- 172.21.15.153:0/3782117202 >> v1:172.21.15.181:6789/0 conn(0x7f9d1009caf0 legacy=0x7f9d1009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient: _finish_hunting 0 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 monclient: found mon.noname-b 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.292+0000 7f9d1484b640 1 -- 172.21.15.153:0/3782117202 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10069e00 con 0x7f9d1006eb80 2024-09-15T03:31:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d1484b640 1 -- 172.21.15.153:0/3782117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 msgr2=0x7f9d1006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d1484b640 1 --2- 172.21.15.153:0/3782117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d1006ef50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009910 tx=0x7f9d040091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1f8d09c44af267c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 1 -- 172.21.15.153:0/3782117202 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0402aea0 con 0x7f9d1006eb80 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: dump: 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 1 -- 172.21.15.153:0/3782117202 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04032440 con 0x7f9d1006eb80 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d16aaf640 5 monclient: authenticate success, global_id 5093 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d16aaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d16aaf640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 1 -- 172.21.15.153:0/3782117202 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040327e0 con 0x7f9d1006eb80 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d16aaf640 4 set_mon_vals no callback set 2024-09-15T03:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.295+0000 7f9d0f7fe640 10 monclient: dump: 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 10 monclient: shutdown 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 -- 172.21.15.153:0/3782117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 msgr2=0x7f9d1006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/3782117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d1006ef50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009910 tx=0x7f9d040091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 -- 172.21.15.153:0/3782117202 shutdown_connections 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/3782117202 >> v2:172.21.15.181:3300/0 conn(0x7f9d10068a20 0x7f9d1009c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/3782117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d1006ef50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 -- 172.21.15.153:0/3782117202 >> 172.21.15.153:0/3782117202 conn(0x7f9d10067cd0 msgr2=0x7f9d100a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 -- 172.21.15.153:0/3782117202 shutdown_connections 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 -- 172.21.15.153:0/3782117202 wait complete. 2024-09-15T03:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 10 monclient: build_initial_monmap 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 10 monclient: monmap: 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.296+0000 7f9d16aaf640 1 Processor -- start 2024-09-15T03:31:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 -- start start 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient: init 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10068a20 0x7f9d10140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9d10068a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d10133ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9d1006eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 0x7f9d1011d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9d1009caf0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d100a2740 con 0x7f9d1009caf0 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d101210e0 con 0x7f9d10068a20 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10141e10 con 0x7f9d1006eb80 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d16aaf640 10 monclient(hunting): authenticate will time out at 911.936709s 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d10133ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d10133ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43942/0 (socket says 172.21.15.153:43942) 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 1 -- 172.21.15.153:0/1947558107 learned_addr learned my addr 172.21.15.153:0/1947558107 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): get_auth_request con 0x7f9d1006eb80 auth_method 0 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 0x7f9d1011d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10068a20 0x7f9d10140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 10 monclient(hunting): get_auth_request con 0x7f9d1009caf0 auth_method 0 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): get_auth_request con 0x7f9d10068a20 auth_method 0 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 msgr2=0x7f9d1011d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 msgr2=0x7f9d1011d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1484b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.297+0000 7f9d1504c640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 0x7f9d1011d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d1504c640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 0x7f9d1011d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 10 monclient(hunting): handle_auth_done global_id 5096 payload 824 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10068a20 msgr2=0x7f9d10140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10068a20 0x7f9d10140fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 msgr2=0x7f9d1011d940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 0x7f9d1011d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 1 monclient: found mon.noname-c 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.298+0000 7f9d0ffff640 1 -- 172.21.15.153:0/1947558107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009450 con 0x7f9d1006eb80 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0ffff640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 msgr2=0x7f9d10133ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0ffff640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d10133ce0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e900 tx=0x7f9cfc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc009480 con 0x7f9d1006eb80 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: dump: 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc00caf0 con 0x7f9d1006eb80 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 5 monclient: authenticate success, global_id 5096 2024-09-15T03:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00ce70 con 0x7f9d1006eb80 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0cff9640 10 monclient: dump: 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 10 monclient: _renew_subs 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d0effd640 4 set_mon_vals no callback set 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d10120e00 con 0x7f9d1006eb80 2024-09-15T03:31:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 10 monclient: _renew_subs 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.300+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d10141500 con 0x7f9d1006eb80 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9cfc013680 con 0x7f9d1006eb80 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d0cff9640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9cf4038790 0x7f9cf403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f9cfc04c1d0 con 0x7f9d1006eb80 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d16aaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d16aaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d16aaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1004bf60 con 0x7f9d1006eb80 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d1484b640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9cf4038790 0x7f9cf403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.301+0000 7f9d1484b640 10 monclient: get_auth_request con 0x7f9cf4038790 auth_method 0 2024-09-15T03:31:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.304+0000 7f9d1484b640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9cf4038790 0x7f9cf403ac40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009910 tx=0x7f9d04032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.306+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9cfc023ad0 con 0x7f9d1006eb80 2024-09-15T03:31:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.306+0000 7f9d0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.306+0000 7f9d0cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:54.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.405+0000 7f29b9500640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:54.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.405+0000 7f29b9500640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:54.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.405+0000 7f29b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.406+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f29b40a13e0 con 0x7f29b41321a0 2024-09-15T03:31:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.520+0000 7f9d16aaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.520+0000 7f9d16aaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.520+0000 7f9d16aaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.520+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d100638d0 con 0x7f9d1006eb80 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.521+0000 7f9d0cff9640 1 -- 172.21.15.153:0/1947558107 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9cfc0232f0 con 0x7f9d1006eb80 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.521+0000 7f9d0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.521+0000 7f9d0cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:54.522 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9cf4038790 msgr2=0x7f9cf403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9cf4038790 0x7f9cf403ac40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009910 tx=0x7f9d04032000 comp rx=0 tx=0).stop 2024-09-15T03:31:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 10 monclient: shutdown 2024-09-15T03:31:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 msgr2=0x7f9d10133ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d10133ce0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e900 tx=0x7f9cfc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 shutdown_connections 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9cf4038790 0x7f9cf403ac40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1009caf0 0x7f9d1011d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d1006eb80 0x7f9d10133ce0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 --2- 172.21.15.153:0/1947558107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10068a20 0x7f9d10140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 >> 172.21.15.153:0/1947558107 conn(0x7f9d10067cd0 msgr2=0x7f9d100a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 shutdown_connections 2024-09-15T03:31:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:54.523+0000 7f9d16aaf640 1 -- 172.21.15.153:0/1947558107 wait complete. 2024-09-15T03:31:54.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.691+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 168+0+0 (secure 0 0 0) 0x7f29ac010770 con 0x7f29b41321a0 2024-09-15T03:31:54.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.691+0000 7f299affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:54.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.691+0000 7f299affd640 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.788+0000 7f29b9500640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.788+0000 7f29b9500640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.788+0000 7f29b9500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.788+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f29b4072640 con 0x7f29b41321a0 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.789+0000 7f299affd640 1 -- 172.21.15.181:0/2609724436 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 177+0+0 (secure 0 0 0) 0x7f29ac010950 con 0x7f29b41321a0 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.789+0000 7f299affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.789+0000 7f299affd640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T03:31:54.790 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a0038740 msgr2=0x7f29a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a0038740 0x7f29a003abf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29b4132a50 tx=0x7f29a4032000 comp rx=0 tx=0).stop 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 10 monclient: shutdown 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 msgr2=0x7f29b411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 0x7f29b411b760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c870 tx=0x7f29ac0018c0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 shutdown_connections 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a0038740 0x7f29a003abf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b41321a0 0x7f29b411b760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b409f020 0x7f29b41309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 --2- 172.21.15.181:0/2609724436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b409ec50 0x7f29b409a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 >> 172.21.15.181:0/2609724436 conn(0x7f29b40a5d40 msgr2=0x7f29b40a48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 shutdown_connections 2024-09-15T03:31:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.791+0000 7f29b9500640 1 -- 172.21.15.181:0/2609724436 wait complete. 2024-09-15T03:31:54.800 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush link host1 root=bar 2024-09-15T03:31:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.873+0000 7ff24e79d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.873+0000 7ff24e79d640 10 monclient: build_initial_monmap 2024-09-15T03:31:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.873+0000 7ff24e79d640 10 monclient: monmap: 2024-09-15T03:31:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:54.874696+0000 2024-09-15T03:31:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:54.874696+0000 2024-09-15T03:31:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:54.877 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.874+0000 7ff24e79d640 1 Processor -- start 2024-09-15T03:31:54.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.874+0000 7ff24e79d640 1 -- start start 2024-09-15T03:31:54.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.874+0000 7ff24e79d640 10 monclient: init 2024-09-15T03:31:54.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:54.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:31:54.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff24809cad0 0x7ff24809c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): picked mon.noname-c con 0x7ff24809cad0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:54.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809cea0 0x7ff24809dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): picked mon.noname-b con 0x7ff24809cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:54.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff24809d270 0x7ff24809e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): picked mon.noname-a con 0x7ff24809d270 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:54.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24809b710 con 0x7ff24809d270 2024-09-15T03:31:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24809c6e0 con 0x7ff24809cad0 2024-09-15T03:31:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24809a650 con 0x7ff24809cea0 2024-09-15T03:31:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:54.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24e79d640 10 monclient(hunting): authenticate will time out at 919.356240s 2024-09-15T03:31:54.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff24809cad0 0x7ff24809c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff24809cad0 0x7ff24809c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46824/0 (socket says 172.21.15.181:46824) 2024-09-15T03:31:54.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 -- 172.21.15.181:0/3775118795 learned_addr learned my addr 172.21.15.181:0/3775118795 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:54.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): get_auth_request con 0x7ff24809cad0 auth_method 0 2024-09-15T03:31:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient(hunting): handle_auth_done global_id 44545 payload 274 2024-09-15T03:31:54.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 -- 172.21.15.181:0/3775118795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809cea0 msgr2=0x7ff24809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:54.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff2477fe640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809cea0 0x7ff24809dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24cd3a640 1 --2- 172.21.15.181:0/3775118795 >> v2:172.21.15.153:3300/0 conn(0x7ff24809d270 0x7ff24809e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809cea0 0x7ff24809dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 -- 172.21.15.181:0/3775118795 >> v2:172.21.15.153:3300/0 conn(0x7ff24809d270 msgr2=0x7ff24809e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 --2- 172.21.15.181:0/3775118795 >> v2:172.21.15.153:3300/0 conn(0x7ff24809d270 0x7ff24809e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 monclient: found mon.noname-c 2024-09-15T03:31:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff247fff640 1 -- 172.21.15.181:0/3775118795 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24809f660 con 0x7ff24809cad0 2024-09-15T03:31:54.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff2477fe640 10 monclient: get_auth_request con 0x7ff24809cea0 auth_method 0 2024-09-15T03:31:54.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff2477fe640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809cea0 0x7ff24809dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:54.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24cd3a640 10 monclient: get_auth_request con 0x7ff24809d270 auth_method 0 2024-09-15T03:31:54.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.875+0000 7ff24cd3a640 1 --2- 172.21.15.181:0/3775118795 >> v2:172.21.15.153:3300/0 conn(0x7ff24809d270 0x7ff24809e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:54.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff247fff640 1 -- 172.21.15.181:0/3775118795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cad0 msgr2=0x7ff24809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff247fff640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cad0 0x7ff24809c1a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff24809eb00 tx=0x7ff2380091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b40e44a0d940ff6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 1 -- 172.21.15.181:0/3775118795 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238032440 con 0x7ff24809cad0 2024-09-15T03:31:54.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:54.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: dump: 2024-09-15T03:31:54.904 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.905 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.905 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.905 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.905 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.905 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.906 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.906 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.906 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.906 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:54.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:54.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff24e79d640 5 monclient: authenticate success, global_id 44545 2024-09-15T03:31:54.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 1 -- 172.21.15.181:0/3775118795 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2380325a0 con 0x7ff24809cad0 2024-09-15T03:31:54.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff24e79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:54.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:54.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 1 -- 172.21.15.181:0/3775118795 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238032940 con 0x7ff24809cad0 2024-09-15T03:31:54.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff24e79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:54.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff24e79d640 4 set_mon_vals no callback set 2024-09-15T03:31:54.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:54.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.878+0000 7ff246ffd640 10 monclient: dump: 2024-09-15T03:31:54.908 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.909 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.909 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.909 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.909 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.909 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.910 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.910 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.910 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.910 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 10 monclient: shutdown 2024-09-15T03:31:54.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:54.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 -- 172.21.15.181:0/3775118795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cad0 msgr2=0x7ff24809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cad0 0x7ff24809c1a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff24809eb00 tx=0x7ff2380091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 -- 172.21.15.181:0/3775118795 shutdown_connections 2024-09-15T03:31:54.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 --2- 172.21.15.181:0/3775118795 >> v2:172.21.15.153:3300/0 conn(0x7ff24809d270 0x7ff24809e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809cea0 0x7ff24809dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 --2- 172.21.15.181:0/3775118795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cad0 0x7ff24809c1a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 -- 172.21.15.181:0/3775118795 >> 172.21.15.181:0/3775118795 conn(0x7ff248070e90 msgr2=0x7ff2480a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:54.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 -- 172.21.15.181:0/3775118795 shutdown_connections 2024-09-15T03:31:54.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 1 -- 172.21.15.181:0/3775118795 wait complete. 2024-09-15T03:31:54.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 10 monclient: build_initial_monmap 2024-09-15T03:31:54.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.879+0000 7ff24e79d640 10 monclient: monmap: 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.913 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:54.914 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:54.914 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:54.914 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 Processor -- start 2024-09-15T03:31:54.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 -- start start 2024-09-15T03:31:54.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient: init 2024-09-15T03:31:54.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:54.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:31:54.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 0x7ff2481329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): picked mon.noname-a con 0x7ff24809cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:54.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 0x7ff24811c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): picked mon.noname-b con 0x7ff24809cea0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809d270 0x7ff24811c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): picked mon.noname-c con 0x7ff24809d270 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24806dbe0 con 0x7ff24809cad0 2024-09-15T03:31:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24811fdf0 con 0x7ff24809cea0 2024-09-15T03:31:54.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:54.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24806eba0 con 0x7ff24809d270 2024-09-15T03:31:54.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:54.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24e79d640 10 monclient(hunting): authenticate will time out at 919.361740s 2024-09-15T03:31:54.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 0x7ff24811c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 0x7ff24811c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46834/0 (socket says 172.21.15.181:46834) 2024-09-15T03:31:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 1 -- 172.21.15.181:0/1952685303 learned_addr learned my addr 172.21.15.181:0/1952685303 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): get_auth_request con 0x7ff24809cea0 auth_method 0 2024-09-15T03:31:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff247fff640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 0x7ff2481329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff2477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff247fff640 10 monclient(hunting): get_auth_request con 0x7ff24809cad0 auth_method 0 2024-09-15T03:31:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24cd3a640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809d270 0x7ff24811c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24cd3a640 10 monclient(hunting): get_auth_request con 0x7ff24809d270 auth_method 0 2024-09-15T03:31:54.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24cd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24cd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.880+0000 7ff24cd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff247fff640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 msgr2=0x7ff2481329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff247fff640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 msgr2=0x7ff2481329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff247fff640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 0x7ff2481329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff247fff640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 0x7ff2481329f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff24cd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff24cd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff24cd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 10 monclient(hunting): handle_auth_done global_id 44548 payload 824 2024-09-15T03:31:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809d270 msgr2=0x7ff24811c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809d270 0x7ff24811c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 msgr2=0x7ff2481329f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 0x7ff2481329f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.881+0000 7ff2477fe640 1 -- 172.21.15.181:0/1952685303 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238009450 con 0x7ff24809cea0 2024-09-15T03:31:54.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff2477fe640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 msgr2=0x7ff24811c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:54.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff2477fe640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 0x7ff24811c4a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00c880 tx=0x7ff23c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c007830 con 0x7ff24809cea0 2024-09-15T03:31:54.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: dump: 2024-09-15T03:31:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.926 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.926 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.926 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.926 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.926 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.926 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: _finish_auth 0 2024-09-15T03:31:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c007990 con 0x7ff24809cea0 2024-09-15T03:31:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 5 monclient: authenticate success, global_id 44548 2024-09-15T03:31:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c007d50 con 0x7ff24809cea0 2024-09-15T03:31:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 10 monclient: _renew_subs 2024-09-15T03:31:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24813f9e0 con 0x7ff24809cea0 2024-09-15T03:31:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff22ffff640 10 monclient: dump: 2024-09-15T03:31:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:54.930 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff2467fc640 4 set_mon_vals no callback set 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 10 monclient: _renew_subs 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.883+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24811ebb0 con 0x7ff24809cea0 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff23c011600 con 0x7ff24809cea0 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff22ffff640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff234038740 0x7ff23403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6197+0+0 (secure 0 0 0) 0x7ff23c04c260 con 0x7ff24809cea0 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff247fff640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff234038740 0x7ff23403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff24e79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff24e79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff24e79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff248120030 con 0x7ff24809cea0 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.884+0000 7ff247fff640 10 monclient: get_auth_request con 0x7ff234038740 auth_method 0 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.886+0000 7ff247fff640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff234038740 0x7ff23403abf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff24806edb0 tx=0x7ff238032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.888+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff23c021ae0 con 0x7ff24809cea0 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.888+0000 7ff22ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:54.888+0000 7ff22ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff24e79d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff24e79d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff24e79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7ff2480008d0 con 0x7ff24809cea0 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 177+0+0 (secure 0 0 0) 0x7ff23c021300 con 0x7ff24809cea0 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff22ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:55.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.046+0000 7ff22ffff640 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.142+0000 7ff24e79d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.142+0000 7ff24e79d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.142+0000 7ff24e79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.142+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7ff24809b430 con 0x7ff24809cea0 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.143+0000 7ff22ffff640 1 -- 172.21.15.181:0/1952685303 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 177+0+0 (secure 0 0 0) 0x7ff23c0214e0 con 0x7ff24809cea0 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.143+0000 7ff22ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.143+0000 7ff22ffff640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T03:31:55.144 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff234038740 msgr2=0x7ff23403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff234038740 0x7ff23403abf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff24806edb0 tx=0x7ff238032000 comp rx=0 tx=0).stop 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 10 monclient: shutdown 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 msgr2=0x7ff24811c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 0x7ff24811c4a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00c880 tx=0x7ff23c008040 comp rx=0 tx=0).stop 2024-09-15T03:31:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 shutdown_connections 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff234038740 0x7ff23403abf0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff24809d270 0x7ff24811c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24809cea0 0x7ff24811c4a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 --2- 172.21.15.181:0/1952685303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff24809cad0 0x7ff2481329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.144+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 >> 172.21.15.181:0/1952685303 conn(0x7ff248070e90 msgr2=0x7ff248098a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.145+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 shutdown_connections 2024-09-15T03:31:55.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.145+0000 7ff24e79d640 1 -- 172.21.15.181:0/1952685303 wait complete. 2024-09-15T03:31:55.154 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush link host2 root=bar 2024-09-15T03:31:55.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.226+0000 7f7ec6b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:55.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.226+0000 7f7ec6b9d640 10 monclient: build_initial_monmap 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.226+0000 7f7ec6b9d640 10 monclient: monmap: 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:55.227542+0000 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:55.227542+0000 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:55.228 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:55.229 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:55.229 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 Processor -- start 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 -- start start 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient: init 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 0x7f7ec009d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec009d140 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7ec009da50 0x7f7ec0094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec009da50 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:55.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec0094a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec00945c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec006f7a0 con 0x7f7ec009d140 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec0098280 con 0x7f7ec009da50 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec009ec20 con 0x7f7ec00945c0 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.227+0000 7f7ec6b9d640 10 monclient(hunting): authenticate will time out at 919.708959s 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7ec009da50 0x7f7ec0094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7ec009da50 0x7f7ec0094080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46838/0 (socket says 172.21.15.181:46838) 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 -- 172.21.15.181:0/2097904758 learned_addr learned my addr 172.21.15.181:0/2097904758 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:55.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): get_auth_request con 0x7f7ec009da50 auth_method 0 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 1 --2- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 0x7f7ec009d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 1 --2- 172.21.15.181:0/2097904758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec0094a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 10 monclient(hunting): get_auth_request con 0x7f7ec009d140 auth_method 0 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): get_auth_request con 0x7f7ec00945c0 auth_method 0 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 1 -- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 msgr2=0x7f7ec009d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 1 -- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 msgr2=0x7f7ec009d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 1 --2- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 0x7f7ec009d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec513a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ec4939640 1 --2- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 0x7f7ec009d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient(hunting): handle_auth_done global_id 44551 payload 274 2024-09-15T03:31:55.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 -- 172.21.15.181:0/2097904758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 msgr2=0x7f7ec0094a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 --2- 172.21.15.181:0/2097904758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec0094a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 -- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 msgr2=0x7f7ec009d510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 --2- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 0x7f7ec009d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 monclient: found mon.noname-c 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.228+0000 7f7ebffff640 1 -- 172.21.15.181:0/2097904758 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec0064cb0 con 0x7f7ec009da50 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebffff640 1 -- 172.21.15.181:0/2097904758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 msgr2=0x7f7ec0094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebffff640 1 --2- 172.21.15.181:0/2097904758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec0094080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00953d0 tx=0x7f7eac0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=208f770ad3636d3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 1 -- 172.21.15.181:0/2097904758 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac032440 con 0x7f7ec009da50 2024-09-15T03:31:55.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: dump: 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:55.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 1 -- 172.21.15.181:0/2097904758 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac0325a0 con 0x7f7ec009da50 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 5 monclient: authenticate success, global_id 44551 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 1 -- 172.21.15.181:0/2097904758 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac032940 con 0x7f7ec009da50 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 4 set_mon_vals no callback set 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ebf7fe640 10 monclient: dump: 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.244 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 10 monclient: shutdown 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/2097904758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 msgr2=0x7f7ec0094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/2097904758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec0094080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00953d0 tx=0x7f7eac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/2097904758 shutdown_connections 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/2097904758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec0094a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/2097904758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec0094080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/2097904758 >> v2:172.21.15.153:3300/0 conn(0x7f7ec009d140 0x7f7ec009d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/2097904758 >> 172.21.15.181:0/2097904758 conn(0x7f7ec00a2190 msgr2=0x7f7ec009e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/2097904758 shutdown_connections 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.231+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/2097904758 wait complete. 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.232+0000 7f7ec6b9d640 10 monclient: build_initial_monmap 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.232+0000 7f7ec6b9d640 10 monclient: monmap: 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:55.246 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.232+0000 7f7ec6b9d640 1 Processor -- start 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.232+0000 7f7ec6b9d640 1 -- start start 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.232+0000 7f7ec6b9d640 10 monclient: init 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec00939e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec00945c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 0x7f7ec008c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec009d140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec008ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec009da50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0090740 con 0x7f7ec009d140 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0095090 con 0x7f7ec009da50 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00a1180 con 0x7f7ec00945c0 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec6b9d640 10 monclient(hunting): authenticate will time out at 919.714259s 2024-09-15T03:31:55.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec008ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec008ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46852/0 (socket says 172.21.15.181:46852) 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 -- 172.21.15.181:0/3034535712 learned_addr learned my addr 172.21.15.181:0/3034535712 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): get_auth_request con 0x7f7ec009da50 auth_method 0 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec00939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 0x7f7ec008c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): get_auth_request con 0x7f7ec00945c0 auth_method 0 2024-09-15T03:31:55.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 10 monclient(hunting): get_auth_request con 0x7f7ec009d140 auth_method 0 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 msgr2=0x7f7ec008c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 msgr2=0x7f7ec008c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec4939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 0x7f7ec008c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient(hunting): handle_auth_done global_id 44554 payload 824 2024-09-15T03:31:55.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ebffff640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 0x7f7ec008c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 msgr2=0x7f7ec00939e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec00939e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 msgr2=0x7f7ec008c520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 0x7f7ec008c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient: _finish_hunting 0 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 monclient: found mon.noname-b 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.233+0000 7f7ec513a640 1 -- 172.21.15.181:0/3034535712 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac009450 con 0x7f7ec009da50 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ec513a640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 msgr2=0x7f7ec008ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ec513a640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec008ca60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009df0 tx=0x7f7eb800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8014480 con 0x7f7ec009da50 2024-09-15T03:31:55.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 10 monclient: dump: 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb800f040 con 0x7f7ec009da50 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ec6b9d640 5 monclient: authenticate success, global_id 44554 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ec6b9d640 10 monclient: _renew_subs 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8014830 con 0x7f7ec009da50 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ec6b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec008e460 con 0x7f7ec009da50 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ebcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ebcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ebcff9640 10 monclient: dump: 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.253 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.235+0000 7f7ebeffd640 4 set_mon_vals no callback set 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 10 monclient: _renew_subs 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec0092f60 con 0x7f7ec009da50 2024-09-15T03:31:55.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7eb8014be0 con 0x7f7ec009da50 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec00008d0 con 0x7f7ec009da50 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.236+0000 7f7ebcff9640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ea4038770 0x7f7ea403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.237+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6197+0+0 (secure 0 0 0) 0x7f7eb804cc00 con 0x7f7ec009da50 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.237+0000 7f7ec4939640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ea4038770 0x7f7ea403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.237+0000 7f7ec4939640 10 monclient: get_auth_request con 0x7f7ea4038770 auth_method 0 2024-09-15T03:31:55.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.239+0000 7f7ec4939640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ea4038770 0x7f7ea403ac20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4006fd0 tx=0x7f7eb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:55.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.241+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7eb801c3b0 con 0x7f7ec009da50 2024-09-15T03:31:55.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.241+0000 7f7ebcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:55.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.241+0000 7f7ebcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:55.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.401+0000 7f7ec6b9d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:55.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.401+0000 7f7ec6b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:55.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.401+0000 7f7ec6b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.401+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f7ec009e7f0 con 0x7f7ec009da50 2024-09-15T03:31:55.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.696+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7eb8018030 con 0x7f7ec009da50 2024-09-15T03:31:55.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.696+0000 7f7ebcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:55.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.696+0000 7f7ebcff9640 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T03:31:55.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ec6b9d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:55.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ec6b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:55.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ec6b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f7ec0064a20 con 0x7f7ec009da50 2024-09-15T03:31:55.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ebcff9640 1 -- 172.21.15.181:0/3034535712 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (secure 0 0 0) 0x7f7eb8018020 con 0x7f7ec009da50 2024-09-15T03:31:55.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ebcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:55.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.793+0000 7f7ebcff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T03:31:55.795 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T03:31:55.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ea4038770 msgr2=0x7f7ea403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ea4038770 0x7f7ea403ac20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4006fd0 tx=0x7f7eb4008040 comp rx=0 tx=0).stop 2024-09-15T03:31:55.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 10 monclient: shutdown 2024-09-15T03:31:55.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 msgr2=0x7f7ec008ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec008ca60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009df0 tx=0x7f7eb800b040 comp rx=0 tx=0).stop 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 shutdown_connections 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ea4038770 0x7f7ea403ac20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ec009da50 0x7f7ec008ca60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ec009d140 0x7f7ec008c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 --2- 172.21.15.181:0/3034535712 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ec00945c0 0x7f7ec00939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 >> 172.21.15.181:0/3034535712 conn(0x7f7ec00a2190 msgr2=0x7f7ec0095ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 shutdown_connections 2024-09-15T03:31:55.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.795+0000 7f7ec6b9d640 1 -- 172.21.15.181:0/3034535712 wait complete. 2024-09-15T03:31:55.805 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush link host2 root=bar 2024-09-15T03:31:55.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.878+0000 7f7f324c9640 10 monclient: get_monmap_and_config 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.878+0000 7f7f324c9640 10 monclient: build_initial_monmap 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.878+0000 7f7f324c9640 10 monclient: monmap: 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:55.879620+0000 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:55.879620+0000 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:55.880 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:55.881 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.879+0000 7f7f324c9640 1 Processor -- start 2024-09-15T03:31:55.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.879+0000 7f7f324c9640 1 -- start start 2024-09-15T03:31:55.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.879+0000 7f7f324c9640 10 monclient: init 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f2c09ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 0x7f7f2c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): picked mon.noname-a con 0x7f7f2c09f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7f2c136d60 0x7f7f2c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): picked mon.noname-c con 0x7f7f2c136d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c067420 con 0x7f7f2c09ec50 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c09fdc0 con 0x7f7f2c09f020 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0a0510 con 0x7f7f2c136d60 2024-09-15T03:31:55.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f324c9640 10 monclient(hunting): authenticate will time out at 920.361299s 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2bfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f7f2c09ec50 0x7f7f2c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50804/0 (socket says 172.21.15.181:50804) 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2bfff640 1 -- 172.21.15.181:0/1002484565 learned_addr learned my addr 172.21.15.181:0/1002484565 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f30a66640 1 --2- 172.21.15.181:0/1002484565 >> v2:172.21.15.181:3300/0 conn(0x7f7f2c136d60 0x7f7f2c139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f30a66640 10 monclient(hunting): get_auth_request con 0x7f7f2c136d60 auth_method 0 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f30a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f30a66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f30a66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 1 --2- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 0x7f7f2c069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 10 monclient(hunting): get_auth_request con 0x7f7f2c09f020 auth_method 0 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 msgr2=0x7f7f2c069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:55.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 msgr2=0x7f7f2c069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:55.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 1 --2- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 0x7f7f2c069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:55.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.880+0000 7f7f2b7fe640 1 --2- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 0x7f7f2c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f2bfff640 1 -- 172.21.15.181:0/1002484565 >> v1:172.21.15.181:6789/0 conn(0x7f7f2c09ec50 legacy=0x7f7f2c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 10 monclient(hunting): handle_auth_done global_id 44557 payload 274 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 1 -- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 msgr2=0x7f7f2c069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 1 --2- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 0x7f7f2c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 1 -- 172.21.15.181:0/1002484565 >> v1:172.21.15.181:6789/0 conn(0x7f7f2c09ec50 legacy=0x7f7f2c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 10 monclient: _finish_hunting 0 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 1 monclient: found mon.noname-c 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:55.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.882+0000 7f7f30a66640 1 -- 172.21.15.181:0/1002484565 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c0a0200 con 0x7f7f2c136d60 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f30a66640 1 -- 172.21.15.181:0/1002484565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c136d60 msgr2=0x7f7f2c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f30a66640 1 --2- 172.21.15.181:0/1002484565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c136d60 0x7f7f2c139140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c132a50 tx=0x7f7f24009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17189928437c588d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 1 -- 172.21.15.181:0/1002484565 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24007760 con 0x7f7f2c136d60 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: dump: 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.892 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: _finish_auth 0 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f324c9640 5 monclient: authenticate success, global_id 44557 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 1 -- 172.21.15.181:0/1002484565 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2402d050 con 0x7f7f2c136d60 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f324c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f324c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f324c9640 4 set_mon_vals no callback set 2024-09-15T03:31:55.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 1 -- 172.21.15.181:0/1002484565 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24007af0 con 0x7f7f2c136d60 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.885+0000 7f7f2affd640 10 monclient: dump: 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 10 monclient: shutdown 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 -- 172.21.15.181:0/1002484565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c136d60 msgr2=0x7f7f2c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 --2- 172.21.15.181:0/1002484565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c136d60 0x7f7f2c139140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c132a50 tx=0x7f7f24009780 comp rx=0 tx=0).stop 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 -- 172.21.15.181:0/1002484565 shutdown_connections 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 --2- 172.21.15.181:0/1002484565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c136d60 0x7f7f2c139140 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 --2- 172.21.15.181:0/1002484565 >> v2:172.21.15.153:3300/0 conn(0x7f7f2c09f020 0x7f7f2c069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 -- 172.21.15.181:0/1002484565 >> 172.21.15.181:0/1002484565 conn(0x7f7f2c0a5d40 msgr2=0x7f7f2c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 -- 172.21.15.181:0/1002484565 shutdown_connections 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 1 -- 172.21.15.181:0/1002484565 wait complete. 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 10 monclient: build_initial_monmap 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.886+0000 7f7f324c9640 10 monclient: monmap: 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:55.895 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 1 Processor -- start 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 1 -- start start 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient: init 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:55.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f2c09ec50 0x7f7f2c0997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient(hunting): picked mon.noname-c con 0x7f7f2c09ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 0x7f7f2c099cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient(hunting): picked mon.noname-b con 0x7f7f2c09f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 0x7f7f2c09a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient(hunting): picked mon.noname-a con 0x7f7f2c136d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.887+0000 7f7f324c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c09b130 con 0x7f7f2c136d60 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f324c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f324c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c09c000 con 0x7f7f2c09f020 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f324c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f324c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0a3ab0 con 0x7f7f2c09ec50 2024-09-15T03:31:55.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f324c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f324c9640 10 monclient(hunting): authenticate will time out at 920.369019s 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 0x7f7f2c099cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 0x7f7f2c099cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46866/0 (socket says 172.21.15.181:46866) 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/2829129547 learned_addr learned my addr 172.21.15.181:0/2829129547 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): get_auth_request con 0x7f7f2c09f020 auth_method 0 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f2c09ec50 0x7f7f2c0997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 0x7f7f2c09a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): get_auth_request con 0x7f7f2c09ec50 auth_method 0 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 10 monclient(hunting): get_auth_request con 0x7f7f2c136d60 auth_method 0 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 msgr2=0x7f7f2c09a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:55.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 msgr2=0x7f7f2c09a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 0x7f7f2c09a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f30a66640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 0x7f7f2c09a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient(hunting): handle_auth_done global_id 44563 payload 824 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f2c09ec50 msgr2=0x7f7f2c0997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f2c09ec50 0x7f7f2c0997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 msgr2=0x7f7f2c09a230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 0x7f7f2c09a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.888+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/2829129547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24009a20 con 0x7f7f2c09f020 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f2b7fe640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 msgr2=0x7f7f2c099cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f2b7fe640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 0x7f7f2c099cf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000d8c0 tx=0x7f7f20008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2000de50 con 0x7f7f2c09f020 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: dump: 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.903 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 5 monclient: authenticate success, global_id 44563 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2000bae0 con 0x7f7f2c09f020 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 10 monclient: _renew_subs 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c11d990 con 0x7f7f2c09f020 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2000bea0 con 0x7f7f2c09f020 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f13fff640 10 monclient: dump: 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:55.905 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 10 monclient: _renew_subs 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f2a7fc640 4 set_mon_vals no callback set 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.890+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c11c3d0 con 0x7f7f2c09f020 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7f20014640 con 0x7f7f2c09f020 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f324c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f324c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f324c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c11c880 con 0x7f7f2c09f020 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f13fff640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f18038740 0x7f7f1803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:55.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.891+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f7f2004c680 con 0x7f7f2c09f020 2024-09-15T03:31:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.892+0000 7f7f2bfff640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f18038740 0x7f7f1803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.892+0000 7f7f2bfff640 10 monclient: get_auth_request con 0x7f7f18038740 auth_method 0 2024-09-15T03:31:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.894+0000 7f7f2bfff640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f18038740 0x7f7f1803abf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c007fd0 tx=0x7f7f1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.896+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7f200148f0 con 0x7f7f2c09f020 2024-09-15T03:31:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.896+0000 7f7f13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:55.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:55.896+0000 7f7f13fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:56.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.055+0000 7f7f324c9640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:56.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.055+0000 7f7f324c9640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:56.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.055+0000 7f7f324c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.055+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f7f2c11cb70 con 0x7f7f2c09f020 2024-09-15T03:31:56.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.055+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (secure 0 0 0) 0x7f7f2001dbb0 con 0x7f7f2c09f020 2024-09-15T03:31:56.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.056+0000 7f7f13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:56.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.056+0000 7f7f13fff640 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T03:31:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f324c9640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f324c9640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f324c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f7f2c0008d0 con 0x7f7f2c09f020 2024-09-15T03:31:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f13fff640 1 -- 172.21.15.181:0/2829129547 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (secure 0 0 0) 0x7f7f2001dd90 con 0x7f7f2c09f020 2024-09-15T03:31:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f13fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-15T03:31:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.153+0000 7f7f13fff640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T03:31:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f18038740 msgr2=0x7f7f1803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f18038740 0x7f7f1803abf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c007fd0 tx=0x7f7f1c009040 comp rx=0 tx=0).stop 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 10 monclient: shutdown 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 msgr2=0x7f7f2c099cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 0x7f7f2c099cf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000d8c0 tx=0x7f7f20008040 comp rx=0 tx=0).stop 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 shutdown_connections 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f18038740 0x7f7f1803abf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f2c136d60 0x7f7f2c09a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f2c09f020 0x7f7f2c099cf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 --2- 172.21.15.181:0/2829129547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f2c09ec50 0x7f7f2c0997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 >> 172.21.15.181:0/2829129547 conn(0x7f7f2c0a5d40 msgr2=0x7f7f2c09cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:56.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 shutdown_connections 2024-09-15T03:31:56.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.155+0000 7f7f324c9640 1 -- 172.21.15.181:0/2829129547 wait complete. 2024-09-15T03:31:56.166 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:31:56.166 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c osd.7 2024-09-15T03:31:56.166 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 2 2024-09-15T03:31:56.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.235+0000 7f357275d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:56.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.235+0000 7f357275d640 10 monclient: build_initial_monmap 2024-09-15T03:31:56.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.235+0000 7f357275d640 10 monclient: monmap: 2024-09-15T03:31:56.236 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:56.236179+0000 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:56.236179+0000 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:56.237 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 Processor -- start 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 -- start start 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient: init 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 0x7f356c131c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): picked mon.noname-b con 0x7f356c1318b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:56.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f356c1321c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 0x7f356c138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): picked mon.noname-a con 0x7f356c09f1e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0a1510 con 0x7f356c1321c0 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c132590 con 0x7f356c09f1e0 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09ff00 con 0x7f356c1318b0 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f357275d640 10 monclient(hunting): authenticate will time out at 920.717613s 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356b7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f356c1321c0 0x7f356c09ead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50810/0 (socket says 172.21.15.181:50810) 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356b7fe640 1 -- 172.21.15.181:0/2518641743 learned_addr learned my addr 172.21.15.181:0/2518641743 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:56.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f3570cfa640 1 --2- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 0x7f356c138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356bfff640 1 --2- 172.21.15.181:0/2518641743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 0x7f356c131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f3570cfa640 10 monclient(hunting): get_auth_request con 0x7f356c09f1e0 auth_method 0 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f3570cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f3570cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f3570cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356bfff640 10 monclient(hunting): get_auth_request con 0x7f356c1318b0 auth_method 0 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.236+0000 7f356bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f3570cfa640 1 -- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 msgr2=0x7f356c138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f3570cfa640 1 -- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 msgr2=0x7f356c138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:56.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f3570cfa640 1 --2- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 0x7f356c138bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f3570cfa640 1 --2- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 0x7f356c138bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 10 monclient(hunting): handle_auth_done global_id 5111 payload 274 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 1 -- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 msgr2=0x7f356c138bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 1 --2- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 0x7f356c138bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 1 -- 172.21.15.181:0/2518641743 >> v1:172.21.15.181:6789/0 conn(0x7f356c1321c0 legacy=0x7f356c09ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 1 monclient: found mon.noname-b 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:56.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.237+0000 7f356bfff640 1 -- 172.21.15.181:0/2518641743 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0a1230 con 0x7f356c1318b0 2024-09-15T03:31:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356bfff640 1 -- 172.21.15.181:0/2518641743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 msgr2=0x7f356c131c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356bfff640 1 --2- 172.21.15.181:0/2518641743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 0x7f356c131c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f356c06a360 tx=0x7f35600090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937057599b2e6f26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 1 -- 172.21.15.181:0/2518641743 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356002adb0 con 0x7f356c1318b0 2024-09-15T03:31:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: dump: 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.250 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: _finish_auth 0 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 1 -- 172.21.15.181:0/2518641743 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560032440 con 0x7f356c1318b0 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f357275d640 5 monclient: authenticate success, global_id 5111 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f357275d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f357275d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 1 -- 172.21.15.181:0/2518641743 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35600327e0 con 0x7f356c1318b0 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f357275d640 4 set_mon_vals no callback set 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.240+0000 7f356affd640 10 monclient: dump: 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 10 monclient: shutdown 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 -- 172.21.15.181:0/2518641743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 msgr2=0x7f356c131c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 --2- 172.21.15.181:0/2518641743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 0x7f356c131c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f356c06a360 tx=0x7f35600090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 -- 172.21.15.181:0/2518641743 shutdown_connections 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 --2- 172.21.15.181:0/2518641743 >> v2:172.21.15.153:3300/0 conn(0x7f356c09f1e0 0x7f356c138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 --2- 172.21.15.181:0/2518641743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c1318b0 0x7f356c131c80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 -- 172.21.15.181:0/2518641743 >> 172.21.15.181:0/2518641743 conn(0x7f356c0a5010 msgr2=0x7f356c0a5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 -- 172.21.15.181:0/2518641743 shutdown_connections 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 -- 172.21.15.181:0/2518641743 wait complete. 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 10 monclient: build_initial_monmap 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 10 monclient: monmap: 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:56.253 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.241+0000 7f357275d640 1 Processor -- start 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 -- start start 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient: init 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c09f1e0 0x7f356c1342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): picked mon.noname-c con 0x7f356c09f1e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 0x7f356c134800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): picked mon.noname-b con 0x7f356c1318b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 0x7f356c134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): picked mon.noname-a con 0x7f356c1321c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09bee0 con 0x7f356c1321c0 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c0a4600 con 0x7f356c1318b0 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c135aa0 con 0x7f356c09f1e0 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f357275d640 10 monclient(hunting): authenticate will time out at 920.723552s 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 0x7f356c134800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 0x7f356c134800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46870/0 (socket says 172.21.15.181:46870) 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 -- 172.21.15.181:0/3058312682 learned_addr learned my addr 172.21.15.181:0/3058312682 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:56.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): get_auth_request con 0x7f356c1318b0 auth_method 0 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 0x7f356c134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c09f1e0 0x7f356c1342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 10 monclient(hunting): get_auth_request con 0x7f356c1321c0 auth_method 0 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): get_auth_request con 0x7f356c09f1e0 auth_method 0 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 msgr2=0x7f356c134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 msgr2=0x7f356c134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient(hunting): handle_auth_done global_id 44566 payload 824 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 0x7f356c134d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f3570cfa640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 0x7f356c134d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c09f1e0 msgr2=0x7f356c1342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c09f1e0 0x7f356c1342c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 msgr2=0x7f356c134d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 0x7f356c134d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.242+0000 7f356b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.243+0000 7f356b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.243+0000 7f356b7fe640 1 -- 172.21.15.181:0/3058312682 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3560009390 con 0x7f356c1318b0 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f356b7fe640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 msgr2=0x7f356c134800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f356b7fe640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 0x7f356c134800 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c5e0 tx=0x7f355c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c005cb0 con 0x7f356c1318b0 2024-09-15T03:31:56.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:56.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: dump: 2024-09-15T03:31:56.260 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.260 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.260 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: _finish_auth 0 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 5 monclient: authenticate success, global_id 44566 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 10 monclient: _renew_subs 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c1375d0 con 0x7f356c1318b0 2024-09-15T03:31:56.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c005e30 con 0x7f356c1318b0 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c007340 con 0x7f356c1318b0 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f3553fff640 10 monclient: dump: 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 10 monclient: _renew_subs 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c135280 con 0x7f356c1318b0 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.245+0000 7f356a7fc640 4 set_mon_vals no callback set 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f355c01b400 con 0x7f356c1318b0 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f3553fff640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3558038770 0x7f355803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f355c04bc30 con 0x7f356c1318b0 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f357275d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f356bfff640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3558038770 0x7f355803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f357275d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f357275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c0636e0 con 0x7f356c1318b0 2024-09-15T03:31:56.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.246+0000 7f356bfff640 10 monclient: get_auth_request con 0x7f3558038770 auth_method 0 2024-09-15T03:31:56.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.248+0000 7f356bfff640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3558038770 0x7f355803ac20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f356c0717a0 tx=0x7f3560032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.250+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f355c01a960 con 0x7f356c1318b0 2024-09-15T03:31:56.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.250+0000 7f3553fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.250+0000 7f3553fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:56.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f357275d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:56.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f357275d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f357275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f356c137990 con 0x7f356c1318b0 2024-09-15T03:31:56.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f355c01a360 con 0x7f356c1318b0 2024-09-15T03:31:56.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f3553fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.404+0000 7f3553fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.495+0000 7f357275d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.495+0000 7f357275d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.495+0000 7f357275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.495+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f356c0649e0 con 0x7f356c1318b0 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.496+0000 7f3553fff640 1 -- 172.21.15.181:0/3058312682 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f355c01a540 con 0x7f356c1318b0 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.496+0000 7f3553fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.496+0000 7f3553fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:56.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3558038770 msgr2=0x7f355803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3558038770 0x7f355803ac20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f356c0717a0 tx=0x7f3560032000 comp rx=0 tx=0).stop 2024-09-15T03:31:56.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 10 monclient: shutdown 2024-09-15T03:31:56.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:56.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 msgr2=0x7f356c134800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 0x7f356c134800 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c5e0 tx=0x7f355c004960 comp rx=0 tx=0).stop 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 shutdown_connections 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3558038770 0x7f355803ac20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f356c1321c0 0x7f356c134d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356c1318b0 0x7f356c134800 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 --2- 172.21.15.181:0/3058312682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f356c09f1e0 0x7f356c1342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 >> 172.21.15.181:0/3058312682 conn(0x7f356c0a5010 msgr2=0x7f356c09ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 shutdown_connections 2024-09-15T03:31:56.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.498+0000 7f357275d640 1 -- 172.21.15.181:0/3058312682 wait complete. 2024-09-15T03:31:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:31:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c host1 2024-09-15T03:31:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 2 2024-09-15T03:31:56.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.578+0000 7f63114d4640 10 monclient: get_monmap_and_config 2024-09-15T03:31:56.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.578+0000 7f63114d4640 10 monclient: build_initial_monmap 2024-09-15T03:31:56.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.578+0000 7f63114d4640 10 monclient: monmap: 2024-09-15T03:31:56.579 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:56.579315+0000 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:56.579315+0000 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:56.580 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:56.581 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 Processor -- start 2024-09-15T03:31:56.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 -- start start 2024-09-15T03:31:56.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient: init 2024-09-15T03:31:56.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:56.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T03:31:56.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f630c131b40 0x7f630c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): picked mon.noname-a con 0x7f630c131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f630c09f240 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f630c136e70 0x7f630c139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): picked mon.noname-c con 0x7f630c136e70 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630c0a0300 con 0x7f630c09f240 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c0a00c0 con 0x7f630c131b40 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c09bee0 con 0x7f630c136e70 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:56.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f63114d4640 10 monclient(hunting): authenticate will time out at 921.060739s 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f630c136e70 0x7f630c139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f630c136e70 0x7f630c139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46886/0 (socket says 172.21.15.181:46886) 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 1 -- 172.21.15.181:0/3417323535 learned_addr learned my addr 172.21.15.181:0/3417323535 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): get_auth_request con 0x7f630c136e70 auth_method 0 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.579+0000 7f630affd640 1 --2- 172.21.15.181:0/3417323535 >> v2:172.21.15.153:3300/0 conn(0x7f630c131b40 0x7f630c09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630affd640 10 monclient(hunting): get_auth_request con 0x7f630c131b40 auth_method 0 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 10 monclient(hunting): handle_auth_done global_id 44569 payload 274 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 1 -- 172.21.15.181:0/3417323535 >> v2:172.21.15.153:3300/0 conn(0x7f630c131b40 msgr2=0x7f630c09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 1 --2- 172.21.15.181:0/3417323535 >> v2:172.21.15.153:3300/0 conn(0x7f630c131b40 0x7f630c09ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 1 -- 172.21.15.181:0/3417323535 >> v1:172.21.15.153:6789/0 conn(0x7f630c09f240 legacy=0x7f630c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.580+0000 7f630b7fe640 1 -- 172.21.15.181:0/3417323535 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c0a14b0 con 0x7f630c136e70 2024-09-15T03:31:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.582+0000 7f630b7fe640 1 -- 172.21.15.181:0/3417323535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 msgr2=0x7f630c139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.582+0000 7f630b7fe640 1 --2- 172.21.15.181:0/3417323535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c139250 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f630c06ab50 tx=0x7f63040091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c9fa4215724f1f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 1 -- 172.21.15.181:0/3417323535 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304032440 con 0x7f630c136e70 2024-09-15T03:31:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: dump: 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:56.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 1 -- 172.21.15.181:0/3417323535 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63040325a0 con 0x7f630c136e70 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 5 monclient: authenticate success, global_id 44569 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 1 -- 172.21.15.181:0/3417323535 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304032940 con 0x7f630c136e70 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 4 set_mon_vals no callback set 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f6309ffb640 10 monclient: dump: 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.596 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 10 monclient: shutdown 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 -- 172.21.15.181:0/3417323535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 msgr2=0x7f630c139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 --2- 172.21.15.181:0/3417323535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c139250 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f630c06ab50 tx=0x7f63040091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 -- 172.21.15.181:0/3417323535 shutdown_connections 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 --2- 172.21.15.181:0/3417323535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c139250 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 --2- 172.21.15.181:0/3417323535 >> v2:172.21.15.153:3300/0 conn(0x7f630c131b40 0x7f630c09ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 -- 172.21.15.181:0/3417323535 >> 172.21.15.181:0/3417323535 conn(0x7f630c0a5100 msgr2=0x7f630c0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 -- 172.21.15.181:0/3417323535 shutdown_connections 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.583+0000 7f63114d4640 1 -- 172.21.15.181:0/3417323535 wait complete. 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.584+0000 7f63114d4640 10 monclient: build_initial_monmap 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.584+0000 7f63114d4640 10 monclient: monmap: 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:56.598 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.584+0000 7f63114d4640 1 Processor -- start 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.584+0000 7f63114d4640 1 -- start start 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.584+0000 7f63114d4640 10 monclient: init 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f630c09f240 0x7f630c098820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): picked mon.noname-c con 0x7f630c09f240 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f630c131b40 0x7f630c098d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): picked mon.noname-a con 0x7f630c131b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c0992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): picked mon.noname-b con 0x7f630c136e70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c11bf90 con 0x7f630c131b40 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c0a0a30 con 0x7f630c136e70 2024-09-15T03:31:56.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c130c90 con 0x7f630c09f240 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f63114d4640 10 monclient(hunting): authenticate will time out at 921.066171s 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c0992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c0992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46890/0 (socket says 172.21.15.181:46890) 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 -- 172.21.15.181:0/375193269 learned_addr learned my addr 172.21.15.181:0/375193269 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): get_auth_request con 0x7f630c136e70 auth_method 0 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630affd640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f630c09f240 0x7f630c098820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630affd640 10 monclient(hunting): get_auth_request con 0x7f630c09f240 auth_method 0 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630a7fc640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f630c131b40 0x7f630c098d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630a7fc640 10 monclient(hunting): get_auth_request con 0x7f630c131b40 auth_method 0 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient(hunting): handle_auth_done global_id 44572 payload 824 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 -- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f630c09f240 msgr2=0x7f630c098820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f630c09f240 0x7f630c098820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 -- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f630c131b40 msgr2=0x7f630c098d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f630c131b40 0x7f630c098d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630b7fe640 1 -- 172.21.15.181:0/375193269 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304009450 con 0x7f630c136e70 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.585+0000 7f630affd640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f630c09f240 0x7f630c098820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f630b7fe640 1 -- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 msgr2=0x7f630c0992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f630b7fe640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c0992a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6304009940 tx=0x7f6304032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304031d40 con 0x7f630c136e70 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: dump: 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.603 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f63114d4640 5 monclient: authenticate success, global_id 44572 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630403d440 con 0x7f630c136e70 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f63114d4640 10 monclient: _renew_subs 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f63114d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c09a120 con 0x7f630c136e70 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630403d800 con 0x7f630c136e70 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f62f2ffd640 10 monclient: dump: 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.588+0000 7f63114d4640 10 monclient: _renew_subs 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.588+0000 7f63114d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.587+0000 7f63097fa640 4 set_mon_vals no callback set 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.588+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c09acf0 con 0x7f630c136e70 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.588+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6304039040 con 0x7f630c136e70 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.588+0000 7f62f2ffd640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62f8038770 0x7f62f803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.588+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f630406c2d0 con 0x7f630c136e70 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.589+0000 7f630affd640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62f8038770 0x7f62f803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.589+0000 7f630affd640 10 monclient: get_auth_request con 0x7f62f8038770 auth_method 0 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.589+0000 7f63114d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.589+0000 7f63114d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.589+0000 7f63114d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.589+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c0636e0 con 0x7f630c136e70 2024-09-15T03:31:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.591+0000 7f630affd640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62f8038770 0x7f62f803ac20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00bb80 tx=0x7f62fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.593+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6304042080 con 0x7f630c136e70 2024-09-15T03:31:56.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.593+0000 7f62f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.593+0000 7f62f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:56.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f63114d4640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:56.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f63114d4640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f63114d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f630c1352a0 con 0x7f630c136e70 2024-09-15T03:31:56.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f630402f4d0 con 0x7f630c136e70 2024-09-15T03:31:56.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f62f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.749+0000 7f62f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f63114d4640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f63114d4640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f63114d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f630c072640 con 0x7f630c136e70 2024-09-15T03:31:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f62f2ffd640 1 -- 172.21.15.181:0/375193269 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f6304030e60 con 0x7f630c136e70 2024-09-15T03:31:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f62f2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:56.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.842+0000 7f62f2ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62f8038770 msgr2=0x7f62f803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62f8038770 0x7f62f803ac20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00bb80 tx=0x7f62fc008040 comp rx=0 tx=0).stop 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 10 monclient: shutdown 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 msgr2=0x7f630c0992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c0992a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6304009940 tx=0x7f6304032000 comp rx=0 tx=0).stop 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 shutdown_connections 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62f8038770 0x7f62f803ac20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630c136e70 0x7f630c0992a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f630c131b40 0x7f630c098d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 --2- 172.21.15.181:0/375193269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f630c09f240 0x7f630c098820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 >> 172.21.15.181:0/375193269 conn(0x7f630c0a5100 msgr2=0x7f630c0a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:56.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 shutdown_connections 2024-09-15T03:31:56.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.844+0000 7f63114d4640 1 -- 172.21.15.181:0/375193269 wait complete. 2024-09-15T03:31:56.854 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:31:56.855 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c osd.9 2024-09-15T03:31:56.855 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 2 2024-09-15T03:31:56.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.924+0000 7f9a53c0f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.924+0000 7f9a53c0f640 10 monclient: build_initial_monmap 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.924+0000 7f9a53c0f640 10 monclient: monmap: 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:56.925566+0000 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:56.925566+0000 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.926 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:56.927 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:56.927 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:56.927 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:56.927 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:56.927 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.925+0000 7f9a53c0f640 1 Processor -- start 2024-09-15T03:31:56.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.925+0000 7f9a53c0f640 1 -- start start 2024-09-15T03:31:56.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.925+0000 7f9a53c0f640 10 monclient: init 2024-09-15T03:31:56.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.925+0000 7f9a53c0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:56.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c069a30 0x7f9a4c12fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a4c069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9a4c1302d0 0x7f9a4c09bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a4c1302d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a4c09c080 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c09e6b0 con 0x7f9a4c09c080 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c069e00 con 0x7f9a4c1302d0 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c09cf00 con 0x7f9a4c069a30 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a53c0f640 10 monclient(hunting): authenticate will time out at 921.407138s 2024-09-15T03:31:56.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9a4c1302d0 0x7f9a4c09bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9a4c1302d0 0x7f9a4c09bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46904/0 (socket says 172.21.15.181:46904) 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 -- 172.21.15.181:0/2207033226 learned_addr learned my addr 172.21.15.181:0/2207033226 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): get_auth_request con 0x7f9a4c1302d0 auth_method 0 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 1 --2- 172.21.15.181:0/2207033226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c069a30 0x7f9a4c12fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): get_auth_request con 0x7f9a4c069a30 auth_method 0 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a519ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient(hunting): handle_auth_done global_id 44575 payload 274 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 -- 172.21.15.181:0/2207033226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c069a30 msgr2=0x7f9a4c12fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 --2- 172.21.15.181:0/2207033226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c069a30 0x7f9a4c12fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 -- 172.21.15.181:0/2207033226 >> v1:172.21.15.153:6789/0 conn(0x7f9a4c09c080 legacy=0x7f9a4c070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient: _finish_hunting 0 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 monclient: found mon.noname-c 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.926+0000 7f9a511aa640 1 -- 172.21.15.181:0/2207033226 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c09e340 con 0x7f9a4c1302d0 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a511aa640 1 -- 172.21.15.181:0/2207033226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c1302d0 msgr2=0x7f9a4c09bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a511aa640 1 --2- 172.21.15.181:0/2207033226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c1302d0 0x7f9a4c09bb40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009910 tx=0x7f9a3c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6e6d5adde154ba2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 1 -- 172.21.15.181:0/2207033226 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c02aea0 con 0x7f9a4c1302d0 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: dump: 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.940 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: _finish_auth 0 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 1 -- 172.21.15.181:0/2207033226 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c032440 con 0x7f9a4c1302d0 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 5 monclient: authenticate success, global_id 44575 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 1 -- 172.21.15.181:0/2207033226 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c032820 con 0x7f9a4c1302d0 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 4 set_mon_vals no callback set 2024-09-15T03:31:56.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a509a9640 10 monclient: dump: 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 10 monclient: shutdown 2024-09-15T03:31:56.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 1 -- 172.21.15.181:0/2207033226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c1302d0 msgr2=0x7f9a4c09bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.929+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/2207033226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c1302d0 0x7f9a4c09bb40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009910 tx=0x7f9a3c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 -- 172.21.15.181:0/2207033226 shutdown_connections 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/2207033226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c1302d0 0x7f9a4c09bb40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/2207033226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c069a30 0x7f9a4c12fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 -- 172.21.15.181:0/2207033226 >> 172.21.15.181:0/2207033226 conn(0x7f9a4c09ea70 msgr2=0x7f9a4c09ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 -- 172.21.15.181:0/2207033226 shutdown_connections 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 -- 172.21.15.181:0/2207033226 wait complete. 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 10 monclient: build_initial_monmap 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 10 monclient: monmap: 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:56.943 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 Processor -- start 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 1 -- start start 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.930+0000 7f9a53c0f640 10 monclient: init 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:56.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 0x7f9a4c12e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a4c069a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c09c080 0x7f9a4c12ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a4c09c080 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a4c1342c0 0x7f9a4c1185e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a4c1342c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c06eef0 con 0x7f9a4c1342c0 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c069760 con 0x7f9a4c069a30 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c11c270 con 0x7f9a4c09c080 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:56.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a53c0f640 10 monclient(hunting): authenticate will time out at 921.412446s 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 0x7f9a4c12e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 0x7f9a4c12e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46918/0 (socket says 172.21.15.181:46918) 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 -- 172.21.15.181:0/581582310 learned_addr learned my addr 172.21.15.181:0/581582310 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): get_auth_request con 0x7f9a4c069a30 auth_method 0 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a511aa640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c09c080 0x7f9a4c12ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a521ac640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a4c1342c0 0x7f9a4c1185e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a511aa640 10 monclient(hunting): get_auth_request con 0x7f9a4c09c080 auth_method 0 2024-09-15T03:31:56.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a511aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a511aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a511aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient(hunting): handle_auth_done global_id 44578 payload 824 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 -- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c09c080 msgr2=0x7f9a4c12ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c09c080 0x7f9a4c12ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 -- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a4c1342c0 msgr2=0x7f9a4c1185e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a4c1342c0 0x7f9a4c1185e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient: _finish_hunting 0 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 monclient: found mon.noname-b 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a519ab640 1 -- 172.21.15.181:0/581582310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009450 con 0x7f9a4c069a30 2024-09-15T03:31:56.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a521ac640 10 monclient: get_auth_request con 0x7f9a4c1342c0 auth_method 0 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.931+0000 7f9a521ac640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a4c1342c0 0x7f9a4c1185e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.933+0000 7f9a519ab640 1 -- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 msgr2=0x7f9a4c12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.933+0000 7f9a519ab640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 0x7f9a4c12e840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800e930 tx=0x7f9a480090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48009480 con 0x7f9a4c069a30 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: dump: 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 5 monclient: authenticate success, global_id 44578 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 10 monclient: _renew_subs 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4800caf0 con 0x7f9a4c069a30 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c119e30 con 0x7f9a4c069a30 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800ce70 con 0x7f9a4c069a30 2024-09-15T03:31:56.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a41ffb640 10 monclient: dump: 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a43fff640 4 set_mon_vals no callback set 2024-09-15T03:31:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 10 monclient: _renew_subs 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.934+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c118b20 con 0x7f9a4c069a30 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9a48013680 con 0x7f9a4c069a30 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a53c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a53c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a53c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c0636e0 con 0x7f9a4c069a30 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a41ffb640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a30038740 0x7f9a3003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f9a4804c260 con 0x7f9a4c069a30 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a511aa640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a30038740 0x7f9a3003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.935+0000 7f9a511aa640 10 monclient: get_auth_request con 0x7f9a30038740 auth_method 0 2024-09-15T03:31:56.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.937+0000 7f9a511aa640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a30038740 0x7f9a3003abf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c034000 tx=0x7f9a3c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:56.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.939+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9a480236b0 con 0x7f9a4c069a30 2024-09-15T03:31:56.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.939+0000 7f9a41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:56.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:56.939+0000 7f9a41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.096+0000 7f9a53c0f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.096+0000 7f9a53c0f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.096+0000 7f9a53c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.096+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f9a4c06ec50 con 0x7f9a4c069a30 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.097+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f9a480232f0 con 0x7f9a4c069a30 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.097+0000 7f9a41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.097+0000 7f9a41ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:57.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a53c0f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:57.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a53c0f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a53c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f9a4c11aa00 con 0x7f9a4c069a30 2024-09-15T03:31:57.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a41ffb640 1 -- 172.21.15.181:0/581582310 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f9a480234d0 con 0x7f9a4c069a30 2024-09-15T03:31:57.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a41ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.192+0000 7f9a41ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:57.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a30038740 msgr2=0x7f9a3003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a30038740 0x7f9a3003abf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c034000 tx=0x7f9a3c032000 comp rx=0 tx=0).stop 2024-09-15T03:31:57.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 10 monclient: shutdown 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 msgr2=0x7f9a4c12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 0x7f9a4c12e840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800e930 tx=0x7f9a480090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 shutdown_connections 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a30038740 0x7f9a3003abf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a4c1342c0 0x7f9a4c1185e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a4c09c080 0x7f9a4c12ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 --2- 172.21.15.181:0/581582310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a4c069a30 0x7f9a4c12e840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 >> 172.21.15.181:0/581582310 conn(0x7f9a4c09ea70 msgr2=0x7f9a4c132aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 shutdown_connections 2024-09-15T03:31:57.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.194+0000 7f9a53c0f640 1 -- 172.21.15.181:0/581582310 wait complete. 2024-09-15T03:31:57.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:31:57.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c host2 2024-09-15T03:31:57.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 2 2024-09-15T03:31:57.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.275+0000 7f334848b640 10 monclient: get_monmap_and_config 2024-09-15T03:31:57.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.275+0000 7f334848b640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.275+0000 7f334848b640 10 monclient: monmap: 2024-09-15T03:31:57.276 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:57.276 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:57.276115+0000 2024-09-15T03:31:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:57.276115+0000 2024-09-15T03:31:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:57.278 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:57.278 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:57.278 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:57.278 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.275+0000 7f334848b640 1 Processor -- start 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.275+0000 7f334848b640 1 -- start start 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.275+0000 7f334848b640 10 monclient: init 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3340069a30 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 0x7f334012eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient(hunting): picked mon.noname-a con 0x7f334012e740 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f334012f080 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:57.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334009d110 con 0x7f3340069a30 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3340132880 con 0x7f334012f080 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334009e520 con 0x7f334012e740 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f334848b640 10 monclient(hunting): authenticate will time out at 921.757535s 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f3346a28640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f334012f080 0x7f33401390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50824/0 (socket says 172.21.15.181:50824) 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f3346a28640 1 -- 172.21.15.181:0/3035798503 learned_addr learned my addr 172.21.15.181:0/3035798503 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 0x7f334012eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.276+0000 7f3345a26640 10 monclient(hunting): get_auth_request con 0x7f334012e740 auth_method 0 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 1 -- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 msgr2=0x7f334012eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 1 -- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 msgr2=0x7f334012eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 0x7f334012eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.277+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 0x7f334012eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.278+0000 7f3346a28640 1 -- 172.21.15.181:0/3035798503 >> v1:172.21.15.181:6789/0 conn(0x7f334012f080 legacy=0x7f33401390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.278+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935326722 0 0) 0x7f3340132880 con 0x7f334012f080 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): my global_id is 44581 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c0034e0 con 0x7f334012f080 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 1 -- 172.21.15.181:0/3035798503 >> v1:172.21.15.153:6789/0 conn(0x7f3340069a30 legacy=0x7f3340132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579433826 0 0) 0x7f332c0034e0 con 0x7f334012f080 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 msgr2=0x7f334012eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 --2- 172.21.15.181:0/3035798503 >> v2:172.21.15.153:3300/0 conn(0x7f334012e740 0x7f334012eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 >> v1:172.21.15.153:6789/0 conn(0x7f3340069a30 legacy=0x7f3340132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340069e00 con 0x7f334012f080 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f333c001aa0 con 0x7f334012f080 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f334848b640 5 monclient: authenticate success, global_id 44581 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: dump: 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 >> v1:172.21.15.181:6789/0 conn(0x7f334012f080 legacy=0x7f33401390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c003b20 0x7f332c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): picked mon.c con 0x7f332c003b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f332c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): picked mon.b con 0x7f332c004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f332c005310 0x7f332c00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): picked mon.a con 0x7f332c005310 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c2a0 con 0x7f332c005310 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c410 con 0x7f332c004780 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c5d0 con 0x7f332c003b20 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345225640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f334848b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f332c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): get_auth_request con 0x7f332c004780 auth_method 0 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c003b20 0x7f332c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 10 monclient(hunting): get_auth_request con 0x7f332c003b20 auth_method 0 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346a28640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f332c005310 0x7f332c00b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346a28640 10 monclient(hunting): get_auth_request con 0x7f332c005310 auth_method 0 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346a28640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346a28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient(hunting): handle_auth_done global_id 44581 payload 293 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 -- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c003b20 msgr2=0x7f332c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c003b20 0x7f332c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 -- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f332c005310 msgr2=0x7f332c00b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f332c005310 0x7f332c00b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 monclient: found mon.b 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346a28640 1 -- 172.21.15.181:0/3035798503 reap_dead start 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 1 -- 172.21.15.181:0/3035798503 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c0034e0 con 0x7f332c004780 2024-09-15T03:31:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3345a26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.279+0000 7f3346227640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c003b20 0x7f332c003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345a26640 1 -- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 msgr2=0x7f332c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345a26640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f332c004bf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3330006fd0 tx=0x7f3330008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=409de9137beceb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333002add0 con 0x7f332c004780 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: dump: 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f334848b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333002e070 con 0x7f332c004780 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f334848b640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f334848b640 4 set_mon_vals no callback set 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 1 -- 172.21.15.181:0/3035798503 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330034660 con 0x7f332c004780 2024-09-15T03:31:57.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.282+0000 7f3345225640 10 monclient: dump: 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 10 monclient: shutdown 2024-09-15T03:31:57.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 -- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 msgr2=0x7f332c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f332c004bf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3330006fd0 tx=0x7f3330008040 comp rx=0 tx=0).stop 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 -- 172.21.15.181:0/3035798503 shutdown_connections 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 --2- 172.21.15.181:0/3035798503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f332c004bf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 -- 172.21.15.181:0/3035798503 >> 172.21.15.181:0/3035798503 conn(0x7f334009ea70 msgr2=0x7f334009ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 -- 172.21.15.181:0/3035798503 shutdown_connections 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 -- 172.21.15.181:0/3035798503 wait complete. 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 10 monclient: monmap: 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:57.299 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.283+0000 7f334848b640 1 Processor -- start 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 -- start start 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient: init 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f334009b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): picked mon.noname-b con 0x7f332c004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:57.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3340118ac0 0x7f334012dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): picked mon.noname-a con 0x7f3340118ac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f334012e1f0 0x7f3340141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): picked mon.noname-c con 0x7f334012e1f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334009c1d0 con 0x7f3340118ac0 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3340098b90 con 0x7f332c004780 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334011add0 con 0x7f334012e1f0 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f334848b640 10 monclient(hunting): authenticate will time out at 921.765499s 2024-09-15T03:31:57.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f334009b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f334009b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46940/0 (socket says 172.21.15.181:46940) 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 -- 172.21.15.181:0/1432062371 learned_addr learned my addr 172.21.15.181:0/1432062371 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): get_auth_request con 0x7f332c004780 auth_method 0 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient(hunting): handle_auth_done global_id 44584 payload 824 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 -- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f334012e1f0 msgr2=0x7f3340141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3345a26640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3340118ac0 0x7f334012dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f334012e1f0 0x7f3340141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 -- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3340118ac0 msgr2=0x7f334012dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3340118ac0 0x7f334012dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 monclient: found mon.noname-b 2024-09-15T03:31:57.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.284+0000 7f3346227640 1 -- 172.21.15.181:0/1432062371 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334011b0b0 con 0x7f332c004780 2024-09-15T03:31:57.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.286+0000 7f3346227640 1 -- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 msgr2=0x7f334009b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.286+0000 7f3346227640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f334009b6a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f333800e5c0 tx=0x7f3338009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33380072e0 con 0x7f332c004780 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: dump: 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3338007440 con 0x7f332c004780 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 5 monclient: authenticate success, global_id 44584 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338005990 con 0x7f332c004780 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 10 monclient: _renew_subs 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334011b390 con 0x7f332c004780 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f33367fc640 10 monclient: dump: 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f3344a24640 4 set_mon_vals no callback set 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 10 monclient: _renew_subs 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.287+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3340119540 con 0x7f332c004780 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3338014480 con 0x7f332c004780 2024-09-15T03:31:57.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f334848b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f334848b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f334848b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33400636e0 con 0x7f332c004780 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f33367fc640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3324038770 0x7f332403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f333804b560 con 0x7f332c004780 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f3345a26640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3324038770 0x7f332403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.288+0000 7f3345a26640 10 monclient: get_auth_request con 0x7f3324038770 auth_method 0 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.290+0000 7f3345a26640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3324038770 0x7f332403ac20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3330034000 tx=0x7f3330035000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.292+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f33380075b0 con 0x7f332c004780 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.292+0000 7f33367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.292+0000 7f33367fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.448+0000 7f334848b640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.448+0000 7f334848b640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.448+0000 7f334848b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.448+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f3340098f30 con 0x7f332c004780 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.449+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f333801e440 con 0x7f332c004780 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.449+0000 7f33367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.449+0000 7f33367fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:57.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:31:57.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f334848b640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:57.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f334848b640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f334848b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f3340072640 con 0x7f332c004780 2024-09-15T03:31:57.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f33367fc640 1 -- 172.21.15.181:0/1432062371 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f3338013cb0 con 0x7f332c004780 2024-09-15T03:31:57.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f33367fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:57.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.542+0000 7f33367fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3324038770 msgr2=0x7f332403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3324038770 0x7f332403ac20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3330034000 tx=0x7f3330035000 comp rx=0 tx=0).stop 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 10 monclient: shutdown 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 msgr2=0x7f334009b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f334009b6a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f333800e5c0 tx=0x7f3338009040 comp rx=0 tx=0).stop 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 shutdown_connections 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3324038770 0x7f332403ac20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f334012e1f0 0x7f3340141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3340118ac0 0x7f334012dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 --2- 172.21.15.181:0/1432062371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f332c004780 0x7f334009b6a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 >> 172.21.15.181:0/1432062371 conn(0x7f334009ea70 msgr2=0x7f33400a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 shutdown_connections 2024-09-15T03:31:57.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.544+0000 7f334848b640 1 -- 172.21.15.181:0/1432062371 wait complete. 2024-09-15T03:31:57.554 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-15T03:31:57.554 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:31:57.554 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm host1 foo 2024-09-15T03:31:57.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.628+0000 7efd77834640 10 monclient: get_monmap_and_config 2024-09-15T03:31:57.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.628+0000 7efd77834640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.628+0000 7efd77834640 10 monclient: monmap: 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:57.628871+0000 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:57.628871+0000 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:57.630 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:57.631 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.628+0000 7efd77834640 1 Processor -- start 2024-09-15T03:31:57.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.628+0000 7efd77834640 1 -- start start 2024-09-15T03:31:57.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.628+0000 7efd77834640 10 monclient: init 2024-09-15T03:31:57.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:31:57.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd7009ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:57.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd7009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:57.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efd70069d10 0x7efd7013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient(hunting): picked mon.noname-c con 0x7efd70069d10 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7006a120 con 0x7efd7009ec50 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd70067420 con 0x7efd7009f020 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7009fdc0 con 0x7efd70069d10 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd77834640 10 monclient(hunting): authenticate will time out at 922.110379s 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efd70069d10 0x7efd7013a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efd70069d10 0x7efd7013a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46942/0 (socket says 172.21.15.181:46942) 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd74dcf640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7efd7009f020 0x7efd70136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:50838/0 (socket says 172.21.15.181:50838) 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 1 -- 172.21.15.181:0/1497828017 learned_addr learned my addr 172.21.15.181:0/1497828017 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:57.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 10 monclient(hunting): get_auth_request con 0x7efd70069d10 auth_method 0 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.629+0000 7efd75dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd74dcf640 1 -- 172.21.15.181:0/1497828017 >> v1:172.21.15.181:6789/0 conn(0x7efd7009f020 legacy=0x7efd70136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd755d0640 1 -- 172.21.15.181:0/1497828017 >> v1:172.21.15.153:6789/0 conn(0x7efd7009ec50 legacy=0x7efd70131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 10 monclient(hunting): handle_auth_done global_id 44587 payload 274 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 1 -- 172.21.15.181:0/1497828017 >> v1:172.21.15.181:6789/0 conn(0x7efd7009f020 legacy=0x7efd70136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 1 -- 172.21.15.181:0/1497828017 >> v1:172.21.15.153:6789/0 conn(0x7efd7009ec50 legacy=0x7efd70131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 1 monclient: found mon.noname-c 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.631+0000 7efd75dd1640 1 -- 172.21.15.181:0/1497828017 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700a0510 con 0x7efd70069d10 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd75dd1640 1 -- 172.21.15.181:0/1497828017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd70069d10 msgr2=0x7efd7013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd75dd1640 1 --2- 172.21.15.181:0/1497828017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd70069d10 0x7efd7013a490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd68009a10 tx=0x7efd680091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c522ccfb4af70c41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 1 -- 172.21.15.181:0/1497828017 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6802f4b0 con 0x7efd70069d10 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: dump: 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.644 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 1 -- 172.21.15.181:0/1497828017 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6802f610 con 0x7efd70069d10 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd77834640 5 monclient: authenticate success, global_id 44587 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd77834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:57.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd77834640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd77834640 4 set_mon_vals no callback set 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 1 -- 172.21.15.181:0/1497828017 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6802f9b0 con 0x7efd70069d10 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.634+0000 7efd6ffff640 10 monclient: dump: 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.646 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 10 monclient: shutdown 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 -- 172.21.15.181:0/1497828017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd70069d10 msgr2=0x7efd7013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 --2- 172.21.15.181:0/1497828017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd70069d10 0x7efd7013a490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd68009a10 tx=0x7efd680091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 -- 172.21.15.181:0/1497828017 shutdown_connections 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 --2- 172.21.15.181:0/1497828017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd70069d10 0x7efd7013a490 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 -- 172.21.15.181:0/1497828017 >> 172.21.15.181:0/1497828017 conn(0x7efd700a5d40 msgr2=0x7efd700a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 -- 172.21.15.181:0/1497828017 shutdown_connections 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 1 -- 172.21.15.181:0/1497828017 wait complete. 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.635+0000 7efd77834640 10 monclient: monmap: 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.648 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 Processor -- start 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 -- start start 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient: init 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd70069d10 0x7efd7009ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): picked mon.noname-c con 0x7efd70069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 0x7efd7011bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): picked mon.noname-a con 0x7efd7009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7009f020 0x7efd70130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): picked mon.noname-b con 0x7efd7009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd700a4090 con 0x7efd7009ec50 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7011dfe0 con 0x7efd7009f020 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7011dd00 con 0x7efd70069d10 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.636+0000 7efd77834640 10 monclient(hunting): authenticate will time out at 922.117963s 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7009f020 0x7efd70130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7009f020 0x7efd70130be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46944/0 (socket says 172.21.15.181:46944) 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 -- 172.21.15.181:0/3912671155 learned_addr learned my addr 172.21.15.181:0/3912671155 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): get_auth_request con 0x7efd7009f020 auth_method 0 2024-09-15T03:31:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 0x7efd7011bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 10 monclient(hunting): get_auth_request con 0x7efd7009ec50 auth_method 0 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd70069d10 0x7efd7009ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): get_auth_request con 0x7efd70069d10 auth_method 0 2024-09-15T03:31:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 1 -- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 msgr2=0x7efd7011bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 1 -- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 msgr2=0x7efd7011bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 0x7efd7011bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd74dcf640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 0x7efd7011bc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd755d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient(hunting): handle_auth_done global_id 44593 payload 824 2024-09-15T03:31:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 -- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd70069d10 msgr2=0x7efd7009ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd70069d10 0x7efd7009ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 -- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 msgr2=0x7efd7011bc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd7009ec50 0x7efd7011bc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 monclient: found mon.noname-b 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.637+0000 7efd75dd1640 1 -- 172.21.15.181:0/3912671155 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68009450 con 0x7efd7009f020 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd75dd1640 1 -- 172.21.15.181:0/3912671155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7009f020 msgr2=0x7efd70130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd75dd1640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7009f020 0x7efd70130be0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efd68035040 tx=0x7efd68032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6802fe20 con 0x7efd7009f020 2024-09-15T03:31:57.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: dump: 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd68032cf0 con 0x7efd7009f020 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 5 monclient: authenticate success, global_id 44593 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6803b650 con 0x7efd7009f020 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 10 monclient: _renew_subs 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 1 -- 172.21.15.181:0/3912671155 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7011c180 con 0x7efd7009f020 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6d7fa640 10 monclient: dump: 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd6f7fe640 4 set_mon_vals no callback set 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 10 monclient: _renew_subs 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.639+0000 7efd77834640 1 -- 172.21.15.181:0/3912671155 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd701312c0 con 0x7efd7009f020 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efd68039040 con 0x7efd7009f020 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd6d7fa640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd5c038740 0x7efd5c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd77834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd77834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd77834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7efd6806c330 con 0x7efd7009f020 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd77834640 1 -- 172.21.15.181:0/3912671155 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7009c070 con 0x7efd7009f020 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd755d0640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd5c038740 0x7efd5c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.640+0000 7efd755d0640 10 monclient: get_auth_request con 0x7efd5c038740 auth_method 0 2024-09-15T03:31:57.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.643+0000 7efd755d0640 1 --2- 172.21.15.181:0/3912671155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd5c038740 0x7efd5c03abf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efd6000f2a0 tx=0x7efd60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.646+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efd7009c070 con 0x7efd7009f020 2024-09-15T03:31:57.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.646+0000 7efd6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.646+0000 7efd6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.641+0000 7fe3bf246640 10 monclient: get_monmap_and_config 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.641+0000 7fe3bf246640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.641+0000 7fe3bf246640 10 monclient: monmap: 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:31:57.641914+0000 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:31:57.641914+0000 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.641+0000 7fe3bf246640 1 Processor -- start 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 1 -- start start 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient: init 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:31:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3b81318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe3b809ea50 0x7fe3b809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): picked mon.noname-c con 0x7fe3b809ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 0x7fe3b8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): picked mon.noname-a con 0x7fe3b8136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b8131cc0 con 0x7fe3b81318f0 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b809fbf0 con 0x7fe3b8136c20 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b809fe30 con 0x7fe3b809ea50 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bf246640 10 monclient(hunting): authenticate will time out at 915.281629s 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bcfe2640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe3b81318f0 0x7fe3b8131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:58638/0 (socket says 172.21.15.153:58638) 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bcfe2640 1 -- 172.21.15.153:0/1890726011 learned_addr learned my addr 172.21.15.153:0/1890726011 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 1 --2- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 0x7fe3b8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 1 --2- 172.21.15.153:0/1890726011 >> v2:172.21.15.181:3300/0 conn(0x7fe3b809ea50 0x7fe3b809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 10 monclient(hunting): get_auth_request con 0x7fe3b8136c20 auth_method 0 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bcfe2640 1 -- 172.21.15.153:0/1890726011 >> v1:172.21.15.153:6789/0 conn(0x7fe3b81318f0 legacy=0x7fe3b8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bcfe2640 1 -- 172.21.15.153:0/1890726011 >> v1:172.21.15.153:6789/0 conn(0x7fe3b81318f0 legacy=0x7fe3b8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): get_auth_request con 0x7fe3b809ea50 auth_method 0 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bcfe2640 1 --1- 172.21.15.153:0/1890726011 >> v1:172.21.15.153:6789/0 conn(0x7fe3b81318f0 0x7fe3b8131f30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-15T03:31:57.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 1 -- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 msgr2=0x7fe3b8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 1 -- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 msgr2=0x7fe3b8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 1 --2- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 0x7fe3b8139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3bd7e3640 1 --2- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 0x7fe3b8139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_done global_id 44596 payload 274 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 1 -- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 msgr2=0x7fe3b8139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 1 --2- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 0x7fe3b8139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 1 -- 172.21.15.153:0/1890726011 >> v1:172.21.15.153:6789/0 conn(0x7fe3b81318f0 legacy=0x7fe3b8131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 1 monclient: found mon.noname-c 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.642+0000 7fe3b7fff640 1 -- 172.21.15.153:0/1890726011 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b80a1210 con 0x7fe3b809ea50 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.645+0000 7fe3b7fff640 1 -- 172.21.15.153:0/1890726011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 msgr2=0x7fe3b809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.645+0000 7fe3b7fff640 1 --2- 172.21.15.153:0/1890726011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b809ee20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3b806f600 tx=0x7fe3a8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dedd6b5dba1a026 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 1 -- 172.21.15.153:0/1890726011 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8033b10 con 0x7fe3b809ea50 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:57.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: dump: 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 1 -- 172.21.15.153:0/1890726011 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8033c70 con 0x7fe3b809ea50 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 5 monclient: authenticate success, global_id 44596 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 1 -- 172.21.15.153:0/1890726011 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a803b440 con 0x7fe3b809ea50 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 4 set_mon_vals no callback set 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3b77fe640 10 monclient: dump: 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.672 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 10 monclient: shutdown 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 -- 172.21.15.153:0/1890726011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 msgr2=0x7fe3b809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 --2- 172.21.15.153:0/1890726011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b809ee20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe3b806f600 tx=0x7fe3a8009780 comp rx=0 tx=0).stop 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 -- 172.21.15.153:0/1890726011 shutdown_connections 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 --2- 172.21.15.153:0/1890726011 >> v2:172.21.15.153:3300/0 conn(0x7fe3b8136c20 0x7fe3b8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 --2- 172.21.15.153:0/1890726011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b809ee20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 -- 172.21.15.153:0/1890726011 >> 172.21.15.153:0/1890726011 conn(0x7fe3b8067110 msgr2=0x7fe3b8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 -- 172.21.15.153:0/1890726011 shutdown_connections 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.646+0000 7fe3bf246640 1 -- 172.21.15.153:0/1890726011 wait complete. 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.647+0000 7fe3bf246640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.647+0000 7fe3bf246640 10 monclient: monmap: 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:31:57.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.647+0000 7fe3bf246640 1 Processor -- start 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.647+0000 7fe3bf246640 1 -- start start 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.647+0000 7fe3bf246640 10 monclient: init 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b806d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): picked mon.noname-b con 0x7fe3b809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 0x7fe3b8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): picked mon.noname-c con 0x7fe3b81318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 0x7fe3b8096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): picked mon.noname-a con 0x7fe3b8136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b809bf80 con 0x7fe3b8136c20 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b809dbd0 con 0x7fe3b809ea50 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b806e090 con 0x7fe3b81318f0 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bf246640 10 monclient(hunting): authenticate will time out at 915.287377s 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 0x7fe3b8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 0x7fe3b8096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 0x7fe3b8096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58130/0 (socket says 172.21.15.153:58130) 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 -- 172.21.15.153:0/831499766 learned_addr learned my addr 172.21.15.153:0/831499766 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): get_auth_request con 0x7fe3b81318f0 auth_method 0 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 10 monclient(hunting): get_auth_request con 0x7fe3b8136c20 auth_method 0 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): get_auth_request con 0x7fe3b809ea50 auth_method 0 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 msgr2=0x7fe3b8096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 msgr2=0x7fe3b8096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:57.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 0x7fe3b8096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bcfe2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3bd7e3640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 0x7fe3b8096fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient(hunting): handle_auth_done global_id 5129 payload 824 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 msgr2=0x7fe3b806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b806d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 msgr2=0x7fe3b8096fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 0x7fe3b8096fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 monclient: found mon.noname-c 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.648+0000 7fe3b7fff640 1 -- 172.21.15.153:0/831499766 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a8009a20 con 0x7fe3b81318f0 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b7fff640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 msgr2=0x7fe3b8096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b7fff640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 0x7fe3b8096a60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8033340 tx=0x7fe3a803e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8001040 con 0x7fe3b81318f0 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: dump: 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.680 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8045830 con 0x7fe3b81318f0 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 5 monclient: authenticate success, global_id 5129 2024-09-15T03:31:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8045bf0 con 0x7fe3b81318f0 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 10 monclient: _renew_subs 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b8098a10 con 0x7fe3b81318f0 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b6ffd640 4 set_mon_vals no callback set 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3b4ff9640 10 monclient: dump: 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 10 monclient: _renew_subs 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.651+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b8099cd0 con 0x7fe3b81318f0 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe3a8042050 con 0x7fe3b81318f0 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3bf246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3bf246640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3bf246640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b804bf60 con 0x7fe3b81318f0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3b4ff9640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe398038740 0x7fe39803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7fe3a8075850 con 0x7fe3b81318f0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3bcfe2640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe398038740 0x7fe39803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.652+0000 7fe3bcfe2640 10 monclient: get_auth_request con 0x7fe398038740 auth_method 0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.655+0000 7fe3bcfe2640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe398038740 0x7fe39803abf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0007fd0 tx=0x7fe3b0007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.656+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe3a803a3a0 con 0x7fe3b81318f0 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.656+0000 7fe3b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.656+0000 7fe3b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.807+0000 7efd77834640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.807+0000 7efd77834640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.807+0000 7efd77834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.807+0000 7efd77834640 1 -- 172.21.15.181:0/3912671155 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7efd70131480 con 0x7efd7009f020 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.808+0000 7efd6d7fa640 1 -- 172.21.15.181:0/3912671155 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v73) v1 ==== 120+0+0 (secure 0 0 0) 0x7efd6803f020 con 0x7efd7009f020 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.808+0000 7efd6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.808+0000 7efd6d7fa640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T03:31:57.809 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T03:31:57.812 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:31:57.812 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush unlink host1 foo 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3bf246640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3bf246640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3bf246640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe3b80638d0 con 0x7fe3b81318f0 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3b4ff9640 1 -- 172.21.15.153:0/831499766 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fe3a8049080 con 0x7fe3b81318f0 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:31:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.870+0000 7fe3b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:57.872 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":32,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:31:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe398038740 msgr2=0x7fe39803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe398038740 0x7fe39803abf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0007fd0 tx=0x7fe3b0007f20 comp rx=0 tx=0).stop 2024-09-15T03:31:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 10 monclient: shutdown 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 msgr2=0x7fe3b8096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 0x7fe3b8096a60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8033340 tx=0x7fe3a803e000 comp rx=0 tx=0).stop 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 shutdown_connections 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe398038740 0x7fe39803abf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3b8136c20 0x7fe3b8096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3b81318f0 0x7fe3b8096a60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 --2- 172.21.15.153:0/831499766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3b809ea50 0x7fe3b806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 >> 172.21.15.153:0/831499766 conn(0x7fe3b8067110 msgr2=0x7fe3b80a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 shutdown_connections 2024-09-15T03:31:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:31:57.872+0000 7fe3bf246640 1 -- 172.21.15.153:0/831499766 wait complete. 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.886+0000 7f026ea6f640 10 monclient: get_monmap_and_config 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.886+0000 7f026ea6f640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.886+0000 7f026ea6f640 10 monclient: monmap: 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:57.887146+0000 2024-09-15T03:31:57.887 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:57.887146+0000 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:57.888 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 Processor -- start 2024-09-15T03:31:57.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 -- start start 2024-09-15T03:31:57.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient: init 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 0x7f02680691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): picked mon.noname-b con 0x7f02680a0ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f026809ee60 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f026809f230 0x7f02681321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): picked mon.noname-c con 0x7f026809f230 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026806a5b0 con 0x7f026809ee60 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02680a0ea0 con 0x7f026809f230 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02680a1bb0 con 0x7f02680a0ad0 2024-09-15T03:31:57.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026ea6f640 10 monclient(hunting): authenticate will time out at 922.368628s 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f026809f230 0x7f02681321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f026809f230 0x7f02681321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46960/0 (socket says 172.21.15.181:46960) 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 1 -- 172.21.15.181:0/3798352486 learned_addr learned my addr 172.21.15.181:0/3798352486 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 10 monclient(hunting): get_auth_request con 0x7f026809f230 auth_method 0 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026d00c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 1 --2- 172.21.15.181:0/3798352486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 0x7f02680691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 10 monclient(hunting): get_auth_request con 0x7f02680a0ad0 auth_method 0 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.887+0000 7f026c80b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 10 monclient(hunting): handle_auth_done global_id 5132 payload 274 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 1 -- 172.21.15.181:0/3798352486 >> v2:172.21.15.181:3300/0 conn(0x7f026809f230 msgr2=0x7f02681321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 1 --2- 172.21.15.181:0/3798352486 >> v2:172.21.15.181:3300/0 conn(0x7f026809f230 0x7f02681321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 1 -- 172.21.15.181:0/3798352486 >> v1:172.21.15.181:6789/0 conn(0x7f026809ee60 legacy=0x7f0268131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 1 monclient: found mon.noname-b 2024-09-15T03:31:57.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:57.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.888+0000 7f026c80b640 1 -- 172.21.15.181:0/3798352486 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680a0460 con 0x7f02680a0ad0 2024-09-15T03:31:57.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026c80b640 1 -- 172.21.15.181:0/3798352486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 msgr2=0x7f02680691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026c80b640 1 --2- 172.21.15.181:0/3798352486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 0x7f02680691d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f025c009850 tx=0x7f025c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248e51f7e51600d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 1 -- 172.21.15.181:0/3798352486 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c02ae80 con 0x7f02680a0ad0 2024-09-15T03:31:57.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: dump: 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 1 -- 172.21.15.181:0/3798352486 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c032440 con 0x7f02680a0ad0 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 5 monclient: authenticate success, global_id 5132 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 4 set_mon_vals no callback set 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 1 -- 172.21.15.181:0/3798352486 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c0327c0 con 0x7f02680a0ad0 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f02677fe640 10 monclient: dump: 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.901 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 10 monclient: shutdown 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 -- 172.21.15.181:0/3798352486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 msgr2=0x7f02680691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3798352486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 0x7f02680691d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f025c009850 tx=0x7f025c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 -- 172.21.15.181:0/3798352486 shutdown_connections 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3798352486 >> v2:172.21.15.181:3300/0 conn(0x7f026809f230 0x7f02681321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3798352486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680a0ad0 0x7f02680691d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 -- 172.21.15.181:0/3798352486 >> 172.21.15.181:0/3798352486 conn(0x7f0268071370 msgr2=0x7f02680a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 -- 172.21.15.181:0/3798352486 shutdown_connections 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.891+0000 7f026ea6f640 1 -- 172.21.15.181:0/3798352486 wait complete. 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.892+0000 7f026ea6f640 10 monclient: build_initial_monmap 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.892+0000 7f026ea6f640 10 monclient: monmap: 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:57.903 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.892+0000 7f026ea6f640 1 Processor -- start 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.892+0000 7f026ea6f640 1 -- start start 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.892+0000 7f026ea6f640 10 monclient: init 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026809ee60 0x7f0268140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): picked mon.noname-c con 0x7f026809ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 0x7f0268133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): picked mon.noname-b con 0x7f026809f230 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 0x7f026811e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): picked mon.noname-a con 0x7f02680a0ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:57.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02681220e0 con 0x7f02680a0ad0 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268121e00 con 0x7f026809f230 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02681223c0 con 0x7f026809ee60 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026ea6f640 10 monclient(hunting): authenticate will time out at 922.374386s 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 0x7f0268133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 0x7f0268133cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46964/0 (socket says 172.21.15.181:46964) 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 -- 172.21.15.181:0/3136836886 learned_addr learned my addr 172.21.15.181:0/3136836886 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): get_auth_request con 0x7f026809f230 auth_method 0 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026809ee60 0x7f0268140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 0x7f026811e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): get_auth_request con 0x7f026809ee60 auth_method 0 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 10 monclient(hunting): get_auth_request con 0x7f02680a0ad0 auth_method 0 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:57.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026c80b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 msgr2=0x7f026811e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 msgr2=0x7f026811e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 0x7f026811e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f026d00c640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 0x7f026811e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient(hunting): handle_auth_done global_id 44605 payload 824 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026809ee60 msgr2=0x7f0268140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026809ee60 0x7f0268140eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 msgr2=0x7f026811e7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 0x7f026811e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:57.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 monclient: found mon.noname-b 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.893+0000 7f0267fff640 1 -- 172.21.15.181:0/3136836886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c009390 con 0x7f026809f230 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.895+0000 7f0267fff640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 msgr2=0x7f0268133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0267fff640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 0x7f0268133cd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f025400b4f0 tx=0x7f02540090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025400f720 con 0x7f026809f230 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: dump: 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.908 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: _finish_auth 0 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025400f8a0 con 0x7f026809f230 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 5 monclient: authenticate success, global_id 44605 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 10 monclient: _renew_subs 2024-09-15T03:31:57.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0268141cf0 con 0x7f026809f230 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025400fc60 con 0x7f026809f230 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0264ff9640 10 monclient: dump: 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:57.910 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f0266ffd640 4 set_mon_vals no callback set 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 10 monclient: _renew_subs 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.896+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02681210a0 con 0x7f026809f230 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0254011610 con 0x7f026809f230 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f026ea6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f026ea6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f026ea6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02680008d0 con 0x7f026809f230 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f0264ff9640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f024c038740 0x7f024c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f025404bbc0 con 0x7f026809f230 2024-09-15T03:31:57.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f026c80b640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f024c038740 0x7f024c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.897+0000 7f026c80b640 10 monclient: get_auth_request con 0x7f024c038740 auth_method 0 2024-09-15T03:31:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.899+0000 7f026c80b640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f024c038740 0x7f024c03abf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f026806f8c0 tx=0x7f025c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.901+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f025401a400 con 0x7f026809f230 2024-09-15T03:31:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.901+0000 7f0264ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:57.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:57.901+0000 7f0264ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:58.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.060+0000 7f026ea6f640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:58.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.060+0000 7f026ea6f640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:58.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.060+0000 7f026ea6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:58.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.060+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f0268064a20 con 0x7f026809f230 2024-09-15T03:31:58.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.707+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v74) v1 ==== 147+0+0 (secure 0 0 0) 0x7f0254017720 con 0x7f026809f230 2024-09-15T03:31:58.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.707+0000 7f0264ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:58.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.707+0000 7f0264ff9640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.804+0000 7f026ea6f640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.804+0000 7f026ea6f640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.804+0000 7f026ea6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.804+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f026811f4e0 con 0x7f026809f230 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.805+0000 7f0264ff9640 1 -- 172.21.15.181:0/3136836886 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v74) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0254017900 con 0x7f026809f230 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.805+0000 7f0264ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.805+0000 7f0264ff9640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-15T03:31:58.806 INFO:tasks.workunit.client.0.smithi181.stderr:item -11 does not appear in that position 2024-09-15T03:31:58.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f024c038740 msgr2=0x7f024c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:58.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f024c038740 0x7f024c03abf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f026806f8c0 tx=0x7f025c032000 comp rx=0 tx=0).stop 2024-09-15T03:31:58.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 10 monclient: shutdown 2024-09-15T03:31:58.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:58.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 msgr2=0x7f0268133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 0x7f0268133cd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f025400b4f0 tx=0x7f02540090f0 comp rx=0 tx=0).stop 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 shutdown_connections 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f024c038740 0x7f024c03abf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680a0ad0 0x7f026811e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026809f230 0x7f0268133cd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 --2- 172.21.15.181:0/3136836886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026809ee60 0x7f0268140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 >> 172.21.15.181:0/3136836886 conn(0x7f0268071370 msgr2=0x7f026809d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 shutdown_connections 2024-09-15T03:31:58.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.807+0000 7f026ea6f640 1 -- 172.21.15.181:0/3136836886 wait complete. 2024-09-15T03:31:58.817 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush unlink host1 foo 2024-09-15T03:31:58.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.890+0000 7f042d71d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:58.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.890+0000 7f042d71d640 10 monclient: build_initial_monmap 2024-09-15T03:31:58.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.890+0000 7f042d71d640 10 monclient: monmap: 2024-09-15T03:31:58.891 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:58.891 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:58.891271+0000 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:58.891271+0000 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:58.892 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:58.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 Processor -- start 2024-09-15T03:31:58.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 -- start start 2024-09-15T03:31:58.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient: init 2024-09-15T03:31:58.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 0x7f0428131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): picked mon.noname-a con 0x7f042809ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): picked mon.noname-c con 0x7f04281321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f042809f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0428067420 con 0x7f042809f020 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042809fdc0 con 0x7f042809ec50 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04280a0510 con 0x7f04281321a0 2024-09-15T03:31:58.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f042d71d640 10 monclient(hunting): authenticate will time out at 923.372600s 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39506/0 (socket says 172.21.15.181:39506) 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 1 -- 172.21.15.181:0/1547161881 learned_addr learned my addr 172.21.15.181:0/1547161881 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): get_auth_request con 0x7f04281321a0 auth_method 0 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04277fe640 1 --1- 172.21.15.181:0/1547161881 >> v1:172.21.15.181:6789/0 conn(0x7f042809f020 0x7f0428069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48422/0 (socket says 172.21.15.181:48422) 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.891+0000 7f04267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:58.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 0x7f0428131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 10 monclient(hunting): get_auth_request con 0x7f042809ec50 auth_method 0 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 1 -- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 msgr2=0x7f0428131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 1 -- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 msgr2=0x7f0428131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 0x7f0428131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.892+0000 7f0426ffd640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 0x7f0428131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.893+0000 7f04277fe640 1 -- 172.21.15.181:0/1547161881 >> v1:172.21.15.181:6789/0 conn(0x7f042809f020 legacy=0x7f0428069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.893+0000 7f04267fc640 10 monclient(hunting): handle_auth_done global_id 44608 payload 274 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 -- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 msgr2=0x7f0428131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.153:3300/0 conn(0x7f042809ec50 0x7f0428131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:58.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 -- 172.21.15.181:0/1547161881 >> v1:172.21.15.181:6789/0 conn(0x7f042809f020 legacy=0x7f0428069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 10 monclient: _finish_hunting 0 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 monclient: found mon.noname-c 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 -- 172.21.15.181:0/1547161881 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280a0200 con 0x7f04281321a0 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 -- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 msgr2=0x7f0428068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 -- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 msgr2=0x7f0428068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f04280a2b00 tx=0x7f04180091d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:58.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:58.894+0000 7f04267fc640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:59.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f04267fc640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f04267fc640 10 monclient: get_auth_request con 0x7f04281321a0 auth_method 0 2024-09-15T03:31:59.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f04267fc640 0 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:31:59.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f04267fc640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient: ms_handle_reset current mon v2:172.21.15.181:3300/0 2024-09-15T03:31:59.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 1 -- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 msgr2=0x7f0428068d00 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 1 --2- 172.21.15.181:0/1547161881 >> v2:172.21.15.181:3300/0 conn(0x7f04281321a0 0x7f0428068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient: _add_conns ranks=[2,0] 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 1 --2- 172.21.15.181:0/1547161881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04140034c0 0x7f0414005900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f04140034c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0414006220 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 1 -- 172.21.15.181:0/1547161881 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041400a180 con 0x7f0414006220 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 1 -- 172.21.15.181:0/1547161881 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041400b380 con 0x7f04140034c0 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0425ffb640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.181:3300/0 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 1 --2- 172.21.15.181:0/1547161881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04140034c0 0x7f0414005900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): get_auth_request con 0x7f04140034c0 auth_method 0 2024-09-15T03:31:59.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.095+0000 7f0426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 10 monclient(hunting): handle_auth_done global_id 44608 payload 293 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 1 -- 172.21.15.181:0/1547161881 >> v1:172.21.15.153:6789/0 conn(0x7f0414006220 legacy=0x7f04140086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:31:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.096+0000 7f0426ffd640 1 -- 172.21.15.181:0/1547161881 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041400b600 con 0x7f04140034c0 2024-09-15T03:31:59.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0426ffd640 1 -- 172.21.15.181:0/1547161881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04140034c0 msgr2=0x7f0414005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0426ffd640 1 --2- 172.21.15.181:0/1547161881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04140034c0 0x7f0414005900 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0410006fd0 tx=0x7f0410008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eb8f16688607 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 1 -- 172.21.15.181:0/1547161881 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041000d9c0 con 0x7f04140034c0 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: dump: 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.104 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 1 -- 172.21.15.181:0/1547161881 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041000db40 con 0x7f04140034c0 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f042d71d640 5 monclient: authenticate success, global_id 44608 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f042d71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f042d71d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 1 -- 172.21.15.181:0/1547161881 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410012420 con 0x7f04140034c0 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f042d71d640 4 set_mon_vals no callback set 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.098+0000 7f0425ffb640 10 monclient: dump: 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.105 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 10 monclient: shutdown 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 1 -- 172.21.15.181:0/1547161881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04140034c0 msgr2=0x7f0414005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 1 --2- 172.21.15.181:0/1547161881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04140034c0 0x7f0414005900 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0410006fd0 tx=0x7f0410008040 comp rx=0 tx=0).stop 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f04277fe640 1 -- 172.21.15.181:0/1547161881 reap_dead start 2024-09-15T03:31:59.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 1 -- 172.21.15.181:0/1547161881 shutdown_connections 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 1 -- 172.21.15.181:0/1547161881 >> 172.21.15.181:0/1547161881 conn(0x7f04280a5d40 msgr2=0x7f04280a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 1 -- 172.21.15.181:0/1547161881 shutdown_connections 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 1 -- 172.21.15.181:0/1547161881 wait complete. 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 10 monclient: build_initial_monmap 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.099+0000 7f042d71d640 10 monclient: monmap: 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:59.107 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 Processor -- start 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 -- start start 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient: init 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 0x7f0428130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): picked mon.noname-b con 0x7f042809a990 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 0x7f04281411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): picked mon.noname-a con 0x7f042811bc40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0428131100 0x7f04281416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): picked mon.noname-c con 0x7f0428131100 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042809c000 con 0x7f042811bc40 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280a4390 con 0x7f042809a990 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042809ad90 con 0x7f0428131100 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f042d71d640 10 monclient(hunting): authenticate will time out at 923.581686s 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 0x7f0428130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 0x7f0428130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39524/0 (socket says 172.21.15.181:39524) 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 1 -- 172.21.15.181:0/2773778801 learned_addr learned my addr 172.21.15.181:0/2773778801 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): get_auth_request con 0x7f042809a990 auth_method 0 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f0426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04267fc640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 0x7f04281411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04277fe640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0428131100 0x7f04281416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04267fc640 10 monclient(hunting): get_auth_request con 0x7f042811bc40 auth_method 0 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04277fe640 10 monclient(hunting): get_auth_request con 0x7f0428131100 auth_method 0 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.100+0000 7f04277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 10 monclient(hunting): handle_auth_done global_id 44611 payload 824 2024-09-15T03:31:59.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f04267fc640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 msgr2=0x7f04281411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f04267fc640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 msgr2=0x7f04281411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f04267fc640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 0x7f04281411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0428131100 msgr2=0x7f04281416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0428131100 0x7f04281416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 msgr2=0x7f04281411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f04267fc640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 0x7f04281411b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 0x7f04281411b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 1 monclient: found mon.noname-b 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f0426ffd640 1 -- 172.21.15.181:0/2773778801 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042811dd10 con 0x7f042809a990 2024-09-15T03:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.101+0000 7f04277fe640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0428131100 0x7f04281416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f0426ffd640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 msgr2=0x7f0428130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f0426ffd640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 0x7f0428130bc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0410014000 tx=0x7f041001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04100128a0 con 0x7f042809a990 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: dump: 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 5 monclient: authenticate success, global_id 44611 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0410023070 con 0x7f042809a990 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 10 monclient: _renew_subs 2024-09-15T03:31:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042811dfc0 con 0x7f042809a990 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410012c10 con 0x7f042809a990 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f041effd640 10 monclient: dump: 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.119 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f0424ff9640 4 set_mon_vals no callback set 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 10 monclient: _renew_subs 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.103+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042811cd80 con 0x7f042809a990 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0410025030 con 0x7f042809a990 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f041effd640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0408038740 0x7f040803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f042d71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f042d71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f041004def0 con 0x7f042809a990 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f042d71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042811cd80 con 0x7f042809a990 2024-09-15T03:31:59.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f04267fc640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0408038740 0x7f040803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.104+0000 7f04267fc640 10 monclient: get_auth_request con 0x7f0408038740 auth_method 0 2024-09-15T03:31:59.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.107+0000 7f04267fc640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0408038740 0x7f040803abf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0428132890 tx=0x7f04180091d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.109+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f041001e420 con 0x7f042809a990 2024-09-15T03:31:59.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.109+0000 7f041effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.109+0000 7f041effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:59.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.267+0000 7f042d71d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:59.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.267+0000 7f042d71d640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:59.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.267+0000 7f042d71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.267+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f042811cd80 con 0x7f042809a990 2024-09-15T03:31:59.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.268+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v74) v1 ==== 141+0+0 (secure 0 0 0) 0x7f04100210e0 con 0x7f042809a990 2024-09-15T03:31:59.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.268+0000 7f041effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:59.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.268+0000 7f041effd640 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-15T03:31:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f042d71d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f042d71d640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f042d71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f042811cd80 con 0x7f042809a990 2024-09-15T03:31:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f041effd640 1 -- 172.21.15.181:0/2773778801 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v74) v1 ==== 141+0+0 (secure 0 0 0) 0x7f04100273d0 con 0x7f042809a990 2024-09-15T03:31:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f041effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-15T03:31:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.364+0000 7f041effd640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-15T03:31:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:item -11 does not appear in that position 2024-09-15T03:31:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0408038740 msgr2=0x7f040803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0408038740 0x7f040803abf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0428132890 tx=0x7f04180091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 10 monclient: shutdown 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 msgr2=0x7f0428130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 0x7f0428130bc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0410014000 tx=0x7f041001a000 comp rx=0 tx=0).stop 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 shutdown_connections 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0408038740 0x7f040803abf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0428131100 0x7f04281416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f042811bc40 0x7f04281411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 --2- 172.21.15.181:0/2773778801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f042809a990 0x7f0428130bc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 >> 172.21.15.181:0/2773778801 conn(0x7f04280a5d40 msgr2=0x7f04280a47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 shutdown_connections 2024-09-15T03:31:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.366+0000 7f042d71d640 1 -- 172.21.15.181:0/2773778801 wait complete. 2024-09-15T03:31:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:31:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c host1 2024-09-15T03:31:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 1 2024-09-15T03:31:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.447+0000 7f1f9b59d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.447+0000 7f1f9b59d640 10 monclient: build_initial_monmap 2024-09-15T03:31:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.447+0000 7f1f9b59d640 10 monclient: monmap: 2024-09-15T03:31:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:59.448403+0000 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:59.448403+0000 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:59.449 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 1 Processor -- start 2024-09-15T03:31:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 1 -- start start 2024-09-15T03:31:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient: init 2024-09-15T03:31:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T03:31:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f94131b40 addr v1:172.21.15.153:6789/0 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 0x7f1f94132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f94131f10 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f94136e70 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9406a230 con 0x7f1f94131b40 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f940a0300 con 0x7f1f94136e70 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940a00c0 con 0x7f1f94131f10 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f9b59d640 10 monclient(hunting): authenticate will time out at 923.929865s 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.448+0000 7f1f99b3a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f1f94136e70 0x7f1f94139250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48428/0 (socket says 172.21.15.181:48428) 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f99b3a640 1 -- 172.21.15.181:0/1857434744 learned_addr learned my addr 172.21.15.181:0/1857434744 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 1 --2- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 0x7f1f94132330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 10 monclient(hunting): get_auth_request con 0x7f1f94131f10 auth_method 0 2024-09-15T03:31:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 1 -- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 msgr2=0x7f1f94132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:31:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 1 -- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 msgr2=0x7f1f94132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 1 --2- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 0x7f1f94132330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.449+0000 7f1f98b38640 1 --2- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 0x7f1f94132330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 1 -- 172.21.15.181:0/1857434744 >> v1:172.21.15.181:6789/0 conn(0x7f1f94136e70 legacy=0x7f1f94139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97298290 0 0) 0x7f1f940a0300 con 0x7f1f94136e70 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): my global_id is 44614 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f800034e0 con 0x7f1f94136e70 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 -- 172.21.15.181:0/1857434744 >> v1:172.21.15.153:6789/0 conn(0x7f1f94131b40 legacy=0x7f1f9409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186128252 0 0) 0x7f1f800034e0 con 0x7f1f94136e70 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 msgr2=0x7f1f94132330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 --2- 172.21.15.181:0/1857434744 >> v2:172.21.15.153:3300/0 conn(0x7f1f94131f10 0x7f1f94132330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 >> v1:172.21.15.153:6789/0 conn(0x7f1f94131b40 legacy=0x7f1f9409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9409bee0 con 0x7f1f94136e70 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f1f8c001aa0 con 0x7f1f94136e70 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f9b59d640 5 monclient: authenticate success, global_id 44614 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f9b59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: dump: 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 >> v1:172.21.15.181:6789/0 conn(0x7f1f94136e70 legacy=0x7f1f94139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:31:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 0x7f1f80003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): picked mon.b con 0x7f1f80003b20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f80004780 0x7f1f80004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): picked mon.c con 0x7f1f80004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f800053b0 0x7f1f8000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): picked mon.a con 0x7f1f800053b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940a00c0 con 0x7f1f800053b0 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8000c380 con 0x7f1f80003b20 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8000c540 con 0x7f1f80004780 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f8bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f9b59d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 0x7f1f80003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): get_auth_request con 0x7f1f80003b20 auth_method 0 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f98b38640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f80004780 0x7f1f80004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f98b38640 10 monclient(hunting): get_auth_request con 0x7f1f80004780 auth_method 0 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f98b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f98b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f98b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f800053b0 0x7f1f8000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 10 monclient(hunting): get_auth_request con 0x7f1f800053b0 auth_method 0 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient(hunting): handle_auth_done global_id 44614 payload 293 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 -- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f80004780 msgr2=0x7f1f80004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f80004780 0x7f1f80004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 -- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f800053b0 msgr2=0x7f1f8000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f800053b0 0x7f1f8000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 monclient: found mon.b 2024-09-15T03:31:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99b3a640 1 -- 172.21.15.181:0/1857434744 reap_dead start 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 1 -- 172.21.15.181:0/1857434744 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800034e0 con 0x7f1f80003b20 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.451+0000 7f1f99339640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f99339640 1 -- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 msgr2=0x7f1f80003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f99339640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 0x7f1f80003f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f90001b00 tx=0x7f1f90007c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874adebfeba2f9d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9002ac50 con 0x7f1f80003b20 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: dump: 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.468 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f9b59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9002adb0 con 0x7f1f80003b20 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 1 -- 172.21.15.181:0/1857434744 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90007340 con 0x7f1f80003b20 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f9b59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f9b59d640 4 set_mon_vals no callback set 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.454+0000 7f1f8bfff640 10 monclient: dump: 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 10 monclient: shutdown 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 -- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 msgr2=0x7f1f80003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 0x7f1f80003f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f90001b00 tx=0x7f1f90007c40 comp rx=0 tx=0).stop 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 -- 172.21.15.181:0/1857434744 shutdown_connections 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/1857434744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80003b20 0x7f1f80003f30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 -- 172.21.15.181:0/1857434744 >> 172.21.15.181:0/1857434744 conn(0x7f1f940a5100 msgr2=0x7f1f940a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 -- 172.21.15.181:0/1857434744 shutdown_connections 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 1 -- 172.21.15.181:0/1857434744 wait complete. 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 10 monclient: build_initial_monmap 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.455+0000 7f1f9b59d640 10 monclient: monmap: 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:59.471 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 Processor -- start 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 -- start start 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient: init 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 0x7f1f941418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f80003b20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9412ff10 0x7f1f94098950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f9412ff10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 0x7f1f94098e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f94141e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f94130f50 con 0x7f1f80003b20 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940a0a30 con 0x7f1f94141e10 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9411c170 con 0x7f1f9412ff10 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.456+0000 7f1f9b59d640 10 monclient(hunting): authenticate will time out at 923.937944s 2024-09-15T03:31:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 0x7f1f94098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 0x7f1f94098e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39554/0 (socket says 172.21.15.181:39554) 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 -- 172.21.15.181:0/4143683719 learned_addr learned my addr 172.21.15.181:0/4143683719 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): get_auth_request con 0x7f1f94141e10 auth_method 0 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 0x7f1f941418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9412ff10 0x7f1f94098950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): get_auth_request con 0x7f1f9412ff10 auth_method 0 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 10 monclient(hunting): get_auth_request con 0x7f1f80003b20 auth_method 0 2024-09-15T03:31:59.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f98b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 msgr2=0x7f1f941418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:59.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 msgr2=0x7f1f941418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 0x7f1f941418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99339640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 0x7f1f941418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient(hunting): handle_auth_done global_id 44617 payload 824 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9412ff10 msgr2=0x7f1f94098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9412ff10 0x7f1f94098950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 msgr2=0x7f1f941418d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 0x7f1f941418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 monclient: found mon.noname-b 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.457+0000 7f1f99b3a640 1 -- 172.21.15.181:0/4143683719 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9409b020 con 0x7f1f94141e10 2024-09-15T03:31:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f99b3a640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 msgr2=0x7f1f94098e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f99b3a640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 0x7f1f94098e90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c001810 tx=0x7f1f8c001b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c010660 con 0x7f1f94141e10 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: dump: 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c0107e0 con 0x7f1f94141e10 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 5 monclient: authenticate success, global_id 44617 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c010ba0 con 0x7f1f94141e10 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f897fa640 10 monclient: dump: 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 10 monclient: _renew_subs 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9409a450 con 0x7f1f94141e10 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f8b7fe640 4 set_mon_vals no callback set 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 10 monclient: _renew_subs 2024-09-15T03:31:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.459+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9409ad60 con 0x7f1f94141e10 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1f8c01d440 con 0x7f1f94141e10 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f9b59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f9b59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f9b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f940636e0 con 0x7f1f94141e10 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f897fa640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f78038740 0x7f1f7803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f1f8c04b860 con 0x7f1f94141e10 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f99339640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f78038740 0x7f1f7803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.460+0000 7f1f99339640 10 monclient: get_auth_request con 0x7f1f78038740 auth_method 0 2024-09-15T03:31:59.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.463+0000 7f1f99339640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f78038740 0x7f1f7803abf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1f9002f000 tx=0x7f1f90030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.465+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1f8c01d6f0 con 0x7f1f94141e10 2024-09-15T03:31:59.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.465+0000 7f1f897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.465+0000 7f1f897fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:59.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.619+0000 7f1f9b59d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:59.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.619+0000 7f1f9b59d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:59.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.619+0000 7f1f9b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.619+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1f94064a70 con 0x7f1f94141e10 2024-09-15T03:31:59.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.620+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f1f8c0112e0 con 0x7f1f94141e10 2024-09-15T03:31:59.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.620+0000 7f1f897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:31:59.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.620+0000 7f1f897fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:31:59.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.713+0000 7f1f9b59d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:31:59.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.713+0000 7f1f9b59d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:59.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.713+0000 7f1f9b59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.713+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1f94072640 con 0x7f1f94141e10 2024-09-15T03:31:59.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.714+0000 7f1f897fa640 1 -- 172.21.15.181:0/4143683719 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f1f8c0114c0 con 0x7f1f94141e10 2024-09-15T03:31:59.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.714+0000 7f1f897fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:31:59.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.714+0000 7f1f897fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:31:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f78038740 msgr2=0x7f1f7803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f78038740 0x7f1f7803abf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1f9002f000 tx=0x7f1f90030000 comp rx=0 tx=0).stop 2024-09-15T03:31:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 10 monclient: shutdown 2024-09-15T03:31:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 msgr2=0x7f1f94098e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 0x7f1f94098e90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c001810 tx=0x7f1f8c001b50 comp rx=0 tx=0).stop 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 shutdown_connections 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f78038740 0x7f1f7803abf0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94141e10 0x7f1f94098e90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9412ff10 0x7f1f94098950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 --2- 172.21.15.181:0/4143683719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f80003b20 0x7f1f941418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 >> 172.21.15.181:0/4143683719 conn(0x7f1f940a5100 msgr2=0x7f1f940a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:59.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 shutdown_connections 2024-09-15T03:31:59.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.716+0000 7f1f9b59d640 1 -- 172.21.15.181:0/4143683719 wait complete. 2024-09-15T03:31:59.728 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush rm foo 2024-09-15T03:31:59.728 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:31:59.728 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm foo 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.798+0000 7f8fda35d640 10 monclient: get_monmap_and_config 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.798+0000 7f8fda35d640 10 monclient: build_initial_monmap 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.798+0000 7f8fda35d640 10 monclient: monmap: 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:31:59.799066+0000 2024-09-15T03:31:59.799 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:31:59.799066+0000 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:31:59.800 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.798+0000 7f8fda35d640 1 Processor -- start 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 1 -- start start 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient: init 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 0x7f8fd4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8fd41321a0 0x7f8fd4068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd41321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fd409f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd4067420 con 0x7f8fd409f020 2024-09-15T03:31:59.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40a0480 con 0x7f8fd409ec50 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40a0200 con 0x7f8fd41321a0 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fda35d640 10 monclient(hunting): authenticate will time out at 924.280546s 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8fd41321a0 0x7f8fd4068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8fd41321a0 0x7f8fd4068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39562/0 (socket says 172.21.15.181:39562) 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 1 -- 172.21.15.181:0/1447320894 learned_addr learned my addr 172.21.15.181:0/1447320894 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): get_auth_request con 0x7f8fd41321a0 auth_method 0 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 0x7f8fd4131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd3fff640 10 monclient(hunting): get_auth_request con 0x7f8fd409ec50 auth_method 0 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.799+0000 7f8fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.800+0000 7f8fd3fff640 1 -- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 msgr2=0x7f8fd4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.800+0000 7f8fd3fff640 1 -- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 msgr2=0x7f8fd4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.800+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 0x7f8fd4131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.800+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 0x7f8fd4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.801+0000 7f8fd88fa640 1 -- 172.21.15.181:0/1447320894 >> v1:172.21.15.181:6789/0 conn(0x7f8fd409f020 legacy=0x7f8fd4069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.801+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_done global_id 44620 payload 274 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 1 -- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 msgr2=0x7f8fd4131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 1 --2- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 0x7f8fd4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 1 -- 172.21.15.181:0/1447320894 >> v1:172.21.15.181:6789/0 conn(0x7f8fd409f020 legacy=0x7f8fd4069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 1 monclient: found mon.noname-c 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.802+0000 7f8fd37fe640 1 -- 172.21.15.181:0/1447320894 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd409fd80 con 0x7f8fd41321a0 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd37fe640 1 -- 172.21.15.181:0/1447320894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 msgr2=0x7f8fd4068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd37fe640 1 --2- 172.21.15.181:0/1447320894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd4068d00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8fd41325a0 tx=0x7f8fc80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46ac634ba4a5a56a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd2ffd640 1 -- 172.21.15.181:0/1447320894 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8032440 con 0x7f8fd41321a0 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd2ffd640 10 monclient: dump: 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fda35d640 5 monclient: authenticate success, global_id 44620 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.804+0000 7f8fda35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fd2ffd640 1 -- 172.21.15.181:0/1447320894 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc80325a0 con 0x7f8fd41321a0 2024-09-15T03:31:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fd2ffd640 1 -- 172.21.15.181:0/1447320894 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8032920 con 0x7f8fd41321a0 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 10 monclient: get_monmap_and_config success 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 4 set_mon_vals no callback set 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fd2ffd640 10 monclient: dump: 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.814 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 10 monclient: shutdown 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 -- 172.21.15.181:0/1447320894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 msgr2=0x7f8fd4068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 --2- 172.21.15.181:0/1447320894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd4068d00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8fd41325a0 tx=0x7f8fc80091d0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 -- 172.21.15.181:0/1447320894 shutdown_connections 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 --2- 172.21.15.181:0/1447320894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd4068d00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 --2- 172.21.15.181:0/1447320894 >> v2:172.21.15.153:3300/0 conn(0x7f8fd409ec50 0x7f8fd4131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 -- 172.21.15.181:0/1447320894 >> 172.21.15.181:0/1447320894 conn(0x7f8fd40a5cf0 msgr2=0x7f8fd40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 -- 172.21.15.181:0/1447320894 shutdown_connections 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.805+0000 7f8fda35d640 1 -- 172.21.15.181:0/1447320894 wait complete. 2024-09-15T03:31:59.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 10 monclient: build_initial_monmap 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 10 monclient: monmap: 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 1 Processor -- start 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 1 -- start start 2024-09-15T03:31:59.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 10 monclient: init 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.806+0000 7f8fda35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 0x7f8fd411e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd409f020 0x7f8fd411e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd411ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd41321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4130c70 con 0x7f8fd409ec50 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd409c020 con 0x7f8fd41321a0 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:31:59.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd409e090 con 0x7f8fd409f020 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fda35d640 10 monclient(hunting): authenticate will time out at 924.288088s 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd411ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd411ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39566/0 (socket says 172.21.15.181:39566) 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 -- 172.21.15.181:0/1829653974 learned_addr learned my addr 172.21.15.181:0/1829653974 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): get_auth_request con 0x7f8fd41321a0 auth_method 0 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:31:59.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 0x7f8fd411e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 10 monclient(hunting): get_auth_request con 0x7f8fd409ec50 auth_method 0 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd37fe640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd409f020 0x7f8fd411e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd37fe640 10 monclient(hunting): get_auth_request con 0x7f8fd409f020 auth_method 0 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:31:59.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient(hunting): handle_auth_done global_id 44623 payload 824 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 1 -- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 msgr2=0x7f8fd411e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 1 -- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 msgr2=0x7f8fd411e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 0x7f8fd411e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 -- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd409f020 msgr2=0x7f8fd411e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd409f020 0x7f8fd411e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 -- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 msgr2=0x7f8fd411e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 0x7f8fd411e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd409ec50 0x7f8fd411e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient: _finish_hunting 0 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 monclient: found mon.noname-b 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:31:59.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd88fa640 1 -- 172.21.15.181:0/1829653974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc8009450 con 0x7f8fd41321a0 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.807+0000 7f8fd37fe640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd409f020 0x7f8fd411e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fd88fa640 1 -- 172.21.15.181:0/1829653974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 msgr2=0x7f8fd411ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fd88fa640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fd41321a0 0x7f8fd411ec30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c840 tx=0x7f8fcc0018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc001950 con 0x7f8fd41321a0 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: dump: 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.821 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: _finish_auth 0 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc00a640 con 0x7f8fd41321a0 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc00aa00 con 0x7f8fd41321a0 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 5 monclient: authenticate success, global_id 44623 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:31:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fbbfff640 10 monclient: dump: 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 10 monclient: _renew_subs 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 1 -- 172.21.15.181:0/1829653974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd411fa70 con 0x7f8fd41321a0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fd27fc640 4 set_mon_vals no callback set 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 10 monclient: _renew_subs 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.809+0000 7f8fda35d640 1 -- 172.21.15.181:0/1829653974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd41315c0 con 0x7f8fd41321a0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.810+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8fcc0070f0 con 0x7f8fd41321a0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.810+0000 7f8fda35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.810+0000 7f8fda35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.810+0000 7f8fda35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.810+0000 7f8fda35d640 1 -- 172.21.15.181:0/1829653974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4121a20 con 0x7f8fd41321a0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.810+0000 7f8fbbfff640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fc0038740 0x7f8fc003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.811+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f8fcc04bdd0 con 0x7f8fd41321a0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.811+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fc0038740 0x7f8fc003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:31:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.811+0000 7f8fd3fff640 10 monclient: get_auth_request con 0x7f8fc0038740 auth_method 0 2024-09-15T03:31:59.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.813+0000 7f8fd3fff640 1 --2- 172.21.15.181:0/1829653974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8fc0038740 0x7f8fc003abf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4006fd0 tx=0x7f8fc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:31:59.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.815+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8fcc01b400 con 0x7f8fd41321a0 2024-09-15T03:31:59.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.815+0000 7f8fbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:31:59.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.815+0000 7f8fbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:31:59.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.974+0000 7f8fda35d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.974+0000 7f8fda35d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.974+0000 7f8fda35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.974+0000 7f8fda35d640 1 -- 172.21.15.181:0/1829653974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f8fd40008d0 con 0x7f8fd41321a0 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.975+0000 7f8fbbfff640 1 -- 172.21.15.181:0/1829653974 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v74) v1 ==== 99+0+0 (secure 0 0 0) 0x7f8fcc010af0 con 0x7f8fd41321a0 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.975+0000 7f8fbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:31:59.975+0000 7f8fbbfff640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T03:31:59.976 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T03:31:59.979 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:31:59.979 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush rm bar 2024-09-15T03:31:59.979 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:31:59.980 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm bar 2024-09-15T03:32:00.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.051+0000 7f71e620f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:00.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.051+0000 7f71e620f640 10 monclient: build_initial_monmap 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.051+0000 7f71e620f640 10 monclient: monmap: 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:00.052474+0000 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:00.052474+0000 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:00.053 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 1 Processor -- start 2024-09-15T03:32:00.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 1 -- start start 2024-09-15T03:32:00.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient: init 2024-09-15T03:32:00.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:00.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f71e0069a30 0x7f71e012f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient(hunting): picked mon.noname-c con 0x7f71e0069a30 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71e009baa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71e009bea0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e0069e00 con 0x7f71e009baa0 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e0071740 con 0x7f71e009bea0 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e009d100 con 0x7f71e0069a30 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.052+0000 7f71e620f640 10 monclient(hunting): authenticate will time out at 924.533939s 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f71e0069a30 0x7f71e012f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f71e0069a30 0x7f71e012f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39570/0 (socket says 172.21.15.181:39570) 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 1 -- 172.21.15.181:0/1701325629 learned_addr learned my addr 172.21.15.181:0/1701325629 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:00.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 10 monclient(hunting): get_auth_request con 0x7f71e0069a30 auth_method 0 2024-09-15T03:32:00.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.053+0000 7f71df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71dffff640 1 -- 172.21.15.181:0/1701325629 >> v1:172.21.15.181:6789/0 conn(0x7f71e009bea0 legacy=0x7f71e0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 10 monclient(hunting): handle_auth_done global_id 44626 payload 274 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 1 -- 172.21.15.181:0/1701325629 >> v1:172.21.15.181:6789/0 conn(0x7f71e009bea0 legacy=0x7f71e0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 1 -- 172.21.15.181:0/1701325629 >> v1:172.21.15.153:6789/0 conn(0x7f71e009baa0 legacy=0x7f71e0070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 1 monclient: found mon.noname-c 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.055+0000 7f71df7fe640 1 -- 172.21.15.181:0/1701325629 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e009e270 con 0x7f71e0069a30 2024-09-15T03:32:00.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71df7fe640 1 -- 172.21.15.181:0/1701325629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e0069a30 msgr2=0x7f71e012f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:00.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71df7fe640 1 --2- 172.21.15.181:0/1701325629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e0069a30 0x7f71e012f840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f71d0009990 tx=0x7f71d00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cadd82dafe0144e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:00.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 1 -- 172.21.15.181:0/1701325629 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d002ae90 con 0x7f71e0069a30 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: dump: 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.061 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 5 monclient: authenticate success, global_id 44626 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 1 -- 172.21.15.181:0/1701325629 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d002f4b0 con 0x7f71e0069a30 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 4 set_mon_vals no callback set 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 1 -- 172.21.15.181:0/1701325629 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d002f850 con 0x7f71e0069a30 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71de7fc640 10 monclient: dump: 2024-09-15T03:32:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 10 monclient: shutdown 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 1 -- 172.21.15.181:0/1701325629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e0069a30 msgr2=0x7f71e012f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 1 --2- 172.21.15.181:0/1701325629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e0069a30 0x7f71e012f840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f71d0009990 tx=0x7f71d00091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 1 -- 172.21.15.181:0/1701325629 shutdown_connections 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 1 --2- 172.21.15.181:0/1701325629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e0069a30 0x7f71e012f840 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 1 -- 172.21.15.181:0/1701325629 >> 172.21.15.181:0/1701325629 conn(0x7f71e009e9f0 msgr2=0x7f71e009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.058+0000 7f71e620f640 1 -- 172.21.15.181:0/1701325629 shutdown_connections 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.059+0000 7f71e620f640 1 -- 172.21.15.181:0/1701325629 wait complete. 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.059+0000 7f71e620f640 10 monclient: build_initial_monmap 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.059+0000 7f71e620f640 10 monclient: monmap: 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:00.064 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:00.065 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:00.065 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.059+0000 7f71e620f640 1 Processor -- start 2024-09-15T03:32:00.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.059+0000 7f71e620f640 1 -- start start 2024-09-15T03:32:00.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.059+0000 7f71e620f640 10 monclient: init 2024-09-15T03:32:00.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71e0069a30 0x7f71e011bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): picked mon.noname-a con 0x7f71e0069a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71e009baa0 0x7f71e011c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): picked mon.noname-c con 0x7f71e009baa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e009bea0 0x7f71e011c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): picked mon.noname-b con 0x7f71e009bea0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00a1ad0 con 0x7f71e0069a30 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e011f430 con 0x7f71e009bea0 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e011f150 con 0x7f71e009baa0 2024-09-15T03:32:00.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71e620f640 10 monclient(hunting): authenticate will time out at 924.541358s 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e009bea0 0x7f71e011c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e009bea0 0x7f71e011c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39572/0 (socket says 172.21.15.181:39572) 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 -- 172.21.15.181:0/4237369126 learned_addr learned my addr 172.21.15.181:0/4237369126 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): get_auth_request con 0x7f71e009bea0 auth_method 0 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71df7fe640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71e0069a30 0x7f71e011bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient(hunting): handle_auth_done global_id 44632 payload 824 2024-09-15T03:32:00.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71deffd640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71e009baa0 0x7f71e011c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 -- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71e009baa0 msgr2=0x7f71e011c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71e009baa0 0x7f71e011c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 -- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71e0069a30 msgr2=0x7f71e011bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71e0069a30 0x7f71e011bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 monclient: found mon.noname-b 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71dffff640 1 -- 172.21.15.181:0/4237369126 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d0009450 con 0x7f71e009bea0 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71df7fe640 10 monclient: get_auth_request con 0x7f71e0069a30 auth_method 0 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71deffd640 10 monclient: get_auth_request con 0x7f71e009baa0 auth_method 0 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71df7fe640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71e0069a30 0x7f71e011bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.060+0000 7f71deffd640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71e009baa0 0x7f71e011c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.062+0000 7f71dffff640 1 -- 172.21.15.181:0/4237369126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e009bea0 msgr2=0x7f71e011c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.062+0000 7f71dffff640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71e009bea0 0x7f71e011c690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c840 tx=0x7f71d80018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8001950 con 0x7f71e009bea0 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: dump: 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.072 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d800a640 con 0x7f71e009bea0 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 5 monclient: authenticate success, global_id 44632 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 10 monclient: _renew_subs 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 1 -- 172.21.15.181:0/4237369126 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e012e6d0 con 0x7f71e009bea0 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d800aa00 con 0x7f71e009bea0 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 10 monclient: dump: 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 10 monclient: _renew_subs 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71e620f640 1 -- 172.21.15.181:0/4237369126 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e011fe10 con 0x7f71e009bea0 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71ddffb640 4 set_mon_vals no callback set 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.063+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f71d80070f0 con 0x7f71e009bea0 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71e620f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71e620f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71e620f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71e620f640 1 -- 172.21.15.181:0/4237369126 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e00008d0 con 0x7f71e009bea0 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71c77fe640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71cc038740 0x7f71cc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f71d804beb0 con 0x7f71e009bea0 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71df7fe640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71cc038740 0x7f71cc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.064+0000 7f71df7fe640 10 monclient: get_auth_request con 0x7f71cc038740 auth_method 0 2024-09-15T03:32:00.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.066+0000 7f71df7fe640 1 --2- 172.21.15.181:0/4237369126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71cc038740 0x7f71cc03abf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f71e009d070 tx=0x7f71d0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:00.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.068+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f71d801a590 con 0x7f71e009bea0 2024-09-15T03:32:00.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.068+0000 7f71c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:00.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.068+0000 7f71c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.233+0000 7f71e620f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.233+0000 7f71e620f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.233+0000 7f71e620f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.233+0000 7f71e620f640 1 -- 172.21.15.181:0/4237369126 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f71e009dc80 con 0x7f71e009bea0 2024-09-15T03:32:00.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.238+0000 7f71c77fe640 1 -- 172.21.15.181:0/4237369126 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v74) v1 ==== 99+0+0 (secure 0 0 0) 0x7f71d8010af0 con 0x7f71e009bea0 2024-09-15T03:32:00.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.238+0000 7f71c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:00.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.238+0000 7f71c77fe640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T03:32:00.239 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T03:32:00.243 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:32:00.243 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush unlink host1 bar 2024-09-15T03:32:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.316+0000 7f9ed9ec0640 10 monclient: get_monmap_and_config 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.316+0000 7f9ed9ec0640 10 monclient: build_initial_monmap 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.316+0000 7f9ed9ec0640 10 monclient: monmap: 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:00.317584+0000 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:00.317584+0000 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:00.319 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:00.319 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.317+0000 7f9ed9ec0640 1 Processor -- start 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.317+0000 7f9ed9ec0640 1 -- start start 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.317+0000 7f9ed9ec0640 10 monclient: init 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.317+0000 7f9ed9ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.317+0000 7f9ed9ec0640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ed4096a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f9ed4099f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ed409d4e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed409cb80 con 0x7f9ed409d4e0 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed4098c50 con 0x7f9ed4096a30 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed4098f30 con 0x7f9ed4099f00 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed9ec0640 10 monclient(hunting): authenticate will time out at 924.799077s 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed37fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f9ed4096a30 0x7f9ed4096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48464/0 (socket says 172.21.15.181:48464) 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed37fe640 1 -- 172.21.15.181:0/2724815249 learned_addr learned my addr 172.21.15.181:0/2724815249 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 1 --2- 172.21.15.181:0/2724815249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): get_auth_request con 0x7f9ed4099f00 auth_method 0 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_done global_id 5147 payload 274 2024-09-15T03:32:00.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 1 -- 172.21.15.181:0/2724815249 >> v1:172.21.15.181:6789/0 conn(0x7f9ed4096a30 legacy=0x7f9ed4096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:00.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 1 -- 172.21.15.181:0/2724815249 >> v1:172.21.15.153:6789/0 conn(0x7f9ed409d4e0 legacy=0x7f9ed41360e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:00.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:00.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:32:00.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:00.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:00.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.318+0000 7f9ed2ffd640 1 -- 172.21.15.181:0/2724815249 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed40a1ba0 con 0x7f9ed4099f00 2024-09-15T03:32:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed2ffd640 1 -- 172.21.15.181:0/2724815249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 msgr2=0x7f9ed4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed2ffd640 1 --2- 172.21.15.181:0/2724815249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed4133b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009880 tx=0x7f9ec80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec4644bc419fe4f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 1 -- 172.21.15.181:0/2724815249 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8032440 con 0x7f9ed4099f00 2024-09-15T03:32:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 10 monclient: dump: 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 1 -- 172.21.15.181:0/2724815249 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec80325a0 con 0x7f9ed4099f00 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.321+0000 7f9ed27fc640 1 -- 172.21.15.181:0/2724815249 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8032940 con 0x7f9ed4099f00 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed27fc640 10 monclient: dump: 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 5 monclient: authenticate success, global_id 5147 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 4 set_mon_vals no callback set 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 10 monclient: shutdown 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/2724815249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 msgr2=0x7f9ed4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/2724815249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed4133b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009880 tx=0x7f9ec80090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/2724815249 shutdown_connections 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/2724815249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed4133b90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/2724815249 >> 172.21.15.181:0/2724815249 conn(0x7f9ed40711a0 msgr2=0x7f9ed40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/2724815249 shutdown_connections 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/2724815249 wait complete. 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 10 monclient: build_initial_monmap 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.322+0000 7f9ed9ec0640 10 monclient: monmap: 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:00.335 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 Processor -- start 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 -- start start 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient: init 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed4096a30 0x7f9ed409b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f9ed4096a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed412ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f9ed4099f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 0x7f9ed412ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f9ed409d4e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed406db00 con 0x7f9ed4096a30 2024-09-15T03:32:00.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed411d120 con 0x7f9ed409d4e0 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed413f950 con 0x7f9ed4099f00 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.323+0000 7f9ed9ec0640 10 monclient(hunting): authenticate will time out at 924.804932s 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 0x7f9ed412ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 0x7f9ed412ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39576/0 (socket says 172.21.15.181:39576) 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 -- 172.21.15.181:0/449985166 learned_addr learned my addr 172.21.15.181:0/449985166 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): get_auth_request con 0x7f9ed409d4e0 auth_method 0 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed4096a30 0x7f9ed409b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed412ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): get_auth_request con 0x7f9ed4099f00 auth_method 0 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): get_auth_request con 0x7f9ed4096a30 auth_method 0 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient(hunting): handle_auth_done global_id 44635 payload 824 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 -- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 msgr2=0x7f9ed412ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed412ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 -- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed4096a30 msgr2=0x7f9ed409b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed4096a30 0x7f9ed409b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 monclient: found mon.noname-b 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.324+0000 7f9ed3fff640 1 -- 172.21.15.181:0/449985166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec8009390 con 0x7f9ed409d4e0 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed3fff640 1 -- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 msgr2=0x7f9ed412ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed3fff640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 0x7f9ed412ef50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c840 tx=0x7f9ecc002620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc00ce30 con 0x7f9ed409d4e0 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: dump: 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.340 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.341 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.341 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 5 monclient: authenticate success, global_id 44635 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 10 monclient: _renew_subs 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed409c570 con 0x7f9ed409d4e0 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc00aa60 con 0x7f9ed409d4e0 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc00ae20 con 0x7f9ed409d4e0 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ebb7fe640 10 monclient: dump: 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:00.346 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 10 monclient: _renew_subs 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed412f490 con 0x7f9ed409d4e0 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.326+0000 7f9ed1ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ecc012630 con 0x7f9ed409d4e0 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ed9ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ed9ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ed9ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed411d290 con 0x7f9ed409d4e0 2024-09-15T03:32:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ebb7fe640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ec0038740 0x7f9ec003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f9ecc04bd60 con 0x7f9ed409d4e0 2024-09-15T03:32:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ed37fe640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ec0038740 0x7f9ec003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.327+0000 7f9ed37fe640 10 monclient: get_auth_request con 0x7f9ec0038740 auth_method 0 2024-09-15T03:32:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.330+0000 7f9ed37fe640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ec0038740 0x7f9ec003abf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4007fd0 tx=0x7f9ec4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.332+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ecc0188c0 con 0x7f9ed409d4e0 2024-09-15T03:32:00.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.332+0000 7f9ebb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:00.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.332+0000 7f9ebb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.492+0000 7f9ed9ec0640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T03:32:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.492+0000 7f9ed9ec0640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T03:32:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.492+0000 7f9ed9ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:00.492+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f9ed40008d0 con 0x7f9ed409d4e0 2024-09-15T03:32:00.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.987+0000 7f646d8c9640 10 monclient: get_monmap_and_config 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.987+0000 7f646d8c9640 10 monclient: build_initial_monmap 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.987+0000 7f646d8c9640 10 monclient: monmap: 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:00.988460+0000 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:00.988460+0000 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:00.990 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.988+0000 7f646d8c9640 1 Processor -- start 2024-09-15T03:32:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.988+0000 7f646d8c9640 1 -- start start 2024-09-15T03:32:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.988+0000 7f646d8c9640 10 monclient: init 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.988+0000 7f646d8c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.988+0000 7f646d8c9640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.988+0000 7f646d8c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6468131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f646809e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 0x7f6468139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f6468137bc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646809faf0 con 0x7f646809e910 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646809fd30 con 0x7f6468131ab0 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468132590 con 0x7f6468137bc0 2024-09-15T03:32:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f646d8c9640 10 monclient(hunting): authenticate will time out at 918.628119s 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f645ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f646809e910 0x7f646809ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:58646/0 (socket says 172.21.15.153:58646) 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f645ffff640 1 -- 172.21.15.153:0/4010566779 learned_addr learned my addr 172.21.15.153:0/4010566779 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 1 --2- 172.21.15.153:0/4010566779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 0x7f6468139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): get_auth_request con 0x7f6468137bc0 auth_method 0 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:00.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient(hunting): handle_auth_done global_id 5153 payload 274 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 1 -- 172.21.15.153:0/4010566779 >> v1:172.21.15.181:6789/0 conn(0x7f6468131ab0 legacy=0x7f6468131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 1 -- 172.21.15.153:0/4010566779 >> v1:172.21.15.153:6789/0 conn(0x7f646809e910 legacy=0x7f646809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 1 monclient: found mon.noname-b 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.989+0000 7f64677fe640 1 -- 172.21.15.153:0/4010566779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680a1110 con 0x7f6468137bc0 2024-09-15T03:32:01.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64677fe640 1 -- 172.21.15.153:0/4010566779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 msgr2=0x7f6468139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64677fe640 1 --2- 172.21.15.153:0/4010566779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 0x7f6468139fa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6460009e90 tx=0x7f6460009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf1dad6f6e6fe1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 1 -- 172.21.15.153:0/4010566779 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460007840 con 0x7f6468137bc0 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: dump: 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 1 -- 172.21.15.153:0/4010566779 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646002d050 con 0x7f6468137bc0 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f646d8c9640 5 monclient: authenticate success, global_id 5153 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f646d8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f646d8c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 1 -- 172.21.15.153:0/4010566779 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460007bd0 con 0x7f6468137bc0 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f646d8c9640 4 set_mon_vals no callback set 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.992+0000 7f64667fc640 10 monclient: dump: 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 10 monclient: shutdown 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 -- 172.21.15.153:0/4010566779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 msgr2=0x7f6468139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 --2- 172.21.15.153:0/4010566779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 0x7f6468139fa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6460009e90 tx=0x7f6460009780 comp rx=0 tx=0).stop 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 -- 172.21.15.153:0/4010566779 shutdown_connections 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 --2- 172.21.15.153:0/4010566779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468137bc0 0x7f6468139fa0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 -- 172.21.15.153:0/4010566779 >> 172.21.15.153:0/4010566779 conn(0x7f6468067d30 msgr2=0x7f64680a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 -- 172.21.15.153:0/4010566779 shutdown_connections 2024-09-15T03:32:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 1 -- 172.21.15.153:0/4010566779 wait complete. 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.993+0000 7f646d8c9640 10 monclient: monmap: 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 Processor -- start 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 -- start start 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient: init 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 0x7f6468137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f646809e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 0x7f646812e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f6468131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6468137bc0 0x7f646812e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f6468137bc0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6468132770 con 0x7f646809e910 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646812f920 con 0x7f6468137bc0 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64680a54d0 con 0x7f6468131ab0 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f646d8c9640 10 monclient(hunting): authenticate will time out at 918.633572s 2024-09-15T03:32:01.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 0x7f646812e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 0x7f646812e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58164/0 (socket says 172.21.15.153:58164) 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 1 -- 172.21.15.153:0/2450936977 learned_addr learned my addr 172.21.15.153:0/2450936977 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): get_auth_request con 0x7f6468131ab0 auth_method 0 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 0x7f6468137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): get_auth_request con 0x7f646809e910 auth_method 0 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f645ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6468137bc0 0x7f646812e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): get_auth_request con 0x7f6468137bc0 auth_method 0 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f64677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 10 monclient(hunting): handle_auth_done global_id 64101 payload 824 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.994+0000 7f6466ffd640 1 -- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 msgr2=0x7f646812e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 0x7f646812e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 1 -- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6468137bc0 msgr2=0x7f646812e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6468137bc0 0x7f646812e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 1 monclient: found mon.noname-a 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f6466ffd640 1 -- 172.21.15.153:0/2450936977 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460009a20 con 0x7f646809e910 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.995+0000 7f645ffff640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 0x7f646812e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f6466ffd640 1 -- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 msgr2=0x7f6468137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f6466ffd640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 0x7f6468137090 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f645800c4e0 tx=0x7f6458007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64580117a0 con 0x7f646809e910 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: dump: 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.012 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6458011920 con 0x7f646809e910 2024-09-15T03:32:01.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 5 monclient: authenticate success, global_id 64101 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 10 monclient: _renew_subs 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6468131300 con 0x7f646809e910 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458011ce0 con 0x7f646809e910 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f645effd640 10 monclient: dump: 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f6465ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 10 monclient: _renew_subs 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.997+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646812fc80 con 0x7f646809e910 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6458012630 con 0x7f646809e910 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f645effd640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6448038740 0x7f644803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f645804bc60 con 0x7f646809e910 2024-09-15T03:32:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f646d8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f646d8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f646d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6468132910 con 0x7f646809e910 2024-09-15T03:32:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f645ffff640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6448038740 0x7f644803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:00.998+0000 7f645ffff640 10 monclient: get_auth_request con 0x7f6448038740 auth_method 0 2024-09-15T03:32:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.001+0000 7f645ffff640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6448038740 0x7f644803abf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6450006fd0 tx=0x7f6450004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.003+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f645801c460 con 0x7f646809e910 2024-09-15T03:32:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.003+0000 7f645effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.003+0000 7f645effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f646d8c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:01.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f646d8c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:01.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f646d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f646804bf60 con 0x7f646809e910 2024-09-15T03:32:01.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f645effd640 1 -- 172.21.15.153:0/2450936977 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f645802b080 con 0x7f646809e910 2024-09-15T03:32:01.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f645effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:01.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.220+0000 7f645effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:01.222 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6448038740 msgr2=0x7f644803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6448038740 0x7f644803abf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6450006fd0 tx=0x7f6450004960 comp rx=0 tx=0).stop 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 10 monclient: shutdown 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 msgr2=0x7f6468137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 0x7f6468137090 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f645800c4e0 tx=0x7f6458007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 shutdown_connections 2024-09-15T03:32:01.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6448038740 0x7f644803abf0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6468137bc0 0x7f646812e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6468131ab0 0x7f646812e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 --2- 172.21.15.153:0/2450936977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f646809e910 0x7f6468137090 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 >> 172.21.15.153:0/2450936977 conn(0x7f6468067d30 msgr2=0x7f64681390e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 shutdown_connections 2024-09-15T03:32:01.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.223+0000 7f646d8c9640 1 -- 172.21.15.153:0/2450936977 wait complete. 2024-09-15T03:32:01.234 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:32:01.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:32:01.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:32:01.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.286+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v75) v1 ==== 147+0+0 (secure 0 0 0) 0x7f9ecc0182c0 con 0x7f9ed409d4e0 2024-09-15T03:32:01.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.286+0000 7f9ebb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T03:32:01.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.286+0000 7f9ebb7fe640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-15T03:32:01.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.329+0000 7f5b2531d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:01.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.329+0000 7f5b2531d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.329+0000 7f5b2531d640 10 monclient: monmap: 2024-09-15T03:32:01.330 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:01.330405+0000 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:01.330405+0000 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:01.331 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 1 Processor -- start 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 1 -- start start 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 10 monclient: init 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5b200981c0 0x7f5b20098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b200981c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b20131da0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b20132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b20098ad0 con 0x7f5b20132170 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.330+0000 7f5b2531d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b20067f30 con 0x7f5b20131da0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b2531d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b2531d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b20131bd0 con 0x7f5b200981c0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b2531d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b2531d640 10 monclient(hunting): authenticate will time out at 918.970017s 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5b20132170 0x7f5b20067820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:58650/0 (socket says 172.21.15.153:58650) 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/94725131 learned_addr learned my addr 172.21.15.153:0/94725131 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 1 --2- 172.21.15.153:0/94725131 >> v2:172.21.15.181:3300/0 conn(0x7f5b200981c0 0x7f5b20098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): get_auth_request con 0x7f5b200981c0 auth_method 0 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.331+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/94725131 >> v1:172.21.15.153:6789/0 conn(0x7f5b20132170 legacy=0x7f5b20067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1e7fc640 1 -- 172.21.15.153:0/94725131 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517528449 0 0) 0x7f5b20098ad0 con 0x7f5b20132170 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b17fff640 1 -- 172.21.15.153:0/94725131 >> v1:172.21.15.181:6789/0 conn(0x7f5b20131da0 legacy=0x7f5b20067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1e7fc640 10 monclient(hunting): my global_id is 64104 2024-09-15T03:32:01.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1e7fc640 1 -- 172.21.15.153:0/94725131 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c0034e0 con 0x7f5b20132170 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 10 monclient(hunting): handle_auth_done global_id 44641 payload 274 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 1 -- 172.21.15.153:0/94725131 >> v1:172.21.15.181:6789/0 conn(0x7f5b20131da0 legacy=0x7f5b20067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 1 -- 172.21.15.153:0/94725131 >> v1:172.21.15.153:6789/0 conn(0x7f5b20132170 legacy=0x7f5b20067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 1 monclient: found mon.noname-c 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.333+0000 7f5b1effd640 1 -- 172.21.15.153:0/94725131 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2009ca60 con 0x7f5b200981c0 2024-09-15T03:32:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1effd640 1 -- 172.21.15.153:0/94725131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b200981c0 msgr2=0x7f5b20098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1effd640 1 --2- 172.21.15.153:0/94725131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b200981c0 0x7f5b20098590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009950 tx=0x7f5b100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b9088008d614839 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 1 -- 172.21.15.153:0/94725131 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1002ae10 con 0x7f5b200981c0 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: dump: 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 1 -- 172.21.15.153:0/94725131 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1002f4b0 con 0x7f5b200981c0 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b2531d640 5 monclient: authenticate success, global_id 44641 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b2531d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b2531d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b2531d640 4 set_mon_vals no callback set 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 1 -- 172.21.15.153:0/94725131 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1002f850 con 0x7f5b200981c0 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.336+0000 7f5b1e7fc640 10 monclient: dump: 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 10 monclient: shutdown 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 -- 172.21.15.153:0/94725131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b200981c0 msgr2=0x7f5b20098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 --2- 172.21.15.153:0/94725131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b200981c0 0x7f5b20098590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009950 tx=0x7f5b100090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 -- 172.21.15.153:0/94725131 shutdown_connections 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 --2- 172.21.15.153:0/94725131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b200981c0 0x7f5b20098590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 -- 172.21.15.153:0/94725131 >> 172.21.15.153:0/94725131 conn(0x7f5b200a1e30 msgr2=0x7f5b200a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 -- 172.21.15.153:0/94725131 shutdown_connections 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.337+0000 7f5b2531d640 1 -- 172.21.15.153:0/94725131 wait complete. 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient: monmap: 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 Processor -- start 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 -- start start 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient: init 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b200981c0 0x7f5b201410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b200981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b20131da0 0x7f5b20130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b20131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 0x7f5b2011a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b20132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200a0660 con 0x7f5b20132170 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2011dec0 con 0x7f5b20131da0 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2011c090 con 0x7f5b200981c0 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b2531d640 10 monclient(hunting): authenticate will time out at 918.977809s 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 0x7f5b2011a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b200981c0 0x7f5b201410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 0x7f5b2011a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43288/0 (socket says 172.21.15.153:43288) 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/3378353647 learned_addr learned my addr 172.21.15.153:0/3378353647 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 10 monclient(hunting): get_auth_request con 0x7f5b20132170 auth_method 0 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1effd640 10 monclient(hunting): get_auth_request con 0x7f5b200981c0 auth_method 0 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.338+0000 7f5b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b20131da0 0x7f5b20130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): get_auth_request con 0x7f5b20131da0 auth_method 0 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 10 monclient(hunting): handle_auth_done global_id 64107 payload 824 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b200981c0 msgr2=0x7f5b201410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b200981c0 0x7f5b201410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b20131da0 msgr2=0x7f5b20130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b20131da0 0x7f5b20130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.339+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10009390 con 0x7f5b20132170 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 msgr2=0x7f5b2011a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b1f7fe640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 0x7f5b2011a500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800cf60 tx=0x7f5b18001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180020e0 con 0x7f5b20132170 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: dump: 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.355 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1800aa60 con 0x7f5b20132170 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 5 monclient: authenticate success, global_id 64107 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 10 monclient: _renew_subs 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2011db50 con 0x7f5b20132170 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800ade0 con 0x7f5b20132170 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b16ffd640 10 monclient: dump: 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b1dffb640 4 set_mon_vals no callback set 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 10 monclient: _renew_subs 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.341+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b20141610 con 0x7f5b20132170 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b2531d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b2531d640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 0x7f5b2011aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b2531d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20065f90 con 0x7f5b20130f70 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5b18007300 con 0x7f5b20132170 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 0x7f5b2011aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient: get_auth_request con 0x7f5b20130f70 auth_method 0 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b16ffd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b00038740 0x7f5b0003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f5b1804c1d0 con 0x7f5b20132170 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 10 monclient(hunting): handle_auth_done global_id 64110 payload 824 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b1effd640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b2004bf60 con 0x7f5b20130f70 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b17fff640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b00038740 0x7f5b0003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.342+0000 7f5b17fff640 10 monclient: get_auth_request con 0x7f5b00038740 auth_method 0 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.344+0000 7f5b1effd640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 msgr2=0x7f5b2011aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.344+0000 7f5b1effd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 0x7f5b2011aa40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5b2006f820 tx=0x7f5b10032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10032940 con 0x7f5b20130f70 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 10 monclient: dump: 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.362 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5b10032aa0 con 0x7f5b20130f70 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 msgr2=0x7f5b2011aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 0x7f5b2011aa40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5b2006f820 tx=0x7f5b10032000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 msgr2=0x7f5b2011aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:01.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b16ffd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20130f70 0x7f5b2011aa40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.345+0000 7f5b17fff640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b00038740 0x7f5b0003abf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5b08007fd0 tx=0x7f5b08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.350+0000 7f5b2531d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b2531d640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 0x7f5b20099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b2531d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20131340 con 0x7f5b2011af80 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 0x7f5b20099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient: get_auth_request con 0x7f5b2011af80 auth_method 0 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 10 monclient(hunting): handle_auth_done global_id 64113 payload 824 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.351+0000 7f5b1effd640 1 -- 172.21.15.153:0/3378353647 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5b2011b380 con 0x7f5b2011af80 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b1effd640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 msgr2=0x7f5b20099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b1effd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 0x7f5b20099310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009130 tx=0x7f5b1003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1004c070 con 0x7f5b2011af80 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 10 monclient: dump: 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.365 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f5b10032730 con 0x7f5b2011af80 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 msgr2=0x7f5b20099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 0x7f5b20099310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009130 tx=0x7f5b1003e000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 msgr2=0x7f5b20099310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.353+0000 7f5b16ffd640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b2011af80 0x7f5b20099310 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009130 tx=0x7f5b1003e000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-15T03:32:01.367 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:01.368 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:01.368 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:01.368 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:01.368 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.368 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:32:01.368 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.369 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.370 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:01.371 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.372 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.374 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.375 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:01.376 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:01.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:01.379 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b00038740 msgr2=0x7f5b0003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b00038740 0x7f5b0003abf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5b08007fd0 tx=0x7f5b08009040 comp rx=0 tx=0).stop 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 10 monclient: shutdown 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 msgr2=0x7f5b2011a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b1f7fe640 1 -- 172.21.15.153:0/3378353647 reap_dead start 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 0x7f5b2011a500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800cf60 tx=0x7f5b18001c70 comp rx=0 tx=0).stop 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 shutdown_connections 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 --2- 172.21.15.153:0/3378353647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b20132170 0x7f5b2011a500 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 >> 172.21.15.153:0/3378353647 conn(0x7f5b200a1e30 msgr2=0x7f5b20096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 shutdown_connections 2024-09-15T03:32:01.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.354+0000 7f5b2531d640 1 -- 172.21.15.153:0/3378353647 wait complete. 2024-09-15T03:32:01.382 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:01.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:32:01.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:32:01.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.387+0000 7f9ed9ec0640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T03:32:01.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.387+0000 7f9ed9ec0640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T03:32:01.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.387+0000 7f9ed9ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.387+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f9ed4072640 con 0x7f9ed409d4e0 2024-09-15T03:32:01.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.389+0000 7f9ebb7fe640 1 -- 172.21.15.181:0/449985166 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9ecc0184a0 con 0x7f9ed409d4e0 2024-09-15T03:32:01.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.389+0000 7f9ebb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-15T03:32:01.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.389+0000 7f9ebb7fe640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-15T03:32:01.390 INFO:tasks.workunit.client.0.smithi181.stderr:item -11 does not appear in that position 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ec0038740 msgr2=0x7f9ec003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ec0038740 0x7f9ec003abf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4007fd0 tx=0x7f9ec4007f00 comp rx=0 tx=0).stop 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 10 monclient: shutdown 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 msgr2=0x7f9ed412ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 0x7f9ed412ef50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c840 tx=0x7f9ecc002620 comp rx=0 tx=0).stop 2024-09-15T03:32:01.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 shutdown_connections 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ec0038740 0x7f9ec003abf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed409d4e0 0x7f9ed412ef50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ed4099f00 0x7f9ed412ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 --2- 172.21.15.181:0/449985166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed4096a30 0x7f9ed409b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 >> 172.21.15.181:0/449985166 conn(0x7f9ed40711a0 msgr2=0x7f9ed4131a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 shutdown_connections 2024-09-15T03:32:01.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.391+0000 7f9ed9ec0640 1 -- 172.21.15.181:0/449985166 wait complete. 2024-09-15T03:32:01.401 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:01.401 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c host1 2024-09-15T03:32:01.402 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 1 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.472+0000 7fd68295d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.472+0000 7fd68295d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.472+0000 7fd68295d640 10 monclient: monmap: 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:01.472979+0000 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:01.472979+0000 2024-09-15T03:32:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:01.474 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.472+0000 7fd68295d640 1 Processor -- start 2024-09-15T03:32:01.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.472+0000 7fd68295d640 1 -- start start 2024-09-15T03:32:01.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.472+0000 7fd68295d640 10 monclient: init 2024-09-15T03:32:01.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd67c131b40 0x7fd67c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): picked mon.noname-a con 0x7fd67c131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd67c09f240 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c136e70 0x7fd67c139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): picked mon.noname-b con 0x7fd67c136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67c06a230 con 0x7fd67c09f240 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c0a0300 con 0x7fd67c131b40 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0a00c0 con 0x7fd67c136e70 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd68295d640 10 monclient(hunting): authenticate will time out at 925.954509s 2024-09-15T03:32:01.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd67c131b40 0x7fd67c09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd67c09f240 0x7fd67c131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:39302/0 (socket says 172.21.15.181:39302) 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67b7fe640 1 -- 172.21.15.181:0/1972953242 learned_addr learned my addr 172.21.15.181:0/1972953242 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c136e70 0x7fd67c139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): get_auth_request con 0x7fd67c136e70 auth_method 0 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): get_auth_request con 0x7fd67c131b40 auth_method 0 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd680efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.473+0000 7fd67bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 10 monclient(hunting): handle_auth_done global_id 64116 payload 274 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 1 -- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c136e70 msgr2=0x7fd67c139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 1 --2- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c136e70 0x7fd67c139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 1 -- 172.21.15.181:0/1972953242 >> v1:172.21.15.153:6789/0 conn(0x7fd67c09f240 legacy=0x7fd67c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 1 monclient: found mon.noname-a 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd67bfff640 1 -- 172.21.15.181:0/1972953242 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c09bee0 con 0x7fd67c131b40 2024-09-15T03:32:01.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.474+0000 7fd680efa640 1 --2- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c136e70 0x7fd67c139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.476+0000 7fd67bfff640 1 -- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c131b40 msgr2=0x7fd67c09ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.476+0000 7fd67bfff640 1 --2- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c131b40 0x7fd67c09ed00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009850 tx=0x7fd66c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9682d354742f1df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 1 -- 172.21.15.181:0/1972953242 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c02adf0 con 0x7fd67c131b40 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: dump: 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.491 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 1 -- 172.21.15.181:0/1972953242 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c032440 con 0x7fd67c131b40 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 1 -- 172.21.15.181:0/1972953242 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0327e0 con 0x7fd67c131b40 2024-09-15T03:32:01.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 5 monclient: authenticate success, global_id 64116 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 4 set_mon_vals no callback set 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd67affd640 10 monclient: dump: 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 10 monclient: shutdown 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 -- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c131b40 msgr2=0x7fd67c09ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 --2- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c131b40 0x7fd67c09ed00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009850 tx=0x7fd66c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 -- 172.21.15.181:0/1972953242 shutdown_connections 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 --2- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c136e70 0x7fd67c139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 --2- 172.21.15.181:0/1972953242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c131b40 0x7fd67c09ed00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 -- 172.21.15.181:0/1972953242 >> 172.21.15.181:0/1972953242 conn(0x7fd67c0a5100 msgr2=0x7fd67c0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 -- 172.21.15.181:0/1972953242 shutdown_connections 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.477+0000 7fd68295d640 1 -- 172.21.15.181:0/1972953242 wait complete. 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.478+0000 7fd68295d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.478+0000 7fd68295d640 10 monclient: monmap: 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.478+0000 7fd68295d640 1 Processor -- start 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.478+0000 7fd68295d640 1 -- start start 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.478+0000 7fd68295d640 10 monclient: init 2024-09-15T03:32:01.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 0x7fd67c141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): picked mon.noname-b con 0x7fd67c09f240 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c131b40 0x7fd67c141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): picked mon.noname-c con 0x7fd67c131b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c136e70 0x7fd67c0988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): picked mon.noname-a con 0x7fd67c136e70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0a0a30 con 0x7fd67c136e70 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c130f90 con 0x7fd67c09f240 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c11be20 con 0x7fd67c131b40 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd68295d640 10 monclient(hunting): authenticate will time out at 925.960164s 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c136e70 0x7fd67c0988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c131b40 0x7fd67c141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c136e70 0x7fd67c0988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:52616/0 (socket says 172.21.15.181:52616) 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): get_auth_request con 0x7fd67c131b40 auth_method 0 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 1 -- 172.21.15.181:0/4200586645 learned_addr learned my addr 172.21.15.181:0/4200586645 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 0x7fd67c141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): get_auth_request con 0x7fd67c136e70 auth_method 0 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): get_auth_request con 0x7fd67c09f240 auth_method 0 2024-09-15T03:32:01.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd680efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient(hunting): handle_auth_done global_id 44653 payload 824 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 -- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c131b40 msgr2=0x7fd67c141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c131b40 0x7fd67c141da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 -- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c136e70 msgr2=0x7fd67c0988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c136e70 0x7fd67c0988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 monclient: found mon.noname-b 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67bfff640 1 -- 172.21.15.181:0/4200586645 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674009a20 con 0x7fd67c09f240 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.479+0000 7fd67b7fe640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c131b40 0x7fd67c141da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd67bfff640 1 -- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 msgr2=0x7fd67c141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd67bfff640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 0x7fd67c141860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd66c034000 tx=0x7fd66c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c031a90 con 0x7fd67c09f240 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd663fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd663fff640 10 monclient: dump: 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd663fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd68295d640 5 monclient: authenticate success, global_id 44653 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd68295d640 10 monclient: _renew_subs 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd68295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.481+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c009390 con 0x7fd67c09f240 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 10 monclient: _renew_subs 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c09af00 con 0x7fd67c09f240 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c031c10 con 0x7fd67c09f240 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c03c400 con 0x7fd67c09f240 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 10 monclient: dump: 2024-09-15T03:32:01.502 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd67a7fc640 4 set_mon_vals no callback set 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd66c0443f0 con 0x7fd67c09f240 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd663fff640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd668038740 0x7fd66803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.482+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c0636e0 con 0x7fd67c09f240 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.483+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6173+0+0 (secure 0 0 0) 0x7fd66c06c520 con 0x7fd67c09f240 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.483+0000 7fd67b7fe640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd668038740 0x7fd66803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.483+0000 7fd67b7fe640 10 monclient: get_auth_request con 0x7fd668038740 auth_method 0 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.485+0000 7fd67b7fe640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd668038740 0x7fd66803abf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd670007fd0 tx=0x7fd670007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.487+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd66c03b3a0 con 0x7fd67c09f240 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.487+0000 7fd663fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.487+0000 7fd663fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.475+0000 7f81c115d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.475+0000 7f81c115d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.475+0000 7f81c115d640 10 monclient: monmap: 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:01.476667+0000 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:01.476667+0000 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 1 Processor -- start 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 1 -- start start 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 10 monclient: init 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f81bc1318f0 0x7f81bc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 10 monclient(hunting): picked mon.noname-c con 0x7f81bc1318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81bc09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.476+0000 7f81c115d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 0x7f81bc139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 10 monclient(hunting): picked mon.noname-b con 0x7f81bc136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81bc131cc0 con 0x7f81bc09ea50 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc09f530 con 0x7f81bc1318f0 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc09fbf0 con 0x7f81bc136c20 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81c115d640 10 monclient(hunting): authenticate will time out at 919.116033s 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 0x7f81bc139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 0x7f81bc139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58186/0 (socket says 172.21.15.153:58186) 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 -- 172.21.15.153:0/3359603254 learned_addr learned my addr 172.21.15.153:0/3359603254 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): get_auth_request con 0x7f81bc136c20 auth_method 0 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bad9d640 1 --2- 172.21.15.153:0/3359603254 >> v2:172.21.15.181:3300/0 conn(0x7f81bc1318f0 0x7f81bc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bad9d640 10 monclient(hunting): get_auth_request con 0x7f81bc1318f0 auth_method 0 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient(hunting): handle_auth_done global_id 5165 payload 274 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 -- 172.21.15.153:0/3359603254 >> v2:172.21.15.181:3300/0 conn(0x7f81bc1318f0 msgr2=0x7f81bc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 --2- 172.21.15.153:0/3359603254 >> v2:172.21.15.181:3300/0 conn(0x7f81bc1318f0 0x7f81bc131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 -- 172.21.15.153:0/3359603254 >> v1:172.21.15.181:6789/0 conn(0x7f81bc09ea50 legacy=0x7f81bc09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 monclient: found mon.noname-b 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.477+0000 7f81bb59e640 1 -- 172.21.15.153:0/3359603254 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc09fe30 con 0x7f81bc136c20 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81bb59e640 1 -- 172.21.15.153:0/3359603254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 msgr2=0x7f81bc139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81bb59e640 1 --2- 172.21.15.153:0/3359603254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 0x7f81bc139000 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009910 tx=0x7f81ac0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad68eaf2d097ad3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 1 -- 172.21.15.153:0/3359603254 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac02aea0 con 0x7f81bc136c20 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: dump: 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 5 monclient: authenticate success, global_id 5165 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 1 -- 172.21.15.153:0/3359603254 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac032440 con 0x7f81bc136c20 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 4 set_mon_vals no callback set 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 1 -- 172.21.15.153:0/3359603254 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0327e0 con 0x7f81bc136c20 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81b9d9b640 10 monclient: dump: 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.515 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 10 monclient: shutdown 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 -- 172.21.15.153:0/3359603254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 msgr2=0x7f81bc139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 --2- 172.21.15.153:0/3359603254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 0x7f81bc139000 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009910 tx=0x7f81ac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 -- 172.21.15.153:0/3359603254 shutdown_connections 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 --2- 172.21.15.153:0/3359603254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc136c20 0x7f81bc139000 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 --2- 172.21.15.153:0/3359603254 >> v2:172.21.15.181:3300/0 conn(0x7f81bc1318f0 0x7f81bc131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 -- 172.21.15.153:0/3359603254 >> 172.21.15.153:0/3359603254 conn(0x7f81bc067110 msgr2=0x7f81bc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 -- 172.21.15.153:0/3359603254 shutdown_connections 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.480+0000 7f81c115d640 1 -- 172.21.15.153:0/3359603254 wait complete. 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.481+0000 7f81c115d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.481+0000 7f81c115d640 10 monclient: monmap: 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.517 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.481+0000 7f81c115d640 1 Processor -- start 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.481+0000 7f81c115d640 1 -- start start 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.481+0000 7f81c115d640 10 monclient: init 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc09ea50 0x7f81bc11daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): picked mon.noname-a con 0x7f81bc09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc1318f0 0x7f81bc130760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): picked mon.noname-c con 0x7f81bc1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc136c20 0x7f81bc099590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): picked mon.noname-b con 0x7f81bc136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc09dd10 con 0x7f81bc09ea50 2024-09-15T03:32:01.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc09bf10 con 0x7f81bc136c20 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc09b830 con 0x7f81bc1318f0 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81c115d640 10 monclient(hunting): authenticate will time out at 919.121347s 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc09ea50 0x7f81bc11daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc1318f0 0x7f81bc130760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc09ea50 0x7f81bc11daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43318/0 (socket says 172.21.15.153:43318) 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 1 -- 172.21.15.153:0/4102205849 learned_addr learned my addr 172.21.15.153:0/4102205849 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): get_auth_request con 0x7f81bc1318f0 auth_method 0 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): get_auth_request con 0x7f81bc09ea50 auth_method 0 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bb59e640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc136c20 0x7f81bc099590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bb59e640 10 monclient(hunting): get_auth_request con 0x7f81bc136c20 auth_method 0 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient(hunting): handle_auth_done global_id 5171 payload 824 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc136c20 msgr2=0x7f81bc099590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc136c20 0x7f81bc099590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc09ea50 msgr2=0x7f81bc11daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc09ea50 0x7f81bc11daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 monclient: found mon.noname-c 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81ba59c640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac009450 con 0x7f81bc1318f0 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.482+0000 7f81bad9d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc09ea50 0x7f81bc11daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.484+0000 7f81ba59c640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc1318f0 msgr2=0x7f81bc130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.484+0000 7f81ba59c640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc1318f0 0x7f81bc130760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f81a400c510 tx=0x7f81a4007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40117a0 con 0x7f81bc1318f0 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: dump: 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.522 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 5 monclient: authenticate success, global_id 5171 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4011920 con 0x7f81bc1318f0 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 10 monclient: _renew_subs 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc09b070 con 0x7f81bc1318f0 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4011ce0 con 0x7f81bc1318f0 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:01.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b2ffd640 10 monclient: dump: 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81b959a640 4 set_mon_vals no callback set 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 10 monclient: _renew_subs 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:01.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc130ca0 con 0x7f81bc1318f0 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 0x7f81bc0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc11c9b0 con 0x7f81bc09a290 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.485+0000 7f81bad9d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 0x7f81bc0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient: get_auth_request con 0x7f81bc09a290 auth_method 0 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f81a4012630 con 0x7f81bc1318f0 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 10 monclient(hunting): handle_auth_done global_id 5174 payload 824 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bad9d640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81bc11cb20 con 0x7f81bc09a290 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81b2ffd640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f819c038740 0x7f819c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f81a404bc60 con 0x7f81bc1318f0 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bb59e640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f819c038740 0x7f819c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.486+0000 7f81bb59e640 10 monclient: get_auth_request con 0x7f819c038740 auth_method 0 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81bad9d640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 msgr2=0x7f81bc0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81bad9d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 0x7f81bc0298c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f81b40027b0 tx=0x7f81b4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4002e80 con 0x7f81bc09a290 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 10 monclient: dump: 2024-09-15T03:32:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81bb59e640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f819c038740 0x7f819c03abf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f81bc06f600 tx=0x7f81ac032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f81b4010ce0 con 0x7f81bc09a290 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 msgr2=0x7f81bc0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.488+0000 7f81b2ffd640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 0x7f81bc0298c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f81b40027b0 tx=0x7f81b4009920 comp rx=0 tx=0).stop 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.489+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 msgr2=0x7f81bc0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.489+0000 7f81b2ffd640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a290 0x7f81bc0298c0 secure :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f81b40027b0 tx=0x7f81b4009920 comp rx=0 tx=0).stop 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81c115d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81c115d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 0x7f81bc029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81c115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc065e00 con 0x7f81bc09a660 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 0x7f81bc029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient: get_auth_request con 0x7f81bc09a660 auth_method 0 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 10 monclient(hunting): handle_auth_done global_id 5177 payload 824 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.494+0000 7f81bad9d640 1 -- 172.21.15.153:0/4102205849 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81bc130f80 con 0x7f81bc09a660 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81bad9d640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 msgr2=0x7f81bc029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81bad9d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 0x7f81bc029e00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f81b401a000 tx=0x7f81b401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4015bb0 con 0x7f81bc09a660 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 10 monclient: dump: 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f81b4015d10 con 0x7f81bc09a660 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 msgr2=0x7f81bc029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 0x7f81bc029e00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f81b401a000 tx=0x7f81b401b000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 msgr2=0x7f81bc029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81b2ffd640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09a660 0x7f81bc029e00 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f81b401a000 tx=0x7f81b401b000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:01.532 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.533 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:01.535 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.537 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.538 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:01.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:01.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.543 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.545 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:01.545 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f819c038740 msgr2=0x7f819c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f819c038740 0x7f819c03abf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f81bc06f600 tx=0x7f81ac032000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 10 monclient: shutdown 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc1318f0 msgr2=0x7f81bc130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 --2- 172.21.15.153:0/4102205849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc1318f0 0x7f81bc130760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f81a400c510 tx=0x7f81a4007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81bb59e640 1 -- 172.21.15.153:0/4102205849 reap_dead start 2024-09-15T03:32:01.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81bb59e640 1 -- 172.21.15.153:0/4102205849 reap_dead start 2024-09-15T03:32:01.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 shutdown_connections 2024-09-15T03:32:01.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 >> 172.21.15.153:0/4102205849 conn(0x7f81bc067110 msgr2=0x7f81bc0a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 shutdown_connections 2024-09-15T03:32:01.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.497+0000 7f81c115d640 1 -- 172.21.15.153:0/4102205849 wait complete. 2024-09-15T03:32:01.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:01.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:32:01.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:32:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.639+0000 7ff3c419d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.639+0000 7ff3c419d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.639+0000 7ff3c419d640 10 monclient: monmap: 2024-09-15T03:32:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:01.640483+0000 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:01.640483+0000 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 1 Processor -- start 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 1 -- start start 2024-09-15T03:32:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 10 monclient: init 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.639+0000 7fd68295d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.639+0000 7fd68295d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.639+0000 7fd68295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.639+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd67c064a70 con 0x7fd67c09f240 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.640+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (secure 0 0 0) 0x7fd66c049070 con 0x7fd67c09f240 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.640+0000 7fd663fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.640+0000 7fd663fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 0x7ff3bc12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc0974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3bc09a8a0 0x7ff3bc09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc09a8a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.640+0000 7ff3c419d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3bc12dd40 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc0978c0 con 0x7ff3bc12dd40 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3bc09ca20 con 0x7ff3bc09a8a0 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc09b1f0 con 0x7ff3bc0974f0 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c419d640 10 monclient(hunting): authenticate will time out at 919.280038s 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3bc09a8a0 0x7ff3bc09acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 0x7ff3bc12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 0x7ff3bc12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58212/0 (socket says 172.21.15.153:58212) 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3bc09a8a0 0x7ff3bc09acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43328/0 (socket says 172.21.15.153:43328) 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 -- 172.21.15.153:0/1634337245 learned_addr learned my addr 172.21.15.153:0/1634337245 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): get_auth_request con 0x7ff3bc0974f0 auth_method 0 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): get_auth_request con 0x7ff3bc09a8a0 auth_method 0 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_done global_id 5180 payload 274 2024-09-15T03:32:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 -- 172.21.15.153:0/1634337245 >> v2:172.21.15.153:3300/0 conn(0x7ff3bc09a8a0 msgr2=0x7ff3bc09acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 --2- 172.21.15.153:0/1634337245 >> v2:172.21.15.153:3300/0 conn(0x7ff3bc09a8a0 0x7ff3bc09acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 -- 172.21.15.153:0/1634337245 >> v1:172.21.15.181:6789/0 conn(0x7ff3bc12dd40 legacy=0x7ff3bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 monclient: found mon.noname-b 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.641+0000 7ff3c1f39640 1 -- 172.21.15.153:0/1634337245 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc098ad0 con 0x7ff3bc0974f0 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c1f39640 1 -- 172.21.15.153:0/1634337245 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 msgr2=0x7ff3bc12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c1f39640 1 --2- 172.21.15.153:0/1634337245 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 0x7ff3bc12ab00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009910 tx=0x7ff3ac0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8d826566a1c12e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 1 -- 172.21.15.153:0/1634337245 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac02aea0 con 0x7ff3bc0974f0 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:01.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: dump: 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 1 -- 172.21.15.153:0/1634337245 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac032440 con 0x7ff3bc0974f0 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 5 monclient: authenticate success, global_id 5180 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 1 -- 172.21.15.153:0/1634337245 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac0327e0 con 0x7ff3bc0974f0 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 4 set_mon_vals no callback set 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c0f37640 10 monclient: dump: 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 10 monclient: shutdown 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 -- 172.21.15.153:0/1634337245 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 msgr2=0x7ff3bc12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 --2- 172.21.15.153:0/1634337245 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 0x7ff3bc12ab00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009910 tx=0x7ff3ac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 -- 172.21.15.153:0/1634337245 shutdown_connections 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 --2- 172.21.15.153:0/1634337245 >> v2:172.21.15.153:3300/0 conn(0x7ff3bc09a8a0 0x7ff3bc09acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 --2- 172.21.15.153:0/1634337245 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc0974f0 0x7ff3bc12ab00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 -- 172.21.15.153:0/1634337245 >> 172.21.15.153:0/1634337245 conn(0x7ff3bc0672b0 msgr2=0x7ff3bc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 -- 172.21.15.153:0/1634337245 shutdown_connections 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.644+0000 7ff3c419d640 1 -- 172.21.15.153:0/1634337245 wait complete. 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.645+0000 7ff3c419d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.645+0000 7ff3c419d640 10 monclient: monmap: 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.660 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.645+0000 7ff3c419d640 1 Processor -- start 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.645+0000 7ff3c419d640 1 -- start start 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.645+0000 7ff3c419d640 10 monclient: init 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 0x7ff3bc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc0974f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc09a8a0 0x7ff3bc06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc09a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc12dd40 0x7ff3bc06de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3bc12dd40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc12efd0 con 0x7ff3bc0974f0 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc098300 con 0x7ff3bc09a8a0 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc09c090 con 0x7ff3bc12dd40 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c419d640 10 monclient(hunting): authenticate will time out at 919.285193s 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 0x7ff3bc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc12dd40 0x7ff3bc06de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 0x7ff3bc06d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43342/0 (socket says 172.21.15.153:43342) 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 -- 172.21.15.153:0/535771673 learned_addr learned my addr 172.21.15.153:0/535771673 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): get_auth_request con 0x7ff3bc0974f0 auth_method 0 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): get_auth_request con 0x7ff3bc12dd40 auth_method 0 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc09a8a0 0x7ff3bc06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): get_auth_request con 0x7ff3bc09a8a0 auth_method 0 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c273a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient(hunting): handle_auth_done global_id 64128 payload 824 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc12dd40 msgr2=0x7ff3bc06de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3bc12dd40 0x7ff3bc06de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc09a8a0 msgr2=0x7ff3bc06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc09a8a0 0x7ff3bc06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 monclient: found mon.noname-a 2024-09-15T03:32:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.646+0000 7ff3c1f39640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac009450 con 0x7ff3bc0974f0 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3c1f39640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 msgr2=0x7ff3bc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3c1f39640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 0x7ff3bc06d3b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac035040 tx=0x7ff3ac032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac02aea0 con 0x7ff3bc0974f0 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 10 monclient: dump: 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac000d10 con 0x7ff3bc0974f0 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3c419d640 5 monclient: authenticate success, global_id 64128 2024-09-15T03:32:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac03d440 con 0x7ff3bc0974f0 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.648+0000 7ff3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3b1ffb640 10 monclient: dump: 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3c419d640 10 monclient: _renew_subs 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3c419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc06ed40 con 0x7ff3bc0974f0 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3b3fff640 4 set_mon_vals no callback set 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3c419d640 10 monclient: _renew_subs 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3c419d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc130470 con 0x7ff3bc0974f0 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.649+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff3ac039040 con 0x7ff3bc0974f0 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c419d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c419d640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 0x7ff3bc0696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc12f3f0 con 0x7ff3bc06e370 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3b1ffb640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff3a0038770 0x7ff3a003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 0x7ff3bc0696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6173+0+0 (secure 0 0 0) 0x7ff3ac06ba80 con 0x7ff3bc0974f0 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient: get_auth_request con 0x7ff3bc06e370 auth_method 0 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c273a640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff3a0038770 0x7ff3a003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c273a640 10 monclient: get_auth_request con 0x7ff3a0038770 auth_method 0 2024-09-15T03:32:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 10 monclient(hunting): handle_auth_done global_id 44662 payload 824 2024-09-15T03:32:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.650+0000 7ff3c1738640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3bc12f9e0 con 0x7ff3bc06e370 2024-09-15T03:32:01.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.652+0000 7ff3c1738640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 msgr2=0x7ff3bc0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.652+0000 7ff3c1738640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 0x7ff3bc0696a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8011bf0 tx=0x7ff3b80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.652+0000 7ff3c273a640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff3a0038770 0x7ff3a003ac20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc06f3f0 tx=0x7ff3b40025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.652+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8014440 con 0x7ff3bc06e370 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.652+0000 7ff3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 10 monclient: dump: 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.674 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff3b80145a0 con 0x7ff3bc06e370 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 msgr2=0x7ff3bc0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 0x7ff3bc0696a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8011bf0 tx=0x7ff3b80090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 msgr2=0x7ff3bc0696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.653+0000 7ff3b1ffb640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc06e370 0x7ff3bc0696a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.658+0000 7ff3c419d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.658+0000 7ff3c419d640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 0x7ff3bc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.658+0000 7ff3c419d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.658+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc065f20 con 0x7ff3bc069be0 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 0x7ff3bc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient: get_auth_request con 0x7ff3bc069be0 auth_method 0 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 10 monclient(hunting): handle_auth_done global_id 44665 payload 824 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.659+0000 7ff3c1738640 1 -- 172.21.15.153:0/535771673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff3bc04bf60 con 0x7ff3bc069be0 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.661+0000 7ff3c1738640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 msgr2=0x7ff3bc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.661+0000 7ff3c1738640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 0x7ff3bc069fe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8011bf0 tx=0x7ff3b801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.661+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b801c5f0 con 0x7ff3bc069be0 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.661+0000 7ff3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.661+0000 7ff3b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 10 monclient: dump: 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.677 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7ff3b800fd60 con 0x7ff3bc069be0 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 msgr2=0x7ff3bc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 0x7ff3bc069fe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8011bf0 tx=0x7ff3b801c000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 msgr2=0x7ff3bc069fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3b1ffb640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3bc069be0 0x7ff3bc069fe0 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8011bf0 tx=0x7ff3b801c000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:01.688 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:01.689 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:01.691 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:01.692 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:01.692 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:01.692 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:01.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:01.692 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:01.692 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:01.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff3a0038770 msgr2=0x7ff3a003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff3a0038770 0x7ff3a003ac20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc06f3f0 tx=0x7ff3b40025b0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 10 monclient: shutdown 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 msgr2=0x7ff3bc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 --2- 172.21.15.153:0/535771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3bc0974f0 0x7ff3bc06d3b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac035040 tx=0x7ff3ac032000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c273a640 1 -- 172.21.15.153:0/535771673 reap_dead start 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c273a640 1 -- 172.21.15.153:0/535771673 reap_dead start 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 shutdown_connections 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 >> 172.21.15.153:0/535771673 conn(0x7ff3bc0672b0 msgr2=0x7ff3bc09d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 shutdown_connections 2024-09-15T03:32:01.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.662+0000 7ff3c419d640 1 -- 172.21.15.153:0/535771673 wait complete. 2024-09-15T03:32:01.695 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:01.695 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:32:01.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:32:01.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd68295d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:01.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd68295d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:01.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd68295d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd67c064a70 con 0x7fd67c09f240 2024-09-15T03:32:01.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd663fff640 1 -- 172.21.15.181:0/4200586645 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (secure 0 0 0) 0x7fd66c042300 con 0x7fd67c09f240 2024-09-15T03:32:01.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd663fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:01.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.734+0000 7fd663fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:01.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd668038740 msgr2=0x7fd66803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd668038740 0x7fd66803abf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd670007fd0 tx=0x7fd670007f00 comp rx=0 tx=0).stop 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 10 monclient: shutdown 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 msgr2=0x7fd67c141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 0x7fd67c141860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd66c034000 tx=0x7fd66c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 shutdown_connections 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd668038740 0x7fd66803abf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd67c136e70 0x7fd67c0988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd67c131b40 0x7fd67c141da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 --2- 172.21.15.181:0/4200586645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd67c09f240 0x7fd67c141860 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 >> 172.21.15.181:0/4200586645 conn(0x7fd67c0a5100 msgr2=0x7fd67c0a3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 shutdown_connections 2024-09-15T03:32:01.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.736+0000 7fd68295d640 1 -- 172.21.15.181:0/4200586645 wait complete. 2024-09-15T03:32:01.746 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.7 host1 2024-09-15T03:32:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.787+0000 7f769099d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.787+0000 7f769099d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.787+0000 7f769099d640 10 monclient: monmap: 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:01.788064+0000 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:01.788064+0000 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.787+0000 7f769099d640 1 Processor -- start 2024-09-15T03:32:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.787+0000 7f769099d640 1 -- start start 2024-09-15T03:32:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.787+0000 7f769099d640 10 monclient: init 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f768c0974f0 0x7f768c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): picked mon.noname-a con 0x7f768c0974f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f768c09a8a0 0x7f768c09ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): picked mon.noname-c con 0x7f768c09a8a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f768c12dd40 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0978c0 con 0x7f768c12dd40 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c09b220 con 0x7f768c0974f0 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c09ca20 con 0x7f768c09a8a0 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f769099d640 10 monclient(hunting): authenticate will time out at 919.427528s 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f768c0974f0 0x7f768c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f768c0974f0 0x7f768c12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43358/0 (socket says 172.21.15.153:43358) 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 -- 172.21.15.153:0/2187421813 learned_addr learned my addr 172.21.15.153:0/2187421813 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): get_auth_request con 0x7f768c0974f0 auth_method 0 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f7689d9b640 1 --2- 172.21.15.153:0/2187421813 >> v2:172.21.15.181:3300/0 conn(0x7f768c09a8a0 0x7f768c09ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f7689d9b640 10 monclient(hunting): get_auth_request con 0x7f768c09a8a0 auth_method 0 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f7689d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f7689d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f7689d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient(hunting): handle_auth_done global_id 64131 payload 274 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 -- 172.21.15.153:0/2187421813 >> v2:172.21.15.181:3300/0 conn(0x7f768c09a8a0 msgr2=0x7f768c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 --2- 172.21.15.153:0/2187421813 >> v2:172.21.15.181:3300/0 conn(0x7f768c09a8a0 0x7f768c09ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 -- 172.21.15.153:0/2187421813 >> v1:172.21.15.181:6789/0 conn(0x7f768c12dd40 legacy=0x7f768c12e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 monclient: found mon.noname-a 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.788+0000 7f768a59c640 1 -- 172.21.15.153:0/2187421813 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c098ad0 con 0x7f768c0974f0 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768a59c640 1 -- 172.21.15.153:0/2187421813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c0974f0 msgr2=0x7f768c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768a59c640 1 --2- 172.21.15.153:0/2187421813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c0974f0 0x7f768c12ab00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f768c09d040 tx=0x7f767c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdad998d29ef2cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 1 -- 172.21.15.153:0/2187421813 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c032440 con 0x7f768c0974f0 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: dump: 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 1 -- 172.21.15.153:0/2187421813 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c0325a0 con 0x7f768c0974f0 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f769099d640 5 monclient: authenticate success, global_id 64131 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f769099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f769099d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f769099d640 4 set_mon_vals no callback set 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 1 -- 172.21.15.153:0/2187421813 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c032940 con 0x7f768c0974f0 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.791+0000 7f768959a640 10 monclient: dump: 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 10 monclient: shutdown 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 -- 172.21.15.153:0/2187421813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c0974f0 msgr2=0x7f768c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 --2- 172.21.15.153:0/2187421813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c0974f0 0x7f768c12ab00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f768c09d040 tx=0x7f767c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 -- 172.21.15.153:0/2187421813 shutdown_connections 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 --2- 172.21.15.153:0/2187421813 >> v2:172.21.15.181:3300/0 conn(0x7f768c09a8a0 0x7f768c09ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 --2- 172.21.15.153:0/2187421813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c0974f0 0x7f768c12ab00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 -- 172.21.15.153:0/2187421813 >> 172.21.15.153:0/2187421813 conn(0x7f768c0672b0 msgr2=0x7f768c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 -- 172.21.15.153:0/2187421813 shutdown_connections 2024-09-15T03:32:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 1 -- 172.21.15.153:0/2187421813 wait complete. 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.792+0000 7f769099d640 10 monclient: monmap: 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 Processor -- start 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 -- start start 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient: init 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f768c0974f0 0x7f768c13f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): picked mon.noname-b con 0x7f768c0974f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f768c09a8a0 0x7f768c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): picked mon.noname-c con 0x7f768c09a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 0x7f768c13fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): picked mon.noname-a con 0x7f768c12dd40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c12e3b0 con 0x7f768c12dd40 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c06f5d0 con 0x7f768c0974f0 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c12f990 con 0x7f768c09a8a0 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f769099d640 10 monclient(hunting): authenticate will time out at 919.432709s 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 0x7f768c13fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f768c09a8a0 0x7f768c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 0x7f768c13fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43362/0 (socket says 172.21.15.153:43362) 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 1 -- 172.21.15.153:0/595063595 learned_addr learned my addr 172.21.15.153:0/595063595 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): get_auth_request con 0x7f768c12dd40 auth_method 0 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): get_auth_request con 0x7f768c09a8a0 auth_method 0 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f768c0974f0 0x7f768c13f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 10 monclient(hunting): get_auth_request con 0x7f768c0974f0 auth_method 0 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f7689d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.793+0000 7f768a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 10 monclient(hunting): handle_auth_done global_id 64134 payload 824 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 1 -- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f768c09a8a0 msgr2=0x7f768c13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f768c09a8a0 0x7f768c13f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 1 -- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f768c0974f0 msgr2=0x7f768c13f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f768c0974f0 0x7f768c13f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 1 monclient: found mon.noname-a 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f768ad9d640 1 -- 172.21.15.153:0/595063595 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c009450 con 0x7f768c12dd40 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.794+0000 7f7689d9b640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f768c09a8a0 0x7f768c13f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f768ad9d640 1 -- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 msgr2=0x7f768c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f768ad9d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 0x7f768c13fe90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f768400c890 tx=0x7f7684002640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768400ce20 con 0x7f768c12dd40 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: dump: 2024-09-15T03:32:01.809 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768400aa00 con 0x7f768c12dd40 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 5 monclient: authenticate success, global_id 64134 2024-09-15T03:32:01.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768400adc0 con 0x7f768c12dd40 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 10 monclient: _renew_subs 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c12f310 con 0x7f768c12dd40 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f767a7fc640 10 monclient: dump: 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:01.811 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f7688d99640 4 set_mon_vals no callback set 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 10 monclient: _renew_subs 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.796+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c06ee80 con 0x7f768c12dd40 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7684012630 con 0x7f768c12dd40 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f769099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f769099d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f769099d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c12fb10 con 0x7f768c12dd40 2024-09-15T03:32:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f767a7fc640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f766c038790 0x7f766c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f768404c3b0 con 0x7f768c12dd40 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f768a59c640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f766c038790 0x7f766c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.797+0000 7f768a59c640 10 monclient: get_auth_request con 0x7f766c038790 auth_method 0 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.800+0000 7f768a59c640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f766c038790 0x7f766c03ac40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f768c098780 tx=0x7f767c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.801+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f768c12fb10 con 0x7f768c12dd40 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.801+0000 7f767a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:01.801+0000 7f767a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.820+0000 7f318e55d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.820+0000 7f318e55d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.820+0000 7f318e55d640 10 monclient: monmap: 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:01.820871+0000 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:01.820871+0000 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:01.821 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:01.822 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.820+0000 7f318e55d640 1 Processor -- start 2024-09-15T03:32:01.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.820+0000 7f318e55d640 1 -- start start 2024-09-15T03:32:01.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.820+0000 7f318e55d640 10 monclient: init 2024-09-15T03:32:01.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f318809ec50 0x7f3188131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): picked mon.noname-c con 0x7f318809ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f31881321a0 0x7f3188068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): picked mon.noname-a con 0x7f31881321a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f318809f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188067420 con 0x7f318809f020 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318809fdc0 con 0x7f31881321a0 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880a0510 con 0x7f318809ec50 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f318e55d640 10 monclient(hunting): authenticate will time out at 926.302562s 2024-09-15T03:32:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f318809ec50 0x7f3188131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f318809ec50 0x7f3188131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39588/0 (socket says 172.21.15.181:39588) 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 -- 172.21.15.181:0/94498132 learned_addr learned my addr 172.21.15.181:0/94498132 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): get_auth_request con 0x7f318809ec50 auth_method 0 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 1 --2- 172.21.15.181:0/94498132 >> v2:172.21.15.153:3300/0 conn(0x7f31881321a0 0x7f3188068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): get_auth_request con 0x7f31881321a0 auth_method 0 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient(hunting): handle_auth_done global_id 44674 payload 274 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 -- 172.21.15.181:0/94498132 >> v2:172.21.15.153:3300/0 conn(0x7f31881321a0 msgr2=0x7f3188068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 --2- 172.21.15.181:0/94498132 >> v2:172.21.15.153:3300/0 conn(0x7f31881321a0 0x7f3188068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 -- 172.21.15.181:0/94498132 >> v1:172.21.15.153:6789/0 conn(0x7f318809f020 legacy=0x7f3188069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 monclient: found mon.noname-c 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.821+0000 7f3187fff640 1 -- 172.21.15.181:0/94498132 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880a0200 con 0x7f318809ec50 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f3187fff640 1 -- 172.21.15.181:0/94498132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809ec50 msgr2=0x7f3188131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f3187fff640 1 --2- 172.21.15.181:0/94498132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809ec50 0x7f3188131c60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31880719d0 tx=0x7f31780091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbd642248fcb2da0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 1 -- 172.21.15.181:0/94498132 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178032440 con 0x7f318809ec50 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: dump: 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.834 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 1 -- 172.21.15.181:0/94498132 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31780325a0 con 0x7f318809ec50 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f318e55d640 5 monclient: authenticate success, global_id 44674 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f318e55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 1 -- 172.21.15.181:0/94498132 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178032940 con 0x7f318809ec50 2024-09-15T03:32:01.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f31877fe640 10 monclient: dump: 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f318e55d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:01.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.824+0000 7f318e55d640 4 set_mon_vals no callback set 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 10 monclient: shutdown 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 -- 172.21.15.181:0/94498132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809ec50 msgr2=0x7f3188131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 --2- 172.21.15.181:0/94498132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809ec50 0x7f3188131c60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31880719d0 tx=0x7f31780091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 -- 172.21.15.181:0/94498132 shutdown_connections 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 --2- 172.21.15.181:0/94498132 >> v2:172.21.15.153:3300/0 conn(0x7f31881321a0 0x7f3188068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 --2- 172.21.15.181:0/94498132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809ec50 0x7f3188131c60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 -- 172.21.15.181:0/94498132 >> 172.21.15.181:0/94498132 conn(0x7f31880a5d40 msgr2=0x7f31880a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 -- 172.21.15.181:0/94498132 shutdown_connections 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.825+0000 7f318e55d640 1 -- 172.21.15.181:0/94498132 wait complete. 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient: build_initial_monmap 2024-09-15T03:32:01.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient: monmap: 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 Processor -- start 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 -- start start 2024-09-15T03:32:01.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient: init 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f318809ec50 0x7f318809a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): picked mon.noname-a con 0x7f318809ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 0x7f3188130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): picked mon.noname-b con 0x7f318809f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31881321a0 0x7f318811b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): picked mon.noname-c con 0x7f31881321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318809dcd0 con 0x7f318809ec50 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318811dd80 con 0x7f318809f020 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:01.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318811cb40 con 0x7f31881321a0 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f318e55d640 10 monclient(hunting): authenticate will time out at 926.307881s 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f317ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 0x7f3188130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.826+0000 7f317ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 0x7f3188130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39600/0 (socket says 172.21.15.181:39600) 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 -- 172.21.15.181:0/4220832161 learned_addr learned my addr 172.21.15.181:0/4220832161 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): get_auth_request con 0x7f318809f020 auth_method 0 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f318809ec50 0x7f318809a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): get_auth_request con 0x7f318809ec50 auth_method 0 2024-09-15T03:32:01.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31881321a0 0x7f318811b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): get_auth_request con 0x7f31881321a0 auth_method 0 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:01.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f318cafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient(hunting): handle_auth_done global_id 44677 payload 824 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 -- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31881321a0 msgr2=0x7f318811b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31881321a0 0x7f318811b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 -- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f318809ec50 msgr2=0x7f318809a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f318809ec50 0x7f318809a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 monclient: found mon.noname-b 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f317ffff640 1 -- 172.21.15.181:0/4220832161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178009450 con 0x7f318809f020 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.827+0000 7f3187fff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f318809ec50 0x7f318809a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f317ffff640 1 -- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 msgr2=0x7f3188130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f317ffff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 0x7f3188130980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f317000ea20 tx=0x7f31700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170009600 con 0x7f318809f020 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: dump: 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.843 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317000cc20 con 0x7f318809f020 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 5 monclient: authenticate success, global_id 44677 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 10 monclient: _renew_subs 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318811daa0 con 0x7f318809f020 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170013470 con 0x7f318809f020 2024-09-15T03:32:01.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3184ff9640 10 monclient: dump: 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 10 monclient: _renew_subs 2024-09-15T03:32:01.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3188141e20 con 0x7f318809f020 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.829+0000 7f3186ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3170013860 con 0x7f318809f020 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f318e55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f318e55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f318e55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f3184ff9640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3168038740 0x7f316803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31880008d0 con 0x7f318809f020 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f317004c450 con 0x7f318809f020 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f3187fff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3168038740 0x7f316803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.830+0000 7f3187fff640 10 monclient: get_auth_request con 0x7f3168038740 auth_method 0 2024-09-15T03:32:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.833+0000 7f3187fff640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3168038740 0x7f316803abf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3188132a50 tx=0x7f3178032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.835+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3170023510 con 0x7f318809f020 2024-09-15T03:32:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.835+0000 7f3184ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.835+0000 7f3184ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:01.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.996+0000 7f318e55d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T03:32:01.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.996+0000 7f318e55d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T03:32:01.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.996+0000 7f318e55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:01.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:01.996+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f31880a13e0 con 0x7f318809f020 2024-09-15T03:32:02.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.018+0000 7f769099d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:32:02.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.018+0000 7f769099d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:32:02.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.018+0000 7f769099d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:02.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.018+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f768c04bf60 con 0x7f768c12dd40 2024-09-15T03:32:02.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.019+0000 7f767a7fc640 1 -- 172.21.15.153:0/595063595 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7684016020 con 0x7f768c12dd40 2024-09-15T03:32:02.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.019+0000 7f767a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:32:02.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.019+0000 7f767a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:02.021 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.019+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.021 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.019+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=1076495460,config=3140514009,health=1791238353,kv=3235807718,logm=3501888497}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=3034187616,config=3140514009,health=1647462659,kv=3235807718,logm=4181601370}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=1076495460,config=3140514009,health=1791238353,kv=3235807718,logm=3501888497}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=2675797323,config=3140514009,health=1993131081,kv=2243821149,logm=223253557}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1991488020}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154765224}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1991488020}) 2024-09-15T03:32:02.022 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.020+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3135126288}) 2024-09-15T03:32:02.023 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.023 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655062681}) 2024-09-15T03:32:02.023 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668528145}) 2024-09-15T03:32:02.023 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.023 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655062681}) 2024-09-15T03:32:02.023 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938546945}) 2024-09-15T03:32:02.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f766c038790 msgr2=0x7f766c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f766c038790 0x7f766c03ac40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f768c098780 tx=0x7f767c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 10 monclient: shutdown 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 msgr2=0x7f768c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 0x7f768c13fe90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f768400c890 tx=0x7f7684002640 comp rx=0 tx=0).stop 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 shutdown_connections 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f766c038790 0x7f766c03ac40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f768c12dd40 0x7f768c13fe90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f768c09a8a0 0x7f768c13f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 --2- 172.21.15.153:0/595063595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f768c0974f0 0x7f768c13f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.021+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 >> 172.21.15.153:0/595063595 conn(0x7f768c0672b0 msgr2=0x7f768c09b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 shutdown_connections 2024-09-15T03:32:02.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f769099d640 1 -- 172.21.15.153:0/595063595 wait complete. 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374809549}) 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384479659}) 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374809549}) 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.022+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076936655}) 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354203007}) 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979110853}) 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354203007}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2504340277}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=739785636}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.023+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1455171685}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=739785636}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324652826}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227972631}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1607312278}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227972631}) 2024-09-15T03:32:02.030 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.024+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753977706}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091144505}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755703592}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091144505}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711356373}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503322744}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204931735}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503322744}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.025+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089269471}) 2024-09-15T03:32:02.031 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.026+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.026+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103213508}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.026+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2409682917}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.026+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.026+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103213508}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.026+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=234907129}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=9} crc {logm=733743195,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgrstat=1403051695}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=3786828760,mdsmap=803431801,mgr=4081113706,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1403051695,mon_config_key=4117641549}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=9} crc {logm=733743195,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgrstat=1403051695}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=2} crc {logm=2258242723,mdsmap=1531151339,mgr=2346544791,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3248699884,mon_config_key=753185158}) 2024-09-15T03:32:02.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=83} crc {mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=3055413492,osd_snap=454132730,osdmap=1121663858}) 2024-09-15T03:32:02.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=83} crc {mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=3055413492,osd_snap=454132730,osdmap=1573996182}) 2024-09-15T03:32:02.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=83} crc {mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=3055413492,osd_snap=454132730,osdmap=1121663858}) 2024-09-15T03:32:02.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.027+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=84} crc {mon_config_key=4117641549,monmap=244636553,osd_metadata=1491145534,osd_pg_creating=3055413492,osd_snap=454132730,osdmap=717188596}) 2024-09-15T03:32:02.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.029+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.030+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=39} crc {osdmap=466615119}) 2024-09-15T03:32:02.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.030+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=3300229870}) 2024-09-15T03:32:02.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.030+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:02.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.030+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=39} crc {osdmap=466615119}) 2024-09-15T03:32:02.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:02.030+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=3747689321}) 2024-09-15T03:32:02.035 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:32:02.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.298+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v76) v1 ==== 143+0+0 (secure 0 0 0) 0x7f317003da10 con 0x7f318809f020 2024-09-15T03:32:02.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.298+0000 7f3184ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T03:32:02.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.298+0000 7f3184ff9640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-15T03:32:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.397+0000 7f318e55d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T03:32:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.397+0000 7f318e55d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T03:32:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.397+0000 7f318e55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.397+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f3188072640 con 0x7f318809f020 2024-09-15T03:32:02.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.398+0000 7f3184ff9640 1 -- 172.21.15.181:0/4220832161 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v76) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3170023330 con 0x7f318809f020 2024-09-15T03:32:02.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.398+0000 7f3184ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-15T03:32:02.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.398+0000 7f3184ff9640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-15T03:32:02.399 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.7' does not appear in the crush map 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3168038740 msgr2=0x7f316803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3168038740 0x7f316803abf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3188132a50 tx=0x7f3178032000 comp rx=0 tx=0).stop 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 10 monclient: shutdown 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 msgr2=0x7f3188130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 0x7f3188130980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f317000ea20 tx=0x7f31700090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 shutdown_connections 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3168038740 0x7f316803abf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31881321a0 0x7f318811b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f318809f020 0x7f3188130980 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 --2- 172.21.15.181:0/4220832161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f318809ec50 0x7f318809a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 >> 172.21.15.181:0/4220832161 conn(0x7f31880a5d40 msgr2=0x7f31880a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 shutdown_connections 2024-09-15T03:32:02.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.400+0000 7f318e55d640 1 -- 172.21.15.181:0/4220832161 wait complete. 2024-09-15T03:32:02.409 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm host1 2024-09-15T03:32:02.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.482+0000 7f8edd64f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:02.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.482+0000 7f8edd64f640 10 monclient: build_initial_monmap 2024-09-15T03:32:02.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.482+0000 7f8edd64f640 10 monclient: monmap: 2024-09-15T03:32:02.483 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:02.483 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:02.483 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:02.482951+0000 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:02.482951+0000 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:02.484 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.482+0000 7f8edd64f640 1 Processor -- start 2024-09-15T03:32:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.482+0000 7f8edd64f640 1 -- start start 2024-09-15T03:32:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.482+0000 7f8edd64f640 10 monclient: init 2024-09-15T03:32:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:32:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ed81321a0 0x7f8ed8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed81321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ed809f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed8067420 con 0x7f8ed809f020 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80a0480 con 0x7f8ed81321a0 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80a0200 con 0x7f8ed809ec50 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8edd64f640 10 monclient(hunting): authenticate will time out at 926.964358s 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ed81321a0 0x7f8ed8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ed81321a0 0x7f8ed8068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39612/0 (socket says 172.21.15.181:39612) 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 -- 172.21.15.181:0/2964360076 learned_addr learned my addr 172.21.15.181:0/2964360076 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ed81321a0 auth_method 0 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:02.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed6ffd640 1 --2- 172.21.15.181:0/2964360076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed809ec50 auth_method 0 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_done global_id 44680 payload 274 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 -- 172.21.15.181:0/2964360076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 msgr2=0x7f8ed8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 --2- 172.21.15.181:0/2964360076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed8131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 -- 172.21.15.181:0/2964360076 >> v1:172.21.15.153:6789/0 conn(0x7f8ed809f020 legacy=0x7f8ed8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 monclient: found mon.noname-c 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.483+0000 7f8ed67fc640 1 -- 172.21.15.181:0/2964360076 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed809fd80 con 0x7f8ed81321a0 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed67fc640 1 -- 172.21.15.181:0/2964360076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 msgr2=0x7f8ed8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed67fc640 1 --2- 172.21.15.181:0/2964360076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed8068d00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009f20 tx=0x7f8ecc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8774e8d5faf32659 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 1 -- 172.21.15.181:0/2964360076 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc007610 con 0x7f8ed81321a0 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: dump: 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 1 -- 172.21.15.181:0/2964360076 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc02d050 con 0x7f8ed81321a0 2024-09-15T03:32:02.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8edd64f640 5 monclient: authenticate success, global_id 44680 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8edd64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 1 -- 172.21.15.181:0/2964360076 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc0079e0 con 0x7f8ed81321a0 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8edd64f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8edd64f640 4 set_mon_vals no callback set 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.486+0000 7f8ed5ffb640 10 monclient: dump: 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:02.494 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 10 monclient: shutdown 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 -- 172.21.15.181:0/2964360076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 msgr2=0x7f8ed8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 --2- 172.21.15.181:0/2964360076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed8068d00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009f20 tx=0x7f8ecc009840 comp rx=0 tx=0).stop 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 -- 172.21.15.181:0/2964360076 shutdown_connections 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 --2- 172.21.15.181:0/2964360076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed8068d00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 --2- 172.21.15.181:0/2964360076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 -- 172.21.15.181:0/2964360076 >> 172.21.15.181:0/2964360076 conn(0x7f8ed80a5cf0 msgr2=0x7f8ed80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:02.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 -- 172.21.15.181:0/2964360076 shutdown_connections 2024-09-15T03:32:02.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.487+0000 7f8edd64f640 1 -- 172.21.15.181:0/2964360076 wait complete. 2024-09-15T03:32:02.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient: build_initial_monmap 2024-09-15T03:32:02.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient: monmap: 2024-09-15T03:32:02.500 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:02.500 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:02.500 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 Processor -- start 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 -- start start 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient: init 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed809a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed809f020 0x7f8ed8130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed809f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed811b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:02.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed81321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8141f90 con 0x7f8ed809f020 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed809b560 con 0x7f8ed81321a0 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8134b70 con 0x7f8ed809ec50 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8edd64f640 10 monclient(hunting): authenticate will time out at 926.969878s 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.488+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed811b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed811b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39616/0 (socket says 172.21.15.181:39616) 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed809a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 -- 172.21.15.181:0/840301091 learned_addr learned my addr 172.21.15.181:0/840301091 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ed81321a0 auth_method 0 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed809f020 0x7f8ed8130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed809ec50 auth_method 0 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ed809f020 auth_method 0 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_done global_id 44683 payload 824 2024-09-15T03:32:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 -- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 msgr2=0x7f8ed809a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed809a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 -- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed809f020 msgr2=0x7f8ed8130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed809f020 0x7f8ed8130a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 monclient: found mon.noname-b 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.489+0000 7f8ed77fe640 1 -- 172.21.15.181:0/840301091 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc009ae0 con 0x7f8ed81321a0 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ed77fe640 1 -- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 msgr2=0x7f8ed811b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ed77fe640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed811b760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800c610 tx=0x7f8ec8007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8012840 con 0x7f8ed81321a0 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: dump: 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 5 monclient: authenticate success, global_id 44683 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec80129e0 con 0x7f8ed81321a0 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8012da0 con 0x7f8ed81321a0 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ebeffd640 10 monclient: dump: 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8ed57fa640 4 set_mon_vals no callback set 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 10 monclient: _renew_subs 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed809b9e0 con 0x7f8ed81321a0 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 10 monclient: _renew_subs 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.491+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed811cc30 con 0x7f8ed81321a0 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8ec8011590 con 0x7f8ed81321a0 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8edd64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:02.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8edd64f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8edd64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed811dcc0 con 0x7f8ed81321a0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8ebeffd640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec4038740 0x7f8ec403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f8ec804ba10 con 0x7f8ed81321a0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8ed6ffd640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec4038740 0x7f8ec403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.492+0000 7f8ed6ffd640 10 monclient: get_auth_request con 0x7f8ec4038740 auth_method 0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.495+0000 7f8ed6ffd640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec4038740 0x7f8ec403abf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000bd40 tx=0x7f8ed0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.497+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8ed811dcc0 con 0x7f8ed81321a0 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.497+0000 7f8ebeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:02.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.497+0000 7f8ebeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.656+0000 7f8edd64f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T03:32:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.656+0000 7f8edd64f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T03:32:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.656+0000 7f8edd64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:02.656+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f8ed80008d0 con 0x7f8ed81321a0 2024-09-15T03:32:03.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:32:03.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.130+0000 7fdf5cd5d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.130+0000 7fdf5cd5d640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.130+0000 7fdf5cd5d640 10 monclient: monmap: 2024-09-15T03:32:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:03.130987+0000 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:03.130987+0000 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:03.132 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.130+0000 7fdf5cd5d640 1 Processor -- start 2024-09-15T03:32:03.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.130+0000 7fdf5cd5d640 1 -- start start 2024-09-15T03:32:03.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.130+0000 7fdf5cd5d640 10 monclient: init 2024-09-15T03:32:03.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf581318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf5809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf5809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdf58136c20 0x7fdf58139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf58136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf58132640 con 0x7fdf581318f0 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf58131d00 con 0x7fdf58136c20 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5809fbf0 con 0x7fdf5809ea50 2024-09-15T03:32:03.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf5cd5d640 10 monclient(hunting): authenticate will time out at 920.770541s 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf5809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf5809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58252/0 (socket says 172.21.15.153:58252) 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 -- 172.21.15.153:0/3943080701 learned_addr learned my addr 172.21.15.153:0/3943080701 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): get_auth_request con 0x7fdf5809ea50 auth_method 0 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf56d9d640 1 --2- 172.21.15.153:0/3943080701 >> v2:172.21.15.181:3300/0 conn(0x7fdf58136c20 0x7fdf58139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf56d9d640 10 monclient(hunting): get_auth_request con 0x7fdf58136c20 auth_method 0 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_done global_id 5198 payload 274 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 -- 172.21.15.153:0/3943080701 >> v2:172.21.15.181:3300/0 conn(0x7fdf58136c20 msgr2=0x7fdf58139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 --2- 172.21.15.153:0/3943080701 >> v2:172.21.15.181:3300/0 conn(0x7fdf58136c20 0x7fdf58139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 -- 172.21.15.153:0/3943080701 >> v1:172.21.15.181:6789/0 conn(0x7fdf581318f0 legacy=0x7fdf58131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 monclient: found mon.noname-b 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.131+0000 7fdf55d9b640 1 -- 172.21.15.153:0/3943080701 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5809fe30 con 0x7fdf5809ea50 2024-09-15T03:32:03.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf55d9b640 1 -- 172.21.15.153:0/3943080701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 msgr2=0x7fdf5809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf55d9b640 1 --2- 172.21.15.153:0/3943080701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf5809ee20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009910 tx=0x7fdf4c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3351751384adb32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 1 -- 172.21.15.153:0/3943080701 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c02aea0 con 0x7fdf5809ea50 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: dump: 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 1 -- 172.21.15.153:0/3943080701 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c032440 con 0x7fdf5809ea50 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 1 -- 172.21.15.153:0/3943080701 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c0327e0 con 0x7fdf5809ea50 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5cd5d640 5 monclient: authenticate success, global_id 5198 2024-09-15T03:32:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5cd5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5cd5d640 4 set_mon_vals no callback set 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.134+0000 7fdf5559a640 10 monclient: dump: 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 10 monclient: shutdown 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/3943080701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 msgr2=0x7fdf5809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/3943080701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf5809ee20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009910 tx=0x7fdf4c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/3943080701 shutdown_connections 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/3943080701 >> v2:172.21.15.181:3300/0 conn(0x7fdf58136c20 0x7fdf58139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/3943080701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf5809ee20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/3943080701 >> 172.21.15.153:0/3943080701 conn(0x7fdf58067110 msgr2=0x7fdf58067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/3943080701 shutdown_connections 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/3943080701 wait complete. 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.135+0000 7fdf5cd5d640 10 monclient: monmap: 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.149 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 Processor -- start 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 -- start start 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient: init 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf58130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf5809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf581318f0 0x7fdf58099760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf581318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf58136c20 0x7fdf58099ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf58136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5811cb10 con 0x7fdf581318f0 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5811d040 con 0x7fdf58136c20 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf58141ef0 con 0x7fdf5809ea50 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5cd5d640 10 monclient(hunting): authenticate will time out at 920.775746s 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf58130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf58130900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58256/0 (socket says 172.21.15.153:58256) 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf581318f0 0x7fdf58099760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 1 -- 172.21.15.153:0/399523547 learned_addr learned my addr 172.21.15.153:0/399523547 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): get_auth_request con 0x7fdf5809ea50 auth_method 0 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): get_auth_request con 0x7fdf581318f0 auth_method 0 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.136+0000 7fdf56d9d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf58136c20 0x7fdf58099ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): get_auth_request con 0x7fdf58136c20 auth_method 0 2024-09-15T03:32:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 10 monclient(hunting): handle_auth_done global_id 5201 payload 824 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 1 -- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf58136c20 msgr2=0x7fdf58099ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf58136c20 0x7fdf58099ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 1 -- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf581318f0 msgr2=0x7fdf58099760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf581318f0 0x7fdf58099760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 1 monclient: found mon.noname-c 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf5659c640 1 -- 172.21.15.153:0/399523547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c009450 con 0x7fdf5809ea50 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.137+0000 7fdf55d9b640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf581318f0 0x7fdf58099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5659c640 1 -- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 msgr2=0x7fdf58130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5659c640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf58130900 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c5e0 tx=0x7fdf50007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50012840 con 0x7fdf5809ea50 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: dump: 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf500129a0 con 0x7fdf5809ea50 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 5 monclient: authenticate success, global_id 5201 2024-09-15T03:32:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 10 monclient: _renew_subs 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5809e030 con 0x7fdf5809ea50 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50012d60 con 0x7fdf5809ea50 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf467fc640 10 monclient: dump: 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf54d99640 4 set_mon_vals no callback set 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 10 monclient: _renew_subs 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.139+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5811b8e0 con 0x7fdf5809ea50 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdf50011590 con 0x7fdf5809ea50 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf5cd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf5cd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf5cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5811cdf0 con 0x7fdf5809ea50 2024-09-15T03:32:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf467fc640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf38038740 0x7fdf3803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.140+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fdf5004c120 con 0x7fdf5809ea50 2024-09-15T03:32:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.141+0000 7fdf55d9b640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf38038740 0x7fdf3803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.141+0000 7fdf55d9b640 10 monclient: get_auth_request con 0x7fdf38038740 auth_method 0 2024-09-15T03:32:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.143+0000 7fdf55d9b640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf38038740 0x7fdf3803abf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdf58076600 tx=0x7fdf4c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.145+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdf50010aa0 con 0x7fdf5809ea50 2024-09-15T03:32:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.145+0000 7fdf467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.145+0000 7fdf467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.358+0000 7fdf5cd5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.358+0000 7fdf5cd5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.358+0000 7fdf5cd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.358+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdf5804bf60 con 0x7fdf5809ea50 2024-09-15T03:32:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.359+0000 7fdf467fc640 1 -- 172.21.15.153:0/399523547 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fdf50016620 con 0x7fdf5809ea50 2024-09-15T03:32:03.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.359+0000 7fdf467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:03.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.359+0000 7fdf467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:03.361 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:03.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf38038740 msgr2=0x7fdf3803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf38038740 0x7fdf3803abf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdf58076600 tx=0x7fdf4c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 10 monclient: shutdown 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 msgr2=0x7fdf58130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf58130900 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c5e0 tx=0x7fdf50007ed0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 shutdown_connections 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf38038740 0x7fdf3803abf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf58136c20 0x7fdf58099ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf581318f0 0x7fdf58099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 --2- 172.21.15.153:0/399523547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf5809ea50 0x7fdf58130900 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 >> 172.21.15.153:0/399523547 conn(0x7fdf58067110 msgr2=0x7fdf5809e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 shutdown_connections 2024-09-15T03:32:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.361+0000 7fdf5cd5d640 1 -- 172.21.15.153:0/399523547 wait complete. 2024-09-15T03:32:03.373 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:32:03.373 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:32:03.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:32:03.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:32:03.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.466+0000 7ff590f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:03.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.466+0000 7ff590f9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.466+0000 7ff590f9d640 10 monclient: monmap: 2024-09-15T03:32:03.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:03.467681+0000 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:03.467681+0000 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:03.470 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:03.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.467+0000 7ff590f9d640 1 Processor -- start 2024-09-15T03:32:03.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.467+0000 7ff590f9d640 1 -- start start 2024-09-15T03:32:03.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.467+0000 7ff590f9d640 10 monclient: init 2024-09-15T03:32:03.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.467+0000 7ff590f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.467+0000 7ff590f9d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff58c09ad90 0x7ff58c09c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff58c09ad90 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff58c09b160 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff58c09b530 0x7ff58c132070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff58c09b530 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:03.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0692c0 con 0x7ff58c09b160 2024-09-15T03:32:03.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c09b990 con 0x7ff58c09ad90 2024-09-15T03:32:03.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c09db70 con 0x7ff58c09b530 2024-09-15T03:32:03.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff590f9d640 10 monclient(hunting): authenticate will time out at 921.107071s 2024-09-15T03:32:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff58c09ad90 0x7ff58c09c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff58c09ad90 0x7ff58c09c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43388/0 (socket says 172.21.15.153:43388) 2024-09-15T03:32:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 -- 172.21.15.153:0/3123593077 learned_addr learned my addr 172.21.15.153:0/3123593077 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): get_auth_request con 0x7ff58c09ad90 auth_method 0 2024-09-15T03:32:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58ad9d640 1 --2- 172.21.15.153:0/3123593077 >> v2:172.21.15.181:3300/0 conn(0x7ff58c09b530 0x7ff58c132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58ad9d640 10 monclient(hunting): get_auth_request con 0x7ff58c09b530 auth_method 0 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient(hunting): handle_auth_done global_id 64149 payload 274 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 -- 172.21.15.153:0/3123593077 >> v2:172.21.15.181:3300/0 conn(0x7ff58c09b530 msgr2=0x7ff58c132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 --2- 172.21.15.153:0/3123593077 >> v2:172.21.15.181:3300/0 conn(0x7ff58c09b530 0x7ff58c132070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 -- 172.21.15.153:0/3123593077 >> v1:172.21.15.181:6789/0 conn(0x7ff58c09b160 legacy=0x7ff58c131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 monclient: found mon.noname-a 2024-09-15T03:32:03.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58a59c640 1 -- 172.21.15.153:0/3123593077 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c09d5b0 con 0x7ff58c09ad90 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.468+0000 7ff58ad9d640 1 --2- 172.21.15.153:0/3123593077 >> v2:172.21.15.181:3300/0 conn(0x7ff58c09b530 0x7ff58c132070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff58a59c640 1 -- 172.21.15.153:0/3123593077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 msgr2=0x7ff58c09c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff58a59c640 1 --2- 172.21.15.153:0/3123593077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c09c5d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff58c066170 tx=0x7ff57c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5aeb9b2ad8bb26b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 1 -- 172.21.15.153:0/3123593077 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c032440 con 0x7ff58c09ad90 2024-09-15T03:32:03.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: dump: 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 1 -- 172.21.15.153:0/3123593077 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c0325a0 con 0x7ff58c09ad90 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 5 monclient: authenticate success, global_id 64149 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 1 -- 172.21.15.153:0/3123593077 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c032980 con 0x7ff58c09ad90 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 4 set_mon_vals no callback set 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:03.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff589d9b640 10 monclient: dump: 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 10 monclient: shutdown 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 1 -- 172.21.15.153:0/3123593077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 msgr2=0x7ff58c09c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.471+0000 7ff590f9d640 1 --2- 172.21.15.153:0/3123593077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c09c5d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff58c066170 tx=0x7ff57c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 -- 172.21.15.153:0/3123593077 shutdown_connections 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 --2- 172.21.15.153:0/3123593077 >> v2:172.21.15.181:3300/0 conn(0x7ff58c09b530 0x7ff58c132070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 --2- 172.21.15.153:0/3123593077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c09c5d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 -- 172.21.15.153:0/3123593077 >> 172.21.15.153:0/3123593077 conn(0x7ff58c0678a0 msgr2=0x7ff58c0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 -- 172.21.15.153:0/3123593077 shutdown_connections 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 -- 172.21.15.153:0/3123593077 wait complete. 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 10 monclient: monmap: 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:03.494 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.472+0000 7ff590f9d640 1 Processor -- start 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 -- start start 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient: init 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:03.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c08f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff58c09ad90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff58c09b160 0x7ff58c08b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff58c09b160 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c09b530 0x7ff58c08b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff58c09b530 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c08eed0 con 0x7ff58c09ad90 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c08e880 con 0x7ff58c09b530 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c08e5a0 con 0x7ff58c09b160 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff590f9d640 10 monclient(hunting): authenticate will time out at 921.112516s 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c08f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c08f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43390/0 (socket says 172.21.15.153:43390) 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 -- 172.21.15.153:0/2332047554 learned_addr learned my addr 172.21.15.153:0/2332047554 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): get_auth_request con 0x7ff58c09ad90 auth_method 0 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff58c09b160 0x7ff58c08b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): get_auth_request con 0x7ff58c09b160 auth_method 0 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58ad9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c09b530 0x7ff58c08b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient(hunting): handle_auth_done global_id 64152 payload 824 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff58c09b160 msgr2=0x7ff58c08b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff58c09b160 0x7ff58c08b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c09b530 msgr2=0x7ff58c08b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c09b530 0x7ff58c08b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 monclient: found mon.noname-a 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58a59c640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c009450 con 0x7ff58c09ad90 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58ad9d640 10 monclient: get_auth_request con 0x7ff58c09b530 auth_method 0 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.473+0000 7ff58ad9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c09b530 0x7ff58c08b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff58a59c640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 msgr2=0x7ff58c08f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff58a59c640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c08f260 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff57c035040 tx=0x7ff57c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c031a40 con 0x7ff58c09ad90 2024-09-15T03:32:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: dump: 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.500 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c031be0 con 0x7ff58c09ad90 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 5 monclient: authenticate success, global_id 64152 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 10 monclient: _renew_subs 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c08eb60 con 0x7ff58c09ad90 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c03d440 con 0x7ff58c09ad90 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff582ffd640 10 monclient: dump: 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff58959a640 4 set_mon_vals no callback set 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 10 monclient: _renew_subs 2024-09-15T03:32:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.476+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c08d230 con 0x7ff58c09ad90 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff590f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff590f9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 0x7ff58c1326f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c065e00 con 0x7ff58c08bb50 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff57c039040 con 0x7ff58c09ad90 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 0x7ff58c1326f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient: get_auth_request con 0x7ff58c08bb50 auth_method 0 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff582ffd640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff56c038740 0x7ff56c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 10 monclient(hunting): handle_auth_done global_id 64155 payload 824 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ff57c06bf10 con 0x7ff58c09ad90 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff581d9b640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff58c04bf60 con 0x7ff58c08bb50 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff58ad9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff56c038740 0x7ff56c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.477+0000 7ff58ad9d640 10 monclient: get_auth_request con 0x7ff56c038740 auth_method 0 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff581d9b640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 msgr2=0x7ff58c1326f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff581d9b640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 0x7ff58c1326f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff58c08d930 tx=0x7ff574007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400d910 con 0x7ff58c08bb50 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 10 monclient: dump: 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.505 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff57400da70 con 0x7ff58c08bb50 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.479+0000 7ff582ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.480+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 msgr2=0x7ff58c1326f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.480+0000 7ff582ffd640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 0x7ff58c1326f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff58c08d930 tx=0x7ff574007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.480+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 msgr2=0x7ff58c1326f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.480+0000 7ff582ffd640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c08bb50 0x7ff58c1326f0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff58c08d930 tx=0x7ff574007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:03.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.480+0000 7ff58ad9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff56c038740 0x7ff56c03abf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff5840094d0 tx=0x7ff58400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff590f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff590f9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 0x7ff58c133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff590f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c08d450 con 0x7ff58c132c30 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 0x7ff58c133030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient: get_auth_request con 0x7ff58c132c30 auth_method 0 2024-09-15T03:32:03.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 10 monclient(hunting): handle_auth_done global_id 64158 payload 824 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.485+0000 7ff581d9b640 1 -- 172.21.15.153:0/2332047554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff58c133570 con 0x7ff58c132c30 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff581d9b640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 msgr2=0x7ff58c133030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff581d9b640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 0x7ff58c133030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff58c08d930 tx=0x7ff57401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574014890 con 0x7ff58c132c30 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 10 monclient: dump: 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7ff5740149f0 con 0x7ff58c132c30 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 msgr2=0x7ff58c133030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 0x7ff58c133030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff58c08d930 tx=0x7ff57401a000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 msgr2=0x7ff58c133030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff582ffd640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c132c30 0x7ff58c133030 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff58c08d930 tx=0x7ff57401a000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:03.512 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:03.513 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:03.514 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:03.515 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:03.516 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.517 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.518 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:03.519 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.520 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.521 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.522 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.523 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.524 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:03.526 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:03.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff56c038740 msgr2=0x7ff56c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff56c038740 0x7ff56c03abf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff5840094d0 tx=0x7ff58400b040 comp rx=0 tx=0).stop 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 10 monclient: shutdown 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 msgr2=0x7ff58c08f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 --2- 172.21.15.153:0/2332047554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff58c09ad90 0x7ff58c08f260 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff57c035040 tx=0x7ff57c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff58ad9d640 1 -- 172.21.15.153:0/2332047554 reap_dead start 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff58ad9d640 1 -- 172.21.15.153:0/2332047554 reap_dead start 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 shutdown_connections 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 >> 172.21.15.153:0/2332047554 conn(0x7ff58c0678a0 msgr2=0x7ff58c134c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 shutdown_connections 2024-09-15T03:32:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.488+0000 7ff590f9d640 1 -- 172.21.15.153:0/2332047554 wait complete. 2024-09-15T03:32:03.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:03.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:32:03.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:32:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.577+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v78) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8ec801a710 con 0x7f8ed81321a0 2024-09-15T03:32:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.578+0000 7f8ebeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T03:32:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.578+0000 7f8ebeffd640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.624+0000 7f1964aef640 10 monclient: get_monmap_and_config 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.624+0000 7f1964aef640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.624+0000 7f1964aef640 10 monclient: monmap: 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:03.624936+0000 2024-09-15T03:32:03.625 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:03.624936+0000 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:03.626 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.624+0000 7f1964aef640 1 Processor -- start 2024-09-15T03:32:03.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.624+0000 7f1964aef640 1 -- start start 2024-09-15T03:32:03.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.624+0000 7f1964aef640 10 monclient: init 2024-09-15T03:32:03.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:32:03.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 0x7f196009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient(hunting): picked mon.noname-b con 0x7f196009c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f196009aba0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f196009af70 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196006a320 con 0x7f196009af70 2024-09-15T03:32:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1960068a20 con 0x7f196009aba0 2024-09-15T03:32:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196009a9d0 con 0x7f196009c860 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f1964aef640 10 monclient(hunting): authenticate will time out at 921.264497s 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 0x7f196009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f196009af70 0x7f196009df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:58658/0 (socket says 172.21.15.153:58658) 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195ed9d640 1 -- 172.21.15.153:0/1848254794 learned_addr learned my addr 172.21.15.153:0/1848254794 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): get_auth_request con 0x7f196009c860 auth_method 0 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient(hunting): handle_auth_done global_id 5207 payload 274 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 1 -- 172.21.15.153:0/1848254794 >> v1:172.21.15.181:6789/0 conn(0x7f196009aba0 legacy=0x7f196009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 1 -- 172.21.15.153:0/1848254794 >> v1:172.21.15.153:6789/0 conn(0x7f196009af70 legacy=0x7f196009df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 1 monclient: found mon.noname-b 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.625+0000 7f195e59c640 1 -- 172.21.15.153:0/1848254794 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960068e60 con 0x7f196009c860 2024-09-15T03:32:03.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195e59c640 1 -- 172.21.15.153:0/1848254794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 msgr2=0x7f196009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195e59c640 1 --2- 172.21.15.153:0/1848254794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 0x7f196009cc30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1950009880 tx=0x7f19500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ba9cdcfe39c617 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 1 -- 172.21.15.153:0/1848254794 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950032440 con 0x7f196009c860 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: dump: 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f1964aef640 5 monclient: authenticate success, global_id 5207 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 1 -- 172.21.15.153:0/1848254794 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500325a0 con 0x7f196009c860 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 1 -- 172.21.15.153:0/1848254794 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950032940 con 0x7f196009c860 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f1964aef640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f1964aef640 4 set_mon_vals no callback set 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.628+0000 7f195d59a640 10 monclient: dump: 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 10 monclient: shutdown 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 -- 172.21.15.153:0/1848254794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 msgr2=0x7f196009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 --2- 172.21.15.153:0/1848254794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 0x7f196009cc30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1950009880 tx=0x7f19500090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 -- 172.21.15.153:0/1848254794 shutdown_connections 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 --2- 172.21.15.153:0/1848254794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009c860 0x7f196009cc30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 -- 172.21.15.153:0/1848254794 >> 172.21.15.153:0/1848254794 conn(0x7f19600a2e40 msgr2=0x7f19600a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 -- 172.21.15.153:0/1848254794 shutdown_connections 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 1 -- 172.21.15.153:0/1848254794 wait complete. 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.629+0000 7f1964aef640 10 monclient: monmap: 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:03.646 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 Processor -- start 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 -- start start 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient: init 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:03.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 0x7f196006eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): picked mon.noname-c con 0x7f196009aba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196009af70 0x7f1960141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): picked mon.noname-a con 0x7f196009af70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f196009c860 0x7f1960132030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): picked mon.noname-b con 0x7f196009c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960120720 con 0x7f196009af70 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960120440 con 0x7f196009c860 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196006f690 con 0x7f196009aba0 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f1964aef640 10 monclient(hunting): authenticate will time out at 921.269687s 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 0x7f196006eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 0x7f196006eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58278/0 (socket says 172.21.15.153:58278) 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 1 -- 172.21.15.153:0/1160522197 learned_addr learned my addr 172.21.15.153:0/1160522197 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): get_auth_request con 0x7f196009aba0 auth_method 0 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196009af70 0x7f1960141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): get_auth_request con 0x7f196009af70 auth_method 0 2024-09-15T03:32:03.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195ed9d640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f196009c860 0x7f1960132030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195ed9d640 10 monclient(hunting): get_auth_request con 0x7f196009c860 auth_method 0 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.630+0000 7f195ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 10 monclient(hunting): handle_auth_done global_id 5210 payload 824 2024-09-15T03:32:03.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f196009c860 msgr2=0x7f1960132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f196009c860 0x7f1960132030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196009af70 msgr2=0x7f1960141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196009af70 0x7f1960141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 1 monclient: found mon.noname-c 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195e59c640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950009390 con 0x7f196009aba0 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195ed9d640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f196009c860 0x7f1960132030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.631+0000 7f195dd9b640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196009af70 0x7f1960141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:03.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f195e59c640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 msgr2=0x7f196006eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f195e59c640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 0x7f196006eb70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1950009880 tx=0x7f1950032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950031990 con 0x7f196009aba0 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: dump: 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 5 monclient: authenticate success, global_id 5210 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 10 monclient: _renew_subs 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1950031af0 con 0x7f196009aba0 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1960120160 con 0x7f196009aba0 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950031e70 con 0x7f196009aba0 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f194e7fc640 10 monclient: dump: 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.655 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f195cd99640 4 set_mon_vals no callback set 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 10 monclient: _renew_subs 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.633+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196011f200 con 0x7f196009aba0 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f1964aef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f1964aef640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 0x7f196011d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f1964aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960065e00 con 0x7f1960132570 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1950039070 con 0x7f196009aba0 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 0x7f196011d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient: get_auth_request con 0x7f1960132570 auth_method 0 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f194e7fc640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1940038740 0x7f194003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f195006ca50 con 0x7f196009aba0 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 10 monclient(hunting): handle_auth_done global_id 5213 payload 824 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195dd9b640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19601208d0 con 0x7f1960132570 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195ed9d640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1940038740 0x7f194003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.634+0000 7f195ed9d640 10 monclient: get_auth_request con 0x7f1940038740 auth_method 0 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.636+0000 7f195dd9b640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 msgr2=0x7f196011d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.636+0000 7f195dd9b640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 0x7f196011d8b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1954007fd0 tx=0x7f1954007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.636+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400dd10 con 0x7f1960132570 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.636+0000 7f194e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.636+0000 7f194e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.636+0000 7f194e7fc640 10 monclient: dump: 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f195400de70 con 0x7f1960132570 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 msgr2=0x7f196011d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 0x7f196011d8b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1954007fd0 tx=0x7f1954007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 msgr2=0x7f196011d8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f194e7fc640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1960132570 0x7f196011d8b0 secure :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1954007fd0 tx=0x7f1954007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:03.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.637+0000 7f195ed9d640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1940038740 0x7f194003abf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f196011f980 tx=0x7f1958009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f1964aef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f1964aef640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 0x7f196011e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f1964aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196011e730 con 0x7f196011ddf0 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 0x7f196011e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 10 monclient: get_auth_request con 0x7f196011ddf0 auth_method 0 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.642+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.643+0000 7f195dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.643+0000 7f195dd9b640 10 monclient(hunting): handle_auth_done global_id 5216 payload 824 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.643+0000 7f195dd9b640 1 -- 172.21.15.153:0/1160522197 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f196004bf60 con 0x7f196011ddf0 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f195dd9b640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 msgr2=0x7f196011e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f195dd9b640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 0x7f196011e1f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1954019000 tx=0x7f1954017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195401ecf0 con 0x7f196011ddf0 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 10 monclient: dump: 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f195401ee50 con 0x7f196011ddf0 2024-09-15T03:32:03.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 msgr2=0x7f196011e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 0x7f196011e1f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1954019000 tx=0x7f1954017000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 msgr2=0x7f196011e1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.645+0000 7f194e7fc640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196011ddf0 0x7f196011e1f0 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1954019000 tx=0x7f1954017000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:03.665 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.666 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.667 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:03.669 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:03.669 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:03.669 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:03.669 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:03.669 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:03.669 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:03.670 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:03.670 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:03.670 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:03.670 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:03.670 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:03.670 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.671 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:03.672 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:03.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.676 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:03.677 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.678 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.679 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.681 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:03.681 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:03.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1940038740 msgr2=0x7f194003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1940038740 0x7f194003abf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f196011f980 tx=0x7f1958009040 comp rx=0 tx=0).stop 2024-09-15T03:32:03.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 10 monclient: shutdown 2024-09-15T03:32:03.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 msgr2=0x7f196006eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 --2- 172.21.15.153:0/1160522197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f196009aba0 0x7f196006eb70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1950009880 tx=0x7f1950032000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f195ed9d640 1 -- 172.21.15.153:0/1160522197 reap_dead start 2024-09-15T03:32:03.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f195ed9d640 1 -- 172.21.15.153:0/1160522197 reap_dead start 2024-09-15T03:32:03.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 shutdown_connections 2024-09-15T03:32:03.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 >> 172.21.15.153:0/1160522197 conn(0x7f19600a2e40 msgr2=0x7f19600a2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 shutdown_connections 2024-09-15T03:32:03.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.646+0000 7f1964aef640 1 -- 172.21.15.153:0/1160522197 wait complete. 2024-09-15T03:32:03.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:03.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:32:03.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:32:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.677+0000 7f8edd64f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T03:32:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.677+0000 7f8edd64f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T03:32:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.677+0000 7f8edd64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.677+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f8ed80a13e0 con 0x7f8ed81321a0 2024-09-15T03:32:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.678+0000 7f8ebeffd640 1 -- 172.21.15.181:0/840301091 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v78) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8ec80198d0 con 0x7f8ed81321a0 2024-09-15T03:32:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.678+0000 7f8ebeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.678+0000 7f8ebeffd640 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:device 'host1' does not appear in the crush map 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec4038740 msgr2=0x7f8ec403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec4038740 0x7f8ec403abf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000bd40 tx=0x7f8ed0008040 comp rx=0 tx=0).stop 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 10 monclient: shutdown 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 msgr2=0x7f8ed811b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed811b760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800c610 tx=0x7f8ec8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 shutdown_connections 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ec4038740 0x7f8ec403abf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ed81321a0 0x7f8ed811b760 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed809f020 0x7f8ed8130a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 --2- 172.21.15.181:0/840301091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ed809ec50 0x7f8ed809a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 >> 172.21.15.181:0/840301091 conn(0x7f8ed80a5cf0 msgr2=0x7f8ed8134d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 shutdown_connections 2024-09-15T03:32:03.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.680+0000 7f8edd64f640 1 -- 172.21.15.181:0/840301091 wait complete. 2024-09-15T03:32:03.692 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:03.692 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c host1 2024-09-15T03:32:03.692 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 0 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.760+0000 7fc82739d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.760+0000 7fc82739d640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.761+0000 7fc82739d640 10 monclient: monmap: 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:03.761775+0000 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:03.761775+0000 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:03.762 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:03.763 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:03.763 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:03.763 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:03.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.761+0000 7fc82739d640 1 Processor -- start 2024-09-15T03:32:03.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.761+0000 7fc82739d640 1 -- start start 2024-09-15T03:32:03.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.761+0000 7fc82739d640 10 monclient: init 2024-09-15T03:32:03.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:32:03.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8200a3c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc820098dd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc820136c50 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc820069f00 con 0x7fc820098dd0 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8200998b0 con 0x7fc820136c50 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc820069c20 con 0x7fc8200a3c00 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82739d640 10 monclient(hunting): authenticate will time out at 928.243248s 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82593a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc820136c50 0x7fc820139030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48466/0 (socket says 172.21.15.181:48466) 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc82593a640 1 -- 172.21.15.181:0/4132969253 learned_addr learned my addr 172.21.15.181:0/4132969253 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 1 --2- 172.21.15.181:0/4132969253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): get_auth_request con 0x7fc8200a3c00 auth_method 0 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient(hunting): handle_auth_done global_id 5219 payload 274 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 1 -- 172.21.15.181:0/4132969253 >> v1:172.21.15.181:6789/0 conn(0x7fc820136c50 legacy=0x7fc820139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 1 -- 172.21.15.181:0/4132969253 >> v1:172.21.15.153:6789/0 conn(0x7fc820098dd0 legacy=0x7fc8200991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 1 monclient: found mon.noname-b 2024-09-15T03:32:03.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:03.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.762+0000 7fc825139640 1 -- 172.21.15.181:0/4132969253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc820069940 con 0x7fc8200a3c00 2024-09-15T03:32:03.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc825139640 1 -- 172.21.15.181:0/4132969253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 msgr2=0x7fc82006a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc825139640 1 --2- 172.21.15.181:0/4132969253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82006a600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009880 tx=0x7fc81c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89adcc74bbdf0ff1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 1 -- 172.21.15.181:0/4132969253 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c032440 con 0x7fc8200a3c00 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: dump: 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.774 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 1 -- 172.21.15.181:0/4132969253 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81c0325a0 con 0x7fc8200a3c00 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc82739d640 5 monclient: authenticate success, global_id 5219 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc82739d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc82739d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 1 -- 172.21.15.181:0/4132969253 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c032940 con 0x7fc8200a3c00 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc82739d640 4 set_mon_vals no callback set 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.765+0000 7fc817fff640 10 monclient: dump: 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:03.775 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 10 monclient: shutdown 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 -- 172.21.15.181:0/4132969253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 msgr2=0x7fc82006a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 --2- 172.21.15.181:0/4132969253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82006a600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009880 tx=0x7fc81c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 -- 172.21.15.181:0/4132969253 shutdown_connections 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 --2- 172.21.15.181:0/4132969253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82006a600 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 -- 172.21.15.181:0/4132969253 >> 172.21.15.181:0/4132969253 conn(0x7fc820071350 msgr2=0x7fc8200a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 -- 172.21.15.181:0/4132969253 shutdown_connections 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.766+0000 7fc82739d640 1 -- 172.21.15.181:0/4132969253 wait complete. 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient: monmap: 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:03.777 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 Processor -- start 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 -- start start 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient: init 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 0x7fc82009aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): picked mon.noname-b con 0x7fc820098dd0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82009af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8200a3c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc820136c50 0x7fc82009b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): picked mon.noname-a con 0x7fc820136c50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82011ed70 con 0x7fc820136c50 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8201367c0 con 0x7fc820098dd0 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82006ec10 con 0x7fc8200a3c00 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.767+0000 7fc82739d640 10 monclient(hunting): authenticate will time out at 928.248955s 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 0x7fc82009aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 0x7fc82009aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39624/0 (socket says 172.21.15.181:39624) 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 -- 172.21.15.181:0/2126842515 learned_addr learned my addr 172.21.15.181:0/2126842515 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): get_auth_request con 0x7fc820098dd0 auth_method 0 2024-09-15T03:32:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc820136c50 0x7fc82009b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82009af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): get_auth_request con 0x7fc820136c50 auth_method 0 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): get_auth_request con 0x7fc8200a3c00 auth_method 0 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc82593a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc824938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient(hunting): handle_auth_done global_id 44698 payload 824 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 -- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 msgr2=0x7fc82009af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82009af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 -- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc820136c50 msgr2=0x7fc82009b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc820136c50 0x7fc82009b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 monclient: found mon.noname-b 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.768+0000 7fc825139640 1 -- 172.21.15.181:0/2126842515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c009390 con 0x7fc820098dd0 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc825139640 1 -- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 msgr2=0x7fc82009aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc825139640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 0x7fc82009aa50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009880 tx=0x7fc81c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c031990 con 0x7fc820098dd0 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: dump: 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:03.782 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 5 monclient: authenticate success, global_id 44698 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 10 monclient: _renew_subs 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81c031af0 con 0x7fc820098dd0 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8200979e0 con 0x7fc820098dd0 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c03b400 con 0x7fc820098dd0 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8157fa640 10 monclient: dump: 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:03.784 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc8177fe640 4 set_mon_vals no callback set 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 10 monclient: _renew_subs 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.770+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82009c5a0 con 0x7fc820098dd0 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc81c039040 con 0x7fc820098dd0 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc82739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc82739d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc82739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8200636e0 con 0x7fc820098dd0 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc8157fa640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc804038740 0x7fc80403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fc81c06c340 con 0x7fc820098dd0 2024-09-15T03:32:03.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc824938640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc804038740 0x7fc80403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.771+0000 7fc824938640 10 monclient: get_auth_request con 0x7fc804038740 auth_method 0 2024-09-15T03:32:03.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.774+0000 7fc824938640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc804038740 0x7fc80403abf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc810007fd0 tx=0x7fc810007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.776+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc81c04b500 con 0x7fc820098dd0 2024-09-15T03:32:03.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.776+0000 7fc8157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.776+0000 7fc8157fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:03.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.784+0000 7fcc2fc57640 10 monclient: get_monmap_and_config 2024-09-15T03:32:03.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.784+0000 7fcc2fc57640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.784+0000 7fcc2fc57640 10 monclient: monmap: 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:03.784984+0000 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:03.784984+0000 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.784+0000 7fcc2fc57640 1 Processor -- start 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 1 -- start start 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient: init 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc281318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc2809ea50 0x7fcc2809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2809ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcc28136c20 0x7fcc28139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): picked mon.noname-c con 0x7fcc28136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc28131cc0 con 0x7fcc281318f0 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2809fbf0 con 0x7fcc2809ea50 2024-09-15T03:32:03.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2809fe30 con 0x7fcc28136c20 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2fc57640 10 monclient(hunting): authenticate will time out at 921.424570s 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc2809ea50 0x7fcc2809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc2809ea50 0x7fcc2809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43432/0 (socket says 172.21.15.153:43432) 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/1430134168 learned_addr learned my addr 172.21.15.153:0/1430134168 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): get_auth_request con 0x7fcc2809ea50 auth_method 0 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2e1f4640 1 --2- 172.21.15.153:0/1430134168 >> v2:172.21.15.181:3300/0 conn(0x7fcc28136c20 0x7fcc28139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2e1f4640 10 monclient(hunting): get_auth_request con 0x7fcc28136c20 auth_method 0 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2e1f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2e1f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2e1f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_done global_id 64167 payload 274 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/1430134168 >> v2:172.21.15.181:3300/0 conn(0x7fcc28136c20 msgr2=0x7fcc28139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 --2- 172.21.15.153:0/1430134168 >> v2:172.21.15.181:3300/0 conn(0x7fcc28136c20 0x7fcc28139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/1430134168 >> v1:172.21.15.181:6789/0 conn(0x7fcc281318f0 legacy=0x7fcc28131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 monclient: found mon.noname-a 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.785+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/1430134168 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280a1210 con 0x7fcc2809ea50 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/1430134168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2809ea50 msgr2=0x7fcc2809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2d1f2640 1 --2- 172.21.15.153:0/1430134168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2809ea50 0x7fcc2809ee20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcc2809c9e0 tx=0x7fcc180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68f2346046fccaf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 1 -- 172.21.15.153:0/1430134168 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18032440 con 0x7fcc2809ea50 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: dump: 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 1 -- 172.21.15.153:0/1430134168 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc180325a0 con 0x7fcc2809ea50 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2fc57640 5 monclient: authenticate success, global_id 64167 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2fc57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2fc57640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 1 -- 172.21.15.153:0/1430134168 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18032980 con 0x7fcc2809ea50 2024-09-15T03:32:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2fc57640 4 set_mon_vals no callback set 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.788+0000 7fcc2c9f1640 10 monclient: dump: 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 10 monclient: shutdown 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 -- 172.21.15.153:0/1430134168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2809ea50 msgr2=0x7fcc2809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/1430134168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2809ea50 0x7fcc2809ee20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcc2809c9e0 tx=0x7fcc180091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 -- 172.21.15.153:0/1430134168 shutdown_connections 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/1430134168 >> v2:172.21.15.181:3300/0 conn(0x7fcc28136c20 0x7fcc28139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/1430134168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2809ea50 0x7fcc2809ee20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 -- 172.21.15.153:0/1430134168 >> 172.21.15.153:0/1430134168 conn(0x7fcc28067110 msgr2=0x7fcc28067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 -- 172.21.15.153:0/1430134168 shutdown_connections 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.789+0000 7fcc2fc57640 1 -- 172.21.15.153:0/1430134168 wait complete. 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient: monmap: 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.802 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 Processor -- start 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 -- start start 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient: init 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:03.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2809ea50 0x7fcc281309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 0x7fcc28099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): picked mon.noname-a con 0x7fcc281318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28136c20 0x7fcc28099dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): picked mon.noname-b con 0x7fcc28136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2811cdc0 con 0x7fcc281318f0 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2809bf10 con 0x7fcc28136c20 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2809df80 con 0x7fcc2809ea50 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2fc57640 10 monclient(hunting): authenticate will time out at 921.429831s 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 0x7fcc28099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d9f3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2809ea50 0x7fcc281309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 0x7fcc28099890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43446/0 (socket says 172.21.15.153:43446) 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/2229432617 learned_addr learned my addr 172.21.15.153:0/2229432617 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 10 monclient(hunting): get_auth_request con 0x7fcc281318f0 auth_method 0 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d1f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d9f3640 10 monclient(hunting): get_auth_request con 0x7fcc2809ea50 auth_method 0 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d9f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d9f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.790+0000 7fcc2d9f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28136c20 0x7fcc28099dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): get_auth_request con 0x7fcc28136c20 auth_method 0 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2e1f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 10 monclient(hunting): handle_auth_done global_id 64170 payload 824 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2809ea50 msgr2=0x7fcc281309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2809ea50 0x7fcc281309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28136c20 msgr2=0x7fcc28099dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28136c20 0x7fcc28099dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 1 monclient: found mon.noname-a 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18009450 con 0x7fcc281318f0 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.791+0000 7fcc2d9f3640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2809ea50 0x7fcc281309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2d1f2640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 msgr2=0x7fcc28099890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2d1f2640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 0x7fcc28099890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcc18035040 tx=0x7fcc18032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18031a40 con 0x7fcc281318f0 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: dump: 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.808 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18031be0 con 0x7fcc281318f0 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 5 monclient: authenticate success, global_id 64170 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 10 monclient: _renew_subs 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2811cae0 con 0x7fcc281318f0 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1803d440 con 0x7fcc281318f0 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1dffb640 10 monclient: dump: 2024-09-15T03:32:03.809 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc1ffff640 4 set_mon_vals no callback set 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 10 monclient: _renew_subs 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.793+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2809b340 con 0x7fcc281318f0 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcc18039040 con 0x7fcc281318f0 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2fc57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 0x7fcc280298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc28065e00 con 0x7fcc28130f10 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc1dffb640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc0c038770 0x7fcc0c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2d9f3640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 0x7fcc280298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2d9f3640 10 monclient: get_auth_request con 0x7fcc28130f10 auth_method 0 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2d9f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2d9f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc2d9f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.794+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fcc1806bc80 con 0x7fcc281318f0 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2e1f4640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc0c038770 0x7fcc0c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2e1f4640 10 monclient: get_auth_request con 0x7fcc0c038770 auth_method 0 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_done global_id 44707 payload 824 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.795+0000 7fcc2d9f3640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc2811cfe0 con 0x7fcc28130f10 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc2d9f3640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 msgr2=0x7fcc280298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc2d9f3640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 0x7fcc280298c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400fa50 tx=0x7fcc24007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2400dda0 con 0x7fcc28130f10 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc2e1f4640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc0c038770 0x7fcc0c03ac20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcc20007640 tx=0x7fcc20009850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 10 monclient: dump: 2024-09-15T03:32:03.812 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcc24014470 con 0x7fcc28130f10 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 msgr2=0x7fcc280298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 0x7fcc280298c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400fa50 tx=0x7fcc24007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:03.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 msgr2=0x7fcc280298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:03.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.797+0000 7fcc1dffb640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28130f10 0x7fcc280298c0 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400fa50 tx=0x7fcc24007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2fc57640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 0x7fcc2802a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2fc57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc281312e0 con 0x7fcc28029e00 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 0x7fcc2802a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient: get_auth_request con 0x7fcc28029e00 auth_method 0 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.803+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.804+0000 7fcc2d9f3640 10 monclient(hunting): handle_auth_done global_id 44710 payload 824 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.804+0000 7fcc2d9f3640 1 -- 172.21.15.153:0/2229432617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcc2804bf60 con 0x7fcc28029e00 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc2d9f3640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 msgr2=0x7fcc2802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc2d9f3640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 0x7fcc2802a200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcc24019000 tx=0x7fcc2401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2401ecb0 con 0x7fcc28029e00 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 10 monclient: dump: 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.816 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fcc2401ee10 con 0x7fcc28029e00 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 msgr2=0x7fcc2802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 0x7fcc2802a200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcc24019000 tx=0x7fcc2401a000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 msgr2=0x7fcc2802a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.806+0000 7fcc1dffb640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc28029e00 0x7fcc2802a200 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcc24019000 tx=0x7fcc2401a000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:03.818 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.819 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.820 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.820 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.820 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:03.820 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:03.820 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.821 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.821 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.821 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.821 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.821 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:03.822 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:03.823 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.824 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.825 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:03.826 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.827 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.828 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.830 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:03.831 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:03.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc0c038770 msgr2=0x7fcc0c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc0c038770 0x7fcc0c03ac20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcc20007640 tx=0x7fcc20009850 comp rx=0 tx=0).stop 2024-09-15T03:32:03.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 10 monclient: shutdown 2024-09-15T03:32:03.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 msgr2=0x7fcc28099890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 --2- 172.21.15.153:0/2229432617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc281318f0 0x7fcc28099890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcc18035040 tx=0x7fcc18032000 comp rx=0 tx=0).stop 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2e1f4640 1 -- 172.21.15.153:0/2229432617 reap_dead start 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2e1f4640 1 -- 172.21.15.153:0/2229432617 reap_dead start 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 shutdown_connections 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 >> 172.21.15.153:0/2229432617 conn(0x7fcc28067110 msgr2=0x7fcc280a3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 shutdown_connections 2024-09-15T03:32:03.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.807+0000 7fcc2fc57640 1 -- 172.21.15.153:0/2229432617 wait complete. 2024-09-15T03:32:03.834 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:03.835 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:32:03.835 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:32:03.835 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:32:03.835 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:32:03.835 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:32:03.835 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:32:03.847 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:32:03.847 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:32:03.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:03.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.938+0000 7fc82739d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:03.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.938+0000 7fc82739d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:03.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.938+0000 7fc82739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.938+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc8200636e0 con 0x7fc820098dd0 2024-09-15T03:32:03.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.939+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1273 (secure 0 0 0) 0x7fc81c03f070 con 0x7fc820098dd0 2024-09-15T03:32:03.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.939+0000 7fc8157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:03.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:03.939+0000 7fc8157fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:03.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.978+0000 7fd051c68640 10 monclient: get_monmap_and_config 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.978+0000 7fd051c68640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.979+0000 7fd051c68640 10 monclient: monmap: 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:03.979725+0000 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:03.979725+0000 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:03.981 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:03.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.979+0000 7fd051c68640 1 Processor -- start 2024-09-15T03:32:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.979+0000 7fd051c68640 1 -- start start 2024-09-15T03:32:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.979+0000 7fd051c68640 10 monclient: init 2024-09-15T03:32:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06eb80 0x7fd04c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): picked mon.noname-b con 0x7fd04c06eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd04c068a20 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd04c06f490 0x7fd04c138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): picked mon.noname-c con 0x7fd04c06f490 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c069b80 con 0x7fd04c068a20 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c099f90 con 0x7fd04c06f490 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c069e00 con 0x7fd04c06eb80 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd051c68640 10 monclient(hunting): authenticate will time out at 921.619271s 2024-09-15T03:32:03.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04b7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06eb80 msgr2=0x7fd04c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06eb80 0x7fd04c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd04c068a20 0x7fd04c09c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:58660/0 (socket says 172.21.15.153:58660) 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04affd640 1 -- 172.21.15.153:0/1964131609 learned_addr learned my addr 172.21.15.153:0/1964131609 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 1 --2- 172.21.15.153:0/1964131609 >> v2:172.21.15.181:3300/0 conn(0x7fd04c06f490 0x7fd04c138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): get_auth_request con 0x7fd04c06f490 auth_method 0 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient(hunting): handle_auth_done global_id 44713 payload 274 2024-09-15T03:32:03.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 1 -- 172.21.15.153:0/1964131609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06eb80 msgr2=0x7fd04c06ef50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 1 --2- 172.21.15.153:0/1964131609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06eb80 0x7fd04c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 1 -- 172.21.15.153:0/1964131609 >> v1:172.21.15.153:6789/0 conn(0x7fd04c068a20 legacy=0x7fd04c09c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 1 monclient: found mon.noname-c 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.980+0000 7fd04bfff640 1 -- 172.21.15.153:0/1964131609 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c09cc50 con 0x7fd04c06f490 2024-09-15T03:32:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.983+0000 7fd04bfff640 1 -- 172.21.15.153:0/1964131609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c06f490 msgr2=0x7fd04c138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.983+0000 7fd04bfff640 1 --2- 172.21.15.153:0/1964131609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c06f490 0x7fd04c138ac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd044009850 tx=0x7fd0440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc963b3563bd5331 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 1 -- 172.21.15.153:0/1964131609 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044032440 con 0x7fd04c06f490 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: dump: 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 1 -- 172.21.15.153:0/1964131609 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0440325a0 con 0x7fd04c06f490 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 5 monclient: authenticate success, global_id 44713 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 4 set_mon_vals no callback set 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 1 -- 172.21.15.153:0/1964131609 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044032940 con 0x7fd04c06f490 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd04a7fc640 10 monclient: dump: 2024-09-15T03:32:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 10 monclient: shutdown 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 -- 172.21.15.153:0/1964131609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c06f490 msgr2=0x7fd04c138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 --2- 172.21.15.153:0/1964131609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c06f490 0x7fd04c138ac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd044009850 tx=0x7fd0440090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 -- 172.21.15.153:0/1964131609 shutdown_connections 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 --2- 172.21.15.153:0/1964131609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c06f490 0x7fd04c138ac0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 --2- 172.21.15.153:0/1964131609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06eb80 0x7fd04c06ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 -- 172.21.15.153:0/1964131609 >> 172.21.15.153:0/1964131609 conn(0x7fd04c067cd0 msgr2=0x7fd04c0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 -- 172.21.15.153:0/1964131609 shutdown_connections 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.984+0000 7fd051c68640 1 -- 172.21.15.153:0/1964131609 wait complete. 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.985+0000 7fd051c68640 10 monclient: build_initial_monmap 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.985+0000 7fd051c68640 10 monclient: monmap: 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.985+0000 7fd051c68640 1 Processor -- start 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.985+0000 7fd051c68640 1 -- start start 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.985+0000 7fd051c68640 10 monclient: init 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 0x7fd04c138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): picked mon.noname-b con 0x7fd04c068a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:03.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 0x7fd04c11e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): picked mon.noname-a con 0x7fd04c06eb80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06f490 0x7fd04c11e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): picked mon.noname-c con 0x7fd04c06f490 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c137f90 con 0x7fd04c06eb80 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c09cf40 con 0x7fd04c068a20 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0a20a0 con 0x7fd04c06f490 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd051c68640 10 monclient(hunting): authenticate will time out at 921.625290s 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04bfff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06f490 msgr2=0x7fd04c11e800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:03.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06f490 0x7fd04c11e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 0x7fd04c11e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 0x7fd04c11e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43482/0 (socket says 172.21.15.153:43482) 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 1 -- 172.21.15.153:0/4214773327 learned_addr learned my addr 172.21.15.153:0/4214773327 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 0x7fd04c138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): get_auth_request con 0x7fd04c06eb80 auth_method 0 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): get_auth_request con 0x7fd04c068a20 auth_method 0 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient(hunting): handle_auth_done global_id 44716 payload 824 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 -- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06f490 msgr2=0x7fd04c11e800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06f490 0x7fd04c11e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 -- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 msgr2=0x7fd04c11e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 0x7fd04c11e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:32:04.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04b7fe640 1 -- 172.21.15.153:0/4214773327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044009390 con 0x7fd04c068a20 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.986+0000 7fd04affd640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 0x7fd04c11e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.988+0000 7fd04b7fe640 1 -- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 msgr2=0x7fd04c138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.988+0000 7fd04b7fe640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 0x7fd04c138a10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00bf20 tx=0x7fd03c009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c016430 con 0x7fd04c068a20 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: dump: 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.001 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c0165b0 con 0x7fd04c068a20 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 5 monclient: authenticate success, global_id 44716 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 10 monclient: _renew_subs 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c121f40 con 0x7fd04c068a20 2024-09-15T03:32:04.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c016760 con 0x7fd04c068a20 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd0337fe640 10 monclient: dump: 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.003 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 10 monclient: _renew_subs 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c11fe50 con 0x7fd04c068a20 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.989+0000 7fd049ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd051c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd051c68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd051c68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c04bf60 con 0x7fd04c068a20 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd03c012020 con 0x7fd04c068a20 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd0337fe640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0380386f0 0x7fd03803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fd03c04c160 con 0x7fd04c068a20 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd04affd640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0380386f0 0x7fd03803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.990+0000 7fd04affd640 10 monclient: get_auth_request con 0x7fd0380386f0 auth_method 0 2024-09-15T03:32:04.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.992+0000 7fd04affd640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0380386f0 0x7fd03803aba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd04c11f8c0 tx=0x7fd040004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.994+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd03c0138e0 con 0x7fd04c068a20 2024-09-15T03:32:04.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.994+0000 7fd0337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:03.994+0000 7fd0337fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc82739d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc82739d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc82739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc8200636e0 con 0x7fc820098dd0 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc8157fa640 1 -- 172.21.15.181:0/2126842515 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1273 (secure 0 0 0) 0x7fc81c043ed0 con 0x7fc820098dd0 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc8157fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.033+0000 7fc8157fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc804038740 msgr2=0x7fc80403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc804038740 0x7fc80403abf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc810007fd0 tx=0x7fc810007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 10 monclient: shutdown 2024-09-15T03:32:04.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 msgr2=0x7fc82009aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 0x7fc82009aa50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009880 tx=0x7fc81c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 shutdown_connections 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc804038740 0x7fc80403abf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc820136c50 0x7fc82009b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8200a3c00 0x7fc82009af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 --2- 172.21.15.181:0/2126842515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc820098dd0 0x7fc82009aa50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 >> 172.21.15.181:0/2126842515 conn(0x7fc820071350 msgr2=0x7fc8200a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 shutdown_connections 2024-09-15T03:32:04.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.035+0000 7fc82739d640 1 -- 172.21.15.181:0/2126842515 wait complete. 2024-09-15T03:32:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd tree-from host1 2024-09-15T03:32:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:32:04.045 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree-from host1 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.117+0000 7fd4e4046640 10 monclient: get_monmap_and_config 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.117+0000 7fd4e4046640 10 monclient: build_initial_monmap 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.117+0000 7fd4e4046640 10 monclient: monmap: 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:04.117995+0000 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:04.117995+0000 2024-09-15T03:32:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:04.119 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.117+0000 7fd4e4046640 1 Processor -- start 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 1 -- start start 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient: init 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4dc131b40 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd4dc131f10 0x7fd4dc132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): picked mon.noname-a con 0x7fd4dc131f10 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc136e70 0x7fd4dc139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): picked mon.noname-b con 0x7fd4dc136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc06a230 con 0x7fd4dc131b40 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0a0300 con 0x7fd4dc131f10 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0a00c0 con 0x7fd4dc136e70 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e4046640 10 monclient(hunting): authenticate will time out at 928.599634s 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e25e3640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc136e70 msgr2=0x7fd4dc139250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e25e3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc136e70 0x7fd4dc139250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e1de2640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd4dc131b40 0x7fd4dc09ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48478/0 (socket says 172.21.15.181:48478) 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e1de2640 1 -- 172.21.15.181:0/284989451 learned_addr learned my addr 172.21.15.181:0/284989451 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 1 --2- 172.21.15.181:0/284989451 >> v2:172.21.15.153:3300/0 conn(0x7fd4dc131f10 0x7fd4dc132330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): get_auth_request con 0x7fd4dc131f10 auth_method 0 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.118+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_done global_id 64176 payload 274 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 1 -- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc136e70 msgr2=0x7fd4dc139250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 1 --2- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc136e70 0x7fd4dc139250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 1 -- 172.21.15.181:0/284989451 >> v1:172.21.15.181:6789/0 conn(0x7fd4dc131b40 legacy=0x7fd4dc09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:04.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 10 monclient: _finish_hunting 0 2024-09-15T03:32:04.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 1 monclient: found mon.noname-a 2024-09-15T03:32:04.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:04.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:04.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.119+0000 7fd4e15e1640 1 -- 172.21.15.181:0/284989451 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc09bee0 con 0x7fd4dc131f10 2024-09-15T03:32:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e15e1640 1 -- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 msgr2=0x7fd4dc132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e15e1640 1 --2- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc132330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009850 tx=0x7fd4cc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6109e8a89eed25fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 1 -- 172.21.15.181:0/284989451 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc032440 con 0x7fd4dc131f10 2024-09-15T03:32:04.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: dump: 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: _finish_auth 0 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 5 monclient: authenticate success, global_id 64176 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 1 -- 172.21.15.181:0/284989451 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc0325a0 con 0x7fd4dc131f10 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 4 set_mon_vals no callback set 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 1 -- 172.21.15.181:0/284989451 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc032940 con 0x7fd4dc131f10 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:04.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e0de0640 10 monclient: dump: 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 10 monclient: shutdown 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:04.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 -- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 msgr2=0x7fd4dc132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 --2- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc132330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009850 tx=0x7fd4cc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 -- 172.21.15.181:0/284989451 shutdown_connections 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 --2- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc136e70 0x7fd4dc139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 --2- 172.21.15.181:0/284989451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc132330 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 -- 172.21.15.181:0/284989451 >> 172.21.15.181:0/284989451 conn(0x7fd4dc0a5100 msgr2=0x7fd4dc0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 -- 172.21.15.181:0/284989451 shutdown_connections 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.122+0000 7fd4e4046640 1 -- 172.21.15.181:0/284989451 wait complete. 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.123+0000 7fd4e4046640 10 monclient: build_initial_monmap 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.123+0000 7fd4e4046640 10 monclient: monmap: 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:04.135 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.123+0000 7fd4e4046640 1 Processor -- start 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.123+0000 7fd4e4046640 1 -- start start 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.123+0000 7fd4e4046640 10 monclient: init 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:04.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc131b40 0x7fd4dc12eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): picked mon.noname-c con 0x7fd4dc131b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc12f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): picked mon.noname-a con 0x7fd4dc131f10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4dc136e70 0x7fd4dc097ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): picked mon.noname-b con 0x7fd4dc136e70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc09b440 con 0x7fd4dc131f10 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0a0a30 con 0x7fd4dc136e70 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc1311d0 con 0x7fd4dc131b40 2024-09-15T03:32:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e4046640 10 monclient(hunting): authenticate will time out at 928.605207s 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4dc136e70 0x7fd4dc097ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e1de2640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc131b40 msgr2=0x7fd4dc12eb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e1de2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc131b40 0x7fd4dc12eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4dc136e70 0x7fd4dc097ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39678/0 (socket says 172.21.15.181:39678) 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 -- 172.21.15.181:0/883493467 learned_addr learned my addr 172.21.15.181:0/883493467 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): get_auth_request con 0x7fd4dc136e70 auth_method 0 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc12f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): get_auth_request con 0x7fd4dc131f10 auth_method 0 2024-09-15T03:32:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient(hunting): handle_auth_done global_id 44719 payload 824 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 -- 172.21.15.181:0/883493467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc131b40 msgr2=0x7fd4dc12eb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4dc131b40 0x7fd4dc12eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 -- 172.21.15.181:0/883493467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 msgr2=0x7fd4dc12f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc12f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient: _finish_hunting 0 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 monclient: found mon.noname-b 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e25e3640 1 -- 172.21.15.181:0/883493467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc009390 con 0x7fd4dc136e70 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.124+0000 7fd4e15e1640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4dc131f10 0x7fd4dc12f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.126+0000 7fd4e25e3640 1 -- 172.21.15.181:0/883493467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4dc136e70 msgr2=0x7fd4dc097ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.126+0000 7fd4e25e3640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4dc136e70 0x7fd4dc097ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400bb10 tx=0x7fd4d40097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400cb60 con 0x7fd4dc136e70 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: dump: 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d400cce0 con 0x7fd4dc136e70 2024-09-15T03:32:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 5 monclient: authenticate success, global_id 44719 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 10 monclient: _renew_subs 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 1 -- 172.21.15.181:0/883493467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc099690 con 0x7fd4dc136e70 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4006150 con 0x7fd4dc136e70 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d1ffb640 10 monclient: dump: 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.142 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 10 monclient: _renew_subs 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4d3fff640 4 set_mon_vals no callback set 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.127+0000 7fd4e4046640 1 -- 172.21.15.181:0/883493467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc137750 con 0x7fd4dc136e70 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd4d4012020 con 0x7fd4dc136e70 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4d1ffb640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4c00386f0 0x7fd4c003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fd4d404be60 con 0x7fd4dc136e70 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4e4046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4e4046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4e4046640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4e1de2640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4c00386f0 0x7fd4c003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4e4046640 1 -- 172.21.15.181:0/883493467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc0636e0 con 0x7fd4dc136e70 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.128+0000 7fd4e1de2640 10 monclient: get_auth_request con 0x7fd4c00386f0 auth_method 0 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.130+0000 7fd4e1de2640 1 --2- 172.21.15.181:0/883493467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4c00386f0 0x7fd4c003aba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80047d0 tx=0x7fd4d8007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.132+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd4d401d8c0 con 0x7fd4dc136e70 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.132+0000 7fd4d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.132+0000 7fd4d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:04.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.209+0000 7fd051c68640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:04.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.209+0000 7fd051c68640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:04.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.209+0000 7fd051c68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.209+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd04c0638d0 con 0x7fd04c068a20 2024-09-15T03:32:04.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.210+0000 7fd0337fe640 1 -- 172.21.15.153:0/4214773327 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd03c01d300 con 0x7fd04c068a20 2024-09-15T03:32:04.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.210+0000 7fd0337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:04.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.210+0000 7fd0337fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:04.211 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:04.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0380386f0 msgr2=0x7fd03803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0380386f0 0x7fd03803aba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd04c11f8c0 tx=0x7fd040004960 comp rx=0 tx=0).stop 2024-09-15T03:32:04.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 10 monclient: shutdown 2024-09-15T03:32:04.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 msgr2=0x7fd04c138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 0x7fd04c138a10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00bf20 tx=0x7fd03c009720 comp rx=0 tx=0).stop 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 shutdown_connections 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd0380386f0 0x7fd03803aba0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c06f490 0x7fd04c11e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c06eb80 0x7fd04c11e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 --2- 172.21.15.153:0/4214773327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c068a20 0x7fd04c138a10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 >> 172.21.15.153:0/4214773327 conn(0x7fd04c067cd0 msgr2=0x7fd04c09b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 shutdown_connections 2024-09-15T03:32:04.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:04.212+0000 7fd051c68640 1 -- 172.21.15.153:0/4214773327 wait complete. 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.287+0000 7fd4e4046640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.287+0000 7fd4e4046640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.287+0000 7fd4e4046640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.287+0000 7fd4e4046640 1 -- 172.21.15.181:0/883493467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) v1 -- 0x7fd4dc064a70 con 0x7fd4dc136e70 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.288+0000 7fd4d1ffb640 1 -- 172.21.15.181:0/883493467 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v78) v1 ==== 109+0+0 (secure 0 0 0) 0x7fd4d401d300 con 0x7fd4dc136e70 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.288+0000 7fd4d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.288+0000 7fd4d1ffb640 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-15T03:32:04.289 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-15T03:32:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:32:04.292 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree-from host2 2024-09-15T03:32:04.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.363+0000 7f1a57820640 10 monclient: get_monmap_and_config 2024-09-15T03:32:04.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.363+0000 7f1a57820640 10 monclient: build_initial_monmap 2024-09-15T03:32:04.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.363+0000 7f1a57820640 10 monclient: monmap: 2024-09-15T03:32:04.364 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:04.364127+0000 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:04.364127+0000 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.363+0000 7f1a57820640 1 Processor -- start 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 1 -- start start 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient: init 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1a500677e0 0x7f1a5012f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): picked mon.noname-a con 0x7f1a500677e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a5012e590 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50138e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5012e990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:04.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5006e780 con 0x7f1a5012e590 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5009d7a0 con 0x7f1a500677e0 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5009d4c0 con 0x7f1a5012e990 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a57820640 10 monclient(hunting): authenticate will time out at 928.845701s 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a55dbd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 msgr2=0x7f1a50138e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a55dbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50138e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a54dbb640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f1a5012e590 0x7f1a5006e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48488/0 (socket says 172.21.15.181:48488) 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a54dbb640 1 -- 172.21.15.181:0/1314524657 learned_addr learned my addr 172.21.15.181:0/1314524657 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a555bc640 1 --2- 172.21.15.181:0/1314524657 >> v2:172.21.15.153:3300/0 conn(0x7f1a500677e0 0x7f1a5012f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a555bc640 10 monclient(hunting): get_auth_request con 0x7f1a500677e0 auth_method 0 2024-09-15T03:32:04.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a555bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a555bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.364+0000 7f1a555bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 10 monclient(hunting): handle_auth_done global_id 64182 payload 274 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 1 -- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 msgr2=0x7f1a50138e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 1 --2- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50138e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 1 -- 172.21.15.181:0/1314524657 >> v1:172.21.15.181:6789/0 conn(0x7f1a5012e590 legacy=0x7f1a5006e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 1 monclient: found mon.noname-a 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:04.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:04.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.365+0000 7f1a555bc640 1 -- 172.21.15.181:0/1314524657 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5009da80 con 0x7f1a500677e0 2024-09-15T03:32:04.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a555bc640 1 -- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 msgr2=0x7f1a5012f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a555bc640 1 --2- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a5012f010 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009850 tx=0x7f1a440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adc7472535206de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 1 -- 172.21.15.181:0/1314524657 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44032440 con 0x7f1a500677e0 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: dump: 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.376 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a57820640 5 monclient: authenticate success, global_id 64182 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a57820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 1 -- 172.21.15.181:0/1314524657 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a440325a0 con 0x7f1a500677e0 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a57820640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a57820640 4 set_mon_vals no callback set 2024-09-15T03:32:04.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 1 -- 172.21.15.181:0/1314524657 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44032940 con 0x7f1a500677e0 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.368+0000 7f1a43fff640 10 monclient: dump: 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 10 monclient: shutdown 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 -- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 msgr2=0x7f1a5012f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 --2- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a5012f010 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009850 tx=0x7f1a440090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 -- 172.21.15.181:0/1314524657 shutdown_connections 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 --2- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50138e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 --2- 172.21.15.181:0/1314524657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a5012f010 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 -- 172.21.15.181:0/1314524657 >> 172.21.15.181:0/1314524657 conn(0x7f1a50071370 msgr2=0x7f1a500a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 -- 172.21.15.181:0/1314524657 shutdown_connections 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 1 -- 172.21.15.181:0/1314524657 wait complete. 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 10 monclient: build_initial_monmap 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.369+0000 7f1a57820640 10 monclient: monmap: 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:04.379 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 Processor -- start 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 -- start start 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient: init 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:04.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a50141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): picked mon.noname-a con 0x7f1a500677e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 0x7f1a50141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5012e590 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50132370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5012e990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a501343f0 con 0x7f1a500677e0 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5012e100 con 0x7f1a5012e590 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5009cfa0 con 0x7f1a5012e990 2024-09-15T03:32:04.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a57820640 10 monclient(hunting): authenticate will time out at 928.851762s 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a55dbd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 msgr2=0x7f1a50132370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 0x7f1a50141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a55dbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50132370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 0x7f1a50141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39694/0 (socket says 172.21.15.181:39694) 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a50141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 1 -- 172.21.15.181:0/1399559564 learned_addr learned my addr 172.21.15.181:0/1399559564 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): get_auth_request con 0x7f1a5012e590 auth_method 0 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 10 monclient(hunting): get_auth_request con 0x7f1a500677e0 auth_method 0 2024-09-15T03:32:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a54dbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.370+0000 7f1a555bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a555bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 10 monclient(hunting): handle_auth_done global_id 44722 payload 824 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 1 -- 172.21.15.181:0/1399559564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 msgr2=0x7f1a50132370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50132370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 1 -- 172.21.15.181:0/1399559564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 msgr2=0x7f1a50141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a50141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 10 monclient: _finish_hunting 0 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 1 monclient: found mon.noname-b 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.371+0000 7f1a54dbb640 1 -- 172.21.15.181:0/1399559564 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a44009390 con 0x7f1a5012e590 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a54dbb640 1 -- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 msgr2=0x7f1a50141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a54dbb640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 0x7f1a50141d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c610 tx=0x7f1a4c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c005cb0 con 0x7f1a5012e590 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: dump: 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 5 monclient: authenticate success, global_id 44722 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c005e30 con 0x7f1a5012e590 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 10 monclient: _renew_subs 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a50133e70 con 0x7f1a5012e590 2024-09-15T03:32:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c007340 con 0x7f1a5012e590 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a417fa640 10 monclient: dump: 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a437fe640 4 set_mon_vals no callback set 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 10 monclient: _renew_subs 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.373+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a50134a40 con 0x7f1a5012e590 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1a4c01b400 con 0x7f1a5012e590 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a57820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a57820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a57820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a500636e0 con 0x7f1a5012e590 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a417fa640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1a340386f0 0x7f1a3403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:04.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f1a4c04bb20 con 0x7f1a5012e590 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a555bc640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1a340386f0 0x7f1a3403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.374+0000 7f1a555bc640 10 monclient: get_auth_request con 0x7f1a340386f0 auth_method 0 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.377+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.377+0000 7f1a555bc640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1a340386f0 0x7f1a3403aba0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1a44031000 tx=0x7f1a44032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.379+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1a4c012760 con 0x7f1a5012e590 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.379+0000 7f1a417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:04.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.379+0000 7f1a417fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:04.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.534+0000 7f1a57820640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T03:32:04.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.534+0000 7f1a57820640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T03:32:04.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.534+0000 7f1a57820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:04.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:04.534+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f1a50064b20 con 0x7f1a5012e590 2024-09-15T03:32:07.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.328+0000 7f688351d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.328+0000 7f688351d640 10 monclient: build_initial_monmap 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.328+0000 7f688351d640 10 monclient: monmap: 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:07.328860+0000 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:07.328860+0000 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:07.329 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:07.330 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:07.330 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:07.330 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:07.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.328+0000 7f688351d640 1 Processor -- start 2024-09-15T03:32:07.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 1 -- start start 2024-09-15T03:32:07.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient: init 2024-09-15T03:32:07.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:07.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T03:32:07.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f687c0981c0 0x7f687c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): picked mon.noname-c con 0x7f687c0981c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f687c131da0 0x7f687c067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): picked mon.noname-a con 0x7f687c131da0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f687c132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c098ad0 con 0x7f687c132170 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687c131bd0 con 0x7f687c131da0 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687c09ca60 con 0x7f687c0981c0 2024-09-15T03:32:07.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f688351d640 10 monclient(hunting): authenticate will time out at 924.968582s 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f687c131da0 0x7f687c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6881aba640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f687c132170 0x7f687c067650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52470/0 (socket says 172.21.15.153:52470) 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6881aba640 1 -- 172.21.15.153:0/394427262 learned_addr learned my addr 172.21.15.153:0/394427262 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): get_auth_request con 0x7f687c131da0 auth_method 0 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f68812b9640 1 --2- 172.21.15.153:0/394427262 >> v2:172.21.15.181:3300/0 conn(0x7f687c0981c0 0x7f687c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f68812b9640 10 monclient(hunting): get_auth_request con 0x7f687c0981c0 auth_method 0 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f68812b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f68812b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f68812b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 10 monclient(hunting): handle_auth_done global_id 64188 payload 274 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 1 -- 172.21.15.153:0/394427262 >> v2:172.21.15.181:3300/0 conn(0x7f687c0981c0 msgr2=0x7f687c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.329+0000 7f6880ab8640 1 --2- 172.21.15.153:0/394427262 >> v2:172.21.15.181:3300/0 conn(0x7f687c0981c0 0x7f687c098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.330+0000 7f6880ab8640 1 -- 172.21.15.153:0/394427262 >> v1:172.21.15.153:6789/0 conn(0x7f687c132170 legacy=0x7f687c067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.330+0000 7f6880ab8640 10 monclient: _finish_hunting 0 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.330+0000 7f6880ab8640 1 monclient: found mon.noname-a 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.330+0000 7f6880ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.330+0000 7f6880ab8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.330+0000 7f6880ab8640 1 -- 172.21.15.153:0/394427262 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c09cca0 con 0x7f687c131da0 2024-09-15T03:32:07.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.332+0000 7f6880ab8640 1 -- 172.21.15.153:0/394427262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 msgr2=0x7f687c067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:07.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.332+0000 7f6880ab8640 1 --2- 172.21.15.153:0/394427262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c067110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f687c066560 tx=0x7f686c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1faa1b8833433117 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:07.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.332+0000 7f6873fff640 1 -- 172.21.15.153:0/394427262 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c02ae00 con 0x7f687c131da0 2024-09-15T03:32:07.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.332+0000 7f6873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:07.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.332+0000 7f6873fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.332+0000 7f6873fff640 10 monclient: dump: 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:07.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 1 -- 172.21.15.153:0/394427262 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c032440 con 0x7f687c131da0 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 5 monclient: authenticate success, global_id 64188 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 1 -- 172.21.15.153:0/394427262 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c032820 con 0x7f687c131da0 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 4 set_mon_vals no callback set 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f6873fff640 10 monclient: dump: 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:07.344 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 10 monclient: shutdown 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:07.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 -- 172.21.15.153:0/394427262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 msgr2=0x7f687c067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 --2- 172.21.15.153:0/394427262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c067110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f687c066560 tx=0x7f686c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 -- 172.21.15.153:0/394427262 shutdown_connections 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 --2- 172.21.15.153:0/394427262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c067110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 --2- 172.21.15.153:0/394427262 >> v2:172.21.15.181:3300/0 conn(0x7f687c0981c0 0x7f687c098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 -- 172.21.15.153:0/394427262 >> 172.21.15.153:0/394427262 conn(0x7f687c0a1e30 msgr2=0x7f687c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 -- 172.21.15.153:0/394427262 shutdown_connections 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.333+0000 7f688351d640 1 -- 172.21.15.153:0/394427262 wait complete. 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.334+0000 7f688351d640 10 monclient: build_initial_monmap 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.334+0000 7f688351d640 10 monclient: monmap: 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:07.346 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:07.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:07.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:07.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.334+0000 7f688351d640 1 Processor -- start 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.334+0000 7f688351d640 1 -- start start 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.334+0000 7f688351d640 10 monclient: init 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687c0981c0 0x7f687c11dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): picked mon.noname-b con 0x7f687c0981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c11e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): picked mon.noname-a con 0x7f687c131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f687c132170 0x7f687c11e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): picked mon.noname-c con 0x7f687c132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c121540 con 0x7f687c131da0 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c098cb0 con 0x7f687c0981c0 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0a07c0 con 0x7f687c132170 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:07.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f688351d640 10 monclient(hunting): authenticate will time out at 924.974309s 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6881aba640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f687c132170 msgr2=0x7f687c11e6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c11e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6881aba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f687c132170 0x7f687c11e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c11e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52426/0 (socket says 172.21.15.153:52426) 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 -- 172.21.15.153:0/1782328891 learned_addr learned my addr 172.21.15.153:0/1782328891 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): get_auth_request con 0x7f687c131da0 auth_method 0 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:07.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687c0981c0 0x7f687c11dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): get_auth_request con 0x7f687c0981c0 auth_method 0 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f68812b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient(hunting): handle_auth_done global_id 64191 payload 824 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 -- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f687c132170 msgr2=0x7f687c11e6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f687c132170 0x7f687c11e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 -- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687c0981c0 msgr2=0x7f687c11dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687c0981c0 0x7f687c11dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient: _finish_hunting 0 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 monclient: found mon.noname-a 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.335+0000 7f6880ab8640 1 -- 172.21.15.153:0/1782328891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c009390 con 0x7f687c131da0 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.337+0000 7f6880ab8640 1 -- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 msgr2=0x7f687c11e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.337+0000 7f6880ab8640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c11e180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f686c035040 tx=0x7f686c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c031d40 con 0x7f687c131da0 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: dump: 2024-09-15T03:32:07.352 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:07.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c03d440 con 0x7f687c131da0 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 5 monclient: authenticate success, global_id 64191 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c031ea0 con 0x7f687c131da0 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 10 monclient: dump: 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:07.354 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68737fe640 4 set_mon_vals no callback set 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 10 monclient: _renew_subs 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c121260 con 0x7f687c131da0 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 10 monclient: _renew_subs 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:07.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c120300 con 0x7f687c131da0 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.338+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f686c039040 con 0x7f687c131da0 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f688351d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f688351d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f688351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c04bf60 con 0x7f687c131da0 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f68717fa640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6860038720 0x7f686003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f686c06c170 con 0x7f687c131da0 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f68812b9640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6860038720 0x7f686003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:07.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.339+0000 7f68812b9640 10 monclient: get_auth_request con 0x7f6860038720 auth_method 0 2024-09-15T03:32:07.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.341+0000 7f68812b9640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6860038720 0x7f686003abd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6878007ba0 tx=0x7f6878008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:07.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.343+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f686c030d40 con 0x7f687c131da0 2024-09-15T03:32:07.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.343+0000 7f68717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:07.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.343+0000 7f68717fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:07.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:07.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:07.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:07.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:07.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:07.371+0000 7f1a3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:07.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.560+0000 7f688351d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:07.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.560+0000 7f688351d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:07.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.560+0000 7f688351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:07.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.560+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f687c098ee0 con 0x7f687c131da0 2024-09-15T03:32:07.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.560+0000 7f68717fa640 1 -- 172.21.15.153:0/1782328891 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f686c03e020 con 0x7f687c131da0 2024-09-15T03:32:07.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.561+0000 7f68717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:07.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.561+0000 7f68717fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:07.562 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:07.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6860038720 msgr2=0x7f686003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:07.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6860038720 0x7f686003abd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6878007ba0 tx=0x7f6878008040 comp rx=0 tx=0).stop 2024-09-15T03:32:07.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 10 monclient: shutdown 2024-09-15T03:32:07.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:07.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 msgr2=0x7f687c11e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:07.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c11e180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f686c035040 tx=0x7f686c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 shutdown_connections 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6860038720 0x7f686003abd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f687c132170 0x7f687c11e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687c131da0 0x7f687c11e180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 --2- 172.21.15.153:0/1782328891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687c0981c0 0x7f687c11dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 >> 172.21.15.153:0/1782328891 conn(0x7f687c0a1e30 msgr2=0x7f687c096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 shutdown_connections 2024-09-15T03:32:07.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:07.563+0000 7f688351d640 1 -- 172.21.15.153:0/1782328891 wait complete. 2024-09-15T03:32:08.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:08.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:08.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:08.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:09.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:09.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:09.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:09.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:10.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:10.372+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:10.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:10.372+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:10.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.679+0000 7fa8a339d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.679+0000 7fa8a339d640 10 monclient: build_initial_monmap 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.679+0000 7fa8a339d640 10 monclient: monmap: 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:10.679970+0000 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:10.679970+0000 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:10.680 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:10.681 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:10.681 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:10.681 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:10.681 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:10.681 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.679+0000 7fa8a339d640 1 Processor -- start 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 1 -- start start 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient: init 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa89c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:10.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa89c09ea50 0x7fa89c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): picked mon.noname-a con 0x7fa89c09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): picked mon.noname-b con 0x7fa89c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c132640 con 0x7fa89c1318f0 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89c131cc0 con 0x7fa89c09ea50 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:10.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c09fbf0 con 0x7fa89c136c20 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a339d640 10 monclient(hunting): authenticate will time out at 928.319681s 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a193a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 msgr2=0x7fa89c139000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a193a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa89c09ea50 0x7fa89c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa89c09ea50 0x7fa89c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52440/0 (socket says 172.21.15.153:52440) 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 1 -- 172.21.15.153:0/1489157688 learned_addr learned my addr 172.21.15.153:0/1489157688 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): get_auth_request con 0x7fa89c09ea50 auth_method 0 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:10.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.680+0000 7fa8a0938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 10 monclient(hunting): handle_auth_done global_id 64194 payload 274 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 1 -- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 msgr2=0x7fa89c139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 1 --2- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 1 -- 172.21.15.153:0/1489157688 >> v1:172.21.15.181:6789/0 conn(0x7fa89c1318f0 legacy=0x7fa89c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 10 monclient: _finish_hunting 0 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 1 monclient: found mon.noname-a 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:10.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.681+0000 7fa8a0938640 1 -- 172.21.15.153:0/1489157688 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c09fe30 con 0x7fa89c09ea50 2024-09-15T03:32:10.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.683+0000 7fa8a0938640 1 -- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c09ea50 msgr2=0x7fa89c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:10.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.683+0000 7fa8a0938640 1 --2- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c09ea50 0x7fa89c09ee20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa890009910 tx=0x7fa8900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845bc36b3788913d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 1 -- 172.21.15.153:0/1489157688 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890032440 con 0x7fa89c09ea50 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: dump: 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: _finish_auth 0 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 5 monclient: authenticate success, global_id 64194 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 1 -- 172.21.15.153:0/1489157688 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8900325a0 con 0x7fa89c09ea50 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 1 -- 172.21.15.153:0/1489157688 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890032940 con 0x7fa89c09ea50 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 4 set_mon_vals no callback set 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:10.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa89bfff640 10 monclient: dump: 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 10 monclient: shutdown 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 -- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c09ea50 msgr2=0x7fa89c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 --2- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c09ea50 0x7fa89c09ee20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa890009910 tx=0x7fa8900091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 -- 172.21.15.153:0/1489157688 shutdown_connections 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 --2- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 --2- 172.21.15.153:0/1489157688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c09ea50 0x7fa89c09ee20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 -- 172.21.15.153:0/1489157688 >> 172.21.15.153:0/1489157688 conn(0x7fa89c067110 msgr2=0x7fa89c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 -- 172.21.15.153:0/1489157688 shutdown_connections 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.684+0000 7fa8a339d640 1 -- 172.21.15.153:0/1489157688 wait complete. 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.685+0000 7fa8a339d640 10 monclient: build_initial_monmap 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.685+0000 7fa8a339d640 10 monclient: monmap: 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:10.696 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.685+0000 7fa8a339d640 1 Processor -- start 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.685+0000 7fa8a339d640 1 -- start start 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.685+0000 7fa8a339d640 10 monclient: init 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:10.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa89c09ea50 0x7fa89c097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): picked mon.noname-b con 0x7fa89c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): picked mon.noname-a con 0x7fa89c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c098080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): picked mon.noname-c con 0x7fa89c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c09bf10 con 0x7fa89c1318f0 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c09db70 con 0x7fa89c09ea50 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c09ac40 con 0x7fa89c136c20 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a339d640 10 monclient(hunting): authenticate will time out at 928.325404s 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a193a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 msgr2=0x7fa89c098080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a193a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c098080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52448/0 (socket says 172.21.15.153:52448) 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 -- 172.21.15.153:0/2425024709 learned_addr learned my addr 172.21.15.153:0/2425024709 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): get_auth_request con 0x7fa89c1318f0 auth_method 0 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:10.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a1139640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa89c09ea50 0x7fa89c097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a1139640 10 monclient(hunting): get_auth_request con 0x7fa89c09ea50 auth_method 0 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a1139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a1139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a1139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient(hunting): handle_auth_done global_id 64197 payload 824 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 -- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 msgr2=0x7fa89c098080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c098080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 -- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa89c09ea50 msgr2=0x7fa89c097600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa89c09ea50 0x7fa89c097600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient: _finish_hunting 0 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 monclient: found mon.noname-a 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.686+0000 7fa8a0938640 1 -- 172.21.15.153:0/2425024709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa890009450 con 0x7fa89c1318f0 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a0938640 1 -- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 msgr2=0x7fa89c097b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a0938640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa890035040 tx=0x7fa890032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89003c960 con 0x7fa89c1318f0 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: dump: 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:10.702 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 5 monclient: authenticate success, global_id 64197 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89003cac0 con 0x7fa89c1318f0 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 10 monclient: _renew_subs 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c098e30 con 0x7fa89c1318f0 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89003ce80 con 0x7fa89c1318f0 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8997fa640 10 monclient: dump: 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:10.703 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 10 monclient: _renew_subs 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa89b7fe640 4 set_mon_vals no callback set 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.689+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c09aa10 con 0x7fa89c1318f0 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa890039040 con 0x7fa89c1318f0 2024-09-15T03:32:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8a339d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8a339d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8a339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c09add0 con 0x7fa89c1318f0 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8997fa640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa888038720 0x7fa88803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fa89006c060 con 0x7fa89c1318f0 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8a1139640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa888038720 0x7fa88803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.690+0000 7fa8a1139640 10 monclient: get_auth_request con 0x7fa888038720 auth_method 0 2024-09-15T03:32:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.693+0000 7fa8a1139640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa888038720 0x7fa88803abd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8940037d0 tx=0x7fa894009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:10.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:10.695+0000 7fa8a0938640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa890035040 tx=0x7fa890032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fa89009de40 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:32:11.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:11.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:11.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:11.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:11.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.523+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa89009de40 con 0x7fa89c1318f0 2024-09-15T03:32:11.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.523+0000 7fa8997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:11.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.523+0000 7fa8997fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.739+0000 7fa8a339d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.739+0000 7fa8a339d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.739+0000 7fa8a339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.740+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa89c04bf60 con 0x7fa89c1318f0 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.740+0000 7fa8997fa640 1 -- 172.21.15.153:0/2425024709 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa890040080 con 0x7fa89c1318f0 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.740+0000 7fa8997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.740+0000 7fa8997fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa888038720 msgr2=0x7fa88803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa888038720 0x7fa88803abd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8940037d0 tx=0x7fa894009040 comp rx=0 tx=0).stop 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 10 monclient: shutdown 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 msgr2=0x7fa89c097b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa890035040 tx=0x7fa890032000 comp rx=0 tx=0).stop 2024-09-15T03:32:11.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 shutdown_connections 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa888038720 0x7fa88803abd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c136c20 0x7fa89c098080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c1318f0 0x7fa89c097b40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 --2- 172.21.15.153:0/2425024709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa89c09ea50 0x7fa89c097600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 >> 172.21.15.153:0/2425024709 conn(0x7fa89c067110 msgr2=0x7fa89c0a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 shutdown_connections 2024-09-15T03:32:11.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:11.742+0000 7fa8a339d640 1 -- 172.21.15.153:0/2425024709 wait complete. 2024-09-15T03:32:12.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:12.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:12.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:12.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:13.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:13.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:13.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:13.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:14.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:14.370+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:14.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:14.370+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:14.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:14.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.857+0000 7f03d6f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:14.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.857+0000 7f03d6f9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:14.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.857+0000 7f03d6f9d640 10 monclient: monmap: 2024-09-15T03:32:14.858 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:14.858 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:14.858312+0000 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:14.858312+0000 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:14.859 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:14.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 Processor -- start 2024-09-15T03:32:14.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 -- start start 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient: init 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03d009c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 0x7f03d009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f03d009e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f03d0136cb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d009ec40 con 0x7f03d009c860 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d009a9d0 con 0x7f03d0136cb0 2024-09-15T03:32:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d006a320 con 0x7f03d009e840 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03d6f9d640 10 monclient(hunting): authenticate will time out at 932.497945s 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.858+0000 7f03cffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 msgr2=0x7f03d009aba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 0x7f03d009aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:39038/0 (socket says 172.21.15.153:39038) 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 learned_addr learned my addr 172.21.15.153:0/2393098203 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 10 monclient(hunting): get_auth_request con 0x7f03d0136cb0 auth_method 0 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:14.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d4d39640 1 --1- 172.21.15.153:0/2393098203 >> v1:172.21.15.181:6789/0 conn(0x7f03d009c860 0x7f03d009cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:52718/0 (socket says 172.21.15.153:52718) 2024-09-15T03:32:14.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:14.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:14.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:14.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.859+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:14.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:14.861+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v1:172.21.15.181:6789/0 conn(0x7f03d009c860 legacy=0x7f03d009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:15.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03cffff640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 msgr2=0x7f03d009aba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:15.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03cffff640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 0x7f03d009aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:15.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:15.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 10 monclient(hunting): get_auth_request con 0x7f03d0136cb0 auth_method 0 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:15.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.059+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:15.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:15.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:15.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:15.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:15.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03cffff640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 msgr2=0x7f03d009aba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:15.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03cffff640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 0x7f03d009aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:32:15.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:15.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 10 monclient(hunting): get_auth_request con 0x7f03d0136cb0 auth_method 0 2024-09-15T03:32:15.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:15.460+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:32:16.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03cffff640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 msgr2=0x7f03d009aba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:16.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03cffff640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 0x7f03d009aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03d553a640 10 monclient(hunting): get_auth_request con 0x7f03d0136cb0 auth_method 0 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03d553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03d553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.261+0000 7f03d553a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.262+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.262+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.262+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:16.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:16.262+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:32:16.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:16.372+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:16.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:16.372+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:17.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:17.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:17.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:17.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:17.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:17.371+0000 7f1a3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:17.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient(hunting): tick 2024-09-15T03:32:17.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:32:17.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:32:17.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 msgr2=0x7f03d009aba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03d009e840 0x7f03d009aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 msgr2=0x7f03d0139090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.181:3300/0 conn(0x7f03d0136cb0 0x7f03d0139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 -- 172.21.15.153:0/2393098203 >> v1:172.21.15.181:6789/0 conn(0x7f03d009c860 legacy=0x7f03d009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient: _add_conns ranks=[3,0] 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f03b4003720 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03b4006260 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 -- 172.21.15.153:0/2393098203 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d0068a20 con 0x7f03b4006260 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 1 -- 172.21.15.153:0/2393098203 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d009a9d0 con 0x7f03b4003720 2024-09-15T03:32:17.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03ce7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 10 monclient(hunting): get_auth_request con 0x7f03b4003720 auth_method 0 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.858+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:17.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:17.860+0000 7f03cffff640 1 -- 172.21.15.153:0/2393098203 >> v1:172.21.15.153:6789/0 conn(0x7f03b4006260 legacy=0x7f03b4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 10 monclient(hunting): get_auth_request con 0x7f03b4003720 auth_method 0 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:18.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:18.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:18.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.059+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:18.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.371+0000 7f1a3bfff640 10 monclient: tick 2024-09-15T03:32:18.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.371+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 10 monclient(hunting): get_auth_request con 0x7f03b4003720 auth_method 0 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.460+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:32:18.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.587+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v78) v1 ==== 80+0+177 (secure 0 0 0) 0x7f1a4c01a360 con 0x7f1a5012e590 2024-09-15T03:32:18.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.587+0000 7f1a417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T03:32:18.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.587+0000 7f1a417fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:18.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973983902 0 0) 0x7f03d0068a20 con 0x7f03b4006260 2024-09-15T03:32:18.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): my global_id is 64200 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c00034e0 con 0x7f03b4006260 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093885652 0 0) 0x7f03c00034e0 con 0x7f03b4006260 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 msgr2=0x7f03b4005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 --2- 172.21.15.153:0/2393098203 >> v2:172.21.15.153:3300/0 conn(0x7f03b4003720 0x7f03b4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b400b060 con 0x7f03b4006260 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03d6f9d640 5 monclient: authenticate success, global_id 64200 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f03bc004380 con 0x7f03b4006260 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: dump: 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.595 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 >> v1:172.21.15.153:6789/0 conn(0x7f03b4006260 legacy=0x7f03b4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 reap_dead start 2024-09-15T03:32:18.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c00039d0 0x7f03c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): picked mon.b con 0x7f03c00039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03c0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): picked mon.c con 0x7f03c0004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 0x7f03c000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): picked mon.a con 0x7f03c0005180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d009a9d0 con 0x7f03c0005180 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c000c120 con 0x7f03c00039d0 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c000c2e0 con 0x7f03c0004780 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:18.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03d6f9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cffff640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 msgr2=0x7f03c0004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03cffff640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03c0004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.589+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 0x7f03c000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): get_auth_request con 0x7f03c0005180 auth_method 0 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c00039d0 0x7f03c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): get_auth_request con 0x7f03c00039d0 auth_method 0 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d4d39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient(hunting): handle_auth_done global_id 64200 payload 293 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 msgr2=0x7f03c0004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03c0004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c00039d0 msgr2=0x7f03c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c00039d0 0x7f03c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 1 monclient: found mon.a 2024-09-15T03:32:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00034e0 con 0x7f03c0005180 2024-09-15T03:32:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.590+0000 7f03d553a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03d553a640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 msgr2=0x7f03c000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03d553a640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 0x7f03c000b9d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03d009f760 tx=0x7f03c8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837bc6a7d8717554 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8030480 con 0x7f03c0005180 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: dump: 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.607 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03d6f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c802d030 con 0x7f03c0005180 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03d6f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 1 -- 172.21.15.153:0/2393098203 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8030830 con 0x7f03c0005180 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03d6f9d640 4 set_mon_vals no callback set 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.593+0000 7f03cf7fe640 10 monclient: dump: 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.608 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 10 monclient: shutdown 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 -- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 msgr2=0x7f03c000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 0x7f03c000b9d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03d009f760 tx=0x7f03c8009040 comp rx=0 tx=0).stop 2024-09-15T03:32:18.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 -- 172.21.15.153:0/2393098203 shutdown_connections 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c0005180 0x7f03c000b9d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03c0004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/2393098203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c00039d0 0x7f03c0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 -- 172.21.15.153:0/2393098203 >> 172.21.15.153:0/2393098203 conn(0x7f03d00a2e40 msgr2=0x7f03d00a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 -- 172.21.15.153:0/2393098203 shutdown_connections 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.594+0000 7f03d6f9d640 1 -- 172.21.15.153:0/2393098203 wait complete. 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.595+0000 7f03d6f9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.595+0000 7f03d6f9d640 10 monclient: monmap: 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.595+0000 7f03d6f9d640 1 Processor -- start 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.595+0000 7f03d6f9d640 1 -- start start 2024-09-15T03:32:18.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.595+0000 7f03d6f9d640 10 monclient: init 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 0x7f03d011e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f03c00039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03d011e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f03c0004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c0005180 0x7f03d011eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f03c0005180 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00a24b0 con 0x7f03c00039d0 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0135d90 con 0x7f03c0005180 2024-09-15T03:32:18.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0099cc0 con 0x7f03c0004780 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d6f9d640 10 monclient(hunting): authenticate will time out at 936.236329s 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03cffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 msgr2=0x7f03d011e760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03d011e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 0x7f03d011e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 0x7f03d011e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33730/0 (socket says 172.21.15.153:33730) 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 -- 172.21.15.153:0/404882485 learned_addr learned my addr 172.21.15.153:0/404882485 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): get_auth_request con 0x7f03c00039d0 auth_method 0 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c0005180 0x7f03d011eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): get_auth_request con 0x7f03c0005180 auth_method 0 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d553a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient(hunting): handle_auth_done global_id 64203 payload 824 2024-09-15T03:32:18.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 -- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 msgr2=0x7f03d011e760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03d011e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 -- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c0005180 msgr2=0x7f03d011eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c0005180 0x7f03d011eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 monclient: found mon.noname-a 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.596+0000 7f03d4d39640 1 -- 172.21.15.153:0/404882485 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d011fc00 con 0x7f03c00039d0 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d4d39640 1 -- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 msgr2=0x7f03d011e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d4d39640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 0x7f03d011e220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03c4000c00 tx=0x7f03c4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4006530 con 0x7f03c00039d0 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: dump: 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c40047a0 con 0x7f03c00039d0 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 5 monclient: authenticate success, global_id 64203 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 10 monclient: _renew_subs 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d01216f0 con 0x7f03c00039d0 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4004b40 con 0x7f03c00039d0 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ccff9640 10 monclient: dump: 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.617 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 10 monclient: _renew_subs 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03ceffd640 4 set_mon_vals no callback set 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.599+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d0120790 con 0x7f03c00039d0 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f03c4017430 con 0x7f03c00039d0 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03d6f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03d6f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03d6f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d004bf60 con 0x7f03c00039d0 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03ccff9640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03b4039110 0x7f03b403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.600+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f03c404ca30 con 0x7f03c00039d0 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.601+0000 7f03cffff640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03b4039110 0x7f03b403b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.601+0000 7f03cffff640 10 monclient: get_auth_request con 0x7f03b4039110 auth_method 0 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.603+0000 7f03cffff640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03b4039110 0x7f03b403b5c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004530 tx=0x7f03bc0043d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.605+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f03c4014b20 con 0x7f03c00039d0 2024-09-15T03:32:18.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.605+0000 7f03ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.605+0000 7f03ccff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:18.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.685+0000 7f1a57820640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.685+0000 7f1a57820640 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.685+0000 7f1a57820640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.685+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f1a50136b30 con 0x7f1a5012e590 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.685+0000 7f1a417fa640 1 -- 172.21.15.181:0/1399559564 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v78) v1 ==== 80+0+177 (secure 0 0 0) 0x7f1a4c01a540 con 0x7f1a5012e590 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.686+0000 7f1a417fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.686+0000 7f1a417fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stdout:-16 1.00000 host host2 2024-09-15T03:32:18.688 INFO:tasks.workunit.client.0.smithi181.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1a340386f0 msgr2=0x7f1a3403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1a340386f0 0x7f1a3403aba0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1a44031000 tx=0x7f1a44032000 comp rx=0 tx=0).stop 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 10 monclient: shutdown 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 msgr2=0x7f1a50141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 0x7f1a50141d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c610 tx=0x7f1a4c004960 comp rx=0 tx=0).stop 2024-09-15T03:32:18.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 shutdown_connections 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1a340386f0 0x7f1a3403aba0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5012e990 0x7f1a50132370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a5012e590 0x7f1a50141d90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 --2- 172.21.15.181:0/1399559564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a500677e0 0x7f1a50141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 >> 172.21.15.181:0/1399559564 conn(0x7f1a50071370 msgr2=0x7f1a500a0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 shutdown_connections 2024-09-15T03:32:18.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.688+0000 7f1a57820640 1 -- 172.21.15.181:0/1399559564 wait complete. 2024-09-15T03:32:18.699 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-15T03:32:18.699 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:32:18.699 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree-from osd.9 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.770+0000 7fe16eb9d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.770+0000 7fe16eb9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.770+0000 7fe16eb9d640 10 monclient: monmap: 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:18.772053+0000 2024-09-15T03:32:18.772 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:18.772053+0000 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:18.773 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.770+0000 7fe16eb9d640 1 Processor -- start 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 -- start start 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient: init 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe168131b40 0x7fe16809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe168131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16809f240 0x7fe168131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe16809f240 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe168132450 0x7fe168068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe168132450 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe168132890 con 0x7fe168131b40 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16806a230 con 0x7fe168132450 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680a0300 con 0x7fe16809f240 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16eb9d640 10 monclient(hunting): authenticate will time out at 943.253522s 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe167fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16809f240 msgr2=0x7fe168131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe167fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16809f240 0x7fe168131f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe168132450 0x7fe168068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe168132450 0x7fe168068e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48986/0 (socket says 172.21.15.181:48986) 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 -- 172.21.15.181:0/1004353936 learned_addr learned my addr 172.21.15.181:0/1004353936 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): get_auth_request con 0x7fe168132450 auth_method 0 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 1 --2- 172.21.15.181:0/1004353936 >> v2:172.21.15.153:3300/0 conn(0x7fe168131b40 0x7fe16809ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): get_auth_request con 0x7fe168131b40 auth_method 0 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16c939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient(hunting): handle_auth_done global_id 44737 payload 274 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 -- 172.21.15.181:0/1004353936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16809f240 msgr2=0x7fe168131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 --2- 172.21.15.181:0/1004353936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16809f240 0x7fe168131f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 -- 172.21.15.181:0/1004353936 >> v2:172.21.15.153:3300/0 conn(0x7fe168131b40 msgr2=0x7fe16809ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 --2- 172.21.15.181:0/1004353936 >> v2:172.21.15.153:3300/0 conn(0x7fe168131b40 0x7fe16809ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 monclient: found mon.noname-c 2024-09-15T03:32:18.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.771+0000 7fe16d13a640 1 -- 172.21.15.181:0/1004353936 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680a00c0 con 0x7fe168132450 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe16d13a640 1 -- 172.21.15.181:0/1004353936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 msgr2=0x7fe168068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe16d13a640 1 --2- 172.21.15.181:0/1004353936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168068e00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe16809cc90 tx=0x7fe160009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fe909ca195fe6d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 1 -- 172.21.15.181:0/1004353936 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160007610 con 0x7fe168132450 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: dump: 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.782 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 1 -- 172.21.15.181:0/1004353936 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16002d050 con 0x7fe168132450 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe16eb9d640 5 monclient: authenticate success, global_id 44737 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe16eb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:18.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe16eb9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 1 -- 172.21.15.181:0/1004353936 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1600079a0 con 0x7fe168132450 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe16eb9d640 4 set_mon_vals no callback set 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.774+0000 7fe1677fe640 10 monclient: dump: 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.784 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 10 monclient: shutdown 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 -- 172.21.15.181:0/1004353936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 msgr2=0x7fe168068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 --2- 172.21.15.181:0/1004353936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168068e00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe16809cc90 tx=0x7fe160009840 comp rx=0 tx=0).stop 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 -- 172.21.15.181:0/1004353936 shutdown_connections 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 --2- 172.21.15.181:0/1004353936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168068e00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 --2- 172.21.15.181:0/1004353936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16809f240 0x7fe168131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 --2- 172.21.15.181:0/1004353936 >> v2:172.21.15.153:3300/0 conn(0x7fe168131b40 0x7fe16809ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 -- 172.21.15.181:0/1004353936 >> 172.21.15.181:0/1004353936 conn(0x7fe1680a5100 msgr2=0x7fe1680a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 -- 172.21.15.181:0/1004353936 shutdown_connections 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.775+0000 7fe16eb9d640 1 -- 172.21.15.181:0/1004353936 wait complete. 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.776+0000 7fe16eb9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.776+0000 7fe16eb9d640 10 monclient: monmap: 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:18.786 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:18.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.776+0000 7fe16eb9d640 1 Processor -- start 2024-09-15T03:32:18.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.776+0000 7fe16eb9d640 1 -- start start 2024-09-15T03:32:18.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.776+0000 7fe16eb9d640 10 monclient: init 2024-09-15T03:32:18.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.776+0000 7fe16eb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:18.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:18.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16809f240 0x7fe168140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe16809f240 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168131b40 0x7fe16812ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe168131b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168098a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe168132450 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16809a5e0 con 0x7fe16809f240 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680a0a90 con 0x7fe168132450 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16811c340 con 0x7fe168131b40 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16eb9d640 10 monclient(hunting): authenticate will time out at 943.259110s 2024-09-15T03:32:18.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe167fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168131b40 msgr2=0x7fe16812ec60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe167fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168131b40 0x7fe16812ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168098a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16809f240 0x7fe168140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168098a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48992/0 (socket says 172.21.15.181:48992) 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 -- 172.21.15.181:0/2356801542 learned_addr learned my addr 172.21.15.181:0/2356801542 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): get_auth_request con 0x7fe168132450 auth_method 0 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): get_auth_request con 0x7fe16809f240 auth_method 0 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16c939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient(hunting): handle_auth_done global_id 44740 payload 824 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 -- 172.21.15.181:0/2356801542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168131b40 msgr2=0x7fe16812ec60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 --2- 172.21.15.181:0/2356801542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168131b40 0x7fe16812ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 -- 172.21.15.181:0/2356801542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16809f240 msgr2=0x7fe168140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 --2- 172.21.15.181:0/2356801542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16809f240 0x7fe168140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 monclient: found mon.noname-b 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.777+0000 7fe16d13a640 1 -- 172.21.15.181:0/2356801542 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160009ae0 con 0x7fe168132450 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16d13a640 1 -- 172.21.15.181:0/2356801542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 msgr2=0x7fe168098a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16d13a640 1 --2- 172.21.15.181:0/2356801542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe168132450 0x7fe168098a90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe160009f50 tx=0x7fe160036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160032ce0 con 0x7fe168132450 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: dump: 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160032e40 con 0x7fe168132450 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 5 monclient: authenticate success, global_id 44740 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 10 monclient: _renew_subs 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 1 -- 172.21.15.181:0/2356801542 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16811bde0 con 0x7fe168132450 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16003b640 con 0x7fe168132450 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe164ff9640 10 monclient: dump: 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.797 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe166ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 10 monclient: _renew_subs 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.779+0000 7fe16eb9d640 1 -- 172.21.15.181:0/2356801542 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1681414d0 con 0x7fe168132450 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe160003020 con 0x7fe168132450 2024-09-15T03:32:18.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe164ff9640 1 --2- 172.21.15.181:0/2356801542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c0386f0 0x7fe14c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fe16006ca50 con 0x7fe168132450 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe16eb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe16eb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe16eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe16c939640 1 --2- 172.21.15.181:0/2356801542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c0386f0 0x7fe14c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe16eb9d640 1 -- 172.21.15.181:0/2356801542 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1680636e0 con 0x7fe168132450 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.780+0000 7fe16c939640 10 monclient: get_auth_request con 0x7fe14c0386f0 auth_method 0 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.783+0000 7fe16c939640 1 --2- 172.21.15.181:0/2356801542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c0386f0 0x7fe14c03aba0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009d20 tx=0x7fe15c0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.785+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe160039ae0 con 0x7fe168132450 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.785+0000 7fe164ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.785+0000 7fe164ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.825+0000 7f03d6f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.825+0000 7f03d6f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.825+0000 7f03d6f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.825+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f03d006cf80 con 0x7f03c00039d0 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.826+0000 7f03ccff9640 1 -- 172.21.15.153:0/404882485 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f03c401b030 con 0x7f03c00039d0 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.826+0000 7f03ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.826+0000 7f03ccff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:18.828 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:18.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03b4039110 msgr2=0x7f03b403b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03b4039110 0x7f03b403b5c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004530 tx=0x7f03bc0043d0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 10 monclient: shutdown 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 msgr2=0x7f03d011e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 0x7f03d011e220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03c4000c00 tx=0x7f03c4007f90 comp rx=0 tx=0).stop 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 shutdown_connections 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03b4039110 0x7f03b403b5c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c0005180 0x7f03d011eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03c0004780 0x7f03d011e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 --2- 172.21.15.153:0/404882485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c00039d0 0x7f03d011e220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 >> 172.21.15.153:0/404882485 conn(0x7f03d00a2e40 msgr2=0x7f03d00a1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 shutdown_connections 2024-09-15T03:32:18.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.828+0000 7f03d6f9d640 1 -- 172.21.15.153:0/404882485 wait complete. 2024-09-15T03:32:18.841 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:32:18.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:32:18.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.932+0000 7f6f79b1d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.932+0000 7f6f79b1d640 10 monclient: build_initial_monmap 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.933+0000 7f6f79b1d640 10 monclient: monmap: 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:18.934736+0000 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:18.934736+0000 2024-09-15T03:32:18.935 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:18.936 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.933+0000 7f6f79b1d640 1 Processor -- start 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.933+0000 7f6f79b1d640 1 -- start start 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.933+0000 7f6f79b1d640 10 monclient: init 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f7409a7f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6f7409abc0 0x7f6f7409afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7409abc0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6f74134a30 0x7f6f74136e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f74134a30 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f740a2050 con 0x7f6f7409a7f0 2024-09-15T03:32:18.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7409c890 con 0x7f6f74134a30 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7409cfb0 con 0x7f6f7409abc0 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f79b1d640 10 monclient(hunting): authenticate will time out at 936.574230s 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6f74134a30 0x7f6f74136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f737fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6f7409a7f0 0x7f6f7409c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45136/0 (socket says 172.21.15.153:45136) 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6f74134a30 0x7f6f74136e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33732/0 (socket says 172.21.15.153:33732) 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f737fe640 1 -- 172.21.15.153:0/1226748635 learned_addr learned my addr 172.21.15.153:0/1226748635 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): get_auth_request con 0x7f6f74134a30 auth_method 0 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 1 --2- 172.21.15.153:0/1226748635 >> v2:172.21.15.181:3300/0 conn(0x7f6f7409abc0 0x7f6f7409afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): get_auth_request con 0x7f6f7409abc0 auth_method 0 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient(hunting): handle_auth_done global_id 64212 payload 274 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 -- 172.21.15.153:0/1226748635 >> v2:172.21.15.181:3300/0 conn(0x7f6f7409abc0 msgr2=0x7f6f7409afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:18.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 --2- 172.21.15.153:0/1226748635 >> v2:172.21.15.181:3300/0 conn(0x7f6f7409abc0 0x7f6f7409afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 -- 172.21.15.153:0/1226748635 >> v1:172.21.15.153:6789/0 conn(0x7f6f7409a7f0 legacy=0x7f6f7409c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 monclient: found mon.noname-a 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.934+0000 7f6f73fff640 1 -- 172.21.15.153:0/1226748635 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7409d290 con 0x7f6f74134a30 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f73fff640 1 -- 172.21.15.153:0/1226748635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f74134a30 msgr2=0x7f6f74136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f73fff640 1 --2- 172.21.15.153:0/1226748635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f74134a30 0x7f6f74136e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6f7409b550 tx=0x7f6f6c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7e3e352c56de03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 1 -- 172.21.15.153:0/1226748635 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c02ae00 con 0x7f6f74134a30 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: dump: 2024-09-15T03:32:18.944 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 1 -- 172.21.15.153:0/1226748635 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c032440 con 0x7f6f74134a30 2024-09-15T03:32:18.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 1 -- 172.21.15.153:0/1226748635 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0327e0 con 0x7f6f74134a30 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f79b1d640 5 monclient: authenticate success, global_id 64212 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f79b1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f79b1d640 4 set_mon_vals no callback set 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.937+0000 7f6f727fc640 10 monclient: dump: 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 10 monclient: shutdown 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:18.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 -- 172.21.15.153:0/1226748635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f74134a30 msgr2=0x7f6f74136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/1226748635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f74134a30 0x7f6f74136e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6f7409b550 tx=0x7f6f6c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 -- 172.21.15.153:0/1226748635 shutdown_connections 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/1226748635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f74134a30 0x7f6f74136e10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/1226748635 >> v2:172.21.15.181:3300/0 conn(0x7f6f7409abc0 0x7f6f7409afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 -- 172.21.15.153:0/1226748635 >> 172.21.15.153:0/1226748635 conn(0x7f6f74067b50 msgr2=0x7f6f740a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 -- 172.21.15.153:0/1226748635 shutdown_connections 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 1 -- 172.21.15.153:0/1226748635 wait complete. 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 10 monclient: build_initial_monmap 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.938+0000 7f6f79b1d640 10 monclient: monmap: 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:18.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 Processor -- start 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 -- start start 2024-09-15T03:32:18.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient: init 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.940+0000 7fe16eb9d640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.940+0000 7fe16eb9d640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.940+0000 7fe16eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.940+0000 7fe16eb9d640 1 -- 172.21.15.181:0/2356801542 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) v1 -- 0x7fe1680a3120 con 0x7fe168132450 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.940+0000 7fe164ff9640 1 -- 172.21.15.181:0/2356801542 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v78) v1 ==== 103+0+0 (secure 0 0 0) 0x7fe16003f070 con 0x7fe168132450 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.941+0000 7fe164ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-15T03:32:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:18.941+0000 7fe164ff9640 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-15T03:32:18.952 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-15T03:32:18.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:32:18.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush rm bar 2024-09-15T03:32:18.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:32:18.952 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm bar 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f7409a7f0 0x7f6f7412b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7409a7f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 0x7f6f7412b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7409abc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f74134a30 0x7f6f7412bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f74134a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7412e180 con 0x7f6f7409abc0 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74135120 con 0x7f6f7409a7f0 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7412e3b0 con 0x7f6f74134a30 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f79b1d640 10 monclient(hunting): authenticate will time out at 936.579772s 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f73fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f74134a30 msgr2=0x7f6f7412bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f74134a30 0x7f6f7412bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 0x7f6f7412b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 0x7f6f7412b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33736/0 (socket says 172.21.15.153:33736) 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 1 -- 172.21.15.153:0/153536713 learned_addr learned my addr 172.21.15.153:0/153536713 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:18.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): get_auth_request con 0x7f6f7409abc0 auth_method 0 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f737fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f7409a7f0 0x7f6f7412b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f737fe640 10 monclient(hunting): get_auth_request con 0x7f6f7409a7f0 auth_method 0 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.939+0000 7f6f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 10 monclient(hunting): handle_auth_done global_id 64215 payload 824 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f74134a30 msgr2=0x7f6f7412bb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f74134a30 0x7f6f7412bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f7409a7f0 msgr2=0x7f6f7412b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f7409a7f0 0x7f6f7412b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 1 monclient: found mon.noname-a 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.940+0000 7f6f72ffd640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c009390 con 0x7f6f7409abc0 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f72ffd640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 msgr2=0x7f6f7412b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f72ffd640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 0x7f6f7412b5f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800ea20 tx=0x7f6f680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68009600 con 0x7f6f7409abc0 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: dump: 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 5 monclient: authenticate success, global_id 64215 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 10 monclient: _renew_subs 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7412e6c0 con 0x7f6f7409abc0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 10 monclient: _renew_subs 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7412ec80 con 0x7f6f7409abc0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6800cc20 con 0x7f6f7409abc0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68013470 con 0x7f6f7409abc0 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f71ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.942+0000 7f6f5b7fe640 10 monclient: dump: 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6f680137e0 con 0x7f6f7409abc0 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f79b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 0x7f6f54004ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f54007c30 con 0x7f6f54002850 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 0x7f6f54004ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient: get_auth_request con 0x7f6f54002850 auth_method 0 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f5b7fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f600386f0 0x7f6f6003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f6f6804c580 con 0x7f6f7409abc0 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 10 monclient(hunting): handle_auth_done global_id 64218 payload 824 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f737fe640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f54007df0 con 0x7f6f54002850 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f73fff640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f600386f0 0x7f6f6003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.943+0000 7f6f73fff640 10 monclient: get_auth_request con 0x7f6f600386f0 auth_method 0 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.945+0000 7f6f737fe640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 msgr2=0x7f6f54004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f737fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 0x7f6f54004ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f640079a0 tx=0x7f6f64005330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64007a90 con 0x7f6f54002850 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 10 monclient: dump: 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f73fff640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f600386f0 0x7f6f6003aba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6f74071310 tx=0x7f6f6c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6f6400fd50 con 0x7f6f54002850 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 msgr2=0x7f6f54004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 0x7f6f54004ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f640079a0 tx=0x7f6f64005330 comp rx=0 tx=0).stop 2024-09-15T03:32:18.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 msgr2=0x7f6f54004ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.946+0000 7f6f5b7fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54002850 0x7f6f54004ce0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f640079a0 tx=0x7f6f64005330 comp rx=0 tx=0).stop 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.951+0000 7f6f79b1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 0x7f6f540070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f79b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540076e0 con 0x7f6f54006c20 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 0x7f6f540070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient: get_auth_request con 0x7f6f54006c20 auth_method 0 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:18.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 10 monclient(hunting): handle_auth_done global_id 64221 payload 824 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.952+0000 7f6f737fe640 1 -- 172.21.15.153:0/153536713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6f540078a0 con 0x7f6f54006c20 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f737fe640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 msgr2=0x7f6f540070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f737fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 0x7f6f540070a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6f64017000 tx=0x7f6f64018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64005e20 con 0x7f6f54006c20 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 10 monclient: dump: 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f6f64021880 con 0x7f6f54006c20 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 msgr2=0x7f6f540070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 0x7f6f540070a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6f64017000 tx=0x7f6f64018000 comp rx=0 tx=0).stop 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 msgr2=0x7f6f540070a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:18.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.954+0000 7f6f5b7fe640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54006c20 0x7f6f540070a0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6f64017000 tx=0x7f6f64018000 comp rx=0 tx=0).stop 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:18.972 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:18.973 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:18.974 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:18.975 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:18.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:18.977 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:18.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:18.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.980 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:18.981 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:18.982 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.983 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:18.984 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f600386f0 msgr2=0x7f6f6003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f600386f0 0x7f6f6003aba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6f74071310 tx=0x7f6f6c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 10 monclient: shutdown 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 msgr2=0x7f6f7412b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 --2- 172.21.15.153:0/153536713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7409abc0 0x7f6f7412b5f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800ea20 tx=0x7f6f680090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f73fff640 1 -- 172.21.15.153:0/153536713 reap_dead start 2024-09-15T03:32:18.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f73fff640 1 -- 172.21.15.153:0/153536713 reap_dead start 2024-09-15T03:32:18.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 shutdown_connections 2024-09-15T03:32:18.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 >> 172.21.15.153:0/153536713 conn(0x7f6f74067b50 msgr2=0x7f6f740a0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:18.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 shutdown_connections 2024-09-15T03:32:18.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:18.955+0000 7f6f79b1d640 1 -- 172.21.15.153:0/153536713 wait complete. 2024-09-15T03:32:18.987 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:32:18.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:32:18.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:32:19.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.017+0000 7fefbe79d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:19.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.017+0000 7fefbe79d640 10 monclient: build_initial_monmap 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.017+0000 7fefbe79d640 10 monclient: monmap: 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:19.019457+0000 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:19.019457+0000 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:19.020 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 1 Processor -- start 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 1 -- start start 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 10 monclient: init 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fefb806e950 0x7fefb809b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.018+0000 7fefbe79d640 10 monclient(hunting): picked mon.noname-a con 0x7fefb806e950 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefb809be20 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefb809c220 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb8097b40 con 0x7fefb809c220 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb809c6c0 con 0x7fefb809be20 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb8098470 con 0x7fefb806e950 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefbe79d640 10 monclient(hunting): authenticate will time out at 943.501047s 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefaffff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fefb809be20 0x7fefb80996f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47032/0 (socket says 172.21.15.181:47032) 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefaffff640 1 -- 172.21.15.181:0/4208612457 learned_addr learned my addr 172.21.15.181:0/4208612457 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 1 --2- 172.21.15.181:0/4208612457 >> v2:172.21.15.153:3300/0 conn(0x7fefb806e950 0x7fefb809b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): get_auth_request con 0x7fefb806e950 auth_method 0 2024-09-15T03:32:19.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient(hunting): handle_auth_done global_id 64224 payload 274 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 1 -- 172.21.15.181:0/4208612457 >> v1:172.21.15.181:6789/0 conn(0x7fefb809be20 legacy=0x7fefb80996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 1 -- 172.21.15.181:0/4208612457 >> v1:172.21.15.153:6789/0 conn(0x7fefb809c220 legacy=0x7fefb8099e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 1 monclient: found mon.noname-a 2024-09-15T03:32:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:19.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:19.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.019+0000 7fefb7fff640 1 -- 172.21.15.181:0/4208612457 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8098750 con 0x7fefb806e950 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb7fff640 1 -- 172.21.15.181:0/4208612457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb806e950 msgr2=0x7fefb809b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb7fff640 1 --2- 172.21.15.181:0/4208612457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb806e950 0x7fefb809b8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009880 tx=0x7fefa80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecefc62d2fd0577 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 1 -- 172.21.15.181:0/4208612457 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8032440 con 0x7fefb806e950 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: dump: 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 1 -- 172.21.15.181:0/4208612457 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa80325a0 con 0x7fefb806e950 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 1 -- 172.21.15.181:0/4208612457 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8032940 con 0x7fefb806e950 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefb77fe640 10 monclient: dump: 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefbe79d640 5 monclient: authenticate success, global_id 64224 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefbe79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.022+0000 7fefbe79d640 4 set_mon_vals no callback set 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 10 monclient: shutdown 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 -- 172.21.15.181:0/4208612457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb806e950 msgr2=0x7fefb809b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 --2- 172.21.15.181:0/4208612457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb806e950 0x7fefb809b8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009880 tx=0x7fefa80090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 -- 172.21.15.181:0/4208612457 shutdown_connections 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 --2- 172.21.15.181:0/4208612457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb806e950 0x7fefb809b8e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 -- 172.21.15.181:0/4208612457 >> 172.21.15.181:0/4208612457 conn(0x7fefb8071100 msgr2=0x7fefb80a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 -- 172.21.15.181:0/4208612457 shutdown_connections 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 1 -- 172.21.15.181:0/4208612457 wait complete. 2024-09-15T03:32:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 10 monclient: build_initial_monmap 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.023+0000 7fefbe79d640 10 monclient: monmap: 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 Processor -- start 2024-09-15T03:32:19.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 -- start start 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient: init 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefb806e950 0x7fefb8135270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): picked mon.noname-b con 0x7fefb806e950 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefb809be20 0x7fefb81357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): picked mon.noname-c con 0x7fefb809be20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb809c220 0x7fefb8135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): picked mon.noname-a con 0x7fefb809c220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81373a0 con 0x7fefb809c220 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81388e0 con 0x7fefb806e950 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8136e20 con 0x7fefb809be20 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbe79d640 10 monclient(hunting): authenticate will time out at 943.506653s 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefaffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefb809be20 msgr2=0x7fefb81357b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefb806e950 0x7fefb8135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefaffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefb809be20 0x7fefb81357b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefb806e950 0x7fefb8135270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48994/0 (socket says 172.21.15.181:48994) 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 -- 172.21.15.181:0/2692637159 learned_addr learned my addr 172.21.15.181:0/2692637159 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): get_auth_request con 0x7fefb806e950 auth_method 0 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb809c220 0x7fefb8135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): get_auth_request con 0x7fefb809c220 auth_method 0 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefbcd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 10 monclient(hunting): handle_auth_done global_id 44749 payload 824 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 -- 172.21.15.181:0/2692637159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefb809be20 msgr2=0x7fefb81357b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefb809be20 0x7fefb81357b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 -- 172.21.15.181:0/2692637159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb809c220 msgr2=0x7fefb8135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.024+0000 7fefb7fff640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefb809c220 0x7fefb8135cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.025+0000 7fefb7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.025+0000 7fefb7fff640 1 monclient: found mon.noname-b 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.025+0000 7fefb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.025+0000 7fefb7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.025+0000 7fefb7fff640 1 -- 172.21.15.181:0/2692637159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8009390 con 0x7fefb806e950 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb7fff640 1 -- 172.21.15.181:0/2692637159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefb806e950 msgr2=0x7fefb8135270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb7fff640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefb806e950 0x7fefb8135270 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fefa80098e0 tx=0x7fefa8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8031910 con 0x7fefb806e950 2024-09-15T03:32:19.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: dump: 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 5 monclient: authenticate success, global_id 44749 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa8031a70 con 0x7fefb806e950 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 10 monclient: _renew_subs 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 1 -- 172.21.15.181:0/2692637159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb809ab20 con 0x7fefb806e950 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8031df0 con 0x7fefb806e950 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb4ff9640 10 monclient: dump: 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefb6ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 10 monclient: _renew_subs 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.027+0000 7fefbe79d640 1 -- 172.21.15.181:0/2692637159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb8137a90 con 0x7fefb806e950 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fefa8039070 con 0x7fefb806e950 2024-09-15T03:32:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefbe79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefbe79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefbe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefb4ff9640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef980386f0 0x7fef9803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefbe79d640 1 -- 172.21.15.181:0/2692637159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb80008d0 con 0x7fefb806e950 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fefa806be40 con 0x7fefb806e950 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefaffff640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef980386f0 0x7fef9803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.028+0000 7fefaffff640 10 monclient: get_auth_request con 0x7fef980386f0 auth_method 0 2024-09-15T03:32:19.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.030+0000 7fefaffff640 1 --2- 172.21.15.181:0/2692637159 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef980386f0 0x7fef9803aba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fefa00047d0 tx=0x7fefa0007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.032+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fefa8037070 con 0x7fefb806e950 2024-09-15T03:32:19.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.032+0000 7fefb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.032+0000 7fefb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:19.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.080+0000 7f99e931d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:19.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.080+0000 7f99e931d640 10 monclient: build_initial_monmap 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.080+0000 7f99e931d640 10 monclient: monmap: 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:19.081981+0000 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:19.081981+0000 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:19.083 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.080+0000 7f99e931d640 1 Processor -- start 2024-09-15T03:32:19.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 1 -- start start 2024-09-15T03:32:19.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient: init 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99e41318f0 0x7f99e4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): picked mon.noname-a con 0x7f99e41318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e409ea50 0x7f99e409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): picked mon.noname-b con 0x7f99e409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99e4068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e4131cc0 con 0x7f99e4068d90 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e409fbf0 con 0x7f99e41318f0 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e409fe30 con 0x7f99e409ea50 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e931d640 10 monclient(hunting): authenticate will time out at 936.721501s 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99dbfff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e409ea50 msgr2=0x7f99e409ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e409ea50 0x7f99e409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99e41318f0 0x7f99e4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99e41318f0 0x7f99e4131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33760/0 (socket says 172.21.15.153:33760) 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 -- 172.21.15.153:0/3956960818 learned_addr learned my addr 172.21.15.153:0/3956960818 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): get_auth_request con 0x7f99e41318f0 auth_method 0 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_done global_id 64230 payload 274 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 -- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e409ea50 msgr2=0x7f99e409ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e409ea50 0x7f99e409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 -- 172.21.15.153:0/3956960818 >> v1:172.21.15.181:6789/0 conn(0x7f99e4068d90 legacy=0x7f99e40691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 monclient: found mon.noname-a 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:19.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.081+0000 7f99e2ffd640 1 -- 172.21.15.153:0/3956960818 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40a1210 con 0x7f99e41318f0 2024-09-15T03:32:19.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.084+0000 7f99e2ffd640 1 -- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e41318f0 msgr2=0x7f99e4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.084+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e41318f0 0x7f99e4131f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009910 tx=0x7f99d40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e039df886e33a5d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.084+0000 7f99e27fc640 1 -- 172.21.15.153:0/3956960818 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4032440 con 0x7f99e41318f0 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.084+0000 7f99e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: dump: 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.096 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 5 monclient: authenticate success, global_id 64230 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 1 -- 172.21.15.153:0/3956960818 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d40325a0 con 0x7f99e41318f0 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 4 set_mon_vals no callback set 2024-09-15T03:32:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 1 -- 172.21.15.153:0/3956960818 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4032980 con 0x7f99e41318f0 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e27fc640 10 monclient: dump: 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 10 monclient: shutdown 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 -- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e41318f0 msgr2=0x7f99e4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 --2- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e41318f0 0x7f99e4131f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009910 tx=0x7f99d40091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 -- 172.21.15.153:0/3956960818 shutdown_connections 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 --2- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e409ea50 0x7f99e409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 --2- 172.21.15.153:0/3956960818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e41318f0 0x7f99e4131f30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 -- 172.21.15.153:0/3956960818 >> 172.21.15.153:0/3956960818 conn(0x7f99e4067110 msgr2=0x7f99e4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 -- 172.21.15.153:0/3956960818 shutdown_connections 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.085+0000 7f99e931d640 1 -- 172.21.15.153:0/3956960818 wait complete. 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient: build_initial_monmap 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient: monmap: 2024-09-15T03:32:19.099 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 Processor -- start 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 -- start start 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient: init 2024-09-15T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e4068d90 0x7f99e409a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): picked mon.noname-b con 0x7f99e4068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 0x7f99e4130930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): picked mon.noname-a con 0x7f99e409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e41318f0 0x7f99e4130e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): picked mon.noname-c con 0x7f99e41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e4141e10 con 0x7f99e409ea50 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e411de30 con 0x7f99e4068d90 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e411e110 con 0x7f99e41318f0 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e931d640 10 monclient(hunting): authenticate will time out at 936.726901s 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e37fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e41318f0 msgr2=0x7f99e4130e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.086+0000 7f99e37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e41318f0 0x7f99e4130e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 0x7f99e4130930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e4068d90 0x7f99e409a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 0x7f99e4130930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33770/0 (socket says 172.21.15.153:33770) 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 -- 172.21.15.153:0/2211438988 learned_addr learned my addr 172.21.15.153:0/2211438988 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): get_auth_request con 0x7f99e409ea50 auth_method 0 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e4068d90 0x7f99e409a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:60802/0 (socket says 172.21.15.153:60802) 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): get_auth_request con 0x7f99e4068d90 auth_method 0 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient(hunting): handle_auth_done global_id 64233 payload 824 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e41318f0 msgr2=0x7f99e4130e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e41318f0 0x7f99e4130e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e4068d90 msgr2=0x7f99e409a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e4068d90 0x7f99e409a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 monclient: found mon.noname-a 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.087+0000 7f99dbfff640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4009450 con 0x7f99e409ea50 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99dbfff640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 msgr2=0x7f99e4130930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99dbfff640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 0x7f99e4130930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00b600 tx=0x7f99cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc015070 con 0x7f99e409ea50 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: dump: 2024-09-15T03:32:19.104 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc003d90 con 0x7f99e409ea50 2024-09-15T03:32:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 5 monclient: authenticate success, global_id 64233 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 10 monclient: _renew_subs 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e41363d0 con 0x7f99e409ea50 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc006380 con 0x7f99e409ea50 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99daffd640 10 monclient: dump: 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.106 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 10 monclient: _renew_subs 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e1ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.089+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e411db50 con 0x7f99e409ea50 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e931d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f99cc010440 con 0x7f99e409ea50 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e931d640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 0x7f99e411baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e4065e00 con 0x7f99e411b720 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99daffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99c40386f0 0x7f99c403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 0x7f99e411baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient: get_auth_request con 0x7f99e411b720 auth_method 0 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f99cc04bae0 con 0x7f99e409ea50 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e37fe640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99c40386f0 0x7f99c403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e37fe640 10 monclient: get_auth_request con 0x7f99c40386f0 auth_method 0 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.090+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_done global_id 44755 payload 824 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.091+0000 7f99e2ffd640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99e404bf60 con 0x7f99e411b720 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99e2ffd640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 msgr2=0x7f99e411baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 0x7f99e411baf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99e4064050 tx=0x7f99d4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99e37fe640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99c40386f0 0x7f99c403aba0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99dc007fd0 tx=0x7f99dc007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d403c930 con 0x7f99e411b720 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 10 monclient: dump: 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.109 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f99d403ca90 con 0x7f99e411b720 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 msgr2=0x7f99e411baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 0x7f99e411baf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99e4064050 tx=0x7f99d4032000 comp rx=0 tx=0).stop 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 msgr2=0x7f99e411baf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:19.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.093+0000 7f99daffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411b720 0x7f99e411baf0 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99e4064050 tx=0x7f99d4032000 comp rx=0 tx=0).stop 2024-09-15T03:32:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e931d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e931d640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 0x7f99e4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e931d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e409b950 con 0x7f99e411c030 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 0x7f99e4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient: get_auth_request con 0x7f99e411c030 auth_method 0 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 10 monclient(hunting): handle_auth_done global_id 44758 payload 824 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.099+0000 7f99e2ffd640 1 -- 172.21.15.153:0/2211438988 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f99e40638d0 con 0x7f99e411c030 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e2ffd640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 msgr2=0x7f99e4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e2ffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 0x7f99e4096a60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99e411c970 tx=0x7f99d403e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d403ce70 con 0x7f99e411c030 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 10 monclient: dump: 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.114 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f99d403b7f0 con 0x7f99e411c030 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 msgr2=0x7f99e4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 0x7f99e4096a60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99e411c970 tx=0x7f99d403e040 comp rx=0 tx=0).stop 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 msgr2=0x7f99e4096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99daffd640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99e411c030 0x7f99e4096a60 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99e411c970 tx=0x7f99d403e040 comp rx=0 tx=0).stop 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:19.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 40, 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:19.116 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:19.117 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:19.119 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:19.120 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:19.120 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:19.120 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:19.120 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.121 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:19.122 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:19.123 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.124 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:19.125 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:19.126 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:19.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:19.129 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:19.129 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99c40386f0 msgr2=0x7f99c403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99c40386f0 0x7f99c403aba0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99dc007fd0 tx=0x7f99dc007df0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 10 monclient: shutdown 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 msgr2=0x7f99e4130930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 1 --2- 172.21.15.153:0/2211438988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e409ea50 0x7f99e4130930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00b600 tx=0x7f99cc0091a0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e37fe640 1 -- 172.21.15.153:0/2211438988 reap_dead start 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e37fe640 1 -- 172.21.15.153:0/2211438988 reap_dead start 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 shutdown_connections 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.102+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 >> 172.21.15.153:0/2211438988 conn(0x7f99e4067110 msgr2=0x7f99e40a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.103+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 shutdown_connections 2024-09-15T03:32:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:19.103+0000 7f99e931d640 1 -- 172.21.15.153:0/2211438988 wait complete. 2024-09-15T03:32:19.131 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:32:19.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:32:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.192+0000 7fefbe79d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.192+0000 7fefbe79d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.192+0000 7fefbe79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.192+0000 7fefbe79d640 1 -- 172.21.15.181:0/2692637159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7fefb80008d0 con 0x7fefb806e950 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.193+0000 7fefb4ff9640 1 -- 172.21.15.181:0/2692637159 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v78) v1 ==== 99+0+0 (secure 0 0 0) 0x7fefa80443b0 con 0x7fefb806e950 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.193+0000 7fefb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.193+0000 7fefb4ff9640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-15T03:32:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-15T03:32:19.199 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:32:19.199 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush unlink host2 2024-09-15T03:32:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.271+0000 7fe715a89640 10 monclient: get_monmap_and_config 2024-09-15T03:32:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.271+0000 7fe715a89640 10 monclient: build_initial_monmap 2024-09-15T03:32:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.271+0000 7fe715a89640 10 monclient: monmap: 2024-09-15T03:32:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:19.273132+0000 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:19.273132+0000 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:19.274 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 Processor -- start 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 -- start start 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient: init 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe71009ec50 0x7fe710131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): picked mon.noname-c con 0x7fe71009ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7101321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe71009f020 0x7fe710069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): picked mon.noname-a con 0x7fe71009f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe710067420 con 0x7fe7101321a0 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100a0480 con 0x7fe71009f020 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100a0200 con 0x7fe71009ec50 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe715a89640 10 monclient(hunting): authenticate will time out at 943.754600s 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe71009ec50 0x7fe710131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70effd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe7101321a0 0x7fe710068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47046/0 (socket says 172.21.15.181:47046) 2024-09-15T03:32:19.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70effd640 1 -- 172.21.15.181:0/1150900852 learned_addr learned my addr 172.21.15.181:0/1150900852 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): get_auth_request con 0x7fe71009ec50 auth_method 0 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 10 monclient(hunting): handle_auth_done global_id 44761 payload 274 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 1 -- 172.21.15.181:0/1150900852 >> v2:172.21.15.153:3300/0 conn(0x7fe71009f020 msgr2=0x7fe710069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70ffff640 1 --2- 172.21.15.181:0/1150900852 >> v2:172.21.15.153:3300/0 conn(0x7fe71009f020 0x7fe710069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.272+0000 7fe70f7fe640 1 --2- 172.21.15.181:0/1150900852 >> v2:172.21.15.153:3300/0 conn(0x7fe71009f020 0x7fe710069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70f7fe640 1 -- 172.21.15.181:0/1150900852 >> v1:172.21.15.181:6789/0 conn(0x7fe7101321a0 legacy=0x7fe710068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:19.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70f7fe640 1 monclient: found mon.noname-c 2024-09-15T03:32:19.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:19.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70f7fe640 1 -- 172.21.15.181:0/1150900852 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71009fd80 con 0x7fe71009ec50 2024-09-15T03:32:19.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70ffff640 10 monclient: get_auth_request con 0x7fe71009f020 auth_method 0 2024-09-15T03:32:19.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.273+0000 7fe70ffff640 1 --2- 172.21.15.181:0/1150900852 >> v2:172.21.15.153:3300/0 conn(0x7fe71009f020 0x7fe710069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70f7fe640 1 -- 172.21.15.181:0/1150900852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe71009ec50 msgr2=0x7fe710131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70f7fe640 1 --2- 172.21.15.181:0/1150900852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe71009ec50 0x7fe710131c60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe7100719d0 tx=0x7fe7000090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b216c00b1ee7b3e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 1 -- 172.21.15.181:0/1150900852 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700032440 con 0x7fe71009ec50 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: dump: 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe715a89640 5 monclient: authenticate success, global_id 44761 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe715a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 1 -- 172.21.15.181:0/1150900852 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7000325a0 con 0x7fe71009ec50 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 1 -- 172.21.15.181:0/1150900852 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700032940 con 0x7fe71009ec50 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe70e7fc640 10 monclient: dump: 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.288 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe715a89640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.275+0000 7fe715a89640 4 set_mon_vals no callback set 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 10 monclient: shutdown 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 -- 172.21.15.181:0/1150900852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe71009ec50 msgr2=0x7fe710131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 --2- 172.21.15.181:0/1150900852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe71009ec50 0x7fe710131c60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe7100719d0 tx=0x7fe7000090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 -- 172.21.15.181:0/1150900852 shutdown_connections 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 --2- 172.21.15.181:0/1150900852 >> v2:172.21.15.153:3300/0 conn(0x7fe71009f020 0x7fe710069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 --2- 172.21.15.181:0/1150900852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe71009ec50 0x7fe710131c60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 -- 172.21.15.181:0/1150900852 >> 172.21.15.181:0/1150900852 conn(0x7fe7100a5cf0 msgr2=0x7fe7100a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 -- 172.21.15.181:0/1150900852 shutdown_connections 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.276+0000 7fe715a89640 1 -- 172.21.15.181:0/1150900852 wait complete. 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.277+0000 7fe715a89640 10 monclient: build_initial_monmap 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.277+0000 7fe715a89640 10 monclient: monmap: 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:19.290 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.277+0000 7fe715a89640 1 Processor -- start 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.277+0000 7fe715a89640 1 -- start start 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.277+0000 7fe715a89640 10 monclient: init 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.277+0000 7fe715a89640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe71009ec50 0x7fe71009acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): picked mon.noname-c con 0x7fe71009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe71009f020 0x7fe710130d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): picked mon.noname-a con 0x7fe71009f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 0x7fe71011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): picked mon.noname-b con 0x7fe7101321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71011c900 con 0x7fe71009f020 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71009e140 con 0x7fe7101321a0 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71009c020 con 0x7fe71009ec50 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe715a89640 10 monclient(hunting): authenticate will time out at 943.760088s 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70f7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe71009ec50 msgr2=0x7fe71009acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe71009ec50 0x7fe71009acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 0x7fe71011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 0x7fe71011b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49014/0 (socket says 172.21.15.181:49014) 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 -- 172.21.15.181:0/1146638233 learned_addr learned my addr 172.21.15.181:0/1146638233 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): get_auth_request con 0x7fe7101321a0 auth_method 0 2024-09-15T03:32:19.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe71009f020 0x7fe710130d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): get_auth_request con 0x7fe71009f020 auth_method 0 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient(hunting): handle_auth_done global_id 44764 payload 824 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 -- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe71009ec50 msgr2=0x7fe71009acc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe71009ec50 0x7fe71009acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 -- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe71009f020 msgr2=0x7fe710130d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe71009f020 0x7fe710130d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 monclient: found mon.noname-b 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70ffff640 1 -- 172.21.15.181:0/1146638233 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe700009390 con 0x7fe7101321a0 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.278+0000 7fe70effd640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe71009f020 0x7fe710130d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe70ffff640 1 -- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 msgr2=0x7fe71011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe70ffff640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 0x7fe71011b760 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe70800a380 tx=0x7fe708007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70800d860 con 0x7fe7101321a0 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: dump: 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.295 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 5 monclient: authenticate success, global_id 44764 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70800d9c0 con 0x7fe7101321a0 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 10 monclient: _renew_subs 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71011cf50 con 0x7fe7101321a0 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70800dd80 con 0x7fe7101321a0 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe6f77fe640 10 monclient: dump: 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:19.296 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 10 monclient: _renew_subs 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71011deb0 con 0x7fe7101321a0 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.280+0000 7fe70dffb640 4 set_mon_vals no callback set 2024-09-15T03:32:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe70801e3b0 con 0x7fe7101321a0 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe6f77fe640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe6fc0386f0 0x7fe6fc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fe70804c970 con 0x7fe7101321a0 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe715a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe715a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe715a89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7100008d0 con 0x7fe7101321a0 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.281+0000 7fe70f7fe640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe6fc0386f0 0x7fe6fc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.282+0000 7fe70f7fe640 10 monclient: get_auth_request con 0x7fe6fc0386f0 auth_method 0 2024-09-15T03:32:19.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.284+0000 7fe70f7fe640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe6fc0386f0 0x7fe6fc03aba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7101325a0 tx=0x7fe700032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:19.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.286+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe70801a020 con 0x7fe7101321a0 2024-09-15T03:32:19.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.286+0000 7fe6f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:19.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.286+0000 7fe6f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:19.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.447+0000 7fe715a89640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T03:32:19.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.447+0000 7fe715a89640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T03:32:19.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.447+0000 7fe715a89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.447+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7fe7100a13e0 con 0x7fe7101321a0 2024-09-15T03:32:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.592+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v79) v1 ==== 128+0+0 (secure 0 0 0) 0x7fe708012a80 con 0x7fe7101321a0 2024-09-15T03:32:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.592+0000 7fe6f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T03:32:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.592+0000 7fe6f77fe640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-15T03:32:19.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.690+0000 7fe715a89640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T03:32:19.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.690+0000 7fe715a89640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T03:32:19.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.690+0000 7fe715a89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:19.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:19.690+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7fe71009b800 con 0x7fe7101321a0 2024-09-15T03:32:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.599+0000 7fe6f77fe640 1 -- 172.21.15.181:0/1146638233 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v80) v1 ==== 128+0+0 (secure 0 0 0) 0x7fe7080122a0 con 0x7fe7101321a0 2024-09-15T03:32:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.599+0000 7fe6f77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-15T03:32:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.599+0000 7fe6f77fe640 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-15T03:32:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:removed item id -16 name 'host2' from crush map 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe6fc0386f0 msgr2=0x7fe6fc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe6fc0386f0 0x7fe6fc03aba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7101325a0 tx=0x7fe700032000 comp rx=0 tx=0).stop 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 10 monclient: shutdown 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 msgr2=0x7fe71011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 0x7fe71011b760 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe70800a380 tx=0x7fe708007900 comp rx=0 tx=0).stop 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 shutdown_connections 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe6fc0386f0 0x7fe6fc03aba0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7101321a0 0x7fe71011b760 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe71009f020 0x7fe710130d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 --2- 172.21.15.181:0/1146638233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe71009ec50 0x7fe71009acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 >> 172.21.15.181:0/1146638233 conn(0x7fe7100a5cf0 msgr2=0x7fe71009e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:20.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 shutdown_connections 2024-09-15T03:32:20.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.601+0000 7fe715a89640 1 -- 172.21.15.181:0/1146638233 wait complete. 2024-09-15T03:32:20.614 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-15T03:32:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.693+0000 7f3a1a0dd640 10 monclient: get_monmap_and_config 2024-09-15T03:32:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.693+0000 7f3a1a0dd640 10 monclient: build_initial_monmap 2024-09-15T03:32:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.693+0000 7f3a1a0dd640 10 monclient: monmap: 2024-09-15T03:32:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:20.694985+0000 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:20.694985+0000 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:20.696 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.693+0000 7f3a1a0dd640 1 Processor -- start 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 1 -- start start 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient: init 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a1409f320 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 0x7f3a1412eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient(hunting): picked mon.noname-b con 0x7f3a1412e740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a1412f080 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a14132880 con 0x7f3a1412f080 2024-09-15T03:32:20.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1409e1f0 con 0x7f3a1409f320 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1409e4d0 con 0x7f3a1412e740 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a1a0dd640 10 monclient(hunting): authenticate will time out at 945.176546s 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a137fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3a1409f320 0x7f3a14132170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47056/0 (socket says 172.21.15.181:47056) 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a137fe640 1 -- 172.21.15.181:0/4055301307 learned_addr learned my addr 172.21.15.181:0/4055301307 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a12ffd640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 msgr2=0x7f3a1412eb40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:20.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.694+0000 7f3a12ffd640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 0x7f3a1412eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:20.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 1 -- 172.21.15.181:0/4055301307 >> v1:172.21.15.181:6789/0 conn(0x7f3a1409f320 legacy=0x7f3a14132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a13fff640 1 -- 172.21.15.181:0/4055301307 >> v1:172.21.15.153:6789/0 conn(0x7f3a1412f080 legacy=0x7f3a14139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902038170 0 0) 0x7f3a14132880 con 0x7f3a1412f080 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): my global_id is 64239 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39fc0034e0 con 0x7f3a1412f080 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210904780 0 0) 0x7f3a1409e1f0 con 0x7f3a1409f320 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:20.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): my global_id is 44767 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a14132880 con 0x7f3a1409f320 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2406956699 0 0) 0x7f3a14132880 con 0x7f3a1409f320 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 msgr2=0x7f3a1412eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 0x7f3a1412eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 >> v1:172.21.15.153:6789/0 conn(0x7f3a1412f080 legacy=0x7f3a14139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1409e7b0 con 0x7f3a1409f320 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:20.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f3a04001720 con 0x7f3a1409f320 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a1a0dd640 5 monclient: authenticate success, global_id 44767 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a1a0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: dump: 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.709 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 >> v1:172.21.15.181:6789/0 conn(0x7f3a1409f320 legacy=0x7f3a14132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f39fc004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): picked mon.b con 0x7f39fc004bf0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 0x7f39fc003720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): picked mon.c con 0x7f3a1412e740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39fc003d80 0x7f39fc0041f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): picked mon.a con 0x7f39fc003d80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c001970 con 0x7f39fc003d80 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1409e4d0 con 0x7f39fc004bf0 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc00bd50 con 0x7f3a1412e740 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a127fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a1a0dd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a12ffd640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 msgr2=0x7f39fc003720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a12ffd640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 0x7f39fc003720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f39fc004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): get_auth_request con 0x7f39fc004bf0 auth_method 0 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a137fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a13fff640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39fc003d80 0x7f39fc0041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a13fff640 10 monclient(hunting): get_auth_request con 0x7f39fc003d80 auth_method 0 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a13fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.697+0000 7f3a13fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:20.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 10 monclient(hunting): handle_auth_done global_id 44767 payload 293 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 msgr2=0x7f39fc003720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a1412e740 0x7f39fc003720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39fc003d80 msgr2=0x7f39fc0041f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39fc003d80 0x7f39fc0041f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 1 monclient: found mon.b 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 1 -- 172.21.15.181:0/4055301307 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14132880 con 0x7f39fc004bf0 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.698+0000 7f3a13fff640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39fc003d80 0x7f39fc0041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:20.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.700+0000 7f3a137fe640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 msgr2=0x7f39fc004fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.700+0000 7f3a137fe640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f39fc004fc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a04001490 tx=0x7f3a04007a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e0e7ff91bae0b1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.700+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04005890 con 0x7f39fc004bf0 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.700+0000 7f3a127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.700+0000 7f3a127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.700+0000 7f3a127fc640 10 monclient: dump: 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.714 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04005a10 con 0x7f39fc004bf0 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 4 set_mon_vals no callback set 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a127fc640 1 -- 172.21.15.181:0/4055301307 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04005dd0 con 0x7f39fc004bf0 2024-09-15T03:32:20.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a127fc640 10 monclient: dump: 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:20.716 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 10 monclient: shutdown 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/4055301307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 msgr2=0x7f39fc004fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/4055301307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f39fc004fc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a04001490 tx=0x7f3a04007a10 comp rx=0 tx=0).stop 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a13fff640 1 -- 172.21.15.181:0/4055301307 reap_dead start 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/4055301307 shutdown_connections 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/4055301307 >> 172.21.15.181:0/4055301307 conn(0x7f3a140a2b70 msgr2=0x7f3a1409ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/4055301307 shutdown_connections 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.701+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/4055301307 wait complete. 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.702+0000 7f3a1a0dd640 10 monclient: build_initial_monmap 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.702+0000 7f3a1a0dd640 10 monclient: monmap: 2024-09-15T03:32:20.717 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.702+0000 7f3a1a0dd640 1 Processor -- start 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.702+0000 7f3a1a0dd640 1 -- start start 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.702+0000 7f3a1a0dd640 10 monclient: init 2024-09-15T03:32:20.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f3a14118ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): picked mon.noname-b con 0x7f39fc004bf0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1412da20 0x7f3a1412ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): picked mon.noname-a con 0x7f3a1412da20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a141410c0 0x7f3a14141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): picked mon.noname-c con 0x7f3a141410c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14118ff0 con 0x7f3a1412da20 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14119620 con 0x7f39fc004bf0 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:20.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1412e330 con 0x7f3a141410c0 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a1a0dd640 10 monclient(hunting): authenticate will time out at 945.185179s 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f3a14118ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a13fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a141410c0 msgr2=0x7f3a14141530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f3a14118ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49032/0 (socket says 172.21.15.181:49032) 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 -- 172.21.15.181:0/777423045 learned_addr learned my addr 172.21.15.181:0/777423045 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): get_auth_request con 0x7f39fc004bf0 auth_method 0 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a13fff640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a141410c0 0x7f3a14141530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:20.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a12ffd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1412da20 0x7f3a1412ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a12ffd640 10 monclient(hunting): get_auth_request con 0x7f3a1412da20 auth_method 0 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:20.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient(hunting): handle_auth_done global_id 44770 payload 824 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 -- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a141410c0 msgr2=0x7f3a14141530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a141410c0 0x7f3a14141530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 -- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1412da20 msgr2=0x7f3a1412ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1412da20 0x7f3a1412ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 monclient: found mon.noname-b 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:20.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.703+0000 7f3a137fe640 1 -- 172.21.15.181:0/777423045 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a140a18b0 con 0x7f39fc004bf0 2024-09-15T03:32:20.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a137fe640 1 -- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 msgr2=0x7f3a14118ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a137fe640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f3a14118ab0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c5c0 tx=0x7f3a04034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04007510 con 0x7f39fc004bf0 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: dump: 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:20.725 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 5 monclient: authenticate success, global_id 44770 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04007670 con 0x7f39fc004bf0 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0403d440 con 0x7f39fc004bf0 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 10 monclient: _renew_subs 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:20.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1411ae20 con 0x7f39fc004bf0 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f39fb7fe640 10 monclient: dump: 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:20.727 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a11ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 10 monclient: _renew_subs 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.705+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1409b500 con 0x7f39fc004bf0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3a04043020 con 0x7f39fc004bf0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f3a1a0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f3a1a0dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f3a1a0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f39fb7fe640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a000386f0 0x7f3a0003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1409b6f0 con 0x7f39fc004bf0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f3a0406cc40 con 0x7f39fc004bf0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f3a12ffd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a000386f0 0x7f3a0003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:20.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.706+0000 7f3a12ffd640 10 monclient: get_auth_request con 0x7f3a000386f0 auth_method 0 2024-09-15T03:32:20.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.709+0000 7f3a12ffd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a000386f0 0x7f3a0003aba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a08006fd0 tx=0x7f3a08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:20.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.711+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3a0402f3b0 con 0x7f39fc004bf0 2024-09-15T03:32:20.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.711+0000 7f39fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:20.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.711+0000 7f39fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:20.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.868+0000 7f3a1a0dd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T03:32:20.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.868+0000 7f3a1a0dd640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T03:32:20.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.868+0000 7f3a1a0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:20.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:20.868+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f3a1411afe0 con 0x7f39fc004bf0 2024-09-15T03:32:21.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.602+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v81) v1 ==== 251+0+0 (secure 0 0 0) 0x7f3a0403b3c0 con 0x7f39fc004bf0 2024-09-15T03:32:21.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.602+0000 7f39fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T03:32:21.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.602+0000 7f39fb7fe640 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-15T03:32:21.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.700+0000 7f3a1a0dd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T03:32:21.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.700+0000 7f3a1a0dd640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T03:32:21.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.700+0000 7f3a1a0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:21.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.700+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f3a1411afe0 con 0x7f39fc004bf0 2024-09-15T03:32:21.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.701+0000 7f39fb7fe640 1 -- 172.21.15.181:0/777423045 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v81) v1 ==== 200+0+0 (secure 0 0 0) 0x7f3a0403ba10 con 0x7f39fc004bf0 2024-09-15T03:32:21.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.701+0000 7f39fb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-15T03:32:21.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.701+0000 7f39fb7fe640 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-15T03:32:21.703 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'host-for-test' already exists 2024-09-15T03:32:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a000386f0 msgr2=0x7f3a0003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a000386f0 0x7f3a0003aba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a08006fd0 tx=0x7f3a08008040 comp rx=0 tx=0).stop 2024-09-15T03:32:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 10 monclient: shutdown 2024-09-15T03:32:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 msgr2=0x7f3a14118ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f3a14118ab0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c5c0 tx=0x7f3a04034000 comp rx=0 tx=0).stop 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 shutdown_connections 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a000386f0 0x7f3a0003aba0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a141410c0 0x7f3a14141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1412da20 0x7f3a1412ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 --2- 172.21.15.181:0/777423045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39fc004bf0 0x7f3a14118ab0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 >> 172.21.15.181:0/777423045 conn(0x7f3a140a2b70 msgr2=0x7f3a14097af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 shutdown_connections 2024-09-15T03:32:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.703+0000 7f3a1a0dd640 1 -- 172.21.15.181:0/777423045 wait complete. 2024-09-15T03:32:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep host-for-test 2024-09-15T03:32:21.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.783+0000 7fd518a9f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:21.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.783+0000 7fd518a9f640 10 monclient: build_initial_monmap 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.783+0000 7fd518a9f640 10 monclient: monmap: 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:21.785542+0000 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:21.785542+0000 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:21.786 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 1 Processor -- start 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 1 -- start start 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 10 monclient: init 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd51409c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 10 monclient(hunting): picked mon.noname-b con 0x7fd514067640 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd51406d8f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.784+0000 7fd518a9f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd514135020 0x7fd514137400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:21.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 10 monclient(hunting): picked mon.noname-a con 0x7fd514135020 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd514067a10 con 0x7fd51406d8f0 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51406c7a0 con 0x7fd514135020 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51406c9a0 con 0x7fd514067640 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd518a9f640 10 monclient(hunting): authenticate will time out at 946.267034s 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd51259c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 msgr2=0x7fd51409c020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd50bfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd51406d8f0 0x7fd51409c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47064/0 (socket says 172.21.15.181:47064) 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd50bfff640 1 -- 172.21.15.181:0/2793269566 learned_addr learned my addr 172.21.15.181:0/2793269566 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd51259c640 1 --2- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd51409c020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 1 --2- 172.21.15.181:0/2793269566 >> v2:172.21.15.153:3300/0 conn(0x7fd514135020 0x7fd514137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:21.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): get_auth_request con 0x7fd514135020 auth_method 0 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient(hunting): handle_auth_done global_id 64245 payload 274 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 1 -- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 msgr2=0x7fd51409c020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 1 --2- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd51409c020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 1 -- 172.21.15.181:0/2793269566 >> v1:172.21.15.181:6789/0 conn(0x7fd51406d8f0 legacy=0x7fd51409c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 1 monclient: found mon.noname-a 2024-09-15T03:32:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:21.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:21.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.785+0000 7fd512d9d640 1 -- 172.21.15.181:0/2793269566 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51406e700 con 0x7fd514135020 2024-09-15T03:32:21.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd512d9d640 1 -- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 msgr2=0x7fd514137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:21.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd512d9d640 1 --2- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514137400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd514067d00 tx=0x7fd50c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3517825e645599 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:21.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 1 -- 172.21.15.181:0/2793269566 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c007840 con 0x7fd514135020 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: dump: 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:21.798 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 1 -- 172.21.15.181:0/2793269566 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50c02d050 con 0x7fd514135020 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 1 -- 172.21.15.181:0/2793269566 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c007bd0 con 0x7fd514135020 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd518a9f640 5 monclient: authenticate success, global_id 64245 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd518a9f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd518a9f640 4 set_mon_vals no callback set 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.788+0000 7fd511d9b640 10 monclient: dump: 2024-09-15T03:32:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 10 monclient: shutdown 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 -- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 msgr2=0x7fd514137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:21.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 --2- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514137400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd514067d00 tx=0x7fd50c009780 comp rx=0 tx=0).stop 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 -- 172.21.15.181:0/2793269566 shutdown_connections 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 --2- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514137400 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 --2- 172.21.15.181:0/2793269566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd51409c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 -- 172.21.15.181:0/2793269566 >> 172.21.15.181:0/2793269566 conn(0x7fd514071370 msgr2=0x7fd5140a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 -- 172.21.15.181:0/2793269566 shutdown_connections 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 1 -- 172.21.15.181:0/2793269566 wait complete. 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 10 monclient: build_initial_monmap 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.789+0000 7fd518a9f640 10 monclient: monmap: 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 Processor -- start 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 -- start start 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient: init 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd5141186c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): picked mon.noname-c con 0x7fd514067640 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 0x7fd514118c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): picked mon.noname-b con 0x7fd51406d8f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514119140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): picked mon.noname-a con 0x7fd514135020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51411bfc0 con 0x7fd514135020 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51411ad80 con 0x7fd51406d8f0 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51409da60 con 0x7fd514067640 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd518a9f640 10 monclient(hunting): authenticate will time out at 946.272828s 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 0x7fd514118c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:21.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd51259c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 msgr2=0x7fd5141186c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 0x7fd514118c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49046/0 (socket says 172.21.15.181:49046) 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 1 -- 172.21.15.181:0/275295944 learned_addr learned my addr 172.21.15.181:0/275295944 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 10 monclient(hunting): get_auth_request con 0x7fd51406d8f0 auth_method 0 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd50bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.790+0000 7fd51259c640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd5141186c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd512d9d640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514119140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd512d9d640 10 monclient(hunting): get_auth_request con 0x7fd514135020 auth_method 0 2024-09-15T03:32:21.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd512d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd512d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd512d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 10 monclient(hunting): handle_auth_done global_id 44773 payload 824 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 1 -- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 msgr2=0x7fd5141186c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd5141186c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 1 -- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 msgr2=0x7fd514119140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514119140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 1 monclient: found mon.noname-b 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.791+0000 7fd50bfff640 1 -- 172.21.15.181:0/275295944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c009a20 con 0x7fd51406d8f0 2024-09-15T03:32:21.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50bfff640 1 -- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 msgr2=0x7fd514118c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50bfff640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 0x7fd514118c00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00c530 tx=0x7fd4fc007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc005cb0 con 0x7fd51406d8f0 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: dump: 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 5 monclient: authenticate success, global_id 44773 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 10 monclient: _renew_subs 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc005e30 con 0x7fd51406d8f0 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5141322d0 con 0x7fd51406d8f0 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc007300 con 0x7fd51406d8f0 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:21.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd50a7fc640 10 monclient: dump: 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 10 monclient: _renew_subs 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd51159a640 4 set_mon_vals no callback set 2024-09-15T03:32:21.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.793+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51411a1b0 con 0x7fd51406d8f0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd4fc01b3b0 con 0x7fd51406d8f0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd518a9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd518a9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd518a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5140636e0 con 0x7fd51406d8f0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd50a7fc640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4f40386f0 0x7fd4f403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7fd4fc04b740 con 0x7fd51406d8f0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd51259c640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4f40386f0 0x7fd4f403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:21.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.794+0000 7fd51259c640 10 monclient: get_auth_request con 0x7fd4f40386f0 auth_method 0 2024-09-15T03:32:21.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.797+0000 7fd51259c640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4f40386f0 0x7fd4f403aba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd5040038f0 tx=0x7fd504006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:21.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.799+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd4fc018680 con 0x7fd51406d8f0 2024-09-15T03:32:21.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.799+0000 7fd50a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:21.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.799+0000 7fd50a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:21.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.954+0000 7fd518a9f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:21.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.954+0000 7fd518a9f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:21.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.954+0000 7fd518a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:21.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.954+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd5140a1ce0 con 0x7fd51406d8f0 2024-09-15T03:32:21.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.955+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7fd4fc0175f0 con 0x7fd51406d8f0 2024-09-15T03:32:21.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.955+0000 7fd50a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:21.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:21.955+0000 7fd50a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:22.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.047+0000 7fd518a9f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:22.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.047+0000 7fd518a9f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:22.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.047+0000 7fd518a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.047+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd51411c130 con 0x7fd51406d8f0 2024-09-15T03:32:22.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.048+0000 7fd50a7fc640 1 -- 172.21.15.181:0/275295944 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7fd4fc017070 con 0x7fd51406d8f0 2024-09-15T03:32:22.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.048+0000 7fd50a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:22.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.048+0000 7fd50a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4f40386f0 msgr2=0x7fd4f403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4f40386f0 0x7fd4f403aba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd5040038f0 tx=0x7fd504006d80 comp rx=0 tx=0).stop 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 10 monclient: shutdown 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 msgr2=0x7fd514118c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 0x7fd514118c00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00c530 tx=0x7fd4fc007f90 comp rx=0 tx=0).stop 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 shutdown_connections 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd4f40386f0 0x7fd4f403aba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd514135020 0x7fd514119140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51406d8f0 0x7fd514118c00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 --2- 172.21.15.181:0/275295944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd514067640 0x7fd5141186c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.049+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 >> 172.21.15.181:0/275295944 conn(0x7fd514071370 msgr2=0x7fd5141344f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:22.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.050+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 shutdown_connections 2024-09-15T03:32:22.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.050+0000 7fd518a9f640 1 -- 172.21.15.181:0/275295944 wait complete. 2024-09-15T03:32:22.060 INFO:tasks.workunit.client.0.smithi181.stdout:-11 0 host host-for-test 2024-09-15T03:32:22.061 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:22.061 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep rack-for-test 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.129+0000 7f037786f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.129+0000 7f037786f640 10 monclient: build_initial_monmap 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.130+0000 7f037786f640 10 monclient: monmap: 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:22.131781+0000 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:22.131781+0000 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:22.133 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.130+0000 7f037786f640 1 Processor -- start 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.130+0000 7f037786f640 1 -- start start 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.130+0000 7f037786f640 10 monclient: init 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037009c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): picked mon.noname-b con 0x7f0370067640 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0370097520 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0370134ba0 0x7f0370136f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): picked mon.noname-c con 0x7f0370134ba0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0370068200 con 0x7f0370097520 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037009b430 con 0x7f0370134ba0 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0370067a10 con 0x7f0370067640 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037786f640 10 monclient(hunting): authenticate will time out at 946.613325s 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037560b640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 msgr2=0x7f037009c530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f037560b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037009c530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:22.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0370134ba0 0x7f0370136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0370134ba0 0x7f0370136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49060/0 (socket says 172.21.15.181:49060) 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 -- 172.21.15.181:0/3155242780 learned_addr learned my addr 172.21.15.181:0/3155242780 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): get_auth_request con 0x7f0370134ba0 auth_method 0 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient(hunting): handle_auth_done global_id 44776 payload 274 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 -- 172.21.15.181:0/3155242780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 msgr2=0x7f037009c530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 --2- 172.21.15.181:0/3155242780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037009c530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 -- 172.21.15.181:0/3155242780 >> v1:172.21.15.153:6789/0 conn(0x7f0370097520 legacy=0x7f0370097910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:22.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:22.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 monclient: found mon.noname-c 2024-09-15T03:32:22.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:22.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.131+0000 7f0375e0c640 1 -- 172.21.15.181:0/3155242780 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037009d230 con 0x7f0370134ba0 2024-09-15T03:32:22.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0375e0c640 1 -- 172.21.15.181:0/3155242780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 msgr2=0x7f0370136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:22.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0375e0c640 1 --2- 172.21.15.181:0/3155242780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f0370136f80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0368009f20 tx=0x7f0368009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f589e160883a839 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:22.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 1 -- 172.21.15.181:0/3155242780 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368007610 con 0x7f0370134ba0 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: dump: 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.144 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f037786f640 5 monclient: authenticate success, global_id 44776 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 1 -- 172.21.15.181:0/3155242780 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036802d050 con 0x7f0370134ba0 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f037786f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f037786f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f037786f640 4 set_mon_vals no callback set 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 1 -- 172.21.15.181:0/3155242780 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680079a0 con 0x7f0370134ba0 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.134+0000 7f0367fff640 10 monclient: dump: 2024-09-15T03:32:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 10 monclient: shutdown 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 -- 172.21.15.181:0/3155242780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 msgr2=0x7f0370136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 --2- 172.21.15.181:0/3155242780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f0370136f80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0368009f20 tx=0x7f0368009840 comp rx=0 tx=0).stop 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 -- 172.21.15.181:0/3155242780 shutdown_connections 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 --2- 172.21.15.181:0/3155242780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f0370136f80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 --2- 172.21.15.181:0/3155242780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037009c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 -- 172.21.15.181:0/3155242780 >> 172.21.15.181:0/3155242780 conn(0x7f03700a1530 msgr2=0x7f03700a1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 -- 172.21.15.181:0/3155242780 shutdown_connections 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 1 -- 172.21.15.181:0/3155242780 wait complete. 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 10 monclient: build_initial_monmap 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.135+0000 7f037786f640 10 monclient: monmap: 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:22.147 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 Processor -- start 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 -- start start 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient: init 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037006d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): picked mon.noname-c con 0x7f0370067640 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370097520 0x7f037006db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): picked mon.noname-a con 0x7f0370097520 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f037006e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): picked mon.noname-b con 0x7f0370134ba0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03701273e0 con 0x7f0370097520 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03701288a0 con 0x7f0370134ba0 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0370127960 con 0x7f0370067640 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037786f640 10 monclient(hunting): authenticate will time out at 946.618766s 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f037006e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037560b640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 msgr2=0x7f037006d5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f037006e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49072/0 (socket says 172.21.15.181:49072) 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f037560b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037006d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 1 -- 172.21.15.181:0/779603256 learned_addr learned my addr 172.21.15.181:0/779603256 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): get_auth_request con 0x7f0370134ba0 auth_method 0 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.136+0000 7f0375e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0374e0a640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370097520 0x7f037006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 10 monclient(hunting): handle_auth_done global_id 44779 payload 824 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 1 -- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 msgr2=0x7f037006d5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:22.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037006d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 1 -- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370097520 msgr2=0x7f037006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370097520 0x7f037006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 1 monclient: found mon.noname-b 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0375e0c640 1 -- 172.21.15.181:0/779603256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0368009ae0 con 0x7f0370134ba0 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0374e0a640 10 monclient: get_auth_request con 0x7f0370097520 auth_method 0 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.137+0000 7f0374e0a640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370097520 0x7f037006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f0375e0c640 1 -- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 msgr2=0x7f037006e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f0375e0c640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f037006e060 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0368009f20 tx=0x7f0368036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368031cb0 con 0x7f0370134ba0 2024-09-15T03:32:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: dump: 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0368031e10 con 0x7f0370134ba0 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 5 monclient: authenticate success, global_id 44779 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368044640 con 0x7f0370134ba0 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03657fa640 10 monclient: dump: 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.153 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 10 monclient: _renew_subs 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f03677fe640 4 set_mon_vals no callback set 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037012ffc0 con 0x7f0370134ba0 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 10 monclient: _renew_subs 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.139+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0370126790 con 0x7f0370134ba0 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0368038070 con 0x7f0370134ba0 2024-09-15T03:32:22.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f037786f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f037786f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f037786f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03700636e0 con 0x7f0370134ba0 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f03657fa640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03540386f0 0x7f035403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7f036806d140 con 0x7f0370134ba0 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f037560b640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03540386f0 0x7f035403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.140+0000 7f037560b640 10 monclient: get_auth_request con 0x7f03540386f0 auth_method 0 2024-09-15T03:32:22.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.143+0000 7f037560b640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03540386f0 0x7f035403aba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f036c0038f0 tx=0x7f036c006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:22.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.145+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0368044a30 con 0x7f0370134ba0 2024-09-15T03:32:22.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.145+0000 7f03657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:22.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.145+0000 7f03657fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:22.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f037786f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:22.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f037786f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:22.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f037786f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f0370072640 con 0x7f0370134ba0 2024-09-15T03:32:22.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7f03680405f0 con 0x7f0370134ba0 2024-09-15T03:32:22.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f03657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:22.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.300+0000 7f03657fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:22.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.392+0000 7f037786f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:22.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.392+0000 7f037786f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:22.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.392+0000 7f037786f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.392+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f0370064a90 con 0x7f0370134ba0 2024-09-15T03:32:22.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.393+0000 7f03657fa640 1 -- 172.21.15.181:0/779603256 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7f0368049020 con 0x7f0370134ba0 2024-09-15T03:32:22.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.393+0000 7f03657fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:22.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.393+0000 7f03657fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.394+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03540386f0 msgr2=0x7f035403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.394+0000 7f037786f640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03540386f0 0x7f035403aba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f036c0038f0 tx=0x7f036c006d80 comp rx=0 tx=0).stop 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.394+0000 7f037786f640 10 monclient: shutdown 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.394+0000 7f037786f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.394+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 msgr2=0x7f037006e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.394+0000 7f037786f640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f037006e060 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0368009f20 tx=0x7f0368036000 comp rx=0 tx=0).stop 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 shutdown_connections 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03540386f0 0x7f035403aba0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370134ba0 0x7f037006e060 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0370097520 0x7f037006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 --2- 172.21.15.181:0/779603256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0370067640 0x7f037006d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 >> 172.21.15.181:0/779603256 conn(0x7f03700a1530 msgr2=0x7f037009f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:22.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 shutdown_connections 2024-09-15T03:32:22.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.395+0000 7f037786f640 1 -- 172.21.15.181:0/779603256 wait complete. 2024-09-15T03:32:22.405 INFO:tasks.workunit.client.0.smithi181.stdout:-12 0 rack rack-for-test 2024-09-15T03:32:22.406 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:22.406 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep root-for-test 2024-09-15T03:32:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.475+0000 7ff0ee2e8640 10 monclient: get_monmap_and_config 2024-09-15T03:32:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.475+0000 7ff0ee2e8640 10 monclient: build_initial_monmap 2024-09-15T03:32:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.475+0000 7ff0ee2e8640 10 monclient: monmap: 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:22.477439+0000 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:22.477439+0000 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:22.478 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 1 Processor -- start 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 1 -- start start 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 10 monclient: init 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0e809cb20 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809afb0 0x7ff0e812f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.476+0000 7ff0ee2e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e809afb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff0e8133c70 0x7ff0e8136050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e8133c70 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e809b3c0 con 0x7ff0e809cb20 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e8097e90 con 0x7ff0e8133c70 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e8099640 con 0x7ff0e809afb0 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ee2e8640 10 monclient(hunting): authenticate will time out at 946.959047s 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0e77fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809afb0 msgr2=0x7ff0e812f050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809afb0 0x7ff0e812f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff0e8133c70 0x7ff0e8136050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff0e8133c70 0x7ff0e8136050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:46594/0 (socket says 172.21.15.181:46594) 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 -- 172.21.15.181:0/1428156520 learned_addr learned my addr 172.21.15.181:0/1428156520 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): get_auth_request con 0x7ff0e8133c70 auth_method 0 2024-09-15T03:32:22.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0e7fff640 1 --1- 172.21.15.181:0/1428156520 >> v1:172.21.15.153:6789/0 conn(0x7ff0e809cb20 0x7ff0e812e940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:38252/0 (socket says 172.21.15.181:38252) 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient(hunting): handle_auth_done global_id 64251 payload 274 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 -- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809afb0 msgr2=0x7ff0e812f050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809afb0 0x7ff0e812f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 -- 172.21.15.181:0/1428156520 >> v1:172.21.15.153:6789/0 conn(0x7ff0e809cb20 legacy=0x7ff0e812e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient: _finish_hunting 0 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 monclient: found mon.noname-a 2024-09-15T03:32:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:22.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:22.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.477+0000 7ff0ec885640 1 -- 172.21.15.181:0/1428156520 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e809d6f0 con 0x7ff0e8133c70 2024-09-15T03:32:22.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0ec885640 1 -- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e8133c70 msgr2=0x7ff0e8136050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e8133c70 0x7ff0e8136050 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000d1a0 tx=0x7ff0e000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c00b35ab3fc2a27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 1 -- 172.21.15.181:0/1428156520 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0009610 con 0x7ff0e8133c70 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 10 monclient: dump: 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.488 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.480+0000 7ff0e6ffd640 1 -- 172.21.15.181:0/1428156520 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e0009770 con 0x7ff0e8133c70 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 5 monclient: authenticate success, global_id 64251 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 4 set_mon_vals no callback set 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0e6ffd640 1 -- 172.21.15.181:0/1428156520 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0009b10 con 0x7ff0e8133c70 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0e6ffd640 10 monclient: dump: 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 10 monclient: shutdown 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e8133c70 msgr2=0x7ff0e8136050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e8133c70 0x7ff0e8136050 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000d1a0 tx=0x7ff0e000ba50 comp rx=0 tx=0).stop 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1428156520 shutdown_connections 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e8133c70 0x7ff0e8136050 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1428156520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809afb0 0x7ff0e812f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1428156520 >> 172.21.15.181:0/1428156520 conn(0x7ff0e80714d0 msgr2=0x7ff0e80a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1428156520 shutdown_connections 2024-09-15T03:32:22.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1428156520 wait complete. 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.481+0000 7ff0ee2e8640 10 monclient: build_initial_monmap 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient: monmap: 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 Processor -- start 2024-09-15T03:32:22.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 -- start start 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient: init 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0e806c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e809afb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0e806c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e809cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e8133c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e812e480 con 0x7ff0e809afb0 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e8098950 con 0x7ff0e8133c70 2024-09-15T03:32:22.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e8118370 con 0x7ff0e809cb20 2024-09-15T03:32:22.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:22.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.482+0000 7ff0ee2e8640 10 monclient(hunting): authenticate will time out at 946.964939s 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e77fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 msgr2=0x7ff0e806c580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0e806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49086/0 (socket says 172.21.15.181:49086) 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0e806c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 -- 172.21.15.181:0/1775117113 learned_addr learned my addr 172.21.15.181:0/1775117113 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): get_auth_request con 0x7ff0e8133c70 auth_method 0 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): get_auth_request con 0x7ff0e809afb0 auth_method 0 2024-09-15T03:32:22.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient(hunting): handle_auth_done global_id 44782 payload 824 2024-09-15T03:32:22.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 msgr2=0x7ff0e806c580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0e806c580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 msgr2=0x7ff0e806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0e806c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient: _finish_hunting 0 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 monclient: found mon.noname-b 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e000bcf0 con 0x7ff0e8133c70 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0e7fff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0e806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 msgr2=0x7ff0e806cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 msgr2=0x7ff0e806cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000d1a0 tx=0x7ff0e0039000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:22.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.483+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:22.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 10 monclient: get_auth_request con 0x7ff0e8133c70 auth_method 0 2024-09-15T03:32:22.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 0 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 msgr2=0x7ff0e806cac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0e8133c70 0x7ff0e806cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 0x7ff0d40058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d4003460 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0d4008430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e809cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0d400aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e809afb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d400b840 con 0x7ff0e809afb0 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d400b980 con 0x7ff0d4003460 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d400bb40 con 0x7ff0e809cb20 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0cffff640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 0x7ff0d40058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): get_auth_request con 0x7ff0d4003460 auth_method 0 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:22.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e77fe640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 msgr2=0x7ff0d4008430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e77fe640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0d4008430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0d400aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 10 monclient(hunting): get_auth_request con 0x7ff0e809afb0 auth_method 0 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0ec885640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_done global_id 44782 payload 293 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 msgr2=0x7ff0d4008430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:22.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0d4008430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 msgr2=0x7ff0d400aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0d400aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 monclient: found mon.noname-b 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.684+0000 7ff0e7fff640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d400bdc0 con 0x7ff0d4003460 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0e7fff640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 msgr2=0x7ff0d40058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0e7fff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 0x7ff0d40058a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8006890 tx=0x7ff0d8007e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80071e0 con 0x7ff0d4003460 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 10 monclient: dump: 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.693 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0e7fff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 0x7ff0d40058a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8006890 tx=0x7ff0d8007e40 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7ff0d80063a0 config(0 keys) v1 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 10 monclient: _finish_auth 0 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 5 monclient: authenticate success, global_id 44782 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 10 monclient: _renew_subs 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d80063a0 con 0x7ff0d4003460 2024-09-15T03:32:22.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e806db90 con 0x7ff0d4003460 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0e67fc640 4 set_mon_vals no callback set 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 10 monclient: _renew_subs 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.686+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e8118930 con 0x7ff0d4003460 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80133f0 con 0x7ff0d4003460 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0cffff640 10 monclient: dump: 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.695 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff0d8013760 con 0x7ff0d4003460 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0cffff640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0e8133c70 0x7ff0d4042ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0ee2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0ee2e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0ee2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e80636e0 con 0x7ff0d4003460 2024-09-15T03:32:22.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0e77fe640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0e8133c70 0x7ff0d4042ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:22.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.687+0000 7ff0e77fe640 10 monclient: get_auth_request con 0x7ff0e8133c70 auth_method 0 2024-09-15T03:32:22.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.690+0000 7ff0e77fe640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0e8133c70 0x7ff0d4042ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc006fd0 tx=0x7ff0dc006e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:22.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:22.692+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7ff0d804b330 con 0x7ff0d4003460 2024-09-15T03:32:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.515+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff0d801d4e0 con 0x7ff0d4003460 2024-09-15T03:32:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.515+0000 7ff0cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.515+0000 7ff0cffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.676+0000 7ff0ee2e8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.676+0000 7ff0ee2e8640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.676+0000 7ff0ee2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.676+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff0e8064b20 con 0x7ff0d4003460 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.677+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff0d8005e60 con 0x7ff0d4003460 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.677+0000 7ff0cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:23.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.677+0000 7ff0cffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.770+0000 7ff0ee2e8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.770+0000 7ff0ee2e8640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.770+0000 7ff0ee2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.770+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff0e81175a0 con 0x7ff0d4003460 2024-09-15T03:32:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.770+0000 7ff0cffff640 1 -- 172.21.15.181:0/1775117113 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff0d801eaf0 con 0x7ff0d4003460 2024-09-15T03:32:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.771+0000 7ff0cffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.771+0000 7ff0cffff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0e8133c70 msgr2=0x7ff0d4042ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0e8133c70 0x7ff0d4042ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc006fd0 tx=0x7ff0dc006e10 comp rx=0 tx=0).stop 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 10 monclient: shutdown 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 msgr2=0x7ff0d40058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 0x7ff0d40058a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8006890 tx=0x7ff0d8007e40 comp rx=0 tx=0).stop 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 shutdown_connections 2024-09-15T03:32:23.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0e8133c70 0x7ff0d4042ef0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0e809afb0 0x7ff0d400aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0e809cb20 0x7ff0d4008430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 --2- 172.21.15.181:0/1775117113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d4003460 0x7ff0d40058a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.772+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 >> 172.21.15.181:0/1775117113 conn(0x7ff0e80714d0 msgr2=0x7ff0e809c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:23.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.773+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 shutdown_connections 2024-09-15T03:32:23.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.773+0000 7ff0ee2e8640 1 -- 172.21.15.181:0/1775117113 wait complete. 2024-09-15T03:32:23.783 INFO:tasks.workunit.client.0.smithi181.stdout:-14 0 root root-for-test 2024-09-15T03:32:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm host-for-test 2024-09-15T03:32:23.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.855+0000 7f1fd58af640 10 monclient: get_monmap_and_config 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.855+0000 7f1fd58af640 10 monclient: build_initial_monmap 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.855+0000 7f1fd58af640 10 monclient: monmap: 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:23.857593+0000 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:23.857593+0000 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:23.858 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:23.859 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 1 Processor -- start 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 1 -- start start 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 10 monclient: init 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fd009ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1fd009f020 0x7f1fd0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:23.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.856+0000 7f1fd58af640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd009f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd0136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd0067420 con 0x7f1fd009ec50 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd00a0480 con 0x7f1fd009f020 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd00a0200 con 0x7f1fd0136d60 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fd58af640 10 monclient(hunting): authenticate will time out at 948.339065s 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1fd009f020 0x7f1fd0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fceffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f1fd009ec50 0x7f1fd0131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47070/0 (socket says 172.21.15.181:47070) 2024-09-15T03:32:23.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1fd009f020 0x7f1fd0069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49110/0 (socket says 172.21.15.181:49110) 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fceffd640 1 -- 172.21.15.181:0/3531852642 learned_addr learned my addr 172.21.15.181:0/3531852642 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): get_auth_request con 0x7f1fd009f020 auth_method 0 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fcf7fe640 1 -- 172.21.15.181:0/3531852642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 msgr2=0x7f1fd0139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fcf7fe640 1 --2- 172.21.15.181:0/3531852642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_done global_id 44785 payload 274 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 -- 172.21.15.181:0/3531852642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 msgr2=0x7f1fd0139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 --2- 172.21.15.181:0/3531852642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 -- 172.21.15.181:0/3531852642 >> v1:172.21.15.181:6789/0 conn(0x7f1fd009ec50 legacy=0x7f1fd0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 monclient: found mon.noname-c 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.857+0000 7f1fce7fc640 1 -- 172.21.15.181:0/3531852642 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd009fd80 con 0x7f1fd009f020 2024-09-15T03:32:23.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fce7fc640 1 -- 172.21.15.181:0/3531852642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 msgr2=0x7f1fd0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fce7fc640 1 --2- 172.21.15.181:0/3531852642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0069d10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009850 tx=0x7f1fc40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84f2eab4bc548997 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 1 -- 172.21.15.181:0/3531852642 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4032440 con 0x7f1fd009f020 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: dump: 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:23.871 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 5 monclient: authenticate success, global_id 44785 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 1 -- 172.21.15.181:0/3531852642 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc40325a0 con 0x7f1fd009f020 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 1 -- 172.21.15.181:0/3531852642 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4032940 con 0x7f1fd009f020 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 4 set_mon_vals no callback set 2024-09-15T03:32:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fcdffb640 10 monclient: dump: 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 10 monclient: shutdown 2024-09-15T03:32:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 -- 172.21.15.181:0/3531852642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 msgr2=0x7f1fd0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 --2- 172.21.15.181:0/3531852642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0069d10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009850 tx=0x7f1fc40090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 -- 172.21.15.181:0/3531852642 shutdown_connections 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 --2- 172.21.15.181:0/3531852642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 --2- 172.21.15.181:0/3531852642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0069d10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 -- 172.21.15.181:0/3531852642 >> 172.21.15.181:0/3531852642 conn(0x7f1fd00a5cf0 msgr2=0x7f1fd00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 -- 172.21.15.181:0/3531852642 shutdown_connections 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.860+0000 7f1fd58af640 1 -- 172.21.15.181:0/3531852642 wait complete. 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.861+0000 7f1fd58af640 10 monclient: build_initial_monmap 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.861+0000 7f1fd58af640 10 monclient: monmap: 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.861+0000 7f1fd58af640 1 Processor -- start 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.861+0000 7f1fd58af640 1 -- start start 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.861+0000 7f1fd58af640 10 monclient: init 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd009ec50 0x7f1fd0097710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:23.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0097c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0098190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd0136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd009af40 con 0x7f1fd009ec50 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0130e40 con 0x7f1fd009f020 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd009b190 con 0x7f1fd0136d60 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:23.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fd58af640 10 monclient(hunting): authenticate will time out at 948.344366s 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0097c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0097c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49116/0 (socket says 172.21.15.181:49116) 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 -- 172.21.15.181:0/460747104 learned_addr learned my addr 172.21.15.181:0/460747104 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): get_auth_request con 0x7f1fd009f020 auth_method 0 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fcf7fe640 1 -- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 msgr2=0x7f1fd0098190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fcf7fe640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0098190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:23.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient(hunting): handle_auth_done global_id 44788 payload 824 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fceffd640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd009ec50 0x7f1fd0097710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 -- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 msgr2=0x7f1fd0098190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0098190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 -- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd009ec50 msgr2=0x7f1fd0097710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd009ec50 0x7f1fd0097710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 monclient: found mon.noname-b 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.862+0000 7f1fce7fc640 1 -- 172.21.15.181:0/460747104 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4009390 con 0x7f1fd009f020 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.864+0000 7f1fce7fc640 1 -- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 msgr2=0x7f1fd0097c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.864+0000 7f1fce7fc640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0097c50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009360 tx=0x7f1fc4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:23.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4031a70 con 0x7f1fd009f020 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: dump: 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:23.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 5 monclient: authenticate success, global_id 44788 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4031bd0 con 0x7f1fd009f020 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 10 monclient: _renew_subs 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd00a4060 con 0x7f1fd009f020 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc403b400 con 0x7f1fd009f020 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fb6ffd640 10 monclient: dump: 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 10 monclient: _renew_subs 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd00995b0 con 0x7f1fd009f020 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.865+0000 7f1fcd7fa640 4 set_mon_vals no callback set 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fd58af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1fc4039040 con 0x7f1fd009f020 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fd58af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fd58af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd00008d0 con 0x7f1fd009f020 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fb6ffd640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fbc038720 0x7f1fbc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6494+0+0 (secure 0 0 0) 0x7f1fc406c420 con 0x7f1fd009f020 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fceffd640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fbc038720 0x7f1fbc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.866+0000 7f1fceffd640 10 monclient: get_auth_request con 0x7f1fbc038720 auth_method 0 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.868+0000 7f1fceffd640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fbc038720 0x7f1fbc03abd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0005070 tx=0x7f1fc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:23.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.870+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1fc4039ad0 con 0x7f1fd009f020 2024-09-15T03:32:23.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.870+0000 7f1fb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:23.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:23.870+0000 7f1fb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.029+0000 7f1fd58af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T03:32:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.029+0000 7f1fd58af640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T03:32:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.029+0000 7f1fd58af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.029+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f1fd009b2d0 con 0x7f1fd009f020 2024-09-15T03:32:24.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.737+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v82) v1 ==== 140+0+0 (secure 0 0 0) 0x7f1fd009b2d0 con 0x7f1fd009f020 2024-09-15T03:32:24.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.737+0000 7f1fb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T03:32:24.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.737+0000 7f1fb6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-15T03:32:24.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.837+0000 7f1fd58af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T03:32:24.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.837+0000 7f1fd58af640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T03:32:24.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.837+0000 7f1fd58af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.837+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f1fd009b2d0 con 0x7f1fd009f020 2024-09-15T03:32:24.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.839+0000 7f1fb6ffd640 1 -- 172.21.15.181:0/460747104 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v82) v1 ==== 140+0+0 (secure 0 0 0) 0x7f1fd009b2d0 con 0x7f1fd009f020 2024-09-15T03:32:24.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.839+0000 7f1fb6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-15T03:32:24.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.839+0000 7f1fb6ffd640 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-15T03:32:24.841 INFO:tasks.workunit.client.0.smithi181.stderr:device 'host-for-test' does not appear in the crush map 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fbc038720 msgr2=0x7f1fbc03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fbc038720 0x7f1fbc03abd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0005070 tx=0x7f1fc0009040 comp rx=0 tx=0).stop 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 10 monclient: shutdown 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 msgr2=0x7f1fd0097c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0097c50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009360 tx=0x7f1fc4032000 comp rx=0 tx=0).stop 2024-09-15T03:32:24.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 shutdown_connections 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fbc038720 0x7f1fbc03abd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fd0136d60 0x7f1fd0098190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd009f020 0x7f1fd0097c50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 --2- 172.21.15.181:0/460747104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd009ec50 0x7f1fd0097710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 >> 172.21.15.181:0/460747104 conn(0x7f1fd00a5cf0 msgr2=0x7f1fd009d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 shutdown_connections 2024-09-15T03:32:24.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.841+0000 7f1fd58af640 1 -- 172.21.15.181:0/460747104 wait complete. 2024-09-15T03:32:24.853 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm rack-for-test 2024-09-15T03:32:24.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.924+0000 7f874dc4f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:24.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.924+0000 7f874dc4f640 10 monclient: build_initial_monmap 2024-09-15T03:32:24.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.924+0000 7f874dc4f640 10 monclient: monmap: 2024-09-15T03:32:24.926 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:24.926347+0000 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:24.926347+0000 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:24.927 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:24.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 Processor -- start 2024-09-15T03:32:24.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 -- start start 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient: init 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f874809c250 0x7f874809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): picked mon.noname-a con 0x7f874809c250 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f874809dfa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f874809e3a0 0x7f874809cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): picked mon.noname-c con 0x7f874809e3a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874809b790 con 0x7f874809dfa0 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874809b230 con 0x7f874809c250 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:24.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874809b4b0 con 0x7f874809e3a0 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.925+0000 7f874dc4f640 10 monclient(hunting): authenticate will time out at 949.407935s 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f874809e3a0 0x7f874809cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f874809e3a0 0x7f874809cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49132/0 (socket says 172.21.15.181:49132) 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 -- 172.21.15.181:0/235407620 learned_addr learned my addr 172.21.15.181:0/235407620 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): get_auth_request con 0x7f874809e3a0 auth_method 0 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:24.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient(hunting): handle_auth_done global_id 44791 payload 274 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 -- 172.21.15.181:0/235407620 >> v2:172.21.15.153:3300/0 conn(0x7f874809c250 msgr2=0x7f874809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f87477fe640 1 --2- 172.21.15.181:0/235407620 >> v2:172.21.15.153:3300/0 conn(0x7f874809c250 0x7f874809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 --2- 172.21.15.181:0/235407620 >> v2:172.21.15.153:3300/0 conn(0x7f874809c250 0x7f874809da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 -- 172.21.15.181:0/235407620 >> v1:172.21.15.153:6789/0 conn(0x7f874809dfa0 legacy=0x7f874809c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 monclient: found mon.noname-c 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.926+0000 7f8747fff640 1 -- 172.21.15.181:0/235407620 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874809ba70 con 0x7f874809e3a0 2024-09-15T03:32:24.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f8747fff640 1 -- 172.21.15.181:0/235407620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809e3a0 msgr2=0x7f874809cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:24.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f8747fff640 1 --2- 172.21.15.181:0/235407620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809e3a0 0x7f874809cd30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f874809c200 tx=0x7f8740009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e053b9a73037d783 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 1 -- 172.21.15.181:0/235407620 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740007610 con 0x7f874809e3a0 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: dump: 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 1 -- 172.21.15.181:0/235407620 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874002d050 con 0x7f874809e3a0 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 5 monclient: authenticate success, global_id 44791 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 1 -- 172.21.15.181:0/235407620 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87400079a0 con 0x7f874809e3a0 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 4 set_mon_vals no callback set 2024-09-15T03:32:24.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f87467fc640 10 monclient: dump: 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 10 monclient: shutdown 2024-09-15T03:32:24.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 -- 172.21.15.181:0/235407620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809e3a0 msgr2=0x7f874809cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 --2- 172.21.15.181:0/235407620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809e3a0 0x7f874809cd30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f874809c200 tx=0x7f8740009840 comp rx=0 tx=0).stop 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 -- 172.21.15.181:0/235407620 shutdown_connections 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 --2- 172.21.15.181:0/235407620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809e3a0 0x7f874809cd30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 --2- 172.21.15.181:0/235407620 >> v2:172.21.15.153:3300/0 conn(0x7f874809c250 0x7f874809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 -- 172.21.15.181:0/235407620 >> 172.21.15.181:0/235407620 conn(0x7f8748071340 msgr2=0x7f87480a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 -- 172.21.15.181:0/235407620 shutdown_connections 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 -- 172.21.15.181:0/235407620 wait complete. 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 10 monclient: build_initial_monmap 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 10 monclient: monmap: 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:24.941 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 Processor -- start 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 1 -- start start 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.930+0000 7f874dc4f640 10 monclient: init 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874809c250 0x7f87481328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:24.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): picked mon.noname-a con 0x7f874809c250 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): picked mon.noname-b con 0x7f874809dfa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874809e3a0 0x7f874811c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): picked mon.noname-c con 0x7f874809e3a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874811fa00 con 0x7f874809c250 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874811e4c0 con 0x7f874809dfa0 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874806e970 con 0x7f874809e3a0 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f874dc4f640 10 monclient(hunting): authenticate will time out at 949.413402s 2024-09-15T03:32:24.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8747fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874809e3a0 msgr2=0x7f874811c980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49140/0 (socket says 172.21.15.181:49140) 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 -- 172.21.15.181:0/2198237321 learned_addr learned my addr 172.21.15.181:0/2198237321 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): get_auth_request con 0x7f874809dfa0 auth_method 0 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8747fff640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874809e3a0 0x7f874811c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f87477fe640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874809c250 0x7f87481328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:24.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient(hunting): handle_auth_done global_id 44794 payload 824 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 -- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874809e3a0 msgr2=0x7f874811c980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874809e3a0 0x7f874811c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 -- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874809c250 msgr2=0x7f87481328e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874809c250 0x7f87481328e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 monclient: found mon.noname-b 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f8746ffd640 1 -- 172.21.15.181:0/2198237321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740009ae0 con 0x7f874809dfa0 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f87477fe640 10 monclient: get_auth_request con 0x7f874809c250 auth_method 0 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.931+0000 7f87477fe640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874809c250 0x7f87481328e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.933+0000 7f8746ffd640 1 -- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 msgr2=0x7f874811c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:24.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.933+0000 7f8746ffd640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f873c00b5e0 tx=0x7f873c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00f880 con 0x7f874809dfa0 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: dump: 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:24.946 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c00fa00 con 0x7f874809dfa0 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 5 monclient: authenticate success, global_id 44794 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 10 monclient: _renew_subs 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87480a2980 con 0x7f874809dfa0 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00fdc0 con 0x7f874809dfa0 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 10 monclient: dump: 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:24.947 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f8745ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 10 monclient: _renew_subs 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874811eb10 con 0x7f874809dfa0 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.934+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f873c010630 con 0x7f874809dfa0 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f874dc4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:24.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f874dc4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f874dc4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87480008d0 con 0x7f874809dfa0 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f872f7fe640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f87340386f0 0x7f873403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f873c04bea0 con 0x7f874809dfa0 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f87477fe640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f87340386f0 0x7f873403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.935+0000 7f87477fe640 10 monclient: get_auth_request con 0x7f87340386f0 auth_method 0 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.937+0000 7f87477fe640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f87340386f0 0x7f873403aba0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87380089f0 tx=0x7f8738008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:24.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:24.939+0000 7f8746ffd640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f873c00b5e0 tx=0x7f873c0090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f873c01a460 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:32:25.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.768+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f873c01a460 con 0x7f874809dfa0 2024-09-15T03:32:25.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.768+0000 7f872f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:25.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.768+0000 7f872f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:25.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.928+0000 7f874dc4f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T03:32:25.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.928+0000 7f874dc4f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T03:32:25.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.928+0000 7f874dc4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:25.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:25.928+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f8748099ed0 con 0x7f874809dfa0 2024-09-15T03:32:26.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.750+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v83) v1 ==== 140+0+0 (secure 0 0 0) 0x7f873c017920 con 0x7f874809dfa0 2024-09-15T03:32:26.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.750+0000 7f872f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T03:32:26.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.750+0000 7f872f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-15T03:32:26.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.849+0000 7f874dc4f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T03:32:26.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.849+0000 7f874dc4f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T03:32:26.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.849+0000 7f874dc4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:26.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.849+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f874811dfa0 con 0x7f874809dfa0 2024-09-15T03:32:26.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.850+0000 7f872f7fe640 1 -- 172.21.15.181:0/2198237321 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v83) v1 ==== 140+0+0 (secure 0 0 0) 0x7f873c029090 con 0x7f874809dfa0 2024-09-15T03:32:26.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.850+0000 7f872f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-15T03:32:26.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.850+0000 7f872f7fe640 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-15T03:32:26.851 INFO:tasks.workunit.client.0.smithi181.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-15T03:32:26.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f87340386f0 msgr2=0x7f873403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:26.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f87340386f0 0x7f873403aba0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87380089f0 tx=0x7f8738008040 comp rx=0 tx=0).stop 2024-09-15T03:32:26.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 10 monclient: shutdown 2024-09-15T03:32:26.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 msgr2=0x7f874811c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f873c00b5e0 tx=0x7f873c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 shutdown_connections 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f87340386f0 0x7f873403aba0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874809e3a0 0x7f874811c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f874809dfa0 0x7f874811c440 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 --2- 172.21.15.181:0/2198237321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874809c250 0x7f87481328e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 >> 172.21.15.181:0/2198237321 conn(0x7f8748071340 msgr2=0x7f874806db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 shutdown_connections 2024-09-15T03:32:26.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.852+0000 7f874dc4f640 1 -- 172.21.15.181:0/2198237321 wait complete. 2024-09-15T03:32:26.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm root-for-test 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.935+0000 7f3afaa6f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.935+0000 7f3afaa6f640 10 monclient: build_initial_monmap 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.935+0000 7f3afaa6f640 10 monclient: monmap: 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:26.935883+0000 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:26.935883+0000 2024-09-15T03:32:26.936 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:26.937 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:26.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.935+0000 7f3afaa6f640 1 Processor -- start 2024-09-15T03:32:26.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.935+0000 7f3afaa6f640 1 -- start start 2024-09-15T03:32:26.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.935+0000 7f3afaa6f640 10 monclient: init 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3af409ec50 0x7f3af4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient(hunting): picked mon.noname-a con 0x7f3af409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3af41321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3af409f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af4067420 con 0x7f3af409f020 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40a0480 con 0x7f3af41321a0 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40a0200 con 0x7f3af409ec50 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:26.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3afaa6f640 10 monclient(hunting): authenticate will time out at 951.417309s 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3aebfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3af41321a0 0x7f3af4068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47082/0 (socket says 172.21.15.181:47082) 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3aebfff640 1 -- 172.21.15.181:0/794379256 learned_addr learned my addr 172.21.15.181:0/794379256 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 1 --2- 172.21.15.181:0/794379256 >> v2:172.21.15.153:3300/0 conn(0x7f3af409ec50 0x7f3af4131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): get_auth_request con 0x7f3af409ec50 auth_method 0 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:26.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient(hunting): handle_auth_done global_id 64257 payload 274 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 1 -- 172.21.15.181:0/794379256 >> v1:172.21.15.181:6789/0 conn(0x7f3af41321a0 legacy=0x7f3af4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 1 -- 172.21.15.181:0/794379256 >> v1:172.21.15.153:6789/0 conn(0x7f3af409f020 legacy=0x7f3af4069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient: _finish_hunting 0 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 1 monclient: found mon.noname-a 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.936+0000 7f3af880b640 1 -- 172.21.15.181:0/794379256 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af409fd80 con 0x7f3af409ec50 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.939+0000 7f3af880b640 1 -- 172.21.15.181:0/794379256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 msgr2=0x7f3af4131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.939+0000 7f3af880b640 1 --2- 172.21.15.181:0/794379256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af4131c60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009e60 tx=0x7f3af0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777e98c352c5f5d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 1 -- 172.21.15.181:0/794379256 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af0007630 con 0x7f3af409ec50 2024-09-15T03:32:26.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: dump: 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 1 -- 172.21.15.181:0/794379256 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af002d050 con 0x7f3af409ec50 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 5 monclient: authenticate success, global_id 64257 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 4 set_mon_vals no callback set 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 1 -- 172.21.15.181:0/794379256 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af00079c0 con 0x7f3af409ec50 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3aeb7fe640 10 monclient: dump: 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:26.952 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.952 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.952 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:26.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:26.952 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:26.952 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:26.953 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 10 monclient: shutdown 2024-09-15T03:32:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 -- 172.21.15.181:0/794379256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 msgr2=0x7f3af4131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/794379256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af4131c60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009e60 tx=0x7f3af0009780 comp rx=0 tx=0).stop 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 -- 172.21.15.181:0/794379256 shutdown_connections 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/794379256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af4131c60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 -- 172.21.15.181:0/794379256 >> 172.21.15.181:0/794379256 conn(0x7f3af40a5cf0 msgr2=0x7f3af40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 -- 172.21.15.181:0/794379256 shutdown_connections 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.940+0000 7f3afaa6f640 1 -- 172.21.15.181:0/794379256 wait complete. 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.941+0000 7f3afaa6f640 10 monclient: build_initial_monmap 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.941+0000 7f3afaa6f640 10 monclient: monmap: 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:26.954 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.941+0000 7f3afaa6f640 1 Processor -- start 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.941+0000 7f3afaa6f640 1 -- start start 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.941+0000 7f3afaa6f640 10 monclient: init 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af409a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): picked mon.noname-a con 0x7f3af409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3af409f020 0x7f3af4130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:26.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): picked mon.noname-c con 0x7f3af409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 0x7f3af4141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): picked mon.noname-b con 0x7f3af41321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af409ddb0 con 0x7f3af409ec50 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af411df50 con 0x7f3af41321a0 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af411dc70 con 0x7f3af409f020 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3afaa6f640 10 monclient(hunting): authenticate will time out at 951.423161s 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3aebfff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3af409f020 msgr2=0x7f3af4130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:26.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3aebfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3af409f020 0x7f3af4130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:26.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 0x7f3af4141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:26.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 0x7f3af4141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49144/0 (socket says 172.21.15.181:49144) 2024-09-15T03:32:26.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 -- 172.21.15.181:0/4073570701 learned_addr learned my addr 172.21.15.181:0/4073570701 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:26.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): get_auth_request con 0x7f3af41321a0 auth_method 0 2024-09-15T03:32:26.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:26.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af409a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): get_auth_request con 0x7f3af409ec50 auth_method 0 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:26.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af880b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient(hunting): handle_auth_done global_id 44797 payload 824 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 -- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3af409f020 msgr2=0x7f3af4130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3af409f020 0x7f3af4130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 -- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 msgr2=0x7f3af409a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:26.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af409a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 monclient: found mon.noname-b 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.942+0000 7f3af900c640 1 -- 172.21.15.181:0/4073570701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0009a20 con 0x7f3af41321a0 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.944+0000 7f3af900c640 1 -- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 msgr2=0x7f3af4141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.944+0000 7f3af900c640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 0x7f3af4141170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00c5e0 tx=0x7f3aec007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:26.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.944+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:32:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec012840 con 0x7f3af41321a0 2024-09-15T03:32:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: dump: 2024-09-15T03:32:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec0129a0 con 0x7f3af41321a0 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3afaa6f640 5 monclient: authenticate success, global_id 44797 2024-09-15T03:32:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3afaa6f640 10 monclient: _renew_subs 2024-09-15T03:32:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3afaa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af411c840 con 0x7f3af41321a0 2024-09-15T03:32:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec012da0 con 0x7f3af41321a0 2024-09-15T03:32:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.946+0000 7f3ae8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:26.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.947+0000 7f3ae8ff9640 10 monclient: dump: 2024-09-15T03:32:26.965 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:26.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:26.965 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.965 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:26.965 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:26.966 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:26.966 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:26.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:26.966 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:26.966 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:26.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.947+0000 7f3afaa6f640 10 monclient: _renew_subs 2024-09-15T03:32:26.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.947+0000 7f3afaa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:26.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.947+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af411be10 con 0x7f3af41321a0 2024-09-15T03:32:26.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.947+0000 7f3aeaffd640 4 set_mon_vals no callback set 2024-09-15T03:32:26.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3aec011590 con 0x7f3af41321a0 2024-09-15T03:32:26.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3afaa6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:26.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3afaa6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:26.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3afaa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:26.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af409c090 con 0x7f3af41321a0 2024-09-15T03:32:26.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3ae8ff9640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ae0038740 0x7f3ae003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:26.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6128+0+0 (secure 0 0 0) 0x7f3aec04c0e0 con 0x7f3af41321a0 2024-09-15T03:32:26.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3af880b640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ae0038740 0x7f3ae003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:26.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.948+0000 7f3af880b640 10 monclient: get_auth_request con 0x7f3ae0038740 auth_method 0 2024-09-15T03:32:26.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.950+0000 7f3af880b640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ae0038740 0x7f3ae003abf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3af409bf50 tx=0x7f3af0037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:26.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.952+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3af409c090 con 0x7f3af41321a0 2024-09-15T03:32:26.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.952+0000 7f3ae8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:26.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:26.952+0000 7f3ae8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:27.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.112+0000 7f3afaa6f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T03:32:27.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.112+0000 7f3afaa6f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T03:32:27.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.112+0000 7f3afaa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.112+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f3af40008d0 con 0x7f3af41321a0 2024-09-15T03:32:27.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.756+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v84) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3aec016090 con 0x7f3af41321a0 2024-09-15T03:32:27.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.756+0000 7f3ae8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T03:32:27.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.756+0000 7f3ae8ff9640 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-15T03:32:27.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.854+0000 7f3afaa6f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T03:32:27.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.854+0000 7f3afaa6f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T03:32:27.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.854+0000 7f3afaa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.854+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f3af40a13e0 con 0x7f3af41321a0 2024-09-15T03:32:27.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.855+0000 7f3ae8ff9640 1 -- 172.21.15.181:0/4073570701 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v84) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3aec0102e0 con 0x7f3af41321a0 2024-09-15T03:32:27.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.855+0000 7f3ae8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-15T03:32:27.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.855+0000 7f3ae8ff9640 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-15T03:32:27.856 INFO:tasks.workunit.client.0.smithi181.stderr:device 'root-for-test' does not appear in the crush map 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ae0038740 msgr2=0x7f3ae003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ae0038740 0x7f3ae003abf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3af409bf50 tx=0x7f3af0037000 comp rx=0 tx=0).stop 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 10 monclient: shutdown 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 msgr2=0x7f3af4141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 0x7f3af4141170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00c5e0 tx=0x7f3aec007ed0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 shutdown_connections 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ae0038740 0x7f3ae003abf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3af41321a0 0x7f3af4141170 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3af409f020 0x7f3af4130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 --2- 172.21.15.181:0/4073570701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3af409ec50 0x7f3af409a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.857+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 >> 172.21.15.181:0/4073570701 conn(0x7f3af40a5cf0 msgr2=0x7f3af409e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:27.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.858+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 shutdown_connections 2024-09-15T03:32:27.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.858+0000 7f3afaa6f640 1 -- 172.21.15.181:0/4073570701 wait complete. 2024-09-15T03:32:27.868 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.944+0000 7f930d154640 10 monclient: get_monmap_and_config 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.944+0000 7f930d154640 10 monclient: build_initial_monmap 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.944+0000 7f930d154640 10 monclient: monmap: 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:27.945744+0000 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:27.945744+0000 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:27.946 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:27.947 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:27.947 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:27.947 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.945+0000 7f930d154640 1 Processor -- start 2024-09-15T03:32:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.945+0000 7f930d154640 1 -- start start 2024-09-15T03:32:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.945+0000 7f930d154640 10 monclient: init 2024-09-15T03:32:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:32:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f930809cad0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f930809c1a0 0x7f93081348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): picked mon.noname-a con 0x7f930809c1a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9308134e10 0x7f93081371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): picked mon.noname-c con 0x7f9308134e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930809d5b0 con 0x7f930809cad0 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930809b710 con 0x7f930809c1a0 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930809a650 con 0x7f9308134e10 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930d154640 10 monclient(hunting): authenticate will time out at 952.427151s 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9308134e10 0x7f93081371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:27.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f9306d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f930809cad0 0x7f930809cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47084/0 (socket says 172.21.15.181:47084) 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9308134e10 0x7f93081371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49146/0 (socket says 172.21.15.181:49146) 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f9306d9d640 1 -- 172.21.15.181:0/1449519594 learned_addr learned my addr 172.21.15.181:0/1449519594 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): get_auth_request con 0x7f9308134e10 auth_method 0 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930659c640 1 --2- 172.21.15.181:0/1449519594 >> v2:172.21.15.153:3300/0 conn(0x7f930809c1a0 0x7f93081348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient(hunting): handle_auth_done global_id 44800 payload 274 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 -- 172.21.15.181:0/1449519594 >> v2:172.21.15.153:3300/0 conn(0x7f930809c1a0 msgr2=0x7f93081348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 --2- 172.21.15.181:0/1449519594 >> v2:172.21.15.153:3300/0 conn(0x7f930809c1a0 0x7f93081348d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 -- 172.21.15.181:0/1449519594 >> v1:172.21.15.181:6789/0 conn(0x7f930809cad0 legacy=0x7f930809cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:27.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient: _finish_hunting 0 2024-09-15T03:32:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 monclient: found mon.noname-c 2024-09-15T03:32:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930759e640 1 -- 172.21.15.181:0/1449519594 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930809f660 con 0x7f9308134e10 2024-09-15T03:32:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930659c640 10 monclient: get_auth_request con 0x7f930809c1a0 auth_method 0 2024-09-15T03:32:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.946+0000 7f930659c640 1 --2- 172.21.15.181:0/1449519594 >> v2:172.21.15.153:3300/0 conn(0x7f930809c1a0 0x7f93081348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:27.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930759e640 1 -- 172.21.15.181:0/1449519594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9308134e10 msgr2=0x7f93081371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:27.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930759e640 1 --2- 172.21.15.181:0/1449519594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9308134e10 0x7f93081371f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009e60 tx=0x7f92f8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=687ec22b1201d4dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:27.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 1 -- 172.21.15.181:0/1449519594 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8007630 con 0x7f9308134e10 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: dump: 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:27.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 1 -- 172.21.15.181:0/1449519594 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f802d050 con 0x7f9308134e10 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 5 monclient: authenticate success, global_id 44800 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 4 set_mon_vals no callback set 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 1 -- 172.21.15.181:0/1449519594 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f80079c0 con 0x7f9308134e10 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f9305d9b640 10 monclient: dump: 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:27.959 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 10 monclient: shutdown 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 1 -- 172.21.15.181:0/1449519594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9308134e10 msgr2=0x7f93081371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.949+0000 7f930d154640 1 --2- 172.21.15.181:0/1449519594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9308134e10 0x7f93081371f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009e60 tx=0x7f92f8009780 comp rx=0 tx=0).stop 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 1 -- 172.21.15.181:0/1449519594 shutdown_connections 2024-09-15T03:32:27.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 1 --2- 172.21.15.181:0/1449519594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9308134e10 0x7f93081371f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 1 --2- 172.21.15.181:0/1449519594 >> v2:172.21.15.153:3300/0 conn(0x7f930809c1a0 0x7f93081348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 1 -- 172.21.15.181:0/1449519594 >> 172.21.15.181:0/1449519594 conn(0x7f9308070e90 msgr2=0x7f93080a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 1 -- 172.21.15.181:0/1449519594 shutdown_connections 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 1 -- 172.21.15.181:0/1449519594 wait complete. 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 10 monclient: build_initial_monmap 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.950+0000 7f930d154640 10 monclient: monmap: 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:27.961 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 Processor -- start 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 -- start start 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient: init 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 0x7f9308134860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): picked mon.noname-b con 0x7f930809c1a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930809cad0 0x7f930811c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): picked mon.noname-a con 0x7f930809cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9308134e10 0x7f930811c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): picked mon.noname-c con 0x7f9308134e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:27.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930806eba0 con 0x7f930809cad0 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080a2100 con 0x7f930809c1a0 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9308120440 con 0x7f9308134e10 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930d154640 10 monclient(hunting): authenticate will time out at 952.432561s 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 0x7f9308134860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 0x7f9308134860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49162/0 (socket says 172.21.15.181:49162) 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 -- 172.21.15.181:0/2831997977 learned_addr learned my addr 172.21.15.181:0/2831997977 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): get_auth_request con 0x7f930809c1a0 auth_method 0 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:27.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930759e640 1 -- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9308134e10 msgr2=0x7f930811c620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930759e640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9308134e10 0x7f930811c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient(hunting): handle_auth_done global_id 44803 payload 824 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 -- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9308134e10 msgr2=0x7f930811c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9308134e10 0x7f930811c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 -- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930809cad0 msgr2=0x7f930811c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930659c640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930809cad0 0x7f930811c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930809cad0 0x7f930811c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:27.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 monclient: found mon.noname-b 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f9306d9d640 1 -- 172.21.15.181:0/2831997977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f8009a20 con 0x7f930809c1a0 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930659c640 10 monclient: get_auth_request con 0x7f930809cad0 auth_method 0 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.951+0000 7f930659c640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930809cad0 0x7f930811c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f9306d9d640 1 -- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 msgr2=0x7f9308134860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f9306d9d640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 0x7f9308134860 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f930000c610 tx=0x7f9300004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300005cb0 con 0x7f930809c1a0 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: dump: 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.966 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.966 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:27.966 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:27.966 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:27.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:27.966 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:27.967 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:27.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:27.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 5 monclient: authenticate success, global_id 44803 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9300005e30 con 0x7f930809c1a0 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300007340 con 0x7f930809c1a0 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 10 monclient: _renew_subs 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930811f780 con 0x7f930809c1a0 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f92feffd640 10 monclient: dump: 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:27.968 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930559a640 4 set_mon_vals no callback set 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 10 monclient: _renew_subs 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.954+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930811dcc0 con 0x7f930809c1a0 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f930001b400 con 0x7f930809c1a0 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f930d154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:27.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f930d154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f930d154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f92feffd640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92e8038720 0x7f92e803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930811e230 con 0x7f930809c1a0 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f930004bc90 con 0x7f930809c1a0 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f930659c640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92e8038720 0x7f92e803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.955+0000 7f930659c640 10 monclient: get_auth_request con 0x7f92e8038720 auth_method 0 2024-09-15T03:32:27.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.957+0000 7f930659c640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92e8038720 0x7f92e803abd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f930811d6e0 tx=0x7f92f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:27.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.959+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f930811e230 con 0x7f930809c1a0 2024-09-15T03:32:27.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.959+0000 7f92feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:27.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:27.959+0000 7f92feffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:28.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.118+0000 7f930d154640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T03:32:28.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.118+0000 7f930d154640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T03:32:28.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.118+0000 7f930d154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.118+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f93080008d0 con 0x7f930809c1a0 2024-09-15T03:32:28.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.758+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v85) v1 ==== 145+0+0 (secure 0 0 0) 0x7f930001a960 con 0x7f930809c1a0 2024-09-15T03:32:28.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.759+0000 7f92feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T03:32:28.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.759+0000 7f92feffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:28.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.857+0000 7f930d154640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T03:32:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.857+0000 7f930d154640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T03:32:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.857+0000 7f930d154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.857+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f930809b430 con 0x7f930809c1a0 2024-09-15T03:32:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.859+0000 7f92feffd640 1 -- 172.21.15.181:0/2831997977 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v85) v1 ==== 173+0+0 (secure 0 0 0) 0x7f930001a360 con 0x7f930809c1a0 2024-09-15T03:32:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.859+0000 7f92feffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-15T03:32:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.859+0000 7f92feffd640 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-15T03:32:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo-rule already exists 2024-09-15T03:32:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92e8038720 msgr2=0x7f92e803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92e8038720 0x7f92e803abd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f930811d6e0 tx=0x7f92f0008040 comp rx=0 tx=0).stop 2024-09-15T03:32:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 10 monclient: shutdown 2024-09-15T03:32:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 msgr2=0x7f9308134860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 0x7f9308134860 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f930000c610 tx=0x7f9300004960 comp rx=0 tx=0).stop 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 shutdown_connections 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92e8038720 0x7f92e803abd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9308134e10 0x7f930811c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f930809cad0 0x7f930811c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 --2- 172.21.15.181:0/2831997977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930809c1a0 0x7f9308134860 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 >> 172.21.15.181:0/2831997977 conn(0x7f9308070e90 msgr2=0x7f93080a0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 shutdown_connections 2024-09-15T03:32:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.861+0000 7f930d154640 1 -- 172.21.15.181:0/2831997977 wait complete. 2024-09-15T03:32:28.871 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush rm foo 2024-09-15T03:32:28.871 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:32:28.872 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm foo 2024-09-15T03:32:28.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.945+0000 7fdab18af640 10 monclient: get_monmap_and_config 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.945+0000 7fdab18af640 10 monclient: build_initial_monmap 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.945+0000 7fdab18af640 10 monclient: monmap: 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:28.946291+0000 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:28.946291+0000 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:28.947 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:28.948 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:28.948 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:28.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 Processor -- start 2024-09-15T03:32:28.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 -- start start 2024-09-15T03:32:28.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient: init 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdaac09ec50 0x7fdaac131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): picked mon.noname-c con 0x7fdaac09ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdaac1321a0 0x7fdaac068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): picked mon.noname-a con 0x7fdaac1321a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac09f020 0x7fdaac069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): picked mon.noname-b con 0x7fdaac09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:28.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaac067420 con 0x7fdaac1321a0 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaac0a0480 con 0x7fdaac09ec50 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac0a0200 con 0x7fdaac09f020 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdab18af640 10 monclient(hunting): authenticate will time out at 953.427705s 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdaac09ec50 0x7fdaac131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdaac09ec50 0x7fdaac131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43606/0 (socket says 172.21.15.181:43606) 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaab7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac09f020 msgr2=0x7fdaac069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 1 -- 172.21.15.181:0/200229466 learned_addr learned my addr 172.21.15.181:0/200229466 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): get_auth_request con 0x7fdaac09ec50 auth_method 0 2024-09-15T03:32:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaab7fe640 1 --2- 172.21.15.181:0/200229466 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac09f020 0x7fdaac069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.946+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaa7fc640 1 --2- 172.21.15.181:0/200229466 >> v2:172.21.15.153:3300/0 conn(0x7fdaac1321a0 0x7fdaac068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaa7fc640 10 monclient(hunting): get_auth_request con 0x7fdaac1321a0 auth_method 0 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_done global_id 44806 payload 274 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 1 -- 172.21.15.181:0/200229466 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac09f020 msgr2=0x7fdaac069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:28.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 1 --2- 172.21.15.181:0/200229466 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac09f020 0x7fdaac069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 1 -- 172.21.15.181:0/200229466 >> v2:172.21.15.153:3300/0 conn(0x7fdaac1321a0 msgr2=0x7fdaac068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 1 --2- 172.21.15.181:0/200229466 >> v2:172.21.15.153:3300/0 conn(0x7fdaac1321a0 0x7fdaac068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 1 monclient: found mon.noname-c 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.947+0000 7fdaaaffd640 1 -- 172.21.15.181:0/200229466 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaac09fd80 con 0x7fdaac09ec50 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.949+0000 7fdaaaffd640 1 -- 172.21.15.181:0/200229466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 msgr2=0x7fdaac131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.949+0000 7fdaaaffd640 1 --2- 172.21.15.181:0/200229466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac131c60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0719d0 tx=0x7fda9c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff9daf1714317ba1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 1 -- 172.21.15.181:0/200229466 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c032440 con 0x7fdaac09ec50 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:28.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: dump: 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:28.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 5 monclient: authenticate success, global_id 44806 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 1 -- 172.21.15.181:0/200229466 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c0325a0 con 0x7fdaac09ec50 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 4 set_mon_vals no callback set 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 1 -- 172.21.15.181:0/200229466 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c032940 con 0x7fdaac09ec50 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdaa9ffb640 10 monclient: dump: 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 10 monclient: shutdown 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 -- 172.21.15.181:0/200229466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 msgr2=0x7fdaac131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 --2- 172.21.15.181:0/200229466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac131c60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0719d0 tx=0x7fda9c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 -- 172.21.15.181:0/200229466 shutdown_connections 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 --2- 172.21.15.181:0/200229466 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac09f020 0x7fdaac069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 --2- 172.21.15.181:0/200229466 >> v2:172.21.15.153:3300/0 conn(0x7fdaac1321a0 0x7fdaac068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 --2- 172.21.15.181:0/200229466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac131c60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 -- 172.21.15.181:0/200229466 >> 172.21.15.181:0/200229466 conn(0x7fdaac0a5cf0 msgr2=0x7fdaac0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 -- 172.21.15.181:0/200229466 shutdown_connections 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.950+0000 7fdab18af640 1 -- 172.21.15.181:0/200229466 wait complete. 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 10 monclient: build_initial_monmap 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 10 monclient: monmap: 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:28.959 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 1 Processor -- start 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 1 -- start start 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 10 monclient: init 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac11de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.951+0000 7fdab18af640 10 monclient(hunting): picked mon.noname-b con 0x7fdaac09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdaac09f020 0x7fdaac11e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): picked mon.noname-a con 0x7fdaac09f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:28.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac1321a0 0x7fdaac11e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): picked mon.noname-c con 0x7fdaac1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac1203c0 con 0x7fdaac09f020 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac09c020 con 0x7fdaac09ec50 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac09e170 con 0x7fdaac1321a0 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdab18af640 10 monclient(hunting): authenticate will time out at 953.433062s 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac11de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaab7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac1321a0 msgr2=0x7fdaac11e8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:28.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaab7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac1321a0 0x7fdaac11e8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac11de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43618/0 (socket says 172.21.15.181:43618) 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 -- 172.21.15.181:0/2988959754 learned_addr learned my addr 172.21.15.181:0/2988959754 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): get_auth_request con 0x7fdaac09ec50 auth_method 0 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdaac09f020 0x7fdaac11e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): get_auth_request con 0x7fdaac09f020 auth_method 0 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:28.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient(hunting): handle_auth_done global_id 44809 payload 824 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 -- 172.21.15.181:0/2988959754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac1321a0 msgr2=0x7fdaac11e8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdaac1321a0 0x7fdaac11e8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 -- 172.21.15.181:0/2988959754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdaac09f020 msgr2=0x7fdaac11e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdaac09f020 0x7fdaac11e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 monclient: found mon.noname-b 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.952+0000 7fdaaaffd640 1 -- 172.21.15.181:0/2988959754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c009450 con 0x7fdaac09ec50 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdaaaffd640 1 -- 172.21.15.181:0/2988959754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 msgr2=0x7fdaac11de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdaaaffd640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaac09ec50 0x7fdaac11de30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda9c035040 tx=0x7fda9c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c031b10 con 0x7fdaac09ec50 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: dump: 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:28.968 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c031c90 con 0x7fdaac09ec50 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 5 monclient: authenticate success, global_id 44809 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 10 monclient: _renew_subs 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 1 -- 172.21.15.181:0/2988959754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaac1314d0 con 0x7fdaac09ec50 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c03d690 con 0x7fdaac09ec50 2024-09-15T03:32:28.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fda92ffd640 10 monclient: dump: 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:28.970 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 10 monclient: _renew_subs 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdaa97fa640 4 set_mon_vals no callback set 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.954+0000 7fdab18af640 1 -- 172.21.15.181:0/2988959754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaac121440 con 0x7fdaac09ec50 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.955+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fda9c039040 con 0x7fdaac09ec50 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.955+0000 7fdab18af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.955+0000 7fdab18af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.955+0000 7fdab18af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.955+0000 7fdab18af640 1 -- 172.21.15.181:0/2988959754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaac0008d0 con 0x7fdaac09ec50 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.956+0000 7fda92ffd640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda980386f0 0x7fda9803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.956+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fda9c06be90 con 0x7fdaac09ec50 2024-09-15T03:32:28.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.956+0000 7fdaaa7fc640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda980386f0 0x7fda9803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:28.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.956+0000 7fdaaa7fc640 10 monclient: get_auth_request con 0x7fda980386f0 auth_method 0 2024-09-15T03:32:28.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.958+0000 7fdaaa7fc640 1 --2- 172.21.15.181:0/2988959754 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda980386f0 0x7fda9803aba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0007ba0 tx=0x7fdaa0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:28.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.960+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fda9c02f2a0 con 0x7fdaac09ec50 2024-09-15T03:32:28.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.960+0000 7fda92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:28.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:28.960+0000 7fda92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:29.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.119+0000 7fdab18af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:29.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.119+0000 7fdab18af640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:29.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.119+0000 7fdab18af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.119+0000 7fdab18af640 1 -- 172.21.15.181:0/2988959754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fdaac0a13e0 con 0x7fdaac09ec50 2024-09-15T03:32:29.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.128+0000 7fda92ffd640 1 -- 172.21.15.181:0/2988959754 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v85) v1 ==== 103+0+0 (secure 0 0 0) 0x7fda9c037090 con 0x7fdaac09ec50 2024-09-15T03:32:29.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.128+0000 7fda92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:29.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.128+0000 7fda92ffd640 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-15T03:32:29.129 INFO:tasks.workunit.client.0.smithi181.stderr:Error EBUSY: (16) Device or resource busy 2024-09-15T03:32:29.132 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:32:29.132 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rule rm foo-rule 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.206+0000 7f860684f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.206+0000 7f860684f640 10 monclient: build_initial_monmap 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.206+0000 7f860684f640 10 monclient: monmap: 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:29.206911+0000 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:29.206911+0000 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:29.207 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:29.208 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.206+0000 7f860684f640 1 Processor -- start 2024-09-15T03:32:29.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.206+0000 7f860684f640 1 -- start start 2024-09-15T03:32:29.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.206+0000 7f860684f640 10 monclient: init 2024-09-15T03:32:29.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:29.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f860009ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f860009f020 0x7f8600069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): picked mon.noname-c con 0x7f860009f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8600136d60 0x7f8600139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): picked mon.noname-a con 0x7f8600136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8600067420 con 0x7f860009ec50 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860009fdc0 con 0x7f8600136d60 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000a0510 con 0x7f860009f020 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f860684f640 10 monclient(hunting): authenticate will time out at 953.688492s 2024-09-15T03:32:29.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f860009f020 0x7f8600069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f860009f020 0x7f8600069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43632/0 (socket says 172.21.15.181:43632) 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 -- 172.21.15.181:0/429577025 learned_addr learned my addr 172.21.15.181:0/429577025 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): get_auth_request con 0x7f860009f020 auth_method 0 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f8604dec640 1 --2- 172.21.15.181:0/429577025 >> v2:172.21.15.153:3300/0 conn(0x7f8600136d60 0x7f8600139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f8604dec640 10 monclient(hunting): get_auth_request con 0x7f8600136d60 auth_method 0 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f8604dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f8604dec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f8604dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_done global_id 44812 payload 274 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 -- 172.21.15.181:0/429577025 >> v2:172.21.15.153:3300/0 conn(0x7f8600136d60 msgr2=0x7f8600139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 --2- 172.21.15.181:0/429577025 >> v2:172.21.15.153:3300/0 conn(0x7f8600136d60 0x7f8600139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 -- 172.21.15.181:0/429577025 >> v1:172.21.15.153:6789/0 conn(0x7f860009ec50 legacy=0x7f8600131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 monclient: found mon.noname-c 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f85ff7fe640 1 -- 172.21.15.181:0/429577025 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000a0200 con 0x7f860009f020 2024-09-15T03:32:29.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.207+0000 7f8604dec640 1 --2- 172.21.15.181:0/429577025 >> v2:172.21.15.153:3300/0 conn(0x7f8600136d60 0x7f8600139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85ff7fe640 1 -- 172.21.15.181:0/429577025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 msgr2=0x7f8600069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85ff7fe640 1 --2- 172.21.15.181:0/429577025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600069d10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f86000a2b00 tx=0x7f85f40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b16413af7d034261 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 1 -- 172.21.15.181:0/429577025 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4032440 con 0x7f860009f020 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: dump: 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.219 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f860684f640 5 monclient: authenticate success, global_id 44812 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 1 -- 172.21.15.181:0/429577025 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f40325a0 con 0x7f860009f020 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f860684f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:29.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f860684f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f860684f640 4 set_mon_vals no callback set 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 1 -- 172.21.15.181:0/429577025 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4032940 con 0x7f860009f020 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.210+0000 7f85feffd640 10 monclient: dump: 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.221 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 10 monclient: shutdown 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 -- 172.21.15.181:0/429577025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 msgr2=0x7f8600069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 --2- 172.21.15.181:0/429577025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600069d10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f86000a2b00 tx=0x7f85f40091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 -- 172.21.15.181:0/429577025 shutdown_connections 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 --2- 172.21.15.181:0/429577025 >> v2:172.21.15.153:3300/0 conn(0x7f8600136d60 0x7f8600139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 --2- 172.21.15.181:0/429577025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600069d10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 -- 172.21.15.181:0/429577025 >> 172.21.15.181:0/429577025 conn(0x7f86000a5d40 msgr2=0x7f86000a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 -- 172.21.15.181:0/429577025 shutdown_connections 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 1 -- 172.21.15.181:0/429577025 wait complete. 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 10 monclient: build_initial_monmap 2024-09-15T03:32:29.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.211+0000 7f860684f640 10 monclient: monmap: 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 Processor -- start 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 -- start start 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient: init 2024-09-15T03:32:29.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f860009ec50 0x7f8600130910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): picked mon.noname-c con 0x7f860009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): picked mon.noname-b con 0x7f860009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8600136d60 0x7f8600099d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): picked mon.noname-a con 0x7f8600136d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000a4170 con 0x7f8600136d60 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860009c070 con 0x7f860009f020 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860009b2e0 con 0x7f860009ec50 2024-09-15T03:32:29.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.212+0000 7f860684f640 10 monclient(hunting): authenticate will time out at 953.693988s 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85fffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f860009ec50 msgr2=0x7f8600130910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85fffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f860009ec50 0x7f8600130910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600099850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43638/0 (socket says 172.21.15.181:43638) 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 -- 172.21.15.181:0/1346694560 learned_addr learned my addr 172.21.15.181:0/1346694560 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): get_auth_request con 0x7f860009f020 auth_method 0 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:29.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f8604dec640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8600136d60 0x7f8600099d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f8604dec640 10 monclient(hunting): get_auth_request con 0x7f8600136d60 auth_method 0 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f8604dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f8604dec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f8604dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient(hunting): handle_auth_done global_id 44815 payload 824 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 -- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f860009ec50 msgr2=0x7f8600130910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f860009ec50 0x7f8600130910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 -- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8600136d60 msgr2=0x7f8600099d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8600136d60 0x7f8600099d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 monclient: found mon.noname-b 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f85ff7fe640 1 -- 172.21.15.181:0/1346694560 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4009450 con 0x7f860009f020 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.213+0000 7f8604dec640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8600136d60 0x7f8600099d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85ff7fe640 1 -- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 msgr2=0x7f8600099850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85ff7fe640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600099850 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f85f4035040 tx=0x7f85f4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40319c0 con 0x7f860009f020 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: dump: 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f4031b20 con 0x7f860009f020 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 5 monclient: authenticate success, global_id 44815 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 10 monclient: _renew_subs 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860011ca20 con 0x7f860009f020 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f403d440 con 0x7f860009f020 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85e7fff640 10 monclient: dump: 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.229 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f85fe7fc640 4 set_mon_vals no callback set 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 10 monclient: _renew_subs 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.215+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860011c730 con 0x7f860009f020 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.216+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85f4039040 con 0x7f860009f020 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f860684f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f860684f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f860684f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86000008d0 con 0x7f860009f020 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f85e7fff640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f85ec0386f0 0x7f85ec03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f85f406bab0 con 0x7f860009f020 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f85fffff640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f85ec0386f0 0x7f85ec03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.217+0000 7f85fffff640 10 monclient: get_auth_request con 0x7f85ec0386f0 auth_method 0 2024-09-15T03:32:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.219+0000 7f85fffff640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f85ec0386f0 0x7f85ec03aba0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f85f0007fd0 tx=0x7f85f0007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.221+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85f402f4d0 con 0x7f860009f020 2024-09-15T03:32:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.221+0000 7f85e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.221+0000 7f85e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:29.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.379+0000 7f860684f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T03:32:29.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.379+0000 7f860684f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T03:32:29.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.379+0000 7f860684f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.379+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f86000a13e0 con 0x7f860009f020 2024-09-15T03:32:29.758 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.757+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v86) v1 ==== 85+0+0 (secure 0 0 0) 0x7f85f4037070 con 0x7f860009f020 2024-09-15T03:32:29.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.758+0000 7f85e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T03:32:29.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.758+0000 7f85e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:29.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.853+0000 7f860684f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T03:32:29.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.853+0000 7f860684f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T03:32:29.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.853+0000 7f860684f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.853+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f86000a13e0 con 0x7f860009f020 2024-09-15T03:32:29.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.854+0000 7f85e7fff640 1 -- 172.21.15.181:0/1346694560 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v86) v1 ==== 113+0+0 (secure 0 0 0) 0x7f85f4030e60 con 0x7f860009f020 2024-09-15T03:32:29.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.854+0000 7f85e7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-15T03:32:29.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.854+0000 7f85e7fff640 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-15T03:32:29.855 INFO:tasks.workunit.client.0.smithi181.stderr:rule foo-rule does not exist 2024-09-15T03:32:29.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f85ec0386f0 msgr2=0x7f85ec03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:29.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f85ec0386f0 0x7f85ec03aba0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f85f0007fd0 tx=0x7f85f0007a70 comp rx=0 tx=0).stop 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 10 monclient: shutdown 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 msgr2=0x7f8600099850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600099850 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f85f4035040 tx=0x7f85f4032000 comp rx=0 tx=0).stop 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 shutdown_connections 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f85ec0386f0 0x7f85ec03aba0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8600136d60 0x7f8600099d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f860009f020 0x7f8600099850 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 --2- 172.21.15.181:0/1346694560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f860009ec50 0x7f8600130910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 >> 172.21.15.181:0/1346694560 conn(0x7f86000a5d40 msgr2=0x7f860009daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 shutdown_connections 2024-09-15T03:32:29.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.856+0000 7f860684f640 1 -- 172.21.15.181:0/1346694560 wait complete. 2024-09-15T03:32:29.866 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm bar 2024-09-15T03:32:29.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.939+0000 7f1bb399d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:29.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.939+0000 7f1bb399d640 10 monclient: build_initial_monmap 2024-09-15T03:32:29.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.939+0000 7f1bb399d640 10 monclient: monmap: 2024-09-15T03:32:29.940 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:29.940132+0000 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:29.940132+0000 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.939+0000 7f1bb399d640 1 Processor -- start 2024-09-15T03:32:29.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 1 -- start start 2024-09-15T03:32:29.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient: init 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1bac06e000 0x7f1bac06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bac06e000 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bac06a710 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bac09e170 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac0699c0 con 0x7f1bac09e170 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac06be20 con 0x7f1bac06a710 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bac06a360 con 0x7f1bac06e000 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb399d640 10 monclient(hunting): authenticate will time out at 954.421726s 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1bac06e000 0x7f1bac06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1bac06e000 0x7f1bac06e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43642/0 (socket says 172.21.15.181:43642) 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 1 -- 172.21.15.181:0/703528328 learned_addr learned my addr 172.21.15.181:0/703528328 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb0f38640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f1bac06a710 0x7f1bac09da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:36926/0 (socket says 172.21.15.181:36926) 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): get_auth_request con 0x7f1bac06e000 auth_method 0 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.940+0000 7f1bb1739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 10 monclient(hunting): handle_auth_done global_id 44818 payload 274 2024-09-15T03:32:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 1 -- 172.21.15.181:0/703528328 >> v1:172.21.15.181:6789/0 conn(0x7f1bac06a710 legacy=0x7f1bac09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 1 -- 172.21.15.181:0/703528328 >> v1:172.21.15.153:6789/0 conn(0x7f1bac09e170 legacy=0x7f1bac09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 10 monclient: _finish_hunting 0 2024-09-15T03:32:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 1 monclient: found mon.noname-c 2024-09-15T03:32:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.941+0000 7f1bb1739640 1 -- 172.21.15.181:0/703528328 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac069f80 con 0x7f1bac06e000 2024-09-15T03:32:29.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.943+0000 7f1bb1739640 1 -- 172.21.15.181:0/703528328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac06e000 msgr2=0x7f1bac06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.943+0000 7f1bb1739640 1 --2- 172.21.15.181:0/703528328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac06e000 0x7f1bac06e3d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1bac063870 tx=0x7f1ba00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa4c6afcf4da3703 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.943+0000 7f1b9ffff640 1 -- 172.21.15.181:0/703528328 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0032440 con 0x7f1bac06e000 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: dump: 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.952 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: _finish_auth 0 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 1 -- 172.21.15.181:0/703528328 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba00325a0 con 0x7f1bac06e000 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 5 monclient: authenticate success, global_id 44818 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 1 -- 172.21.15.181:0/703528328 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0032940 con 0x7f1bac06e000 2024-09-15T03:32:29.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 4 set_mon_vals no callback set 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1b9ffff640 10 monclient: dump: 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 10 monclient: shutdown 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 -- 172.21.15.181:0/703528328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac06e000 msgr2=0x7f1bac06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 --2- 172.21.15.181:0/703528328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac06e000 0x7f1bac06e3d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1bac063870 tx=0x7f1ba00091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 -- 172.21.15.181:0/703528328 shutdown_connections 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 --2- 172.21.15.181:0/703528328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac06e000 0x7f1bac06e3d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 -- 172.21.15.181:0/703528328 >> 172.21.15.181:0/703528328 conn(0x7f1bac071340 msgr2=0x7f1bac0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 -- 172.21.15.181:0/703528328 shutdown_connections 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.944+0000 7f1bb399d640 1 -- 172.21.15.181:0/703528328 wait complete. 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.945+0000 7f1bb399d640 10 monclient: build_initial_monmap 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.945+0000 7f1bb399d640 10 monclient: monmap: 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.945+0000 7f1bb399d640 1 Processor -- start 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.945+0000 7f1bb399d640 1 -- start start 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.945+0000 7f1bb399d640 10 monclient: init 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:29.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac06a710 0x7f1bac141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bac06a710 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bac06e000 0x7f1bac133f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bac06e000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 0x7f1bac11dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bac09e170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac06ea00 con 0x7f1bac06a710 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac11fa50 con 0x7f1bac09e170 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac141850 con 0x7f1bac06e000 2024-09-15T03:32:29.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb399d640 10 monclient(hunting): authenticate will time out at 954.427162s 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb0f38640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bac06e000 msgr2=0x7f1bac133f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb0f38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bac06e000 0x7f1bac133f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 0x7f1bac11dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 0x7f1bac11dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43644/0 (socket says 172.21.15.181:43644) 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 -- 172.21.15.181:0/1374635374 learned_addr learned my addr 172.21.15.181:0/1374635374 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): get_auth_request con 0x7f1bac09e170 auth_method 0 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1739640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac06a710 0x7f1bac141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1739640 10 monclient(hunting): get_auth_request con 0x7f1bac06a710 auth_method 0 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient(hunting): handle_auth_done global_id 44821 payload 824 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 -- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bac06e000 msgr2=0x7f1bac133f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bac06e000 0x7f1bac133f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 -- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac06a710 msgr2=0x7f1bac141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac06a710 0x7f1bac141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 monclient: found mon.noname-b 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.946+0000 7f1bb1f3a640 1 -- 172.21.15.181:0/1374635374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba0009450 con 0x7f1bac09e170 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb1f3a640 1 -- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 msgr2=0x7f1bac11dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb1f3a640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 0x7f1bac11dae0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400c890 tx=0x7f1ba4002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400ce80 con 0x7f1bac09e170 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: dump: 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.960 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba400aa00 con 0x7f1bac09e170 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 5 monclient: authenticate success, global_id 44821 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 10 monclient: _renew_subs 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac120110 con 0x7f1bac09e170 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400ae00 con 0x7f1bac09e170 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9d7fa640 10 monclient: dump: 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1b9f7fe640 4 set_mon_vals no callback set 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 10 monclient: _renew_subs 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.948+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac121350 con 0x7f1bac09e170 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.949+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1ba4012630 con 0x7f1bac09e170 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1bb399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1bb399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1bb399d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac0008d0 con 0x7f1bac09e170 2024-09-15T03:32:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1b9d7fa640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b900386f0 0x7f1b9003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f1ba404c380 con 0x7f1bac09e170 2024-09-15T03:32:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1bb1739640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b900386f0 0x7f1b9003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.950+0000 7f1bb1739640 10 monclient: get_auth_request con 0x7f1b900386f0 auth_method 0 2024-09-15T03:32:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.952+0000 7f1bb1739640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b900386f0 0x7f1b9003aba0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1bac09d680 tx=0x7f1ba0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:29.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.954+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1ba4022d70 con 0x7f1bac09e170 2024-09-15T03:32:29.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.954+0000 7f1b9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:29.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:29.954+0000 7f1b9d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:30.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.111+0000 7f1bb399d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:30.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.111+0000 7f1bb399d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:30.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.111+0000 7f1bb399d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.111+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f1bac0a0f70 con 0x7f1bac09e170 2024-09-15T03:32:30.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.765+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1ba4016020 con 0x7f1bac09e170 2024-09-15T03:32:30.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.765+0000 7f1b9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:30.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.765+0000 7f1b9d7fa640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-15T03:32:30.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.862+0000 7f1bb399d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:30.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.862+0000 7f1bb399d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:30.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.862+0000 7f1bb399d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.862+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f1bac142150 con 0x7f1bac09e170 2024-09-15T03:32:30.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.863+0000 7f1b9d7fa640 1 -- 172.21.15.181:0/1374635374 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1ba4022770 con 0x7f1bac09e170 2024-09-15T03:32:30.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.863+0000 7f1b9d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-15T03:32:30.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.863+0000 7f1b9d7fa640 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-15T03:32:30.864 INFO:tasks.workunit.client.0.smithi181.stderr:device 'bar' does not appear in the crush map 2024-09-15T03:32:30.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b900386f0 msgr2=0x7f1b9003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:30.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b900386f0 0x7f1b9003aba0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1bac09d680 tx=0x7f1ba0032000 comp rx=0 tx=0).stop 2024-09-15T03:32:30.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 10 monclient: shutdown 2024-09-15T03:32:30.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:30.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 msgr2=0x7f1bac11dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 0x7f1bac11dae0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400c890 tx=0x7f1ba4002640 comp rx=0 tx=0).stop 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 shutdown_connections 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b900386f0 0x7f1b9003aba0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac09e170 0x7f1bac11dae0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bac06e000 0x7f1bac133f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 --2- 172.21.15.181:0/1374635374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac06a710 0x7f1bac141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 >> 172.21.15.181:0/1374635374 conn(0x7f1bac071340 msgr2=0x7f1bac134e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 shutdown_connections 2024-09-15T03:32:30.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.865+0000 7f1bb399d640 1 -- 172.21.15.181:0/1374635374 wait complete. 2024-09-15T03:32:30.874 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm foo 2024-09-15T03:32:30.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.947+0000 7fecefb09640 10 monclient: get_monmap_and_config 2024-09-15T03:32:30.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.947+0000 7fecefb09640 10 monclient: build_initial_monmap 2024-09-15T03:32:30.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.947+0000 7fecefb09640 10 monclient: monmap: 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:30.948434+0000 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:30.948434+0000 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:30.949 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:30.950 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:30.950 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:30.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 Processor -- start 2024-09-15T03:32:30.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 -- start start 2024-09-15T03:32:30.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient: init 2024-09-15T03:32:30.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:30.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): picked mon.noname-b con 0x7fece8131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fece80a0b00 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fece8132510 0x7fece8069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): picked mon.noname-c con 0x7fece8132510 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80a1b70 con 0x7fece80a0b00 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80a08c0 con 0x7fece8132510 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:30.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece8132910 con 0x7fece8131c00 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.948+0000 7fecefb09640 10 monclient(hunting): authenticate will time out at 955.429923s 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7feced8a5640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 msgr2=0x7fece8131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7feced8a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fece8132510 0x7fece8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fece8132510 0x7fece8069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43650/0 (socket says 172.21.15.181:43650) 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7feced0a4640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fece80a0b00 0x7fece8068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:36940/0 (socket says 172.21.15.181:36940) 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 -- 172.21.15.181:0/909164454 learned_addr learned my addr 172.21.15.181:0/909164454 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): get_auth_request con 0x7fece8132510 auth_method 0 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient(hunting): handle_auth_done global_id 44824 payload 274 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 -- 172.21.15.181:0/909164454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 msgr2=0x7fece8131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 --2- 172.21.15.181:0/909164454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 -- 172.21.15.181:0/909164454 >> v1:172.21.15.181:6789/0 conn(0x7fece80a0b00 legacy=0x7fece8068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient: _finish_hunting 0 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 monclient: found mon.noname-c 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.949+0000 7fecee0a6640 1 -- 172.21.15.181:0/909164454 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80a0200 con 0x7fece8132510 2024-09-15T03:32:30.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecee0a6640 1 -- 172.21.15.181:0/909164454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 msgr2=0x7fece8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:30.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecee0a6640 1 --2- 172.21.15.181:0/909164454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece8069410 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fece0009f20 tx=0x7fece0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dfc521f88f6e022 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:30.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 1 -- 172.21.15.181:0/909164454 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0007610 con 0x7fece8132510 2024-09-15T03:32:30.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: dump: 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:30.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: _finish_auth 0 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 1 -- 172.21.15.181:0/909164454 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece002d050 con 0x7fece8132510 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecefb09640 5 monclient: authenticate success, global_id 44824 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecefb09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecefb09640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 1 -- 172.21.15.181:0/909164454 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece00079a0 con 0x7fece8132510 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecefb09640 4 set_mon_vals no callback set 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.952+0000 7fecec8a3640 10 monclient: dump: 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:30.963 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 10 monclient: shutdown 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 -- 172.21.15.181:0/909164454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 msgr2=0x7fece8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 --2- 172.21.15.181:0/909164454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece8069410 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fece0009f20 tx=0x7fece0009840 comp rx=0 tx=0).stop 2024-09-15T03:32:30.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 -- 172.21.15.181:0/909164454 shutdown_connections 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 --2- 172.21.15.181:0/909164454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece8069410 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 --2- 172.21.15.181:0/909164454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 -- 172.21.15.181:0/909164454 >> 172.21.15.181:0/909164454 conn(0x7fece8071370 msgr2=0x7fece80a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 -- 172.21.15.181:0/909164454 shutdown_connections 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 -- 172.21.15.181:0/909164454 wait complete. 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 10 monclient: build_initial_monmap 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 10 monclient: monmap: 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:30.965 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.953+0000 7fecefb09640 1 Processor -- start 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 -- start start 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient: init 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece80a0b00 0x7fece809a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): picked mon.noname-a con 0x7fece80a0b00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): picked mon.noname-c con 0x7fece8131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): picked mon.noname-b con 0x7fece8132510 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece8134b00 con 0x7fece80a0b00 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece809b390 con 0x7fece8132510 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece811db40 con 0x7fece8131c00 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecefb09640 10 monclient(hunting): authenticate will time out at 955.435466s 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43660/0 (socket says 172.21.15.181:43660) 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 -- 172.21.15.181:0/4248057590 learned_addr learned my addr 172.21.15.181:0/4248057590 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): get_auth_request con 0x7fece8132510 auth_method 0 2024-09-15T03:32:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7feced0a4640 1 -- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 msgr2=0x7fece8130a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7feced0a4640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8130a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7feced8a5640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece80a0b00 0x7fece809a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient(hunting): handle_auth_done global_id 44827 payload 824 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 -- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 msgr2=0x7fece8130a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8130a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 -- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece80a0b00 msgr2=0x7fece809a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece80a0b00 0x7fece809a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient: _finish_hunting 0 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 monclient: found mon.noname-b 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7fecee0a6640 1 -- 172.21.15.181:0/4248057590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0009ae0 con 0x7fece8132510 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7feced8a5640 10 monclient: get_auth_request con 0x7fece80a0b00 auth_method 0 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.954+0000 7feced8a5640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece80a0b00 0x7fece809a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecee0a6640 1 -- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 msgr2=0x7fece811b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecee0a6640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fece0009ab0 tx=0x7fece0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0032da0 con 0x7fece8132510 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: dump: 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 5 monclient: authenticate success, global_id 44827 2024-09-15T03:32:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece003b410 con 0x7fece8132510 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 10 monclient: _renew_subs 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece811de20 con 0x7fece8132510 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece003b7b0 con 0x7fece8132510 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecddffb640 10 monclient: dump: 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:30.971 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecdffff640 4 set_mon_vals no callback set 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 10 monclient: _renew_subs 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.957+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece8141d80 con 0x7fece8132510 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fece0003020 con 0x7fece8132510 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecefb09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecefb09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecefb09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece811e060 con 0x7fece8132510 2024-09-15T03:32:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecddffb640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feccc0386f0 0x7feccc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5840+0+0 (secure 0 0 0) 0x7fece006cad0 con 0x7fece8132510 2024-09-15T03:32:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7feced8a5640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feccc0386f0 0x7feccc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.958+0000 7feced8a5640 10 monclient: get_auth_request con 0x7feccc0386f0 auth_method 0 2024-09-15T03:32:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.960+0000 7feced8a5640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feccc0386f0 0x7feccc03aba0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fecd80038f0 tx=0x7fecd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:30.962+0000 7fecee0a6640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fece0009ab0 tx=0x7fece0036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fece811e060 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:32:31.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.790+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fece811e060 con 0x7fece8132510 2024-09-15T03:32:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.790+0000 7fecddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.790+0000 7fecddffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:31.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.949+0000 7fecefb09640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:31.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.949+0000 7fecefb09640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:31.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.949+0000 7fecefb09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:31.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:31.949+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fece80008d0 con 0x7fece8132510 2024-09-15T03:32:32.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.771+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v88) v1 ==== 120+0+0 (secure 0 0 0) 0x7fece003f070 con 0x7fece8132510 2024-09-15T03:32:32.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.771+0000 7fecddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:32.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.771+0000 7fecddffb640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-15T03:32:32.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.867+0000 7fecefb09640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:32.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.867+0000 7fecefb09640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:32.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.867+0000 7fecefb09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:32.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.867+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fece80a1890 con 0x7fece8132510 2024-09-15T03:32:32.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.868+0000 7fecddffb640 1 -- 172.21.15.181:0/4248057590 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v88) v1 ==== 120+0+0 (secure 0 0 0) 0x7fece003bba0 con 0x7fece8132510 2024-09-15T03:32:32.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.868+0000 7fecddffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:32.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.868+0000 7fecddffb640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-15T03:32:32.870 INFO:tasks.workunit.client.0.smithi181.stderr:device 'foo' does not appear in the crush map 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feccc0386f0 msgr2=0x7feccc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feccc0386f0 0x7feccc03aba0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fecd80038f0 tx=0x7fecd8008040 comp rx=0 tx=0).stop 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 10 monclient: shutdown 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 msgr2=0x7fece811b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fece0009ab0 tx=0x7fece0036000 comp rx=0 tx=0).stop 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 shutdown_connections 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feccc0386f0 0x7feccc03aba0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fece8132510 0x7fece811b900 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece8131c00 0x7fece8130a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 --2- 172.21.15.181:0/4248057590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece80a0b00 0x7fece809a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.870+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 >> 172.21.15.181:0/4248057590 conn(0x7fece8071370 msgr2=0x7fece80a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:32.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.871+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 shutdown_connections 2024-09-15T03:32:32.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.871+0000 7fecefb09640 1 -- 172.21.15.181:0/4248057590 wait complete. 2024-09-15T03:32:32.880 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.9 host2 2024-09-15T03:32:32.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.953+0000 7f889c689640 10 monclient: get_monmap_and_config 2024-09-15T03:32:32.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.953+0000 7f889c689640 10 monclient: build_initial_monmap 2024-09-15T03:32:32.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.953+0000 7f889c689640 10 monclient: monmap: 2024-09-15T03:32:32.954 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:32.954480+0000 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:32.954480+0000 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:32.955 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:32.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 Processor -- start 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 -- start start 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient: init 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8894131c90 0x7f8894132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): picked mon.noname-a con 0x7f8894131c90 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f889409d210 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88940a0ab0 0x7f88941325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): picked mon.noname-b con 0x7f88940a0ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940a1800 con 0x7f889409d210 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889409fea0 con 0x7f8894131c90 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88940a1520 con 0x7f88940a0ab0 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.954+0000 7f889c689640 10 monclient(hunting): authenticate will time out at 957.435909s 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889ac26640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88940a0ab0 msgr2=0x7f88941325a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889ac26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88940a0ab0 0x7f88941325a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8894131c90 0x7f8894132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8894131c90 0x7f8894132060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:52212/0 (socket says 172.21.15.181:52212) 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f8899c24640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f889409d210 0x7f8894068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:43504/0 (socket says 172.21.15.181:43504) 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 -- 172.21.15.181:0/1812356814 learned_addr learned my addr 172.21.15.181:0/1812356814 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): get_auth_request con 0x7f8894131c90 auth_method 0 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:32.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient(hunting): handle_auth_done global_id 64278 payload 274 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 -- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88940a0ab0 msgr2=0x7f88941325a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 --2- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88940a0ab0 0x7f88941325a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 -- 172.21.15.181:0/1812356814 >> v1:172.21.15.153:6789/0 conn(0x7f889409d210 legacy=0x7f8894068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient: _finish_hunting 0 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 monclient: found mon.noname-a 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.955+0000 7f889a425640 1 -- 172.21.15.181:0/1812356814 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894069510 con 0x7f8894131c90 2024-09-15T03:32:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.958+0000 7f889a425640 1 -- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894131c90 msgr2=0x7f8894132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.958+0000 7f889a425640 1 --2- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894131c90 0x7f8894132060 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8894076b50 tx=0x7f88900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90829c611bc936d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 1 -- 172.21.15.181:0/1812356814 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890032440 con 0x7f8894131c90 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: dump: 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:32.970 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: _finish_auth 0 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 1 -- 172.21.15.181:0/1812356814 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900325a0 con 0x7f8894131c90 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 5 monclient: authenticate success, global_id 64278 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 1 -- 172.21.15.181:0/1812356814 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890032940 con 0x7f8894131c90 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 4 set_mon_vals no callback set 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:32.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f8899423640 10 monclient: dump: 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 10 monclient: shutdown 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 -- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894131c90 msgr2=0x7f8894132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 --2- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894131c90 0x7f8894132060 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8894076b50 tx=0x7f88900091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 -- 172.21.15.181:0/1812356814 shutdown_connections 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 --2- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88940a0ab0 0x7f88941325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 --2- 172.21.15.181:0/1812356814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894131c90 0x7f8894132060 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 -- 172.21.15.181:0/1812356814 >> 172.21.15.181:0/1812356814 conn(0x7f8894071370 msgr2=0x7f88940a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 -- 172.21.15.181:0/1812356814 shutdown_connections 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.959+0000 7f889c689640 1 -- 172.21.15.181:0/1812356814 wait complete. 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.960+0000 7f889c689640 10 monclient: build_initial_monmap 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.960+0000 7f889c689640 10 monclient: monmap: 2024-09-15T03:32:32.973 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.960+0000 7f889c689640 1 Processor -- start 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.960+0000 7f889c689640 1 -- start start 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.960+0000 7f889c689640 10 monclient: init 2024-09-15T03:32:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940a0ab0 0x7f8894141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): picked mon.noname-a con 0x7f88940a0ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 0x7f88941418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): picked mon.noname-b con 0x7f8894131c90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f889411e4e0 0x7f889411e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): picked mon.noname-c con 0x7f889411e4e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88940a0fb0 con 0x7f88940a0ab0 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889409f270 con 0x7f8894131c90 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8894120c90 con 0x7f889411e4e0 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889c689640 10 monclient(hunting): authenticate will time out at 957.442138s 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889ac26640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f889411e4e0 msgr2=0x7f889411e8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 0x7f88941418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889ac26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f889411e4e0 0x7f889411e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 0x7f88941418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43674/0 (socket says 172.21.15.181:43674) 2024-09-15T03:32:32.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 -- 172.21.15.181:0/2331301069 learned_addr learned my addr 172.21.15.181:0/2331301069 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): get_auth_request con 0x7f8894131c90 auth_method 0 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940a0ab0 0x7f8894141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): get_auth_request con 0x7f88940a0ab0 auth_method 0 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f889a425640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient(hunting): handle_auth_done global_id 44830 payload 824 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 -- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f889411e4e0 msgr2=0x7f889411e8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f889411e4e0 0x7f889411e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 -- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940a0ab0 msgr2=0x7f8894141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940a0ab0 0x7f8894141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient: _finish_hunting 0 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 monclient: found mon.noname-b 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.961+0000 7f8899c24640 1 -- 172.21.15.181:0/2331301069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009450 con 0x7f8894131c90 2024-09-15T03:32:32.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f8899c24640 1 -- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 msgr2=0x7f88941418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f8899c24640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 0x7f88941418c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f888400c5b0 tx=0x7f8884007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884007510 con 0x7f8894131c90 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 10 monclient: dump: 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:32.979 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f889c689640 5 monclient: authenticate success, global_id 44830 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f889c689640 10 monclient: _renew_subs 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f889c689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88840057d0 con 0x7f8894131c90 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.963+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8894122140 con 0x7f8894131c90 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f888a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884007690 con 0x7f8894131c90 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f888a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f888a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:32.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f888a7fc640 10 monclient: dump: 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f8898c22640 4 set_mon_vals no callback set 2024-09-15T03:32:32.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 10 monclient: _renew_subs 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88941209e0 con 0x7f8894131c90 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f888401b400 con 0x7f8894131c90 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.964+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8894122dd0 con 0x7f8894131c90 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.965+0000 7f888a7fc640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88780386f0 0x7f887803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.965+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5711+0+0 (secure 0 0 0) 0x7f888404ba10 con 0x7f8894131c90 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.965+0000 7f889a425640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88780386f0 0x7f887803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:32.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.965+0000 7f889a425640 10 monclient: get_auth_request con 0x7f88780386f0 auth_method 0 2024-09-15T03:32:32.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.967+0000 7f889a425640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88780386f0 0x7f887803aba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f88940769f0 tx=0x7f8890032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:32.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.969+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f888401ab40 con 0x7f8894131c90 2024-09-15T03:32:32.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.969+0000 7f888a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:32.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:32.969+0000 7f888a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:33.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.127+0000 7f889c689640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T03:32:33.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.127+0000 7f889c689640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T03:32:33.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.127+0000 7f889c689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.127+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f88940008d0 con 0x7f8894131c90 2024-09-15T03:32:33.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.772+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v89) v1 ==== 143+0+0 (secure 0 0 0) 0x7f888401a360 con 0x7f8894131c90 2024-09-15T03:32:33.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.772+0000 7f888a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T03:32:33.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.772+0000 7f888a7fc640 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-15T03:32:33.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.873+0000 7f889c689640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T03:32:33.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.873+0000 7f889c689640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T03:32:33.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.873+0000 7f889c689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.873+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f8894121cd0 con 0x7f8894131c90 2024-09-15T03:32:33.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.874+0000 7f888a7fc640 1 -- 172.21.15.181:0/2331301069 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v89) v1 ==== 145+0+0 (secure 0 0 0) 0x7f888401a540 con 0x7f8894131c90 2024-09-15T03:32:33.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.874+0000 7f888a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-15T03:32:33.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.874+0000 7f888a7fc640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-15T03:32:33.875 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.9' does not appear in the crush map 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88780386f0 msgr2=0x7f887803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88780386f0 0x7f887803aba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f88940769f0 tx=0x7f8890032000 comp rx=0 tx=0).stop 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 10 monclient: shutdown 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 msgr2=0x7f88941418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 0x7f88941418c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f888400c5b0 tx=0x7f8884007f00 comp rx=0 tx=0).stop 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 shutdown_connections 2024-09-15T03:32:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88780386f0 0x7f887803aba0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f889411e4e0 0x7f889411e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8894131c90 0x7f88941418c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 --2- 172.21.15.181:0/2331301069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940a0ab0 0x7f8894141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 >> 172.21.15.181:0/2331301069 conn(0x7f8894071370 msgr2=0x7f88940a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 shutdown_connections 2024-09-15T03:32:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.876+0000 7f889c689640 1 -- 172.21.15.181:0/2331301069 wait complete. 2024-09-15T03:32:33.886 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm host2 2024-09-15T03:32:33.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.958+0000 7f7183e34640 10 monclient: get_monmap_and_config 2024-09-15T03:32:33.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.958+0000 7f7183e34640 10 monclient: build_initial_monmap 2024-09-15T03:32:33.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.958+0000 7f7183e34640 10 monclient: monmap: 2024-09-15T03:32:33.959 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:33.959 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:33.958976+0000 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:33.958976+0000 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.958+0000 7f7183e34640 1 Processor -- start 2024-09-15T03:32:33.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.958+0000 7f7183e34640 1 -- start start 2024-09-15T03:32:33.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.958+0000 7f7183e34640 10 monclient: init 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f717c06e130 0x7f717c09da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): picked mon.noname-c con 0x7f717c06e130 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f717c06a790 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f717c06e500 0x7f717c09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): picked mon.noname-a con 0x7f717c06e500 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c06a0b0 con 0x7f717c06a790 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c068c30 con 0x7f717c06e500 2024-09-15T03:32:33.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c069750 con 0x7f717c06e130 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7183e34640 10 monclient(hunting): authenticate will time out at 958.440442s 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f717c06e130 0x7f717c09da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f717c06e130 0x7f717c09da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43682/0 (socket says 172.21.15.181:43682) 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 -- 172.21.15.181:0/3939629766 learned_addr learned my addr 172.21.15.181:0/3939629766 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): get_auth_request con 0x7f717c06e130 auth_method 0 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f71813cf640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f717c06a790 0x7f717c09dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:36942/0 (socket says 172.21.15.181:36942) 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:33.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f71823d1640 1 --2- 172.21.15.181:0/3939629766 >> v2:172.21.15.153:3300/0 conn(0x7f717c06e500 0x7f717c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient(hunting): handle_auth_done global_id 44833 payload 274 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 -- 172.21.15.181:0/3939629766 >> v2:172.21.15.153:3300/0 conn(0x7f717c06e500 msgr2=0x7f717c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 --2- 172.21.15.181:0/3939629766 >> v2:172.21.15.153:3300/0 conn(0x7f717c06e500 0x7f717c09e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 -- 172.21.15.181:0/3939629766 >> v1:172.21.15.181:6789/0 conn(0x7f717c06a790 legacy=0x7f717c09dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient: _finish_hunting 0 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 monclient: found mon.noname-c 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f7181bd0640 1 -- 172.21.15.181:0/3939629766 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c069a60 con 0x7f717c06e130 2024-09-15T03:32:33.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f71823d1640 10 monclient: get_auth_request con 0x7f717c06e500 auth_method 0 2024-09-15T03:32:33.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.959+0000 7f71823d1640 1 --2- 172.21.15.181:0/3939629766 >> v2:172.21.15.153:3300/0 conn(0x7f717c06e500 0x7f717c09e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:33.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7181bd0640 1 -- 172.21.15.181:0/3939629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06e130 msgr2=0x7f717c09da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7181bd0640 1 --2- 172.21.15.181:0/3939629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06e130 0x7f717c09da60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f717c09fac0 tx=0x7f71780091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec4e2a66ce4e79a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 1 -- 172.21.15.181:0/3939629766 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802ae30 con 0x7f717c06e130 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: dump: 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:33.973 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: _finish_auth 0 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 1 -- 172.21.15.181:0/3939629766 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717802f4b0 con 0x7f717c06e130 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7183e34640 5 monclient: authenticate success, global_id 44833 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7183e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7183e34640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 1 -- 172.21.15.181:0/3939629766 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802f850 con 0x7f717c06e130 2024-09-15T03:32:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7183e34640 4 set_mon_vals no callback set 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.962+0000 7f7180bce640 10 monclient: dump: 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 10 monclient: shutdown 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 -- 172.21.15.181:0/3939629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06e130 msgr2=0x7f717c09da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 --2- 172.21.15.181:0/3939629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06e130 0x7f717c09da60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f717c09fac0 tx=0x7f71780091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 -- 172.21.15.181:0/3939629766 shutdown_connections 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 --2- 172.21.15.181:0/3939629766 >> v2:172.21.15.153:3300/0 conn(0x7f717c06e500 0x7f717c09e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 --2- 172.21.15.181:0/3939629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06e130 0x7f717c09da60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 -- 172.21.15.181:0/3939629766 >> 172.21.15.181:0/3939629766 conn(0x7f717c071340 msgr2=0x7f717c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 -- 172.21.15.181:0/3939629766 shutdown_connections 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 1 -- 172.21.15.181:0/3939629766 wait complete. 2024-09-15T03:32:33.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 10 monclient: build_initial_monmap 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.963+0000 7f7183e34640 10 monclient: monmap: 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 Processor -- start 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 -- start start 2024-09-15T03:32:33.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient: init 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 0x7f717c145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): picked mon.noname-b con 0x7f717c06a790 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f717c06e130 0x7f717c142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): picked mon.noname-a con 0x7f717c06e130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f717c06e500 0x7f717c1427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): picked mon.noname-c con 0x7f717c06e500 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c11bc50 con 0x7f717c06e130 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c06e970 con 0x7f717c06a790 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c1265d0 con 0x7f717c06e500 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7183e34640 10 monclient(hunting): authenticate will time out at 958.445856s 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 0x7f717c145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 0x7f717c145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43696/0 (socket says 172.21.15.181:43696) 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 1 -- 172.21.15.181:0/2435120683 learned_addr learned my addr 172.21.15.181:0/2435120683 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:33.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 10 monclient(hunting): get_auth_request con 0x7f717c06a790 auth_method 0 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f71823d1640 1 -- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f717c06e500 msgr2=0x7f717c1427a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f7181bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.964+0000 7f71823d1640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f717c06e500 0x7f717c1427a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f71813cf640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f717c06e130 0x7f717c142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f71813cf640 10 monclient(hunting): get_auth_request con 0x7f717c06e130 auth_method 0 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f71813cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f71813cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:33.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f71813cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 10 monclient(hunting): handle_auth_done global_id 44836 payload 824 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 1 -- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f717c06e500 msgr2=0x7f717c1427a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f717c06e500 0x7f717c1427a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 1 -- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f717c06e130 msgr2=0x7f717c142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f717c06e130 0x7f717c142260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 10 monclient: _finish_hunting 0 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 1 monclient: found mon.noname-b 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f7181bd0640 1 -- 172.21.15.181:0/2435120683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178009450 con 0x7f717c06a790 2024-09-15T03:32:33.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.965+0000 7f71813cf640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f717c06e130 0x7f717c142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:33.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7181bd0640 1 -- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 msgr2=0x7f717c145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:33.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7181bd0640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 0x7f717c145350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7178009940 tx=0x7f7178032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:33.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71780328e0 con 0x7f717c06a790 2024-09-15T03:32:33.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: dump: 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:33.985 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7178032a80 con 0x7f717c06a790 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 5 monclient: authenticate success, global_id 44836 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 10 monclient: _renew_subs 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717c11c300 con 0x7f717c06a790 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178032e40 con 0x7f717c06a790 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:33.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7171ffb640 10 monclient: dump: 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 10 monclient: _renew_subs 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7173fff640 4 set_mon_vals no callback set 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.967+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717c1262f0 con 0x7f717c06a790 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7178039040 con 0x7f717c06a790 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7183e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7183e34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7183e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717c0008d0 con 0x7f717c06a790 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7171ffb640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71600386f0 0x7f716003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.968+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5690+0+0 (secure 0 0 0) 0x7f717806c140 con 0x7f717c06a790 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.969+0000 7f71813cf640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71600386f0 0x7f716003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:33.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.969+0000 7f71813cf640 10 monclient: get_auth_request con 0x7f71600386f0 auth_method 0 2024-09-15T03:32:33.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.971+0000 7f71813cf640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71600386f0 0x7f716003aba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f717c06b8f0 tx=0x7f716c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:33.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.973+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f717803a960 con 0x7f717c06a790 2024-09-15T03:32:33.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.973+0000 7f7171ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:33.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:33.973+0000 7f7171ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:34.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.133+0000 7f7183e34640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T03:32:34.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.133+0000 7f7183e34640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T03:32:34.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.133+0000 7f7183e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.133+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f717c126940 con 0x7f717c06a790 2024-09-15T03:32:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.780+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v90) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7178037090 con 0x7f717c06a790 2024-09-15T03:32:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.781+0000 7f7171ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T03:32:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.781+0000 7f7171ffb640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-15T03:32:34.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.879+0000 7f7183e34640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T03:32:34.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.879+0000 7f7183e34640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T03:32:34.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.879+0000 7f7183e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.879+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f717c126940 con 0x7f717c06a790 2024-09-15T03:32:34.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.880+0000 7f7171ffb640 1 -- 172.21.15.181:0/2435120683 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v90) v1 ==== 124+0+0 (secure 0 0 0) 0x7f717803f020 con 0x7f717c06a790 2024-09-15T03:32:34.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.880+0000 7f7171ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-15T03:32:34.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.880+0000 7f7171ffb640 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-15T03:32:34.881 INFO:tasks.workunit.client.0.smithi181.stderr:device 'host2' does not appear in the crush map 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71600386f0 msgr2=0x7f716003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71600386f0 0x7f716003aba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f717c06b8f0 tx=0x7f716c009040 comp rx=0 tx=0).stop 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 10 monclient: shutdown 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 msgr2=0x7f717c145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 0x7f717c145350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7178009940 tx=0x7f7178032000 comp rx=0 tx=0).stop 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 shutdown_connections 2024-09-15T03:32:34.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f71600386f0 0x7f716003aba0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f717c06e500 0x7f717c1427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f717c06e130 0x7f717c142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 --2- 172.21.15.181:0/2435120683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f717c06a790 0x7f717c145350 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 >> 172.21.15.181:0/2435120683 conn(0x7f717c071340 msgr2=0x7f717c09c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:34.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 shutdown_connections 2024-09-15T03:32:34.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.883+0000 7f7183e34640 1 -- 172.21.15.181:0/2435120683 wait complete. 2024-09-15T03:32:34.892 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket foo host 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.965+0000 7fdd1b2dd640 10 monclient: get_monmap_and_config 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.965+0000 7fdd1b2dd640 10 monclient: build_initial_monmap 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.965+0000 7fdd1b2dd640 10 monclient: monmap: 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:34.965824+0000 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:34.965824+0000 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:34.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:34.967 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:34.967 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:34.967 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:34.967 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:34.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.965+0000 7fdd1b2dd640 1 Processor -- start 2024-09-15T03:32:34.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.965+0000 7fdd1b2dd640 1 -- start start 2024-09-15T03:32:34.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.965+0000 7fdd1b2dd640 10 monclient: init 2024-09-15T03:32:34.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdd1409ec50 0x7fdd14131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdd141321a0 0x7fdd14068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdd141321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd1409f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd14067420 con 0x7fdd1409f020 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1409fdc0 con 0x7fdd1409ec50 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd140a0510 con 0x7fdd141321a0 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd1b2dd640 10 monclient(hunting): authenticate will time out at 959.447352s 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdd141321a0 0x7fdd14068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdd141321a0 0x7fdd14068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43710/0 (socket says 172.21.15.181:43710) 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 -- 172.21.15.181:0/1583469499 learned_addr learned my addr 172.21.15.181:0/1583469499 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): get_auth_request con 0x7fdd141321a0 auth_method 0 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd19079640 1 --2- 172.21.15.181:0/1583469499 >> v2:172.21.15.153:3300/0 conn(0x7fdd1409ec50 0x7fdd14131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd19079640 10 monclient(hunting): get_auth_request con 0x7fdd1409ec50 auth_method 0 2024-09-15T03:32:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd19079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd19079640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd19079640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient(hunting): handle_auth_done global_id 44839 payload 274 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 -- 172.21.15.181:0/1583469499 >> v2:172.21.15.153:3300/0 conn(0x7fdd1409ec50 msgr2=0x7fdd14131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 --2- 172.21.15.181:0/1583469499 >> v2:172.21.15.153:3300/0 conn(0x7fdd1409ec50 0x7fdd14131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 -- 172.21.15.181:0/1583469499 >> v1:172.21.15.153:6789/0 conn(0x7fdd1409f020 legacy=0x7fdd14069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient: _finish_hunting 0 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 monclient: found mon.noname-c 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd18878640 1 -- 172.21.15.181:0/1583469499 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd140a0200 con 0x7fdd141321a0 2024-09-15T03:32:34.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.966+0000 7fdd19079640 1 --2- 172.21.15.181:0/1583469499 >> v2:172.21.15.153:3300/0 conn(0x7fdd1409ec50 0x7fdd14131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:34.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd18878640 1 -- 172.21.15.181:0/1583469499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 msgr2=0x7fdd14068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:34.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd18878640 1 --2- 172.21.15.181:0/1583469499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd14068d00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdd140a2b00 tx=0x7fdd040091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f650265c36d4557 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:34.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 1 -- 172.21.15.181:0/1583469499 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04032440 con 0x7fdd141321a0 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: dump: 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:34.980 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: _finish_auth 0 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd1b2dd640 5 monclient: authenticate success, global_id 44839 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd1b2dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 1 -- 172.21.15.181:0/1583469499 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd040325a0 con 0x7fdd141321a0 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 1 -- 172.21.15.181:0/1583469499 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04032940 con 0x7fdd141321a0 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd1b2dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd1b2dd640 4 set_mon_vals no callback set 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.969+0000 7fdd0bfff640 10 monclient: dump: 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 10 monclient: shutdown 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1583469499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 msgr2=0x7fdd14068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1583469499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd14068d00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdd140a2b00 tx=0x7fdd040091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1583469499 shutdown_connections 2024-09-15T03:32:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1583469499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd14068d00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1583469499 >> v2:172.21.15.153:3300/0 conn(0x7fdd1409ec50 0x7fdd14131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1583469499 >> 172.21.15.181:0/1583469499 conn(0x7fdd140a5d40 msgr2=0x7fdd140a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1583469499 shutdown_connections 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1583469499 wait complete. 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 10 monclient: build_initial_monmap 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.970+0000 7fdd1b2dd640 10 monclient: monmap: 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:34.983 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 Processor -- start 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 -- start start 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient: init 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd1409ec50 0x7fdd1409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd1409f020 0x7fdd141309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd1411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:34.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdd141321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1409c000 con 0x7fdd1409ec50 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1409dd10 con 0x7fdd141321a0 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14141d10 con 0x7fdd1409f020 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1b2dd640 10 monclient(hunting): authenticate will time out at 959.452790s 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd1411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd18878640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd1409f020 msgr2=0x7fdd141309c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd1411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43718/0 (socket says 172.21.15.181:43718) 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd18878640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd1409f020 0x7fdd141309c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:34.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 1 -- 172.21.15.181:0/1430748746 learned_addr learned my addr 172.21.15.181:0/1430748746 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): get_auth_request con 0x7fdd141321a0 auth_method 0 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.971+0000 7fdd1987a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd19079640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd1409ec50 0x7fdd1409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd19079640 10 monclient(hunting): get_auth_request con 0x7fdd1409ec50 auth_method 0 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd19079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd19079640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd19079640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 10 monclient(hunting): handle_auth_done global_id 44842 payload 824 2024-09-15T03:32:34.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 1 -- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd1409f020 msgr2=0x7fdd141309c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd1409f020 0x7fdd141309c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 1 -- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd1409ec50 msgr2=0x7fdd1409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd1409ec50 0x7fdd1409aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 1 monclient: found mon.noname-b 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd1987a640 1 -- 172.21.15.181:0/1430748746 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04009450 con 0x7fdd141321a0 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.972+0000 7fdd19079640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd1409ec50 0x7fdd1409aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1987a640 1 -- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 msgr2=0x7fdd1411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1987a640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd1411b760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00c8c0 tx=0x7fdd0c002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00aa00 con 0x7fdd141321a0 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:34.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: dump: 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:34.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c00ab80 con 0x7fdd141321a0 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 5 monclient: authenticate success, global_id 44842 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c012420 con 0x7fdd141321a0 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 10 monclient: _renew_subs 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1409b5e0 con 0x7fdd141321a0 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd097fa640 10 monclient: dump: 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd0b7fe640 4 set_mon_vals no callback set 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 10 monclient: _renew_subs 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.974+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1411cbf0 con 0x7fdd141321a0 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdd0c012890 con 0x7fdd141321a0 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd1b2dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd1b2dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:34.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd1b2dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1411df80 con 0x7fdd141321a0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd097fa640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcf8038720 0x7fdcf803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5555+0+0 (secure 0 0 0) 0x7fdd0c04c190 con 0x7fdd141321a0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd19079640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcf8038720 0x7fdcf803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.975+0000 7fdd19079640 10 monclient: get_auth_request con 0x7fdcf8038720 auth_method 0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.978+0000 7fdd19079640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcf8038720 0x7fdcf803abd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd10006e70 tx=0x7fdd10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.979+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdd0c010510 con 0x7fdd141321a0 2024-09-15T03:32:34.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.979+0000 7fdd097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:34.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:34.979+0000 7fdd097fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:35.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.134+0000 7fdd1b2dd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T03:32:35.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.134+0000 7fdd1b2dd640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T03:32:35.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.134+0000 7fdd1b2dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.134+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fdd140008d0 con 0x7fdd141321a0 2024-09-15T03:32:35.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.782+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdd0c0160a0 con 0x7fdd141321a0 2024-09-15T03:32:35.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.782+0000 7fdd097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T03:32:35.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.782+0000 7fdd097fa640 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-15T03:32:35.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.878+0000 7fdd1b2dd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T03:32:35.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.878+0000 7fdd1b2dd640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T03:32:35.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.878+0000 7fdd1b2dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.878+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fdd140a13e0 con 0x7fdd141321a0 2024-09-15T03:32:35.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.880+0000 7fdd097fa640 1 -- 172.21.15.181:0/1430748746 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v91) v1 ==== 126+0+0 (secure 0 0 0) 0x7fdd0c022af0 con 0x7fdd141321a0 2024-09-15T03:32:35.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.880+0000 7fdd097fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-15T03:32:35.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.880+0000 7fdd097fa640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-15T03:32:35.881 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'foo' already exists 2024-09-15T03:32:35.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcf8038720 msgr2=0x7fdcf803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:35.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcf8038720 0x7fdcf803abd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd10006e70 tx=0x7fdd10008040 comp rx=0 tx=0).stop 2024-09-15T03:32:35.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 10 monclient: shutdown 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 msgr2=0x7fdd1411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd1411b760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00c8c0 tx=0x7fdd0c002640 comp rx=0 tx=0).stop 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 shutdown_connections 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcf8038720 0x7fdcf803abd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd141321a0 0x7fdd1411b760 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd1409f020 0x7fdd141309c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 --2- 172.21.15.181:0/1430748746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd1409ec50 0x7fdd1409aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 >> 172.21.15.181:0/1430748746 conn(0x7fdd140a5d40 msgr2=0x7fdd1409df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 shutdown_connections 2024-09-15T03:32:35.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.882+0000 7fdd1b2dd640 1 -- 172.21.15.181:0/1430748746 wait complete. 2024-09-15T03:32:35.891 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-15T03:32:35.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.964+0000 7ff25486f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:35.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.964+0000 7ff25486f640 10 monclient: build_initial_monmap 2024-09-15T03:32:35.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.964+0000 7ff25486f640 10 monclient: monmap: 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:35.965497+0000 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:35.965497+0000 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:35.966 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 Processor -- start 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 -- start start 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient: init 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff2500a19a0 0x7ff25009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): picked mon.noname-a con 0x7ff2500a19a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25009be20 0x7ff25009c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): picked mon.noname-b con 0x7ff25009be20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:35.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff250099250 0x7ff2500996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): picked mon.noname-c con 0x7ff250099250 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25009c730 con 0x7ff2500a19a0 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2500a1d70 con 0x7ff250099250 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff250098600 con 0x7ff25009be20 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.965+0000 7ff25486f640 10 monclient(hunting): authenticate will time out at 960.446980s 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24dd9b640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25009be20 msgr2=0x7ff25009c1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25009be20 0x7ff25009c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff250099250 0x7ff2500996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:35.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff250099250 0x7ff2500996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43720/0 (socket says 172.21.15.181:43720) 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 -- 172.21.15.181:0/1168243266 learned_addr learned my addr 172.21.15.181:0/1168243266 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): get_auth_request con 0x7ff250099250 auth_method 0 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24e59c640 1 --2- 172.21.15.181:0/1168243266 >> v2:172.21.15.153:3300/0 conn(0x7ff2500a19a0 0x7ff25009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24e59c640 10 monclient(hunting): get_auth_request con 0x7ff2500a19a0 auth_method 0 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:35.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient(hunting): handle_auth_done global_id 44845 payload 274 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 -- 172.21.15.181:0/1168243266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25009be20 msgr2=0x7ff25009c1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 --2- 172.21.15.181:0/1168243266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25009be20 0x7ff25009c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 -- 172.21.15.181:0/1168243266 >> v2:172.21.15.153:3300/0 conn(0x7ff2500a19a0 msgr2=0x7ff25009b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 --2- 172.21.15.181:0/1168243266 >> v2:172.21.15.153:3300/0 conn(0x7ff2500a19a0 0x7ff25009b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.966+0000 7ff24ed9d640 1 -- 172.21.15.181:0/1168243266 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250097a70 con 0x7ff250099250 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24ed9d640 1 -- 172.21.15.181:0/1168243266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 msgr2=0x7ff2500996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24ed9d640 1 --2- 172.21.15.181:0/1168243266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff2500996d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff25009cf50 tx=0x7ff24800ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70dd567ae33dac92 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:35.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 1 -- 172.21.15.181:0/1168243266 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2480336e0 con 0x7ff250099250 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: dump: 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:35.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: _finish_auth 0 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 1 -- 172.21.15.181:0/1168243266 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248033840 con 0x7ff250099250 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 1 -- 172.21.15.181:0/1168243266 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff248033be0 con 0x7ff250099250 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 5 monclient: authenticate success, global_id 44845 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 4 set_mon_vals no callback set 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff24d59a640 10 monclient: dump: 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:35.975 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:35.976 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 10 monclient: shutdown 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 -- 172.21.15.181:0/1168243266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 msgr2=0x7ff2500996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 --2- 172.21.15.181:0/1168243266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff2500996d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff25009cf50 tx=0x7ff24800ba50 comp rx=0 tx=0).stop 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 -- 172.21.15.181:0/1168243266 shutdown_connections 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 --2- 172.21.15.181:0/1168243266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff2500996d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 --2- 172.21.15.181:0/1168243266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25009be20 0x7ff25009c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 --2- 172.21.15.181:0/1168243266 >> v2:172.21.15.153:3300/0 conn(0x7ff2500a19a0 0x7ff25009b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 -- 172.21.15.181:0/1168243266 >> 172.21.15.181:0/1168243266 conn(0x7ff2500711a0 msgr2=0x7ff2500a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 -- 172.21.15.181:0/1168243266 shutdown_connections 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.969+0000 7ff25486f640 1 -- 172.21.15.181:0/1168243266 wait complete. 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.970+0000 7ff25486f640 10 monclient: build_initial_monmap 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.970+0000 7ff25486f640 10 monclient: monmap: 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.970+0000 7ff25486f640 1 Processor -- start 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.970+0000 7ff25486f640 1 -- start start 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.970+0000 7ff25486f640 10 monclient: init 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff250139490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): picked mon.noname-b con 0x7ff250099250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff25009be20 0x7ff250138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): picked mon.noname-a con 0x7ff25009be20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2500a19a0 0x7ff25013ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): picked mon.noname-c con 0x7ff2500a19a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25011e480 con 0x7ff25009be20 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25011dec0 con 0x7ff250099250 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25011b9e0 con 0x7ff2500a19a0 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff25486f640 10 monclient(hunting): authenticate will time out at 960.452192s 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff250139490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff250139490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43734/0 (socket says 172.21.15.181:43734) 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 -- 172.21.15.181:0/3977257150 learned_addr learned my addr 172.21.15.181:0/3977257150 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): get_auth_request con 0x7ff250099250 auth_method 0 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24ed9d640 1 -- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2500a19a0 msgr2=0x7ff25013ca00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24ed9d640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2500a19a0 0x7ff25013ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff25009be20 0x7ff250138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): get_auth_request con 0x7ff25009be20 auth_method 0 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient(hunting): handle_auth_done global_id 44848 payload 824 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 -- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2500a19a0 msgr2=0x7ff25013ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2500a19a0 0x7ff25013ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 -- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff25009be20 msgr2=0x7ff250138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff25009be20 0x7ff250138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 monclient: found mon.noname-b 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.971+0000 7ff24e59c640 1 -- 172.21.15.181:0/3977257150 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24800bcf0 con 0x7ff250099250 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff24e59c640 1 -- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 msgr2=0x7ff250139490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff24e59c640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff250139490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff24000d890 tx=0x7ff240008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24000de20 con 0x7ff250099250 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: dump: 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24000bae0 con 0x7ff250099250 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff25486f640 5 monclient: authenticate success, global_id 44848 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24000be80 con 0x7ff250099250 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff25486f640 10 monclient: _renew_subs 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff25486f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25011dbe0 con 0x7ff250099250 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff23e7fc640 10 monclient: dump: 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.973+0000 7ff24cd99640 4 set_mon_vals no callback set 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 10 monclient: _renew_subs 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff250139220 con 0x7ff250099250 2024-09-15T03:32:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25011e2c0 con 0x7ff250099250 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.974+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff240014640 con 0x7ff250099250 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.975+0000 7ff23e7fc640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2300386f0 0x7ff23003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.975+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5582+0+0 (secure 0 0 0) 0x7ff24004c3b0 con 0x7ff250099250 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.975+0000 7ff24dd9b640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2300386f0 0x7ff23003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.975+0000 7ff24dd9b640 10 monclient: get_auth_request con 0x7ff2300386f0 auth_method 0 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.977+0000 7ff24dd9b640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2300386f0 0x7ff23003aba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff250099100 tx=0x7ff2440090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.979+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff240011570 con 0x7ff250099250 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.979+0000 7ff23e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:35.979+0000 7ff23e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:36.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.140+0000 7ff25486f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T03:32:36.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.140+0000 7ff25486f640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T03:32:36.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.140+0000 7ff25486f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:36.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.140+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7ff2500008d0 con 0x7ff250099250 2024-09-15T03:32:36.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.794+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 204+0+0 (secure 0 0 0) 0x7ff24001dbb0 con 0x7ff250099250 2024-09-15T03:32:36.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.794+0000 7ff23e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T03:32:36.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.794+0000 7ff23e7fc640 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-15T03:32:36.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.894+0000 7ff25486f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T03:32:36.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.894+0000 7ff25486f640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T03:32:36.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.894+0000 7ff25486f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:36.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.894+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7ff25009ba30 con 0x7ff250099250 2024-09-15T03:32:36.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.895+0000 7ff23e7fc640 1 -- 172.21.15.181:0/3977257150 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 214+0+0 (secure 0 0 0) 0x7ff240011340 con 0x7ff250099250 2024-09-15T03:32:36.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.895+0000 7ff23e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-15T03:32:36.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.895+0000 7ff23e7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-15T03:32:36.896 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2300386f0 msgr2=0x7ff23003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2300386f0 0x7ff23003aba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff250099100 tx=0x7ff2440090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 10 monclient: shutdown 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 msgr2=0x7ff250139490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff250139490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff24000d890 tx=0x7ff240008040 comp rx=0 tx=0).stop 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 shutdown_connections 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff2300386f0 0x7ff23003aba0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2500a19a0 0x7ff25013ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff25009be20 0x7ff250138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 --2- 172.21.15.181:0/3977257150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff250099250 0x7ff250139490 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 >> 172.21.15.181:0/3977257150 conn(0x7ff2500711a0 msgr2=0x7ff25009d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:36.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 shutdown_connections 2024-09-15T03:32:36.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.897+0000 7ff25486f640 1 -- 172.21.15.181:0/3977257150 wait complete. 2024-09-15T03:32:36.907 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.981+0000 7f255919d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.981+0000 7f255919d640 10 monclient: build_initial_monmap 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.981+0000 7f255919d640 10 monclient: monmap: 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:36.981895+0000 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:36.981895+0000 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:36.983 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:36.983 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:36.983 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:36.983 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:36.983 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:36.983 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:36.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.981+0000 7f255919d640 1 Processor -- start 2024-09-15T03:32:36.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.981+0000 7f255919d640 1 -- start start 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.981+0000 7f255919d640 10 monclient: init 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409ec50 0x7f2554131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): picked mon.noname-b con 0x7f255409ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25541321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f255409f020 0x7f2554069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): picked mon.noname-a con 0x7f255409f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2554067420 con 0x7f25541321a0 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255409fdc0 con 0x7f255409f020 2024-09-15T03:32:36.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25540a0510 con 0x7f255409ec50 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255919d640 10 monclient(hunting): authenticate will time out at 961.463301s 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f2552d9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409ec50 msgr2=0x7f2554131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f2552d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409ec50 0x7f2554131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f255409f020 0x7f2554069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f255409f020 0x7f2554069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:52294/0 (socket says 172.21.15.181:52294) 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 1 -- 172.21.15.181:0/91524195 learned_addr learned my addr 172.21.15.181:0/91524195 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f25541321a0 0x7f2554068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:43522/0 (socket says 172.21.15.181:43522) 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): get_auth_request con 0x7f255409f020 auth_method 0 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:36.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 10 monclient(hunting): handle_auth_done global_id 64299 payload 274 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.982+0000 7f255359e640 1 -- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409ec50 msgr2=0x7f2554131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 1 --2- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409ec50 0x7f2554131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 1 -- 172.21.15.181:0/91524195 >> v1:172.21.15.153:6789/0 conn(0x7f25541321a0 legacy=0x7f2554068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 10 monclient: _finish_hunting 0 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 1 monclient: found mon.noname-a 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:36.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.983+0000 7f255359e640 1 -- 172.21.15.181:0/91524195 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25540a0200 con 0x7f255409f020 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.985+0000 7f255359e640 1 -- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f255409f020 msgr2=0x7f2554069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.985+0000 7f255359e640 1 --2- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f255409f020 0x7f2554069410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f254c009f20 tx=0x7f254c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c240a91f3d6e48 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 1 -- 172.21.15.181:0/91524195 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c007610 con 0x7f255409f020 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: dump: 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:36.994 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 5 monclient: authenticate success, global_id 64299 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 1 -- 172.21.15.181:0/91524195 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c02d050 con 0x7f255409f020 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:36.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 4 set_mon_vals no callback set 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 1 -- 172.21.15.181:0/91524195 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0079a0 con 0x7f255409f020 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f2551d9b640 10 monclient: dump: 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:36.996 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 10 monclient: shutdown 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 1 -- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f255409f020 msgr2=0x7f2554069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 1 --2- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f255409f020 0x7f2554069410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f254c009f20 tx=0x7f254c009840 comp rx=0 tx=0).stop 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 1 -- 172.21.15.181:0/91524195 shutdown_connections 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 1 --2- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f255409f020 0x7f2554069410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 1 --2- 172.21.15.181:0/91524195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409ec50 0x7f2554131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.986+0000 7f255919d640 1 -- 172.21.15.181:0/91524195 >> 172.21.15.181:0/91524195 conn(0x7f25540a5d40 msgr2=0x7f25540a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:36.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 1 -- 172.21.15.181:0/91524195 shutdown_connections 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 1 -- 172.21.15.181:0/91524195 wait complete. 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 10 monclient: build_initial_monmap 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 10 monclient: monmap: 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:36.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 1 Processor -- start 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 1 -- start start 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.987+0000 7f255919d640 10 monclient: init 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): picked mon.noname-b con 0x7f255409ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409f020 0x7f2554130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): picked mon.noname-c con 0x7f255409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25541321a0 0x7f255411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): picked mon.noname-a con 0x7f25541321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255409c000 con 0x7f25541321a0 2024-09-15T03:32:36.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255409e240 con 0x7f255409ec50 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2554141be0 con 0x7f255409f020 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255919d640 10 monclient(hunting): authenticate will time out at 961.469414s 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255259c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409f020 msgr2=0x7f2554130a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409f020 0x7f2554130a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43738/0 (socket says 172.21.15.181:43738) 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 -- 172.21.15.181:0/586567001 learned_addr learned my addr 172.21.15.181:0/586567001 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): get_auth_request con 0x7f255409ec50 auth_method 0 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255359e640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25541321a0 0x7f255411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255359e640 10 monclient(hunting): get_auth_request con 0x7f25541321a0 auth_method 0 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient(hunting): handle_auth_done global_id 44851 payload 824 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 -- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409f020 msgr2=0x7f2554130a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409f020 0x7f2554130a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 -- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25541321a0 msgr2=0x7f255411b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25541321a0 0x7f255411b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 monclient: found mon.noname-b 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f2552d9d640 1 -- 172.21.15.181:0/586567001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c009ae0 con 0x7f255409ec50 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.988+0000 7f255359e640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25541321a0 0x7f255411b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f2552d9d640 1 -- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 msgr2=0x7f255409a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f2552d9d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f254400b600 tx=0x7f25440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f2552d9d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f254400b600 tx=0x7f25440091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f2544015070 mon_map magic: 0 v1 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2544015070 con 0x7f255409ec50 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:37.002 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 10 monclient: dump: 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 10 monclient: _finish_auth 0 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2544003d90 con 0x7f255409ec50 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 5 monclient: authenticate success, global_id 44851 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 10 monclient: _renew_subs 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255411db90 con 0x7f255409ec50 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f253affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255159a640 4 set_mon_vals no callback set 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 10 monclient: _renew_subs 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.991+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25541361a0 con 0x7f255409ec50 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2544006380 con 0x7f255409ec50 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f253affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f253affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f253affd640 10 monclient: dump: 2024-09-15T03:32:37.004 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2544010440 con 0x7f255409ec50 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f253affd640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2540038740 0x7f254003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f255919d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:37.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f255919d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f255919d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25540008d0 con 0x7f255409ec50 2024-09-15T03:32:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f255259c640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2540038740 0x7f254003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.992+0000 7f255259c640 10 monclient: get_auth_request con 0x7f2540038740 auth_method 0 2024-09-15T03:32:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.994+0000 7f255259c640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2540038740 0x7f254003abf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f25540a07a0 tx=0x7f2548007ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:36.996+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f254404ba00 con 0x7f255409ec50 2024-09-15T03:32:37.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.614+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2544006770 con 0x7f255409ec50 2024-09-15T03:32:37.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.614+0000 7f253affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:37.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.614+0000 7f253affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.773+0000 7f255919d640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T03:32:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.773+0000 7f255919d640 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T03:32:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.773+0000 7f255919d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:37.773+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7f25540a13e0 con 0x7f255409ec50 2024-09-15T03:32:38.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.588+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v94) v1 ==== 212+0+0 (secure 0 0 0) 0x7f2544018780 con 0x7f255409ec50 2024-09-15T03:32:38.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.589+0000 7f253affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T03:32:38.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.589+0000 7f253affd640 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-15T03:32:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.685+0000 7f255919d640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T03:32:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.685+0000 7f255919d640 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T03:32:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.685+0000 7f255919d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.685+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7f2554064900 con 0x7f255409ec50 2024-09-15T03:32:38.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.686+0000 7f253affd640 1 -- 172.21.15.181:0/586567001 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v94) v1 ==== 211+0+0 (secure 0 0 0) 0x7f2544018d60 con 0x7f255409ec50 2024-09-15T03:32:38.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.686+0000 7f253affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-15T03:32:38.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.686+0000 7f253affd640 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-15T03:32:38.688 INFO:tasks.workunit.client.0.smithi181.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2540038740 msgr2=0x7f254003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2540038740 0x7f254003abf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f25540a07a0 tx=0x7f2548007ac0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 10 monclient: shutdown 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 msgr2=0x7f255409a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f254400b600 tx=0x7f25440091a0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 shutdown_connections 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2540038740 0x7f254003abf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25541321a0 0x7f255411b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f255409f020 0x7f2554130a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 --2- 172.21.15.181:0/586567001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255409ec50 0x7f255409a760 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 >> 172.21.15.181:0/586567001 conn(0x7f25540a5d40 msgr2=0x7f2554134ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:38.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 shutdown_connections 2024-09-15T03:32:38.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.688+0000 7f255919d640 1 -- 172.21.15.181:0/586567001 wait complete. 2024-09-15T03:32:38.700 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-15T03:32:38.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.771+0000 7f2d2ee68640 10 monclient: get_monmap_and_config 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.771+0000 7f2d2ee68640 10 monclient: build_initial_monmap 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.771+0000 7f2d2ee68640 10 monclient: monmap: 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:38.772678+0000 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:38.772678+0000 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:38.774 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:38.774 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.772+0000 7f2d2ee68640 1 Processor -- start 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.772+0000 7f2d2ee68640 1 -- start start 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.772+0000 7f2d2ee68640 10 monclient: init 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d28098c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:38.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2d28099c70 0x7f2d28136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): picked mon.noname-c con 0x7f2d28099c70 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d28137200 0x7f2d281395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): picked mon.noname-a con 0x7f2d28137200 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2809f680 con 0x7f2d28098c70 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2809f420 con 0x7f2d28137200 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d28068a20 con 0x7f2d28099c70 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2ee68640 10 monclient(hunting): authenticate will time out at 963.254127s 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2d28099c70 0x7f2d28136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2d28099c70 0x7f2d28136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37540/0 (socket says 172.21.15.181:37540) 2024-09-15T03:32:38.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 -- 172.21.15.181:0/2386620718 learned_addr learned my addr 172.21.15.181:0/2386620718 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): get_auth_request con 0x7f2d28099c70 auth_method 0 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2d405640 1 --2- 172.21.15.181:0/2386620718 >> v2:172.21.15.153:3300/0 conn(0x7f2d28137200 0x7f2d281395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2d405640 10 monclient(hunting): get_auth_request con 0x7f2d28137200 auth_method 0 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2d405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2d405640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d2d405640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:38.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient(hunting): handle_auth_done global_id 44854 payload 274 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 -- 172.21.15.181:0/2386620718 >> v2:172.21.15.153:3300/0 conn(0x7f2d28137200 msgr2=0x7f2d281395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 --2- 172.21.15.181:0/2386620718 >> v2:172.21.15.153:3300/0 conn(0x7f2d28137200 0x7f2d281395e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 -- 172.21.15.181:0/2386620718 >> v1:172.21.15.181:6789/0 conn(0x7f2d28098c70 legacy=0x7f2d28099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 monclient: found mon.noname-c 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.773+0000 7f2d27fff640 1 -- 172.21.15.181:0/2386620718 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28096a30 con 0x7f2d28099c70 2024-09-15T03:32:38.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d27fff640 1 -- 172.21.15.181:0/2386620718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28099c70 msgr2=0x7f2d28136cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d27fff640 1 --2- 172.21.15.181:0/2386620718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28099c70 0x7f2d28136cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2d28096c20 tx=0x7f2d1c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33f69842355c0cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 1 -- 172.21.15.181:0/2386620718 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c032440 con 0x7f2d28099c70 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: dump: 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:38.787 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 1 -- 172.21.15.181:0/2386620718 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0325a0 con 0x7f2d28099c70 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d2ee68640 5 monclient: authenticate success, global_id 44854 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d2ee68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d2ee68640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 1 -- 172.21.15.181:0/2386620718 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c032940 con 0x7f2d28099c70 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d2ee68640 4 set_mon_vals no callback set 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:38.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.776+0000 7f2d277fe640 10 monclient: dump: 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 10 monclient: shutdown 2024-09-15T03:32:38.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2386620718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28099c70 msgr2=0x7f2d28136cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2386620718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28099c70 0x7f2d28136cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2d28096c20 tx=0x7f2d1c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2386620718 shutdown_connections 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2386620718 >> v2:172.21.15.153:3300/0 conn(0x7f2d28137200 0x7f2d281395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2386620718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28099c70 0x7f2d28136cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2386620718 >> 172.21.15.181:0/2386620718 conn(0x7f2d28070e90 msgr2=0x7f2d280a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2386620718 shutdown_connections 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2386620718 wait complete. 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 10 monclient: build_initial_monmap 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 10 monclient: monmap: 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:38.790 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.777+0000 7f2d2ee68640 1 Processor -- start 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 -- start start 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient: init 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 0x7f2d2811ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:38.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): picked mon.noname-b con 0x7f2d28098c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d28099c70 0x7f2d2811ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): picked mon.noname-c con 0x7f2d28099c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d28137200 0x7f2d2811f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): picked mon.noname-a con 0x7f2d28137200 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d281364a0 con 0x7f2d28137200 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d281208e0 con 0x7f2d28098c70 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2806eba0 con 0x7f2d28099c70 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2ee68640 10 monclient(hunting): authenticate will time out at 963.259553s 2024-09-15T03:32:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 0x7f2d2811ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:38.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d27fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d28099c70 msgr2=0x7f2d2811ef40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:32:38.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d27fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d28099c70 0x7f2d2811ef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:38.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 0x7f2d2811ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37542/0 (socket says 172.21.15.181:37542) 2024-09-15T03:32:38.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 -- 172.21.15.181:0/2903044407 learned_addr learned my addr 172.21.15.181:0/2903044407 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:38.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): get_auth_request con 0x7f2d28098c70 auth_method 0 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d28137200 0x7f2d2811f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:38.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): get_auth_request con 0x7f2d28137200 auth_method 0 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2d405640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient(hunting): handle_auth_done global_id 44857 payload 824 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 -- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d28099c70 msgr2=0x7f2d2811ef40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d28099c70 0x7f2d2811ef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 -- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d28137200 msgr2=0x7f2d2811f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d28137200 0x7f2d2811f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient: _finish_hunting 0 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 monclient: found mon.noname-b 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.778+0000 7f2d2cc04640 1 -- 172.21.15.181:0/2903044407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c009450 con 0x7f2d28098c70 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2cc04640 1 -- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 msgr2=0x7f2d2811ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2cc04640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 0x7f2d2811ea00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c8e0 tx=0x7f2d200026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2000aa60 con 0x7f2d28098c70 2024-09-15T03:32:38.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: dump: 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:38.797 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2000abe0 con 0x7f2d28098c70 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 5 monclient: authenticate success, global_id 44857 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 10 monclient: _renew_subs 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d280a1f90 con 0x7f2d28098c70 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20012420 con 0x7f2d28098c70 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d24ff9640 10 monclient: dump: 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 10 monclient: _renew_subs 2024-09-15T03:32:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d28136f00 con 0x7f2d28098c70 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.781+0000 7f2d26ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2d20012850 con 0x7f2d28098c70 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d2ee68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d2ee68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d2ee68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2809a070 con 0x7f2d28098c70 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d24ff9640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d140386f0 0x7f2d1403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f2d2004c310 con 0x7f2d28098c70 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d27fff640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d140386f0 0x7f2d1403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.782+0000 7f2d27fff640 10 monclient: get_auth_request con 0x7f2d140386f0 auth_method 0 2024-09-15T03:32:38.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.784+0000 7f2d27fff640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d140386f0 0x7f2d1403aba0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2d280a0c20 tx=0x7f2d1c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:38.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.786+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2d20022950 con 0x7f2d28098c70 2024-09-15T03:32:38.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.786+0000 7f2d24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:38.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.786+0000 7f2d24ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:38.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.947+0000 7f2d2ee68640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T03:32:38.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.947+0000 7f2d2ee68640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T03:32:38.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.947+0000 7f2d2ee68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:38.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:38.947+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f2d280008d0 con 0x7f2d28098c70 2024-09-15T03:32:39.132 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:32:39.133 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:32:39.133 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:32:39.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:32:39.136 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:32:39.136 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:32:39.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:39.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.239+0000 7f8a90ae8640 10 monclient: get_monmap_and_config 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.239+0000 7f8a90ae8640 10 monclient: build_initial_monmap 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.239+0000 7f8a90ae8640 10 monclient: monmap: 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:39.240662+0000 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:39.240662+0000 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:39.241 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:39.242 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.240+0000 7f8a90ae8640 1 Processor -- start 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.240+0000 7f8a90ae8640 1 -- start start 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.240+0000 7f8a90ae8640 10 monclient: init 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a8c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8a8c09ea50 0x7f8a8c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a8c09ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8a8c068d90 0x7f8a8c0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a8c068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c131cc0 con 0x7f8a8c068d90 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c09fbf0 con 0x7f8a8c09ea50 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c09fe30 con 0x7f8a8c1318f0 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a90ae8640 10 monclient(hunting): authenticate will time out at 956.880259s 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8a8c068d90 0x7f8a8c0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49422/0 (socket says 172.21.15.153:49422) 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8a8c068d90 0x7f8a8c0691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36486/0 (socket says 172.21.15.153:36486) 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 -- 172.21.15.153:0/2778721749 learned_addr learned my addr 172.21.15.153:0/2778721749 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 10 monclient(hunting): get_auth_request con 0x7f8a8c1318f0 auth_method 0 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): get_auth_request con 0x7f8a8c068d90 auth_method 0 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:39.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 -- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 msgr2=0x7f8a8c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 -- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 msgr2=0x7f8a8c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a89d9b640 1 --2- 172.21.15.153:0/2778721749 >> v2:172.21.15.181:3300/0 conn(0x7f8a8c09ea50 0x7f8a8c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a89d9b640 10 monclient(hunting): get_auth_request con 0x7f8a8c09ea50 auth_method 0 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8a59c640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient(hunting): handle_auth_done global_id 64311 payload 274 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 -- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 msgr2=0x7f8a8c131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 -- 172.21.15.153:0/2778721749 >> v2:172.21.15.181:3300/0 conn(0x7f8a8c09ea50 msgr2=0x7f8a8c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:39.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 --2- 172.21.15.153:0/2778721749 >> v2:172.21.15.181:3300/0 conn(0x7f8a8c09ea50 0x7f8a8c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:39.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 monclient: found mon.noname-a 2024-09-15T03:32:39.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:39.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:39.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a8ad9d640 1 -- 172.21.15.153:0/2778721749 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c0a1210 con 0x7f8a8c068d90 2024-09-15T03:32:39.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.241+0000 7f8a89d9b640 1 --2- 172.21.15.153:0/2778721749 >> v2:172.21.15.181:3300/0 conn(0x7f8a8c09ea50 0x7f8a8c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:39.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8ad9d640 1 -- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 msgr2=0x7f8a8c0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:39.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8ad9d640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c0691f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c068160 tx=0x7f8a84009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7265c03b22fc047 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:39.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 1 -- 172.21.15.153:0/2778721749 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84007760 con 0x7f8a8c068d90 2024-09-15T03:32:39.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: dump: 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: _finish_auth 0 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:39.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 1 -- 172.21.15.153:0/2778721749 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8402d050 con 0x7f8a8c068d90 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a90ae8640 5 monclient: authenticate success, global_id 64311 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a90ae8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a90ae8640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 1 -- 172.21.15.153:0/2778721749 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84007b30 con 0x7f8a8c068d90 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a90ae8640 4 set_mon_vals no callback set 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.244+0000 7f8a8959a640 10 monclient: dump: 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.255 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 10 monclient: shutdown 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 -- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 msgr2=0x7f8a8c0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c0691f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c068160 tx=0x7f8a84009780 comp rx=0 tx=0).stop 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 -- 172.21.15.153:0/2778721749 shutdown_connections 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c0691f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/2778721749 >> v2:172.21.15.181:3300/0 conn(0x7f8a8c09ea50 0x7f8a8c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/2778721749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 -- 172.21.15.153:0/2778721749 >> 172.21.15.153:0/2778721749 conn(0x7f8a8c067110 msgr2=0x7f8a8c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 -- 172.21.15.153:0/2778721749 shutdown_connections 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 1 -- 172.21.15.153:0/2778721749 wait complete. 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 10 monclient: build_initial_monmap 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.245+0000 7f8a90ae8640 10 monclient: monmap: 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:39.257 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 Processor -- start 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 -- start start 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient: init 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c09a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a8c068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a8c09ea50 0x7f8a8c130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a8c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a8c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c11e4d0 con 0x7f8a8c068d90 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c11dca0 con 0x7f8a8c09ea50 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c09b4c0 con 0x7f8a8c1318f0 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a90ae8640 10 monclient(hunting): authenticate will time out at 956.885832s 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c09a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c09a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36498/0 (socket says 172.21.15.153:36498) 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 1 -- 172.21.15.153:0/3099816040 learned_addr learned my addr 172.21.15.153:0/3099816040 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): get_auth_request con 0x7f8a8c068d90 auth_method 0 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8ad9d640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8ad9d640 10 monclient(hunting): get_auth_request con 0x7f8a8c1318f0 auth_method 0 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.246+0000 7f8a8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8ad9d640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 msgr2=0x7f8a8c140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a8c09ea50 0x7f8a8c130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8ad9d640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 msgr2=0x7f8a8c140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8ad9d640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): get_auth_request con 0x7f8a8c09ea50 auth_method 0 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8ad9d640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c140ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 10 monclient(hunting): handle_auth_done global_id 64314 payload 824 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 msgr2=0x7f8a8c140ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c140ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a8c09ea50 msgr2=0x7f8a8c130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a8c09ea50 0x7f8a8c130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 1 monclient: found mon.noname-a 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.247+0000 7f8a8a59c640 1 -- 172.21.15.153:0/3099816040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a84009a20 con 0x7f8a8c068d90 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a8a59c640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 msgr2=0x7f8a8c09a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a8a59c640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c09a940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00e8a0 tx=0x7f8a7c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c00c6d0 con 0x7f8a8c068d90 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: dump: 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c00c830 con 0x7f8a8c068d90 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 5 monclient: authenticate success, global_id 64314 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c00cbf0 con 0x7f8a8c068d90 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a827fc640 10 monclient: dump: 2024-09-15T03:32:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 10 monclient: _renew_subs 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a88d99640 4 set_mon_vals no callback set 2024-09-15T03:32:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c11e1f0 con 0x7f8a8c068d90 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 10 monclient: _renew_subs 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.249+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c11bbf0 con 0x7f8a8c068d90 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8a7c013410 con 0x7f8a8c068d90 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a90ae8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a90ae8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a90ae8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8c04bf60 con 0x7f8a8c068d90 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a827fc640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a6c038740 0x7f8a6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f8a7c04bf30 con 0x7f8a8c068d90 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a89d9b640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a6c038740 0x7f8a6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.250+0000 7f8a89d9b640 10 monclient: get_auth_request con 0x7f8a6c038740 auth_method 0 2024-09-15T03:32:39.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.253+0000 7f8a89d9b640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a6c038740 0x7f8a6c03abf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c11d0e0 tx=0x7f8a74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:39.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.255+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8a7c023300 con 0x7f8a8c068d90 2024-09-15T03:32:39.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.255+0000 7f8a827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:39.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.255+0000 7f8a827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a90ae8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a90ae8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a90ae8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8a8c0638d0 con 0x7f8a8c068d90 2024-09-15T03:32:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a827fc640 1 -- 172.21.15.153:0/3099816040 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f8a7c021070 con 0x7f8a8c068d90 2024-09-15T03:32:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.471+0000 7f8a827fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:39.473 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.473+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a6c038740 msgr2=0x7f8a6c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.473+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a6c038740 0x7f8a6c03abf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c11d0e0 tx=0x7f8a74008040 comp rx=0 tx=0).stop 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.473+0000 7f8a90ae8640 10 monclient: shutdown 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.473+0000 7f8a90ae8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.473+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 msgr2=0x7f8a8c09a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.473+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c09a940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00e8a0 tx=0x7f8a7c008040 comp rx=0 tx=0).stop 2024-09-15T03:32:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 shutdown_connections 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a6c038740 0x7f8a6c03abf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a8c1318f0 0x7f8a8c140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a8c09ea50 0x7f8a8c130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 --2- 172.21.15.153:0/3099816040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a8c068d90 0x7f8a8c09a940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 >> 172.21.15.153:0/3099816040 conn(0x7f8a8c067110 msgr2=0x7f8a8c09d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 shutdown_connections 2024-09-15T03:32:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:39.474+0000 7f8a90ae8640 1 -- 172.21.15.153:0/3099816040 wait complete. 2024-09-15T03:32:39.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.591+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v95) v1 ==== 165+0+0 (secure 0 0 0) 0x7f2d20016020 con 0x7f2d28098c70 2024-09-15T03:32:39.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.591+0000 7f2d24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T03:32:39.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.591+0000 7f2d24ff9640 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-15T03:32:39.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.693+0000 7f2d2ee68640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T03:32:39.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.693+0000 7f2d2ee68640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T03:32:39.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.693+0000 7f2d2ee68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.693+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f2d28064a20 con 0x7f2d28098c70 2024-09-15T03:32:39.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.694+0000 7f2d24ff9640 1 -- 172.21.15.181:0/2903044407 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v95) v1 ==== 175+0+0 (secure 0 0 0) 0x7f2d20016070 con 0x7f2d28098c70 2024-09-15T03:32:39.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.694+0000 7f2d24ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-15T03:32:39.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.694+0000 7f2d24ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-15T03:32:39.695 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d140386f0 msgr2=0x7f2d1403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d140386f0 0x7f2d1403aba0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2d280a0c20 tx=0x7f2d1c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 10 monclient: shutdown 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 msgr2=0x7f2d2811ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 0x7f2d2811ea00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c8e0 tx=0x7f2d200026c0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 shutdown_connections 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d140386f0 0x7f2d1403aba0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d28137200 0x7f2d2811f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d28099c70 0x7f2d2811ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 --2- 172.21.15.181:0/2903044407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d28098c70 0x7f2d2811ea00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.696+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 >> 172.21.15.181:0/2903044407 conn(0x7f2d28070e90 msgr2=0x7f2d28097990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:39.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.697+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 shutdown_connections 2024-09-15T03:32:39.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.697+0000 7f2d2ee68640 1 -- 172.21.15.181:0/2903044407 wait complete. 2024-09-15T03:32:39.707 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd find osd.7 2024-09-15T03:32:39.707 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep host 2024-09-15T03:32:39.707 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep foo 2024-09-15T03:32:39.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.778+0000 7f093db5d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:39.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.778+0000 7f093db5d640 10 monclient: build_initial_monmap 2024-09-15T03:32:39.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.778+0000 7f093db5d640 10 monclient: monmap: 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:39.779392+0000 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:39.779392+0000 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:39.780 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:39.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 Processor -- start 2024-09-15T03:32:39.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 -- start start 2024-09-15T03:32:39.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient: init 2024-09-15T03:32:39.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093809afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): picked mon.noname-b con 0x7f093809cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f093812e940 0x7f093812ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): picked mon.noname-c con 0x7f093812e940 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f093812f290 0x7f0938096a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): picked mon.noname-a con 0x7f093812f290 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0938097e90 con 0x7f093812f290 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0938099640 con 0x7f093812e940 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093809d6f0 con 0x7f093809cb20 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f093db5d640 10 monclient(hunting): authenticate will time out at 964.260870s 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.779+0000 7f0936ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f093812e940 0x7f093812ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f093812e940 0x7f093812ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37556/0 (socket says 172.21.15.181:37556) 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 -- 172.21.15.181:0/3887502785 learned_addr learned my addr 172.21.15.181:0/3887502785 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): get_auth_request con 0x7f093812e940 auth_method 0 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0937fff640 1 --2- 172.21.15.181:0/3887502785 >> v2:172.21.15.153:3300/0 conn(0x7f093812f290 0x7f0938096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f09377fe640 1 --2- 172.21.15.181:0/3887502785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093809afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient(hunting): handle_auth_done global_id 44866 payload 274 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 -- 172.21.15.181:0/3887502785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 msgr2=0x7f093809afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 --2- 172.21.15.181:0/3887502785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093809afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 -- 172.21.15.181:0/3887502785 >> v2:172.21.15.153:3300/0 conn(0x7f093812f290 msgr2=0x7f0938096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 --2- 172.21.15.181:0/3887502785 >> v2:172.21.15.153:3300/0 conn(0x7f093812f290 0x7f0938096a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 monclient: found mon.noname-c 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0936ffd640 1 -- 172.21.15.181:0/3887502785 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938098da0 con 0x7f093812e940 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0937fff640 10 monclient: get_auth_request con 0x7f093812f290 auth_method 0 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.780+0000 7f0937fff640 1 --2- 172.21.15.181:0/3887502785 >> v2:172.21.15.153:3300/0 conn(0x7f093812f290 0x7f0938096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f0936ffd640 1 -- 172.21.15.181:0/3887502785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 msgr2=0x7f093812ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f0936ffd640 1 --2- 172.21.15.181:0/3887502785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812ed50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0938068eb0 tx=0x7f092c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19351d9977aa84c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 1 -- 172.21.15.181:0/3887502785 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c032440 con 0x7f093812e940 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: dump: 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 5 monclient: authenticate success, global_id 44866 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 1 -- 172.21.15.181:0/3887502785 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c0325a0 con 0x7f093812e940 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 1 -- 172.21.15.181:0/3887502785 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c032940 con 0x7f093812e940 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 4 set_mon_vals no callback set 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:39.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f09367fc640 10 monclient: dump: 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 10 monclient: shutdown 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 -- 172.21.15.181:0/3887502785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 msgr2=0x7f093812ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 --2- 172.21.15.181:0/3887502785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812ed50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0938068eb0 tx=0x7f092c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 -- 172.21.15.181:0/3887502785 shutdown_connections 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 --2- 172.21.15.181:0/3887502785 >> v2:172.21.15.153:3300/0 conn(0x7f093812f290 0x7f0938096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 --2- 172.21.15.181:0/3887502785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812ed50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 --2- 172.21.15.181:0/3887502785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093809afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 -- 172.21.15.181:0/3887502785 >> 172.21.15.181:0/3887502785 conn(0x7f09380714d0 msgr2=0x7f09380a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 -- 172.21.15.181:0/3887502785 shutdown_connections 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.783+0000 7f093db5d640 1 -- 172.21.15.181:0/3887502785 wait complete. 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.784+0000 7f093db5d640 10 monclient: build_initial_monmap 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.784+0000 7f093db5d640 10 monclient: monmap: 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.784+0000 7f093db5d640 1 Processor -- start 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.784+0000 7f093db5d640 1 -- start start 2024-09-15T03:32:39.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.784+0000 7f093db5d640 10 monclient: init 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093813ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): picked mon.noname-c con 0x7f093809cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): picked mon.noname-b con 0x7f093812e940 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:39.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f093812f290 0x7f0938119840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): picked mon.noname-a con 0x7f093812f290 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938098950 con 0x7f093812f290 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093809ddd0 con 0x7f093812e940 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093811bfd0 con 0x7f093809cb20 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f093db5d640 10 monclient(hunting): authenticate will time out at 964.266356s 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37570/0 (socket says 172.21.15.181:37570) 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 -- 172.21.15.181:0/2324340622 learned_addr learned my addr 172.21.15.181:0/2324340622 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:39.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): get_auth_request con 0x7f093812e940 auth_method 0 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f093812f290 0x7f0938119840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): get_auth_request con 0x7f093812f290 auth_method 0 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 10 monclient(hunting): get_auth_request con 0x7f093809cb20 auth_method 0 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0937fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 msgr2=0x7f093813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 msgr2=0x7f093813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient(hunting): handle_auth_done global_id 44869 payload 824 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093813ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 msgr2=0x7f093813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f09377fe640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093813ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093813ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f093812f290 msgr2=0x7f0938119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f093812f290 0x7f0938119840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 monclient: found mon.noname-b 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.785+0000 7f0936ffd640 1 -- 172.21.15.181:0/2324340622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c009450 con 0x7f093812e940 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.787+0000 7f0936ffd640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 msgr2=0x7f093812bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.787+0000 7f0936ffd640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812bdd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f092c035040 tx=0x7f092c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c031a40 con 0x7f093812e940 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: dump: 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:39.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c031ba0 con 0x7f093812e940 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 5 monclient: authenticate success, global_id 44869 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 10 monclient: _renew_subs 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093813fc70 con 0x7f093812e940 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c03b400 con 0x7f093812e940 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f091f7fe640 10 monclient: dump: 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:39.804 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 10 monclient: _renew_subs 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093811b400 con 0x7f093812e940 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.788+0000 7f0935ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f092c039040 con 0x7f093812e940 2024-09-15T03:32:39.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f093db5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f093db5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f093db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09380636e0 con 0x7f093812e940 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f091f7fe640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0924038790 0x7f092403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6074+0+0 (secure 0 0 0) 0x7f092c06bdb0 con 0x7f093812e940 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f09377fe640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0924038790 0x7f092403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.789+0000 7f09377fe640 10 monclient: get_auth_request con 0x7f0924038790 auth_method 0 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.792+0000 7f09377fe640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0924038790 0x7f092403ac40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0928006fa0 tx=0x7f0928008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.793+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f092c049a90 con 0x7f093812e940 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.793+0000 7f091f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:39.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.793+0000 7f091f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.950+0000 7f093db5d640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-15T03:32:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.950+0000 7f093db5d640 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-15T03:32:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.950+0000 7f093db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.950+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7f0938097750 con 0x7f093812e940 2024-09-15T03:32:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.951+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v95) v1 ==== 65+0+226 (secure 0 0 0) 0x7f092c03f070 con 0x7f093812e940 2024-09-15T03:32:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.951+0000 7f091f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-15T03:32:39.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:39.951+0000 7f091f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:40.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.046+0000 7f093db5d640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-15T03:32:40.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.046+0000 7f093db5d640 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-15T03:32:40.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.046+0000 7f093db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.046+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7f093811bb60 con 0x7f093812e940 2024-09-15T03:32:40.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.047+0000 7f091f7fe640 1 -- 172.21.15.181:0/2324340622 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v95) v1 ==== 65+0+226 (secure 0 0 0) 0x7f092c037030 con 0x7f093812e940 2024-09-15T03:32:40.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.047+0000 7f091f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-15T03:32:40.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.047+0000 7f091f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:40.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0924038790 msgr2=0x7f092403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0924038790 0x7f092403ac40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0928006fa0 tx=0x7f0928008040 comp rx=0 tx=0).stop 2024-09-15T03:32:40.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 10 monclient: shutdown 2024-09-15T03:32:40.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:40.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 msgr2=0x7f093812bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812bdd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f092c035040 tx=0x7f092c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 shutdown_connections 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0924038790 0x7f092403ac40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f093812f290 0x7f0938119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f093812e940 0x7f093812bdd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 --2- 172.21.15.181:0/2324340622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f093809cb20 0x7f093813ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 >> 172.21.15.181:0/2324340622 conn(0x7f09380714d0 msgr2=0x7f093806db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 shutdown_connections 2024-09-15T03:32:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.049+0000 7f093db5d640 1 -- 172.21.15.181:0/2324340622 wait complete. 2024-09-15T03:32:40.059 INFO:tasks.workunit.client.0.smithi181.stdout: "host": "foo", 2024-09-15T03:32:40.059 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.7 2024-09-15T03:32:40.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.131+0000 7f2da4cc9640 10 monclient: get_monmap_and_config 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.131+0000 7f2da4cc9640 10 monclient: build_initial_monmap 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.131+0000 7f2da4cc9640 10 monclient: monmap: 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:40.132426+0000 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:40.132426+0000 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:40.133 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:40.134 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:40.134 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 Processor -- start 2024-09-15T03:32:40.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 -- start start 2024-09-15T03:32:40.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient: init 2024-09-15T03:32:40.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:40.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:32:40.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 0x7f2da009d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f2da0096a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2da009b0b0 0x7f2da0096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f2da009b0b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2da0097340 0x7f2da012c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f2da0097340 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da009c470 con 0x7f2da0097340 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da009c190 con 0x7f2da009b0b0 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da009c750 con 0x7f2da0096a30 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2da4cc9640 10 monclient(hunting): authenticate will time out at 964.613874s 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.132+0000 7f2d9dd9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2da009b0b0 0x7f2da0096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2da009b0b0 0x7f2da0096e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37576/0 (socket says 172.21.15.181:37576) 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 -- 172.21.15.181:0/1650752735 learned_addr learned my addr 172.21.15.181:0/1650752735 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): get_auth_request con 0x7f2da009b0b0 auth_method 0 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 0x7f2da009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 1 --2- 172.21.15.181:0/1650752735 >> v2:172.21.15.153:3300/0 conn(0x7f2da0097340 0x7f2da012c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): get_auth_request con 0x7f2da0097340 auth_method 0 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 10 monclient(hunting): get_auth_request con 0x7f2da0096a30 auth_method 0 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_done global_id 44872 payload 274 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 1 -- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 msgr2=0x7f2da009d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 1 -- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 msgr2=0x7f2da009d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 0x7f2da009d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 -- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 msgr2=0x7f2da009d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9e59c640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 0x7f2da009d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 0x7f2da009d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 -- 172.21.15.181:0/1650752735 >> v2:172.21.15.153:3300/0 conn(0x7f2da0097340 msgr2=0x7f2da012c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 --2- 172.21.15.181:0/1650752735 >> v2:172.21.15.153:3300/0 conn(0x7f2da0097340 0x7f2da012c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 monclient: found mon.noname-c 2024-09-15T03:32:40.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.133+0000 7f2d9dd9b640 1 -- 172.21.15.181:0/1650752735 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da00977a0 con 0x7f2da009b0b0 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9dd9b640 1 -- 172.21.15.181:0/1650752735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da009b0b0 msgr2=0x7f2da0096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9dd9b640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da009b0b0 0x7f2da0096e00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2da00a1e70 tx=0x7f2d940091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf7015b7d186d475 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 1 -- 172.21.15.181:0/1650752735 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d94032440 con 0x7f2da009b0b0 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: dump: 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.145 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: _finish_auth 0 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 5 monclient: authenticate success, global_id 44872 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 1 -- 172.21.15.181:0/1650752735 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d940325a0 con 0x7f2da009b0b0 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:40.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 4 set_mon_vals no callback set 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 1 -- 172.21.15.181:0/1650752735 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d94032940 con 0x7f2da009b0b0 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2d9d59a640 10 monclient: dump: 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 10 monclient: shutdown 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 -- 172.21.15.181:0/1650752735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da009b0b0 msgr2=0x7f2da0096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da009b0b0 0x7f2da0096e00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2da00a1e70 tx=0x7f2d940091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 -- 172.21.15.181:0/1650752735 shutdown_connections 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/1650752735 >> v2:172.21.15.153:3300/0 conn(0x7f2da0097340 0x7f2da012c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da009b0b0 0x7f2da0096e00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/1650752735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da0096a30 0x7f2da009d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 -- 172.21.15.181:0/1650752735 >> 172.21.15.181:0/1650752735 conn(0x7f2da0071100 msgr2=0x7f2da00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 -- 172.21.15.181:0/1650752735 shutdown_connections 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.136+0000 7f2da4cc9640 1 -- 172.21.15.181:0/1650752735 wait complete. 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.137+0000 7f2da4cc9640 10 monclient: build_initial_monmap 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.137+0000 7f2da4cc9640 10 monclient: monmap: 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.137+0000 7f2da4cc9640 1 Processor -- start 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.137+0000 7f2da4cc9640 1 -- start start 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.137+0000 7f2da4cc9640 10 monclient: init 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 0x7f2da0131820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f2da0096a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da0097340 0x7f2da011b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f2da0097340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 0x7f2da011b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f2da009b0b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da01327e0 con 0x7f2da0097340 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da011ec20 con 0x7f2da0096a30 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da011ce10 con 0x7f2da009b0b0 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2da4cc9640 10 monclient(hunting): authenticate will time out at 964.619138s 2024-09-15T03:32:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 0x7f2da0131820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 0x7f2da0131820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37580/0 (socket says 172.21.15.181:37580) 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 -- 172.21.15.181:0/3613778317 learned_addr learned my addr 172.21.15.181:0/3613778317 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): get_auth_request con 0x7f2da0096a30 auth_method 0 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da0097340 0x7f2da011b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 0x7f2da011b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): get_auth_request con 0x7f2da0097340 auth_method 0 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 10 monclient(hunting): get_auth_request con 0x7f2da009b0b0 auth_method 0 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 msgr2=0x7f2da011b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 msgr2=0x7f2da011b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 0x7f2da011b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient(hunting): handle_auth_done global_id 44875 payload 824 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9ed9d640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 0x7f2da011b860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 msgr2=0x7f2da011b860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 0x7f2da011b860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da0097340 msgr2=0x7f2da011b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da0097340 0x7f2da011b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:40.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 monclient: found mon.noname-b 2024-09-15T03:32:40.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:40.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.138+0000 7f2d9e59c640 1 -- 172.21.15.181:0/3613778317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94009450 con 0x7f2da0096a30 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d9e59c640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 msgr2=0x7f2da0131820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d9e59c640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 0x7f2da0131820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000d890 tx=0x7f2d90008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000de20 con 0x7f2da0096a30 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: dump: 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.158 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:40.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 5 monclient: authenticate success, global_id 44875 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9000bae0 con 0x7f2da0096a30 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 10 monclient: _renew_subs 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da011e940 con 0x7f2da0096a30 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000bea0 con 0x7f2da0096a30 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d8e7fc640 10 monclient: dump: 2024-09-15T03:32:40.160 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 10 monclient: _renew_subs 2024-09-15T03:32:40.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2d9cd99640 4 set_mon_vals no callback set 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.140+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da011d9e0 con 0x7f2da0096a30 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2d90014640 con 0x7f2da0096a30 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2da4cc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2da4cc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2da4cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da011ee60 con 0x7f2da0096a30 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2d8e7fc640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d80038740 0x7f2d8003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6074+0+0 (secure 0 0 0) 0x7f2d9004c620 con 0x7f2da0096a30 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.141+0000 7f2d9dd9b640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d80038740 0x7f2d8003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.142+0000 7f2d9dd9b640 10 monclient: get_auth_request con 0x7f2d80038740 auth_method 0 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.144+0000 7f2d9dd9b640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d80038740 0x7f2d8003abf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2da009d860 tx=0x7f2d94032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.146+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2da011ee60 con 0x7f2da0096a30 2024-09-15T03:32:40.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.146+0000 7f2d8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:40.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.146+0000 7f2d8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:40.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.306+0000 7f2da4cc9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T03:32:40.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.306+0000 7f2da4cc9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T03:32:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.306+0000 7f2da4cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.306+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f2da00008d0 con 0x7f2da0096a30 2024-09-15T03:32:40.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.592+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v96) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d900148f0 con 0x7f2da0096a30 2024-09-15T03:32:40.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.592+0000 7f2d8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T03:32:40.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.592+0000 7f2d8e7fc640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-15T03:32:40.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.691+0000 7f2da4cc9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T03:32:40.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.691+0000 7f2da4cc9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T03:32:40.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.691+0000 7f2da4cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.691+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f2da00a1a30 con 0x7f2da0096a30 2024-09-15T03:32:40.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.692+0000 7f2d8e7fc640 1 -- 172.21.15.181:0/3613778317 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v96) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2d9001dbb0 con 0x7f2da0096a30 2024-09-15T03:32:40.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.692+0000 7f2d8e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-15T03:32:40.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.692+0000 7f2d8e7fc640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-15T03:32:40.693 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.7' does not appear in the crush map 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d80038740 msgr2=0x7f2d8003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d80038740 0x7f2d8003abf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2da009d860 tx=0x7f2d94032000 comp rx=0 tx=0).stop 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 10 monclient: shutdown 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 msgr2=0x7f2da0131820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 0x7f2da0131820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000d890 tx=0x7f2d90008040 comp rx=0 tx=0).stop 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 shutdown_connections 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d80038740 0x7f2d8003abf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da009b0b0 0x7f2da011b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da0097340 0x7f2da011b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 --2- 172.21.15.181:0/3613778317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da0096a30 0x7f2da0131820 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 >> 172.21.15.181:0/3613778317 conn(0x7f2da0071100 msgr2=0x7f2da0069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:40.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 shutdown_connections 2024-09-15T03:32:40.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.694+0000 7f2da4cc9640 1 -- 172.21.15.181:0/3613778317 wait complete. 2024-09-15T03:32:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.9 2024-09-15T03:32:40.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.777+0000 7fe50883f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:40.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.777+0000 7fe50883f640 10 monclient: build_initial_monmap 2024-09-15T03:32:40.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.777+0000 7fe50883f640 10 monclient: monmap: 2024-09-15T03:32:40.778 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:40.777991+0000 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:40.777991+0000 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.777+0000 7fe50883f640 1 Processor -- start 2024-09-15T03:32:40.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.777+0000 7fe50883f640 1 -- start start 2024-09-15T03:32:40.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.777+0000 7fe50883f640 10 monclient: init 2024-09-15T03:32:40.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:40.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T03:32:40.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe504069a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe50409baa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe50409bea0 0x7fe5041382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient(hunting): picked mon.noname-a con 0x7fe50409bea0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50412ff50 con 0x7fe50409baa0 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe504069e00 con 0x7fe504069a30 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50409d100 con 0x7fe50409bea0 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50883f640 10 monclient(hunting): authenticate will time out at 965.259503s 2024-09-15T03:32:40.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50259c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe504069a30 0x7fe50412f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:45756/0 (socket says 172.21.15.181:45756) 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe50259c640 1 -- 172.21.15.181:0/340622201 learned_addr learned my addr 172.21.15.181:0/340622201 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 1 --2- 172.21.15.181:0/340622201 >> v2:172.21.15.153:3300/0 conn(0x7fe50409bea0 0x7fe5041382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): get_auth_request con 0x7fe50409bea0 auth_method 0 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.778+0000 7fe502d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 10 monclient(hunting): handle_auth_done global_id 64326 payload 274 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 1 -- 172.21.15.181:0/340622201 >> v1:172.21.15.181:6789/0 conn(0x7fe504069a30 legacy=0x7fe50412f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 1 -- 172.21.15.181:0/340622201 >> v1:172.21.15.153:6789/0 conn(0x7fe50409baa0 legacy=0x7fe504134b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:40.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 1 monclient: found mon.noname-a 2024-09-15T03:32:40.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:40.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:40.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.779+0000 7fe502d9d640 1 -- 172.21.15.181:0/340622201 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50409e270 con 0x7fe50409bea0 2024-09-15T03:32:40.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe502d9d640 1 -- 172.21.15.181:0/340622201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 msgr2=0x7fe5041382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe502d9d640 1 --2- 172.21.15.181:0/340622201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041382a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009e60 tx=0x7fe4f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9816b8bd9758c89 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 1 -- 172.21.15.181:0/340622201 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4007630 con 0x7fe50409bea0 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: dump: 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: _finish_auth 0 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 1 -- 172.21.15.181:0/340622201 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f402d050 con 0x7fe50409bea0 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 5 monclient: authenticate success, global_id 64326 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 1 -- 172.21.15.181:0/340622201 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f40079c0 con 0x7fe50409bea0 2024-09-15T03:32:40.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 4 set_mon_vals no callback set 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50159a640 10 monclient: dump: 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.792 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 10 monclient: shutdown 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 1 -- 172.21.15.181:0/340622201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 msgr2=0x7fe5041382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.782+0000 7fe50883f640 1 --2- 172.21.15.181:0/340622201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041382a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009e60 tx=0x7fe4f4009780 comp rx=0 tx=0).stop 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 -- 172.21.15.181:0/340622201 shutdown_connections 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 --2- 172.21.15.181:0/340622201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041382a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 -- 172.21.15.181:0/340622201 >> 172.21.15.181:0/340622201 conn(0x7fe50409e9f0 msgr2=0x7fe50409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 -- 172.21.15.181:0/340622201 shutdown_connections 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 -- 172.21.15.181:0/340622201 wait complete. 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 10 monclient: build_initial_monmap 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 10 monclient: monmap: 2024-09-15T03:32:40.793 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 Processor -- start 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 1 -- start start 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.783+0000 7fe50883f640 10 monclient: init 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:40.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 0x7fe50413ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): picked mon.noname-c con 0x7fe504069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 0x7fe50412eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): picked mon.noname-b con 0x7fe50409baa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041187c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): picked mon.noname-a con 0x7fe50409bea0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504119ca0 con 0x7fe50409bea0 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50411bf40 con 0x7fe50409baa0 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50409dea0 con 0x7fe504069a30 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:40.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50883f640 10 monclient(hunting): authenticate will time out at 965.265436s 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 0x7fe50412eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 0x7fe50412eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37590/0 (socket says 172.21.15.181:37590) 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 -- 172.21.15.181:0/2529265552 learned_addr learned my addr 172.21.15.181:0/2529265552 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): get_auth_request con 0x7fe50409baa0 auth_method 0 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 0x7fe50413ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 10 monclient(hunting): get_auth_request con 0x7fe504069a30 auth_method 0 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): get_auth_request con 0x7fe50409bea0 auth_method 0 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 msgr2=0x7fe50413ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 msgr2=0x7fe50413ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 0x7fe50413ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:40.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe50259c640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 0x7fe50413ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe502d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient(hunting): handle_auth_done global_id 44878 payload 824 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 msgr2=0x7fe50413ee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 0x7fe50413ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 msgr2=0x7fe5041187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 monclient: found mon.noname-b 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.784+0000 7fe501d9b640 1 -- 172.21.15.181:0/2529265552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4009a20 con 0x7fe50409baa0 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe501d9b640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 msgr2=0x7fe50412eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe501d9b640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 0x7fe50412eb40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c610 tx=0x7fe4ec007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:40.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec012840 con 0x7fe50409baa0 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: dump: 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 5 monclient: authenticate success, global_id 44878 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 10 monclient: _renew_subs 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50411bc60 con 0x7fe50409baa0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 10 monclient: _renew_subs 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50413f350 con 0x7fe50409baa0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec0129c0 con 0x7fe50409baa0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec012d80 con 0x7fe50409baa0 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:40.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe4fa7fc640 10 monclient: dump: 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.787+0000 7fe500d99640 4 set_mon_vals no callback set 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe4ec011590 con 0x7fe50409baa0 2024-09-15T03:32:40.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe50883f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe50883f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe50883f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50413f620 con 0x7fe50409baa0 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe4fa7fc640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe4e4038740 0x7fe4e403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6024+0+0 (secure 0 0 0) 0x7fe4ec04c1e0 con 0x7fe50409baa0 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe50259c640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe4e4038740 0x7fe4e403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:40.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.788+0000 7fe50259c640 10 monclient: get_auth_request con 0x7fe4e4038740 auth_method 0 2024-09-15T03:32:40.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.791+0000 7fe50259c640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe4e4038740 0x7fe4e403abf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc007d10 tx=0x7fe4fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:40.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.792+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe4ec010ac0 con 0x7fe50409baa0 2024-09-15T03:32:40.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.792+0000 7fe4fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:40.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.792+0000 7fe4fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.951+0000 7fe50883f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T03:32:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.951+0000 7fe50883f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T03:32:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.951+0000 7fe50883f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.951+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7fe5040008d0 con 0x7fe50409baa0 2024-09-15T03:32:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.952+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v96) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe4ec016030 con 0x7fe50409baa0 2024-09-15T03:32:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.952+0000 7fe4fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T03:32:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:40.952+0000 7fe4fa7fc640 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-15T03:32:41.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.047+0000 7fe50883f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.047+0000 7fe50883f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.047+0000 7fe50883f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.047+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7fe50411c0b0 con 0x7fe50409baa0 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.048+0000 7fe4fa7fc640 1 -- 172.21.15.181:0/2529265552 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v96) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe4ec016020 con 0x7fe50409baa0 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.048+0000 7fe4fa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.048+0000 7fe4fa7fc640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-15T03:32:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.9' does not appear in the crush map 2024-09-15T03:32:41.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe4e4038740 msgr2=0x7fe4e403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe4e4038740 0x7fe4e403abf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc007d10 tx=0x7fe4fc009040 comp rx=0 tx=0).stop 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 10 monclient: shutdown 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 msgr2=0x7fe50412eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 0x7fe50412eb40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c610 tx=0x7fe4ec007ed0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 shutdown_connections 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe4e4038740 0x7fe4e403abf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe50409bea0 0x7fe5041187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe50409baa0 0x7fe50412eb40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 --2- 172.21.15.181:0/2529265552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe504069a30 0x7fe50413ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 >> 172.21.15.181:0/2529265552 conn(0x7fe50409e9f0 msgr2=0x7fe5041327b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 shutdown_connections 2024-09-15T03:32:41.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.050+0000 7fe50883f640 1 -- 172.21.15.181:0/2529265552 wait complete. 2024-09-15T03:32:41.060 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm foo 2024-09-15T03:32:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.132+0000 7f42b559d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.132+0000 7f42b559d640 10 monclient: build_initial_monmap 2024-09-15T03:32:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.132+0000 7f42b559d640 10 monclient: monmap: 2024-09-15T03:32:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:41.133681+0000 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:41.133681+0000 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 1 Processor -- start 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 1 -- start start 2024-09-15T03:32:41.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 10 monclient: init 2024-09-15T03:32:41.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:41.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:32:41.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f42b006e130 0x7f42b009da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 10 monclient(hunting): picked mon.noname-c con 0x7f42b006e130 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:41.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.133+0000 7f42b559d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42b006a790 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 0x7f42b009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 10 monclient(hunting): picked mon.noname-b con 0x7f42b006e500 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b006a0b0 con 0x7f42b006a790 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b0068c30 con 0x7f42b006e130 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b0069750 con 0x7f42b006e500 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42b559d640 10 monclient(hunting): authenticate will time out at 965.615036s 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f42b006e130 0x7f42b009da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f42b006e130 0x7f42b009da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37594/0 (socket says 172.21.15.181:37594) 2024-09-15T03:32:41.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 -- 172.21.15.181:0/4067078600 learned_addr learned my addr 172.21.15.181:0/4067078600 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): get_auth_request con 0x7f42b006e130 auth_method 0 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42ae7fc640 1 --1- 172.21.15.181:0/4067078600 >> v1:172.21.15.181:6789/0 conn(0x7f42b006a790 0x7f42b009dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:45766/0 (socket says 172.21.15.181:45766) 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 0x7f42b009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 10 monclient(hunting): get_auth_request con 0x7f42b006e500 auth_method 0 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 1 -- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 msgr2=0x7f42b009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 1 -- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 msgr2=0x7f42b009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 0x7f42b009e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42af7fe640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 0x7f42b009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient(hunting): handle_auth_done global_id 44881 payload 274 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 -- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 msgr2=0x7f42b009e6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 0x7f42b009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 -- 172.21.15.181:0/4067078600 >> v1:172.21.15.181:6789/0 conn(0x7f42b006a790 legacy=0x7f42b009dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:41.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 monclient: found mon.noname-c 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.134+0000 7f42aeffd640 1 -- 172.21.15.181:0/4067078600 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0069a60 con 0x7f42b006e130 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42aeffd640 1 -- 172.21.15.181:0/4067078600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e130 msgr2=0x7f42b009da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42aeffd640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e130 0x7f42b009da60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009910 tx=0x7f42a00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b09a0a65dfe5313 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 1 -- 172.21.15.181:0/4067078600 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0032440 con 0x7f42b006e130 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: dump: 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.148 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 1 -- 172.21.15.181:0/4067078600 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a00325a0 con 0x7f42b006e130 2024-09-15T03:32:41.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 5 monclient: authenticate success, global_id 44881 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 1 -- 172.21.15.181:0/4067078600 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0032940 con 0x7f42b006e130 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 4 set_mon_vals no callback set 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42adffb640 10 monclient: dump: 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 10 monclient: shutdown 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 -- 172.21.15.181:0/4067078600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e130 msgr2=0x7f42b009da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e130 0x7f42b009da60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009910 tx=0x7f42a00091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 -- 172.21.15.181:0/4067078600 shutdown_connections 2024-09-15T03:32:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e500 0x7f42b009e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 --2- 172.21.15.181:0/4067078600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e130 0x7f42b009da60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 -- 172.21.15.181:0/4067078600 >> 172.21.15.181:0/4067078600 conn(0x7f42b0071340 msgr2=0x7f42b00a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 -- 172.21.15.181:0/4067078600 shutdown_connections 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.137+0000 7f42b559d640 1 -- 172.21.15.181:0/4067078600 wait complete. 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.138+0000 7f42b559d640 10 monclient: build_initial_monmap 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.138+0000 7f42b559d640 10 monclient: monmap: 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.138+0000 7f42b559d640 1 Processor -- start 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.138+0000 7f42b559d640 1 -- start start 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.138+0000 7f42b559d640 10 monclient: init 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b006a790 0x7f42b0141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): picked mon.noname-a con 0x7f42b006a790 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 0x7f42b0133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): picked mon.noname-c con 0x7f42b006e130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 0x7f42b011ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): picked mon.noname-b con 0x7f42b006e500 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b01222f0 con 0x7f42b006a790 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b0120630 con 0x7f42b006e500 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b0120ad0 con 0x7f42b006e130 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:41.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42b559d640 10 monclient(hunting): authenticate will time out at 965.620333s 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 0x7f42b011ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 0x7f42b011ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37608/0 (socket says 172.21.15.181:37608) 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 -- 172.21.15.181:0/4142676065 learned_addr learned my addr 172.21.15.181:0/4142676065 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): get_auth_request con 0x7f42b006e500 auth_method 0 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:41.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 0x7f42b0133ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42aeffd640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b006a790 0x7f42b0141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 10 monclient(hunting): get_auth_request con 0x7f42b006e130 auth_method 0 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42aeffd640 10 monclient(hunting): get_auth_request con 0x7f42b006a790 auth_method 0 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 msgr2=0x7f42b0133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:32:41.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 msgr2=0x7f42b0133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient(hunting): handle_auth_done global_id 44884 payload 824 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 0x7f42b0133ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 msgr2=0x7f42b0133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42ae7fc640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 0x7f42b0133ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 0x7f42b0133ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b006a790 msgr2=0x7f42b0141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b006a790 0x7f42b0141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 monclient: found mon.noname-b 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:41.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42af7fe640 1 -- 172.21.15.181:0/4142676065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a0009450 con 0x7f42b006e500 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.139+0000 7f42aeffd640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b006a790 0x7f42b0141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.141+0000 7f42af7fe640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 msgr2=0x7f42b011ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.141+0000 7f42af7fe640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 0x7f42b011ea00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f42a8007f50 tx=0x7f42a8009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8014420 con 0x7f42b006e500 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: dump: 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 5 monclient: authenticate success, global_id 44884 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a8014580 con 0x7f42b006e500 2024-09-15T03:32:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8014940 con 0x7f42b006e500 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 10 monclient: _renew_subs 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b006e9d0 con 0x7f42b006e500 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 10 monclient: dump: 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42ad7fa640 4 set_mon_vals no callback set 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 10 monclient: _renew_subs 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b011fde0 con 0x7f42b006e500 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.142+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f42a801e460 con 0x7f42b006e500 2024-09-15T03:32:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f4296ffd640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f429c038770 0x7f429c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6024+0+0 (secure 0 0 0) 0x7f42a804c6e0 con 0x7f42b006e500 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f42b559d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f42b559d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f42b559d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f42aeffd640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f429c038770 0x7f429c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b00008d0 con 0x7f42b006e500 2024-09-15T03:32:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.143+0000 7f42aeffd640 10 monclient: get_auth_request con 0x7f429c038770 auth_method 0 2024-09-15T03:32:41.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.145+0000 7f42aeffd640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f429c038770 0x7f429c03ac20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f42b006edc0 tx=0x7f42a0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:41.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.147+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f42a8013380 con 0x7f42b006e500 2024-09-15T03:32:41.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.147+0000 7f4296ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:41.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.147+0000 7f4296ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:41.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.307+0000 7f42b559d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:41.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.307+0000 7f42b559d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:41.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.307+0000 7f42b559d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.307+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f42b01220a0 con 0x7f42b006e500 2024-09-15T03:32:41.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.593+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v97) v1 ==== 120+0+0 (secure 0 0 0) 0x7f42a800d0d0 con 0x7f42b006e500 2024-09-15T03:32:41.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.593+0000 7f4296ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:41.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.593+0000 7f4296ffd640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-15T03:32:41.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.690+0000 7f42b559d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:41.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.690+0000 7f42b559d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:41.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.690+0000 7f42b559d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.690+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f42b0072640 con 0x7f42b006e500 2024-09-15T03:32:41.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.692+0000 7f4296ffd640 1 -- 172.21.15.181:0/4142676065 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v97) v1 ==== 120+0+0 (secure 0 0 0) 0x7f42a801b310 con 0x7f42b006e500 2024-09-15T03:32:41.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.692+0000 7f4296ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-15T03:32:41.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.692+0000 7f4296ffd640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-15T03:32:41.693 INFO:tasks.workunit.client.0.smithi181.stderr:device 'foo' does not appear in the crush map 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f429c038770 msgr2=0x7f429c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f429c038770 0x7f429c03ac20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f42b006edc0 tx=0x7f42a0032000 comp rx=0 tx=0).stop 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 10 monclient: shutdown 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 msgr2=0x7f42b011ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 0x7f42b011ea00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f42a8007f50 tx=0x7f42a8009830 comp rx=0 tx=0).stop 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 shutdown_connections 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f429c038770 0x7f429c03ac20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42b006e500 0x7f42b011ea00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42b006e130 0x7f42b0133ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 --2- 172.21.15.181:0/4142676065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b006a790 0x7f42b0141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 >> 172.21.15.181:0/4142676065 conn(0x7f42b0071340 msgr2=0x7f42b00a28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 shutdown_connections 2024-09-15T03:32:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.694+0000 7f42b559d640 1 -- 172.21.15.181:0/4142676065 wait complete. 2024-09-15T03:32:41.704 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.774+0000 7f1c266a8640 10 monclient: get_monmap_and_config 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.774+0000 7f1c266a8640 10 monclient: build_initial_monmap 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.774+0000 7f1c266a8640 10 monclient: monmap: 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:41.774928+0000 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:41.774928+0000 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:41.775 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.776 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:41.776 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:41.776 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:41.776 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:41.776 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:41.776 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.774+0000 7f1c266a8640 1 Processor -- start 2024-09-15T03:32:41.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.774+0000 7f1c266a8640 1 -- start start 2024-09-15T03:32:41.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.774+0000 7f1c266a8640 10 monclient: init 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1c20131b40 0x7f1c2009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): picked mon.noname-a con 0x7f1c20131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1c2009f240 0x7f1c20131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2009f240 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): picked mon.noname-b con 0x7f1c20132450 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2006aa60 con 0x7f1c20131b40 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2006a230 con 0x7f1c2009f240 2024-09-15T03:32:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200a0300 con 0x7f1c20132450 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c266a8640 10 monclient(hunting): authenticate will time out at 966.256441s 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1c2009f240 0x7f1c20131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1c2009f240 0x7f1c20131f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37622/0 (socket says 172.21.15.181:37622) 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 -- 172.21.15.181:0/1801768627 learned_addr learned my addr 172.21.15.181:0/1801768627 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request con 0x7f1c2009f240 auth_method 0 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c24c45640 1 --2- 172.21.15.181:0/1801768627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c24c45640 10 monclient(hunting): get_auth_request con 0x7f1c20132450 auth_method 0 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c24c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c24c45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c24c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1ffff640 1 --2- 172.21.15.181:0/1801768627 >> v2:172.21.15.153:3300/0 conn(0x7f1c20131b40 0x7f1c2009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient(hunting): handle_auth_done global_id 44887 payload 274 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 -- 172.21.15.181:0/1801768627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 msgr2=0x7f1c20068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 --2- 172.21.15.181:0/1801768627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20068e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 -- 172.21.15.181:0/1801768627 >> v2:172.21.15.153:3300/0 conn(0x7f1c20131b40 msgr2=0x7f1c2009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 --2- 172.21.15.181:0/1801768627 >> v2:172.21.15.153:3300/0 conn(0x7f1c20131b40 0x7f1c2009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:41.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 monclient: found mon.noname-c 2024-09-15T03:32:41.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:41.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1f7fe640 1 -- 172.21.15.181:0/1801768627 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200a00c0 con 0x7f1c2009f240 2024-09-15T03:32:41.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1ffff640 10 monclient: get_auth_request con 0x7f1c20131b40 auth_method 0 2024-09-15T03:32:41.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.775+0000 7f1c1ffff640 1 --2- 172.21.15.181:0/1801768627 >> v2:172.21.15.153:3300/0 conn(0x7f1c20131b40 0x7f1c2009ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:41.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1f7fe640 1 -- 172.21.15.181:0/1801768627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 msgr2=0x7f1c20131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:41.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1f7fe640 1 --2- 172.21.15.181:0/1801768627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c20131f10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1c2009f660 tx=0x7f1c140091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a05d43061447374 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:41.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 1 -- 172.21.15.181:0/1801768627 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14032440 con 0x7f1c2009f240 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: dump: 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: _finish_auth 0 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 1 -- 172.21.15.181:0/1801768627 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c140325a0 con 0x7f1c2009f240 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c266a8640 5 monclient: authenticate success, global_id 44887 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c266a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c266a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 1 -- 172.21.15.181:0/1801768627 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14032940 con 0x7f1c2009f240 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c266a8640 4 set_mon_vals no callback set 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.778+0000 7f1c1effd640 10 monclient: dump: 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.790 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 10 monclient: shutdown 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 -- 172.21.15.181:0/1801768627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 msgr2=0x7f1c20131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 --2- 172.21.15.181:0/1801768627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c20131f10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1c2009f660 tx=0x7f1c140091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 -- 172.21.15.181:0/1801768627 shutdown_connections 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 --2- 172.21.15.181:0/1801768627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 --2- 172.21.15.181:0/1801768627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c20131f10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 --2- 172.21.15.181:0/1801768627 >> v2:172.21.15.153:3300/0 conn(0x7f1c20131b40 0x7f1c2009ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 -- 172.21.15.181:0/1801768627 >> 172.21.15.181:0/1801768627 conn(0x7f1c200a5100 msgr2=0x7f1c200a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 -- 172.21.15.181:0/1801768627 shutdown_connections 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 1 -- 172.21.15.181:0/1801768627 wait complete. 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 10 monclient: build_initial_monmap 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.779+0000 7f1c266a8640 10 monclient: monmap: 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:41.792 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 Processor -- start 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 -- start start 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient: init 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c2012eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): picked mon.noname-b con 0x7f1c2009f240 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c20131b40 0x7f1c2012f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): picked mon.noname-a con 0x7f1c20131b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:41.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20099b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): picked mon.noname-c con 0x7f1c20132450 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2011ccb0 con 0x7f1c20131b40 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2011d270 con 0x7f1c2009f240 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20141cc0 con 0x7f1c20132450 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c266a8640 10 monclient(hunting): authenticate will time out at 966.261737s 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c2012eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c2012eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37630/0 (socket says 172.21.15.181:37630) 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 1 -- 172.21.15.181:0/2940301162 learned_addr learned my addr 172.21.15.181:0/2940301162 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:41.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): get_auth_request con 0x7f1c2009f240 auth_method 0 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c1f7fe640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c20131b40 0x7f1c2012f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.780+0000 7f1c24c45640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c24c45640 10 monclient(hunting): get_auth_request con 0x7f1c20132450 auth_method 0 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c24c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c24c45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c24c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request con 0x7f1c20131b40 auth_method 0 2024-09-15T03:32:41.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 10 monclient(hunting): handle_auth_done global_id 44890 payload 824 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 1 -- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 msgr2=0x7f1c20099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20099b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 1 -- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c20131b40 msgr2=0x7f1c2012f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c20131b40 0x7f1c2012f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 1 monclient: found mon.noname-b 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.781+0000 7f1c1ffff640 1 -- 172.21.15.181:0/2940301162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14009450 con 0x7f1c2009f240 2024-09-15T03:32:41.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c1ffff640 1 -- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 msgr2=0x7f1c2012eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c1ffff640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c2012eb80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c880 tx=0x7f1c10008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1000ce10 con 0x7f1c2009f240 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: dump: 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.797 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: _finish_auth 0 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c10007aa0 con 0x7f1c2009f240 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 5 monclient: authenticate success, global_id 44890 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 10 monclient: _renew_subs 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2011bdc0 con 0x7f1c2009f240 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10007e60 con 0x7f1c2009f240 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:41.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c07fff640 10 monclient: dump: 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 10 monclient: _renew_subs 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c20135290 con 0x7f1c2009f240 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.783+0000 7f1c1e7fc640 4 set_mon_vals no callback set 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1c10011600 con 0x7f1c2009f240 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c266a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c266a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c266a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2011d430 con 0x7f1c2009f240 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c07fff640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c0c038740 0x7f1c0c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5813+0+0 (secure 0 0 0) 0x7f1c1004c070 con 0x7f1c2009f240 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c1f7fe640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c0c038740 0x7f1c0c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.784+0000 7f1c1f7fe640 10 monclient: get_auth_request con 0x7f1c0c038740 auth_method 0 2024-09-15T03:32:41.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.787+0000 7f1c1f7fe640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c0c038740 0x7f1c0c03abf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1c2006a6b0 tx=0x7f1c14032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:41.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.788+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1c10017ae0 con 0x7f1c2009f240 2024-09-15T03:32:41.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.788+0000 7f1c07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:41.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.788+0000 7f1c07fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:41.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.956+0000 7f1c266a8640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:32:41.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.956+0000 7f1c266a8640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T03:32:41.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.956+0000 7f1c266a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:41.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:41.956+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f1c200636e0 con 0x7f1c2009f240 2024-09-15T03:32:42.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.590+0000 7f29c9d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.590+0000 7f29c9d5d640 10 monclient: build_initial_monmap 2024-09-15T03:32:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.590+0000 7f29c9d5d640 10 monclient: monmap: 2024-09-15T03:32:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:42.591 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:42.590842+0000 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:42.590842+0000 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:42.592 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:42.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 Processor -- start 2024-09-15T03:32:42.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 -- start start 2024-09-15T03:32:42.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient: init 2024-09-15T03:32:42.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29c41318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f29c409ea50 0x7f29c409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f29c409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29c4136c20 0x7f29c4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f29c4136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c4131cc0 con 0x7f29c41318f0 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c409fbf0 con 0x7f29c4136c20 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c409fe30 con 0x7f29c409ea50 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:42.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c9d5d640 10 monclient(hunting): authenticate will time out at 960.230768s 2024-09-15T03:32:42.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f29c41318f0 0x7f29c4131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38146/0 (socket says 172.21.15.153:38146) 2024-09-15T03:32:42.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29c4136c20 0x7f29c4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:42.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c37fe640 1 -- 172.21.15.153:0/2062991158 learned_addr learned my addr 172.21.15.153:0/2062991158 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 10 monclient(hunting): get_auth_request con 0x7f29c4136c20 auth_method 0 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29bbfff640 1 --2- 172.21.15.153:0/2062991158 >> v2:172.21.15.181:3300/0 conn(0x7f29c409ea50 0x7f29c409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29bbfff640 10 monclient(hunting): get_auth_request con 0x7f29c409ea50 auth_method 0 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.591+0000 7f29c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:42.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:42.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:42.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:42.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 10 monclient(hunting): handle_auth_done global_id 64338 payload 274 2024-09-15T03:32:42.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 1 -- 172.21.15.153:0/2062991158 >> v2:172.21.15.181:3300/0 conn(0x7f29c409ea50 msgr2=0x7f29c409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:42.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 1 --2- 172.21.15.153:0/2062991158 >> v2:172.21.15.181:3300/0 conn(0x7f29c409ea50 0x7f29c409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 1 -- 172.21.15.153:0/2062991158 >> v1:172.21.15.153:6789/0 conn(0x7f29c41318f0 legacy=0x7f29c4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 1 monclient: found mon.noname-a 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.592+0000 7f29c3fff640 1 -- 172.21.15.153:0/2062991158 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40a1210 con 0x7f29c4136c20 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c3fff640 1 -- 172.21.15.153:0/2062991158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4136c20 msgr2=0x7f29c4139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c3fff640 1 --2- 172.21.15.153:0/2062991158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4136c20 0x7f29c4139000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f29c406f600 tx=0x7f29bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd64a69bc08e4ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 1 -- 172.21.15.153:0/2062991158 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc007760 con 0x7f29c4136c20 2024-09-15T03:32:42.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: dump: 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:42.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 1 -- 172.21.15.153:0/2062991158 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc02d050 con 0x7f29c4136c20 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 1 -- 172.21.15.153:0/2062991158 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc007af0 con 0x7f29c4136c20 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c2ffd640 10 monclient: dump: 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:42.607 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c9d5d640 5 monclient: authenticate success, global_id 64338 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c9d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.595+0000 7f29c9d5d640 4 set_mon_vals no callback set 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 10 monclient: shutdown 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 -- 172.21.15.153:0/2062991158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4136c20 msgr2=0x7f29c4139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/2062991158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4136c20 0x7f29c4139000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f29c406f600 tx=0x7f29bc009780 comp rx=0 tx=0).stop 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 -- 172.21.15.153:0/2062991158 shutdown_connections 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/2062991158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c4136c20 0x7f29c4139000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/2062991158 >> v2:172.21.15.181:3300/0 conn(0x7f29c409ea50 0x7f29c409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 -- 172.21.15.153:0/2062991158 >> 172.21.15.153:0/2062991158 conn(0x7f29c4067110 msgr2=0x7f29c4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:42.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 -- 172.21.15.153:0/2062991158 shutdown_connections 2024-09-15T03:32:42.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.596+0000 7f29c9d5d640 1 -- 172.21.15.153:0/2062991158 wait complete. 2024-09-15T03:32:42.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.595+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v98) v1 ==== 58+0+3 (secure 0 0 0) 0x7f1c1001e070 con 0x7f1c2009f240 2024-09-15T03:32:42.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.595+0000 7f1c07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T03:32:42.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.595+0000 7f1c07fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.597+0000 7f29c9d5d640 10 monclient: build_initial_monmap 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.597+0000 7f29c9d5d640 10 monclient: monmap: 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:42.614 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.597+0000 7f29c9d5d640 1 Processor -- start 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.597+0000 7f29c9d5d640 1 -- start start 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.597+0000 7f29c9d5d640 10 monclient: init 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 0x7f29c41307b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f29c409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f29c41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c4136c20 0x7f29c4099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f29c4136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c411b720 con 0x7f29c409ea50 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c411d160 con 0x7f29c4136c20 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c411c5b0 con 0x7f29c41318f0 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c9d5d640 10 monclient(hunting): authenticate will time out at 960.237211s 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 0x7f29c41307b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49448/0 (socket says 172.21.15.153:49448) 2024-09-15T03:32:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 0x7f29c41307b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36568/0 (socket says 172.21.15.153:36568) 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 -- 172.21.15.153:0/3124782394 learned_addr learned my addr 172.21.15.153:0/3124782394 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 10 monclient(hunting): get_auth_request con 0x7f29c41318f0 auth_method 0 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): get_auth_request con 0x7f29c409ea50 auth_method 0 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c4136c20 0x7f29c4099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): get_auth_request con 0x7f29c4136c20 auth_method 0 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 msgr2=0x7f29c40996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 msgr2=0x7f29c40996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29bbfff640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient(hunting): handle_auth_done global_id 64341 payload 824 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 msgr2=0x7f29c40996c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c4136c20 msgr2=0x7f29c4099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c4136c20 0x7f29c4099c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:42.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c37fe640 1 -- 172.21.15.153:0/3124782394 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc009a20 con 0x7f29c409ea50 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.598+0000 7f29c3fff640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c4136c20 0x7f29c4099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.600+0000 7f29c37fe640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 msgr2=0x7f29c41307b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.600+0000 7f29c37fe640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 0x7f29c41307b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f29b400c490 tx=0x7f29b4007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4005cb0 con 0x7f29c409ea50 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: dump: 2024-09-15T03:32:42.620 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:42.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b4005e30 con 0x7f29c409ea50 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 5 monclient: authenticate success, global_id 64341 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 10 monclient: _renew_subs 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c409b290 con 0x7f29c409ea50 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4007320 con 0x7f29c409ea50 2024-09-15T03:32:42.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29bb7fe640 10 monclient: dump: 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:42.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c27fc640 4 set_mon_vals no callback set 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 10 monclient: _renew_subs 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.601+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c411c970 con 0x7f29c409ea50 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f29b401b3b0 con 0x7f29c409ea50 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29c9d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29bb7fe640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a4038740 0x7f29a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29c9d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c404bf60 con 0x7f29c409ea50 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5928+0+0 (secure 0 0 0) 0x7f29b404b3e0 con 0x7f29c409ea50 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29bbfff640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a4038740 0x7f29a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.602+0000 7f29bbfff640 10 monclient: get_auth_request con 0x7f29a4038740 auth_method 0 2024-09-15T03:32:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.604+0000 7f29bbfff640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a4038740 0x7f29a403abf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f29c411bc50 tx=0x7f29ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.606+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f29b4017020 con 0x7f29c409ea50 2024-09-15T03:32:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.607+0000 7f29bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.607+0000 7f29bb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.699+0000 7f1c266a8640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:32:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.699+0000 7f1c266a8640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T03:32:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.700+0000 7f1c266a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:42.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:42.700+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f1c2011d050 con 0x7f1c2009f240 2024-09-15T03:32:42.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.822+0000 7f29c9d5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:42.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.822+0000 7f29c9d5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:42.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.822+0000 7f29c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:42.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.822+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f29c411cda0 con 0x7f29c409ea50 2024-09-15T03:32:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.823+0000 7f29bb7fe640 1 -- 172.21.15.153:0/3124782394 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f29b4018840 con 0x7f29c409ea50 2024-09-15T03:32:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.823+0000 7f29bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:42.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.823+0000 7f29bb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:42.825 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.824+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a4038740 msgr2=0x7f29a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.824+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a4038740 0x7f29a403abf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f29c411bc50 tx=0x7f29ac008040 comp rx=0 tx=0).stop 2024-09-15T03:32:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.824+0000 7f29c9d5d640 10 monclient: shutdown 2024-09-15T03:32:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.824+0000 7f29c9d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:42.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.824+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 msgr2=0x7f29c41307b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.824+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 0x7f29c41307b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f29b400c490 tx=0x7f29b4007f00 comp rx=0 tx=0).stop 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 shutdown_connections 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29a4038740 0x7f29a403abf0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c4136c20 0x7f29c4099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c41318f0 0x7f29c40996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 --2- 172.21.15.153:0/3124782394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c409ea50 0x7f29c41307b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 >> 172.21.15.153:0/3124782394 conn(0x7f29c4067110 msgr2=0x7f29c40a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 shutdown_connections 2024-09-15T03:32:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:42.825+0000 7f29c9d5d640 1 -- 172.21.15.153:0/3124782394 wait complete. 2024-09-15T03:32:43.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.596+0000 7f1c07fff640 1 -- 172.21.15.181:0/2940301162 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v99) v1 ==== 58+0+3 (secure 0 0 0) 0x7f1c10017300 con 0x7f1c2009f240 2024-09-15T03:32:43.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.596+0000 7f1c07fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T03:32:43.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.596+0000 7f1c07fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:43.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c0c038740 msgr2=0x7f1c0c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:43.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c0c038740 0x7f1c0c03abf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1c2006a6b0 tx=0x7f1c14032000 comp rx=0 tx=0).stop 2024-09-15T03:32:43.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 10 monclient: shutdown 2024-09-15T03:32:43.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 msgr2=0x7f1c2012eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c2012eb80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c880 tx=0x7f1c10008040 comp rx=0 tx=0).stop 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 shutdown_connections 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c0c038740 0x7f1c0c03abf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c20132450 0x7f1c20099b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c20131b40 0x7f1c2012f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 --2- 172.21.15.181:0/2940301162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c2009f240 0x7f1c2012eb80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 >> 172.21.15.181:0/2940301162 conn(0x7f1c200a5100 msgr2=0x7f1c20130220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 shutdown_connections 2024-09-15T03:32:43.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.599+0000 7f1c266a8640 1 -- 172.21.15.181:0/2940301162 wait complete. 2024-09-15T03:32:43.611 INFO:tasks.workunit.client.0.smithi181.stderr:+ o3=11 2024-09-15T03:32:43.611 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 11 123 root=default 2024-09-15T03:32:43.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.688+0000 7f741f4c0640 10 monclient: get_monmap_and_config 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.688+0000 7f741f4c0640 10 monclient: build_initial_monmap 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.688+0000 7f741f4c0640 10 monclient: monmap: 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:43.689499+0000 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:43.689499+0000 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:43.691 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:43.691 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:43.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 1 Processor -- start 2024-09-15T03:32:43.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 1 -- start start 2024-09-15T03:32:43.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient: init 2024-09-15T03:32:43.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:43.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f741809f320 0x7f74181321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f741809f320 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7418070920 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7418071430 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418096b60 con 0x7f7418071430 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741809e4d0 con 0x7f7418070920 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741809e7b0 con 0x7f741809f320 2024-09-15T03:32:43.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.689+0000 7f741f4c0640 10 monclient(hunting): authenticate will time out at 968.170998s 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f741809f320 0x7f74181321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f741809f320 0x7f74181321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37646/0 (socket says 172.21.15.181:37646) 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 -- 172.21.15.181:0/2258816787 learned_addr learned my addr 172.21.15.181:0/2258816787 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): get_auth_request con 0x7f741809f320 auth_method 0 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741ca5b640 1 --1- 172.21.15.181:0/2258816787 >> v1:172.21.15.181:6789/0 conn(0x7f7418070920 0x7f7418070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:45772/0 (socket says 172.21.15.181:45772) 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:43.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient(hunting): handle_auth_done global_id 44899 payload 274 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 -- 172.21.15.181:0/2258816787 >> v1:172.21.15.181:6789/0 conn(0x7f7418070920 legacy=0x7f7418070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 -- 172.21.15.181:0/2258816787 >> v1:172.21.15.153:6789/0 conn(0x7f7418071430 legacy=0x7f7418138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 monclient: found mon.noname-c 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.690+0000 7f741d25c640 1 -- 172.21.15.181:0/2258816787 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741809e1f0 con 0x7f741809f320 2024-09-15T03:32:43.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741d25c640 1 -- 172.21.15.181:0/2258816787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f741809f320 msgr2=0x7f74181321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741d25c640 1 --2- 172.21.15.181:0/2258816787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f741809f320 0x7f74181321b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7408009910 tx=0x7f74080091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1faf265dc97073d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 1 -- 172.21.15.181:0/2258816787 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408032440 con 0x7f741809f320 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: dump: 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:43.699 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: _finish_auth 0 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 1 -- 172.21.15.181:0/2258816787 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74080325a0 con 0x7f741809f320 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 5 monclient: authenticate success, global_id 44899 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 1 -- 172.21.15.181:0/2258816787 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408032940 con 0x7f741809f320 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 4 set_mon_vals no callback set 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:43.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f740ffff640 10 monclient: dump: 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 10 monclient: shutdown 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 1 -- 172.21.15.181:0/2258816787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f741809f320 msgr2=0x7f74181321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 1 --2- 172.21.15.181:0/2258816787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f741809f320 0x7f74181321b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7408009910 tx=0x7f74080091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 1 -- 172.21.15.181:0/2258816787 shutdown_connections 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 1 --2- 172.21.15.181:0/2258816787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f741809f320 0x7f74181321b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.693+0000 7f741f4c0640 1 -- 172.21.15.181:0/2258816787 >> 172.21.15.181:0/2258816787 conn(0x7f74180a2b70 msgr2=0x7f741809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.694+0000 7f741f4c0640 1 -- 172.21.15.181:0/2258816787 shutdown_connections 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.694+0000 7f741f4c0640 1 -- 172.21.15.181:0/2258816787 wait complete. 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.694+0000 7f741f4c0640 10 monclient: build_initial_monmap 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.694+0000 7f741f4c0640 10 monclient: monmap: 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:43.703 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:43.703 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:43.703 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:43.703 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:43.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 Processor -- start 2024-09-15T03:32:43.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 -- start start 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient: init 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 0x7f7418136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f7418070920 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 0x7f7418137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f7418071430 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f741809f320 0x7f74181375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f741809f320 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418139b70 con 0x7f741809f320 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:43.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418138a10 con 0x7f7418070920 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741809d5b0 con 0x7f7418071430 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741f4c0640 10 monclient(hunting): authenticate will time out at 968.176897s 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.695+0000 7f741d25c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 0x7f7418136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 0x7f7418136b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37658/0 (socket says 172.21.15.181:37658) 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 -- 172.21.15.181:0/3408169968 learned_addr learned my addr 172.21.15.181:0/3408169968 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): get_auth_request con 0x7f7418070920 auth_method 0 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f741809f320 0x7f74181375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 0x7f7418137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): get_auth_request con 0x7f741809f320 auth_method 0 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 10 monclient(hunting): get_auth_request con 0x7f7418071430 auth_method 0 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741da5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 msgr2=0x7f7418137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 msgr2=0x7f7418137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 0x7f7418137090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741ca5b640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 0x7f7418137090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient(hunting): handle_auth_done global_id 44902 payload 824 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 msgr2=0x7f7418137090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 0x7f7418137090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f741809f320 msgr2=0x7f74181375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f741809f320 0x7f74181375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 monclient: found mon.noname-b 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:43.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.696+0000 7f741d25c640 1 -- 172.21.15.181:0/3408169968 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408009450 con 0x7f7418070920 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741d25c640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 msgr2=0x7f7418136b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741d25c640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 0x7f7418136b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7408009970 tx=0x7f7408032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408031c70 con 0x7f7418070920 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: dump: 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7408031dd0 con 0x7f7418070920 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 5 monclient: authenticate success, global_id 44902 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 10 monclient: _renew_subs 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74181390d0 con 0x7f7418070920 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740803c650 con 0x7f7418070920 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:43.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740d7fa640 10 monclient: dump: 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:43.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 10 monclient: _renew_subs 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f740f7fe640 4 set_mon_vals no callback set 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.698+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7418138500 con 0x7f7418070920 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7408044400 con 0x7f7418070920 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f741f4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f741f4c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f741f4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74180008d0 con 0x7f7418070920 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f740d7fa640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f73fc038770 0x7f73fc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6043+0+0 (secure 0 0 0) 0x7f740806c220 con 0x7f7418070920 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f741ca5b640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f73fc038770 0x7f73fc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.699+0000 7f741ca5b640 10 monclient: get_auth_request con 0x7f73fc038770 auth_method 0 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.702+0000 7f741ca5b640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f73fc038770 0x7f73fc03ac20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f74180633c0 tx=0x7f7414009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.704+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f74080382b0 con 0x7f7418070920 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.704+0000 7f740d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.704+0000 7f740d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:43.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.863+0000 7f741f4c0640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T03:32:43.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.863+0000 7f741f4c0640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T03:32:43.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.863+0000 7f741f4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:43.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:43.863+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f74180008d0 con 0x7f7418070920 2024-09-15T03:32:45.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.942+0000 7fe868cdd640 10 monclient: get_monmap_and_config 2024-09-15T03:32:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.942+0000 7fe868cdd640 10 monclient: build_initial_monmap 2024-09-15T03:32:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.942+0000 7fe868cdd640 10 monclient: monmap: 2024-09-15T03:32:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:45.943 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:45.943184+0000 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:45.943184+0000 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:45.944 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.942+0000 7fe868cdd640 1 Processor -- start 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 1 -- start start 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient: init 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): picked mon.noname-a con 0x7fe8640981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe864131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): picked mon.noname-c con 0x7fe864132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe864098ad0 con 0x7fe864131da0 2024-09-15T03:32:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe864131bd0 con 0x7fe8640981c0 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86409ca60 con 0x7fe864132170 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe868cdd640 10 monclient(hunting): authenticate will time out at 963.582773s 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe86259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe861d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe864131da0 0x7fe864067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38242/0 (socket says 172.21.15.153:38242) 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe861d9b640 1 -- 172.21.15.153:0/3750118492 learned_addr learned my addr 172.21.15.153:0/3750118492 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe86259c640 10 monclient(hunting): get_auth_request con 0x7fe8640981c0 auth_method 0 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:45.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe86259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.943+0000 7fe86259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 10 monclient(hunting): get_auth_request con 0x7fe864132170 auth_method 0 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:45.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.944+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:45.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:45.946+0000 7fe861d9b640 1 -- 172.21.15.153:0/3750118492 >> v1:172.21.15.153:6789/0 conn(0x7fe864131da0 legacy=0x7fe864067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 10 monclient(hunting): get_auth_request con 0x7fe8640981c0 auth_method 0 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:46.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 10 monclient(hunting): get_auth_request con 0x7fe864132170 auth_method 0 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.145+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:46.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 10 monclient(hunting): get_auth_request con 0x7fe8640981c0 auth_method 0 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:46.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 10 monclient(hunting): get_auth_request con 0x7fe864132170 auth_method 0 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:46.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:46.546+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:32:46.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:46.696+0000 7f7403fff640 10 monclient: tick 2024-09-15T03:32:46.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:46.696+0000 7f7403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:46.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:46.696+0000 7f7403fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:47.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:47.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 10 monclient(hunting): get_auth_request con 0x7fe8640981c0 auth_method 0 2024-09-15T03:32:47.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:47.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:47.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 10 monclient(hunting): get_auth_request con 0x7fe864132170 auth_method 0 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:47.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:47.347+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:32:47.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:47.696+0000 7f7403fff640 10 monclient: tick 2024-09-15T03:32:47.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:47.696+0000 7f7403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:48.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:48.696+0000 7f7403fff640 10 monclient: tick 2024-09-15T03:32:48.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:48.696+0000 7f7403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:48.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient(hunting): tick 2024-09-15T03:32:48.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:32:48.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:32:48.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 msgr2=0x7fe864067820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.181:3300/0 conn(0x7fe864132170 0x7fe864067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 -- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 msgr2=0x7fe864098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 --2- 172.21.15.153:0/3750118492 >> v2:172.21.15.153:3300/0 conn(0x7fe8640981c0 0x7fe864098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 -- 172.21.15.153:0/3750118492 >> v1:172.21.15.153:6789/0 conn(0x7fe864131da0 legacy=0x7fe864067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient: _add_conns ranks=[1,2] 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8440036d0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe844007390 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 -- 172.21.15.153:0/3750118492 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86409cca0 con 0x7fe8440036d0 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 1 -- 172.21.15.153:0/3750118492 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe864131bd0 con 0x7fe844007390 2024-09-15T03:32:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.943+0000 7fe853fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 10 monclient(hunting): get_auth_request con 0x7fe844007390 auth_method 0 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 msgr2=0x7fe844009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 msgr2=0x7fe844009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.944+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:48.946+0000 7fe86259c640 1 -- 172.21.15.153:0/3750118492 >> v1:172.21.15.181:6789/0 conn(0x7fe8440036d0 legacy=0x7fe844005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 10 monclient(hunting): get_auth_request con 0x7fe844007390 auth_method 0 2024-09-15T03:32:49.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:49.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 msgr2=0x7fe844009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:32:49.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 msgr2=0x7fe844009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:49.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:49.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.145+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874724083 0 0) 0x7fe86409cca0 con 0x7fe8440036d0 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 10 monclient(hunting): my global_id is 44905 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84c0034e0 con 0x7fe8440036d0 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86259c640 1 --1- 172.21.15.153:0/3750118492 >> v1:172.21.15.181:6789/0 conn(0x7fe8440036d0 0x7fe844005b20 :-1 s=OPENED pgs=351 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fe84c0034e0 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:32:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568692102 0 0) 0x7fe84c0034e0 con 0x7fe8440036d0 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 msgr2=0x7fe844009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe844007390 0x7fe844009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84400b070 con 0x7fe8440036d0 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe86159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe868cdd640 5 monclient: authenticate success, global_id 44905 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.281+0000 7fe868cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:49.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fe85400d480 con 0x7fe8440036d0 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient: dump: 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:49.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 >> v1:172.21.15.181:6789/0 conn(0x7fe8440036d0 legacy=0x7fe844005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 reap_dead start 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c0039d0 0x7fe84c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): picked mon.c con 0x7fe84c0039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c004780 0x7fe84c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): picked mon.b con 0x7fe84c004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 0x7fe84c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): picked mon.a con 0x7fe84c005250 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864131bd0 con 0x7fe84c005250 2024-09-15T03:32:49.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c00c1a0 con 0x7fe84c004780 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c00c360 con 0x7fe84c0039d0 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.1 v1:172.21.15.181:6789/0 5 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fe85400d960 con 0x7fe8440036d0 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe868cdd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86159a640 10 monclient(hunting): discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c0039d0 0x7fe84c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 0x7fe84c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): get_auth_request con 0x7fe84c0039d0 auth_method 0 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): get_auth_request con 0x7fe84c005250 auth_method 0 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c004780 0x7fe84c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): get_auth_request con 0x7fe84c004780 auth_method 0 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe86259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe861d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient(hunting): handle_auth_done global_id 44905 payload 293 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c0039d0 msgr2=0x7fe84c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c0039d0 0x7fe84c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c004780 msgr2=0x7fe84c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:49.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c004780 0x7fe84c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 monclient: found mon.a 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0034e0 con 0x7fe84c005250 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.282+0000 7fe862d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe862d9d640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 msgr2=0x7fe84c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe862d9d640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 0x7fe84c00b9d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe864132590 tx=0x7fe85c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f066d5921d284fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c031600 con 0x7fe84c005250 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 10 monclient: dump: 2024-09-15T03:32:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85c00f040 con 0x7fe84c005250 2024-09-15T03:32:49.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe868cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe868cdd640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe868cdd640 4 set_mon_vals no callback set 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 1 -- 172.21.15.153:0/3750118492 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c0319b0 con 0x7fe84c005250 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.285+0000 7fe86159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe86159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe86159a640 10 monclient: dump: 2024-09-15T03:32:49.301 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 10 monclient: shutdown 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:49.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 -- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 msgr2=0x7fe84c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 0x7fe84c00b9d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe864132590 tx=0x7fe85c00b040 comp rx=0 tx=0).stop 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 -- 172.21.15.153:0/3750118492 shutdown_connections 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c005250 0x7fe84c00b9d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c004780 0x7fe84c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 --2- 172.21.15.153:0/3750118492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c0039d0 0x7fe84c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.286+0000 7fe868cdd640 1 -- 172.21.15.153:0/3750118492 >> 172.21.15.153:0/3750118492 conn(0x7fe8640a1e30 msgr2=0x7fe8640a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.287+0000 7fe868cdd640 1 -- 172.21.15.153:0/3750118492 shutdown_connections 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.287+0000 7fe868cdd640 1 -- 172.21.15.153:0/3750118492 wait complete. 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.287+0000 7fe868cdd640 10 monclient: build_initial_monmap 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.287+0000 7fe868cdd640 10 monclient: monmap: 2024-09-15T03:32:49.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.287+0000 7fe868cdd640 1 Processor -- start 2024-09-15T03:32:49.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 -- start start 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient: init 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 0x7fe864140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): picked mon.noname-a con 0x7fe84c0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): picked mon.noname-c con 0x7fe84c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c005250 0x7fe86411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): picked mon.noname-b con 0x7fe84c005250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864098cb0 con 0x7fe84c0039d0 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86409b550 con 0x7fe84c005250 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86411de30 con 0x7fe84c004780 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe868cdd640 10 monclient(hunting): authenticate will time out at 966.927801s 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 0x7fe864140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe861d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 0x7fe864140b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53622/0 (socket says 172.21.15.153:53622) 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe861d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57996/0 (socket says 172.21.15.153:57996) 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 1 -- 172.21.15.153:0/274823976 learned_addr learned my addr 172.21.15.153:0/274823976 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 10 monclient(hunting): get_auth_request con 0x7fe84c0039d0 auth_method 0 2024-09-15T03:32:49.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe86259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe861d9b640 10 monclient(hunting): get_auth_request con 0x7fe84c004780 auth_method 0 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe861d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe861d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.288+0000 7fe861d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c005250 0x7fe86411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): get_auth_request con 0x7fe84c005250 auth_method 0 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe861d9b640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 msgr2=0x7fe864130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe861d9b640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 msgr2=0x7fe864130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe861d9b640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe861d9b640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 10 monclient(hunting): handle_auth_done global_id 64347 payload 824 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 msgr2=0x7fe864130a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:49.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c005250 msgr2=0x7fe86411a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c005250 0x7fe86411a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 1 monclient: found mon.noname-a 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe86259c640 1 -- 172.21.15.153:0/274823976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86411c020 con 0x7fe84c0039d0 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.289+0000 7fe862d9d640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c005250 0x7fe86411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe86259c640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 msgr2=0x7fe864140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe86259c640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 0x7fe864140b90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe85400d310 tx=0x7fe854011450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854013ea0 con 0x7fe84c0039d0 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: dump: 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.311 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:49.311 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.311 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.311 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:49.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe854016880 con 0x7fe84c0039d0 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe868cdd640 5 monclient: authenticate success, global_id 64347 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe868cdd640 10 monclient: _renew_subs 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe868cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86411dbb0 con 0x7fe84c0039d0 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854016c40 con 0x7fe84c0039d0 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe8527fc640 10 monclient: dump: 2024-09-15T03:32:49.312 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.292+0000 7fe868cdd640 10 monclient: _renew_subs 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.292+0000 7fe868cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.292+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8641410d0 con 0x7fe84c0039d0 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.291+0000 7fe860d99640 4 set_mon_vals no callback set 2024-09-15T03:32:49.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.292+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe8540153c0 con 0x7fe84c0039d0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe868cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe868cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe868cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe864141a50 con 0x7fe84c0039d0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe8527fc640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe844039070 0x7fe84403b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6043+0+0 (secure 0 0 0) 0x7fe85404d2d0 con 0x7fe84c0039d0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe861d9b640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe844039070 0x7fe84403b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.293+0000 7fe861d9b640 10 monclient: get_auth_request con 0x7fe844039070 auth_method 0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.296+0000 7fe861d9b640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe844039070 0x7fe84403b520 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe858000c00 tx=0x7fe858009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.297+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe854026900 con 0x7fe84c0039d0 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.297+0000 7fe8527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.297+0000 7fe8527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:49.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.344+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v100) v1 ==== 193+0+0 (secure 0 0 0) 0x7f7408040070 con 0x7f7418070920 2024-09-15T03:32:49.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.344+0000 7f740d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T03:32:49.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.344+0000 7f740d7fa640 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-15T03:32:49.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.445+0000 7f741f4c0640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T03:32:49.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.445+0000 7f741f4c0640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T03:32:49.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.445+0000 7f741f4c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.445+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f74180008d0 con 0x7f7418070920 2024-09-15T03:32:49.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.446+0000 7f740d7fa640 1 -- 172.21.15.181:0/3408169968 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v100) v1 ==== 191+0+0 (secure 0 0 0) 0x7f7408043360 con 0x7f7418070920 2024-09-15T03:32:49.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.446+0000 7f740d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-15T03:32:49.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.446+0000 7f740d7fa640 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-15T03:32:49.448 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f73fc038770 msgr2=0x7f73fc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f73fc038770 0x7f73fc03ac20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f74180633c0 tx=0x7f7414009040 comp rx=0 tx=0).stop 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 10 monclient: shutdown 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 msgr2=0x7f7418136b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 0x7f7418136b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7408009970 tx=0x7f7408032000 comp rx=0 tx=0).stop 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 shutdown_connections 2024-09-15T03:32:49.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f73fc038770 0x7f73fc03ac20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f741809f320 0x7f74181375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7418071430 0x7f7418137090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 --2- 172.21.15.181:0/3408169968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7418070920 0x7f7418136b50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 >> 172.21.15.181:0/3408169968 conn(0x7f74180a2b70 msgr2=0x7f7418131480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 shutdown_connections 2024-09-15T03:32:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.448+0000 7f741f4c0640 1 -- 172.21.15.181:0/3408169968 wait complete. 2024-09-15T03:32:49.459 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:49.459 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep osd.11 2024-09-15T03:32:49.459 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 123 2024-09-15T03:32:49.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe868cdd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:49.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe868cdd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe868cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe86404bf60 con 0x7fe84c0039d0 2024-09-15T03:32:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe8527fc640 1 -- 172.21.15.153:0/274823976 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fe854024030 con 0x7fe84c0039d0 2024-09-15T03:32:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe8527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.518+0000 7fe8527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:49.521 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe844039070 msgr2=0x7fe84403b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe844039070 0x7fe84403b520 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe858000c00 tx=0x7fe858009040 comp rx=0 tx=0).stop 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 10 monclient: shutdown 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 msgr2=0x7fe864140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 0x7fe864140b90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe85400d310 tx=0x7fe854011450 comp rx=0 tx=0).stop 2024-09-15T03:32:49.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 shutdown_connections 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe844039070 0x7fe84403b520 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe84c005250 0x7fe86411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe84c004780 0x7fe864130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 --2- 172.21.15.153:0/274823976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe84c0039d0 0x7fe864140b90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 >> 172.21.15.153:0/274823976 conn(0x7fe8640a1e30 msgr2=0x7fe864096e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 shutdown_connections 2024-09-15T03:32:49.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:49.520+0000 7fe868cdd640 1 -- 172.21.15.153:0/274823976 wait complete. 2024-09-15T03:32:49.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.531+0000 7fa8b9c0f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:49.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.531+0000 7fa8b9c0f640 10 monclient: build_initial_monmap 2024-09-15T03:32:49.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.531+0000 7fa8b9c0f640 10 monclient: monmap: 2024-09-15T03:32:49.532 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:49.532 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.532 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:49.531867+0000 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:49.531867+0000 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:49.533 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.531+0000 7fa8b9c0f640 1 Processor -- start 2024-09-15T03:32:49.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 1 -- start start 2024-09-15T03:32:49.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient: init 2024-09-15T03:32:49.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:49.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8b4131b40 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa8b4131f10 0x7fa8b4132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b4131f10 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa8b4136e70 0x7fa8b4139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b4136e70 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b406a230 con 0x7fa8b4131b40 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40a0300 con 0x7fa8b4136e70 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40a00c0 con 0x7fa8b4131f10 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:49.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b9c0f640 10 monclient(hunting): authenticate will time out at 974.013551s 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa8b4131f10 0x7fa8b4132330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa8b4131f10 0x7fa8b4132330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44456/0 (socket says 172.21.15.181:44456) 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 -- 172.21.15.181:0/791361041 learned_addr learned my addr 172.21.15.181:0/791361041 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): get_auth_request con 0x7fa8b4131f10 auth_method 0 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b3fff640 1 --2- 172.21.15.181:0/791361041 >> v2:172.21.15.153:3300/0 conn(0x7fa8b4136e70 0x7fa8b4139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient(hunting): handle_auth_done global_id 44911 payload 274 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 -- 172.21.15.181:0/791361041 >> v2:172.21.15.153:3300/0 conn(0x7fa8b4136e70 msgr2=0x7fa8b4139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 --2- 172.21.15.181:0/791361041 >> v2:172.21.15.153:3300/0 conn(0x7fa8b4136e70 0x7fa8b4139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 -- 172.21.15.181:0/791361041 >> v1:172.21.15.153:6789/0 conn(0x7fa8b4131b40 legacy=0x7fa8b409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 monclient: found mon.noname-c 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8abfff640 1 -- 172.21.15.181:0/791361041 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b409bee0 con 0x7fa8b4131f10 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b3fff640 10 monclient: get_auth_request con 0x7fa8b4136e70 auth_method 0 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.532+0000 7fa8b3fff640 1 --2- 172.21.15.181:0/791361041 >> v2:172.21.15.153:3300/0 conn(0x7fa8b4136e70 0x7fa8b4139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.535+0000 7fa8abfff640 1 -- 172.21.15.181:0/791361041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 msgr2=0x7fa8b4132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.535+0000 7fa8abfff640 1 --2- 172.21.15.181:0/791361041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b4132330 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa8b409cbe0 tx=0x7fa89c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a960847f59da6ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.535+0000 7fa8b2ffd640 1 -- 172.21.15.181:0/791361041 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c032440 con 0x7fa8b4131f10 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.535+0000 7fa8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: dump: 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 1 -- 172.21.15.181:0/791361041 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c0325a0 con 0x7fa8b4131f10 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 5 monclient: authenticate success, global_id 44911 2024-09-15T03:32:49.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 1 -- 172.21.15.181:0/791361041 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c032940 con 0x7fa8b4131f10 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b2ffd640 10 monclient: dump: 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 4 set_mon_vals no callback set 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 10 monclient: shutdown 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/791361041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 msgr2=0x7fa8b4132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/791361041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b4132330 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa8b409cbe0 tx=0x7fa89c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/791361041 shutdown_connections 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/791361041 >> v2:172.21.15.153:3300/0 conn(0x7fa8b4136e70 0x7fa8b4139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/791361041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b4132330 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/791361041 >> 172.21.15.181:0/791361041 conn(0x7fa8b40a5100 msgr2=0x7fa8b40a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:49.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/791361041 shutdown_connections 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.536+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/791361041 wait complete. 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.537+0000 7fa8b9c0f640 10 monclient: build_initial_monmap 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.537+0000 7fa8b9c0f640 10 monclient: monmap: 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:49.547 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:49.547 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:49.547 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.537+0000 7fa8b9c0f640 1 Processor -- start 2024-09-15T03:32:49.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.537+0000 7fa8b9c0f640 1 -- start start 2024-09-15T03:32:49.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.537+0000 7fa8b9c0f640 10 monclient: init 2024-09-15T03:32:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4131b40 0x7fa8b412f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b4131b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:49.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b412f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b4131f10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 0x7fa8b412fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b4136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b409adf0 con 0x7fa8b4131b40 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4131010 con 0x7fa8b4131f10 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b409b0d0 con 0x7fa8b4136e70 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b9c0f640 10 monclient(hunting): authenticate will time out at 974.019158s 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b412f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b412f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44468/0 (socket says 172.21.15.181:44468) 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 -- 172.21.15.181:0/3868366832 learned_addr learned my addr 172.21.15.181:0/3868366832 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): get_auth_request con 0x7fa8b4131f10 auth_method 0 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4131b40 0x7fa8b412f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 0x7fa8b412fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): get_auth_request con 0x7fa8b4131b40 auth_method 0 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 10 monclient(hunting): get_auth_request con 0x7fa8b4136e70 auth_method 0 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 msgr2=0x7fa8b412fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 msgr2=0x7fa8b412fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 0x7fa8b412fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient(hunting): handle_auth_done global_id 44914 payload 824 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8b3fff640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 0x7fa8b412fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 msgr2=0x7fa8b412fb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 0x7fa8b412fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4131b40 msgr2=0x7fa8b412f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4131b40 0x7fa8b412f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 monclient: found mon.noname-b 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.538+0000 7fa8abfff640 1 -- 172.21.15.181:0/3868366832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c009450 con 0x7fa8b4131f10 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8abfff640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 msgr2=0x7fa8b412f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8abfff640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b412f5d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa89c035040 tx=0x7fa89c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c000c40 con 0x7fa8b4131f10 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: dump: 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.558 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c031c70 con 0x7fa8b4131f10 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 5 monclient: authenticate success, global_id 44914 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c03b400 con 0x7fa8b4131f10 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 10 monclient: _renew_subs 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b409a900 con 0x7fa8b4131f10 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8ab7fe640 10 monclient: dump: 2024-09-15T03:32:49.559 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b27fc640 4 set_mon_vals no callback set 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 10 monclient: _renew_subs 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.540+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b411c150 con 0x7fa8b4131f10 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa89c039040 con 0x7fa8b4131f10 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8b9c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8b9c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8b9c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b409a450 con 0x7fa8b4131f10 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8ab7fe640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa894038740 0x7fa89403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.541+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5978+0+0 (secure 0 0 0) 0x7fa89c06c460 con 0x7fa8b4131f10 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.542+0000 7fa8b37fe640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa894038740 0x7fa89403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.542+0000 7fa8b37fe640 10 monclient: get_auth_request con 0x7fa894038740 auth_method 0 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.544+0000 7fa8b37fe640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa894038740 0x7fa89403abf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4007fd0 tx=0x7fa8a4007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.546+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa89c03b870 con 0x7fa8b4131f10 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.546+0000 7fa8ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.546+0000 7fa8ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:49.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8b9c0f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:49.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8b9c0f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:49.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8b9c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa8b40636e0 con 0x7fa8b4131f10 2024-09-15T03:32:49.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v100) v1 ==== 56+0+1173 (secure 0 0 0) 0x7fa89c037070 con 0x7fa8b4131f10 2024-09-15T03:32:49.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:49.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.703+0000 7fa8ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.795+0000 7fa8b9c0f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.795+0000 7fa8b9c0f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.795+0000 7fa8b9c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.795+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa8b40636e0 con 0x7fa8b4131f10 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.796+0000 7fa8ab7fe640 1 -- 172.21.15.181:0/3868366832 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v100) v1 ==== 56+0+1173 (secure 0 0 0) 0x7fa89c049070 con 0x7fa8b4131f10 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.796+0000 7fa8ab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:49.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.796+0000 7fa8ab7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:49.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa894038740 msgr2=0x7fa89403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa894038740 0x7fa89403abf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4007fd0 tx=0x7fa8a4007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:49.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 10 monclient: shutdown 2024-09-15T03:32:49.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:49.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 msgr2=0x7fa8b412f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b412f5d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa89c035040 tx=0x7fa89c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 shutdown_connections 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa894038740 0x7fa89403abf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8b4136e70 0x7fa8b412fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8b4131f10 0x7fa8b412f5d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 --2- 172.21.15.181:0/3868366832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8b4131b40 0x7fa8b412f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 >> 172.21.15.181:0/3868366832 conn(0x7fa8b40a5100 msgr2=0x7fa8b40a4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 shutdown_connections 2024-09-15T03:32:49.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.798+0000 7fa8b9c0f640 1 -- 172.21.15.181:0/3868366832 wait complete. 2024-09-15T03:32:49.808 INFO:tasks.workunit.client.0.smithi181.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-15T03:32:49.808 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush reweight osd.11 113 2024-09-15T03:32:49.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.881+0000 7f06ee8af640 10 monclient: get_monmap_and_config 2024-09-15T03:32:49.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.881+0000 7f06ee8af640 10 monclient: build_initial_monmap 2024-09-15T03:32:49.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.881+0000 7f06ee8af640 10 monclient: monmap: 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:49.882357+0000 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:49.882357+0000 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:49.883 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 Processor -- start 2024-09-15T03:32:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 -- start start 2024-09-15T03:32:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient: init 2024-09-15T03:32:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06e80a0ad0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f06e809ee60 0x7f06e809f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): picked mon.noname-a con 0x7f06e809ee60 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 0x7f06e81391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): picked mon.noname-b con 0x7f06e8136e00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e806a5b0 con 0x7f06e80a0ad0 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e809f7a0 con 0x7f06e809ee60 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e80a0ea0 con 0x7f06e8136e00 2024-09-15T03:32:49.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06ee8af640 10 monclient(hunting): authenticate will time out at 974.363812s 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06e7fff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f06e80a0ad0 0x7f06e80691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34662/0 (socket says 172.21.15.181:34662) 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06e7fff640 1 -- 172.21.15.181:0/1036530821 learned_addr learned my addr 172.21.15.181:0/1036530821 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.882+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1036530821 >> v2:172.21.15.153:3300/0 conn(0x7f06e809ee60 0x7f06e809f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): get_auth_request con 0x7f06e809ee60 auth_method 0 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 0x7f06e81391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 10 monclient(hunting): get_auth_request con 0x7f06e8136e00 auth_method 0 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 1 -- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 msgr2=0x7f06e81391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:49.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 1 -- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 msgr2=0x7f06e81391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 0x7f06e81391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06ece4c640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 0x7f06e81391e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient(hunting): handle_auth_done global_id 64353 payload 274 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 1 -- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 msgr2=0x7f06e81391e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 0x7f06e81391e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 1 -- 172.21.15.181:0/1036530821 >> v1:172.21.15.181:6789/0 conn(0x7f06e80a0ad0 legacy=0x7f06e80691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.883+0000 7f06e77fe640 1 -- 172.21.15.181:0/1036530821 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e80a0460 con 0x7f06e809ee60 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.885+0000 7f06e77fe640 1 -- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e809ee60 msgr2=0x7f06e809f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.885+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e809ee60 0x7f06e809f260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009850 tx=0x7f06dc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f2674916eff628 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 1 -- 172.21.15.181:0/1036530821 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc02ae80 con 0x7f06e809ee60 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: dump: 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 1 -- 172.21.15.181:0/1036530821 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc032440 con 0x7f06e809ee60 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 5 monclient: authenticate success, global_id 64353 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 4 set_mon_vals no callback set 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 1 -- 172.21.15.181:0/1036530821 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc032800 con 0x7f06e809ee60 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06e6ffd640 10 monclient: dump: 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.900 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 10 monclient: shutdown 2024-09-15T03:32:49.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:49.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 -- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e809ee60 msgr2=0x7f06e809f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e809ee60 0x7f06e809f260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009850 tx=0x7f06dc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 -- 172.21.15.181:0/1036530821 shutdown_connections 2024-09-15T03:32:49.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e8136e00 0x7f06e81391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1036530821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e809ee60 0x7f06e809f260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 -- 172.21.15.181:0/1036530821 >> 172.21.15.181:0/1036530821 conn(0x7f06e8071370 msgr2=0x7f06e80a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 -- 172.21.15.181:0/1036530821 shutdown_connections 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.886+0000 7f06ee8af640 1 -- 172.21.15.181:0/1036530821 wait complete. 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.887+0000 7f06ee8af640 10 monclient: build_initial_monmap 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.887+0000 7f06ee8af640 10 monclient: monmap: 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:49.903 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.887+0000 7f06ee8af640 1 Processor -- start 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.887+0000 7f06ee8af640 1 -- start start 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.887+0000 7f06ee8af640 10 monclient: init 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 0x7f06e8135c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): picked mon.noname-b con 0x7f06e809ee60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 0x7f06e811e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): picked mon.noname-c con 0x7f06e80a0ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8136e00 0x7f06e811eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): picked mon.noname-a con 0x7f06e8136e00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e8121d70 con 0x7f06e8136e00 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e81223c0 con 0x7f06e809ee60 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e8120830 con 0x7f06e80a0ad0 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ee8af640 10 monclient(hunting): authenticate will time out at 974.369114s 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 0x7f06e8135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 0x7f06e8135c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44484/0 (socket says 172.21.15.181:44484) 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 -- 172.21.15.181:0/1358728977 learned_addr learned my addr 172.21.15.181:0/1358728977 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): get_auth_request con 0x7f06e809ee60 auth_method 0 2024-09-15T03:32:49.905 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8136e00 0x7f06e811eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 0x7f06e811e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 10 monclient(hunting): get_auth_request con 0x7f06e80a0ad0 auth_method 0 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): get_auth_request con 0x7f06e8136e00 auth_method 0 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 msgr2=0x7f06e811e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 msgr2=0x7f06e811e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 0x7f06e811e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 0x7f06e811e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06ece4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:49.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient(hunting): handle_auth_done global_id 44917 payload 824 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 msgr2=0x7f06e811e760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 0x7f06e811e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8136e00 msgr2=0x7f06e811eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8136e00 0x7f06e811eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 monclient: found mon.noname-b 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.888+0000 7f06e7fff640 1 -- 172.21.15.181:0/1358728977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc009390 con 0x7f06e809ee60 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06e7fff640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 msgr2=0x7f06e8135c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06e7fff640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 0x7f06e8135c60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f06d800c640 tx=0x7f06d8004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8005cb0 con 0x7f06e809ee60 2024-09-15T03:32:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: dump: 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: _finish_auth 0 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d8005e30 con 0x7f06e809ee60 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06ee8af640 5 monclient: authenticate success, global_id 44917 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06ee8af640 10 monclient: _renew_subs 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8007340 con 0x7f06e809ee60 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06ee8af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e809e810 con 0x7f06e809ee60 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06cffff640 10 monclient: dump: 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06e67fc640 4 set_mon_vals no callback set 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.890+0000 7f06ee8af640 10 monclient: _renew_subs 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06ee8af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e8120e80 con 0x7f06e809ee60 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f06d801b400 con 0x7f06e809ee60 2024-09-15T03:32:49.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06ee8af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06ee8af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06ee8af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.891+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e80008d0 con 0x7f06e809ee60 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.892+0000 7f06cffff640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f06d4038740 0x7f06d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.892+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5978+0+0 (secure 0 0 0) 0x7f06d804bb00 con 0x7f06e809ee60 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.892+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f06d4038740 0x7f06d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:49.912 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.892+0000 7f06e77fe640 10 monclient: get_auth_request con 0x7f06d4038740 auth_method 0 2024-09-15T03:32:49.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.894+0000 7f06e77fe640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f06d4038740 0x7f06d403abf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f06dc034000 tx=0x7f06dc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:49.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.896+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f06d8007730 con 0x7f06e809ee60 2024-09-15T03:32:49.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.896+0000 7f06cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:49.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:49.896+0000 7f06cffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:50.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.054+0000 7f06ee8af640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T03:32:50.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.054+0000 7f06ee8af640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T03:32:50.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.054+0000 7f06ee8af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:50.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.054+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7f06e8121a60 con 0x7f06e809ee60 2024-09-15T03:32:50.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.348+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v101) v1 ==== 156+0+0 (secure 0 0 0) 0x7f06d804be60 con 0x7f06e809ee60 2024-09-15T03:32:50.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.348+0000 7f06cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T03:32:50.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.348+0000 7f06cffff640 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-15T03:32:50.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.445+0000 7f06ee8af640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T03:32:50.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.445+0000 7f06ee8af640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T03:32:50.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.445+0000 7f06ee8af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:50.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:50.445+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7f06e8072640 con 0x7f06e809ee60 2024-09-15T03:32:51.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.351+0000 7f06cffff640 1 -- 172.21.15.181:0/1358728977 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v102) v1 ==== 156+0+0 (secure 0 0 0) 0x7f06d801a360 con 0x7f06e809ee60 2024-09-15T03:32:51.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.351+0000 7f06cffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-15T03:32:51.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.351+0000 7f06cffff640 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-15T03:32:51.353 INFO:tasks.workunit.client.0.smithi181.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.354+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f06d4038740 msgr2=0x7f06d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f06d4038740 0x7f06d403abf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f06dc034000 tx=0x7f06dc032000 comp rx=0 tx=0).stop 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 10 monclient: shutdown 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 msgr2=0x7f06e8135c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 0x7f06e8135c60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f06d800c640 tx=0x7f06d8004960 comp rx=0 tx=0).stop 2024-09-15T03:32:51.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 shutdown_connections 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f06d4038740 0x7f06d403abf0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8136e00 0x7f06e811eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a0ad0 0x7f06e811e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 --2- 172.21.15.181:0/1358728977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06e809ee60 0x7f06e8135c60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 >> 172.21.15.181:0/1358728977 conn(0x7f06e8071370 msgr2=0x7f06e809ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 shutdown_connections 2024-09-15T03:32:51.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.355+0000 7f06ee8af640 1 -- 172.21.15.181:0/1358728977 wait complete. 2024-09-15T03:32:51.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-15T03:32:51.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:32:51.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.445+0000 7fd03115d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.445+0000 7fd03115d640 10 monclient: build_initial_monmap 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.445+0000 7fd03115d640 10 monclient: monmap: 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:51.446753+0000 2024-09-15T03:32:51.447 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:51.446753+0000 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:51.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.446+0000 7fd03115d640 1 Processor -- start 2024-09-15T03:32:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.446+0000 7fd03115d640 1 -- start start 2024-09-15T03:32:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.446+0000 7fd03115d640 10 monclient: init 2024-09-15T03:32:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd02c0a0ad0 0x7fd02c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): picked mon.noname-a con 0x7fd02c0a0ad0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd02c09ee60 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): picked mon.noname-b con 0x7fd02c09f260 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c06a5b0 con 0x7fd02c09ee60 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c0a0ea0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0a1bb0 con 0x7fd02c09f260 2024-09-15T03:32:51.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd03115d640 10 monclient(hunting): authenticate will time out at 975.928113s 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd02c0a0ad0 0x7fd02c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02a59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd02c09ee60 0x7fd02c131ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:43276/0 (socket says 172.21.15.181:43276) 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd02c0a0ad0 0x7fd02c0691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:41730/0 (socket says 172.21.15.181:41730) 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02a59c640 1 -- 172.21.15.181:0/2444102464 learned_addr learned my addr 172.21.15.181:0/2444102464 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 10 monclient(hunting): get_auth_request con 0x7fd02c09f260 auth_method 0 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): get_auth_request con 0x7fd02c0a0ad0 auth_method 0 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 1 -- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 msgr2=0x7fd02c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:51.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 1 -- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 msgr2=0x7fd02c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c1321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02b59e640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c1321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_done global_id 64359 payload 274 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 -- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 msgr2=0x7fd02c1321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c1321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 -- 172.21.15.181:0/2444102464 >> v1:172.21.15.153:6789/0 conn(0x7fd02c09ee60 legacy=0x7fd02c131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 monclient: found mon.noname-a 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.447+0000 7fd02ad9d640 1 -- 172.21.15.181:0/2444102464 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0a0460 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.450+0000 7fd02ad9d640 1 -- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c0a0ad0 msgr2=0x7fd02c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:51.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.450+0000 7fd02ad9d640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c0691d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd024009850 tx=0x7fd0240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882bc589c2becd77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 1 -- 172.21.15.181:0/2444102464 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02402adf0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: dump: 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.463 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 5 monclient: authenticate success, global_id 64359 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 1 -- 172.21.15.181:0/2444102464 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd024032440 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 1 -- 172.21.15.181:0/2444102464 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0240327e0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 4 set_mon_vals no callback set 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd029d9b640 10 monclient: dump: 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.465 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.466 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 10 monclient: shutdown 2024-09-15T03:32:51.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:51.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 -- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c0a0ad0 msgr2=0x7fd02c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c0691d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd024009850 tx=0x7fd0240090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 -- 172.21.15.181:0/2444102464 shutdown_connections 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 --2- 172.21.15.181:0/2444102464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c0691d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 -- 172.21.15.181:0/2444102464 >> 172.21.15.181:0/2444102464 conn(0x7fd02c071370 msgr2=0x7fd02c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.451+0000 7fd03115d640 1 -- 172.21.15.181:0/2444102464 shutdown_connections 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.452+0000 7fd03115d640 1 -- 172.21.15.181:0/2444102464 wait complete. 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.452+0000 7fd03115d640 10 monclient: build_initial_monmap 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.452+0000 7fd03115d640 10 monclient: monmap: 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.467 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:51.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:51.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:51.468 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.452+0000 7fd03115d640 1 Processor -- start 2024-09-15T03:32:51.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 -- start start 2024-09-15T03:32:51.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient: init 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c09ee60 0x7fd02c1411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): picked mon.noname-a con 0x7fd02c09ee60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): picked mon.noname-c con 0x7fd02c09f260 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c133f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): picked mon.noname-b con 0x7fd02c0a0ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0a11e0 con 0x7fd02c09ee60 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c1226c0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c121180 con 0x7fd02c09f260 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd03115d640 10 monclient(hunting): authenticate will time out at 975.934507s 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c133f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44486/0 (socket says 172.21.15.181:44486) 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 -- 172.21.15.181:0/2694582309 learned_addr learned my addr 172.21.15.181:0/2694582309 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): get_auth_request con 0x7fd02c0a0ad0 auth_method 0 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c09ee60 0x7fd02c1411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): get_auth_request con 0x7fd02c09ee60 auth_method 0 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 10 monclient(hunting): get_auth_request con 0x7fd02c09f260 auth_method 0 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 1 -- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 msgr2=0x7fd02c141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 1 -- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 msgr2=0x7fd02c141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02a59c640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient(hunting): handle_auth_done global_id 44920 payload 824 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 -- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 msgr2=0x7fd02c141700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:51.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd02c09f260 0x7fd02c141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 -- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c09ee60 msgr2=0x7fd02c1411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd02c09ee60 0x7fd02c1411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient: _finish_hunting 0 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 monclient: found mon.noname-b 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.453+0000 7fd02b59e640 1 -- 172.21.15.181:0/2694582309 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024009390 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd02b59e640 1 -- 172.21.15.181:0/2694582309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c0a0ad0 msgr2=0x7fd02c133f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd02b59e640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c0a0ad0 0x7fd02c133f40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009fd0 tx=0x7fd01c009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00ee30 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: dump: 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.474 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 5 monclient: authenticate success, global_id 44920 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c013590 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 10 monclient: _renew_subs 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c013950 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 1 -- 172.21.15.181:0/2694582309 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c122070 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd022ffd640 10 monclient: dump: 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.476 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd028d99640 4 set_mon_vals no callback set 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 10 monclient: _renew_subs 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.456+0000 7fd03115d640 1 -- 172.21.15.181:0/2694582309 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c134480 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd01c01e460 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd03115d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:51.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd03115d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd03115d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd03115d640 1 -- 172.21.15.181:0/2694582309 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c0008d0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd022ffd640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd00c038770 0x7fd00c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fd01c04c6c0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd02ad9d640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd00c038770 0x7fd00c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.457+0000 7fd02ad9d640 10 monclient: get_auth_request con 0x7fd00c038770 auth_method 0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.459+0000 7fd02ad9d640 1 --2- 172.21.15.181:0/2694582309 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd00c038770 0x7fd00c03ac20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd024031000 tx=0x7fd024032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.461+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd01c0128e0 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.461+0000 7fd022ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:51.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.461+0000 7fd022ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:51.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.621+0000 7fd03115d640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-15T03:32:51.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.621+0000 7fd03115d640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-15T03:32:51.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.621+0000 7fd03115d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.621+0000 7fd03115d640 1 -- 172.21.15.181:0/2694582309 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) v1 -- 0x7fd02c0a2c70 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.622+0000 7fd022ffd640 1 -- 172.21.15.181:0/2694582309 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v102) v1 ==== 146+0+0 (secure 0 0 0) 0x7fd01c01b310 con 0x7fd02c0a0ad0 2024-09-15T03:32:51.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.622+0000 7fd022ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-15T03:32:51.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.622+0000 7fd022ffd640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-15T03:32:51.623 INFO:tasks.workunit.client.0.smithi181.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-15T03:32:51.627 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:32:51.627 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:32:51.628 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep osd.11 2024-09-15T03:32:51.628 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 113 2024-09-15T03:32:51.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.697+0000 7fb270c2b640 10 monclient: get_monmap_and_config 2024-09-15T03:32:51.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.697+0000 7fb270c2b640 10 monclient: build_initial_monmap 2024-09-15T03:32:51.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.697+0000 7fb270c2b640 10 monclient: monmap: 2024-09-15T03:32:51.698 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:51.698101+0000 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:51.698101+0000 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.697+0000 7fb270c2b640 1 Processor -- start 2024-09-15T03:32:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.697+0000 7fb270c2b640 1 -- start start 2024-09-15T03:32:51.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.697+0000 7fb270c2b640 10 monclient: init 2024-09-15T03:32:51.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:51.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c09cb20 0x7fb26c09afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb26c09cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb26c12e940 0x7fb26c12ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb26c12e940 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb26c12f290 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c09d6f0 con 0x7fb26c12f290 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c097e90 con 0x7fb26c12e940 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c099640 con 0x7fb26c09cb20 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:51.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb270c2b640 10 monclient(hunting): authenticate will time out at 976.179539s 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb26c12e940 0x7fb26c12ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb26c12e940 0x7fb26c12ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44490/0 (socket says 172.21.15.181:44490) 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 -- 172.21.15.181:0/1624675283 learned_addr learned my addr 172.21.15.181:0/1624675283 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): get_auth_request con 0x7fb26c12e940 auth_method 0 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb26a59c640 1 --2- 172.21.15.181:0/1624675283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c09cb20 0x7fb26c09afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb26a59c640 10 monclient(hunting): get_auth_request con 0x7fb26c09cb20 auth_method 0 2024-09-15T03:32:51.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb26a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb26a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb26a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient(hunting): handle_auth_done global_id 44923 payload 274 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 -- 172.21.15.181:0/1624675283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c09cb20 msgr2=0x7fb26c09afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 --2- 172.21.15.181:0/1624675283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c09cb20 0x7fb26c09afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 -- 172.21.15.181:0/1624675283 >> v1:172.21.15.153:6789/0 conn(0x7fb26c12f290 legacy=0x7fb26c096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 monclient: found mon.noname-c 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.698+0000 7fb269d9b640 1 -- 172.21.15.181:0/1624675283 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c098da0 con 0x7fb26c12e940 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb269d9b640 1 -- 172.21.15.181:0/1624675283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c12e940 msgr2=0x7fb26c12ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:51.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb269d9b640 1 --2- 172.21.15.181:0/1624675283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c12e940 0x7fb26c12ed50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb26000bbc0 tx=0x7fb26000b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd7b974ec42a9d83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 1 -- 172.21.15.181:0/1624675283 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260030050 con 0x7fb26c12e940 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: dump: 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.707 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: _finish_auth 0 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb270c2b640 5 monclient: authenticate success, global_id 44923 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb270c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 1 -- 172.21.15.181:0/1624675283 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb260032710 con 0x7fb26c12e940 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb270c2b640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 1 -- 172.21.15.181:0/1624675283 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260032af0 con 0x7fb26c12e940 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb270c2b640 4 set_mon_vals no callback set 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.701+0000 7fb26959a640 10 monclient: dump: 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 10 monclient: shutdown 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 -- 172.21.15.181:0/1624675283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c12e940 msgr2=0x7fb26c12ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 --2- 172.21.15.181:0/1624675283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c12e940 0x7fb26c12ed50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb26000bbc0 tx=0x7fb26000b460 comp rx=0 tx=0).stop 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 -- 172.21.15.181:0/1624675283 shutdown_connections 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 --2- 172.21.15.181:0/1624675283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c12e940 0x7fb26c12ed50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 --2- 172.21.15.181:0/1624675283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c09cb20 0x7fb26c09afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 -- 172.21.15.181:0/1624675283 >> 172.21.15.181:0/1624675283 conn(0x7fb26c0714d0 msgr2=0x7fb26c0a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 -- 172.21.15.181:0/1624675283 shutdown_connections 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 1 -- 172.21.15.181:0/1624675283 wait complete. 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 10 monclient: build_initial_monmap 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.702+0000 7fb270c2b640 10 monclient: monmap: 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:51.711 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 Processor -- start 2024-09-15T03:32:51.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 -- start start 2024-09-15T03:32:51.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient: init 2024-09-15T03:32:51.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:51.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:51.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 0x7fb26c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb26c09cb20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb26c12e940 0x7fb26c13d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb26c12e940 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c12f290 0x7fb26c12ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb26c12f290 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c119ea0 con 0x7fb26c12e940 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c13d640 con 0x7fb26c09cb20 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c118e80 con 0x7fb26c12f290 2024-09-15T03:32:51.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb270c2b640 10 monclient(hunting): authenticate will time out at 976.184799s 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 0x7fb26c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 0x7fb26c06d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44496/0 (socket says 172.21.15.181:44496) 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 1 -- 172.21.15.181:0/348560540 learned_addr learned my addr 172.21.15.181:0/348560540 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): get_auth_request con 0x7fb26c09cb20 auth_method 0 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.703+0000 7fb26a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 10 monclient(hunting): handle_auth_done global_id 44926 payload 824 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 1 -- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c12f290 msgr2=0x7fb26c12ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c12f290 0x7fb26c12ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb269d9b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb26c12e940 0x7fb26c13d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 1 -- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb26c12e940 msgr2=0x7fb26c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb26c12e940 0x7fb26c13d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 1 monclient: found mon.noname-b 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.704+0000 7fb26a59c640 1 -- 172.21.15.181:0/348560540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26000b700 con 0x7fb26c09cb20 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb26a59c640 1 -- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 msgr2=0x7fb26c06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb26a59c640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 0x7fb26c06d800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00d890 tx=0x7fb25c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c00de20 con 0x7fb26c09cb20 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: dump: 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:51.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c009ac0 con 0x7fb26c09cb20 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 5 monclient: authenticate success, global_id 44926 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c009e80 con 0x7fb26c09cb20 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb25a7fc640 10 monclient: dump: 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 10 monclient: _renew_subs 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c119bc0 con 0x7fb26c09cb20 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb268d99640 4 set_mon_vals no callback set 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 10 monclient: _renew_subs 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.706+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c12bfd0 con 0x7fb26c09cb20 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb25c014620 con 0x7fb26c09cb20 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb270c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:51.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb270c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb270c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c0636e0 con 0x7fb26c09cb20 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb25a7fc640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb24c038740 0x7fb24c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fb25c04c8e0 con 0x7fb26c09cb20 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb269d9b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb24c038740 0x7fb24c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.707+0000 7fb269d9b640 10 monclient: get_auth_request con 0x7fb24c038740 auth_method 0 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.710+0000 7fb269d9b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb24c038740 0x7fb24c03abf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb260037000 tx=0x7fb26003a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.711+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb25c025b10 con 0x7fb26c09cb20 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.711+0000 7fb25a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:51.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.711+0000 7fb25a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.867+0000 7fb270c2b640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.867+0000 7fb270c2b640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.867+0000 7fb270c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.867+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb26c12c250 con 0x7fb26c09cb20 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.868+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v102) v1 ==== 56+0+1173 (secure 0 0 0) 0x7fb25c025350 con 0x7fb26c09cb20 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.868+0000 7fb25a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:32:51.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.868+0000 7fb25a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.961+0000 7fb270c2b640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.961+0000 7fb270c2b640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.961+0000 7fb270c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.961+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb26c072640 con 0x7fb26c09cb20 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.962+0000 7fb25a7fc640 1 -- 172.21.15.181:0/348560540 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v102) v1 ==== 56+0+1173 (secure 0 0 0) 0x7fb25c025530 con 0x7fb26c09cb20 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.962+0000 7fb25a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:32:51.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.962+0000 7fb25a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:51.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb24c038740 msgr2=0x7fb24c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb24c038740 0x7fb24c03abf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb260037000 tx=0x7fb26003a000 comp rx=0 tx=0).stop 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 10 monclient: shutdown 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 msgr2=0x7fb26c06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 0x7fb26c06d800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00d890 tx=0x7fb25c008040 comp rx=0 tx=0).stop 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 shutdown_connections 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb24c038740 0x7fb24c03abf0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb26c12f290 0x7fb26c12ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb26c12e940 0x7fb26c13d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 --2- 172.21.15.181:0/348560540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb26c09cb20 0x7fb26c06d800 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:51.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 >> 172.21.15.181:0/348560540 conn(0x7fb26c0714d0 msgr2=0x7fb26c130110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:51.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 shutdown_connections 2024-09-15T03:32:51.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:51.964+0000 7fb270c2b640 1 -- 172.21.15.181:0/348560540 wait complete. 2024-09-15T03:32:51.974 INFO:tasks.workunit.client.0.smithi181.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-15T03:32:51.975 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.11 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.047+0000 7f601d300640 10 monclient: get_monmap_and_config 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.047+0000 7f601d300640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.047+0000 7f601d300640 10 monclient: monmap: 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:52.047829+0000 2024-09-15T03:32:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:52.047829+0000 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:52.049 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.047+0000 7f601d300640 1 Processor -- start 2024-09-15T03:32:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.047+0000 7f601d300640 1 -- start start 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.047+0000 7f601d300640 10 monclient: init 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f601809ec50 0x7f6018131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): picked mon.noname-a con 0x7f601809ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f60181321a0 0x7f6018068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): picked mon.noname-c con 0x7f60181321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f601809f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6018067420 con 0x7f601809f020 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60180a0480 con 0x7f601809ec50 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60180a0200 con 0x7f60181321a0 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f601d300640 10 monclient(hunting): authenticate will time out at 976.529234s 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f60181321a0 0x7f6018068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f60181321a0 0x7f6018068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44506/0 (socket says 172.21.15.181:44506) 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 -- 172.21.15.181:0/2775131294 learned_addr learned my addr 172.21.15.181:0/2775131294 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60177fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f601809f020 0x7f6018069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34664/0 (socket says 172.21.15.181:34664) 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): get_auth_request con 0x7f60181321a0 auth_method 0 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f6016ffd640 1 --2- 172.21.15.181:0/2775131294 >> v2:172.21.15.153:3300/0 conn(0x7f601809ec50 0x7f6018131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f6016ffd640 10 monclient(hunting): get_auth_request con 0x7f601809ec50 auth_method 0 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f6016ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f6016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f6016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient(hunting): handle_auth_done global_id 44929 payload 274 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 -- 172.21.15.181:0/2775131294 >> v2:172.21.15.153:3300/0 conn(0x7f601809ec50 msgr2=0x7f6018131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 --2- 172.21.15.181:0/2775131294 >> v2:172.21.15.153:3300/0 conn(0x7f601809ec50 0x7f6018131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 -- 172.21.15.181:0/2775131294 >> v1:172.21.15.181:6789/0 conn(0x7f601809f020 legacy=0x7f6018069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 monclient: found mon.noname-c 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.048+0000 7f60167fc640 1 -- 172.21.15.181:0/2775131294 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601809fd80 con 0x7f60181321a0 2024-09-15T03:32:52.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f60167fc640 1 -- 172.21.15.181:0/2775131294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60181321a0 msgr2=0x7f6018068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f60167fc640 1 --2- 172.21.15.181:0/2775131294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60181321a0 0x7f6018068d00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f60181325a0 tx=0x7f60080091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f4657e4495019c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 1 -- 172.21.15.181:0/2775131294 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008032440 con 0x7f60181321a0 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: dump: 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.060 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 1 -- 172.21.15.181:0/2775131294 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60080325a0 con 0x7f60181321a0 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f601d300640 5 monclient: authenticate success, global_id 44929 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f601d300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f601d300640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f601d300640 4 set_mon_vals no callback set 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 1 -- 172.21.15.181:0/2775131294 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008032980 con 0x7f60181321a0 2024-09-15T03:32:52.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.051+0000 7f6015ffb640 10 monclient: dump: 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 10 monclient: shutdown 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 -- 172.21.15.181:0/2775131294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60181321a0 msgr2=0x7f6018068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 --2- 172.21.15.181:0/2775131294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60181321a0 0x7f6018068d00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f60181325a0 tx=0x7f60080091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 -- 172.21.15.181:0/2775131294 shutdown_connections 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 --2- 172.21.15.181:0/2775131294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60181321a0 0x7f6018068d00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 --2- 172.21.15.181:0/2775131294 >> v2:172.21.15.153:3300/0 conn(0x7f601809ec50 0x7f6018131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 -- 172.21.15.181:0/2775131294 >> 172.21.15.181:0/2775131294 conn(0x7f60180a5cf0 msgr2=0x7f60180a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 -- 172.21.15.181:0/2775131294 shutdown_connections 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.052+0000 7f601d300640 1 -- 172.21.15.181:0/2775131294 wait complete. 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient: monmap: 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:52.063 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 Processor -- start 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 -- start start 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient: init 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:52.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f601809ec50 0x7f601811bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): picked mon.noname-c con 0x7f601809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 0x7f6018130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): picked mon.noname-b con 0x7f601809f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60181321a0 0x7f6018131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): picked mon.noname-a con 0x7f60181321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601811ca60 con 0x7f60181321a0 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601809c020 con 0x7f601809f020 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601809e0b0 con 0x7f601809ec50 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.053+0000 7f601d300640 10 monclient(hunting): authenticate will time out at 976.534988s 2024-09-15T03:32:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 0x7f6018130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 0x7f6018130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44512/0 (socket says 172.21.15.181:44512) 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 -- 172.21.15.181:0/959290242 learned_addr learned my addr 172.21.15.181:0/959290242 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): get_auth_request con 0x7f601809f020 auth_method 0 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60177fe640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60181321a0 0x7f6018131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f6016ffd640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f601809ec50 0x7f601811bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient(hunting): handle_auth_done global_id 44932 payload 824 2024-09-15T03:32:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 -- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f601809ec50 msgr2=0x7f601811bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f601809ec50 0x7f601811bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 -- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60181321a0 msgr2=0x7f6018131100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60181321a0 0x7f6018131100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 monclient: found mon.noname-b 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60167fc640 1 -- 172.21.15.181:0/959290242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6008009450 con 0x7f601809f020 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60177fe640 10 monclient: get_auth_request con 0x7f60181321a0 auth_method 0 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.054+0000 7f60177fe640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60181321a0 0x7f6018131100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f60167fc640 1 -- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 msgr2=0x7f6018130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f60167fc640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 0x7f6018130bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6008009940 tx=0x7f6008032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008031b20 con 0x7f601809f020 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: dump: 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.068 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 5 monclient: authenticate success, global_id 44932 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6008031cc0 con 0x7f601809f020 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 10 monclient: _renew_subs 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600803d690 con 0x7f601809f020 2024-09-15T03:32:52.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601811dfc0 con 0x7f601809f020 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f5ffeffd640 10 monclient: dump: 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.070 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f60157fa640 4 set_mon_vals no callback set 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 10 monclient: _renew_subs 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.056+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601811dce0 con 0x7f601809f020 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6008039040 con 0x7f601809f020 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f601d300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f601d300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f601d300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60180008d0 con 0x7f601809f020 2024-09-15T03:32:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f5ffeffd640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6004038740 0x7f600403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f600806bf20 con 0x7f601809f020 2024-09-15T03:32:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.057+0000 7f6016ffd640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6004038740 0x7f600403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.058+0000 7f6016ffd640 10 monclient: get_auth_request con 0x7f6004038740 auth_method 0 2024-09-15T03:32:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.060+0000 7f6016ffd640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6004038740 0x7f600403abf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f600c006fd0 tx=0x7f600c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.062+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6008030570 con 0x7f601809f020 2024-09-15T03:32:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.062+0000 7f5ffeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.062+0000 7f5ffeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:52.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.224+0000 7f601d300640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T03:32:52.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.224+0000 7f601d300640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T03:32:52.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.224+0000 7f601d300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.225+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7f60180a13e0 con 0x7f601809f020 2024-09-15T03:32:52.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.364+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v103) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6008037090 con 0x7f601809f020 2024-09-15T03:32:52.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.364+0000 7f5ffeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T03:32:52.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.364+0000 7f5ffeffd640 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-15T03:32:52.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.463+0000 7f601d300640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T03:32:52.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.463+0000 7f601d300640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T03:32:52.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.463+0000 7f601d300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.463+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7f601811d950 con 0x7f601809f020 2024-09-15T03:32:52.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.464+0000 7f5ffeffd640 1 -- 172.21.15.181:0/959290242 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v103) v1 ==== 126+0+0 (secure 0 0 0) 0x7f6008042070 con 0x7f601809f020 2024-09-15T03:32:52.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.464+0000 7f5ffeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-15T03:32:52.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.464+0000 7f5ffeffd640 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-15T03:32:52.466 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.11' does not appear in the crush map 2024-09-15T03:32:52.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.466+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6004038740 msgr2=0x7f600403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6004038740 0x7f600403abf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f600c006fd0 tx=0x7f600c008040 comp rx=0 tx=0).stop 2024-09-15T03:32:52.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 10 monclient: shutdown 2024-09-15T03:32:52.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 msgr2=0x7f6018130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 0x7f6018130bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6008009940 tx=0x7f6008032000 comp rx=0 tx=0).stop 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 shutdown_connections 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6004038740 0x7f600403abf0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60181321a0 0x7f6018131100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f601809f020 0x7f6018130bc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 --2- 172.21.15.181:0/959290242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f601809ec50 0x7f601811bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 >> 172.21.15.181:0/959290242 conn(0x7f60180a5cf0 msgr2=0x7f6018134930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 shutdown_connections 2024-09-15T03:32:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.467+0000 7f601d300640 1 -- 172.21.15.181:0/959290242 wait complete. 2024-09-15T03:32:52.477 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm osd.11 2024-09-15T03:32:52.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:52.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.548+0000 7f612cc0f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.548+0000 7f612cc0f640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.548+0000 7f612cc0f640 10 monclient: monmap: 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:52.549283+0000 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:52.549283+0000 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 Processor -- start 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 -- start start 2024-09-15T03:32:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient: init 2024-09-15T03:32:52.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:32:52.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6128099a60 0x7f612809a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f6128099a60 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:52.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f612809ad00 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:52.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f612809e010 0x7f612809e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f612809e010 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:52.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6128069bc0 con 0x7f612809ad00 2024-09-15T03:32:52.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612806a180 con 0x7f6128099a60 2024-09-15T03:32:52.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6128099e30 con 0x7f612809e010 2024-09-15T03:32:52.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f612cc0f640 10 monclient(hunting): authenticate will time out at 977.030777s 2024-09-15T03:32:52.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6126d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f612809e010 0x7f612809e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6125d9b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f612809ad00 0x7f612809d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34678/0 (socket says 172.21.15.181:34678) 2024-09-15T03:32:52.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6126d9d640 10 monclient(hunting): get_auth_request con 0x7f612809e010 auth_method 0 2024-09-15T03:32:52.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6125d9b640 1 -- 172.21.15.181:0/2783748517 learned_addr learned my addr 172.21.15.181:0/2783748517 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:52.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6126d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6126d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6126d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.549+0000 7f6126d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f612659c640 1 --2- 172.21.15.181:0/2783748517 >> v2:172.21.15.153:3300/0 conn(0x7f6128099a60 0x7f612809a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f612659c640 10 monclient(hunting): get_auth_request con 0x7f6128099a60 auth_method 0 2024-09-15T03:32:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f612659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f612659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f612659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 10 monclient(hunting): handle_auth_done global_id 44935 payload 274 2024-09-15T03:32:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 1 -- 172.21.15.181:0/2783748517 >> v2:172.21.15.153:3300/0 conn(0x7f6128099a60 msgr2=0x7f612809a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 1 --2- 172.21.15.181:0/2783748517 >> v2:172.21.15.153:3300/0 conn(0x7f6128099a60 0x7f612809a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 1 -- 172.21.15.181:0/2783748517 >> v1:172.21.15.181:6789/0 conn(0x7f612809ad00 legacy=0x7f612809d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 1 monclient: found mon.noname-c 2024-09-15T03:32:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f6126d9d640 1 -- 172.21.15.181:0/2783748517 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612806a880 con 0x7f612809e010 2024-09-15T03:32:52.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.550+0000 7f612659c640 1 --2- 172.21.15.181:0/2783748517 >> v2:172.21.15.153:3300/0 conn(0x7f6128099a60 0x7f612809a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:32:52.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.552+0000 7f6126d9d640 1 -- 172.21.15.181:0/2783748517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 msgr2=0x7f612809e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.552+0000 7f6126d9d640 1 --2- 172.21.15.181:0/2783748517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612809e410 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6128064940 tx=0x7f6120009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b277f25377af8ae4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 1 -- 172.21.15.181:0/2783748517 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120007760 con 0x7f612809e010 2024-09-15T03:32:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: dump: 2024-09-15T03:32:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.576 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.576 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.576 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 1 -- 172.21.15.181:0/2783748517 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612002d050 con 0x7f612809e010 2024-09-15T03:32:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 5 monclient: authenticate success, global_id 44935 2024-09-15T03:32:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 1 -- 172.21.15.181:0/2783748517 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120007af0 con 0x7f612809e010 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 4 set_mon_vals no callback set 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612559a640 10 monclient: dump: 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 10 monclient: shutdown 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 -- 172.21.15.181:0/2783748517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 msgr2=0x7f612809e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 --2- 172.21.15.181:0/2783748517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612809e410 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6128064940 tx=0x7f6120009780 comp rx=0 tx=0).stop 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 -- 172.21.15.181:0/2783748517 shutdown_connections 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 --2- 172.21.15.181:0/2783748517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612809e410 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 --2- 172.21.15.181:0/2783748517 >> v2:172.21.15.153:3300/0 conn(0x7f6128099a60 0x7f612809a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 -- 172.21.15.181:0/2783748517 >> 172.21.15.181:0/2783748517 conn(0x7f61280a2f10 msgr2=0x7f61280a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 -- 172.21.15.181:0/2783748517 shutdown_connections 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.553+0000 7f612cc0f640 1 -- 172.21.15.181:0/2783748517 wait complete. 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient: monmap: 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 Processor -- start 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 -- start start 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient: init 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 0x7f6128140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f6128099a60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f612809ad00 0x7f6128133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f612809ad00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612811e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f612809e010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612806eb10 con 0x7f612809ad00 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6128122110 con 0x7f612809e010 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6128121e30 con 0x7f6128099a60 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.554+0000 7f612cc0f640 10 monclient(hunting): authenticate will time out at 977.035949s 2024-09-15T03:32:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612811e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612811e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44530/0 (socket says 172.21.15.181:44530) 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 -- 172.21.15.181:0/1688077610 learned_addr learned my addr 172.21.15.181:0/1688077610 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): get_auth_request con 0x7f612809e010 auth_method 0 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f612809ad00 0x7f6128133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 0x7f6128140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): get_auth_request con 0x7f612809ad00 auth_method 0 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 10 monclient(hunting): get_auth_request con 0x7f6128099a60 auth_method 0 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6125d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient(hunting): handle_auth_done global_id 44938 payload 824 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 msgr2=0x7f6128140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 msgr2=0x7f6128140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 msgr2=0x7f6128140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 0x7f6128140e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f612659c640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 0x7f6128140e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 0x7f6128140e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f612809ad00 msgr2=0x7f6128133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f612809ad00 0x7f6128133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 monclient: found mon.noname-b 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.555+0000 7f6126d9d640 1 -- 172.21.15.181:0/1688077610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6120009a20 con 0x7f612809e010 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f6126d9d640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 msgr2=0x7f612811e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f6126d9d640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612811e810 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f61200099f0 tx=0x7f6120037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120031c10 con 0x7f612809e010 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:52.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: dump: 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6120040070 con 0x7f612809e010 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 5 monclient: authenticate success, global_id 44938 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 10 monclient: _renew_subs 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120045480 con 0x7f612809e010 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61281223f0 con 0x7f612809e010 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f6124d99640 4 set_mon_vals no callback set 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f611e7fc640 10 monclient: dump: 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:52.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 10 monclient: _renew_subs 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.557+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6128121070 con 0x7f612809e010 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6120042030 con 0x7f612809e010 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f612cc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f612cc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f612cc0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f611e7fc640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6108038740 0x7f610803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612804bf60 con 0x7f612809e010 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f612006d660 con 0x7f612809e010 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f612659c640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6108038740 0x7f610803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.558+0000 7f612659c640 10 monclient: get_auth_request con 0x7f6108038740 auth_method 0 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.561+0000 7f612659c640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6108038740 0x7f610803abf0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6110006e70 tx=0x7f6110008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.563+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f61200309a0 con 0x7f612809e010 2024-09-15T03:32:52.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.563+0000 7f611e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.563+0000 7f611e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:52.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.642+0000 7f994a460640 10 monclient: get_monmap_and_config 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.642+0000 7f994a460640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.642+0000 7f994a460640 10 monclient: monmap: 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:52.642811+0000 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:52.642811+0000 2024-09-15T03:32:52.648 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.642+0000 7f994a460640 1 Processor -- start 2024-09-15T03:32:52.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.642+0000 7f994a460640 1 -- start start 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.642+0000 7f994a460640 10 monclient: init 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99441318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f994409ea50 0x7f994409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient(hunting): picked mon.noname-a con 0x7f994409ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9944136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9944131cc0 con 0x7f99441318f0 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994409fbf0 con 0x7f9944136c20 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994409fe30 con 0x7f994409ea50 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f994a460640 10 monclient(hunting): authenticate will time out at 970.282506s 2024-09-15T03:32:52.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f994409ea50 0x7f994409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f9943fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f99441318f0 0x7f9944131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38248/0 (socket says 172.21.15.153:38248) 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f9943fff640 1 -- 172.21.15.153:0/669723777 learned_addr learned my addr 172.21.15.153:0/669723777 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): get_auth_request con 0x7f994409ea50 auth_method 0 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient(hunting): handle_auth_done global_id 64374 payload 274 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 1 -- 172.21.15.153:0/669723777 >> v1:172.21.15.181:6789/0 conn(0x7f9944136c20 legacy=0x7f9944139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 1 -- 172.21.15.153:0/669723777 >> v1:172.21.15.153:6789/0 conn(0x7f99441318f0 legacy=0x7f9944131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.643+0000 7f99437fe640 1 -- 172.21.15.153:0/669723777 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440a1210 con 0x7f994409ea50 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f99437fe640 1 -- 172.21.15.153:0/669723777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994409ea50 msgr2=0x7f994409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f99437fe640 1 --2- 172.21.15.153:0/669723777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994409ea50 0x7f994409ee20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9938009850 tx=0x7f99380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8eb64f328cd875 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 1 -- 172.21.15.153:0/669723777 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938032440 con 0x7f994409ea50 2024-09-15T03:32:52.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: dump: 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 1 -- 172.21.15.153:0/669723777 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99380325a0 con 0x7f994409ea50 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f994a460640 5 monclient: authenticate success, global_id 64374 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f994a460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f994a460640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 1 -- 172.21.15.153:0/669723777 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938032940 con 0x7f994409ea50 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f994a460640 4 set_mon_vals no callback set 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.646+0000 7f9942ffd640 10 monclient: dump: 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.654 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.655 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 10 monclient: shutdown 2024-09-15T03:32:52.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:52.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 -- 172.21.15.153:0/669723777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994409ea50 msgr2=0x7f994409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 --2- 172.21.15.153:0/669723777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994409ea50 0x7f994409ee20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9938009850 tx=0x7f99380090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 -- 172.21.15.153:0/669723777 shutdown_connections 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 --2- 172.21.15.153:0/669723777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994409ea50 0x7f994409ee20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 -- 172.21.15.153:0/669723777 >> 172.21.15.153:0/669723777 conn(0x7f9944067110 msgr2=0x7f9944067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 -- 172.21.15.153:0/669723777 shutdown_connections 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.647+0000 7f994a460640 1 -- 172.21.15.153:0/669723777 wait complete. 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 10 monclient: monmap: 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:52.662 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 1 Processor -- start 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 1 -- start start 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 10 monclient: init 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 10 monclient(hunting): picked mon.noname-c con 0x7f994409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.648+0000 7f994a460640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 0x7f9944099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): picked mon.noname-a con 0x7f99441318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9944136c20 0x7f9944099e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): picked mon.noname-b con 0x7f9944136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994409b8e0 con 0x7f99441318f0 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994409e000 con 0x7f9944136c20 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994411bba0 con 0x7f994409ea50 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f994a460640 10 monclient(hunting): authenticate will time out at 970.288048s 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 0x7f9944099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 0x7f9944099920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53632/0 (socket says 172.21.15.153:53632) 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 -- 172.21.15.153:0/1124707997 learned_addr learned my addr 172.21.15.153:0/1124707997 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): get_auth_request con 0x7f99441318f0 auth_method 0 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58020/0 (socket says 172.21.15.153:58020) 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 10 monclient(hunting): get_auth_request con 0x7f994409ea50 auth_method 0 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9944136c20 0x7f9944099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 msgr2=0x7f9944130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): get_auth_request con 0x7f9944136c20 auth_method 0 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 msgr2=0x7f9944130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f9943fff640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient(hunting): handle_auth_done global_id 64377 payload 824 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 msgr2=0x7f9944130980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9944136c20 msgr2=0x7f9944099e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9944136c20 0x7f9944099e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99437fe640 1 -- 172.21.15.153:0/1124707997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9938009390 con 0x7f99441318f0 2024-09-15T03:32:52.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.649+0000 7f99489fd640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9944136c20 0x7f9944099e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.651+0000 7f99437fe640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 msgr2=0x7f9944099920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.651+0000 7f99437fe640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 0x7f9944099920 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9938009360 tx=0x7f9938032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99380318c0 con 0x7f99441318f0 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: dump: 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938031a20 con 0x7f99441318f0 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 5 monclient: authenticate success, global_id 64377 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 10 monclient: _renew_subs 2024-09-15T03:32:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994409bf10 con 0x7f99441318f0 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938031dc0 con 0x7f99441318f0 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f992bfff640 10 monclient: dump: 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 10 monclient: _renew_subs 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f99427fc640 4 set_mon_vals no callback set 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.652+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994411cb40 con 0x7f99441318f0 2024-09-15T03:32:52.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9938039040 con 0x7f99441318f0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f994a460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f994a460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f994a460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f992bfff640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9930038740 0x7f993003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994404bf60 con 0x7f99441318f0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f993806c1f0 con 0x7f99441318f0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f9943fff640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9930038740 0x7f993003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.653+0000 7f9943fff640 10 monclient: get_auth_request con 0x7f9930038740 auth_method 0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.655+0000 7f9943fff640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9930038740 0x7f993003abf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9934007fd0 tx=0x7f9934009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.657+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9938037080 con 0x7f99441318f0 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.657+0000 7f992bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.657+0000 7f992bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:52.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.728+0000 7f612cc0f640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T03:32:52.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.728+0000 7f612cc0f640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T03:32:52.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.728+0000 7f612cc0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:52.728+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7f61280677c0 con 0x7f612809e010 2024-09-15T03:32:52.730 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:52.729+0000 7f7635bf1640 -1 mon.a@0(leader).osd e103 definitely_dead 0 2024-09-15T03:32:52.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f994a460640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:52.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f994a460640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:52.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f994a460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:52.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f994404bf60 con 0x7f99441318f0 2024-09-15T03:32:52.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f992bfff640 1 -- 172.21.15.153:0/1124707997 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f993803f020 con 0x7f99441318f0 2024-09-15T03:32:52.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f992bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:52.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.872+0000 7f992bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:52.874 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9930038740 msgr2=0x7f993003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9930038740 0x7f993003abf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9934007fd0 tx=0x7f9934009040 comp rx=0 tx=0).stop 2024-09-15T03:32:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 10 monclient: shutdown 2024-09-15T03:32:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:52.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 msgr2=0x7f9944099920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 0x7f9944099920 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9938009360 tx=0x7f9938032000 comp rx=0 tx=0).stop 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 shutdown_connections 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9930038740 0x7f993003abf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9944136c20 0x7f9944099e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99441318f0 0x7f9944099920 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 --2- 172.21.15.153:0/1124707997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994409ea50 0x7f9944130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 >> 172.21.15.153:0/1124707997 conn(0x7f9944067110 msgr2=0x7f99440a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 shutdown_connections 2024-09-15T03:32:52.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.874+0000 7f994a460640 1 -- 172.21.15.153:0/1124707997 wait complete. 2024-09-15T03:32:52.885 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:32:52.886 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:32:52.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:32:52.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.979+0000 7fd8b5d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:52.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.979+0000 7fd8b5d9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.979+0000 7fd8b5d9d640 10 monclient: monmap: 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:52.980482+0000 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:52.980482+0000 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:52.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 1 Processor -- start 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 1 -- start start 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 10 monclient: init 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd8b0096c60 0x7fd8b009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b0096c60 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8b0097030 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b0133b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.980+0000 7fd8b5d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00696a0 con 0x7fd8b0097030 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8b5d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b009ac00 con 0x7fd8b0096c60 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8b5d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b009d6b0 con 0x7fd8b0133b40 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8b5d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8b5d9d640 10 monclient(hunting): authenticate will time out at 970.620017s 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58028/0 (socket says 172.21.15.153:58028) 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 -- 172.21.15.153:0/53219529 learned_addr learned my addr 172.21.15.153:0/53219529 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 10 monclient(hunting): get_auth_request con 0x7fd8b0133b40 auth_method 0 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 -- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 msgr2=0x7fd8b0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:32:52.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 -- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 msgr2=0x7fd8b0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/53219529 >> v2:172.21.15.181:3300/0 conn(0x7fd8b0096c60 0x7fd8b009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8affff640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): get_auth_request con 0x7fd8b0096c60 auth_method 0 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_done global_id 44944 payload 274 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 1 -- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 msgr2=0x7fd8b0135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 1 -- 172.21.15.153:0/53219529 >> v1:172.21.15.181:6789/0 conn(0x7fd8b0097030 legacy=0x7fd8b0097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:52.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:52.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 1 monclient: found mon.noname-c 2024-09-15T03:32:52.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:52.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:52.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.981+0000 7fd8af7fe640 1 -- 172.21.15.153:0/53219529 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b009d2e0 con 0x7fd8b0096c60 2024-09-15T03:32:52.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8af7fe640 1 -- 172.21.15.153:0/53219529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0096c60 msgr2=0x7fd8b009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:52.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0096c60 0x7fd8b009a6c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009850 tx=0x7fd8a00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7c4464a2eb78cca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:52.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 1 -- 172.21.15.153:0/53219529 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a002adf0 con 0x7fd8b0096c60 2024-09-15T03:32:52.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: dump: 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 1 -- 172.21.15.153:0/53219529 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0032440 con 0x7fd8b0096c60 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8b5d9d640 5 monclient: authenticate success, global_id 44944 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8b5d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8b5d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 1 -- 172.21.15.153:0/53219529 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00327e0 con 0x7fd8b0096c60 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8b5d9d640 4 set_mon_vals no callback set 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.984+0000 7fd8ae7fc640 10 monclient: dump: 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:52.996 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 10 monclient: shutdown 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/53219529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0096c60 msgr2=0x7fd8b009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0096c60 0x7fd8b009a6c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009850 tx=0x7fd8a00090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/53219529 shutdown_connections 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0133b40 0x7fd8b0135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/53219529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0096c60 0x7fd8b009a6c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/53219529 >> 172.21.15.153:0/53219529 conn(0x7fd8b00672b0 msgr2=0x7fd8b00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/53219529 shutdown_connections 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/53219529 wait complete. 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.985+0000 7fd8b5d9d640 10 monclient: monmap: 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 Processor -- start 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 -- start start 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient: init 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 0x7fd8b006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b0096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0097030 0x7fd8b012ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd8b0097030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0133b40 0x7fd8b01411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b0133b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:52.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b009add0 con 0x7fd8b0097030 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0133670 con 0x7fd8b0133b40 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b011ded0 con 0x7fd8b0096c60 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8b5d9d640 10 monclient(hunting): authenticate will time out at 970.625626s 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 0x7fd8b006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 0x7fd8b006e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58032/0 (socket says 172.21.15.153:58032) 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 learned_addr learned my addr 172.21.15.153:0/1816561298 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 10 monclient(hunting): get_auth_request con 0x7fd8b0096c60 auth_method 0 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0097030 0x7fd8b012ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): get_auth_request con 0x7fd8b0097030 auth_method 0 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 msgr2=0x7fd8b006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 msgr2=0x7fd8b006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 0x7fd8b006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0133b40 0x7fd8b01411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 0x7fd8b006e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:53.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): get_auth_request con 0x7fd8b0133b40 auth_method 0 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 10 monclient(hunting): handle_auth_done global_id 64380 payload 824 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 msgr2=0x7fd8b006e470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.986+0000 7fd8aeffd640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8b0096c60 0x7fd8b006e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0133b40 msgr2=0x7fd8b01411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8b0133b40 0x7fd8b01411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 1 monclient: found mon.noname-a 2024-09-15T03:32:53.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.987+0000 7fd8aeffd640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0009390 con 0x7fd8b0097030 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8aeffd640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0097030 msgr2=0x7fd8b012ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8aeffd640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0097030 0x7fd8b012ea20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c4e0 tx=0x7fd8a4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40117a0 con 0x7fd8b0097030 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: dump: 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 5 monclient: authenticate success, global_id 64380 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a4011920 con 0x7fd8b0097030 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 10 monclient: _renew_subs 2024-09-15T03:32:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b011c290 con 0x7fd8b0097030 2024-09-15T03:32:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4011ce0 con 0x7fd8b0097030 2024-09-15T03:32:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8977fe640 10 monclient: dump: 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 10 monclient: _renew_subs 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b011bd60 con 0x7fd8b0097030 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.989+0000 7fd8adffb640 4 set_mon_vals no callback set 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8b5d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 0x7fd8b00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0065f20 con 0x7fd8b012ef60 2024-09-15T03:32:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd8a4012630 con 0x7fd8b0097030 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 0x7fd8b00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient: get_auth_request con 0x7fd8b012ef60 auth_method 0 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8977fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd89c038770 0x7fd89c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5956+0+0 (secure 0 0 0) 0x7fd8a404bb60 con 0x7fd8b0097030 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_done global_id 64383 payload 824 2024-09-15T03:32:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8b011dd00 con 0x7fd8b012ef60 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8affff640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd89c038770 0x7fd89c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.990+0000 7fd8affff640 10 monclient: get_auth_request con 0x7fd89c038770 auth_method 0 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.992+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 msgr2=0x7fd8b00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.992+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 0x7fd8b00298c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009850 tx=0x7fd8a0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0031a80 con 0x7fd8b012ef60 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 10 monclient: dump: 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8affff640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd89c038770 0x7fd89c03ac20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd8b011de80 tx=0x7fd8a8009250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd8a0031be0 con 0x7fd8b012ef60 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 msgr2=0x7fd8b00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 0x7fd8b00298c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009850 tx=0x7fd8a0032000 comp rx=0 tx=0).stop 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 msgr2=0x7fd8b00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:53.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.993+0000 7fd8977fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b012ef60 0x7fd8b00298c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.998+0000 7fd8b5d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.998+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 0x7fd8b002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.998+0000 7fd8b5d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.998+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b011bfe0 con 0x7fd8b0029e00 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 0x7fd8b002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient: get_auth_request con 0x7fd8b0029e00 auth_method 0 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 10 monclient(hunting): handle_auth_done global_id 64386 payload 824 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:52.999+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd8b004bf60 con 0x7fd8b0029e00 2024-09-15T03:32:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8af7fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 msgr2=0x7fd8b002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8af7fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 0x7fd8b002a200 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd8b006f1b0 tx=0x7fd8a003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a004b070 con 0x7fd8b0029e00 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 10 monclient: dump: 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.014 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7fd8a0031870 con 0x7fd8b0029e00 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 msgr2=0x7fd8b002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 0x7fd8b002a200 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd8b006f1b0 tx=0x7fd8a003d000 comp rx=0 tx=0).stop 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 msgr2=0x7fd8b002a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.001+0000 7fd8977fe640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0029e00 0x7fd8b002a200 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd8b006f1b0 tx=0x7fd8a003d000 comp rx=0 tx=0).stop 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:53.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:53.016 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:53.017 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:53.018 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:53.019 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.020 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:53.021 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:53.022 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:53.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.024 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:53.025 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.026 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:53.027 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:53.028 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd89c038770 msgr2=0x7fd89c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd89c038770 0x7fd89c03ac20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd8b011de80 tx=0x7fd8a8009250 comp rx=0 tx=0).stop 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 10 monclient: shutdown 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0097030 msgr2=0x7fd8b012ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 --2- 172.21.15.153:0/1816561298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8b0097030 0x7fd8b012ea20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c4e0 tx=0x7fd8a4007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8affff640 1 -- 172.21.15.153:0/1816561298 reap_dead start 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8affff640 1 -- 172.21.15.153:0/1816561298 reap_dead start 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 shutdown_connections 2024-09-15T03:32:53.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 >> 172.21.15.153:0/1816561298 conn(0x7fd8b00672b0 msgr2=0x7fd8b009f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 shutdown_connections 2024-09-15T03:32:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.002+0000 7fd8b5d9d640 1 -- 172.21.15.153:0/1816561298 wait complete. 2024-09-15T03:32:53.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:53.031 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:32:53.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:32:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.124+0000 7f6f8ef54640 10 monclient: get_monmap_and_config 2024-09-15T03:32:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.124+0000 7f6f8ef54640 10 monclient: build_initial_monmap 2024-09-15T03:32:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.124+0000 7f6f8ef54640 10 monclient: monmap: 2024-09-15T03:32:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:53.125300+0000 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:53.125300+0000 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:53.126 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 Processor -- start 2024-09-15T03:32:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 -- start start 2024-09-15T03:32:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient: init 2024-09-15T03:32:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): picked mon.noname-b con 0x7f6f8809caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6f8809a540 0x7f6f8806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): picked mon.noname-c con 0x7f6f8809a540 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6f8806f490 0x7f6f8809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8806f490 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8809ed10 con 0x7f6f8806f490 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8809d3b0 con 0x7f6f8809a540 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8809abd0 con 0x7f6f8809caa0 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ef54640 10 monclient(hunting): authenticate will time out at 970.764890s 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8ccf0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58048/0 (socket says 172.21.15.153:58048) 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/425488685 learned_addr learned my addr 172.21.15.153:0/425488685 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request con 0x7f6f8809caa0 auth_method 0 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.125+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/425488685 >> v2:172.21.15.153:3300/0 conn(0x7f6f8806f490 0x7f6f8809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): get_auth_request con 0x7f6f8806f490 auth_method 0 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 msgr2=0x7f6f8809ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 msgr2=0x7f6f8809ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f87fff640 1 --2- 172.21.15.153:0/425488685 >> v2:172.21.15.181:3300/0 conn(0x7f6f8809a540 0x7f6f8806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f87fff640 10 monclient(hunting): get_auth_request con 0x7f6f8809a540 auth_method 0 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient(hunting): handle_auth_done global_id 64389 payload 274 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 msgr2=0x7f6f8809ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/425488685 >> v2:172.21.15.181:3300/0 conn(0x7f6f8809a540 msgr2=0x7f6f8806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/425488685 >> v2:172.21.15.181:3300/0 conn(0x7f6f8809a540 0x7f6f8806ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient: _finish_hunting 0 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 1 monclient: found mon.noname-a 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.126+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/425488685 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8809b920 con 0x7f6f8806f490 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8806f490 msgr2=0x7f6f8809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8806f490 0x7f6f8809d8c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6f880a2e10 tx=0x7f6f80009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceff55067e544c32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 1 -- 172.21.15.153:0/425488685 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80007760 con 0x7f6f8806f490 2024-09-15T03:32:53.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:53.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: dump: 2024-09-15T03:32:53.140 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.140 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 1 -- 172.21.15.153:0/425488685 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8002d050 con 0x7f6f8806f490 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 5 monclient: authenticate success, global_id 64389 2024-09-15T03:32:53.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 1 -- 172.21.15.153:0/425488685 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80007af0 con 0x7f6f8806f490 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 4 set_mon_vals no callback set 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f877fe640 10 monclient: dump: 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.142 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 10 monclient: shutdown 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 -- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8806f490 msgr2=0x7f6f8809d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8806f490 0x7f6f8809d8c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6f880a2e10 tx=0x7f6f80009780 comp rx=0 tx=0).stop 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 -- 172.21.15.153:0/425488685 shutdown_connections 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8806f490 0x7f6f8809d8c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/425488685 >> v2:172.21.15.181:3300/0 conn(0x7f6f8809a540 0x7f6f8806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/425488685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8809ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 -- 172.21.15.153:0/425488685 >> 172.21.15.153:0/425488685 conn(0x7f6f88067cd0 msgr2=0x7f6f880a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 -- 172.21.15.153:0/425488685 shutdown_connections 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.129+0000 7f6f8ef54640 1 -- 172.21.15.153:0/425488685 wait complete. 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.130+0000 7f6f8ef54640 10 monclient: build_initial_monmap 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.130+0000 7f6f8ef54640 10 monclient: monmap: 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.144 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.130+0000 7f6f8ef54640 1 Processor -- start 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.130+0000 7f6f8ef54640 1 -- start start 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.130+0000 7f6f8ef54640 10 monclient: init 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f8806f490 0x7f6f8813b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): picked mon.noname-b con 0x7f6f8806f490 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 0x7f6f8813aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8809a540 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8813e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): picked mon.noname-c con 0x7f6f8809caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f88099d50 con 0x7f6f8809a540 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880a07f0 con 0x7f6f8806f490 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8813bf50 con 0x7f6f8809caa0 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ef54640 10 monclient(hunting): authenticate will time out at 970.770242s 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 0x7f6f8813aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 0x7f6f8813aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53682/0 (socket says 172.21.15.153:53682) 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 -- 172.21.15.153:0/1275069498 learned_addr learned my addr 172.21.15.153:0/1275069498 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): get_auth_request con 0x7f6f8809a540 auth_method 0 2024-09-15T03:32:53.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8813e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 10 monclient(hunting): get_auth_request con 0x7f6f8809caa0 auth_method 0 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 msgr2=0x7f6f8813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 msgr2=0x7f6f8813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:53.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8813e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f8806f490 0x7f6f8813b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8813e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request con 0x7f6f8806f490 auth_method 0 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient(hunting): handle_auth_done global_id 64392 payload 824 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 msgr2=0x7f6f8813e8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8809caa0 0x7f6f8813e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f8806f490 msgr2=0x7f6f8813b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f8806f490 0x7f6f8813b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 monclient: found mon.noname-a 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.131+0000 7f6f87fff640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f80009a20 con 0x7f6f8809a540 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.133+0000 7f6f87fff640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 msgr2=0x7f6f8813aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.133+0000 7f6f87fff640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 0x7f6f8813aa30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400d890 tx=0x7f6f74008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.133+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400de20 con 0x7f6f8809a540 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.133+0000 7f6f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.133+0000 7f6f84ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.133+0000 7f6f84ff9640 10 monclient: dump: 2024-09-15T03:32:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7400bae0 con 0x7f6f8809a540 2024-09-15T03:32:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 5 monclient: authenticate success, global_id 64392 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 10 monclient: _renew_subs 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8811fad0 con 0x7f6f8809a540 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400be80 con 0x7f6f8809a540 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 10 monclient: dump: 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f86ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 10 monclient: _renew_subs 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8813af70 con 0x7f6f8809a540 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8811ff50 con 0x7f6f8811ebb0 2024-09-15T03:32:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6f74014640 con 0x7f6f8809a540 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ccf0640 10 monclient: get_auth_request con 0x7f6f8811ebb0 auth_method 0 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.134+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f84ff9640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f6c038740 0x7f6f6c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f6f7404c5a0 con 0x7f6f8809a540 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f6c038740 0x7f6f6c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.135+0000 7f6f8d4f1640 10 monclient: get_auth_request con 0x7f6f6c038740 auth_method 0 2024-09-15T03:32:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.137+0000 7f6f8d4f1640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f6c038740 0x7f6f6c03abf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6f8809bc90 tx=0x7f6f80037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 10 monclient: get_auth_request con 0x7f6f8811ebb0 auth_method 0 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:53.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:53.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.336+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:32:53.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.427+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v104) v1 ==== 87+0+0 (secure 0 0 0) 0x7f612004a020 con 0x7f612809e010 2024-09-15T03:32:53.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.427+0000 7f611e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T03:32:53.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.427+0000 7f611e7fc640 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-15T03:32:53.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.531+0000 7f612cc0f640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T03:32:53.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.531+0000 7f612cc0f640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T03:32:53.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.531+0000 7f612cc0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.531+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7f6128072640 con 0x7f612809e010 2024-09-15T03:32:53.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:53.532+0000 7f7635bf1640 -1 mon.a@0(leader).osd e104 definitely_dead 0 2024-09-15T03:32:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.532+0000 7f611e7fc640 1 -- 172.21.15.181:0/1688077610 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v104) v1 ==== 96+0+0 (secure 0 0 0) 0x7f612003a7d0 con 0x7f612809e010 2024-09-15T03:32:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.533+0000 7f611e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-15T03:32:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.533+0000 7f611e7fc640 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-15T03:32:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:osd.11 does not exist. 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6108038740 msgr2=0x7f610803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6108038740 0x7f610803abf0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6110006e70 tx=0x7f6110008040 comp rx=0 tx=0).stop 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 10 monclient: shutdown 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 msgr2=0x7f612811e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612811e810 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f61200099f0 tx=0x7f6120037000 comp rx=0 tx=0).stop 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 shutdown_connections 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6108038740 0x7f610803abf0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f612809e010 0x7f612811e810 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f612809ad00 0x7f6128133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 --2- 172.21.15.181:0/1688077610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128099a60 0x7f6128140e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 >> 172.21.15.181:0/1688077610 conn(0x7f61280a2f10 msgr2=0x7f61280a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:53.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.534+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 shutdown_connections 2024-09-15T03:32:53.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.535+0000 7f612cc0f640 1 -- 172.21.15.181:0/1688077610 wait complete. 2024-09-15T03:32:53.545 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 2024-09-15T03:32:53.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.614+0000 7f3293d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:53.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.614+0000 7f3293d9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:53.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.614+0000 7f3293d9d640 10 monclient: monmap: 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:53.615258+0000 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:53.615258+0000 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:53.617 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:53.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 Processor -- start 2024-09-15T03:32:53.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 -- start start 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient: init 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f328c0a37b0 0x7f328c06e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f328c0a37b0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f328c06be40 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f328c06a6f0 0x7f328c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f328c06a6f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c099ff0 con 0x7f328c06be40 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c069d10 con 0x7f328c0a37b0 2024-09-15T03:32:53.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c0a3b80 con 0x7f328c06a6f0 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3293d9d640 10 monclient(hunting): authenticate will time out at 978.096850s 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f329233a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f328c06a6f0 0x7f328c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3291338640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f328c06be40 0x7f328c09d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34686/0 (socket says 172.21.15.181:34686) 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f3291338640 1 -- 172.21.15.181:0/1356589003 learned_addr learned my addr 172.21.15.181:0/1356589003 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.615+0000 7f329233a640 10 monclient(hunting): get_auth_request con 0x7f328c06a6f0 auth_method 0 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f3291b39640 1 --2- 172.21.15.181:0/1356589003 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a37b0 0x7f328c06e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f3291b39640 10 monclient(hunting): get_auth_request con 0x7f328c0a37b0 auth_method 0 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f3291b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f3291b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f3291b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient(hunting): handle_auth_done global_id 44956 payload 274 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 1 -- 172.21.15.181:0/1356589003 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a37b0 msgr2=0x7f328c06e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 1 --2- 172.21.15.181:0/1356589003 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a37b0 0x7f328c06e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 1 -- 172.21.15.181:0/1356589003 >> v1:172.21.15.181:6789/0 conn(0x7f328c06be40 legacy=0x7f328c09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient: _finish_hunting 0 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 1 monclient: found mon.noname-c 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:53.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.616+0000 7f329233a640 1 -- 172.21.15.181:0/1356589003 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c09f520 con 0x7f328c06a6f0 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f329233a640 1 -- 172.21.15.181:0/1356589003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06a6f0 msgr2=0x7f328c09e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f329233a640 1 --2- 172.21.15.181:0/1356589003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06a6f0 0x7f328c09e070 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f328c09f2d0 tx=0x7f3284009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42786f5457c9750f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 1 -- 172.21.15.181:0/1356589003 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284007760 con 0x7f328c06a6f0 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: dump: 2024-09-15T03:32:53.625 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: _finish_auth 0 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 1 -- 172.21.15.181:0/1356589003 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328402d050 con 0x7f328c06a6f0 2024-09-15T03:32:53.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 5 monclient: authenticate success, global_id 44956 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 1 -- 172.21.15.181:0/1356589003 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284007af0 con 0x7f328c06a6f0 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 4 set_mon_vals no callback set 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3290b37640 10 monclient: dump: 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.627 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 10 monclient: shutdown 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 1 -- 172.21.15.181:0/1356589003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06a6f0 msgr2=0x7f328c09e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.619+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1356589003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06a6f0 0x7f328c09e070 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f328c09f2d0 tx=0x7f3284009780 comp rx=0 tx=0).stop 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 -- 172.21.15.181:0/1356589003 shutdown_connections 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1356589003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06a6f0 0x7f328c09e070 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1356589003 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a37b0 0x7f328c06e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 -- 172.21.15.181:0/1356589003 >> 172.21.15.181:0/1356589003 conn(0x7f328c070fd0 msgr2=0x7f328c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 -- 172.21.15.181:0/1356589003 shutdown_connections 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 -- 172.21.15.181:0/1356589003 wait complete. 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 10 monclient: build_initial_monmap 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 10 monclient: monmap: 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:53.629 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:53.630 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:53.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 Processor -- start 2024-09-15T03:32:53.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 1 -- start start 2024-09-15T03:32:53.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.620+0000 7f3293d9d640 10 monclient: init 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 0x7f328c13d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f328c06a6f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 0x7f328c13cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f328c06be40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a37b0 0x7f328c140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f328c0a37b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c06eaa0 con 0x7f328c0a37b0 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0a24c0 con 0x7f328c06be40 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c13e2a0 con 0x7f328c06a6f0 2024-09-15T03:32:53.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:53.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3293d9d640 10 monclient(hunting): authenticate will time out at 978.102359s 2024-09-15T03:32:53.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 0x7f328c13cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 0x7f328c13cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44554/0 (socket says 172.21.15.181:44554) 2024-09-15T03:32:53.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 -- 172.21.15.181:0/1975743567 learned_addr learned my addr 172.21.15.181:0/1975743567 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:53.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): get_auth_request con 0x7f328c06be40 auth_method 0 2024-09-15T03:32:53.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 0x7f328c13d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 10 monclient(hunting): get_auth_request con 0x7f328c06a6f0 auth_method 0 2024-09-15T03:32:53.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a37b0 0x7f328c140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): get_auth_request con 0x7f328c0a37b0 auth_method 0 2024-09-15T03:32:53.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:53.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 msgr2=0x7f328c13d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:32:53.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 msgr2=0x7f328c13d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:53.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 0x7f328c13d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:53.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:53.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291b39640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 0x7f328c13d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:53.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f329233a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:53.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient(hunting): handle_auth_done global_id 44959 payload 824 2024-09-15T03:32:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 msgr2=0x7f328c13d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 0x7f328c13d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a37b0 msgr2=0x7f328c140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a37b0 0x7f328c140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient: _finish_hunting 0 2024-09-15T03:32:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 monclient: found mon.noname-b 2024-09-15T03:32:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.621+0000 7f3291338640 1 -- 172.21.15.181:0/1975743567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284009a20 con 0x7f328c06be40 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.623+0000 7f3291338640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 msgr2=0x7f328c13cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.623+0000 7f3291338640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 0x7f328c13cd80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c610 tx=0x7f327c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c005cb0 con 0x7f328c06be40 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: dump: 2024-09-15T03:32:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c005e30 con 0x7f328c06be40 2024-09-15T03:32:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 5 monclient: authenticate success, global_id 44959 2024-09-15T03:32:53.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 10 monclient: _renew_subs 2024-09-15T03:32:53.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c121e20 con 0x7f328c06be40 2024-09-15T03:32:53.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:53.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c007320 con 0x7f328c06be40 2024-09-15T03:32:53.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:53.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:53.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3281ffb640 10 monclient: dump: 2024-09-15T03:32:53.651 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:53.651 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:53.651 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.651 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:53.652 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:53.652 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:53.652 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:53.652 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:53.652 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:53.653 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:53.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 10 monclient: _renew_subs 2024-09-15T03:32:53.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3283fff640 4 set_mon_vals no callback set 2024-09-15T03:32:53.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.624+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c13d2c0 con 0x7f328c06be40 2024-09-15T03:32:53.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f327c01b3b0 con 0x7f328c06be40 2024-09-15T03:32:53.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3281ffb640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3270038740 0x7f327003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:53.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f327c04b4e0 con 0x7f328c06be40 2024-09-15T03:32:53.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3293d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:53.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3293d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:53.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3293d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3291b39640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3270038740 0x7f327003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c0636e0 con 0x7f328c06be40 2024-09-15T03:32:53.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.625+0000 7f3291b39640 10 monclient: get_auth_request con 0x7f3270038740 auth_method 0 2024-09-15T03:32:53.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.627+0000 7f3291b39640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3270038740 0x7f327003abf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3288006fd0 tx=0x7f3288008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:53.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.629+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f327c01a360 con 0x7f328c06be40 2024-09-15T03:32:53.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.629+0000 7f3281ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:53.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.629+0000 7f3281ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 10 monclient: get_auth_request con 0x7f6f8811ebb0 auth_method 0 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:32:53.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:32:53.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:32:53.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:53.737+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:32:53.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.794+0000 7f3293d9d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:32:53.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.794+0000 7f3293d9d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T03:32:53.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.794+0000 7f3293d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:53.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:53.794+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f328c13d450 con 0x7f328c06be40 2024-09-15T03:32:54.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.441+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v105) v1 ==== 58+0+3 (secure 0 0 0) 0x7f327c017030 con 0x7f328c06be40 2024-09-15T03:32:54.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.441+0000 7f3281ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T03:32:54.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.441+0000 7f3281ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:54.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient: get_auth_request con 0x7f6f8811ebb0 auth_method 0 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_done global_id 74099 payload 824 2024-09-15T03:32:54.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.538+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f8813b130 con 0x7f6f8811ebb0 2024-09-15T03:32:54.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c015eb0 con 0x7f6f8811ebb0 2024-09-15T03:32:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 10 monclient: dump: 2024-09-15T03:32:54.542 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6f7c01c070 con 0x7f6f8811ebb0 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:54.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).stop 2024-09-15T03:32:54.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 msgr2=0x7f6f8813ee10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:54.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.541+0000 7f6f84ff9640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811ebb0 0x7f6f8813ee10 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).stop 2024-09-15T03:32:54.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.545+0000 7f3293d9d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:32:54.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.545+0000 7f3293d9d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T03:32:54.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.545+0000 7f3293d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:54.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:54.545+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f328c072640 con 0x7f328c06be40 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ef54640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 0x7f6f8813f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ef54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f88065e00 con 0x7f6f8811d6e0 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 0x7f6f8813f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient: get_auth_request con 0x7f6f8811d6e0 auth_method 0 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 10 monclient(hunting): handle_auth_done global_id 74102 payload 824 2024-09-15T03:32:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.548+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6f8804bf60 con 0x7f6f8811d6e0 2024-09-15T03:32:54.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f8ccf0640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 msgr2=0x7f6f8813f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f8ccf0640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 0x7f6f8813f350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c016010 tx=0x7f6f7c023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c018a10 con 0x7f6f8811d6e0 2024-09-15T03:32:54.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 10 monclient: dump: 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f6f7c00c050 con 0x7f6f8811d6e0 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 msgr2=0x7f6f8813f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 0x7f6f8813f350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c016010 tx=0x7f6f7c023000 comp rx=0 tx=0).stop 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 msgr2=0x7f6f8813f350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.551+0000 7f6f84ff9640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8811d6e0 0x7f6f8813f350 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c016010 tx=0x7f6f7c023000 comp rx=0 tx=0).stop 2024-09-15T03:32:54.556 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:54.556 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:54.556 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:54.556 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:54.556 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2024-09-15T03:32:54.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:54.557 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:54.558 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:54.559 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:54.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:54.559 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:54.559 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:54.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:54.560 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:54.561 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:54.562 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:54.563 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:54.563 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:54.563 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:54.563 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:54.563 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:54.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:54.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:54.567 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:54.569 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.570 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:54.571 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:54.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.573 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.573 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:54.573 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:54.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f6c038740 msgr2=0x7f6f6c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6f6c038740 0x7f6f6c03abf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6f8809bc90 tx=0x7f6f80037000 comp rx=0 tx=0).stop 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 10 monclient: shutdown 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 msgr2=0x7f6f8813aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 --2- 172.21.15.153:0/1275069498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f8809a540 0x7f6f8813aa30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400d890 tx=0x7f6f74008040 comp rx=0 tx=0).stop 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/1275069498 reap_dead start 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8d4f1640 1 -- 172.21.15.153:0/1275069498 reap_dead start 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 shutdown_connections 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 >> 172.21.15.153:0/1275069498 conn(0x7f6f88067cd0 msgr2=0x7f6f880a1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 shutdown_connections 2024-09-15T03:32:54.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.552+0000 7f6f8ef54640 1 -- 172.21.15.153:0/1275069498 wait complete. 2024-09-15T03:32:54.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:54.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:32:54.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:32:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.678+0000 7f4c8d82b640 10 monclient: get_monmap_and_config 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.678+0000 7f4c8d82b640 10 monclient: build_initial_monmap 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.678+0000 7f4c8d82b640 10 monclient: monmap: 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:54.679171+0000 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:54.679171+0000 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.678+0000 7f4c8d82b640 1 Processor -- start 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 1 -- start start 2024-09-15T03:32:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient: init 2024-09-15T03:32:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T03:32:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c88131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c8809e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4c88137bc0 0x7f4c88139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient(hunting): picked mon.noname-a con 0x7f4c88137bc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8809faf0 con 0x7f4c88131ab0 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8809fd30 con 0x7f4c8809e910 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c88132590 con 0x7f4c88137bc0 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c8d82b640 10 monclient(hunting): authenticate will time out at 972.318598s 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c86ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4c88131ab0 0x7f4c88131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38260/0 (socket says 172.21.15.153:38260) 2024-09-15T03:32:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c86ffd640 1 -- 172.21.15.153:0/158391508 learned_addr learned my addr 172.21.15.153:0/158391508 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 1 --2- 172.21.15.153:0/158391508 >> v2:172.21.15.153:3300/0 conn(0x7f4c88137bc0 0x7f4c88139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): get_auth_request con 0x7f4c88137bc0 auth_method 0 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.679+0000 7f4c877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 10 monclient(hunting): handle_auth_done global_id 64401 payload 274 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 1 -- 172.21.15.153:0/158391508 >> v1:172.21.15.181:6789/0 conn(0x7f4c8809e910 legacy=0x7f4c8809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 1 -- 172.21.15.153:0/158391508 >> v1:172.21.15.153:6789/0 conn(0x7f4c88131ab0 legacy=0x7f4c88131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.680+0000 7f4c877fe640 1 -- 172.21.15.153:0/158391508 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880a1110 con 0x7f4c88137bc0 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.682+0000 7f4c877fe640 1 -- 172.21.15.153:0/158391508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 msgr2=0x7f4c88139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.682+0000 7f4c877fe640 1 --2- 172.21.15.153:0/158391508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88139fa0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009880 tx=0x7f4c780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baae1dda8f9e3d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 1 -- 172.21.15.153:0/158391508 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78032440 con 0x7f4c88137bc0 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: dump: 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: _finish_auth 0 2024-09-15T03:32:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 1 -- 172.21.15.153:0/158391508 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c780325a0 con 0x7f4c88137bc0 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 5 monclient: authenticate success, global_id 64401 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 1 -- 172.21.15.153:0/158391508 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78032940 con 0x7f4c88137bc0 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 4 set_mon_vals no callback set 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c85ffb640 10 monclient: dump: 2024-09-15T03:32:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 10 monclient: shutdown 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:54.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 -- 172.21.15.153:0/158391508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 msgr2=0x7f4c88139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 --2- 172.21.15.153:0/158391508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88139fa0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009880 tx=0x7f4c780090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 -- 172.21.15.153:0/158391508 shutdown_connections 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 --2- 172.21.15.153:0/158391508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88139fa0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 -- 172.21.15.153:0/158391508 >> 172.21.15.153:0/158391508 conn(0x7f4c88067d30 msgr2=0x7f4c880a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 -- 172.21.15.153:0/158391508 shutdown_connections 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.683+0000 7f4c8d82b640 1 -- 172.21.15.153:0/158391508 wait complete. 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.684+0000 7f4c8d82b640 10 monclient: build_initial_monmap 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.684+0000 7f4c8d82b640 10 monclient: monmap: 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.684+0000 7f4c8d82b640 1 Processor -- start 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.684+0000 7f4c8d82b640 1 -- start start 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.684+0000 7f4c8d82b640 10 monclient: init 2024-09-15T03:32:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c8809e910 0x7f4c8809b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): picked mon.noname-b con 0x7f4c8809e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c88131ab0 0x7f4c88096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): picked mon.noname-c con 0x7f4c88131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): picked mon.noname-a con 0x7f4c88137bc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8809a300 con 0x7f4c88137bc0 2024-09-15T03:32:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8809bde0 con 0x7f4c8809e910 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c880a54d0 con 0x7f4c88131ab0 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c8d82b640 10 monclient(hunting): authenticate will time out at 972.324278s 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53696/0 (socket says 172.21.15.153:53696) 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 learned_addr learned my addr 172.21.15.153:0/4131944381 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): get_auth_request con 0x7f4c88137bc0 auth_method 0 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c88131ab0 0x7f4c88096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): get_auth_request con 0x7f4c88131ab0 auth_method 0 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c8809e910 0x7f4c8809b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): get_auth_request con 0x7f4c8809e910 auth_method 0 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient(hunting): handle_auth_done global_id 64404 payload 824 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c88131ab0 msgr2=0x7f4c88096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c88131ab0 0x7f4c88096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c8809e910 msgr2=0x7f4c8809b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c8809e910 0x7f4c8809b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c78009390 con 0x7f4c88137bc0 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.685+0000 7f4c867fc640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c88131ab0 0x7f4c88096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.687+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 msgr2=0x7f4c88096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.687+0000 7f4c877fe640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88096fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c780097d0 tx=0x7f4c78032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78031c70 con 0x7f4c88137bc0 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: dump: 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.711 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: _finish_auth 0 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 5 monclient: authenticate success, global_id 64404 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c78031dd0 con 0x7f4c88137bc0 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 10 monclient: _renew_subs 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c88098020 con 0x7f4c88137bc0 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7803b610 con 0x7f4c88137bc0 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c7effd640 10 monclient: dump: 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c857fa640 4 set_mon_vals no callback set 2024-09-15T03:32:54.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 10 monclient: _renew_subs 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.688+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8809aa90 con 0x7f4c88137bc0 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c8d82b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c8d82b640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 0x7f4c880974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c8d82b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88063740 con 0x7f4c880986e0 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4c78039040 con 0x7f4c88137bc0 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 0x7f4c880974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient: get_auth_request con 0x7f4c880986e0 auth_method 0 2024-09-15T03:32:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c7effd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c68038740 0x7f4c6803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f4c7806c4b0 con 0x7f4c88137bc0 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_done global_id 44965 payload 824 2024-09-15T03:32:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c86ffd640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c88099c30 con 0x7f4c880986e0 2024-09-15T03:32:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c867fc640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c68038740 0x7f4c6803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.689+0000 7f4c867fc640 10 monclient: get_auth_request con 0x7f4c68038740 auth_method 0 2024-09-15T03:32:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.691+0000 7f4c86ffd640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 msgr2=0x7f4c880974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.691+0000 7f4c86ffd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 0x7f4c880974e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c800079a0 tx=0x7f4c80005330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.691+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80007a90 con 0x7f4c880986e0 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.691+0000 7f4c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.691+0000 7f4c7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.691+0000 7f4c7effd640 10 monclient: dump: 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c867fc640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c68038740 0x7f4c6803abf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4c881372e0 tx=0x7f4c70007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4c8000fd50 con 0x7f4c880986e0 2024-09-15T03:32:54.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:54.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:54.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 msgr2=0x7f4c880974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 0x7f4c880974e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c800079a0 tx=0x7f4c80005330 comp rx=0 tx=0).stop 2024-09-15T03:32:54.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 msgr2=0x7f4c880974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:54.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.692+0000 7f4c7effd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c880986e0 0x7f4c880974e0 secure :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c800079a0 tx=0x7f4c80005330 comp rx=0 tx=0).stop 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.697+0000 7f4c8d82b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.697+0000 7f4c8d82b640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 0x7f4c88132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.697+0000 7f4c8d82b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.697+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88099830 con 0x7f4c881326d0 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 0x7f4c88132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient: get_auth_request con 0x7f4c881326d0 auth_method 0 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 10 monclient(hunting): handle_auth_done global_id 44968 payload 824 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.698+0000 7f4c86ffd640 1 -- 172.21.15.153:0/4131944381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c88097b70 con 0x7f4c881326d0 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.700+0000 7f4c86ffd640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 msgr2=0x7f4c88132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.700+0000 7f4c86ffd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 0x7f4c88132ad0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c80017000 tx=0x7f4c80018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.700+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80005ea0 con 0x7f4c881326d0 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.700+0000 7f4c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.700+0000 7f4c7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.700+0000 7f4c7effd640 10 monclient: dump: 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.725 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f4c80021880 con 0x7f4c881326d0 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 msgr2=0x7f4c88132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 0x7f4c88132ad0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c80017000 tx=0x7f4c80018000 comp rx=0 tx=0).stop 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 msgr2=0x7f4c88132ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c7effd640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c881326d0 0x7f4c88132ad0 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c80017000 tx=0x7f4c80018000 comp rx=0 tx=0).stop 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:54.733 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.734 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:54.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:54.736 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:54.737 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:54.738 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.739 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:54.740 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:54.741 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:54.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c68038740 msgr2=0x7f4c6803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c68038740 0x7f4c6803abf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4c881372e0 tx=0x7f4c70007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:54.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 10 monclient: shutdown 2024-09-15T03:32:54.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:54.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 msgr2=0x7f4c88096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 --2- 172.21.15.153:0/4131944381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c88137bc0 0x7f4c88096fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c780097d0 tx=0x7f4c78032000 comp rx=0 tx=0).stop 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 reap_dead start 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c877fe640 1 -- 172.21.15.153:0/4131944381 reap_dead start 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 shutdown_connections 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 >> 172.21.15.153:0/4131944381 conn(0x7f4c88067d30 msgr2=0x7f4c8809e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 shutdown_connections 2024-09-15T03:32:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.701+0000 7f4c8d82b640 1 -- 172.21.15.153:0/4131944381 wait complete. 2024-09-15T03:32:54.745 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:54.745 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:32:54.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:32:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.839+0000 7f3b892dd640 10 monclient: get_monmap_and_config 2024-09-15T03:32:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.839+0000 7f3b892dd640 10 monclient: build_initial_monmap 2024-09-15T03:32:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.839+0000 7f3b892dd640 10 monclient: monmap: 2024-09-15T03:32:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:54.839968+0000 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:54.839968+0000 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:54.841 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.839+0000 7f3b892dd640 1 Processor -- start 2024-09-15T03:32:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 1 -- start start 2024-09-15T03:32:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient: init 2024-09-15T03:32:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b841318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b8409ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3b84137c30 0x7f3b8413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient(hunting): picked mon.noname-a con 0x7f3b84137c30 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b84131cc0 con 0x7f3b8409ea50 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8409f530 con 0x7f3b841318f0 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8409fbf0 con 0x7f3b84137c30 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b892dd640 10 monclient(hunting): authenticate will time out at 972.479583s 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3b84137c30 0x7f3b8413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b7bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3b8409ea50 0x7f3b8409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38270/0 (socket says 172.21.15.153:38270) 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b7bfff640 1 -- 172.21.15.153:0/1376540540 learned_addr learned my addr 172.21.15.153:0/1376540540 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): get_auth_request con 0x7f3b84137c30 auth_method 0 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient(hunting): handle_auth_done global_id 64407 payload 274 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 1 -- 172.21.15.153:0/1376540540 >> v1:172.21.15.181:6789/0 conn(0x7f3b841318f0 legacy=0x7f3b84131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 1 -- 172.21.15.153:0/1376540540 >> v1:172.21.15.153:6789/0 conn(0x7f3b8409ea50 legacy=0x7f3b8409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.840+0000 7f3b837fe640 1 -- 172.21.15.153:0/1376540540 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8409fe30 con 0x7f3b84137c30 2024-09-15T03:32:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.843+0000 7f3b837fe640 1 -- 172.21.15.153:0/1376540540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 msgr2=0x7f3b8413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.843+0000 7f3b837fe640 1 --2- 172.21.15.153:0/1376540540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b8413a010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009e60 tx=0x7f3b7c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9522dbc9f61952 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 1 -- 172.21.15.153:0/1376540540 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c007630 con 0x7f3b84137c30 2024-09-15T03:32:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: dump: 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 1 -- 172.21.15.153:0/1376540540 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7c02d050 con 0x7f3b84137c30 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 1 -- 172.21.15.153:0/1376540540 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c0079c0 con 0x7f3b84137c30 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 5 monclient: authenticate success, global_id 64407 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 4 set_mon_vals no callback set 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b827fc640 10 monclient: dump: 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 10 monclient: shutdown 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 -- 172.21.15.153:0/1376540540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 msgr2=0x7f3b8413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 --2- 172.21.15.153:0/1376540540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b8413a010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009e60 tx=0x7f3b7c009780 comp rx=0 tx=0).stop 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 -- 172.21.15.153:0/1376540540 shutdown_connections 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 --2- 172.21.15.153:0/1376540540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b8413a010 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 -- 172.21.15.153:0/1376540540 >> 172.21.15.153:0/1376540540 conn(0x7f3b84067110 msgr2=0x7f3b84067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 -- 172.21.15.153:0/1376540540 shutdown_connections 2024-09-15T03:32:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.844+0000 7f3b892dd640 1 -- 172.21.15.153:0/1376540540 wait complete. 2024-09-15T03:32:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.845+0000 7f3b892dd640 10 monclient: build_initial_monmap 2024-09-15T03:32:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.845+0000 7f3b892dd640 10 monclient: monmap: 2024-09-15T03:32:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.845+0000 7f3b892dd640 1 Processor -- start 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.845+0000 7f3b892dd640 1 -- start start 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.845+0000 7f3b892dd640 10 monclient: init 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8409ea50 0x7f3b84136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): picked mon.noname-b con 0x7f3b8409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 0x7f3b84096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): picked mon.noname-c con 0x7f3b841318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b84096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): picked mon.noname-a con 0x7f3b84137c30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b84099c50 con 0x7f3b84137c30 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8409bf10 con 0x7f3b8409ea50 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8409d7c0 con 0x7f3b841318f0 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b892dd640 10 monclient(hunting): authenticate will time out at 972.485130s 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b84096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 0x7f3b84096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b84096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53714/0 (socket says 172.21.15.153:53714) 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 0x7f3b84096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58122/0 (socket says 172.21.15.153:58122) 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 -- 172.21.15.153:0/3142194679 learned_addr learned my addr 172.21.15.153:0/3142194679 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): get_auth_request con 0x7f3b84137c30 auth_method 0 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): get_auth_request con 0x7f3b841318f0 auth_method 0 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8409ea50 0x7f3b84136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): get_auth_request con 0x7f3b8409ea50 auth_method 0 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient(hunting): handle_auth_done global_id 64410 payload 824 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 -- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 msgr2=0x7f3b84096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 0x7f3b84096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 -- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8409ea50 msgr2=0x7f3b84136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8409ea50 0x7f3b84136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient: _finish_hunting 0 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 monclient: found mon.noname-a 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b837fe640 1 -- 172.21.15.153:0/3142194679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c009a20 con 0x7f3b84137c30 2024-09-15T03:32:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b7bfff640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 0x7f3b84096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.846+0000 7f3b82ffd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8409ea50 0x7f3b84136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b837fe640 1 -- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 msgr2=0x7f3b84096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b837fe640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b84096fa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0099f0 tx=0x7f3b7c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c031ac0 con 0x7f3b84137c30 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 10 monclient: dump: 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 10 monclient: _finish_auth 0 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7c040070 con 0x7f3b84137c30 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b892dd640 5 monclient: authenticate success, global_id 64410 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b892dd640 10 monclient: _renew_subs 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c031e50 con 0x7f3b84137c30 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b892dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b84098070 con 0x7f3b84137c30 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.848+0000 7f3b81ffb640 4 set_mon_vals no callback set 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b7affd640 10 monclient: dump: 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b892dd640 10 monclient: _renew_subs 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b892dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8409aa50 con 0x7f3b84137c30 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3b7c042030 con 0x7f3b84137c30 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b7affd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b64038740 0x7f3b6403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.849+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f3b7c06d8d0 con 0x7f3b84137c30 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.850+0000 7f3b892dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.850+0000 7f3b892dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.850+0000 7f3b892dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.850+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8404bf60 con 0x7f3b84137c30 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.850+0000 7f3b82ffd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b64038740 0x7f3b6403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.850+0000 7f3b82ffd640 10 monclient: get_auth_request con 0x7f3b64038740 auth_method 0 2024-09-15T03:32:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.852+0000 7f3b82ffd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b64038740 0x7f3b6403abf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3b74007fd0 tx=0x7f3b74007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.854+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3b7c03b5e0 con 0x7f3b84137c30 2024-09-15T03:32:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.854+0000 7f3b7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:54.854+0000 7f3b7affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.070+0000 7f3b892dd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:32:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.070+0000 7f3b892dd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:32:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.070+0000 7f3b892dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.070+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3b8404bf60 con 0x7f3b84137c30 2024-09-15T03:32:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.071+0000 7f3b7affd640 1 -- 172.21.15.153:0/3142194679 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3b7c03a3a0 con 0x7f3b84137c30 2024-09-15T03:32:55.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.071+0000 7f3b7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:32:55.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.071+0000 7f3b7affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.072+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.072+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=1824141720,config=3140514009,health=1361790972,kv=3235807718,logm=2612601905}) 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.072+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=1436101693,config=3140514009,health=1361790972,kv=3235807718,logm=1195507515}) 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.072+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.072+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=1824141720,config=3140514009,health=1361790972,kv=3235807718,logm=2612601905}) 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.072+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=1,health=11,kv=7,logm=67} crc {auth=1824141720,config=1228700967,health=3602034646,kv=1382144082,logm=3290342847}) 2024-09-15T03:32:55.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.073+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.073+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819235826}) 2024-09-15T03:32:55.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.073+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2176376787}) 2024-09-15T03:32:55.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.073+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.073+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819235826}) 2024-09-15T03:32:55.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.073+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=807143061}) 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808948730}) 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437493925}) 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808948730}) 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865913371}) 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354129061}) 2024-09-15T03:32:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b64038740 msgr2=0x7f3b6403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b64038740 0x7f3b6403abf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3b74007fd0 tx=0x7f3b74007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 10 monclient: shutdown 2024-09-15T03:32:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:55.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3251948380}) 2024-09-15T03:32:55.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 msgr2=0x7f3b84096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b84096fa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0099f0 tx=0x7f3b7c037000 comp rx=0 tx=0).stop 2024-09-15T03:32:55.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354129061}) 2024-09-15T03:32:55.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481170010}) 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 shutdown_connections 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b64038740 0x7f3b6403abf0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b84137c30 0x7f3b84096fa0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b841318f0 0x7f3b84096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.074+0000 7f3b892dd640 1 --2- 172.21.15.153:0/3142194679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8409ea50 0x7f3b84136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 >> 172.21.15.153:0/3142194679 conn(0x7f3b84067110 msgr2=0x7f3b840a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 shutdown_connections 2024-09-15T03:32:55.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f3b892dd640 1 -- 172.21.15.153:0/3142194679 wait complete. 2024-09-15T03:32:55.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408170036}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1575571575}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408170036}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.075+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511371438}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.076+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.076+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924018239}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.076+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076854525}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.076+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.076+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924018239}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.076+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2956241268}) 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.077+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.085 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.077+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976518307}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.077+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626668444}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.077+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.077+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976518307}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.077+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792075580}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.078+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.078+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695855053}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.078+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=359144367}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.078+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.078+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695855053}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.078+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590434941}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.079+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.079+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884410601}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.079+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002882327}) 2024-09-15T03:32:55.086 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.079+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.079+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884410601}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.079+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626112373}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.080+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.080+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042711416}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.080+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4010403416}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.080+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.080+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042711416}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.080+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3851198095}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1831409472}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1306706853}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1831409472}) 2024-09-15T03:32:55.087 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109485770}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.081+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286211412}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3594510052}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286211412}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1120413899}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511761891}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3277969436}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.082+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511761891}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880322979}) 2024-09-15T03:32:55.088 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.089 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759633078}) 2024-09-15T03:32:55.089 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2213180305}) 2024-09-15T03:32:55.089 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.089 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759633078}) 2024-09-15T03:32:55.089 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.083+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067086576}) 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.084+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.084+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23868827}) 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.084+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350894863}) 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.084+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.084+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23868827}) 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.084+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078614393}) 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.085+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.085+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=1754720439,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2087550717,mon_config_key=4117641549,monmap=626991505,osd_metadata=851455416,osd_pg_creating=1611600085,osd_snap=454132730,osdmap=534172571}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.086+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=52,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=852927847,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1961931344,mon_config_key=4117641549,monmap=244636554,osd_metadata=1491145534,osd_pg_creating=1611600085,osd_snap=1121295165,osdmap=2800334780}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.086+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.086+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=1754720439,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2087550717,mon_config_key=4117641549,monmap=626991505,osd_metadata=851455416,osd_pg_creating=1611600085,osd_snap=454132730,osdmap=534172571}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.086+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=8,mgrstat=8,mon_config_key=5,monmap=1,osd_metadata=6,osd_snap=2,osdmap=15} crc {logm=1222768165,mdsmap=1531151339,mgr=2938112866,mgrstat=2706660553,mon_config_key=4117641549,monmap=1228700967,osd_metadata=851455416,osd_snap=454132730,osdmap=1490971763}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.087+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.087+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=80647150}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.087+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=780523759}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.087+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.087+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=80647150}) 2024-09-15T03:32:55.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.087+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2027993157}) 2024-09-15T03:32:55.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:32:55.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.089+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.089+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=2202029914}) 2024-09-15T03:32:55.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.089+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=55} crc {osdmap=514897589}) 2024-09-15T03:32:55.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.089+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:32:55.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.089+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=2202029914}) 2024-09-15T03:32:55.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:32:55.089+0000 7f7635bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=55} crc {osdmap=3347998955}) 2024-09-15T03:32:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.447+0000 7f3281ffb640 1 -- 172.21.15.181:0/1975743567 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v106) v1 ==== 58+0+3 (secure 0 0 0) 0x7f327c017020 con 0x7f328c06be40 2024-09-15T03:32:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.447+0000 7f3281ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T03:32:55.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.447+0000 7f3281ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3270038740 msgr2=0x7f327003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3270038740 0x7f327003abf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3288006fd0 tx=0x7f3288008040 comp rx=0 tx=0).stop 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 10 monclient: shutdown 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 msgr2=0x7f328c13cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 0x7f328c13cd80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c610 tx=0x7f327c004960 comp rx=0 tx=0).stop 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 shutdown_connections 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3270038740 0x7f327003abf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a37b0 0x7f328c140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f328c06be40 0x7f328c13cd80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 --2- 172.21.15.181:0/1975743567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c06a6f0 0x7f328c13d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 >> 172.21.15.181:0/1975743567 conn(0x7f328c070fd0 msgr2=0x7f328c06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:55.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 shutdown_connections 2024-09-15T03:32:55.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.449+0000 7f3293d9d640 1 -- 172.21.15.181:0/1975743567 wait complete. 2024-09-15T03:32:55.461 INFO:tasks.workunit.client.0.smithi181.stderr:+ o4=12 2024-09-15T03:32:55.461 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.533+0000 7fc61911d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.533+0000 7fc61911d640 10 monclient: build_initial_monmap 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.533+0000 7fc61911d640 10 monclient: monmap: 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:32:55.534755+0000 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:32:55.534755+0000 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:55.535 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:55.536 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:55.536 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:55.536 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.534+0000 7fc61911d640 1 Processor -- start 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.534+0000 7fc61911d640 1 -- start start 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.534+0000 7fc61911d640 10 monclient: init 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 0x7fc61409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): picked mon.noname-a con 0x7fc614131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc61409f240 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): picked mon.noname-b con 0x7fc614136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61406a230 con 0x7fc61409f240 2024-09-15T03:32:55.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140a0300 con 0x7fc614131b40 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140a00c0 con 0x7fc614136e70 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61911d640 10 monclient(hunting): authenticate will time out at 980.016166s 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 0x7fc61409ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc61409f240 0x7fc614131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:43290/0 (socket says 172.21.15.181:43290) 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 0x7fc61409ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:41818/0 (socket says 172.21.15.181:41818) 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61259c640 1 -- 172.21.15.181:0/3317624567 learned_addr learned my addr 172.21.15.181:0/3317624567 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:36174/0 (socket says 172.21.15.181:36174) 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): get_auth_request con 0x7fc614136e70 auth_method 0 2024-09-15T03:32:55.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): get_auth_request con 0x7fc614131b40 auth_method 0 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:55.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient(hunting): handle_auth_done global_id 74111 payload 274 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 -- 172.21.15.181:0/3317624567 >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 msgr2=0x7fc61409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 --2- 172.21.15.181:0/3317624567 >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 0x7fc61409ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 -- 172.21.15.181:0/3317624567 >> v1:172.21.15.153:6789/0 conn(0x7fc61409f240 legacy=0x7fc614131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient: _finish_hunting 0 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 monclient: found mon.noname-b 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc61359e640 1 -- 172.21.15.181:0/3317624567 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61409bee0 con 0x7fc614136e70 2024-09-15T03:32:55.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.535+0000 7fc612d9d640 1 --2- 172.21.15.181:0/3317624567 >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 0x7fc61409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:55.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.538+0000 7fc61359e640 1 -- 172.21.15.181:0/3317624567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 msgr2=0x7fc614139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.538+0000 7fc61359e640 1 --2- 172.21.15.181:0/3317624567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614139250 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc604009e60 tx=0x7fc604009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94b0af1564e3e19 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.538+0000 7fc611d9b640 1 -- 172.21.15.181:0/3317624567 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604007760 con 0x7fc614136e70 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: dump: 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:55.550 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: _finish_auth 0 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 1 -- 172.21.15.181:0/3317624567 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60402d050 con 0x7fc614136e70 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 1 -- 172.21.15.181:0/3317624567 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604007af0 con 0x7fc614136e70 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 5 monclient: authenticate success, global_id 74111 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 4 set_mon_vals no callback set 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:55.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc611d9b640 10 monclient: dump: 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:55.552 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 10 monclient: shutdown 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 -- 172.21.15.181:0/3317624567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 msgr2=0x7fc614139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 --2- 172.21.15.181:0/3317624567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614139250 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc604009e60 tx=0x7fc604009780 comp rx=0 tx=0).stop 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 -- 172.21.15.181:0/3317624567 shutdown_connections 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 --2- 172.21.15.181:0/3317624567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614139250 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 --2- 172.21.15.181:0/3317624567 >> v2:172.21.15.153:3300/0 conn(0x7fc614131b40 0x7fc61409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 -- 172.21.15.181:0/3317624567 >> 172.21.15.181:0/3317624567 conn(0x7fc6140a5100 msgr2=0x7fc6140a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:55.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 -- 172.21.15.181:0/3317624567 shutdown_connections 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.539+0000 7fc61911d640 1 -- 172.21.15.181:0/3317624567 wait complete. 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.540+0000 7fc61911d640 10 monclient: build_initial_monmap 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.540+0000 7fc61911d640 10 monclient: monmap: 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:55.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.540+0000 7fc61911d640 1 Processor -- start 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.540+0000 7fc61911d640 1 -- start start 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.540+0000 7fc61911d640 10 monclient: init 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 0x7fc614141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): picked mon.noname-b con 0x7fc61409f240 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc614131b40 0x7fc614141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): picked mon.noname-a con 0x7fc614131b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614098730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): picked mon.noname-c con 0x7fc614136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc614130b60 con 0x7fc614131b40 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61411c020 con 0x7fc61409f240 2024-09-15T03:32:55.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61411cbf0 con 0x7fc614136e70 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61911d640 10 monclient(hunting): authenticate will time out at 980.022145s 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 0x7fc614141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 0x7fc614141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44556/0 (socket says 172.21.15.181:44556) 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 -- 172.21.15.181:0/1011238246 learned_addr learned my addr 172.21.15.181:0/1011238246 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): get_auth_request con 0x7fc61409f240 auth_method 0 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc614131b40 0x7fc614141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614098730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): get_auth_request con 0x7fc614131b40 auth_method 0 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): get_auth_request con 0x7fc614136e70 auth_method 0 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc61359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient(hunting): handle_auth_done global_id 44974 payload 824 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 -- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 msgr2=0x7fc614098730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614098730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 -- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc614131b40 msgr2=0x7fc614141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc614131b40 0x7fc614141a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 monclient: found mon.noname-b 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.541+0000 7fc612d9d640 1 -- 172.21.15.181:0/1011238246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c009390 con 0x7fc61409f240 2024-09-15T03:32:55.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc612d9d640 1 -- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 msgr2=0x7fc614141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc612d9d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 0x7fc614141510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00ec80 tx=0x7fc60c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c00eec0 con 0x7fc61409f240 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 10 monclient: dump: 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.559 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:55.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 10 monclient: _finish_auth 0 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c00cd50 con 0x7fc61409f240 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc61911d640 5 monclient: authenticate success, global_id 44974 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c013680 con 0x7fc61409f240 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc61911d640 10 monclient: _renew_subs 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc61911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc604009a20 con 0x7fc61409f240 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc60affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc60affd640 10 monclient: dump: 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:55.561 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.543+0000 7fc61159a640 4 set_mon_vals no callback set 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 10 monclient: _renew_subs 2024-09-15T03:32:55.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:55.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61411bd40 con 0x7fc61409f240 2024-09-15T03:32:55.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc60c0139f0 con 0x7fc61409f240 2024-09-15T03:32:55.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:55.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:55.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:55.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.544+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6140636e0 con 0x7fc61409f240 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.545+0000 7fc60affd640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5f4038740 0x7fc5f403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.545+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6123+0+0 (secure 0 0 0) 0x7fc60c04c3e0 con 0x7fc61409f240 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.545+0000 7fc61259c640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5f4038740 0x7fc5f403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.545+0000 7fc61259c640 10 monclient: get_auth_request con 0x7fc5f4038740 auth_method 0 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.547+0000 7fc61259c640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5f4038740 0x7fc5f403abf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc61409a5e0 tx=0x7fc5fc007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.549+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc60c0234d0 con 0x7fc61409f240 2024-09-15T03:32:55.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.549+0000 7fc60affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:55.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.549+0000 7fc60affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:55.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.714+0000 7fc61911d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:32:55.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.714+0000 7fc61911d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-15T03:32:55.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.714+0000 7fc61911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:55.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:55.714+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fc61409ae30 con 0x7fc61409f240 2024-09-15T03:32:56.094 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:32:56.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:56.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.198+0000 7fcb6f51d640 10 monclient: get_monmap_and_config 2024-09-15T03:32:56.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.198+0000 7fcb6f51d640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.198+0000 7fcb6f51d640 10 monclient: monmap: 2024-09-15T03:32:56.199 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.199 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:56.199369+0000 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:56.199369+0000 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:56.200 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 1 Processor -- start 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 1 -- start start 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient: init 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb68096e10 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb680971e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcb68134b40 0x7fcb68136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb68134b40 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6809d4a0 con 0x7fcb680971e0 2024-09-15T03:32:56.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb68099620 con 0x7fcb68096e10 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb68099340 con 0x7fcb68134b40 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.199+0000 7fcb6f51d640 10 monclient(hunting): authenticate will time out at 973.838968s 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6cab8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcb680971e0 0x7fcb680975e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41572/0 (socket says 172.21.15.153:41572) 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6cab8640 1 -- 172.21.15.153:0/8417556 learned_addr learned my addr 172.21.15.153:0/8417556 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 1 --2- 172.21.15.153:0/8417556 >> v2:172.21.15.181:3300/0 conn(0x7fcb68134b40 0x7fcb68136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): get_auth_request con 0x7fcb68134b40 auth_method 0 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient(hunting): handle_auth_done global_id 44977 payload 274 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 1 -- 172.21.15.153:0/8417556 >> v1:172.21.15.181:6789/0 conn(0x7fcb68096e10 legacy=0x7fcb6809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 1 -- 172.21.15.153:0/8417556 >> v1:172.21.15.153:6789/0 conn(0x7fcb680971e0 legacy=0x7fcb680975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 1 monclient: found mon.noname-c 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:56.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.200+0000 7fcb6daba640 1 -- 172.21.15.153:0/8417556 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68099900 con 0x7fcb68134b40 2024-09-15T03:32:56.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb6daba640 1 -- 172.21.15.153:0/8417556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb68134b40 msgr2=0x7fcb68136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb6daba640 1 --2- 172.21.15.153:0/8417556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb68134b40 0x7fcb68136f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcb68076be0 tx=0x7fcb60009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3794e04b3fd034a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 1 -- 172.21.15.153:0/8417556 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60007630 con 0x7fcb68134b40 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: dump: 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.212 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 1 -- 172.21.15.153:0/8417556 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6002d050 con 0x7fcb68134b40 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb6f51d640 5 monclient: authenticate success, global_id 44977 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb6f51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb6f51d640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb6f51d640 4 set_mon_vals no callback set 2024-09-15T03:32:56.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 1 -- 172.21.15.153:0/8417556 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb600079c0 con 0x7fcb68134b40 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.203+0000 7fcb5ffff640 10 monclient: dump: 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 10 monclient: shutdown 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 -- 172.21.15.153:0/8417556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb68134b40 msgr2=0x7fcb68136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/8417556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb68134b40 0x7fcb68136f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcb68076be0 tx=0x7fcb60009780 comp rx=0 tx=0).stop 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 -- 172.21.15.153:0/8417556 shutdown_connections 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/8417556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb68134b40 0x7fcb68136f20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 -- 172.21.15.153:0/8417556 >> 172.21.15.153:0/8417556 conn(0x7fcb680672b0 msgr2=0x7fcb680676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 -- 172.21.15.153:0/8417556 shutdown_connections 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 1 -- 172.21.15.153:0/8417556 wait complete. 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.204+0000 7fcb6f51d640 10 monclient: monmap: 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:56.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 Processor -- start 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 -- start start 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient: init 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb68096e10 0x7fcb6809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb68096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb680971e0 0x7fcb68132cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb680971e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 0x7fcb68133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb68134b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6809b050 con 0x7fcb68134b40 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6809e660 con 0x7fcb680971e0 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6806a650 con 0x7fcb68096e10 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.205+0000 7fcb6f51d640 10 monclient(hunting): authenticate will time out at 973.844940s 2024-09-15T03:32:56.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 0x7fcb68133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb68096e10 0x7fcb6809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 0x7fcb68133210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53666/0 (socket says 172.21.15.153:53666) 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 -- 172.21.15.153:0/2126755610 learned_addr learned my addr 172.21.15.153:0/2126755610 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): get_auth_request con 0x7fcb68134b40 auth_method 0 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): get_auth_request con 0x7fcb68096e10 auth_method 0 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb680971e0 0x7fcb68132cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): get_auth_request con 0x7fcb680971e0 auth_method 0 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient(hunting): handle_auth_done global_id 64419 payload 824 2024-09-15T03:32:56.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 -- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb68096e10 msgr2=0x7fcb6809b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb68096e10 0x7fcb6809b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 -- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb680971e0 msgr2=0x7fcb68132cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb680971e0 0x7fcb68132cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 monclient: found mon.noname-a 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6daba640 1 -- 172.21.15.153:0/2126755610 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb60009a20 con 0x7fcb68134b40 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.206+0000 7fcb6d2b9640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb68096e10 0x7fcb6809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6daba640 1 -- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 msgr2=0x7fcb68133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6daba640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 0x7fcb68133210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009de0 tx=0x7fcb60037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60031ac0 con 0x7fcb68134b40 2024-09-15T03:32:56.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: dump: 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb60040070 con 0x7fcb68134b40 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 5 monclient: authenticate success, global_id 64419 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 10 monclient: _renew_subs 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6806a2e0 con 0x7fcb68134b40 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60031e50 con 0x7fcb68134b40 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5d7fa640 10 monclient: dump: 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.222 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 10 monclient: _renew_subs 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6806f250 con 0x7fcb68134b40 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.208+0000 7fcb5f7fe640 4 set_mon_vals no callback set 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.209+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb60042030 con 0x7fcb68134b40 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.209+0000 7fcb6f51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.209+0000 7fcb6f51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.209+0000 7fcb6f51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.209+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6804bf60 con 0x7fcb68134b40 2024-09-15T03:32:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.210+0000 7fcb5d7fa640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.210+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6123+0+0 (secure 0 0 0) 0x7fcb6006d880 con 0x7fcb68134b40 2024-09-15T03:32:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.210+0000 7fcb6d2b9640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.210+0000 7fcb6d2b9640 10 monclient: get_auth_request con 0x7fcb4c038740 auth_method 0 2024-09-15T03:32:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.212+0000 7fcb6d2b9640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb64007fd0 tx=0x7fcb64007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.214+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcb60044bb0 con 0x7fcb68134b40 2024-09-15T03:32:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.214+0000 7fcb5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.214+0000 7fcb5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.431+0000 7fcb6f51d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.431+0000 7fcb6f51d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.431+0000 7fcb6f51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.432+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcb6804bf60 con 0x7fcb68134b40 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.432+0000 7fcb5d7fa640 1 -- 172.21.15.153:0/2126755610 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fcb600443d0 con 0x7fcb68134b40 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.432+0000 7fcb5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.432+0000 7fcb5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:56.433 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 msgr2=0x7fcb4c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb64007fd0 tx=0x7fcb64007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 10 monclient: shutdown 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 msgr2=0x7fcb68133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 0x7fcb68133210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009de0 tx=0x7fcb60037000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 shutdown_connections 2024-09-15T03:32:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb68134b40 0x7fcb68133210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb680971e0 0x7fcb68132cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 --2- 172.21.15.153:0/2126755610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb68096e10 0x7fcb6809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 >> 172.21.15.153:0/2126755610 conn(0x7fcb680672b0 msgr2=0x7fcb68129610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 shutdown_connections 2024-09-15T03:32:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.434+0000 7fcb6f51d640 1 -- 172.21.15.153:0/2126755610 wait complete. 2024-09-15T03:32:56.445 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:32:56.446 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:32:56.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:32:56.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:32:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.452+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v107) v1 ==== 58+0+3 (secure 0 0 0) 0x7fc60c0232f0 con 0x7fc61409f240 2024-09-15T03:32:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.452+0000 7fc60affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-15T03:32:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.452+0000 7fc60affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:56.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.540+0000 7ff058754640 10 monclient: get_monmap_and_config 2024-09-15T03:32:56.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.540+0000 7ff058754640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.540+0000 7ff058754640 10 monclient: monmap: 2024-09-15T03:32:56.541 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.541 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:56.540930+0000 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:56.540930+0000 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.540+0000 7ff058754640 1 Processor -- start 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.540+0000 7ff058754640 1 -- start start 2024-09-15T03:32:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.540+0000 7ff058754640 10 monclient: init 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff0501318f0 0x7ff050131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): picked mon.noname-c con 0x7ff0501318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff05009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 0x7ff050139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): picked mon.noname-b con 0x7ff050136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050131cc0 con 0x7ff05009ea50 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05009f530 con 0x7ff0501318f0 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05009fbf0 con 0x7ff050136c20 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff058754640 10 monclient(hunting): authenticate will time out at 974.180455s 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 0x7ff050139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 0x7ff050139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43392/0 (socket says 172.21.15.153:43392) 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 -- 172.21.15.153:0/3294161888 learned_addr learned my addr 172.21.15.153:0/3294161888 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): get_auth_request con 0x7ff050136c20 auth_method 0 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 1 --2- 172.21.15.153:0/3294161888 >> v2:172.21.15.181:3300/0 conn(0x7ff0501318f0 0x7ff050131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): get_auth_request con 0x7ff0501318f0 auth_method 0 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient(hunting): handle_auth_done global_id 74120 payload 274 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 -- 172.21.15.153:0/3294161888 >> v2:172.21.15.181:3300/0 conn(0x7ff0501318f0 msgr2=0x7ff050131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 --2- 172.21.15.153:0/3294161888 >> v2:172.21.15.181:3300/0 conn(0x7ff0501318f0 0x7ff050131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 -- 172.21.15.153:0/3294161888 >> v1:172.21.15.181:6789/0 conn(0x7ff05009ea50 legacy=0x7ff05009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 monclient: found mon.noname-b 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.541+0000 7ff056cf1640 1 -- 172.21.15.153:0/3294161888 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05009fe30 con 0x7ff050136c20 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff056cf1640 1 -- 172.21.15.153:0/3294161888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 msgr2=0x7ff050139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff056cf1640 1 --2- 172.21.15.153:0/3294161888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 0x7ff050139000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff040009910 tx=0x7ff0400091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2bf674add5ec0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 1 -- 172.21.15.153:0/3294161888 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04002aea0 con 0x7ff050136c20 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: dump: 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.556 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff058754640 5 monclient: authenticate success, global_id 74120 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 1 -- 172.21.15.153:0/3294161888 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040032440 con 0x7ff050136c20 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff058754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:56.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff058754640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 1 -- 172.21.15.153:0/3294161888 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0400327e0 con 0x7ff050136c20 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff058754640 4 set_mon_vals no callback set 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.544+0000 7ff0554ee640 10 monclient: dump: 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.558 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 10 monclient: shutdown 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 -- 172.21.15.153:0/3294161888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 msgr2=0x7ff050139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 --2- 172.21.15.153:0/3294161888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 0x7ff050139000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff040009910 tx=0x7ff0400091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 -- 172.21.15.153:0/3294161888 shutdown_connections 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 --2- 172.21.15.153:0/3294161888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050136c20 0x7ff050139000 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 --2- 172.21.15.153:0/3294161888 >> v2:172.21.15.181:3300/0 conn(0x7ff0501318f0 0x7ff050131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 -- 172.21.15.153:0/3294161888 >> 172.21.15.153:0/3294161888 conn(0x7ff050067110 msgr2=0x7ff050067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 -- 172.21.15.153:0/3294161888 shutdown_connections 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 1 -- 172.21.15.153:0/3294161888 wait complete. 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.545+0000 7ff058754640 10 monclient: monmap: 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:56.560 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 Processor -- start 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 -- start start 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient: init 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff05009ea50 0x7ff05011daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): picked mon.noname-c con 0x7ff05009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0501318f0 0x7ff050130760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): picked mon.noname-a con 0x7ff0501318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff050136c20 0x7ff050099590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): picked mon.noname-b con 0x7ff050136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05009dd10 con 0x7ff0501318f0 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05009bf10 con 0x7ff050136c20 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05009b830 con 0x7ff05009ea50 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff058754640 10 monclient(hunting): authenticate will time out at 974.185618s 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff05009ea50 0x7ff05011daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff05009ea50 0x7ff05011daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43398/0 (socket says 172.21.15.153:43398) 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 1 -- 172.21.15.153:0/3125880864 learned_addr learned my addr 172.21.15.153:0/3125880864 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0501318f0 0x7ff050130760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): get_auth_request con 0x7ff05009ea50 auth_method 0 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): get_auth_request con 0x7ff0501318f0 auth_method 0 2024-09-15T03:32:56.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff050136c20 0x7ff050099590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): get_auth_request con 0x7ff050136c20 auth_method 0 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff056cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient(hunting): handle_auth_done global_id 64422 payload 824 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff05009ea50 msgr2=0x7ff05011daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff05009ea50 0x7ff05011daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff050136c20 msgr2=0x7ff050099590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff050136c20 0x7ff050099590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 monclient: found mon.noname-a 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.546+0000 7ff055cef640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040009450 con 0x7ff0501318f0 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.547+0000 7ff0564f0640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff05009ea50 0x7ff05011daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff055cef640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0501318f0 msgr2=0x7ff050130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff055cef640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0501318f0 0x7ff050130760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c860 tx=0x7ff044002630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04400a850 con 0x7ff0501318f0 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: dump: 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 5 monclient: authenticate success, global_id 64422 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04400a9b0 con 0x7ff0501318f0 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 10 monclient: _renew_subs 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05009b070 con 0x7ff0501318f0 2024-09-15T03:32:56.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04400ad70 con 0x7ff0501318f0 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff04e7fc640 10 monclient: dump: 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.569 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff054ced640 4 set_mon_vals no callback set 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 10 monclient: _renew_subs 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.549+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff050130ca0 con 0x7ff0501318f0 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff058754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff058754640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 0x7ff0500298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05011c9b0 con 0x7ff05009a290 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 0x7ff0500298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient: get_auth_request con 0x7ff05009a290 auth_method 0 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff044012630 con 0x7ff0501318f0 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 10 monclient(hunting): handle_auth_done global_id 64425 payload 824 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff0564f0640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff05011cb20 con 0x7ff05009a290 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff04e7fc640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff03c038740 0x7ff03c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (secure 0 0 0) 0x7ff04404c420 con 0x7ff0501318f0 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff056cf1640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff03c038740 0x7ff03c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.550+0000 7ff056cf1640 10 monclient: get_auth_request con 0x7ff03c038740 auth_method 0 2024-09-15T03:32:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.559+0000 7fc61911d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-15T03:32:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.559+0000 7fc61911d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-15T03:32:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.559+0000 7fc61911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:32:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:56.559+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fc614072640 con 0x7fc61409f240 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.552+0000 7ff0564f0640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 msgr2=0x7ff0500298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.552+0000 7ff0564f0640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 0x7ff0500298c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff048009f60 tx=0x7ff0480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.552+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048014860 con 0x7ff05009a290 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.552+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.552+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.552+0000 7ff04e7fc640 10 monclient: dump: 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.575 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff0480149c0 con 0x7ff05009a290 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 msgr2=0x7ff0500298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 0x7ff0500298c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff048009f60 tx=0x7ff0480090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 msgr2=0x7ff0500298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff04e7fc640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a290 0x7ff0500298c0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff048009f60 tx=0x7ff0480090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.553+0000 7ff056cf1640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff03c038740 0x7ff03c03abf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff05006f600 tx=0x7ff040032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff058754640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff058754640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 0x7ff050029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff058754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff050065e00 con 0x7ff05009a660 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 0x7ff050029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 10 monclient: get_auth_request con 0x7ff05009a660 auth_method 0 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.558+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.559+0000 7ff0564f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.559+0000 7ff0564f0640 10 monclient(hunting): handle_auth_done global_id 64428 payload 824 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.559+0000 7ff0564f0640 1 -- 172.21.15.153:0/3125880864 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff050130f80 con 0x7ff05009a660 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff0564f0640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 msgr2=0x7ff050029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff0564f0640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 0x7ff050029e00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff04801b000 tx=0x7ff04801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048024040 con 0x7ff05009a660 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 10 monclient: dump: 2024-09-15T03:32:56.578 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7ff0480133a0 con 0x7ff05009a660 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 msgr2=0x7ff050029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 0x7ff050029e00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff04801b000 tx=0x7ff04801a000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 msgr2=0x7ff050029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.561+0000 7ff04e7fc640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff05009a660 0x7ff050029e00 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff04801b000 tx=0x7ff04801a000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:56.580 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.581 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.582 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:56.583 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.584 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:56.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:56.588 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:56.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.590 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.591 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.592 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:56.593 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:56.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff03c038740 msgr2=0x7ff03c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff03c038740 0x7ff03c03abf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff05006f600 tx=0x7ff040032000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 10 monclient: shutdown 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0501318f0 msgr2=0x7ff050130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 --2- 172.21.15.153:0/3125880864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0501318f0 0x7ff050130760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c860 tx=0x7ff044002630 comp rx=0 tx=0).stop 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff056cf1640 1 -- 172.21.15.153:0/3125880864 reap_dead start 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff056cf1640 1 -- 172.21.15.153:0/3125880864 reap_dead start 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 shutdown_connections 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 >> 172.21.15.153:0/3125880864 conn(0x7ff050067110 msgr2=0x7ff0500a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 shutdown_connections 2024-09-15T03:32:56.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.562+0000 7ff058754640 1 -- 172.21.15.153:0/3125880864 wait complete. 2024-09-15T03:32:56.596 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:56.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:32:56.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.691+0000 7fea6aec0640 10 monclient: get_monmap_and_config 2024-09-15T03:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.691+0000 7fea6aec0640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.691+0000 7fea6aec0640 10 monclient: monmap: 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:56.692353+0000 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:56.692353+0000 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:56.695 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 Processor -- start 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 -- start start 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient: init 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea64096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): picked mon.noname-b con 0x7fea6409a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea640971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fea640975b0 0x7fea6412fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): picked mon.noname-c con 0x7fea640975b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6406ee10 con 0x7fea640971e0 2024-09-15T03:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6409cb20 con 0x7fea640975b0 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6409a8e0 con 0x7fea6409a510 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea6aec0640 10 monclient(hunting): authenticate will time out at 974.331839s 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea64096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea64096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43404/0 (socket says 172.21.15.153:43404) 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 1 -- 172.21.15.153:0/2939014951 learned_addr learned my addr 172.21.15.153:0/2939014951 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 10 monclient(hunting): get_auth_request con 0x7fea6409a510 auth_method 0 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.692+0000 7fea68c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 1 --2- 172.21.15.153:0/2939014951 >> v2:172.21.15.181:3300/0 conn(0x7fea640975b0 0x7fea6412fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): get_auth_request con 0x7fea640975b0 auth_method 0 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea6945d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 10 monclient(hunting): handle_auth_done global_id 74126 payload 274 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 1 -- 172.21.15.153:0/2939014951 >> v2:172.21.15.181:3300/0 conn(0x7fea640975b0 msgr2=0x7fea6412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 1 --2- 172.21.15.153:0/2939014951 >> v2:172.21.15.181:3300/0 conn(0x7fea640975b0 0x7fea6412fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 1 -- 172.21.15.153:0/2939014951 >> v1:172.21.15.181:6789/0 conn(0x7fea640971e0 legacy=0x7fea6412f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 1 monclient: found mon.noname-b 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.693+0000 7fea68c5c640 1 -- 172.21.15.153:0/2939014951 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6409d590 con 0x7fea6409a510 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea68c5c640 1 -- 172.21.15.153:0/2939014951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 msgr2=0x7fea64096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea68c5c640 1 --2- 172.21.15.153:0/2939014951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea64096ca0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea58009910 tx=0x7fea580091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fd609f71afaa57b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 1 -- 172.21.15.153:0/2939014951 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea58032440 con 0x7fea6409a510 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: dump: 2024-09-15T03:32:56.704 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 5 monclient: authenticate success, global_id 74126 2024-09-15T03:32:56.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 1 -- 172.21.15.153:0/2939014951 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea580325a0 con 0x7fea6409a510 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 4 set_mon_vals no callback set 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 1 -- 172.21.15.153:0/2939014951 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea58032940 con 0x7fea6409a510 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea637fe640 10 monclient: dump: 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.706 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 10 monclient: shutdown 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 -- 172.21.15.153:0/2939014951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 msgr2=0x7fea64096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 --2- 172.21.15.153:0/2939014951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea64096ca0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea58009910 tx=0x7fea580091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 -- 172.21.15.153:0/2939014951 shutdown_connections 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 --2- 172.21.15.153:0/2939014951 >> v2:172.21.15.181:3300/0 conn(0x7fea640975b0 0x7fea6412fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 --2- 172.21.15.153:0/2939014951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea64096ca0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 -- 172.21.15.153:0/2939014951 >> 172.21.15.153:0/2939014951 conn(0x7fea640672b0 msgr2=0x7fea640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 -- 172.21.15.153:0/2939014951 shutdown_connections 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.696+0000 7fea6aec0640 1 -- 172.21.15.153:0/2939014951 wait complete. 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.697+0000 7fea6aec0640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.697+0000 7fea6aec0640 10 monclient: monmap: 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:56.708 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.697+0000 7fea6aec0640 1 Processor -- start 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.697+0000 7fea6aec0640 1 -- start start 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.697+0000 7fea6aec0640 10 monclient: init 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640971e0 0x7fea64131be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): picked mon.noname-b con 0x7fea640971e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea640975b0 0x7fea6413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): picked mon.noname-a con 0x7fea640975b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea6413f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): picked mon.noname-c con 0x7fea6409a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6413f7c0 con 0x7fea640975b0 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6409c370 con 0x7fea640971e0 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6411ede0 con 0x7fea6409a510 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6aec0640 10 monclient(hunting): authenticate will time out at 974.337216s 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea6413f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea640975b0 0x7fea6413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea6413f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43416/0 (socket says 172.21.15.153:43416) 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 -- 172.21.15.153:0/1545240825 learned_addr learned my addr 172.21.15.153:0/1545240825 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): get_auth_request con 0x7fea6409a510 auth_method 0 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640971e0 0x7fea64131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): get_auth_request con 0x7fea640975b0 auth_method 0 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): get_auth_request con 0x7fea640971e0 auth_method 0 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient(hunting): handle_auth_done global_id 74129 payload 824 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640971e0 msgr2=0x7fea64131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640971e0 0x7fea64131be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea640975b0 msgr2=0x7fea6413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea640975b0 0x7fea6413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 monclient: found mon.noname-c 2024-09-15T03:32:56.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea6945d640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea58009450 con 0x7fea6409a510 2024-09-15T03:32:56.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.698+0000 7fea63fff640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea640975b0 0x7fea6413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:56.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea6945d640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 msgr2=0x7fea6413f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea6945d640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea6413f280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009e90 tx=0x7fea5c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c014480 con 0x7fea6409a510 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: dump: 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.719 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c00f040 con 0x7fea6409a510 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea6aec0640 5 monclient: authenticate success, global_id 74129 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c014830 con 0x7fea6409a510 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea60ff9640 10 monclient: dump: 2024-09-15T03:32:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea6aec0640 10 monclient: _renew_subs 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea62ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea6aec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.700+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6411fff0 con 0x7fea6409a510 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 10 monclient: _renew_subs 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6411db10 con 0x7fea6409a510 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fea5c014be0 con 0x7fea6409a510 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 0x7fea6411c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6411e1a0 con 0x7fea64132120 2024-09-15T03:32:56.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea68c5c640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 0x7fea6411c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea68c5c640 10 monclient: get_auth_request con 0x7fea64132120 auth_method 0 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea68c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea68c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea68c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.701+0000 7fea60ff9640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea48038740 0x7fea4803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fea5c04cfb0 con 0x7fea6409a510 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea68c5c640 10 monclient(hunting): handle_auth_done global_id 74132 payload 824 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea68c5c640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea6404bf60 con 0x7fea64132120 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea63fff640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea48038740 0x7fea4803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.702+0000 7fea63fff640 10 monclient: get_auth_request con 0x7fea48038740 auth_method 0 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea68c5c640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 msgr2=0x7fea6411c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea68c5c640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 0x7fea6411c730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fea58034000 tx=0x7fea58032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5803bb70 con 0x7fea64132120 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 10 monclient: dump: 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.724 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea63fff640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea48038740 0x7fea4803abf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fea641207c0 tx=0x7fea50007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fea58040070 con 0x7fea64132120 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 msgr2=0x7fea6411c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 0x7fea6411c730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fea58034000 tx=0x7fea58032000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 msgr2=0x7fea6411c730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.704+0000 7fea60ff9640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64132120 0x7fea6411c730 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea6aec0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea6aec0640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 0x7fea6411d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea6aec0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea64065f20 con 0x7fea6411cc70 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 0x7fea6411d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient: get_auth_request con 0x7fea6411cc70 auth_method 0 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 10 monclient(hunting): handle_auth_done global_id 74135 payload 824 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.710+0000 7fea68c5c640 1 -- 172.21.15.153:0/1545240825 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fea6411d5b0 con 0x7fea6411cc70 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea68c5c640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 msgr2=0x7fea6411d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea68c5c640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 0x7fea6411d070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fea64068720 tx=0x7fea5803e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea58046c70 con 0x7fea6411cc70 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 10 monclient: dump: 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.727 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fea5803b960 con 0x7fea6411cc70 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 msgr2=0x7fea6411d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 0x7fea6411d070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fea64068720 tx=0x7fea5803e070 comp rx=0 tx=0).stop 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 msgr2=0x7fea6411d070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea60ff9640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411cc70 0x7fea6411d070 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fea64068720 tx=0x7fea5803e070 comp rx=0 tx=0).stop 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:56.729 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.730 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.731 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:56.732 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.733 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.734 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:56.735 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:56.736 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.738 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.739 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.740 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:56.741 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.713+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea48038740 msgr2=0x7fea4803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea48038740 0x7fea4803abf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fea641207c0 tx=0x7fea50007e10 comp rx=0 tx=0).stop 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 10 monclient: shutdown 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 msgr2=0x7fea6413f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6945d640 1 -- 172.21.15.153:0/1545240825 reap_dead start 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea6413f280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009e90 tx=0x7fea5c00b040 comp rx=0 tx=0).stop 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 shutdown_connections 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 --2- 172.21.15.153:0/1545240825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6409a510 0x7fea6413f280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 >> 172.21.15.153:0/1545240825 conn(0x7fea640672b0 msgr2=0x7fea6406cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 shutdown_connections 2024-09-15T03:32:56.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.714+0000 7fea6aec0640 1 -- 172.21.15.153:0/1545240825 wait complete. 2024-09-15T03:32:56.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:56.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:32:56.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.837+0000 7f892b24f640 10 monclient: get_monmap_and_config 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.837+0000 7f892b24f640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.837+0000 7f892b24f640 10 monclient: monmap: 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:56.837894+0000 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:56.837894+0000 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:56.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:56.839 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:56.839 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:56.839 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:56.839 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.837+0000 7f892b24f640 1 Processor -- start 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.837+0000 7f892b24f640 1 -- start start 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.837+0000 7f892b24f640 10 monclient: init 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f89241318f0 0x7f8924131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): picked mon.noname-c con 0x7f89241318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 0x7f892409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): picked mon.noname-a con 0x7f892409ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 0x7f89240691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): picked mon.noname-b con 0x7f8924068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8924131cc0 con 0x7f892409ea50 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892409fbf0 con 0x7f89241318f0 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892409fe30 con 0x7f8924068d90 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f892b24f640 10 monclient(hunting): authenticate will time out at 974.477573s 2024-09-15T03:32:56.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 0x7f89240691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 0x7f892409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 0x7f89240691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43448/0 (socket says 172.21.15.153:43448) 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 -- 172.21.15.153:0/2327350458 learned_addr learned my addr 172.21.15.153:0/2327350458 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 0x7f892409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53708/0 (socket says 172.21.15.153:53708) 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): get_auth_request con 0x7f8924068d90 auth_method 0 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f892409ea50 auth_method 0 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8928feb640 1 --2- 172.21.15.153:0/2327350458 >> v2:172.21.15.181:3300/0 conn(0x7f89241318f0 0x7f8924131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8928feb640 10 monclient(hunting): get_auth_request con 0x7f89241318f0 auth_method 0 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8928feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8928feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f8928feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient(hunting): handle_auth_done global_id 74138 payload 274 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 -- 172.21.15.153:0/2327350458 >> v2:172.21.15.181:3300/0 conn(0x7f89241318f0 msgr2=0x7f8924131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 --2- 172.21.15.153:0/2327350458 >> v2:172.21.15.181:3300/0 conn(0x7f89241318f0 0x7f8924131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 -- 172.21.15.153:0/2327350458 >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 msgr2=0x7f892409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 --2- 172.21.15.153:0/2327350458 >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 0x7f892409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.838+0000 7f89297ec640 1 monclient: found mon.noname-b 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.839+0000 7f89297ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.839+0000 7f89297ec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.839+0000 7f89297ec640 1 -- 172.21.15.153:0/2327350458 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240a1210 con 0x7f8924068d90 2024-09-15T03:32:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.839+0000 7f8923fff640 1 --2- 172.21.15.153:0/2327350458 >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 0x7f892409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:56.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89297ec640 1 -- 172.21.15.153:0/2327350458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 msgr2=0x7f89240691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89297ec640 1 --2- 172.21.15.153:0/2327350458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 0x7f89240691d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8924076600 tx=0x7f891c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=708500c395d7feac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 1 -- 172.21.15.153:0/2327350458 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c032440 con 0x7f8924068d90 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: dump: 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.856 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 1 -- 172.21.15.153:0/2327350458 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c0325a0 con 0x7f8924068d90 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f892b24f640 5 monclient: authenticate success, global_id 74138 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f892b24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f892b24f640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 1 -- 172.21.15.153:0/2327350458 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c032980 con 0x7f8924068d90 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f892b24f640 4 set_mon_vals no callback set 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.841+0000 7f89237fe640 10 monclient: dump: 2024-09-15T03:32:56.858 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.859 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 10 monclient: shutdown 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 -- 172.21.15.153:0/2327350458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 msgr2=0x7f89240691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 --2- 172.21.15.153:0/2327350458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 0x7f89240691d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8924076600 tx=0x7f891c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 -- 172.21.15.153:0/2327350458 shutdown_connections 2024-09-15T03:32:56.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 --2- 172.21.15.153:0/2327350458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8924068d90 0x7f89240691d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 --2- 172.21.15.153:0/2327350458 >> v2:172.21.15.153:3300/0 conn(0x7f892409ea50 0x7f892409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 --2- 172.21.15.153:0/2327350458 >> v2:172.21.15.181:3300/0 conn(0x7f89241318f0 0x7f8924131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 -- 172.21.15.153:0/2327350458 >> 172.21.15.153:0/2327350458 conn(0x7f8924067110 msgr2=0x7f8924067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 -- 172.21.15.153:0/2327350458 shutdown_connections 2024-09-15T03:32:56.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.842+0000 7f892b24f640 1 -- 172.21.15.153:0/2327350458 wait complete. 2024-09-15T03:32:56.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.843+0000 7f892b24f640 10 monclient: build_initial_monmap 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.843+0000 7f892b24f640 10 monclient: monmap: 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:56.862 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:56.863 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.843+0000 7f892b24f640 1 Processor -- start 2024-09-15T03:32:56.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.843+0000 7f892b24f640 1 -- start start 2024-09-15T03:32:56.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.843+0000 7f892b24f640 10 monclient: init 2024-09-15T03:32:56.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.843+0000 7f892b24f640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:56.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8924068d90 0x7f892409a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): picked mon.noname-a con 0x7f8924068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892409ea50 0x7f8924130a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): picked mon.noname-b con 0x7f892409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 0x7f892411b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): picked mon.noname-c con 0x7f89241318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:56.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892411de30 con 0x7f8924068d90 2024-09-15T03:32:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892411db50 con 0x7f892409ea50 2024-09-15T03:32:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892411e110 con 0x7f89241318f0 2024-09-15T03:32:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f892b24f640 10 monclient(hunting): authenticate will time out at 974.483084s 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8924068d90 0x7f892409a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 0x7f892411b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8924068d90 0x7f892409a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53716/0 (socket says 172.21.15.153:53716) 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 1 -- 172.21.15.153:0/882684891 learned_addr learned my addr 172.21.15.153:0/882684891 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): get_auth_request con 0x7f8924068d90 auth_method 0 2024-09-15T03:32:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): get_auth_request con 0x7f89241318f0 auth_method 0 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892409ea50 0x7f8924130a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f892409ea50 auth_method 0 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient(hunting): handle_auth_done global_id 74141 payload 824 2024-09-15T03:32:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892409ea50 msgr2=0x7f8924130a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892409ea50 0x7f8924130a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8924068d90 msgr2=0x7f892409a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8924068d90 0x7f892409a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient: _finish_hunting 0 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 monclient: found mon.noname-c 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f89297ec640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918009a20 con 0x7f89241318f0 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.844+0000 7f8928feb640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8924068d90 0x7f892409a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f89297ec640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 msgr2=0x7f892411b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f89297ec640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 0x7f892411b720 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f891c031000 tx=0x7f891c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c031c70 con 0x7f89241318f0 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:56.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: dump: 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: _finish_auth 0 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:56.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c031df0 con 0x7f89241318f0 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 5 monclient: authenticate success, global_id 74141 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 10 monclient: _renew_subs 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891c009450 con 0x7f89241318f0 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c03d690 con 0x7f89241318f0 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8920ff9640 10 monclient: dump: 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 10 monclient: _renew_subs 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f8922ffd640 4 set_mon_vals no callback set 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.846+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8924141890 con 0x7f89241318f0 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f892b24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f89297ec640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 0x7f892411b720 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f891c031000 tx=0x7f891c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f891c039040 mgrmap(e 5) v1 2024-09-15T03:32:56.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f892b24f640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 0x7f892411bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8924065e00 con 0x7f8924130f50 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f891c039040 con 0x7f89241318f0 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f8920ff9640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8910038790 0x7f891003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.847+0000 7f8928feb640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 0x7f892411bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient: get_auth_request con 0x7f8924130f50 auth_method 0 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8923fff640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8910038790 0x7f891003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8923fff640 10 monclient: get_auth_request con 0x7f8910038790 auth_method 0 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 10 monclient(hunting): handle_auth_done global_id 45001 payload 824 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.848+0000 7f8928feb640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f892409b3e0 con 0x7f8924130f50 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8928feb640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 msgr2=0x7f892411bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8928feb640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 0x7f892411bc60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8914010ce0 tx=0x7f8914008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8923fff640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8910038790 0x7f891003ac40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f89240a0470 tx=0x7f891800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914014640 con 0x7f8924130f50 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 10 monclient: dump: 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.875 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f89140147a0 con 0x7f8924130f50 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 msgr2=0x7f892411bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 0x7f892411bc60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8914010ce0 tx=0x7f8914008040 comp rx=0 tx=0).stop 2024-09-15T03:32:56.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 msgr2=0x7f892411bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:56.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.850+0000 7f8920ff9640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8924130f50 0x7f892411bc60 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8914010ce0 tx=0x7f8914008040 comp rx=0 tx=0).stop 2024-09-15T03:32:56.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f892b24f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f892b24f640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 0x7f8924096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f892b24f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892411dfa0 con 0x7f892411c1a0 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f8928feb640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 0x7f8924096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f8928feb640 10 monclient: get_auth_request con 0x7f892411c1a0 auth_method 0 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f8928feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f8928feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.856+0000 7f8928feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.857+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.857+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.857+0000 7f8928feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.857+0000 7f8928feb640 10 monclient(hunting): handle_auth_done global_id 45004 payload 824 2024-09-15T03:32:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.857+0000 7f8928feb640 1 -- 172.21.15.153:0/882684891 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f892411c5a0 con 0x7f892411c1a0 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.859+0000 7f8928feb640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 msgr2=0x7f8924096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.859+0000 7f8928feb640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 0x7f8924096a60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8914007b50 tx=0x7f891401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.859+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914027400 con 0x7f892411c1a0 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.859+0000 7f8920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.859+0000 7f8920ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.859+0000 7f8920ff9640 10 monclient: dump: 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f89140133d0 con 0x7f892411c1a0 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 msgr2=0x7f8924096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 0x7f8924096a60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8914007b50 tx=0x7f891401c000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 msgr2=0x7f8924096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:32:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892411c1a0 0x7f8924096a60 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8914007b50 tx=0x7f891401c000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 46, 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:56.882 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:32:56.883 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:32:56.884 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:32:56.885 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:32:56.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.887 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.888 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:32:56.890 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:32:56.891 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:32:56.892 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.894 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:32:56.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:32:56.896 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:32:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8910038790 msgr2=0x7f891003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8910038790 0x7f891003ac40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f89240a0470 tx=0x7f891800b040 comp rx=0 tx=0).stop 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 10 monclient: shutdown 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 msgr2=0x7f892411b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 --2- 172.21.15.153:0/882684891 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89241318f0 0x7f892411b720 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f891c031000 tx=0x7f891c032000 comp rx=0 tx=0).stop 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f89297ec640 1 -- 172.21.15.153:0/882684891 reap_dead start 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f89297ec640 1 -- 172.21.15.153:0/882684891 reap_dead start 2024-09-15T03:32:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f8920ff9640 1 -- 172.21.15.153:0/882684891 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (unknown 0 0 0) 0x7f891c06be00 con 0x7f89241318f0 2024-09-15T03:32:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 shutdown_connections 2024-09-15T03:32:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 >> 172.21.15.153:0/882684891 conn(0x7f8924067110 msgr2=0x7f89240a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 shutdown_connections 2024-09-15T03:32:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:56.860+0000 7f892b24f640 1 -- 172.21.15.153:0/882684891 wait complete. 2024-09-15T03:32:56.900 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:32:56.901 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:32:56.901 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:32:56.901 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:32:56.901 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:32:56.901 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:32:56.901 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:32:56.913 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:32:56.913 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:32:56.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:32:57.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.044+0000 7f2ce5ea8640 10 monclient: get_monmap_and_config 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.044+0000 7f2ce5ea8640 10 monclient: build_initial_monmap 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.044+0000 7f2ce5ea8640 10 monclient: monmap: 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:32:57.044952+0000 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:32:57.044952+0000 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 Processor -- start 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 -- start start 2024-09-15T03:32:57.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient: init 2024-09-15T03:32:57.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:57.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T03:32:57.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ce00981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:32:57.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2ce0131da0 0x7f2ce0136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:57.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce0131da0 addr v2:172.21.15.153:3300/0 2024-09-15T03:32:57.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2ce0132170 0x7f2ce01390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce0132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce009ca60 con 0x7f2ce00981c0 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce0131bd0 con 0x7f2ce0131da0 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce009cca0 con 0x7f2ce0132170 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2ce5ea8640 10 monclient(hunting): authenticate will time out at 974.684712s 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2cdf7fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f2ce00981c0 legacy=0x7f2ce0098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2cd7fff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f2ce0131da0 msgr2=0x7f2ce0136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.045+0000 7f2cd7fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2ce0131da0 0x7f2ce0136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2ce0132170 0x7f2ce01390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2ce0132170 0x7f2ce01390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:40328/0 (socket says 172.21.15.153:40328) 2024-09-15T03:32:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 -- 172.21.15.153:0/1108020039 learned_addr learned my addr 172.21.15.153:0/1108020039 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): get_auth_request con 0x7f2ce0132170 auth_method 0 2024-09-15T03:32:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:57.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:57.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient(hunting): handle_auth_done global_id 45007 payload 274 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 -- 172.21.15.153:0/1108020039 >> v2:172.21.15.153:3300/0 conn(0x7f2ce0131da0 msgr2=0x7f2ce0136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 --2- 172.21.15.153:0/1108020039 >> v2:172.21.15.153:3300/0 conn(0x7f2ce0131da0 0x7f2ce0136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 -- 172.21.15.153:0/1108020039 >> v1:172.21.15.153:6789/0 conn(0x7f2ce00981c0 legacy=0x7f2ce0098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 monclient: found mon.noname-c 2024-09-15T03:32:57.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.046+0000 7f2cdffff640 1 -- 172.21.15.153:0/1108020039 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce009e010 con 0x7f2ce0132170 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdffff640 1 -- 172.21.15.153:0/1108020039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce0132170 msgr2=0x7f2ce01390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdffff640 1 --2- 172.21.15.153:0/1108020039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce0132170 0x7f2ce01390c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ce006f820 tx=0x7f2cd8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb673463fbd85ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 1 -- 172.21.15.153:0/1108020039 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd802aea0 con 0x7f2ce0132170 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: dump: 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:57.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: _finish_auth 0 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 1 -- 172.21.15.153:0/1108020039 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd802d050 con 0x7f2ce0132170 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:57.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 1 -- 172.21.15.153:0/1108020039 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8007840 con 0x7f2ce0132170 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2ce5ea8640 5 monclient: authenticate success, global_id 45007 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2ce5ea8640 10 monclient: get_monmap_and_config success 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2ce5ea8640 4 set_mon_vals no callback set 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.049+0000 7f2cdeffd640 10 monclient: dump: 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:57.065 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 10 monclient: shutdown 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/1108020039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce0132170 msgr2=0x7f2ce01390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/1108020039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce0132170 0x7f2ce01390c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ce006f820 tx=0x7f2cd8009840 comp rx=0 tx=0).stop 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/1108020039 shutdown_connections 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/1108020039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce0132170 0x7f2ce01390c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/1108020039 >> v2:172.21.15.153:3300/0 conn(0x7f2ce0131da0 0x7f2ce0136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/1108020039 >> 172.21.15.153:0/1108020039 conn(0x7f2ce00a1e30 msgr2=0x7f2ce00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/1108020039 shutdown_connections 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/1108020039 wait complete. 2024-09-15T03:32:57.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 10 monclient: build_initial_monmap 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 10 monclient: monmap: 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:57.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 Processor -- start 2024-09-15T03:32:57.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 1 -- start start 2024-09-15T03:32:57.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.050+0000 7f2ce5ea8640 10 monclient: init 2024-09-15T03:32:57.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient: _reopen_session rank -1 2024-09-15T03:32:57.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:32:57.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce00981c0 0x7f2ce0138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:57.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce00981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:32:57.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ce0131da0 0x7f2ce0118490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:57.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce0131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:32:57.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 0x7f2ce01189d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:57.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce0132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:32:57.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce011a4b0 con 0x7f2ce0131da0 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce011ab00 con 0x7f2ce00981c0 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce0139220 con 0x7f2ce0132170 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): _renew_subs 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2ce5ea8640 10 monclient(hunting): authenticate will time out at 974.690351s 2024-09-15T03:32:57.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cd7fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ce0131da0 msgr2=0x7f2ce0118490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cd7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ce0131da0 0x7f2ce0118490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 0x7f2ce01189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 0x7f2ce01189d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43484/0 (socket says 172.21.15.153:43484) 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 -- 172.21.15.153:0/596809394 learned_addr learned my addr 172.21.15.153:0/596809394 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): get_auth_request con 0x7f2ce0132170 auth_method 0 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:57.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:57.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:57.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce00981c0 0x7f2ce0138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:57.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:57.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:57.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:57.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): get_auth_request con 0x7f2ce00981c0 auth_method 0 2024-09-15T03:32:57.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:32:57.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:32:57.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient(hunting): handle_auth_done global_id 74144 payload 824 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 -- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce00981c0 msgr2=0x7f2ce0138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce00981c0 0x7f2ce0138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 -- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ce0131da0 msgr2=0x7f2ce0118490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ce0131da0 0x7f2ce0118490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient: _finish_hunting 0 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 monclient: found mon.noname-c 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.051+0000 7f2cdffff640 1 -- 172.21.15.153:0/596809394 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8009ae0 con 0x7f2ce0132170 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cdffff640 1 -- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 msgr2=0x7f2ce01189d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cdffff640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 0x7f2ce01189d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009ab0 tx=0x7f2cd8036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031cd0 con 0x7f2ce0132170 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:57.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: dump: 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:57.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: _finish_auth 0 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8031e30 con 0x7f2ce0132170 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 5 monclient: authenticate success, global_id 74144 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd803c630 con 0x7f2ce0132170 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 10 monclient: _renew_subs 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce0130b30 con 0x7f2ce0132170 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cd77fe640 10 monclient: dump: 2024-09-15T03:32:57.079 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2cde7fc640 4 set_mon_vals no callback set 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 10 monclient: _renew_subs 2024-09-15T03:32:57.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.054+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce0138ad0 con 0x7f2ce0132170 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2cd803ca20 con 0x7f2ce0132170 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2ce5ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2ce5ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2ce5ea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce004bf60 con 0x7f2ce0132170 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2cd77fe640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2cc00386f0 0x7f2cc003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f2cd806d3f0 con 0x7f2ce0132170 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2cdf7fe640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2cc00386f0 0x7f2cc003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:32:57.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.055+0000 7f2cdf7fe640 10 monclient: get_auth_request con 0x7f2cc00386f0 auth_method 0 2024-09-15T03:32:57.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.057+0000 7f2cdf7fe640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2cc00386f0 0x7f2cc003aba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0003940 tx=0x7f2cd00090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:32:57.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.059+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2cd803ccd0 con 0x7f2ce0132170 2024-09-15T03:32:57.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.059+0000 7f2cd77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:32:57.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.059+0000 7f2cd77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:32:57.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2ce5ea8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:32:57.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2ce5ea8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:57.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2ce5ea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:32:57.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2ce0098cb0 con 0x7f2ce0132170 2024-09-15T03:32:57.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2cd77fe640 1 -- 172.21.15.153:0/596809394 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2cd804a080 con 0x7f2ce0132170 2024-09-15T03:32:57.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2cd77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:32:57.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.277+0000 7f2cd77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:32:57.281 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:32:57.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2cc00386f0 msgr2=0x7f2cc003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2cc00386f0 0x7f2cc003aba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0003940 tx=0x7f2cd00090f0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 10 monclient: shutdown 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 msgr2=0x7f2ce01189d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 0x7f2ce01189d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009ab0 tx=0x7f2cd8036000 comp rx=0 tx=0).stop 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 shutdown_connections 2024-09-15T03:32:57.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2cc00386f0 0x7f2cc003aba0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ce0132170 0x7f2ce01189d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ce0131da0 0x7f2ce0118490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 --2- 172.21.15.153:0/596809394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ce00981c0 0x7f2ce0138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:32:57.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.279+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 >> 172.21.15.153:0/596809394 conn(0x7f2ce00a1e30 msgr2=0x7f2ce009bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:32:57.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.280+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 shutdown_connections 2024-09-15T03:32:57.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:32:57.280+0000 7f2ce5ea8640 1 -- 172.21.15.153:0/596809394 wait complete. 2024-09-15T03:32:58.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:58.541+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:32:58.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:58.541+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:32:58.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:58.541+0000 7fc609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:32:59.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:59.541+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:32:59.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:32:59.541+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:00.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.396+0000 7f3624f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.396+0000 7f3624f9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.396+0000 7f3624f9d640 10 monclient: monmap: 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:00.397364+0000 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:00.397364+0000 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:00.408 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:00.409 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:00.409 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:00.409 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:00.409 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:00.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:00.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 Processor -- start 2024-09-15T03:33:00.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 -- start start 2024-09-15T03:33:00.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient: init 2024-09-15T03:33:00.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:00.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:33:00.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3620067110 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:00.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 0x7f362009b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:00.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f362009b010 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:00.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f36201349a0 0x7f3620136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:00.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f36201349a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:00.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3620130540 con 0x7f3620067110 2024-09-15T03:33:00.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:00.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362009b940 con 0x7f36201349a0 2024-09-15T03:33:00.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:00.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3620067ed0 con 0x7f362009b010 2024-09-15T03:33:00.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:00.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.397+0000 7f3624f9d640 10 monclient(hunting): authenticate will time out at 978.036959s 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 0x7f362009b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 0x7f362009b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43494/0 (socket says 172.21.15.153:43494) 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 -- 172.21.15.153:0/1237668291 learned_addr learned my addr 172.21.15.153:0/1237668291 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): get_auth_request con 0x7f362009b010 auth_method 0 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:00.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:00.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:00.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f361ed9d640 1 --2- 172.21.15.153:0/1237668291 >> v2:172.21.15.181:3300/0 conn(0x7f36201349a0 0x7f3620136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:00.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f361ed9d640 10 monclient(hunting): get_auth_request con 0x7f36201349a0 auth_method 0 2024-09-15T03:33:00.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f361ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:00.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f361ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:00.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f361ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient(hunting): handle_auth_done global_id 74147 payload 274 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 -- 172.21.15.153:0/1237668291 >> v2:172.21.15.181:3300/0 conn(0x7f36201349a0 msgr2=0x7f3620136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 --2- 172.21.15.153:0/1237668291 >> v2:172.21.15.181:3300/0 conn(0x7f36201349a0 0x7f3620136d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 -- 172.21.15.153:0/1237668291 >> v1:172.21.15.181:6789/0 conn(0x7f3620067110 legacy=0x7f362012f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 monclient: found mon.noname-b 2024-09-15T03:33:00.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:00.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:00.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.398+0000 7f3615d9b640 1 -- 172.21.15.153:0/1237668291 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620099800 con 0x7f362009b010 2024-09-15T03:33:00.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3615d9b640 1 -- 172.21.15.153:0/1237668291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 msgr2=0x7f362009b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:00.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3615d9b640 1 --2- 172.21.15.153:0/1237668291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 0x7f362009b400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3608009910 tx=0x7f36080091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01da6e6f4fdea26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 1 -- 172.21.15.153:0/1237668291 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608032440 con 0x7f362009b010 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: dump: 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:00.417 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:33:00.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:00.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 5 monclient: authenticate success, global_id 74147 2024-09-15T03:33:00.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:00.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 1 -- 172.21.15.153:0/1237668291 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080325a0 con 0x7f362009b010 2024-09-15T03:33:00.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:00.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 1 -- 172.21.15.153:0/1237668291 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608032940 con 0x7f362009b010 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 4 set_mon_vals no callback set 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f361dd9b640 10 monclient: dump: 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:00.420 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:00.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 10 monclient: shutdown 2024-09-15T03:33:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 -- 172.21.15.153:0/1237668291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 msgr2=0x7f362009b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 --2- 172.21.15.153:0/1237668291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 0x7f362009b400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3608009910 tx=0x7f36080091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 -- 172.21.15.153:0/1237668291 shutdown_connections 2024-09-15T03:33:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 --2- 172.21.15.153:0/1237668291 >> v2:172.21.15.181:3300/0 conn(0x7f36201349a0 0x7f3620136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 --2- 172.21.15.153:0/1237668291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f362009b010 0x7f362009b400 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 -- 172.21.15.153:0/1237668291 >> 172.21.15.153:0/1237668291 conn(0x7f36200a1e30 msgr2=0x7f36200a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 -- 172.21.15.153:0/1237668291 shutdown_connections 2024-09-15T03:33:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.401+0000 7f3624f9d640 1 -- 172.21.15.153:0/1237668291 wait complete. 2024-09-15T03:33:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.402+0000 7f3624f9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.402+0000 7f3624f9d640 10 monclient: monmap: 2024-09-15T03:33:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:00.424 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.402+0000 7f3624f9d640 1 Processor -- start 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.402+0000 7f3624f9d640 1 -- start start 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.402+0000 7f3624f9d640 10 monclient: init 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3620067110 0x7f362006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3620067110 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f362009b010 0x7f3620096b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f362009b010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 0x7f3620097050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f36201349a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362006f250 con 0x7f3620067110 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36200a0740 con 0x7f362009b010 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362009e330 con 0x7f36201349a0 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3624f9d640 10 monclient(hunting): authenticate will time out at 978.042423s 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361e59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3620067110 msgr2=0x7f362006ed10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3620067110 0x7f362006ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 0x7f3620097050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 0x7f3620097050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43500/0 (socket says 172.21.15.153:43500) 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 -- 172.21.15.153:0/3226366319 learned_addr learned my addr 172.21.15.153:0/3226366319 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): get_auth_request con 0x7f36201349a0 auth_method 0 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3615d9b640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f362009b010 0x7f3620096b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3615d9b640 10 monclient(hunting): get_auth_request con 0x7f362009b010 auth_method 0 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3615d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3615d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:00.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f3615d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient(hunting): handle_auth_done global_id 74150 payload 824 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 -- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f362009b010 msgr2=0x7f3620096b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f362009b010 0x7f3620096b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 -- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3620067110 msgr2=0x7f362006ed10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3620067110 0x7f362006ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.403+0000 7f361ed9d640 1 -- 172.21.15.153:0/3226366319 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608009450 con 0x7f36201349a0 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.406+0000 7f361ed9d640 1 -- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 msgr2=0x7f3620097050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.406+0000 7f361ed9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 0x7f3620097050 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3618007f30 tx=0x7f36180098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:00.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.406+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:33:00.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361800ee30 con 0x7f36201349a0 2024-09-15T03:33:00.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: dump: 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: _finish_auth 0 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618013590 con 0x7f36201349a0 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 5 monclient: authenticate success, global_id 74150 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 10 monclient: _renew_subs 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618013950 con 0x7f36201349a0 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362006cec0 con 0x7f36201349a0 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3616ffd640 10 monclient: dump: 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:00.432 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f361d59a640 4 set_mon_vals no callback set 2024-09-15T03:33:00.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 10 monclient: _renew_subs 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.408+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362006ddb0 con 0x7f36201349a0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3618013d40 con 0x7f36201349a0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3624f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3624f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3624f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362004bf60 con 0x7f36201349a0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3616ffd640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3600038740 0x7f360003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f361804c8b0 con 0x7f36201349a0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f361e59c640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3600038740 0x7f360003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.409+0000 7f361e59c640 10 monclient: get_auth_request con 0x7f3600038740 auth_method 0 2024-09-15T03:33:00.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.412+0000 7f361e59c640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3600038740 0x7f360003abf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3610007fd0 tx=0x7f3610007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:00.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.413+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3618014420 con 0x7f36201349a0 2024-09-15T03:33:00.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.413+0000 7f3616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:00.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.413+0000 7f3616ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:00.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:00.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:00.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:00.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.629+0000 7f3624f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.629+0000 7f3624f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.629+0000 7f3624f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.629+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f36201370a0 con 0x7f36201349a0 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.630+0000 7f3616ffd640 1 -- 172.21.15.153:0/3226366319 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f361801b310 con 0x7f36201349a0 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.630+0000 7f3616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.630+0000 7f3616ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:00.631 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.631+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3600038740 msgr2=0x7f360003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.631+0000 7f3624f9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3600038740 0x7f360003abf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3610007fd0 tx=0x7f3610007a70 comp rx=0 tx=0).stop 2024-09-15T03:33:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 10 monclient: shutdown 2024-09-15T03:33:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 msgr2=0x7f3620097050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 0x7f3620097050 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3618007f30 tx=0x7f36180098e0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 shutdown_connections 2024-09-15T03:33:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3600038740 0x7f360003abf0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36201349a0 0x7f3620097050 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f362009b010 0x7f3620096b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 --2- 172.21.15.153:0/3226366319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3620067110 0x7f362006ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 >> 172.21.15.153:0/3226366319 conn(0x7f36200a1e30 msgr2=0x7f36201306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 shutdown_connections 2024-09-15T03:33:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:00.632+0000 7f3624f9d640 1 -- 172.21.15.153:0/3226366319 wait complete. 2024-09-15T03:33:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:01.541+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:01.541+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:02.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:02.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:02.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:02.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:03.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:03.541+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:03.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:03.541+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:03.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:03.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.747+0000 7fa31206f640 10 monclient: get_monmap_and_config 2024-09-15T03:33:03.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.747+0000 7fa31206f640 10 monclient: build_initial_monmap 2024-09-15T03:33:03.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.747+0000 7fa31206f640 10 monclient: monmap: 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:03.748512+0000 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:03.748512+0000 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:03.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:03.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.748+0000 7fa31206f640 1 Processor -- start 2024-09-15T03:33:03.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.748+0000 7fa31206f640 1 -- start start 2024-09-15T03:33:03.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.748+0000 7fa31206f640 10 monclient: init 2024-09-15T03:33:03.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:03.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa30c096c60 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa30c097030 0x7fa30c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient(hunting): picked mon.noname-c con 0x7fa30c097030 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa30c133b40 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30c09d2e0 con 0x7fa30c133b40 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30c09d6b0 con 0x7fa30c096c60 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30bfff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa30c133b40 legacy=0x7fa30c135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:03.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30c0992f0 con 0x7fa30c097030 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa31206f640 10 monclient(hunting): authenticate will time out at 981.388320s 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa30c097030 0x7fa30c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa30c097030 0x7fa30c097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:40374/0 (socket says 172.21.15.153:40374) 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 -- 172.21.15.153:0/3390634550 learned_addr learned my addr 172.21.15.153:0/3390634550 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30b7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa30c096c60 0x7fa30c09a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:48800/0 (socket says 172.21.15.153:48800) 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): get_auth_request con 0x7fa30c097030 auth_method 0 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:03.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient(hunting): handle_auth_done global_id 45019 payload 274 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 -- 172.21.15.153:0/3390634550 >> v1:172.21.15.181:6789/0 conn(0x7fa30c096c60 legacy=0x7fa30c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 -- 172.21.15.153:0/3390634550 >> v1:172.21.15.153:6789/0 conn(0x7fa30c133b40 legacy=0x7fa30c135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 monclient: found mon.noname-c 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:03.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.749+0000 7fa30affd640 1 -- 172.21.15.153:0/3390634550 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c0995d0 con 0x7fa30c097030 2024-09-15T03:33:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.752+0000 7fa30affd640 1 -- 172.21.15.153:0/3390634550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c097030 msgr2=0x7fa30c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.752+0000 7fa30affd640 1 --2- 172.21.15.153:0/3390634550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c097030 0x7fa30c097430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa300009910 tx=0x7fa3000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7e77527d9dec384 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 1 -- 172.21.15.153:0/3390634550 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300032440 con 0x7fa30c097030 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: dump: 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 1 -- 172.21.15.153:0/3390634550 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3000325a0 con 0x7fa30c097030 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 5 monclient: authenticate success, global_id 45019 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 4 set_mon_vals no callback set 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 1 -- 172.21.15.153:0/3390634550 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300032940 con 0x7fa30c097030 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa30a7fc640 10 monclient: dump: 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 10 monclient: shutdown 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:03.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 -- 172.21.15.153:0/3390634550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c097030 msgr2=0x7fa30c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 --2- 172.21.15.153:0/3390634550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c097030 0x7fa30c097430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa300009910 tx=0x7fa3000091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 -- 172.21.15.153:0/3390634550 shutdown_connections 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 --2- 172.21.15.153:0/3390634550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c097030 0x7fa30c097430 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 -- 172.21.15.153:0/3390634550 >> 172.21.15.153:0/3390634550 conn(0x7fa30c0672b0 msgr2=0x7fa30c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 -- 172.21.15.153:0/3390634550 shutdown_connections 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.753+0000 7fa31206f640 1 -- 172.21.15.153:0/3390634550 wait complete. 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.754+0000 7fa31206f640 10 monclient: build_initial_monmap 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.754+0000 7fa31206f640 10 monclient: monmap: 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:03.764 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.754+0000 7fa31206f640 1 Processor -- start 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 -- start start 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient: init 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 0x7fa30c06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): picked mon.noname-c con 0x7fa30c096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa30c097030 0x7fa30c06c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:03.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): picked mon.noname-a con 0x7fa30c097030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c133b40 0x7fa30c06ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): picked mon.noname-b con 0x7fa30c133b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c06e050 con 0x7fa30c097030 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c136b50 con 0x7fa30c133b40 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c117520 con 0x7fa30c096c60 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa31206f640 10 monclient(hunting): authenticate will time out at 981.394509s 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30affd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa30c097030 msgr2=0x7fa30c06c4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa30c097030 0x7fa30c06c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 0x7fa30c06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:03.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 0x7fa30c06bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43502/0 (socket says 172.21.15.153:43502) 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 -- 172.21.15.153:0/567540564 learned_addr learned my addr 172.21.15.153:0/567540564 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): get_auth_request con 0x7fa30c096c60 auth_method 0 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c133b40 0x7fa30c06ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): get_auth_request con 0x7fa30c133b40 auth_method 0 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:03.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient(hunting): handle_auth_done global_id 74153 payload 824 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 -- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c133b40 msgr2=0x7fa30c06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c133b40 0x7fa30c06ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 -- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa30c097030 msgr2=0x7fa30c06c4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa30c097030 0x7fa30c06c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.755+0000 7fa30b7fe640 1 -- 172.21.15.153:0/567540564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa300009450 con 0x7fa30c096c60 2024-09-15T03:33:03.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa30b7fe640 1 -- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 msgr2=0x7fa30c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa30b7fe640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 0x7fa30c06bf90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00c960 tx=0x7fa2fc004f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00a5e0 con 0x7fa30c096c60 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: dump: 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:03.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 5 monclient: authenticate success, global_id 74153 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc00a740 con 0x7fa30c096c60 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 10 monclient: _renew_subs 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30c09add0 con 0x7fa30c096c60 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00ab00 con 0x7fa30c096c60 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa2f37fe640 10 monclient: dump: 2024-09-15T03:33:03.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa309ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 10 monclient: _renew_subs 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.758+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30c12f2e0 con 0x7fa30c096c60 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa2fc0070f0 con 0x7fa30c096c60 2024-09-15T03:33:03.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa31206f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa31206f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa31206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30c04bf60 con 0x7fa30c096c60 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa2f37fe640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f80386f0 0x7fa2f803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.759+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fa2fc04be90 con 0x7fa30c096c60 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.760+0000 7fa30affd640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f80386f0 0x7fa2f803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:03.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.760+0000 7fa30affd640 10 monclient: get_auth_request con 0x7fa2f80386f0 auth_method 0 2024-09-15T03:33:03.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.762+0000 7fa30affd640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f80386f0 0x7fa2f803aba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa300034000 tx=0x7fa300032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:03.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.764+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa2fc0073a0 con 0x7fa30c096c60 2024-09-15T03:33:03.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.764+0000 7fa2f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:03.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.764+0000 7fa2f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:03.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa31206f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:03.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa31206f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:03.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa31206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:03.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa30c0983b0 con 0x7fa30c096c60 2024-09-15T03:33:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa2f37fe640 1 -- 172.21.15.153:0/567540564 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa2fc010af0 con 0x7fa30c096c60 2024-09-15T03:33:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa2f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.978+0000 7fa2f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:03.980 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:03.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f80386f0 msgr2=0x7fa2f803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f80386f0 0x7fa2f803aba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa300034000 tx=0x7fa300032000 comp rx=0 tx=0).stop 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 10 monclient: shutdown 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 msgr2=0x7fa30c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 0x7fa30c06bf90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00c960 tx=0x7fa2fc004f70 comp rx=0 tx=0).stop 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 shutdown_connections 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f80386f0 0x7fa2f803aba0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa30c133b40 0x7fa30c06ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa30c097030 0x7fa30c06c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 --2- 172.21.15.153:0/567540564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa30c096c60 0x7fa30c06bf90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 >> 172.21.15.153:0/567540564 conn(0x7fa30c0672b0 msgr2=0x7fa30c1352e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 shutdown_connections 2024-09-15T03:33:03.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:03.980+0000 7fa31206f640 1 -- 172.21.15.153:0/567540564 wait complete. 2024-09-15T03:33:04.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:04.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:04.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:04.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:05.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:05.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:06.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:06.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:06.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:06.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:06.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:07.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.096+0000 7f49e519d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:07.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.096+0000 7f49e519d640 10 monclient: build_initial_monmap 2024-09-15T03:33:07.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.096+0000 7f49e519d640 10 monclient: monmap: 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:07.096928+0000 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:07.096928+0000 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:07.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.096+0000 7f49e519d640 1 Processor -- start 2024-09-15T03:33:07.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.096+0000 7f49e519d640 1 -- start start 2024-09-15T03:33:07.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.096+0000 7f49e519d640 10 monclient: init 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): picked mon.noname-c con 0x7f49e006eb80 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): picked mon.noname-b con 0x7f49e009c6d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:07.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 0x7f49e009de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): picked mon.noname-a con 0x7f49e009cac0 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e0068a20 con 0x7f49e009cac0 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e006ef50 con 0x7f49e006eb80 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e006f700 con 0x7f49e009c6d0 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49e519d640 10 monclient(hunting): authenticate will time out at 984.736298s 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49df59e640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 msgr2=0x7f49e009de00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:07.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49df59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 0x7f49e009de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45616/0 (socket says 172.21.15.153:45616) 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 learned_addr learned my addr 172.21.15.153:0/1412998640 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49e009c6d0 auth_method 0 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 10 monclient(hunting): get_auth_request con 0x7f49e006eb80 auth_method 0 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:07.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:07.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.097+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 msgr2=0x7f49e009de00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 0x7f49e009de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49e009c6d0 auth_method 0 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:07.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 10 monclient(hunting): get_auth_request con 0x7f49e006eb80 auth_method 0 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:07.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.298+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:07.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:07.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:07.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:07.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 msgr2=0x7f49e009de00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 0x7f49e009de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49e009c6d0 auth_method 0 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:07.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 10 monclient(hunting): get_auth_request con 0x7f49e006eb80 auth_method 0 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:07.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:07.699+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:08.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 msgr2=0x7f49e009de00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 0x7f49e009de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49e009c6d0 auth_method 0 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 10 monclient(hunting): get_auth_request con 0x7f49e006eb80 auth_method 0 2024-09-15T03:33:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:08.500+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:08.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:08.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:08.542+0000 7fc609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:09.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:09.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:09.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:09.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:10.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 10 monclient(hunting): tick 2024-09-15T03:33:10.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 monclient(hunting): continuing hunt 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 msgr2=0x7f49e009d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49e009c6d0 0x7f49e009d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 msgr2=0x7f49e006f1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49e006eb80 0x7f49e006f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 msgr2=0x7f49e009de00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49e009cac0 0x7f49e009de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 10 monclient: _add_conns ranks=[0,1] 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49c0003650 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49c0007390 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e0099ef0 con 0x7f49c0003650 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c000c0e0 con 0x7f49c0007390 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49dcd99640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:10.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.097+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c0003650 legacy=0x7f49c0005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:10.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.100+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.181:6789/0 conn(0x7f49c0007390 legacy=0x7f49c0009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:10.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.298+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c0003650 legacy=0x7f49c0005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:10.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:10.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:10.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:10.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:10.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:10.699+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c0003650 legacy=0x7f49c0005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:11.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:11.500+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c0003650 legacy=0x7f49c0005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:11.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:11.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:11.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:11.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:12.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:12.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:12.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:12.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 10 monclient(hunting): tick 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 1 monclient(hunting): continuing hunt 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.181:6789/0 conn(0x7f49c0007390 legacy=0x7f49c0009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c0003650 legacy=0x7f49c0005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 reap_dead start 2024-09-15T03:33:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49c000c460 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-c con 0x7f49c00077c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 0x7f49c000a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-a con 0x7f49c000a530 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c000c460 legacy=0x7f49c000c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.096+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c000d2e0 con 0x7f49c000c460 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49dcd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e0068a20 con 0x7f49c000a530 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49dcd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e006ef50 con 0x7f49c00077c0 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49dcd99640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 msgr2=0x7f49c000a9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:13.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 0x7f49c000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49c00077c0 auth_method 0 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:13.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:13.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c000c460 legacy=0x7f49c000c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:13.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 msgr2=0x7f49c000a9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 0x7f49c000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49c00077c0 auth_method 0 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:13.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:13.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:13.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:13.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:13.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:13.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 msgr2=0x7f49c000a9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:13.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c000c460 legacy=0x7f49c000c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:13.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 0x7f49c000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49c00077c0 auth_method 0 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:13.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:13.699+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:14.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 msgr2=0x7f49c000a9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:14.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c000c460 legacy=0x7f49c000c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:14.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 0x7f49c000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49c00077c0 auth_method 0 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:14.500+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:14.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:14.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:14.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:14.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:15.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:15.542+0000 7fc609ffb640 10 monclient: tick 2024-09-15T03:33:15.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:15.542+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient(hunting): tick 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 monclient(hunting): continuing hunt 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 msgr2=0x7f49c0009ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.181:3300/0 conn(0x7f49c00077c0 0x7f49c0009ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 msgr2=0x7f49c000a9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> v2:172.21.15.153:3300/0 conn(0x7f49c000a530 0x7f49c000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.153:6789/0 conn(0x7f49c000c460 legacy=0x7f49c000c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient: _add_conns ranks=[1,2] 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49c0005a60 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient(hunting): picked mon.noname-b con 0x7f49c0006980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e0099ef0 con 0x7f49c0005a60 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 1 -- 172.21.15.153:0/1412998640 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0068a20 con 0x7f49c0006980 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49dcd99640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49c0006980 auth_method 0 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 msgr2=0x7f49c0014610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 msgr2=0x7f49c0014610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:16.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.097+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.099+0000 7f49ded9d640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.181:6789/0 conn(0x7f49c0005a60 legacy=0x7f49c0006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49c0006980 auth_method 0 2024-09-15T03:33:16.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:16.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 msgr2=0x7f49c0014610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:16.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 msgr2=0x7f49c0014610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:16.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:16.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.298+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.461+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967244781 0 0) 0x7f49e0099ef0 con 0x7f49c0005a60 2024-09-15T03:33:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.461+0000 7f49ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.461+0000 7f49ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.461+0000 7f49ddd9b640 10 monclient(hunting): my global_id is 45025 2024-09-15T03:33:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.461+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc0034e0 con 0x7f49c0005a60 2024-09-15T03:33:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.462+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 969251502 0 0) 0x7f49cc0034e0 con 0x7f49c0005a60 2024-09-15T03:33:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 msgr2=0x7f49c0014610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49c0006980 0x7f49c0014610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c000d2e0 con 0x7f49c0005a60 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f49d8009b80 con 0x7f49c0005a60 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49e519d640 5 monclient: authenticate success, global_id 45025 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49e519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: dump: 2024-09-15T03:33:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 >> v1:172.21.15.181:6789/0 conn(0x7f49c0005a60 legacy=0x7f49c0006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 reap_dead start 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc0039d0 0x7f49cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): picked mon.b con 0x7f49cc0039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 0x7f49cc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): picked mon.c con 0x7f49cc004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc005180 0x7f49cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): picked mon.a con 0x7f49cc005180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0068a20 con 0x7f49cc005180 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc00c120 con 0x7f49cc0039d0 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc00c2e0 con 0x7f49cc004780 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ddd9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49df59e640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc005180 msgr2=0x7f49cc00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49e519d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49df59e640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc005180 0x7f49cc00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 0x7f49cc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49cc004780 auth_method 0 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc0039d0 0x7f49cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): get_auth_request con 0x7f49cc0039d0 auth_method 0 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient(hunting): handle_auth_done global_id 45025 payload 293 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc0039d0 msgr2=0x7f49cc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc0039d0 0x7f49cc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc005180 msgr2=0x7f49cc00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc005180 0x7f49cc00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 monclient: found mon.c 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0034e0 con 0x7f49cc004780 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.463+0000 7f49de59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.466+0000 7f49de59c640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 msgr2=0x7f49cc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.466+0000 7f49de59c640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 0x7f49cc004bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49c8006f60 tx=0x7f49c8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d98640ac93f4ea9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8033440 con 0x7f49cc004780 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: dump: 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.479 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c802f050 con 0x7f49cc004780 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 4 set_mon_vals no callback set 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 1 -- 172.21.15.153:0/1412998640 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8033600 con 0x7f49cc004780 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49ddd9b640 10 monclient: dump: 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 10 monclient: shutdown 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 -- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 msgr2=0x7f49cc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 0x7f49cc004bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49c8006f60 tx=0x7f49c8008040 comp rx=0 tx=0).stop 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 -- 172.21.15.153:0/1412998640 shutdown_connections 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc005180 0x7f49cc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc004780 0x7f49cc004bf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 --2- 172.21.15.153:0/1412998640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc0039d0 0x7f49cc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.467+0000 7f49e519d640 1 -- 172.21.15.153:0/1412998640 >> 172.21.15.153:0/1412998640 conn(0x7f49e0067cd0 msgr2=0x7f49e00a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.468+0000 7f49e519d640 1 -- 172.21.15.153:0/1412998640 shutdown_connections 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.468+0000 7f49e519d640 1 -- 172.21.15.153:0/1412998640 wait complete. 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient: monmap: 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 1 Processor -- start 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 1 -- start start 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient: init 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc0039d0 0x7f49e013d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient(hunting): picked mon.noname-a con 0x7f49cc0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:16.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc004780 0x7f49e013cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.469+0000 7f49e519d640 10 monclient(hunting): picked mon.noname-b con 0x7f49cc004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 0x7f49e0140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 10 monclient(hunting): picked mon.noname-c con 0x7f49cc005180 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0122690 con 0x7f49cc0039d0 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e01220d0 con 0x7f49cc004780 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e011fb60 con 0x7f49cc005180 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49e519d640 10 monclient(hunting): authenticate will time out at 994.110039s 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49ded9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc0039d0 msgr2=0x7f49e013d6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49ded9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc0039d0 0x7f49e013d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 0x7f49e0140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 0x7f49e0140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51944/0 (socket says 172.21.15.153:51944) 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 -- 172.21.15.153:0/2420499947 learned_addr learned my addr 172.21.15.153:0/2420499947 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): get_auth_request con 0x7f49cc005180 auth_method 0 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49de59c640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc004780 0x7f49e013cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49de59c640 10 monclient(hunting): get_auth_request con 0x7f49cc004780 auth_method 0 2024-09-15T03:33:16.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient(hunting): handle_auth_done global_id 74156 payload 824 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 -- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc004780 msgr2=0x7f49e013cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc004780 0x7f49e013cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 -- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc0039d0 msgr2=0x7f49e013d6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc0039d0 0x7f49e013d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 monclient: found mon.noname-c 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.470+0000 7f49df59e640 1 -- 172.21.15.153:0/2420499947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e0121df0 con 0x7f49cc005180 2024-09-15T03:33:16.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49df59e640 1 -- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 msgr2=0x7f49e0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49df59e640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 0x7f49e0140c10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49c000b060 tx=0x7f49d00091c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000ba20 con 0x7f49cc005180 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: dump: 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.487 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d000bba0 con 0x7f49cc005180 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49e519d640 5 monclient: authenticate success, global_id 74156 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49e519d640 10 monclient: _renew_subs 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0006150 con 0x7f49cc005180 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49e519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e01223b0 con 0x7f49cc005180 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49d6ffd640 10 monclient: dump: 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.472+0000 7f49dd59a640 4 set_mon_vals no callback set 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.473+0000 7f49e519d640 10 monclient: _renew_subs 2024-09-15T03:33:16.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.473+0000 7f49e519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.473+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e013d490 con 0x7f49cc005180 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.473+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f49d0006540 con 0x7f49cc005180 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49d6ffd640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49c003a410 0x7f49c003c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49e519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49e519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49e519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f49d004ad60 con 0x7f49cc005180 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e013d490 con 0x7f49cc005180 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49ded9d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49c003a410 0x7f49c003c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.474+0000 7f49ded9d640 10 monclient: get_auth_request con 0x7f49c003a410 auth_method 0 2024-09-15T03:33:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.476+0000 7f49ded9d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49c003a410 0x7f49c003c8c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f49d8000f80 tx=0x7f49d800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.478+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f49e013d490 con 0x7f49cc005180 2024-09-15T03:33:16.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.479+0000 7f49d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.479+0000 7f49d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:16.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.521+0000 7fc60affd640 1 -- 172.21.15.181:0/1011238246 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v108) v1 ==== 58+0+3 (secure 0 0 0) 0x7fc60c0234d0 con 0x7fc61409f240 2024-09-15T03:33:16.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.521+0000 7fc60affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-15T03:33:16.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.521+0000 7fc60affd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:16.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5f4038740 msgr2=0x7fc5f403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5f4038740 0x7fc5f403abf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc61409a5e0 tx=0x7fc5fc007f00 comp rx=0 tx=0).stop 2024-09-15T03:33:16.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 10 monclient: shutdown 2024-09-15T03:33:16.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 msgr2=0x7fc614141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 0x7fc614141510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00ec80 tx=0x7fc60c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 shutdown_connections 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc5f4038740 0x7fc5f403abf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc614136e70 0x7fc614098730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc614131b40 0x7fc614141a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 --2- 172.21.15.181:0/1011238246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61409f240 0x7fc614141510 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 >> 172.21.15.181:0/1011238246 conn(0x7fc6140a5100 msgr2=0x7fc6140a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 shutdown_connections 2024-09-15T03:33:16.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.525+0000 7fc61911d640 1 -- 172.21.15.181:0/1011238246 wait complete. 2024-09-15T03:33:16.538 INFO:tasks.workunit.client.0.smithi181.stderr:+ o5=14 2024-09-15T03:33:16.539 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.617+0000 7f4148880640 10 monclient: get_monmap_and_config 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.617+0000 7f4148880640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.617+0000 7f4148880640 10 monclient: monmap: 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:16.618865+0000 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:16.618865+0000 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:16.620 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:16.620 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:16.620 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:16.620 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.617+0000 7f4148880640 1 Processor -- start 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.617+0000 7f4148880640 1 -- start start 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.617+0000 7f4148880640 10 monclient: init 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f414409ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f414409f020 0x7f4144069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): picked mon.noname-c con 0x7f414409f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4144136d60 0x7f4144139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): picked mon.noname-a con 0x7f4144136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4144067420 con 0x7f414409ec50 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414409fdc0 con 0x7f4144136d60 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41440a0510 con 0x7f414409f020 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4148880640 10 monclient(hunting): authenticate will time out at 1001.100424s 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4142d9d640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f4144136d60 msgr2=0x7f4144139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f414409f020 0x7f4144069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f414259c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f414409ec50 0x7f4144131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48318/0 (socket says 172.21.15.181:48318) 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f414259c640 1 -- 172.21.15.181:0/1409048299 learned_addr learned my addr 172.21.15.181:0/1409048299 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:16.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): get_auth_request con 0x7f414409f020 auth_method 0 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4142d9d640 1 --2- 172.21.15.181:0/1409048299 >> v2:172.21.15.153:3300/0 conn(0x7f4144136d60 0x7f4144139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient(hunting): handle_auth_done global_id 45031 payload 274 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 1 -- 172.21.15.181:0/1409048299 >> v2:172.21.15.153:3300/0 conn(0x7f4144136d60 msgr2=0x7f4144139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 1 --2- 172.21.15.181:0/1409048299 >> v2:172.21.15.153:3300/0 conn(0x7f4144136d60 0x7f4144139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 1 -- 172.21.15.181:0/1409048299 >> v1:172.21.15.181:6789/0 conn(0x7f414409ec50 legacy=0x7f4144131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 1 monclient: found mon.noname-c 2024-09-15T03:33:16.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.618+0000 7f4141d9b640 1 -- 172.21.15.181:0/1409048299 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41440a0200 con 0x7f414409f020 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f4141d9b640 1 -- 172.21.15.181:0/1409048299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 msgr2=0x7f4144069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f4141d9b640 1 --2- 172.21.15.181:0/1409048299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144069d10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f41440a2b00 tx=0x7f41380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd2ebd621a6ef538 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 1 -- 172.21.15.181:0/1409048299 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138032440 con 0x7f414409f020 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: dump: 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 1 -- 172.21.15.181:0/1409048299 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41380325a0 con 0x7f414409f020 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f4148880640 5 monclient: authenticate success, global_id 45031 2024-09-15T03:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f4148880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f4148880640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 1 -- 172.21.15.181:0/1409048299 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138032940 con 0x7f414409f020 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f4148880640 4 set_mon_vals no callback set 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.621+0000 7f414159a640 10 monclient: dump: 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 10 monclient: shutdown 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 -- 172.21.15.181:0/1409048299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 msgr2=0x7f4144069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 --2- 172.21.15.181:0/1409048299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144069d10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f41440a2b00 tx=0x7f41380090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 -- 172.21.15.181:0/1409048299 shutdown_connections 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 --2- 172.21.15.181:0/1409048299 >> v2:172.21.15.153:3300/0 conn(0x7f4144136d60 0x7f4144139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 --2- 172.21.15.181:0/1409048299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144069d10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 -- 172.21.15.181:0/1409048299 >> 172.21.15.181:0/1409048299 conn(0x7f41440a5d40 msgr2=0x7f41440a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 -- 172.21.15.181:0/1409048299 shutdown_connections 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 1 -- 172.21.15.181:0/1409048299 wait complete. 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.622+0000 7f4148880640 10 monclient: monmap: 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:16.635 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 Processor -- start 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 -- start start 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient: init 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f414409ec50 0x7f41441418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): picked mon.noname-c con 0x7f414409ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144098860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): picked mon.noname-b con 0x7f414409f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:16.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4144136d60 0x7f4144098da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): picked mon.noname-a con 0x7f4144136d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414411bd20 con 0x7f4144136d60 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414411b760 con 0x7f414409f020 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41440a4490 con 0x7f414409ec50 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4148880640 10 monclient(hunting): authenticate will time out at 1001.105760s 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4142d9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4144136d60 msgr2=0x7f4144098da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144098860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4142d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4144136d60 0x7f4144098da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144098860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48740/0 (socket says 172.21.15.181:48740) 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 1 -- 172.21.15.181:0/282156431 learned_addr learned my addr 172.21.15.181:0/282156431 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): get_auth_request con 0x7f414409f020 auth_method 0 2024-09-15T03:33:16.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f4141d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f414259c640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f414409ec50 0x7f41441418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f414259c640 10 monclient(hunting): get_auth_request con 0x7f414409ec50 auth_method 0 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f414259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f414259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.623+0000 7f414259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f414259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f414259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f414259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 10 monclient(hunting): handle_auth_done global_id 45034 payload 824 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f414409ec50 msgr2=0x7f41441418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f414409ec50 0x7f41441418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4144136d60 msgr2=0x7f4144098da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4144136d60 0x7f4144098da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 1 monclient: found mon.noname-b 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.624+0000 7f4141d9b640 1 -- 172.21.15.181:0/282156431 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4138009390 con 0x7f414409f020 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4141d9b640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 msgr2=0x7f4144098860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4141d9b640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144098860 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4138009360 tx=0x7f4138032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138000d10 con 0x7f414409f020 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: dump: 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:16.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4138031c70 con 0x7f414409f020 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 5 monclient: authenticate success, global_id 45034 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 10 monclient: _renew_subs 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413803d440 con 0x7f414409f020 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414411c2e0 con 0x7f414409f020 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f41327fc640 10 monclient: dump: 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.642 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4140d99640 4 set_mon_vals no callback set 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 10 monclient: _renew_subs 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414409a230 con 0x7f414409f020 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4138039040 con 0x7f414409f020 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f4148880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f4148880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f4148880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414411be90 con 0x7f414409f020 2024-09-15T03:33:16.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f41327fc640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6361+0+0 (secure 0 0 0) 0x7f413806bda0 con 0x7f414409f020 2024-09-15T03:33:16.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f414259c640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f414259c640 10 monclient: get_auth_request con 0x7f41240386f0 auth_method 0 2024-09-15T03:33:16.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f414259c640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 msgr2=0x7f412403aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:33:16.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f414259c640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 msgr2=0x7f412403aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:16.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f414259c640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:16.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.627+0000 7f414259c640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.632+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4138044af0 con 0x7f414409f020 2024-09-15T03:33:16.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.632+0000 7f41327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.632+0000 7f41327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:16.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.699+0000 7f49e519d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:16.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.699+0000 7f49e519d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:16.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.699+0000 7f49e519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.699+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f49e013d490 con 0x7f49cc005180 2024-09-15T03:33:16.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.700+0000 7f49d6ffd640 1 -- 172.21.15.153:0/2420499947 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f49e013d490 con 0x7f49cc005180 2024-09-15T03:33:16.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.700+0000 7f49d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:16.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.700+0000 7f49d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:16.702 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":50,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49c003a410 msgr2=0x7f49c003c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49c003a410 0x7f49c003c8c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f49d8000f80 tx=0x7f49d800b040 comp rx=0 tx=0).stop 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 10 monclient: shutdown 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 msgr2=0x7f49e0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 0x7f49e0140c10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49c000b060 tx=0x7f49d00091c0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 shutdown_connections 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49c003a410 0x7f49c003c8c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49cc005180 0x7f49e0140c10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49cc004780 0x7f49e013cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 --2- 172.21.15.153:0/2420499947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49cc0039d0 0x7f49e013d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 >> 172.21.15.153:0/2420499947 conn(0x7f49e0067cd0 msgr2=0x7f49e00a18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 shutdown_connections 2024-09-15T03:33:16.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.702+0000 7f49e519d640 1 -- 172.21.15.153:0/2420499947 wait complete. 2024-09-15T03:33:16.715 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:33:16.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:33:16.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:33:16.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.790+0000 7f4148880640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:16.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.790+0000 7f4148880640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:16.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.790+0000 7f4148880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.790+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f4144000940 con 0x7f414409f020 2024-09-15T03:33:16.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.808+0000 7fb4e815d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:16.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.808+0000 7fb4e815d640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.808+0000 7fb4e815d640 10 monclient: monmap: 2024-09-15T03:33:16.810 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:16.810422+0000 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:16.810422+0000 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:16.811 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 Processor -- start 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 -- start start 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient: init 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4e00a1ef0 0x7fb4e009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e00a1ef0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4e009abd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb4e0136d00 0x7fb4e01390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e0136d00 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e006f010 con 0x7fb4e009abd0 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e009da00 con 0x7fb4e0136d00 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e009dcc0 con 0x7fb4e00a1ef0 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e815d640 10 monclient(hunting): authenticate will time out at 994.449894s 2024-09-15T03:33:16.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e56f8640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fb4e009abd0 legacy=0x7fb4e009afd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e66fa640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fb4e0136d00 msgr2=0x7fb4e01390e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.809+0000 7fb4e66fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb4e0136d00 0x7fb4e01390e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4e00a1ef0 0x7fb4e009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb4e00a1ef0 0x7fb4e009c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:58268/0 (socket says 172.21.15.153:58268) 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/430820224 learned_addr learned my addr 172.21.15.153:0/430820224 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): get_auth_request con 0x7fb4e00a1ef0 auth_method 0 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_done global_id 45037 payload 274 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/430820224 >> v2:172.21.15.153:3300/0 conn(0x7fb4e0136d00 msgr2=0x7fb4e01390e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 --2- 172.21.15.153:0/430820224 >> v2:172.21.15.153:3300/0 conn(0x7fb4e0136d00 0x7fb4e01390e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/430820224 >> v1:172.21.15.153:6789/0 conn(0x7fb4e009abd0 legacy=0x7fb4e009afd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 monclient: found mon.noname-c 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:16.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.810+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/430820224 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e00a22c0 con 0x7fb4e00a1ef0 2024-09-15T03:33:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/430820224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e00a1ef0 msgr2=0x7fb4e009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e5ef9640 1 --2- 172.21.15.153:0/430820224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e00a1ef0 0x7fb4e009c6a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00645f0 tx=0x7fb4dc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e2def5a3ac570d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 1 -- 172.21.15.153:0/430820224 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc02ae50 con 0x7fb4e00a1ef0 2024-09-15T03:33:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: dump: 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.826 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 1 -- 172.21.15.153:0/430820224 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc02f4b0 con 0x7fb4e00a1ef0 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 1 -- 172.21.15.153:0/430820224 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc02f850 con 0x7fb4e00a1ef0 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e815d640 5 monclient: authenticate success, global_id 45037 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e815d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e815d640 4 set_mon_vals no callback set 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.813+0000 7fb4e4ef7640 10 monclient: dump: 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 10 monclient: shutdown 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 -- 172.21.15.153:0/430820224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e00a1ef0 msgr2=0x7fb4e009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 --2- 172.21.15.153:0/430820224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e00a1ef0 0x7fb4e009c6a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00645f0 tx=0x7fb4dc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 -- 172.21.15.153:0/430820224 shutdown_connections 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 --2- 172.21.15.153:0/430820224 >> v2:172.21.15.153:3300/0 conn(0x7fb4e0136d00 0x7fb4e01390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 --2- 172.21.15.153:0/430820224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e00a1ef0 0x7fb4e009c6a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 -- 172.21.15.153:0/430820224 >> 172.21.15.153:0/430820224 conn(0x7fb4e00678a0 msgr2=0x7fb4e00a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 -- 172.21.15.153:0/430820224 shutdown_connections 2024-09-15T03:33:16.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 -- 172.21.15.153:0/430820224 wait complete. 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 10 monclient: monmap: 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:16.830 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 Processor -- start 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 1 -- start start 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.814+0000 7fb4e815d640 10 monclient: init 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 0x7fb4e01411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e009abd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a1ef0 0x7fb4e012eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e00a1ef0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0136d00 0x7fb4e0133960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e0136d00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e0126c70 con 0x7fb4e00a1ef0 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e009b910 con 0x7fb4e0136d00 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e012fa20 con 0x7fb4e009abd0 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e815d640 10 monclient(hunting): authenticate will time out at 994.455405s 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e56f8640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a1ef0 msgr2=0x7fb4e012eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e56f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a1ef0 0x7fb4e012eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 0x7fb4e01411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 0x7fb4e01411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51960/0 (socket says 172.21.15.153:51960) 2024-09-15T03:33:16.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/2069036534 learned_addr learned my addr 172.21.15.153:0/2069036534 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): get_auth_request con 0x7fb4e009abd0 auth_method 0 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0136d00 0x7fb4e0133960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): get_auth_request con 0x7fb4e0136d00 auth_method 0 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient(hunting): handle_auth_done global_id 74162 payload 824 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0136d00 msgr2=0x7fb4e0133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0136d00 0x7fb4e0133960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a1ef0 msgr2=0x7fb4e012eaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e00a1ef0 0x7fb4e012eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 monclient: found mon.noname-c 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc009450 con 0x7fb4e009abd0 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.815+0000 7fb4e66fa640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0136d00 0x7fb4e0133960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e5ef9640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 msgr2=0x7fb4e01411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e5ef9640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 0x7fb4e01411e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009940 tx=0x7fb4dc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc0329b0 con 0x7fb4e009abd0 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: dump: 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.835 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 5 monclient: authenticate success, global_id 74162 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc032b30 con 0x7fb4e009abd0 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 10 monclient: _renew_subs 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e0135390 con 0x7fb4e009abd0 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc03b400 con 0x7fb4e009abd0 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d5ffb640 10 monclient: dump: 2024-09-15T03:33:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4d7fff640 4 set_mon_vals no callback set 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 10 monclient: _renew_subs 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.818+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e0142050 con 0x7fb4e009abd0 2024-09-15T03:33:16.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e815d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb4dc043030 con 0x7fb4e009abd0 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e815d640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 0x7fb4e0133ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e0065e00 con 0x7fb4e012efe0 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 0x7fb4e0133ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient: get_auth_request con 0x7fb4e012efe0 auth_method 0 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4d5ffb640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4c40386f0 0x7fb4c403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6361+0+0 (secure 0 0 0) 0x7fb4dc041070 con 0x7fb4e009abd0 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_done global_id 74165 payload 824 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e56f8640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4e01270e0 con 0x7fb4e012efe0 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e66fa640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4c40386f0 0x7fb4c403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.819+0000 7fb4e66fa640 10 monclient: get_auth_request con 0x7fb4c40386f0 auth_method 0 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.821+0000 7fb4e56f8640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 msgr2=0x7fb4e0133ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.821+0000 7fb4e56f8640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 0x7fb4e0133ea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0003940 tx=0x7fb4d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.821+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00050f0 con 0x7fb4e012efe0 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.821+0000 7fb4d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.821+0000 7fb4d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.821+0000 7fb4d5ffb640 10 monclient: dump: 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.839 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4e66fa640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4c40386f0 0x7fb4c403aba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8003cc0 tx=0x7fb4d80096a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb4d0005250 con 0x7fb4e012efe0 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 msgr2=0x7fb4e0133ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 0x7fb4e0133ea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0003940 tx=0x7fb4d00091a0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 msgr2=0x7fb4e0133ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:16.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.822+0000 7fb4d5ffb640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e012efe0 0x7fb4e0133ea0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0003940 tx=0x7fb4d00091a0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.829+0000 7f414259c640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.829+0000 7f414259c640 10 monclient: get_auth_request con 0x7f41240386f0 auth_method 0 2024-09-15T03:33:16.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:16.831+0000 7f414259c640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4134006bf0 tx=0x7f4134005330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.827+0000 7fb4e815d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:16.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.827+0000 7fb4e815d640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 0x7fb4e0131810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.827+0000 7fb4e815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.827+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e0127710 con 0x7fb4e01343e0 2024-09-15T03:33:16.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 0x7fb4e0131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient: get_auth_request con 0x7fb4e01343e0 auth_method 0 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 10 monclient(hunting): handle_auth_done global_id 74168 payload 824 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.828+0000 7fb4e56f8640 1 -- 172.21.15.153:0/2069036534 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4e01347e0 con 0x7fb4e01343e0 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4e56f8640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 msgr2=0x7fb4e0131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4e56f8640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 0x7fb4e0131810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0014000 tx=0x7fb4d0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0018660 con 0x7fb4e01343e0 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 10 monclient: dump: 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fb4d00187c0 con 0x7fb4e01343e0 2024-09-15T03:33:16.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 msgr2=0x7fb4e0131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 0x7fb4e0131810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0014000 tx=0x7fb4d0015000 comp rx=0 tx=0).stop 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 msgr2=0x7fb4e0131810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.830+0000 7fb4d5ffb640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e01343e0 0x7fb4e0131810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:33:16.846 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 50, 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:16.847 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:16.848 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:16.849 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:16.850 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:16.851 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:16.852 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.853 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:16.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:16.856 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:16.857 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:16.858 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:33:16.858 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:16.858 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.858 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:16.858 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:16.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4c40386f0 msgr2=0x7fb4c403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4c40386f0 0x7fb4c403aba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8003cc0 tx=0x7fb4d80096a0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 10 monclient: shutdown 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 msgr2=0x7fb4e01411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 --2- 172.21.15.153:0/2069036534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4e009abd0 0x7fb4e01411e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009940 tx=0x7fb4dc032000 comp rx=0 tx=0).stop 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e66fa640 1 -- 172.21.15.153:0/2069036534 reap_dead start 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e66fa640 1 -- 172.21.15.153:0/2069036534 reap_dead start 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 shutdown_connections 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 >> 172.21.15.153:0/2069036534 conn(0x7fb4e00678a0 msgr2=0x7fb4e00942c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 shutdown_connections 2024-09-15T03:33:16.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.831+0000 7fb4e815d640 1 -- 172.21.15.153:0/2069036534 wait complete. 2024-09-15T03:33:16.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:33:16.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:33:16.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:33:16.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.953+0000 7fa2fc15d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.953+0000 7fa2fc15d640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.953+0000 7fa2fc15d640 10 monclient: monmap: 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:16.955211+0000 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:16.955211+0000 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:16.957 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 Processor -- start 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 -- start start 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient: init 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 0x7fa2f409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f409c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa2f409aba0 0x7fa2f409d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f409aba0 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2f409af70 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f406a320 con 0x7fa2f409af70 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f409a9d0 con 0x7fa2f409aba0 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4068a20 con 0x7fa2f409c860 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2f96f8640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa2f409aba0 msgr2=0x7fa2f409d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2fc15d640 10 monclient(hunting): authenticate will time out at 994.594874s 2024-09-15T03:33:16.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2f96f8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa2f409aba0 0x7fa2f409d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2f9ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 0x7fa2f409cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.954+0000 7fa2f9ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 0x7fa2f409cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52000/0 (socket says 172.21.15.153:52000) 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/2738544122 learned_addr learned my addr 172.21.15.153:0/2738544122 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): get_auth_request con 0x7fa2f409c860 auth_method 0 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_done global_id 74171 payload 274 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/2738544122 >> v2:172.21.15.153:3300/0 conn(0x7fa2f409aba0 msgr2=0x7fa2f409d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:16.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 1 --2- 172.21.15.153:0/2738544122 >> v2:172.21.15.153:3300/0 conn(0x7fa2f409aba0 0x7fa2f409d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/2738544122 >> v1:172.21.15.181:6789/0 conn(0x7fa2f409af70 legacy=0x7fa2f409dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 1 monclient: found mon.noname-b 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.955+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/2738544122 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4068e60 con 0x7fa2f409c860 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.957+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/2738544122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 msgr2=0x7fa2f409cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.957+0000 7fa2f9ef9640 1 --2- 172.21.15.153:0/2738544122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 0x7fa2f409cc30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009910 tx=0x7fa2ec0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e3ee5ee8dd8352d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 1 -- 172.21.15.153:0/2738544122 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec032440 con 0x7fa2f409c860 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: dump: 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.970 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 5 monclient: authenticate success, global_id 74171 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 1 -- 172.21.15.153:0/2738544122 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec0325a0 con 0x7fa2f409c860 2024-09-15T03:33:16.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 4 set_mon_vals no callback set 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 1 -- 172.21.15.153:0/2738544122 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec032940 con 0x7fa2f409c860 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2f8ef7640 10 monclient: dump: 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 10 monclient: shutdown 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 -- 172.21.15.153:0/2738544122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 msgr2=0x7fa2f409cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/2738544122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 0x7fa2f409cc30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009910 tx=0x7fa2ec0091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 -- 172.21.15.153:0/2738544122 shutdown_connections 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/2738544122 >> v2:172.21.15.153:3300/0 conn(0x7fa2f409aba0 0x7fa2f409d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/2738544122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409c860 0x7fa2f409cc30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 -- 172.21.15.153:0/2738544122 >> 172.21.15.153:0/2738544122 conn(0x7fa2f40a2e40 msgr2=0x7fa2f40a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 -- 172.21.15.153:0/2738544122 shutdown_connections 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.958+0000 7fa2fc15d640 1 -- 172.21.15.153:0/2738544122 wait complete. 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.959+0000 7fa2fc15d640 10 monclient: build_initial_monmap 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.959+0000 7fa2fc15d640 10 monclient: monmap: 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:16.974 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.959+0000 7fa2fc15d640 1 Processor -- start 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.959+0000 7fa2fc15d640 1 -- start start 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.959+0000 7fa2fc15d640 10 monclient: init 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 0x7fa2f4141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f409aba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f409af70 0x7fa2f4131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f409af70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f409c860 0x7fa2f406d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f409c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f411f020 con 0x7fa2f409af70 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4099cc0 con 0x7fa2f409c860 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4134ac0 con 0x7fa2f409aba0 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f96f8640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f409af70 msgr2=0x7fa2f4131c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fc15d640 10 monclient(hunting): authenticate will time out at 994.600083s 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f96f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f409af70 0x7fa2f4131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 0x7fa2f4141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 0x7fa2f4141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52008/0 (socket says 172.21.15.153:52008) 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/3524861367 learned_addr learned my addr 172.21.15.153:0/3524861367 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): get_auth_request con 0x7fa2f409aba0 auth_method 0 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f409c860 0x7fa2f406d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): get_auth_request con 0x7fa2f409c860 auth_method 0 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2fa6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient(hunting): handle_auth_done global_id 74174 payload 824 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f409c860 msgr2=0x7fa2f406d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f409c860 0x7fa2f406d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f409af70 msgr2=0x7fa2f4131c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f409af70 0x7fa2f4131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 monclient: found mon.noname-c 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.960+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec009450 con 0x7fa2f409aba0 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2f9ef9640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 msgr2=0x7fa2f4141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2f9ef9640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 0x7fa2f4141010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009970 tx=0x7fa2ec032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec031a10 con 0x7fa2f409aba0 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: dump: 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec031b70 con 0x7fa2f409aba0 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2fc15d640 5 monclient: authenticate success, global_id 74174 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2fc15d640 10 monclient: _renew_subs 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2fc15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f411eda0 con 0x7fa2f409aba0 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec03c410 con 0x7fa2f409aba0 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:16.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2e9ffb640 10 monclient: dump: 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 10 monclient: _renew_subs 2024-09-15T03:33:16.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.962+0000 7fa2ebfff640 4 set_mon_vals no callback set 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f4141de0 con 0x7fa2f409aba0 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa2ec039050 con 0x7fa2f409aba0 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 0x7fa2f406dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4065e00 con 0x7fa2f4132160 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2e9ffb640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2d80386f0 0x7fa2d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2f96f8640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 0x7fa2f406dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6361+0+0 (secure 0 0 0) 0x7fa2ec06c580 con 0x7fa2f409aba0 2024-09-15T03:33:16.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2f96f8640 10 monclient: get_auth_request con 0x7fa2f4132160 auth_method 0 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2f96f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.963+0000 7fa2f96f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2f96f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2fa6fa640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2d80386f0 0x7fa2d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2fa6fa640 10 monclient: get_auth_request con 0x7fa2d80386f0 auth_method 0 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_done global_id 45046 payload 824 2024-09-15T03:33:16.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.964+0000 7fa2f96f8640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2f4141660 con 0x7fa2f4132160 2024-09-15T03:33:16.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2f96f8640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 msgr2=0x7fa2f406dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2f96f8640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 0x7fa2f406dc10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400b440 tx=0x7fa2e4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2fa6fa640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2d80386f0 0x7fa2d803aba0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0007fd0 tx=0x7fa2f00025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4003e70 con 0x7fa2f4132160 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 10 monclient: dump: 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.988 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa2e40050f0 con 0x7fa2f4132160 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 msgr2=0x7fa2f406dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 0x7fa2f406dc10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400b440 tx=0x7fa2e4009120 comp rx=0 tx=0).stop 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 msgr2=0x7fa2f406dc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.966+0000 7fa2e9ffb640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f4132160 0x7fa2f406dc10 secure :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400b440 tx=0x7fa2e4009120 comp rx=0 tx=0).stop 2024-09-15T03:33:16.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.972+0000 7fa2fc15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.972+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 0x7fa2f4097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.972+0000 7fa2fc15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.972+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f406e550 con 0x7fa2f406e150 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 0x7fa2f4097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient: get_auth_request con 0x7fa2f406e150 auth_method 0 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:16.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 10 monclient(hunting): handle_auth_done global_id 45049 payload 824 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.973+0000 7fa2f96f8640 1 -- 172.21.15.153:0/3524861367 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa2f404bf60 con 0x7fa2f406e150 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2f96f8640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 msgr2=0x7fa2f4097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2f96f8640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 0x7fa2f4097a70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4003fd0 tx=0x7fa2e4015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e401e620 con 0x7fa2f406e150 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 10 monclient: dump: 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:16.991 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fa2e401e780 con 0x7fa2f406e150 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 msgr2=0x7fa2f4097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 0x7fa2f4097a70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4003fd0 tx=0x7fa2e4015000 comp rx=0 tx=0).stop 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 msgr2=0x7fa2f4097a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:16.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.975+0000 7fa2e9ffb640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f406e150 0x7fa2f4097a70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 50, 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.993 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:16.994 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:16.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:16.996 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:16.999 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:17.000 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:17.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:17.002 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:17.003 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:17.004 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.005 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:33:17.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:17.007 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:17.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2d80386f0 msgr2=0x7fa2d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:17.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2d80386f0 0x7fa2d803aba0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0007fd0 tx=0x7fa2f00025b0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 10 monclient: shutdown 2024-09-15T03:33:17.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:17.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 msgr2=0x7fa2f4141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:17.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 0x7fa2f4141010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009970 tx=0x7fa2ec032000 comp rx=0 tx=0).stop 2024-09-15T03:33:17.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fa6fa640 1 -- 172.21.15.153:0/3524861367 reap_dead start 2024-09-15T03:33:17.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 shutdown_connections 2024-09-15T03:33:17.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 --2- 172.21.15.153:0/3524861367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2f409aba0 0x7fa2f4141010 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 >> 172.21.15.153:0/3524861367 conn(0x7fa2f40a2e40 msgr2=0x7fa2f4134cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:17.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 shutdown_connections 2024-09-15T03:33:17.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:16.976+0000 7fa2fc15d640 1 -- 172.21.15.153:0/3524861367 wait complete. 2024-09-15T03:33:17.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:33:17.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:33:17.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.522+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v109) v1 ==== 220+0+0 (secure 0 0 0) 0x7f4144000940 con 0x7f414409f020 2024-09-15T03:33:17.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.522+0000 7f41327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:17.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.522+0000 7f41327fc640 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-15T03:33:17.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.622+0000 7f4148880640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:17.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.623+0000 7f4148880640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:17.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.623+0000 7f4148880640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:17.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.623+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f4144000940 con 0x7f414409f020 2024-09-15T03:33:17.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.623+0000 7f41327fc640 1 -- 172.21.15.181:0/282156431 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v109) v1 ==== 218+0+0 (secure 0 0 0) 0x7f4144000940 con 0x7f414409f020 2024-09-15T03:33:17.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.623+0000 7f41327fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:17.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.623+0000 7f41327fc640 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T03:33:17.626 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 msgr2=0x7f412403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4134006bf0 tx=0x7f4134005330 comp rx=0 tx=0).stop 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 10 monclient: shutdown 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 msgr2=0x7f4144098860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144098860 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4138009360 tx=0x7f4138032000 comp rx=0 tx=0).stop 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 shutdown_connections 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f41240386f0 0x7f412403aba0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4144136d60 0x7f4144098da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f414409f020 0x7f4144098860 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 --2- 172.21.15.181:0/282156431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f414409ec50 0x7f41441418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 >> 172.21.15.181:0/282156431 conn(0x7f41440a5d40 msgr2=0x7f414409de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:17.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 shutdown_connections 2024-09-15T03:33:17.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.626+0000 7f4148880640 1 -- 172.21.15.181:0/282156431 wait complete. 2024-09-15T03:33:17.636 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-15T03:33:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.710+0000 7f1dec857640 10 monclient: get_monmap_and_config 2024-09-15T03:33:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.710+0000 7f1dec857640 10 monclient: build_initial_monmap 2024-09-15T03:33:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.710+0000 7f1dec857640 10 monclient: monmap: 2024-09-15T03:33:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:17.712640+0000 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:17.712640+0000 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.711+0000 7f1dec857640 1 Processor -- start 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.711+0000 7f1dec857640 1 -- start start 2024-09-15T03:33:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.711+0000 7f1dec857640 10 monclient: init 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1de809cad0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 0x7f1de81348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): picked mon.noname-b con 0x7f1de809c1a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1de8134e10 0x7f1de81371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): picked mon.noname-a con 0x7f1de8134e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de809d5b0 con 0x7f1de809cad0 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de809b710 con 0x7f1de8134e10 2024-09-15T03:33:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de809a650 con 0x7f1de809c1a0 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1dec857640 10 monclient(hunting): authenticate will time out at 1002.194268s 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de659c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f1de809cad0 legacy=0x7f1de809cea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de6d9d640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f1de8134e10 msgr2=0x7f1de81371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de6d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1de8134e10 0x7f1de81371f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 0x7f1de81348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 0x7f1de81348d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:54966/0 (socket says 172.21.15.181:54966) 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 -- 172.21.15.181:0/2859811099 learned_addr learned my addr 172.21.15.181:0/2859811099 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): get_auth_request con 0x7f1de809c1a0 auth_method 0 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient(hunting): handle_auth_done global_id 74177 payload 274 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 -- 172.21.15.181:0/2859811099 >> v2:172.21.15.153:3300/0 conn(0x7f1de8134e10 msgr2=0x7f1de81371f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 --2- 172.21.15.181:0/2859811099 >> v2:172.21.15.153:3300/0 conn(0x7f1de8134e10 0x7f1de81371f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 -- 172.21.15.181:0/2859811099 >> v1:172.21.15.153:6789/0 conn(0x7f1de809cad0 legacy=0x7f1de809cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 monclient: found mon.noname-b 2024-09-15T03:33:17.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.712+0000 7f1de5d9b640 1 -- 172.21.15.181:0/2859811099 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de809f660 con 0x7f1de809c1a0 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.715+0000 7f1de5d9b640 1 -- 172.21.15.181:0/2859811099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 msgr2=0x7f1de81348d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.715+0000 7f1de5d9b640 1 --2- 172.21.15.181:0/2859811099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 0x7f1de81348d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009910 tx=0x7f1ddc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fffa1fd0fcaf4d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 1 -- 172.21.15.181:0/2859811099 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc032440 con 0x7f1de809c1a0 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: dump: 2024-09-15T03:33:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:17.723 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: _finish_auth 0 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 5 monclient: authenticate success, global_id 74177 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 1 -- 172.21.15.181:0/2859811099 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0325a0 con 0x7f1de809c1a0 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 4 set_mon_vals no callback set 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 1 -- 172.21.15.181:0/2859811099 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc032940 con 0x7f1de809c1a0 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1de559a640 10 monclient: dump: 2024-09-15T03:33:17.724 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 10 monclient: shutdown 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 -- 172.21.15.181:0/2859811099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 msgr2=0x7f1de81348d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 --2- 172.21.15.181:0/2859811099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 0x7f1de81348d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009910 tx=0x7f1ddc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 -- 172.21.15.181:0/2859811099 shutdown_connections 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 --2- 172.21.15.181:0/2859811099 >> v2:172.21.15.153:3300/0 conn(0x7f1de8134e10 0x7f1de81371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 --2- 172.21.15.181:0/2859811099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de809c1a0 0x7f1de81348d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 -- 172.21.15.181:0/2859811099 >> 172.21.15.181:0/2859811099 conn(0x7f1de8070e90 msgr2=0x7f1de80a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 -- 172.21.15.181:0/2859811099 shutdown_connections 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.716+0000 7f1dec857640 1 -- 172.21.15.181:0/2859811099 wait complete. 2024-09-15T03:33:17.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient: build_initial_monmap 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient: monmap: 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:17.727 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:17.728 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:17.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 Processor -- start 2024-09-15T03:33:17.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 -- start start 2024-09-15T03:33:17.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient: init 2024-09-15T03:33:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): picked mon.noname-b con 0x7f1de809c1a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de809cad0 0x7f1de813f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): picked mon.noname-a con 0x7f1de809cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de8134e10 0x7f1de813fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): picked mon.noname-c con 0x7f1de8134e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80a2710 con 0x7f1de809cad0 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de811eb00 con 0x7f1de809c1a0 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de806eba0 con 0x7f1de8134e10 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.717+0000 7f1dec857640 10 monclient(hunting): authenticate will time out at 1002.199939s 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de5d9b640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de809cad0 msgr2=0x7f1de813f7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de5d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de809cad0 0x7f1de813f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48742/0 (socket says 172.21.15.181:48742) 2024-09-15T03:33:17.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 -- 172.21.15.181:0/1690374391 learned_addr learned my addr 172.21.15.181:0/1690374391 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): get_auth_request con 0x7f1de809c1a0 auth_method 0 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de8134e10 0x7f1de813fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:17.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): get_auth_request con 0x7f1de8134e10 auth_method 0 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient(hunting): handle_auth_done global_id 45052 payload 824 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 -- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de8134e10 msgr2=0x7f1de813fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de8134e10 0x7f1de813fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 -- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de809cad0 msgr2=0x7f1de813f7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de809cad0 0x7f1de813f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 monclient: found mon.noname-b 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.718+0000 7f1de659c640 1 -- 172.21.15.181:0/1690374391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc009450 con 0x7f1de809c1a0 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1de659c640 1 -- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 msgr2=0x7f1de813f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1de659c640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b630 tx=0x7f1dd80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8015070 con 0x7f1de809c1a0 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1de659c640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b630 tx=0x7f1dd80091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f1dd8003d90 config(0 keys) v1 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 10 monclient: dump: 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:17.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd8003d90 con 0x7f1de809c1a0 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 5 monclient: authenticate success, global_id 45052 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 10 monclient: _renew_subs 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de811e610 con 0x7f1de809c1a0 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dd67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1de4d99640 4 set_mon_vals no callback set 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 10 monclient: _renew_subs 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.720+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de81201f0 con 0x7f1de809c1a0 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8006380 con 0x7f1de809c1a0 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:17.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dd67fc640 10 monclient: dump: 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1dd8010440 con 0x7f1de809c1a0 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dec857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dec857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dec857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de80008d0 con 0x7f1de809c1a0 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1dd67fc640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dc80386f0 0x7f1dc803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1de5d9b640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dc80386f0 0x7f1dc803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.721+0000 7f1de5d9b640 10 monclient: get_auth_request con 0x7f1dc80386f0 auth_method 0 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.724+0000 7f1de5d9b640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dc80386f0 0x7f1dc803aba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc031000 tx=0x7f1ddc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:17.726+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(109..109 src has 40..109) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f1dd804bcd0 con 0x7f1de809c1a0 2024-09-15T03:33:18.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.343+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1dd8006770 con 0x7f1de809c1a0 2024-09-15T03:33:18.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.343+0000 7f1dd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:18.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.343+0000 7f1dd67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:18.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.503+0000 7f1dec857640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:18.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.503+0000 7f1dec857640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:18.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.503+0000 7f1dec857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.503+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f1de8064a20 con 0x7f1de809c1a0 2024-09-15T03:33:18.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.615+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v110) v1 ==== 220+0+0 (secure 0 0 0) 0x7f1dd8018780 con 0x7f1de809c1a0 2024-09-15T03:33:18.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.615+0000 7f1dd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:18.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.615+0000 7f1dd67fc640 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-15T03:33:18.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dec857640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:18.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dec857640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dec857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f1de811d800 con 0x7f1de809c1a0 2024-09-15T03:33:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dd67fc640 1 -- 172.21.15.181:0/1690374391 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v110) v1 ==== 218+0+0 (secure 0 0 0) 0x7f1dd8018d60 con 0x7f1de809c1a0 2024-09-15T03:33:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dd67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-15T03:33:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.711+0000 7f1dd67fc640 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T03:33:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-15T03:33:18.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dc80386f0 msgr2=0x7f1dc803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dc80386f0 0x7f1dc803aba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc031000 tx=0x7f1ddc032000 comp rx=0 tx=0).stop 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 10 monclient: shutdown 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 msgr2=0x7f1de813f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b630 tx=0x7f1dd80091a0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 shutdown_connections 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1dc80386f0 0x7f1dc803aba0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1de8134e10 0x7f1de813fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de809cad0 0x7f1de813f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 --2- 172.21.15.181:0/1690374391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1de809c1a0 0x7f1de813f280 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 >> 172.21.15.181:0/1690374391 conn(0x7f1de8070e90 msgr2=0x7f1de806e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:18.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 shutdown_connections 2024-09-15T03:33:18.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.713+0000 7f1dec857640 1 -- 172.21.15.181:0/1690374391 wait complete. 2024-09-15T03:33:18.724 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:33:18.724 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep osd.12 2024-09-15T03:33:18.724 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 123 2024-09-15T03:33:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.793+0000 7f21acd9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.793+0000 7f21acd9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.793+0000 7f21acd9d640 10 monclient: monmap: 2024-09-15T03:33:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:18.794 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:18.794176+0000 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:18.794176+0000 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:18.795 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:18.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.793+0000 7f21acd9d640 1 Processor -- start 2024-09-15T03:33:18.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.793+0000 7f21acd9d640 1 -- start start 2024-09-15T03:33:18.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.793+0000 7f21acd9d640 10 monclient: init 2024-09-15T03:33:18.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:18.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:33:18.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21a80a37b0 0x7f21a806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f21a80a37b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a809d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f21a806be40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21a806a6f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a8099ff0 con 0x7f21a806a6f0 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a8069d10 con 0x7f21a80a37b0 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80a3b80 con 0x7f21a806be40 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:18.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21acd9d640 10 monclient(hunting): authenticate will time out at 1003.275662s 2024-09-15T03:33:18.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21a80a37b0 0x7f21a806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:18.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21a80a37b0 0x7f21a806e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39676/0 (socket says 172.21.15.181:39676) 2024-09-15T03:33:18.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 1 -- 172.21.15.181:0/3304287668 learned_addr learned my addr 172.21.15.181:0/3304287668 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:18.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): get_auth_request con 0x7f21a80a37b0 auth_method 0 2024-09-15T03:33:18.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:18.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a6d9d640 1 -- 172.21.15.181:0/3304287668 >> v1:172.21.15.153:6789/0 conn(0x7f21a806a6f0 legacy=0x7f21a809dea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a5d9b640 1 --2- 172.21.15.181:0/3304287668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a809d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a5d9b640 10 monclient(hunting): get_auth_request con 0x7f21a806be40 auth_method 0 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.794+0000 7f21a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 10 monclient(hunting): handle_auth_done global_id 45055 payload 274 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 1 -- 172.21.15.181:0/3304287668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 msgr2=0x7f21a809d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:18.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 1 --2- 172.21.15.181:0/3304287668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a809d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 1 -- 172.21.15.181:0/3304287668 >> v1:172.21.15.153:6789/0 conn(0x7f21a806a6f0 legacy=0x7f21a809dea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 1 monclient: found mon.noname-c 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.795+0000 7f21a659c640 1 -- 172.21.15.181:0/3304287668 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a809f520 con 0x7f21a80a37b0 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.797+0000 7f21a659c640 1 -- 172.21.15.181:0/3304287668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 msgr2=0x7f21a806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.797+0000 7f21a659c640 1 --2- 172.21.15.181:0/3304287668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a806e2d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2190009910 tx=0x7f21900091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ae02a3eba6c397f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 1 -- 172.21.15.181:0/3304287668 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219002aea0 con 0x7f21a80a37b0 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: dump: 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:18.809 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: _finish_auth 0 2024-09-15T03:33:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 1 -- 172.21.15.181:0/3304287668 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2190032440 con 0x7f21a80a37b0 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 5 monclient: authenticate success, global_id 45055 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 1 -- 172.21.15.181:0/3304287668 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21900327e0 con 0x7f21a80a37b0 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 4 set_mon_vals no callback set 2024-09-15T03:33:18.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:18.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21a559a640 10 monclient: dump: 2024-09-15T03:33:18.812 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 10 monclient: shutdown 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:18.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 -- 172.21.15.181:0/3304287668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 msgr2=0x7f21a806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 --2- 172.21.15.181:0/3304287668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a806e2d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2190009910 tx=0x7f21900091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 -- 172.21.15.181:0/3304287668 shutdown_connections 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 --2- 172.21.15.181:0/3304287668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a809d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 --2- 172.21.15.181:0/3304287668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a806e2d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 -- 172.21.15.181:0/3304287668 >> 172.21.15.181:0/3304287668 conn(0x7f21a8070fd0 msgr2=0x7f21a80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 -- 172.21.15.181:0/3304287668 shutdown_connections 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.798+0000 7f21acd9d640 1 -- 172.21.15.181:0/3304287668 wait complete. 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient: monmap: 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:18.814 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 Processor -- start 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 -- start start 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient: init 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a806a6f0 0x7f21a8135010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:18.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f21a806a6f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a8133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f21a806be40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a811e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f21a80a37b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8122580 con 0x7f21a806a6f0 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a806eb10 con 0x7f21a80a37b0 2024-09-15T03:33:18.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81221a0 con 0x7f21a806be40 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.799+0000 7f21acd9d640 10 monclient(hunting): authenticate will time out at 1003.280931s 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a659c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a806a6f0 msgr2=0x7f21a8135010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a811e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a806a6f0 0x7f21a8135010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a811e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39684/0 (socket says 172.21.15.181:39684) 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 -- 172.21.15.181:0/1010569407 learned_addr learned my addr 172.21.15.181:0/1010569407 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): get_auth_request con 0x7f21a80a37b0 auth_method 0 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:18.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a8133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): get_auth_request con 0x7f21a806be40 auth_method 0 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient(hunting): handle_auth_done global_id 45058 payload 824 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 msgr2=0x7f21a8133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:18.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a8133d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a806a6f0 msgr2=0x7f21a8135010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a806a6f0 0x7f21a8135010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 monclient: found mon.noname-b 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.800+0000 7f21a6d9d640 1 -- 172.21.15.181:0/1010569407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190009450 con 0x7f21a80a37b0 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21a6d9d640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 msgr2=0x7f21a811e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21a6d9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a811e870 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f21a000bb10 tx=0x7f21a00097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a000cbc0 con 0x7f21a80a37b0 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: dump: 2024-09-15T03:33:18.819 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a000cd40 con 0x7f21a80a37b0 2024-09-15T03:33:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 5 monclient: authenticate success, global_id 45058 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0006150 con 0x7f21a80a37b0 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f219e7fc640 10 monclient: dump: 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:18.821 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 10 monclient: _renew_subs 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a8141d60 con 0x7f21a80a37b0 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21a4d99640 4 set_mon_vals no callback set 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 10 monclient: _renew_subs 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.802+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a81342d0 con 0x7f21a80a37b0 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f21a0012020 con 0x7f21a80a37b0 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f21acd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f21acd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:18.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f21acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a80636e0 con 0x7f21a80a37b0 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f219e7fc640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(110..110 src has 40..110) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f21a004c130 con 0x7f21a80a37b0 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f21a659c640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.803+0000 7f21a659c640 10 monclient: get_auth_request con 0x7f21880386f0 auth_method 0 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.804+0000 7f21a659c640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 msgr2=0x7f218803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.804+0000 7f21a659c640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 msgr2=0x7f218803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.804+0000 7f21a659c640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f21a809d680 tx=0x7f2190032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:18.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.804+0000 7f21a659c640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:18.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.808+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f21a001daa0 con 0x7f21a80a37b0 2024-09-15T03:33:18.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.808+0000 7f219e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:18.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.808+0000 7f219e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:18.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f21acd9d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:18.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f21acd9d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:18.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f21acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:18.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f21a8064b20 con 0x7f21a80a37b0 2024-09-15T03:33:18.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f21a001d680 con 0x7f21a80a37b0 2024-09-15T03:33:18.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f219e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:18.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:18.962+0000 7f219e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:19.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.005+0000 7f21a659c640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:19.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.005+0000 7f21a659c640 10 monclient: get_auth_request con 0x7f21880386f0 auth_method 0 2024-09-15T03:33:19.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.007+0000 7f21a659c640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2190009910 tx=0x7f219003c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:19.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.054+0000 7f21acd9d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.054+0000 7f21acd9d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.054+0000 7f21acd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.054+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f21a8121ec0 con 0x7f21a80a37b0 2024-09-15T03:33:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.055+0000 7f219e7fc640 1 -- 172.21.15.181:0/1010569407 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f21a001daa0 con 0x7f21a80a37b0 2024-09-15T03:33:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.055+0000 7f219e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.055+0000 7f219e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 msgr2=0x7f218803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2190009910 tx=0x7f219003c000 comp rx=0 tx=0).stop 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 10 monclient: shutdown 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 msgr2=0x7f21a811e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a811e870 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f21a000bb10 tx=0x7f21a00097d0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 shutdown_connections 2024-09-15T03:33:19.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f21880386f0 0x7f218803aba0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a80a37b0 0x7f21a811e870 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a806be40 0x7f21a8133d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 --2- 172.21.15.181:0/1010569407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a806a6f0 0x7f21a8135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.056+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 >> 172.21.15.181:0/1010569407 conn(0x7f21a8070fd0 msgr2=0x7f21a806d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:19.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.057+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 shutdown_connections 2024-09-15T03:33:19.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.057+0000 7f21acd9d640 1 -- 172.21.15.181:0/1010569407 wait complete. 2024-09-15T03:33:19.066 INFO:tasks.workunit.client.0.smithi181.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-15T03:33:19.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:33:19.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep osd.14 2024-09-15T03:33:19.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 123 2024-09-15T03:33:19.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.136+0000 7fab99f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.136+0000 7fab99f5d640 10 monclient: build_initial_monmap 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.136+0000 7fab99f5d640 10 monclient: monmap: 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:19.137546+0000 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:19.137546+0000 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:19.139 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 1 Processor -- start 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 1 -- start start 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 10 monclient: init 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab9409a780 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab94134d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fab941300f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.137+0000 7fab99f5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fab94130500 0x7fab94137260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fab94130500 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9406ec50 con 0x7fab9409a780 2024-09-15T03:33:19.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab94097a40 con 0x7fab94130500 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9409c570 con 0x7fab941300f0 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab99f5d640 10 monclient(hunting): authenticate will time out at 1003.619045s 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab937fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fab9409a780 0x7fab9412f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40684/0 (socket says 172.21.15.181:40684) 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab937fe640 1 -- 172.21.15.181:0/1318253260 learned_addr learned my addr 172.21.15.181:0/1318253260 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab93fff640 1 -- 172.21.15.181:0/1318253260 >> v2:172.21.15.153:3300/0 conn(0x7fab94130500 msgr2=0x7fab94137260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab93fff640 1 --2- 172.21.15.181:0/1318253260 >> v2:172.21.15.153:3300/0 conn(0x7fab94130500 0x7fab94137260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 1 --2- 172.21.15.181:0/1318253260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab94134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): get_auth_request con 0x7fab941300f0 auth_method 0 2024-09-15T03:33:19.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient(hunting): handle_auth_done global_id 74189 payload 274 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 1 -- 172.21.15.181:0/1318253260 >> v2:172.21.15.153:3300/0 conn(0x7fab94130500 msgr2=0x7fab94137260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 1 --2- 172.21.15.181:0/1318253260 >> v2:172.21.15.153:3300/0 conn(0x7fab94130500 0x7fab94137260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 1 -- 172.21.15.181:0/1318253260 >> v1:172.21.15.181:6789/0 conn(0x7fab9409a780 legacy=0x7fab9412f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 1 monclient: found mon.noname-b 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:19.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.138+0000 7fab92ffd640 1 -- 172.21.15.181:0/1318253260 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab94097de0 con 0x7fab941300f0 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab92ffd640 1 -- 172.21.15.181:0/1318253260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 msgr2=0x7fab94134d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab92ffd640 1 --2- 172.21.15.181:0/1318253260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab94134d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab84009850 tx=0x7fab840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e6e54e38155b799 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 1 -- 172.21.15.181:0/1318253260 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84032440 con 0x7fab941300f0 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 10 monclient: dump: 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:19.150 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 1 -- 172.21.15.181:0/1318253260 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab840325a0 con 0x7fab941300f0 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab99f5d640 5 monclient: authenticate success, global_id 74189 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab99f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.141+0000 7fab927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab927fc640 1 -- 172.21.15.181:0/1318253260 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84032940 con 0x7fab941300f0 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 4 set_mon_vals no callback set 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:19.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab927fc640 10 monclient: dump: 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 10 monclient: shutdown 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 -- 172.21.15.181:0/1318253260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 msgr2=0x7fab94134d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 --2- 172.21.15.181:0/1318253260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab94134d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab84009850 tx=0x7fab840090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 -- 172.21.15.181:0/1318253260 shutdown_connections 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 --2- 172.21.15.181:0/1318253260 >> v2:172.21.15.153:3300/0 conn(0x7fab94130500 0x7fab94137260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 --2- 172.21.15.181:0/1318253260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab94134d10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 -- 172.21.15.181:0/1318253260 >> 172.21.15.181:0/1318253260 conn(0x7fab940714d0 msgr2=0x7fab940a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 -- 172.21.15.181:0/1318253260 shutdown_connections 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.142+0000 7fab99f5d640 1 -- 172.21.15.181:0/1318253260 wait complete. 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.143+0000 7fab99f5d640 10 monclient: build_initial_monmap 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.143+0000 7fab99f5d640 10 monclient: monmap: 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:19.153 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.143+0000 7fab99f5d640 1 Processor -- start 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.143+0000 7fab99f5d640 1 -- start start 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.143+0000 7fab99f5d640 10 monclient: init 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fab9409a780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab9412c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fab941300f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab94130500 0x7fab9412ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fab94130500 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941187d0 con 0x7fab94130500 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9406ede0 con 0x7fab9409a780 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab94132a90 con 0x7fab941300f0 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab99f5d640 10 monclient(hunting): authenticate will time out at 1003.625212s 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab93fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab94130500 msgr2=0x7fab9412ce80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39696/0 (socket says 172.21.15.181:39696) 2024-09-15T03:33:19.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 -- 172.21.15.181:0/3747103219 learned_addr learned my addr 172.21.15.181:0/3747103219 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): get_auth_request con 0x7fab9409a780 auth_method 0 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab93fff640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab94130500 0x7fab9412ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab92ffd640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab9412c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient(hunting): handle_auth_done global_id 45061 payload 824 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 -- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 msgr2=0x7fab9412c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab9412c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 -- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab94130500 msgr2=0x7fab9412ce80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:19.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab94130500 0x7fab9412ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:19.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:19.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:19.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:19.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:19.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.144+0000 7fab937fe640 1 -- 172.21.15.181:0/3747103219 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84009390 con 0x7fab9409a780 2024-09-15T03:33:19.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab937fe640 1 -- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 msgr2=0x7fab9406d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:19.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab937fe640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fab8800b510 tx=0x7fab880090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:19.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88011840 con 0x7fab9409a780 2024-09-15T03:33:19.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:19.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:19.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab937fe640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fab8800b510 tx=0x7fab880090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fab880119a0 config(0 keys) v1 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 10 monclient: dump: 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:19.160 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab99f5d640 5 monclient: authenticate success, global_id 45061 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab99f5d640 10 monclient: _renew_subs 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab880119a0 con 0x7fab9409a780 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab99f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9413fe30 con 0x7fab9409a780 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab99f5d640 10 monclient: _renew_subs 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab99f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab94134390 con 0x7fab9409a780 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.146+0000 7fab91ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88011d60 con 0x7fab9409a780 2024-09-15T03:33:19.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab7b7fe640 10 monclient: dump: 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fab88010590 con 0x7fab9409a780 2024-09-15T03:33:19.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab99f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab99f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab99f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab7b7fe640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab800386f0 0x7fab8003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.147+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94119f40 con 0x7fab9409a780 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.148+0000 7fab92ffd640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab800386f0 0x7fab8003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.148+0000 7fab92ffd640 10 monclient: get_auth_request con 0x7fab800386f0 auth_method 0 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.150+0000 7fab92ffd640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab800386f0 0x7fab8003aba0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fab940a1c00 tx=0x7fab84032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:19.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.152+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(110..110 src has 40..110) v4 ==== 6567+0+0 (secure 0 0 0) 0x7fab8804c3f0 con 0x7fab9409a780 2024-09-15T03:33:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.769+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fab8800f4c0 con 0x7fab9409a780 2024-09-15T03:33:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.769+0000 7fab7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.769+0000 7fab7b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab99f5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab99f5d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab99f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fab940636e0 con 0x7fab9409a780 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fab880155c0 con 0x7fab9409a780 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:19.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:19.925+0000 7fab7b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:20.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab99f5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:20.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab99f5d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab99f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fab94119ac0 con 0x7fab9409a780 2024-09-15T03:33:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab7b7fe640 1 -- 172.21.15.181:0/3747103219 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fab88015070 con 0x7fab9409a780 2024-09-15T03:33:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab7b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:20.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.018+0000 7fab7b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab800386f0 msgr2=0x7fab8003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab800386f0 0x7fab8003aba0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fab940a1c00 tx=0x7fab84032000 comp rx=0 tx=0).stop 2024-09-15T03:33:20.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 10 monclient: shutdown 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 msgr2=0x7fab9406d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fab8800b510 tx=0x7fab880090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 shutdown_connections 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab800386f0 0x7fab8003aba0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab94130500 0x7fab9412ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab941300f0 0x7fab9412c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 --2- 172.21.15.181:0/3747103219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9409a780 0x7fab9406d550 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 >> 172.21.15.181:0/3747103219 conn(0x7fab940714d0 msgr2=0x7fab94132ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 shutdown_connections 2024-09-15T03:33:20.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.020+0000 7fab99f5d640 1 -- 172.21.15.181:0/3747103219 wait complete. 2024-09-15T03:33:20.030 INFO:tasks.workunit.client.0.smithi181.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-15T03:33:20.030 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-15T03:33:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.105+0000 7f1588b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.105+0000 7f1588b9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.105+0000 7f1588b9d640 10 monclient: monmap: 2024-09-15T03:33:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:20.105816+0000 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:20.105816+0000 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:20.108 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.105+0000 7f1588b9d640 1 Processor -- start 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.105+0000 7f1588b9d640 1 -- start start 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.105+0000 7f1588b9d640 10 monclient: init 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1584131c90 0x7f1584132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:20.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1584131c90 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f158409d210 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f15840a0ab0 0x7f15841325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f15840a0ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15840a1800 con 0x7f158409d210 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f158409fea0 con 0x7f1584131c90 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840a1520 con 0x7f15840a0ab0 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1588b9d640 10 monclient(hunting): authenticate will time out at 1004.587347s 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f15840a0ab0 0x7f15841325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1581d9b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f158409d210 0x7f1584068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40688/0 (socket says 172.21.15.181:40688) 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1581d9b640 1 -- 172.21.15.181:0/3460685708 learned_addr learned my addr 172.21.15.181:0/3460685708 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:20.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f15840a0ab0 0x7f15841325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39710/0 (socket says 172.21.15.181:39710) 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): get_auth_request con 0x7f15840a0ab0 auth_method 0 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f158259c640 1 -- 172.21.15.181:0/3460685708 >> v2:172.21.15.153:3300/0 conn(0x7f1584131c90 msgr2=0x7f1584132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f158259c640 1 --2- 172.21.15.181:0/3460685708 >> v2:172.21.15.153:3300/0 conn(0x7f1584131c90 0x7f1584132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient(hunting): handle_auth_done global_id 45064 payload 274 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 -- 172.21.15.181:0/3460685708 >> v2:172.21.15.153:3300/0 conn(0x7f1584131c90 msgr2=0x7f1584132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 --2- 172.21.15.181:0/3460685708 >> v2:172.21.15.153:3300/0 conn(0x7f1584131c90 0x7f1584132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 -- 172.21.15.181:0/3460685708 >> v1:172.21.15.181:6789/0 conn(0x7f158409d210 legacy=0x7f1584068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:20.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 monclient: found mon.noname-c 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.106+0000 7f1582d9d640 1 -- 172.21.15.181:0/3460685708 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1584069510 con 0x7f15840a0ab0 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f1582d9d640 1 -- 172.21.15.181:0/3460685708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15840a0ab0 msgr2=0x7f15841325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f1582d9d640 1 --2- 172.21.15.181:0/3460685708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15840a0ab0 0x7f15841325a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1584076b50 tx=0x7f157c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b377d56c32dcfc25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 1 -- 172.21.15.181:0/3460685708 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c007840 con 0x7f15840a0ab0 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: dump: 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.117 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: _finish_auth 0 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 1 -- 172.21.15.181:0/3460685708 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c02d050 con 0x7f15840a0ab0 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f1588b9d640 5 monclient: authenticate success, global_id 45064 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f1588b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f1588b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f1588b9d640 4 set_mon_vals no callback set 2024-09-15T03:33:20.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 1 -- 172.21.15.181:0/3460685708 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c007bd0 con 0x7f15840a0ab0 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.109+0000 7f158159a640 10 monclient: dump: 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:20.119 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 10 monclient: shutdown 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 -- 172.21.15.181:0/3460685708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15840a0ab0 msgr2=0x7f15841325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3460685708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15840a0ab0 0x7f15841325a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1584076b50 tx=0x7f157c009780 comp rx=0 tx=0).stop 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 -- 172.21.15.181:0/3460685708 shutdown_connections 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3460685708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15840a0ab0 0x7f15841325a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3460685708 >> v2:172.21.15.153:3300/0 conn(0x7f1584131c90 0x7f1584132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 -- 172.21.15.181:0/3460685708 >> 172.21.15.181:0/3460685708 conn(0x7f1584071370 msgr2=0x7f15840a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 -- 172.21.15.181:0/3460685708 shutdown_connections 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 1 -- 172.21.15.181:0/3460685708 wait complete. 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.110+0000 7f1588b9d640 10 monclient: monmap: 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 Processor -- start 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 -- start start 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient: init 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15840a0ab0 0x7f1584141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f15840a0ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f1584131c90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f158411e470 0x7f158411e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f158411e470 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15840a0fb0 con 0x7f158411e470 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158409ee80 con 0x7f1584131c90 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15841220d0 con 0x7f15840a0ab0 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1588b9d640 10 monclient(hunting): authenticate will time out at 1004.592554s 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39724/0 (socket says 172.21.15.181:39724) 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 -- 172.21.15.181:0/3628279928 learned_addr learned my addr 172.21.15.181:0/3628279928 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): get_auth_request con 0x7f1584131c90 auth_method 0 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1582d9d640 1 -- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f158411e470 msgr2=0x7f158411e8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1582d9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f158411e470 0x7f158411e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f158259c640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15840a0ab0 0x7f1584141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient(hunting): handle_auth_done global_id 45067 payload 824 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 -- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15840a0ab0 msgr2=0x7f1584141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15840a0ab0 0x7f1584141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 -- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f158411e470 msgr2=0x7f158411e8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f158411e470 0x7f158411e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 monclient: found mon.noname-b 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f1581d9b640 1 -- 172.21.15.181:0/3628279928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c009a20 con 0x7f1584131c90 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f158259c640 10 monclient: get_auth_request con 0x7f15840a0ab0 auth_method 0 2024-09-15T03:33:20.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.111+0000 7f158259c640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15840a0ab0 0x7f1584141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1581d9b640 1 -- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 msgr2=0x7f15841418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1581d9b640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f157800c610 tx=0x7f1578004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578005cb0 con 0x7f1584131c90 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: dump: 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:20.127 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 5 monclient: authenticate success, global_id 45067 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1578005e30 con 0x7f1584131c90 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 10 monclient: _renew_subs 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1584122d20 con 0x7f1584131c90 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578007340 con 0x7f1584131c90 2024-09-15T03:33:20.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f15727fc640 10 monclient: dump: 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 10 monclient: _renew_subs 2024-09-15T03:33:20.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1584120050 con 0x7f1584131c90 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.114+0000 7f1580d99640 4 set_mon_vals no callback set 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f157801b400 con 0x7f1584131c90 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f1588b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f1588b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f1588b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15840008d0 con 0x7f1584131c90 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f15727fc640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15640386f0 0x7f156403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(110..110 src has 40..110) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f157804bf60 con 0x7f1584131c90 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f158259c640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15640386f0 0x7f156403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.115+0000 7f158259c640 10 monclient: get_auth_request con 0x7f15640386f0 auth_method 0 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.117+0000 7f158259c640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15640386f0 0x7f156403aba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f15740038f0 tx=0x7f1574008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:20.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.119+0000 7f1581d9b640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f157800c610 tx=0x7f1578004960 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f157801a720 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:33:20.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.947+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f157801a720 con 0x7f1584131c90 2024-09-15T03:33:20.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.947+0000 7f15727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:20.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:20.947+0000 7f15727fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:21.112+0000 7f1588b9d640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T03:33:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:21.112+0000 7f1588b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T03:33:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:21.112+0000 7f1588b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:21.112+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f1584069830 con 0x7f1584131c90 2024-09-15T03:33:22.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.072+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v111) v1 ==== 168+0+0 (secure 0 0 0) 0x7f157801a360 con 0x7f1584131c90 2024-09-15T03:33:22.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.072+0000 7f15727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T03:33:22.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.072+0000 7f15727fc640 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-15T03:33:22.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.174+0000 7f1588b9d640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T03:33:22.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.174+0000 7f1588b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T03:33:22.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.174+0000 7f1588b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:22.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:22.174+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f158409de50 con 0x7f1584131c90 2024-09-15T03:33:23.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.073+0000 7f15727fc640 1 -- 172.21.15.181:0/3628279928 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v112) v1 ==== 168+0+0 (secure 0 0 0) 0x7f157801a540 con 0x7f1584131c90 2024-09-15T03:33:23.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.073+0000 7f15727fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-15T03:33:23.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.073+0000 7f15727fc640 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-15T03:33:23.074 INFO:tasks.workunit.client.0.smithi181.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-15T03:33:23.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15640386f0 msgr2=0x7f156403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15640386f0 0x7f156403aba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f15740038f0 tx=0x7f1574008040 comp rx=0 tx=0).stop 2024-09-15T03:33:23.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 10 monclient: shutdown 2024-09-15T03:33:23.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 msgr2=0x7f15841418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f157800c610 tx=0x7f1578004960 comp rx=0 tx=0).stop 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 shutdown_connections 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15640386f0 0x7f156403aba0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f158411e470 0x7f158411e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1584131c90 0x7f15841418c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 --2- 172.21.15.181:0/3628279928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15840a0ab0 0x7f1584141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 >> 172.21.15.181:0/3628279928 conn(0x7f1584071370 msgr2=0x7f15840a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 shutdown_connections 2024-09-15T03:33:23.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.076+0000 7f1588b9d640 1 -- 172.21.15.181:0/3628279928 wait complete. 2024-09-15T03:33:23.088 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-15T03:33:23.088 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:33:23.089 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.166+0000 7f008a2ef640 10 monclient: get_monmap_and_config 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.166+0000 7f008a2ef640 10 monclient: build_initial_monmap 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.166+0000 7f008a2ef640 10 monclient: monmap: 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:23.167113+0000 2024-09-15T03:33:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:23.167113+0000 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:23.168 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 Processor -- start 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 -- start start 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient: init 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0084096a30 0x7f0084099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): picked mon.noname-a con 0x7f0084096a30 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f008409d530 0x7f0084096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): picked mon.noname-b con 0x7f008409d530 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0084097340 0x7f008412e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): picked mon.noname-c con 0x7f0084097340 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0084097e80 con 0x7f0084096a30 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0084098be0 con 0x7f0084097340 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0084098ec0 con 0x7f008409d530 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008a2ef640 10 monclient(hunting): authenticate will time out at 1007.648614s 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f0083fff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0084096a30 msgr2=0x7f0084099d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0084097340 0x7f008412e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f0083fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0084096a30 0x7f0084099d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0084097340 0x7f008412e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39726/0 (socket says 172.21.15.181:39726) 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 1 -- 172.21.15.181:0/3230722494 learned_addr learned my addr 172.21.15.181:0/3230722494 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): get_auth_request con 0x7f0084097340 auth_method 0 2024-09-15T03:33:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.167+0000 7f008888c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 10 monclient(hunting): handle_auth_done global_id 45070 payload 274 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 1 -- 172.21.15.181:0/3230722494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f008409d530 msgr2=0x7f0084096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f00837fe640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f008409d530 0x7f0084096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f008409d530 0x7f0084096e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 1 -- 172.21.15.181:0/3230722494 >> v2:172.21.15.153:3300/0 conn(0x7f0084096a30 msgr2=0x7f0084099d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 1 --2- 172.21.15.181:0/3230722494 >> v2:172.21.15.153:3300/0 conn(0x7f0084096a30 0x7f0084099d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 1 monclient: found mon.noname-c 2024-09-15T03:33:23.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:23.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f008888c640 1 -- 172.21.15.181:0/3230722494 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008409cd10 con 0x7f0084097340 2024-09-15T03:33:23.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f00837fe640 10 monclient: get_auth_request con 0x7f008409d530 auth_method 0 2024-09-15T03:33:23.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.168+0000 7f00837fe640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f008409d530 0x7f0084096e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:23.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.170+0000 7f008888c640 1 -- 172.21.15.181:0/3230722494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0084097340 msgr2=0x7f008412e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:23.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.170+0000 7f008888c640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0084097340 0x7f008412e7b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f008409d170 tx=0x7f007c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=498564c0031eb183 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 1 -- 172.21.15.181:0/3230722494 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c007610 con 0x7f0084097340 2024-09-15T03:33:23.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: dump: 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.181 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: _finish_auth 0 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 1 -- 172.21.15.181:0/3230722494 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c02d050 con 0x7f0084097340 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 5 monclient: authenticate success, global_id 45070 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 1 -- 172.21.15.181:0/3230722494 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c0079a0 con 0x7f0084097340 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 4 set_mon_vals no callback set 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f0082ffd640 10 monclient: dump: 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 10 monclient: shutdown 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 -- 172.21.15.181:0/3230722494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0084097340 msgr2=0x7f008412e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0084097340 0x7f008412e7b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f008409d170 tx=0x7f007c009840 comp rx=0 tx=0).stop 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 -- 172.21.15.181:0/3230722494 shutdown_connections 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0084097340 0x7f008412e7b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 --2- 172.21.15.181:0/3230722494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f008409d530 0x7f0084096e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 --2- 172.21.15.181:0/3230722494 >> v2:172.21.15.153:3300/0 conn(0x7f0084096a30 0x7f0084099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 -- 172.21.15.181:0/3230722494 >> 172.21.15.181:0/3230722494 conn(0x7f00840a0fe0 msgr2=0x7f00840a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 -- 172.21.15.181:0/3230722494 shutdown_connections 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.171+0000 7f008a2ef640 1 -- 172.21.15.181:0/3230722494 wait complete. 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.172+0000 7f008a2ef640 10 monclient: build_initial_monmap 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.172+0000 7f008a2ef640 10 monclient: monmap: 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:23.184 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.172+0000 7f008a2ef640 1 Processor -- start 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.172+0000 7f008a2ef640 1 -- start start 2024-09-15T03:33:23.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.172+0000 7f008a2ef640 10 monclient: init 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0084096a30 0x7f008413d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): picked mon.noname-a con 0x7f0084096a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0084097340 0x7f008413d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): picked mon.noname-c con 0x7f0084097340 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008409d530 0x7f0084140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): picked mon.noname-b con 0x7f008409d530 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084122940 con 0x7f0084096a30 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084122380 con 0x7f008409d530 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008411fe10 con 0x7f0084097340 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008a2ef640 10 monclient(hunting): authenticate will time out at 1007.654267s 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f0083fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0084096a30 msgr2=0x7f008413d950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f0083fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0084096a30 0x7f008413d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008409d530 0x7f0084140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008409d530 0x7f0084140ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39740/0 (socket says 172.21.15.181:39740) 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 -- 172.21.15.181:0/2054645193 learned_addr learned my addr 172.21.15.181:0/2054645193 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): get_auth_request con 0x7f008409d530 auth_method 0 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f00837fe640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0084097340 0x7f008413d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f00837fe640 10 monclient(hunting): get_auth_request con 0x7f0084097340 auth_method 0 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f00837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f00837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f00837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient(hunting): handle_auth_done global_id 45073 payload 824 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 -- 172.21.15.181:0/2054645193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0084097340 msgr2=0x7f008413d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0084097340 0x7f008413d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 -- 172.21.15.181:0/2054645193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0084096a30 msgr2=0x7f008413d950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0084096a30 0x7f008413d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 monclient: found mon.noname-b 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.173+0000 7f008888c640 1 -- 172.21.15.181:0/2054645193 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c009ae0 con 0x7f008409d530 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.175+0000 7f008888c640 1 -- 172.21.15.181:0/2054645193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008409d530 msgr2=0x7f0084140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.175+0000 7f008888c640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008409d530 0x7f0084140ec0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f007c009880 tx=0x7f007c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c0094e0 con 0x7f008409d530 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: dump: 2024-09-15T03:33:23.189 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: _finish_auth 0 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 5 monclient: authenticate success, global_id 45073 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c032cc0 con 0x7f008409d530 2024-09-15T03:33:23.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 10 monclient: _renew_subs 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 1 -- 172.21.15.181:0/2054645193 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00841220a0 con 0x7f008409d530 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c03b660 con 0x7f008409d530 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 10 monclient: dump: 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.191 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 10 monclient: _renew_subs 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f008a2ef640 1 -- 172.21.15.181:0/2054645193 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008413d540 con 0x7f008409d530 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f00827fc640 4 set_mon_vals no callback set 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.176+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f007c038050 con 0x7f008409d530 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f006bfff640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f00700386f0 0x7f007003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(112..112 src has 40..112) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f007c06cec0 con 0x7f008409d530 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f0083fff640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f00700386f0 0x7f007003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f0083fff640 10 monclient: get_auth_request con 0x7f00700386f0 auth_method 0 2024-09-15T03:33:23.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f008a2ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f008a2ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f008a2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.177+0000 7f008a2ef640 1 -- 172.21.15.181:0/2054645193 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008404bf60 con 0x7f008409d530 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.179+0000 7f0083fff640 1 --2- 172.21.15.181:0/2054645193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f00700386f0 0x7f007003aba0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f00740038f0 tx=0x7f0074006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.181+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f007c03f080 con 0x7f008409d530 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.181+0000 7f006bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.181+0000 7f006bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f008a2ef640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-15T03:33:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f008a2ef640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-15T03:33:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f008a2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f008a2ef640 1 -- 172.21.15.181:0/2054645193 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) v1 -- 0x7f0084064b40 con 0x7f008409d530 2024-09-15T03:33:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f006bfff640 1 -- 172.21.15.181:0/2054645193 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v112) v1 ==== 154+0+0 (secure 0 0 0) 0x7f007c049020 con 0x7f008409d530 2024-09-15T03:33:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f006bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-15T03:33:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.343+0000 7f006bfff640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-15T03:33:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-15T03:33:23.348 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:33:23.349 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:33:23.349 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep osd.12 2024-09-15T03:33:23.349 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 155 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.418+0000 7fc893c6f640 10 monclient: get_monmap_and_config 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.418+0000 7fc893c6f640 10 monclient: build_initial_monmap 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.419+0000 7fc893c6f640 10 monclient: monmap: 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:23.419787+0000 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:23.419787+0000 2024-09-15T03:33:23.420 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:23.421 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.419+0000 7fc893c6f640 1 Processor -- start 2024-09-15T03:33:23.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.419+0000 7fc893c6f640 1 -- start start 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.419+0000 7fc893c6f640 10 monclient: init 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc88c0972c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c09a7a0 0x7fc88c09be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): picked mon.noname-b con 0x7fc88c09a7a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:23.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc88c133680 0x7fc88c135a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): picked mon.noname-c con 0x7fc88c133680 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c097690 con 0x7fc88c0972c0 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c09d6a0 con 0x7fc88c133680 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c09c880 con 0x7fc88c09a7a0 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc893c6f640 10 monclient(hunting): authenticate will time out at 1007.901294s 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc891a0b640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fc88c0972c0 legacy=0x7fc88c09b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc88c133680 0x7fc88c135a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc88c133680 0x7fc88c135a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39752/0 (socket says 172.21.15.181:39752) 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 -- 172.21.15.181:0/3935529749 learned_addr learned my addr 172.21.15.181:0/3935529749 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): get_auth_request con 0x7fc88c133680 auth_method 0 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89120a640 1 --2- 172.21.15.181:0/3935529749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c09a7a0 0x7fc88c09be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89120a640 10 monclient(hunting): get_auth_request con 0x7fc88c09a7a0 auth_method 0 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89120a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89120a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89120a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient(hunting): handle_auth_done global_id 45076 payload 274 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 -- 172.21.15.181:0/3935529749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c09a7a0 msgr2=0x7fc88c09be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 --2- 172.21.15.181:0/3935529749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c09a7a0 0x7fc88c09be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 -- 172.21.15.181:0/3935529749 >> v1:172.21.15.153:6789/0 conn(0x7fc88c0972c0 legacy=0x7fc88c09b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 monclient: found mon.noname-c 2024-09-15T03:33:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.420+0000 7fc89220c640 1 -- 172.21.15.181:0/3935529749 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c098500 con 0x7fc88c133680 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc89220c640 1 -- 172.21.15.181:0/3935529749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 msgr2=0x7fc88c135a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc89220c640 1 --2- 172.21.15.181:0/3935529749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c135a60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc884009f20 tx=0x7fc884009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=264ba81a48e49018 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 1 -- 172.21.15.181:0/3935529749 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884007610 con 0x7fc88c133680 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: dump: 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.441 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: _finish_auth 0 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc893c6f640 5 monclient: authenticate success, global_id 45076 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc893c6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 1 -- 172.21.15.181:0/3935529749 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88402d050 con 0x7fc88c133680 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc893c6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:23.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc893c6f640 4 set_mon_vals no callback set 2024-09-15T03:33:23.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 1 -- 172.21.15.181:0/3935529749 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8840079a0 con 0x7fc88c133680 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.423+0000 7fc890a09640 10 monclient: dump: 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 10 monclient: shutdown 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 -- 172.21.15.181:0/3935529749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 msgr2=0x7fc88c135a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 --2- 172.21.15.181:0/3935529749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c135a60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc884009f20 tx=0x7fc884009840 comp rx=0 tx=0).stop 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 -- 172.21.15.181:0/3935529749 shutdown_connections 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 --2- 172.21.15.181:0/3935529749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c135a60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 --2- 172.21.15.181:0/3935529749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c09a7a0 0x7fc88c09be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 -- 172.21.15.181:0/3935529749 >> 172.21.15.181:0/3935529749 conn(0x7fc88c0714d0 msgr2=0x7fc88c0a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 -- 172.21.15.181:0/3935529749 shutdown_connections 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 1 -- 172.21.15.181:0/3935529749 wait complete. 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 10 monclient: build_initial_monmap 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.424+0000 7fc893c6f640 10 monclient: monmap: 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 Processor -- start 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 -- start start 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient: init 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c0972c0 0x7fc88c1174e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): picked mon.noname-c con 0x7fc88c0972c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88c09a7a0 0x7fc88c117a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): picked mon.noname-a con 0x7fc88c09a7a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c117f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): picked mon.noname-b con 0x7fc88c133680 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c118740 con 0x7fc88c09a7a0 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c11e8d0 con 0x7fc88c133680 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c09e6c0 con 0x7fc88c0972c0 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc893c6f640 10 monclient(hunting): authenticate will time out at 1007.906545s 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c117f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89120a640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88c09a7a0 msgr2=0x7fc88c117a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c117f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39766/0 (socket says 172.21.15.181:39766) 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 -- 172.21.15.181:0/1480957783 learned_addr learned my addr 172.21.15.181:0/1480957783 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): get_auth_request con 0x7fc88c133680 auth_method 0 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89120a640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88c09a7a0 0x7fc88c117a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c0972c0 0x7fc88c1174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): get_auth_request con 0x7fc88c0972c0 auth_method 0 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc891a0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient(hunting): handle_auth_done global_id 45079 payload 824 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 -- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c0972c0 msgr2=0x7fc88c1174e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c0972c0 0x7fc88c1174e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 -- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88c09a7a0 msgr2=0x7fc88c117a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88c09a7a0 0x7fc88c117a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 monclient: found mon.noname-b 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.425+0000 7fc89220c640 1 -- 172.21.15.181:0/1480957783 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884009ae0 con 0x7fc88c133680 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc89220c640 1 -- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 msgr2=0x7fc88c117f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc89220c640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c117f60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc884009ab0 tx=0x7fc884036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884032c60 con 0x7fc88c133680 2024-09-15T03:33:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: dump: 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc884032dc0 con 0x7fc88c133680 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 5 monclient: authenticate success, global_id 45079 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 10 monclient: _renew_subs 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c11e5f0 con 0x7fc88c133680 2024-09-15T03:33:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88403b640 con 0x7fc88c133680 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87dffb640 10 monclient: dump: 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc87ffff640 4 set_mon_vals no callback set 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 10 monclient: _renew_subs 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.428+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c06e170 con 0x7fc88c133680 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc884003020 con 0x7fc88c133680 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc893c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc893c6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc893c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c11ecd0 con 0x7fc88c133680 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc87dffb640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8700386f0 0x7fc87003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(112..112 src has 40..112) v4 ==== 6567+0+0 (secure 0 0 0) 0x7fc88402ae40 con 0x7fc88c133680 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc891a0b640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8700386f0 0x7fc87003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.429+0000 7fc891a0b640 10 monclient: get_auth_request con 0x7fc8700386f0 auth_method 0 2024-09-15T03:33:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.432+0000 7fc891a0b640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8700386f0 0x7fc87003aba0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8800038f0 tx=0x7fc8800090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.435+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc8840396c0 con 0x7fc88c133680 2024-09-15T03:33:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.435+0000 7fc87dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.435+0000 7fc87dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:23.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc893c6f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:23.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc893c6f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:23.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc893c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc88c0636e0 con 0x7fc88c133680 2024-09-15T03:33:23.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fc88403f020 con 0x7fc88c133680 2024-09-15T03:33:23.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc87dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:23.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.590+0000 7fc87dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.683+0000 7fc893c6f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.683+0000 7fc893c6f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.683+0000 7fc893c6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.683+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc88c1327d0 con 0x7fc88c133680 2024-09-15T03:33:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.683+0000 7fc87dffb640 1 -- 172.21.15.181:0/1480957783 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fc884037280 con 0x7fc88c133680 2024-09-15T03:33:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.684+0000 7fc87dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.684+0000 7fc87dffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8700386f0 msgr2=0x7fc87003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8700386f0 0x7fc87003aba0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8800038f0 tx=0x7fc8800090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 10 monclient: shutdown 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 msgr2=0x7fc88c117f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c117f60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc884009ab0 tx=0x7fc884036000 comp rx=0 tx=0).stop 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 shutdown_connections 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8700386f0 0x7fc87003aba0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc88c133680 0x7fc88c117f60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88c09a7a0 0x7fc88c117a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.685+0000 7fc893c6f640 1 --2- 172.21.15.181:0/1480957783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc88c0972c0 0x7fc88c1174e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.686+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 >> 172.21.15.181:0/1480957783 conn(0x7fc88c0714d0 msgr2=0x7fc88c136460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:23.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.686+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 shutdown_connections 2024-09-15T03:33:23.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.686+0000 7fc893c6f640 1 -- 172.21.15.181:0/1480957783 wait complete. 2024-09-15T03:33:23.695 INFO:tasks.workunit.client.0.smithi181.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-15T03:33:23.696 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd tree 2024-09-15T03:33:23.696 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep osd.14 2024-09-15T03:33:23.696 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 155 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.765+0000 7f04168e8640 10 monclient: get_monmap_and_config 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.765+0000 7f04168e8640 10 monclient: build_initial_monmap 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.765+0000 7f04168e8640 10 monclient: monmap: 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:23.766635+0000 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:23.766635+0000 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:23.768 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:23.768 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:23.768 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:23.768 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.766+0000 7f04168e8640 1 Processor -- start 2024-09-15T03:33:23.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.766+0000 7f04168e8640 1 -- start start 2024-09-15T03:33:23.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.766+0000 7f04168e8640 10 monclient: init 2024-09-15T03:33:23.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:23.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f041009cb20 0x7f041009afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): picked mon.noname-c con 0x7f041009cb20 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f041012ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): picked mon.noname-b con 0x7f041012e940 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f041012f290 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041009d6f0 con 0x7f041012f290 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0410097e90 con 0x7f041009cb20 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0410099640 con 0x7f041012e940 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f04168e8640 10 monclient(hunting): authenticate will time out at 1008.248225s 2024-09-15T03:33:23.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f041009cb20 0x7f041009afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f0414e85640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f041012f290 0x7f0410096a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40696/0 (socket says 172.21.15.181:40696) 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f0414e85640 1 -- 172.21.15.181:0/1945864540 learned_addr learned my addr 172.21.15.181:0/1945864540 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): get_auth_request con 0x7f041009cb20 auth_method 0 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040f7fe640 1 --2- 172.21.15.181:0/1945864540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f041012ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040f7fe640 10 monclient(hunting): get_auth_request con 0x7f041012e940 auth_method 0 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient(hunting): handle_auth_done global_id 45082 payload 274 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 1 -- 172.21.15.181:0/1945864540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 msgr2=0x7f041012ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 1 --2- 172.21.15.181:0/1945864540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f041012ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 1 -- 172.21.15.181:0/1945864540 >> v1:172.21.15.181:6789/0 conn(0x7f041012f290 legacy=0x7f0410096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 1 monclient: found mon.noname-c 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.767+0000 7f040ffff640 1 -- 172.21.15.181:0/1945864540 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410098da0 con 0x7f041009cb20 2024-09-15T03:33:23.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040ffff640 1 -- 172.21.15.181:0/1945864540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 msgr2=0x7f041009afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040ffff640 1 --2- 172.21.15.181:0/1945864540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f041009afb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0400009850 tx=0x7f04000090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e440d9772a1cb2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 1 -- 172.21.15.181:0/1945864540 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040002adf0 con 0x7f041009cb20 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: dump: 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.779 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: _finish_auth 0 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 1 -- 172.21.15.181:0/1945864540 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0400032440 con 0x7f041009cb20 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f04168e8640 5 monclient: authenticate success, global_id 45082 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f04168e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f04168e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f04168e8640 4 set_mon_vals no callback set 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 1 -- 172.21.15.181:0/1945864540 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000327e0 con 0x7f041009cb20 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.770+0000 7f040effd640 10 monclient: dump: 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 10 monclient: shutdown 2024-09-15T03:33:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 -- 172.21.15.181:0/1945864540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 msgr2=0x7f041009afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 --2- 172.21.15.181:0/1945864540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f041009afb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0400009850 tx=0x7f04000090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 -- 172.21.15.181:0/1945864540 shutdown_connections 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 --2- 172.21.15.181:0/1945864540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f041012ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 --2- 172.21.15.181:0/1945864540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f041009afb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 -- 172.21.15.181:0/1945864540 >> 172.21.15.181:0/1945864540 conn(0x7f04100714d0 msgr2=0x7f04100a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 -- 172.21.15.181:0/1945864540 shutdown_connections 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 1 -- 172.21.15.181:0/1945864540 wait complete. 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 10 monclient: build_initial_monmap 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.771+0000 7f04168e8640 10 monclient: monmap: 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 Processor -- start 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 -- start start 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient: init 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f0410131c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): picked mon.noname-b con 0x7f041009cb20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:23.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f0410132190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): picked mon.noname-c con 0x7f041012e940 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f041012f290 0x7f04101326d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): picked mon.noname-a con 0x7f041012f290 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041009dd60 con 0x7f041012f290 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410133d20 con 0x7f041009cb20 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0410098950 con 0x7f041012e940 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f04168e8640 10 monclient(hunting): authenticate will time out at 1008.253815s 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f0410131c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f0410131c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39778/0 (socket says 172.21.15.181:39778) 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 1 -- 172.21.15.181:0/2541288080 learned_addr learned my addr 172.21.15.181:0/2541288080 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): get_auth_request con 0x7f041009cb20 auth_method 0 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f0414e85640 1 -- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f041012f290 msgr2=0x7f04101326d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f0414e85640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f041012f290 0x7f04101326d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.772+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 10 monclient(hunting): handle_auth_done global_id 45085 payload 824 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 1 -- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 msgr2=0x7f0410132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:23.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040f7fe640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f0410132190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f0410132190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 1 -- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f041012f290 msgr2=0x7f04101326d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f041012f290 0x7f04101326d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 1 monclient: found mon.noname-b 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.773+0000 7f040ffff640 1 -- 172.21.15.181:0/2541288080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400009390 con 0x7f041009cb20 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f040ffff640 1 -- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 msgr2=0x7f0410131c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f040ffff640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f0410131c50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f04000098b0 tx=0x7f0400032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400031af0 con 0x7f041009cb20 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: dump: 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: _finish_auth 0 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 5 monclient: authenticate success, global_id 45085 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0400031c70 con 0x7f041009cb20 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 10 monclient: _renew_subs 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0410135340 con 0x7f041009cb20 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040003c400 con 0x7f041009cb20 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f03f7fff640 10 monclient: dump: 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.788 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 10 monclient: _renew_subs 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0410137730 con 0x7f041009cb20 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.775+0000 7f040e7fc640 4 set_mon_vals no callback set 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f040003c870 con 0x7f041009cb20 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f03f7fff640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03fc0386f0 0x7f03fc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(112..112 src has 40..112) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f040006cb20 con 0x7f041009cb20 2024-09-15T03:33:23.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f040f7fe640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03fc0386f0 0x7f03fc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:23.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f04168e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f04168e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f04168e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04100636e0 con 0x7f041009cb20 2024-09-15T03:33:23.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.776+0000 7f040f7fe640 10 monclient: get_auth_request con 0x7f03fc0386f0 auth_method 0 2024-09-15T03:33:23.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.779+0000 7f040f7fe640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03fc0386f0 0x7f03fc03aba0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0404006fd0 tx=0x7f0404008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:23.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.781+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f040003b3a0 con 0x7f041009cb20 2024-09-15T03:33:23.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.781+0000 7f03f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:23.791 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.781+0000 7f03f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:23.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f04168e8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:23.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f04168e8640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:23.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f04168e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:23.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f041009e800 con 0x7f041009cb20 2024-09-15T03:33:23.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f0400037070 con 0x7f041009cb20 2024-09-15T03:33:23.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f03f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-15T03:33:23.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:23.936+0000 7f03f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.028+0000 7f04168e8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.028+0000 7f04168e8640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.029+0000 7f04168e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.029+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f041009e800 con 0x7f041009cb20 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.029+0000 7f03f7fff640 1 -- 172.21.15.181:0/2541288080 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f040004a720 con 0x7f041009cb20 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.029+0000 7f03f7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-15T03:33:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.029+0000 7f03f7fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03fc0386f0 msgr2=0x7f03fc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03fc0386f0 0x7f03fc03aba0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0404006fd0 tx=0x7f0404008040 comp rx=0 tx=0).stop 2024-09-15T03:33:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 10 monclient: shutdown 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 msgr2=0x7f0410131c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f0410131c50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f04000098b0 tx=0x7f0400032000 comp rx=0 tx=0).stop 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 shutdown_connections 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f03fc0386f0 0x7f03fc03aba0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f041012f290 0x7f04101326d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f041012e940 0x7f0410132190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 --2- 172.21.15.181:0/2541288080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f041009cb20 0x7f0410131c50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 >> 172.21.15.181:0/2541288080 conn(0x7f04100714d0 msgr2=0x7f041009fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 shutdown_connections 2024-09-15T03:33:24.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.031+0000 7f04168e8640 1 -- 172.21.15.181:0/2541288080 wait complete. 2024-09-15T03:33:24.041 INFO:tasks.workunit.client.0.smithi181.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-15T03:33:24.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.12 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.114+0000 7fcb792a8640 10 monclient: get_monmap_and_config 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.114+0000 7fcb792a8640 10 monclient: build_initial_monmap 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.114+0000 7fcb792a8640 10 monclient: monmap: 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:24.115019+0000 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:24.115019+0000 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:24.116 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:24.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.114+0000 7fcb792a8640 1 Processor -- start 2024-09-15T03:33:24.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.114+0000 7fcb792a8640 1 -- start start 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.114+0000 7fcb792a8640 10 monclient: init 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcb7409ec50 0x7fcb74131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7409ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb741321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb74067420 con 0x7fcb741321a0 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb740a0480 con 0x7fcb7409ec50 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:24.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740a0200 con 0x7fcb7409f020 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb792a8640 10 monclient(hunting): authenticate will time out at 1008.596427s 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb72ffd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fcb7409ec50 msgr2=0x7fcb74131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb72ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcb7409ec50 0x7fcb74131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb727fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fcb741321a0 0x7fcb74068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40710/0 (socket says 172.21.15.181:40710) 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb727fc640 1 -- 172.21.15.181:0/2959876504 learned_addr learned my addr 172.21.15.181:0/2959876504 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 1 --2- 172.21.15.181:0/2959876504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): get_auth_request con 0x7fcb7409f020 auth_method 0 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.115+0000 7fcb737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 10 monclient(hunting): handle_auth_done global_id 74204 payload 274 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 1 -- 172.21.15.181:0/2959876504 >> v2:172.21.15.153:3300/0 conn(0x7fcb7409ec50 msgr2=0x7fcb74131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 1 --2- 172.21.15.181:0/2959876504 >> v2:172.21.15.153:3300/0 conn(0x7fcb7409ec50 0x7fcb74131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 1 -- 172.21.15.181:0/2959876504 >> v1:172.21.15.181:6789/0 conn(0x7fcb741321a0 legacy=0x7fcb74068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:24.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.116+0000 7fcb737fe640 1 -- 172.21.15.181:0/2959876504 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7409fd80 con 0x7fcb7409f020 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb737fe640 1 -- 172.21.15.181:0/2959876504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 msgr2=0x7fcb74069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb737fe640 1 --2- 172.21.15.181:0/2959876504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74069410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009850 tx=0x7fcb6c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66fcd038a3a2cbb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 1 -- 172.21.15.181:0/2959876504 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c032440 con 0x7fcb7409f020 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: dump: 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:24.127 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 1 -- 172.21.15.181:0/2959876504 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c0325a0 con 0x7fcb7409f020 2024-09-15T03:33:24.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 5 monclient: authenticate success, global_id 74204 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 1 -- 172.21.15.181:0/2959876504 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c032940 con 0x7fcb7409f020 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 4 set_mon_vals no callback set 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb71ffb640 10 monclient: dump: 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.129 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 10 monclient: shutdown 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 1 -- 172.21.15.181:0/2959876504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 msgr2=0x7fcb74069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 1 --2- 172.21.15.181:0/2959876504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74069410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009850 tx=0x7fcb6c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 1 -- 172.21.15.181:0/2959876504 shutdown_connections 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 1 --2- 172.21.15.181:0/2959876504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74069410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.119+0000 7fcb792a8640 1 --2- 172.21.15.181:0/2959876504 >> v2:172.21.15.153:3300/0 conn(0x7fcb7409ec50 0x7fcb74131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 1 -- 172.21.15.181:0/2959876504 >> 172.21.15.181:0/2959876504 conn(0x7fcb740a5cf0 msgr2=0x7fcb740a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 1 -- 172.21.15.181:0/2959876504 shutdown_connections 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 1 -- 172.21.15.181:0/2959876504 wait complete. 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 10 monclient: build_initial_monmap 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 10 monclient: monmap: 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:24.131 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 1 Processor -- start 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 1 -- start start 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.120+0000 7fcb792a8640 10 monclient: init 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb7409ec50 0x7fcb7409aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb741321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:24.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7411c7c0 con 0x7fcb7409ec50 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7411dce0 con 0x7fcb741321a0 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7411dfc0 con 0x7fcb7409f020 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb792a8640 10 monclient(hunting): authenticate will time out at 1008.602403s 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39782/0 (socket says 172.21.15.181:39782) 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 -- 172.21.15.181:0/551537869 learned_addr learned my addr 172.21.15.181:0/551537869 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): get_auth_request con 0x7fcb741321a0 auth_method 0 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb72ffd640 1 -- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb7409ec50 msgr2=0x7fcb7409aad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb72ffd640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb7409ec50 0x7fcb7409aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb727fc640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb727fc640 10 monclient(hunting): get_auth_request con 0x7fcb7409f020 auth_method 0 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient(hunting): handle_auth_done global_id 45088 payload 824 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 -- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 msgr2=0x7fcb74130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 -- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb7409ec50 msgr2=0x7fcb7409aad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb7409ec50 0x7fcb7409aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb737fe640 1 -- 172.21.15.181:0/551537869 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c009390 con 0x7fcb741321a0 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.121+0000 7fcb727fc640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.123+0000 7fcb737fe640 1 -- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 msgr2=0x7fcb74131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.123+0000 7fcb737fe640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009360 tx=0x7fcb6c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb737fe640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009360 tx=0x7fcb6c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fcb6c032e80 mon_map magic: 0 v1 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c032e80 con 0x7fcb741321a0 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 10 monclient: dump: 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 10 monclient: _finish_auth 0 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c031c70 con 0x7fcb741321a0 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 5 monclient: authenticate success, global_id 45088 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 10 monclient: _renew_subs 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7411e2a0 con 0x7fcb741321a0 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb5affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 10 monclient: _renew_subs 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7411bc30 con 0x7fcb741321a0 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.124+0000 7fcb717fa640 4 set_mon_vals no callback set 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c03c410 con 0x7fcb741321a0 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb5affd640 10 monclient: dump: 2024-09-15T03:33:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:24.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb6c039070 con 0x7fcb741321a0 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb792a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb792a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb792a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7409df90 con 0x7fcb741321a0 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb5affd640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb60038740 0x7fcb6003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb72ffd640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb60038740 0x7fcb6003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.125+0000 7fcb72ffd640 10 monclient: get_auth_request con 0x7fcb60038740 auth_method 0 2024-09-15T03:33:24.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.127+0000 7fcb72ffd640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb60038740 0x7fcb6003abf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcb64003c70 tx=0x7fcb64009310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:24.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.129+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(112..112 src has 40..112) v4 ==== 6567+0+0 (secure 0 0 0) 0x7fcb6c06c7b0 con 0x7fcb741321a0 2024-09-15T03:33:24.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.747+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcb7409df90 con 0x7fcb741321a0 2024-09-15T03:33:24.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.747+0000 7fcb5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:24.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.747+0000 7fcb5affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:24.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.908+0000 7fcb792a8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T03:33:24.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.908+0000 7fcb792a8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T03:33:24.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.908+0000 7fcb792a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:24.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:24.908+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7fcb7409df90 con 0x7fcb741321a0 2024-09-15T03:33:25.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.088+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v113) v1 ==== 125+0+0 (secure 0 0 0) 0x7fcb7409df90 con 0x7fcb741321a0 2024-09-15T03:33:25.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.088+0000 7fcb5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T03:33:25.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.088+0000 7fcb5affd640 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.185+0000 7fcb792a8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.185+0000 7fcb792a8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.185+0000 7fcb792a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.185+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7fcb7409df90 con 0x7fcb741321a0 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.186+0000 7fcb5affd640 1 -- 172.21.15.181:0/551537869 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v113) v1 ==== 126+0+0 (secure 0 0 0) 0x7fcb7409df90 con 0x7fcb741321a0 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.186+0000 7fcb5affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.186+0000 7fcb5affd640 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-15T03:33:25.187 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.12' does not appear in the crush map 2024-09-15T03:33:25.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb60038740 msgr2=0x7fcb6003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:25.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb60038740 0x7fcb6003abf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcb64003c70 tx=0x7fcb64009310 comp rx=0 tx=0).stop 2024-09-15T03:33:25.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 10 monclient: shutdown 2024-09-15T03:33:25.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:25.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 msgr2=0x7fcb74131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009360 tx=0x7fcb6c032000 comp rx=0 tx=0).stop 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 shutdown_connections 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb60038740 0x7fcb6003abf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb741321a0 0x7fcb74131100 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb7409f020 0x7fcb74130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 --2- 172.21.15.181:0/551537869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb7409ec50 0x7fcb7409aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 >> 172.21.15.181:0/551537869 conn(0x7fcb740a5cf0 msgr2=0x7fcb740a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 shutdown_connections 2024-09-15T03:33:25.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.188+0000 7fcb792a8640 1 -- 172.21.15.181:0/551537869 wait complete. 2024-09-15T03:33:25.198 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm osd.14 2024-09-15T03:33:25.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.270+0000 7fae5e19d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:25.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.270+0000 7fae5e19d640 10 monclient: build_initial_monmap 2024-09-15T03:33:25.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.270+0000 7fae5e19d640 10 monclient: monmap: 2024-09-15T03:33:25.271 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:25.271328+0000 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:25.271328+0000 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:25.272 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:25.273 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:25.273 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:25.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 Processor -- start 2024-09-15T03:33:25.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 -- start start 2024-09-15T03:33:25.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient: init 2024-09-15T03:33:25.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:25.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:33:25.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fae5809ec50 0x7fae58131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): picked mon.noname-a con 0x7fae5809ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae581321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 0x7fae58069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): picked mon.noname-b con 0x7fae5809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae58067420 con 0x7fae581321a0 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae580a0480 con 0x7fae5809ec50 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580a0200 con 0x7fae5809f020 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae5e19d640 10 monclient(hunting): authenticate will time out at 1009.752738s 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae56ffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fae581321a0 0x7fae58068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40726/0 (socket says 172.21.15.181:40726) 2024-09-15T03:33:25.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae56ffd640 1 -- 172.21.15.181:0/3861925246 learned_addr learned my addr 172.21.15.181:0/3861925246 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae577fe640 1 -- 172.21.15.181:0/3861925246 >> v2:172.21.15.153:3300/0 conn(0x7fae5809ec50 msgr2=0x7fae58131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.271+0000 7fae577fe640 1 --2- 172.21.15.181:0/3861925246 >> v2:172.21.15.153:3300/0 conn(0x7fae5809ec50 0x7fae58131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 1 --2- 172.21.15.181:0/3861925246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 0x7fae58069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): get_auth_request con 0x7fae5809f020 auth_method 0 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient(hunting): handle_auth_done global_id 74210 payload 274 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 1 -- 172.21.15.181:0/3861925246 >> v2:172.21.15.153:3300/0 conn(0x7fae5809ec50 msgr2=0x7fae58131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 1 --2- 172.21.15.181:0/3861925246 >> v2:172.21.15.153:3300/0 conn(0x7fae5809ec50 0x7fae58131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 1 -- 172.21.15.181:0/3861925246 >> v1:172.21.15.181:6789/0 conn(0x7fae581321a0 legacy=0x7fae58068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 1 monclient: found mon.noname-b 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.272+0000 7fae57fff640 1 -- 172.21.15.181:0/3861925246 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5809fd80 con 0x7fae5809f020 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae57fff640 1 -- 172.21.15.181:0/3861925246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 msgr2=0x7fae58069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae57fff640 1 --2- 172.21.15.181:0/3861925246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 0x7fae58069410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fae50009e60 tx=0x7fae50009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b9b5c145b45b99 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 1 -- 172.21.15.181:0/3861925246 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50007840 con 0x7fae5809f020 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: dump: 2024-09-15T03:33:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 1 -- 172.21.15.181:0/3861925246 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5002d050 con 0x7fae5809f020 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 5 monclient: authenticate success, global_id 74210 2024-09-15T03:33:25.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 1 -- 172.21.15.181:0/3861925246 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50007bd0 con 0x7fae5809f020 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 4 set_mon_vals no callback set 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae567fc640 10 monclient: dump: 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 10 monclient: shutdown 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 1 -- 172.21.15.181:0/3861925246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 msgr2=0x7fae58069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.275+0000 7fae5e19d640 1 --2- 172.21.15.181:0/3861925246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 0x7fae58069410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fae50009e60 tx=0x7fae50009780 comp rx=0 tx=0).stop 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 -- 172.21.15.181:0/3861925246 shutdown_connections 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 --2- 172.21.15.181:0/3861925246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809f020 0x7fae58069410 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 --2- 172.21.15.181:0/3861925246 >> v2:172.21.15.153:3300/0 conn(0x7fae5809ec50 0x7fae58131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 -- 172.21.15.181:0/3861925246 >> 172.21.15.181:0/3861925246 conn(0x7fae580a5cf0 msgr2=0x7fae580a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 -- 172.21.15.181:0/3861925246 shutdown_connections 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 -- 172.21.15.181:0/3861925246 wait complete. 2024-09-15T03:33:25.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 10 monclient: build_initial_monmap 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 10 monclient: monmap: 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 Processor -- start 2024-09-15T03:33:25.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 1 -- start start 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.276+0000 7fae5e19d640 10 monclient: init 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809ec50 0x7fae5809aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): picked mon.noname-c con 0x7fae5809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5809f020 0x7fae58130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): picked mon.noname-a con 0x7fae5809f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 0x7fae58131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): picked mon.noname-b con 0x7fae581321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5811c7c0 con 0x7fae5809f020 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5811dce0 con 0x7fae581321a0 2024-09-15T03:33:25.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5811dfc0 con 0x7fae5809ec50 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae5e19d640 10 monclient(hunting): authenticate will time out at 1009.758411s 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae56ffd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5809f020 msgr2=0x7fae58130bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 0x7fae58131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae56ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5809f020 0x7fae58130bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 0x7fae58131100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39794/0 (socket says 172.21.15.181:39794) 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 -- 172.21.15.181:0/1090332640 learned_addr learned my addr 172.21.15.181:0/1090332640 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): get_auth_request con 0x7fae581321a0 auth_method 0 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809ec50 0x7fae5809aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): get_auth_request con 0x7fae5809ec50 auth_method 0 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient(hunting): handle_auth_done global_id 45091 payload 824 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 -- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809ec50 msgr2=0x7fae5809aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809ec50 0x7fae5809aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 -- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5809f020 msgr2=0x7fae58130bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5809f020 0x7fae58130bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 monclient: found mon.noname-b 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.277+0000 7fae57fff640 1 -- 172.21.15.181:0/1090332640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50009a20 con 0x7fae581321a0 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae57fff640 1 -- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 msgr2=0x7fae58131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae57fff640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 0x7fae58131100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae50009e60 tx=0x7fae50037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50031cd0 con 0x7fae581321a0 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: dump: 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 5 monclient: authenticate success, global_id 45091 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 10 monclient: _renew_subs 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5811e2a0 con 0x7fae581321a0 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50040070 con 0x7fae581321a0 2024-09-15T03:33:25.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50045640 con 0x7fae581321a0 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae3f7fe640 10 monclient: dump: 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:25.295 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 10 monclient: _renew_subs 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5811bc30 con 0x7fae581321a0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.280+0000 7fae55ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fae50003020 con 0x7fae581321a0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae5e19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae5e19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae5e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5809df90 con 0x7fae581321a0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae3f7fe640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae440386f0 0x7fae4403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(113..113 src has 40..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fae5006dea0 con 0x7fae581321a0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae577fe640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae440386f0 0x7fae4403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:25.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.281+0000 7fae577fe640 10 monclient: get_auth_request con 0x7fae440386f0 auth_method 0 2024-09-15T03:33:25.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.283+0000 7fae577fe640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae440386f0 0x7fae4403aba0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae480038f0 tx=0x7fae480090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:25.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.283+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:33:25.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.285+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fae5809df90 con 0x7fae581321a0 2024-09-15T03:33:25.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.286+0000 7fae3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:25.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.286+0000 7fae3f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.444+0000 7fae5e19d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T03:33:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.444+0000 7fae5e19d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T03:33:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.444+0000 7fae5e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:25.444+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7fae5809df90 con 0x7fae581321a0 2024-09-15T03:33:26.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.092+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v114) v1 ==== 125+0+0 (secure 0 0 0) 0x7fae5809df90 con 0x7fae581321a0 2024-09-15T03:33:26.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.092+0000 7fae3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T03:33:26.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.092+0000 7fae3f7fe640 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.192+0000 7fae5e19d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.192+0000 7fae5e19d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.192+0000 7fae5e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.192+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7fae5809df90 con 0x7fae581321a0 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.193+0000 7fae3f7fe640 1 -- 172.21.15.181:0/1090332640 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v114) v1 ==== 126+0+0 (secure 0 0 0) 0x7fae5809df90 con 0x7fae581321a0 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.193+0000 7fae3f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-15T03:33:26.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.193+0000 7fae3f7fe640 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-15T03:33:26.195 INFO:tasks.workunit.client.0.smithi181.stderr:device 'osd.14' does not appear in the crush map 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae440386f0 msgr2=0x7fae4403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae440386f0 0x7fae4403aba0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae480038f0 tx=0x7fae480090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 10 monclient: shutdown 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 msgr2=0x7fae58131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 0x7fae58131100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae50009e60 tx=0x7fae50037000 comp rx=0 tx=0).stop 2024-09-15T03:33:26.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 shutdown_connections 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae440386f0 0x7fae4403aba0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae581321a0 0x7fae58131100 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5809f020 0x7fae58130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 --2- 172.21.15.181:0/1090332640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae5809ec50 0x7fae5809aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 >> 172.21.15.181:0/1090332640 conn(0x7fae580a5cf0 msgr2=0x7fae580a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 shutdown_connections 2024-09-15T03:33:26.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.195+0000 7fae5e19d640 1 -- 172.21.15.181:0/1090332640 wait complete. 2024-09-15T03:33:26.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm osd.12 2024-09-15T03:33:26.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.276+0000 7f0fee280640 10 monclient: get_monmap_and_config 2024-09-15T03:33:26.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.276+0000 7f0fee280640 10 monclient: build_initial_monmap 2024-09-15T03:33:26.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.276+0000 7f0fee280640 10 monclient: monmap: 2024-09-15T03:33:26.278 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:26.278 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:26.277634+0000 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:26.277634+0000 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:26.279 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:26.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.277+0000 7f0fee280640 1 Processor -- start 2024-09-15T03:33:26.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.277+0000 7f0fee280640 1 -- start start 2024-09-15T03:33:26.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.277+0000 7f0fee280640 10 monclient: init 2024-09-15T03:33:26.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:26.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:33:26.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0fe8099a60 0x7f0fe809a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:26.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe8099a60 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:26.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe809ad00 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:26.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0fe809e010 0x7f0fe809e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:26.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): picked mon.noname-c con 0x7f0fe809e010 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe8069bc0 con 0x7f0fe809ad00 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe806a180 con 0x7f0fe8099a60 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe8099e30 con 0x7f0fe809e010 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fee280640 10 monclient(hunting): authenticate will time out at 1010.759142s 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0fe809e010 0x7f0fe809e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0fe809e010 0x7f0fe809e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39804/0 (socket says 172.21.15.181:39804) 2024-09-15T03:33:26.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 -- 172.21.15.181:0/2098223765 learned_addr learned my addr 172.21.15.181:0/2098223765 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): get_auth_request con 0x7f0fe809e010 auth_method 0 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fe7fff640 1 -- 172.21.15.181:0/2098223765 >> v2:172.21.15.153:3300/0 conn(0x7f0fe8099a60 msgr2=0x7f0fe809a7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fe77fe640 1 -- 172.21.15.181:0/2098223765 >> v1:172.21.15.153:6789/0 conn(0x7f0fe809ad00 legacy=0x7f0fe809d900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fe7fff640 1 --2- 172.21.15.181:0/2098223765 >> v2:172.21.15.153:3300/0 conn(0x7f0fe8099a60 0x7f0fe809a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fe6ffd640 1 -- 172.21.15.181:0/2098223765 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:26.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient(hunting): handle_auth_done global_id 45094 payload 274 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 -- 172.21.15.181:0/2098223765 >> v2:172.21.15.153:3300/0 conn(0x7f0fe8099a60 msgr2=0x7f0fe809a7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 --2- 172.21.15.181:0/2098223765 >> v2:172.21.15.153:3300/0 conn(0x7f0fe8099a60 0x7f0fe809a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 -- 172.21.15.181:0/2098223765 >> v1:172.21.15.153:6789/0 conn(0x7f0fe809ad00 legacy=0x7f0fe809d900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient: _finish_hunting 0 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 monclient: found mon.noname-c 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.278+0000 7f0fec81d640 1 -- 172.21.15.181:0/2098223765 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe806a880 con 0x7f0fe809e010 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fec81d640 1 -- 172.21.15.181:0/2098223765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809e010 msgr2=0x7f0fe809e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fec81d640 1 --2- 172.21.15.181:0/2098223765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809e010 0x7f0fe809e410 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8064940 tx=0x7f0fe0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a02be080c09b4b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:26.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 1 -- 172.21.15.181:0/2098223765 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe002aea0 con 0x7f0fe809e010 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: dump: 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:26.296 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 1 -- 172.21.15.181:0/2098223765 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe002d050 con 0x7f0fe809e010 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 5 monclient: authenticate success, global_id 45094 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 4 set_mon_vals no callback set 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 1 -- 172.21.15.181:0/2098223765 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0007820 con 0x7f0fe809e010 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:26.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fe6ffd640 10 monclient: dump: 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:26.298 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 10 monclient: shutdown 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 -- 172.21.15.181:0/2098223765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809e010 msgr2=0x7f0fe809e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 --2- 172.21.15.181:0/2098223765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809e010 0x7f0fe809e410 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8064940 tx=0x7f0fe0009840 comp rx=0 tx=0).stop 2024-09-15T03:33:26.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 -- 172.21.15.181:0/2098223765 shutdown_connections 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 --2- 172.21.15.181:0/2098223765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809e010 0x7f0fe809e410 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 --2- 172.21.15.181:0/2098223765 >> v2:172.21.15.153:3300/0 conn(0x7f0fe8099a60 0x7f0fe809a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 -- 172.21.15.181:0/2098223765 >> 172.21.15.181:0/2098223765 conn(0x7f0fe80a2f10 msgr2=0x7f0fe80a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 -- 172.21.15.181:0/2098223765 shutdown_connections 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.281+0000 7f0fee280640 1 -- 172.21.15.181:0/2098223765 wait complete. 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.282+0000 7f0fee280640 10 monclient: build_initial_monmap 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.282+0000 7f0fee280640 10 monclient: monmap: 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:26.300 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.282+0000 7f0fee280640 1 Processor -- start 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.282+0000 7f0fee280640 1 -- start start 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.282+0000 7f0fee280640 10 monclient: init 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe8099a60 0x7f0fe8141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): picked mon.noname-c con 0x7f0fe8099a60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:26.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe809ad00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe809e010 0x7f0fe811e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe809e010 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe806eb10 con 0x7f0fe809e010 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe809c420 con 0x7f0fe809ad00 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81224e0 con 0x7f0fe8099a60 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fee280640 10 monclient(hunting): authenticate will time out at 1010.764328s 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39812/0 (socket says 172.21.15.181:39812) 2024-09-15T03:33:26.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 -- 172.21.15.181:0/3127742831 learned_addr learned my addr 172.21.15.181:0/3127742831 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): get_auth_request con 0x7f0fe809ad00 auth_method 0 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fec81d640 1 -- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe809e010 msgr2=0x7f0fe811e900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fec81d640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe809e010 0x7f0fe811e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe7fff640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe8099a60 0x7f0fe8141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient(hunting): handle_auth_done global_id 45097 payload 824 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 -- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe8099a60 msgr2=0x7f0fe8141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe8099a60 0x7f0fe8141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 -- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe809e010 msgr2=0x7f0fe811e900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe809e010 0x7f0fe811e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe77fe640 1 -- 172.21.15.181:0/3127742831 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0009ae0 con 0x7f0fe809ad00 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe7fff640 10 monclient: get_auth_request con 0x7f0fe8099a60 auth_method 0 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.283+0000 7f0fe7fff640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe8099a60 0x7f0fe8141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.285+0000 7f0fe77fe640 1 -- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 msgr2=0x7f0fe8134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.285+0000 7f0fe77fe640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b600 tx=0x7f0fdc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.285+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc015070 con 0x7f0fe809ad00 2024-09-15T03:33:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fe77fe640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b600 tx=0x7f0fdc0091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f0fdc006380 mon_map magic: 0 v1 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 10 monclient: dump: 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 10 monclient: _finish_auth 0 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 5 monclient: authenticate success, global_id 45097 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc003d90 con 0x7f0fe809ad00 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fcffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 10 monclient: _renew_subs 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8141560 con 0x7f0fe809ad00 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fe67fc640 4 set_mon_vals no callback set 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 10 monclient: _renew_subs 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.286+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe8141e60 con 0x7f0fe809ad00 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc006380 con 0x7f0fe809ad00 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fcffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fcffff640 10 monclient: dump: 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:26.307 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0fdc010440 con 0x7f0fe809ad00 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fee280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fee280640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fee280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe804bf60 con 0x7f0fe809ad00 2024-09-15T03:33:26.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fcffff640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd40386f0 0x7f0fd403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:26.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fe7fff640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd40386f0 0x7f0fd403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:26.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.287+0000 7f0fe7fff640 10 monclient: get_auth_request con 0x7f0fd40386f0 auth_method 0 2024-09-15T03:33:26.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.289+0000 7f0fe7fff640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd40386f0 0x7f0fd403aba0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80038f0 tx=0x7f0fd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:26.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.291+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(114..114 src has 40..114) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f0fdc04bdb0 con 0x7f0fe809ad00 2024-09-15T03:33:26.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.909+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0fdc018d60 con 0x7f0fe809ad00 2024-09-15T03:33:27.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.909+0000 7f0fcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:26.909+0000 7f0fcffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.073+0000 7f0fee280640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T03:33:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.073+0000 7f0fee280640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T03:33:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.073+0000 7f0fee280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.073+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f0fe80677e0 con 0x7f0fe809ad00 2024-09-15T03:33:27.275 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:33:27.073+0000 7fb5705f1640 -1 mon.b@1(leader).osd e114 definitely_dead 0 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.094+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v115) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0fdc018780 con 0x7f0fe809ad00 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.094+0000 7f0fcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.094+0000 7f0fcffff640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fee280640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fee280640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fee280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f0fe81203f0 con 0x7f0fe809ad00 2024-09-15T03:33:27.276 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:33:27.201+0000 7fb5705f1640 -1 mon.b@1(leader).osd e115 definitely_dead 0 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fcffff640 1 -- 172.21.15.181:0/3127742831 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v115) v1 ==== 96+0+0 (secure 0 0 0) 0x7f0fdc018780 con 0x7f0fe809ad00 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fcffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.201+0000 7f0fcffff640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:osd.12 does not exist. 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd40386f0 msgr2=0x7f0fd403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd40386f0 0x7f0fd403aba0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80038f0 tx=0x7f0fd8008040 comp rx=0 tx=0).stop 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 10 monclient: shutdown 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 msgr2=0x7f0fe8134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b600 tx=0x7f0fdc0091a0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 shutdown_connections 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fd40386f0 0x7f0fd403aba0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe809e010 0x7f0fe811e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fe809ad00 0x7f0fe8134e00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 --2- 172.21.15.181:0/3127742831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe8099a60 0x7f0fe8141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 >> 172.21.15.181:0/3127742831 conn(0x7f0fe80a2f10 msgr2=0x7f0fe80a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 shutdown_connections 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.203+0000 7f0fee280640 1 -- 172.21.15.181:0/3127742831 wait complete. 2024-09-15T03:33:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm osd.14 2024-09-15T03:33:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.285+0000 7f2db26d4640 10 monclient: get_monmap_and_config 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.285+0000 7f2db26d4640 10 monclient: build_initial_monmap 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.285+0000 7f2db26d4640 10 monclient: monmap: 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:27.285851+0000 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:27.285851+0000 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:27.287 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.285+0000 7f2db26d4640 1 Processor -- start 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.285+0000 7f2db26d4640 1 -- start start 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.285+0000 7f2db26d4640 10 monclient: init 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2dac1319c0 0x7f2dac131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): picked mon.noname-a con 0x7f2dac1319c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2dac09eb30 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 0x7f2dac1390d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): picked mon.noname-b con 0x7f2dac136cf0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac131d90 con 0x7f2dac09eb30 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac06a260 con 0x7f2dac1319c0 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0a0050 con 0x7f2dac136cf0 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db26d4640 10 monclient(hunting): authenticate will time out at 1011.767314s 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2dabfff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f2dac1319c0 msgr2=0x7f2dac131fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2dabfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2dac1319c0 0x7f2dac131fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2dab7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f2dac09eb30 0x7f2dac09ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40728/0 (socket says 172.21.15.181:40728) 2024-09-15T03:33:27.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2dab7fe640 1 -- 172.21.15.181:0/1201914193 learned_addr learned my addr 172.21.15.181:0/1201914193 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 1 --2- 172.21.15.181:0/1201914193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 0x7f2dac1390d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): get_auth_request con 0x7f2dac136cf0 auth_method 0 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient(hunting): handle_auth_done global_id 74216 payload 274 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 1 -- 172.21.15.181:0/1201914193 >> v2:172.21.15.153:3300/0 conn(0x7f2dac1319c0 msgr2=0x7f2dac131fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 1 --2- 172.21.15.181:0/1201914193 >> v2:172.21.15.153:3300/0 conn(0x7f2dac1319c0 0x7f2dac131fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 1 -- 172.21.15.181:0/1201914193 >> v1:172.21.15.181:6789/0 conn(0x7f2dac09eb30 legacy=0x7f2dac09ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient: _finish_hunting 0 2024-09-15T03:33:27.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 1 monclient: found mon.noname-b 2024-09-15T03:33:27.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:27.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:27.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.286+0000 7f2db0c71640 1 -- 172.21.15.181:0/1201914193 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac09fe10 con 0x7f2dac136cf0 2024-09-15T03:33:27.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.289+0000 7f2db0c71640 1 -- 172.21.15.181:0/1201914193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 msgr2=0x7f2dac1390d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:27.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.289+0000 7f2db0c71640 1 --2- 172.21.15.181:0/1201914193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 0x7f2dac1390d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009e90 tx=0x7f2da4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6900d59dca8c43 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 1 -- 172.21.15.181:0/1201914193 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4007840 con 0x7f2dac136cf0 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: dump: 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: _finish_auth 0 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 5 monclient: authenticate success, global_id 74216 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 1 -- 172.21.15.181:0/1201914193 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da402d050 con 0x7f2dac136cf0 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 4 set_mon_vals no callback set 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 1 -- 172.21.15.181:0/1201914193 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4007bd0 con 0x7f2dac136cf0 2024-09-15T03:33:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2daaffd640 10 monclient: dump: 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 10 monclient: shutdown 2024-09-15T03:33:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 -- 172.21.15.181:0/1201914193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 msgr2=0x7f2dac1390d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1201914193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 0x7f2dac1390d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009e90 tx=0x7f2da4009780 comp rx=0 tx=0).stop 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 -- 172.21.15.181:0/1201914193 shutdown_connections 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1201914193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac136cf0 0x7f2dac1390d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1201914193 >> v2:172.21.15.153:3300/0 conn(0x7f2dac1319c0 0x7f2dac131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 -- 172.21.15.181:0/1201914193 >> 172.21.15.181:0/1201914193 conn(0x7f2dac071370 msgr2=0x7f2dac0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 -- 172.21.15.181:0/1201914193 shutdown_connections 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.290+0000 7f2db26d4640 1 -- 172.21.15.181:0/1201914193 wait complete. 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.291+0000 7f2db26d4640 10 monclient: build_initial_monmap 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.291+0000 7f2db26d4640 10 monclient: monmap: 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.291+0000 7f2db26d4640 1 Processor -- start 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.291+0000 7f2db26d4640 1 -- start start 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.291+0000 7f2db26d4640 10 monclient: init 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eb30 0x7f2dac1308f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:27.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): picked mon.noname-c con 0x7f2dac09eb30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 0x7f2dac099790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): picked mon.noname-b con 0x7f2dac1319c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac136cf0 0x7f2dac099cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): picked mon.noname-a con 0x7f2dac136cf0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac09bee0 con 0x7f2dac136cf0 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac09e180 con 0x7f2dac1319c0 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac11c7a0 con 0x7f2dac09eb30 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db26d4640 10 monclient(hunting): authenticate will time out at 1011.773278s 2024-09-15T03:33:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 0x7f2dac099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 0x7f2dac099790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:39826/0 (socket says 172.21.15.181:39826) 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 -- 172.21.15.181:0/1114364521 learned_addr learned my addr 172.21.15.181:0/1114364521 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): get_auth_request con 0x7f2dac1319c0 auth_method 0 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db0c71640 1 -- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac136cf0 msgr2=0x7f2dac099cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2db0c71640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac136cf0 0x7f2dac099cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient(hunting): handle_auth_done global_id 45100 payload 824 2024-09-15T03:33:27.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dabfff640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eb30 0x7f2dac1308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 -- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eb30 msgr2=0x7f2dac1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eb30 0x7f2dac1308f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 -- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac136cf0 msgr2=0x7f2dac099cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac136cf0 0x7f2dac099cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.292+0000 7f2dab7fe640 1 -- 172.21.15.181:0/1114364521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da4009a20 con 0x7f2dac1319c0 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2dab7fe640 1 -- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 msgr2=0x7f2dac099790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2dab7fe640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 0x7f2dac099790 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2da000c610 tx=0x7f2da0004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0005cb0 con 0x7f2dac1319c0 2024-09-15T03:33:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:27.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:27.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 10 monclient: dump: 2024-09-15T03:33:27.307 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:27.307 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 10 monclient: _finish_auth 0 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2db26d4640 5 monclient: authenticate success, global_id 45100 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.294+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da0005e30 con 0x7f2dac1319c0 2024-09-15T03:33:27.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 monclient: _renew_subs 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac11baa0 con 0x7f2dac1319c0 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2d93fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0007340 con 0x7f2dac1319c0 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2d93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2d93fff640 10 monclient: dump: 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:27.309 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 monclient: _renew_subs 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2daa7fc640 4 set_mon_vals no callback set 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac09b260 con 0x7f2dac1319c0 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2da001b400 con 0x7f2dac1319c0 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.295+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac1375f0 con 0x7f2dac1319c0 2024-09-15T03:33:27.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.296+0000 7f2d93fff640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d980386f0 0x7f2d9803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:27.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.296+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(115..115 src has 40..115) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f2da004bf00 con 0x7f2dac1319c0 2024-09-15T03:33:27.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.296+0000 7f2dabfff640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d980386f0 0x7f2d9803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:27.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.296+0000 7f2dabfff640 10 monclient: get_auth_request con 0x7f2d980386f0 auth_method 0 2024-09-15T03:33:27.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.298+0000 7f2dabfff640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d980386f0 0x7f2d9803aba0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c006fd0 tx=0x7f2d9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:27.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.300+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2da001a930 con 0x7f2dac1319c0 2024-09-15T03:33:27.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.300+0000 7f2d93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:27.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.300+0000 7f2d93fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.463+0000 7f2db26d4640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T03:33:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.463+0000 7f2db26d4640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T03:33:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.463+0000 7f2db26d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:27.463+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7f2dac04bf60 con 0x7f2dac1319c0 2024-09-15T03:33:27.466 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:33:27.464+0000 7fb5705f1640 -1 mon.b@1(leader).osd e115 definitely_dead 0 2024-09-15T03:33:28.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.102+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v116) v1 ==== 87+0+0 (secure 0 0 0) 0x7f2da001a360 con 0x7f2dac1319c0 2024-09-15T03:33:28.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.102+0000 7f2d93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T03:33:28.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.102+0000 7f2d93fff640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-15T03:33:28.205 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.204+0000 7f2db26d4640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T03:33:28.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.204+0000 7f2db26d4640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T03:33:28.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.204+0000 7f2db26d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.204+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7f2dac11ce00 con 0x7f2dac1319c0 2024-09-15T03:33:28.206 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:33:28.205+0000 7fb5705f1640 -1 mon.b@1(leader).osd e116 definitely_dead 0 2024-09-15T03:33:28.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.205+0000 7f2d93fff640 1 -- 172.21.15.181:0/1114364521 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v116) v1 ==== 96+0+0 (secure 0 0 0) 0x7f2da001a540 con 0x7f2dac1319c0 2024-09-15T03:33:28.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.205+0000 7f2d93fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-15T03:33:28.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.205+0000 7f2d93fff640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-15T03:33:28.207 INFO:tasks.workunit.client.0.smithi181.stderr:osd.14 does not exist. 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d980386f0 msgr2=0x7f2d9803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d980386f0 0x7f2d9803aba0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c006fd0 tx=0x7f2d9c008040 comp rx=0 tx=0).stop 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 10 monclient: shutdown 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 msgr2=0x7f2dac099790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 0x7f2dac099790 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2da000c610 tx=0x7f2da0004960 comp rx=0 tx=0).stop 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 shutdown_connections 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d980386f0 0x7f2d9803aba0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac136cf0 0x7f2dac099cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac1319c0 0x7f2dac099790 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.206+0000 7f2db26d4640 1 --2- 172.21.15.181:0/1114364521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eb30 0x7f2dac1308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.207+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 >> 172.21.15.181:0/1114364521 conn(0x7f2dac071370 msgr2=0x7f2dac09d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:28.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.207+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 shutdown_connections 2024-09-15T03:33:28.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.207+0000 7f2db26d4640 1 -- 172.21.15.181:0/1114364521 wait complete. 2024-09-15T03:33:28.217 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-15T03:33:28.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.288+0000 7f205655d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.288+0000 7f205655d640 10 monclient: build_initial_monmap 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.288+0000 7f205655d640 10 monclient: monmap: 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:28.289700+0000 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:28.289700+0000 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:28.290 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:28.291 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.289+0000 7f205655d640 1 Processor -- start 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.289+0000 7f205655d640 1 -- start start 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.289+0000 7f205655d640 10 monclient: init 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2050069a30 0x7f20501320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient(hunting): picked mon.noname-a con 0x7f2050069a30 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2050070920 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2050071430 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2050069e00 con 0x7f2050071430 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205009e560 con 0x7f2050070920 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205009cc60 con 0x7f2050069a30 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f205655d640 10 monclient(hunting): authenticate will time out at 1012.771224s 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f2054afa640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f2050071430 legacy=0x7f20501390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f204ffff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f2050069a30 msgr2=0x7f20501320d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f204ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2050069a30 0x7f20501320d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f204f7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f2050070920 0x7f2050070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56884/0 (socket says 172.21.15.181:56884) 2024-09-15T03:33:28.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.290+0000 7f204f7fe640 1 -- 172.21.15.181:0/2691571076 learned_addr learned my addr 172.21.15.181:0/2691571076 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204f7fe640 1 -- 172.21.15.181:0/2691571076 >> v1:172.21.15.181:6789/0 conn(0x7f2050070920 legacy=0x7f2050070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756017893 0 0) 0x7f205009e560 con 0x7f2050070920 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): my global_id is 45103 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20380034e0 con 0x7f2050070920 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2712686468 0 0) 0x7f20380034e0 con 0x7f2050070920 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 >> v2:172.21.15.153:3300/0 conn(0x7f2050069a30 msgr2=0x7f20501320d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 --2- 172.21.15.181:0/2691571076 >> v2:172.21.15.153:3300/0 conn(0x7f2050069a30 0x7f20501320d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 >> v1:172.21.15.153:6789/0 conn(0x7f2050071430 legacy=0x7f20501390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205009d0a0 con 0x7f2050070920 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: _finish_auth 0 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f2044001720 con 0x7f2050070920 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: dump: 2024-09-15T03:33:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 >> v1:172.21.15.181:6789/0 conn(0x7f2050070920 legacy=0x7f2050070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038003a50 0x7f2038003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): picked mon.c con 0x7f2038003a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 0x7f2038004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): picked mon.b con 0x7f2038004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2038005330 0x7f203800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): picked mon.a con 0x7f2038005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205009cc60 con 0x7f2038005330 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203800c2d0 con 0x7f2038004780 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203800c490 con 0x7f2038003a50 2024-09-15T03:33:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.292+0000 7f204effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f2054afa640 1 -- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2038005330 msgr2=0x7f203800b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f2054afa640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2038005330 0x7f203800b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 0x7f2038004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): get_auth_request con 0x7f2038004780 auth_method 0 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204ffff640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038003a50 0x7f2038003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204ffff640 10 monclient(hunting): get_auth_request con 0x7f2038003a50 auth_method 0 2024-09-15T03:33:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient(hunting): handle_auth_done global_id 45103 payload 293 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 -- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038003a50 msgr2=0x7f2038003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038003a50 0x7f2038003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 -- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2038005330 msgr2=0x7f203800b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2038005330 0x7f203800b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 monclient: found mon.b 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f2054afa640 1 -- 172.21.15.181:0/2691571076 reap_dead start 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 1 -- 172.21.15.181:0/2691571076 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20380034e0 con 0x7f2038004780 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f205655d640 5 monclient: authenticate success, global_id 45103 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f204ffff640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038003a50 0x7f2038003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.293+0000 7f205655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204f7fe640 1 -- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 msgr2=0x7f2038004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204f7fe640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 0x7f2038004bf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2044001490 tx=0x7f205006a690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a1ea34b1a7ddac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044005890 con 0x7f2038004780 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: dump: 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044005a10 con 0x7f2038004780 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 1 -- 172.21.15.181:0/2691571076 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044005dd0 con 0x7f2038004780 2024-09-15T03:33:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 4 set_mon_vals no callback set 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f204effd640 10 monclient: dump: 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:28.309 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 10 monclient: shutdown 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 -- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 msgr2=0x7f2038004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 0x7f2038004bf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2044001490 tx=0x7f205006a690 comp rx=0 tx=0).stop 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 -- 172.21.15.181:0/2691571076 shutdown_connections 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 --2- 172.21.15.181:0/2691571076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2038004780 0x7f2038004bf0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 -- 172.21.15.181:0/2691571076 >> 172.21.15.181:0/2691571076 conn(0x7f205009ea70 msgr2=0x7f205009ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 -- 172.21.15.181:0/2691571076 shutdown_connections 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.296+0000 7f205655d640 1 -- 172.21.15.181:0/2691571076 wait complete. 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.297+0000 7f205655d640 10 monclient: build_initial_monmap 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.297+0000 7f205655d640 10 monclient: monmap: 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:28.310 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.297+0000 7f205655d640 1 Processor -- start 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.297+0000 7f205655d640 1 -- start start 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.297+0000 7f205655d640 10 monclient: init 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038004780 0x7f2050140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): picked mon.noname-c con 0x7f2038004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:28.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205012edb0 0x7f205012f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): picked mon.noname-a con 0x7f205012edb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 0x7f2050118e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): picked mon.noname-b con 0x7f20501189a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205012f6c0 con 0x7f205012edb0 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205011a990 con 0x7f20501189a0 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205009d5d0 con 0x7f2038004780 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f205655d640 10 monclient(hunting): authenticate will time out at 1012.779249s 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204f7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205012edb0 msgr2=0x7f205012f180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:28.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205012edb0 0x7f205012f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 0x7f2050118e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 0x7f2050118e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49578/0 (socket says 172.21.15.181:49578) 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 -- 172.21.15.181:0/4567940 learned_addr learned my addr 172.21.15.181:0/4567940 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): get_auth_request con 0x7f20501189a0 auth_method 0 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038004780 0x7f2050140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): get_auth_request con 0x7f2038004780 auth_method 0 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:28.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f204ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient(hunting): handle_auth_done global_id 45106 payload 824 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 -- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038004780 msgr2=0x7f2050140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038004780 0x7f2050140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 -- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205012edb0 msgr2=0x7f205012f180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205012edb0 0x7f205012f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient: _finish_hunting 0 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 monclient: found mon.noname-b 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.298+0000 7f2054afa640 1 -- 172.21.15.181:0/4567940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500a0580 con 0x7f20501189a0 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.300+0000 7f2054afa640 1 -- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 msgr2=0x7f2050118e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.300+0000 7f2054afa640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 0x7f2050118e10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2050071800 tx=0x7f2048009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204800c9a0 con 0x7f20501189a0 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: dump: 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.317 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: _finish_auth 0 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204800cb20 con 0x7f20501189a0 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 5 monclient: authenticate success, global_id 45106 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 10 monclient: _renew_subs 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205011a430 con 0x7f20501189a0 2024-09-15T03:33:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204800ccd0 con 0x7f20501189a0 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f2037fff640 10 monclient: dump: 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:28.319 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 10 monclient: _renew_subs 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205011bed0 con 0x7f20501189a0 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.301+0000 7f204e7fc640 4 set_mon_vals no callback set 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2048012020 con 0x7f20501189a0 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f205655d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f205655d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f205655d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20500636e0 con 0x7f20501189a0 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f2037fff640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f203c0386f0 0x7f203c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(116..116 src has 40..116) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f204804bc50 con 0x7f20501189a0 2024-09-15T03:33:28.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f204ffff640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f203c0386f0 0x7f203c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:28.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.302+0000 7f204ffff640 10 monclient: get_auth_request con 0x7f203c0386f0 auth_method 0 2024-09-15T03:33:28.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.304+0000 7f204ffff640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f203c0386f0 0x7f203c03aba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2050066f10 tx=0x7f2040008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:28.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.306+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f204800ce30 con 0x7f20501189a0 2024-09-15T03:33:28.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.306+0000 7f2037fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:28.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.306+0000 7f2037fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.469+0000 7f205655d640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T03:33:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.469+0000 7f205655d640 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T03:33:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.469+0000 7f205655d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:28.469+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7f205009e280 con 0x7f20501189a0 2024-09-15T03:33:29.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.105+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v117) v1 ==== 145+0+0 (secure 0 0 0) 0x7f2048014e60 con 0x7f20501189a0 2024-09-15T03:33:29.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.105+0000 7f2037fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T03:33:29.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.105+0000 7f2037fff640 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-15T03:33:29.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.205+0000 7f205655d640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T03:33:29.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.205+0000 7f205655d640 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T03:33:29.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.205+0000 7f205655d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:29.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:29.205+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7f205011c290 con 0x7f20501189a0 2024-09-15T03:33:30.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.115+0000 7f2037fff640 1 -- 172.21.15.181:0/4567940 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v118) v1 ==== 145+0+0 (secure 0 0 0) 0x7f204801daf0 con 0x7f20501189a0 2024-09-15T03:33:30.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.115+0000 7f2037fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-15T03:33:30.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.115+0000 7f2037fff640 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-15T03:33:30.117 INFO:tasks.workunit.client.0.smithi181.stderr:set require_min_compat_client to luminous 2024-09-15T03:33:30.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f203c0386f0 msgr2=0x7f203c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:30.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f203c0386f0 0x7f203c03aba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2050066f10 tx=0x7f2040008040 comp rx=0 tx=0).stop 2024-09-15T03:33:30.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 10 monclient: shutdown 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 msgr2=0x7f2050118e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 0x7f2050118e10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2050071800 tx=0x7f2048009660 comp rx=0 tx=0).stop 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 shutdown_connections 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f203c0386f0 0x7f203c03aba0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20501189a0 0x7f2050118e10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205012edb0 0x7f205012f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 --2- 172.21.15.181:0/4567940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2038004780 0x7f2050140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 >> 172.21.15.181:0/4567940 conn(0x7f205009ea70 msgr2=0x7f2050097080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 shutdown_connections 2024-09-15T03:33:30.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.118+0000 7f205655d640 1 -- 172.21.15.181:0/4567940 wait complete. 2024-09-15T03:33:30.129 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set dump 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.204+0000 7fd6a8b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.204+0000 7fd6a8b9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.204+0000 7fd6a8b9d640 10 monclient: monmap: 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:30.205398+0000 2024-09-15T03:33:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:30.205398+0000 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 Processor -- start 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 -- start start 2024-09-15T03:33:30.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient: init 2024-09-15T03:33:30.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:30.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6a4069a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd6a4099c80 0x7fd6a4071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a4099c80 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd6a4135710 0x7fd6a4137af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a4135710 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a409e7e0 con 0x7fd6a4069a30 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a4069e20 con 0x7fd6a4135710 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a409cef0 con 0x7fd6a4099c80 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a8b9d640 10 monclient(hunting): authenticate will time out at 1014.686777s 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd6a4099c80 0x7fd6a4071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd6a4099c80 0x7fd6a4071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49592/0 (socket says 172.21.15.181:49592) 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a259c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd6a4069a30 0x7fd6a4070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56894/0 (socket says 172.21.15.181:56894) 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 1 -- 172.21.15.181:0/3711065145 learned_addr learned my addr 172.21.15.181:0/3711065145 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): get_auth_request con 0x7fd6a4099c80 auth_method 0 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a2d9d640 1 -- 172.21.15.181:0/3711065145 >> v2:172.21.15.153:3300/0 conn(0x7fd6a4135710 msgr2=0x7fd6a4137af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd6a2d9d640 1 --2- 172.21.15.181:0/3711065145 >> v2:172.21.15.153:3300/0 conn(0x7fd6a4135710 0x7fd6a4137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.205+0000 7fd699d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 10 monclient(hunting): handle_auth_done global_id 45109 payload 274 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 1 -- 172.21.15.181:0/3711065145 >> v2:172.21.15.153:3300/0 conn(0x7fd6a4135710 msgr2=0x7fd6a4137af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 1 --2- 172.21.15.181:0/3711065145 >> v2:172.21.15.153:3300/0 conn(0x7fd6a4135710 0x7fd6a4137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 1 -- 172.21.15.181:0/3711065145 >> v1:172.21.15.181:6789/0 conn(0x7fd6a4069a30 legacy=0x7fd6a4070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 1 monclient: found mon.noname-c 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.206+0000 7fd699d9b640 1 -- 172.21.15.181:0/3711065145 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a409d390 con 0x7fd6a4099c80 2024-09-15T03:33:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.208+0000 7fd699d9b640 1 -- 172.21.15.181:0/3711065145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4099c80 msgr2=0x7fd6a4071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.208+0000 7fd699d9b640 1 --2- 172.21.15.181:0/3711065145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4099c80 0x7fd6a4071030 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd6a406a7a0 tx=0x7fd68c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27602fc99a76a72 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 1 -- 172.21.15.181:0/3711065145 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c032440 con 0x7fd6a4099c80 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: dump: 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:30.639 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 1 -- 172.21.15.181:0/3711065145 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c0325a0 con 0x7fd6a4099c80 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 5 monclient: authenticate success, global_id 45109 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:30.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 1 -- 172.21.15.181:0/3711065145 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c032940 con 0x7fd6a4099c80 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 4 set_mon_vals no callback set 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a1d9b640 10 monclient: dump: 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:30.641 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:30.642 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 10 monclient: shutdown 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/3711065145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4099c80 msgr2=0x7fd6a4071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/3711065145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4099c80 0x7fd6a4071030 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd6a406a7a0 tx=0x7fd68c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/3711065145 shutdown_connections 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/3711065145 >> v2:172.21.15.153:3300/0 conn(0x7fd6a4135710 0x7fd6a4137af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/3711065145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4099c80 0x7fd6a4071030 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/3711065145 >> 172.21.15.181:0/3711065145 conn(0x7fd6a409e9f0 msgr2=0x7fd6a409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/3711065145 shutdown_connections 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.209+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/3711065145 wait complete. 2024-09-15T03:33:30.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.210+0000 7fd6a8b9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:30.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.210+0000 7fd6a8b9d640 10 monclient: monmap: 2024-09-15T03:33:30.644 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:30.644 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:30.644 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:30.644 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:30.645 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:30.645 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:30.645 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:30.645 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:30.645 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:30.645 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:30.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.210+0000 7fd6a8b9d640 1 Processor -- start 2024-09-15T03:33:30.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.210+0000 7fd6a8b9d640 1 -- start start 2024-09-15T03:33:30.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.210+0000 7fd6a8b9d640 10 monclient: init 2024-09-15T03:33:30.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:30.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:30.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a4069a30 0x7fd6a413f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a4069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a4099c80 0x7fd6a4130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a4099c80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a4135710 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a411c1a0 con 0x7fd6a4099c80 2024-09-15T03:33:30.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:30.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a409d8c0 con 0x7fd6a4135710 2024-09-15T03:33:30.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:30.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a41328c0 con 0x7fd6a4069a30 2024-09-15T03:33:30.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:30.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a8b9d640 10 monclient(hunting): authenticate will time out at 1014.692188s 2024-09-15T03:33:30.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49604/0 (socket says 172.21.15.181:49604) 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd699d9b640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a4099c80 msgr2=0x7fd6a4130dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 -- 172.21.15.181:0/1089244156 learned_addr learned my addr 172.21.15.181:0/1089244156 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): get_auth_request con 0x7fd6a4135710 auth_method 0 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd699d9b640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a4099c80 0x7fd6a4130dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:30.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a259c640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a4069a30 0x7fd6a413f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient(hunting): handle_auth_done global_id 45112 payload 824 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 -- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a4069a30 msgr2=0x7fd6a413f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a4069a30 0x7fd6a413f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 -- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a4099c80 msgr2=0x7fd6a4130dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a4099c80 0x7fd6a4130dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 monclient: found mon.noname-b 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:30.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a2d9d640 1 -- 172.21.15.181:0/1089244156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c009450 con 0x7fd6a4135710 2024-09-15T03:33:30.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a259c640 10 monclient: get_auth_request con 0x7fd6a4069a30 auth_method 0 2024-09-15T03:33:30.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.211+0000 7fd6a259c640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a4069a30 0x7fd6a413f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:30.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a2d9d640 1 -- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 msgr2=0x7fd6a4118960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a2d9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00bb40 tx=0x7fd69c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00cbf0 con 0x7fd6a4135710 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: dump: 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:30.654 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: _finish_auth 0 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c00cd70 con 0x7fd6a4135710 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 5 monclient: authenticate success, global_id 45112 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 10 monclient: _renew_subs 2024-09-15T03:33:30.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c006150 con 0x7fd6a4135710 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a411bec0 con 0x7fd6a4135710 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd69affd640 10 monclient: dump: 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:30.656 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a159a640 4 set_mon_vals no callback set 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 10 monclient: _renew_subs 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.213+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a413fda0 con 0x7fd6a4135710 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd69c012020 con 0x7fd6a4135710 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd6a8b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd6a8b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd6a8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a411a460 con 0x7fd6a4135710 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd69affd640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd6840386f0 0x7fd68403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.214+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(118..118 src has 40..118) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fd69c04c1c0 con 0x7fd6a4135710 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.215+0000 7fd6a259c640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd6840386f0 0x7fd68403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:30.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.215+0000 7fd6a259c640 10 monclient: get_auth_request con 0x7fd6840386f0 auth_method 0 2024-09-15T03:33:30.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.217+0000 7fd6a259c640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd6840386f0 0x7fd68403aba0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd694005090 tx=0x7fd694009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:30.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:30.219+0000 7fd6a2d9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00bb40 tx=0x7fd69c0097d0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fd69c01d8c0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:33:31.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.047+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd69c01d8c0 con 0x7fd6a4135710 2024-09-15T03:33:31.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.047+0000 7fd69affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.047+0000 7fd69affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:31.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd6a8b9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:31.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd6a8b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:31.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd6a8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fd6a404bf60 con 0x7fd6a4135710 2024-09-15T03:33:31.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v118) v1 ==== 73+0+3 (secure 0 0 0) 0x7fd69c01d8c0 con 0x7fd6a4135710 2024-09-15T03:33:31.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd69affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:31.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.214+0000 7fd69affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd6a8b9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd6a8b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd6a8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fd6a4098eb0 con 0x7fd6a4135710 2024-09-15T03:33:31.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd69affd640 1 -- 172.21.15.181:0/1089244156 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v118) v1 ==== 73+0+3 (secure 0 0 0) 0x7fd69c01d8c0 con 0x7fd6a4135710 2024-09-15T03:33:31.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd69affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:31.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.313+0000 7fd69affd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:31.315 INFO:tasks.workunit.client.0.smithi181.stdout:{} 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd6840386f0 msgr2=0x7fd68403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd6840386f0 0x7fd68403aba0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd694005090 tx=0x7fd694009040 comp rx=0 tx=0).stop 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 10 monclient: shutdown 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 msgr2=0x7fd6a4118960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00bb40 tx=0x7fd69c0097d0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 shutdown_connections 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd6840386f0 0x7fd68403aba0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a4135710 0x7fd6a4118960 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a4099c80 0x7fd6a4130dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 --2- 172.21.15.181:0/1089244156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a4069a30 0x7fd6a413f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 >> 172.21.15.181:0/1089244156 conn(0x7fd6a409e9f0 msgr2=0x7fd6a4134d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:31.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 shutdown_connections 2024-09-15T03:33:31.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.315+0000 7fd6a8b9d640 1 -- 172.21.15.181:0/1089244156 wait complete. 2024-09-15T03:33:31.325 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:33:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.397+0000 7f5118c20640 10 monclient: get_monmap_and_config 2024-09-15T03:33:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.397+0000 7f5118c20640 10 monclient: build_initial_monmap 2024-09-15T03:33:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.397+0000 7f5118c20640 10 monclient: monmap: 2024-09-15T03:33:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:31.398203+0000 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:31.398203+0000 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:31.399 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 1 Processor -- start 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 1 -- start start 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient: init 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5114069bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f51140981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient(hunting): picked mon.noname-b con 0x7f511409cf80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51140986f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511409cd40 con 0x7f5114069bd0 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511409e2d0 con 0x7f51140986f0 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511409d710 con 0x7f511409cf80 2024-09-15T03:33:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5118c20640 10 monclient(hunting): authenticate will time out at 1015.879719s 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5112d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f51140986f0 0x7f5114138e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56900/0 (socket says 172.21.15.181:56900) 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f511259c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f5114069bd0 legacy=0x7f5114131f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.398+0000 7f5112d9d640 1 -- 172.21.15.181:0/3558978301 learned_addr learned my addr 172.21.15.181:0/3558978301 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 1 --2- 172.21.15.181:0/3558978301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f51140981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): get_auth_request con 0x7f511409cf80 auth_method 0 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient(hunting): handle_auth_done global_id 74222 payload 274 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 1 -- 172.21.15.181:0/3558978301 >> v1:172.21.15.181:6789/0 conn(0x7f51140986f0 legacy=0x7f5114138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 1 -- 172.21.15.181:0/3558978301 >> v1:172.21.15.153:6789/0 conn(0x7f5114069bd0 legacy=0x7f5114131f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 1 monclient: found mon.noname-b 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.399+0000 7f510bfff640 1 -- 172.21.15.181:0/3558978301 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114098b50 con 0x7f511409cf80 2024-09-15T03:33:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f510bfff640 1 -- 172.21.15.181:0/3558978301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 msgr2=0x7f51140981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f510bfff640 1 --2- 172.21.15.181:0/3558978301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f51140981b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009880 tx=0x7f50fc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e4af4ba3cfae0d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 1 -- 172.21.15.181:0/3558978301 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc032440 con 0x7f511409cf80 2024-09-15T03:33:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: dump: 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: _finish_auth 0 2024-09-15T03:33:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 1 -- 172.21.15.181:0/3558978301 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50fc0325a0 con 0x7f511409cf80 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 1 -- 172.21.15.181:0/3558978301 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc032940 con 0x7f511409cf80 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5111d9b640 10 monclient: dump: 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5118c20640 5 monclient: authenticate success, global_id 74222 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5118c20640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.402+0000 7f5118c20640 4 set_mon_vals no callback set 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 10 monclient: shutdown 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 -- 172.21.15.181:0/3558978301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 msgr2=0x7f51140981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 --2- 172.21.15.181:0/3558978301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f51140981b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009880 tx=0x7f50fc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 -- 172.21.15.181:0/3558978301 shutdown_connections 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 --2- 172.21.15.181:0/3558978301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f51140981b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 -- 172.21.15.181:0/3558978301 >> 172.21.15.181:0/3558978301 conn(0x7f511409e9f0 msgr2=0x7f511409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 -- 172.21.15.181:0/3558978301 shutdown_connections 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 -- 172.21.15.181:0/3558978301 wait complete. 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 10 monclient: build_initial_monmap 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 10 monclient: monmap: 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 Processor -- start 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 1 -- start start 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.403+0000 7f5118c20640 10 monclient: init 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 0x7f511409b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): picked mon.noname-b con 0x7f5114069bd0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51140986f0 0x7f511412aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): picked mon.noname-a con 0x7f51140986f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f511412b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): picked mon.noname-c con 0x7f511409cf80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511412bfc0 con 0x7f51140986f0 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511412d510 con 0x7f5114069bd0 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511412d7f0 con 0x7f511409cf80 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5118c20640 10 monclient(hunting): authenticate will time out at 1015.885337s 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 0x7f511409b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 0x7f511409b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49608/0 (socket says 172.21.15.181:49608) 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f510bfff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51140986f0 msgr2=0x7f511412aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 -- 172.21.15.181:0/598299714 learned_addr learned my addr 172.21.15.181:0/598299714 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): get_auth_request con 0x7f5114069bd0 auth_method 0 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:31.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f510bfff640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51140986f0 0x7f511412aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f511412b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): get_auth_request con 0x7f511409cf80 auth_method 0 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:31.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f5112d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient(hunting): handle_auth_done global_id 45115 payload 824 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 -- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 msgr2=0x7f511412b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f511412b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 -- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51140986f0 msgr2=0x7f511412aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51140986f0 0x7f511412aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 monclient: found mon.noname-b 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.404+0000 7f511259c640 1 -- 172.21.15.181:0/598299714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc009390 con 0x7f5114069bd0 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.406+0000 7f511259c640 1 -- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 msgr2=0x7f511409b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.406+0000 7f511259c640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 0x7f511409b7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f510400b630 tx=0x7f51040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104015070 con 0x7f5114069bd0 2024-09-15T03:33:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: dump: 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:31.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 5 monclient: authenticate success, global_id 45115 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104003d90 con 0x7f5114069bd0 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 10 monclient: _renew_subs 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511412e8f0 con 0x7f5114069bd0 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104006380 con 0x7f5114069bd0 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 10 monclient: dump: 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.420 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f511159a640 4 set_mon_vals no callback set 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 10 monclient: _renew_subs 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511412c140 con 0x7f5114069bd0 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5104010440 con 0x7f5114069bd0 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.407+0000 7f5118c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f5118c20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f5118c20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511412dc70 con 0x7f5114069bd0 2024-09-15T03:33:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f510a7fc640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f40386f0 0x7f50f403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(118..118 src has 40..118) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f510404bcb0 con 0x7f5114069bd0 2024-09-15T03:33:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f510bfff640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f40386f0 0x7f50f403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.408+0000 7f510bfff640 10 monclient: get_auth_request con 0x7f50f40386f0 auth_method 0 2024-09-15T03:33:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.410+0000 7f510bfff640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f40386f0 0x7f50f403aba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f511406a4e0 tx=0x7f50fc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:31.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.412+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5104018a70 con 0x7f5114069bd0 2024-09-15T03:33:31.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.412+0000 7f510a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.412+0000 7f510a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f5118c20640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f5118c20640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f5118c20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f511404bf60 con 0x7f5114069bd0 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v118) v1 ==== 71+0+0 (secure 0 0 0) 0x7f5104018a70 con 0x7f5114069bd0 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f510a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:31.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.575+0000 7f510a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:31.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f5118c20640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:31.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f5118c20640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:31.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f5118c20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f511413a540 con 0x7f5114069bd0 2024-09-15T03:33:31.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f510a7fc640 1 -- 172.21.15.181:0/598299714 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v118) v1 ==== 71+0+0 (secure 0 0 0) 0x7f5104018a70 con 0x7f5114069bd0 2024-09-15T03:33:31.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f510a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:31.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.676+0000 7f510a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:31.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f40386f0 msgr2=0x7f50f403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f40386f0 0x7f50f403aba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f511406a4e0 tx=0x7f50fc032000 comp rx=0 tx=0).stop 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 10 monclient: shutdown 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 msgr2=0x7f511409b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 0x7f511409b7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f510400b630 tx=0x7f51040091a0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 shutdown_connections 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f40386f0 0x7f50f403aba0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409cf80 0x7f511412b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51140986f0 0x7f511412aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 --2- 172.21.15.181:0/598299714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114069bd0 0x7f511409b7f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 >> 172.21.15.181:0/598299714 conn(0x7f511409e9f0 msgr2=0x7f51140a08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 shutdown_connections 2024-09-15T03:33:31.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.678+0000 7f5118c20640 1 -- 172.21.15.181:0/598299714 wait complete. 2024-09-15T03:33:31.688 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T03:33:31.688 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:33:31.688 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T03:33:31.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.763+0000 7f924d79d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:31.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.763+0000 7f924d79d640 10 monclient: build_initial_monmap 2024-09-15T03:33:31.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.763+0000 7f924d79d640 10 monclient: monmap: 2024-09-15T03:33:31.764 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:31.764 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.764 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:31.764063+0000 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:31.764063+0000 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:31.765 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.763+0000 7f924d79d640 1 Processor -- start 2024-09-15T03:33:31.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.763+0000 7f924d79d640 1 -- start start 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.763+0000 7f924d79d640 10 monclient: init 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9248096a30 0x7f9248096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): picked mon.noname-c con 0x7f9248096a30 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f9248097340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): picked mon.noname-b con 0x7f9248099f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:31.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f924809d4e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9248097880 con 0x7f924809d4e0 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924809cb80 con 0x7f9248096a30 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9248098c50 con 0x7f9248099f00 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f924d79d640 10 monclient(hunting): authenticate will time out at 1016.245336s 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9248096a30 0x7f9248096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f92477fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f924809d4e0 0x7f924812e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56916/0 (socket says 172.21.15.181:56916) 2024-09-15T03:33:31.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f92477fe640 1 -- 172.21.15.181:0/2174757336 learned_addr learned my addr 172.21.15.181:0/2174757336 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): get_auth_request con 0x7f9248096a30 auth_method 0 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient(hunting): handle_auth_done global_id 45118 payload 274 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f92467fc640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f9248097340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 1 -- 172.21.15.181:0/2174757336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 msgr2=0x7f9248097340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f9248097340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 1 -- 172.21.15.181:0/2174757336 >> v1:172.21.15.181:6789/0 conn(0x7f924809d4e0 legacy=0x7f924812e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 1 monclient: found mon.noname-c 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f9246ffd640 1 -- 172.21.15.181:0/2174757336 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9248098f30 con 0x7f9248096a30 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f92467fc640 10 monclient: get_auth_request con 0x7f9248099f00 auth_method 0 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.764+0000 7f92467fc640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f9248097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9246ffd640 1 -- 172.21.15.181:0/2174757336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 msgr2=0x7f9248096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:31.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9246ffd640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f9248096e00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9240009850 tx=0x7f92400090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=660ad07b07d0e578 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 1 -- 172.21.15.181:0/2174757336 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240032440 con 0x7f9248096a30 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: dump: 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.777 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 1 -- 172.21.15.181:0/2174757336 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92400325a0 con 0x7f9248096a30 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f924d79d640 5 monclient: authenticate success, global_id 45118 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f924d79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f924d79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 1 -- 172.21.15.181:0/2174757336 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240032940 con 0x7f9248096a30 2024-09-15T03:33:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f924d79d640 4 set_mon_vals no callback set 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.767+0000 7f9245ffb640 10 monclient: dump: 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 10 monclient: shutdown 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 -- 172.21.15.181:0/2174757336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 msgr2=0x7f9248096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f9248096e00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9240009850 tx=0x7f92400090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 -- 172.21.15.181:0/2174757336 shutdown_connections 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f9248097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 --2- 172.21.15.181:0/2174757336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f9248096e00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 -- 172.21.15.181:0/2174757336 >> 172.21.15.181:0/2174757336 conn(0x7f92480711a0 msgr2=0x7f92480a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 -- 172.21.15.181:0/2174757336 shutdown_connections 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.768+0000 7f924d79d640 1 -- 172.21.15.181:0/2174757336 wait complete. 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient: build_initial_monmap 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient: monmap: 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:31.781 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 Processor -- start 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 -- start start 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient: init 2024-09-15T03:33:31.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f924813f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): picked mon.noname-b con 0x7f9248096a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f924813f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): picked mon.noname-c con 0x7f9248099f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f924809d4e0 0x7f924811c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): picked mon.noname-a con 0x7f924809d4e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924811fc70 con 0x7f924809d4e0 2024-09-15T03:33:31.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9248098460 con 0x7f9248096a30 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924809fb10 con 0x7f9248099f00 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.769+0000 7f924d79d640 10 monclient(hunting): authenticate will time out at 1016.250945s 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f924813f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92477fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f924809d4e0 msgr2=0x7f924811c3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:31.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f924813f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49618/0 (socket says 172.21.15.181:49618) 2024-09-15T03:33:31.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 -- 172.21.15.181:0/590675859 learned_addr learned my addr 172.21.15.181:0/590675859 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:31.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): get_auth_request con 0x7f9248096a30 auth_method 0 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92477fe640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f924809d4e0 0x7f924811c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92467fc640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f924813f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92467fc640 10 monclient(hunting): get_auth_request con 0x7f9248099f00 auth_method 0 2024-09-15T03:33:31.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f92467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient(hunting): handle_auth_done global_id 45121 payload 824 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 -- 172.21.15.181:0/590675859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 msgr2=0x7f924813f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9248099f00 0x7f924813f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 -- 172.21.15.181:0/590675859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f924809d4e0 msgr2=0x7f924811c3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f924809d4e0 0x7f924811c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:31.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 monclient: found mon.noname-b 2024-09-15T03:33:31.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:31.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.788 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.770+0000 7f9246ffd640 1 -- 172.21.15.181:0/590675859 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240009390 con 0x7f9248096a30 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f9246ffd640 1 -- 172.21.15.181:0/590675859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 msgr2=0x7f924813f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f9246ffd640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9248096a30 0x7f924813f200 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9240009850 tx=0x7f9240032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240031c70 con 0x7f9248096a30 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: dump: 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.792 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.793 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: _finish_auth 0 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 5 monclient: authenticate success, global_id 45121 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240031dd0 con 0x7f9248096a30 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 10 monclient: _renew_subs 2024-09-15T03:33:31.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924003d650 con 0x7f9248096a30 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 1 -- 172.21.15.181:0/590675859 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924811ea30 con 0x7f9248096a30 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f922effd640 10 monclient: dump: 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f92457fa640 4 set_mon_vals no callback set 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 10 monclient: _renew_subs 2024-09-15T03:33:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.772+0000 7f924d79d640 1 -- 172.21.15.181:0/590675859 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9248120c20 con 0x7f9248096a30 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9240039040 con 0x7f9248096a30 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f922effd640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234038740 0x7f923403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f924d79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(118..118 src has 40..118) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f924006c2b0 con 0x7f9248096a30 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f924d79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f924d79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f924d79d640 1 -- 172.21.15.181:0/590675859 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924811fe70 con 0x7f9248096a30 2024-09-15T03:33:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f92467fc640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234038740 0x7f923403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.773+0000 7f92467fc640 10 monclient: get_auth_request con 0x7f9234038740 auth_method 0 2024-09-15T03:33:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.776+0000 7f92467fc640 1 --2- 172.21.15.181:0/590675859 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234038740 0x7f923403abf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f923c00ab30 tx=0x7f923c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.778+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9240042020 con 0x7f9248096a30 2024-09-15T03:33:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.778+0000 7f922effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.778+0000 7f922effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f924d79d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f924d79d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f924d79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f924d79d640 1 -- 172.21.15.181:0/590675859 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f924811faa0 con 0x7f9248096a30 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f922effd640 1 -- 172.21.15.181:0/590675859 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v118) v1 ==== 152+0+0 (secure 0 0 0) 0x7f9240037070 con 0x7f9248096a30 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f922effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:31.939+0000 7f922effd640 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-15T03:33:31.941 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOENT: pool 'fooset' not found 2024-09-15T03:33:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:33:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create fooset 8 2024-09-15T03:33:32.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.017+0000 7fca4055d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:32.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.017+0000 7fca4055d640 10 monclient: build_initial_monmap 2024-09-15T03:33:32.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.017+0000 7fca4055d640 10 monclient: monmap: 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:32.018394+0000 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:32.018394+0000 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:32.019 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 1 Processor -- start 2024-09-15T03:33:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 1 -- start start 2024-09-15T03:33:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient: init 2024-09-15T03:33:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T03:33:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fca3809f320 0x7fca38132170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient(hunting): picked mon.noname-a con 0x7fca3809f320 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca38070920 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca38071430 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3809e1f0 con 0x7fca38070920 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3809e7b0 con 0x7fca38071430 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3809e4d0 con 0x7fca3809f320 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca4055d640 10 monclient(hunting): authenticate will time out at 1016.499894s 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca3daf8640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fca38070920 legacy=0x7fca38070d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.018+0000 7fca3e2f9640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fca3809f320 msgr2=0x7fca38132170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.019+0000 7fca3eafa640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fca38071430 0x7fca38139040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56918/0 (socket says 172.21.15.181:56918) 2024-09-15T03:33:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.019+0000 7fca3e2f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fca3809f320 0x7fca38132170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.019+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 learned_addr learned my addr 172.21.15.181:0/1851401552 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:32.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 >> v1:172.21.15.181:6789/0 conn(0x7fca38071430 legacy=0x7fca38139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728437231 0 0) 0x7fca3809e7b0 con 0x7fca38071430 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): my global_id is 45124 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca240034e0 con 0x7fca38071430 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620501402 0 0) 0x7fca240034e0 con 0x7fca38071430 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 >> v2:172.21.15.153:3300/0 conn(0x7fca3809f320 msgr2=0x7fca38132170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 --2- 172.21.15.181:0/1851401552 >> v2:172.21.15.153:3300/0 conn(0x7fca3809f320 0x7fca38132170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 >> v1:172.21.15.153:6789/0 conn(0x7fca38070920 legacy=0x7fca38070d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: _finish_hunting 0 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:33:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3809cae0 con 0x7fca38071430 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: _finish_auth 0 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fca30001720 con 0x7fca38071430 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca4055d640 5 monclient: authenticate success, global_id 45124 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca4055d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: dump: 2024-09-15T03:33:32.031 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 >> v1:172.21.15.181:6789/0 conn(0x7fca38071430 legacy=0x7fca38139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca240039d0 0x7fca24003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): picked mon.a con 0x7fca240039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca24004780 0x7fca24004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): picked mon.c con 0x7fca24004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 0x7fca2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): picked mon.b con 0x7fca24005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3809e4d0 con 0x7fca240039d0 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2400c290 con 0x7fca24005330 2024-09-15T03:33:32.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2400c450 con 0x7fca24004780 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3d2f7640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca4055d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 0x7fca2400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): get_auth_request con 0x7fca24005330 auth_method 0 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3e2f9640 1 -- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca240039d0 msgr2=0x7fca24003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3e2f9640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca240039d0 0x7fca24003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient(hunting): handle_auth_done global_id 45124 payload 293 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca24004780 msgr2=0x7fca24004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca24004780 0x7fca24004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca240039d0 msgr2=0x7fca24003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca240039d0 0x7fca24003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient: _finish_hunting 0 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 monclient: found mon.b 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca240034e0 con 0x7fca24005330 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 10 monclient: _finish_auth 0 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.021+0000 7fca3eafa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.022+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 reap_dead start 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3eafa640 1 -- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 msgr2=0x7fca2400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3eafa640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 0x7fca2400b9d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fca30001490 tx=0x7fca30007a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33423b91b5a436a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30033070 con 0x7fca24005330 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: dump: 2024-09-15T03:33:32.036 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.037 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.037 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.037 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.037 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.037 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca4055d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca30005890 con 0x7fca24005330 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca4055d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca4055d640 4 set_mon_vals no callback set 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 1 -- 172.21.15.181:0/1851401552 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30005c70 con 0x7fca24005330 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.024+0000 7fca3d2f7640 10 monclient: dump: 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 10 monclient: shutdown 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:32.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 -- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 msgr2=0x7fca2400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 0x7fca2400b9d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fca30001490 tx=0x7fca30007a90 comp rx=0 tx=0).stop 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 -- 172.21.15.181:0/1851401552 shutdown_connections 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 --2- 172.21.15.181:0/1851401552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca24005330 0x7fca2400b9d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 -- 172.21.15.181:0/1851401552 >> 172.21.15.181:0/1851401552 conn(0x7fca380a2b70 msgr2=0x7fca3809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 -- 172.21.15.181:0/1851401552 shutdown_connections 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.025+0000 7fca4055d640 1 -- 172.21.15.181:0/1851401552 wait complete. 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient: build_initial_monmap 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient: monmap: 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:32.040 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 Processor -- start 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 -- start start 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient: init 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca24005330 0x7fca3812eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): picked mon.noname-a con 0x7fca24005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:32.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca38133040 0x7fca3811b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): picked mon.noname-c con 0x7fca38133040 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 0x7fca3811bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): picked mon.noname-b con 0x7fca3812efe0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3809d620 con 0x7fca24005330 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3811eed0 con 0x7fca3812efe0 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca38141bd0 con 0x7fca38133040 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca4055d640 10 monclient(hunting): authenticate will time out at 1016.507874s 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca3e2f9640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca24005330 msgr2=0x7fca3812eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:32.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca3e2f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca24005330 0x7fca3812eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca3eafa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 0x7fca3811bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca3eafa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 0x7fca3811bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49646/0 (socket says 172.21.15.181:49646) 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.026+0000 7fca3eafa640 1 -- 172.21.15.181:0/446964370 learned_addr learned my addr 172.21.15.181:0/446964370 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): get_auth_request con 0x7fca3812efe0 auth_method 0 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3daf8640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca38133040 0x7fca3811b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3daf8640 10 monclient(hunting): get_auth_request con 0x7fca38133040 auth_method 0 2024-09-15T03:33:32.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3daf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3daf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3daf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient(hunting): handle_auth_done global_id 45127 payload 824 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 1 -- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca38133040 msgr2=0x7fca3811b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca38133040 0x7fca3811b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 1 -- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca24005330 msgr2=0x7fca3812eaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca24005330 0x7fca3812eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient: _finish_hunting 0 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 1 monclient: found mon.noname-b 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.027+0000 7fca3eafa640 1 -- 172.21.15.181:0/446964370 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38097760 con 0x7fca3812efe0 2024-09-15T03:33:32.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca3eafa640 1 -- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 msgr2=0x7fca3811bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:32.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca3eafa640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 0x7fca3811bae0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fca3000c5f0 tx=0x7fca30033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:32.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca300056b0 con 0x7fca3812efe0 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: dump: 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.047 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca300047a0 con 0x7fca3812efe0 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30004b60 con 0x7fca3812efe0 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 5 monclient: authenticate success, global_id 45127 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 10 monclient: _renew_subs 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3811ebf0 con 0x7fca3812efe0 2024-09-15T03:33:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca3caf6640 4 set_mon_vals no callback set 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca2e7fc640 10 monclient: dump: 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 10 monclient: _renew_subs 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.029+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3811c020 con 0x7fca3812efe0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fca30007070 con 0x7fca3812efe0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca4055d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca4055d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca2e7fc640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca1c0386f0 0x7fca1c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca4055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca380008d0 con 0x7fca3812efe0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(118..118 src has 40..118) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fca3006c910 con 0x7fca3812efe0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca3e2f9640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca1c0386f0 0x7fca1c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.030+0000 7fca3e2f9640 10 monclient: get_auth_request con 0x7fca1c0386f0 auth_method 0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.033+0000 7fca3e2f9640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca1c0386f0 0x7fca1c03aba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fca34003c70 tx=0x7fca34009350 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.035+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fca3003b440 con 0x7fca3812efe0 2024-09-15T03:33:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.035+0000 7fca2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:32.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.035+0000 7fca2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:32.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.192+0000 7fca4055d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T03:33:32.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.192+0000 7fca4055d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T03:33:32.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.192+0000 7fca4055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.192+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7fca3811f110 con 0x7fca3812efe0 2024-09-15T03:33:32.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.618+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v119) v1 ==== 115+0+0 (secure 0 0 0) 0x7fca30037bb0 con 0x7fca3812efe0 2024-09-15T03:33:32.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.618+0000 7fca2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T03:33:32.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.618+0000 7fca2e7fc640 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-15T03:33:32.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca4055d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca4055d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca4055d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7fca38072640 con 0x7fca3812efe0 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca2e7fc640 1 -- 172.21.15.181:0/446964370 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v119) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca30037d90 con 0x7fca3812efe0 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca2e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.714+0000 7fca2e7fc640 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-15T03:33:32.716 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'fooset' already exists 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca1c0386f0 msgr2=0x7fca1c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca1c0386f0 0x7fca1c03aba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fca34003c70 tx=0x7fca34009350 comp rx=0 tx=0).stop 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 10 monclient: shutdown 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 msgr2=0x7fca3811bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 0x7fca3811bae0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fca3000c5f0 tx=0x7fca30033000 comp rx=0 tx=0).stop 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 shutdown_connections 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca1c0386f0 0x7fca1c03aba0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca3812efe0 0x7fca3811bae0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca38133040 0x7fca3811b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 --2- 172.21.15.181:0/446964370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca24005330 0x7fca3812eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.716+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 >> 172.21.15.181:0/446964370 conn(0x7fca380a2b70 msgr2=0x7fca3809fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:32.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.717+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 shutdown_connections 2024-09-15T03:33:32.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.717+0000 7fca4055d640 1 -- 172.21.15.181:0/446964370 wait complete. 2024-09-15T03:33:32.726 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create barset 8 2024-09-15T03:33:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.800+0000 7ff13d42b640 10 monclient: get_monmap_and_config 2024-09-15T03:33:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.800+0000 7ff13d42b640 10 monclient: build_initial_monmap 2024-09-15T03:33:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.800+0000 7ff13d42b640 10 monclient: monmap: 2024-09-15T03:33:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:32.801736+0000 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:32.801736+0000 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:32.804 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:32.804 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.801+0000 7ff13d42b640 1 Processor -- start 2024-09-15T03:33:32.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.801+0000 7ff13d42b640 1 -- start start 2024-09-15T03:33:32.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.801+0000 7ff13d42b640 10 monclient: init 2024-09-15T03:33:32.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:32.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T03:33:32.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1380a0ad0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:32.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff13809ee60 0x7ff13809f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): picked mon.noname-c con 0x7ff13809ee60 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff138136e00 0x7ff1381391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): picked mon.noname-b con 0x7ff138136e00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13806a5b0 con 0x7ff1380a0ad0 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13809f7a0 con 0x7ff13809ee60 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1380a0ea0 con 0x7ff138136e00 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff13d42b640 10 monclient(hunting): authenticate will time out at 1017.283166s 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff136ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7ff1380a0ad0 legacy=0x7ff1380691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:32.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff13809ee60 0x7ff13809f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff13809ee60 0x7ff13809f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49660/0 (socket says 172.21.15.181:49660) 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 -- 172.21.15.181:0/3982420629 learned_addr learned my addr 172.21.15.181:0/3982420629 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): get_auth_request con 0x7ff13809ee60 auth_method 0 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1377fe640 1 --2- 172.21.15.181:0/3982420629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff138136e00 0x7ff1381391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1377fe640 10 monclient(hunting): get_auth_request con 0x7ff138136e00 auth_method 0 2024-09-15T03:33:32.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient(hunting): handle_auth_done global_id 45130 payload 274 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 -- 172.21.15.181:0/3982420629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff138136e00 msgr2=0x7ff1381391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 --2- 172.21.15.181:0/3982420629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff138136e00 0x7ff1381391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 -- 172.21.15.181:0/3982420629 >> v1:172.21.15.153:6789/0 conn(0x7ff1380a0ad0 legacy=0x7ff1380691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 monclient: found mon.noname-c 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.802+0000 7ff1367fc640 1 -- 172.21.15.181:0/3982420629 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380a0460 con 0x7ff13809ee60 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff1367fc640 1 -- 172.21.15.181:0/3982420629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff13809ee60 msgr2=0x7ff13809f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff1367fc640 1 --2- 172.21.15.181:0/3982420629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff13809ee60 0x7ff13809f260 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009910 tx=0x7ff12c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=581ca3891ff8033a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:32.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 1 -- 172.21.15.181:0/3982420629 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c032440 con 0x7ff13809ee60 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: dump: 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.814 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 1 -- 172.21.15.181:0/3982420629 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c0325a0 con 0x7ff13809ee60 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff13d42b640 5 monclient: authenticate success, global_id 45130 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff13d42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff13d42b640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 1 -- 172.21.15.181:0/3982420629 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c032940 con 0x7ff13809ee60 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff13d42b640 4 set_mon_vals no callback set 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.805+0000 7ff135ffb640 10 monclient: dump: 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.816 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 10 monclient: shutdown 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 -- 172.21.15.181:0/3982420629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff13809ee60 msgr2=0x7ff13809f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 --2- 172.21.15.181:0/3982420629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff13809ee60 0x7ff13809f260 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009910 tx=0x7ff12c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 -- 172.21.15.181:0/3982420629 shutdown_connections 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 --2- 172.21.15.181:0/3982420629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff138136e00 0x7ff1381391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 --2- 172.21.15.181:0/3982420629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff13809ee60 0x7ff13809f260 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 -- 172.21.15.181:0/3982420629 >> 172.21.15.181:0/3982420629 conn(0x7ff138071370 msgr2=0x7ff1380a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:32.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 -- 172.21.15.181:0/3982420629 shutdown_connections 2024-09-15T03:33:32.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 1 -- 172.21.15.181:0/3982420629 wait complete. 2024-09-15T03:33:32.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 10 monclient: build_initial_monmap 2024-09-15T03:33:32.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.806+0000 7ff13d42b640 10 monclient: monmap: 2024-09-15T03:33:32.818 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:32.819 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:32.819 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 Processor -- start 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 -- start start 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient: init 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:32.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff13809ee60 0x7ff138141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): picked mon.noname-c con 0x7ff13809ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1380a0ad0 0x7ff1381417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1380a0ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 0x7ff138141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): picked mon.noname-b con 0x7ff138136e00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138121a60 con 0x7ff1380a0ad0 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138120c10 con 0x7ff138136e00 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff138120520 con 0x7ff13809ee60 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff13d42b640 10 monclient(hunting): authenticate will time out at 1017.288660s 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 0x7ff138141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 0x7ff138141cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49674/0 (socket says 172.21.15.181:49674) 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1367fc640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1380a0ad0 msgr2=0x7ff1381417b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 -- 172.21.15.181:0/1007750942 learned_addr learned my addr 172.21.15.181:0/1007750942 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): get_auth_request con 0x7ff138136e00 auth_method 0 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1367fc640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1380a0ad0 0x7ff1381417b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff136ffd640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff13809ee60 0x7ff138141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff136ffd640 10 monclient(hunting): get_auth_request con 0x7ff13809ee60 auth_method 0 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff136ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff136ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient(hunting): handle_auth_done global_id 45133 payload 824 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 -- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff13809ee60 msgr2=0x7ff138141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff13809ee60 0x7ff138141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 -- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1380a0ad0 msgr2=0x7ff1381417b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:32.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1380a0ad0 0x7ff1381417b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:32.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:32.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:32.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:32.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.807+0000 7ff1377fe640 1 -- 172.21.15.181:0/1007750942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c009450 con 0x7ff138136e00 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff1377fe640 1 -- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 msgr2=0x7ff138141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff1377fe640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 0x7ff138141cf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff128009fd0 tx=0x7ff128009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12800ee90 con 0x7ff138136e00 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: dump: 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.826 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: _finish_auth 0 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff128013590 con 0x7ff138136e00 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 5 monclient: authenticate success, global_id 45133 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:32.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128013950 con 0x7ff138136e00 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 10 monclient: _renew_subs 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1380a11e0 con 0x7ff138136e00 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff11effd640 10 monclient: dump: 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:32.828 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff1357fa640 4 set_mon_vals no callback set 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 10 monclient: _renew_subs 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.810+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13811ffa0 con 0x7ff138136e00 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff128013d00 con 0x7ff138136e00 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff11effd640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff1240386f0 0x7ff12403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(119..119 src has 40..119) v4 ==== 6868+0+0 (secure 0 0 0) 0x7ff12804cbf0 con 0x7ff138136e00 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff13d42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff13d42b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff13d42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff138121da0 con 0x7ff138136e00 2024-09-15T03:33:32.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff136ffd640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff1240386f0 0x7ff12403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:32.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.811+0000 7ff136ffd640 10 monclient: get_auth_request con 0x7ff1240386f0 auth_method 0 2024-09-15T03:33:32.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.814+0000 7ff136ffd640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff1240386f0 0x7ff12403aba0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff1300038f0 tx=0x7ff130006ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:32.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.816+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff138121da0 con 0x7ff138136e00 2024-09-15T03:33:32.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.816+0000 7ff11effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:32.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.816+0000 7ff11effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:32.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.973+0000 7ff13d42b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T03:33:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.973+0000 7ff13d42b640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T03:33:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.973+0000 7ff13d42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:32.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:32.973+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7ff138121da0 con 0x7ff138136e00 2024-09-15T03:33:33.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.617+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v120) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff138121da0 con 0x7ff138136e00 2024-09-15T03:33:33.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.617+0000 7ff11effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T03:33:33.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.617+0000 7ff11effd640 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.718+0000 7ff13d42b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.718+0000 7ff13d42b640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.718+0000 7ff13d42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.718+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7ff138121da0 con 0x7ff138136e00 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.719+0000 7ff11effd640 1 -- 172.21.15.181:0/1007750942 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v120) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff138121da0 con 0x7ff138136e00 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.719+0000 7ff11effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.719+0000 7ff11effd640 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-15T03:33:33.720 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'barset' already exists 2024-09-15T03:33:33.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.720+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff1240386f0 msgr2=0x7ff12403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:33.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.720+0000 7ff13d42b640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff1240386f0 0x7ff12403aba0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff1300038f0 tx=0x7ff130006ef0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.720+0000 7ff13d42b640 10 monclient: shutdown 2024-09-15T03:33:33.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.720+0000 7ff13d42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:33.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.720+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 msgr2=0x7ff138141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.720+0000 7ff13d42b640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 0x7ff138141cf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff128009fd0 tx=0x7ff128009900 comp rx=0 tx=0).stop 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 shutdown_connections 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff1240386f0 0x7ff12403aba0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138136e00 0x7ff138141cf0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1380a0ad0 0x7ff1381417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 --2- 172.21.15.181:0/1007750942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff13809ee60 0x7ff138141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 >> 172.21.15.181:0/1007750942 conn(0x7ff138071370 msgr2=0x7ff13809ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 shutdown_connections 2024-09-15T03:33:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.721+0000 7ff13d42b640 1 -- 172.21.15.181:0/1007750942 wait complete. 2024-09-15T03:33:33.731 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set barset size 3 2024-09-15T03:33:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.804+0000 7fd45fa6f640 10 monclient: get_monmap_and_config 2024-09-15T03:33:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.804+0000 7fd45fa6f640 10 monclient: build_initial_monmap 2024-09-15T03:33:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.804+0000 7fd45fa6f640 10 monclient: monmap: 2024-09-15T03:33:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:33.805127+0000 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:33.805127+0000 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:33.806 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.804+0000 7fd45fa6f640 1 Processor -- start 2024-09-15T03:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 1 -- start start 2024-09-15T03:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient: init 2024-09-15T03:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd45809f320 0x7fd4581321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): picked mon.noname-c con 0x7fd45809f320 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd458070920 0x7fd458070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): picked mon.noname-a con 0x7fd458070920 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd458071260 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd458096b60 con 0x7fd458071260 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45809e4d0 con 0x7fd458070920 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45809e7b0 con 0x7fd45809f320 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45fa6f640 10 monclient(hunting): authenticate will time out at 1018.286653s 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd45809f320 0x7fd4581321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd45809f320 0x7fd4581321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49686/0 (socket says 172.21.15.181:49686) 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d00a640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fd458070920 msgr2=0x7fd458070d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45e00c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd458071260 0x7fd45812e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56926/0 (socket says 172.21.15.181:56926) 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 1 -- 172.21.15.181:0/2463610471 learned_addr learned my addr 172.21.15.181:0/2463610471 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:33.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): get_auth_request con 0x7fd45809f320 auth_method 0 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d00a640 1 --2- 172.21.15.181:0/2463610471 >> v2:172.21.15.153:3300/0 conn(0x7fd458070920 0x7fd458070d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:33.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.805+0000 7fd45d80b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 10 monclient(hunting): handle_auth_done global_id 45136 payload 274 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 1 -- 172.21.15.181:0/2463610471 >> v2:172.21.15.153:3300/0 conn(0x7fd458070920 msgr2=0x7fd458070d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 1 --2- 172.21.15.181:0/2463610471 >> v2:172.21.15.153:3300/0 conn(0x7fd458070920 0x7fd458070d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 1 -- 172.21.15.181:0/2463610471 >> v1:172.21.15.181:6789/0 conn(0x7fd458071260 legacy=0x7fd45812e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 1 monclient: found mon.noname-c 2024-09-15T03:33:33.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.806+0000 7fd45d80b640 1 -- 172.21.15.181:0/2463610471 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45809e1f0 con 0x7fd45809f320 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.808+0000 7fd45d80b640 1 -- 172.21.15.181:0/2463610471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd45809f320 msgr2=0x7fd4581321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.808+0000 7fd45d80b640 1 --2- 172.21.15.181:0/2463610471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd45809f320 0x7fd4581321b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd45809e9d0 tx=0x7fd4540091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90ce9e9996e4c4f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 1 -- 172.21.15.181:0/2463610471 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454032440 con 0x7fd45809f320 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: dump: 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: _finish_auth 0 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 1 -- 172.21.15.181:0/2463610471 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4540325a0 con 0x7fd45809f320 2024-09-15T03:33:33.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 5 monclient: authenticate success, global_id 45136 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 1 -- 172.21.15.181:0/2463610471 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454032980 con 0x7fd45809f320 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 4 set_mon_vals no callback set 2024-09-15T03:33:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:33.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45c809640 10 monclient: dump: 2024-09-15T03:33:33.829 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:33.829 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:33.829 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.829 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.829 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 10 monclient: shutdown 2024-09-15T03:33:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 -- 172.21.15.181:0/2463610471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd45809f320 msgr2=0x7fd4581321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/2463610471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd45809f320 0x7fd4581321b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd45809e9d0 tx=0x7fd4540091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 -- 172.21.15.181:0/2463610471 shutdown_connections 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/2463610471 >> v2:172.21.15.153:3300/0 conn(0x7fd458070920 0x7fd458070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/2463610471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd45809f320 0x7fd4581321b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 -- 172.21.15.181:0/2463610471 >> 172.21.15.181:0/2463610471 conn(0x7fd4580a2b70 msgr2=0x7fd45809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 -- 172.21.15.181:0/2463610471 shutdown_connections 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.809+0000 7fd45fa6f640 1 -- 172.21.15.181:0/2463610471 wait complete. 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient: build_initial_monmap 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient: monmap: 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:33.832 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 Processor -- start 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 -- start start 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient: init 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 0x7fd45812fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): picked mon.noname-b con 0x7fd458070920 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd458071260 0x7fd4581198b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): picked mon.noname-a con 0x7fd458071260 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd45809f320 0x7fd458119df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): picked mon.noname-c con 0x7fd45809f320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45811cec0 con 0x7fd458071260 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45811b980 con 0x7fd458070920 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd458130eb0 con 0x7fd45809f320 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45fa6f640 10 monclient(hunting): authenticate will time out at 1018.291898s 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.810+0000 7fd45d80b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 0x7fd45812fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 0x7fd45812fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49698/0 (socket says 172.21.15.181:49698) 2024-09-15T03:33:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 -- 172.21.15.181:0/411891488 learned_addr learned my addr 172.21.15.181:0/411891488 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): get_auth_request con 0x7fd458070920 auth_method 0 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d00a640 1 -- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd458071260 msgr2=0x7fd4581198b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d00a640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd458071260 0x7fd4581198b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient(hunting): handle_auth_done global_id 45139 payload 824 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 -- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd45809f320 msgr2=0x7fd458119df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45e00c640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd45809f320 0x7fd458119df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd45809f320 0x7fd458119df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 -- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd458071260 msgr2=0x7fd4581198b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd458071260 0x7fd4581198b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 monclient: found mon.noname-b 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.811+0000 7fd45d80b640 1 -- 172.21.15.181:0/411891488 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454009450 con 0x7fd458070920 2024-09-15T03:33:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45d80b640 1 -- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 msgr2=0x7fd45812fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45d80b640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 0x7fd45812fda0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd454009210 tx=0x7fd454032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45403ca40 con 0x7fd458070920 2024-09-15T03:33:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: dump: 2024-09-15T03:33:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:33.842 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:33.842 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:33.842 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:33.842 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45403cba0 con 0x7fd458070920 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 5 monclient: authenticate success, global_id 45139 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 10 monclient: _renew_subs 2024-09-15T03:33:33.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45809d5b0 con 0x7fd458070920 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4540313a0 con 0x7fd458070920 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44dffb640 10 monclient: dump: 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd44ffff640 4 set_mon_vals no callback set 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 10 monclient: _renew_subs 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.813+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45811b400 con 0x7fd458070920 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.814+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd454039040 con 0x7fd458070920 2024-09-15T03:33:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.814+0000 7fd45fa6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.814+0000 7fd45fa6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.814+0000 7fd45fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.814+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45811d330 con 0x7fd458070920 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.814+0000 7fd44dffb640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd43c038720 0x7fd43c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.815+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(120..120 src has 40..120) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fd45406c6b0 con 0x7fd458070920 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.815+0000 7fd45d00a640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd43c038720 0x7fd43c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.815+0000 7fd45d00a640 10 monclient: get_auth_request con 0x7fd43c038720 auth_method 0 2024-09-15T03:33:33.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.817+0000 7fd45d00a640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd43c038720 0x7fd43c03abd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd45811b090 tx=0x7fd448006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:33.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.819+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd454042910 con 0x7fd458070920 2024-09-15T03:33:33.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.819+0000 7fd44dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:33.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.819+0000 7fd44dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.976+0000 7fd45fa6f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T03:33:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.976+0000 7fd45fa6f640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T03:33:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.976+0000 7fd45fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:33.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:33.976+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fd4580008d0 con 0x7fd458070920 2024-09-15T03:33:34.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.621+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v121) v1 ==== 126+0+0 (secure 0 0 0) 0x7fd4540370b0 con 0x7fd458070920 2024-09-15T03:33:34.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.622+0000 7fd44dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T03:33:34.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.622+0000 7fd44dffb640 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-15T03:33:34.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.718+0000 7fd45fa6f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T03:33:34.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.718+0000 7fd45fa6f640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T03:33:34.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.718+0000 7fd45fa6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:34.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:34.718+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fd4580008d0 con 0x7fd458070920 2024-09-15T03:33:35.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.627+0000 7fd44dffb640 1 -- 172.21.15.181:0/411891488 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v122) v1 ==== 126+0+0 (secure 0 0 0) 0x7fd454040030 con 0x7fd458070920 2024-09-15T03:33:35.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.627+0000 7fd44dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-15T03:33:35.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.627+0000 7fd44dffb640 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-15T03:33:35.629 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 10 size to 3 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd43c038720 msgr2=0x7fd43c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd43c038720 0x7fd43c03abd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd45811b090 tx=0x7fd448006d80 comp rx=0 tx=0).stop 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 10 monclient: shutdown 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 msgr2=0x7fd45812fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 0x7fd45812fda0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd454009210 tx=0x7fd454032000 comp rx=0 tx=0).stop 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 shutdown_connections 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd43c038720 0x7fd43c03abd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd45809f320 0x7fd458119df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd458071260 0x7fd4581198b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 --2- 172.21.15.181:0/411891488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd458070920 0x7fd45812fda0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 >> 172.21.15.181:0/411891488 conn(0x7fd4580a2b70 msgr2=0x7fd458097eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 shutdown_connections 2024-09-15T03:33:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.632+0000 7fd45fa6f640 1 -- 172.21.15.181:0/411891488 wait complete. 2024-09-15T03:33:35.646 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T03:33:35.646 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:33:35.646 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T03:33:35.727 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.726+0000 7f397939d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.726+0000 7f397939d640 10 monclient: build_initial_monmap 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.726+0000 7f397939d640 10 monclient: monmap: 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:35.727154+0000 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:35.727154+0000 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.726+0000 7f397939d640 1 Processor -- start 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.726+0000 7f397939d640 1 -- start start 2024-09-15T03:33:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.726+0000 7f397939d640 10 monclient: init 2024-09-15T03:33:35.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:35.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f397409ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f397409f020 0x7f3974069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): picked mon.noname-a con 0x7f397409f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 0x7f3974139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): picked mon.noname-b con 0x7f3974136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3974067420 con 0x7f397409ec50 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397409fdc0 con 0x7f397409f020 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39740a0510 con 0x7f3974136d60 2024-09-15T03:33:35.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f397939d640 10 monclient(hunting): authenticate will time out at 1020.208459s 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f3972ffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f397409ec50 0x7f3974131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56928/0 (socket says 172.21.15.181:56928) 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39727fc640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f397409f020 msgr2=0x7f3974069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f3972ffd640 1 -- 172.21.15.181:0/4288717533 learned_addr learned my addr 172.21.15.181:0/4288717533 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39727fc640 1 --2- 172.21.15.181:0/4288717533 >> v2:172.21.15.153:3300/0 conn(0x7f397409f020 0x7f3974069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 1 --2- 172.21.15.181:0/4288717533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 0x7f3974139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): get_auth_request con 0x7f3974136d60 auth_method 0 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:35.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.727+0000 7f39737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 10 monclient(hunting): handle_auth_done global_id 74240 payload 274 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 1 -- 172.21.15.181:0/4288717533 >> v2:172.21.15.153:3300/0 conn(0x7f397409f020 msgr2=0x7f3974069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 1 --2- 172.21.15.181:0/4288717533 >> v2:172.21.15.153:3300/0 conn(0x7f397409f020 0x7f3974069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 1 -- 172.21.15.181:0/4288717533 >> v1:172.21.15.181:6789/0 conn(0x7f397409ec50 legacy=0x7f3974131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:35.734 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:35.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.728+0000 7f39737fe640 1 -- 172.21.15.181:0/4288717533 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740a0200 con 0x7f3974136d60 2024-09-15T03:33:35.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f39737fe640 1 -- 172.21.15.181:0/4288717533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 msgr2=0x7f3974139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:35.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f39737fe640 1 --2- 172.21.15.181:0/4288717533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 0x7f3974139140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f396c009850 tx=0x7f396c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c557a6f765ca034f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 1 -- 172.21.15.181:0/4288717533 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c032440 con 0x7f3974136d60 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: dump: 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 1 -- 172.21.15.181:0/4288717533 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c0325a0 con 0x7f3974136d60 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 5 monclient: authenticate success, global_id 74240 2024-09-15T03:33:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 4 set_mon_vals no callback set 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 1 -- 172.21.15.181:0/4288717533 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c032940 con 0x7f3974136d60 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f3971ffb640 10 monclient: dump: 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.744 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 10 monclient: shutdown 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 1 -- 172.21.15.181:0/4288717533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 msgr2=0x7f3974139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.731+0000 7f397939d640 1 --2- 172.21.15.181:0/4288717533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 0x7f3974139140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f396c009850 tx=0x7f396c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 1 -- 172.21.15.181:0/4288717533 shutdown_connections 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 1 --2- 172.21.15.181:0/4288717533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3974136d60 0x7f3974139140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 1 --2- 172.21.15.181:0/4288717533 >> v2:172.21.15.153:3300/0 conn(0x7f397409f020 0x7f3974069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 1 -- 172.21.15.181:0/4288717533 >> 172.21.15.181:0/4288717533 conn(0x7f39740a5d40 msgr2=0x7f39740a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 1 -- 172.21.15.181:0/4288717533 shutdown_connections 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 1 -- 172.21.15.181:0/4288717533 wait complete. 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 10 monclient: build_initial_monmap 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.732+0000 7f397939d640 10 monclient: monmap: 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.746 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 Processor -- start 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 -- start start 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient: init 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f397409ec50 0x7f3974097280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): picked mon.noname-a con 0x7f397409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f397409f020 0x7f39740977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): picked mon.noname-c con 0x7f397409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:35.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3974136d60 0x7f3974097d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): picked mon.noname-b con 0x7f3974136d60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397409c000 con 0x7f397409ec50 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397409d320 con 0x7f3974136d60 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397409abd0 con 0x7f397409f020 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f397939d640 10 monclient(hunting): authenticate will time out at 1020.214610s 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f3972ffd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f397409ec50 msgr2=0x7f3974097280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f3972ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f397409ec50 0x7f3974097280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:35.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3974136d60 0x7f3974097d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3974136d60 0x7f3974097d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49704/0 (socket says 172.21.15.181:49704) 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 1 -- 172.21.15.181:0/603680145 learned_addr learned my addr 172.21.15.181:0/603680145 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): get_auth_request con 0x7f3974136d60 auth_method 0 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f397409f020 0x7f39740977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): get_auth_request con 0x7f397409f020 auth_method 0 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:35.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.733+0000 7f39737fe640 10 monclient(hunting): handle_auth_done global_id 45142 payload 824 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 1 -- 172.21.15.181:0/603680145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f397409f020 msgr2=0x7f39740977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f397409f020 0x7f39740977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 1 -- 172.21.15.181:0/603680145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f397409ec50 msgr2=0x7f3974097280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f397409ec50 0x7f3974097280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:35.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.734+0000 7f39737fe640 1 -- 172.21.15.181:0/603680145 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c009390 con 0x7f3974136d60 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f39737fe640 1 -- 172.21.15.181:0/603680145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3974136d60 msgr2=0x7f3974097d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f39737fe640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3974136d60 0x7f3974097d00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f396c009360 tx=0x7f396c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c031d40 con 0x7f3974136d60 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: dump: 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.751 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.752 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:35.752 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.752 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:35.752 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:35.752 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:35.752 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: _finish_auth 0 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c03d440 con 0x7f3974136d60 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 5 monclient: authenticate success, global_id 45142 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 10 monclient: _renew_subs 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c031ea0 con 0x7f3974136d60 2024-09-15T03:33:35.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 1 -- 172.21.15.181:0/603680145 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397409b690 con 0x7f3974136d60 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f396affd640 10 monclient: dump: 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:35.754 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f39717fa640 4 set_mon_vals no callback set 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 10 monclient: _renew_subs 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.736+0000 7f397939d640 1 -- 172.21.15.181:0/603680145 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397409a9a0 con 0x7f3974136d60 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f396c039040 con 0x7f3974136d60 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f397939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f397939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f396affd640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3954038720 0x7f395403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f397939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:35.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f397939d640 1 -- 172.21.15.181:0/603680145 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39740008d0 con 0x7f3974136d60 2024-09-15T03:33:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(122..122 src has 40..122) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f396c06c1e0 con 0x7f3974136d60 2024-09-15T03:33:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f3972ffd640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3954038720 0x7f395403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.737+0000 7f3972ffd640 10 monclient: get_auth_request con 0x7f3954038720 auth_method 0 2024-09-15T03:33:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.740+0000 7f3972ffd640 1 --2- 172.21.15.181:0/603680145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3954038720 0x7f395403abd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3964004b50 tx=0x7f3964005100 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.742+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f396c037080 con 0x7f3974136d60 2024-09-15T03:33:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.742+0000 7f396affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:35.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.742+0000 7f396affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.902+0000 7f397939d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.902+0000 7f397939d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.902+0000 7f397939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.902+0000 7f397939d640 1 -- 172.21.15.181:0/603680145 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f3974064900 con 0x7f3974136d60 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.903+0000 7f396affd640 1 -- 172.21.15.181:0/603680145 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v122) v1 ==== 160+0+0 (secure 0 0 0) 0x7f396c042020 con 0x7f3974136d60 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.903+0000 7f396affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.903+0000 7f396affd640 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-15T03:33:35.904 INFO:tasks.workunit.client.0.smithi181.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-15T03:33:35.908 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:33:35.908 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-15T03:33:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.981+0000 7f0b7179d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:35.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.981+0000 7f0b7179d640 10 monclient: build_initial_monmap 2024-09-15T03:33:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.982+0000 7f0b7179d640 10 monclient: monmap: 2024-09-15T03:33:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:35.982778+0000 2024-09-15T03:33:35.986 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:35.982778+0000 2024-09-15T03:33:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:35.987 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:35.988 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:35.989 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.982+0000 7f0b7179d640 1 Processor -- start 2024-09-15T03:33:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.982+0000 7f0b7179d640 1 -- start start 2024-09-15T03:33:35.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.982+0000 7f0b7179d640 10 monclient: init 2024-09-15T03:33:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:33:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0b6c09ec50 0x7f0b6c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6c09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b6c1321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b6c09f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c067420 con 0x7f0b6c09f020 2024-09-15T03:33:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c09fdc0 con 0x7f0b6c1321a0 2024-09-15T03:33:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b6c0a0510 con 0x7f0b6c09ec50 2024-09-15T03:33:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b7179d640 10 monclient(hunting): authenticate will time out at 1020.464153s 2024-09-15T03:33:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b627fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f0b6c1321a0 0x7f0b6c068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56932/0 (socket says 172.21.15.181:56932) 2024-09-15T03:33:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b6affd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0b6c09ec50 msgr2=0x7f0b6c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b6b7fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0b6c09f020 legacy=0x7f0b6c069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:33:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b627fc640 1 -- 172.21.15.181:0/3673240674 learned_addr learned my addr 172.21.15.181:0/3673240674 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:35.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.983+0000 7f0b6affd640 1 --2- 172.21.15.181:0/3673240674 >> v2:172.21.15.153:3300/0 conn(0x7f0b6c09ec50 0x7f0b6c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b627fc640 1 -- 172.21.15.181:0/3673240674 >> v1:172.21.15.181:6789/0 conn(0x7f0b6c1321a0 legacy=0x7f0b6c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744526953 0 0) 0x7f0b6c09fdc0 con 0x7f0b6c1321a0 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): my global_id is 45145 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b580034e0 con 0x7f0b6c1321a0 2024-09-15T03:33:36.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2303621280 0 0) 0x7f0b580034e0 con 0x7f0b6c1321a0 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 >> v2:172.21.15.153:3300/0 conn(0x7f0b6c09ec50 msgr2=0x7f0b6c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 --2- 172.21.15.181:0/3673240674 >> v2:172.21.15.153:3300/0 conn(0x7f0b6c09ec50 0x7f0b6c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 >> v1:172.21.15.153:6789/0 conn(0x7f0b6c09f020 legacy=0x7f0b6c069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:33:36.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0a0200 con 0x7f0b6c1321a0 2024-09-15T03:33:36.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:36.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:36.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f0b54001720 con 0x7f0b6c1321a0 2024-09-15T03:33:36.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b7179d640 5 monclient: authenticate success, global_id 45145 2024-09-15T03:33:36.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b7179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:36.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:33:36.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: dump: 2024-09-15T03:33:36.016 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.016 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:36.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 >> v1:172.21.15.181:6789/0 conn(0x7f0b6c1321a0 legacy=0x7f0b6c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b58003a50 0x7f0b58003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): picked mon.a con 0x7f0b58003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b58004780 0x7f0b58004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): picked mon.c con 0x7f0b58004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 0x7f0b5800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): picked mon.b con 0x7f0b580099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c0a0510 con 0x7f0b58003a50 2024-09-15T03:33:36.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5800c360 con 0x7f0b580099c0 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5800c520 con 0x7f0b58004780 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b6a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.985+0000 7f0b7179d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6affd640 1 -- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b58003a50 msgr2=0x7f0b58003e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6affd640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b58003a50 0x7f0b58003e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 0x7f0b5800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): get_auth_request con 0x7f0b580099c0 auth_method 0 2024-09-15T03:33:36.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b58004780 0x7f0b58004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): get_auth_request con 0x7f0b58004780 auth_method 0 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:36.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b627fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient(hunting): handle_auth_done global_id 45145 payload 293 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 -- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b58004780 msgr2=0x7f0b58004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b58004780 0x7f0b58004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 -- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b58003a50 msgr2=0x7f0b58003e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b58003a50 0x7f0b58003e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 monclient: found mon.b 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 -- 172.21.15.181:0/3673240674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580034e0 con 0x7f0b580099c0 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:36.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.986+0000 7f0b6b7fe640 1 -- 172.21.15.181:0/3673240674 reap_dead start 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6b7fe640 1 -- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 msgr2=0x7f0b5800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6b7fe640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 0x7f0b5800bda0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009f90 tx=0x7f0b640095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21959da0c4e957a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b640050d0 con 0x7f0b580099c0 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: dump: 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.022 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.023 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.023 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.023 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.023 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.023 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.024 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.024 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6402d070 con 0x7f0b580099c0 2024-09-15T03:33:36.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:36.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:36.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 1 -- 172.21.15.181:0/3673240674 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b640054c0 con 0x7f0b580099c0 2024-09-15T03:33:36.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:36.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 4 set_mon_vals no callback set 2024-09-15T03:33:36.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:36.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b6a7fc640 10 monclient: dump: 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 10 monclient: shutdown 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 -- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 msgr2=0x7f0b5800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 0x7f0b5800bda0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009f90 tx=0x7f0b640095d0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 -- 172.21.15.181:0/3673240674 shutdown_connections 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 --2- 172.21.15.181:0/3673240674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b580099c0 0x7f0b5800bda0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 -- 172.21.15.181:0/3673240674 >> 172.21.15.181:0/3673240674 conn(0x7f0b6c0a5d40 msgr2=0x7f0b6c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 -- 172.21.15.181:0/3673240674 shutdown_connections 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.989+0000 7f0b7179d640 1 -- 172.21.15.181:0/3673240674 wait complete. 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.990+0000 7f0b7179d640 10 monclient: build_initial_monmap 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.990+0000 7f0b7179d640 10 monclient: monmap: 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:36.027 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:36.028 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:36.028 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:36.028 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:36.028 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.029 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:36.029 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:36.029 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:36.029 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.990+0000 7f0b7179d640 1 Processor -- start 2024-09-15T03:33:36.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.990+0000 7f0b7179d640 1 -- start start 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.990+0000 7f0b7179d640 10 monclient: init 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b580099c0 0x7f0b6c09a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b580099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 0x7f0b6c130d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6c1309c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b6c11b760 0x7f0b6c11bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6c11b760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c1312d0 con 0x7f0b580099c0 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c09c000 con 0x7f0b6c1309c0 2024-09-15T03:33:36.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c09e440 con 0x7f0b6c11b760 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b7179d640 10 monclient(hunting): authenticate will time out at 1020.472261s 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6affd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b580099c0 msgr2=0x7f0b6c09a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b580099c0 0x7f0b6c09a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 0x7f0b6c130d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 0x7f0b6c130d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49724/0 (socket says 172.21.15.181:49724) 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 -- 172.21.15.181:0/1478400235 learned_addr learned my addr 172.21.15.181:0/1478400235 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:36.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): get_auth_request con 0x7f0b6c1309c0 auth_method 0 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6b7fe640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b6c11b760 0x7f0b6c11bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6b7fe640 10 monclient(hunting): get_auth_request con 0x7f0b6c11b760 auth_method 0 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient(hunting): handle_auth_done global_id 45148 payload 824 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 -- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b6c11b760 msgr2=0x7f0b6c11bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b6c11b760 0x7f0b6c11bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 -- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b580099c0 msgr2=0x7f0b6c09a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b580099c0 0x7f0b6c09a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 monclient: found mon.noname-b 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.991+0000 7f0b627fc640 1 -- 172.21.15.181:0/1478400235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c141c80 con 0x7f0b6c1309c0 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b627fc640 1 -- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 msgr2=0x7f0b6c130d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:36.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b627fc640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 0x7f0b6c130d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007810 tx=0x7f0b6c068b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:36.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54018030 con 0x7f0b6c1309c0 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: dump: 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.036 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5400f900 con 0x7f0b6c1309c0 2024-09-15T03:33:36.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b7179d640 5 monclient: authenticate success, global_id 45148 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b7179d640 10 monclient: _renew_subs 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b7179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6c11db40 con 0x7f0b6c1309c0 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5400fca0 con 0x7f0b6c1309c0 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b637fe640 10 monclient: dump: 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.993+0000 7f0b69ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 10 monclient: _renew_subs 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6c11cbf0 con 0x7f0b6c1309c0 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0b5401b070 con 0x7f0b6c1309c0 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6c0008d0 con 0x7f0b6c1309c0 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.994+0000 7f0b637fe640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b4c038720 0x7f0b4c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.995+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(122..122 src has 40..122) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f0b54019070 con 0x7f0b6c1309c0 2024-09-15T03:33:36.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.995+0000 7f0b6affd640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b4c038720 0x7f0b4c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.995+0000 7f0b6affd640 10 monclient: get_auth_request con 0x7f0b4c038720 auth_method 0 2024-09-15T03:33:36.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.997+0000 7f0b6affd640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b4c038720 0x7f0b4c03abd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c006fd0 tx=0x7f0b5c006dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:36.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.999+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0b540133f0 con 0x7f0b6c1309c0 2024-09-15T03:33:36.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.999+0000 7f0b637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:36.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:35.999+0000 7f0b637fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:36.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.163+0000 7f0b7179d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T03:33:36.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.163+0000 7f0b7179d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T03:33:36.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.163+0000 7f0b7179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.163+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f0b6c141c80 con 0x7f0b6c1309c0 2024-09-15T03:33:36.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.636+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v123) v1 ==== 109+0+0 (secure 0 0 0) 0x7f0b6c141c80 con 0x7f0b6c1309c0 2024-09-15T03:33:36.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.636+0000 7f0b637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T03:33:36.638 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.636+0000 7f0b637fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:36.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.738+0000 7f0b7179d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T03:33:36.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.738+0000 7f0b7179d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T03:33:36.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.738+0000 7f0b7179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.738+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f0b6c141c80 con 0x7f0b6c1309c0 2024-09-15T03:33:36.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.739+0000 7f0b637fe640 1 -- 172.21.15.181:0/1478400235 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v123) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0b6c141c80 con 0x7f0b6c1309c0 2024-09-15T03:33:36.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.739+0000 7f0b637fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-15T03:33:36.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.739+0000 7f0b637fe640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-15T03:33:36.741 INFO:tasks.workunit.client.0.smithi181.stderr:weight-set for pool 'fooset' already created 2024-09-15T03:33:36.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b4c038720 msgr2=0x7f0b4c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b4c038720 0x7f0b4c03abd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c006fd0 tx=0x7f0b5c006dd0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 10 monclient: shutdown 2024-09-15T03:33:36.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:36.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 msgr2=0x7f0b6c130d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 0x7f0b6c130d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007810 tx=0x7f0b6c068b50 comp rx=0 tx=0).stop 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 shutdown_connections 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b4c038720 0x7f0b4c03abd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b6c11b760 0x7f0b6c11bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b6c1309c0 0x7f0b6c130d90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 --2- 172.21.15.181:0/1478400235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b580099c0 0x7f0b6c09a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 >> 172.21.15.181:0/1478400235 conn(0x7f0b6c0a5d40 msgr2=0x7f0b6c069490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 shutdown_connections 2024-09-15T03:33:36.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.741+0000 7f0b7179d640 1 -- 172.21.15.181:0/1478400235 wait complete. 2024-09-15T03:33:36.751 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create barset positional 2024-09-15T03:33:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.825+0000 7f964660f640 10 monclient: get_monmap_and_config 2024-09-15T03:33:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.825+0000 7f964660f640 10 monclient: build_initial_monmap 2024-09-15T03:33:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.825+0000 7f964660f640 10 monclient: monmap: 2024-09-15T03:33:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:36.826654+0000 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:36.826654+0000 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:36.828 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.826+0000 7f964660f640 1 Processor -- start 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.826+0000 7f964660f640 1 -- start start 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.826+0000 7f964660f640 10 monclient: init 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f964009f320 0x7f9640132170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): picked mon.noname-b con 0x7f964009f320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9640070920 0x7f9640070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): picked mon.noname-a con 0x7f9640070920 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9640071230 0x7f964012e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): picked mon.noname-c con 0x7f9640071230 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964009e4d0 con 0x7f9640070920 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964009e1f0 con 0x7f9640071230 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964009e7b0 con 0x7f964009f320 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f964660f640 10 monclient(hunting): authenticate will time out at 1021.308123s 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963f7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9640070920 msgr2=0x7f9640070cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9640070920 0x7f9640070cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:36.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9640071230 0x7f964012e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9640071230 0x7f964012e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49734/0 (socket says 172.21.15.181:49734) 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 -- 172.21.15.181:0/574173228 learned_addr learned my addr 172.21.15.181:0/574173228 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): get_auth_request con 0x7f9640071230 auth_method 0 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:36.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 1 --2- 172.21.15.181:0/574173228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f964009f320 0x7f9640132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): get_auth_request con 0x7f964009f320 auth_method 0 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f963ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient(hunting): handle_auth_done global_id 45151 payload 274 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 -- 172.21.15.181:0/574173228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f964009f320 msgr2=0x7f9640132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 --2- 172.21.15.181:0/574173228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f964009f320 0x7f9640132170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 -- 172.21.15.181:0/574173228 >> v2:172.21.15.153:3300/0 conn(0x7f9640070920 msgr2=0x7f9640070cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 --2- 172.21.15.181:0/574173228 >> v2:172.21.15.153:3300/0 conn(0x7f9640070920 0x7f9640070cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient: _finish_hunting 0 2024-09-15T03:33:36.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 monclient: found mon.noname-c 2024-09-15T03:33:36.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:36.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.827+0000 7f9644bac640 1 -- 172.21.15.181:0/574173228 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640071690 con 0x7f9640071230 2024-09-15T03:33:36.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f9644bac640 1 -- 172.21.15.181:0/574173228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9640071230 msgr2=0x7f964012e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f9644bac640 1 --2- 172.21.15.181:0/574173228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9640071230 0x7f964012e740 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9640064db0 tx=0x7f9638009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d7a2d6eca92b80a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 1 -- 172.21.15.181:0/574173228 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638007610 con 0x7f9640071230 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: dump: 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.844 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: _finish_auth 0 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 1 -- 172.21.15.181:0/574173228 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963802d050 con 0x7f9640071230 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:36.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 1 -- 172.21.15.181:0/574173228 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380079e0 con 0x7f9640071230 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f964660f640 5 monclient: authenticate success, global_id 45151 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f964660f640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f964660f640 4 set_mon_vals no callback set 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.830+0000 7f963effd640 10 monclient: dump: 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.846 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 10 monclient: shutdown 2024-09-15T03:33:36.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 -- 172.21.15.181:0/574173228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9640071230 msgr2=0x7f964012e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 --2- 172.21.15.181:0/574173228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9640071230 0x7f964012e740 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9640064db0 tx=0x7f9638009840 comp rx=0 tx=0).stop 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 -- 172.21.15.181:0/574173228 shutdown_connections 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 --2- 172.21.15.181:0/574173228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9640071230 0x7f964012e740 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 --2- 172.21.15.181:0/574173228 >> v2:172.21.15.153:3300/0 conn(0x7f9640070920 0x7f9640070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 --2- 172.21.15.181:0/574173228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f964009f320 0x7f9640132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 -- 172.21.15.181:0/574173228 >> 172.21.15.181:0/574173228 conn(0x7f96400a2b70 msgr2=0x7f964009ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 -- 172.21.15.181:0/574173228 shutdown_connections 2024-09-15T03:33:36.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 1 -- 172.21.15.181:0/574173228 wait complete. 2024-09-15T03:33:36.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 10 monclient: build_initial_monmap 2024-09-15T03:33:36.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.831+0000 7f964660f640 10 monclient: monmap: 2024-09-15T03:33:36.849 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:36.849 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:36.850 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:36.850 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:36.850 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:36.850 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.850 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:36.850 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 Processor -- start 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 -- start start 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient: init 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f9640140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): picked mon.noname-a con 0x7f9640070920 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f964012eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): picked mon.noname-c con 0x7f9640071230 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:36.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): picked mon.noname-b con 0x7f964009f320 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:36.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9640141a80 con 0x7f9640070920 2024-09-15T03:33:36.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964011ec80 con 0x7f964009f320 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96400976a0 con 0x7f9640071230 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f964660f640 10 monclient(hunting): authenticate will time out at 1021.313841s 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f963ffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 msgr2=0x7f9640140ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:33:36.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49736/0 (socket says 172.21.15.181:49736) 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f963ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f9640140ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 learned_addr learned my addr 172.21.15.181:0/869881945 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 10 monclient(hunting): get_auth_request con 0x7f964009f320 auth_method 0 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.832+0000 7f9644bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:36.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_done global_id 45154 payload 824 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 msgr2=0x7f964012eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f963f7fe640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f964012eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f964012eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 msgr2=0x7f9640140ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f9640140ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 10 monclient: _finish_hunting 0 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 1 monclient: found mon.noname-b 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638009ae0 con 0x7f964009f320 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f963f7fe640 10 monclient: get_auth_request con 0x7f9640071230 auth_method 0 2024-09-15T03:33:36.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.833+0000 7f963f7fe640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f964012eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 msgr2=0x7f964011b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9638009f50 tx=0x7f9638036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638007170 con 0x7f964009f320 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: dump: 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.856 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: _finish_auth 0 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9638032c60 con 0x7f964009f320 2024-09-15T03:33:36.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:36.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963803b410 con 0x7f964009f320 2024-09-15T03:33:36.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:36.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:36.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f9627fff640 10 monclient: dump: 2024-09-15T03:33:36.858 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:36.858 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:36.859 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 5 monclient: authenticate success, global_id 45154 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f963e7fc640 4 set_mon_vals no callback set 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 10 monclient: _renew_subs 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964011ef60 con 0x7f964009f320 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 10 monclient: _renew_subs 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.835+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964011df30 con 0x7f964009f320 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9638038050 con 0x7f964009f320 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f964660f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f964660f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f964660f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96400008d0 con 0x7f964009f320 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f9627fff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f962c0386f0 0x7f962c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(123..123 src has 40..123) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f963806d1a0 con 0x7f964009f320 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f963ffff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f962c0386f0 0x7f962c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:36.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.836+0000 7f963ffff640 10 monclient: get_auth_request con 0x7f962c0386f0 auth_method 0 2024-09-15T03:33:36.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.839+0000 7f963ffff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f962c0386f0 0x7f962c03aba0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9630009430 tx=0x7f96300090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:36.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.841+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f96380435e0 con 0x7f964009f320 2024-09-15T03:33:36.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.841+0000 7f9627fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:36.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:36.841+0000 7f9627fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.005+0000 7f964660f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.005+0000 7f964660f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.005+0000 7f964660f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:37.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.005+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f964009cac0 con 0x7f964009f320 2024-09-15T03:33:37.011 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:33:37.012 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:33:37.012 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:33:37.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:33:37.016 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:33:37.016 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:33:37.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:37.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.119+0000 7fd050b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:37.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.119+0000 7fd050b9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.119+0000 7fd050b9d640 10 monclient: monmap: 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:37.120530+0000 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:37.120530+0000 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:37.121 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:37.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.120+0000 7fd050b9d640 1 Processor -- start 2024-09-15T03:33:37.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.120+0000 7fd050b9d640 1 -- start start 2024-09-15T03:33:37.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.120+0000 7fd050b9d640 10 monclient: init 2024-09-15T03:33:37.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd04c09e950 0x7fd04c09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd04c09e950 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd04c09ed20 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd04c068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c09fb60 con 0x7fd04c068d90 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c09d4f0 con 0x7fd04c09ed20 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c0a1180 con 0x7fd04c09e950 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd050b9d640 10 monclient(hunting): authenticate will time out at 1014.760175s 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42484/0 (socket says 172.21.15.153:42484) 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 -- 172.21.15.153:0/3157881500 learned_addr learned my addr 172.21.15.153:0/3157881500 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 10 monclient(hunting): get_auth_request con 0x7fd04c09ed20 auth_method 0 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 1 --2- 172.21.15.153:0/3157881500 >> v2:172.21.15.181:3300/0 conn(0x7fd04c09e950 0x7fd04c09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): get_auth_request con 0x7fd04c09e950 auth_method 0 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 -- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 msgr2=0x7fd04c09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:33:37.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 -- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 msgr2=0x7fd04c09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 --2- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd049d9b640 1 --2- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient(hunting): handle_auth_done global_id 45157 payload 274 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 1 -- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 msgr2=0x7fd04c09f120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 1 --2- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 1 -- 172.21.15.153:0/3157881500 >> v1:172.21.15.181:6789/0 conn(0x7fd04c068d90 legacy=0x7fd04c069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 1 monclient: found mon.noname-c 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:37.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:37.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.121+0000 7fd04a59c640 1 -- 172.21.15.153:0/3157881500 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c09ff60 con 0x7fd04c09e950 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.124+0000 7fd04a59c640 1 -- 172.21.15.153:0/3157881500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09e950 msgr2=0x7fd04c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.124+0000 7fd04a59c640 1 --2- 172.21.15.153:0/3157881500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09e950 0x7fd04c09cef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd04c076760 tx=0x7fd03c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af217bdbc211e0c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 1 -- 172.21.15.153:0/3157881500 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c02ae00 con 0x7fd04c09e950 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: dump: 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:37.135 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: _finish_auth 0 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 1 -- 172.21.15.153:0/3157881500 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c032440 con 0x7fd04c09e950 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 5 monclient: authenticate success, global_id 45157 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:37.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:37.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:37.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 1 -- 172.21.15.153:0/3157881500 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0327e0 con 0x7fd04c09e950 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 4 set_mon_vals no callback set 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd04959a640 10 monclient: dump: 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:37.138 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 10 monclient: shutdown 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 -- 172.21.15.153:0/3157881500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09e950 msgr2=0x7fd04c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 --2- 172.21.15.153:0/3157881500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09e950 0x7fd04c09cef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd04c076760 tx=0x7fd03c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 -- 172.21.15.153:0/3157881500 shutdown_connections 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 --2- 172.21.15.153:0/3157881500 >> v2:172.21.15.153:3300/0 conn(0x7fd04c09ed20 0x7fd04c09f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 --2- 172.21.15.153:0/3157881500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09e950 0x7fd04c09cef0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 -- 172.21.15.153:0/3157881500 >> 172.21.15.153:0/3157881500 conn(0x7fd04c067b90 msgr2=0x7fd04c0a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 -- 172.21.15.153:0/3157881500 shutdown_connections 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.125+0000 7fd050b9d640 1 -- 172.21.15.153:0/3157881500 wait complete. 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.126+0000 7fd050b9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:37.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.126+0000 7fd050b9d640 10 monclient: monmap: 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.126+0000 7fd050b9d640 1 Processor -- start 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.126+0000 7fd050b9d640 1 -- start start 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.126+0000 7fd050b9d640 10 monclient: init 2024-09-15T03:33:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd04c068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 0x7fd04c133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd04c09e950 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09ed20 0x7fd04c11d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd04c09ed20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c141dd0 con 0x7fd04c068d90 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c11efe0 con 0x7fd04c09ed20 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:37.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c120d80 con 0x7fd04c09e950 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd050b9d640 10 monclient(hunting): authenticate will time out at 1014.766219s 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 0x7fd04c133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42486/0 (socket says 172.21.15.153:42486) 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 -- 172.21.15.153:0/4138533824 learned_addr learned my addr 172.21.15.153:0/4138533824 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): get_auth_request con 0x7fd04c09e950 auth_method 0 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 10 monclient(hunting): get_auth_request con 0x7fd04c068d90 auth_method 0 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:37.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09ed20 0x7fd04c11d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): get_auth_request con 0x7fd04c09ed20 auth_method 0 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 msgr2=0x7fd04c135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 msgr2=0x7fd04c135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04a59c640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd04ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient(hunting): handle_auth_done global_id 74252 payload 824 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09ed20 msgr2=0x7fd04c11d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09ed20 0x7fd04c11d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 msgr2=0x7fd04c135e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 monclient: found mon.noname-c 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.127+0000 7fd049d9b640 1 -- 172.21.15.153:0/4138533824 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c009390 con 0x7fd04c09e950 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.129+0000 7fd049d9b640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 msgr2=0x7fd04c133980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:37.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.129+0000 7fd049d9b640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 0x7fd04c133980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd040006f60 tx=0x7fd040008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000c6d0 con 0x7fd04c09e950 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: dump: 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:37.145 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04000c830 con 0x7fd04c09e950 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 5 monclient: authenticate success, global_id 74252 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 10 monclient: _renew_subs 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c120aa0 con 0x7fd04c09e950 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000cbf0 con 0x7fd04c09e950 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd03a7fc640 10 monclient: dump: 2024-09-15T03:33:37.146 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd048d99640 4 set_mon_vals no callback set 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 10 monclient: _renew_subs 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.130+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c133ec0 con 0x7fd04c09e950 2024-09-15T03:33:37.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd040013620 con 0x7fd04c09e950 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd050b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd050b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd050b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c04bf60 con 0x7fd04c09e950 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd03a7fc640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd02c038740 0x7fd02c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(123..123 src has 40..123) v4 ==== 7545+0+0 (secure 0 0 0) 0x7fd04004c610 con 0x7fd04c09e950 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd04a59c640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd02c038740 0x7fd02c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.131+0000 7fd04a59c640 10 monclient: get_auth_request con 0x7fd02c038740 auth_method 0 2024-09-15T03:33:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.134+0000 7fd04a59c640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd02c038740 0x7fd02c03abf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd03c035040 tx=0x7fd03c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.135+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd0400138d0 con 0x7fd04c09e950 2024-09-15T03:33:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.135+0000 7fd03a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:37.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.135+0000 7fd03a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:37.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd050b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:37.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd050b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:37.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd050b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:37.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd04c0638d0 con 0x7fd04c09e950 2024-09-15T03:33:37.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd03a7fc640 1 -- 172.21.15.153:0/4138533824 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fd040017610 con 0x7fd04c09e950 2024-09-15T03:33:37.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd03a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:37.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.351+0000 7fd03a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:37.353 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":50,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd02c038740 msgr2=0x7fd02c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd02c038740 0x7fd02c03abf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd03c035040 tx=0x7fd03c032000 comp rx=0 tx=0).stop 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 10 monclient: shutdown 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 msgr2=0x7fd04c133980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 0x7fd04c133980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd040006f60 tx=0x7fd040008040 comp rx=0 tx=0).stop 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 shutdown_connections 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd02c038740 0x7fd02c03abf0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd04c09ed20 0x7fd04c11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd04c09e950 0x7fd04c133980 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 --2- 172.21.15.153:0/4138533824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd04c068d90 0x7fd04c135e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:37.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 >> 172.21.15.153:0/4138533824 conn(0x7fd04c067b90 msgr2=0x7fd04c134c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:37.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 shutdown_connections 2024-09-15T03:33:37.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:37.353+0000 7fd050b9d640 1 -- 172.21.15.153:0/4138533824 wait complete. 2024-09-15T03:33:37.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.642+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v124) v1 ==== 115+0+0 (secure 0 0 0) 0x7f963803f020 con 0x7f964009f320 2024-09-15T03:33:37.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.642+0000 7f9627fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:37.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.642+0000 7f9627fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:37.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.744+0000 7f964660f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:37.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.744+0000 7f964660f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:37.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.744+0000 7f964660f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:37.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:37.744+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f9640064b00 con 0x7f964009f320 2024-09-15T03:33:39.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.832+0000 7f9626ffd640 10 monclient: tick 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.832+0000 7f9626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.832+0000 7f9626ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 msgr2=0x7f964011b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 msgr2=0x7f964011b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9638009f50 tx=0x7f9638036000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9638009f50 tx=0x7f9638036000 comp rx=0 tx=0).stop 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 msgr2=0x7f964011b5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f964009f320 0x7f964011b5e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f962c043a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): picked mon.a con 0x7f9640070920 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 0x7f962c046570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): picked mon.b con 0x7f962c0440f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f962c048fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): picked mon.c con 0x7f9640071230 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c049940 con 0x7f9640070920 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c049a80 con 0x7f962c0440f0 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:39.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c049bc0 con 0x7f9640071230 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9627fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 0x7f962c046570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): get_auth_request con 0x7f962c0440f0 auth_method 0 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963f7fe640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f962c043a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963ffff640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f962c048fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963ffff640 10 monclient(hunting): get_auth_request con 0x7f9640071230 auth_method 0 2024-09-15T03:33:39.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient(hunting): handle_auth_done global_id 45154 payload 293 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 msgr2=0x7f962c048fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f962c048fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 msgr2=0x7f962c043a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f962c043a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient: _finish_hunting 0 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 monclient: found mon.b 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f962c049d30 con 0x7f962c0440f0 2024-09-15T03:33:39.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f9640064b00 con 0x7f962c0440f0 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 10 monclient: _finish_auth 0 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f9644bac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963f7fe640 10 monclient: get_auth_request con 0x7f9640070920 auth_method 0 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.833+0000 7f963f7fe640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f962c043a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9644bac640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 msgr2=0x7f962c046570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9644bac640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 0x7f962c046570 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f963806b010 tx=0x7f9638042440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380424b0 con 0x7f962c0440f0 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9627fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9627fff640 10 monclient: dump: 2024-09-15T03:33:39.842 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96380426e0 con 0x7f962c0440f0 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f9627fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.836+0000 7f963dffb640 4 set_mon_vals no callback set 2024-09-15T03:33:39.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:39.837+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f963806f100 con 0x7f962c0440f0 2024-09-15T03:33:40.366 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.470+0000 7f1542a68640 10 monclient: get_monmap_and_config 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.470+0000 7f1542a68640 10 monclient: build_initial_monmap 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.470+0000 7f1542a68640 10 monclient: monmap: 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:40.470801+0000 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:40.470801+0000 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:40.472 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.470+0000 7f1542a68640 1 Processor -- start 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.470+0000 7f1542a68640 1 -- start start 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.470+0000 7f1542a68640 10 monclient: init 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f153c131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f153c09e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient(hunting): picked mon.noname-b con 0x7f153c137bc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c09faf0 con 0x7f153c131ab0 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c09fd30 con 0x7f153c09e910 2024-09-15T03:33:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c132590 con 0x7f153c137bc0 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1542a68640 10 monclient(hunting): authenticate will time out at 1018.110565s 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38284/0 (socket says 172.21.15.153:38284) 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 learned_addr learned my addr 172.21.15.153:0/2740971811 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 10 monclient(hunting): get_auth_request con 0x7f153c137bc0 auth_method 0 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:40.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.471+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:40.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.473+0000 7f1540804640 1 -- 172.21.15.153:0/2740971811 >> v1:172.21.15.153:6789/0 conn(0x7f153c131ab0 legacy=0x7f153c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:40.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.474+0000 7f153bfff640 1 -- 172.21.15.153:0/2740971811 >> v1:172.21.15.181:6789/0 conn(0x7f153c09e910 legacy=0x7f153c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:40.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 10 monclient(hunting): get_auth_request con 0x7f153c137bc0 auth_method 0 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:40.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:40.672+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:40.833+0000 7f9626ffd640 10 monclient: tick 2024-09-15T03:33:40.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:40.833+0000 7f9626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:41.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 10 monclient(hunting): get_auth_request con 0x7f153c137bc0 auth_method 0 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:41.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.073+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:41.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:41.833+0000 7f9626ffd640 10 monclient: tick 2024-09-15T03:33:41.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:41.833+0000 7f9626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 10 monclient(hunting): get_auth_request con 0x7f153c137bc0 auth_method 0 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:41.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:41.874+0000 7f1541005640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:42.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:42.833+0000 7f9626ffd640 10 monclient: tick 2024-09-15T03:33:42.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:42.833+0000 7f9626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:43.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): tick 2024-09-15T03:33:43.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 msgr2=0x7f153c139fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c137bc0 0x7f153c139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 -- 172.21.15.153:0/2740971811 >> v1:172.21.15.181:6789/0 conn(0x7f153c09e910 legacy=0x7f153c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 -- 172.21.15.153:0/2740971811 >> v1:172.21.15.153:6789/0 conn(0x7f153c131ab0 legacy=0x7f153c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient: _add_conns ranks=[4,3] 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.181:3300/0 conn(0x7f151c0037d0 0x7f151c005c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f151c0037d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 0x7f151c008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f151c006270 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 -- 172.21.15.153:0/2740971811 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c132590 con 0x7f151c006270 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 1 -- 172.21.15.153:0/2740971811 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c008de0 con 0x7f151c0037d0 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.471+0000 7f153bfff640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 0x7f151c008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 10 monclient(hunting): get_auth_request con 0x7f151c006270 auth_method 0 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 1 -- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 msgr2=0x7f151c008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 1 -- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 msgr2=0x7f151c008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 0x7f151c008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:43.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f153bfff640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 0x7f151c008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.181:3300/0 conn(0x7f151c0037d0 0x7f151c005c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): get_auth_request con 0x7f151c0037d0 auth_method 0 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient(hunting): handle_auth_done global_id 45163 payload 274 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 1 -- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 msgr2=0x7f151c008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 1 --2- 172.21.15.153:0/2740971811 >> v2:172.21.15.153:3300/0 conn(0x7f151c006270 0x7f151c008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient: _finish_hunting 0 2024-09-15T03:33:43.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 1 monclient: found mon.noname-c 2024-09-15T03:33:43.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:43.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.472+0000 7f1540804640 1 -- 172.21.15.153:0/2740971811 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0a1110 con 0x7f151c0037d0 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f1540804640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 msgr2=0x7f151c005c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f1540804640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f151c005c20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f151c003740 tx=0x7f152c007cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62b810fdb9e84f48 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 1 -- 172.21.15.153:0/2740971811 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c02ac90 con 0x7f151c0037d0 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 10 monclient: dump: 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 1 -- 172.21.15.153:0/2740971811 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c02adf0 con 0x7f151c0037d0 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f1542a68640 5 monclient: authenticate success, global_id 45163 2024-09-15T03:33:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f1542a68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f1542a68640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.475+0000 7f153b7fe640 1 -- 172.21.15.153:0/2740971811 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c007320 con 0x7f151c0037d0 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 4 set_mon_vals no callback set 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f153b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f153b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f153b7fe640 10 monclient: dump: 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 10 monclient: shutdown 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 -- 172.21.15.153:0/2740971811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 msgr2=0x7f151c005c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f151c005c20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f151c003740 tx=0x7f152c007cf0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1541005640 1 -- 172.21.15.153:0/2740971811 reap_dead start 2024-09-15T03:33:43.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 -- 172.21.15.153:0/2740971811 shutdown_connections 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 --2- 172.21.15.153:0/2740971811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f151c005c20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 -- 172.21.15.153:0/2740971811 >> 172.21.15.153:0/2740971811 conn(0x7f153c067d30 msgr2=0x7f153c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 -- 172.21.15.153:0/2740971811 shutdown_connections 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.476+0000 7f1542a68640 1 -- 172.21.15.153:0/2740971811 wait complete. 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.477+0000 7f1542a68640 10 monclient: build_initial_monmap 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.477+0000 7f1542a68640 10 monclient: monmap: 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.477+0000 7f1542a68640 1 Processor -- start 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 -- start start 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient: init 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f153c137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): picked mon.noname-b con 0x7f151c0037d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): picked mon.noname-a con 0x7f153c098a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 0x7f153c0973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): picked mon.noname-c con 0x7f153c096fa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c097910 con 0x7f153c098a30 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c09bde0 con 0x7f151c0037d0 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c09dbb0 con 0x7f153c096fa0 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.478+0000 7f1542a68640 10 monclient(hunting): authenticate will time out at 1021.117944s 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 0x7f153c0973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42496/0 (socket says 172.21.15.153:42496) 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 -- 172.21.15.153:0/804226184 learned_addr learned my addr 172.21.15.153:0/804226184 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:43.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 10 monclient(hunting): get_auth_request con 0x7f153c098a30 auth_method 0 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 10 monclient(hunting): get_auth_request con 0x7f153c096fa0 auth_method 0 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f153c137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): get_auth_request con 0x7f151c0037d0 auth_method 0 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 msgr2=0x7f153c0973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 msgr2=0x7f153c0973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 msgr2=0x7f153c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 msgr2=0x7f153c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 0x7f153c0973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1541005640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 0x7f153c0973d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f153bfff640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient(hunting): handle_auth_done global_id 45166 payload 824 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 msgr2=0x7f153c0973d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 0x7f153c0973d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 msgr2=0x7f153c096a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:43.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient: _finish_hunting 0 2024-09-15T03:33:43.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 monclient: found mon.noname-b 2024-09-15T03:33:43.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:43.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.479+0000 7f1540804640 1 -- 172.21.15.153:0/804226184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c09faf0 con 0x7f151c0037d0 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1540804640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 msgr2=0x7f153c137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1540804640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f153c137110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f151c008f20 tx=0x7f152c030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c005e40 con 0x7f151c0037d0 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: dump: 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.493 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c001240 con 0x7f151c0037d0 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 5 monclient: authenticate success, global_id 45166 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 10 monclient: _renew_subs 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c099ee0 con 0x7f151c0037d0 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:43.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c0338b0 con 0x7f151c0037d0 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1538ff9640 10 monclient: dump: 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:43.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f153affd640 4 set_mon_vals no callback set 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 10 monclient: _renew_subs 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.482+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c099920 con 0x7f151c0037d0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1542a68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1542a68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1542a68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f152c030070 con 0x7f151c0037d0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c04bf60 con 0x7f151c0037d0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1538ff9640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f151c039530 0x7f151c03b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(124..124 src has 40..124) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f152c06c200 con 0x7f151c0037d0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f153bfff640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f151c039530 0x7f151c03b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.484+0000 7f153bfff640 10 monclient: get_auth_request con 0x7f151c039530 auth_method 0 2024-09-15T03:33:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.487+0000 7f153bfff640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f151c039530 0x7f151c03b9e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1524007ca0 tx=0x7f1524009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.489+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f152c030920 con 0x7f151c0037d0 2024-09-15T03:33:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.489+0000 7f1538ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.489+0000 7f1538ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:43.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.550+0000 7f9627fff640 1 -- 172.21.15.181:0/869881945 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v124) v1 ==== 159+0+0 (secure 0 0 0) 0x7f9638042850 con 0x7f962c0440f0 2024-09-15T03:33:43.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.550+0000 7f9627fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-15T03:33:43.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.550+0000 7f9627fff640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-15T03:33:43.551 INFO:tasks.workunit.client.0.smithi181.stderr:weight-set for pool 'barset' already created 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.552+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f962c0386f0 msgr2=0x7f962c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.552+0000 7f964660f640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f962c0386f0 0x7f962c03aba0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9630009430 tx=0x7f96300090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.552+0000 7f964660f640 10 monclient: shutdown 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.552+0000 7f964660f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.552+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 msgr2=0x7f962c046570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.552+0000 7f964660f640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 0x7f962c046570 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f963806b010 tx=0x7f9638042440 comp rx=0 tx=0).stop 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 shutdown_connections 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9640071230 0x7f962c048fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f962c0440f0 0x7f962c046570 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9640070920 0x7f962c043a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 --2- 172.21.15.181:0/869881945 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f962c0386f0 0x7f962c03aba0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 >> 172.21.15.181:0/869881945 conn(0x7f96400a2b70 msgr2=0x7f964009fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:43.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 shutdown_connections 2024-09-15T03:33:43.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.553+0000 7f964660f640 1 -- 172.21.15.181:0/869881945 wait complete. 2024-09-15T03:33:43.565 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:33:43.565 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep fooset 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.644+0000 7f7fbaf9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.644+0000 7f7fbaf9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.644+0000 7f7fbaf9d640 10 monclient: monmap: 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:43.644997+0000 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:43.644997+0000 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:43.646 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:43.646 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:43.646 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:43.646 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:43.646 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:43.646 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:43.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.644+0000 7f7fbaf9d640 1 Processor -- start 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 1 -- start start 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient: init 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7fb4069bd0 0x7f7fb4132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb4069bd0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7fb4098330 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7fb4136e30 0x7f7fb4139210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb4136e30 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb409d140 con 0x7f7fb4098330 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb409e2b0 con 0x7f7fb4136e30 2024-09-15T03:33:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb4098e40 con 0x7f7fb4069bd0 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fbaf9d640 10 monclient(hunting): authenticate will time out at 1028.126503s 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7fb4069bd0 0x7f7fb4132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7fb4069bd0 0x7f7fb4132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:41704/0 (socket says 172.21.15.181:41704) 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 1 -- 172.21.15.181:0/774203748 learned_addr learned my addr 172.21.15.181:0/774203748 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 10 monclient(hunting): get_auth_request con 0x7f7fb4069bd0 auth_method 0 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb8d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 1 --2- 172.21.15.181:0/774203748 >> v2:172.21.15.153:3300/0 conn(0x7f7fb4136e30 0x7f7fb4139210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): get_auth_request con 0x7f7fb4136e30 auth_method 0 2024-09-15T03:33:43.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient(hunting): handle_auth_done global_id 84099 payload 274 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 1 -- 172.21.15.181:0/774203748 >> v2:172.21.15.181:3300/0 conn(0x7f7fb4069bd0 msgr2=0x7f7fb4132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 1 --2- 172.21.15.181:0/774203748 >> v2:172.21.15.181:3300/0 conn(0x7f7fb4069bd0 0x7f7fb4132140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 1 -- 172.21.15.181:0/774203748 >> v1:172.21.15.181:6789/0 conn(0x7f7fb4098330 legacy=0x7f7fb4098730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient: _finish_hunting 0 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 1 monclient: found mon.noname-a 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:43.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:43.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.645+0000 7f7fb953a640 1 -- 172.21.15.181:0/774203748 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb409cb80 con 0x7f7fb4136e30 2024-09-15T03:33:43.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.648+0000 7f7fb953a640 1 -- 172.21.15.181:0/774203748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4136e30 msgr2=0x7f7fb4139210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:43.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.648+0000 7f7fb953a640 1 --2- 172.21.15.181:0/774203748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4136e30 0x7f7fb4139210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7fb409f9d0 tx=0x7f7fa40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46421573a0159c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:43.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 1 -- 172.21.15.181:0/774203748 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa402ae00 con 0x7f7fb4136e30 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: dump: 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.660 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 1 -- 172.21.15.181:0/774203748 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4032440 con 0x7f7fb4136e30 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 1 -- 172.21.15.181:0/774203748 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40327e0 con 0x7f7fb4136e30 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 5 monclient: authenticate success, global_id 84099 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 4 set_mon_vals no callback set 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fb37fe640 10 monclient: dump: 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 10 monclient: shutdown 2024-09-15T03:33:43.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/774203748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4136e30 msgr2=0x7f7fb4139210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/774203748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4136e30 0x7f7fb4139210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7fb409f9d0 tx=0x7f7fa40090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/774203748 shutdown_connections 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/774203748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4136e30 0x7f7fb4139210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/774203748 >> v2:172.21.15.181:3300/0 conn(0x7f7fb4069bd0 0x7f7fb4132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/774203748 >> 172.21.15.181:0/774203748 conn(0x7f7fb409e9f0 msgr2=0x7f7fb409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/774203748 shutdown_connections 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.649+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/774203748 wait complete. 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.650+0000 7f7fbaf9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.650+0000 7f7fbaf9d640 10 monclient: monmap: 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:43.663 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.650+0000 7f7fbaf9d640 1 Processor -- start 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.650+0000 7f7fbaf9d640 1 -- start start 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.650+0000 7f7fbaf9d640 10 monclient: init 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 0x7f7fb4118560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb4069bd0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4098330 0x7f7fb4118aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb4098330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 0x7f7fb4118fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb4136e30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb411a170 con 0x7f7fb4098330 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb409fea0 con 0x7f7fb4069bd0 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4119bf0 con 0x7f7fb4136e30 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fbaf9d640 10 monclient(hunting): authenticate will time out at 1028.132090s 2024-09-15T03:33:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 0x7f7fb4118560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 0x7f7fb4118560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:41706/0 (socket says 172.21.15.181:41706) 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 -- 172.21.15.181:0/253537255 learned_addr learned my addr 172.21.15.181:0/253537255 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): get_auth_request con 0x7f7fb4069bd0 auth_method 0 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 0x7f7fb4118fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4098330 0x7f7fb4118aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): get_auth_request con 0x7f7fb4098330 auth_method 0 2024-09-15T03:33:43.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 10 monclient(hunting): get_auth_request con 0x7f7fb4136e30 auth_method 0 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 msgr2=0x7f7fb4118fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 msgr2=0x7f7fb4118fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 0x7f7fb4118fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:43.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb953a640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 0x7f7fb4118fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient(hunting): handle_auth_done global_id 45172 payload 824 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 msgr2=0x7f7fb4118fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 0x7f7fb4118fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4098330 msgr2=0x7f7fb4118aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4098330 0x7f7fb4118aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient: _finish_hunting 0 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 monclient: found mon.noname-b 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.651+0000 7f7fb8d39640 1 -- 172.21.15.181:0/253537255 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4009390 con 0x7f7fb4069bd0 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb8d39640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 msgr2=0x7f7fb4118560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb8d39640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 0x7f7fb4118560 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7fac007e80 tx=0x7f7fac009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:43.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac009e90 con 0x7f7fb4069bd0 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: dump: 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 5 monclient: authenticate success, global_id 45172 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac00f040 con 0x7f7fb4069bd0 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac015630 con 0x7f7fb4069bd0 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 10 monclient: _renew_subs 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb409d670 con 0x7f7fb4069bd0 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb0ff9640 10 monclient: dump: 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fb2ffd640 4 set_mon_vals no callback set 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 10 monclient: _renew_subs 2024-09-15T03:33:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.653+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb411b7a0 con 0x7f7fb4069bd0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7fac0159e0 con 0x7f7fb4069bd0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fb0ff9640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f94038740 0x7f7f9403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fbaf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fbaf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fbaf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(124..124 src has 40..124) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f7fac04cf00 con 0x7f7fb4069bd0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.654+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb4130890 con 0x7f7fb4069bd0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.655+0000 7f7fb3fff640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f94038740 0x7f7f9403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.655+0000 7f7fb3fff640 10 monclient: get_auth_request con 0x7f7f94038740 auth_method 0 2024-09-15T03:33:43.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.657+0000 7f7fb3fff640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f94038740 0x7f7f9403abf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4136bf0 tx=0x7f7f9c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:43.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.659+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7fac01d360 con 0x7f7fb4069bd0 2024-09-15T03:33:43.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.659+0000 7f7fb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:43.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.659+0000 7f7fb0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.713+0000 7f1542a68640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.713+0000 7f1542a68640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.713+0000 7f1542a68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.713+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f153c066280 con 0x7f151c0037d0 2024-09-15T03:33:43.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.714+0000 7f1538ff9640 1 -- 172.21.15.153:0/804226184 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f152c0323d0 con 0x7f151c0037d0 2024-09-15T03:33:43.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.714+0000 7f1538ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:43.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.714+0000 7f1538ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:43.715 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f151c039530 msgr2=0x7f151c03b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f151c039530 0x7f151c03b9e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1524007ca0 tx=0x7f1524009040 comp rx=0 tx=0).stop 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 10 monclient: shutdown 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 msgr2=0x7f153c137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f153c137110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f151c008f20 tx=0x7f152c030000 comp rx=0 tx=0).stop 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 shutdown_connections 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f151c039530 0x7f151c03b9e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f153c096fa0 0x7f153c0973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153c098a30 0x7f153c096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 --2- 172.21.15.153:0/804226184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c0037d0 0x7f153c137110 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:43.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.715+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 >> 172.21.15.153:0/804226184 conn(0x7f153c067d30 msgr2=0x7f153c138dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:43.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.716+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 shutdown_connections 2024-09-15T03:33:43.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:43.716+0000 7f1542a68640 1 -- 172.21.15.153:0/804226184 wait complete. 2024-09-15T03:33:43.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.822+0000 7f7fbaf9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:43.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.822+0000 7f7fbaf9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:43.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.822+0000 7f7fbaf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:43.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:43.822+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f7fb404bf60 con 0x7f7fb4069bd0 2024-09-15T03:33:46.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:46.651+0000 7f7fab7fe640 10 monclient: tick 2024-09-15T03:33:46.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:46.651+0000 7f7fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:46.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:46.651+0000 7f7fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:46.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.831+0000 7f9357020640 10 monclient: get_monmap_and_config 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.831+0000 7f9357020640 10 monclient: build_initial_monmap 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.831+0000 7f9357020640 10 monclient: monmap: 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:46.831835+0000 2024-09-15T03:33:46.832 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:46.831835+0000 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:46.833 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:46.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.831+0000 7f9357020640 1 Processor -- start 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.831+0000 7f9357020640 1 -- start start 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.831+0000 7f9357020640 10 monclient: init 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): picked mon.noname-a con 0x7f935009e950 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f935009ed20 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:46.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): picked mon.noname-b con 0x7f935009f120 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935009f520 con 0x7f935009ed20 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935009fc20 con 0x7f935009e950 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935009d4f0 con 0x7f935009f120 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9357020640 10 monclient(hunting): authenticate will time out at 1024.471474s 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f934ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f935009ed20 0x7f9350068d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37850/0 (socket says 172.21.15.153:37850) 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f934ffff640 1 -- 172.21.15.153:0/1401914796 learned_addr learned my addr 172.21.15.153:0/1401914796 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:33:46.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 10 monclient(hunting): get_auth_request con 0x7f935009e950 auth_method 0 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 10 monclient(hunting): get_auth_request con 0x7f935009f120 auth_method 0 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.832+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:46.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:46.834+0000 7f934ffff640 1 -- 172.21.15.153:0/1401914796 >> v1:172.21.15.153:6789/0 conn(0x7f935009ed20 legacy=0x7f9350068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:47.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:47.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 10 monclient(hunting): get_auth_request con 0x7f935009e950 auth_method 0 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 10 monclient(hunting): get_auth_request con 0x7f935009f120 auth_method 0 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:47.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:47.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:47.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.033+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:33:47.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:47.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:47.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 10 monclient(hunting): get_auth_request con 0x7f935009f120 auth_method 0 2024-09-15T03:33:47.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 10 monclient(hunting): get_auth_request con 0x7f935009e950 auth_method 0 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:47.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:47.434+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:33:47.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:47.651+0000 7f7fab7fe640 10 monclient: tick 2024-09-15T03:33:47.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:47.651+0000 7f7fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 10 monclient(hunting): get_auth_request con 0x7f935009e950 auth_method 0 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 10 monclient(hunting): get_auth_request con 0x7f935009f120 auth_method 0 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:33:48.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.235+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:33:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.652+0000 7f7fab7fe640 10 monclient: tick 2024-09-15T03:33:48.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.652+0000 7f7fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.745+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (secure 0 0 0) 0x7f7fac040020 con 0x7f7fb4069bd0 2024-09-15T03:33:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.745+0000 7f7fb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:48.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.745+0000 7f7fb0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.747+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093155283 0 0) 0x7f935009f520 con 0x7f935009ed20 2024-09-15T03:33:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.747+0000 7f934f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.747+0000 7f934f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.747+0000 7f934f7fe640 10 monclient(hunting): my global_id is 84105 2024-09-15T03:33:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.747+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c0034e0 con 0x7f935009ed20 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1130582102 0 0) 0x7f933c0034e0 con 0x7f935009ed20 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f93500694a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f93500694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 msgr2=0x7f935009cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 --2- 172.21.15.153:0/1401914796 >> v2:172.21.15.153:3300/0 conn(0x7f935009e950 0x7f935009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500a1180 con 0x7f935009ed20 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f9338001720 con 0x7f935009ed20 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f9357020640 5 monclient: authenticate success, global_id 84105 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f9357020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: dump: 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 >> v1:172.21.15.153:6789/0 conn(0x7f935009ed20 legacy=0x7f9350068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c003ad0 0x7f933c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): picked mon.b con 0x7f933c003ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f933c004780 0x7f933c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): picked mon.a con 0x7f933c004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f933c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): picked mon.c con 0x7f935009f120 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935009fc20 con 0x7f933c004780 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935009d4f0 con 0x7f933c003ad0 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c005560 con 0x7f935009f120 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.748+0000 7f934f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:33:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9357020640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f933c004780 0x7f933c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): get_auth_request con 0x7f933c004780 auth_method 0 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f933c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c003ad0 0x7f933c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): get_auth_request con 0x7f935009f120 auth_method 0 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): get_auth_request con 0x7f933c003ad0 auth_method 0 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f93555bd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient(hunting): handle_auth_done global_id 84105 payload 293 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 msgr2=0x7f933c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f935009f120 0x7f933c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c003ad0 msgr2=0x7f933c003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c003ad0 0x7f933c003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 monclient: found mon.a 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 1 -- 172.21.15.153:0/1401914796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0034e0 con 0x7f933c004780 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 10 monclient: _finish_auth 0 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f934ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.749+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c003ad0 0x7f933c003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934ffff640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f933c004780 msgr2=0x7f933c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934ffff640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f933c004780 0x7f933c004bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9338001490 tx=0x7f935009c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ca907f96805ee0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338005890 con 0x7f933c004780 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: dump: 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9338005a10 con 0x7f933c004780 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f9357020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f9357020640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f9357020640 4 set_mon_vals no callback set 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 1 -- 172.21.15.153:0/1401914796 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338005dd0 con 0x7f933c004780 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.752+0000 7f934f7fe640 10 monclient: dump: 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 10 monclient: shutdown 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 1 -- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f933c004780 msgr2=0x7f933c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 1 --2- 172.21.15.153:0/1401914796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f933c004780 0x7f933c004bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9338001490 tx=0x7f935009c860 comp rx=0 tx=0).stop 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f93555bd640 1 -- 172.21.15.153:0/1401914796 reap_dead start 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 1 -- 172.21.15.153:0/1401914796 shutdown_connections 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 1 -- 172.21.15.153:0/1401914796 >> 172.21.15.153:0/1401914796 conn(0x7f9350067b90 msgr2=0x7f93500a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 1 -- 172.21.15.153:0/1401914796 shutdown_connections 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.753+0000 7f9357020640 1 -- 172.21.15.153:0/1401914796 wait complete. 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.754+0000 7f9357020640 10 monclient: build_initial_monmap 2024-09-15T03:33:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.754+0000 7f9357020640 10 monclient: monmap: 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.754+0000 7f9357020640 1 Processor -- start 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 -- start start 2024-09-15T03:33:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient: init 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 0x7f9350131980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): picked mon.noname-b con 0x7f933c004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 0x7f935011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): picked mon.noname-c con 0x7f9350131ec0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 0x7f935011c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): picked mon.noname-a con 0x7f935011bc60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935009be70 con 0x7f935011bc60 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935011ed80 con 0x7f933c004780 2024-09-15T03:33:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93500a3390 con 0x7f9350131ec0 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9357020640 10 monclient(hunting): authenticate will time out at 1026.394741s 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 0x7f935011c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f934ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 0x7f935011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 0x7f935011c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44224/0 (socket says 172.21.15.153:44224) 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 1 -- 172.21.15.153:0/1739259328 learned_addr learned my addr 172.21.15.153:0/1739259328 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 10 monclient(hunting): get_auth_request con 0x7f935011bc60 auth_method 0 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f93555bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 0x7f9350131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9354dbc640 10 monclient(hunting): get_auth_request con 0x7f933c004780 auth_method 0 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9354dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9354dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f9354dbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f934ffff640 10 monclient(hunting): get_auth_request con 0x7f9350131ec0 auth_method 0 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f934ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f934ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.755+0000 7f934ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f93555bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f93555bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f93555bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f934ffff640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 msgr2=0x7f935011b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f934ffff640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 msgr2=0x7f935011b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f934ffff640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 0x7f935011b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f934ffff640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 0x7f935011b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 10 monclient(hunting): handle_auth_done global_id 45175 payload 824 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 msgr2=0x7f935011b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 0x7f935011b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 msgr2=0x7f935011c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 0x7f935011c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 1 monclient: found mon.noname-b 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f9354dbc640 1 -- 172.21.15.153:0/1739259328 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350141ab0 con 0x7f933c004780 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.756+0000 7f93555bd640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 0x7f935011c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f9354dbc640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 msgr2=0x7f9350131980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f9354dbc640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 0x7f9350131980 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f93400076c0 tx=0x7f934000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340018910 con 0x7f933c004780 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 10 monclient: dump: 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.776 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9340018a70 con 0x7f933c004780 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.758+0000 7f9357020640 5 monclient: authenticate success, global_id 45175 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f9357020640 10 monclient: _renew_subs 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f9357020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935011eaa0 con 0x7f933c004780 2024-09-15T03:33:48.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f934cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340018e10 con 0x7f933c004780 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f934cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f934cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f934cff9640 10 monclient: dump: 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.778 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f9357020640 10 monclient: _renew_subs 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f934effd640 4 set_mon_vals no callback set 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f9357020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.759+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935011cfe0 con 0x7f933c004780 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9340003040 con 0x7f933c004780 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f9357020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f9357020640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f9357020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f934cff9640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9330038790 0x7f933003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935004bf60 con 0x7f933c004780 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(124..124 src has 40..124) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f934004e770 con 0x7f933c004780 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f934ffff640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9330038790 0x7f933003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.760+0000 7f934ffff640 10 monclient: get_auth_request con 0x7f9330038790 auth_method 0 2024-09-15T03:33:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.763+0000 7f934ffff640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9330038790 0x7f933003ac40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9338030000 tx=0x7f9338034000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.765+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f934001fb40 con 0x7f933c004780 2024-09-15T03:33:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.765+0000 7f934cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.765+0000 7f934cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:48.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.849+0000 7f7fbaf9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:48.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.849+0000 7f7fbaf9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:48.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.849+0000 7f7fbaf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.849+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f7fb4072640 con 0x7f7fb4069bd0 2024-09-15T03:33:48.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.850+0000 7f7fb0ff9640 1 -- 172.21.15.181:0/253537255 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (secure 0 0 0) 0x7f7fac049020 con 0x7f7fb4069bd0 2024-09-15T03:33:48.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.850+0000 7f7fb0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:48.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.850+0000 7f7fb0ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f94038740 msgr2=0x7f7f9403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f94038740 0x7f7f9403abf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4136bf0 tx=0x7f7f9c007e10 comp rx=0 tx=0).stop 2024-09-15T03:33:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 10 monclient: shutdown 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 msgr2=0x7f7fb4118560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 0x7f7fb4118560 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7fac007e80 tx=0x7f7fac009330 comp rx=0 tx=0).stop 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 shutdown_connections 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7f94038740 0x7f7f9403abf0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fb4136e30 0x7f7fb4118fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb4098330 0x7f7fb4118aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 --2- 172.21.15.181:0/253537255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fb4069bd0 0x7f7fb4118560 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 >> 172.21.15.181:0/253537255 conn(0x7f7fb409e9f0 msgr2=0x7f7fb40a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 shutdown_connections 2024-09-15T03:33:48.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.852+0000 7f7fbaf9d640 1 -- 172.21.15.181:0/253537255 wait complete. 2024-09-15T03:33:48.863 INFO:tasks.workunit.client.0.smithi181.stdout:fooset 2024-09-15T03:33:48.863 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:33:48.864 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep barset 2024-09-15T03:33:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.936+0000 7fae36c8b640 10 monclient: get_monmap_and_config 2024-09-15T03:33:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.936+0000 7fae36c8b640 10 monclient: build_initial_monmap 2024-09-15T03:33:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.936+0000 7fae36c8b640 10 monclient: monmap: 2024-09-15T03:33:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:48.937200+0000 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:48.937200+0000 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:48.939 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 1 Processor -- start 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 1 -- start start 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient: init 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae3009c9f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:48.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae30096f00 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fae3009d580 0x7fae30137300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient(hunting): picked mon.noname-a con 0x7fae3009d580 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3009cdc0 con 0x7fae3009c9f0 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae300992f0 con 0x7fae30096f00 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae300983d0 con 0x7fae3009d580 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae36c8b640 10 monclient(hunting): authenticate will time out at 1033.418796s 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae2ffff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fae30096f00 0x7fae30133bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32894/0 (socket says 172.21.15.181:32894) 2024-09-15T03:33:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae2ffff640 1 -- 172.21.15.181:0/3500076849 learned_addr learned my addr 172.21.15.181:0/3500076849 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae35228640 1 --2- 172.21.15.181:0/3500076849 >> v2:172.21.15.153:3300/0 conn(0x7fae3009d580 0x7fae30137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae35228640 10 monclient(hunting): get_auth_request con 0x7fae3009d580 auth_method 0 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae35228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.937+0000 7fae35228640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.938+0000 7fae35228640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae2ffff640 1 -- 172.21.15.181:0/3500076849 >> v1:172.21.15.181:6789/0 conn(0x7fae30096f00 legacy=0x7fae30133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae34a27640 1 -- 172.21.15.181:0/3500076849 >> v1:172.21.15.153:6789/0 conn(0x7fae3009c9f0 legacy=0x7fae3009ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae2f7fe640 1 -- 172.21.15.181:0/3500076849 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154225115 0 0) 0x7fae300992f0 con 0x7fae30096f00 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae2f7fe640 10 monclient(hunting): my global_id is 45178 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae2f7fe640 1 -- 172.21.15.181:0/3500076849 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae200034e0 con 0x7fae30096f00 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 10 monclient(hunting): handle_auth_done global_id 84111 payload 274 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 1 -- 172.21.15.181:0/3500076849 >> v1:172.21.15.181:6789/0 conn(0x7fae30096f00 legacy=0x7fae30133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 1 -- 172.21.15.181:0/3500076849 >> v1:172.21.15.153:6789/0 conn(0x7fae3009c9f0 legacy=0x7fae3009ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 10 monclient: _finish_hunting 0 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 1 monclient: found mon.noname-a 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:48.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.940+0000 7fae35228640 1 -- 172.21.15.181:0/3500076849 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30098d30 con 0x7fae3009d580 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae35228640 1 -- 172.21.15.181:0/3500076849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 msgr2=0x7fae30137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae35228640 1 --2- 172.21.15.181:0/3500076849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae30137300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae28009f60 tx=0x7fae28009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50973c36afbec06 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 1 -- 172.21.15.181:0/3500076849 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28007760 con 0x7fae3009d580 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: dump: 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:48.952 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 5 monclient: authenticate success, global_id 84111 2024-09-15T03:33:48.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 1 -- 172.21.15.181:0/3500076849 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2802d050 con 0x7fae3009d580 2024-09-15T03:33:48.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:48.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 1 -- 172.21.15.181:0/3500076849 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28007af0 con 0x7fae3009d580 2024-09-15T03:33:48.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:48.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 4 set_mon_vals no callback set 2024-09-15T03:33:48.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:48.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae2f7fe640 10 monclient: dump: 2024-09-15T03:33:48.955 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:48.955 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.955 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.955 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 10 monclient: shutdown 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 -- 172.21.15.181:0/3500076849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 msgr2=0x7fae30137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3500076849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae30137300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae28009f60 tx=0x7fae28009780 comp rx=0 tx=0).stop 2024-09-15T03:33:48.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 -- 172.21.15.181:0/3500076849 shutdown_connections 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3500076849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae30137300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 -- 172.21.15.181:0/3500076849 >> 172.21.15.181:0/3500076849 conn(0x7fae30071510 msgr2=0x7fae300a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 -- 172.21.15.181:0/3500076849 shutdown_connections 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.943+0000 7fae36c8b640 1 -- 172.21.15.181:0/3500076849 wait complete. 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.944+0000 7fae36c8b640 10 monclient: build_initial_monmap 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.944+0000 7fae36c8b640 10 monclient: monmap: 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.944+0000 7fae36c8b640 1 Processor -- start 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.944+0000 7fae36c8b640 1 -- start start 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.944+0000 7fae36c8b640 10 monclient: init 2024-09-15T03:33:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 0x7fae3013ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): picked mon.noname-c con 0x7fae30096f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 0x7fae30131aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): picked mon.noname-b con 0x7fae3009c9f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae3011b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): picked mon.noname-a con 0x7fae3009d580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3011d680 con 0x7fae3009d580 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae30098810 con 0x7fae3009c9f0 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae30132700 con 0x7fae30096f00 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae36c8b640 10 monclient(hunting): authenticate will time out at 1033.426379s 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 0x7fae30131aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 0x7fae30131aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47616/0 (socket says 172.21.15.181:47616) 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 -- 172.21.15.181:0/3957059680 learned_addr learned my addr 172.21.15.181:0/3957059680 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): get_auth_request con 0x7fae3009c9f0 auth_method 0 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 0x7fae3013ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 10 monclient(hunting): get_auth_request con 0x7fae30096f00 auth_method 0 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae3011b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): get_auth_request con 0x7fae3009d580 auth_method 0 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 msgr2=0x7fae3013ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 msgr2=0x7fae3013ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 0x7fae3013ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae35228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae34a27640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 0x7fae3013ee60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient(hunting): handle_auth_done global_id 45181 payload 824 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 msgr2=0x7fae3013ee60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 0x7fae3013ee60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 msgr2=0x7fae3011b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae3011b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 monclient: found mon.noname-b 2024-09-15T03:33:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.945+0000 7fae2ffff640 1 -- 172.21.15.181:0/3957059680 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28009a20 con 0x7fae3009c9f0 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.947+0000 7fae2ffff640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 msgr2=0x7fae30131aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.947+0000 7fae2ffff640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 0x7fae30131aa0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00c790 tx=0x7fae1c007bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c00ce60 con 0x7fae3009c9f0 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: dump: 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c005aa0 con 0x7fae3009c9f0 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 5 monclient: authenticate success, global_id 45181 2024-09-15T03:33:48.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 10 monclient: _renew_subs 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3011dcd0 con 0x7fae3009c9f0 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c005e20 con 0x7fae3009c9f0 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 10 monclient: dump: 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:48.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:48.966 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.966 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:48.966 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:48.966 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:48.966 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:48.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:48.967 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2effd640 4 set_mon_vals no callback set 2024-09-15T03:33:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 10 monclient: _renew_subs 2024-09-15T03:33:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3011d100 con 0x7fae3009c9f0 2024-09-15T03:33:48.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.948+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fae1c007300 con 0x7fae3009c9f0 2024-09-15T03:33:48.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae36c8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:48.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae36c8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:48.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae36c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3004bf60 con 0x7fae3009c9f0 2024-09-15T03:33:48.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae2cff9640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae14038740 0x7fae1403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:48.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(124..124 src has 40..124) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fae1c04c2f0 con 0x7fae3009c9f0 2024-09-15T03:33:48.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae34a27640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae14038740 0x7fae1403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:48.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.949+0000 7fae34a27640 10 monclient: get_auth_request con 0x7fae14038740 auth_method 0 2024-09-15T03:33:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.951+0000 7fae34a27640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae14038740 0x7fae1403abf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae24007a80 tx=0x7fae24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.953+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fae1c0108c0 con 0x7fae3009c9f0 2024-09-15T03:33:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.953+0000 7fae2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:48.953+0000 7fae2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:48.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f9357020640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:48.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f9357020640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:48.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f9357020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:48.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f93500638d0 con 0x7f933c004780 2024-09-15T03:33:48.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f934cff9640 1 -- 172.21.15.153:0/1739259328 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f934000a270 con 0x7f933c004780 2024-09-15T03:33:48.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f934cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:48.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.984+0000 7f934cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:48.986 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:48.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9330038790 msgr2=0x7f933003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9330038790 0x7f933003ac40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9338030000 tx=0x7f9338034000 comp rx=0 tx=0).stop 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 10 monclient: shutdown 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 msgr2=0x7f9350131980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 0x7f9350131980 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f93400076c0 tx=0x7f934000b040 comp rx=0 tx=0).stop 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 shutdown_connections 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9330038790 0x7f933003ac40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f935011bc60 0x7f935011c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9350131ec0 0x7f935011b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 --2- 172.21.15.153:0/1739259328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f933c004780 0x7f9350131980 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:48.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 >> 172.21.15.153:0/1739259328 conn(0x7f9350067b90 msgr2=0x7f93500a4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:48.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 shutdown_connections 2024-09-15T03:33:48.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:48.986+0000 7f9357020640 1 -- 172.21.15.153:0/1739259328 wait complete. 2024-09-15T03:33:49.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae36c8b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:49.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae36c8b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:49.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae36c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fae30064b20 con 0x7fae3009c9f0 2024-09-15T03:33:49.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (secure 0 0 0) 0x7fae1c0108c0 con 0x7fae3009c9f0 2024-09-15T03:33:49.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:49.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.114+0000 7fae2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:49.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae36c8b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:49.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae36c8b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:49.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae36c8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fae3013f3a0 con 0x7fae3009c9f0 2024-09-15T03:33:49.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae2cff9640 1 -- 172.21.15.181:0/3957059680 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (secure 0 0 0) 0x7fae1c0108c0 con 0x7fae3009c9f0 2024-09-15T03:33:49.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae2cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:49.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.212+0000 7fae2cff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae14038740 msgr2=0x7fae1403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae14038740 0x7fae1403abf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae24007a80 tx=0x7fae24009040 comp rx=0 tx=0).stop 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 10 monclient: shutdown 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 msgr2=0x7fae30131aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 0x7fae30131aa0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00c790 tx=0x7fae1c007bc0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 shutdown_connections 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fae14038740 0x7fae1403abf0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae3009d580 0x7fae3011b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae3009c9f0 0x7fae30131aa0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 --2- 172.21.15.181:0/3957059680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae30096f00 0x7fae3013ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 >> 172.21.15.181:0/3957059680 conn(0x7fae30071510 msgr2=0x7fae3006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:49.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 shutdown_connections 2024-09-15T03:33:49.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.214+0000 7fae36c8b640 1 -- 172.21.15.181:0/3957059680 wait complete. 2024-09-15T03:33:49.224 INFO:tasks.workunit.client.0.smithi181.stdout:barset 2024-09-15T03:33:49.224 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set dump 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.297+0000 7fe270f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.297+0000 7fe270f9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.297+0000 7fe270f9d640 10 monclient: monmap: 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:49.297996+0000 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:49.297996+0000 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:49.298 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.299 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:49.299 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:49.299 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:49.299 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:49.299 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:49.299 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.297+0000 7fe270f9d640 1 Processor -- start 2024-09-15T03:33:49.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.297+0000 7fe270f9d640 1 -- start start 2024-09-15T03:33:49.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.297+0000 7fe270f9d640 10 monclient: init 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c09ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe26c09c9f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe26c096f00 0x7fe26c12e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe26c096f00 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe26c09d580 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c09cdc0 con 0x7fe26c09d580 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0983d0 con 0x7fe26c096f00 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0992f0 con 0x7fe26c09c9f0 2024-09-15T03:33:49.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe270f9d640 10 monclient(hunting): authenticate will time out at 1033.779615s 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26ad9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe26c09d580 0x7fe26c12ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32904/0 (socket says 172.21.15.181:32904) 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26ad9d640 1 -- 172.21.15.181:0/824625230 learned_addr learned my addr 172.21.15.181:0/824625230 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 1 --2- 172.21.15.181:0/824625230 >> v2:172.21.15.153:3300/0 conn(0x7fe26c096f00 0x7fe26c12e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c09ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): get_auth_request con 0x7fe26c096f00 auth_method 0 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 10 monclient(hunting): get_auth_request con 0x7fe26c09c9f0 auth_method 0 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 1 -- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 msgr2=0x7fe26c09ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 1 -- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 msgr2=0x7fe26c09ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe269d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c09ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.298+0000 7fe26a59c640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c09ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 10 monclient(hunting): handle_auth_done global_id 84120 payload 274 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 1 -- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 msgr2=0x7fe26c09ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:33:49.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c09ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 1 -- 172.21.15.181:0/824625230 >> v1:172.21.15.181:6789/0 conn(0x7fe26c09d580 legacy=0x7fe26c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:49.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:33:49.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 1 monclient: found mon.noname-a 2024-09-15T03:33:49.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:49.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.299+0000 7fe269d9b640 1 -- 172.21.15.181:0/824625230 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c098d30 con 0x7fe26c096f00 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.301+0000 7fe269d9b640 1 -- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c096f00 msgr2=0x7fe26c12e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.301+0000 7fe269d9b640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c096f00 0x7fe26c12e8a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe260009850 tx=0x7fe2600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683cbe4ab6f0d433 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 1 -- 172.21.15.181:0/824625230 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26002adf0 con 0x7fe26c096f00 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: dump: 2024-09-15T03:33:49.320 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.321 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.321 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.321 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.321 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: _finish_auth 0 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 1 -- 172.21.15.181:0/824625230 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe260032440 con 0x7fe26c096f00 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 5 monclient: authenticate success, global_id 84120 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 1 -- 172.21.15.181:0/824625230 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260032820 con 0x7fe26c096f00 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 4 set_mon_vals no callback set 2024-09-15T03:33:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe26959a640 10 monclient: dump: 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.324 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 10 monclient: shutdown 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:49.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 -- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c096f00 msgr2=0x7fe26c12e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c096f00 0x7fe26c12e8a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe260009850 tx=0x7fe2600090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 -- 172.21.15.181:0/824625230 shutdown_connections 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c096f00 0x7fe26c12e8a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 --2- 172.21.15.181:0/824625230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c09ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 -- 172.21.15.181:0/824625230 >> 172.21.15.181:0/824625230 conn(0x7fe26c071510 msgr2=0x7fe26c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 -- 172.21.15.181:0/824625230 shutdown_connections 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.302+0000 7fe270f9d640 1 -- 172.21.15.181:0/824625230 wait complete. 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.303+0000 7fe270f9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.303+0000 7fe270f9d640 10 monclient: monmap: 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.303+0000 7fe270f9d640 1 Processor -- start 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.303+0000 7fe270f9d640 1 -- start start 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.303+0000 7fe270f9d640 10 monclient: init 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 0x7fe26c11bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe26c096f00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c11c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe26c09c9f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c09d580 0x7fe26c11c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe26c09d580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c122840 con 0x7fe26c09d580 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c123710 con 0x7fe26c096f00 2024-09-15T03:33:49.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c122b20 con 0x7fe26c09c9f0 2024-09-15T03:33:49.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:49.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe270f9d640 10 monclient(hunting): authenticate will time out at 1033.785114s 2024-09-15T03:33:49.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 0x7fe26c11bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 0x7fe26c11bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47640/0 (socket says 172.21.15.181:47640) 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 -- 172.21.15.181:0/2202950728 learned_addr learned my addr 172.21.15.181:0/2202950728 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): get_auth_request con 0x7fe26c096f00 auth_method 0 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c11c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:49.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 10 monclient(hunting): get_auth_request con 0x7fe26c09c9f0 auth_method 0 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c09d580 0x7fe26c11c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): get_auth_request con 0x7fe26c09d580 auth_method 0 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 msgr2=0x7fe26c11c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 msgr2=0x7fe26c11c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c11c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe269d9b640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c11c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient(hunting): handle_auth_done global_id 45184 payload 824 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 msgr2=0x7fe26c11c230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c11c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c09d580 msgr2=0x7fe26c11c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c09d580 0x7fe26c11c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:49.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 monclient: found mon.noname-b 2024-09-15T03:33:49.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:49.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.304+0000 7fe26a59c640 1 -- 172.21.15.181:0/2202950728 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe260009390 con 0x7fe26c096f00 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe26a59c640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 msgr2=0x7fe26c11bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe26a59c640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 0x7fe26c11bcf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00d890 tx=0x7fe25c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00de20 con 0x7fe26c096f00 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: dump: 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:49.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 5 monclient: authenticate success, global_id 45184 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 10 monclient: _renew_subs 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c00bae0 con 0x7fe26c096f00 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c131550 con 0x7fe26c096f00 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00bea0 con 0x7fe26c096f00 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe25a7fc640 10 monclient: dump: 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.337 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe268d99640 4 set_mon_vals no callback set 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 10 monclient: _renew_subs 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.306+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c122260 con 0x7fe26c096f00 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.307+0000 7fe270f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.307+0000 7fe270f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:49.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.307+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe25c014640 con 0x7fe26c096f00 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.307+0000 7fe270f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.307+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c122670 con 0x7fe26c096f00 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.307+0000 7fe25a7fc640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe24c038740 0x7fe24c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.308+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(124..124 src has 40..124) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fe25c04cdf0 con 0x7fe26c096f00 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.308+0000 7fe269d9b640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe24c038740 0x7fe24c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.308+0000 7fe269d9b640 10 monclient: get_auth_request con 0x7fe24c038740 auth_method 0 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.310+0000 7fe269d9b640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe24c038740 0x7fe24c03abf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe260034000 tx=0x7fe260032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.312+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe25c050030 con 0x7fe26c096f00 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.312+0000 7fe25a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.312+0000 7fe25a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:49.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.473+0000 7fe270f9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:49.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.473+0000 7fe270f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:49.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.473+0000 7fe270f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.473+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fe26c04bf60 con 0x7fe26c096f00 2024-09-15T03:33:49.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.474+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v124) v1 ==== 73+0+6748 (secure 0 0 0) 0x7fe25c011570 con 0x7fe26c096f00 2024-09-15T03:33:49.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.474+0000 7fe25a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:49.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.474+0000 7fe25a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:49.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe270f9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:49.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe270f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:49.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe270f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fe26c0a0460 con 0x7fe26c096f00 2024-09-15T03:33:49.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe25a7fc640 1 -- 172.21.15.181:0/2202950728 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v124) v1 ==== 73+0+6748 (secure 0 0 0) 0x7fe25c01dbb0 con 0x7fe26c096f00 2024-09-15T03:33:49.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe25a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-15T03:33:49.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.572+0000 7fe25a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:49.574 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-15T03:33:49.574 INFO:tasks.workunit.client.0.smithi181.stdout: "9": [ 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -1, 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.575 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -2, 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -3, 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.576 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -4, 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -5, 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.577 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -6, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.578 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -7, 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.579 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -8, 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -9, 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.580 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -10, 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -11, 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.581 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -12, 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -13, 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.582 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -14, 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -15, 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:33:49.583 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: "10": [ 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -1, 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.584 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0.263671875, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.585 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -2, 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.586 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -3, 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.588 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.589 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -4, 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -5, 2024-09-15T03:33:49.590 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.591 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -6, 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.592 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.593 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -7, 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.594 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -8, 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -9, 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.595 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: [ 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0.087890625, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.596 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -10, 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.597 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -11, 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -12, 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.598 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -13, 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -14, 2024-09-15T03:33:49.599 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: "bucket_id": -15, 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: "weight_set": [ 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: [], 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: [] 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.600 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-15T03:33:49.601 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-15T03:33:49.601 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe24c038740 msgr2=0x7fe24c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe24c038740 0x7fe24c03abf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe260034000 tx=0x7fe260032000 comp rx=0 tx=0).stop 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 10 monclient: shutdown 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 msgr2=0x7fe26c11bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 0x7fe26c11bcf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00d890 tx=0x7fe25c008040 comp rx=0 tx=0).stop 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 shutdown_connections 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe24c038740 0x7fe24c03abf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c09d580 0x7fe26c11c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe26c09c9f0 0x7fe26c11c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 --2- 172.21.15.181:0/2202950728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe26c096f00 0x7fe26c11bcf0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 >> 172.21.15.181:0/2202950728 conn(0x7fe26c071510 msgr2=0x7fe26c06bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:49.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.574+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 shutdown_connections 2024-09-15T03:33:49.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.575+0000 7fe270f9d640 1 -- 172.21.15.181:0/2202950728 wait complete. 2024-09-15T03:33:49.603 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-15T03:33:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.660+0000 7f1f79834640 10 monclient: get_monmap_and_config 2024-09-15T03:33:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.660+0000 7f1f79834640 10 monclient: build_initial_monmap 2024-09-15T03:33:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.660+0000 7f1f79834640 10 monclient: monmap: 2024-09-15T03:33:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:49.661400+0000 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:49.661400+0000 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:49.662 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 Processor -- start 2024-09-15T03:33:49.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 -- start start 2024-09-15T03:33:49.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient: init 2024-09-15T03:33:49.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:49.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f7409ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1f7409f020 0x7f1f74069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): picked mon.noname-a con 0x7f1f7409f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f74136d60 0x7f1f74139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): picked mon.noname-c con 0x7f1f74136d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f74067420 con 0x7f1f7409ec50 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7409fdc0 con 0x7f1f7409f020 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f740a0510 con 0x7f1f74136d60 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f79834640 10 monclient(hunting): authenticate will time out at 1034.142740s 2024-09-15T03:33:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f74136d60 0x7f1f74139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f74136d60 0x7f1f74139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47646/0 (socket says 172.21.15.181:47646) 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 1 -- 172.21.15.181:0/1084432689 learned_addr learned my addr 172.21.15.181:0/1084432689 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 10 monclient(hunting): get_auth_request con 0x7f1f74136d60 auth_method 0 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.661+0000 7f1f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f6a7fc640 1 --2- 172.21.15.181:0/1084432689 >> v2:172.21.15.153:3300/0 conn(0x7f1f7409f020 0x7f1f74069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 10 monclient(hunting): handle_auth_done global_id 45187 payload 274 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 1 -- 172.21.15.181:0/1084432689 >> v2:172.21.15.153:3300/0 conn(0x7f1f7409f020 msgr2=0x7f1f74069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 1 --2- 172.21.15.181:0/1084432689 >> v2:172.21.15.153:3300/0 conn(0x7f1f7409f020 0x7f1f74069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 1 -- 172.21.15.181:0/1084432689 >> v1:172.21.15.153:6789/0 conn(0x7f1f7409ec50 legacy=0x7f1f74131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 1 monclient: found mon.noname-c 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f737fe640 1 -- 172.21.15.181:0/1084432689 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f740a0200 con 0x7f1f74136d60 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f6a7fc640 10 monclient: get_auth_request con 0x7f1f7409f020 auth_method 0 2024-09-15T03:33:49.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.662+0000 7f1f6a7fc640 1 --2- 172.21.15.181:0/1084432689 >> v2:172.21.15.153:3300/0 conn(0x7f1f7409f020 0x7f1f74069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.664+0000 7f1f737fe640 1 -- 172.21.15.181:0/1084432689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 msgr2=0x7f1f74139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f737fe640 1 --2- 172.21.15.181:0/1084432689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74139140 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1f74132a50 tx=0x7f1f6c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b07932bf080ce91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 1 -- 172.21.15.181:0/1084432689 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c007610 con 0x7f1f74136d60 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: dump: 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.675 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 1 -- 172.21.15.181:0/1084432689 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c02d050 con 0x7f1f74136d60 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 1 -- 172.21.15.181:0/1084432689 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0079a0 con 0x7f1f74136d60 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 5 monclient: authenticate success, global_id 45187 2024-09-15T03:33:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 4 set_mon_vals no callback set 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f727fc640 10 monclient: dump: 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 10 monclient: shutdown 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 -- 172.21.15.181:0/1084432689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 msgr2=0x7f1f74139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 --2- 172.21.15.181:0/1084432689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74139140 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1f74132a50 tx=0x7f1f6c009840 comp rx=0 tx=0).stop 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 -- 172.21.15.181:0/1084432689 shutdown_connections 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 --2- 172.21.15.181:0/1084432689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74139140 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 --2- 172.21.15.181:0/1084432689 >> v2:172.21.15.153:3300/0 conn(0x7f1f7409f020 0x7f1f74069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 -- 172.21.15.181:0/1084432689 >> 172.21.15.181:0/1084432689 conn(0x7f1f740a5d40 msgr2=0x7f1f740a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 -- 172.21.15.181:0/1084432689 shutdown_connections 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.665+0000 7f1f79834640 1 -- 172.21.15.181:0/1084432689 wait complete. 2024-09-15T03:33:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.666+0000 7f1f79834640 10 monclient: build_initial_monmap 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.666+0000 7f1f79834640 10 monclient: monmap: 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.666+0000 7f1f79834640 1 Processor -- start 2024-09-15T03:33:49.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.666+0000 7f1f79834640 1 -- start start 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.666+0000 7f1f79834640 10 monclient: init 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7409ec50 0x7f1f74141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): picked mon.noname-a con 0x7f1f7409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f7409f020 0x7f1f74141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): picked mon.noname-c con 0x7f1f7409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74099540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): picked mon.noname-b con 0x7f1f74136d60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7411cb30 con 0x7f1f7409ec50 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7409c000 con 0x7f1f74136d60 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740a3e50 con 0x7f1f7409f020 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f79834640 10 monclient(hunting): authenticate will time out at 1034.148379s 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74099540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74099540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47662/0 (socket says 172.21.15.181:47662) 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 -- 172.21.15.181:0/1797528283 learned_addr learned my addr 172.21.15.181:0/1797528283 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): get_auth_request con 0x7f1f74136d60 auth_method 0 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f72ffd640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7409ec50 0x7f1f74141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f6a7fc640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f7409f020 0x7f1f74141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f6a7fc640 10 monclient(hunting): get_auth_request con 0x7f1f7409f020 auth_method 0 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f6a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f72ffd640 10 monclient(hunting): get_auth_request con 0x7f1f7409ec50 auth_method 0 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:49.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient(hunting): handle_auth_done global_id 45190 payload 824 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 -- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f7409f020 msgr2=0x7f1f74141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f7409f020 0x7f1f74141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 -- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7409ec50 msgr2=0x7f1f74141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7409ec50 0x7f1f74141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.667+0000 7f1f737fe640 1 -- 172.21.15.181:0/1797528283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c009ae0 con 0x7f1f74136d60 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.669+0000 7f1f737fe640 1 -- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 msgr2=0x7f1f74099540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f737fe640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74099540 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009880 tx=0x7f1f6c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c032c60 con 0x7f1f74136d60 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: dump: 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.684 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c032dc0 con 0x7f1f74136d60 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 5 monclient: authenticate success, global_id 45190 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 10 monclient: _renew_subs 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7411c850 con 0x7f1f74136d60 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c03b660 con 0x7f1f74136d60 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 10 monclient: dump: 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:49.685 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 10 monclient: _renew_subs 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f71ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7411b960 con 0x7f1f74136d60 2024-09-15T03:33:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.670+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1f6c038050 con 0x7f1f74136d60 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f6b7fe640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f54038740 0x7f1f5403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(124..124 src has 40..124) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f1f6c06d5f0 con 0x7f1f74136d60 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f79834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f72ffd640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f54038740 0x7f1f5403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f79834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f79834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f740008d0 con 0x7f1f74136d60 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.671+0000 7f1f72ffd640 10 monclient: get_auth_request con 0x7f1f54038740 auth_method 0 2024-09-15T03:33:49.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.673+0000 7f1f72ffd640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f54038740 0x7f1f5403abf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009950 tx=0x7f1f64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:49.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.675+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1f6c03f070 con 0x7f1f74136d60 2024-09-15T03:33:49.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.675+0000 7f1f6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:49.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.675+0000 7f1f6b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:49.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.839+0000 7f1f79834640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:49.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.839+0000 7f1f79834640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:49.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.839+0000 7f1f79834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:49.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:49.839+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f1f740a13e0 con 0x7f1f74136d60 2024-09-15T03:33:51.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.238+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v125) v1 ==== 129+0+0 (secure 0 0 0) 0x7f1f6c03ba10 con 0x7f1f74136d60 2024-09-15T03:33:51.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.238+0000 7f1f6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:51.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.238+0000 7f1f6b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:51.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.346+0000 7f1f79834640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:51.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.346+0000 7f1f79834640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:51.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.346+0000 7f1f79834640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:51.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:51.346+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f1f74072640 con 0x7f1f74136d60 2024-09-15T03:33:52.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:52.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.103+0000 7f688ac2b640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.103+0000 7f688ac2b640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.103+0000 7f688ac2b640 10 monclient: monmap: 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:52.104582+0000 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:52.104582+0000 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.106 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.104+0000 7f688ac2b640 1 Processor -- start 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.104+0000 7f688ac2b640 1 -- start start 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.104+0000 7f688ac2b640 10 monclient: init 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f68840974f0 0x7f688412ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): picked mon.noname-a con 0x7f68840974f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f688409a8a0 0x7f688409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): picked mon.noname-c con 0x7f688409a8a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f688412dd40 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68840978c0 con 0x7f688412dd40 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688409b220 con 0x7f68840974f0 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688409ca20 con 0x7f688409a8a0 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f688ac2b640 10 monclient(hunting): authenticate will time out at 1029.744323s 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f68840974f0 0x7f688412ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f68840974f0 0x7f688412ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44230/0 (socket says 172.21.15.153:44230) 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 -- 172.21.15.153:0/339782635 learned_addr learned my addr 172.21.15.153:0/339782635 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): get_auth_request con 0x7f68840974f0 auth_method 0 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 1 --2- 172.21.15.153:0/339782635 >> v2:172.21.15.181:3300/0 conn(0x7f688409a8a0 0x7f688409ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): get_auth_request con 0x7f688409a8a0 auth_method 0 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient(hunting): handle_auth_done global_id 84129 payload 274 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 -- 172.21.15.153:0/339782635 >> v2:172.21.15.181:3300/0 conn(0x7f688409a8a0 msgr2=0x7f688409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 --2- 172.21.15.153:0/339782635 >> v2:172.21.15.181:3300/0 conn(0x7f688409a8a0 0x7f688409ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 -- 172.21.15.153:0/339782635 >> v1:172.21.15.181:6789/0 conn(0x7f688412dd40 legacy=0x7f688412e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.105+0000 7f68889c7640 1 -- 172.21.15.153:0/339782635 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884098ad0 con 0x7f68840974f0 2024-09-15T03:33:52.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68889c7640 1 -- 172.21.15.153:0/339782635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68840974f0 msgr2=0x7f688412ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68889c7640 1 --2- 172.21.15.153:0/339782635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68840974f0 0x7f688412ab00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f688409d040 tx=0x7f68740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6835c507a61b0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 1 -- 172.21.15.153:0/339782635 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6874032440 con 0x7f68840974f0 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: dump: 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 1 -- 172.21.15.153:0/339782635 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68740325a0 con 0x7f68840974f0 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f688ac2b640 5 monclient: authenticate success, global_id 84129 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f688ac2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f688ac2b640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 1 -- 172.21.15.153:0/339782635 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6874032940 con 0x7f68840974f0 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f688ac2b640 4 set_mon_vals no callback set 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.108+0000 7f68837fe640 10 monclient: dump: 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 10 monclient: shutdown 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 -- 172.21.15.153:0/339782635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68840974f0 msgr2=0x7f688412ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 --2- 172.21.15.153:0/339782635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68840974f0 0x7f688412ab00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f688409d040 tx=0x7f68740091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 -- 172.21.15.153:0/339782635 shutdown_connections 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 --2- 172.21.15.153:0/339782635 >> v2:172.21.15.181:3300/0 conn(0x7f688409a8a0 0x7f688409ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 --2- 172.21.15.153:0/339782635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68840974f0 0x7f688412ab00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 -- 172.21.15.153:0/339782635 >> 172.21.15.153:0/339782635 conn(0x7f68840672b0 msgr2=0x7f68840676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 -- 172.21.15.153:0/339782635 shutdown_connections 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 1 -- 172.21.15.153:0/339782635 wait complete. 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.109+0000 7f688ac2b640 10 monclient: monmap: 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:52.121 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 1 Processor -- start 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 1 -- start start 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient: init 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 0x7f68841176b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient(hunting): picked mon.noname-c con 0x7f68840974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f688409a8a0 0x7f6884117bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient(hunting): picked mon.noname-b con 0x7f688409a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 0x7f6884118130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient(hunting): picked mon.noname-a con 0x7f688412dd40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.110+0000 7f688ac2b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6884119570 con 0x7f688412dd40 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f688ac2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f688ac2b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6884098300 con 0x7f688409a8a0 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f688ac2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f688ac2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688409e060 con 0x7f68840974f0 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f688ac2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f688ac2b640 10 monclient(hunting): authenticate will time out at 1029.750049s 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 0x7f68841176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 0x7f68841176b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45642/0 (socket says 172.21.15.153:45642) 2024-09-15T03:33:52.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 0x7f6884118130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 1 -- 172.21.15.153:0/508546048 learned_addr learned my addr 172.21.15.153:0/508546048 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): get_auth_request con 0x7f68840974f0 auth_method 0 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): get_auth_request con 0x7f688412dd40 auth_method 0 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f688409a8a0 0x7f6884117bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): get_auth_request con 0x7f688409a8a0 auth_method 0 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient(hunting): handle_auth_done global_id 84132 payload 824 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 -- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 msgr2=0x7f68841176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 0x7f68841176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 -- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f688409a8a0 msgr2=0x7f6884117bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f688409a8a0 0x7f6884117bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68891c8640 1 -- 172.21.15.153:0/508546048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874009450 con 0x7f688412dd40 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.111+0000 7f68889c7640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 0x7f68841176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f68891c8640 1 -- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 msgr2=0x7f6884118130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f68891c8640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 0x7f6884118130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c460 tx=0x7f687c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0117a0 con 0x7f688412dd40 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: dump: 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c011920 con 0x7f688412dd40 2024-09-15T03:33:52.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 5 monclient: authenticate success, global_id 84132 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 10 monclient: _renew_subs 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6884119080 con 0x7f688412dd40 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c011ce0 con 0x7f688412dd40 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6880ff9640 10 monclient: dump: 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.128 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 10 monclient: _renew_subs 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688411a630 con 0x7f688412dd40 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.113+0000 7f6882ffd640 4 set_mon_vals no callback set 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.114+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f687c012630 con 0x7f688412dd40 2024-09-15T03:33:52.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f688ac2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f688ac2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f688ac2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f6880ff9640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6870038740 0x7f687003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688411a970 con 0x7f688412dd40 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(125..125 src has 40..125) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f687c04c430 con 0x7f688412dd40 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f68889c7640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6870038740 0x7f687003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.115+0000 7f68889c7640 10 monclient: get_auth_request con 0x7f6870038740 auth_method 0 2024-09-15T03:33:52.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.117+0000 7f68889c7640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6870038740 0x7f687003abf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6874009420 tx=0x7f6874032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.119+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f688411a970 con 0x7f688412dd40 2024-09-15T03:33:52.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.119+0000 7f6880ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.119+0000 7f6880ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f688ac2b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f688ac2b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f688ac2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f688411a970 con 0x7f688412dd40 2024-09-15T03:33:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f6880ff9640 1 -- 172.21.15.153:0/508546048 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f688411a970 con 0x7f688412dd40 2024-09-15T03:33:52.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f6880ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:52.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.332+0000 7f6880ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:52.334 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6870038740 msgr2=0x7f687003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6870038740 0x7f687003abf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6874009420 tx=0x7f6874032000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 10 monclient: shutdown 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 msgr2=0x7f6884118130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 0x7f6884118130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c460 tx=0x7f687c007e10 comp rx=0 tx=0).stop 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 shutdown_connections 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6870038740 0x7f687003abf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f688412dd40 0x7f6884118130 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f688409a8a0 0x7f6884117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 --2- 172.21.15.153:0/508546048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68840974f0 0x7f68841176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 >> 172.21.15.153:0/508546048 conn(0x7f68840672b0 msgr2=0x7f688412d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 shutdown_connections 2024-09-15T03:33:52.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.334+0000 7f688ac2b640 1 -- 172.21.15.153:0/508546048 wait complete. 2024-09-15T03:33:52.346 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:33:52.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:33:52.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:33:52.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.379+0000 7f1f6b7fe640 1 -- 172.21.15.181:0/1797528283 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v126) v1 ==== 129+0+0 (secure 0 0 0) 0x7f1f6c0433b0 con 0x7f1f74136d60 2024-09-15T03:33:52.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.379+0000 7f1f6b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:52.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.379+0000 7f1f6b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f54038740 msgr2=0x7f1f5403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f54038740 0x7f1f5403abf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009950 tx=0x7f1f64008040 comp rx=0 tx=0).stop 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 10 monclient: shutdown 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 msgr2=0x7f1f74099540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74099540 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009880 tx=0x7f1f6c036000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 shutdown_connections 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f54038740 0x7f1f5403abf0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f74136d60 0x7f1f74099540 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f7409f020 0x7f1f74141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 --2- 172.21.15.181:0/1797528283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7409ec50 0x7f1f74141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 >> 172.21.15.181:0/1797528283 conn(0x7f1f740a5d40 msgr2=0x7f1f7409d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 shutdown_connections 2024-09-15T03:33:52.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.383+0000 7f1f79834640 1 -- 172.21.15.181:0/1797528283 wait complete. 2024-09-15T03:33:52.396 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-15T03:33:52.396 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:33:52.397 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-15T03:33:52.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.445+0000 7f93de094640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.445+0000 7f93de094640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.445+0000 7f93de094640 10 monclient: monmap: 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:52.446386+0000 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:52.446386+0000 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.447 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 1 Processor -- start 2024-09-15T03:33:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 1 -- start start 2024-09-15T03:33:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 10 monclient: init 2024-09-15T03:33:52.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f93d80981c0 0x7f93d8098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 10 monclient(hunting): picked mon.noname-a con 0x7f93d80981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93d8131da0 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.446+0000 7f93de094640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93d8132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93de094640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d8098ad0 con 0x7f93d8132170 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93de094640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d8067f30 con 0x7f93d8131da0 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93de094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93de094640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d8131bd0 con 0x7f93d80981c0 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93de094640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93de094640 10 monclient(hunting): authenticate will time out at 1030.086047s 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f93d80981c0 0x7f93d8098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f93d80981c0 0x7f93d8098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44242/0 (socket says 172.21.15.153:44242) 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 1 -- 172.21.15.153:0/1774129581 learned_addr learned my addr 172.21.15.153:0/1774129581 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): get_auth_request con 0x7f93d80981c0 auth_method 0 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.447+0000 7f93d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d7fff640 1 -- 172.21.15.153:0/1774129581 >> v1:172.21.15.153:6789/0 conn(0x7f93d8132170 legacy=0x7f93d8067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d6ffd640 1 -- 172.21.15.153:0/1774129581 >> v1:172.21.15.181:6789/0 conn(0x7f93d8131da0 legacy=0x7f93d8067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 10 monclient(hunting): handle_auth_done global_id 84135 payload 274 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 1 -- 172.21.15.153:0/1774129581 >> v1:172.21.15.181:6789/0 conn(0x7f93d8131da0 legacy=0x7f93d8067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 1 -- 172.21.15.153:0/1774129581 >> v1:172.21.15.153:6789/0 conn(0x7f93d8132170 legacy=0x7f93d8067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.449+0000 7f93d77fe640 1 -- 172.21.15.153:0/1774129581 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d809ca60 con 0x7f93d80981c0 2024-09-15T03:33:52.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d77fe640 1 -- 172.21.15.153:0/1774129581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80981c0 msgr2=0x7f93d8098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d77fe640 1 --2- 172.21.15.153:0/1774129581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80981c0 0x7f93d8098590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009a10 tx=0x7f93c80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75545b58155c664 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 1 -- 172.21.15.153:0/1774129581 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c802f4b0 con 0x7f93d80981c0 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: dump: 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 1 -- 172.21.15.153:0/1774129581 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c802f610 con 0x7f93d80981c0 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 5 monclient: authenticate success, global_id 84135 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 1 -- 172.21.15.153:0/1774129581 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c802f9f0 con 0x7f93d80981c0 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 4 set_mon_vals no callback set 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93d67fc640 10 monclient: dump: 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.456 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 10 monclient: shutdown 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 1 -- 172.21.15.153:0/1774129581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80981c0 msgr2=0x7f93d8098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.452+0000 7f93de094640 1 --2- 172.21.15.153:0/1774129581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80981c0 0x7f93d8098590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009a10 tx=0x7f93c80091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 1 -- 172.21.15.153:0/1774129581 shutdown_connections 2024-09-15T03:33:52.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 1 --2- 172.21.15.153:0/1774129581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80981c0 0x7f93d8098590 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 1 -- 172.21.15.153:0/1774129581 >> 172.21.15.153:0/1774129581 conn(0x7f93d80a1e30 msgr2=0x7f93d80a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 1 -- 172.21.15.153:0/1774129581 shutdown_connections 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 1 -- 172.21.15.153:0/1774129581 wait complete. 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 10 monclient: monmap: 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.458 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.453+0000 7f93de094640 1 Processor -- start 2024-09-15T03:33:52.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 -- start start 2024-09-15T03:33:52.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient: init 2024-09-15T03:33:52.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d80981c0 0x7f93d811dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): picked mon.noname-b con 0x7f93d80981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d8131da0 0x7f93d811e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): picked mon.noname-c con 0x7f93d8131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 0x7f93d811ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): picked mon.noname-a con 0x7f93d8132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80a0a60 con 0x7f93d8132170 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d81216c0 con 0x7f93d80981c0 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d811f8b0 con 0x7f93d8131da0 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93de094640 10 monclient(hunting): authenticate will time out at 1030.093501s 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 0x7f93d811ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d8131da0 0x7f93d811e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 0x7f93d811ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44254/0 (socket says 172.21.15.153:44254) 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 learned_addr learned my addr 172.21.15.153:0/763614609 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): get_auth_request con 0x7f93d8132170 auth_method 0 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): get_auth_request con 0x7f93d8131da0 auth_method 0 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d80981c0 0x7f93d811dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): get_auth_request con 0x7f93d80981c0 auth_method 0 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient(hunting): handle_auth_done global_id 84138 payload 824 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d8131da0 msgr2=0x7f93d811e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d8131da0 0x7f93d811e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d80981c0 msgr2=0x7f93d811dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d80981c0 0x7f93d811dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.454+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c8009450 con 0x7f93d8132170 2024-09-15T03:33:52.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 msgr2=0x7f93d811ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93d7fff640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 0x7f93d811ea30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f93d000c840 tx=0x7f93d00018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0001950 con 0x7f93d8132170 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: dump: 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d000a640 con 0x7f93d8132170 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 5 monclient: authenticate success, global_id 84138 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 10 monclient: _renew_subs 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d81213e0 con 0x7f93d8132170 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d000aa00 con 0x7f93d8132170 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93bf7fe640 10 monclient: dump: 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 10 monclient: _renew_subs 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93d5ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.457+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d8120fd0 con 0x7f93d8132170 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f93d00070f0 con 0x7f93d8132170 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93de094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93bf7fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f93c4038740 0x7f93c403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93de094640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 0x7f93d811a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93de094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d8065f90 con 0x7f93d811ef70 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f93d004c540 con 0x7f93d8132170 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 0x7f93d811a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient: get_auth_request con 0x7f93d811ef70 auth_method 0 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d77fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f93c4038740 0x7f93c403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d77fe640 10 monclient: get_auth_request con 0x7f93c4038740 auth_method 0 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_done global_id 84141 payload 824 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.458+0000 7f93d6ffd640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93d8120620 con 0x7f93d811ef70 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93d6ffd640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 msgr2=0x7f93d811a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93d6ffd640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 0x7f93d811a4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00f780 tx=0x7f93cc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0072e0 con 0x7f93d811ef70 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93bf7fe640 10 monclient: dump: 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.472 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.460+0000 7f93d77fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f93c4038740 0x7f93c403abf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034000 tx=0x7f93c8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f93cc007440 con 0x7f93d811ef70 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 msgr2=0x7f93d811a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 0x7f93d811a4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00f780 tx=0x7f93cc009040 comp rx=0 tx=0).stop 2024-09-15T03:33:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 msgr2=0x7f93d811a4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:52.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.461+0000 7f93bf7fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811ef70 0x7f93d811a4a0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00f780 tx=0x7f93cc009040 comp rx=0 tx=0).stop 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93de094640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93de094640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 0x7f93d811ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93de094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d811f340 con 0x7f93d811a9e0 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93d6ffd640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 0x7f93d811ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93d6ffd640 10 monclient: get_auth_request con 0x7f93d811a9e0 auth_method 0 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.466+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.467+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.467+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.467+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.467+0000 7f93d6ffd640 10 monclient(hunting): handle_auth_done global_id 84144 payload 824 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.467+0000 7f93d6ffd640 1 -- 172.21.15.153:0/763614609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f93d804bf60 con 0x7f93d811a9e0 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93d6ffd640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 msgr2=0x7f93d811ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93d6ffd640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 0x7f93d811ade0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93cc017000 tx=0x7f93cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc021830 con 0x7f93d811a9e0 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 10 monclient: dump: 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.477 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f93cc0070d0 con 0x7f93d811a9e0 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 msgr2=0x7f93d811ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 0x7f93d811ade0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93cc017000 tx=0x7f93cc018000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 msgr2=0x7f93d811ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93bf7fe640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d811a9e0 0x7f93d811ade0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93cc017000 tx=0x7f93cc018000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 60, 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:52.479 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:52.480 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:52.481 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:52.482 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.483 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.484 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.485 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.486 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.487 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:33:52.489 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.490 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:52.490 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:52.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.469+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f93c4038740 msgr2=0x7f93c403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f93c4038740 0x7f93c403abf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034000 tx=0x7f93c8032000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 10 monclient: shutdown 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 msgr2=0x7f93d811ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 --2- 172.21.15.153:0/763614609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d8132170 0x7f93d811ea30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f93d000c840 tx=0x7f93d00018c0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 reap_dead start 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93d7fff640 1 -- 172.21.15.153:0/763614609 reap_dead start 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 shutdown_connections 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 >> 172.21.15.153:0/763614609 conn(0x7f93d80a1e30 msgr2=0x7f93d809b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 shutdown_connections 2024-09-15T03:33:52.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.470+0000 7f93de094640 1 -- 172.21.15.153:0/763614609 wait complete. 2024-09-15T03:33:52.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:33:52.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:33:52.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.478+0000 7f2eed99d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.478+0000 7f2eed99d640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.478+0000 7f2eed99d640 10 monclient: monmap: 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:52.478985+0000 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:52.478985+0000 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.478+0000 7f2eed99d640 1 Processor -- start 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.478+0000 7f2eed99d640 1 -- start start 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.478+0000 7f2eed99d640 10 monclient: init 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee809ec50 0x7f2ee8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2ee81321a0 0x7f2ee8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee81321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ee809f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee8067420 con 0x7f2ee809f020 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee809fdc0 con 0x7f2ee81321a0 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80a0510 con 0x7f2ee809ec50 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2eed99d640 10 monclient(hunting): authenticate will time out at 1036.960295s 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee77fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f2ee809f020 0x7f2ee8069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32910/0 (socket says 172.21.15.181:32910) 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2ee81321a0 0x7f2ee8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee77fe640 1 -- 172.21.15.181:0/3604973927 learned_addr learned my addr 172.21.15.181:0/3604973927 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): get_auth_request con 0x7f2ee81321a0 auth_method 0 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 1 --2- 172.21.15.181:0/3604973927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee809ec50 0x7f2ee8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request con 0x7f2ee809ec50 auth_method 0 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_done global_id 45205 payload 274 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 1 -- 172.21.15.181:0/3604973927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee809ec50 msgr2=0x7f2ee8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 1 --2- 172.21.15.181:0/3604973927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee809ec50 0x7f2ee8131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 1 -- 172.21.15.181:0/3604973927 >> v1:172.21.15.181:6789/0 conn(0x7f2ee809f020 legacy=0x7f2ee8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 1 monclient: found mon.noname-c 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.479+0000 7f2ede7fc640 1 -- 172.21.15.181:0/3604973927 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80a0200 con 0x7f2ee81321a0 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ede7fc640 1 -- 172.21.15.181:0/3604973927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee81321a0 msgr2=0x7f2ee8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ede7fc640 1 --2- 172.21.15.181:0/3604973927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee81321a0 0x7f2ee8068d00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009850 tx=0x7f2ed00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e1780e4d9c22a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 1 -- 172.21.15.181:0/3604973927 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed002adf0 con 0x7f2ee81321a0 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: dump: 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.506 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 1 -- 172.21.15.181:0/3604973927 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed0032440 con 0x7f2ee81321a0 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2eed99d640 5 monclient: authenticate success, global_id 45205 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2eed99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2eed99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 1 -- 172.21.15.181:0/3604973927 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00327e0 con 0x7f2ee81321a0 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2eed99d640 4 set_mon_vals no callback set 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.482+0000 7f2ee67fc640 10 monclient: dump: 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 10 monclient: shutdown 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 -- 172.21.15.181:0/3604973927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee81321a0 msgr2=0x7f2ee8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 --2- 172.21.15.181:0/3604973927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee81321a0 0x7f2ee8068d00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009850 tx=0x7f2ed00090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 -- 172.21.15.181:0/3604973927 shutdown_connections 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 --2- 172.21.15.181:0/3604973927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee81321a0 0x7f2ee8068d00 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 --2- 172.21.15.181:0/3604973927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee809ec50 0x7f2ee8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 -- 172.21.15.181:0/3604973927 >> 172.21.15.181:0/3604973927 conn(0x7f2ee80a5d40 msgr2=0x7f2ee80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 -- 172.21.15.181:0/3604973927 shutdown_connections 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 1 -- 172.21.15.181:0/3604973927 wait complete. 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.483+0000 7f2eed99d640 10 monclient: monmap: 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 Processor -- start 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 -- start start 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient: init 2024-09-15T03:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee809ec50 0x7f2ee809aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee809ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee809f020 0x7f2ee8130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee809f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee81321a0 0x7f2ee8131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee81321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee809dd60 con 0x7f2ee809f020 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee809b600 con 0x7f2ee809ec50 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee811dc70 con 0x7f2ee81321a0 2024-09-15T03:33:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2eed99d640 10 monclient(hunting): authenticate will time out at 1036.965822s 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee809ec50 0x7f2ee809aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee809ec50 0x7f2ee809aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47688/0 (socket says 172.21.15.181:47688) 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 1 -- 172.21.15.181:0/1177390825 learned_addr learned my addr 172.21.15.181:0/1177390825 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request con 0x7f2ee809ec50 auth_method 0 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.484+0000 7f2ee77fe640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee81321a0 0x7f2ee8131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): get_auth_request con 0x7f2ee81321a0 auth_method 0 2024-09-15T03:33:52.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee809f020 0x7f2ee8130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): get_auth_request con 0x7f2ee809f020 auth_method 0 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ede7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_done global_id 45208 payload 824 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 1 -- 172.21.15.181:0/1177390825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee81321a0 msgr2=0x7f2ee8131090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ee81321a0 0x7f2ee8131090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 1 -- 172.21.15.181:0/1177390825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee809f020 msgr2=0x7f2ee8130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee809f020 0x7f2ee8130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 1 monclient: found mon.noname-b 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.485+0000 7f2ee6ffd640 1 -- 172.21.15.181:0/1177390825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0009390 con 0x7f2ee809ec50 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2ee6ffd640 1 -- 172.21.15.181:0/1177390825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee809ec50 msgr2=0x7f2ee809aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2ee6ffd640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee809ec50 0x7f2ee809aac0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800ea20 tx=0x7f2ed80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8009600 con 0x7f2ee809ec50 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: dump: 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed800cc20 con 0x7f2ee809ec50 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 5 monclient: authenticate success, global_id 45208 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 10 monclient: _renew_subs 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8013470 con 0x7f2ee809ec50 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 1 -- 172.21.15.181:0/1177390825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee811df50 con 0x7f2ee809ec50 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2edf7fe640 10 monclient: dump: 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2ee5ffb640 4 set_mon_vals no callback set 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 10 monclient: _renew_subs 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.487+0000 7f2eed99d640 1 -- 172.21.15.181:0/1177390825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee811bc70 con 0x7f2ee809ec50 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2ed8013860 con 0x7f2ee809ec50 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2edf7fe640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2ec8038740 0x7f2ec803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f2ed804cc10 con 0x7f2ee809ec50 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2eed99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2eed99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2eed99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2eed99d640 1 -- 172.21.15.181:0/1177390825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee811bef0 con 0x7f2ee809ec50 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2ede7fc640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2ec8038740 0x7f2ec803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.488+0000 7f2ede7fc640 10 monclient: get_auth_request con 0x7f2ec8038740 auth_method 0 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.491+0000 7f2ede7fc640 1 --2- 172.21.15.181:0/1177390825 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2ec8038740 0x7f2ec803abf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80a0150 tx=0x7f2ed0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.493+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2ed8023510 con 0x7f2ee809ec50 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.493+0000 7f2edf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.493+0000 7f2edf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:52.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.588+0000 7f71bba4f640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.588+0000 7f71bba4f640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.588+0000 7f71bba4f640 10 monclient: monmap: 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:52.589627+0000 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:52.589627+0000 2024-09-15T03:33:52.590 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.589+0000 7f71bba4f640 1 Processor -- start 2024-09-15T03:33:52.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.589+0000 7f71bba4f640 1 -- start start 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.589+0000 7f71bba4f640 10 monclient: init 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71b409a510 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f71b4097370 0x7f71b41349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient(hunting): picked mon.noname-a con 0x7f71b4097370 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71b4097770 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b409cb00 con 0x7f71b409a510 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b409a8e0 con 0x7f71b4097770 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b406ed80 con 0x7f71b4097370 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71bba4f640 10 monclient(hunting): authenticate will time out at 1030.229512s 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f71b4097370 0x7f71b41349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f71b4097370 0x7f71b41349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44296/0 (socket says 172.21.15.153:44296) 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 1 -- 172.21.15.153:0/3067822832 learned_addr learned my addr 172.21.15.153:0/3067822832 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): get_auth_request con 0x7f71b4097370 auth_method 0 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b97eb640 1 --1- 172.21.15.153:0/3067822832 >> v1:172.21.15.153:6789/0 conn(0x7f71b409a510 0x7f71b4096c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37864/0 (socket says 172.21.15.153:37864) 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.590+0000 7f71b8fea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.592+0000 7f71b97eb640 1 -- 172.21.15.153:0/3067822832 >> v1:172.21.15.153:6789/0 conn(0x7f71b409a510 legacy=0x7f71b4096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.592+0000 7f71b9fec640 1 -- 172.21.15.153:0/3067822832 >> v1:172.21.15.181:6789/0 conn(0x7f71b4097770 legacy=0x7f71b4136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.592+0000 7f71b8fea640 10 monclient(hunting): handle_auth_done global_id 84150 payload 274 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 1 -- 172.21.15.153:0/3067822832 >> v1:172.21.15.181:6789/0 conn(0x7f71b4097770 legacy=0x7f71b4136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 1 -- 172.21.15.153:0/3067822832 >> v1:172.21.15.153:6789/0 conn(0x7f71b409a510 legacy=0x7f71b4096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.593+0000 7f71b8fea640 1 -- 172.21.15.153:0/3067822832 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b409c820 con 0x7f71b4097370 2024-09-15T03:33:52.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.595+0000 7f71b8fea640 1 -- 172.21.15.153:0/3067822832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097370 msgr2=0x7f71b41349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.595+0000 7f71b8fea640 1 --2- 172.21.15.153:0/3067822832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097370 0x7f71b41349a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009a40 tx=0x7f71a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c098614ba5d9185e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 1 -- 172.21.15.153:0/3067822832 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a402f4b0 con 0x7f71b4097370 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: dump: 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.601 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 1 -- 172.21.15.153:0/3067822832 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a402f610 con 0x7f71b4097370 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 5 monclient: authenticate success, global_id 84150 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 1 -- 172.21.15.153:0/3067822832 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a402f9f0 con 0x7f71b4097370 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 4 set_mon_vals no callback set 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71abfff640 10 monclient: dump: 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 10 monclient: shutdown 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 -- 172.21.15.153:0/3067822832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097370 msgr2=0x7f71b41349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 --2- 172.21.15.153:0/3067822832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097370 0x7f71b41349a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009a40 tx=0x7f71a40091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 -- 172.21.15.153:0/3067822832 shutdown_connections 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 --2- 172.21.15.153:0/3067822832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097370 0x7f71b41349a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 -- 172.21.15.153:0/3067822832 >> 172.21.15.153:0/3067822832 conn(0x7f71b40672b0 msgr2=0x7f71b40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 -- 172.21.15.153:0/3067822832 shutdown_connections 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.596+0000 7f71bba4f640 1 -- 172.21.15.153:0/3067822832 wait complete. 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.597+0000 7f71bba4f640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.597+0000 7f71bba4f640 10 monclient: monmap: 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.597+0000 7f71bba4f640 1 Processor -- start 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.597+0000 7f71bba4f640 1 -- start start 2024-09-15T03:33:52.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.597+0000 7f71bba4f640 10 monclient: init 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b4097370 0x7f71b413eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): picked mon.noname-b con 0x7f71b4097370 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 0x7f71b412fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): picked mon.noname-a con 0x7f71b4097770 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b409a510 0x7f71b41300f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): picked mon.noname-c con 0x7f71b409a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b411d290 con 0x7f71b4097770 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b406d990 con 0x7f71b4097370 2024-09-15T03:33:52.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b413fd80 con 0x7f71b409a510 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71bba4f640 10 monclient(hunting): authenticate will time out at 1030.237388s 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 0x7f71b412fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b409a510 0x7f71b41300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 0x7f71b412fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44308/0 (socket says 172.21.15.153:44308) 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 -- 172.21.15.153:0/4030842614 learned_addr learned my addr 172.21.15.153:0/4030842614 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): get_auth_request con 0x7f71b4097770 auth_method 0 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): get_auth_request con 0x7f71b409a510 auth_method 0 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b4097370 0x7f71b413eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): get_auth_request con 0x7f71b4097370 auth_method 0 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient(hunting): handle_auth_done global_id 84153 payload 824 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b409a510 msgr2=0x7f71b41300f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b409a510 0x7f71b41300f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b4097370 msgr2=0x7f71b413eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b4097370 0x7f71b413eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b8fea640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a4009450 con 0x7f71b4097770 2024-09-15T03:33:52.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.598+0000 7f71b9fec640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b409a510 0x7f71b41300f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:52.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71b8fea640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 msgr2=0x7f71b412fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71b8fea640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 0x7f71b412fbb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f71a4035040 tx=0x7f71a4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4032a30 con 0x7f71b4097770 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: dump: 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4032bd0 con 0x7f71b4097770 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 5 monclient: authenticate success, global_id 84153 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 10 monclient: _renew_subs 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b411bdd0 con 0x7f71b4097770 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a403d440 con 0x7f71b4097770 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71a97fa640 10 monclient: dump: 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71ab7fe640 4 set_mon_vals no callback set 2024-09-15T03:33:52.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 10 monclient: _renew_subs 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b411b1a0 con 0x7f71b4097770 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.601+0000 7f71bba4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f71a4039040 con 0x7f71b4097770 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71bba4f640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 0x7f71b4119640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71bba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b411d150 con 0x7f71b413f420 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 0x7f71b4119640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient: get_auth_request con 0x7f71b413f420 auth_method 0 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71a97fa640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7198038740 0x7f719803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f71a406bf30 con 0x7f71b4097770 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 10 monclient(hunting): handle_auth_done global_id 74270 payload 824 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b97eb640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71b411d4d0 con 0x7f71b413f420 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b9fec640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7198038740 0x7f719803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.602+0000 7f71b9fec640 10 monclient: get_auth_request con 0x7f7198038740 auth_method 0 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71b97eb640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 msgr2=0x7f71b4119640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71b97eb640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 0x7f71b4119640 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71b0004f70 tx=0x7f71b0007cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0012850 con 0x7f71b413f420 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 10 monclient: dump: 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.620 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71b9fec640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7198038740 0x7f719803abf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f71ac002230 tx=0x7f71ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f71b00129b0 con 0x7f71b413f420 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 msgr2=0x7f71b4119640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 0x7f71b4119640 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71b0004f70 tx=0x7f71b0007cc0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 msgr2=0x7f71b4119640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.604+0000 7f71a97fa640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b413f420 0x7f71b4119640 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71b0004f70 tx=0x7f71b0007cc0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71bba4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71bba4f640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 0x7f71b4119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71bba4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4065f20 con 0x7f71b4119b80 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 0x7f71b4119f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient: get_auth_request con 0x7f71b4119b80 auth_method 0 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 10 monclient(hunting): handle_auth_done global_id 74273 payload 824 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.610+0000 7f71b97eb640 1 -- 172.21.15.153:0/4030842614 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f71b411ce00 con 0x7f71b4119b80 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71b97eb640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 msgr2=0x7f71b4119f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71b97eb640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 0x7f71b4119f80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71b0017000 tx=0x7f71b0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0021830 con 0x7f71b4119b80 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:52.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 10 monclient: dump: 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f71b0021990 con 0x7f71b4119b80 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:52.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 msgr2=0x7f71b4119f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 0x7f71b4119f80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71b0017000 tx=0x7f71b0015000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 msgr2=0x7f71b4119f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71a97fa640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b4119b80 0x7f71b4119f80 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71b0017000 tx=0x7f71b0015000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:33:52.624 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 60, 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.625 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.626 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:52.627 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.628 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.629 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.630 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:52.632 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.633 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.634 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.635 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.636 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.636 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:52.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:52.638 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7198038740 msgr2=0x7f719803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71bba4f640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7198038740 0x7f719803abf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f71ac002230 tx=0x7f71ac009040 comp rx=0 tx=0).stop 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71bba4f640 10 monclient: shutdown 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71bba4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 msgr2=0x7f71b412fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71bba4f640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 0x7f71b412fbb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f71a4035040 tx=0x7f71a4032000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.613+0000 7f71b9fec640 1 -- 172.21.15.153:0/4030842614 reap_dead start 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.614+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 shutdown_connections 2024-09-15T03:33:52.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.614+0000 7f71bba4f640 1 --2- 172.21.15.153:0/4030842614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b4097770 0x7f71b412fbb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.614+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 >> 172.21.15.153:0/4030842614 conn(0x7f71b40672b0 msgr2=0x7f71b409fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.614+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 shutdown_connections 2024-09-15T03:33:52.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.614+0000 7f71bba4f640 1 -- 172.21.15.153:0/4030842614 wait complete. 2024-09-15T03:33:52.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:33:52.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:33:52.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:33:52.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.658+0000 7f2eed99d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-15T03:33:52.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.658+0000 7f2eed99d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-15T03:33:52.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.658+0000 7f2eed99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.658+0000 7f2eed99d640 1 -- 172.21.15.181:0/1177390825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) v1 -- 0x7f2ee80008d0 con 0x7f2ee809ec50 2024-09-15T03:33:52.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.659+0000 7f2edf7fe640 1 -- 172.21.15.181:0/1177390825 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v126) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2ed8023330 con 0x7f2ee809ec50 2024-09-15T03:33:52.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.659+0000 7f2edf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-15T03:33:52.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.659+0000 7f2edf7fe640 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-15T03:33:52.661 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-15T03:33:52.665 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:33:52.665 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-15T03:33:52.665 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -x 2024-09-15T03:33:52.665 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.736+0000 7ff85c020640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.736+0000 7ff85c020640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.736+0000 7ff85c020640 10 monclient: monmap: 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:52.736915+0000 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:52.736915+0000 2024-09-15T03:33:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.738 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.736+0000 7ff85c020640 1 Processor -- start 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 -- start start 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient: init 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 0x7ff854131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): picked mon.noname-b con 0x7ff8541318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff85409ea50 0x7ff85409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): picked mon.noname-c con 0x7ff85409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff854068d90 0x7ff8540691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): picked mon.noname-a con 0x7ff854068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff854131cc0 con 0x7ff854068d90 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85409fbf0 con 0x7ff85409ea50 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85409fe30 con 0x7ff8541318f0 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85c020640 10 monclient(hunting): authenticate will time out at 1030.376591s 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff854068d90 0x7ff8540691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 0x7ff854131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 0x7ff854131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45662/0 (socket says 172.21.15.153:45662) 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff854068d90 0x7ff8540691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44324/0 (socket says 172.21.15.153:44324) 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 1 -- 172.21.15.153:0/703320745 learned_addr learned my addr 172.21.15.153:0/703320745 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): get_auth_request con 0x7ff8541318f0 auth_method 0 2024-09-15T03:33:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): get_auth_request con 0x7ff854068d90 auth_method 0 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 1 --2- 172.21.15.153:0/703320745 >> v2:172.21.15.181:3300/0 conn(0x7ff85409ea50 0x7ff85409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff859dbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff85a5bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 10 monclient(hunting): get_auth_request con 0x7ff85409ea50 auth_method 0 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.737+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 10 monclient(hunting): handle_auth_done global_id 84156 payload 274 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 1 -- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 msgr2=0x7ff854131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 1 --2- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 0x7ff854131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 1 -- 172.21.15.153:0/703320745 >> v2:172.21.15.181:3300/0 conn(0x7ff85409ea50 msgr2=0x7ff85409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 1 --2- 172.21.15.153:0/703320745 >> v2:172.21.15.181:3300/0 conn(0x7ff85409ea50 0x7ff85409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff85a5bd640 1 -- 172.21.15.153:0/703320745 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540a1210 con 0x7ff854068d90 2024-09-15T03:33:52.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.738+0000 7ff859dbc640 1 --2- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 0x7ff854131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:52.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.740+0000 7ff85a5bd640 1 -- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff854068d90 msgr2=0x7ff8540691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.740+0000 7ff85a5bd640 1 --2- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff854068d90 0x7ff8540691f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff854068160 tx=0x7ff84c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d8677ecd86af6e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 1 -- 172.21.15.153:0/703320745 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c007760 con 0x7ff854068d90 2024-09-15T03:33:52.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: dump: 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 1 -- 172.21.15.153:0/703320745 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c02d050 con 0x7ff854068d90 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 5 monclient: authenticate success, global_id 84156 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 4 set_mon_vals no callback set 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 1 -- 172.21.15.153:0/703320745 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c007af0 con 0x7ff854068d90 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff858dba640 10 monclient: dump: 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 10 monclient: shutdown 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 -- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff854068d90 msgr2=0x7ff8540691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 --2- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff854068d90 0x7ff8540691f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff854068160 tx=0x7ff84c009780 comp rx=0 tx=0).stop 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 -- 172.21.15.153:0/703320745 shutdown_connections 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 --2- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff854068d90 0x7ff8540691f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 --2- 172.21.15.153:0/703320745 >> v2:172.21.15.181:3300/0 conn(0x7ff85409ea50 0x7ff85409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 --2- 172.21.15.153:0/703320745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8541318f0 0x7ff854131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 -- 172.21.15.153:0/703320745 >> 172.21.15.153:0/703320745 conn(0x7ff854067110 msgr2=0x7ff854067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 -- 172.21.15.153:0/703320745 shutdown_connections 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.741+0000 7ff85c020640 1 -- 172.21.15.153:0/703320745 wait complete. 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.742+0000 7ff85c020640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.742+0000 7ff85c020640 10 monclient: monmap: 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.766 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.742+0000 7ff85c020640 1 Processor -- start 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.742+0000 7ff85c020640 1 -- start start 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.742+0000 7ff85c020640 10 monclient: init 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.742+0000 7ff85c020640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 0x7ff85409a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): picked mon.noname-c con 0x7ff854068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff85409ea50 0x7ff854130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): picked mon.noname-a con 0x7ff85409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8541318f0 0x7ff85411b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): picked mon.noname-b con 0x7ff8541318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85409e470 con 0x7ff85409ea50 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854141a30 con 0x7ff8541318f0 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85411de30 con 0x7ff854068d90 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff85c020640 10 monclient(hunting): authenticate will time out at 1030.382097s 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 0x7ff85409a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff85409ea50 0x7ff854130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 0x7ff85409a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45668/0 (socket says 172.21.15.153:45668) 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 -- 172.21.15.153:0/923680467 learned_addr learned my addr 172.21.15.153:0/923680467 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): get_auth_request con 0x7ff854068d90 auth_method 0 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): get_auth_request con 0x7ff85409ea50 auth_method 0 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient(hunting): handle_auth_done global_id 74282 payload 824 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8541318f0 msgr2=0x7ff85411b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8541318f0 0x7ff85411b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff85409ea50 msgr2=0x7ff854130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff85409ea50 0x7ff854130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 monclient: found mon.noname-c 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff859dbc640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850009450 con 0x7ff854068d90 2024-09-15T03:33:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.743+0000 7ff8595bb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff85409ea50 0x7ff854130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff859dbc640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 msgr2=0x7ff85409a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff859dbc640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 0x7ff85409a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff85000ec80 tx=0x7ff8500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850009730 con 0x7ff854068d90 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: dump: 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85000eec0 con 0x7ff854068d90 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff85c020640 5 monclient: authenticate success, global_id 74282 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850013470 con 0x7ff854068d90 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff849ffb640 10 monclient: dump: 2024-09-15T03:33:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff85c020640 10 monclient: _renew_subs 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff85c020640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c009a20 con 0x7ff854068d90 2024-09-15T03:33:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.745+0000 7ff84bfff640 4 set_mon_vals no callback set 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 10 monclient: _renew_subs 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff854141030 con 0x7ff854068d90 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff850013820 con 0x7ff854068d90 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 0x7ff85411bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.746+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85411dcf0 con 0x7ff854130f70 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff849ffb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff838038740 0x7ff83803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 0x7ff85411bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7ff85004ccc0 con 0x7ff854068d90 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient: get_auth_request con 0x7ff854130f70 auth_method 0 2024-09-15T03:33:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff85a5bd640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff838038740 0x7ff83803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff85a5bd640 10 monclient: get_auth_request con 0x7ff838038740 auth_method 0 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 10 monclient(hunting): handle_auth_done global_id 45226 payload 824 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.747+0000 7ff8595bb640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff85411dff0 con 0x7ff854130f70 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff8595bb640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 msgr2=0x7ff85411bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff8595bb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 0x7ff85411bc60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff844009f30 tx=0x7ff8440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff85a5bd640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff838038740 0x7ff83803abf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff854064050 tx=0x7ff84c037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844014860 con 0x7ff854130f70 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff849ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.749+0000 7ff849ffb640 10 monclient: dump: 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.776 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff8440149c0 con 0x7ff854130f70 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 msgr2=0x7ff85411bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 0x7ff85411bc60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff844009f30 tx=0x7ff8440090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 msgr2=0x7ff85411bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.750+0000 7ff849ffb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff854130f70 0x7ff85411bc60 secure :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff844009f30 tx=0x7ff8440090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.755+0000 7ff85c020640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.755+0000 7ff85c020640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 0x7ff854096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.755+0000 7ff85c020640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.755+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854065e00 con 0x7ff85411c1a0 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.755+0000 7ff8595bb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 0x7ff854096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient: get_auth_request con 0x7ff85411c1a0 auth_method 0 2024-09-15T03:33:52.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 10 monclient(hunting): handle_auth_done global_id 45229 payload 824 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.756+0000 7ff8595bb640 1 -- 172.21.15.153:0/923680467 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff85411c5a0 con 0x7ff85411c1a0 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff8595bb640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 msgr2=0x7ff854096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff8595bb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 0x7ff854096a60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff844009f30 tx=0x7ff84401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844024040 con 0x7ff85411c1a0 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 10 monclient: dump: 2024-09-15T03:33:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7ff8440133a0 con 0x7ff85411c1a0 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 msgr2=0x7ff854096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 0x7ff854096a60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff844009f30 tx=0x7ff84401c000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.758+0000 7ff849ffb640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 msgr2=0x7ff854096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff849ffb640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff85411c1a0 0x7ff854096a60 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff844009f30 tx=0x7ff84401c000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:52.780 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:33:52.781 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 60, 2024-09-15T03:33:52.781 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:52.785 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:33:52.785 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:52.785 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.786 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.787 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:52.789 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.791 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:52.794 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:52.795 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.796 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:52.797 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:52.798 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.740+0000 7f633af09640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.740+0000 7f633af09640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.740+0000 7f633af09640 10 monclient: monmap: 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:52.741147+0000 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:52.741147+0000 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.800 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 Processor -- start 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 -- start start 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient: init 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f633409d140 0x7f633409d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): picked mon.noname-b con 0x7f633409d140 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f633409da50 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:52.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6334094790 0x7f6334137d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): picked mon.noname-c con 0x7f6334094790 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6334098570 con 0x7f633409da50 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63340670b0 con 0x7f6334094790 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6334097fb0 con 0x7f633409d140 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f633af09640 10 monclient(hunting): authenticate will time out at 1037.222601s 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6334094790 0x7f6334137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6333fff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f633409da50 0x7f6334094080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32912/0 (socket says 172.21.15.181:32912) 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6333fff640 1 -- 172.21.15.181:0/3399108640 learned_addr learned my addr 172.21.15.181:0/3399108640 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): get_auth_request con 0x7f6334094790 auth_method 0 2024-09-15T03:33:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 1 --2- 172.21.15.181:0/3399108640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f633409d140 0x7f633409d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): get_auth_request con 0x7f633409d140 auth_method 0 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f6338ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient(hunting): handle_auth_done global_id 45220 payload 274 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 1 -- 172.21.15.181:0/3399108640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f633409d140 msgr2=0x7f633409d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 1 --2- 172.21.15.181:0/3399108640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f633409d140 0x7f633409d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 1 -- 172.21.15.181:0/3399108640 >> v1:172.21.15.181:6789/0 conn(0x7f633409da50 legacy=0x7f6334094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 1 monclient: found mon.noname-c 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.741+0000 7f63394a6640 1 -- 172.21.15.181:0/3399108640 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334064cb0 con 0x7f6334094790 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63394a6640 1 -- 172.21.15.181:0/3399108640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6334094790 msgr2=0x7f6334137d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63394a6640 1 --2- 172.21.15.181:0/3399108640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6334094790 0x7f6334137d10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f632c009e60 tx=0x7f632c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=782cd2f3ace99a0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 1 -- 172.21.15.181:0/3399108640 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c007760 con 0x7f6334094790 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:52.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: dump: 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 1 -- 172.21.15.181:0/3399108640 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c02d050 con 0x7f6334094790 2024-09-15T03:33:52.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f633af09640 5 monclient: authenticate success, global_id 45220 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f633af09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f633af09640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 1 -- 172.21.15.181:0/3399108640 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c007af0 con 0x7f6334094790 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f633af09640 4 set_mon_vals no callback set 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.744+0000 7f63337fe640 10 monclient: dump: 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.806 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 10 monclient: shutdown 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 -- 172.21.15.181:0/3399108640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6334094790 msgr2=0x7f6334137d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 --2- 172.21.15.181:0/3399108640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6334094790 0x7f6334137d10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f632c009e60 tx=0x7f632c009780 comp rx=0 tx=0).stop 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 -- 172.21.15.181:0/3399108640 shutdown_connections 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 --2- 172.21.15.181:0/3399108640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6334094790 0x7f6334137d10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 --2- 172.21.15.181:0/3399108640 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f633409d140 0x7f633409d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 -- 172.21.15.181:0/3399108640 >> 172.21.15.181:0/3399108640 conn(0x7f63340a2190 msgr2=0x7f633409e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 -- 172.21.15.181:0/3399108640 shutdown_connections 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 1 -- 172.21.15.181:0/3399108640 wait complete. 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.745+0000 7f633af09640 10 monclient: monmap: 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.808 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.809 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 Processor -- start 2024-09-15T03:33:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 -- start start 2024-09-15T03:33:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient: init 2024-09-15T03:33:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6334094790 0x7f633413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): picked mon.noname-c con 0x7f6334094790 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f633409d140 0x7f63340934c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): picked mon.noname-b con 0x7f633409d140 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f633409da50 0x7f633408a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): picked mon.noname-a con 0x7f633409da50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633408bc80 con 0x7f633409da50 2024-09-15T03:33:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633409baf0 con 0x7f633409d140 2024-09-15T03:33:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63340a0f30 con 0x7f6334094790 2024-09-15T03:33:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f633af09640 10 monclient(hunting): authenticate will time out at 1037.227847s 2024-09-15T03:33:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f633409d140 0x7f63340934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f633409d140 0x7f63340934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47710/0 (socket says 172.21.15.181:47710) 2024-09-15T03:33:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 1 -- 172.21.15.181:0/2320785409 learned_addr learned my addr 172.21.15.181:0/2320785409 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 10 monclient(hunting): get_auth_request con 0x7f633409d140 auth_method 0 2024-09-15T03:33:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6333fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.746+0000 7f6338ca5640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6334094790 0x7f633413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f633409da50 0x7f633408a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): get_auth_request con 0x7f6334094790 auth_method 0 2024-09-15T03:33:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): get_auth_request con 0x7f633409da50 auth_method 0 2024-09-15T03:33:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6338ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f63394a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 10 monclient(hunting): handle_auth_done global_id 45223 payload 824 2024-09-15T03:33:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 1 -- 172.21.15.181:0/2320785409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6334094790 msgr2=0x7f633413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6334094790 0x7f633413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 1 -- 172.21.15.181:0/2320785409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f633409da50 msgr2=0x7f633408a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f633409da50 0x7f633408a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 1 monclient: found mon.noname-b 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.747+0000 7f6333fff640 1 -- 172.21.15.181:0/2320785409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c009a20 con 0x7f633409d140 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6333fff640 1 -- 172.21.15.181:0/2320785409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f633409d140 msgr2=0x7f63340934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6333fff640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f633409d140 0x7f63340934c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f632000c610 tx=0x7f6320004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320005cb0 con 0x7f633409d140 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: dump: 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.819 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320005e30 con 0x7f633409d140 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 5 monclient: authenticate success, global_id 45223 2024-09-15T03:33:52.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 10 monclient: _renew_subs 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 1 -- 172.21.15.181:0/2320785409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633408d6b0 con 0x7f633409d140 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320007320 con 0x7f633409d140 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6330ff9640 10 monclient: dump: 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.821 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 10 monclient: _renew_subs 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f633af09640 1 -- 172.21.15.181:0/2320785409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633413f1c0 con 0x7f633409d140 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.749+0000 7f6332ffd640 4 set_mon_vals no callback set 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f632001b3b0 con 0x7f633409d140 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f633af09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f633af09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f633af09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f633af09640 1 -- 172.21.15.181:0/2320785409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633413f4a0 con 0x7f633409d140 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f6330ff9640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6318038740 0x7f631803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f632004bd40 con 0x7f633409d140 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f6338ca5640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6318038740 0x7f631803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.750+0000 7f6338ca5640 10 monclient: get_auth_request con 0x7f6318038740 auth_method 0 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.753+0000 7f6338ca5640 1 --2- 172.21.15.181:0/2320785409 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6318038740 0x7f631803abf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6328009d20 tx=0x7f63280090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.755+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6320007710 con 0x7f633409d140 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.755+0000 7f6330ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.755+0000 7f6330ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff838038740 msgr2=0x7ff83803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff838038740 0x7ff83803abf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff854064050 tx=0x7ff84c037000 comp rx=0 tx=0).stop 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 10 monclient: shutdown 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 msgr2=0x7ff85409a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 --2- 172.21.15.153:0/923680467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff854068d90 0x7ff85409a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff85000ec80 tx=0x7ff8500091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85a5bd640 1 -- 172.21.15.153:0/923680467 reap_dead start 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85a5bd640 1 -- 172.21.15.153:0/923680467 reap_dead start 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 shutdown_connections 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 >> 172.21.15.153:0/923680467 conn(0x7ff854067110 msgr2=0x7ff854134790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 shutdown_connections 2024-09-15T03:33:52.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.759+0000 7ff85c020640 1 -- 172.21.15.153:0/923680467 wait complete. 2024-09-15T03:33:52.825 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:33:52.825 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:33:52.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:33:52.917 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.916+0000 7f633af09640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:52.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.916+0000 7f633af09640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:52.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.916+0000 7f633af09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:52.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.916+0000 7f633af09640 1 -- 172.21.15.181:0/2320785409 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f63340008d0 con 0x7f633409d140 2024-09-15T03:33:52.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.917+0000 7f6330ff9640 1 -- 172.21.15.181:0/2320785409 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v126) v1 ==== 163+0+0 (secure 0 0 0) 0x7f6320017070 con 0x7f633409d140 2024-09-15T03:33:52.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.917+0000 7f6330ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-15T03:33:52.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.917+0000 7f6330ff9640 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-15T03:33:52.919 INFO:tasks.workunit.client.0.smithi181.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-15T03:33:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.917+0000 7f43b999d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.917+0000 7f43b999d640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.917+0000 7f43b999d640 10 monclient: monmap: 2024-09-15T03:33:52.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:52.918439+0000 2024-09-15T03:33:52.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:52.918439+0000 2024-09-15T03:33:52.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 Processor -- start 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 -- start start 2024-09-15T03:33:52.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient: init 2024-09-15T03:33:52.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T03:33:52.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43b41318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:52.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f43b409ea50 0x7f43b409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): picked mon.noname-a con 0x7f43b409ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:52.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): picked mon.noname-b con 0x7f43b4136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b4132640 con 0x7f43b41318f0 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b4131cc0 con 0x7f43b409ea50 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b409fbf0 con 0x7f43b4136c20 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b999d640 10 monclient(hunting): authenticate will time out at 1030.557813s 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b27fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f43b409ea50 0x7f43b409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f43b41318f0 0x7f43b4131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37876/0 (socket says 172.21.15.153:37876) 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b27fc640 10 monclient(hunting): get_auth_request con 0x7f43b409ea50 auth_method 0 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b2ffd640 1 -- 172.21.15.153:0/903770376 learned_addr learned my addr 172.21.15.153:0/903770376 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b37fe640 10 monclient(hunting): get_auth_request con 0x7f43b4136c20 auth_method 0 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.918+0000 7f43b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 10 monclient(hunting): handle_auth_done global_id 84165 payload 274 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 1 -- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 msgr2=0x7f43b4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 1 --2- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 1 -- 172.21.15.153:0/903770376 >> v1:172.21.15.153:6789/0 conn(0x7f43b41318f0 legacy=0x7f43b4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b27fc640 1 -- 172.21.15.153:0/903770376 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b409fe30 con 0x7f43b409ea50 2024-09-15T03:33:52.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.919+0000 7f43b37fe640 1 --2- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:52.931 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:33:52.931 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-15T03:33:52.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.921+0000 7f43b27fc640 1 -- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b409ea50 msgr2=0x7f43b409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.921+0000 7f43b27fc640 1 --2- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b409ea50 0x7f43b409ee20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009850 tx=0x7f43a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddd7d61f932d865 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 1 -- 172.21.15.153:0/903770376 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a802adf0 con 0x7f43b409ea50 2024-09-15T03:33:52.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: dump: 2024-09-15T03:33:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.939 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 1 -- 172.21.15.153:0/903770376 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8032440 con 0x7f43b409ea50 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 5 monclient: authenticate success, global_id 84165 2024-09-15T03:33:52.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 1 -- 172.21.15.153:0/903770376 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a80327e0 con 0x7f43b409ea50 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 4 set_mon_vals no callback set 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b1ffb640 10 monclient: dump: 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 10 monclient: shutdown 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 -- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b409ea50 msgr2=0x7f43b409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 --2- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b409ea50 0x7f43b409ee20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009850 tx=0x7f43a80090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 -- 172.21.15.153:0/903770376 shutdown_connections 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 --2- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 --2- 172.21.15.153:0/903770376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b409ea50 0x7f43b409ee20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 -- 172.21.15.153:0/903770376 >> 172.21.15.153:0/903770376 conn(0x7f43b4067110 msgr2=0x7f43b4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 -- 172.21.15.153:0/903770376 shutdown_connections 2024-09-15T03:33:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.922+0000 7f43b999d640 1 -- 172.21.15.153:0/903770376 wait complete. 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.923+0000 7f43b999d640 10 monclient: build_initial_monmap 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.923+0000 7f43b999d640 10 monclient: monmap: 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.923+0000 7f43b999d640 1 Processor -- start 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.923+0000 7f43b999d640 1 -- start start 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.923+0000 7f43b999d640 10 monclient: init 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43b409ea50 0x7f43b41308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): picked mon.noname-b con 0x7f43b409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 0x7f43b4099750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): picked mon.noname-a con 0x7f43b41318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4099c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): picked mon.noname-c con 0x7f43b4136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b411c9c0 con 0x7f43b41318f0 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b409bf10 con 0x7f43b409ea50 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b409dfb0 con 0x7f43b4136c20 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b999d640 10 monclient(hunting): authenticate will time out at 1030.563417s 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4099c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 0x7f43b4099750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4099c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45688/0 (socket says 172.21.15.153:45688) 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 1 -- 172.21.15.153:0/2893430193 learned_addr learned my addr 172.21.15.153:0/2893430193 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): get_auth_request con 0x7f43b4136c20 auth_method 0 2024-09-15T03:33:52.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): get_auth_request con 0x7f43b41318f0 auth_method 0 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43b409ea50 0x7f43b41308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): get_auth_request con 0x7f43b409ea50 auth_method 0 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient(hunting): handle_auth_done global_id 84168 payload 824 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 -- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 msgr2=0x7f43b4099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4099c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 -- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43b409ea50 msgr2=0x7f43b41308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43b409ea50 0x7f43b41308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:52.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 monclient: found mon.noname-a 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b27fc640 1 -- 172.21.15.153:0/2893430193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac009a20 con 0x7f43b41318f0 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.924+0000 7f43b37fe640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4099c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b27fc640 1 -- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 msgr2=0x7f43b4099750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b27fc640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 0x7f43b4099750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43a8031000 tx=0x7f43a8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8031ae0 con 0x7f43b41318f0 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: dump: 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.948 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: _finish_auth 0 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8031c40 con 0x7f43b41318f0 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 5 monclient: authenticate success, global_id 84168 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 10 monclient: _renew_subs 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a8009390 con 0x7f43b41318f0 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a803d440 con 0x7f43b41318f0 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f439affd640 10 monclient: dump: 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:52.950 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 10 monclient: _renew_subs 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b411bad0 con 0x7f43b41318f0 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.927+0000 7f43b17fa640 4 set_mon_vals no callback set 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f43a8039040 con 0x7f43b41318f0 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f43b999d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f43b999d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f43b999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404bf60 con 0x7f43b41318f0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f439affd640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43a0038790 0x7f43a003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f43a806c050 con 0x7f43b41318f0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f43b2ffd640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43a0038790 0x7f43a003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.928+0000 7f43b2ffd640 10 monclient: get_auth_request con 0x7f43a0038790 auth_method 0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.931+0000 7f43b2ffd640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43a0038790 0x7f43a003ac40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f43a4007fd0 tx=0x7f43a4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.933+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f43a809de40 con 0x7f43b41318f0 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.933+0000 7f439affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:52.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:52.933+0000 7f439affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:53.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.999+0000 7f218a6a8640 10 monclient: get_monmap_and_config 2024-09-15T03:33:53.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.999+0000 7f218a6a8640 10 monclient: build_initial_monmap 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:52.999+0000 7f218a6a8640 10 monclient: monmap: 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:53.000593+0000 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:53.000593+0000 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:53.001 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:53.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.000+0000 7f218a6a8640 1 Processor -- start 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.000+0000 7f218a6a8640 1 -- start start 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.000+0000 7f218a6a8640 10 monclient: init 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21840a0ad0 0x7f21840691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f21840a0ad0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f218409ee60 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f218409f260 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218406a5b0 con 0x7f218409f260 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21840a0460 con 0x7f218409ee60 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:53.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21840a0ea0 con 0x7f21840a0ad0 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f218a6a8640 10 monclient(hunting): authenticate will time out at 1037.482174s 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21840a0ad0 0x7f21840691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f21837fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f218409ee60 0x7f2184131ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32926/0 (socket says 172.21.15.181:32926) 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f21837fe640 1 -- 172.21.15.181:0/3297117458 learned_addr learned my addr 172.21.15.181:0/3297117458 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): get_auth_request con 0x7f21840a0ad0 auth_method 0 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient(hunting): handle_auth_done global_id 45235 payload 274 2024-09-15T03:33:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 1 -- 172.21.15.181:0/3297117458 >> v1:172.21.15.181:6789/0 conn(0x7f218409ee60 legacy=0x7f2184131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:53.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 1 -- 172.21.15.181:0/3297117458 >> v1:172.21.15.153:6789/0 conn(0x7f218409f260 legacy=0x7f21841321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:53.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:53.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 1 monclient: found mon.noname-c 2024-09-15T03:33:53.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:53.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.001+0000 7f2183fff640 1 -- 172.21.15.181:0/3297117458 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840a1bb0 con 0x7f21840a0ad0 2024-09-15T03:33:53.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2183fff640 1 -- 172.21.15.181:0/3297117458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21840a0ad0 msgr2=0x7f21840691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:53.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2183fff640 1 --2- 172.21.15.181:0/3297117458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21840a0ad0 0x7f21840691d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2174009850 tx=0x7f21740090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=401759f579c9a828 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:53.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 1 -- 172.21.15.181:0/3297117458 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174032440 con 0x7f21840a0ad0 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: dump: 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:53.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: _finish_auth 0 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 1 -- 172.21.15.181:0/3297117458 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21740325a0 con 0x7f21840a0ad0 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f218a6a8640 5 monclient: authenticate success, global_id 45235 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f218a6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f218a6a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 1 -- 172.21.15.181:0/3297117458 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174032940 con 0x7f21840a0ad0 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f218a6a8640 4 set_mon_vals no callback set 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.004+0000 7f2182ffd640 10 monclient: dump: 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:53.014 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 10 monclient: shutdown 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 -- 172.21.15.181:0/3297117458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21840a0ad0 msgr2=0x7f21840691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3297117458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21840a0ad0 0x7f21840691d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2174009850 tx=0x7f21740090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 -- 172.21.15.181:0/3297117458 shutdown_connections 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3297117458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21840a0ad0 0x7f21840691d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 -- 172.21.15.181:0/3297117458 >> 172.21.15.181:0/3297117458 conn(0x7f2184071370 msgr2=0x7f21840a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 -- 172.21.15.181:0/3297117458 shutdown_connections 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 1 -- 172.21.15.181:0/3297117458 wait complete. 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 10 monclient: build_initial_monmap 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.005+0000 7f218a6a8640 10 monclient: monmap: 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:53.016 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 Processor -- start 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 -- start start 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient: init 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f218409ee60 0x7f218411bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f218409ee60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 0x7f218409a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f218409f260 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21840a0ad0 0x7f2184130cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f21840a0ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:53.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218411e3b0 con 0x7f21840a0ad0 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218411cf00 con 0x7f218409f260 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218411cab0 con 0x7f218409ee60 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f218a6a8640 10 monclient(hunting): authenticate will time out at 1037.487830s 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 0x7f218409a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 0x7f218409a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47730/0 (socket says 172.21.15.181:47730) 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 1 -- 172.21.15.181:0/3936077901 learned_addr learned my addr 172.21.15.181:0/3936077901 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): get_auth_request con 0x7f218409f260 auth_method 0 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:53.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.006+0000 7f21837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2183fff640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f218409ee60 0x7f218411bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2188c45640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21840a0ad0 0x7f2184130cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2183fff640 10 monclient(hunting): get_auth_request con 0x7f218409ee60 auth_method 0 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2183fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2188c45640 10 monclient(hunting): get_auth_request con 0x7f21840a0ad0 auth_method 0 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2188c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2188c45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:53.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2188c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 10 monclient(hunting): handle_auth_done global_id 45238 payload 824 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 1 -- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f218409ee60 msgr2=0x7f218411bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f218409ee60 0x7f218411bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 1 -- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21840a0ad0 msgr2=0x7f2184130cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21840a0ad0 0x7f2184130cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 10 monclient: _finish_hunting 0 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 1 monclient: found mon.noname-b 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f21837fe640 1 -- 172.21.15.181:0/3936077901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2174009390 con 0x7f218409f260 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2183fff640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f218409ee60 0x7f218411bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.007+0000 7f2188c45640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21840a0ad0 0x7f2184130cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:33:53.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f21837fe640 1 -- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 msgr2=0x7f218409a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f21837fe640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 0x7f218409a9a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f217800c5e0 tx=0x7f2178004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178005cb0 con 0x7f218409f260 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: dump: 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:53.021 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: _finish_auth 0 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2178005e30 con 0x7f218409f260 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 5 monclient: authenticate success, global_id 45238 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 10 monclient: _renew_subs 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218411e0d0 con 0x7f218409f260 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178007340 con 0x7f218409f260 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f216bfff640 10 monclient: dump: 2024-09-15T03:33:53.022 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 10 monclient: _renew_subs 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f21827fc640 4 set_mon_vals no callback set 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.009+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2184131210 con 0x7f218409f260 2024-09-15T03:33:53.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f217801b400 con 0x7f218409f260 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f216bfff640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2170038770 0x7f217003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(126..126 src has 40..126) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f217804c3c0 con 0x7f218409f260 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f218a6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f218a6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f218a6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f2183fff640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2170038770 0x7f217003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21840008d0 con 0x7f218409f260 2024-09-15T03:33:53.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.010+0000 7f2183fff640 10 monclient: get_auth_request con 0x7f2170038770 auth_method 0 2024-09-15T03:33:53.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.013+0000 7f2183fff640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2170038770 0x7f217003ac20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f21840675c0 tx=0x7f2174032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:53.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.015+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f217801ab40 con 0x7f218409f260 2024-09-15T03:33:53.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.015+0000 7f216bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:53.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.015+0000 7f216bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.148+0000 7f43b999d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:33:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.148+0000 7f43b999d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:33:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.148+0000 7f43b999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:53.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.148+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f43b4141e60 con 0x7f43b41318f0 2024-09-15T03:33:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f439affd640 1 -- 172.21.15.153:0/2893430193 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f43a8037080 con 0x7f43b41318f0 2024-09-15T03:33:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f439affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:33:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f439affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:53.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=3743544865,config=3140514009,health=2864126176,kv=3235807718,logm=1986960227}) 2024-09-15T03:33:53.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=10,kv=7,logm=66} crc {auth=3716191395,config=3140514009,health=2657470472,kv=2145712985,logm=495369516}) 2024-09-15T03:33:53.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=3743544865,config=3140514009,health=2864126176,kv=3235807718,logm=1986960227}) 2024-09-15T03:33:53.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.149+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=1,health=12,kv=7,logm=66} crc {auth=3297601659,config=1228700967,health=2864126176,kv=1382144082,logm=2744644761}) 2024-09-15T03:33:53.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.150+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.150+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085140080}) 2024-09-15T03:33:53.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.150+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268432142}) 2024-09-15T03:33:53.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.150+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085140080}) 2024-09-15T03:33:53.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387269286}) 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43a0038790 msgr2=0x7f43a003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43a0038790 0x7f43a003ac40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f43a4007fd0 tx=0x7f43a4007f00 comp rx=0 tx=0).stop 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 10 monclient: shutdown 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 msgr2=0x7f43b4099750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 0x7f43b4099750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43a8031000 tx=0x7f43a8032000 comp rx=0 tx=0).stop 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 shutdown_connections 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43a0038790 0x7f43a003ac40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43b4136c20 0x7f43b4099c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43b41318f0 0x7f43b4099750 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 --2- 172.21.15.153:0/2893430193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43b409ea50 0x7f43b41308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 >> 172.21.15.153:0/2893430193 conn(0x7f43b4067110 msgr2=0x7f43b40a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:53.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 shutdown_connections 2024-09-15T03:33:53.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f43b999d640 1 -- 172.21.15.153:0/2893430193 wait complete. 2024-09-15T03:33:53.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822811773}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=125167374}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.151+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822811773}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2824339387}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800323489}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675476985}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800323489}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.152+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=627303565}) 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.153+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.153+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848487339}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.153+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953653367}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.153+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.153+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848487339}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.153+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225942779}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.154+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.154+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047652233}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.154+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=140645327}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.154+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.154+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047652233}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.154+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855245065}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.155+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.155+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022752132}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.155+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2678967350}) 2024-09-15T03:33:53.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.155+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.155+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022752132}) 2024-09-15T03:33:53.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.155+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2776389773}) 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.156+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.156+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004805355}) 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.156+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745317850}) 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.156+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.156+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004805355}) 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.156+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2194814942}) 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.157+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.157+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642360234}) 2024-09-15T03:33:53.162 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.157+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733757960}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.157+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.157+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642360234}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.157+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4067827888}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.158+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.158+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4112806887}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.158+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2810803661}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.158+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.158+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4112806887}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.158+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186818378}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.159+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.159+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67604293}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.159+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134580163}) 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.159+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.163 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.159+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67604293}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.159+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922044295}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.160+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.160+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019117972}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.160+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889796374}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.160+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.160+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019117972}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.160+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965195982}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.161+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.161+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204930425}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.161+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005332853}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.161+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.161+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204930425}) 2024-09-15T03:33:53.164 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.161+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309385814}) 2024-09-15T03:33:53.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.162+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.162+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1956441014}) 2024-09-15T03:33:53.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.162+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575733724}) 2024-09-15T03:33:53.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.162+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.162+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1956441014}) 2024-09-15T03:33:53.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.162+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159870398}) 2024-09-15T03:33:53.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910669972}) 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2035742382}) 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910669972}) 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025608404}) 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146302980}) 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1984397429}) 2024-09-15T03:33:53.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146302980}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.163+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538398151}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.164+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.164+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644313668}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.164+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961433757}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.164+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.164+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644313668}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.164+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1923629070}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.165+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.165+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959009196}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.165+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839718454}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.165+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.165+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959009196}) 2024-09-15T03:33:53.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.165+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134136388}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.166+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.166+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2648988104}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.166+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032410265}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.166+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.166+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2648988104}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.166+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777779167}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.167+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.167+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158478174}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.167+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1324756942}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.167+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.167+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158478174}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.167+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=2,mgr=2} crc {logm=845808286,mdsmap=3585761405,mgr=2666571515}) 2024-09-15T03:33:53.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.168+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.168+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=63} crc {mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=584889315,mon_config_key=4117641549,monmap=244636553,osd_metadata=2347033642,osd_pg_creating=3450098465,osd_snap=454132730,osdmap=411202068}) 2024-09-15T03:33:53.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.168+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=62} crc {mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1955124759,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=3450098465,osd_snap=454132730,osdmap=3784356255}) 2024-09-15T03:33:53.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.168+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.168+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=63} crc {mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=584889315,mon_config_key=4117641549,monmap=244636553,osd_metadata=2347033642,osd_pg_creating=3450098465,osd_snap=454132730,osdmap=411202068}) 2024-09-15T03:33:53.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.168+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=2,osdmap=65} crc {mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3563932354,mon_config_key=4117641549,monmap=244636553,osd_metadata=2347033642,osd_snap=454132730,osdmap=1153701166}) 2024-09-15T03:33:53.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.172+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.172+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=83} crc {osdmap=1671664110}) 2024-09-15T03:33:53.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.172+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=80} crc {osdmap=1565840574}) 2024-09-15T03:33:53.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.172+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:33:53.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.172+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=83} crc {osdmap=1671664110}) 2024-09-15T03:33:53.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:33:53.172+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=83} crc {osdmap=4268916544}) 2024-09-15T03:33:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.178+0000 7f218a6a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T03:33:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.178+0000 7f218a6a8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T03:33:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.178+0000 7f218a6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.178+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f218411de50 con 0x7f218409f260 2024-09-15T03:33:53.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.379+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v127) v1 ==== 139+0+0 (secure 0 0 0) 0x7f217801a360 con 0x7f218409f260 2024-09-15T03:33:53.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.379+0000 7f216bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T03:33:53.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.379+0000 7f216bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:53.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.482+0000 7f218a6a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T03:33:53.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.482+0000 7f218a6a8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T03:33:53.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.482+0000 7f218a6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:53.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:53.482+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f2184064a20 con 0x7f218409f260 2024-09-15T03:33:54.167 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:33:54.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:33:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.271+0000 7fa8e52ef640 10 monclient: get_monmap_and_config 2024-09-15T03:33:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.271+0000 7fa8e52ef640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.271+0000 7fa8e52ef640 10 monclient: monmap: 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:54.272609+0000 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:54.272609+0000 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:54.273 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.272+0000 7fa8e52ef640 1 Processor -- start 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.272+0000 7fa8e52ef640 1 -- start start 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.272+0000 7fa8e52ef640 10 monclient: init 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8e0131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 0x7fa8e009e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e009fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa8e009ef00 0x7fa8e0138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e009ef00 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e00a1180 con 0x7fa8e0131980 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00a1460 con 0x7fa8e009ef00 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e0132460 con 0x7fa8e009fda0 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8e52ef640 10 monclient(hunting): authenticate will time out at 1031.912106s 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 0x7fa8e009e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa8e009ef00 0x7fa8e0138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 0x7fa8e009e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45698/0 (socket says 172.21.15.153:45698) 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 -- 172.21.15.153:0/3399459826 learned_addr learned my addr 172.21.15.153:0/3399459826 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): get_auth_request con 0x7fa8e009fda0 auth_method 0 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): get_auth_request con 0x7fa8e009ef00 auth_method 0 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_done global_id 74297 payload 274 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 -- 172.21.15.153:0/3399459826 >> v2:172.21.15.153:3300/0 conn(0x7fa8e009ef00 msgr2=0x7fa8e0138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 --2- 172.21.15.153:0/3399459826 >> v2:172.21.15.153:3300/0 conn(0x7fa8e009ef00 0x7fa8e0138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 -- 172.21.15.153:0/3399459826 >> v1:172.21.15.153:6789/0 conn(0x7fa8e0131980 legacy=0x7fa8e0131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 monclient: found mon.noname-b 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8d67fc640 1 -- 172.21.15.153:0/3399459826 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e01325d0 con 0x7fa8e009fda0 2024-09-15T03:33:54.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.273+0000 7fa8df7fe640 1 --2- 172.21.15.153:0/3399459826 >> v2:172.21.15.153:3300/0 conn(0x7fa8e009ef00 0x7fa8e0138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8d67fc640 1 -- 172.21.15.153:0/3399459826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 msgr2=0x7fa8e009e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8d67fc640 1 --2- 172.21.15.153:0/3399459826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 0x7fa8e009e9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009910 tx=0x7fa8c80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17677f39ba05ce6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 1 -- 172.21.15.153:0/3399459826 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c802aea0 con 0x7fa8e009fda0 2024-09-15T03:33:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: dump: 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 1 -- 172.21.15.153:0/3399459826 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8032440 con 0x7fa8e009fda0 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8e52ef640 5 monclient: authenticate success, global_id 74297 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8e52ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 1 -- 172.21.15.153:0/3399459826 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80327e0 con 0x7fa8e009fda0 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8de7fc640 10 monclient: dump: 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.288 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8e52ef640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.276+0000 7fa8e52ef640 4 set_mon_vals no callback set 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 10 monclient: shutdown 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3399459826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 msgr2=0x7fa8e009e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3399459826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 0x7fa8e009e9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009910 tx=0x7fa8c80091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3399459826 shutdown_connections 2024-09-15T03:33:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3399459826 >> v2:172.21.15.153:3300/0 conn(0x7fa8e009ef00 0x7fa8e0138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3399459826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009fda0 0x7fa8e009e9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3399459826 >> 172.21.15.153:0/3399459826 conn(0x7fa8e0067bd0 msgr2=0x7fa8e00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3399459826 shutdown_connections 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3399459826 wait complete. 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.277+0000 7fa8e52ef640 10 monclient: monmap: 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.290 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 Processor -- start 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 -- start start 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient: init 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 0x7fa8e0134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e009ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e009fda0 0x7fa8e0134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e009fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8e0131980 0x7fa8e0134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e0131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e009bf10 con 0x7fa8e009fda0 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00a4010 con 0x7fa8e0131980 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e0136660 con 0x7fa8e009ef00 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.278+0000 7fa8e52ef640 10 monclient(hunting): authenticate will time out at 1031.917950s 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 0x7fa8e0134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e009fda0 0x7fa8e0134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 0x7fa8e0134020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45700/0 (socket says 172.21.15.153:45700) 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e009fda0 0x7fa8e0134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44386/0 (socket says 172.21.15.153:44386) 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 -- 172.21.15.153:0/3870072728 learned_addr learned my addr 172.21.15.153:0/3870072728 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): get_auth_request con 0x7fa8e009ef00 auth_method 0 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): get_auth_request con 0x7fa8e009fda0 auth_method 0 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8df7fe640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8e0131980 0x7fa8e0134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8df7fe640 10 monclient(hunting): get_auth_request con 0x7fa8e0131980 auth_method 0 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient(hunting): handle_auth_done global_id 74300 payload 824 2024-09-15T03:33:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 -- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8e0131980 msgr2=0x7fa8e0134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8e0131980 0x7fa8e0134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 -- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e009fda0 msgr2=0x7fa8e0134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e009fda0 0x7fa8e0134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 monclient: found mon.noname-c 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8deffd640 1 -- 172.21.15.153:0/3870072728 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8009a20 con 0x7fa8e009ef00 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.279+0000 7fa8df7fe640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8e0131980 0x7fa8e0134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8deffd640 1 -- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 msgr2=0x7fa8e0134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8deffd640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 0x7fa8e0134020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c5e0 tx=0x7fa8d0007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0012840 con 0x7fa8e009ef00 2024-09-15T03:33:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: dump: 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d00129a0 con 0x7fa8e009ef00 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 5 monclient: authenticate success, global_id 74300 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 10 monclient: _renew_subs 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c8009450 con 0x7fa8e009ef00 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0012da0 con 0x7fa8e009ef00 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8d77fe640 10 monclient: dump: 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 10 monclient: _renew_subs 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8ddffb640 4 set_mon_vals no callback set 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.281+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e0138280 con 0x7fa8e009ef00 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa8d0011590 con 0x7fa8e009ef00 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8e52ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8e52ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8e52ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e004bf60 con 0x7fa8e009ef00 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8d77fe640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8c0038740 0x7fa8c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.282+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(127..127 src has 40..127) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fa8d004c9b0 con 0x7fa8e009ef00 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.283+0000 7fa8d67fc640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8c0038740 0x7fa8c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.283+0000 7fa8d67fc640 10 monclient: get_auth_request con 0x7fa8c0038740 auth_method 0 2024-09-15T03:33:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.285+0000 7fa8d67fc640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8c0038740 0x7fa8c003abf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00a4fb0 tx=0x7fa8c8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.287+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa8d0010ac0 con 0x7fa8e009ef00 2024-09-15T03:33:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.287+0000 7fa8d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.287+0000 7fa8d77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:54.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.400+0000 7f216bfff640 1 -- 172.21.15.181:0/3936077901 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v128) v1 ==== 139+0+0 (secure 0 0 0) 0x7f217801a540 con 0x7f218409f260 2024-09-15T03:33:54.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.400+0000 7f216bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-15T03:33:54.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.400+0000 7f216bfff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:54.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2170038770 msgr2=0x7f217003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2170038770 0x7f217003ac20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f21840675c0 tx=0x7f2174032000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 10 monclient: shutdown 2024-09-15T03:33:54.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 msgr2=0x7f218409a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 0x7f218409a9a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f217800c5e0 tx=0x7f2178004960 comp rx=0 tx=0).stop 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 shutdown_connections 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2170038770 0x7f217003ac20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21840a0ad0 0x7f2184130cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f218409f260 0x7f218409a9a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 --2- 172.21.15.181:0/3936077901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f218409ee60 0x7f218411bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 >> 172.21.15.181:0/3936077901 conn(0x7f2184071370 msgr2=0x7f2184131d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 shutdown_connections 2024-09-15T03:33:54.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.403+0000 7f218a6a8640 1 -- 172.21.15.181:0/3936077901 wait complete. 2024-09-15T03:33:54.416 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:33:54.416 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c fooset 2024-09-15T03:33:54.416 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 1 2024-09-15T03:33:54.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.496+0000 7fb1a3d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:54.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.496+0000 7fb1a3d9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.496+0000 7fb1a3d9d640 10 monclient: monmap: 2024-09-15T03:33:54.497 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:54.497 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.497 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:54.497074+0000 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:54.497074+0000 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:54.498 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.496+0000 7fb1a3d9d640 1 Processor -- start 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 1 -- start start 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient: init 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb19c069bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb19c098330 0x7fb19c098730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c098330 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:54.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb19c098c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c132850 con 0x7fb19c069bd0 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c09d140 con 0x7fb19c098c70 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19c09e590 con 0x7fb19c098330 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a3d9d640 10 monclient(hunting): authenticate will time out at 1038.978615s 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a233a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fb19c098c70 0x7fb19c138e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32940/0 (socket says 172.21.15.181:32940) 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a233a640 1 -- 172.21.15.181:0/1359783679 learned_addr learned my addr 172.21.15.181:0/1359783679 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a1338640 1 --2- 172.21.15.181:0/1359783679 >> v2:172.21.15.153:3300/0 conn(0x7fb19c098330 0x7fb19c098730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a1338640 10 monclient(hunting): get_auth_request con 0x7fb19c098330 auth_method 0 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a1338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a1338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.497+0000 7fb1a1338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 10 monclient(hunting): handle_auth_done global_id 84180 payload 274 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 1 -- 172.21.15.181:0/1359783679 >> v1:172.21.15.181:6789/0 conn(0x7fb19c098c70 legacy=0x7fb19c138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 1 -- 172.21.15.181:0/1359783679 >> v1:172.21.15.153:6789/0 conn(0x7fb19c069bd0 legacy=0x7fb19c132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 1 monclient: found mon.noname-a 2024-09-15T03:33:54.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.498+0000 7fb1a1338640 1 -- 172.21.15.181:0/1359783679 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c09e2b0 con 0x7fb19c098330 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a1338640 1 -- 172.21.15.181:0/1359783679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 msgr2=0x7fb19c098730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a1338640 1 --2- 172.21.15.181:0/1359783679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c098730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009850 tx=0x7fb18c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c317d7cb9c389d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 1 -- 172.21.15.181:0/1359783679 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c032440 con 0x7fb19c098330 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: dump: 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.511 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 1 -- 172.21.15.181:0/1359783679 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c0325a0 con 0x7fb19c098330 2024-09-15T03:33:54.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 1 -- 172.21.15.181:0/1359783679 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c032940 con 0x7fb19c098330 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a0b37640 10 monclient: dump: 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.513 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 5 monclient: authenticate success, global_id 84180 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 4 set_mon_vals no callback set 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 10 monclient: shutdown 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/1359783679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 msgr2=0x7fb19c098730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.501+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/1359783679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c098730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009850 tx=0x7fb18c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/1359783679 shutdown_connections 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/1359783679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c098730 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/1359783679 >> 172.21.15.181:0/1359783679 conn(0x7fb19c09e9f0 msgr2=0x7fb19c09edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/1359783679 shutdown_connections 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/1359783679 wait complete. 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 10 monclient: monmap: 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:54.515 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 Processor -- start 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 1 -- start start 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.502+0000 7fb1a3d9d640 10 monclient: init 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb19c069bd0 0x7fb19c1307c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c069bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c1367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c098330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 0x7fb19c136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb19c098c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c09d6e0 con 0x7fb19c098330 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c12fe10 con 0x7fb19c098c70 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c0a0000 con 0x7fb19c069bd0 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a3d9d640 10 monclient(hunting): authenticate will time out at 1038.984302s 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 0x7fb19c136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 0x7fb19c136ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47740/0 (socket says 172.21.15.181:47740) 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 -- 172.21.15.181:0/2331941202 learned_addr learned my addr 172.21.15.181:0/2331941202 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): get_auth_request con 0x7fb19c098c70 auth_method 0 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c1367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb19c069bd0 0x7fb19c1307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): get_auth_request con 0x7fb19c098330 auth_method 0 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): get_auth_request con 0x7fb19c069bd0 auth_method 0 2024-09-15T03:33:54.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a1b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient(hunting): handle_auth_done global_id 45244 payload 824 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 -- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb19c069bd0 msgr2=0x7fb19c1307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb19c069bd0 0x7fb19c1307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 -- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 msgr2=0x7fb19c1367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c1367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 monclient: found mon.noname-b 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.503+0000 7fb1a233a640 1 -- 172.21.15.181:0/2331941202 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c009390 con 0x7fb19c098c70 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.505+0000 7fb1a233a640 1 -- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 msgr2=0x7fb19c136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.505+0000 7fb1a233a640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 0x7fb19c136ce0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb19400ccc0 tx=0x7fb194005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400ce80 con 0x7fb19c098c70 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: dump: 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 5 monclient: authenticate success, global_id 45244 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 10 monclient: _renew_subs 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19400aa60 con 0x7fb19c098c70 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19c118740 con 0x7fb19c098c70 2024-09-15T03:33:54.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400ae20 con 0x7fb19c098c70 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 10 monclient: dump: 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.522 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb193fff640 4 set_mon_vals no callback set 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 10 monclient: _renew_subs 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19c119400 con 0x7fb19c098c70 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.506+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb194007300 con 0x7fb19c098c70 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb191ffb640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb180038740 0x7fb18003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(128..128 src has 40..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fb19404c9b0 con 0x7fb19c098c70 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb1a3d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb1a3d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb1a3d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19c118b30 con 0x7fb19c098c70 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb1a1b39640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb180038740 0x7fb18003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.507+0000 7fb1a1b39640 10 monclient: get_auth_request con 0x7fb180038740 auth_method 0 2024-09-15T03:33:54.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.509+0000 7fb1a1b39640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb180038740 0x7fb18003abf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb198007fd0 tx=0x7fb198007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.511+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb1940075b0 con 0x7fb19c098c70 2024-09-15T03:33:54.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.511+0000 7fb191ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.511+0000 7fb191ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8e52ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8e52ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8e52ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa8e0138560 con 0x7fa8e009ef00 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8d77fe640 1 -- 172.21.15.153:0/3870072728 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fa8d00102e0 con 0x7fa8e009ef00 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.514+0000 7fa8d77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:54.525 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:33:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8c0038740 msgr2=0x7fa8c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8c0038740 0x7fa8c003abf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00a4fb0 tx=0x7fa8c8032000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 10 monclient: shutdown 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 msgr2=0x7fa8e0134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 0x7fa8e0134020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c5e0 tx=0x7fa8d0007ed0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 shutdown_connections 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8c0038740 0x7fa8c003abf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8e0131980 0x7fa8e0134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8e009fda0 0x7fa8e0134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 --2- 172.21.15.153:0/3870072728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8e009ef00 0x7fa8e0134020 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 >> 172.21.15.153:0/3870072728 conn(0x7fa8e0067bd0 msgr2=0x7fa8e009d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 shutdown_connections 2024-09-15T03:33:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.516+0000 7fa8e52ef640 1 -- 172.21.15.153:0/3870072728 wait complete. 2024-09-15T03:33:54.529 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:33:54.529 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:33:54.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:33:54.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.622+0000 7fa4c4c99640 10 monclient: get_monmap_and_config 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.622+0000 7fa4c4c99640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.622+0000 7fa4c4c99640 10 monclient: monmap: 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:54.622891+0000 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:54.622891+0000 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.626 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.622+0000 7fa4c4c99640 1 Processor -- start 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.622+0000 7fa4c4c99640 1 -- start start 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.622+0000 7fa4c4c99640 10 monclient: init 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:33:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): picked mon.noname-b con 0x7fa4c01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4c009ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa4c0136c20 0x7fa4c0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c0136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c0131cc0 con 0x7fa4c009ea50 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c009f530 con 0x7fa4c0136c20 2024-09-15T03:33:54.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c009fbf0 con 0x7fa4c01318f0 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4c4c99640 10 monclient(hunting): authenticate will time out at 1032.262627s 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45702/0 (socket says 172.21.15.153:45702) 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa4c0136c20 0x7fa4c0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 1 -- 172.21.15.153:0/4192040373 learned_addr learned my addr 172.21.15.153:0/4192040373 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): get_auth_request con 0x7fa4c01318f0 auth_method 0 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): get_auth_request con 0x7fa4c0136c20 auth_method 0 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.623+0000 7fa4bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 10 monclient(hunting): handle_auth_done global_id 74306 payload 274 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 1 -- 172.21.15.153:0/4192040373 >> v2:172.21.15.153:3300/0 conn(0x7fa4c0136c20 msgr2=0x7fa4c0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 1 --2- 172.21.15.153:0/4192040373 >> v2:172.21.15.153:3300/0 conn(0x7fa4c0136c20 0x7fa4c0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 1 -- 172.21.15.153:0/4192040373 >> v1:172.21.15.153:6789/0 conn(0x7fa4c009ea50 legacy=0x7fa4c009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 1 monclient: found mon.noname-b 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4be59c640 1 -- 172.21.15.153:0/4192040373 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c009fe30 con 0x7fa4c01318f0 2024-09-15T03:33:54.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.624+0000 7fa4bed9d640 1 --2- 172.21.15.153:0/4192040373 >> v2:172.21.15.153:3300/0 conn(0x7fa4c0136c20 0x7fa4c0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:54.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.626+0000 7fa4be59c640 1 -- 172.21.15.153:0/4192040373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 msgr2=0x7fa4c0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.626+0000 7fa4be59c640 1 --2- 172.21.15.153:0/4192040373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c0131f30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009f20 tx=0x7fa4b8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fecacc3f6d2f11d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.626+0000 7fa4bd59a640 1 -- 172.21.15.153:0/4192040373 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8007610 con 0x7fa4c01318f0 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.626+0000 7fa4bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: dump: 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 1 -- 172.21.15.153:0/4192040373 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b802d050 con 0x7fa4c01318f0 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 5 monclient: authenticate success, global_id 74306 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 1 -- 172.21.15.153:0/4192040373 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80079a0 con 0x7fa4c01318f0 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 4 set_mon_vals no callback set 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4bd59a640 10 monclient: dump: 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 10 monclient: shutdown 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 -- 172.21.15.153:0/4192040373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 msgr2=0x7fa4c0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/4192040373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c0131f30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009f20 tx=0x7fa4b8009840 comp rx=0 tx=0).stop 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 -- 172.21.15.153:0/4192040373 shutdown_connections 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/4192040373 >> v2:172.21.15.153:3300/0 conn(0x7fa4c0136c20 0x7fa4c0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/4192040373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c0131f30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 -- 172.21.15.153:0/4192040373 >> 172.21.15.153:0/4192040373 conn(0x7fa4c0067110 msgr2=0x7fa4c0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 -- 172.21.15.153:0/4192040373 shutdown_connections 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.627+0000 7fa4c4c99640 1 -- 172.21.15.153:0/4192040373 wait complete. 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient: monmap: 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:54.650 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 Processor -- start 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 -- start start 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient: init 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 0x7fa4c01308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c00999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4c0136c20 0x7fa4c0099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): picked mon.noname-b con 0x7fa4c0136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c009bf10 con 0x7fa4c009ea50 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c009e060 con 0x7fa4c0136c20 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01374c0 con 0x7fa4c01318f0 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4c4c99640 10 monclient(hunting): authenticate will time out at 1032.267895s 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4be59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 0x7fa4c01308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.628+0000 7fa4bdd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c00999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 0x7fa4c01308d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44412/0 (socket says 172.21.15.153:44412) 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c00999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45712/0 (socket says 172.21.15.153:45712) 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 -- 172.21.15.153:0/1444893540 learned_addr learned my addr 172.21.15.153:0/1444893540 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): get_auth_request con 0x7fa4c009ea50 auth_method 0 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): get_auth_request con 0x7fa4c01318f0 auth_method 0 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4c0136c20 0x7fa4c0099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): get_auth_request con 0x7fa4c0136c20 auth_method 0 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient(hunting): handle_auth_done global_id 84189 payload 824 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 msgr2=0x7fa4c00999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c01318f0 0x7fa4c00999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4c0136c20 msgr2=0x7fa4c0099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4c0136c20 0x7fa4c0099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 monclient: found mon.noname-a 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.629+0000 7fa4be59c640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009390 con 0x7fa4c009ea50 2024-09-15T03:33:54.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4be59c640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 msgr2=0x7fa4c01308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4be59c640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 0x7fa4c01308d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4b802abc0 tx=0x7fa4b8036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b80328b0 con 0x7fa4c009ea50 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: dump: 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.656 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b8032a50 con 0x7fa4c009ea50 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8032e10 con 0x7fa4c009ea50 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4b67fc640 10 monclient: dump: 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.657 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 5 monclient: authenticate success, global_id 84189 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 10 monclient: _renew_subs 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b8009ae0 con 0x7fa4c009ea50 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4bcd99640 4 set_mon_vals no callback set 2024-09-15T03:33:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 10 monclient: _renew_subs 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.631+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c011d000 con 0x7fa4c009ea50 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa4b8038050 con 0x7fa4c009ea50 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4c4c99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 0x7fa4c00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0065e00 con 0x7fa4c0130e10 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4b67fc640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4a0038740 0x7fa4a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 0x7fa4c00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient: get_auth_request con 0x7fa4c0130e10 auth_method 0 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(128..128 src has 40..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fa4b802ae40 con 0x7fa4c009ea50 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.632+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.633+0000 7fa4bed9d640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4a0038740 0x7fa4a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.633+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_done global_id 84192 payload 824 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.633+0000 7fa4bdd9b640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4c004bf60 con 0x7fa4c0130e10 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.633+0000 7fa4bed9d640 10 monclient: get_auth_request con 0x7fa4a0038740 auth_method 0 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4bdd9b640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 msgr2=0x7fa4c00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4bdd9b640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 0x7fa4c00298c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800f780 tx=0x7fa4a8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800d8b0 con 0x7fa4c0130e10 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 10 monclient: dump: 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4bed9d640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4a0038740 0x7fa4a003abf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00a0240 tx=0x7fa4b00090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa4a800da10 con 0x7fa4c0130e10 2024-09-15T03:33:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 msgr2=0x7fa4c00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 0x7fa4c00298c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800f780 tx=0x7fa4a8007e10 comp rx=0 tx=0).stop 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 msgr2=0x7fa4c00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.635+0000 7fa4b67fc640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0130e10 0x7fa4c00298c0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800f780 tx=0x7fa4a8007e10 comp rx=0 tx=0).stop 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4c4c99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 0x7fa4c002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4c4c99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c011cea0 con 0x7fa4c0029e00 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 0x7fa4c002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient: get_auth_request con 0x7fa4c0029e00 auth_method 0 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 10 monclient(hunting): handle_auth_done global_id 84195 payload 824 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.641+0000 7fa4bdd9b640 1 -- 172.21.15.153:0/1444893540 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa4c0141e70 con 0x7fa4c0029e00 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.643+0000 7fa4bdd9b640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 msgr2=0x7fa4c002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.643+0000 7fa4bdd9b640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 0x7fa4c002a200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800f780 tx=0x7fa4a8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a801d610 con 0x7fa4c0029e00 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 10 monclient: dump: 2024-09-15T03:33:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7fa4a801d770 con 0x7fa4c0029e00 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 msgr2=0x7fa4c002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 0x7fa4c002a200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800f780 tx=0x7fa4a8019000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 msgr2=0x7fa4c002a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4b67fc640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c0029e00 0x7fa4c002a200 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800f780 tx=0x7fa4a8019000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:33:54.665 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 60, 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.667 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:54.668 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.669 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.670 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.671 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:54.672 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.673 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:54.675 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.676 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.677 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:54.677 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4a0038740 msgr2=0x7fa4a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa4a0038740 0x7fa4a003abf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00a0240 tx=0x7fa4b00090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 10 monclient: shutdown 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 msgr2=0x7fa4c01308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 --2- 172.21.15.153:0/1444893540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4c009ea50 0x7fa4c01308d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4b802abc0 tx=0x7fa4b8036000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4bed9d640 1 -- 172.21.15.153:0/1444893540 reap_dead start 2024-09-15T03:33:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4bed9d640 1 -- 172.21.15.153:0/1444893540 reap_dead start 2024-09-15T03:33:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 shutdown_connections 2024-09-15T03:33:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 >> 172.21.15.153:0/1444893540 conn(0x7fa4c0067110 msgr2=0x7fa4c00a4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 shutdown_connections 2024-09-15T03:33:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.644+0000 7fa4c4c99640 1 -- 172.21.15.153:0/1444893540 wait complete. 2024-09-15T03:33:54.679 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:33:54.679 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:33:54.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:33:54.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.673+0000 7fb1a3d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:54.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.673+0000 7fb1a3d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:54.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.673+0000 7fb1a3d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.673+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb19c04bf60 con 0x7fb19c098c70 2024-09-15T03:33:54.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.674+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v128) v1 ==== 71+0+14 (secure 0 0 0) 0x7fb194029080 con 0x7fb19c098c70 2024-09-15T03:33:54.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.674+0000 7fb191ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:54.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.674+0000 7fb191ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.772+0000 7fb1a3d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.772+0000 7fb1a3d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.772+0000 7fb1a3d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.772+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb19c1188d0 con 0x7fb19c098c70 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.773+0000 7fb191ffb640 1 -- 172.21.15.181:0/2331941202 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v128) v1 ==== 71+0+14 (secure 0 0 0) 0x7fb194010af0 con 0x7fb19c098c70 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.773+0000 7fb191ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:33:54.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.773+0000 7fb191ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:33:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.773+0000 7f2e46c34640 10 monclient: get_monmap_and_config 2024-09-15T03:33:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.773+0000 7f2e46c34640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.773+0000 7f2e46c34640 10 monclient: monmap: 2024-09-15T03:33:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:54.774490+0000 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:54.774490+0000 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.774+0000 7f2e46c34640 1 Processor -- start 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.774+0000 7f2e46c34640 1 -- start start 2024-09-15T03:33:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.774+0000 7f2e46c34640 10 monclient: init 2024-09-15T03:33:54.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb180038740 msgr2=0x7fb18003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb180038740 0x7fb18003abf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb198007fd0 tx=0x7fb198007e10 comp rx=0 tx=0).stop 2024-09-15T03:33:54.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 10 monclient: shutdown 2024-09-15T03:33:54.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 msgr2=0x7fb19c136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 0x7fb19c136ce0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb19400ccc0 tx=0x7fb194005370 comp rx=0 tx=0).stop 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 shutdown_connections 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb180038740 0x7fb18003abf0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb19c098c70 0x7fb19c136ce0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb19c098330 0x7fb19c1367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 --2- 172.21.15.181:0/2331941202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb19c069bd0 0x7fb19c1307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 >> 172.21.15.181:0/2331941202 conn(0x7fb19c09e9f0 msgr2=0x7fb19c131210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 shutdown_connections 2024-09-15T03:33:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.775+0000 7fb1a3d9d640 1 -- 172.21.15.181:0/2331941202 wait complete. 2024-09-15T03:33:54.789 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set rm fooset 2024-09-15T03:33:54.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.774+0000 7f2e46c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.774+0000 7f2e46c34640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:33:54.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 0x7f2e40131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): picked mon.noname-a con 0x7f2e401318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:54.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2e4009ea50 0x7f2e4009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): picked mon.noname-c con 0x7f2e4009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 0x7f2e400691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): picked mon.noname-b con 0x7f2e40068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e40131cc0 con 0x7f2e401318f0 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4009fbf0 con 0x7f2e4009ea50 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4009fe30 con 0x7f2e40068d90 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e46c34640 10 monclient(hunting): authenticate will time out at 1032.414087s 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 0x7f2e40131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 0x7f2e40131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44438/0 (socket says 172.21.15.153:44438) 2024-09-15T03:33:54.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 0x7f2e400691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 1 -- 172.21.15.153:0/1954761572 learned_addr learned my addr 172.21.15.153:0/1954761572 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): get_auth_request con 0x7f2e401318f0 auth_method 0 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): get_auth_request con 0x7f2e40068d90 auth_method 0 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 1 --2- 172.21.15.153:0/1954761572 >> v2:172.21.15.181:3300/0 conn(0x7f2e4009ea50 0x7f2e4009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request con 0x7f2e4009ea50 auth_method 0 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient(hunting): handle_auth_done global_id 74312 payload 274 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 -- 172.21.15.153:0/1954761572 >> v2:172.21.15.181:3300/0 conn(0x7f2e4009ea50 msgr2=0x7f2e4009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 --2- 172.21.15.153:0/1954761572 >> v2:172.21.15.181:3300/0 conn(0x7f2e4009ea50 0x7f2e4009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 -- 172.21.15.153:0/1954761572 >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 msgr2=0x7f2e40131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 --2- 172.21.15.153:0/1954761572 >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 0x7f2e40131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 monclient: found mon.noname-b 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e451d1640 1 -- 172.21.15.153:0/1954761572 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e400a1210 con 0x7f2e40068d90 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.775+0000 7f2e449d0640 1 --2- 172.21.15.153:0/1954761572 >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 0x7f2e40131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e451d1640 1 -- 172.21.15.153:0/1954761572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 msgr2=0x7f2e400691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e451d1640 1 --2- 172.21.15.153:0/1954761572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 0x7f2e400691d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2e4006f600 tx=0x7f2e38009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15519fb56e52627c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 1 -- 172.21.15.153:0/1954761572 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38007760 con 0x7f2e40068d90 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: dump: 2024-09-15T03:33:54.793 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 1 -- 172.21.15.153:0/1954761572 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3802d050 con 0x7f2e40068d90 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 5 monclient: authenticate success, global_id 74312 2024-09-15T03:33:54.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 1 -- 172.21.15.153:0/1954761572 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38007af0 con 0x7f2e40068d90 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 4 set_mon_vals no callback set 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e3f7fe640 10 monclient: dump: 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 10 monclient: shutdown 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 1 -- 172.21.15.153:0/1954761572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 msgr2=0x7f2e400691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.778+0000 7f2e46c34640 1 --2- 172.21.15.153:0/1954761572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 0x7f2e400691d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2e4006f600 tx=0x7f2e38009780 comp rx=0 tx=0).stop 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 -- 172.21.15.153:0/1954761572 shutdown_connections 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 --2- 172.21.15.153:0/1954761572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40068d90 0x7f2e400691d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 --2- 172.21.15.153:0/1954761572 >> v2:172.21.15.181:3300/0 conn(0x7f2e4009ea50 0x7f2e4009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 --2- 172.21.15.153:0/1954761572 >> v2:172.21.15.153:3300/0 conn(0x7f2e401318f0 0x7f2e40131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 -- 172.21.15.153:0/1954761572 >> 172.21.15.153:0/1954761572 conn(0x7f2e40067110 msgr2=0x7f2e40067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 -- 172.21.15.153:0/1954761572 shutdown_connections 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 -- 172.21.15.153:0/1954761572 wait complete. 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 10 monclient: monmap: 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.779+0000 7f2e46c34640 1 Processor -- start 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 -- start start 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient: init 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 0x7f2e4009a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): picked mon.noname-a con 0x7f2e40068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e4009ea50 0x7f2e4011bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): picked mon.noname-c con 0x7f2e4009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e401318f0 0x7f2e40130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): picked mon.noname-b con 0x7f2e401318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4011c920 con 0x7f2e40068d90 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4011de80 con 0x7f2e401318f0 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4011dba0 con 0x7f2e4009ea50 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e46c34640 10 monclient(hunting): authenticate will time out at 1032.419567s 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 0x7f2e4009a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e4009ea50 0x7f2e4011bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e4009ea50 0x7f2e4011bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45724/0 (socket says 172.21.15.153:45724) 2024-09-15T03:33:54.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 0x7f2e4009a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44452/0 (socket says 172.21.15.153:44452) 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 1 -- 172.21.15.153:0/4014670055 learned_addr learned my addr 172.21.15.153:0/4014670055 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request con 0x7f2e4009ea50 auth_method 0 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): get_auth_request con 0x7f2e40068d90 auth_method 0 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e401318f0 0x7f2e40130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): get_auth_request con 0x7f2e401318f0 auth_method 0 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient(hunting): handle_auth_done global_id 84201 payload 824 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e4009ea50 msgr2=0x7f2e4011bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e4009ea50 0x7f2e4011bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e401318f0 msgr2=0x7f2e40130a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e401318f0 0x7f2e40130a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 monclient: found mon.noname-a 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e449d0640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e34009450 con 0x7f2e40068d90 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.780+0000 7f2e3ffff640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e4009ea50 0x7f2e4011bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e449d0640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 msgr2=0x7f2e4009a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e449d0640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 0x7f2e4009a890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400ea20 tx=0x7f2e340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34009680 con 0x7f2e40068d90 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: dump: 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3400cc20 con 0x7f2e40068d90 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 5 monclient: authenticate success, global_id 84201 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 10 monclient: _renew_subs 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e38009a20 con 0x7f2e40068d90 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34013470 con 0x7f2e40068d90 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3cff9640 10 monclient: dump: 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e3effd640 4 set_mon_vals no callback set 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 10 monclient: _renew_subs 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.783+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4011e210 con 0x7f2e40068d90 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2e34013880 con 0x7f2e40068d90 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e3cff9640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2e24038740 0x7f2e2403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e46c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e46c34640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 0x7f2e40140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e40065e00 con 0x7f2e40130fc0 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(128..128 src has 40..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f2e3404cab0 con 0x7f2e40068d90 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 0x7f2e40140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient: get_auth_request con 0x7f2e40130fc0 auth_method 0 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e3ffff640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2e24038740 0x7f2e2403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e3ffff640 10 monclient: get_auth_request con 0x7f2e24038740 auth_method 0 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 10 monclient(hunting): handle_auth_done global_id 74318 payload 824 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.784+0000 7f2e451d1640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e4004bf60 con 0x7f2e40130fc0 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.786+0000 7f2e451d1640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 msgr2=0x7f2e40140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.786+0000 7f2e451d1640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 0x7f2e40140b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e38034000 tx=0x7f2e38037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3ffff640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2e24038740 0x7f2e2403abf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2e4009ba80 tx=0x7f2e2c0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38041070 con 0x7f2e40130fc0 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 10 monclient: dump: 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2e3803cc50 con 0x7f2e40130fc0 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 msgr2=0x7f2e40140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 0x7f2e40140b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e38034000 tx=0x7f2e38037000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 msgr2=0x7f2e40140b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.787+0000 7f2e3cff9640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40130fc0 0x7f2e40140b50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.792+0000 7f2e46c34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:54.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e46c34640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 0x7f2e40141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e46c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e40131390 con 0x7f2e40141090 2024-09-15T03:33:54.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 0x7f2e40141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient: get_auth_request con 0x7f2e40141090 auth_method 0 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 10 monclient(hunting): handle_auth_done global_id 74321 payload 824 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.793+0000 7f2e451d1640 1 -- 172.21.15.153:0/4014670055 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e401419d0 con 0x7f2e40141090 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e451d1640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 msgr2=0x7f2e40141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e451d1640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 0x7f2e40141490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e380097c0 tx=0x7f2e38041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38031c20 con 0x7f2e40141090 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 10 monclient: dump: 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f2e3803ca40 con 0x7f2e40141090 2024-09-15T03:33:54.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 msgr2=0x7f2e40141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 0x7f2e40141490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e380097c0 tx=0x7f2e38041000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 msgr2=0x7f2e40141490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.795+0000 7f2e3cff9640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e40141090 0x7f2e40141490 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e380097c0 tx=0x7f2e38041000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:33:54.813 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 60, 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.814 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.815 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:54.816 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.817 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.818 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.819 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.821 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.822 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.823 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.824 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:54.825 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:54.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2e24038740 msgr2=0x7f2e2403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2e24038740 0x7f2e2403abf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2e4009ba80 tx=0x7f2e2c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 10 monclient: shutdown 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 msgr2=0x7f2e4009a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e451d1640 1 -- 172.21.15.153:0/4014670055 reap_dead start 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 0x7f2e4009a890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400ea20 tx=0x7f2e340091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 shutdown_connections 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 --2- 172.21.15.153:0/4014670055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40068d90 0x7f2e4009a890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 >> 172.21.15.153:0/4014670055 conn(0x7f2e40067110 msgr2=0x7f2e400a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 shutdown_connections 2024-09-15T03:33:54.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.796+0000 7f2e46c34640 1 -- 172.21.15.153:0/4014670055 wait complete. 2024-09-15T03:33:54.827 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:33:54.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:33:54.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:33:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.858+0000 7f069f11d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.858+0000 7f069f11d640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.858+0000 7f069f11d640 10 monclient: monmap: 2024-09-15T03:33:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:33:54.859232+0000 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:33:54.859232+0000 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:54.860 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 Processor -- start 2024-09-15T03:33:54.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 -- start start 2024-09-15T03:33:54.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient: init 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f069809f180 0x7f0698130040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): picked mon.noname-c con 0x7f069809f180 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809cee0 0x7f069809bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): picked mon.noname-b con 0x7f069809cee0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f069809c080 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069809e610 con 0x7f069809c080 2024-09-15T03:33:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069809f550 con 0x7f069809f180 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069809e2a0 con 0x7f069809cee0 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069f11d640 10 monclient(hunting): authenticate will time out at 1039.340677s 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f069809f180 0x7f0698130040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069d6ba640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f069809c080 0x7f0698070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:32952/0 (socket says 172.21.15.181:32952) 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069d6ba640 1 -- 172.21.15.181:0/2581568490 learned_addr learned my addr 172.21.15.181:0/2581568490 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): get_auth_request con 0x7f069809f180 auth_method 0 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f0697fff640 1 --2- 172.21.15.181:0/2581568490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809cee0 0x7f069809bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f0697fff640 10 monclient(hunting): get_auth_request con 0x7f069809cee0 auth_method 0 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f0697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f0697fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.859+0000 7f0697fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 10 monclient(hunting): handle_auth_done global_id 45256 payload 274 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 1 -- 172.21.15.181:0/2581568490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809cee0 msgr2=0x7f069809bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 1 --2- 172.21.15.181:0/2581568490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809cee0 0x7f069809bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 1 -- 172.21.15.181:0/2581568490 >> v1:172.21.15.181:6789/0 conn(0x7f069809c080 legacy=0x7f0698070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 1 monclient: found mon.noname-c 2024-09-15T03:33:54.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.860+0000 7f069ceb9640 1 -- 172.21.15.181:0/2581568490 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698130580 con 0x7f069809f180 2024-09-15T03:33:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f069ceb9640 1 -- 172.21.15.181:0/2581568490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809f180 msgr2=0x7f0698130040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f069ceb9640 1 --2- 172.21.15.181:0/2581568490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809f180 0x7f0698130040 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f068c009850 tx=0x7f068c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bf0b6a7c10f3010 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 1 -- 172.21.15.181:0/2581568490 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c02ae80 con 0x7f069809f180 2024-09-15T03:33:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 10 monclient: dump: 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 1 -- 172.21.15.181:0/2581568490 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068c032440 con 0x7f069809f180 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f069f11d640 5 monclient: authenticate success, global_id 45256 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f069f11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.862+0000 7f06977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f06977fe640 1 -- 172.21.15.181:0/2581568490 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c0327c0 con 0x7f069809f180 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f06977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f06977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f06977fe640 10 monclient: dump: 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.874 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 4 set_mon_vals no callback set 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 10 monclient: shutdown 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 -- 172.21.15.181:0/2581568490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809f180 msgr2=0x7f0698130040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 --2- 172.21.15.181:0/2581568490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809f180 0x7f0698130040 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f068c009850 tx=0x7f068c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 -- 172.21.15.181:0/2581568490 shutdown_connections 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 --2- 172.21.15.181:0/2581568490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809cee0 0x7f069809bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 --2- 172.21.15.181:0/2581568490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809f180 0x7f0698130040 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 -- 172.21.15.181:0/2581568490 >> 172.21.15.181:0/2581568490 conn(0x7f06980a2b70 msgr2=0x7f069809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 -- 172.21.15.181:0/2581568490 shutdown_connections 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.863+0000 7f069f11d640 1 -- 172.21.15.181:0/2581568490 wait complete. 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 10 monclient: monmap: 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:54.877 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 1 Processor -- start 2024-09-15T03:33:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 1 -- start start 2024-09-15T03:33:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 10 monclient: init 2024-09-15T03:33:54.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.864+0000 7f069f11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f069813efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): picked mon.noname-c con 0x7f069809c080 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): picked mon.noname-b con 0x7f069809cee0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f06981185e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): picked mon.noname-a con 0x7f069809f180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069811c1a0 con 0x7f069809f180 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069806eca0 con 0x7f069809cee0 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069811a500 con 0x7f069809c080 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069f11d640 10 monclient(hunting): authenticate will time out at 1039.346058s 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:47764/0 (socket says 172.21.15.181:47764) 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f069813efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 learned_addr learned my addr 172.21.15.181:0/833609021 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): get_auth_request con 0x7f069809cee0 auth_method 0 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): get_auth_request con 0x7f069809c080 auth_method 0 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f06981185e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): get_auth_request con 0x7f069809f180 auth_method 0 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f069d6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient(hunting): handle_auth_done global_id 45259 payload 824 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 msgr2=0x7f069813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f069813efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 msgr2=0x7f06981185e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f06981185e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 monclient: found mon.noname-b 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.865+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c009390 con 0x7f069809cee0 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 msgr2=0x7f069812ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0697fff640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f068400d890 tx=0x7f0684008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400de20 con 0x7f069809cee0 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: dump: 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 5 monclient: authenticate success, global_id 45259 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068400bae0 con 0x7f069809cee0 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 10 monclient: _renew_subs 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069811abc0 con 0x7f069809cee0 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400be80 con 0x7f069809cee0 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0694ff9640 10 monclient: dump: 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 10 monclient: _renew_subs 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f0696ffd640 4 set_mon_vals no callback set 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.867+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069813fde0 con 0x7f069809cee0 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0684014640 con 0x7f069809cee0 2024-09-15T03:33:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f069f11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f069f11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f069f11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069811bf20 con 0x7f069809cee0 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f0694ff9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f067c038740 0x7f067c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(128..128 src has 40..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f068404ce10 con 0x7f069809cee0 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.868+0000 7f069ceb9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f067c038740 0x7f067c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.869+0000 7f069ceb9640 10 monclient: get_auth_request con 0x7f067c038740 auth_method 0 2024-09-15T03:33:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.871+0000 7f069ceb9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f067c038740 0x7f067c03abf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f069809fd80 tx=0x7f068c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.873+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0684011570 con 0x7f069809cee0 2024-09-15T03:33:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.873+0000 7f0694ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.873+0000 7f0694ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.921+0000 7f39c2f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.921+0000 7f39c2f9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.921+0000 7f39c2f9d640 10 monclient: monmap: 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:33:54.921918+0000 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:33:54.921918+0000 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.923 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:33:54.923 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:33:54.923 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:33:54.923 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:33:54.923 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:33:54.923 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.921+0000 7f39c2f9d640 1 Processor -- start 2024-09-15T03:33:54.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.921+0000 7f39c2f9d640 1 -- start start 2024-09-15T03:33:54.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.921+0000 7f39c2f9d640 10 monclient: init 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f39bc1318f0 0x7f39bc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f39bc1318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f39bc09ea50 0x7f39bc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f39bc09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39bc068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc09fbf0 con 0x7f39bc068d90 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc131d50 con 0x7f39bc09ea50 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc09fe30 con 0x7f39bc1318f0 2024-09-15T03:33:54.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c2f9d640 10 monclient(hunting): authenticate will time out at 1032.561480s 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f39bc09ea50 0x7f39bc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f39bc09ea50 0x7f39bc09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44458/0 (socket says 172.21.15.153:44458) 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 -- 172.21.15.153:0/1912583727 learned_addr learned my addr 172.21.15.153:0/1912583727 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): get_auth_request con 0x7f39bc09ea50 auth_method 0 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 1 --2- 172.21.15.153:0/1912583727 >> v2:172.21.15.181:3300/0 conn(0x7f39bc1318f0 0x7f39bc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): get_auth_request con 0x7f39bc1318f0 auth_method 0 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient(hunting): handle_auth_done global_id 84207 payload 274 2024-09-15T03:33:54.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 -- 172.21.15.153:0/1912583727 >> v2:172.21.15.181:3300/0 conn(0x7f39bc1318f0 msgr2=0x7f39bc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 --2- 172.21.15.153:0/1912583727 >> v2:172.21.15.181:3300/0 conn(0x7f39bc1318f0 0x7f39bc131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 -- 172.21.15.153:0/1912583727 >> v1:172.21.15.181:6789/0 conn(0x7f39bc068d90 legacy=0x7f39bc0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 monclient: found mon.noname-a 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.922+0000 7f39bbfff640 1 -- 172.21.15.153:0/1912583727 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc0a1210 con 0x7f39bc09ea50 2024-09-15T03:33:54.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bbfff640 1 -- 172.21.15.153:0/1912583727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 msgr2=0x7f39bc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bbfff640 1 --2- 172.21.15.153:0/1912583727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc09ee20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39bc076600 tx=0x7f39a80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2fa495dd0218bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 1 -- 172.21.15.153:0/1912583727 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8032440 con 0x7f39bc09ea50 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: dump: 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.936 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39c2f9d640 5 monclient: authenticate success, global_id 84207 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 1 -- 172.21.15.153:0/1912583727 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a80325a0 con 0x7f39bc09ea50 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39c2f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39c2f9d640 4 set_mon_vals no callback set 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 1 -- 172.21.15.153:0/1912583727 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8032940 con 0x7f39bc09ea50 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.925+0000 7f39bb7fe640 10 monclient: dump: 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 10 monclient: shutdown 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 -- 172.21.15.153:0/1912583727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 msgr2=0x7f39bc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/1912583727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc09ee20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39bc076600 tx=0x7f39a80091d0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 -- 172.21.15.153:0/1912583727 shutdown_connections 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/1912583727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc09ee20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/1912583727 >> v2:172.21.15.181:3300/0 conn(0x7f39bc1318f0 0x7f39bc131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 -- 172.21.15.153:0/1912583727 >> 172.21.15.153:0/1912583727 conn(0x7f39bc067110 msgr2=0x7f39bc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 -- 172.21.15.153:0/1912583727 shutdown_connections 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 1 -- 172.21.15.153:0/1912583727 wait complete. 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 10 monclient: build_initial_monmap 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.926+0000 7f39c2f9d640 10 monclient: monmap: 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:33:54.939 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 Processor -- start 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 -- start start 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient: init 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc068d90 0x7f39bc09ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f39bc068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f39bc09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39bc1318f0 0x7f39bc11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f39bc1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0a4410 con 0x7f39bc09ea50 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc11e310 con 0x7f39bc068d90 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc11e080 con 0x7f39bc1318f0 2024-09-15T03:33:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39c2f9d640 10 monclient(hunting): authenticate will time out at 1032.566908s 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.927+0000 7f39bbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc130c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44472/0 (socket says 172.21.15.153:44472) 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 -- 172.21.15.153:0/2331038085 learned_addr learned my addr 172.21.15.153:0/2331038085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): get_auth_request con 0x7f39bc09ea50 auth_method 0 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39bc1318f0 0x7f39bc11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): get_auth_request con 0x7f39bc1318f0 auth_method 0 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc068d90 0x7f39bc09ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): get_auth_request con 0x7f39bc068d90 auth_method 0 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient(hunting): handle_auth_done global_id 84210 payload 824 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39bc1318f0 msgr2=0x7f39bc11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39bc1318f0 0x7f39bc11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc068d90 msgr2=0x7f39bc09ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc068d90 0x7f39bc09ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 monclient: found mon.noname-a 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39bbfff640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8009450 con 0x7f39bc09ea50 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c153a640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39bc1318f0 0x7f39bc11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.928+0000 7f39c0d39640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc068d90 0x7f39bc09ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:33:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39bbfff640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 msgr2=0x7f39bc130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39bbfff640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc130c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39a8035040 tx=0x7f39a8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a80319c0 con 0x7f39bc09ea50 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: dump: 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8031b40 con 0x7f39bc09ea50 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 5 monclient: authenticate success, global_id 84210 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 10 monclient: _renew_subs 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc09b7c0 con 0x7f39bc09ea50 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a803d440 con 0x7f39bc09ea50 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39b8ff9640 10 monclient: dump: 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39baffd640 4 set_mon_vals no callback set 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 10 monclient: _renew_subs 2024-09-15T03:33:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.930+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc11ce30 con 0x7f39bc09ea50 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c2f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 0x7f39bc11bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc065e00 con 0x7f39bc131140 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f39a8039040 con 0x7f39bc09ea50 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 0x7f39bc11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient: get_auth_request con 0x7f39bc131140 auth_method 0 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39b8ff9640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f39a0038740 0x7f39a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(128..128 src has 40..128) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f39a806c580 con 0x7f39bc09ea50 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 10 monclient(hunting): handle_auth_done global_id 45268 payload 824 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c0d39640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39bc11e490 con 0x7f39bc131140 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.931+0000 7f39c153a640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f39a0038740 0x7f39a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.932+0000 7f39c153a640 10 monclient: get_auth_request con 0x7f39a0038740 auth_method 0 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39c0d39640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 msgr2=0x7f39bc11bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39c0d39640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 0x7f39bc11bc60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f30 tx=0x7f39b00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0014860 con 0x7f39bc131140 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 10 monclient: dump: 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39c153a640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f39a0038740 0x7f39a003abf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f39b4007fd0 tx=0x7f39b40025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f39b00149c0 con 0x7f39bc131140 2024-09-15T03:33:54.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:33:54.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:33:54.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 msgr2=0x7f39bc11bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 0x7f39bc11bc60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f30 tx=0x7f39b00090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 msgr2=0x7f39bc11bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.934+0000 7f39b8ff9640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc131140 0x7f39bc11bc60 secure :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f30 tx=0x7f39b00090f0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c2f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:33:54.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 0x7f39bc096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c2f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc131510 con 0x7f39bc11c1a0 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 0x7f39bc096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient: get_auth_request con 0x7f39bc11c1a0 auth_method 0 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 10 monclient(hunting): handle_auth_done global_id 45271 payload 824 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.940+0000 7f39c0d39640 1 -- 172.21.15.153:0/2331038085 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f39bc11c5a0 con 0x7f39bc11c1a0 2024-09-15T03:33:54.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39c0d39640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 msgr2=0x7f39bc096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39c0d39640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 0x7f39bc096a60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f30 tx=0x7f39b001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0024040 con 0x7f39bc11c1a0 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 10 monclient: dump: 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.954 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f39b00133a0 con 0x7f39bc11c1a0 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 msgr2=0x7f39bc096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 0x7f39bc096a60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f30 tx=0x7f39b001c000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 msgr2=0x7f39bc096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.943+0000 7f39b8ff9640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39bc11c1a0 0x7f39bc096a60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:33:54.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 60, 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:33:54.956 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:33:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:33:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:33:54.960 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:33:54.961 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.962 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.963 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:33:54.964 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:33:54.966 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.967 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:33:54.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:33:54.969 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:33:54.969 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f39a0038740 msgr2=0x7f39a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f39a0038740 0x7f39a003abf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f39b4007fd0 tx=0x7f39b40025b0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 10 monclient: shutdown 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 msgr2=0x7f39bc130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 --2- 172.21.15.153:0/2331038085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc09ea50 0x7f39bc130c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39a8035040 tx=0x7f39a8032000 comp rx=0 tx=0).stop 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c153a640 1 -- 172.21.15.153:0/2331038085 reap_dead start 2024-09-15T03:33:54.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c153a640 1 -- 172.21.15.153:0/2331038085 reap_dead start 2024-09-15T03:33:54.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 shutdown_connections 2024-09-15T03:33:54.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 >> 172.21.15.153:0/2331038085 conn(0x7f39bc067110 msgr2=0x7f39bc0a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:33:54.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 shutdown_connections 2024-09-15T03:33:54.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:33:54.944+0000 7f39c2f9d640 1 -- 172.21.15.153:0/2331038085 wait complete. 2024-09-15T03:33:54.971 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:33:54.971 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:33:54.972 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-15T03:33:54.972 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:33:54.972 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:33:54.972 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:33:54.972 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:33:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 msgr2=0x7f069812ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:33:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0697fff640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 msgr2=0x7f069812ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:33:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0697fff640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f068400d890 tx=0x7f0684008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:33:54.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0697fff640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f068400d890 tx=0x7f0684008040 comp rx=0 tx=0).stop 2024-09-15T03:33:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 10 monclient: _reopen_session rank -1 2024-09-15T03:33:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 msgr2=0x7f069812ea00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:33:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069809cee0 0x7f069812ea00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f067c044680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 10 monclient(hunting): picked mon.c con 0x7f069809c080 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.976+0000 7f0694ff9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f067c044e50 0x7f067c0472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 10 monclient(hunting): picked mon.b con 0x7f067c044e50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f067c049db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069d6ba640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f067c044e50 msgr2=0x7f067c0472d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069d6ba640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f067c044e50 0x7f067c0472d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 10 monclient(hunting): picked mon.a con 0x7f069809f180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c04a730 con 0x7f069809f180 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c04a870 con 0x7f067c044e50 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c04aa00 con 0x7f069809c080 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0694ff9640 10 monclient(hunting): _renew_subs 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f067c044680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): get_auth_request con 0x7f069809c080 auth_method 0 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f067c049db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): get_auth_request con 0x7f069809f180 auth_method 0 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:33:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f0697fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient(hunting): handle_auth_done global_id 45259 payload 293 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 msgr2=0x7f067c044680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f067c044680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f067c044e50 msgr2=0x7f067c0472d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f067c044e50 0x7f067c0472d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient: _finish_hunting 0 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 monclient: found mon.a 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f067c04abc0 con 0x7f069809f180 2024-09-15T03:33:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 10 monclient: _finish_auth 0 2024-09-15T03:33:54.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.977+0000 7f069ceb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:54.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.982+0000 7f069ceb9640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 msgr2=0x7f067c049db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:33:54.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.982+0000 7f069ceb9640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f067c049db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f068c031000 tx=0x7f068c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c03c880 con 0x7f069809f180 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0694ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0694ff9640 10 monclient: dump: 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:33:54.985 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:33:54.986 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:33:54.986 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:33:54.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068c03c9e0 con 0x7f069809f180 2024-09-15T03:33:54.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0694ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:33:54.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.983+0000 7f0696ffd640 4 set_mon_vals no callback set 2024-09-15T03:33:54.987 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:33:54.987 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T03:33:54.987 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T03:33:54.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:33:54.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:54.984+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f068c039040 con 0x7f069809f180 2024-09-15T03:33:55.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:55.041+0000 7f069f11d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T03:33:55.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:55.041+0000 7f069f11d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T03:33:55.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:55.041+0000 7f069f11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:33:55.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:55.041+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f069811bc30 con 0x7f069809f180 2024-09-15T03:33:57.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:57.864+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:33:57.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:57.864+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:57.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:57.864+0000 7f06837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:33:58.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:58.865+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:33:58.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:58.865+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:33:59.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:59.865+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:33:59.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:33:59.865+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:00.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:00.865+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:00.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:00.865+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:01.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:01.864+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:01.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:01.864+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:02.865+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:02.865+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:03.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:03.864+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:03.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:03.864+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:04.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:04.865+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:05.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:04.865+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:05.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:05.866+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:05.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:05.866+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:06.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:06.866+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:06.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:06.866+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:07.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:07.866+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:07.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:07.866+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:07.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:07.866+0000 7f06837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:08.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:08.866+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:08.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:08.866+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:09.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:09.866+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:09.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:09.866+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:09.988 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T03:34:09.989 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T03:34:09.989 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:34:09.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:34:10.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.087+0000 7f692359d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:10.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.087+0000 7f692359d640 10 monclient: build_initial_monmap 2024-09-15T03:34:10.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.087+0000 7f692359d640 10 monclient: monmap: 2024-09-15T03:34:10.089 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:10.089324+0000 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:10.089324+0000 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:10.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 1 Processor -- start 2024-09-15T03:34:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 1 -- start start 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient: init 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f691c096c60 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 0x7f691c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient(hunting): picked mon.noname-c con 0x7f691c097030 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f691c133b40 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c09d2e0 con 0x7f691c096c60 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c09d6b0 con 0x7f691c133b40 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0992f0 con 0x7f691c097030 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.088+0000 7f692359d640 10 monclient(hunting): authenticate will time out at 1047.728981s 2024-09-15T03:34:10.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.089+0000 7f6921339640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f691c096c60 0x7f691c09a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55842/0 (socket says 172.21.15.153:55842) 2024-09-15T03:34:10.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.089+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 learned_addr learned my addr 172.21.15.153:0/1664850054 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:34:10.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.089+0000 7f6921b3a640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f691c135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:10.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.089+0000 7f6920b38640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 msgr2=0x7f691c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:10.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.089+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 0x7f691c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:10.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.091+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.153:6789/0 conn(0x7f691c096c60 legacy=0x7f691c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.290+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 msgr2=0x7f691c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.290+0000 7f6921b3a640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f691c135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:10.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.290+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 0x7f691c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:34:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.691+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 msgr2=0x7f691c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.691+0000 7f6921b3a640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f691c135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:10.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:10.691+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 0x7f691c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:34:10.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:10.866+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:10.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:10.866+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:11.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:11.492+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 msgr2=0x7f691c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:11.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:11.492+0000 7f6921b3a640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f691c135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:11.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:11.492+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 0x7f691c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:34:11.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:11.867+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:11.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:11.867+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:12.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:12.867+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:12.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:12.867+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:13.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): tick 2024-09-15T03:34:13.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:34:13.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:34:13.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 msgr2=0x7f691c097430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:13.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f691c097030 0x7f691c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:13.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f691c135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.153:6789/0 conn(0x7f691c096c60 legacy=0x7f691c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient: _add_conns ranks=[2,3] 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f6900003700 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f6900006270 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0992f0 con 0x7f6900006270 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6900008de0 con 0x7f6900003700 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6912ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:13.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 10 monclient(hunting): get_auth_request con 0x7f6900006270 auth_method 0 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 10 monclient(hunting): get_auth_request con 0x7f6900003700 auth_method 0 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:13.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:13.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:13.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.089+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 10 monclient(hunting): get_auth_request con 0x7f6900006270 auth_method 0 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 10 monclient(hunting): get_auth_request con 0x7f6900003700 auth_method 0 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:13.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:13.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:13.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:13.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:34:13.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.290+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:34:13.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:13.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:13.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 10 monclient(hunting): get_auth_request con 0x7f6900006270 auth_method 0 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 10 monclient(hunting): get_auth_request con 0x7f6900003700 auth_method 0 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:13.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:13.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:13.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:34:13.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:13.691+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:34:13.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:13.867+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:13.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:13.867+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:14.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 10 monclient(hunting): get_auth_request con 0x7f6900003700 auth_method 0 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 10 monclient(hunting): get_auth_request con 0x7f6900006270 auth_method 0 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:14.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:34:14.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:14.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:14.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:14.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:14.492+0000 7f6920b38640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:34:14.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:14.867+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:14.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:14.867+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:15.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:15.867+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:15.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:15.867+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.010+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v130) v1 ==== 89+0+0 (secure 0 0 0) 0x7f068c037070 con 0x7f069809f180 2024-09-15T03:34:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.010+0000 7f0694ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T03:34:16.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.010+0000 7f0694ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): tick 2024-09-15T03:34:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:34:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:34:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 msgr2=0x7f6900008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.153:3300/0 conn(0x7f6900006270 0x7f6900008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f6900003700 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921b3a640 1 -- 172.21.15.153:0/1664850054 reap_dead start 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f691c133b40 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f6900006690 0x7f690000eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f6900006690 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0995d0 con 0x7f691c133b40 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0992f0 con 0x7f6900006690 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 1 -- 172.21.15.153:0/1664850054 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6900008de0 con 0x7f6900003700 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6912ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921b3a640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f6900006690 msgr2=0x7f690000eb40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6920b38640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f690000d420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921b3a640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f6900006690 0x7f690000eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): get_auth_request con 0x7f6900003700 auth_method 0 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient(hunting): handle_auth_done global_id 74333 payload 274 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f6900006690 msgr2=0x7f690000eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f6900006690 0x7f690000eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> v1:172.21.15.181:6789/0 conn(0x7f691c133b40 legacy=0x7f690000d420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 10 monclient: _finish_hunting 0 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.089+0000 7f6921339640 1 monclient: found mon.noname-b 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.090+0000 7f6921339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.090+0000 7f6921339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:16.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.090+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c09d6b0 con 0x7f6900003700 2024-09-15T03:34:16.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.092+0000 7f6921339640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.092+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005ae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6918007fd0 tx=0x7f6918009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2cf6a0133386c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6921339640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005ae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6918007fd0 tx=0x7f6918009040 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f691802f070 config(0 keys) v1 2024-09-15T03:34:16.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 1 -- 172.21.15.153:0/1664850054 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691802b870 con 0x7f6900003700 2024-09-15T03:34:16.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 10 monclient: dump: 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.104 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 10 monclient: _finish_auth 0 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 1 -- 172.21.15.153:0/1664850054 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691802f070 con 0x7f6900003700 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 5 monclient: authenticate success, global_id 74333 2024-09-15T03:34:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f6913fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 4 set_mon_vals no callback set 2024-09-15T03:34:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 10 monclient: shutdown 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 1 -- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 msgr2=0x7f6900005ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.093+0000 7f692359d640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005ae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6918007fd0 tx=0x7f6918009040 comp rx=0 tx=0).stop 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f6913fff640 1 -- 172.21.15.153:0/1664850054 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f691802bc60 con 0x7f6900003700 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f6913fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 1 -- 172.21.15.153:0/1664850054 shutdown_connections 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 1 --2- 172.21.15.153:0/1664850054 >> v2:172.21.15.181:3300/0 conn(0x7f6900006690 0x7f690000eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 1 --2- 172.21.15.153:0/1664850054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900003700 0x7f6900005ae0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 1 -- 172.21.15.153:0/1664850054 >> 172.21.15.153:0/1664850054 conn(0x7f691c0672b0 msgr2=0x7f691c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 1 -- 172.21.15.153:0/1664850054 shutdown_connections 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 1 -- 172.21.15.153:0/1664850054 wait complete. 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 10 monclient: build_initial_monmap 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.094+0000 7f692359d640 10 monclient: monmap: 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.095+0000 7f692359d640 1 Processor -- start 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.095+0000 7f692359d640 1 -- start start 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.095+0000 7f692359d640 10 monclient: init 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 0x7f691c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): picked mon.noname-a con 0x7f6900003700 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900006690 0x7f691c12e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): picked mon.noname-c con 0x7f6900006690 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f691c133b40 0x7f691c1410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): picked mon.noname-b con 0x7f691c133b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c11dcf0 con 0x7f6900003700 2024-09-15T03:34:16.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c11dfd0 con 0x7f691c133b40 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c11e2b0 con 0x7f6900006690 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f692359d640 10 monclient(hunting): authenticate will time out at 1053.736117s 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921b3a640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f691c133b40 msgr2=0x7f691c1410c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921b3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f691c133b40 0x7f691c1410c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 0x7f691c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900006690 0x7f691c12e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 0x7f691c06e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41394/0 (socket says 172.21.15.153:41394) 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 -- 172.21.15.153:0/1987402786 learned_addr learned my addr 172.21.15.153:0/1987402786 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): get_auth_request con 0x7f6900003700 auth_method 0 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): get_auth_request con 0x7f6900006690 auth_method 0 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient(hunting): handle_auth_done global_id 84213 payload 824 2024-09-15T03:34:16.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 -- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900006690 msgr2=0x7f691c12e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900006690 0x7f691c12e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 -- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f691c133b40 msgr2=0x7f691c1410c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f691c133b40 0x7f691c1410c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient: _finish_hunting 0 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 monclient: found mon.noname-a 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6921339640 1 -- 172.21.15.153:0/1987402786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691800f040 con 0x7f6900003700 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.096+0000 7f6920b38640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900006690 0x7f691c12e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f6921339640 1 -- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 msgr2=0x7f691c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f6921339640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 0x7f691c06e3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6918000c00 tx=0x7f6918037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918043070 con 0x7f6900003700 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: dump: 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.117 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: _finish_auth 0 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918035cb0 con 0x7f6900003700 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 5 monclient: authenticate success, global_id 84213 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 10 monclient: _renew_subs 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c11c7b0 con 0x7f6900003700 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918005280 con 0x7f6900003700 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69117fa640 10 monclient: dump: 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f69137fe640 4 set_mon_vals no callback set 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 10 monclient: _renew_subs 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.099+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c11cd40 con 0x7f6900003700 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f69180047a0 con 0x7f6900003700 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f69117fa640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6900039930 0x7f690003bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f692359d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f692359d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f692359d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(130..130 src has 40..130) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f691806d4f0 con 0x7f6900003700 2024-09-15T03:34:16.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c04bf60 con 0x7f6900003700 2024-09-15T03:34:16.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f6920b38640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6900039930 0x7f690003bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.100+0000 7f6920b38640 10 monclient: get_auth_request con 0x7f6900039930 auth_method 0 2024-09-15T03:34:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.114+0000 7f069f11d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T03:34:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.114+0000 7f069f11d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T03:34:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.114+0000 7f069f11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.114+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f06980008d0 con 0x7f069809f180 2024-09-15T03:34:16.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.103+0000 7f6920b38640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6900039930 0x7f690003bde0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f690c004fd0 tx=0x7f690c005040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.105+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6918004a50 con 0x7f6900003700 2024-09-15T03:34:16.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.105+0000 7f69117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.105+0000 7f69117fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.324+0000 7f692359d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:34:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.324+0000 7f692359d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:34:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.324+0000 7f692359d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.324+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f691c04bf60 con 0x7f6900003700 2024-09-15T03:34:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.325+0000 7f69117fa640 1 -- 172.21.15.153:0/1987402786 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f691803e3b0 con 0x7f6900003700 2024-09-15T03:34:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.325+0000 7f69117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:34:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.325+0000 7f69117fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:16.327 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.326+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6900039930 msgr2=0x7f690003bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.326+0000 7f692359d640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6900039930 0x7f690003bde0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f690c004fd0 tx=0x7f690c005040 comp rx=0 tx=0).stop 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.326+0000 7f692359d640 10 monclient: shutdown 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.326+0000 7f692359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.326+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 msgr2=0x7f691c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.326+0000 7f692359d640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 0x7f691c06e3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6918000c00 tx=0x7f6918037000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 shutdown_connections 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6900039930 0x7f690003bde0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f691c133b40 0x7f691c1410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6900006690 0x7f691c12e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 --2- 172.21.15.153:0/1987402786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6900003700 0x7f691c06e3c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 >> 172.21.15.153:0/1987402786 conn(0x7f691c0672b0 msgr2=0x7f691c131df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 shutdown_connections 2024-09-15T03:34:16.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.327+0000 7f692359d640 1 -- 172.21.15.153:0/1987402786 wait complete. 2024-09-15T03:34:16.339 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:34:16.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:34:16.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:34:16.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.430+0000 7f1868d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:16.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.430+0000 7f1868d9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:16.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.430+0000 7f1868d9d640 10 monclient: monmap: 2024-09-15T03:34:16.433 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:16.433 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.433 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:16.432502+0000 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:16.432502+0000 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.431+0000 7f1868d9d640 1 Processor -- start 2024-09-15T03:34:16.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.431+0000 7f1868d9d640 1 -- start start 2024-09-15T03:34:16.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.431+0000 7f1868d9d640 10 monclient: init 2024-09-15T03:34:16.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:16.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:34:16.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f18641318f0 0x7f1864131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f18641318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f186409ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1864136c20 0x7f1864139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1864136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186409f530 con 0x7f186409ea50 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1864131d50 con 0x7f1864136c20 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186409fbf0 con 0x7f18641318f0 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1868d9d640 10 monclient(hunting): authenticate will time out at 1054.072238s 2024-09-15T03:34:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1861d9b640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f186409ea50 legacy=0x7f186409ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:16.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f186259c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f18641318f0 msgr2=0x7f1864131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:16.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f186259c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f18641318f0 0x7f1864131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:16.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1864136c20 0x7f1864139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1864136c20 0x7f1864139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41410/0 (socket says 172.21.15.153:41410) 2024-09-15T03:34:16.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 -- 172.21.15.153:0/3509836320 learned_addr learned my addr 172.21.15.153:0/3509836320 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): get_auth_request con 0x7f1864136c20 auth_method 0 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient(hunting): handle_auth_done global_id 84216 payload 274 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 -- 172.21.15.153:0/3509836320 >> v2:172.21.15.181:3300/0 conn(0x7f18641318f0 msgr2=0x7f1864131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 --2- 172.21.15.153:0/3509836320 >> v2:172.21.15.181:3300/0 conn(0x7f18641318f0 0x7f1864131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 -- 172.21.15.153:0/3509836320 >> v1:172.21.15.181:6789/0 conn(0x7f186409ea50 legacy=0x7f186409ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 monclient: found mon.noname-a 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.432+0000 7f1862d9d640 1 -- 172.21.15.153:0/3509836320 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186409fe30 con 0x7f1864136c20 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f1862d9d640 1 -- 172.21.15.153:0/3509836320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864136c20 msgr2=0x7f1864139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f1862d9d640 1 --2- 172.21.15.153:0/3509836320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864136c20 0x7f1864139000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1864076600 tx=0x7f185c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0af717b79d135b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 1 -- 172.21.15.153:0/3509836320 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c02aea0 con 0x7f1864136c20 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: dump: 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: _finish_auth 0 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 1 -- 172.21.15.153:0/3509836320 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c02d050 con 0x7f1864136c20 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 1 -- 172.21.15.153:0/3509836320 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c007840 con 0x7f1864136c20 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f186159a640 10 monclient: dump: 2024-09-15T03:34:16.452 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f1868d9d640 5 monclient: authenticate success, global_id 84216 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f1868d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.435+0000 7f1868d9d640 4 set_mon_vals no callback set 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 10 monclient: shutdown 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 -- 172.21.15.153:0/3509836320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864136c20 msgr2=0x7f1864139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 --2- 172.21.15.153:0/3509836320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864136c20 0x7f1864139000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1864076600 tx=0x7f185c009840 comp rx=0 tx=0).stop 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 -- 172.21.15.153:0/3509836320 shutdown_connections 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 --2- 172.21.15.153:0/3509836320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864136c20 0x7f1864139000 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 --2- 172.21.15.153:0/3509836320 >> v2:172.21.15.181:3300/0 conn(0x7f18641318f0 0x7f1864131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 -- 172.21.15.153:0/3509836320 >> 172.21.15.153:0/3509836320 conn(0x7f1864067110 msgr2=0x7f1864067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:16.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 -- 172.21.15.153:0/3509836320 shutdown_connections 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 1 -- 172.21.15.153:0/3509836320 wait complete. 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.436+0000 7f1868d9d640 10 monclient: monmap: 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:16.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 Processor -- start 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 -- start start 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient: init 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f186409ea50 0x7f1864130770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f186409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 0x7f1864099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f18641318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1864136c20 0x7f1864099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f1864136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186409b910 con 0x7f18641318f0 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186409bf10 con 0x7f1864136c20 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186409de60 con 0x7f186409ea50 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1868d9d640 10 monclient(hunting): authenticate will time out at 1054.077741s 2024-09-15T03:34:16.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 0x7f1864099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f186409ea50 0x7f1864130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 0x7f1864099570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41416/0 (socket says 172.21.15.153:41416) 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 1 -- 172.21.15.153:0/2964788374 learned_addr learned my addr 172.21.15.153:0/2964788374 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f186409ea50 0x7f1864130770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38714/0 (socket says 172.21.15.153:38714) 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): get_auth_request con 0x7f18641318f0 auth_method 0 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1862d9d640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1864136c20 msgr2=0x7f1864099ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): get_auth_request con 0x7f186409ea50 auth_method 0 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1862d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1864136c20 0x7f1864099ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f1861d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.437+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 10 monclient(hunting): handle_auth_done global_id 84219 payload 824 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f186409ea50 msgr2=0x7f1864130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f186409ea50 0x7f1864130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1864136c20 msgr2=0x7f1864099ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1864136c20 0x7f1864099ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:34:16.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 1 monclient: found mon.noname-a 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.438+0000 7f1861d9b640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c009ae0 con 0x7f18641318f0 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1861d9b640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 msgr2=0x7f1864099570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1861d9b640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 0x7f1864099570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f185800b600 tx=0x7f18580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858015070 con 0x7f18641318f0 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: dump: 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.461 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1858003d90 con 0x7f18641318f0 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 5 monclient: authenticate success, global_id 84219 2024-09-15T03:34:16.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 10 monclient: _renew_subs 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186409b0e0 con 0x7f18641318f0 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858006380 con 0x7f18641318f0 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f18527fc640 10 monclient: dump: 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.463 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.464 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1860d99640 4 set_mon_vals no callback set 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 10 monclient: _renew_subs 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.440+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186411b920 con 0x7f18641318f0 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1858010440 con 0x7f18641318f0 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f1868d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f1868d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 0x7f18640298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1864065e00 con 0x7f1864141c10 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 0x7f18640298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient: get_auth_request con 0x7f1864141c10 auth_method 0 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f18527fc640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18440386f0 0x7f184403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(130..130 src has 40..130) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f185804c160 con 0x7f18641318f0 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 10 monclient(hunting): handle_auth_done global_id 84222 payload 824 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f186259c640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f186411c990 con 0x7f1864141c10 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f1862d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18440386f0 0x7f184403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.441+0000 7f1862d9d640 10 monclient: get_auth_request con 0x7f18440386f0 auth_method 0 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.443+0000 7f186259c640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 msgr2=0x7f18640298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.443+0000 7f186259c640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 0x7f18640298c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1854009590 tx=0x7f1854006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185400e990 con 0x7f1864141c10 2024-09-15T03:34:16.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 10 monclient: dump: 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f185400eaf0 con 0x7f1864141c10 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 msgr2=0x7f18640298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 0x7f18640298c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1854009590 tx=0x7f1854006f90 comp rx=0 tx=0).stop 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 msgr2=0x7f18640298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f18527fc640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864141c10 0x7f18640298c0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1854009590 tx=0x7f1854006f90 comp rx=0 tx=0).stop 2024-09-15T03:34:16.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.444+0000 7f1862d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18440386f0 0x7f184403aba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f186406f600 tx=0x7f185c036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.449+0000 7f1868d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f1868d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 0x7f1864029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f1868d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186404bf60 con 0x7f1864130cb0 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 0x7f1864029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient: get_auth_request con 0x7f1864130cb0 auth_method 0 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 10 monclient(hunting): handle_auth_done global_id 84225 payload 824 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.450+0000 7f186259c640 1 -- 172.21.15.153:0/2964788374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1864135eb0 con 0x7f1864130cb0 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f186259c640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 msgr2=0x7f1864029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f186259c640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 0x7f1864029e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1854009590 tx=0x7f1854019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854022070 con 0x7f1864130cb0 2024-09-15T03:34:16.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 10 monclient: dump: 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f185401d2e0 con 0x7f1864130cb0 2024-09-15T03:34:16.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 msgr2=0x7f1864029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 0x7f1864029e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1854009590 tx=0x7f1854019000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 msgr2=0x7f1864029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.452+0000 7f18527fc640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1864130cb0 0x7f1864029e00 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1854009590 tx=0x7f1854019000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:34:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 62, 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:34:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:34:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:34:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:34:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:34:16.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18440386f0 msgr2=0x7f184403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18440386f0 0x7f184403aba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f186406f600 tx=0x7f185c036000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 10 monclient: shutdown 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 msgr2=0x7f1864099570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 --2- 172.21.15.153:0/2964788374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18641318f0 0x7f1864099570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f185800b600 tx=0x7f18580091a0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1862d9d640 1 -- 172.21.15.153:0/2964788374 reap_dead start 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1862d9d640 1 -- 172.21.15.153:0/2964788374 reap_dead start 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 shutdown_connections 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 >> 172.21.15.153:0/2964788374 conn(0x7f1864067110 msgr2=0x7f18640a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 shutdown_connections 2024-09-15T03:34:16.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.453+0000 7f1868d9d640 1 -- 172.21.15.153:0/2964788374 wait complete. 2024-09-15T03:34:16.498 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:34:16.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:34:16.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:34:16.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.591+0000 7fde5675d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:16.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.591+0000 7fde5675d640 10 monclient: build_initial_monmap 2024-09-15T03:34:16.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.591+0000 7fde5675d640 10 monclient: monmap: 2024-09-15T03:34:16.593 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:16.593 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:16.592871+0000 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:16.592871+0000 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:16.594 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.591+0000 7fde5675d640 1 Processor -- start 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 1 -- start start 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient: init 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde500a5760 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 0x7fde5009f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): picked mon.noname-b con 0x7fde5009ce40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fde50136b60 0x7fde50138f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): picked mon.noname-c con 0x7fde50136b60 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:16.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5009ffc0 con 0x7fde500a5760 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde500a1080 con 0x7fde50136b60 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde500a5b30 con 0x7fde5009ce40 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde5675d640 10 monclient(hunting): authenticate will time out at 1054.232557s 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 0x7fde5009f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde54cfa640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fde50136b60 msgr2=0x7fde50138f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde4ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fde500a5760 0x7fde5009e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45872/0 (socket says 172.21.15.153:45872) 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde4ffff640 1 -- 172.21.15.153:0/1454274763 learned_addr learned my addr 172.21.15.153:0/1454274763 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:34:16.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): get_auth_request con 0x7fde5009ce40 auth_method 0 2024-09-15T03:34:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde54cfa640 1 --2- 172.21.15.153:0/1454274763 >> v2:172.21.15.181:3300/0 conn(0x7fde50136b60 0x7fde50138f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient(hunting): handle_auth_done global_id 74342 payload 274 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 1 -- 172.21.15.153:0/1454274763 >> v2:172.21.15.181:3300/0 conn(0x7fde50136b60 msgr2=0x7fde50138f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 1 --2- 172.21.15.153:0/1454274763 >> v2:172.21.15.181:3300/0 conn(0x7fde50136b60 0x7fde50138f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 1 -- 172.21.15.153:0/1454274763 >> v1:172.21.15.153:6789/0 conn(0x7fde500a5760 legacy=0x7fde5009e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient: _finish_hunting 0 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 1 monclient: found mon.noname-b 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.592+0000 7fde47fff640 1 -- 172.21.15.153:0/1454274763 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5009d240 con 0x7fde5009ce40 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde47fff640 1 -- 172.21.15.153:0/1454274763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 msgr2=0x7fde5009f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde47fff640 1 --2- 172.21.15.153:0/1454274763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 0x7fde5009f010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fde38009850 tx=0x7fde380090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4085a04707dac5f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 1 -- 172.21.15.153:0/1454274763 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38032440 con 0x7fde5009ce40 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: dump: 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde5675d640 5 monclient: authenticate success, global_id 74342 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 1 -- 172.21.15.153:0/1454274763 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde380325a0 con 0x7fde5009ce40 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde5675d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde5675d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde5675d640 4 set_mon_vals no callback set 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 1 -- 172.21.15.153:0/1454274763 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38032940 con 0x7fde5009ce40 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.595+0000 7fde4f7fe640 10 monclient: dump: 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 10 monclient: shutdown 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 -- 172.21.15.153:0/1454274763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 msgr2=0x7fde5009f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 --2- 172.21.15.153:0/1454274763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 0x7fde5009f010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fde38009850 tx=0x7fde380090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 -- 172.21.15.153:0/1454274763 shutdown_connections 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 --2- 172.21.15.153:0/1454274763 >> v2:172.21.15.181:3300/0 conn(0x7fde50136b60 0x7fde50138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 --2- 172.21.15.153:0/1454274763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde5009ce40 0x7fde5009f010 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 -- 172.21.15.153:0/1454274763 >> 172.21.15.153:0/1454274763 conn(0x7fde50067680 msgr2=0x7fde500a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 -- 172.21.15.153:0/1454274763 shutdown_connections 2024-09-15T03:34:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.596+0000 7fde5675d640 1 -- 172.21.15.153:0/1454274763 wait complete. 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.597+0000 7fde5675d640 10 monclient: build_initial_monmap 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.597+0000 7fde5675d640 10 monclient: monmap: 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:16.613 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.597+0000 7fde5675d640 1 Processor -- start 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.597+0000 7fde5675d640 1 -- start start 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.597+0000 7fde5675d640 10 monclient: init 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde5009ce40 0x7fde50134f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): picked mon.noname-b con 0x7fde5009ce40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde500a5760 0x7fde50135490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): picked mon.noname-c con 0x7fde500a5760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:16.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 0x7fde501359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): picked mon.noname-a con 0x7fde50136b60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5009bde0 con 0x7fde50136b60 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5011f7b0 con 0x7fde5009ce40 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5011ed80 con 0x7fde500a5760 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde5675d640 10 monclient(hunting): authenticate will time out at 1054.238291s 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde4ffff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde5009ce40 msgr2=0x7fde50134f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde5009ce40 0x7fde50134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde500a5760 0x7fde50135490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 0x7fde501359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde500a5760 0x7fde50135490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38736/0 (socket says 172.21.15.153:38736) 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 1 -- 172.21.15.153:0/2745012551 learned_addr learned my addr 172.21.15.153:0/2745012551 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): get_auth_request con 0x7fde50136b60 auth_method 0 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): get_auth_request con 0x7fde500a5760 auth_method 0 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient(hunting): handle_auth_done global_id 84228 payload 824 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde500a5760 msgr2=0x7fde50135490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde500a5760 0x7fde50135490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde5009ce40 msgr2=0x7fde50134f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde5009ce40 0x7fde50134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 monclient: found mon.noname-a 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.598+0000 7fde54cfa640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde38009390 con 0x7fde50136b60 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.600+0000 7fde54cfa640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 msgr2=0x7fde501359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.600+0000 7fde54cfa640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 0x7fde501359d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde4800bf10 tx=0x7fde48009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48006150 con 0x7fde50136b60 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: dump: 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.619 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: _finish_auth 0 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde480062d0 con 0x7fde50136b60 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 5 monclient: authenticate success, global_id 84228 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 10 monclient: _renew_subs 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5011eae0 con 0x7fde50136b60 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48006480 con 0x7fde50136b60 2024-09-15T03:34:16.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4cff9640 10 monclient: dump: 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 10 monclient: _renew_subs 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5011fcb0 con 0x7fde50136b60 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.601+0000 7fde4effd640 4 set_mon_vals no callback set 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde5675d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fde48012020 con 0x7fde50136b60 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde5675d640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 0x7fde50098e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde50065e70 con 0x7fde50098a70 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4ffff640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 0x7fde50098e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4cff9640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde300386f0 0x7fde3003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4ffff640 10 monclient: get_auth_request con 0x7fde50098a70 auth_method 0 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.602+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(130..130 src has 40..130) v4 ==== 7825+0+0 (secure 0 0 0) 0x7fde4804c8d0 con 0x7fde50136b60 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde4ffff640 10 monclient(hunting): handle_auth_done global_id 74348 payload 824 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde47fff640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde300386f0 0x7fde3003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde4ffff640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde5011dfd0 con 0x7fde50098a70 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.603+0000 7fde47fff640 10 monclient: get_auth_request con 0x7fde300386f0 auth_method 0 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4ffff640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 msgr2=0x7fde50098e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4ffff640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 0x7fde50098e40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde400047d0 tx=0x7fde40007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40005d50 con 0x7fde50098a70 2024-09-15T03:34:16.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 10 monclient: dump: 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.624 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde47fff640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde300386f0 0x7fde3003aba0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde38009130 tx=0x7fde38032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fde400070f0 con 0x7fde50098a70 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 msgr2=0x7fde50098e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 0x7fde50098e40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde400047d0 tx=0x7fde40007f00 comp rx=0 tx=0).stop 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 msgr2=0x7fde50098e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:16.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.605+0000 7fde4cff9640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50098a70 0x7fde50098e40 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde400047d0 tx=0x7fde40007f00 comp rx=0 tx=0).stop 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde5675d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde5675d640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 0x7fde5011c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde5675d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde50120040 con 0x7fde50099380 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 0x7fde5011c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient: get_auth_request con 0x7fde50099380 auth_method 0 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:16.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 10 monclient(hunting): handle_auth_done global_id 74351 payload 824 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.611+0000 7fde4ffff640 1 -- 172.21.15.153:0/2745012551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fde5004bf60 con 0x7fde50099380 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4ffff640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 msgr2=0x7fde5011c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4ffff640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 0x7fde5011c330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fde40007fd0 tx=0x7fde40016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40021630 con 0x7fde50099380 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 10 monclient: dump: 2024-09-15T03:34:16.629 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fde40021790 con 0x7fde50099380 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:16.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 msgr2=0x7fde5011c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 0x7fde5011c330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fde40007fd0 tx=0x7fde40016000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 msgr2=0x7fde5011c330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde4cff9640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde50099380 0x7fde5011c330 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fde40007fd0 tx=0x7fde40016000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:34:16.631 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 62, 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:16.632 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:16.633 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.634 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:34:16.635 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:16.636 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.637 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.638 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:16.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:34:16.642 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:16.643 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.644 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.645 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:16.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.645 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.645 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:34:16.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:16.646 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:34:16.647 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:34:16.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde300386f0 msgr2=0x7fde3003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde300386f0 0x7fde3003aba0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde38009130 tx=0x7fde38032000 comp rx=0 tx=0).stop 2024-09-15T03:34:16.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 10 monclient: shutdown 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 msgr2=0x7fde501359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 0x7fde501359d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde4800bf10 tx=0x7fde48009740 comp rx=0 tx=0).stop 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde54cfa640 1 -- 172.21.15.153:0/2745012551 reap_dead start 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 shutdown_connections 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 --2- 172.21.15.153:0/2745012551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde50136b60 0x7fde501359d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.614+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 >> 172.21.15.153:0/2745012551 conn(0x7fde50067680 msgr2=0x7fde5009d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.615+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 shutdown_connections 2024-09-15T03:34:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:16.615+0000 7fde5675d640 1 -- 172.21.15.153:0/2745012551 wait complete. 2024-09-15T03:34:16.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:34:16.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:34:16.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.867+0000 7f06837fe640 10 monclient: tick 2024-09-15T03:34:16.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:16.867+0000 7f06837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.013+0000 7f0694ff9640 1 -- 172.21.15.181:0/833609021 <== mon.0 v2:172.21.15.153:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v131) v1 ==== 89+0+0 (secure 0 0 0) 0x7f068c05e0a0 con 0x7f069809f180 2024-09-15T03:34:17.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.013+0000 7f0694ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-15T03:34:17.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.013+0000 7f0694ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:17.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f067c038740 msgr2=0x7f067c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f067c038740 0x7f067c03abf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f069809fd80 tx=0x7f068c032000 comp rx=0 tx=0).stop 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 10 monclient: shutdown 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 msgr2=0x7f067c049db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f067c049db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f068c031000 tx=0x7f068c03d000 comp rx=0 tx=0).stop 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 shutdown_connections 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f069809f180 0x7f067c049db0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f067c044e50 0x7f067c0472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f069809c080 0x7f067c044680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 --2- 172.21.15.181:0/833609021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f067c038740 0x7f067c03abf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 >> 172.21.15.181:0/833609021 conn(0x7f06980a2b70 msgr2=0x7f069809b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:17.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 shutdown_connections 2024-09-15T03:34:17.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.015+0000 7f069f11d640 1 -- 172.21.15.181:0/833609021 wait complete. 2024-09-15T03:34:17.028 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:34:17.028 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c fooset 2024-09-15T03:34:17.028 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 0 2024-09-15T03:34:17.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.104+0000 7f82aa08b640 10 monclient: get_monmap_and_config 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.104+0000 7f82aa08b640 10 monclient: build_initial_monmap 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.104+0000 7f82aa08b640 10 monclient: monmap: 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:17.105706+0000 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:17.105706+0000 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:17.107 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.105+0000 7f82aa08b640 1 Processor -- start 2024-09-15T03:34:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.105+0000 7f82aa08b640 1 -- start start 2024-09-15T03:34:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.105+0000 7f82aa08b640 10 monclient: init 2024-09-15T03:34:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:34:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82a409a780 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82a409d610 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4137f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient(hunting): picked mon.noname-b con 0x7f82a409b8e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a409c590 con 0x7f82a409d610 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a409b170 con 0x7f82a409a780 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a41307e0 con 0x7f82a409b8e0 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82aa08b640 10 monclient(hunting): authenticate will time out at 1061.587265s 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a37fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f82a409a780 legacy=0x7f82a412f9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f82a409d610 0x7f82a41300d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:58766/0 (socket says 172.21.15.181:58766) 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a2ffd640 1 -- 172.21.15.181:0/1896886199 learned_addr learned my addr 172.21.15.181:0/1896886199 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 1 --2- 172.21.15.181:0/1896886199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): get_auth_request con 0x7f82a409b8e0 auth_method 0 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient(hunting): handle_auth_done global_id 74354 payload 274 2024-09-15T03:34:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 1 -- 172.21.15.181:0/1896886199 >> v1:172.21.15.181:6789/0 conn(0x7f82a409a780 legacy=0x7f82a412f9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 1 -- 172.21.15.181:0/1896886199 >> v1:172.21.15.153:6789/0 conn(0x7f82a409d610 legacy=0x7f82a41300d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 1 monclient: found mon.noname-b 2024-09-15T03:34:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.106+0000 7f82a3fff640 1 -- 172.21.15.181:0/1896886199 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a409c2b0 con 0x7f82a409b8e0 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.109+0000 7f82a3fff640 1 -- 172.21.15.181:0/1896886199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 msgr2=0x7f82a4137f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.109+0000 7f82a3fff640 1 --2- 172.21.15.181:0/1896886199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4137f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f829c009e90 tx=0x7f829c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00be1f5981346f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 1 -- 172.21.15.181:0/1896886199 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c007840 con 0x7f82a409b8e0 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: dump: 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 1 -- 172.21.15.181:0/1896886199 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c02d050 con 0x7f82a409b8e0 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 1 -- 172.21.15.181:0/1896886199 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c007bd0 con 0x7f82a409b8e0 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82a27fc640 10 monclient: dump: 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82aa08b640 5 monclient: authenticate success, global_id 74354 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82aa08b640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.111+0000 7f82aa08b640 4 set_mon_vals no callback set 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 10 monclient: shutdown 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 -- 172.21.15.181:0/1896886199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 msgr2=0x7f82a4137f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 --2- 172.21.15.181:0/1896886199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4137f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f829c009e90 tx=0x7f829c009780 comp rx=0 tx=0).stop 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 -- 172.21.15.181:0/1896886199 shutdown_connections 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 --2- 172.21.15.181:0/1896886199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4137f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 -- 172.21.15.181:0/1896886199 >> 172.21.15.181:0/1896886199 conn(0x7f82a4071510 msgr2=0x7f82a40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 -- 172.21.15.181:0/1896886199 shutdown_connections 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 1 -- 172.21.15.181:0/1896886199 wait complete. 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 10 monclient: build_initial_monmap 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.112+0000 7f82aa08b640 10 monclient: monmap: 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 Processor -- start 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 -- start start 2024-09-15T03:34:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient: init 2024-09-15T03:34:17.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:17.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 0x7f82a41370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): picked mon.noname-a con 0x7f82a409a780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:17.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4132cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): picked mon.noname-c con 0x7f82a409b8e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a409d610 0x7f82a4133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): picked mon.noname-b con 0x7f82a409d610 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a409bdd0 con 0x7f82a409a780 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a40a0940 con 0x7f82a409d610 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a41347e0 con 0x7f82a409b8e0 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82aa08b640 10 monclient(hunting): authenticate will time out at 1061.594741s 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a3fff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a409d610 msgr2=0x7f82a4133230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a409d610 0x7f82a4133230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 0x7f82a41370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4132cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 0x7f82a41370b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58438/0 (socket says 172.21.15.181:58438) 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 1 -- 172.21.15.181:0/3996393615 learned_addr learned my addr 172.21.15.181:0/3996393615 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 10 monclient(hunting): get_auth_request con 0x7f82a409a780 auth_method 0 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a2ffd640 10 monclient(hunting): get_auth_request con 0x7f82a409b8e0 auth_method 0 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.113+0000 7f82a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 10 monclient(hunting): handle_auth_done global_id 74357 payload 824 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 1 -- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a409d610 msgr2=0x7f82a4133230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a409d610 0x7f82a4133230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 1 -- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 msgr2=0x7f82a41370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 0x7f82a41370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 1 monclient: found mon.noname-c 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a2ffd640 1 -- 172.21.15.181:0/3996393615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c009a20 con 0x7f82a409b8e0 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.114+0000 7f82a37fe640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 0x7f82a41370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82a2ffd640 1 -- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 msgr2=0x7f82a4132cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82a2ffd640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4132cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f829800c490 tx=0x7f8298007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298005cb0 con 0x7f82a409b8e0 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: dump: 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8298005e30 con 0x7f82a409b8e0 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 5 monclient: authenticate success, global_id 74357 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 10 monclient: _renew_subs 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a4136370 con 0x7f82a409b8e0 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298007300 con 0x7f82a409b8e0 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f828b7fe640 10 monclient: dump: 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.131 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 10 monclient: _renew_subs 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82a1ffb640 4 set_mon_vals no callback set 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.116+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a4135dd0 con 0x7f82a409b8e0 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.117+0000 7f82aa08b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.117+0000 7f82aa08b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.117+0000 7f82aa08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.117+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a4136050 con 0x7f82a409b8e0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.117+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f829801b3b0 con 0x7f82a409b8e0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.118+0000 7f828b7fe640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8290038740 0x7f829003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.118+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(131..131 src has 106..131) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f829804bbc0 con 0x7f82a409b8e0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.118+0000 7f82a37fe640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8290038740 0x7f829003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.118+0000 7f82a37fe640 10 monclient: get_auth_request con 0x7f8290038740 auth_method 0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.120+0000 7f82a37fe640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8290038740 0x7f829003abf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8294003850 tx=0x7f8294009120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.122+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f829801a360 con 0x7f82a409b8e0 2024-09-15T03:34:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.122+0000 7f828b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.122+0000 7f828b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.282+0000 7f82aa08b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.282+0000 7f82aa08b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.282+0000 7f82aa08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.282+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f82a404bf60 con 0x7f82a409b8e0 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.283+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) v1 ==== 71+0+7 (secure 0 0 0) 0x7f8298017070 con 0x7f82a409b8e0 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.283+0000 7f828b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.283+0000 7f828b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:17.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.379+0000 7f82aa08b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.379+0000 7f82aa08b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.379+0000 7f82aa08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.379+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f82a406ccf0 con 0x7f82a409b8e0 2024-09-15T03:34:17.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.380+0000 7f828b7fe640 1 -- 172.21.15.181:0/3996393615 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) v1 ==== 71+0+7 (secure 0 0 0) 0x7f8298018260 con 0x7f82a409b8e0 2024-09-15T03:34:17.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.380+0000 7f828b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.380+0000 7f828b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:17.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8290038740 msgr2=0x7f829003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8290038740 0x7f829003abf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8294003850 tx=0x7f8294009120 comp rx=0 tx=0).stop 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 10 monclient: shutdown 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 msgr2=0x7f82a4132cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4132cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f829800c490 tx=0x7f8298007f00 comp rx=0 tx=0).stop 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 shutdown_connections 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8290038740 0x7f829003abf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82a409d610 0x7f82a4133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a409b8e0 0x7f82a4132cf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 --2- 172.21.15.181:0/3996393615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a409a780 0x7f82a41370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 >> 172.21.15.181:0/3996393615 conn(0x7f82a4071510 msgr2=0x7f82a409f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 shutdown_connections 2024-09-15T03:34:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.382+0000 7f82aa08b640 1 -- 172.21.15.181:0/3996393615 wait complete. 2024-09-15T03:34:17.393 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:34:17.393 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep barset 2024-09-15T03:34:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.464+0000 7f0007f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.464+0000 7f0007f5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.464+0000 7f0007f5d640 10 monclient: monmap: 2024-09-15T03:34:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:17.465231+0000 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:17.465231+0000 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:17.466 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 1 Processor -- start 2024-09-15T03:34:17.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 1 -- start start 2024-09-15T03:34:17.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient: init 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 0x7f0000099c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0000069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0000070920 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0000071400 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000009e7e0 con 0x7f0000071400 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000009a1c0 con 0x7f0000070920 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0000069e00 con 0x7f0000069a30 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f0007f5d640 10 monclient(hunting): authenticate will time out at 1061.946838s 2024-09-15T03:34:17.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.465+0000 7f00054f8640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f0000070920 legacy=0x7f0000070cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f00064fa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0000071400 0x7f0000137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:58782/0 (socket says 172.21.15.181:58782) 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f00064fa640 1 -- 172.21.15.181:0/3235770248 learned_addr learned my addr 172.21.15.181:0/3235770248 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 1 --2- 172.21.15.181:0/3235770248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 0x7f0000099c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): get_auth_request con 0x7f0000069a30 auth_method 0 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient(hunting): handle_auth_done global_id 74360 payload 274 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 1 -- 172.21.15.181:0/3235770248 >> v1:172.21.15.181:6789/0 conn(0x7f0000070920 legacy=0x7f0000070cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 1 -- 172.21.15.181:0/3235770248 >> v1:172.21.15.153:6789/0 conn(0x7f0000071400 legacy=0x7f0000137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:17.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 1 monclient: found mon.noname-b 2024-09-15T03:34:17.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.466+0000 7f0005cf9640 1 -- 172.21.15.181:0/3235770248 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000009cef0 con 0x7f0000069a30 2024-09-15T03:34:17.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0005cf9640 1 -- 172.21.15.181:0/3235770248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 msgr2=0x7f0000099c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0005cf9640 1 --2- 172.21.15.181:0/3235770248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 0x7f0000099c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efff4009e90 tx=0x7efff4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c594130de2a507d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 1 -- 172.21.15.181:0/3235770248 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4007840 con 0x7f0000069a30 2024-09-15T03:34:17.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: dump: 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 1 -- 172.21.15.181:0/3235770248 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff402d050 con 0x7f0000069a30 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 1 -- 172.21.15.181:0/3235770248 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4007bd0 con 0x7f0000069a30 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 5 monclient: authenticate success, global_id 74360 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 4 set_mon_vals no callback set 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0004cf7640 10 monclient: dump: 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.480 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 10 monclient: shutdown 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 1 -- 172.21.15.181:0/3235770248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 msgr2=0x7f0000099c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.469+0000 7f0007f5d640 1 --2- 172.21.15.181:0/3235770248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 0x7f0000099c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efff4009e90 tx=0x7efff4009780 comp rx=0 tx=0).stop 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 -- 172.21.15.181:0/3235770248 shutdown_connections 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 --2- 172.21.15.181:0/3235770248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000069a30 0x7f0000099c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 -- 172.21.15.181:0/3235770248 >> 172.21.15.181:0/3235770248 conn(0x7f000009e9f0 msgr2=0x7f000009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 -- 172.21.15.181:0/3235770248 shutdown_connections 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 -- 172.21.15.181:0/3235770248 wait complete. 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 10 monclient: monmap: 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 Processor -- start 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 1 -- start start 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.470+0000 7f0007f5d640 10 monclient: init 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 0x7f000013f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0000069a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0000070920 0x7f000012f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0000070920 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000071400 0x7f00001193d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f0000071400 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000011aff0 con 0x7f0000069a30 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000009d8c0 con 0x7f0000070920 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000006df10 con 0x7f0000071400 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0007f5d640 10 monclient(hunting): authenticate will time out at 1061.952387s 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00054f8640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0000070920 msgr2=0x7f000012f900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:17.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00054f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0000070920 0x7f000012f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 0x7f000013f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 0x7f000013f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58454/0 (socket says 172.21.15.181:58454) 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 -- 172.21.15.181:0/2291386682 learned_addr learned my addr 172.21.15.181:0/2291386682 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): get_auth_request con 0x7f0000069a30 auth_method 0 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000071400 0x7f00001193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): get_auth_request con 0x7f0000071400 auth_method 0 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f00064fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient(hunting): handle_auth_done global_id 84234 payload 824 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 -- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000071400 msgr2=0x7f00001193d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000071400 0x7f00001193d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 -- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0000070920 msgr2=0x7f000012f900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0000070920 0x7f000012f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 monclient: found mon.noname-a 2024-09-15T03:34:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.471+0000 7f0005cf9640 1 -- 172.21.15.181:0/2291386682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff4009a20 con 0x7f0000069a30 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0005cf9640 1 -- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 msgr2=0x7f000013f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0005cf9640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 0x7f000013f020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efff4009de0 tx=0x7efff4037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4031a20 con 0x7f0000069a30 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: dump: 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff4040070 con 0x7f0000069a30 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 5 monclient: authenticate success, global_id 84234 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4031d90 con 0x7f0000069a30 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 10 monclient: _renew_subs 2024-09-15T03:34:17.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000011ca20 con 0x7f0000069a30 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff1ffb640 10 monclient: dump: 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7efff3fff640 4 set_mon_vals no callback set 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 10 monclient: _renew_subs 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.474+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000013f560 con 0x7f0000069a30 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efff4042030 con 0x7f0000069a30 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7f0007f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7f0007f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7f0007f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000011d1b0 con 0x7f0000069a30 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7efff1ffb640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7effe40386f0 0x7effe403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(131..131 src has 106..131) v4 ==== 7825+0+0 (secure 0 0 0) 0x7efff406de10 con 0x7f0000069a30 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7f00054f8640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7effe40386f0 0x7effe403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.475+0000 7f00054f8640 10 monclient: get_auth_request con 0x7effe40386f0 auth_method 0 2024-09-15T03:34:17.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.478+0000 7f00054f8640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7effe40386f0 0x7effe403aba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efffc003940 tx=0x7efffc006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.481+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f000011d1b0 con 0x7f0000069a30 2024-09-15T03:34:17.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.481+0000 7efff1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.481+0000 7efff1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:17.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.639+0000 7f0007f5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.639+0000 7f0007f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.639+0000 7f0007f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.639+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f000011d1b0 con 0x7f0000069a30 2024-09-15T03:34:17.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.640+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) v1 ==== 71+0+7 (secure 0 0 0) 0x7f000011d1b0 con 0x7f0000069a30 2024-09-15T03:34:17.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.640+0000 7efff1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.640+0000 7efff1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.736+0000 7f0007f5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.736+0000 7f0007f5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.736+0000 7f0007f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.736+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f000011d1b0 con 0x7f0000069a30 2024-09-15T03:34:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.737+0000 7efff1ffb640 1 -- 172.21.15.181:0/2291386682 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) v1 ==== 71+0+7 (secure 0 0 0) 0x7f000011d1b0 con 0x7f0000069a30 2024-09-15T03:34:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.737+0000 7efff1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:17.738 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.737+0000 7efff1ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7effe40386f0 msgr2=0x7effe403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7effe40386f0 0x7effe403aba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efffc003940 tx=0x7efffc006d80 comp rx=0 tx=0).stop 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 10 monclient: shutdown 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 msgr2=0x7f000013f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 0x7f000013f020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efff4009de0 tx=0x7efff4037000 comp rx=0 tx=0).stop 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 shutdown_connections 2024-09-15T03:34:17.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7effe40386f0 0x7effe403aba0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0000071400 0x7f00001193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0000070920 0x7f000012f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 --2- 172.21.15.181:0/2291386682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0000069a30 0x7f000013f020 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 >> 172.21.15.181:0/2291386682 conn(0x7f000009e9f0 msgr2=0x7f000009fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 shutdown_connections 2024-09-15T03:34:17.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.739+0000 7f0007f5d640 1 -- 172.21.15.181:0/2291386682 wait complete. 2024-09-15T03:34:17.748 INFO:tasks.workunit.client.0.smithi181.stdout:barset 2024-09-15T03:34:17.748 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set rm barset 2024-09-15T03:34:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.821+0000 7f8d94d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.821+0000 7f8d94d5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.821+0000 7f8d94d5d640 10 monclient: monmap: 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:17.822528+0000 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:17.822528+0000 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:17.823 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 1 Processor -- start 2024-09-15T03:34:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 1 -- start start 2024-09-15T03:34:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient: init 2024-09-15T03:34:17.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8d9009f180 0x7f8d90130040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f8d9009f180 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d9009cee0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d90134c60 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9009e610 con 0x7f8d9009cee0 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9009f550 con 0x7f8d90134c60 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9009e2a0 con 0x7f8d9009f180 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d94d5d640 10 monclient(hunting): authenticate will time out at 1062.303867s 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d8e59c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f8d9009f180 msgr2=0x7f8d90130040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:17.826 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d8ed9d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f8d90134c60 legacy=0x7f8d90137040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:17.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.822+0000 7f8d8e59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8d9009f180 0x7f8d90130040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:17.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.823+0000 7f8d85d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8d9009cee0 0x7f8d9009bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:58788/0 (socket says 172.21.15.181:58788) 2024-09-15T03:34:17.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.823+0000 7f8d85d9b640 1 -- 172.21.15.181:0/1300061254 learned_addr learned my addr 172.21.15.181:0/1300061254 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:17.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d85d9b640 1 -- 172.21.15.181:0/1300061254 >> v1:172.21.15.153:6789/0 conn(0x7f8d9009cee0 legacy=0x7f8d9009bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194279915 0 0) 0x7f8d9009e610 con 0x7f8d9009cee0 2024-09-15T03:34:17.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): my global_id is 84237 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c0034e0 con 0x7f8d9009cee0 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428630486 0 0) 0x7f8d7c0034e0 con 0x7f8d9009cee0 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 >> v2:172.21.15.181:3300/0 conn(0x7f8d9009f180 msgr2=0x7f8d90130040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 --2- 172.21.15.181:0/1300061254 >> v2:172.21.15.181:3300/0 conn(0x7f8d9009f180 0x7f8d90130040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 >> v1:172.21.15.181:6789/0 conn(0x7f8d90134c60 legacy=0x7f8d90137040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9009c250 con 0x7f8d9009cee0 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f8d780017e0 con 0x7f8d9009cee0 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d94d5d640 5 monclient: authenticate success, global_id 84237 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d94d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: dump: 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.840 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 >> v1:172.21.15.153:6789/0 conn(0x7f8d9009cee0 legacy=0x7f8d9009bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 0x7f8d7c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): picked mon.a con 0x7f8d7c0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d7c004780 0x7f8d7c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): picked mon.c con 0x7f8d7c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c005280 0x7f8d7c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): picked mon.b con 0x7f8d7c005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9009e2a0 con 0x7f8d7c0039d0 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00c240 con 0x7f8d7c005280 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00c400 con 0x7f8d7c004780 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d94d5d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:17.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8ed9d640 1 -- 172.21.15.181:0/1300061254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c005280 msgr2=0x7f8d7c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:17.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8ed9d640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c005280 0x7f8d7c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:17.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8e59c640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 0x7f8d7c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8e59c640 10 monclient(hunting): get_auth_request con 0x7f8d7c0039d0 auth_method 0 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.825+0000 7f8d8e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d7c004780 0x7f8d7c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): get_auth_request con 0x7f8d7c004780 auth_method 0 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient(hunting): handle_auth_done global_id 84237 payload 293 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 1 -- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d7c004780 msgr2=0x7f8d7c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d7c004780 0x7f8d7c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 1 -- 172.21.15.181:0/1300061254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c005280 msgr2=0x7f8d7c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c005280 0x7f8d7c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 1 monclient: found mon.a 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 1 -- 172.21.15.181:0/1300061254 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0034e0 con 0x7f8d7c0039d0 2024-09-15T03:34:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.826+0000 7f8d8ed9d640 1 -- 172.21.15.181:0/1300061254 reap_dead start 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8e59c640 1 -- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 msgr2=0x7f8d7c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8e59c640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 0x7f8d7c003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8d800099c0 tx=0x7f8d80009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5406e642d1d43528 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80003b80 con 0x7f8d7c0039d0 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: dump: 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d80003d20 con 0x7f8d7c0039d0 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 4 set_mon_vals no callback set 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 1 -- 172.21.15.181:0/1300061254 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8002ab80 con 0x7f8d7c0039d0 2024-09-15T03:34:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d8dd9b640 10 monclient: dump: 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.848 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 10 monclient: shutdown 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 -- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 msgr2=0x7f8d7c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 0x7f8d7c003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8d800099c0 tx=0x7f8d80009060 comp rx=0 tx=0).stop 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 -- 172.21.15.181:0/1300061254 shutdown_connections 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/1300061254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7c0039d0 0x7f8d7c003dc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 -- 172.21.15.181:0/1300061254 >> 172.21.15.181:0/1300061254 conn(0x7f8d900a2b70 msgr2=0x7f8d9009ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 -- 172.21.15.181:0/1300061254 shutdown_connections 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.829+0000 7f8d94d5d640 1 -- 172.21.15.181:0/1300061254 wait complete. 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.830+0000 7f8d94d5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.830+0000 7f8d94d5d640 10 monclient: monmap: 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.830+0000 7f8d94d5d640 1 Processor -- start 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.830+0000 7f8d94d5d640 1 -- start start 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.830+0000 7f8d94d5d640 10 monclient: init 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c0039d0 0x7f8d9012e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7c0039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 0x7f8d901194e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f8d9012eec0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 0x7f8d90119e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f8d90119a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9006ec70 con 0x7f8d9012eec0 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900a1370 con 0x7f8d7c0039d0 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90141fc0 con 0x7f8d90119a20 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d94d5d640 10 monclient(hunting): authenticate will time out at 1062.312295s 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8e59c640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c0039d0 msgr2=0x7f8d9012e980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c0039d0 0x7f8d9012e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 0x7f8d901194e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 0x7f8d90119e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 0x7f8d901194e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58482/0 (socket says 172.21.15.181:58482) 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 0x7f8d90119e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:40768/0 (socket says 172.21.15.181:40768) 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 1 -- 172.21.15.181:0/2032257436 learned_addr learned my addr 172.21.15.181:0/2032257436 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): get_auth_request con 0x7f8d9012eec0 auth_method 0 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): get_auth_request con 0x7f8d90119a20 auth_method 0 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient(hunting): handle_auth_done global_id 74366 payload 824 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 -- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c0039d0 msgr2=0x7f8d9012e980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c0039d0 0x7f8d9012e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 -- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 msgr2=0x7f8d901194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 0x7f8d901194e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d8ed9d640 1 -- 172.21.15.181:0/2032257436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9011cd90 con 0x7f8d90119a20 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.831+0000 7f8d85d9b640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 0x7f8d901194e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d8ed9d640 1 -- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 msgr2=0x7f8d90119e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d8ed9d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 0x7f8d90119e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8d9009e580 tx=0x7f8d88009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800d940 con 0x7f8d90119a20 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: dump: 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.854 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: _finish_auth 0 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8800dac0 con 0x7f8d90119a20 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 5 monclient: authenticate success, global_id 74366 2024-09-15T03:34:17.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 10 monclient: _renew_subs 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800de80 con 0x7f8d90119a20 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9011afe0 con 0x7f8d90119a20 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d86ffd640 10 monclient: dump: 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:17.856 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d8d59a640 4 set_mon_vals no callback set 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 10 monclient: _renew_subs 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.834+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d901356e0 con 0x7f8d90119a20 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8d88006360 con 0x7f8d90119a20 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d94d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d94d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d94d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d900008d0 con 0x7f8d90119a20 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d86ffd640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d700386f0 0x7f8d7003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(131..131 src has 106..131) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f8d8804c7c0 con 0x7f8d90119a20 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d8e59c640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d700386f0 0x7f8d7003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.835+0000 7f8d8e59c640 10 monclient: get_auth_request con 0x7f8d700386f0 auth_method 0 2024-09-15T03:34:17.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.837+0000 7f8d8e59c640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d700386f0 0x7f8d7003aba0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8d80003940 tx=0x7f8d80030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.840+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8d8801eae0 con 0x7f8d90119a20 2024-09-15T03:34:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.840+0000 7f8d86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:17.840+0000 7f8d86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:18.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.004+0000 7f8d94d5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T03:34:18.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.004+0000 7f8d94d5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T03:34:18.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.004+0000 7f8d94d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:18.010 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.004+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f8d90099b50 con 0x7f8d90119a20 2024-09-15T03:34:18.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.116+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v132) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8d8801e300 con 0x7f8d90119a20 2024-09-15T03:34:18.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.116+0000 7f8d86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T03:34:18.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.116+0000 7f8d86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:18.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.217+0000 7f8d94d5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T03:34:18.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.217+0000 7f8d94d5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T03:34:18.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.217+0000 7f8d94d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:18.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:18.217+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f8d9009df10 con 0x7f8d90119a20 2024-09-15T03:34:19.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.118+0000 7f8d86ffd640 1 -- 172.21.15.181:0/2032257436 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v133) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8d8801e4e0 con 0x7f8d90119a20 2024-09-15T03:34:19.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.118+0000 7f8d86ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-15T03:34:19.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.118+0000 7f8d86ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:19.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d700386f0 msgr2=0x7f8d7003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d700386f0 0x7f8d7003aba0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8d80003940 tx=0x7f8d80030000 comp rx=0 tx=0).stop 2024-09-15T03:34:19.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 10 monclient: shutdown 2024-09-15T03:34:19.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 msgr2=0x7f8d90119e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 0x7f8d90119e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8d9009e580 tx=0x7f8d88009620 comp rx=0 tx=0).stop 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 shutdown_connections 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8d700386f0 0x7f8d7003aba0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d90119a20 0x7f8d90119e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d9012eec0 0x7f8d901194e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 --2- 172.21.15.181:0/2032257436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d7c0039d0 0x7f8d9012e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 >> 172.21.15.181:0/2032257436 conn(0x7f8d900a2b70 msgr2=0x7f8d9009a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 shutdown_connections 2024-09-15T03:34:19.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.123+0000 7f8d94d5d640 1 -- 172.21.15.181:0/2032257436 wait complete. 2024-09-15T03:34:19.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:34:19.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -c barset 2024-09-15T03:34:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep -q 0 2024-09-15T03:34:19.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.214+0000 7f615c6d4640 10 monclient: get_monmap_and_config 2024-09-15T03:34:19.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.214+0000 7f615c6d4640 10 monclient: build_initial_monmap 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.214+0000 7f615c6d4640 10 monclient: monmap: 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:19.214907+0000 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:19.214907+0000 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:19.217 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.214+0000 7f615c6d4640 1 Processor -- start 2024-09-15T03:34:19.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.214+0000 7f615c6d4640 1 -- start start 2024-09-15T03:34:19.217 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.214+0000 7f615c6d4640 10 monclient: init 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61541319c0 0x7f6154131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f61541319c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f615409eb30 0x7f615409ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f615409eb30 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6154068e00 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6154131d90 con 0x7f6154068e00 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615406a260 con 0x7f615409eb30 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61540a0050 con 0x7f61541319c0 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615c6d4640 10 monclient(hunting): authenticate will time out at 1063.696417s 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615ac71640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f6154068e00 legacy=0x7f6154069280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f615409eb30 0x7f615409ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61541319c0 0x7f6154131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f615409eb30 0x7f615409ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58548/0 (socket says 172.21.15.181:58548) 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): get_auth_request con 0x7f61541319c0 auth_method 0 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 1 -- 172.21.15.181:0/2907041462 learned_addr learned my addr 172.21.15.181:0/2907041462 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): get_auth_request con 0x7f615409eb30 auth_method 0 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f6159c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.215+0000 7f615a470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 10 monclient(hunting): handle_auth_done global_id 84243 payload 274 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 1 -- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61541319c0 msgr2=0x7f6154131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 1 --2- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61541319c0 0x7f6154131fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 1 -- 172.21.15.181:0/2907041462 >> v1:172.21.15.181:6789/0 conn(0x7f6154068e00 legacy=0x7f6154069280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:19.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 10 monclient: _finish_hunting 0 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 1 monclient: found mon.noname-a 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f6159c6f640 1 -- 172.21.15.181:0/2907041462 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615409fe10 con 0x7f615409eb30 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.216+0000 7f615a470640 1 --2- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61541319c0 0x7f6154131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.218+0000 7f6159c6f640 1 -- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f615409eb30 msgr2=0x7f615409ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.218+0000 7f6159c6f640 1 --2- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f615409eb30 0x7f615409ef00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6148009910 tx=0x7f61480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0aa39870ca76804 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 1 -- 172.21.15.181:0/2907041462 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614802aea0 con 0x7f615409eb30 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: dump: 2024-09-15T03:34:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: _finish_auth 0 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 1 -- 172.21.15.181:0/2907041462 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6148032440 con 0x7f615409eb30 2024-09-15T03:34:19.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 5 monclient: authenticate success, global_id 84243 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 1 -- 172.21.15.181:0/2907041462 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148032820 con 0x7f615409eb30 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 4 set_mon_vals no callback set 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615946e640 10 monclient: dump: 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.230 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 10 monclient: shutdown 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 -- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f615409eb30 msgr2=0x7f615409ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 --2- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f615409eb30 0x7f615409ef00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6148009910 tx=0x7f61480091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 -- 172.21.15.181:0/2907041462 shutdown_connections 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 --2- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f615409eb30 0x7f615409ef00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 --2- 172.21.15.181:0/2907041462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61541319c0 0x7f6154131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 -- 172.21.15.181:0/2907041462 >> 172.21.15.181:0/2907041462 conn(0x7f6154071370 msgr2=0x7f61540a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 -- 172.21.15.181:0/2907041462 shutdown_connections 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.219+0000 7f615c6d4640 1 -- 172.21.15.181:0/2907041462 wait complete. 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.220+0000 7f615c6d4640 10 monclient: build_initial_monmap 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.220+0000 7f615c6d4640 10 monclient: monmap: 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.220+0000 7f615c6d4640 1 Processor -- start 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.220+0000 7f615c6d4640 1 -- start start 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.220+0000 7f615c6d4640 10 monclient: init 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:34:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6154068e00 0x7f615409a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f6154068e00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 0x7f6154130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f615409eb30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61541319c0 0x7f6154141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f61541319c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615411de30 con 0x7f61541319c0 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615411c7c0 con 0x7f6154068e00 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615411db50 con 0x7f615409eb30 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615a470640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6154068e00 msgr2=0x7f615409a930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615c6d4640 10 monclient(hunting): authenticate will time out at 1063.702237s 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615a470640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6154068e00 0x7f615409a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 0x7f6154130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61541319c0 0x7f6154141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 0x7f6154130a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42284/0 (socket says 172.21.15.181:42284) 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 -- 172.21.15.181:0/1087505382 learned_addr learned my addr 172.21.15.181:0/1087505382 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): get_auth_request con 0x7f615409eb30 auth_method 0 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): get_auth_request con 0x7f61541319c0 auth_method 0 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f615ac71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient(hunting): handle_auth_done global_id 74372 payload 824 2024-09-15T03:34:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 -- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6154068e00 msgr2=0x7f615409a930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6154068e00 0x7f615409a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 -- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61541319c0 msgr2=0x7f6154141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61541319c0 0x7f6154141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient: _finish_hunting 0 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 monclient: found mon.noname-c 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.221+0000 7f6159c6f640 1 -- 172.21.15.181:0/1087505382 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c009390 con 0x7f615409eb30 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f6159c6f640 1 -- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 msgr2=0x7f6154130a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f6159c6f640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 0x7f6154130a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6148009910 tx=0x7f6148032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614802aea0 con 0x7f615409eb30 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: dump: 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 5 monclient: authenticate success, global_id 74372 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6148000c40 con 0x7f615409eb30 2024-09-15T03:34:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614803c400 con 0x7f615409eb30 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 10 monclient: _renew_subs 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148009450 con 0x7f615409eb30 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f61527fc640 10 monclient: dump: 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.244 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.244 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.244 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.244 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.244 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.244 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f6158c6d640 4 set_mon_vals no callback set 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 10 monclient: _renew_subs 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.224+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615411cbf0 con 0x7f615409eb30 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f615c6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:19.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f615c6d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f615c6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f614803c7b0 con 0x7f615409eb30 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615409b3c0 con 0x7f615409eb30 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f61527fc640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f61400386f0 0x7f614003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(133..133 src has 106..133) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f614806c990 con 0x7f615409eb30 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f615a470640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f61400386f0 0x7f614003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.225+0000 7f615a470640 10 monclient: get_auth_request con 0x7f61400386f0 auth_method 0 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.228+0000 7f615a470640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f61400386f0 0x7f614003aba0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f614c009610 tx=0x7f614c0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.230+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f614803b3a0 con 0x7f615409eb30 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.230+0000 7f61527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:19.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.230+0000 7f61527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.391+0000 7f615c6d4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.391+0000 7f615c6d4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.391+0000 7f615c6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.391+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f615411ce40 con 0x7f615409eb30 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.392+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+0 (secure 0 0 0) 0x7f614802f710 con 0x7f615409eb30 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.392+0000 7f61527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:19.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.392+0000 7f61527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:19.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.489+0000 7f615c6d4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:19.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.489+0000 7f615c6d4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:19.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.489+0000 7f615c6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.489+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f615411ce40 con 0x7f615409eb30 2024-09-15T03:34:19.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.490+0000 7f61527fc640 1 -- 172.21.15.181:0/1087505382 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+0 (secure 0 0 0) 0x7f614802f8f0 con 0x7f615409eb30 2024-09-15T03:34:19.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.490+0000 7f61527fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:19.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.490+0000 7f61527fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:19.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f61400386f0 msgr2=0x7f614003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f61400386f0 0x7f614003aba0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f614c009610 tx=0x7f614c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 10 monclient: shutdown 2024-09-15T03:34:19.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:19.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 msgr2=0x7f6154130a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 0x7f6154130a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6148009910 tx=0x7f6148032000 comp rx=0 tx=0).stop 2024-09-15T03:34:19.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 shutdown_connections 2024-09-15T03:34:19.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f61400386f0 0x7f614003aba0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61541319c0 0x7f6154141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f615409eb30 0x7f6154130a30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 --2- 172.21.15.181:0/1087505382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6154068e00 0x7f615409a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 >> 172.21.15.181:0/1087505382 conn(0x7f6154071370 msgr2=0x7f61540a35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:19.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 shutdown_connections 2024-09-15T03:34:19.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.491+0000 7f615c6d4640 1 -- 172.21.15.181:0/1087505382 wait complete. 2024-09-15T03:34:19.502 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T03:34:19.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.573+0000 7ff4cbd9d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:19.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.573+0000 7ff4cbd9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:19.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.573+0000 7ff4cbd9d640 10 monclient: monmap: 2024-09-15T03:34:19.579 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:19.579 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.579 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:19.574480+0000 2024-09-15T03:34:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:19.574480+0000 2024-09-15T03:34:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 Processor -- start 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 -- start start 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient: init 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff4c40a1a70 0x7ff4c409c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c40a1a70 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4c409ac40 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff4c4134b70 0x7ff4c4136f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c4134b70 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c40a1e40 con 0x7ff4c409ac40 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c409b7f0 con 0x7ff4c4134b70 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c40a2230 con 0x7ff4c40a1a70 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4c9b39640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7ff4c40a1a70 msgr2=0x7ff4c409c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4cbd9d640 10 monclient(hunting): authenticate will time out at 1064.055992s 2024-09-15T03:34:19.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.574+0000 7ff4c9338640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7ff4c409ac40 legacy=0x7ff4c409b0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4c9b39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff4c40a1a70 0x7ff4c409c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff4c4134b70 0x7ff4c4136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff4c4134b70 0x7ff4c4136f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58564/0 (socket says 172.21.15.181:58564) 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 -- 172.21.15.181:0/630780102 learned_addr learned my addr 172.21.15.181:0/630780102 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): get_auth_request con 0x7ff4c4134b70 auth_method 0 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient(hunting): handle_auth_done global_id 84249 payload 274 2024-09-15T03:34:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 -- 172.21.15.181:0/630780102 >> v2:172.21.15.181:3300/0 conn(0x7ff4c40a1a70 msgr2=0x7ff4c409c2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 --2- 172.21.15.181:0/630780102 >> v2:172.21.15.181:3300/0 conn(0x7ff4c40a1a70 0x7ff4c409c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 -- 172.21.15.181:0/630780102 >> v1:172.21.15.181:6789/0 conn(0x7ff4c409ac40 legacy=0x7ff4c409b0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient: _finish_hunting 0 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 monclient: found mon.noname-a 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:19.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.575+0000 7ff4ca33a640 1 -- 172.21.15.181:0/630780102 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c409d3a0 con 0x7ff4c4134b70 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.578+0000 7ff4ca33a640 1 -- 172.21.15.181:0/630780102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c4134b70 msgr2=0x7ff4c4136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.578+0000 7ff4ca33a640 1 --2- 172.21.15.181:0/630780102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c4134b70 0x7ff4c4136f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40a12c0 tx=0x7ff4bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf59219a8bff949 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.578+0000 7ff4c8b37640 1 -- 172.21.15.181:0/630780102 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc02aea0 con 0x7ff4c4134b70 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.578+0000 7ff4c8b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: dump: 2024-09-15T03:34:19.591 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: _finish_auth 0 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 1 -- 172.21.15.181:0/630780102 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc02d050 con 0x7ff4c4134b70 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:19.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 1 -- 172.21.15.181:0/630780102 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc007840 con 0x7ff4c4134b70 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 5 monclient: authenticate success, global_id 84249 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 4 set_mon_vals no callback set 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4c8b37640 10 monclient: dump: 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 10 monclient: shutdown 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/630780102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c4134b70 msgr2=0x7ff4c4136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/630780102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c4134b70 0x7ff4c4136f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40a12c0 tx=0x7ff4bc009840 comp rx=0 tx=0).stop 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/630780102 shutdown_connections 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/630780102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c4134b70 0x7ff4c4136f50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/630780102 >> v2:172.21.15.181:3300/0 conn(0x7ff4c40a1a70 0x7ff4c409c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/630780102 >> 172.21.15.181:0/630780102 conn(0x7ff4c4067e90 msgr2=0x7ff4c40a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/630780102 shutdown_connections 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.579+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/630780102 wait complete. 2024-09-15T03:34:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.580+0000 7ff4cbd9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.580+0000 7ff4cbd9d640 10 monclient: monmap: 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.580+0000 7ff4cbd9d640 1 Processor -- start 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.580+0000 7ff4cbd9d640 1 -- start start 2024-09-15T03:34:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.580+0000 7ff4cbd9d640 10 monclient: init 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.580+0000 7ff4cbd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 0x7ff4c408ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c409ac40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 0x7ff4c408aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c40a1a70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4c4134b70 0x7ff4c408b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c4134b70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4095000 con 0x7ff4c40a1a70 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c408e880 con 0x7ff4c4134b70 2024-09-15T03:34:19.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c408eb60 con 0x7ff4c409ac40 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4cbd9d640 10 monclient(hunting): authenticate will time out at 1064.062102s 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4ca33a640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4c4134b70 msgr2=0x7ff4c408b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4ca33a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4c4134b70 0x7ff4c408b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 0x7ff4c408aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 0x7ff4c408ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 0x7ff4c408aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58580/0 (socket says 172.21.15.181:58580) 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 1 -- 172.21.15.181:0/4019758872 learned_addr learned my addr 172.21.15.181:0/4019758872 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): get_auth_request con 0x7ff4c40a1a70 auth_method 0 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 0x7ff4c408ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42300/0 (socket says 172.21.15.181:42300) 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): get_auth_request con 0x7ff4c409ac40 auth_method 0 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient(hunting): handle_auth_done global_id 74375 payload 824 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 -- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4c4134b70 msgr2=0x7ff4c408b400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:19.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4c4134b70 0x7ff4c408b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 -- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 msgr2=0x7ff4c408aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:19.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 0x7ff4c408aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:19.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient: _finish_hunting 0 2024-09-15T03:34:19.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 monclient: found mon.noname-c 2024-09-15T03:34:19.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:19.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9b39640 1 -- 172.21.15.181:0/4019758872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc009ae0 con 0x7ff4c409ac40 2024-09-15T03:34:19.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.581+0000 7ff4c9338640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 0x7ff4c408aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.583+0000 7ff4c9b39640 1 -- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 msgr2=0x7ff4c408ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.583+0000 7ff4c9b39640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 0x7ff4c408ef80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000b5b0 tx=0x7ff4c00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0003d90 con 0x7ff4c409ac40 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: dump: 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 5 monclient: authenticate success, global_id 74375 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c0006150 con 0x7ff4c409ac40 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 10 monclient: _renew_subs 2024-09-15T03:34:19.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0006550 con 0x7ff4c409ac40 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c408cee0 con 0x7ff4c409ac40 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4bbfff640 4 set_mon_vals no callback set 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4b9ffb640 10 monclient: dump: 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 10 monclient: _renew_subs 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.584+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c408f660 con 0x7ff4c409ac40 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff4c00103e0 con 0x7ff4c409ac40 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4b9ffb640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4a80386f0 0x7ff4a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(133..133 src has 106..133) v4 ==== 7213+0+0 (secure 0 0 0) 0x7ff4c004b850 con 0x7ff4c409ac40 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4cbd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4cbd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4cbd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4c9338640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4a80386f0 0x7ff4a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c40008d0 con 0x7ff4c409ac40 2024-09-15T03:34:19.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.585+0000 7ff4c9338640 10 monclient: get_auth_request con 0x7ff4a80386f0 auth_method 0 2024-09-15T03:34:19.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.587+0000 7ff4c9338640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4a80386f0 0x7ff4a803aba0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff4c409c510 tx=0x7ff4b4009120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:19.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.587+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:34:19.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.590+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff4c0017020 con 0x7ff4c409ac40 2024-09-15T03:34:19.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.590+0000 7ff4b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:19.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.590+0000 7ff4b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:19.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.752+0000 7ff4cbd9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.752+0000 7ff4cbd9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.752+0000 7ff4cbd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:19.752+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7ff4c409d960 con 0x7ff4c409ac40 2024-09-15T03:34:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.133+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v134) v1 ==== 82+0+0 (secure 0 0 0) 0x7ff4c000f6e0 con 0x7ff4c409ac40 2024-09-15T03:34:20.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.133+0000 7ff4b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:20.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.133+0000 7ff4b9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:20.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.233+0000 7ff4cbd9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:20.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.233+0000 7ff4cbd9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:20.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.233+0000 7ff4cbd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:20.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.233+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7ff4c4072640 con 0x7ff4c409ac40 2024-09-15T03:34:20.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.235+0000 7ff4b9ffb640 1 -- 172.21.15.181:0/4019758872 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v134) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff4c0019a10 con 0x7ff4c409ac40 2024-09-15T03:34:20.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.235+0000 7ff4b9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:20.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.235+0000 7ff4b9ffb640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T03:34:20.237 INFO:tasks.workunit.client.0.smithi181.stderr:compat weight-set already created 2024-09-15T03:34:20.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4a80386f0 msgr2=0x7ff4a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:20.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4a80386f0 0x7ff4a803aba0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff4c409c510 tx=0x7ff4b4009120 comp rx=0 tx=0).stop 2024-09-15T03:34:20.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 10 monclient: shutdown 2024-09-15T03:34:20.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:20.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 msgr2=0x7ff4c408ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:20.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 0x7ff4c408ef80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000b5b0 tx=0x7ff4c00090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 shutdown_connections 2024-09-15T03:34:20.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4a80386f0 0x7ff4a803aba0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4c4134b70 0x7ff4c408b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c40a1a70 0x7ff4c408aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 --2- 172.21.15.181:0/4019758872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4c409ac40 0x7ff4c408ef80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.236+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 >> 172.21.15.181:0/4019758872 conn(0x7ff4c4067e90 msgr2=0x7ff4c4099ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:20.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.237+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 shutdown_connections 2024-09-15T03:34:20.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.237+0000 7ff4cbd9d640 1 -- 172.21.15.181:0/4019758872 wait complete. 2024-09-15T03:34:20.247 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set ls 2024-09-15T03:34:20.247 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '(compat)' 2024-09-15T03:34:20.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.318+0000 7f02d2eef640 10 monclient: get_monmap_and_config 2024-09-15T03:34:20.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.318+0000 7f02d2eef640 10 monclient: build_initial_monmap 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.318+0000 7f02d2eef640 10 monclient: monmap: 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:20.319370+0000 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:20.319370+0000 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:20.320 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:20.321 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:20.321 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 1 Processor -- start 2024-09-15T03:34:20.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 1 -- start start 2024-09-15T03:34:20.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient: init 2024-09-15T03:34:20.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:20.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02cc09c2b0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02cc09aeb0 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f02cc09c680 0x7f02cc137f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient(hunting): picked mon.noname-c con 0x7f02cc09c680 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc0a0a30 con 0x7f02cc09c2b0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02cbfff640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f02cc09aeb0 legacy=0x7f02cc1347f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc09ca70 con 0x7f02cc09aeb0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02cc09b7d0 con 0x7f02cc09c680 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d2eef640 10 monclient(hunting): authenticate will time out at 1064.800990s 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.319+0000 7f02d148c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f02cc09c680 msgr2=0x7f02cc137f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:20.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.320+0000 7f02d148c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f02cc09c680 0x7f02cc137f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:20.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.320+0000 7f02d0c8b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f02cc09c2b0 0x7f02cc09a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:57028/0 (socket says 172.21.15.181:57028) 2024-09-15T03:34:20.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.320+0000 7f02d0c8b640 1 -- 172.21.15.181:0/1550123081 learned_addr learned my addr 172.21.15.181:0/1550123081 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02d0c8b640 1 -- 172.21.15.181:0/1550123081 >> v1:172.21.15.153:6789/0 conn(0x7f02cc09c2b0 legacy=0x7f02cc09a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677127054 0 0) 0x7f02cc0a0a30 con 0x7f02cc09c2b0 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient(hunting): my global_id is 84255 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02bc0034e0 con 0x7f02cc09c2b0 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3470591366 0 0) 0x7f02bc0034e0 con 0x7f02cc09c2b0 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 >> v2:172.21.15.181:3300/0 conn(0x7f02cc09c680 msgr2=0x7f02cc137f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 --2- 172.21.15.181:0/1550123081 >> v2:172.21.15.181:3300/0 conn(0x7f02cc09c680 0x7f02cc137f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 >> v1:172.21.15.181:6789/0 conn(0x7f02cc09aeb0 legacy=0x7f02cc1347f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc09b280 con 0x7f02cc09c2b0 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f02c00017e0 con 0x7f02cc09c2b0 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02d2eef640 5 monclient: authenticate success, global_id 84255 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02d2eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: dump: 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:34:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 >> v1:172.21.15.153:6789/0 conn(0x7f02cc09c2b0 legacy=0x7f02cc09a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.322+0000 7f02cb7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02bc0039d0 0x7f02bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): picked mon.b con 0x7f02bc0039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 0x7f02bc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): picked mon.c con 0x7f02bc004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc005280 0x7f02bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): picked mon.a con 0x7f02bc005280 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc09b7d0 con 0x7f02bc005280 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc00c240 con 0x7f02bc0039d0 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc00c400 con 0x7f02bc004780 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d0c8b640 1 -- 172.21.15.181:0/1550123081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02bc0039d0 msgr2=0x7f02bc003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d2eef640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d0c8b640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02bc0039d0 0x7f02bc003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 0x7f02bc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc005280 0x7f02bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): get_auth_request con 0x7f02bc005280 auth_method 0 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): get_auth_request con 0x7f02bc004780 auth_method 0 2024-09-15T03:34:20.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient(hunting): handle_auth_done global_id 84255 payload 293 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 -- 172.21.15.181:0/1550123081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02bc0039d0 msgr2=0x7f02bc003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02bc0039d0 0x7f02bc003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 -- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc005280 msgr2=0x7f02bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:20.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc005280 0x7f02bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 monclient: found mon.c 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 1 -- 172.21.15.181:0/1550123081 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0034e0 con 0x7f02bc004780 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 10 monclient: _finish_auth 0 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc005280 0x7f02bc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.323+0000 7f02d148c640 1 -- 172.21.15.181:0/1550123081 reap_dead start 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cbfff640 1 -- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 msgr2=0x7f02bc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cbfff640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 0x7f02bc004bf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f02b80099c0 tx=0x7f02b8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97eb63f3de3327e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8003b80 con 0x7f02bc004780 2024-09-15T03:34:20.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: dump: 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02d2eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:20.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8003d20 con 0x7f02bc004780 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02d2eef640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02d2eef640 4 set_mon_vals no callback set 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 1 -- 172.21.15.181:0/1550123081 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b802ab80 con 0x7f02bc004780 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.326+0000 7f02cb7fe640 10 monclient: dump: 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:20.342 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 10 monclient: shutdown 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 -- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 msgr2=0x7f02bc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 0x7f02bc004bf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f02b80099c0 tx=0x7f02b8009060 comp rx=0 tx=0).stop 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 -- 172.21.15.181:0/1550123081 shutdown_connections 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 --2- 172.21.15.181:0/1550123081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02bc004780 0x7f02bc004bf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 -- 172.21.15.181:0/1550123081 >> 172.21.15.181:0/1550123081 conn(0x7f02cc0a3a50 msgr2=0x7f02cc09faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 -- 172.21.15.181:0/1550123081 shutdown_connections 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 1 -- 172.21.15.181:0/1550123081 wait complete. 2024-09-15T03:34:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 10 monclient: build_initial_monmap 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.327+0000 7f02d2eef640 10 monclient: monmap: 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 Processor -- start 2024-09-15T03:34:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 -- start start 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient: init 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 0x7f02cc131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): picked mon.noname-a con 0x7f02bc004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02cc131da0 0x7f02cc11b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): picked mon.noname-b con 0x7f02cc131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 0x7f02cc11bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): picked mon.noname-c con 0x7f02cc11b8e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc06e9c0 con 0x7f02bc004780 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc132170 con 0x7f02cc131da0 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc11ec60 con 0x7f02cc11b8e0 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02cbfff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02cc131da0 msgr2=0x7f02cc11b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02d2eef640 10 monclient(hunting): authenticate will time out at 1064.809946s 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.328+0000 7f02cbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02cc131da0 0x7f02cc11b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 0x7f02cc131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 0x7f02cc11bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 0x7f02cc131860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58590/0 (socket says 172.21.15.181:58590) 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 1 -- 172.21.15.181:0/3708452485 learned_addr learned my addr 172.21.15.181:0/3708452485 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): get_auth_request con 0x7f02bc004780 auth_method 0 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:20.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): get_auth_request con 0x7f02cc11b8e0 auth_method 0 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient(hunting): handle_auth_done global_id 74378 payload 824 2024-09-15T03:34:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 -- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02cc131da0 msgr2=0x7f02cc11b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02cc131da0 0x7f02cc11b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 -- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 msgr2=0x7f02cc131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 0x7f02cc131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient: _finish_hunting 0 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 monclient: found mon.noname-c 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d148c640 1 -- 172.21.15.181:0/3708452485 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc13fac0 con 0x7f02cc11b8e0 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.329+0000 7f02d0c8b640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 0x7f02cc131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.331+0000 7f02d148c640 1 -- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 msgr2=0x7f02cc11bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.331+0000 7f02d148c640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 0x7f02cc11bd80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0a07f0 tx=0x7f02c40095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.331+0000 7f02d148c640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 0x7f02cc11bd80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0a07f0 tx=0x7f02c40095d0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f02c4011070 mon_map magic: 0 v1 2024-09-15T03:34:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4011070 con 0x7f02cc11b8e0 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: dump: 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:20.349 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c400b9c0 con 0x7f02cc11b8e0 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 5 monclient: authenticate success, global_id 74378 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 monclient: _renew_subs 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc11e9e0 con 0x7f02cc11b8e0 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02caffd640 4 set_mon_vals no callback set 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 monclient: _renew_subs 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc1340f0 con 0x7f02cc11b8e0 2024-09-15T03:34:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c400bd60 con 0x7f02cc11b8e0 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 10 monclient: dump: 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f02c4005110 con 0x7f02cc11b8e0 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.332+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc04bf60 con 0x7f02cc11b8e0 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.333+0000 7f02c8ff9640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02b0038740 0x7f02b003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.333+0000 7f02d0c8b640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02b0038740 0x7f02b003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.333+0000 7f02d0c8b640 10 monclient: get_auth_request con 0x7f02b0038740 auth_method 0 2024-09-15T03:34:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.335+0000 7f02d0c8b640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02b0038740 0x7f02b003abf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02cc06eda0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:20.338+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(134..134 src has 106..134) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f02c4006e40 con 0x7f02cc11b8e0 2024-09-15T03:34:21.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.159+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f02c40134b0 con 0x7f02cc11b8e0 2024-09-15T03:34:21.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.159+0000 7f02c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:21.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.159+0000 7f02c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:21.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02d2eef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:21.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02d2eef640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:21.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02d2eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:21.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f02cc11efa0 con 0x7f02cc11b8e0 2024-09-15T03:34:21.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v134) v1 ==== 71+0+9 (secure 0 0 0) 0x7f02c40132d0 con 0x7f02cc11b8e0 2024-09-15T03:34:21.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:21.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.319+0000 7f02c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02d2eef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02d2eef640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02d2eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f02cc072640 con 0x7f02cc11b8e0 2024-09-15T03:34:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02c8ff9640 1 -- 172.21.15.181:0/3708452485 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v134) v1 ==== 71+0+9 (secure 0 0 0) 0x7f02c40134b0 con 0x7f02cc11b8e0 2024-09-15T03:34:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02c8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-15T03:34:21.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.417+0000 7f02c8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02b0038740 msgr2=0x7f02b003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02b0038740 0x7f02b003abf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02cc06eda0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 10 monclient: shutdown 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 msgr2=0x7f02cc11bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 0x7f02cc11bd80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0a07f0 tx=0x7f02c40095d0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 shutdown_connections 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02b0038740 0x7f02b003abf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02cc11b8e0 0x7f02cc11bd80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02cc131da0 0x7f02cc11b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 --2- 172.21.15.181:0/3708452485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02bc004780 0x7f02cc131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 >> 172.21.15.181:0/3708452485 conn(0x7f02cc0a3a50 msgr2=0x7f02cc0a0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 shutdown_connections 2024-09-15T03:34:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.419+0000 7f02d2eef640 1 -- 172.21.15.181:0/3708452485 wait complete. 2024-09-15T03:34:21.429 INFO:tasks.workunit.client.0.smithi181.stdout:(compat) 2024-09-15T03:34:21.429 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T03:34:21.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.501+0000 7f916479d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:21.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.501+0000 7f916479d640 10 monclient: build_initial_monmap 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.501+0000 7f916479d640 10 monclient: monmap: 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:21.502453+0000 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:21.502453+0000 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:21.503 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:21.504 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:21.504 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:21.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 Processor -- start 2024-09-15T03:34:21.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 -- start start 2024-09-15T03:34:21.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient: init 2024-09-15T03:34:21.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:21.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f915c069a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f915c09baa0 0x7f915c09bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): picked mon.noname-c con 0x7f915c09baa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f915c134b70 0x7f915c136f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): picked mon.noname-a con 0x7f915c134b70 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c12ff50 con 0x7f915c069a30 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c069e00 con 0x7f915c134b70 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c09d100 con 0x7f915c09baa0 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f916479d640 10 monclient(hunting): authenticate will time out at 1065.983910s 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f9162539640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f915c069a30 legacy=0x7f915c12f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f9161d38640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f915c09baa0 msgr2=0x7f915c09bea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.502+0000 7f9161d38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f915c09baa0 0x7f915c09bea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f915c134b70 0x7f915c136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f915c134b70 0x7f915c136f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58598/0 (socket says 172.21.15.181:58598) 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 -- 172.21.15.181:0/1080458271 learned_addr learned my addr 172.21.15.181:0/1080458271 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): get_auth_request con 0x7f915c134b70 auth_method 0 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient(hunting): handle_auth_done global_id 84261 payload 274 2024-09-15T03:34:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 -- 172.21.15.181:0/1080458271 >> v2:172.21.15.181:3300/0 conn(0x7f915c09baa0 msgr2=0x7f915c09bea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 --2- 172.21.15.181:0/1080458271 >> v2:172.21.15.181:3300/0 conn(0x7f915c09baa0 0x7f915c09bea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 -- 172.21.15.181:0/1080458271 >> v1:172.21.15.181:6789/0 conn(0x7f915c069a30 legacy=0x7f915c12f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient: _finish_hunting 0 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 monclient: found mon.noname-a 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.503+0000 7f9162d3a640 1 -- 172.21.15.181:0/1080458271 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c09e270 con 0x7f915c134b70 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9162d3a640 1 -- 172.21.15.181:0/1080458271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 msgr2=0x7f915c136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9162d3a640 1 --2- 172.21.15.181:0/1080458271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c136f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f915c09d070 tx=0x7f9154009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ae0cd0f6e1096a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 1 -- 172.21.15.181:0/1080458271 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915402aea0 con 0x7f915c134b70 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: dump: 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.515 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: _finish_auth 0 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 1 -- 172.21.15.181:0/1080458271 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915402d050 con 0x7f915c134b70 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f916479d640 5 monclient: authenticate success, global_id 84261 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f916479d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f916479d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 1 -- 172.21.15.181:0/1080458271 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154007840 con 0x7f915c134b70 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f916479d640 4 set_mon_vals no callback set 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.506+0000 7f9161537640 10 monclient: dump: 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 10 monclient: shutdown 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 -- 172.21.15.181:0/1080458271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 msgr2=0x7f915c136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 --2- 172.21.15.181:0/1080458271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c136f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f915c09d070 tx=0x7f9154009840 comp rx=0 tx=0).stop 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 -- 172.21.15.181:0/1080458271 shutdown_connections 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 --2- 172.21.15.181:0/1080458271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c136f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 --2- 172.21.15.181:0/1080458271 >> v2:172.21.15.181:3300/0 conn(0x7f915c09baa0 0x7f915c09bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 -- 172.21.15.181:0/1080458271 >> 172.21.15.181:0/1080458271 conn(0x7f915c09e9f0 msgr2=0x7f915c09edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 -- 172.21.15.181:0/1080458271 shutdown_connections 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 -- 172.21.15.181:0/1080458271 wait complete. 2024-09-15T03:34:21.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 10 monclient: build_initial_monmap 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 10 monclient: monmap: 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.507+0000 7f916479d640 1 Processor -- start 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 -- start start 2024-09-15T03:34:21.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient: init 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f915c069a30 0x7f915c12cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): picked mon.noname-b con 0x7f915c069a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f915c09baa0 0x7f915c12d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): picked mon.noname-c con 0x7f915c09baa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): picked mon.noname-a con 0x7f915c134b70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c097f60 con 0x7f915c134b70 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0a0fe0 con 0x7f915c069a30 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:21.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162539640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f915c069a30 msgr2=0x7f915c12cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c06c7e0 con 0x7f915c09baa0 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f916479d640 10 monclient(hunting): authenticate will time out at 1065.989467s 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162539640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f915c069a30 0x7f915c12cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f915c09baa0 0x7f915c12d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58614/0 (socket says 172.21.15.181:58614) 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 1 -- 172.21.15.181:0/2974036657 learned_addr learned my addr 172.21.15.181:0/2974036657 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): get_auth_request con 0x7f915c134b70 auth_method 0 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f915c09baa0 0x7f915c12d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42330/0 (socket says 172.21.15.181:42330) 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:21.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): get_auth_request con 0x7f915c09baa0 auth_method 0 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9162d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.508+0000 7f9161d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 10 monclient(hunting): handle_auth_done global_id 84264 payload 824 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 1 -- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f915c09baa0 msgr2=0x7f915c12d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f915c09baa0 0x7f915c12d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 1 -- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f915c069a30 msgr2=0x7f915c12cca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:21.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f915c069a30 0x7f915c12cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 10 monclient: _finish_hunting 0 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 1 monclient: found mon.noname-a 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.509+0000 7f9162d3a640 1 -- 172.21.15.181:0/2974036657 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154009ae0 con 0x7f915c134b70 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f9162d3a640 1 -- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 msgr2=0x7f915c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f9162d3a640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c096a60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9154009f50 tx=0x7f9154036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154031c50 con 0x7f915c134b70 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:21.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: dump: 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:21.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 5 monclient: authenticate success, global_id 84264 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9154031db0 con 0x7f915c134b70 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 10 monclient: _renew_subs 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915403c670 con 0x7f915c134b70 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c06cb20 con 0x7f915c134b70 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f91527fc640 10 monclient: dump: 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:21.525 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f9160d36640 4 set_mon_vals no callback set 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 10 monclient: _renew_subs 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.511+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c06c4d0 con 0x7f915c134b70 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f916479d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f916479d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:21.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f916479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f915403ca60 con 0x7f915c134b70 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c0008d0 con 0x7f915c134b70 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f91527fc640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f91400386f0 0x7f914003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.512+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(134..134 src has 106..134) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f915406d0b0 con 0x7f915c134b70 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.513+0000 7f9162539640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f91400386f0 0x7f914003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.513+0000 7f9162539640 10 monclient: get_auth_request con 0x7f91400386f0 auth_method 0 2024-09-15T03:34:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.515+0000 7f9162539640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f91400386f0 0x7f914003aba0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9158003940 tx=0x7f9158006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:21.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.517+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f915403aaf0 con 0x7f915c134b70 2024-09-15T03:34:21.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.517+0000 7f91527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:21.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.517+0000 7f91527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:21.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.677+0000 7f916479d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:34:21.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.677+0000 7f916479d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:34:21.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.677+0000 7f916479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:21.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:21.677+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f915c09dc40 con 0x7f915c134b70 2024-09-15T03:34:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.144+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v135) v1 ==== 78+0+0 (secure 0 0 0) 0x7f9154040020 con 0x7f915c134b70 2024-09-15T03:34:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.144+0000 7f91527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:34:22.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.144+0000 7f91527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:22.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.242+0000 7f916479d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:34:22.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.242+0000 7f916479d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:34:22.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.242+0000 7f916479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:22.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:22.242+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f915c072640 con 0x7f915c134b70 2024-09-15T03:34:23.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.145+0000 7f91527fc640 1 -- 172.21.15.181:0/2974036657 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v136) v1 ==== 78+0+0 (secure 0 0 0) 0x7f915403a310 con 0x7f915c134b70 2024-09-15T03:34:23.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.145+0000 7f91527fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:34:23.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.145+0000 7f91527fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:23.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f91400386f0 msgr2=0x7f914003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:23.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f91400386f0 0x7f914003aba0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9158003940 tx=0x7f9158006d80 comp rx=0 tx=0).stop 2024-09-15T03:34:23.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 10 monclient: shutdown 2024-09-15T03:34:23.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:23.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 msgr2=0x7f915c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c096a60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9154009f50 tx=0x7f9154036000 comp rx=0 tx=0).stop 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 shutdown_connections 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f91400386f0 0x7f914003aba0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f915c134b70 0x7f915c096a60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f915c09baa0 0x7f915c12d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 --2- 172.21.15.181:0/2974036657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f915c069a30 0x7f915c12cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 >> 172.21.15.181:0/2974036657 conn(0x7f915c09e9f0 msgr2=0x7f915c0a14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 shutdown_connections 2024-09-15T03:34:23.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.147+0000 7f916479d640 1 -- 172.21.15.181:0/2974036657 wait complete. 2024-09-15T03:34:23.160 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create cool 2 2024-09-15T03:34:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.237+0000 7f8829d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.237+0000 7f8829d9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.237+0000 7f8829d9d640 10 monclient: monmap: 2024-09-15T03:34:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:23.237938+0000 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:23.237938+0000 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:23.239 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.237+0000 7f8829d9d640 1 Processor -- start 2024-09-15T03:34:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.237+0000 7f8829d9d640 1 -- start start 2024-09-15T03:34:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.237+0000 7f8829d9d640 10 monclient: init 2024-09-15T03:34:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T03:34:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f882409ec50 0x7f8824131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f882409ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f88241321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 0x7f8824069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f882409f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8824067420 con 0x7f882409f020 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882409fdc0 con 0x7f882409ec50 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88240a0510 con 0x7f88241321a0 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:23.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8829d9d640 10 monclient(hunting): authenticate will time out at 1067.719415s 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f88237fe640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f882409ec50 msgr2=0x7f8824131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f88237fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f882409ec50 0x7f8824131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 0x7f8824069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 0x7f8824069240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58630/0 (socket says 172.21.15.181:58630) 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 1 -- 172.21.15.181:0/3183679166 learned_addr learned my addr 172.21.15.181:0/3183679166 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): get_auth_request con 0x7f882409f020 auth_method 0 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 10 monclient(hunting): get_auth_request con 0x7f88241321a0 auth_method 0 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:23.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8823fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.238+0000 7f8822ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 10 monclient(hunting): handle_auth_done global_id 74384 payload 274 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 1 -- 172.21.15.181:0/3183679166 >> v2:172.21.15.181:3300/0 conn(0x7f882409ec50 msgr2=0x7f8824131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 1 --2- 172.21.15.181:0/3183679166 >> v2:172.21.15.181:3300/0 conn(0x7f882409ec50 0x7f8824131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 1 -- 172.21.15.181:0/3183679166 >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 msgr2=0x7f8824069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 1 --2- 172.21.15.181:0/3183679166 >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 0x7f8824069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:34:23.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 1 monclient: found mon.noname-b 2024-09-15T03:34:23.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:23.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:23.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8822ffd640 1 -- 172.21.15.181:0/3183679166 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240a0200 con 0x7f88241321a0 2024-09-15T03:34:23.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.239+0000 7f8823fff640 1 --2- 172.21.15.181:0/3183679166 >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 0x7f8824069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f8822ffd640 1 -- 172.21.15.181:0/3183679166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 msgr2=0x7f8824068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f8822ffd640 1 --2- 172.21.15.181:0/3183679166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824068d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f88240719d0 tx=0x7f88180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7591560ddb0dcae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 1 -- 172.21.15.181:0/3183679166 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881802ae10 con 0x7f88241321a0 2024-09-15T03:34:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: dump: 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:23.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 1 -- 172.21.15.181:0/3183679166 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8818032440 con 0x7f88241321a0 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 1 -- 172.21.15.181:0/3183679166 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88180327e0 con 0x7f88241321a0 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f88227fc640 10 monclient: dump: 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f8829d9d640 5 monclient: authenticate success, global_id 74384 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f8829d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.242+0000 7f8829d9d640 4 set_mon_vals no callback set 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 10 monclient: shutdown 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 -- 172.21.15.181:0/3183679166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 msgr2=0x7f8824068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 --2- 172.21.15.181:0/3183679166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824068d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f88240719d0 tx=0x7f88180090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 -- 172.21.15.181:0/3183679166 shutdown_connections 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 --2- 172.21.15.181:0/3183679166 >> v2:172.21.15.153:3300/0 conn(0x7f882409f020 0x7f8824069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 --2- 172.21.15.181:0/3183679166 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824068d00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 --2- 172.21.15.181:0/3183679166 >> v2:172.21.15.181:3300/0 conn(0x7f882409ec50 0x7f8824131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 -- 172.21.15.181:0/3183679166 >> 172.21.15.181:0/3183679166 conn(0x7f88240a5d40 msgr2=0x7f88240a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 -- 172.21.15.181:0/3183679166 shutdown_connections 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 1 -- 172.21.15.181:0/3183679166 wait complete. 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.243+0000 7f8829d9d640 10 monclient: monmap: 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 Processor -- start 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 -- start start 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient: init 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f882409ec50 0x7f882409aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f882409ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882409f020 0x7f882411bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f882409f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:23.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f88241321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882411c170 con 0x7f882409f020 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882409c000 con 0x7f882409ec50 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882409e070 con 0x7f88241321a0 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f88237fe640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f882409ec50 msgr2=0x7f882409aaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f8829d9d640 10 monclient(hunting): authenticate will time out at 1067.725930s 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.244+0000 7f88237fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f882409ec50 0x7f882409aaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882409f020 0x7f882411bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42350/0 (socket says 172.21.15.181:42350) 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882409f020 0x7f882411bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58634/0 (socket says 172.21.15.181:58634) 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 -- 172.21.15.181:0/1410821264 learned_addr learned my addr 172.21.15.181:0/1410821264 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): get_auth_request con 0x7f88241321a0 auth_method 0 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): get_auth_request con 0x7f882409f020 auth_method 0 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8822ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient(hunting): handle_auth_done global_id 74387 payload 824 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 -- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f882409ec50 msgr2=0x7f882409aaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f882409ec50 0x7f882409aaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 -- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882409f020 msgr2=0x7f882411bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882409f020 0x7f882411bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient: _finish_hunting 0 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 monclient: found mon.noname-c 2024-09-15T03:34:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.245+0000 7f8823fff640 1 -- 172.21.15.181:0/1410821264 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c009ae0 con 0x7f88241321a0 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.247+0000 7f8823fff640 1 -- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 msgr2=0x7f8824130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.247+0000 7f8823fff640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824130bc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f881c009fd0 tx=0x7f881c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c018070 con 0x7f88241321a0 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: dump: 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c00f040 con 0x7f88241321a0 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 5 monclient: authenticate success, global_id 74387 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c0146b0 con 0x7f88241321a0 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 10 monclient: _renew_subs 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8818009390 con 0x7f88241321a0 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 10 monclient: dump: 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8821ffb640 4 set_mon_vals no callback set 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 10 monclient: _renew_subs 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882409b5a0 con 0x7f88241321a0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.248+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f881c003020 con 0x7f88241321a0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f880b7fe640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88100386f0 0x7f881003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(136..136 src has 106..136) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f881c04c6b0 con 0x7f88241321a0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f8829d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f8829d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f8829d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88240008d0 con 0x7f88241321a0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f88237fe640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88100386f0 0x7f881003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.249+0000 7f88237fe640 10 monclient: get_auth_request con 0x7f88100386f0 auth_method 0 2024-09-15T03:34:23.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.251+0000 7f88237fe640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88100386f0 0x7f881003aba0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8814003940 tx=0x7f8814006d20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:23.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.254+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f881c0092e0 con 0x7f88241321a0 2024-09-15T03:34:23.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.254+0000 7f880b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:23.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.254+0000 7f880b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:23.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.408+0000 7f8829d9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T03:34:23.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.408+0000 7f8829d9d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T03:34:23.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.408+0000 7f8829d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:23.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:23.409+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f882411d950 con 0x7f88241321a0 2024-09-15T03:34:24.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.159+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v137) v1 ==== 111+0+0 (secure 0 0 0) 0x7f881c0410a0 con 0x7f88241321a0 2024-09-15T03:34:24.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.159+0000 7f880b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T03:34:24.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.159+0000 7f880b7fe640 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-15T03:34:24.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.252+0000 7f8829d9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T03:34:24.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.252+0000 7f8829d9d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T03:34:24.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.252+0000 7f8829d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:24.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.252+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f882409e520 con 0x7f88241321a0 2024-09-15T03:34:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.253+0000 7f880b7fe640 1 -- 172.21.15.181:0/1410821264 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v137) v1 ==== 118+0+0 (secure 0 0 0) 0x7f881c012340 con 0x7f88241321a0 2024-09-15T03:34:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.253+0000 7f880b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-15T03:34:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.253+0000 7f880b7fe640 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-15T03:34:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'cool' already exists 2024-09-15T03:34:24.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88100386f0 msgr2=0x7f881003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:24.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88100386f0 0x7f881003aba0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8814003940 tx=0x7f8814006d20 comp rx=0 tx=0).stop 2024-09-15T03:34:24.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 10 monclient: shutdown 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 msgr2=0x7f8824130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824130bc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f881c009fd0 tx=0x7f881c00b040 comp rx=0 tx=0).stop 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 shutdown_connections 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88100386f0 0x7f881003aba0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88241321a0 0x7f8824130bc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882409f020 0x7f882411bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 --2- 172.21.15.181:0/1410821264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f882409ec50 0x7f882409aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 >> 172.21.15.181:0/1410821264 conn(0x7f88240a5d40 msgr2=0x7f8824134c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:24.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 shutdown_connections 2024-09-15T03:34:24.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.255+0000 7f8829d9d640 1 -- 172.21.15.181:0/1410821264 wait complete. 2024-09-15T03:34:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool create cold 2 2024-09-15T03:34:24.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.338+0000 7f07b479d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:24.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.338+0000 7f07b479d640 10 monclient: build_initial_monmap 2024-09-15T03:34:24.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.338+0000 7f07b479d640 10 monclient: monmap: 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:24.339143+0000 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:24.339143+0000 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:24.340 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.338+0000 7f07b479d640 1 Processor -- start 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.338+0000 7f07b479d640 1 -- start start 2024-09-15T03:34:24.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.338+0000 7f07b479d640 10 monclient: init 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): picked mon.noname-b con 0x7f07ac098c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07ac099580 0x7f07ac09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): picked mon.noname-a con 0x7f07ac099580 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07ac099c70 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac09f680 con 0x7f07ac099c70 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:24.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ac09f420 con 0x7f07ac099580 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac068a20 con 0x7f07ac098c70 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b479d640 10 monclient(hunting): authenticate will time out at 1068.820579s 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b2d3a640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f07ac099c70 legacy=0x7f07ac09e010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b1d38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07ac099580 0x7f07ac09dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.339+0000 7f07b2539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07ac099580 0x7f07ac09dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58638/0 (socket says 172.21.15.181:58638) 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 -- 172.21.15.181:0/1113120070 learned_addr learned my addr 172.21.15.181:0/1113120070 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): get_auth_request con 0x7f07ac099580 auth_method 0 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:24.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): get_auth_request con 0x7f07ac098c70 auth_method 0 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:24.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient(hunting): handle_auth_done global_id 84273 payload 274 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 -- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 msgr2=0x7f07ac099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 --2- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 -- 172.21.15.181:0/1113120070 >> v1:172.21.15.181:6789/0 conn(0x7f07ac099c70 legacy=0x7f07ac09e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient: _finish_hunting 0 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 monclient: found mon.noname-a 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:24.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b1d38640 1 -- 172.21.15.181:0/1113120070 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac096a30 con 0x7f07ac099580 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.340+0000 7f07b2539640 1 --2- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1d38640 1 -- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 msgr2=0x7f07ac09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1d38640 1 --2- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac09dad0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009910 tx=0x7f07a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d03558237bcc7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 1 -- 172.21.15.181:0/1113120070 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a002aea0 con 0x7f07ac099580 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: dump: 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.352 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: _finish_auth 0 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 1 -- 172.21.15.181:0/1113120070 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0032440 con 0x7f07ac099580 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 1 -- 172.21.15.181:0/1113120070 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a00327e0 con 0x7f07ac099580 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b1537640 10 monclient: dump: 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:24.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 5 monclient: authenticate success, global_id 84273 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 4 set_mon_vals no callback set 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 10 monclient: shutdown 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 1 -- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 msgr2=0x7f07ac09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 1 --2- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac09dad0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009910 tx=0x7f07a00091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 1 -- 172.21.15.181:0/1113120070 shutdown_connections 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 1 --2- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac09dad0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 1 --2- 172.21.15.181:0/1113120070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.343+0000 7f07b479d640 1 -- 172.21.15.181:0/1113120070 >> 172.21.15.181:0/1113120070 conn(0x7f07ac070e90 msgr2=0x7f07ac0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 1 -- 172.21.15.181:0/1113120070 shutdown_connections 2024-09-15T03:34:24.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 1 -- 172.21.15.181:0/1113120070 wait complete. 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 10 monclient: build_initial_monmap 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 10 monclient: monmap: 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:24.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 1 Processor -- start 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 1 -- start start 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.344+0000 7f07b479d640 10 monclient: init 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): picked mon.noname-c con 0x7f07ac098c70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): picked mon.noname-a con 0x7f07ac099580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07ac099c70 0x7f07ac11db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): picked mon.noname-b con 0x7f07ac099c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac06eb80 con 0x7f07ac099580 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac134c80 con 0x7f07ac099c70 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac120f90 con 0x7f07ac098c70 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b479d640 10 monclient(hunting): authenticate will time out at 1068.826241s 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2d3a640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07ac099c70 msgr2=0x7f07ac11db50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2d3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07ac099c70 0x7f07ac11db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58654/0 (socket says 172.21.15.181:58654) 2024-09-15T03:34:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 -- 172.21.15.181:0/1337661768 learned_addr learned my addr 172.21.15.181:0/1337661768 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): get_auth_request con 0x7f07ac099580 auth_method 0 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): get_auth_request con 0x7f07ac098c70 auth_method 0 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:24.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient(hunting): handle_auth_done global_id 84276 payload 824 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 -- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 msgr2=0x7f07ac141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 -- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07ac099c70 msgr2=0x7f07ac11db50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07ac099c70 0x7f07ac11db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient: _finish_hunting 0 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 monclient: found mon.noname-a 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b1d38640 1 -- 172.21.15.181:0/1337661768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8009390 con 0x7f07ac099580 2024-09-15T03:34:24.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.345+0000 7f07b2539640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b1d38640 1 -- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 msgr2=0x7f07ac133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b1d38640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac133e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f07a0035040 tx=0x7f07a0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a002aea0 con 0x7f07ac099580 2024-09-15T03:34:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: dump: 2024-09-15T03:34:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0000d10 con 0x7f07ac099580 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a003d440 con 0x7f07ac099580 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:24.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f079e7fc640 10 monclient: dump: 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 5 monclient: authenticate success, global_id 84276 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 10 monclient: _renew_subs 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b0d36640 4 set_mon_vals no callback set 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a0009450 con 0x7f07ac099580 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 10 monclient: _renew_subs 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.348+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac141750 con 0x7f07ac099580 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f07b479d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f07b479d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f07b479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac0008d0 con 0x7f07ac099580 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f07a0039040 con 0x7f07ac099580 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f079e7fc640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0790038740 0x7f079003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.349+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(137..137 src has 106..137) v4 ==== 7556+0+0 (secure 0 0 0) 0x7f07a006c410 con 0x7f07ac099580 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.350+0000 7f07b2539640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0790038740 0x7f079003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.350+0000 7f07b2539640 10 monclient: get_auth_request con 0x7f0790038740 auth_method 0 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.352+0000 7f07b2539640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0790038740 0x7f079003abf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f07a8011580 tx=0x7f07a80090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.354+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f07a002f2a0 con 0x7f07ac099580 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.354+0000 7f079e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.354+0000 7f079e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:24.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.510+0000 7f07b479d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T03:34:24.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.510+0000 7f07b479d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T03:34:24.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.510+0000 7f07b479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:24.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:24.510+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f07ac121600 con 0x7f07ac099580 2024-09-15T03:34:25.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.162+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v138) v1 ==== 111+0+0 (secure 0 0 0) 0x7f07a0042020 con 0x7f07ac099580 2024-09-15T03:34:25.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.162+0000 7f079e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T03:34:25.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.162+0000 7f079e7fc640 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-15T03:34:25.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.256+0000 7f07b479d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T03:34:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.256+0000 7f07b479d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T03:34:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.256+0000 7f07b479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.256+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f07ac121600 con 0x7f07ac099580 2024-09-15T03:34:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.257+0000 7f079e7fc640 1 -- 172.21.15.181:0/1337661768 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v138) v1 ==== 118+0+0 (secure 0 0 0) 0x7f07a0042070 con 0x7f07ac099580 2024-09-15T03:34:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.257+0000 7f079e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-15T03:34:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.257+0000 7f079e7fc640 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-15T03:34:25.259 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'cold' already exists 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0790038740 msgr2=0x7f079003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0790038740 0x7f079003abf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f07a8011580 tx=0x7f07a80090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 10 monclient: shutdown 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 msgr2=0x7f07ac133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac133e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f07a0035040 tx=0x7f07a0032000 comp rx=0 tx=0).stop 2024-09-15T03:34:25.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 shutdown_connections 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0790038740 0x7f079003abf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07ac099c70 0x7f07ac11db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07ac099580 0x7f07ac133e70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.259+0000 7f07b479d640 1 --2- 172.21.15.181:0/1337661768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07ac098c70 0x7f07ac141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.260+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 >> 172.21.15.181:0/1337661768 conn(0x7f07ac070e90 msgr2=0x7f07ac134ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.260+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 shutdown_connections 2024-09-15T03:34:25.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.260+0000 7f07b479d640 1 -- 172.21.15.181:0/1337661768 wait complete. 2024-09-15T03:34:25.270 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool set cold size 2 2024-09-15T03:34:25.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.344+0000 7fa1a8e90640 10 monclient: get_monmap_and_config 2024-09-15T03:34:25.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.344+0000 7fa1a8e90640 10 monclient: build_initial_monmap 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.344+0000 7fa1a8e90640 10 monclient: monmap: 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:25.345393+0000 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:25.345393+0000 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:25.346 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:25.347 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:25.347 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 1 Processor -- start 2024-09-15T03:34:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 1 -- start start 2024-09-15T03:34:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient: init 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa1a409cad0 0x7fa1a409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a409cad0 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1a409cea0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1a409d2a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a409b710 con 0x7fa1a409cea0 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a409c6e0 con 0x7fa1a409d2a0 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a409e8f0 con 0x7fa1a409cad0 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a8e90640 10 monclient(hunting): authenticate will time out at 1069.826758s 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a2d9d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fa1a409d2a0 legacy=0x7fa1a409e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.345+0000 7fa1a259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa1a409cad0 0x7fa1a409c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa1a259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa1a409cad0 0x7fa1a409c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58666/0 (socket says 172.21.15.181:58666) 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa1a259c640 1 -- 172.21.15.181:0/2623973251 learned_addr learned my addr 172.21.15.181:0/2623973251 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa1a259c640 10 monclient(hunting): get_auth_request con 0x7fa1a409cad0 auth_method 0 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa1a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa199d9b640 1 --1- 172.21.15.181:0/2623973251 >> v1:172.21.15.153:6789/0 conn(0x7fa1a409cea0 0x7fa1a409dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:57038/0 (socket says 172.21.15.181:57038) 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa1a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.346+0000 7fa1a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa199d9b640 1 -- 172.21.15.181:0/2623973251 >> v1:172.21.15.153:6789/0 conn(0x7fa1a409cea0 legacy=0x7fa1a409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 10 monclient(hunting): handle_auth_done global_id 84279 payload 274 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 1 -- 172.21.15.181:0/2623973251 >> v1:172.21.15.181:6789/0 conn(0x7fa1a409d2a0 legacy=0x7fa1a409e1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 1 -- 172.21.15.181:0/2623973251 >> v1:172.21.15.153:6789/0 conn(0x7fa1a409cea0 legacy=0x7fa1a409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 10 monclient: _finish_hunting 0 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 1 monclient: found mon.noname-a 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a259c640 1 -- 172.21.15.181:0/2623973251 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a409a650 con 0x7fa1a409cad0 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a1d9b640 1 -- 172.21.15.181:0/2623973251 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100424011 0 0) 0x7fa1a409b710 con 0x7fa1a409cea0 2024-09-15T03:34:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.348+0000 7fa1a1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:34:25.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a259c640 1 -- 172.21.15.181:0/2623973251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 msgr2=0x7fa1a409c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a259c640 1 --2- 172.21.15.181:0/2623973251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a409c1a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa194009990 tx=0x7fa1940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1622f9904fe20f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 1 -- 172.21.15.181:0/2623973251 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19402f4b0 con 0x7fa1a409cad0 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: dump: 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a8e90640 5 monclient: authenticate success, global_id 84279 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 1 -- 172.21.15.181:0/2623973251 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19402f610 con 0x7fa1a409cad0 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a8e90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a8e90640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a8e90640 4 set_mon_vals no callback set 2024-09-15T03:34:25.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 1 -- 172.21.15.181:0/2623973251 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19402f9b0 con 0x7fa1a409cad0 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.351+0000 7fa1a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a1d9b640 10 monclient: dump: 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:25.359 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 10 monclient: shutdown 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 -- 172.21.15.181:0/2623973251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 msgr2=0x7fa1a409c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/2623973251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a409c1a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa194009990 tx=0x7fa1940091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 -- 172.21.15.181:0/2623973251 shutdown_connections 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/2623973251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a409c1a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 -- 172.21.15.181:0/2623973251 >> 172.21.15.181:0/2623973251 conn(0x7fa1a4070e90 msgr2=0x7fa1a40a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 -- 172.21.15.181:0/2623973251 shutdown_connections 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.352+0000 7fa1a8e90640 1 -- 172.21.15.181:0/2623973251 wait complete. 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient: build_initial_monmap 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient: monmap: 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:25.361 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 Processor -- start 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 -- start start 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient: init 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a412f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a409cad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 0x7fa1a4119740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a409cea0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1a409d2a0 0x7fa1a4119c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:25.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a409d2a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a411cdc0 con 0x7fa1a409cad0 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a411cae0 con 0x7fa1a409d2a0 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a411b040 con 0x7fa1a409cea0 2024-09-15T03:34:25.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a8e90640 10 monclient(hunting): authenticate will time out at 1069.834921s 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a2d9d640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1a409d2a0 msgr2=0x7fa1a4119c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.353+0000 7fa1a2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1a409d2a0 0x7fa1a4119c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a412f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 0x7fa1a4119740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a412f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58674/0 (socket says 172.21.15.181:58674) 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 0x7fa1a4119740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42370/0 (socket says 172.21.15.181:42370) 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 1 -- 172.21.15.181:0/471097495 learned_addr learned my addr 172.21.15.181:0/471097495 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): get_auth_request con 0x7fa1a409cad0 auth_method 0 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:25.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): get_auth_request con 0x7fa1a409cea0 auth_method 0 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient(hunting): handle_auth_done global_id 74396 payload 824 2024-09-15T03:34:25.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 -- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1a409d2a0 msgr2=0x7fa1a4119c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1a409d2a0 0x7fa1a4119c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 -- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 msgr2=0x7fa1a412f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a412f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 monclient: found mon.noname-c 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:25.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa199d9b640 1 -- 172.21.15.181:0/471097495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa194009450 con 0x7fa1a409cea0 2024-09-15T03:34:25.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.354+0000 7fa1a259c640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a412f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.356+0000 7fa199d9b640 1 -- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 msgr2=0x7fa1a4119740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.356+0000 7fa199d9b640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 0x7fa1a4119740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa18c001490 tx=0x7fa18c007b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0074f0 con 0x7fa1a409cea0 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: dump: 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.370 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: _finish_auth 0 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c007670 con 0x7fa1a409cea0 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 5 monclient: authenticate success, global_id 74396 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 10 monclient: _renew_subs 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a413fbd0 con 0x7fa1a409cea0 2024-09-15T03:34:25.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0049f0 con 0x7fa1a409cea0 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa19affd640 10 monclient: dump: 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:25.372 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 10 monclient: _renew_subs 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a159a640 4 set_mon_vals no callback set 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.357+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a411bd90 con 0x7fa1a409cea0 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa1a8e90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa1a8e90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa1a8e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a411cf80 con 0x7fa1a409cea0 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa18c013400 con 0x7fa1a409cea0 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa19affd640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa184038740 0x7fa18403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(138..138 src has 106..138) v4 ==== 7899+0+0 (secure 0 0 0) 0x7fa18c04b8e0 con 0x7fa1a409cea0 2024-09-15T03:34:25.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa1a259c640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa184038740 0x7fa18403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:25.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.358+0000 7fa1a259c640 10 monclient: get_auth_request con 0x7fa184038740 auth_method 0 2024-09-15T03:34:25.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.360+0000 7fa1a259c640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa184038740 0x7fa18403abf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa194009210 tx=0x7fa194032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:25.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.362+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa18c0136b0 con 0x7fa1a409cea0 2024-09-15T03:34:25.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.362+0000 7fa19affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:25.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.362+0000 7fa19affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.516+0000 7fa1a8e90640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T03:34:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.516+0000 7fa1a8e90640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T03:34:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.516+0000 7fa1a8e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:25.516+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7fa1a411bd90 con 0x7fa1a409cea0 2024-09-15T03:34:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.168+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v139) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa1a411bd90 con 0x7fa1a409cea0 2024-09-15T03:34:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.168+0000 7fa19affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T03:34:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.168+0000 7fa19affd640 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-15T03:34:26.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.263+0000 7fa1a8e90640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T03:34:26.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.263+0000 7fa1a8e90640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T03:34:26.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.263+0000 7fa1a8e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:26.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:26.263+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7fa1a411bd90 con 0x7fa1a409cea0 2024-09-15T03:34:27.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.169+0000 7fa19affd640 1 -- 172.21.15.181:0/471097495 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v140) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa1a411bd90 con 0x7fa1a409cea0 2024-09-15T03:34:27.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.169+0000 7fa19affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.169+0000 7fa19affd640 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 12 size to 2 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa184038740 msgr2=0x7fa18403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa184038740 0x7fa18403abf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa194009210 tx=0x7fa194032000 comp rx=0 tx=0).stop 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 10 monclient: shutdown 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 msgr2=0x7fa1a4119740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 0x7fa1a4119740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa18c001490 tx=0x7fa18c007b60 comp rx=0 tx=0).stop 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 shutdown_connections 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa184038740 0x7fa18403abf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1a409d2a0 0x7fa1a4119c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1a409cea0 0x7fa1a4119740 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 --2- 172.21.15.181:0/471097495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1a409cad0 0x7fa1a412f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 >> 172.21.15.181:0/471097495 conn(0x7fa1a4070e90 msgr2=0x7fa1a4132a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 shutdown_connections 2024-09-15T03:34:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.171+0000 7fa1a8e90640 1 -- 172.21.15.181:0/471097495 wait complete. 2024-09-15T03:34:27.183 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T03:34:27.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.259+0000 7fa54779d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:27.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.259+0000 7fa54779d640 10 monclient: build_initial_monmap 2024-09-15T03:34:27.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.259+0000 7fa54779d640 10 monclient: monmap: 2024-09-15T03:34:27.260 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:27.259964+0000 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:27.259964+0000 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:27.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.259+0000 7fa54779d640 1 Processor -- start 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.259+0000 7fa54779d640 1 -- start start 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.259+0000 7fa54779d640 10 monclient: init 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 0x7fa540131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): picked mon.noname-a con 0x7fa54009ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 0x7fa540068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5401321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa54009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:27.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540067420 con 0x7fa54009f020 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400a0480 con 0x7fa54009ec50 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400a0200 con 0x7fa5401321a0 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa54779d640 10 monclient(hunting): authenticate will time out at 1071.741515s 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545d3a640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fa54009f020 legacy=0x7fa540069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa544d38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 0x7fa540068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 0x7fa540131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 0x7fa540131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58688/0 (socket says 172.21.15.181:58688) 2024-09-15T03:34:27.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa544d38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 0x7fa540068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42380/0 (socket says 172.21.15.181:42380) 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 1 -- 172.21.15.181:0/1410687748 learned_addr learned my addr 172.21.15.181:0/1410687748 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 10 monclient(hunting): get_auth_request con 0x7fa54009ec50 auth_method 0 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa545539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa544d38640 10 monclient(hunting): get_auth_request con 0x7fa5401321a0 auth_method 0 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa544d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa544d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.260+0000 7fa544d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa545539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa545539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:27.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa545539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 10 monclient(hunting): handle_auth_done global_id 74399 payload 274 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 1 -- 172.21.15.181:0/1410687748 >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 msgr2=0x7fa540131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 1 --2- 172.21.15.181:0/1410687748 >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 0x7fa540131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 1 -- 172.21.15.181:0/1410687748 >> v1:172.21.15.181:6789/0 conn(0x7fa54009f020 legacy=0x7fa540069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 10 monclient: _finish_hunting 0 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 1 monclient: found mon.noname-b 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:27.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa544d38640 1 -- 172.21.15.181:0/1410687748 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54009fd80 con 0x7fa5401321a0 2024-09-15T03:34:27.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.261+0000 7fa545539640 1 --2- 172.21.15.181:0/1410687748 >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 0x7fa540131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa544d38640 1 -- 172.21.15.181:0/1410687748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 msgr2=0x7fa540068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa544d38640 1 --2- 172.21.15.181:0/1410687748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 0x7fa540068d00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa534009850 tx=0x7fa5340090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa5a99c13ae2495f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 1 -- 172.21.15.181:0/1410687748 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53402ae80 con 0x7fa5401321a0 2024-09-15T03:34:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:27.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: dump: 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: _finish_auth 0 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:27.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 1 -- 172.21.15.181:0/1410687748 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa534032440 con 0x7fa5401321a0 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 1 -- 172.21.15.181:0/1410687748 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534032800 con 0x7fa5401321a0 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa533fff640 10 monclient: dump: 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:27.276 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 5 monclient: authenticate success, global_id 74399 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 4 set_mon_vals no callback set 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 10 monclient: shutdown 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 -- 172.21.15.181:0/1410687748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 msgr2=0x7fa540068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 --2- 172.21.15.181:0/1410687748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 0x7fa540068d00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa534009850 tx=0x7fa5340090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 -- 172.21.15.181:0/1410687748 shutdown_connections 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 --2- 172.21.15.181:0/1410687748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5401321a0 0x7fa540068d00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 --2- 172.21.15.181:0/1410687748 >> v2:172.21.15.153:3300/0 conn(0x7fa54009ec50 0x7fa540131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 -- 172.21.15.181:0/1410687748 >> 172.21.15.181:0/1410687748 conn(0x7fa5400a5cf0 msgr2=0x7fa5400a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:27.277 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 -- 172.21.15.181:0/1410687748 shutdown_connections 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.264+0000 7fa54779d640 1 -- 172.21.15.181:0/1410687748 wait complete. 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.265+0000 7fa54779d640 10 monclient: build_initial_monmap 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.265+0000 7fa54779d640 10 monclient: monmap: 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:27.278 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.265+0000 7fa54779d640 1 Processor -- start 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.265+0000 7fa54779d640 1 -- start start 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.265+0000 7fa54779d640 10 monclient: init 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 0x7fa54009a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): picked mon.noname-c con 0x7fa54009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa54009f020 0x7fa540130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): picked mon.noname-b con 0x7fa54009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5401321a0 0x7fa5401410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5401321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54011dee0 con 0x7fa5401321a0 2024-09-15T03:34:27.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54011e1c0 con 0x7fa54009f020 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54011e4a0 con 0x7fa54009ec50 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa54779d640 10 monclient(hunting): authenticate will time out at 1071.747349s 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa544d38640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa54009f020 msgr2=0x7fa540130ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa544d38640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa54009f020 0x7fa540130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 0x7fa54009a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5401321a0 0x7fa5401410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 0x7fa54009a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:42390/0 (socket says 172.21.15.181:42390) 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 -- 172.21.15.181:0/3315023145 learned_addr learned my addr 172.21.15.181:0/3315023145 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): get_auth_request con 0x7fa54009ec50 auth_method 0 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): get_auth_request con 0x7fa5401321a0 auth_method 0 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient(hunting): handle_auth_done global_id 74402 payload 824 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 -- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa54009f020 msgr2=0x7fa540130ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa54009f020 0x7fa540130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 -- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5401321a0 msgr2=0x7fa5401410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5401321a0 0x7fa5401410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient: _finish_hunting 0 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 monclient: found mon.noname-c 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545539640 1 -- 172.21.15.181:0/3315023145 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c009450 con 0x7fa54009ec50 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.266+0000 7fa545d3a640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5401321a0 0x7fa5401410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:27.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa545539640 1 -- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 msgr2=0x7fa54009a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa545539640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 0x7fa54009a940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00ea30 tx=0x7fa53c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c009690 con 0x7fa54009ec50 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: dump: 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: _finish_auth 0 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c00cc20 con 0x7fa54009ec50 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 5 monclient: authenticate success, global_id 74402 2024-09-15T03:34:27.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 10 monclient: _renew_subs 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c013470 con 0x7fa54009ec50 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa534009390 con 0x7fa54009ec50 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5337fe640 4 set_mon_vals no callback set 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa5317fa640 10 monclient: dump: 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 10 monclient: _renew_subs 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.269+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54011cd90 con 0x7fa54009ec50 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa53c013820 con 0x7fa54009ec50 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa5317fa640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa524038740 0x7fa52403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(140..140 src has 106..140) v4 ==== 7899+0+0 (secure 0 0 0) 0x7fa53c04ca40 con 0x7fa54009ec50 2024-09-15T03:34:27.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa54779d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa54779d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa54779d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa544d38640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa524038740 0x7fa52403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5400008d0 con 0x7fa54009ec50 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.270+0000 7fa544d38640 10 monclient: get_auth_request con 0x7fa524038740 auth_method 0 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.273+0000 7fa544d38640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa524038740 0x7fa52403abf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa534034000 tx=0x7fa534032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.275+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa53c021020 con 0x7fa54009ec50 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.275+0000 7fa5317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:27.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.275+0000 7fa5317fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:27.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.435+0000 7fa54779d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:27.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.436+0000 7fa54779d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:27.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.436+0000 7fa54779d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:27.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:27.436+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fa5400a13e0 con 0x7fa54009ec50 2024-09-15T03:34:28.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.179+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v141) v1 ==== 82+0+0 (secure 0 0 0) 0x7fa53c0236b0 con 0x7fa54009ec50 2024-09-15T03:34:28.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.179+0000 7fa5317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:28.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.179+0000 7fa5317fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:28.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.279+0000 7fa54779d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:28.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.279+0000 7fa54779d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:28.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.279+0000 7fa54779d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:28.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.279+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fa540072640 con 0x7fa54009ec50 2024-09-15T03:34:28.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.280+0000 7fa5317fa640 1 -- 172.21.15.181:0/3315023145 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v141) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa53c0232f0 con 0x7fa54009ec50 2024-09-15T03:34:28.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.280+0000 7fa5317fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:34:28.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.280+0000 7fa5317fa640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T03:34:28.281 INFO:tasks.workunit.client.0.smithi181.stderr:compat weight-set already created 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa524038740 msgr2=0x7fa52403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa524038740 0x7fa52403abf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa534034000 tx=0x7fa534032000 comp rx=0 tx=0).stop 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 10 monclient: shutdown 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 msgr2=0x7fa54009a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 0x7fa54009a940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00ea30 tx=0x7fa53c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 shutdown_connections 2024-09-15T03:34:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa524038740 0x7fa52403abf0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5401321a0 0x7fa5401410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa54009f020 0x7fa540130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 --2- 172.21.15.181:0/3315023145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa54009ec50 0x7fa54009a940 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 >> 172.21.15.181:0/3315023145 conn(0x7fa5400a5cf0 msgr2=0x7fa54009e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:28.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 shutdown_connections 2024-09-15T03:34:28.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.282+0000 7fa54779d640 1 -- 172.21.15.181:0/3315023145 wait complete. 2024-09-15T03:34:28.292 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create cool flat 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.366+0000 7f01bc35d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.366+0000 7f01bc35d640 10 monclient: build_initial_monmap 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.366+0000 7f01bc35d640 10 monclient: monmap: 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:28.366863+0000 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:28.366863+0000 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.367 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:28.368 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:28.368 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:28.368 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:28.368 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:28.368 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.366+0000 7f01bc35d640 1 Processor -- start 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.366+0000 7f01bc35d640 1 -- start start 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.366+0000 7f01bc35d640 10 monclient: init 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01b4131bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f01b4069e10 0x7f01b4131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient(hunting): picked mon.noname-c con 0x7f01b4069e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01b4136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40a06d0 con 0x7f01b4131bd0 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b4071780 con 0x7f01b4136d60 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b406a210 con 0x7f01b4069e10 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01bc35d640 10 monclient(hunting): authenticate will time out at 1072.848341s 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01ba8fa640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f01b4136d60 legacy=0x7f01b4139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01b98f8640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f01b4069e10 msgr2=0x7f01b4131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01b98f8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f01b4069e10 0x7f01b4131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01ba0f9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f01b4131bd0 0x7f01b409ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:52964/0 (socket says 172.21.15.181:52964) 2024-09-15T03:34:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.367+0000 7f01ba0f9640 1 -- 172.21.15.181:0/3420186593 learned_addr learned my addr 172.21.15.181:0/3420186593 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.369+0000 7f01ba0f9640 1 -- 172.21.15.181:0/3420186593 >> v1:172.21.15.153:6789/0 conn(0x7f01b4131bd0 legacy=0x7f01b409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034320725 0 0) 0x7f01b40a06d0 con 0x7f01b4131bd0 2024-09-15T03:34:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): my global_id is 84294 2024-09-15T03:34:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019c0034e0 con 0x7f01b4131bd0 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968482646 0 0) 0x7f019c0034e0 con 0x7f01b4131bd0 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 >> v2:172.21.15.181:3300/0 conn(0x7f01b4069e10 msgr2=0x7f01b4131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 --2- 172.21.15.181:0/3420186593 >> v2:172.21.15.181:3300/0 conn(0x7f01b4069e10 0x7f01b4131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 >> v1:172.21.15.181:6789/0 conn(0x7f01b4136d60 legacy=0x7f01b4139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: _finish_hunting 0 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b40a0010 con 0x7f01b4131bd0 2024-09-15T03:34:28.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: _finish_auth 0 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01bc35d640 5 monclient: authenticate success, global_id 84294 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01bc35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f01ac0017e0 con 0x7f01b4131bd0 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: dump: 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.386 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 >> v1:172.21.15.153:6789/0 conn(0x7f01b4131bd0 legacy=0x7f01b409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 0x7f019c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): picked mon.a con 0x7f019c0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c004780 0x7f019c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): picked mon.c con 0x7f019c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f019c005330 0x7f019c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): picked mon.b con 0x7f019c005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b406a210 con 0x7f019c0039d0 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c00c290 con 0x7f019c005330 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c00c450 con 0x7f019c004780 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b90f7640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3420186593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f019c005330 msgr2=0x7f019c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01bc35d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:28.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba8fa640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f019c005330 0x7f019c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 0x7f019c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): get_auth_request con 0x7f019c0039d0 auth_method 0 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c004780 0x7f019c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:28.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): get_auth_request con 0x7f019c004780 auth_method 0 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01b98f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_done global_id 84294 payload 293 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 -- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c004780 msgr2=0x7f019c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c004780 0x7f019c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 -- 172.21.15.181:0/3420186593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f019c005330 msgr2=0x7f019c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f019c005330 0x7f019c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 monclient: found mon.a 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3420186593 reap_dead start 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 1 -- 172.21.15.181:0/3420186593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c0034e0 con 0x7f019c0039d0 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 10 monclient: _finish_auth 0 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.370+0000 7f01ba0f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.373+0000 7f01ba0f9640 1 -- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 msgr2=0x7f019c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.373+0000 7f01ba0f9640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 0x7f019c003dc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01ac00c5c0 tx=0x7f01b406a350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af84ebd9c12bcb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:28.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac005890 con 0x7f019c0039d0 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: dump: 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:28.392 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac005a10 con 0x7f019c0039d0 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 1 -- 172.21.15.181:0/3420186593 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac005dd0 con 0x7f019c0039d0 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 4 set_mon_vals no callback set 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01b90f7640 10 monclient: dump: 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:28.393 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 10 monclient: shutdown 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 -- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 msgr2=0x7f019c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 0x7f019c003dc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01ac00c5c0 tx=0x7f01b406a350 comp rx=0 tx=0).stop 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 -- 172.21.15.181:0/3420186593 shutdown_connections 2024-09-15T03:34:28.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3420186593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f019c0039d0 0x7f019c003dc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 -- 172.21.15.181:0/3420186593 >> 172.21.15.181:0/3420186593 conn(0x7f01b40713b0 msgr2=0x7f01b40a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 -- 172.21.15.181:0/3420186593 shutdown_connections 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.374+0000 7f01bc35d640 1 -- 172.21.15.181:0/3420186593 wait complete. 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient: build_initial_monmap 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient: monmap: 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:28.395 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 1 Processor -- start 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 1 -- start start 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient: init 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c0039d0 0x7f01b40997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient(hunting): picked mon.noname-c con 0x7f019c0039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01b4099d30 0x7f01b409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient(hunting): picked mon.noname-b con 0x7f01b4099d30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.375+0000 7f01bc35d640 10 monclient(hunting): picked mon.noname-a con 0x7f01b409a660 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b411c8d0 con 0x7f01b409a660 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b409b660 con 0x7f01b4099d30 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b40a4890 con 0x7f019c0039d0 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:28.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01bc35d640 10 monclient(hunting): authenticate will time out at 1072.857021s 2024-09-15T03:34:28.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01b98f8640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01b4099d30 msgr2=0x7f01b409a120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01b98f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01b4099d30 0x7f01b409a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01b27fc640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c0039d0 0x7f01b40997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:57476/0 (socket says 172.21.15.181:57476) 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3180880319 learned_addr learned my addr 172.21.15.181:0/3180880319 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): get_auth_request con 0x7f01b409a660 auth_method 0 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:28.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): get_auth_request con 0x7f019c0039d0 auth_method 0 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient(hunting): handle_auth_done global_id 84297 payload 824 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c0039d0 msgr2=0x7f01b40997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c0039d0 0x7f01b40997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01b4099d30 msgr2=0x7f01b409a120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01b4099d30 0x7f01b409a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient: _finish_hunting 0 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 monclient: found mon.noname-a 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.376+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3180880319 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b409bec0 con 0x7f01b409a660 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.378+0000 7f01ba8fa640 1 -- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 msgr2=0x7f01b4147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.378+0000 7f01ba8fa640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01b40a02f0 tx=0x7f01a4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a400e840 con 0x7f01b409a660 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 10 monclient: dump: 2024-09-15T03:34:28.404 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:28.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01ba8fa640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01b40a02f0 tx=0x7f01a4009580 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f01a400e9c0 config(0 keys) v1 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a400e9c0 con 0x7f01b409a660 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 5 monclient: authenticate success, global_id 84297 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 10 monclient: _renew_subs 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b411d170 con 0x7f01b409a660 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01b88f6640 4 set_mon_vals no callback set 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 10 monclient: _renew_subs 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.379+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b409b0e0 con 0x7f01b409a660 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01bc35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01bc35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:28.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01bc35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b404bf60 con 0x7f01b409a660 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a400ed80 con 0x7f01b409a660 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01b27fc640 10 monclient: dump: 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:28.407 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f01a4006360 con 0x7f01b409a660 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01b27fc640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01a00386f0 0x7f01a003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01ba0f9640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01a00386f0 0x7f01a003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.380+0000 7f01ba0f9640 10 monclient: get_auth_request con 0x7f01a00386f0 auth_method 0 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.382+0000 7f01ba0f9640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01a00386f0 0x7f01a003aba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f01ac001490 tx=0x7f01ac034000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:28.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:28.385+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(141..141 src has 106..141) v4 ==== 8231+0+0 (secure 0 0 0) 0x7f01a404c6b0 con 0x7f01b409a660 2024-09-15T03:34:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.207+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01a400cd40 con 0x7f01b409a660 2024-09-15T03:34:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.207+0000 7f01b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:29.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.207+0000 7f01b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:29.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.370+0000 7f01bc35d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T03:34:29.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.370+0000 7f01bc35d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T03:34:29.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.370+0000 7f01bc35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:29.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:29.370+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f01b4136210 con 0x7f01b409a660 2024-09-15T03:34:30.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.190+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v142) v1 ==== 107+0+0 (secure 0 0 0) 0x7f01a4012030 con 0x7f01b409a660 2024-09-15T03:34:30.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.191+0000 7f01b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T03:34:30.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.191+0000 7f01b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:30.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.291+0000 7f01bc35d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.291+0000 7f01bc35d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.291+0000 7f01bc35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.291+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f01b4072640 con 0x7f01b409a660 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.292+0000 7f01b27fc640 1 -- 172.21.15.181:0/3180880319 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v142) v1 ==== 149+0+0 (secure 0 0 0) 0x7f01a4012030 con 0x7f01b409a660 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.292+0000 7f01b27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.292+0000 7f01b27fc640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-15T03:34:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:weight-set for pool 'cool' already created 2024-09-15T03:34:30.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01a00386f0 msgr2=0x7f01a003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:30.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01a00386f0 0x7f01a003aba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f01ac001490 tx=0x7f01ac034000 comp rx=0 tx=0).stop 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 10 monclient: shutdown 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 msgr2=0x7f01b4147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01b40a02f0 tx=0x7f01a4009580 comp rx=0 tx=0).stop 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 shutdown_connections 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01a00386f0 0x7f01a003aba0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01b409a660 0x7f01b4147200 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01b4099d30 0x7f01b409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 --2- 172.21.15.181:0/3180880319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f019c0039d0 0x7f01b40997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 >> 172.21.15.181:0/3180880319 conn(0x7f01b40713b0 msgr2=0x7f01b40a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 shutdown_connections 2024-09-15T03:34:30.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.294+0000 7f01bc35d640 1 -- 172.21.15.181:0/3180880319 wait complete. 2024-09-15T03:34:30.304 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create cold positional 2024-09-15T03:34:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.378+0000 7f0bd3f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.378+0000 7f0bd3f9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.378+0000 7f0bd3f9d640 10 monclient: monmap: 2024-09-15T03:34:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:30.379406+0000 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:30.379406+0000 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:30.380 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:30.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 Processor -- start 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 -- start start 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient: init 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 0x7f0bcc09d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc09d140 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0bcc09da50 0x7f0bcc094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0bcc09da50 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bcc0945c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc098570 con 0x7f0bcc0945c0 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc0670b0 con 0x7f0bcc09da50 2024-09-15T03:34:30.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc097fb0 con 0x7f0bcc09d140 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd3f9d640 10 monclient(hunting): authenticate will time out at 1074.860886s 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd1538640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f0bcc09da50 msgr2=0x7f0bcc094080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.379+0000 7f0bd1538640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0bcc09da50 0x7f0bcc094080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd253a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0bcc0945c0 0x7f0bcc136d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:52980/0 (socket says 172.21.15.181:52980) 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd253a640 1 -- 172.21.15.181:0/2433676822 learned_addr learned my addr 172.21.15.181:0/2433676822 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 1 --2- 172.21.15.181:0/2433676822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 0x7f0bcc09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:30.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): get_auth_request con 0x7f0bcc09d140 auth_method 0 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_done global_id 74408 payload 274 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 1 -- 172.21.15.181:0/2433676822 >> v2:172.21.15.181:3300/0 conn(0x7f0bcc09da50 msgr2=0x7f0bcc094080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 1 --2- 172.21.15.181:0/2433676822 >> v2:172.21.15.181:3300/0 conn(0x7f0bcc09da50 0x7f0bcc094080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 1 -- 172.21.15.181:0/2433676822 >> v1:172.21.15.153:6789/0 conn(0x7f0bcc0945c0 legacy=0x7f0bcc136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient: _finish_hunting 0 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 1 monclient: found mon.noname-b 2024-09-15T03:34:30.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:30.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:30.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.380+0000 7f0bd1d39640 1 -- 172.21.15.181:0/2433676822 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc064cb0 con 0x7f0bcc09d140 2024-09-15T03:34:30.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd1d39640 1 -- 172.21.15.181:0/2433676822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 msgr2=0x7f0bcc09d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd1d39640 1 --2- 172.21.15.181:0/2433676822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 0x7f0bcc09d510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009850 tx=0x7f0bc80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c417b724d1abc3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 1 -- 172.21.15.181:0/2433676822 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8032440 con 0x7f0bcc09d140 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: dump: 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:30.394 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: _finish_auth 0 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 1 -- 172.21.15.181:0/2433676822 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc80325a0 con 0x7f0bcc09d140 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 1 -- 172.21.15.181:0/2433676822 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8032940 con 0x7f0bcc09d140 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd3f9d640 5 monclient: authenticate success, global_id 74408 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd3f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:30.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd3f9d640 4 set_mon_vals no callback set 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.383+0000 7f0bd0d37640 10 monclient: dump: 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:30.396 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 10 monclient: shutdown 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/2433676822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 msgr2=0x7f0bcc09d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/2433676822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 0x7f0bcc09d510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009850 tx=0x7f0bc80090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/2433676822 shutdown_connections 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/2433676822 >> v2:172.21.15.181:3300/0 conn(0x7f0bcc09da50 0x7f0bcc094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/2433676822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc09d140 0x7f0bcc09d510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/2433676822 >> 172.21.15.181:0/2433676822 conn(0x7f0bcc0a2190 msgr2=0x7f0bcc09e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/2433676822 shutdown_connections 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/2433676822 wait complete. 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:30.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.384+0000 7f0bd3f9d640 10 monclient: monmap: 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 Processor -- start 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 -- start start 2024-09-15T03:34:30.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient: init 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 0x7f0bcc0935e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0bcc0945c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bcc09d140 0x7f0bcc093b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0bcc09d140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bcc09da50 0x7f0bcc13ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc09da50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc0a11f0 con 0x7f0bcc09d140 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc08b690 con 0x7f0bcc09da50 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:30.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc08d7d0 con 0x7f0bcc0945c0 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd3f9d640 10 monclient(hunting): authenticate will time out at 1074.866513s 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd253a640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bcc09da50 msgr2=0x7f0bcc13ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd253a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bcc09da50 0x7f0bcc13ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 0x7f0bcc0935e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1538640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bcc09d140 0x7f0bcc093b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 0x7f0bcc0935e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38418/0 (socket says 172.21.15.181:38418) 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 1 -- 172.21.15.181:0/471284472 learned_addr learned my addr 172.21.15.181:0/471284472 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): get_auth_request con 0x7f0bcc0945c0 auth_method 0 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:30.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1538640 10 monclient(hunting): get_auth_request con 0x7f0bcc09d140 auth_method 0 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1538640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1538640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.385+0000 7f0bd1538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 10 monclient(hunting): handle_auth_done global_id 74411 payload 824 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 1 -- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bcc09da50 msgr2=0x7f0bcc13ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bcc09da50 0x7f0bcc13ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 1 -- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bcc09d140 msgr2=0x7f0bcc093b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bcc09d140 0x7f0bcc093b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 10 monclient: _finish_hunting 0 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 1 monclient: found mon.noname-c 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.386+0000 7f0bd1d39640 1 -- 172.21.15.181:0/471284472 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8009390 con 0x7f0bcc0945c0 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bd1d39640 1 -- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 msgr2=0x7f0bcc0935e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bd1d39640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 0x7f0bcc0935e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80098b0 tx=0x7f0bc8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8031940 con 0x7f0bcc0945c0 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: dump: 2024-09-15T03:34:30.402 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:30.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc8031aa0 con 0x7f0bcc0945c0 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bd3f9d640 5 monclient: authenticate success, global_id 74411 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bd3f9d640 10 monclient: _renew_subs 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bd3f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc08dab0 con 0x7f0bcc0945c0 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8031e40 con 0x7f0bcc0945c0 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc1ffb640 10 monclient: dump: 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 10 monclient: _renew_subs 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.388+0000 7f0bc3fff640 4 set_mon_vals no callback set 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc13f460 con 0x7f0bcc0945c0 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:30.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.389+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc0008d0 con 0x7f0bcc0945c0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.390+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0bc8039070 con 0x7f0bcc0945c0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.390+0000 7f0bc1ffb640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0bb00386f0 0x7f0bb003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.390+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(142..142 src has 106..142) v4 ==== 8563+0+0 (secure 0 0 0) 0x7f0bc806cfd0 con 0x7f0bcc0945c0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.390+0000 7f0bd1538640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0bb00386f0 0x7f0bb003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.390+0000 7f0bd1538640 10 monclient: get_auth_request con 0x7f0bb00386f0 auth_method 0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.392+0000 7f0bd1538640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0bb00386f0 0x7f0bb003aba0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc08c0d0 tx=0x7f0bbc0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.394+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0bc803b5a0 con 0x7f0bcc0945c0 2024-09-15T03:34:30.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.394+0000 7f0bc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:30.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.394+0000 7f0bc1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:30.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.556+0000 7f0bd3f9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T03:34:30.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.556+0000 7f0bd3f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T03:34:30.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.556+0000 7f0bd3f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:30.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:30.556+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f0bcc09ec20 con 0x7f0bcc0945c0 2024-09-15T03:34:31.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.188+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v143) v1 ==== 113+0+0 (secure 0 0 0) 0x7f0bc8040030 con 0x7f0bcc0945c0 2024-09-15T03:34:31.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.188+0000 7f0bc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T03:34:31.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.188+0000 7f0bc1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:31.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.290+0000 7f0bd3f9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T03:34:31.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.290+0000 7f0bd3f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T03:34:31.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.290+0000 7f0bd3f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:31.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.290+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f0bcc09ec20 con 0x7f0bcc0945c0 2024-09-15T03:34:31.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.291+0000 7f0bc1ffb640 1 -- 172.21.15.181:0/471284472 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v143) v1 ==== 155+0+0 (secure 0 0 0) 0x7f0bc8040020 con 0x7f0bcc0945c0 2024-09-15T03:34:31.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.291+0000 7f0bc1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-15T03:34:31.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.291+0000 7f0bc1ffb640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-15T03:34:31.293 INFO:tasks.workunit.client.0.smithi181.stderr:weight-set for pool 'cold' already created 2024-09-15T03:34:31.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0bb00386f0 msgr2=0x7f0bb003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:31.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0bb00386f0 0x7f0bb003aba0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc08c0d0 tx=0x7f0bbc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 10 monclient: shutdown 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 msgr2=0x7f0bcc0935e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 0x7f0bcc0935e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80098b0 tx=0x7f0bc8032000 comp rx=0 tx=0).stop 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 shutdown_connections 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0bb00386f0 0x7f0bb003aba0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bcc09da50 0x7f0bcc13ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bcc09d140 0x7f0bcc093b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 --2- 172.21.15.181:0/471284472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bcc0945c0 0x7f0bcc0935e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 >> 172.21.15.181:0/471284472 conn(0x7f0bcc0a2190 msgr2=0x7f0bcc09f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 shutdown_connections 2024-09-15T03:34:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.293+0000 7f0bd3f9d640 1 -- 172.21.15.181:0/471284472 wait complete. 2024-09-15T03:34:31.303 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.376+0000 7f962b15d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.376+0000 7f962b15d640 10 monclient: build_initial_monmap 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.376+0000 7f962b15d640 10 monclient: monmap: 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:31.377102+0000 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:31.377102+0000 2024-09-15T03:34:31.377 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:31.378 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 1 Processor -- start 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 1 -- start start 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient: init 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f962409ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f962409f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f962413a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient(hunting): picked mon.noname-b con 0x7f9624069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9624067420 con 0x7f962409f020 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240a0480 con 0x7f962409ec50 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962406a170 con 0x7f9624069d10 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f962b15d640 10 monclient(hunting): authenticate will time out at 1075.858647s 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f9628ef9640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f962409ec50 legacy=0x7f9624131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f96296fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f962413a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f9623fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f962409f020 0x7f9624136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:52992/0 (socket says 172.21.15.181:52992) 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f9623fff640 1 -- 172.21.15.181:0/3606107475 learned_addr learned my addr 172.21.15.181:0/3606107475 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f96296fa640 10 monclient(hunting): get_auth_request con 0x7f9624069d10 auth_method 0 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f96296fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f96296fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.377+0000 7f96296fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.378+0000 7f96296fa640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 msgr2=0x7f962413a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.378+0000 7f96296fa640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 msgr2=0x7f962413a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:31.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.378+0000 7f96296fa640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f962413a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:31.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.378+0000 7f96296fa640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f962413a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9623fff640 1 -- 172.21.15.181:0/3606107475 >> v1:172.21.15.153:6789/0 conn(0x7f962409f020 legacy=0x7f9624136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994898069 0 0) 0x7f9624067420 con 0x7f962409f020 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): my global_id is 84303 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96140034e0 con 0x7f962409f020 2024-09-15T03:34:31.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535423734 0 0) 0x7f96140034e0 con 0x7f962409f020 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 msgr2=0x7f962413a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f962413a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 >> v1:172.21.15.181:6789/0 conn(0x7f962409ec50 legacy=0x7f9624131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: _finish_hunting 0 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:34:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240a0200 con 0x7f962409f020 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f96100017e0 con 0x7f962409f020 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f962b15d640 5 monclient: authenticate success, global_id 84303 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f962b15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:34:31.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: dump: 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:31.392 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 >> v1:172.21.15.153:6789/0 conn(0x7f962409f020 legacy=0x7f9624136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9614003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): picked mon.a con 0x7f9614003a00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9614004780 0x7f9614004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): picked mon.b con 0x7f9614004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:31.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f961400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): picked mon.c con 0x7f9624069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614005460 con 0x7f9614003a00 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140055d0 con 0x7f9614004780 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961400bf10 con 0x7f9624069d10 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:31.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96237fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f962b15d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9623fff640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9614004780 msgr2=0x7f9614004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9623fff640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9614004780 0x7f9614004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9614003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): get_auth_request con 0x7f9614003a00 auth_method 0 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f961400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:31.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): get_auth_request con 0x7f9624069d10 auth_method 0 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f9628ef9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.380+0000 7f96296fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 10 monclient(hunting): handle_auth_done global_id 84303 payload 293 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 msgr2=0x7f961400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9624069d10 0x7f961400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9614004780 msgr2=0x7f9614004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9614004780 0x7f9614004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 1 monclient: found mon.a 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:31.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 1 -- 172.21.15.181:0/3606107475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140034e0 con 0x7f9614003a00 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 10 monclient: _finish_auth 0 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.381+0000 7f9628ef9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.383+0000 7f9628ef9640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 msgr2=0x7f9614003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.383+0000 7f9628ef9640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9614003df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f96180099c0 tx=0x7f9618009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2a6b90e350138c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618003b80 con 0x7f9614003a00 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:31.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: dump: 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618003d20 con 0x7f9614003a00 2024-09-15T03:34:31.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 1 -- 172.21.15.181:0/3606107475 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961802ab80 con 0x7f9614003a00 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 4 set_mon_vals no callback set 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96237fe640 10 monclient: dump: 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.400 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 10 monclient: shutdown 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 1 -- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 msgr2=0x7f9614003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 1 --2- 172.21.15.181:0/3606107475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9614003df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f96180099c0 tx=0x7f9618009060 comp rx=0 tx=0).stop 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f96296fa640 1 -- 172.21.15.181:0/3606107475 reap_dead start 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 1 -- 172.21.15.181:0/3606107475 shutdown_connections 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 1 -- 172.21.15.181:0/3606107475 >> 172.21.15.181:0/3606107475 conn(0x7f96240a5cf0 msgr2=0x7f96240a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 1 -- 172.21.15.181:0/3606107475 shutdown_connections 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.384+0000 7f962b15d640 1 -- 172.21.15.181:0/3606107475 wait complete. 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.385+0000 7f962b15d640 10 monclient: build_initial_monmap 2024-09-15T03:34:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.385+0000 7f962b15d640 10 monclient: monmap: 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.385+0000 7f962b15d640 1 Processor -- start 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.385+0000 7f962b15d640 1 -- start start 2024-09-15T03:34:31.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.385+0000 7f962b15d640 10 monclient: init 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9624130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): picked mon.noname-a con 0x7f9614003a00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9624130e40 0x7f962411d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): picked mon.noname-b con 0x7f9624130e40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 0x7f962411df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): picked mon.noname-c con 0x7f962411daa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9624131210 con 0x7f9614003a00 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962409c020 con 0x7f9624130e40 2024-09-15T03:34:31.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962409d830 con 0x7f962411daa0 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f962b15d640 10 monclient(hunting): authenticate will time out at 1075.867269s 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9623fff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9624130e40 msgr2=0x7f962411d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9623fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9624130e40 0x7f962411d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 0x7f962411df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9624130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 0x7f962411df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38458/0 (socket says 172.21.15.181:38458) 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 -- 172.21.15.181:0/2637229061 learned_addr learned my addr 172.21.15.181:0/2637229061 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): get_auth_request con 0x7f962411daa0 auth_method 0 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:31.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): get_auth_request con 0x7f9614003a00 auth_method 0 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient(hunting): handle_auth_done global_id 74417 payload 824 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 -- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9624130e40 msgr2=0x7f962411d560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9624130e40 0x7f962411d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 -- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 msgr2=0x7f9624130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:31.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9624130900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:31.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient: _finish_hunting 0 2024-09-15T03:34:31.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 monclient: found mon.noname-c 2024-09-15T03:34:31.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:31.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:31.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f96296fa640 1 -- 172.21.15.181:0/2637229061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96241210a0 con 0x7f962411daa0 2024-09-15T03:34:31.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.386+0000 7f9628ef9640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9624130900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:31.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f96296fa640 1 -- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 msgr2=0x7f962411df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:31.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f96296fa640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 0x7f962411df10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f961c009590 tx=0x7f961c009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:31.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c00f910 con 0x7f962411daa0 2024-09-15T03:34:31.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:31.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: dump: 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:31.409 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: _finish_auth 0 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c00fab0 con 0x7f962411daa0 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c00fe70 con 0x7f962411daa0 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9620ff9640 10 monclient: dump: 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 5 monclient: authenticate success, global_id 74417 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 10 monclient: _renew_subs 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962411ef20 con 0x7f962411daa0 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f9622ffd640 4 set_mon_vals no callback set 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 10 monclient: _renew_subs 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.389+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962411faf0 con 0x7f962411daa0 2024-09-15T03:34:31.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f961c014600 con 0x7f962411daa0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f962b15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f962b15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f962b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9624120b70 con 0x7f962411daa0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f9620ff9640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9608038740 0x7f960803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(143..143 src has 106..143) v4 ==== 9035+0+0 (secure 0 0 0) 0x7f961c04cd10 con 0x7f962411daa0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f9628ef9640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9608038740 0x7f960803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.390+0000 7f9628ef9640 10 monclient: get_auth_request con 0x7f9608038740 auth_method 0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.393+0000 7f9628ef9640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9608038740 0x7f960803abf0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f961802f000 tx=0x7f9618030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.395+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f961c01cb90 con 0x7f962411daa0 2024-09-15T03:34:31.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.395+0000 7f9620ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:31.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.395+0000 7f9620ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:31.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.555+0000 7f962b15d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:34:31.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.555+0000 7f962b15d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:34:31.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.555+0000 7f962b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:31.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:31.555+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f96240008d0 con 0x7f962411daa0 2024-09-15T03:34:32.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.199+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v144) v1 ==== 123+0+0 (secure 0 0 0) 0x7f961c01c3b0 con 0x7f962411daa0 2024-09-15T03:34:32.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.199+0000 7f9620ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:34:32.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.199+0000 7f9620ff9640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-15T03:34:32.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.295+0000 7f962b15d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:34:32.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.295+0000 7f962b15d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:34:32.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.295+0000 7f962b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:32.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:32.295+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f962409fd80 con 0x7f962411daa0 2024-09-15T03:34:33.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.199+0000 7f9620ff9640 1 -- 172.21.15.181:0/2637229061 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v145) v1 ==== 149+0+0 (secure 0 0 0) 0x7f961c01c590 con 0x7f962411daa0 2024-09-15T03:34:33.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.199+0000 7f9620ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:34:33.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.199+0000 7f9620ff9640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T03:34:33.201 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T03:34:33.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9608038740 msgr2=0x7f960803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:33.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9608038740 0x7f960803abf0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f961802f000 tx=0x7f9618030000 comp rx=0 tx=0).stop 2024-09-15T03:34:33.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 10 monclient: shutdown 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 msgr2=0x7f962411df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 0x7f962411df10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f961c009590 tx=0x7f961c009720 comp rx=0 tx=0).stop 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 shutdown_connections 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9608038740 0x7f960803abf0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962411daa0 0x7f962411df10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9624130e40 0x7f962411d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 --2- 172.21.15.181:0/2637229061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614003a00 0x7f9624130900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 >> 172.21.15.181:0/2637229061 conn(0x7f96240a5cf0 msgr2=0x7f96240a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 shutdown_connections 2024-09-15T03:34:33.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.202+0000 7f962b15d640 1 -- 172.21.15.181:0/2637229061 wait complete. 2024-09-15T03:34:33.214 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-15T03:34:33.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.294+0000 7f441599d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:33.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.294+0000 7f441599d640 10 monclient: build_initial_monmap 2024-09-15T03:34:33.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.294+0000 7f441599d640 10 monclient: monmap: 2024-09-15T03:34:33.295 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:33.295315+0000 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:33.295315+0000 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:33.296 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 1 Processor -- start 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 1 -- start start 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient: init 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f441009ec50 0x7f4410131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient(hunting): picked mon.noname-a con 0x7f441009ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:33.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44101321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f441009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4410067420 con 0x7f44101321a0 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441009fdc0 con 0x7f441009f020 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44100a0510 con 0x7f441009ec50 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f441599d640 10 monclient(hunting): authenticate will time out at 1077.776701s 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.295+0000 7f440f7fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f441009f020 legacy=0x7f4410069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f441009ec50 0x7f4410131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f441009ec50 0x7f4410131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:57522/0 (socket says 172.21.15.181:57522) 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 1 -- 172.21.15.181:0/3903610036 learned_addr learned my addr 172.21.15.181:0/3903610036 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): get_auth_request con 0x7f441009ec50 auth_method 0 2024-09-15T03:34:33.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:33.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:33.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:33.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:33.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:33.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.296+0000 7f440effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f44067fc640 1 -- 172.21.15.181:0/3903610036 >> v1:172.21.15.153:6789/0 conn(0x7f44101321a0 legacy=0x7f4410068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 10 monclient(hunting): handle_auth_done global_id 84309 payload 274 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 1 -- 172.21.15.181:0/3903610036 >> v1:172.21.15.181:6789/0 conn(0x7f441009f020 legacy=0x7f4410069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 1 -- 172.21.15.181:0/3903610036 >> v1:172.21.15.153:6789/0 conn(0x7f44101321a0 legacy=0x7f4410068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 10 monclient: _finish_hunting 0 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 1 monclient: found mon.noname-a 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:33.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.298+0000 7f440effd640 1 -- 172.21.15.181:0/3903610036 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100a0200 con 0x7f441009ec50 2024-09-15T03:34:33.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.301+0000 7f440effd640 1 -- 172.21.15.181:0/3903610036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009ec50 msgr2=0x7f4410131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:33.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.301+0000 7f440effd640 1 --2- 172.21.15.181:0/3903610036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009ec50 0x7f4410131c60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4400009990 tx=0x7f44000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42cd046238b5ff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:33.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 1 -- 172.21.15.181:0/3903610036 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440002ae90 con 0x7f441009ec50 2024-09-15T03:34:33.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:33.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: dump: 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:33.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 1 -- 172.21.15.181:0/3903610036 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440002f4b0 con 0x7f441009ec50 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 5 monclient: authenticate success, global_id 84309 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 1 -- 172.21.15.181:0/3903610036 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440002f850 con 0x7f441009ec50 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 4 set_mon_vals no callback set 2024-09-15T03:34:33.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:33.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f440e7fc640 10 monclient: dump: 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:33.309 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 10 monclient: shutdown 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 -- 172.21.15.181:0/3903610036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009ec50 msgr2=0x7f4410131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 --2- 172.21.15.181:0/3903610036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009ec50 0x7f4410131c60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4400009990 tx=0x7f44000091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 -- 172.21.15.181:0/3903610036 shutdown_connections 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 --2- 172.21.15.181:0/3903610036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009ec50 0x7f4410131c60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 -- 172.21.15.181:0/3903610036 >> 172.21.15.181:0/3903610036 conn(0x7f44100a5d40 msgr2=0x7f44100a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 -- 172.21.15.181:0/3903610036 shutdown_connections 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.302+0000 7f441599d640 1 -- 172.21.15.181:0/3903610036 wait complete. 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.303+0000 7f441599d640 10 monclient: build_initial_monmap 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.303+0000 7f441599d640 10 monclient: monmap: 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:33.310 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.303+0000 7f441599d640 1 Processor -- start 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.303+0000 7f441599d640 1 -- start start 2024-09-15T03:34:33.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.303+0000 7f441599d640 10 monclient: init 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f441009ec50 0x7f441009a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): picked mon.noname-c con 0x7f441009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 0x7f4410130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): picked mon.noname-a con 0x7f441009f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44101321a0 0x7f4410131120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): picked mon.noname-b con 0x7f44101321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:33.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441009df50 con 0x7f441009f020 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441011c7c0 con 0x7f44101321a0 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441011dfe0 con 0x7f441009ec50 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f441599d640 10 monclient(hunting): authenticate will time out at 1077.785200s 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440f7fe640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44101321a0 msgr2=0x7f4410131120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44101321a0 0x7f4410131120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 0x7f4410130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 0x7f4410130be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:57532/0 (socket says 172.21.15.181:57532) 2024-09-15T03:34:33.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 -- 172.21.15.181:0/1377536515 learned_addr learned my addr 172.21.15.181:0/1377536515 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): get_auth_request con 0x7f441009f020 auth_method 0 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f441009ec50 0x7f441009a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): get_auth_request con 0x7f441009ec50 auth_method 0 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:33.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f440effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient(hunting): handle_auth_done global_id 84315 payload 824 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 -- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f441009ec50 msgr2=0x7f441009a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f441009ec50 0x7f441009a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 -- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44101321a0 msgr2=0x7f4410131120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44101321a0 0x7f4410131120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient: _finish_hunting 0 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 monclient: found mon.noname-a 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.304+0000 7f44067fc640 1 -- 172.21.15.181:0/1377536515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400009450 con 0x7f441009f020 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44067fc640 1 -- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 msgr2=0x7f4410130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:33.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44067fc640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 0x7f4410130be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43f800c540 tx=0x7f43f8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80074f0 con 0x7f441009f020 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: dump: 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:33.320 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f8005780 con 0x7f441009f020 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 5 monclient: authenticate success, global_id 84315 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 10 monclient: _renew_subs 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441009b600 con 0x7f441009f020 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8005b80 con 0x7f441009f020 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:33.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f44077fe640 10 monclient: dump: 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 10 monclient: _renew_subs 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f440dffb640 4 set_mon_vals no callback set 2024-09-15T03:34:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.307+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441011bc10 con 0x7f441009f020 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f441599d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f441599d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f441599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44100008d0 con 0x7f441009f020 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f43f801b3b0 con 0x7f441009f020 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f44077fe640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43f00386f0 0x7f43f003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(145..145 src has 106..145) v4 ==== 9003+0+0 (secure 0 0 0) 0x7f43f804c030 con 0x7f441009f020 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f440effd640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43f00386f0 0x7f43f003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:33.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.308+0000 7f440effd640 10 monclient: get_auth_request con 0x7f43f00386f0 auth_method 0 2024-09-15T03:34:33.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.311+0000 7f440effd640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43f00386f0 0x7f43f003aba0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f44100a2b00 tx=0x7f4400032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:33.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.313+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f43f8018820 con 0x7f441009f020 2024-09-15T03:34:33.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.313+0000 7f44077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:33.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.313+0000 7f44077fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:33.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.476+0000 7f441599d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T03:34:33.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.476+0000 7f441599d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T03:34:33.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.476+0000 7f441599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:33.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:33.476+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f44100a13e0 con 0x7f441009f020 2024-09-15T03:34:35.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.053+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v146) v1 ==== 119+0+0 (secure 0 0 0) 0x7f43f8017070 con 0x7f441009f020 2024-09-15T03:34:35.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.054+0000 7f44077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T03:34:35.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.054+0000 7f44077fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:35.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.161+0000 7f441599d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T03:34:35.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.161+0000 7f441599d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T03:34:35.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.161+0000 7f441599d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:35.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:35.161+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f441011be30 con 0x7f441009f020 2024-09-15T03:34:36.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.056+0000 7f44077fe640 1 -- 172.21.15.181:0/1377536515 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v147) v1 ==== 119+0+0 (secure 0 0 0) 0x7f43f8018260 con 0x7f441009f020 2024-09-15T03:34:36.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.056+0000 7f44077fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-15T03:34:36.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.056+0000 7f44077fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.060+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43f00386f0 msgr2=0x7f43f003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.060+0000 7f441599d640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43f00386f0 0x7f43f003aba0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f44100a2b00 tx=0x7f4400032000 comp rx=0 tx=0).stop 2024-09-15T03:34:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.060+0000 7f441599d640 10 monclient: shutdown 2024-09-15T03:34:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.060+0000 7f441599d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.060+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 msgr2=0x7f4410130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.060+0000 7f441599d640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 0x7f4410130be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43f800c540 tx=0x7f43f8007d30 comp rx=0 tx=0).stop 2024-09-15T03:34:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.061+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 shutdown_connections 2024-09-15T03:34:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.061+0000 7f441599d640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43f00386f0 0x7f43f003aba0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.061+0000 7f441599d640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44101321a0 0x7f4410131120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.061+0000 7f441599d640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f441009f020 0x7f4410130be0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.061+0000 7f441599d640 1 --2- 172.21.15.181:0/1377536515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f441009ec50 0x7f441009a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.061+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 >> 172.21.15.181:0/1377536515 conn(0x7f44100a5d40 msgr2=0x7f4410134c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.062+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 shutdown_connections 2024-09-15T03:34:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.062+0000 7f441599d640 1 -- 172.21.15.181:0/1377536515 wait complete. 2024-09-15T03:34:36.076 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.156+0000 7f74ffd5d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.156+0000 7f74ffd5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.156+0000 7f74ffd5d640 10 monclient: monmap: 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:36.156840+0000 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:36.156840+0000 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.157 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:36.158 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:36.158 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:36.158 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:36.158 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:36.158 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.156+0000 7f74ffd5d640 1 Processor -- start 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.156+0000 7f74ffd5d640 1 -- start start 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.156+0000 7f74ffd5d640 10 monclient: init 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f74f809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:36.160 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f74f81321a0 0x7f74f8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f74f81321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74f809f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8067420 con 0x7f74f809f020 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f809fdc0 con 0x7f74f81321a0 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f80a0510 con 0x7f74f809ec50 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74ffd5d640 10 monclient(hunting): authenticate will time out at 1080.638423s 2024-09-15T03:34:36.161 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fd2f8640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f74f81321a0 msgr2=0x7f74f8068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fd2f8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f74f81321a0 0x7f74f8068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fe2fa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f74f809f020 0x7f74f8069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:52996/0 (socket says 172.21.15.181:52996) 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fe2fa640 1 -- 172.21.15.181:0/3546839586 learned_addr learned my addr 172.21.15.181:0/3546839586 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request con 0x7f74f809ec50 auth_method 0 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.157+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_done global_id 74423 payload 274 2024-09-15T03:34:36.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 1 -- 172.21.15.181:0/3546839586 >> v2:172.21.15.181:3300/0 conn(0x7f74f81321a0 msgr2=0x7f74f8068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/3546839586 >> v2:172.21.15.181:3300/0 conn(0x7f74f81321a0 0x7f74f8068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 1 -- 172.21.15.181:0/3546839586 >> v1:172.21.15.153:6789/0 conn(0x7f74f809f020 legacy=0x7f74f8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 1 monclient: found mon.noname-b 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.158+0000 7f74fdaf9640 1 -- 172.21.15.181:0/3546839586 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80a0200 con 0x7f74f809ec50 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fdaf9640 1 -- 172.21.15.181:0/3546839586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 msgr2=0x7f74f8131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/3546839586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f8131c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009850 tx=0x7f74f40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8f9e3c5d808b44 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 1 -- 172.21.15.181:0/3546839586 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4032440 con 0x7f74f809ec50 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: dump: 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:36.167 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: _finish_auth 0 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 1 -- 172.21.15.181:0/3546839586 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f40325a0 con 0x7f74f809ec50 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 1 -- 172.21.15.181:0/3546839586 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4032940 con 0x7f74f809ec50 2024-09-15T03:34:36.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74fcaf7640 10 monclient: dump: 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74ffd5d640 5 monclient: authenticate success, global_id 74423 2024-09-15T03:34:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74ffd5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.161+0000 7f74ffd5d640 4 set_mon_vals no callback set 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 10 monclient: shutdown 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 -- 172.21.15.181:0/3546839586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 msgr2=0x7f74f8131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/3546839586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f8131c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009850 tx=0x7f74f40090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 -- 172.21.15.181:0/3546839586 shutdown_connections 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/3546839586 >> v2:172.21.15.181:3300/0 conn(0x7f74f81321a0 0x7f74f8068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/3546839586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f8131c60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 -- 172.21.15.181:0/3546839586 >> 172.21.15.181:0/3546839586 conn(0x7f74f80a5d40 msgr2=0x7f74f80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 -- 172.21.15.181:0/3546839586 shutdown_connections 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.162+0000 7f74ffd5d640 1 -- 172.21.15.181:0/3546839586 wait complete. 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient: monmap: 2024-09-15T03:34:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 Processor -- start 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 -- start start 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient: init 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:36.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f74f809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74f8130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f74f809f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74f8131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f74f81321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f809de10 con 0x7f74f809f020 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f809b6e0 con 0x7f74f81321a0 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f811dce0 con 0x7f74f809ec50 2024-09-15T03:34:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74ffd5d640 10 monclient(hunting): authenticate will time out at 1080.644846s 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74fe2fa640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 msgr2=0x7f74f8131100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.163+0000 7f74fe2fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74f8131100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74f8130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38490/0 (socket says 172.21.15.181:38490) 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 learned_addr learned my addr 172.21.15.181:0/4003749215 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request con 0x7f74f809ec50 auth_method 0 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): get_auth_request con 0x7f74f809f020 auth_method 0 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:36.173 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_done global_id 74426 payload 824 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 msgr2=0x7f74f8131100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74f8131100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 msgr2=0x7f74f8130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74f8130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:36.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 monclient: found mon.noname-c 2024-09-15T03:34:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4009390 con 0x7f74f809ec50 2024-09-15T03:34:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.164+0000 7f74fd2f8640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74f8130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:36.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 msgr2=0x7f74f809ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:36.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009360 tx=0x7f74f4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4031990 con 0x7f74f809ec50 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: dump: 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: _finish_auth 0 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f4031af0 con 0x7f74f809ec50 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74ffd5d640 5 monclient: authenticate success, global_id 74426 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74ffd5d640 10 monclient: _renew_subs 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74ffd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f811dfc0 con 0x7f74f809ec50 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4031e70 con 0x7f74f809ec50 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74edffb640 10 monclient: dump: 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:36.182 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.166+0000 7f74effff640 4 set_mon_vals no callback set 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 10 monclient: _renew_subs 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f811c5c0 con 0x7f74f809ec50 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:36.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f74f4039040 con 0x7f74f809ec50 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.167+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f80008d0 con 0x7f74f809ec50 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 msgr2=0x7f74f809ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 msgr2=0x7f74f809ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009360 tx=0x7f74f4032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74dc0386f0 0x7f74dc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009360 tx=0x7f74f4032000 comp rx=0 tx=0).stop 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:36.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:36.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 msgr2=0x7f74f809ab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:36.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74f809ec50 0x7f74f809ab20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:34:36.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 0x7f74dc03d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): picked mon.c con 0x7f74dc03b2b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74dc040070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): picked mon.b con 0x7f74f81321a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74dc042a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fd2f8640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74dc0386f0 0x7f74dc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): picked mon.a con 0x7f74f809f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fe2fa640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 msgr2=0x7f74dc040070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:34:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0433a0 con 0x7f74f809f020 2024-09-15T03:34:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0434e0 con 0x7f74f81321a0 2024-09-15T03:34:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fe2fa640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74dc040070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc043620 con 0x7f74dc03b2b0 2024-09-15T03:34:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74edffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fd2f8640 10 monclient(hunting): get_auth_request con 0x7f74dc0386f0 auth_method 0 2024-09-15T03:34:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 0x7f74dc03d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request con 0x7f74dc03b2b0 auth_method 0 2024-09-15T03:34:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:36.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:36.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:36.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74dc042a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:36.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request con 0x7f74f809f020 auth_method 0 2024-09-15T03:34:36.189 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:36.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:36.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:34:36.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient(hunting): handle_auth_done global_id 74426 payload 293 2024-09-15T03:34:36.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 msgr2=0x7f74dc040070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:34:36.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74dc040070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 msgr2=0x7f74dc042a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74dc042a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 monclient: found mon.c 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f74dc043760 con 0x7f74dc03b2b0 2024-09-15T03:34:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f4072020 con 0x7f74dc03b2b0 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 10 monclient: _finish_auth 0 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.168+0000 7f74fdaf9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.170+0000 7f74fd2f8640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74dc0386f0 0x7f74dc03aba0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f74e8003940 tx=0x7f74e80090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.170+0000 7f74fdaf9640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 msgr2=0x7f74dc03d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:36.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.170+0000 7f74fdaf9640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 0x7f74dc03d690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f74f402a630 tx=0x7f74f4071000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:36.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4054070 con 0x7f74dc03b2b0 2024-09-15T03:34:36.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:36.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:36.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74edffb640 10 monclient: dump: 2024-09-15T03:34:36.195 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:36.195 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:36.195 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.195 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:36.196 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:36.196 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:36.196 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:36.196 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:36.196 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:36.196 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f4039540 con 0x7f74dc03b2b0 2024-09-15T03:34:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.172+0000 7f74ef7fe640 4 set_mon_vals no callback set 2024-09-15T03:34:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.173+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f74f403b610 con 0x7f74dc03b2b0 2024-09-15T03:34:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.173+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 4 ==== osd_map(147..147 src has 106..147) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f74f406ccd0 con 0x7f74dc03b2b0 2024-09-15T03:34:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.177+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f74f4065370 con 0x7f74dc03b2b0 2024-09-15T03:34:36.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.177+0000 7f74edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:36.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.177+0000 7f74edffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.340+0000 7f74ffd5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T03:34:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.340+0000 7f74ffd5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T03:34:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.340+0000 7f74ffd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:36.340+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f74f811e150 con 0x7f74dc03b2b0 2024-09-15T03:34:36.652 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:34:36.653 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:34:36.653 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:34:36.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:34:36.656 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:34:36.656 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T03:34:36.656 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T03:34:36.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:34:39.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:39.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:39.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:39.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:39.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:39.164+0000 7f74ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:40.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:40.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:40.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:40.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:41.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:41.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:41.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:41.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:42.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:42.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:42.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:42.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:43.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:43.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:43.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:43.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:44.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:44.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:44.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:44.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:45.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:45.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:46.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:46.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:46.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:46.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:47.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:47.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:47.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:47.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:48.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:48.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:48.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:48.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:49.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:49.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:49.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:49.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:49.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:49.165+0000 7f74ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:50.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:50.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:50.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:50.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:51.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:51.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:51.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:51.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:51.657 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T03:34:51.658 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T03:34:51.658 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:34:51.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:34:51.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.762+0000 7f8931046640 10 monclient: get_monmap_and_config 2024-09-15T03:34:51.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.762+0000 7f8931046640 10 monclient: build_initial_monmap 2024-09-15T03:34:51.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.762+0000 7f8931046640 10 monclient: monmap: 2024-09-15T03:34:51.764 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:51.764 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:51.764 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:51.763486+0000 2024-09-15T03:34:51.764 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:51.763486+0000 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:51.765 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 1 Processor -- start 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 1 -- start start 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 10 monclient: init 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.763+0000 7f8931046640 10 monclient(hunting): picked mon.noname-a con 0x7f892c0974f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f892c09a8a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f892c133070 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0978c0 con 0x7f892c09a8a0 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c09ca20 con 0x7f892c133070 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c098ad0 con 0x7f892c0974f0 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8931046640 10 monclient(hunting): authenticate will time out at 1089.403053s 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52980/0 (socket says 172.21.15.153:52980) 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 learned_addr learned my addr 172.21.15.153:0/3686961377 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:51.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f892c0974f0 auth_method 0 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f8923fff640 1 --1- 172.21.15.153:0/3686961377 >> v1:172.21.15.153:6789/0 conn(0x7f892c09a8a0 0x7f892c09ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60630/0 (socket says 172.21.15.153:60630) 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.764+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.766+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v1:172.21.15.153:6789/0 conn(0x7f892c09a8a0 legacy=0x7f892c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.766+0000 7f892b59e640 1 -- 172.21.15.153:0/3686961377 >> v1:172.21.15.181:6789/0 conn(0x7f892c133070 legacy=0x7f892c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f892c0974f0 auth_method 0 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:51.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:51.965+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:34:52.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:52.164+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:52.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:52.164+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f892c0974f0 auth_method 0 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:52.366+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:34:53.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:53.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:53.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:53.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f892c0974f0 auth_method 0 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:53.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:53.167+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:34:54.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:54.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:54.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:54.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): tick 2024-09-15T03:34:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 monclient(hunting): continuing hunt 2024-09-15T03:34:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 msgr2=0x7f892c12ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.153:3300/0 conn(0x7f892c0974f0 0x7f892c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 >> v1:172.21.15.181:6789/0 conn(0x7f892c133070 legacy=0x7f892c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 >> v1:172.21.15.153:6789/0 conn(0x7f892c09a8a0 legacy=0x7f892c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient: _add_conns ranks=[2,4] 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): picked mon.noname-b con 0x7f890c005a00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): picked mon.noname-c con 0x7f890c008550 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c098ad0 con 0x7f890c008550 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c00b0c0 con 0x7f890c005a00 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892959a640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f890c005a00 auth_method 0 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f890c008550 auth_method 0 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.764+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f890c005a00 auth_method 0 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f890c008550 auth_method 0 2024-09-15T03:34:54.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:54.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:54.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:54.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:54.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:54.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:54.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:54.965+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:34:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:55.165+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:55.165+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:55.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f890c005a00 auth_method 0 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f890c008550 auth_method 0 2024-09-15T03:34:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:55.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:55.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:55.366+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:34:56.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:56.166+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:56.167 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:56.166+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f890c005a00 auth_method 0 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f890c008550 auth_method 0 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:34:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:34:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:34:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:34:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:56.167+0000 7f8923fff640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:34:57.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.166+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:57.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.166+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:57.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.339+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v149) v1 ==== 128+0+0 (secure 0 0 0) 0x7f74f4037050 con 0x7f74dc03b2b0 2024-09-15T03:34:57.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.339+0000 7f74edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T03:34:57.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.339+0000 7f74edffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.443+0000 7f74ffd5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T03:34:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.443+0000 7f74ffd5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T03:34:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.443+0000 7f74ffd5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:57.443+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f74f811e150 con 0x7f74dc03b2b0 2024-09-15T03:34:57.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): tick 2024-09-15T03:34:57.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 monclient(hunting): continuing hunt 2024-09-15T03:34:57.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 msgr2=0x7f890c007e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c005a00 0x7f890c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 msgr2=0x7f890c00aa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 --2- 172.21.15.153:0/3686961377 >> v2:172.21.15.181:3300/0 conn(0x7f890c008550 0x7f890c00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892b59e640 1 -- 172.21.15.153:0/3686961377 reap_dead start 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 0x7f890c00d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): picked mon.noname-b con 0x7f890c00b4c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f892c09a8a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:57.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f892c133070 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:57.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c098db0 con 0x7f892c09a8a0 2024-09-15T03:34:57.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c007e80 con 0x7f892c133070 2024-09-15T03:34:57.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:57.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 1 -- 172.21.15.153:0/3686961377 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c098ad0 con 0x7f890c00b4c0 2024-09-15T03:34:57.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892959a640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:57.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 0x7f890c00d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:57.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f890c00b4c0 auth_method 0 2024-09-15T03:34:57.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.764+0000 7f892ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 10 monclient(hunting): handle_auth_done global_id 74429 payload 274 2024-09-15T03:34:57.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v1:172.21.15.181:6789/0 conn(0x7f892c133070 legacy=0x7f890c0115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:57.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> v1:172.21.15.153:6789/0 conn(0x7f892c09a8a0 legacy=0x7f890c00dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:57.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:34:57.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 1 monclient: found mon.noname-b 2024-09-15T03:34:57.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:57.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.765+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c008060 con 0x7f890c00b4c0 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892ad9d640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 msgr2=0x7f890c00d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892ad9d640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 0x7f890c00d8e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f892c09d040 tx=0x7f891c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad498374b5875c1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 1 -- 172.21.15.153:0/3686961377 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c031750 con 0x7f890c00b4c0 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: dump: 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:57.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: _finish_auth 0 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 1 -- 172.21.15.153:0/3686961377 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c0318f0 con 0x7f890c00b4c0 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:57.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 1 -- 172.21.15.153:0/3686961377 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c031cb0 con 0x7f890c00b4c0 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f8931046640 5 monclient: authenticate success, global_id 74429 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f8931046640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f8931046640 4 set_mon_vals no callback set 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.768+0000 7f892a59c640 10 monclient: dump: 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:57.781 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 10 monclient: shutdown 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 -- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 msgr2=0x7f890c00d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 0x7f890c00d8e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f892c09d040 tx=0x7f891c008040 comp rx=0 tx=0).stop 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 -- 172.21.15.153:0/3686961377 shutdown_connections 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 --2- 172.21.15.153:0/3686961377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f890c00b4c0 0x7f890c00d8e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 -- 172.21.15.153:0/3686961377 >> 172.21.15.153:0/3686961377 conn(0x7f892c0672b0 msgr2=0x7f892c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 -- 172.21.15.153:0/3686961377 shutdown_connections 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 1 -- 172.21.15.153:0/3686961377 wait complete. 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 10 monclient: build_initial_monmap 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.769+0000 7f8931046640 10 monclient: monmap: 2024-09-15T03:34:57.782 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 Processor -- start 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 -- start start 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient: init 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:57.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f890c00b4c0 0x7f892c12bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): picked mon.noname-a con 0x7f890c00b4c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 0x7f892c12c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): picked mon.noname-c con 0x7f892c09a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892c133070 0x7f892c12c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): picked mon.noname-b con 0x7f892c133070 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c132c30 con 0x7f890c00b4c0 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c12a4f0 con 0x7f892c133070 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c12d4b0 con 0x7f892c09a8a0 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:57.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f8931046640 10 monclient(hunting): authenticate will time out at 1095.409871s 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.770+0000 7f892ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f890c00b4c0 0x7f892c12bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 0x7f892c12c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 0x7f892c12c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34920/0 (socket says 172.21.15.153:34920) 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 -- 172.21.15.153:0/1883493115 learned_addr learned my addr 172.21.15.153:0/1883493115 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f892c09a8a0 auth_method 0 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892c133070 0x7f892c12c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): get_auth_request con 0x7f890c00b4c0 auth_method 0 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): get_auth_request con 0x7f892c133070 auth_method 0 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:57.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f892b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient(hunting): handle_auth_done global_id 74432 payload 824 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 -- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892c133070 msgr2=0x7f892c12c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892c133070 0x7f892c12c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 -- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f890c00b4c0 msgr2=0x7f892c12bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f890c00b4c0 0x7f892c12bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient: _finish_hunting 0 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 monclient: found mon.noname-c 2024-09-15T03:34:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.771+0000 7f8923fff640 1 -- 172.21.15.153:0/1883493115 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c011890 con 0x7f892c09a8a0 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.773+0000 7f8923fff640 1 -- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 msgr2=0x7f892c12c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.773+0000 7f8923fff640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 0x7f892c12c090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8914005c40 tx=0x7f8914009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891401c070 con 0x7f892c09a8a0 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: dump: 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: _finish_auth 0 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89140165d0 con 0x7f892c09a8a0 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914016970 con 0x7f892c09a8a0 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 5 monclient: authenticate success, global_id 74432 2024-09-15T03:34:57.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8922ffd640 10 monclient: dump: 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 10 monclient: _renew_subs 2024-09-15T03:34:57.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c12f290 con 0x7f892c09a8a0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8929d9b640 4 set_mon_vals no callback set 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 10 monclient: _renew_subs 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.774+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c12e050 con 0x7f892c09a8a0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8914016d20 con 0x7f892c09a8a0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8931046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8931046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8931046640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c04bf60 con 0x7f892c09a8a0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8922ffd640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f890c03bf50 0x7f890c03e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.775+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(149..149 src has 106..149) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f891404d040 con 0x7f892c09a8a0 2024-09-15T03:34:57.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.776+0000 7f892ad9d640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f890c03bf50 0x7f890c03e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:57.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.776+0000 7f892ad9d640 10 monclient: get_auth_request con 0x7f890c03bf50 auth_method 0 2024-09-15T03:34:57.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.778+0000 7f892ad9d640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f890c03bf50 0x7f890c03e400 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f892c071470 tx=0x7f891c037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:57.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.780+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f891407ee40 con 0x7f892c09a8a0 2024-09-15T03:34:57.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.780+0000 7f8922ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:57.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.780+0000 7f8922ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:57.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.998+0000 7f8931046640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:34:57.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.998+0000 7f8931046640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:34:57.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.998+0000 7f8931046640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:57.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.998+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f892c0638d0 con 0x7f892c09a8a0 2024-09-15T03:34:58.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.999+0000 7f8922ffd640 1 -- 172.21.15.153:0/1883493115 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f891401faf0 con 0x7f892c09a8a0 2024-09-15T03:34:58.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.999+0000 7f8922ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:34:58.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:57.999+0000 7f8922ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:58.000 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:34:58.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f890c03bf50 msgr2=0x7f890c03e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f890c03bf50 0x7f890c03e400 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f892c071470 tx=0x7f891c037000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 10 monclient: shutdown 2024-09-15T03:34:58.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 msgr2=0x7f892c12c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 0x7f892c12c090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8914005c40 tx=0x7f8914009040 comp rx=0 tx=0).stop 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 shutdown_connections 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f890c03bf50 0x7f890c03e400 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f892c133070 0x7f892c12c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f892c09a8a0 0x7f892c12c090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 --2- 172.21.15.153:0/1883493115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f890c00b4c0 0x7f892c12bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 >> 172.21.15.153:0/1883493115 conn(0x7f892c0672b0 msgr2=0x7f892c0a04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 shutdown_connections 2024-09-15T03:34:58.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.001+0000 7f8931046640 1 -- 172.21.15.153:0/1883493115 wait complete. 2024-09-15T03:34:58.012 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:34:58.013 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:34:58.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:34:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.107+0000 7fe82795d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.107+0000 7fe82795d640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.107+0000 7fe82795d640 10 monclient: monmap: 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:58.107944+0000 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:58.107944+0000 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:58.109 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:58.110 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.107+0000 7fe82795d640 1 Processor -- start 2024-09-15T03:34:58.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.107+0000 7fe82795d640 1 -- start start 2024-09-15T03:34:58.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.107+0000 7fe82795d640 10 monclient: init 2024-09-15T03:34:58.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 0x7fe820131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): picked mon.noname-b con 0x7fe820131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe82009e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 0x7fe820138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): picked mon.noname-a con 0x7fe820136bb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82009faf0 con 0x7fe82009e910 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82009fd30 con 0x7fe820136bb0 2024-09-15T03:34:58.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8200a1110 con 0x7fe820131ab0 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe82795d640 10 monclient(hunting): authenticate will time out at 1095.747433s 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 0x7fe820131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 0x7fe820138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 0x7fe820131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34922/0 (socket says 172.21.15.153:34922) 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 0x7fe820138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46480/0 (socket says 172.21.15.153:46480) 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 -- 172.21.15.153:0/1660964027 learned_addr learned my addr 172.21.15.153:0/1660964027 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe824ef8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe82009e910 0x7fe82009ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33472/0 (socket says 172.21.15.153:33472) 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): get_auth_request con 0x7fe820131ab0 auth_method 0 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): get_auth_request con 0x7fe820136bb0 auth_method 0 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient(hunting): handle_auth_done global_id 74435 payload 274 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 -- 172.21.15.153:0/1660964027 >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 msgr2=0x7fe820138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 --2- 172.21.15.153:0/1660964027 >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 0x7fe820138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 -- 172.21.15.153:0/1660964027 >> v1:172.21.15.153:6789/0 conn(0x7fe82009e910 legacy=0x7fe82009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 monclient: found mon.noname-b 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe8256f9640 1 -- 172.21.15.153:0/1660964027 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8200a13f0 con 0x7fe820131ab0 2024-09-15T03:34:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.108+0000 7fe825efa640 1 --2- 172.21.15.153:0/1660964027 >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 0x7fe820138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe8256f9640 1 -- 172.21.15.153:0/1660964027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 msgr2=0x7fe820131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe8256f9640 1 --2- 172.21.15.153:0/1660964027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 0x7fe820131e80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe814009910 tx=0x7fe8140091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6682a0927a21d9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 1 -- 172.21.15.153:0/1660964027 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe814032440 con 0x7fe820131ab0 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: dump: 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 1 -- 172.21.15.153:0/1660964027 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8140325a0 con 0x7fe820131ab0 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe82795d640 5 monclient: authenticate success, global_id 74435 2024-09-15T03:34:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe82795d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe82795d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 1 -- 172.21.15.153:0/1660964027 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe814032940 con 0x7fe820131ab0 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe82795d640 4 set_mon_vals no callback set 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.111+0000 7fe813fff640 10 monclient: dump: 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 10 monclient: shutdown 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 -- 172.21.15.153:0/1660964027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 msgr2=0x7fe820131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 --2- 172.21.15.153:0/1660964027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 0x7fe820131e80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe814009910 tx=0x7fe8140091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 -- 172.21.15.153:0/1660964027 shutdown_connections 2024-09-15T03:34:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 --2- 172.21.15.153:0/1660964027 >> v2:172.21.15.153:3300/0 conn(0x7fe820136bb0 0x7fe820138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 --2- 172.21.15.153:0/1660964027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe820131ab0 0x7fe820131e80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 -- 172.21.15.153:0/1660964027 >> 172.21.15.153:0/1660964027 conn(0x7fe820067d30 msgr2=0x7fe8200a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 -- 172.21.15.153:0/1660964027 shutdown_connections 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 1 -- 172.21.15.153:0/1660964027 wait complete. 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.112+0000 7fe82795d640 10 monclient: monmap: 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:58.132 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:58.133 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.133 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.133 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:58.133 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:58.133 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:58.133 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 Processor -- start 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 -- start start 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient: init 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 0x7fe820097c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): picked mon.noname-c con 0x7fe82009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe820131ab0 0x7fe8200981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): picked mon.noname-a con 0x7fe820131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe820136bb0 0x7fe8200986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): picked mon.noname-b con 0x7fe820136bb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82009aa60 con 0x7fe820131ab0 2024-09-15T03:34:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe820131650 con 0x7fe820136bb0 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82009d900 con 0x7fe82009e910 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe82795d640 10 monclient(hunting): authenticate will time out at 1095.752890s 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe8256f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 0x7fe820097c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe824ef8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe820131ab0 0x7fe8200981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe8256f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 0x7fe820097c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34932/0 (socket says 172.21.15.153:34932) 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.113+0000 7fe8256f9640 1 -- 172.21.15.153:0/2571011146 learned_addr learned my addr 172.21.15.153:0/2571011146 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): get_auth_request con 0x7fe82009e910 auth_method 0 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): get_auth_request con 0x7fe820131ab0 auth_method 0 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient(hunting): handle_auth_done global_id 74438 payload 824 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe820136bb0 msgr2=0x7fe8200986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe820136bb0 0x7fe8200986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe820131ab0 msgr2=0x7fe8200981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe820131ab0 0x7fe8200981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 1 monclient: found mon.noname-c 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe8256f9640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe818009a20 con 0x7fe82009e910 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.114+0000 7fe824ef8640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe820131ab0 0x7fe8200981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8256f9640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 msgr2=0x7fe820097c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8256f9640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 0x7fe820097c60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe814035040 tx=0x7fe814032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8140319c0 con 0x7fe82009e910 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: dump: 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe814031b40 con 0x7fe82009e910 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 5 monclient: authenticate success, global_id 74438 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 10 monclient: _renew_subs 2024-09-15T03:34:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe814009450 con 0x7fe82009e910 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81403d440 con 0x7fe82009e910 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8117fa640 10 monclient: dump: 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe8137fe640 4 set_mon_vals no callback set 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 10 monclient: _renew_subs 2024-09-15T03:34:58.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.116+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82009b720 con 0x7fe82009e910 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe814039040 con 0x7fe82009e910 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe82795d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe82795d640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 0x7fe820098c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe820063740 con 0x7fe82002aa40 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 0x7fe820098c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient: get_auth_request con 0x7fe82002aa40 auth_method 0 2024-09-15T03:34:58.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe8117fa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe804038740 0x7fe80403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(149..149 src has 106..149) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fe81406c8a0 con 0x7fe82009e910 2024-09-15T03:34:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 10 monclient(hunting): handle_auth_done global_id 84330 payload 824 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.117+0000 7fe824ef8640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe82002ae10 con 0x7fe82002aa40 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.118+0000 7fe825efa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe804038740 0x7fe80403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.118+0000 7fe825efa640 10 monclient: get_auth_request con 0x7fe804038740 auth_method 0 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe824ef8640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 msgr2=0x7fe820098c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe824ef8640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 0x7fe820098c20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0050f0 tx=0x7fe81c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c00fd50 con 0x7fe82002aa40 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 10 monclient: dump: 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe81c00feb0 con 0x7fe82002aa40 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 msgr2=0x7fe820098c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 0x7fe820098c20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0050f0 tx=0x7fe81c007e40 comp rx=0 tx=0).stop 2024-09-15T03:34:58.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 msgr2=0x7fe820098c20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:58.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe8117fa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82002aa40 0x7fe820098c20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.120+0000 7fe825efa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe804038740 0x7fe80403abf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe820066040 tx=0x7fe81800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe82795d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:34:58.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe82795d640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 0x7fe82005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe82795d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82009aed0 con 0x7fe82005e310 2024-09-15T03:34:58.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 0x7fe82005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient: get_auth_request con 0x7fe82005e310 auth_method 0 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 10 monclient(hunting): handle_auth_done global_id 84333 payload 824 2024-09-15T03:34:58.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.126+0000 7fe824ef8640 1 -- 172.21.15.153:0/2571011146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe82009b910 con 0x7fe82005e310 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.128+0000 7fe824ef8640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 msgr2=0x7fe82005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.128+0000 7fe824ef8640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 0x7fe82005e710 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0050f0 tx=0x7fe81c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.128+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c01caa0 con 0x7fe82005e310 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.128+0000 7fe8117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.128+0000 7fe8117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.128+0000 7fe8117fa640 10 monclient: dump: 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.152 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fe81c00fb40 con 0x7fe82005e310 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 msgr2=0x7fe82005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 0x7fe82005e710 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0050f0 tx=0x7fe81c015000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 msgr2=0x7fe82005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:58.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe8117fa640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe82005e310 0x7fe82005e710 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0050f0 tx=0x7fe81c015000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 68, 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:34:58.154 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.155 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:34:58.156 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.158 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.158 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.158 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.158 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:34:58.158 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:34:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:34:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:34:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:34:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:34:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:34:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.167 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:58.168 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:34:58.174 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:34:58.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe804038740 msgr2=0x7fe80403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe804038740 0x7fe80403abf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe820066040 tx=0x7fe81800b040 comp rx=0 tx=0).stop 2024-09-15T03:34:58.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 10 monclient: shutdown 2024-09-15T03:34:58.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 msgr2=0x7fe820097c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 --2- 172.21.15.153:0/2571011146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe82009e910 0x7fe820097c60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe814035040 tx=0x7fe814032000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe825efa640 1 -- 172.21.15.153:0/2571011146 reap_dead start 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe825efa640 1 -- 172.21.15.153:0/2571011146 reap_dead start 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 shutdown_connections 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 >> 172.21.15.153:0/2571011146 conn(0x7fe820067d30 msgr2=0x7fe8200a3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 shutdown_connections 2024-09-15T03:34:58.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.129+0000 7fe82795d640 1 -- 172.21.15.153:0/2571011146 wait complete. 2024-09-15T03:34:58.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.166+0000 7f74ecff9640 10 monclient: tick 2024-09-15T03:34:58.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.166+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:34:58.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:34:58.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:34:58.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.273+0000 7f2a0fd9d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.273+0000 7f2a0fd9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.273+0000 7f2a0fd9d640 10 monclient: monmap: 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:58.274569+0000 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:58.274569+0000 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:58.275 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:58.276 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:58.276 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:58.276 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.274+0000 7f2a0fd9d640 1 Processor -- start 2024-09-15T03:34:58.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.274+0000 7f2a0fd9d640 1 -- start start 2024-09-15T03:34:58.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.274+0000 7f2a0fd9d640 10 monclient: init 2024-09-15T03:34:58.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a0809c860 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2a0809e840 0x7f2a0809aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0809e840 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 0x7f2a08139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a08136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0809ec40 con 0x7f2a0809c860 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0809a9d0 con 0x7f2a0809e840 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0806a320 con 0x7f2a08136cb0 2024-09-15T03:34:58.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0fd9d640 10 monclient(hunting): authenticate will time out at 1095.914168s 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0db39640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2a0809c860 0x7f2a0809cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33484/0 (socket says 172.21.15.153:33484) 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0db39640 1 -- 172.21.15.153:0/1558346282 learned_addr learned my addr 172.21.15.153:0/1558346282 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 1 --2- 172.21.15.153:0/1558346282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 0x7f2a08139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): get_auth_request con 0x7f2a08136cb0 auth_method 0 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient(hunting): handle_auth_done global_id 74441 payload 274 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0d338640 1 --2- 172.21.15.153:0/1558346282 >> v2:172.21.15.181:3300/0 conn(0x7f2a0809e840 0x7f2a0809aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 1 -- 172.21.15.153:0/1558346282 >> v2:172.21.15.181:3300/0 conn(0x7f2a0809e840 msgr2=0x7f2a0809aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 1 --2- 172.21.15.153:0/1558346282 >> v2:172.21.15.181:3300/0 conn(0x7f2a0809e840 0x7f2a0809aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 1 -- 172.21.15.153:0/1558346282 >> v1:172.21.15.153:6789/0 conn(0x7f2a0809c860 legacy=0x7f2a0809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 1 monclient: found mon.noname-b 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.275+0000 7f2a0e33a640 1 -- 172.21.15.153:0/1558346282 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08068a20 con 0x7f2a08136cb0 2024-09-15T03:34:58.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0e33a640 1 -- 172.21.15.153:0/1558346282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 msgr2=0x7f2a08139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0e33a640 1 --2- 172.21.15.153:0/1558346282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 0x7f2a08139090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009e60 tx=0x7f2a00009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ca0fbee0774f1ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 1 -- 172.21.15.153:0/1558346282 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007840 con 0x7f2a08136cb0 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: dump: 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.290 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 1 -- 172.21.15.153:0/1558346282 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0002d050 con 0x7f2a08136cb0 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0fd9d640 5 monclient: authenticate success, global_id 74441 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0fd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0fd9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:58.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 1 -- 172.21.15.153:0/1558346282 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007c10 con 0x7f2a08136cb0 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0fd9d640 4 set_mon_vals no callback set 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.278+0000 7f2a0cb37640 10 monclient: dump: 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.292 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 10 monclient: shutdown 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/1558346282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 msgr2=0x7f2a08139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/1558346282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 0x7f2a08139090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009e60 tx=0x7f2a00009780 comp rx=0 tx=0).stop 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/1558346282 shutdown_connections 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/1558346282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08136cb0 0x7f2a08139090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/1558346282 >> v2:172.21.15.181:3300/0 conn(0x7f2a0809e840 0x7f2a0809aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/1558346282 >> 172.21.15.153:0/1558346282 conn(0x7f2a080a2e40 msgr2=0x7f2a080a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/1558346282 shutdown_connections 2024-09-15T03:34:58.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/1558346282 wait complete. 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 10 monclient: monmap: 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.279+0000 7f2a0fd9d640 1 Processor -- start 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 -- start start 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient: init 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 0x7f2a08098a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0809c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0809e840 0x7f2a08132f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0809e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a08136cb0 0x7f2a08133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a08136cb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08099cc0 con 0x7f2a0809c860 2024-09-15T03:34:58.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0809e0c0 con 0x7f2a08136cb0 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08135620 con 0x7f2a0809e840 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0fd9d640 10 monclient(hunting): authenticate will time out at 1095.919454s 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 0x7f2a08098a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0809e840 0x7f2a08132f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 0x7f2a08098a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46532/0 (socket says 172.21.15.153:46532) 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 -- 172.21.15.153:0/2604478599 learned_addr learned my addr 172.21.15.153:0/2604478599 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): get_auth_request con 0x7f2a0809c860 auth_method 0 2024-09-15T03:34:58.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): get_auth_request con 0x7f2a0809e840 auth_method 0 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0e33a640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a08136cb0 0x7f2a08133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0e33a640 10 monclient(hunting): get_auth_request con 0x7f2a08136cb0 auth_method 0 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0e33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0e33a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0e33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient(hunting): handle_auth_done global_id 84336 payload 824 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0809e840 msgr2=0x7f2a08132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0809e840 0x7f2a08132f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a08136cb0 msgr2=0x7f2a08133480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a08136cb0 0x7f2a08133480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 monclient: found mon.noname-a 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0db39640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a00009a20 con 0x7f2a0809c860 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.280+0000 7f2a0d338640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0809e840 0x7f2a08132f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0db39640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 msgr2=0x7f2a08098a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0db39640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 0x7f2a08098a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400c5e0 tx=0x7f2a04004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04005cb0 con 0x7f2a0809c860 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: dump: 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.299 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a04005e30 con 0x7f2a0809c860 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 5 monclient: authenticate success, global_id 84336 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 10 monclient: _renew_subs 2024-09-15T03:34:58.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a08135900 con 0x7f2a0809c860 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04007340 con 0x7f2a0809c860 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fdffb640 10 monclient: dump: 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.301 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f29fffff640 4 set_mon_vals no callback set 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 10 monclient: _renew_subs 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.283+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a08136370 con 0x7f2a0809c860 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0fd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 0x7f2a081339c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08065e00 con 0x7f2a08134730 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2a0401b400 con 0x7f2a0809c860 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 0x7f2a081339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient: get_auth_request con 0x7f2a08134730 auth_method 0 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f29fdffb640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29ec038740 0x7f29ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 10 monclient(hunting): handle_auth_done global_id 74447 payload 824 2024-09-15T03:34:58.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0d338640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a0804bf60 con 0x7f2a08134730 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(149..149 src has 106..149) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f2a0404c7b0 con 0x7f2a0809c860 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0e33a640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29ec038740 0x7f29ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.284+0000 7f2a0e33a640 10 monclient: get_auth_request con 0x7f29ec038740 auth_method 0 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.286+0000 7f2a0d338640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 msgr2=0x7f2a081339c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.286+0000 7f2a0d338640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 0x7f2a081339c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f29f800fca0 tx=0x7f29f8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.286+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800fee0 con 0x7f2a08134730 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 10 monclient: dump: 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f29f800c040 con 0x7f2a08134730 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 msgr2=0x7f2a081339c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 0x7f2a081339c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f29f800fca0 tx=0x7f29f8008040 comp rx=0 tx=0).stop 2024-09-15T03:34:58.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 msgr2=0x7f2a081339c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:58.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f29fdffb640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a08134730 0x7f2a081339c0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f29f800fca0 tx=0x7f29f8008040 comp rx=0 tx=0).stop 2024-09-15T03:34:58.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.287+0000 7f2a0e33a640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29ec038740 0x7f29ec03abf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2a00034000 tx=0x7f2a00037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.292+0000 7f2a0fd9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.292+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 0x7f2a0812fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.292+0000 7f2a0fd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.292+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08135b40 con 0x7f2a0812fad0 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 0x7f2a0812fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient: get_auth_request con 0x7f2a0812fad0 auth_method 0 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 10 monclient(hunting): handle_auth_done global_id 74450 payload 824 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.293+0000 7f2a0d338640 1 -- 172.21.15.153:0/2604478599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a0809e620 con 0x7f2a0812fad0 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f2a0d338640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 msgr2=0x7f2a0812fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f2a0d338640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 0x7f2a0812fed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29f801a000 tx=0x7f29f801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8023040 con 0x7f2a0812fad0 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 10 monclient: dump: 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f29f80123c0 con 0x7f2a0812fad0 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 msgr2=0x7f2a0812fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 0x7f2a0812fed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29f801a000 tx=0x7f29f801d000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 msgr2=0x7f2a0812fed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:58.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.295+0000 7f29fdffb640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a0812fad0 0x7f2a0812fed0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29f801a000 tx=0x7f29f801d000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 68, 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:34:58.312 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.313 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:34:58.315 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:34:58.316 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.317 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.320 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.321 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.323 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:34:58.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.326 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:34:58.326 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:34:58.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29ec038740 msgr2=0x7f29ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29ec038740 0x7f29ec03abf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2a00034000 tx=0x7f2a00037000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 10 monclient: shutdown 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 msgr2=0x7f2a08098a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 --2- 172.21.15.153:0/2604478599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a0809c860 0x7f2a08098a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400c5e0 tx=0x7f2a04004960 comp rx=0 tx=0).stop 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0e33a640 1 -- 172.21.15.153:0/2604478599 reap_dead start 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0e33a640 1 -- 172.21.15.153:0/2604478599 reap_dead start 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 shutdown_connections 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 >> 172.21.15.153:0/2604478599 conn(0x7f2a080a2e40 msgr2=0x7f2a080a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 shutdown_connections 2024-09-15T03:34:58.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.296+0000 7f2a0fd9d640 1 -- 172.21.15.153:0/2604478599 wait complete. 2024-09-15T03:34:58.329 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:34:58.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:34:58.329 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:34:58.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.343+0000 7f74edffb640 1 -- 172.21.15.181:0/4003749215 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v150) v1 ==== 128+0+0 (secure 0 0 0) 0x7f74f403f0c0 con 0x7f74dc03b2b0 2024-09-15T03:34:58.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.343+0000 7f74edffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-15T03:34:58.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.343+0000 7f74edffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:34:58.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74dc0386f0 msgr2=0x7f74dc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74dc0386f0 0x7f74dc03aba0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f74e8003940 tx=0x7f74e80090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 10 monclient: shutdown 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 msgr2=0x7f74dc03d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 0x7f74dc03d690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f74f402a630 tx=0x7f74f4071000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 shutdown_connections 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74f809f020 0x7f74dc042a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f81321a0 0x7f74dc040070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74dc03b2b0 0x7f74dc03d690 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 --2- 172.21.15.181:0/4003749215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74dc0386f0 0x7f74dc03aba0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 >> 172.21.15.181:0/4003749215 conn(0x7f74f80a5d40 msgr2=0x7f74f809e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 shutdown_connections 2024-09-15T03:34:58.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.346+0000 7f74ffd5d640 1 -- 172.21.15.181:0/4003749215 wait complete. 2024-09-15T03:34:58.359 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.423+0000 7f09cde6f640 10 monclient: get_monmap_and_config 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.423+0000 7f09cde6f640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.423+0000 7f09cde6f640 10 monclient: monmap: 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:58.423820+0000 2024-09-15T03:34:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:58.423820+0000 2024-09-15T03:34:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:58.426 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.423+0000 7f09cde6f640 1 Processor -- start 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.423+0000 7f09cde6f640 1 -- start start 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.423+0000 7f09cde6f640 10 monclient: init 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09c81318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 0x7f09c809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): picked mon.noname-b con 0x7f09c809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f09c8136c20 0x7f09c8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): picked mon.noname-a con 0x7f09c8136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c8132640 con 0x7f09c81318f0 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c8131d00 con 0x7f09c8136c20 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c809fbf0 con 0x7f09c809ea50 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09cde6f640 10 monclient(hunting): authenticate will time out at 1096.063392s 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f09c81318f0 0x7f09c8131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33498/0 (socket says 172.21.15.153:33498) 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 0x7f09c809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c77fe640 1 -- 172.21.15.153:0/2309089586 learned_addr learned my addr 172.21.15.153:0/2309089586 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09c809ea50 auth_method 0 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 1 --2- 172.21.15.153:0/2309089586 >> v2:172.21.15.153:3300/0 conn(0x7f09c8136c20 0x7f09c8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): get_auth_request con 0x7f09c8136c20 auth_method 0 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_done global_id 74453 payload 274 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 1 -- 172.21.15.153:0/2309089586 >> v2:172.21.15.153:3300/0 conn(0x7f09c8136c20 msgr2=0x7f09c8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 1 --2- 172.21.15.153:0/2309089586 >> v2:172.21.15.153:3300/0 conn(0x7f09c8136c20 0x7f09c8139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 1 -- 172.21.15.153:0/2309089586 >> v1:172.21.15.153:6789/0 conn(0x7f09c81318f0 legacy=0x7f09c8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 1 monclient: found mon.noname-b 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c6ffd640 1 -- 172.21.15.153:0/2309089586 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c809fe30 con 0x7f09c809ea50 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.424+0000 7f09c7fff640 1 --2- 172.21.15.153:0/2309089586 >> v2:172.21.15.153:3300/0 conn(0x7f09c8136c20 0x7f09c8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c6ffd640 1 -- 172.21.15.153:0/2309089586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 msgr2=0x7f09c809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c6ffd640 1 --2- 172.21.15.153:0/2309089586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 0x7f09c809ee20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009850 tx=0x7f09bc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a98ba9af7a63fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 1 -- 172.21.15.153:0/2309089586 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc02adf0 con 0x7f09c809ea50 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: dump: 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 1 -- 172.21.15.153:0/2309089586 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc032440 con 0x7f09c809ea50 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09cde6f640 5 monclient: authenticate success, global_id 74453 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09cde6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09cde6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 1 -- 172.21.15.153:0/2309089586 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0327e0 con 0x7f09c809ea50 2024-09-15T03:34:58.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09cde6f640 4 set_mon_vals no callback set 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.427+0000 7f09c67fc640 10 monclient: dump: 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.438 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 10 monclient: shutdown 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 -- 172.21.15.153:0/2309089586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 msgr2=0x7f09c809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 --2- 172.21.15.153:0/2309089586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 0x7f09c809ee20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009850 tx=0x7f09bc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 -- 172.21.15.153:0/2309089586 shutdown_connections 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 --2- 172.21.15.153:0/2309089586 >> v2:172.21.15.153:3300/0 conn(0x7f09c8136c20 0x7f09c8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 --2- 172.21.15.153:0/2309089586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c809ea50 0x7f09c809ee20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 -- 172.21.15.153:0/2309089586 >> 172.21.15.153:0/2309089586 conn(0x7f09c8067110 msgr2=0x7f09c8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 -- 172.21.15.153:0/2309089586 shutdown_connections 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 1 -- 172.21.15.153:0/2309089586 wait complete. 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.428+0000 7f09cde6f640 10 monclient: monmap: 2024-09-15T03:34:58.439 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 Processor -- start 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 -- start start 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient: init 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c809ea50 0x7f09c8130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): picked mon.noname-a con 0x7f09c809ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c81318f0 0x7f09c8099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): picked mon.noname-b con 0x7f09c81318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 0x7f09c8099d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): picked mon.noname-c con 0x7f09c8136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c811ca90 con 0x7f09c809ea50 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c809b8e0 con 0x7f09c81318f0 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c809bf10 con 0x7f09c8136c20 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09cde6f640 10 monclient(hunting): authenticate will time out at 1096.068731s 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c809ea50 0x7f09c8130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 0x7f09c8099d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c809ea50 0x7f09c8130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46556/0 (socket says 172.21.15.153:46556) 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 0x7f09c8099d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35008/0 (socket says 172.21.15.153:35008) 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 1 -- 172.21.15.153:0/156968077 learned_addr learned my addr 172.21.15.153:0/156968077 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09c809ea50 auth_method 0 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): get_auth_request con 0x7f09c8136c20 auth_method 0 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.429+0000 7f09c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c6ffd640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c81318f0 0x7f09c8099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09c81318f0 auth_method 0 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 10 monclient(hunting): handle_auth_done global_id 74456 payload 824 2024-09-15T03:34:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c81318f0 msgr2=0x7f09c8099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c81318f0 0x7f09c8099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c809ea50 msgr2=0x7f09c8130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c809ea50 0x7f09c8130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 1 monclient: found mon.noname-c 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c7fff640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009a20 con 0x7f09c8136c20 2024-09-15T03:34:58.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.430+0000 7f09c77fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c809ea50 0x7f09c8130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:34:58.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09c7fff640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 msgr2=0x7f09c8099d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09c7fff640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 0x7f09c8099d50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009f00 tx=0x7f09c000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0014480 con 0x7f09c8136c20 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: dump: 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.455 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c000f040 con 0x7f09c8136c20 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 5 monclient: authenticate success, global_id 74456 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 10 monclient: _renew_subs 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc009390 con 0x7f09c8136c20 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0014830 con 0x7f09c8136c20 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09af7fe640 10 monclient: dump: 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 10 monclient: _renew_subs 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09c5ffb640 4 set_mon_vals no callback set 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.432+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c8141eb0 con 0x7f09c8136c20 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09cde6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09cde6f640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 0x7f09c80298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c811ce30 con 0x7f09c809aa50 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f09c0014ba0 con 0x7f09c8136c20 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 0x7f09c80298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient: get_auth_request con 0x7f09c809aa50 auth_method 0 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09af7fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09b4038740 0x7f09b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(150..150 src has 106..150) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f09c004dbf0 con 0x7f09c8136c20 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 10 monclient(hunting): handle_auth_done global_id 94108 payload 824 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c77fe640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09c811bc60 con 0x7f09c809aa50 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c6ffd640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09b4038740 0x7f09b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.433+0000 7f09c6ffd640 10 monclient: get_auth_request con 0x7f09b4038740 auth_method 0 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09c77fe640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 msgr2=0x7f09c80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09c77fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 0x7f09c80298c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007320 tx=0x7f09b8004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b800fd50 con 0x7f09c809aa50 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 10 monclient: dump: 2024-09-15T03:34:58.459 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09c6ffd640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09b4038740 0x7f09b403abf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f09c8076600 tx=0x7f09bc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f09b800feb0 con 0x7f09c809aa50 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 msgr2=0x7f09c80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 0x7f09c80298c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007320 tx=0x7f09b8004960 comp rx=0 tx=0).stop 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 msgr2=0x7f09c80298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.436+0000 7f09af7fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c809aa50 0x7f09c80298c0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007320 tx=0x7f09b8004960 comp rx=0 tx=0).stop 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09cde6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09cde6f640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 0x7f09c8029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09cde6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c8065e00 con 0x7f09c8130ec0 2024-09-15T03:34:58.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 0x7f09c8029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient: get_auth_request con 0x7f09c8130ec0 auth_method 0 2024-09-15T03:34:58.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 10 monclient(hunting): handle_auth_done global_id 94111 payload 824 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.442+0000 7f09c77fe640 1 -- 172.21.15.153:0/156968077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09c804bf60 con 0x7f09c8130ec0 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09c77fe640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 msgr2=0x7f09c8029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09c77fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 0x7f09c8029e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007320 tx=0x7f09b8018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8021870 con 0x7f09c8130ec0 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 10 monclient: dump: 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f09b80219d0 con 0x7f09c8130ec0 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 msgr2=0x7f09c8029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 0x7f09c8029e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007320 tx=0x7f09b8018000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 msgr2=0x7f09c8029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:34:58.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09af7fe640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8130ec0 0x7f09c8029e00 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007320 tx=0x7f09b8018000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 68, 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:34:58.466 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.467 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.468 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:58.469 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:34:58.470 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:34:58.472 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.474 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:34:58.475 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:34:58.476 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:34:58.477 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09b4038740 msgr2=0x7f09b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09cde6f640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09b4038740 0x7f09b403abf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f09c8076600 tx=0x7f09bc032000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09cde6f640 10 monclient: shutdown 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09cde6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 msgr2=0x7f09c8099d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09cde6f640 1 --2- 172.21.15.153:0/156968077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c8136c20 0x7f09c8099d50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009f00 tx=0x7f09c000b040 comp rx=0 tx=0).stop 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09c7fff640 1 -- 172.21.15.153:0/156968077 reap_dead start 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.445+0000 7f09c7fff640 1 -- 172.21.15.153:0/156968077 reap_dead start 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.446+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 shutdown_connections 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.446+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 >> 172.21.15.153:0/156968077 conn(0x7f09c8067110 msgr2=0x7f09c80a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.446+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 shutdown_connections 2024-09-15T03:34:58.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.446+0000 7f09cde6f640 1 -- 172.21.15.153:0/156968077 wait complete. 2024-09-15T03:34:58.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:34:58.482 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:34:58.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:34:58.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.439+0000 7f33c3b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.439+0000 7f33c3b5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.439+0000 7f33c3b5d640 10 monclient: monmap: 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:34:58.440939+0000 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:34:58.440939+0000 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:58.485 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.439+0000 7f33c3b5d640 1 Processor -- start 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.439+0000 7f33c3b5d640 1 -- start start 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.439+0000 7f33c3b5d640 10 monclient: init 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 0x7f33bc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f33bc09ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33bc1321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f33bc09f020 0x7f33bc069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f33bc09f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc067420 con 0x7f33bc1321a0 2024-09-15T03:34:58.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33bc09fdc0 con 0x7f33bc09f020 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0a0510 con 0x7f33bc09ec50 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c3b5d640 10 monclient(hunting): authenticate will time out at 1102.922472s 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c10f8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f33bc1321a0 0x7f33bc068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:46860/0 (socket says 172.21.15.181:46860) 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c10f8640 1 -- 172.21.15.181:0/3595108634 learned_addr learned my addr 172.21.15.181:0/3595108634 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 1 --2- 172.21.15.181:0/3595108634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 0x7f33bc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 1 --2- 172.21.15.181:0/3595108634 >> v2:172.21.15.153:3300/0 conn(0x7f33bc09f020 0x7f33bc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): get_auth_request con 0x7f33bc09ec50 auth_method 0 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): get_auth_request con 0x7f33bc09f020 auth_method 0 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c20fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient(hunting): handle_auth_done global_id 74459 payload 274 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 1 -- 172.21.15.181:0/3595108634 >> v2:172.21.15.153:3300/0 conn(0x7f33bc09f020 msgr2=0x7f33bc069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 1 --2- 172.21.15.181:0/3595108634 >> v2:172.21.15.153:3300/0 conn(0x7f33bc09f020 0x7f33bc069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 1 -- 172.21.15.181:0/3595108634 >> v1:172.21.15.153:6789/0 conn(0x7f33bc1321a0 legacy=0x7f33bc068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 1 monclient: found mon.noname-b 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.440+0000 7f33c18f9640 1 -- 172.21.15.181:0/3595108634 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc0a0200 con 0x7f33bc09ec50 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.443+0000 7f33c18f9640 1 -- 172.21.15.181:0/3595108634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 msgr2=0x7f33bc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.443+0000 7f33c18f9640 1 --2- 172.21.15.181:0/3595108634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 0x7f33bc131c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009850 tx=0x7f33b80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5befde665225e07 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 1 -- 172.21.15.181:0/3595108634 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b802adf0 con 0x7f33bc09ec50 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: dump: 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:58.489 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 1 -- 172.21.15.181:0/3595108634 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b8032440 con 0x7f33bc09ec50 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 5 monclient: authenticate success, global_id 74459 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:58.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 4 set_mon_vals no callback set 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 1 -- 172.21.15.181:0/3595108634 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b80327e0 con 0x7f33bc09ec50 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c08f7640 10 monclient: dump: 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.491 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 10 monclient: shutdown 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 -- 172.21.15.181:0/3595108634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 msgr2=0x7f33bc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/3595108634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 0x7f33bc131c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009850 tx=0x7f33b80090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 -- 172.21.15.181:0/3595108634 shutdown_connections 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/3595108634 >> v2:172.21.15.153:3300/0 conn(0x7f33bc09f020 0x7f33bc069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/3595108634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09ec50 0x7f33bc131c60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 -- 172.21.15.181:0/3595108634 >> 172.21.15.181:0/3595108634 conn(0x7f33bc0a5d40 msgr2=0x7f33bc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 -- 172.21.15.181:0/3595108634 shutdown_connections 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.444+0000 7f33c3b5d640 1 -- 172.21.15.181:0/3595108634 wait complete. 2024-09-15T03:34:58.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.445+0000 7f33c3b5d640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.445+0000 7f33c3b5d640 10 monclient: monmap: 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.445+0000 7f33c3b5d640 1 Processor -- start 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.445+0000 7f33c3b5d640 1 -- start start 2024-09-15T03:34:58.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.445+0000 7f33c3b5d640 10 monclient: init 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 0x7f33bc09a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f33bc09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09f020 0x7f33bc11bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f33bc09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33bc1321a0 0x7f33bc130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f33bc1321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc11c7b0 con 0x7f33bc1321a0 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc09de10 con 0x7f33bc09ec50 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc11ce30 con 0x7f33bc09f020 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c3b5d640 10 monclient(hunting): authenticate will time out at 1102.928320s 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 0x7f33bc09a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 0x7f33bc09a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43334/0 (socket says 172.21.15.181:43334) 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 -- 172.21.15.181:0/1401129832 learned_addr learned my addr 172.21.15.181:0/1401129832 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): get_auth_request con 0x7f33bc09ec50 auth_method 0 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09f020 0x7f33bc11bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33bc1321a0 0x7f33bc130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): get_auth_request con 0x7f33bc1321a0 auth_method 0 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): get_auth_request con 0x7f33bc09f020 auth_method 0 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c20fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c10f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient(hunting): handle_auth_done global_id 94114 payload 824 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 -- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09f020 msgr2=0x7f33bc11bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09f020 0x7f33bc11bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 -- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33bc1321a0 msgr2=0x7f33bc130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33bc1321a0 0x7f33bc130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 monclient: found mon.noname-b 2024-09-15T03:34:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.446+0000 7f33c18f9640 1 -- 172.21.15.181:0/1401129832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b8009390 con 0x7f33bc09ec50 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.448+0000 7f33c18f9640 1 -- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 msgr2=0x7f33bc09a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.448+0000 7f33c18f9640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 0x7f33bc09a970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009360 tx=0x7f33b8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8031a10 con 0x7f33bc09ec50 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:34:58.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: dump: 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 5 monclient: authenticate success, global_id 94114 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b8031b70 con 0x7f33bc09ec50 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 10 monclient: _renew_subs 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc11dd20 con 0x7f33bc09ec50 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b803e450 con 0x7f33bc09ec50 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b1ffb640 10 monclient: dump: 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33b3fff640 4 set_mon_vals no callback set 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 10 monclient: _renew_subs 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.449+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc11e2e0 con 0x7f33bc09ec50 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f33b803c400 con 0x7f33bc09ec50 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33c3b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33c3b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33c3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc0008d0 con 0x7f33bc09ec50 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33b1ffb640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33a0038740 0x7f33a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(150..150 src has 106..150) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f33b806ccc0 con 0x7f33bc09ec50 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33c10f8640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33a0038740 0x7f33a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.450+0000 7f33c10f8640 10 monclient: get_auth_request con 0x7f33a0038740 auth_method 0 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.452+0000 7f33c10f8640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33a0038740 0x7f33a003abf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f33bc09b020 tx=0x7f33ac007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.455+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f33b803a900 con 0x7f33bc09ec50 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.455+0000 7f33b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.455+0000 7f33b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.595+0000 7f3155ac0640 10 monclient: get_monmap_and_config 2024-09-15T03:34:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.595+0000 7f3155ac0640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.595+0000 7f3155ac0640 10 monclient: monmap: 2024-09-15T03:34:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:58.596183+0000 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:58.596183+0000 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:58.597 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 1 Processor -- start 2024-09-15T03:34:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 1 -- start start 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient: init 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f31500a1890 0x7f315012eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f31500a1890 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31500975c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3150136c80 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315006e900 con 0x7f31500975c0 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31500a1c60 con 0x7f3150136c80 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3150098150 con 0x7f31500a1890 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f3155ac0640 10 monclient(hunting): authenticate will time out at 1096.235771s 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f31500a1890 0x7f315012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f31500975c0 0x7f3150097a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33506/0 (socket says 172.21.15.153:33506) 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314effd640 1 -- 172.21.15.153:0/232753750 learned_addr learned my addr 172.21.15.153:0/232753750 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): get_auth_request con 0x7f31500a1890 auth_method 0 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.596+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 10 monclient(hunting): handle_auth_done global_id 84351 payload 274 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 1 -- 172.21.15.153:0/232753750 >> v1:172.21.15.181:6789/0 conn(0x7f3150136c80 legacy=0x7f3150139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 1 -- 172.21.15.153:0/232753750 >> v1:172.21.15.153:6789/0 conn(0x7f31500975c0 legacy=0x7f3150097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:34:58.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.597+0000 7f314f7fe640 1 -- 172.21.15.153:0/232753750 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31500641d0 con 0x7f31500a1890 2024-09-15T03:34:58.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.599+0000 7f314f7fe640 1 -- 172.21.15.153:0/232753750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500a1890 msgr2=0x7f315012eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.599+0000 7f314f7fe640 1 --2- 172.21.15.153:0/232753750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500a1890 0x7f315012eae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3140009850 tx=0x7f31400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b99a6841ad876d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.599+0000 7f314e7fc640 1 -- 172.21.15.153:0/232753750 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140032440 con 0x7f31500a1890 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: dump: 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.608 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 1 -- 172.21.15.153:0/232753750 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31400325a0 con 0x7f31500a1890 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 5 monclient: authenticate success, global_id 84351 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:58.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 1 -- 172.21.15.153:0/232753750 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140032940 con 0x7f31500a1890 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 4 set_mon_vals no callback set 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f314e7fc640 10 monclient: dump: 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 10 monclient: shutdown 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 -- 172.21.15.153:0/232753750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500a1890 msgr2=0x7f315012eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 --2- 172.21.15.153:0/232753750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500a1890 0x7f315012eae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3140009850 tx=0x7f31400090f0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 -- 172.21.15.153:0/232753750 shutdown_connections 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 --2- 172.21.15.153:0/232753750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500a1890 0x7f315012eae0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 -- 172.21.15.153:0/232753750 >> 172.21.15.153:0/232753750 conn(0x7f3150067b50 msgr2=0x7f31500a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 -- 172.21.15.153:0/232753750 shutdown_connections 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.600+0000 7f3155ac0640 1 -- 172.21.15.153:0/232753750 wait complete. 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.601+0000 7f3155ac0640 10 monclient: build_initial_monmap 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.601+0000 7f3155ac0640 10 monclient: monmap: 2024-09-15T03:34:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.601+0000 7f3155ac0640 1 Processor -- start 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.601+0000 7f3155ac0640 1 -- start start 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.601+0000 7f3155ac0640 10 monclient: init 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:58.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 0x7f315009a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f31500975c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31500a1890 0x7f3150126250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): picked mon.noname-b con 0x7f31500a1890 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3150136c80 0x7f3150126790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): picked mon.noname-c con 0x7f3150136c80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3150127fd0 con 0x7f31500975c0 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3150128160 con 0x7f31500a1890 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315009aee0 con 0x7f3150136c80 2024-09-15T03:34:58.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f3155ac0640 10 monclient(hunting): authenticate will time out at 1096.241091s 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 0x7f315009a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 0x7f315009a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46574/0 (socket says 172.21.15.153:46574) 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 -- 172.21.15.153:0/897285044 learned_addr learned my addr 172.21.15.153:0/897285044 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): get_auth_request con 0x7f31500975c0 auth_method 0 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3150136c80 0x7f3150126790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31500a1890 0x7f3150126250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): get_auth_request con 0x7f31500a1890 auth_method 0 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): get_auth_request con 0x7f3150136c80 auth_method 0 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient(hunting): handle_auth_done global_id 84354 payload 824 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 -- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3150136c80 msgr2=0x7f3150126790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3150136c80 0x7f3150126790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 -- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31500a1890 msgr2=0x7f3150126250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31500a1890 0x7f3150126250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.602+0000 7f314f7fe640 1 -- 172.21.15.153:0/897285044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3140009390 con 0x7f31500975c0 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f314f7fe640 1 -- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 msgr2=0x7f315009a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f314f7fe640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 0x7f315009a2c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3140009360 tx=0x7f3140032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31400318c0 con 0x7f31500975c0 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: dump: 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: _finish_auth 0 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3140031a20 con 0x7f31500975c0 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 5 monclient: authenticate success, global_id 84354 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 10 monclient: _renew_subs 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315009b2d0 con 0x7f31500975c0 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140031dc0 con 0x7f31500975c0 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f31377fe640 10 monclient: dump: 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:58.618 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 10 monclient: _renew_subs 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31501296c0 con 0x7f31500975c0 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.604+0000 7f314dffb640 4 set_mon_vals no callback set 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.605+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3140039040 con 0x7f31500975c0 2024-09-15T03:34:58.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.605+0000 7f3155ac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.605+0000 7f3155ac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.605+0000 7f3155ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.605+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3150128e40 con 0x7f31500975c0 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.605+0000 7f31377fe640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f313c038740 0x7f313c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.606+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(150..150 src has 106..150) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f314006ce90 con 0x7f31500975c0 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.606+0000 7f314effd640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f313c038740 0x7f313c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:58.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.606+0000 7f314effd640 10 monclient: get_auth_request con 0x7f313c038740 auth_method 0 2024-09-15T03:34:58.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.617+0000 7f33c3b5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T03:34:58.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.617+0000 7f33c3b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T03:34:58.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.617+0000 7f33c3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:34:58.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:58.617+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f33bc0a13e0 con 0x7f33bc09ec50 2024-09-15T03:34:58.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.608+0000 7f314effd640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f313c038740 0x7f313c03abf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f315009aa40 tx=0x7f3144006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:58.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.610+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f314003a360 con 0x7f31500975c0 2024-09-15T03:34:58.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.610+0000 7f31377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:58.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.610+0000 7f31377fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:34:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.825+0000 7f3155ac0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:34:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.825+0000 7f3155ac0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:34:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.825+0000 7f3155ac0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.825+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3150128e40 con 0x7f31500975c0 2024-09-15T03:34:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f31377fe640 1 -- 172.21.15.153:0/897285044 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f314003f070 con 0x7f31500975c0 2024-09-15T03:34:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f31377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:34:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f31377fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:58.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=7,logm=65} crc {auth=635279322,config=3140514009,health=2572758929,kv=2243821149,logm=73916999}) 2024-09-15T03:34:58.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=10,kv=8,logm=64} crc {auth=3940776089,config=3140514009,health=1228072951,kv=3235807718,logm=1283753914}) 2024-09-15T03:34:58.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=7,logm=65} crc {auth=635279322,config=3140514009,health=2572758929,kv=2243821149,logm=73916999}) 2024-09-15T03:34:58.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.826+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=7,logm=64} crc {auth=635279322,config=3140514009,health=438406894,kv=3367408631,logm=3946451445}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123677257}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912492415}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123677257}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717877629}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046731284}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757630007}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046731284}) 2024-09-15T03:34:58.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.827+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010606764}) 2024-09-15T03:34:58.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f313c038740 msgr2=0x7f313c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f313c038740 0x7f313c03abf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f315009aa40 tx=0x7f3144006f50 comp rx=0 tx=0).stop 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 10 monclient: shutdown 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 msgr2=0x7f315009a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 0x7f315009a2c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3140009360 tx=0x7f3140032000 comp rx=0 tx=0).stop 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 shutdown_connections 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f313c038740 0x7f313c03abf0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3150136c80 0x7f3150126790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31500a1890 0x7f3150126250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 --2- 172.21.15.153:0/897285044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31500975c0 0x7f315009a2c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 >> 172.21.15.153:0/897285044 conn(0x7f3150067b50 msgr2=0x7f31500955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 shutdown_connections 2024-09-15T03:34:58.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f3155ac0640 1 -- 172.21.15.153:0/897285044 wait complete. 2024-09-15T03:34:58.841 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.841 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431164111}) 2024-09-15T03:34:58.841 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=985229537}) 2024-09-15T03:34:58.841 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.841 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431164111}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.828+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502965068}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.829+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.829+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030686318}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.829+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1688976331}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.829+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.829+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030686318}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.829+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=748629422}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909656277}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733265005}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909656277}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015021780}) 2024-09-15T03:34:58.842 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67079042}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916594583}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67079042}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.830+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645702999}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.831+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.831+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192521753}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.831+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404230746}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.831+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.831+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192521753}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.831+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2408922040}) 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367006301}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45534059}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367006301}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314815409}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456672978}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879020851}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456672978}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.832+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2381366124}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.833+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.833+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784135284}) 2024-09-15T03:34:58.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.833+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778004491}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.833+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.833+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784135284}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.833+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1017641058}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.834+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.834+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753897839}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.834+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561418495}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.834+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.834+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753897839}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.834+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554753987}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.835+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.835+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593198008}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.835+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836735281}) 2024-09-15T03:34:58.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.835+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.835+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593198008}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.835+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257302779}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.836+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.836+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949699894}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.836+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956677851}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.836+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.836+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949699894}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.836+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881499065}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159947792}) 2024-09-15T03:34:58.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128506548}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159947792}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2012649730}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135594634}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3390020277}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135594634}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.837+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249689199}) 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.838+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.838+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947581948}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.838+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1245053952}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.838+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.838+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947581948}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.838+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591882931}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.839+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.839+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861469987}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.839+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550569053}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.839+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.839+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861469987}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.839+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950742339}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.840+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.840+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562232308}) 2024-09-15T03:34:58.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.840+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2516707448}) 2024-09-15T03:34:58.849 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.840+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.849 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.840+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562232308}) 2024-09-15T03:34:58.849 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.840+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610407149}) 2024-09-15T03:34:58.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:34:58.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918584983}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717998414}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918584983}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046813278}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810379232}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1556530482}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810379232}) 2024-09-15T03:34:58.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:58.841+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2886056078}) 2024-09-15T03:34:59.342 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.341+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.342 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.341+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916246095}) 2024-09-15T03:34:59.343 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.341+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3763885038}) 2024-09-15T03:34:59.343 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.341+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.343 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.341+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916246095}) 2024-09-15T03:34:59.343 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.341+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3416903630}) 2024-09-15T03:34:59.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.345+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.345+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340187005}) 2024-09-15T03:34:59.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.345+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030643425}) 2024-09-15T03:34:59.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.345+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.345+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340187005}) 2024-09-15T03:34:59.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.345+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2551771243}) 2024-09-15T03:34:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.346+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v151) v1 ==== 134+0+0 (secure 0 0 0) 0x7f33b8037070 con 0x7f33bc09ec50 2024-09-15T03:34:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.346+0000 7f33b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T03:34:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.346+0000 7f33b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:34:59.354 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.352+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.354 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.352+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553268270}) 2024-09-15T03:34:59.354 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.353+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915870164}) 2024-09-15T03:34:59.354 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.353+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.354 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.353+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553268270}) 2024-09-15T03:34:59.354 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.353+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364885181}) 2024-09-15T03:34:59.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.448+0000 7f33c3b5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T03:34:59.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.448+0000 7f33c3b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T03:34:59.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.448+0000 7f33c3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:34:59.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:34:59.448+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f33bc0a13e0 con 0x7f33bc09ec50 2024-09-15T03:34:59.661 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.660+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.661 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.660+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476304970}) 2024-09-15T03:34:59.661 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.660+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605793019}) 2024-09-15T03:34:59.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.660+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.660+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476304970}) 2024-09-15T03:34:59.665 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.660+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=932181331}) 2024-09-15T03:34:59.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.678+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.678+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=54,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=1086102033,mdsmap=1531151339,mgr=2572398688,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4113934990,mon_config_key=4117641549,monmap=626991505,osd_metadata=2347033642,osd_pg_creating=1754776702,osd_snap=454132730,osdmap=2234272626}) 2024-09-15T03:34:59.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.678+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=4} crc {logm=3862586232,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4113934990,mon_config_key=4117641549,monmap=626991505,osd_metadata=851455416,osd_pg_creating=1754776702,osd_snap=454132730,osdmap=3121881595}) 2024-09-15T03:34:59.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.678+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.678+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=54,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=1086102033,mdsmap=1531151339,mgr=2572398688,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4113934990,mon_config_key=4117641549,monmap=626991505,osd_metadata=2347033642,osd_pg_creating=1754776702,osd_snap=454132730,osdmap=2234272626}) 2024-09-15T03:34:59.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.678+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=1653953412,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4113934990,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754776702,osd_snap=454132730,osdmap=630872362}) 2024-09-15T03:34:59.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.682+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.682+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=2752897343}) 2024-09-15T03:34:59.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.682+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=76} crc {osdmap=3293954776}) 2024-09-15T03:34:59.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.682+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:34:59.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.682+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=2752897343}) 2024-09-15T03:34:59.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:34:59.682+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=3460676535}) 2024-09-15T03:34:59.850 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:34:59.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.954+0000 7fb6d04a8640 10 monclient: get_monmap_and_config 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.954+0000 7fb6d04a8640 10 monclient: build_initial_monmap 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.954+0000 7fb6d04a8640 10 monclient: monmap: 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:34:59.955283+0000 2024-09-15T03:34:59.955 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:34:59.955283+0000 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:34:59.956 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 Processor -- start 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 -- start start 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient: init 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb6c809ad90 0x7fb6c809c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c809ad90 addr v2:172.21.15.181:3300/0 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb6c809b160 0x7fb6c809b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c809b160 addr v2:172.21.15.153:3300/0 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6c8131960 addr v1:172.21.15.181:6789/0 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c80692c0 con 0x7fb6c8131960 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c809d890 con 0x7fb6c809b160 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c809db70 con 0x7fb6c809ad90 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6d04a8640 10 monclient(hunting): authenticate will time out at 1097.594834s 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb6c809b160 0x7fb6c809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb6c809b160 0x7fb6c809b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46588/0 (socket says 172.21.15.153:46588) 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 1 -- 172.21.15.153:0/3345550612 learned_addr learned my addr 172.21.15.153:0/3345550612 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:59.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 10 monclient(hunting): get_auth_request con 0x7fb6c809b160 auth_method 0 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.955+0000 7fb6cda43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 10 monclient(hunting): handle_auth_done global_id 84357 payload 274 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 1 -- 172.21.15.153:0/3345550612 >> v2:172.21.15.181:3300/0 conn(0x7fb6c809ad90 msgr2=0x7fb6c809c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 1 --2- 172.21.15.153:0/3345550612 >> v2:172.21.15.181:3300/0 conn(0x7fb6c809ad90 0x7fb6c809c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 1 -- 172.21.15.153:0/3345550612 >> v1:172.21.15.181:6789/0 conn(0x7fb6c8131960 legacy=0x7fb6c8131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 10 monclient: _finish_hunting 0 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 1 monclient: found mon.noname-a 2024-09-15T03:34:59.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:59.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:59.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.956+0000 7fb6cda43640 1 -- 172.21.15.153:0/3345550612 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c809d5b0 con 0x7fb6c809b160 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cda43640 1 -- 172.21.15.153:0/3345550612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 msgr2=0x7fb6c809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cda43640 1 --2- 172.21.15.153:0/3345550612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c809b530 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8068070 tx=0x7fb6b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725887c65b13906a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 1 -- 172.21.15.153:0/3345550612 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8032440 con 0x7fb6c809b160 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: dump: 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:59.966 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: _finish_auth 0 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 1 -- 172.21.15.153:0/3345550612 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b80325a0 con 0x7fb6c809b160 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 5 monclient: authenticate success, global_id 84357 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:34:59.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 1 -- 172.21.15.153:0/3345550612 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8032980 con 0x7fb6c809b160 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 4 set_mon_vals no callback set 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6cd242640 10 monclient: dump: 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 10 monclient: shutdown 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3345550612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 msgr2=0x7fb6c809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3345550612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c809b530 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8068070 tx=0x7fb6b80091d0 comp rx=0 tx=0).stop 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3345550612 shutdown_connections 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3345550612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c809b530 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3345550612 >> v2:172.21.15.181:3300/0 conn(0x7fb6c809ad90 0x7fb6c809c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3345550612 >> 172.21.15.153:0/3345550612 conn(0x7fb6c80678a0 msgr2=0x7fb6c80a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3345550612 shutdown_connections 2024-09-15T03:34:59.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.959+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3345550612 wait complete. 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.960+0000 7fb6d04a8640 10 monclient: build_initial_monmap 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.960+0000 7fb6d04a8640 10 monclient: monmap: 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.960+0000 7fb6d04a8640 1 Processor -- start 2024-09-15T03:34:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.960+0000 7fb6d04a8640 1 -- start start 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.960+0000 7fb6d04a8640 10 monclient: init 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c809ad90 0x7fb6c8134e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c809ad90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c8126630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c809b160 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6c8131960 0x7fb6c8126b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c8131960 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8128f00 con 0x7fb6c809b160 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:59.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8127d50 con 0x7fb6c809ad90 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8128c20 con 0x7fb6c8131960 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6d04a8640 10 monclient(hunting): authenticate will time out at 1097.600135s 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c8126630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c8126630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46596/0 (socket says 172.21.15.153:46596) 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 -- 172.21.15.153:0/3363858471 learned_addr learned my addr 172.21.15.153:0/3363858471 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): get_auth_request con 0x7fb6c809b160 auth_method 0 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6c8131960 0x7fb6c8126b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:59.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): get_auth_request con 0x7fb6c8131960 auth_method 0 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c809ad90 0x7fb6c8134e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): get_auth_request con 0x7fb6c809ad90 auth_method 0 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:59.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cea45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6ce244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient(hunting): handle_auth_done global_id 84360 payload 824 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 -- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6c8131960 msgr2=0x7fb6c8126b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6c8131960 0x7fb6c8126b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 -- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c809ad90 msgr2=0x7fb6c8134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c809ad90 0x7fb6c8134e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient: _finish_hunting 0 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 monclient: found mon.noname-a 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:34:59.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.961+0000 7fb6cda43640 1 -- 172.21.15.153:0/3363858471 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8009450 con 0x7fb6c809b160 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6cda43640 1 -- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 msgr2=0x7fb6c8126630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6cda43640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c8126630 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009210 tx=0x7fb6b8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8031990 con 0x7fb6c809b160 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 10 monclient: dump: 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:59.975 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b8031b30 con 0x7fb6c809b160 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6d04a8640 5 monclient: authenticate success, global_id 84360 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6d04a8640 10 monclient: _renew_subs 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.963+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b803b400 con 0x7fb6c809b160 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6d04a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c81291e0 con 0x7fb6c809b160 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:34:59.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6be7fc640 10 monclient: dump: 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6cca41640 4 set_mon_vals no callback set 2024-09-15T03:34:59.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6d04a8640 10 monclient: _renew_subs 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6d04a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c81359b0 con 0x7fb6c809b160 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.964+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb6b8039040 con 0x7fb6c809b160 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6d04a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6d04a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6d04a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c804bf60 con 0x7fb6c809b160 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6be7fc640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6ac038740 0x7fb6ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(151..151 src has 106..151) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fb6b806cf60 con 0x7fb6c809b160 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6ce244640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6ac038740 0x7fb6ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:34:59.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.965+0000 7fb6ce244640 10 monclient: get_auth_request con 0x7fb6ac038740 auth_method 0 2024-09-15T03:34:59.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.967+0000 7fb6ce244640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6ac038740 0x7fb6ac03abf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009b50 tx=0x7fb6c4009120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:34:59.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.969+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb6b803b7f0 con 0x7fb6c809b160 2024-09-15T03:34:59.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.970+0000 7fb6be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:34:59.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:34:59.970+0000 7fb6be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:00.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6d04a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:00.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6d04a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:00.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6d04a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb6c8129550 con 0x7fb6c809b160 2024-09-15T03:35:00.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6be7fc640 1 -- 172.21.15.153:0/3363858471 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fb6b803f030 con 0x7fb6c809b160 2024-09-15T03:35:00.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:00.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.188+0000 7fb6be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:00.190 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.190+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6ac038740 msgr2=0x7fb6ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.190+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6ac038740 0x7fb6ac03abf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009b50 tx=0x7fb6c4009120 comp rx=0 tx=0).stop 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.190+0000 7fb6d04a8640 10 monclient: shutdown 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.190+0000 7fb6d04a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.190+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 msgr2=0x7fb6c8126630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.190+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c8126630 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009210 tx=0x7fb6b8032000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 shutdown_connections 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6ac038740 0x7fb6ac03abf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6c8131960 0x7fb6c8126b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6c809b160 0x7fb6c8126630 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 --2- 172.21.15.153:0/3363858471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c809ad90 0x7fb6c8134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 >> 172.21.15.153:0/3363858471 conn(0x7fb6c80678a0 msgr2=0x7fb6c8095430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 shutdown_connections 2024-09-15T03:35:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.191+0000 7fb6d04a8640 1 -- 172.21.15.153:0/3363858471 wait complete. 2024-09-15T03:35:00.201 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:35:00.202 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:35:00.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:35:00.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.294+0000 7fdc3f754640 10 monclient: get_monmap_and_config 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.294+0000 7fdc3f754640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.294+0000 7fdc3f754640 10 monclient: monmap: 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:00.295014+0000 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:00.295014+0000 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.295 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.296 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:00.296 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:00.296 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:00.296 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:00.296 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:00.296 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.294+0000 7fdc3f754640 1 Processor -- start 2024-09-15T03:35:00.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 1 -- start start 2024-09-15T03:35:00.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient: init 2024-09-15T03:35:00.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc38131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdc3809fda0 0x7fdc3809e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3809fda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdc3809ef00 0x7fdc38138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3809ef00 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc380a1180 con 0x7fdc38131980 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc380a1460 con 0x7fdc3809ef00 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc38132460 con 0x7fdc3809fda0 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3f754640 10 monclient(hunting): authenticate will time out at 1097.934587s 2024-09-15T03:35:00.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdc3809ef00 0x7fdc38138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdc3809ef00 0x7fdc38138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46606/0 (socket says 172.21.15.153:46606) 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/229837313 learned_addr learned my addr 172.21.15.153:0/229837313 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): get_auth_request con 0x7fdc3809ef00 auth_method 0 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3ccef640 1 --2- 172.21.15.153:0/229837313 >> v2:172.21.15.181:3300/0 conn(0x7fdc3809fda0 0x7fdc3809e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_done global_id 84363 payload 274 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/229837313 >> v2:172.21.15.181:3300/0 conn(0x7fdc3809fda0 msgr2=0x7fdc3809e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 --2- 172.21.15.153:0/229837313 >> v2:172.21.15.181:3300/0 conn(0x7fdc3809fda0 0x7fdc3809e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/229837313 >> v1:172.21.15.181:6789/0 conn(0x7fdc38131980 legacy=0x7fdc38131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 monclient: found mon.noname-a 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/229837313 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc381325d0 con 0x7fdc3809ef00 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3ccef640 10 monclient: get_auth_request con 0x7fdc3809fda0 auth_method 0 2024-09-15T03:35:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.295+0000 7fdc3ccef640 1 --2- 172.21.15.153:0/229837313 >> v2:172.21.15.181:3300/0 conn(0x7fdc3809fda0 0x7fdc3809e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:35:00.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/229837313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809ef00 msgr2=0x7fdc38138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc3dcf1640 1 --2- 172.21.15.153:0/229837313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809ef00 0x7fdc38138cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc38064050 tx=0x7fdc30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975b920f9c3b1e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 1 -- 172.21.15.153:0/229837313 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30007610 con 0x7fdc3809ef00 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: dump: 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.309 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 1 -- 172.21.15.153:0/229837313 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3002d050 con 0x7fdc3809ef00 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 1 -- 172.21.15.153:0/229837313 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc300079a0 con 0x7fdc3809ef00 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc3f754640 5 monclient: authenticate success, global_id 84363 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc3f754640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc3f754640 4 set_mon_vals no callback set 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.298+0000 7fdc2ffff640 10 monclient: dump: 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 10 monclient: shutdown 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 -- 172.21.15.153:0/229837313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809ef00 msgr2=0x7fdc38138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 --2- 172.21.15.153:0/229837313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809ef00 0x7fdc38138cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc38064050 tx=0x7fdc30009840 comp rx=0 tx=0).stop 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 -- 172.21.15.153:0/229837313 shutdown_connections 2024-09-15T03:35:00.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 --2- 172.21.15.153:0/229837313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809ef00 0x7fdc38138cc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 --2- 172.21.15.153:0/229837313 >> v2:172.21.15.181:3300/0 conn(0x7fdc3809fda0 0x7fdc3809e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 -- 172.21.15.153:0/229837313 >> 172.21.15.153:0/229837313 conn(0x7fdc38067bd0 msgr2=0x7fdc380a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 -- 172.21.15.153:0/229837313 shutdown_connections 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.299+0000 7fdc3f754640 1 -- 172.21.15.153:0/229837313 wait complete. 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.300+0000 7fdc3f754640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.300+0000 7fdc3f754640 10 monclient: monmap: 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.312 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.300+0000 7fdc3f754640 1 Processor -- start 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.300+0000 7fdc3f754640 1 -- start start 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.300+0000 7fdc3f754640 10 monclient: init 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc3809ef00 0x7fdc3809b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3809ef00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc3809fda0 0x7fdc38096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3809fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 0x7fdc38096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): picked mon.noname-a con 0x7fdc38131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38099980 con 0x7fdc38131980 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380996a0 con 0x7fdc3809ef00 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38099c60 con 0x7fdc3809fda0 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3f754640 10 monclient(hunting): authenticate will time out at 1097.940113s 2024-09-15T03:35:00.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 0x7fdc38096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc3809fda0 0x7fdc38096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 0x7fdc38096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46612/0 (socket says 172.21.15.153:46612) 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/2451047220 learned_addr learned my addr 172.21.15.153:0/2451047220 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): get_auth_request con 0x7fdc38131980 auth_method 0 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): get_auth_request con 0x7fdc3809fda0 auth_method 0 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc3809ef00 0x7fdc3809b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): get_auth_request con 0x7fdc3809ef00 auth_method 0 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient(hunting): handle_auth_done global_id 84366 payload 824 2024-09-15T03:35:00.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc3809fda0 msgr2=0x7fdc38096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc3809fda0 0x7fdc38096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc3809ef00 msgr2=0x7fdc3809b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc3809ef00 0x7fdc3809b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 monclient: found mon.noname-a 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30009ae0 con 0x7fdc38131980 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.301+0000 7fdc3ccef640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc3809fda0 0x7fdc38096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 msgr2=0x7fdc38096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc3dcf1640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 0x7fdc38096fa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009880 tx=0x7fdc30036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30032ce0 con 0x7fdc38131980 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: dump: 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc30032e40 con 0x7fdc38131980 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc3f754640 5 monclient: authenticate success, global_id 84366 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc3f754640 10 monclient: _renew_subs 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc3f754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3809aad0 con 0x7fdc38131980 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3003b640 con 0x7fdc38131980 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2d7fa640 10 monclient: dump: 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.319 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.303+0000 7fdc2f7fe640 4 set_mon_vals no callback set 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 10 monclient: _renew_subs 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc380988e0 con 0x7fdc38131980 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdc30038050 con 0x7fdc38131980 2024-09-15T03:35:00.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 0x7fdc380974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38065e00 con 0x7fdc38098050 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc2d7fa640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc1c038740 0x7fdc1c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.304+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(151..151 src has 106..151) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fdc3006da50 con 0x7fdc38131980 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 0x7fdc380974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient: get_auth_request con 0x7fdc38098050 auth_method 0 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3ccef640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc1c038740 0x7fdc1c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_done global_id 84369 payload 824 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3d4f0640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc38099810 con 0x7fdc38098050 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.305+0000 7fdc3ccef640 10 monclient: get_auth_request con 0x7fdc1c038740 auth_method 0 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc3d4f0640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 msgr2=0x7fdc380974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc3d4f0640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 0x7fdc380974e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400e780 tx=0x7fdc34009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3400cac0 con 0x7fdc38098050 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 10 monclient: dump: 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.322 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc3ccef640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc1c038740 0x7fdc1c03abf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800f360 tx=0x7fdc28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdc3400cc20 con 0x7fdc38098050 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:00.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 msgr2=0x7fdc380974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 0x7fdc380974e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400e780 tx=0x7fdc34009060 comp rx=0 tx=0).stop 2024-09-15T03:35:00.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 msgr2=0x7fdc380974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:00.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.307+0000 7fdc2d7fa640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38098050 0x7fdc380974e0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400e780 tx=0x7fdc34009060 comp rx=0 tx=0).stop 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3f754640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3f754640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 0x7fdc3809e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3f754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38097c80 con 0x7fdc3809dce0 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 0x7fdc3809e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient: get_auth_request con 0x7fdc3809dce0 auth_method 0 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 10 monclient(hunting): handle_auth_done global_id 84372 payload 824 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.313+0000 7fdc3d4f0640 1 -- 172.21.15.153:0/2451047220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdc3804bf60 con 0x7fdc3809dce0 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3d4f0640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 msgr2=0x7fdc3809e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3d4f0640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 0x7fdc3809e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009fd0 tx=0x7fdc34015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340209a0 con 0x7fdc3809dce0 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 10 monclient: dump: 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.327 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fdc34020b00 con 0x7fdc3809dce0 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 msgr2=0x7fdc3809e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 0x7fdc3809e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009fd0 tx=0x7fdc34015000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 msgr2=0x7fdc3809e0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:00.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc2d7fa640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc3809dce0 0x7fdc3809e0e0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009fd0 tx=0x7fdc34015000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 68, 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:35:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:35:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:00.341 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc1c038740 msgr2=0x7fdc1c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc1c038740 0x7fdc1c03abf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800f360 tx=0x7fdc28008040 comp rx=0 tx=0).stop 2024-09-15T03:35:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 10 monclient: shutdown 2024-09-15T03:35:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 msgr2=0x7fdc38096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3dcf1640 1 -- 172.21.15.153:0/2451047220 reap_dead start 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 0x7fdc38096fa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009880 tx=0x7fdc30036000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 shutdown_connections 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 --2- 172.21.15.153:0/2451047220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc38131980 0x7fdc38096fa0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 >> 172.21.15.153:0/2451047220 conn(0x7fdc38067bd0 msgr2=0x7fdc3812caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 shutdown_connections 2024-09-15T03:35:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.316+0000 7fdc3f754640 1 -- 172.21.15.153:0/2451047220 wait complete. 2024-09-15T03:35:00.343 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:00.343 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:35:00.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:35:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.358+0000 7f33b1ffb640 1 -- 172.21.15.181:0/1401129832 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v152) v1 ==== 134+0+0 (secure 0 0 0) 0x7f33b8043030 con 0x7f33bc09ec50 2024-09-15T03:35:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.358+0000 7f33b1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-15T03:35:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.358+0000 7f33b1ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33a0038740 msgr2=0x7f33a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33a0038740 0x7f33a003abf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f33bc09b020 tx=0x7f33ac007f00 comp rx=0 tx=0).stop 2024-09-15T03:35:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 10 monclient: shutdown 2024-09-15T03:35:00.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 msgr2=0x7f33bc09a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 0x7f33bc09a970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009360 tx=0x7f33b8032000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 shutdown_connections 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f33a0038740 0x7f33a003abf0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33bc1321a0 0x7f33bc130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33bc09f020 0x7f33bc11bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 --2- 172.21.15.181:0/1401129832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33bc09ec50 0x7f33bc09a970 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 >> 172.21.15.181:0/1401129832 conn(0x7f33bc0a5d40 msgr2=0x7f33bc068c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 shutdown_connections 2024-09-15T03:35:00.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.361+0000 7f33c3b5d640 1 -- 172.21.15.181:0/1401129832 wait complete. 2024-09-15T03:35:00.373 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.438+0000 7f274e8af640 10 monclient: get_monmap_and_config 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.438+0000 7f274e8af640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.438+0000 7f274e8af640 10 monclient: monmap: 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:00.439175+0000 2024-09-15T03:35:00.439 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:00.439175+0000 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:00.440 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 1 Processor -- start 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 1 -- start start 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient: init 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f274809ad90 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 0x7f274809b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient(hunting): picked mon.noname-b con 0x7f274809b160 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2748136c90 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27480692c0 con 0x7f2748136c90 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274809cce0 con 0x7f274809ad90 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274809d890 con 0x7f274809b160 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274e8af640 10 monclient(hunting): authenticate will time out at 1098.078651s 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 0x7f274809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274ce4c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2748136c90 0x7f2748139070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33522/0 (socket says 172.21.15.153:33522) 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f274ce4c640 1 -- 172.21.15.153:0/4020915078 learned_addr learned my addr 172.21.15.153:0/4020915078 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): get_auth_request con 0x7f274809b160 auth_method 0 2024-09-15T03:35:00.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.439+0000 7f27477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 10 monclient(hunting): handle_auth_done global_id 74474 payload 274 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 1 -- 172.21.15.153:0/4020915078 >> v1:172.21.15.181:6789/0 conn(0x7f274809ad90 legacy=0x7f274809c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 1 -- 172.21.15.153:0/4020915078 >> v1:172.21.15.153:6789/0 conn(0x7f2748136c90 legacy=0x7f2748139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 1 monclient: found mon.noname-b 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:00.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.440+0000 7f27477fe640 1 -- 172.21.15.153:0/4020915078 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274809db70 con 0x7f274809b160 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.442+0000 7f27477fe640 1 -- 172.21.15.153:0/4020915078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 msgr2=0x7f274809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.442+0000 7f27477fe640 1 --2- 172.21.15.153:0/4020915078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 0x7f274809b530 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f273c009880 tx=0x7f273c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25573e697028b099 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 1 -- 172.21.15.153:0/4020915078 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c032440 con 0x7f274809b160 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: dump: 2024-09-15T03:35:00.449 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 1 -- 172.21.15.153:0/4020915078 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c0325a0 con 0x7f274809b160 2024-09-15T03:35:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 5 monclient: authenticate success, global_id 74474 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 1 -- 172.21.15.153:0/4020915078 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c032940 con 0x7f274809b160 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 4 set_mon_vals no callback set 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f2746ffd640 10 monclient: dump: 2024-09-15T03:35:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 10 monclient: shutdown 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 -- 172.21.15.153:0/4020915078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 msgr2=0x7f274809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 --2- 172.21.15.153:0/4020915078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 0x7f274809b530 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f273c009880 tx=0x7f273c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 -- 172.21.15.153:0/4020915078 shutdown_connections 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 --2- 172.21.15.153:0/4020915078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274809b160 0x7f274809b530 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 -- 172.21.15.153:0/4020915078 >> 172.21.15.153:0/4020915078 conn(0x7f27480678a0 msgr2=0x7f27480a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 -- 172.21.15.153:0/4020915078 shutdown_connections 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.443+0000 7f274e8af640 1 -- 172.21.15.153:0/4020915078 wait complete. 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.444+0000 7f274e8af640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.444+0000 7f274e8af640 10 monclient: monmap: 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.453 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.444+0000 7f274e8af640 1 Processor -- start 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.444+0000 7f274e8af640 1 -- start start 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.444+0000 7f274e8af640 10 monclient: init 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f274809ad90 0x7f2748089940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): picked mon.noname-b con 0x7f274809ad90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 0x7f2748089e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): picked mon.noname-a con 0x7f274809b160 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748136c90 0x7f274808a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): picked mon.noname-c con 0x7f2748136c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274808e1c0 con 0x7f274809b160 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274806f020 con 0x7f274809ad90 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274808caf0 con 0x7f2748136c90 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274e8af640 10 monclient(hunting): authenticate will time out at 1098.084266s 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 0x7f2748089e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 0x7f2748089e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46634/0 (socket says 172.21.15.153:46634) 2024-09-15T03:35:00.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748136c90 0x7f274808a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 -- 172.21.15.153:0/1593046120 learned_addr learned my addr 172.21.15.153:0/1593046120 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): get_auth_request con 0x7f274809b160 auth_method 0 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): get_auth_request con 0x7f2748136c90 auth_method 0 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f274809ad90 0x7f2748089940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): get_auth_request con 0x7f274809ad90 auth_method 0 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient(hunting): handle_auth_done global_id 84375 payload 824 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748136c90 msgr2=0x7f274808a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748136c90 0x7f274808a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f274809ad90 msgr2=0x7f2748089940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f274809ad90 0x7f2748089940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 monclient: found mon.noname-a 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f27477fe640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c009390 con 0x7f274809b160 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.445+0000 7f274ce4c640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748136c90 0x7f274808a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.447+0000 7f27477fe640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 msgr2=0x7f2748089e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.447+0000 7f27477fe640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 0x7f2748089e80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f273c009360 tx=0x7f273c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c031990 con 0x7f274809b160 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:00.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: dump: 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c031af0 con 0x7f274809b160 2024-09-15T03:35:00.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 5 monclient: authenticate success, global_id 84375 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 10 monclient: _renew_subs 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2748098b40 con 0x7f274809b160 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c031e70 con 0x7f274809b160 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f272ffff640 10 monclient: dump: 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.460 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 10 monclient: _renew_subs 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f27467fc640 4 set_mon_vals no callback set 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.448+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274808d270 con 0x7f274809b160 2024-09-15T03:35:00.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f274e8af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f273c039040 con 0x7f274809b160 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f274e8af640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 0x7f274808a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f274e8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2748065e00 con 0x7f274802aa40 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 0x7f274808a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient: get_auth_request con 0x7f274802aa40 auth_method 0 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f272ffff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2734038740 0x7f273403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f273c06cf40 con 0x7f274809b160 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 10 monclient(hunting): handle_auth_done global_id 74480 payload 824 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f2747fff640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f274802ae10 con 0x7f274802aa40 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f274ce4c640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2734038740 0x7f273403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.449+0000 7f274ce4c640 10 monclient: get_auth_request con 0x7f2734038740 auth_method 0 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f2747fff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 msgr2=0x7f274808a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 0x7f274808a900 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f273800f780 tx=0x7f2738007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273800d8b0 con 0x7f274802aa40 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 10 monclient: dump: 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f274ce4c640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2734038740 0x7f273403abf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f274808bd80 tx=0x7f2740009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f273800da10 con 0x7f274802aa40 2024-09-15T03:35:00.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 msgr2=0x7f274808a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 0x7f274808a900 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f273800f780 tx=0x7f2738007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 msgr2=0x7f274808a900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.451+0000 7f272ffff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274802aa40 0x7f274808a900 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f273800f780 tx=0x7f2738007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f274e8af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f274e8af640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f274e8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274808ae40 con 0x7f274805e310 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 10 monclient: get_auth_request con 0x7f274805e310 auth_method 0 2024-09-15T03:35:00.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 msgr2=0x7f274805e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:35:00.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 msgr2=0x7f274805e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:00.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:00.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.457+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.449+0000 7fad45b54640 10 monclient: get_monmap_and_config 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.449+0000 7fad45b54640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.449+0000 7fad45b54640 10 monclient: monmap: 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:00.451752+0000 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:00.451752+0000 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:00.481 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.450+0000 7fad45b54640 1 Processor -- start 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.450+0000 7fad45b54640 1 -- start start 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.450+0000 7fad45b54640 10 monclient: init 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fad4009ec50 0x7fad40131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): picked mon.noname-c con 0x7fad4009ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad401321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fad4009f020 0x7fad40069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): picked mon.noname-a con 0x7fad4009f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad40067420 con 0x7fad401321a0 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4009fdc0 con 0x7fad4009f020 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad400a0510 con 0x7fad4009ec50 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad45b54640 10 monclient(hunting): authenticate will time out at 1104.933308s 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fad4009ec50 0x7fad40131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fad4009f020 0x7fad40069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fad4009ec50 0x7fad40131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43340/0 (socket says 172.21.15.181:43340) 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 -- 172.21.15.181:0/2328024398 learned_addr learned my addr 172.21.15.181:0/2328024398 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request con 0x7fad4009ec50 auth_method 0 2024-09-15T03:35:00.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): get_auth_request con 0x7fad4009f020 auth_method 0 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_done global_id 94129 payload 274 2024-09-15T03:35:00.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 -- 172.21.15.181:0/2328024398 >> v2:172.21.15.153:3300/0 conn(0x7fad4009f020 msgr2=0x7fad40069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/2328024398 >> v2:172.21.15.153:3300/0 conn(0x7fad4009f020 0x7fad40069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 -- 172.21.15.181:0/2328024398 >> v1:172.21.15.153:6789/0 conn(0x7fad401321a0 legacy=0x7fad40068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 monclient: found mon.noname-c 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.451+0000 7fad3f7fe640 1 -- 172.21.15.181:0/2328024398 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400a0200 con 0x7fad4009ec50 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3f7fe640 1 -- 172.21.15.181:0/2328024398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad40131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/2328024398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad40131c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad400719d0 tx=0x7fad30009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=514dc5e6ccc1c3c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 1 -- 172.21.15.181:0/2328024398 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30007610 con 0x7fad4009ec50 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:00.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: dump: 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 1 -- 172.21.15.181:0/2328024398 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3002d050 con 0x7fad4009ec50 2024-09-15T03:35:00.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad45b54640 5 monclient: authenticate success, global_id 94129 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad45b54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad45b54640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 1 -- 172.21.15.181:0/2328024398 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300079a0 con 0x7fad4009ec50 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad45b54640 4 set_mon_vals no callback set 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.454+0000 7fad3e7fc640 10 monclient: dump: 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.487 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 10 monclient: shutdown 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 -- 172.21.15.181:0/2328024398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad40131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 --2- 172.21.15.181:0/2328024398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad40131c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad400719d0 tx=0x7fad30009840 comp rx=0 tx=0).stop 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 -- 172.21.15.181:0/2328024398 shutdown_connections 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 --2- 172.21.15.181:0/2328024398 >> v2:172.21.15.153:3300/0 conn(0x7fad4009f020 0x7fad40069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 --2- 172.21.15.181:0/2328024398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad40131c60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 -- 172.21.15.181:0/2328024398 >> 172.21.15.181:0/2328024398 conn(0x7fad400a5d40 msgr2=0x7fad400a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 -- 172.21.15.181:0/2328024398 shutdown_connections 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 1 -- 172.21.15.181:0/2328024398 wait complete. 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.455+0000 7fad45b54640 10 monclient: monmap: 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:00.489 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 Processor -- start 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 -- start start 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient: init 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): picked mon.noname-b con 0x7fad4009ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad40130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): picked mon.noname-c con 0x7fad4009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad401310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): picked mon.noname-a con 0x7fad401321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4009b660 con 0x7fad401321a0 2024-09-15T03:35:00.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4009c000 con 0x7fad4009ec50 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4009dda0 con 0x7fad4009f020 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad45b54640 10 monclient(hunting): authenticate will time out at 1104.938689s 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43354/0 (socket says 172.21.15.181:43354) 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 learned_addr learned my addr 172.21.15.181:0/904558051 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request con 0x7fad4009ec50 auth_method 0 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad401310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad40130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): get_auth_request con 0x7fad401321a0 auth_method 0 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): get_auth_request con 0x7fad4009f020 auth_method 0 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.456+0000 7fad3f7fe640 10 monclient(hunting): handle_auth_done global_id 94132 payload 824 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 msgr2=0x7fad40130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad40130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 msgr2=0x7fad401310c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad401310c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 1 monclient: found mon.noname-b 2024-09-15T03:35:00.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.457+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad30009ae0 con 0x7fad4009ec50 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad4009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fad3002ac00 tx=0x7fad30036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300328b0 con 0x7fad4009ec50 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: dump: 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.494 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30032a30 con 0x7fad4009ec50 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 5 monclient: authenticate success, global_id 94132 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 10 monclient: _renew_subs 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4011dca0 con 0x7fad4009ec50 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30032df0 con 0x7fad4009ec50 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad277fe640 10 monclient: dump: 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.496 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 10 monclient: _renew_subs 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4011c570 con 0x7fad4009ec50 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.459+0000 7fad3dffb640 4 set_mon_vals no callback set 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad30038050 con 0x7fad4009ec50 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad45b54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad45b54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad45b54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad400008d0 con 0x7fad4009ec50 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fad3006d660 con 0x7fad4009ec50 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad3effd640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.460+0000 7fad3effd640 10 monclient: get_auth_request con 0x7fad2c038740 auth_method 0 2024-09-15T03:35:00.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.463+0000 7fad3effd640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fad34007fd0 tx=0x7fad34007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad4009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad4009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fad3002ac00 tx=0x7fad30036000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fad3002ac00 tx=0x7fad30036000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad4009aa10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad4009aa10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c045040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): picked mon.a con 0x7fad401321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 0x7fad2c047d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): picked mon.b con 0x7fad2c0458f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): picked mon.c con 0x7fad4009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c04b1d0 con 0x7fad401321a0 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c04b310 con 0x7fad2c0458f0 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c04b4d0 con 0x7fad4009f020 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad277fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 0x7fad2c047d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): get_auth_request con 0x7fad2c0458f0 auth_method 0 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request con 0x7fad4009f020 auth_method 0 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c045040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request con 0x7fad401321a0 auth_method 0 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient(hunting): handle_auth_done global_id 94132 payload 293 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 msgr2=0x7fad2c04a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 msgr2=0x7fad2c045040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c045040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 monclient: found mon.b 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fad2c04b750 con 0x7fad2c0458f0 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad38009dc0 con 0x7fad2c0458f0 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.465+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.467+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 msgr2=0x7fad2c047d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.467+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 0x7fad2c047d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad38011520 tx=0x7fad38011550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38020070 con 0x7fad2c0458f0 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 10 monclient: dump: 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.502 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38011e90 con 0x7fad2c0458f0 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad3d7fa640 4 set_mon_vals no callback set 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.468+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad38014650 con 0x7fad2c0458f0 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.472+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fad380192d0 con 0x7fad2c0458f0 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.472+0000 7fad277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.472+0000 7fad277fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:00.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.634+0000 7fad45b54640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:00.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.634+0000 7fad45b54640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:00.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.634+0000 7fad45b54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:00.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:00.634+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fad400a13e0 con 0x7fad2c0458f0 2024-09-15T03:35:00.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient: get_auth_request con 0x7f274805e310 auth_method 0 2024-09-15T03:35:00.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 10 monclient(hunting): handle_auth_done global_id 74486 payload 824 2024-09-15T03:35:00.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.658+0000 7f2747fff640 1 -- 172.21.15.153:0/1593046120 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f274808c100 con 0x7f274805e310 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.660+0000 7f2747fff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 msgr2=0x7f274805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.660+0000 7f2747fff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f273800f780 tx=0x7f2738019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.660+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273802b430 con 0x7f274805e310 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.660+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.660+0000 7f272ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.660+0000 7f272ffff640 10 monclient: dump: 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f273801d3f0 con 0x7f274805e310 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 msgr2=0x7f274805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f273800f780 tx=0x7f2738019000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 msgr2=0x7f274805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:00.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f272ffff640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274805e310 0x7f274805e710 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f273800f780 tx=0x7f2738019000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 68, 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.664 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:00.665 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.666 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:00.668 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.671 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.673 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:00.674 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:00.675 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:00.676 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:00.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2734038740 msgr2=0x7f273403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2734038740 0x7f273403abf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f274808bd80 tx=0x7f2740009040 comp rx=0 tx=0).stop 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 10 monclient: shutdown 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 msgr2=0x7f2748089e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 0x7f2748089e80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f273c009360 tx=0x7f273c032000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274ce4c640 1 -- 172.21.15.153:0/1593046120 reap_dead start 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 shutdown_connections 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 --2- 172.21.15.153:0/1593046120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f274809b160 0x7f2748089e80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 >> 172.21.15.153:0/1593046120 conn(0x7f27480678a0 msgr2=0x7f2748139890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 shutdown_connections 2024-09-15T03:35:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.661+0000 7f274e8af640 1 -- 172.21.15.153:0/1593046120 wait complete. 2024-09-15T03:35:00.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:00.679 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:35:00.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:35:00.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.772+0000 7fc46c79d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:00.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.772+0000 7fc46c79d640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.772+0000 7fc46c79d640 10 monclient: monmap: 2024-09-15T03:35:00.773 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:00.772954+0000 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:00.772954+0000 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.772+0000 7fc46c79d640 1 Processor -- start 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.772+0000 7fc46c79d640 1 -- start start 2024-09-15T03:35:00.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.772+0000 7fc46c79d640 10 monclient: init 2024-09-15T03:35:00.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc4641318f0 0x7fc464131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4641318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc46409ea50 0x7fc46409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): picked mon.noname-c con 0x7fc46409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc464068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46409fbf0 con 0x7fc464068d90 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc464131d50 con 0x7fc4641318f0 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46409fe30 con 0x7fc46409ea50 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46c79d640 10 monclient(hunting): authenticate will time out at 1098.412499s 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc4641318f0 0x7fc464131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc4641318f0 0x7fc464131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46646/0 (socket says 172.21.15.153:46646) 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 -- 172.21.15.153:0/1233559050 learned_addr learned my addr 172.21.15.153:0/1233559050 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): get_auth_request con 0x7fc4641318f0 auth_method 0 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc469d38640 1 --2- 172.21.15.153:0/1233559050 >> v2:172.21.15.181:3300/0 conn(0x7fc46409ea50 0x7fc46409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient(hunting): handle_auth_done global_id 84384 payload 274 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 -- 172.21.15.153:0/1233559050 >> v2:172.21.15.181:3300/0 conn(0x7fc46409ea50 msgr2=0x7fc46409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 --2- 172.21.15.153:0/1233559050 >> v2:172.21.15.181:3300/0 conn(0x7fc46409ea50 0x7fc46409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 -- 172.21.15.153:0/1233559050 >> v1:172.21.15.181:6789/0 conn(0x7fc464068d90 legacy=0x7fc4640691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 monclient: found mon.noname-a 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc46a539640 1 -- 172.21.15.153:0/1233559050 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4640a1210 con 0x7fc4641318f0 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc469d38640 10 monclient: get_auth_request con 0x7fc46409ea50 auth_method 0 2024-09-15T03:35:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.773+0000 7fc469d38640 1 --2- 172.21.15.153:0/1233559050 >> v2:172.21.15.181:3300/0 conn(0x7fc46409ea50 0x7fc46409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:35:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc46a539640 1 -- 172.21.15.153:0/1233559050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4641318f0 msgr2=0x7fc464131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc46a539640 1 --2- 172.21.15.153:0/1233559050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4641318f0 0x7fc464131f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc46406f600 tx=0x7fc4580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17904b5826446ae8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 1 -- 172.21.15.153:0/1233559050 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458032440 con 0x7fc4641318f0 2024-09-15T03:35:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: dump: 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 1 -- 172.21.15.153:0/1233559050 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4580325a0 con 0x7fc4641318f0 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc46c79d640 5 monclient: authenticate success, global_id 84384 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc46c79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 1 -- 172.21.15.153:0/1233559050 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458032940 con 0x7fc4641318f0 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc469537640 10 monclient: dump: 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.787 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc46c79d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.776+0000 7fc46c79d640 4 set_mon_vals no callback set 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 10 monclient: shutdown 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 -- 172.21.15.153:0/1233559050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4641318f0 msgr2=0x7fc464131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1233559050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4641318f0 0x7fc464131f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc46406f600 tx=0x7fc4580091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 -- 172.21.15.153:0/1233559050 shutdown_connections 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1233559050 >> v2:172.21.15.181:3300/0 conn(0x7fc46409ea50 0x7fc46409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1233559050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4641318f0 0x7fc464131f30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 -- 172.21.15.153:0/1233559050 >> 172.21.15.153:0/1233559050 conn(0x7fc464067110 msgr2=0x7fc464067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 -- 172.21.15.153:0/1233559050 shutdown_connections 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.777+0000 7fc46c79d640 1 -- 172.21.15.153:0/1233559050 wait complete. 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient: build_initial_monmap 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient: monmap: 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:00.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 Processor -- start 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 -- start start 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient: init 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc464068d90 0x7fc46409abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): picked mon.noname-b con 0x7fc464068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 0x7fc464130cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): picked mon.noname-a con 0x7fc46409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4641318f0 0x7fc464131200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4641318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc464131740 con 0x7fc46409ea50 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640a4400 con 0x7fc464068d90 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc464141780 con 0x7fc4641318f0 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc46c79d640 10 monclient(hunting): authenticate will time out at 1098.417937s 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.778+0000 7fc469d38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 0x7fc464130cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 0x7fc464130cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46650/0 (socket says 172.21.15.153:46650) 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 -- 172.21.15.153:0/1955581211 learned_addr learned my addr 172.21.15.153:0/1955581211 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:00.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): get_auth_request con 0x7fc46409ea50 auth_method 0 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4641318f0 0x7fc464131200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): get_auth_request con 0x7fc4641318f0 auth_method 0 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc464068d90 0x7fc46409abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46ad3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46a539640 10 monclient(hunting): get_auth_request con 0x7fc464068d90 auth_method 0 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46a539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46a539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46a539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient(hunting): handle_auth_done global_id 84387 payload 824 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4641318f0 msgr2=0x7fc464131200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4641318f0 0x7fc464131200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc464068d90 msgr2=0x7fc46409abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc464068d90 0x7fc46409abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient: _finish_hunting 0 2024-09-15T03:35:00.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 monclient: found mon.noname-a 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc469d38640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458009450 con 0x7fc46409ea50 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.779+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc464068d90 0x7fc46409abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc469d38640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 msgr2=0x7fc464130cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc469d38640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 0x7fc464130cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc46000c880 tx=0x7fc460008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000ce10 con 0x7fc46409ea50 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: dump: 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc460007aa0 con 0x7fc46409ea50 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460007e20 con 0x7fc46409ea50 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 5 monclient: authenticate success, global_id 84387 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 10 monclient: _renew_subs 2024-09-15T03:35:00.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4641414a0 con 0x7fc46409ea50 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc4567fc640 10 monclient: dump: 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.796 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc468d36640 4 set_mon_vals no callback set 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 10 monclient: _renew_subs 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.781+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46411e1c0 con 0x7fc46409ea50 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46c79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc460011600 con 0x7fc46409ea50 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc464065e00 con 0x7fc46411b720 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc4567fc640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc448038740 0x7fc44803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fc46004cd90 con 0x7fc46409ea50 2024-09-15T03:35:00.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46a539640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 msgr2=0x7fc46411baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46a539640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 msgr2=0x7fc46411baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46ad3a640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc448038740 0x7fc44803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.782+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.783+0000 7fc46ad3a640 10 monclient: get_auth_request con 0x7fc448038740 auth_method 0 2024-09-15T03:35:00.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.785+0000 7fc46ad3a640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc448038740 0x7fc44803abf0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc45c007fd0 tx=0x7fc45c0025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient: get_auth_request con 0x7fc46411b720 auth_method 0 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 10 monclient(hunting): handle_auth_done global_id 94138 payload 824 2024-09-15T03:35:00.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.984+0000 7fc46a539640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc46404bf60 con 0x7fc46411b720 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.987+0000 7fc46a539640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 msgr2=0x7fc46411baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.987+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc458034000 tx=0x7fc458032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.987+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45803bc80 con 0x7fc46411b720 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.987+0000 7fc4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.987+0000 7fc4567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.987+0000 7fc4567fc640 10 monclient: dump: 2024-09-15T03:35:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc458040070 con 0x7fc46411b720 2024-09-15T03:35:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 msgr2=0x7fc46411baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc458034000 tx=0x7fc458032000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 msgr2=0x7fc46411baf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.988+0000 7fc4567fc640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411b720 0x7fc46411baf0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc458034000 tx=0x7fc458032000 comp rx=0 tx=0).stop 2024-09-15T03:35:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46c79d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 0x7fc464096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46c79d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640a3320 con 0x7fc46411c030 2024-09-15T03:35:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 0x7fc464096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:00.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient: get_auth_request con 0x7fc46411c030 auth_method 0 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 10 monclient(hunting): handle_auth_done global_id 94141 payload 824 2024-09-15T03:35:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.994+0000 7fc46a539640 1 -- 172.21.15.153:0/1955581211 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc46411cf80 con 0x7fc46411c030 2024-09-15T03:35:00.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc46a539640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 msgr2=0x7fc464096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:00.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc46a539640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 0x7fc464096a60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc458009210 tx=0x7fc45803e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:00.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4580469e0 con 0x7fc46411c030 2024-09-15T03:35:00.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:00.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 10 monclient: dump: 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fc458046b40 con 0x7fc46411c030 2024-09-15T03:35:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 msgr2=0x7fc464096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 0x7fc464096a60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc458009210 tx=0x7fc45803e070 comp rx=0 tx=0).stop 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 msgr2=0x7fc464096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.997+0000 7fc4567fc640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc46411c030 0x7fc464096a60 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc458009210 tx=0x7fc45803e070 comp rx=0 tx=0).stop 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:01.000 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 68, 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:01.001 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:01.002 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:01.003 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:01.004 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.005 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:01.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:01.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.009 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:01.011 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:01.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:01.013 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:01.013 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc448038740 msgr2=0x7fc44803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc448038740 0x7fc44803abf0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc45c007fd0 tx=0x7fc45c0025b0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 10 monclient: shutdown 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 msgr2=0x7fc464130cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 0x7fc464130cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc46000c880 tx=0x7fc460008040 comp rx=0 tx=0).stop 2024-09-15T03:35:01.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46ad3a640 1 -- 172.21.15.153:0/1955581211 reap_dead start 2024-09-15T03:35:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 shutdown_connections 2024-09-15T03:35:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 --2- 172.21.15.153:0/1955581211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc46409ea50 0x7fc464130cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 >> 172.21.15.153:0/1955581211 conn(0x7fc464067110 msgr2=0x7fc46409ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 shutdown_connections 2024-09-15T03:35:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:00.998+0000 7fc46c79d640 1 -- 172.21.15.153:0/1955581211 wait complete. 2024-09-15T03:35:01.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:01.015 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:35:01.015 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:35:01.015 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:35:01.016 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:35:01.016 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:35:01.016 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 msgr2=0x7fad2c047d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 msgr2=0x7fad2c047d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 0x7fad2c047d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad38011520 tx=0x7fad38011550 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 0x7fad2c047d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad38011520 tx=0x7fad38011550 comp rx=0 tx=0).stop 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 msgr2=0x7fad2c047d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad2c0458f0 0x7fad2c047d70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c044fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): picked mon.a con 0x7fad401321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): picked mon.c con 0x7fad4009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad2c04e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): picked mon.b con 0x7fad4009ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c04ec10 con 0x7fad401321a0 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c04ed50 con 0x7fad4009ec50 2024-09-15T03:35:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c04ee90 con 0x7fad4009f020 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad277fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad2c04e330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad2c04e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c044fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 msgr2=0x7fad2c044fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 msgr2=0x7fad2c044fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c044fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3f7fe640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c044fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): get_auth_request con 0x7fad4009f020 auth_method 0 2024-09-15T03:35:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient(hunting): handle_auth_done global_id 94132 payload 293 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 msgr2=0x7fad2c04e330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad2c04e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 msgr2=0x7fad2c044fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c044fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 monclient: found mon.c 2024-09-15T03:35:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fad2c04efd0 con 0x7fad4009f020 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fad400a13e0 con 0x7fad4009f020 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 10 monclient: _finish_auth 0 2024-09-15T03:35:01.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.018+0000 7fad3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad3ffff640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 msgr2=0x7fad2c04a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad3ffff640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fad380490c0 tx=0x7fad3801cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38033070 con 0x7fad4009f020 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad277fe640 10 monclient: dump: 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:01.027 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38048cd0 con 0x7fad4009f020 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.021+0000 7fad3d7fa640 4 set_mon_vals no callback set 2024-09-15T03:35:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:01.022+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad38035030 con 0x7fad4009f020 2024-09-15T03:35:01.029 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:35:01.029 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:35:01.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:01.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.120+0000 7fd56bc9f640 10 monclient: get_monmap_and_config 2024-09-15T03:35:01.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.120+0000 7fd56bc9f640 10 monclient: build_initial_monmap 2024-09-15T03:35:01.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.120+0000 7fd56bc9f640 10 monclient: monmap: 2024-09-15T03:35:01.121 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:01.121 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:01.121365+0000 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:01.121365+0000 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:01.122 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:01.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 1 Processor -- start 2024-09-15T03:35:01.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 1 -- start start 2024-09-15T03:35:01.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient: init 2024-09-15T03:35:01.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:01.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd564096c60 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 0x7fd564097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient(hunting): picked mon.noname-b con 0x7fd564097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd564133b40 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56409d2e0 con 0x7fd564133b40 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56409d6b0 con 0x7fd564096c60 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640992f0 con 0x7fd564097030 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56bc9f640 10 monclient(hunting): authenticate will time out at 1098.760864s 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd56923a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 0x7fd564097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.121+0000 7fd569a3b640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fd564096c60 legacy=0x7fd56409a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:01.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 0x7fd564097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35098/0 (socket says 172.21.15.153:35098) 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56a23c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd564133b40 0x7fd564135f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33528/0 (socket says 172.21.15.153:33528) 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 1 -- 172.21.15.153:0/1133087537 learned_addr learned my addr 172.21.15.153:0/1133087537 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): get_auth_request con 0x7fd564097030 auth_method 0 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient(hunting): handle_auth_done global_id 74492 payload 274 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 1 -- 172.21.15.153:0/1133087537 >> v1:172.21.15.181:6789/0 conn(0x7fd564096c60 legacy=0x7fd56409a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 1 -- 172.21.15.153:0/1133087537 >> v1:172.21.15.153:6789/0 conn(0x7fd564133b40 legacy=0x7fd564135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:01.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 1 monclient: found mon.noname-b 2024-09-15T03:35:01.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:01.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:01.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.122+0000 7fd56923a640 1 -- 172.21.15.153:0/1133087537 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640995d0 con 0x7fd564097030 2024-09-15T03:35:01.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.124+0000 7fd56923a640 1 -- 172.21.15.153:0/1133087537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 msgr2=0x7fd564097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:01.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.124+0000 7fd56923a640 1 --2- 172.21.15.153:0/1133087537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 0x7fd564097430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd554009940 tx=0x7fd5540091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde50ba7192feb7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:01.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 1 -- 172.21.15.153:0/1133087537 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554032440 con 0x7fd564097030 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: dump: 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:01.133 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: _finish_auth 0 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 5 monclient: authenticate success, global_id 74492 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 1 -- 172.21.15.153:0/1133087537 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5540325a0 con 0x7fd564097030 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 4 set_mon_vals no callback set 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 1 -- 172.21.15.153:0/1133087537 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554032940 con 0x7fd564097030 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:01.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd568a39640 10 monclient: dump: 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 10 monclient: shutdown 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:01.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 -- 172.21.15.153:0/1133087537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 msgr2=0x7fd564097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/1133087537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 0x7fd564097430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd554009940 tx=0x7fd5540091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 -- 172.21.15.153:0/1133087537 shutdown_connections 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/1133087537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564097030 0x7fd564097430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 -- 172.21.15.153:0/1133087537 >> 172.21.15.153:0/1133087537 conn(0x7fd5640672b0 msgr2=0x7fd5640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 -- 172.21.15.153:0/1133087537 shutdown_connections 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.125+0000 7fd56bc9f640 1 -- 172.21.15.153:0/1133087537 wait complete. 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient: build_initial_monmap 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient: monmap: 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:01.136 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 1 Processor -- start 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 1 -- start start 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient: init 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd564096c60 0x7fd56406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient(hunting): picked mon.noname-b con 0x7fd564096c60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 0x7fd56406e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient(hunting): picked mon.noname-a con 0x7fd564097030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 0x7fd56406e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.126+0000 7fd56bc9f640 10 monclient(hunting): picked mon.noname-c con 0x7fd564133b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5641359d0 con 0x7fd564097030 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564136c30 con 0x7fd564096c60 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5641346d0 con 0x7fd564133b40 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56bc9f640 10 monclient(hunting): authenticate will time out at 1098.766017s 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd569a3b640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd564096c60 msgr2=0x7fd56406dee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 0x7fd56406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 0x7fd56406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:01.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd569a3b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd564096c60 0x7fd56406dee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 0x7fd56406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46658/0 (socket says 172.21.15.153:46658) 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 -- 172.21.15.153:0/2535801047 learned_addr learned my addr 172.21.15.153:0/2535801047 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 0x7fd56406e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35112/0 (socket says 172.21.15.153:35112) 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): get_auth_request con 0x7fd564097030 auth_method 0 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): get_auth_request con 0x7fd564133b40 auth_method 0 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:01.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient(hunting): handle_auth_done global_id 84390 payload 824 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 -- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 msgr2=0x7fd56406e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 0x7fd56406e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 -- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd564096c60 msgr2=0x7fd56406dee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd564096c60 0x7fd56406dee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 monclient: found mon.noname-a 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:01.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56923a640 1 -- 172.21.15.153:0/2535801047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554009450 con 0x7fd564097030 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.127+0000 7fd56a23c640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 0x7fd56406e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd56923a640 1 -- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 msgr2=0x7fd56406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd56923a640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 0x7fd56406e420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd5540099a0 tx=0x7fd554032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55403c960 con 0x7fd564097030 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: dump: 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:01.141 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55403cae0 con 0x7fd564097030 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd56bc9f640 5 monclient: authenticate success, global_id 84390 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:01.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd56bc9f640 10 monclient: _renew_subs 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55403cea0 con 0x7fd564097030 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd56bc9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd564134420 con 0x7fd564097030 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd559ffb640 10 monclient: dump: 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:01.143 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.129+0000 7fd55bfff640 4 set_mon_vals no callback set 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 10 monclient: _renew_subs 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd564134ae0 con 0x7fd564097030 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd554039040 con 0x7fd564097030 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56404bf60 con 0x7fd564097030 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.130+0000 7fd559ffb640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5480386f0 0x7fd54803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:01.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.131+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fd55406cb70 con 0x7fd564097030 2024-09-15T03:35:01.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.131+0000 7fd569a3b640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5480386f0 0x7fd54803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:01.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.131+0000 7fd569a3b640 10 monclient: get_auth_request con 0x7fd5480386f0 auth_method 0 2024-09-15T03:35:01.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.133+0000 7fd569a3b640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5480386f0 0x7fd54803aba0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd5600038f0 tx=0x7fd560006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:01.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.135+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd55404bc40 con 0x7fd564097030 2024-09-15T03:35:01.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.135+0000 7fd559ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:01.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.135+0000 7fd559ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:01.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.347+0000 7fd56bc9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.347+0000 7fd56bc9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.348+0000 7fd56bc9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.348+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd56404bf60 con 0x7fd564097030 2024-09-15T03:35:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.348+0000 7fd559ffb640 1 -- 172.21.15.153:0/2535801047 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd554040070 con 0x7fd564097030 2024-09-15T03:35:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.348+0000 7fd559ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.348+0000 7fd559ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:01.350 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5480386f0 msgr2=0x7fd54803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5480386f0 0x7fd54803aba0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd5600038f0 tx=0x7fd560006d80 comp rx=0 tx=0).stop 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 10 monclient: shutdown 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 msgr2=0x7fd56406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 0x7fd56406e420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd5540099a0 tx=0x7fd554032000 comp rx=0 tx=0).stop 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 shutdown_connections 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5480386f0 0x7fd54803aba0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd564133b40 0x7fd56406e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564097030 0x7fd56406e420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 --2- 172.21.15.153:0/2535801047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd564096c60 0x7fd56406dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 >> 172.21.15.153:0/2535801047 conn(0x7fd5640672b0 msgr2=0x7fd564136e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 shutdown_connections 2024-09-15T03:35:01.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:01.350+0000 7fd56bc9f640 1 -- 172.21.15.153:0/2535801047 wait complete. 2024-09-15T03:35:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:03.457+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:03.457+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:03.457+0000 7fad267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:04.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:04.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:04.457+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:04.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:04.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:04.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.466+0000 7f722395d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:04.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.466+0000 7f722395d640 10 monclient: build_initial_monmap 2024-09-15T03:35:04.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.466+0000 7f722395d640 10 monclient: monmap: 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:04.467614+0000 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:04.467614+0000 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:04.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.467+0000 7f722395d640 1 Processor -- start 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.467+0000 7f722395d640 1 -- start start 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.467+0000 7f722395d640 10 monclient: init 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f721c096e10 0x7f721c09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): picked mon.noname-a con 0x7f721c096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f721c09aaa0 0x7f721c0971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): picked mon.noname-c con 0x7f721c09aaa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): picked mon.noname-b con 0x7f721c097720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c099620 con 0x7f721c096e10 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c099340 con 0x7f721c09aaa0 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c099900 con 0x7f721c097720 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f722395d640 10 monclient(hunting): authenticate will time out at 1102.107269s 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f721c096e10 0x7f721c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7220ef8640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f721c09aaa0 msgr2=0x7f721c0971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7220ef8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f721c09aaa0 0x7f721c0971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35134/0 (socket says 172.21.15.153:35134) 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 -- 172.21.15.153:0/331992810 learned_addr learned my addr 172.21.15.153:0/331992810 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): get_auth_request con 0x7f721c097720 auth_method 0 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): get_auth_request con 0x7f721c096e10 auth_method 0 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:04.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient(hunting): handle_auth_done global_id 74498 payload 274 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 -- 172.21.15.153:0/331992810 >> v2:172.21.15.181:3300/0 conn(0x7f721c09aaa0 msgr2=0x7f721c0971e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 --2- 172.21.15.153:0/331992810 >> v2:172.21.15.181:3300/0 conn(0x7f721c09aaa0 0x7f721c0971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 -- 172.21.15.153:0/331992810 >> v2:172.21.15.153:3300/0 conn(0x7f721c096e10 msgr2=0x7f721c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 --2- 172.21.15.153:0/331992810 >> v2:172.21.15.153:3300/0 conn(0x7f721c096e10 0x7f721c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient: _finish_hunting 0 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 monclient: found mon.noname-b 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:04.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f7221efa640 1 -- 172.21.15.153:0/331992810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c097af0 con 0x7f721c097720 2024-09-15T03:35:04.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.468+0000 7f72216f9640 1 --2- 172.21.15.153:0/331992810 >> v2:172.21.15.153:3300/0 conn(0x7f721c096e10 0x7f721c09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f7221efa640 1 -- 172.21.15.153:0/331992810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 msgr2=0x7f721c12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f7221efa640 1 --2- 172.21.15.153:0/331992810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12e5f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f721c09aff0 tx=0x7f720c00ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1083319be888ad5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 1 -- 172.21.15.153:0/331992810 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c009610 con 0x7f721c097720 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: dump: 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:04.480 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:04.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: _finish_auth 0 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 1 -- 172.21.15.153:0/331992810 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c009770 con 0x7f721c097720 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f722395d640 5 monclient: authenticate success, global_id 74498 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f722395d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f722395d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 1 -- 172.21.15.153:0/331992810 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c009b50 con 0x7f721c097720 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f722395d640 4 set_mon_vals no callback set 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.471+0000 7f721bfff640 10 monclient: dump: 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:04.482 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 10 monclient: shutdown 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 -- 172.21.15.153:0/331992810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 msgr2=0x7f721c12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 --2- 172.21.15.153:0/331992810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12e5f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f721c09aff0 tx=0x7f720c00ba50 comp rx=0 tx=0).stop 2024-09-15T03:35:04.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 -- 172.21.15.153:0/331992810 shutdown_connections 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 --2- 172.21.15.153:0/331992810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12e5f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 --2- 172.21.15.153:0/331992810 >> v2:172.21.15.181:3300/0 conn(0x7f721c09aaa0 0x7f721c0971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 --2- 172.21.15.153:0/331992810 >> v2:172.21.15.153:3300/0 conn(0x7f721c096e10 0x7f721c09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 -- 172.21.15.153:0/331992810 >> 172.21.15.153:0/331992810 conn(0x7f721c0672b0 msgr2=0x7f721c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 -- 172.21.15.153:0/331992810 shutdown_connections 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.472+0000 7f722395d640 1 -- 172.21.15.153:0/331992810 wait complete. 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.473+0000 7f722395d640 10 monclient: build_initial_monmap 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.473+0000 7f722395d640 10 monclient: monmap: 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:04.484 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.473+0000 7f722395d640 1 Processor -- start 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.473+0000 7f722395d640 1 -- start start 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.473+0000 7f722395d640 10 monclient: init 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 0x7f721c140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): picked mon.noname-a con 0x7f721c096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:04.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): picked mon.noname-c con 0x7f721c097720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f721c09aaa0 0x7f721c11b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): picked mon.noname-b con 0x7f721c09aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c12e890 con 0x7f721c096e10 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c11ef90 con 0x7f721c09aaa0 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c09b050 con 0x7f721c097720 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f722395d640 10 monclient(hunting): authenticate will time out at 1102.113118s 2024-09-15T03:35:04.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7221efa640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f721c09aaa0 msgr2=0x7f721c11b890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 0x7f721c140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7221efa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f721c09aaa0 0x7f721c11b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 0x7f721c140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46668/0 (socket says 172.21.15.153:46668) 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 -- 172.21.15.153:0/786057514 learned_addr learned my addr 172.21.15.153:0/786057514 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): get_auth_request con 0x7f721c096e10 auth_method 0 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35140/0 (socket says 172.21.15.153:35140) 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): get_auth_request con 0x7f721c097720 auth_method 0 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:04.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient(hunting): handle_auth_done global_id 84396 payload 824 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 -- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 msgr2=0x7f721c12db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 -- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f721c09aaa0 msgr2=0x7f721c11b890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f721c09aaa0 0x7f721c11b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient: _finish_hunting 0 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 monclient: found mon.noname-a 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f72216f9640 1 -- 172.21.15.153:0/786057514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7214009390 con 0x7f721c096e10 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.474+0000 7f7220ef8640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.476+0000 7f72216f9640 1 -- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 msgr2=0x7f721c140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.476+0000 7f72216f9640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 0x7f721c140c10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f721400ec80 tx=0x7f72140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721400eec0 con 0x7f721c096e10 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: dump: 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:04.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 5 monclient: authenticate success, global_id 84396 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721400cdd0 con 0x7f721c096e10 2024-09-15T03:35:04.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 10 monclient: _renew_subs 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720c00bcf0 con 0x7f721c096e10 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214013680 con 0x7f721c096e10 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f72197fa640 10 monclient: dump: 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:04.491 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f721b7fe640 4 set_mon_vals no callback set 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 10 monclient: _renew_subs 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.477+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c11ddc0 con 0x7f721c096e10 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f72140139f0 con 0x7f721c096e10 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f72197fa640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72080386f0 0x7f720803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f721404d130 con 0x7f721c096e10 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f722395d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:04.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f722395d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f722395d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c11d110 con 0x7f721c096e10 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f7220ef8640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72080386f0 0x7f720803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.478+0000 7f7220ef8640 10 monclient: get_auth_request con 0x7f72080386f0 auth_method 0 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.481+0000 7f7220ef8640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72080386f0 0x7f720803aba0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72100038f0 tx=0x7f72100090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.482+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7214023ad0 con 0x7f721c096e10 2024-09-15T03:35:04.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.482+0000 7f72197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:04.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.482+0000 7f72197fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f722395d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f722395d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f722395d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f721c04bf60 con 0x7f721c096e10 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f72197fa640 1 -- 172.21.15.153:0/786057514 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f72140232f0 con 0x7f721c096e10 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f72197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.698+0000 7f72197fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:04.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72080386f0 msgr2=0x7f720803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72080386f0 0x7f720803aba0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72100038f0 tx=0x7f72100090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 10 monclient: shutdown 2024-09-15T03:35:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 msgr2=0x7f721c140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 0x7f721c140c10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f721400ec80 tx=0x7f72140090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 shutdown_connections 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72080386f0 0x7f720803aba0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f721c09aaa0 0x7f721c11b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f721c097720 0x7f721c12db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 --2- 172.21.15.153:0/786057514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f721c096e10 0x7f721c140c10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 >> 172.21.15.153:0/786057514 conn(0x7f721c0672b0 msgr2=0x7f721c0696a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 shutdown_connections 2024-09-15T03:35:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:04.700+0000 7f722395d640 1 -- 172.21.15.153:0/786057514 wait complete. 2024-09-15T03:35:05.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:05.458+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:05.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:05.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:06.458+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:06.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:07.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:07.458+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:07.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:07.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:07.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.818+0000 7f7155900640 10 monclient: get_monmap_and_config 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.818+0000 7f7155900640 10 monclient: build_initial_monmap 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.818+0000 7f7155900640 10 monclient: monmap: 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:07.819966+0000 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:07.819966+0000 2024-09-15T03:35:07.820 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:07.821 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:07.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.818+0000 7f7155900640 1 Processor -- start 2024-09-15T03:35:07.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.818+0000 7f7155900640 1 -- start start 2024-09-15T03:35:07.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.818+0000 7f7155900640 10 monclient: init 2024-09-15T03:35:07.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:07.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:35:07.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 0x7f7150131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): picked mon.noname-b con 0x7f71501318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f715009ea50 0x7f715009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): picked mon.noname-c con 0x7f715009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7150068d90 0x7f71500691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): picked mon.noname-a con 0x7f7150068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7150131cc0 con 0x7f7150068d90 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715009fbf0 con 0x7f715009ea50 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715009fe30 con 0x7f71501318f0 2024-09-15T03:35:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f7155900640 10 monclient(hunting): authenticate will time out at 1105.459410s 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714e7fc640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f715009ea50 msgr2=0x7f715009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f715009ea50 0x7f715009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 0x7f7150131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7150068d90 0x7f71500691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7150068d90 0x7f71500691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57644/0 (socket says 172.21.15.153:57644) 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 -- 172.21.15.153:0/2141199864 learned_addr learned my addr 172.21.15.153:0/2141199864 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 0x7f7150131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55112/0 (socket says 172.21.15.153:55112) 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): get_auth_request con 0x7f7150068d90 auth_method 0 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:07.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): get_auth_request con 0x7f71501318f0 auth_method 0 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient(hunting): handle_auth_done global_id 84399 payload 274 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 -- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 msgr2=0x7f7150131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 --2- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 0x7f7150131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 -- 172.21.15.153:0/2141199864 >> v2:172.21.15.181:3300/0 conn(0x7f715009ea50 msgr2=0x7f715009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 --2- 172.21.15.153:0/2141199864 >> v2:172.21.15.181:3300/0 conn(0x7f715009ea50 0x7f715009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714f7fe640 1 -- 172.21.15.153:0/2141199864 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71500a1210 con 0x7f7150068d90 2024-09-15T03:35:07.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.819+0000 7f714effd640 1 --2- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 0x7f7150131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:07.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714f7fe640 1 -- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 msgr2=0x7f71500691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:07.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714f7fe640 1 --2- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f71500691f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7150068160 tx=0x7f7148009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51b7dd1cb57499b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:07.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 1 -- 172.21.15.153:0/2141199864 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148007610 con 0x7f7150068d90 2024-09-15T03:35:07.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:07.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: dump: 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:07.835 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 1 -- 172.21.15.153:0/2141199864 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714802d050 con 0x7f7150068d90 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f7155900640 5 monclient: authenticate success, global_id 84399 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f7155900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f7155900640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f7155900640 4 set_mon_vals no callback set 2024-09-15T03:35:07.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 1 -- 172.21.15.153:0/2141199864 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480079e0 con 0x7f7150068d90 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.822+0000 7f714dffb640 10 monclient: dump: 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:07.837 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 10 monclient: shutdown 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 -- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 msgr2=0x7f71500691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 --2- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f71500691f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7150068160 tx=0x7f7148009840 comp rx=0 tx=0).stop 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 -- 172.21.15.153:0/2141199864 shutdown_connections 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 --2- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f71500691f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 --2- 172.21.15.153:0/2141199864 >> v2:172.21.15.181:3300/0 conn(0x7f715009ea50 0x7f715009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 --2- 172.21.15.153:0/2141199864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71501318f0 0x7f7150131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 -- 172.21.15.153:0/2141199864 >> 172.21.15.153:0/2141199864 conn(0x7f7150067110 msgr2=0x7f7150067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 -- 172.21.15.153:0/2141199864 shutdown_connections 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.823+0000 7f7155900640 1 -- 172.21.15.153:0/2141199864 wait complete. 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.824+0000 7f7155900640 10 monclient: build_initial_monmap 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.824+0000 7f7155900640 10 monclient: monmap: 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:07.839 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:07.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:07.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:07.840 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.824+0000 7f7155900640 1 Processor -- start 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.824+0000 7f7155900640 1 -- start start 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.824+0000 7f7155900640 10 monclient: init 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f715009ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): picked mon.noname-a con 0x7f7150068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f715009ea50 0x7f7150130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): picked mon.noname-c con 0x7f715009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:07.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501318f0 0x7f715011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): picked mon.noname-b con 0x7f71501318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71500a4150 con 0x7f7150068d90 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715011e180 con 0x7f71501318f0 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715011dea0 con 0x7f715009ea50 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:07.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f7155900640 10 monclient(hunting): authenticate will time out at 1105.465135s 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f715009ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f715009ea50 0x7f7150130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f715009ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57646/0 (socket says 172.21.15.153:57646) 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 -- 172.21.15.153:0/1214677112 learned_addr learned my addr 172.21.15.153:0/1214677112 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): get_auth_request con 0x7f7150068d90 auth_method 0 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:07.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): get_auth_request con 0x7f715009ea50 auth_method 0 2024-09-15T03:35:07.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:07.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:07.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:07.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714f7fe640 1 -- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501318f0 msgr2=0x7f715011b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:07.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714f7fe640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501318f0 0x7f715011b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:07.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:07.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:07.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:07.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:07.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient(hunting): handle_auth_done global_id 84402 payload 824 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 -- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f715009ea50 msgr2=0x7f7150130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f715009ea50 0x7f7150130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 -- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501318f0 msgr2=0x7f715011b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501318f0 0x7f715011b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 monclient: found mon.noname-a 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714effd640 1 -- 172.21.15.153:0/1214677112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140009390 con 0x7f7150068d90 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.825+0000 7f714e7fc640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f715009ea50 0x7f7150130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:07.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f714effd640 1 -- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 msgr2=0x7f715009ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f714effd640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f715009ab60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f714000ec80 tx=0x7f71400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714000eec0 con 0x7f7150068d90 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: dump: 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714000cdd0 con 0x7f7150068d90 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 5 monclient: authenticate success, global_id 84402 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140013680 con 0x7f7150068d90 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 10 monclient: _renew_subs 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7148009ae0 con 0x7f7150068d90 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:07.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7136ffd640 10 monclient: dump: 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f714d7fa640 4 set_mon_vals no callback set 2024-09-15T03:35:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 10 monclient: _renew_subs 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.827+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715011dbc0 con 0x7f7150068d90 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.828+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f71400139f0 con 0x7f7150068d90 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f7155900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f7155900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f7155900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715009b640 con 0x7f7150068d90 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f7136ffd640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f713c0386f0 0x7f713c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f714004cfc0 con 0x7f7150068d90 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f714e7fc640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f713c0386f0 0x7f713c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.829+0000 7f714e7fc640 10 monclient: get_auth_request con 0x7f713c0386f0 auth_method 0 2024-09-15T03:35:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.832+0000 7f714e7fc640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f713c0386f0 0x7f713c03aba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7150066560 tx=0x7f71440090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:07.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.833+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f71400234d0 con 0x7f7150068d90 2024-09-15T03:35:07.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.833+0000 7f7136ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:07.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:07.833+0000 7f7136ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.047+0000 7f7155900640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.047+0000 7f7155900640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.047+0000 7f7155900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.047+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f715004bf60 con 0x7f7150068d90 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.048+0000 7f7136ffd640 1 -- 172.21.15.153:0/1214677112 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f71400232f0 con 0x7f7150068d90 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.048+0000 7f7136ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.048+0000 7f7136ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:08.050 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:08.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f713c0386f0 msgr2=0x7f713c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:08.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f713c0386f0 0x7f713c03aba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7150066560 tx=0x7f71440090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:08.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 10 monclient: shutdown 2024-09-15T03:35:08.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:08.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 msgr2=0x7f715009ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:08.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f715009ab60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f714000ec80 tx=0x7f71400090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 shutdown_connections 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f713c0386f0 0x7f713c03aba0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501318f0 0x7f715011b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f715009ea50 0x7f7150130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 --2- 172.21.15.153:0/1214677112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7150068d90 0x7f715009ab60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 >> 172.21.15.153:0/1214677112 conn(0x7f7150067110 msgr2=0x7f71500a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 shutdown_connections 2024-09-15T03:35:08.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:08.050+0000 7f7155900640 1 -- 172.21.15.153:0/1214677112 wait complete. 2024-09-15T03:35:08.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:08.458+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:08.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:08.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:09.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:09.458+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:09.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:09.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:10.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:10.458+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:10.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:10.458+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:11.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.166+0000 7f527739d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.166+0000 7f527739d640 10 monclient: build_initial_monmap 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.166+0000 7f527739d640 10 monclient: monmap: 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:11.167919+0000 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:11.167919+0000 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:11.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:11.169 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:11.169 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:11.169 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.166+0000 7f527739d640 1 Processor -- start 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.166+0000 7f527739d640 1 -- start start 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.166+0000 7f527739d640 10 monclient: init 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5270131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 0x7f527009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): picked mon.noname-c con 0x7f527009e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): picked mon.noname-b con 0x7f527009f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527009faf0 con 0x7f5270131ab0 2024-09-15T03:35:11.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5270132590 con 0x7f527009e910 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527009fd30 con 0x7f527009f250 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527739d640 10 monclient(hunting): authenticate will time out at 1108.807515s 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f5275139640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5270131ab0 0x7f5270131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:49288/0 (socket says 172.21.15.153:49288) 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f5275139640 1 -- 172.21.15.153:0/1696152251 learned_addr learned my addr 172.21.15.153:0/1696152251 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 10 monclient(hunting): get_auth_request con 0x7f527009f250 auth_method 0 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 msgr2=0x7f527009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:11.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 0x7f527009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:11.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:11.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:11.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:11.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.167+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:11.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.169+0000 7f5275139640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.153:6789/0 conn(0x7f5270131ab0 legacy=0x7f5270131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 msgr2=0x7f527009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 0x7f527009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 10 monclient(hunting): get_auth_request con 0x7f527009f250 auth_method 0 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:11.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:11.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:11.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.368+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:11.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:11.459+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:11.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:11.459+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 msgr2=0x7f527009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 0x7f527009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 10 monclient(hunting): get_auth_request con 0x7f527009f250 auth_method 0 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:11.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:11.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:11.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:11.769+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:12.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:12.459+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:12.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:12.459+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:12.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 msgr2=0x7f527009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:12.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 0x7f527009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 10 monclient(hunting): get_auth_request con 0x7f527009f250 auth_method 0 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:12.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:12.570+0000 7f527593a640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:35:13.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:13.459+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:13.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:13.459+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:13.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:13.459+0000 7fad267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient(hunting): tick 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 msgr2=0x7f5270138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f527009f250 0x7f5270138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 msgr2=0x7f527009ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.181:3300/0 conn(0x7f527009e910 0x7f527009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.153:6789/0 conn(0x7f5270131ab0 legacy=0x7f5270131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient: _add_conns ranks=[1,3] 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5254003700 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f52540073a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 -- 172.21.15.153:0/1696152251 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52700a1110 con 0x7f5254003700 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 1 -- 172.21.15.153:0/1696152251 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5270132590 con 0x7f52540073a0 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.167+0000 7f5262ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5275139640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.181:6789/0 conn(0x7f5254003700 legacy=0x7f5254005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 10 monclient(hunting): get_auth_request con 0x7f52540073a0 auth_method 0 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 msgr2=0x7f5254009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 msgr2=0x7f5254009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.168+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5275139640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.181:6789/0 conn(0x7f5254003700 legacy=0x7f5254005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 10 monclient(hunting): get_auth_request con 0x7f52540073a0 auth_method 0 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 msgr2=0x7f5254009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 msgr2=0x7f5254009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.369+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:14.459+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:14.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:14.459+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:14.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5275139640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.181:6789/0 conn(0x7f5254003700 legacy=0x7f5254005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:14.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:14.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 10 monclient(hunting): get_auth_request con 0x7f52540073a0 auth_method 0 2024-09-15T03:35:14.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:14.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:14.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:14.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 msgr2=0x7f5254009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:35:14.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 msgr2=0x7f5254009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:14.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:14.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:14.770+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:15.459+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:15.459+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:15.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5275139640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.181:6789/0 conn(0x7f5254003700 legacy=0x7f5254005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> v2:172.21.15.153:3300/0 conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): get_auth_request con 0x7f52540073a0 auth_method 0 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient(hunting): handle_auth_done global_id 84405 payload 274 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> v1:172.21.15.181:6789/0 conn(0x7f5254003700 legacy=0x7f5254005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient: _finish_hunting 0 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 1 monclient: found mon.noname-a 2024-09-15T03:35:15.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:15.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.571+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525400af60 con 0x7f52540073a0 2024-09-15T03:35:15.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.574+0000 7f5274938640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 msgr2=0x7f5254009850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.574+0000 7f5274938640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5254009850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f52640038f0 tx=0x7f526400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c009569c4b4fee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 1 -- 172.21.15.153:0/1696152251 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264033430 con 0x7f52540073a0 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: dump: 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.577 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 1 -- 172.21.15.153:0/1696152251 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5264033590 con 0x7f52540073a0 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 5 monclient: authenticate success, global_id 84405 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 1 -- 172.21.15.153:0/1696152251 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264033930 con 0x7f52540073a0 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 4 set_mon_vals no callback set 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f5263fff640 10 monclient: dump: 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 10 monclient: shutdown 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 1 -- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 msgr2=0x7f5254009850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527739d640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5254009850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f52640038f0 tx=0x7f526400b040 comp rx=0 tx=0).stop 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.575+0000 7f527593a640 1 -- 172.21.15.153:0/1696152251 reap_dead start 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 1 -- 172.21.15.153:0/1696152251 shutdown_connections 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 1 --2- 172.21.15.153:0/1696152251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5254009850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 1 -- 172.21.15.153:0/1696152251 >> 172.21.15.153:0/1696152251 conn(0x7f5270067d30 msgr2=0x7f52700a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 1 -- 172.21.15.153:0/1696152251 shutdown_connections 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 1 -- 172.21.15.153:0/1696152251 wait complete. 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 10 monclient: build_initial_monmap 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.576+0000 7f527739d640 10 monclient: monmap: 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:15.585 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 Processor -- start 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 -- start start 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient: init 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:15.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5270098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): picked mon.noname-a con 0x7f52540073a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5270131070 0x7f5270096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): picked mon.noname-b con 0x7f5270131070 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5270096fa0 0x7f52700973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): picked mon.noname-c con 0x7f5270096fa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5270097930 con 0x7f52540073a0 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5270099f70 con 0x7f5270131070 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5270131440 con 0x7f5270096fa0 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:15.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.577+0000 7f527739d640 10 monclient(hunting): authenticate will time out at 1113.217960s 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5270098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5274938640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5270131070 msgr2=0x7f5270096a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5270096fa0 0x7f52700973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5274938640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5270131070 0x7f5270096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5270098180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43250/0 (socket says 172.21.15.153:43250) 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 -- 172.21.15.153:0/1283068277 learned_addr learned my addr 172.21.15.153:0/1283068277 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): get_auth_request con 0x7f52540073a0 auth_method 0 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): get_auth_request con 0x7f5270096fa0 auth_method 0 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient(hunting): handle_auth_done global_id 84408 payload 824 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 -- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5270096fa0 msgr2=0x7f52700973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5270096fa0 0x7f52700973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 -- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5270131070 msgr2=0x7f5270096a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5270131070 0x7f5270096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient: _finish_hunting 0 2024-09-15T03:35:15.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 monclient: found mon.noname-a 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f5275139640 1 -- 172.21.15.153:0/1283068277 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5264009a70 con 0x7f52540073a0 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.578+0000 7f527593a640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5270096fa0 0x7f52700973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.580+0000 7f5275139640 1 -- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 msgr2=0x7f5270098180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.580+0000 7f5275139640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5270098180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c790 tx=0x7f526c007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.580+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c005aa0 con 0x7f52540073a0 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.580+0000 7f52617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: dump: 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.590 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 5 monclient: authenticate success, global_id 84408 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 10 monclient: _renew_subs 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c005c00 con 0x7f52540073a0 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527009ab20 con 0x7f52540073a0 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c0070f0 con 0x7f52540073a0 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52617fa640 10 monclient: dump: 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.592 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f52637fe640 4 set_mon_vals no callback set 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 10 monclient: _renew_subs 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.581+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52700988a0 con 0x7f52540073a0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f526c0074e0 con 0x7f52540073a0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f527739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f527739d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f527739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527004bf60 con 0x7f52540073a0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f52617fa640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5254039470 0x7f525403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.582+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f526c04c7a0 con 0x7f52540073a0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.583+0000 7f5274938640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5254039470 0x7f525403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.583+0000 7f5274938640 10 monclient: get_auth_request con 0x7f5254039470 auth_method 0 2024-09-15T03:35:15.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.585+0000 7f5274938640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5254039470 0x7f525403b920 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5254003670 tx=0x7f5264035000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.587+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f526c00a640 con 0x7f52540073a0 2024-09-15T03:35:15.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.587+0000 7f52617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:15.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.587+0000 7f52617fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:15.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f527739d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:15.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f527739d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:15.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f527739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5270099bf0 con 0x7f52540073a0 2024-09-15T03:35:15.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f52617fa640 1 -- 172.21.15.153:0/1283068277 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f526c010770 con 0x7f52540073a0 2024-09-15T03:35:15.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f52617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:15.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.808+0000 7f52617fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:15.811 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":72,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:15.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5254039470 msgr2=0x7f525403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5254039470 0x7f525403b920 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5254003670 tx=0x7f5264035000 comp rx=0 tx=0).stop 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 10 monclient: shutdown 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 msgr2=0x7f5270098180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5270098180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c790 tx=0x7f526c007d20 comp rx=0 tx=0).stop 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 shutdown_connections 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5254039470 0x7f525403b920 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5270096fa0 0x7f52700973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5270131070 0x7f5270096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 --2- 172.21.15.153:0/1283068277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52540073a0 0x7f5270098180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 >> 172.21.15.153:0/1283068277 conn(0x7f5270067d30 msgr2=0x7f52700a19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 shutdown_connections 2024-09-15T03:35:15.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.810+0000 7f527739d640 1 -- 172.21.15.153:0/1283068277 wait complete. 2024-09-15T03:35:15.823 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:35:15.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:35:15.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:35:15.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.915+0000 7fbbcf15d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.915+0000 7fbbcf15d640 10 monclient: build_initial_monmap 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.915+0000 7fbbcf15d640 10 monclient: monmap: 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:15.917591+0000 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:15.917591+0000 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:15.918 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:15.919 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:15.919 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:15.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.916+0000 7fbbcf15d640 1 Processor -- start 2024-09-15T03:35:15.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.916+0000 7fbbcf15d640 1 -- start start 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.916+0000 7fbbcf15d640 10 monclient: init 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbbc80975f0 0x7fbbc8095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc80975f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 0x7fbbc8097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc80979c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbc809bf50 0x7fbbc809c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc809bf50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc809d700 con 0x7fbbc80975f0 2024-09-15T03:35:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc809d1b0 con 0x7fbbc809bf50 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc809d420 con 0x7fbbc80979c0 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcf15d640 10 monclient(hunting): authenticate will time out at 1113.557153s 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcd6fa640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fbbc809bf50 msgr2=0x7fbbc809c3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbcd6fa640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbc809bf50 0x7fbbc809c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 0x7fbbc8097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbbc80975f0 0x7fbbc8095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 0x7fbbc8097dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37582/0 (socket says 172.21.15.153:37582) 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 1 -- 172.21.15.153:0/4212117534 learned_addr learned my addr 172.21.15.153:0/4212117534 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): get_auth_request con 0x7fbbc80979c0 auth_method 0 2024-09-15T03:35:15.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): get_auth_request con 0x7fbbc80975f0 auth_method 0 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient(hunting): handle_auth_done global_id 84411 payload 274 2024-09-15T03:35:15.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 -- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 msgr2=0x7fbbc8097dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 --2- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 0x7fbbc8097dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 -- 172.21.15.153:0/4212117534 >> v2:172.21.15.181:3300/0 conn(0x7fbbc809bf50 msgr2=0x7fbbc809c3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 --2- 172.21.15.153:0/4212117534 >> v2:172.21.15.181:3300/0 conn(0x7fbbc809bf50 0x7fbbc809c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient: _finish_hunting 0 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 monclient: found mon.noname-a 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbccef9640 1 -- 172.21.15.153:0/4212117534 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc809d9e0 con 0x7fbbc80975f0 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.917+0000 7fbbc7fff640 1 --2- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 0x7fbbc8097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbccef9640 1 -- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80975f0 msgr2=0x7fbbc8095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbccef9640 1 --2- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80975f0 0x7fbbc8095ad0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbbc809ad50 tx=0x7fbbb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fa1a6f37d470ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 1 -- 172.21.15.153:0/4212117534 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8007760 con 0x7fbbc80975f0 2024-09-15T03:35:15.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: dump: 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 1 -- 172.21.15.153:0/4212117534 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb802d050 con 0x7fbbc80975f0 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbcf15d640 5 monclient: authenticate success, global_id 84411 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbcf15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbcf15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 1 -- 172.21.15.153:0/4212117534 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8007af0 con 0x7fbbc80975f0 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbcf15d640 4 set_mon_vals no callback set 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.920+0000 7fbbc77fe640 10 monclient: dump: 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.931 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 10 monclient: shutdown 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 -- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80975f0 msgr2=0x7fbbc8095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80975f0 0x7fbbc8095ad0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbbc809ad50 tx=0x7fbbb8009780 comp rx=0 tx=0).stop 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 -- 172.21.15.153:0/4212117534 shutdown_connections 2024-09-15T03:35:15.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/4212117534 >> v2:172.21.15.181:3300/0 conn(0x7fbbc809bf50 0x7fbbc809c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc80979c0 0x7fbbc8097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/4212117534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80975f0 0x7fbbc8095ad0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 -- 172.21.15.153:0/4212117534 >> 172.21.15.153:0/4212117534 conn(0x7fbbc80678a0 msgr2=0x7fbbc80a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 -- 172.21.15.153:0/4212117534 shutdown_connections 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 1 -- 172.21.15.153:0/4212117534 wait complete. 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 10 monclient: build_initial_monmap 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.921+0000 7fbbcf15d640 10 monclient: monmap: 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 Processor -- start 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 -- start start 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient: init 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbc80975f0 0x7fbbc813eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc80975f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 0x7fbbc812f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc80979c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc809bf50 0x7fbbc8126630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc809bf50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc80983c0 con 0x7fbbc80979c0 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc806eb20 con 0x7fbbc80975f0 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81286e0 con 0x7fbbc809bf50 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcf15d640 10 monclient(hunting): authenticate will time out at 1113.562788s 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 0x7fbbc812f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbccef9640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbc80975f0 msgr2=0x7fbbc813eea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc809bf50 0x7fbbc8126630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbccef9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbc80975f0 0x7fbbc813eea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 0x7fbbc812f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43270/0 (socket says 172.21.15.153:43270) 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 1 -- 172.21.15.153:0/3975705012 learned_addr learned my addr 172.21.15.153:0/3975705012 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): get_auth_request con 0x7fbbc80979c0 auth_method 0 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): get_auth_request con 0x7fbbc809bf50 auth_method 0 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.922+0000 7fbbcd6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_done global_id 84414 payload 824 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc809bf50 msgr2=0x7fbbc8126630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbc809bf50 0x7fbbc8126630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbc80975f0 msgr2=0x7fbbc813eea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbc80975f0 0x7fbbc813eea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 1 monclient: found mon.noname-a 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.923+0000 7fbbc7fff640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc009450 con 0x7fbbc80979c0 2024-09-15T03:35:15.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc7fff640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 msgr2=0x7fbbc812f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:15.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc7fff640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 0x7fbbc812f820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00ec80 tx=0x7fbbbc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc009730 con 0x7fbbc80979c0 2024-09-15T03:35:15.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: dump: 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc00eec0 con 0x7fbbc80979c0 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 5 monclient: authenticate success, global_id 84414 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 10 monclient: _renew_subs 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb8009a20 con 0x7fbbc80979c0 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc013470 con 0x7fbbc80979c0 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc4ff9640 10 monclient: dump: 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.944 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbc6ffd640 4 set_mon_vals no callback set 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 10 monclient: _renew_subs 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.925+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc8127f50 con 0x7fbbc80979c0 2024-09-15T03:35:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbcf15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbbbc013820 con 0x7fbbc80979c0 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 0x7fbbc8126b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8065e00 con 0x7fbbc812fd60 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 0x7fbbc8126b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient: get_auth_request con 0x7fbbc812fd60 auth_method 0 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbc4ff9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbb4038740 0x7fbbb403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fbbbc04cfe0 con 0x7fbbc80979c0 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 10 monclient(hunting): handle_auth_done global_id 84417 payload 824 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbccef9640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbc809aa50 con 0x7fbbc812fd60 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbcd6fa640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbb4038740 0x7fbbb403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.926+0000 7fbbcd6fa640 10 monclient: get_auth_request con 0x7fbbb4038740 auth_method 0 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.928+0000 7fbbccef9640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 msgr2=0x7fbbc8126b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.928+0000 7fbbccef9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 0x7fbbc8126b70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8034000 tx=0x7fbbb8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8031910 con 0x7fbbc812fd60 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 10 monclient: dump: 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbbb8040070 con 0x7fbbc812fd60 2024-09-15T03:35:15.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 msgr2=0x7fbbc8126b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 0x7fbbc8126b70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8034000 tx=0x7fbbb8037000 comp rx=0 tx=0).stop 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 msgr2=0x7fbbc8126b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbc4ff9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc812fd60 0x7fbbc8126b70 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8034000 tx=0x7fbbb8037000 comp rx=0 tx=0).stop 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.929+0000 7fbbcd6fa640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbb4038740 0x7fbbb403abf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00038f0 tx=0x7fbbc000d500 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.934+0000 7fbbcf15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.934+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 0x7fbbc806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.934+0000 7fbbcf15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.934+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc813f4a0 con 0x7fbbc81270b0 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 0x7fbbc806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient: get_auth_request con 0x7fbbc81270b0 auth_method 0 2024-09-15T03:35:15.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 10 monclient(hunting): handle_auth_done global_id 84420 payload 824 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.935+0000 7fbbccef9640 1 -- 172.21.15.153:0/3975705012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbbc81274b0 con 0x7fbbc81270b0 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbccef9640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 msgr2=0x7fbbc806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbccef9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 0x7fbbc806a7d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80097c0 tx=0x7fbbb803e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb803c840 con 0x7fbbc81270b0 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:15.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 10 monclient: dump: 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fbbb803c9a0 con 0x7fbbc81270b0 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 msgr2=0x7fbbc806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 0x7fbbc806a7d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80097c0 tx=0x7fbbb803e070 comp rx=0 tx=0).stop 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 msgr2=0x7fbbc806a7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.937+0000 7fbbc4ff9640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc81270b0 0x7fbbc806a7d0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80097c0 tx=0x7fbbb803e070 comp rx=0 tx=0).stop 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 72, 2024-09-15T03:35:15.952 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:15.953 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:15.954 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:15.955 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:15.956 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.957 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:15.958 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:15.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:15.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:15.961 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.963 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:15.964 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:15.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:15.965 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:15.965 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:15.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbb4038740 msgr2=0x7fbbb403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbb4038740 0x7fbbb403abf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00038f0 tx=0x7fbbc000d500 comp rx=0 tx=0).stop 2024-09-15T03:35:15.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 10 monclient: shutdown 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 msgr2=0x7fbbc812f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 --2- 172.21.15.153:0/3975705012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbc80979c0 0x7fbbc812f820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00ec80 tx=0x7fbbbc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcd6fa640 1 -- 172.21.15.153:0/3975705012 reap_dead start 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcd6fa640 1 -- 172.21.15.153:0/3975705012 reap_dead start 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 shutdown_connections 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 >> 172.21.15.153:0/3975705012 conn(0x7fbbc80678a0 msgr2=0x7fbbc8096290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 shutdown_connections 2024-09-15T03:35:15.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:15.938+0000 7fbbcf15d640 1 -- 172.21.15.153:0/3975705012 wait complete. 2024-09-15T03:35:15.967 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:35:15.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:35:15.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:35:16.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.059+0000 7f4beb20f640 10 monclient: get_monmap_and_config 2024-09-15T03:35:16.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.059+0000 7f4beb20f640 10 monclient: build_initial_monmap 2024-09-15T03:35:16.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.059+0000 7f4beb20f640 10 monclient: monmap: 2024-09-15T03:35:16.061 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:16.061 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:16.061163+0000 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:16.061163+0000 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:16.062 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 Processor -- start 2024-09-15T03:35:16.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 -- start start 2024-09-15T03:35:16.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient: init 2024-09-15T03:35:16.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4be4096e10 0x7f4be409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): picked mon.noname-a con 0x7f4be4096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4be409aaa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4be40978f0 0x7f4be4135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): picked mon.noname-c con 0x7f4be40978f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be409d4a0 con 0x7f4be409aaa0 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be4099620 con 0x7f4be4096e10 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be4099340 con 0x7f4be40978f0 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:16.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4beb20f640 10 monclient(hunting): authenticate will time out at 1113.700680s 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4be4096e10 0x7f4be409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be97ac640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f4be40978f0 msgr2=0x7f4be4135b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4be4096e10 0x7f4be409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43290/0 (socket says 172.21.15.153:43290) 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 1 -- 172.21.15.153:0/1826867413 learned_addr learned my addr 172.21.15.153:0/1826867413 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be97ac640 1 --2- 172.21.15.153:0/1826867413 >> v2:172.21.15.181:3300/0 conn(0x7f4be40978f0 0x7f4be4135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): get_auth_request con 0x7f4be4096e10 auth_method 0 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be3fff640 1 -- 172.21.15.153:0/1826867413 >> v1:172.21.15.181:6789/0 conn(0x7f4be409aaa0 legacy=0x7f4be40971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:16.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.060+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 10 monclient(hunting): handle_auth_done global_id 84423 payload 274 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 1 -- 172.21.15.153:0/1826867413 >> v2:172.21.15.181:3300/0 conn(0x7f4be40978f0 msgr2=0x7f4be4135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1826867413 >> v2:172.21.15.181:3300/0 conn(0x7f4be40978f0 0x7f4be4135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 1 -- 172.21.15.153:0/1826867413 >> v1:172.21.15.181:6789/0 conn(0x7f4be409aaa0 legacy=0x7f4be40971e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 10 monclient: _finish_hunting 0 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 1 monclient: found mon.noname-a 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.061+0000 7f4be8fab640 1 -- 172.21.15.153:0/1826867413 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4099900 con 0x7f4be4096e10 2024-09-15T03:35:16.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.063+0000 7f4be8fab640 1 -- 172.21.15.153:0/1826867413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 msgr2=0x7f4be409a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1826867413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be409a560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4be409d0e0 tx=0x7f4bd800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2b3a1ad73c03ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 1 -- 172.21.15.153:0/1826867413 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8032840 con 0x7f4be4096e10 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: dump: 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 5 monclient: authenticate success, global_id 84423 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 1 -- 172.21.15.153:0/1826867413 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd80329a0 con 0x7f4be4096e10 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 1 -- 172.21.15.153:0/1826867413 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8032d80 con 0x7f4be4096e10 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 4 set_mon_vals no callback set 2024-09-15T03:35:16.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4be37fe640 10 monclient: dump: 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 10 monclient: shutdown 2024-09-15T03:35:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 1 -- 172.21.15.153:0/1826867413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 msgr2=0x7f4be409a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1826867413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be409a560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4be409d0e0 tx=0x7f4bd800b460 comp rx=0 tx=0).stop 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 1 -- 172.21.15.153:0/1826867413 shutdown_connections 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1826867413 >> v2:172.21.15.181:3300/0 conn(0x7f4be40978f0 0x7f4be4135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1826867413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be409a560 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.064+0000 7f4beb20f640 1 -- 172.21.15.153:0/1826867413 >> 172.21.15.153:0/1826867413 conn(0x7f4be40672b0 msgr2=0x7f4be40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.065+0000 7f4beb20f640 1 -- 172.21.15.153:0/1826867413 shutdown_connections 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.065+0000 7f4beb20f640 1 -- 172.21.15.153:0/1826867413 wait complete. 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.065+0000 7f4beb20f640 10 monclient: build_initial_monmap 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.065+0000 7f4beb20f640 10 monclient: monmap: 2024-09-15T03:35:16.077 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.065+0000 7f4beb20f640 1 Processor -- start 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 -- start start 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient: init 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be412e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): picked mon.noname-a con 0x7f4be4096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 0x7f4be4118200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): picked mon.noname-c con 0x7f4be40978f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be409aaa0 0x7f4be4118740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): picked mon.noname-b con 0x7f4be409aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be411bb80 con 0x7f4be4096e10 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be409b050 con 0x7f4be409aaa0 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be4130700 con 0x7f4be40978f0 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4beb20f640 10 monclient(hunting): authenticate will time out at 1113.706612s 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be412e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 0x7f4be4118200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be412e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43306/0 (socket says 172.21.15.153:43306) 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be97ac640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be409aaa0 msgr2=0x7f4be4118740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 1 -- 172.21.15.153:0/1495209520 learned_addr learned my addr 172.21.15.153:0/1495209520 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): get_auth_request con 0x7f4be4096e10 auth_method 0 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be97ac640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be409aaa0 0x7f4be4118740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): get_auth_request con 0x7f4be40978f0 auth_method 0 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 10 monclient(hunting): handle_auth_done global_id 74519 payload 824 2024-09-15T03:35:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be409aaa0 msgr2=0x7f4be4118740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.066+0000 7f4be3fff640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be409aaa0 0x7f4be4118740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 msgr2=0x7f4be412e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be412e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 1 monclient: found mon.noname-c 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be3fff640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd800b700 con 0x7f4be40978f0 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.067+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be4096e10 0x7f4be412e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be3fff640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 msgr2=0x7f4be4118200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be3fff640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 0x7f4be4118200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b5b0 tx=0x7f4bd00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be3fff640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 0x7f4be4118200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b5b0 tx=0x7f4bd00090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f4bd0003d90 mon_map magic: 0 v1 2024-09-15T03:35:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0003d90 con 0x7f4be40978f0 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 10 monclient: dump: 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:16.083 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd0006150 con 0x7f4be40978f0 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 5 monclient: authenticate success, global_id 74519 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 10 monclient: _renew_subs 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be411b8a0 con 0x7f4be40978f0 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4be2ffd640 4 set_mon_vals no callback set 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 10 monclient: _renew_subs 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.069+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be4132000 con 0x7f4be40978f0 2024-09-15T03:35:16.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4beb20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 0x7f4be4118c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be4065f20 con 0x7f4be412ecc0 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0006510 con 0x7f4be40978f0 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 0x7f4be4118c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be0ff9640 10 monclient: dump: 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4bd001a360 con 0x7f4be40978f0 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient: get_auth_request con 0x7f4be412ecc0 auth_method 0 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:16.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:16.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 10 monclient(hunting): handle_auth_done global_id 74522 payload 824 2024-09-15T03:35:16.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be8fab640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4be411bdc0 con 0x7f4be412ecc0 2024-09-15T03:35:16.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be0ff9640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bc8038740 0x7f4bc803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be97ac640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bc8038740 0x7f4bc803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:16.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.070+0000 7f4be97ac640 10 monclient: get_auth_request con 0x7f4bc8038740 auth_method 0 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.072+0000 7f4be8fab640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 msgr2=0x7f4be4118c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.072+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 0x7f4be4118c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800b4a0 tx=0x7f4bd803a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8044070 con 0x7f4be412ecc0 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 10 monclient: dump: 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.090 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4bd803f840 con 0x7f4be412ecc0 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 msgr2=0x7f4be4118c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 0x7f4be4118c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800b4a0 tx=0x7f4bd803a000 comp rx=0 tx=0).stop 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 msgr2=0x7f4be4118c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be0ff9640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be412ecc0 0x7f4be4118c80 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800b4a0 tx=0x7f4bd803a000 comp rx=0 tx=0).stop 2024-09-15T03:35:16.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.073+0000 7f4be97ac640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bc8038740 0x7f4bc803abf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc003c70 tx=0x7f4bdc009310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.078+0000 7f4beb20f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.078+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 0x7f4be406be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.078+0000 7f4beb20f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.078+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be412f090 con 0x7f4be41191c0 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 0x7f4be406be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient: get_auth_request con 0x7f4be41191c0 auth_method 0 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 10 monclient(hunting): handle_auth_done global_id 74525 payload 824 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.079+0000 7f4be8fab640 1 -- 172.21.15.153:0/1495209520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4be41195c0 con 0x7f4be41191c0 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be8fab640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 msgr2=0x7f4be406be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be8fab640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 0x7f4be406be70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8041000 tx=0x7f4bd8044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8048400 con 0x7f4be41191c0 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 10 monclient: dump: 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:16.093 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f4bd8048560 con 0x7f4be41191c0 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 msgr2=0x7f4be406be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 0x7f4be406be70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8041000 tx=0x7f4bd8044000 comp rx=0 tx=0).stop 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 msgr2=0x7f4be406be70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.081+0000 7f4be0ff9640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be41191c0 0x7f4be406be70 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8041000 tx=0x7f4bd8044000 comp rx=0 tx=0).stop 2024-09-15T03:35:16.094 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 72, 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:16.095 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:16.096 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:16.097 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:16.098 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:16.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:16.100 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:16.101 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:16.102 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:16.105 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:16.106 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:16.107 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:16.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:16.108 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:16.108 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:16.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bc8038740 msgr2=0x7f4bc803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:16.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bc8038740 0x7f4bc803abf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc003c70 tx=0x7f4bdc009310 comp rx=0 tx=0).stop 2024-09-15T03:35:16.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 10 monclient: shutdown 2024-09-15T03:35:16.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 msgr2=0x7f4be4118200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 --2- 172.21.15.153:0/1495209520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be40978f0 0x7f4be4118200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b5b0 tx=0x7f4bd00090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4be97ac640 1 -- 172.21.15.153:0/1495209520 reap_dead start 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4be0ff9640 1 -- 172.21.15.153:0/1495209520 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(152..152 src has 106..152) v4 ==== 9160+0+0 (unknown 0 0 0) 0x7f4bd004c110 con 0x7f4be40978f0 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4be97ac640 1 -- 172.21.15.153:0/1495209520 reap_dead start 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 shutdown_connections 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 >> 172.21.15.153:0/1495209520 conn(0x7f4be40672b0 msgr2=0x7f4be412f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 shutdown_connections 2024-09-15T03:35:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:16.082+0000 7f4beb20f640 1 -- 172.21.15.153:0/1495209520 wait complete. 2024-09-15T03:35:16.113 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:35:16.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:35:16.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.370+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v153) v1 ==== 137+0+0 (secure 0 0 0) 0x7fad38029070 con 0x7fad4009f020 2024-09-15T03:35:16.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.370+0000 7fad277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:16.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.370+0000 7fad277fe640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-15T03:35:16.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.459+0000 7fad267fc640 10 monclient: tick 2024-09-15T03:35:16.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.459+0000 7fad267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.471+0000 7fad45b54640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:16.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.471+0000 7fad45b54640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:16.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.471+0000 7fad45b54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:16.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:16.471+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fad4011de30 con 0x7fad4009f020 2024-09-15T03:35:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.117+0000 7fad277fe640 1 -- 172.21.15.181:0/904558051 <== mon.2 v2:172.21.15.153:3301/0 5 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v154) v1 ==== 241+0+0 (secure 0 0 0) 0x7fad380490f0 con 0x7fad4009f020 2024-09-15T03:35:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.117+0000 7fad277fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-15T03:35:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.117+0000 7fad277fe640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-15T03:35:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-15T03:35:17.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 msgr2=0x7fad2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fad34007fd0 tx=0x7fad34007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 10 monclient: shutdown 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 msgr2=0x7fad2c04a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fad380490c0 tx=0x7fad3801cd10 comp rx=0 tx=0).stop 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 shutdown_connections 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4009ec50 0x7fad2c04e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4009f020 0x7fad2c04a850 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad401321a0 0x7fad2c044fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 --2- 172.21.15.181:0/904558051 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 >> 172.21.15.181:0/904558051 conn(0x7fad400a5d40 msgr2=0x7fad4009e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 shutdown_connections 2024-09-15T03:35:17.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.120+0000 7fad45b54640 1 -- 172.21.15.181:0/904558051 wait complete. 2024-09-15T03:35:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep fish 2024-09-15T03:35:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '10\.' 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.210+0000 7fd350b1d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.210+0000 7fd350b1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.210+0000 7fd350b1d640 10 monclient: monmap: 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:17.211006+0000 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:17.211006+0000 2024-09-15T03:35:17.211 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:17.212 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.210+0000 7fd350b1d640 1 Processor -- start 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.210+0000 7fd350b1d640 1 -- start start 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.210+0000 7fd350b1d640 10 monclient: init 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd34c09ec50 0x7fd34c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd34c09ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd34c1321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd34c09f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c067420 con 0x7fd34c1321a0 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c0a0480 con 0x7fd34c09f020 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0a0200 con 0x7fd34c09ec50 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd350b1d640 10 monclient(hunting): authenticate will time out at 1121.692484s 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd34a59c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fd34c09ec50 msgr2=0x7fd34c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd34ad9d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fd34c09f020 legacy=0x7fd34c069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd34a59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd34c09ec50 0x7fd34c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd349d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd34c1321a0 0x7fd34c068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:58392/0 (socket says 172.21.15.181:58392) 2024-09-15T03:35:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.211+0000 7fd349d9b640 1 -- 172.21.15.181:0/2132880531 learned_addr learned my addr 172.21.15.181:0/2132880531 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:17.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd349d9b640 1 -- 172.21.15.181:0/2132880531 >> v1:172.21.15.153:6789/0 conn(0x7fd34c1321a0 legacy=0x7fd34c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952092303 0 0) 0x7fd34c067420 con 0x7fd34c1321a0 2024-09-15T03:35:17.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): my global_id is 84429 2024-09-15T03:35:17.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3340034e0 con 0x7fd34c1321a0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495222506 0 0) 0x7fd3340034e0 con 0x7fd34c1321a0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 >> v2:172.21.15.181:3300/0 conn(0x7fd34c09ec50 msgr2=0x7fd34c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 --2- 172.21.15.181:0/2132880531 >> v2:172.21.15.181:3300/0 conn(0x7fd34c09ec50 0x7fd34c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 >> v1:172.21.15.181:6789/0 conn(0x7fd34c09f020 legacy=0x7fd34c069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c09fd80 con 0x7fd34c1321a0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fd340001720 con 0x7fd34c1321a0 2024-09-15T03:35:17.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd350b1d640 5 monclient: authenticate success, global_id 84429 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd350b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: dump: 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.220 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 >> v1:172.21.15.153:6789/0 conn(0x7fd34c1321a0 legacy=0x7fd34c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3340039d0 0x7fd334003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): picked mon.b con 0x7fd3340039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd334004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): picked mon.a con 0x7fd334004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd334005330 0x7fd33400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): picked mon.c con 0x7fd334005330 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:17.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0a0200 con 0x7fd334004780 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400c290 con 0x7fd3340039d0 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400c450 con 0x7fd334005330 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd350b1d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34a59c640 1 -- 172.21.15.181:0/2132880531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3340039d0 msgr2=0x7fd334003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd34a59c640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3340039d0 0x7fd334003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.214+0000 7fd349d9b640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd334004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): get_auth_request con 0x7fd334004780 auth_method 0 2024-09-15T03:35:17.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd334005330 0x7fd33400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): get_auth_request con 0x7fd334005330 auth_method 0 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient(hunting): handle_auth_done global_id 84429 payload 293 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 1 -- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd334005330 msgr2=0x7fd33400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd334005330 0x7fd33400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 1 -- 172.21.15.181:0/2132880531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3340039d0 msgr2=0x7fd334003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3340039d0 0x7fd334003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 1 monclient: found mon.a 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 1 -- 172.21.15.181:0/2132880531 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3340034e0 con 0x7fd334004780 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd349d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.215+0000 7fd34ad9d640 1 -- 172.21.15.181:0/2132880531 reap_dead start 2024-09-15T03:35:17.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd349d9b640 1 -- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 msgr2=0x7fd334004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd349d9b640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd334004bf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd340001490 tx=0x7fd340007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3ecf0f7a1e8e9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340005890 con 0x7fd334004780 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: dump: 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.232 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd340005a10 con 0x7fd334004780 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 1 -- 172.21.15.181:0/2132880531 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340005dd0 con 0x7fd334004780 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 4 set_mon_vals no callback set 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd34959a640 10 monclient: dump: 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 10 monclient: shutdown 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 -- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 msgr2=0x7fd334004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd334004bf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd340001490 tx=0x7fd340007a90 comp rx=0 tx=0).stop 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 -- 172.21.15.181:0/2132880531 shutdown_connections 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 --2- 172.21.15.181:0/2132880531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd334004bf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 -- 172.21.15.181:0/2132880531 >> 172.21.15.181:0/2132880531 conn(0x7fd34c0a5cf0 msgr2=0x7fd34c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 -- 172.21.15.181:0/2132880531 shutdown_connections 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.218+0000 7fd350b1d640 1 -- 172.21.15.181:0/2132880531 wait complete. 2024-09-15T03:35:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.219+0000 7fd350b1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.219+0000 7fd350b1d640 10 monclient: monmap: 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.219+0000 7fd350b1d640 1 Processor -- start 2024-09-15T03:35:17.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.219+0000 7fd350b1d640 1 -- start start 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.219+0000 7fd350b1d640 10 monclient: init 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd34c09a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd334004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd34c11bc30 0x7fd34c130ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd34c11bc30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd34c131000 0x7fd34c140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd34c131000 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c11c960 con 0x7fd334004780 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c11dec0 con 0x7fd34c131000 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c11c060 con 0x7fd34c11bc30 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd350b1d640 10 monclient(hunting): authenticate will time out at 1121.701123s 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34ad9d640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd34c131000 msgr2=0x7fd34c140d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34ad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd34c131000 0x7fd34c140d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd34c09a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd34c11bc30 0x7fd34c130ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd34c09a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:53506/0 (socket says 172.21.15.181:53506) 2024-09-15T03:35:17.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 -- 172.21.15.181:0/3715096636 learned_addr learned my addr 172.21.15.181:0/3715096636 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): get_auth_request con 0x7fd334004780 auth_method 0 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): get_auth_request con 0x7fd34c11bc30 auth_method 0 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd349d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient(hunting): handle_auth_done global_id 84432 payload 824 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 -- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd34c11bc30 msgr2=0x7fd34c130ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd34c11bc30 0x7fd34c130ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 -- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd34c131000 msgr2=0x7fd34c140d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd34c131000 0x7fd34c140d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 monclient: found mon.noname-a 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.220+0000 7fd34a59c640 1 -- 172.21.15.181:0/3715096636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c11dbe0 con 0x7fd334004780 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.222+0000 7fd34a59c640 1 -- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 msgr2=0x7fd34c09a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.222+0000 7fd34a59c640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd34c09a9b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00b810 tx=0x7fd33c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.222+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c003de0 con 0x7fd334004780 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.222+0000 7fd33a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.222+0000 7fd33a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 10 monclient: dump: 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c006150 con 0x7fd334004780 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 5 monclient: authenticate success, global_id 84432 2024-09-15T03:35:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c006340 con 0x7fd334004780 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd33a7fc640 10 monclient: dump: 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 monclient: _renew_subs 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c11e1a0 con 0x7fd334004780 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd348d99640 4 set_mon_vals no callback set 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 monclient: _renew_subs 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c141280 con 0x7fd334004780 2024-09-15T03:35:17.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.223+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c09b530 con 0x7fd334004780 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.224+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd33c012020 con 0x7fd334004780 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.224+0000 7fd33a7fc640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd32c0386f0 0x7fd32c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.224+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(154..154 src has 106..154) v4 ==== 9825+0+0 (secure 0 0 0) 0x7fd33c04c230 con 0x7fd334004780 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.224+0000 7fd349d9b640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd32c0386f0 0x7fd32c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.224+0000 7fd349d9b640 10 monclient: get_auth_request con 0x7fd32c0386f0 auth_method 0 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.226+0000 7fd349d9b640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd32c0386f0 0x7fd32c03aba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd340001900 tx=0x7fd340034000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.228+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd33c014570 con 0x7fd334004780 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.228+0000 7fd33a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.228+0000 7fd33a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:17.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.390+0000 7fd350b1d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.390+0000 7fd350b1d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.390+0000 7fd350b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.390+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fd34c0008d0 con 0x7fd334004780 2024-09-15T03:35:17.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.391+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fd33c013ad0 con 0x7fd334004780 2024-09-15T03:35:17.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.391+0000 7fd33a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.391+0000 7fd33a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.488+0000 7fd350b1d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.488+0000 7fd350b1d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.488+0000 7fd350b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.488+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fd34c0a13e0 con 0x7fd334004780 2024-09-15T03:35:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.489+0000 7fd33a7fc640 1 -- 172.21.15.181:0/3715096636 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fd33c014340 con 0x7fd334004780 2024-09-15T03:35:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.489+0000 7fd33a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.489+0000 7fd33a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd32c0386f0 msgr2=0x7fd32c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd32c0386f0 0x7fd32c03aba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd340001900 tx=0x7fd340034000 comp rx=0 tx=0).stop 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 10 monclient: shutdown 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 msgr2=0x7fd34c09a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd34c09a9b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00b810 tx=0x7fd33c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 shutdown_connections 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd32c0386f0 0x7fd32c03aba0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd34c131000 0x7fd34c140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd34c11bc30 0x7fd34c130ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 --2- 172.21.15.181:0/3715096636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd334004780 0x7fd34c09a9b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 >> 172.21.15.181:0/3715096636 conn(0x7fd34c0a5cf0 msgr2=0x7fd34c0a4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 shutdown_connections 2024-09-15T03:35:17.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.491+0000 7fd350b1d640 1 -- 172.21.15.181:0/3715096636 wait complete. 2024-09-15T03:35:17.501 INFO:tasks.workunit.client.0.smithi181.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:17.502 INFO:tasks.workunit.client.0.smithi181.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:17.503 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:17.503 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:17.503 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep fish 2024-09-15T03:35:17.503 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '11\.' 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.573+0000 7f97e208b640 10 monclient: get_monmap_and_config 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.573+0000 7f97e208b640 10 monclient: build_initial_monmap 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.573+0000 7f97e208b640 10 monclient: monmap: 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:17.574750+0000 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:17.574750+0000 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.576 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:17.576 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:17.576 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:17.576 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:17.576 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:17.576 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.574+0000 7f97e208b640 1 Processor -- start 2024-09-15T03:35:17.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.574+0000 7f97e208b640 1 -- start start 2024-09-15T03:35:17.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.574+0000 7f97e208b640 10 monclient: init 2024-09-15T03:35:17.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:35:17.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97dc131c00 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:17.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97dc1326e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:17.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 0x7f97dc069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient(hunting): picked mon.noname-b con 0x7f97dc0a0b00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:17.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc09d230 con 0x7f97dc1326e0 2024-09-15T03:35:17.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc0a1b70 con 0x7f97dc131c00 2024-09-15T03:35:17.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc0a08c0 con 0x7f97dc0a0b00 2024-09-15T03:35:17.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97e208b640 10 monclient(hunting): authenticate will time out at 1122.056230s 2024-09-15T03:35:17.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97db7fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f97dc131c00 legacy=0x7f97dc131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:17.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 0x7f97dc069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97daffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f97dc1326e0 0x7f97dc136fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:58400/0 (socket says 172.21.15.181:58400) 2024-09-15T03:35:17.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97daffd640 1 -- 172.21.15.181:0/3988621456 learned_addr learned my addr 172.21.15.181:0/3988621456 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:17.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): get_auth_request con 0x7f97dc0a0b00 auth_method 0 2024-09-15T03:35:17.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient(hunting): handle_auth_done global_id 74531 payload 274 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 1 -- 172.21.15.181:0/3988621456 >> v1:172.21.15.181:6789/0 conn(0x7f97dc131c00 legacy=0x7f97dc131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 1 -- 172.21.15.181:0/3988621456 >> v1:172.21.15.153:6789/0 conn(0x7f97dc1326e0 legacy=0x7f97dc136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:17.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 1 monclient: found mon.noname-b 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.575+0000 7f97dbfff640 1 -- 172.21.15.181:0/3988621456 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc0a0200 con 0x7f97dc0a0b00 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.578+0000 7f97dbfff640 1 -- 172.21.15.181:0/3988621456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 msgr2=0x7f97dc069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.578+0000 7f97dbfff640 1 --2- 172.21.15.181:0/3988621456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 0x7f97dc069d10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009e90 tx=0x7f97d4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f38fb386f085293 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 1 -- 172.21.15.181:0/3988621456 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4007840 con 0x7f97dc0a0b00 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: dump: 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.593 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 1 -- 172.21.15.181:0/3988621456 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d402d050 con 0x7f97dc0a0b00 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 1 -- 172.21.15.181:0/3988621456 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4007bd0 con 0x7f97dc0a0b00 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 5 monclient: authenticate success, global_id 74531 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 4 set_mon_vals no callback set 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97da7fc640 10 monclient: dump: 2024-09-15T03:35:17.594 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 10 monclient: shutdown 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 -- 172.21.15.181:0/3988621456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 msgr2=0x7f97dc069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 --2- 172.21.15.181:0/3988621456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 0x7f97dc069d10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009e90 tx=0x7f97d4009780 comp rx=0 tx=0).stop 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 -- 172.21.15.181:0/3988621456 shutdown_connections 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 --2- 172.21.15.181:0/3988621456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc0a0b00 0x7f97dc069d10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 -- 172.21.15.181:0/3988621456 >> 172.21.15.181:0/3988621456 conn(0x7f97dc071370 msgr2=0x7f97dc0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 -- 172.21.15.181:0/3988621456 shutdown_connections 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.579+0000 7f97e208b640 1 -- 172.21.15.181:0/3988621456 wait complete. 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.580+0000 7f97e208b640 10 monclient: build_initial_monmap 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.580+0000 7f97e208b640 10 monclient: monmap: 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:17.596 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.580+0000 7f97e208b640 1 Processor -- start 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.580+0000 7f97e208b640 1 -- start start 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.580+0000 7f97e208b640 10 monclient: init 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97dc0a0b00 0x7f97dc13a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): picked mon.noname-b con 0x7f97dc0a0b00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 0x7f97dc134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): picked mon.noname-a con 0x7f97dc131c00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc1326e0 0x7f97dc1350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): picked mon.noname-c con 0x7f97dc1326e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1384a0 con 0x7f97dc131c00 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc0a0fd0 con 0x7f97dc0a0b00 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc09f0b0 con 0x7f97dc1326e0 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97e208b640 10 monclient(hunting): authenticate will time out at 1122.062182s 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97db7fe640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97dc0a0b00 msgr2=0x7f97dc13a100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97db7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97dc0a0b00 0x7f97dc13a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc1326e0 0x7f97dc1350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 0x7f97dc134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 0x7f97dc134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:53520/0 (socket says 172.21.15.181:53520) 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc1326e0 0x7f97dc1350e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:60964/0 (socket says 172.21.15.181:60964) 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 -- 172.21.15.181:0/2235149764 learned_addr learned my addr 172.21.15.181:0/2235149764 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:17.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): get_auth_request con 0x7f97dc131c00 auth_method 0 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): get_auth_request con 0x7f97dc1326e0 auth_method 0 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient(hunting): handle_auth_done global_id 84435 payload 824 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 -- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc1326e0 msgr2=0x7f97dc1350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc1326e0 0x7f97dc1350e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 -- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97dc0a0b00 msgr2=0x7f97dc13a100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97dc0a0b00 0x7f97dc13a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 monclient: found mon.noname-a 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.581+0000 7f97daffd640 1 -- 172.21.15.181:0/2235149764 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d4009a20 con 0x7f97dc131c00 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97daffd640 1 -- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 msgr2=0x7f97dc134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97daffd640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 0x7f97dc134ba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c610 tx=0x7f97d0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0005cb0 con 0x7f97dc131c00 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: dump: 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.602 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0005e30 con 0x7f97dc131c00 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 5 monclient: authenticate success, global_id 84435 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 10 monclient: _renew_subs 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc1381c0 con 0x7f97dc131c00 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0007320 con 0x7f97dc131c00 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97c37fe640 10 monclient: dump: 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 10 monclient: _renew_subs 2024-09-15T03:35:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc138e80 con 0x7f97dc131c00 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.584+0000 7f97d9ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f97d001b3b0 con 0x7f97dc131c00 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97e208b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97e208b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97e208b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc0008d0 con 0x7f97dc131c00 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97c37fe640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f97c80386f0 0x7f97c803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(154..154 src has 106..154) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f97d004c390 con 0x7f97dc131c00 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97db7fe640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f97c80386f0 0x7f97c803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.585+0000 7f97db7fe640 10 monclient: get_auth_request con 0x7f97c80386f0 auth_method 0 2024-09-15T03:35:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.587+0000 7f97db7fe640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f97c80386f0 0x7f97c803aba0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f97cc006fd0 tx=0x7f97cc006d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.590+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f97d0018260 con 0x7f97dc131c00 2024-09-15T03:35:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.590+0000 7f97c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.590+0000 7f97c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:17.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.749+0000 7f97e208b640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.749+0000 7f97e208b640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.749+0000 7f97e208b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.749+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f97dc0a0440 con 0x7f97dc131c00 2024-09-15T03:35:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.750+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f97d0017020 con 0x7f97dc131c00 2024-09-15T03:35:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.750+0000 7f97c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.750+0000 7f97c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:17.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.847+0000 7f97e208b640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.847+0000 7f97e208b640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.847+0000 7f97e208b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.847+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f97dc072640 con 0x7f97dc131c00 2024-09-15T03:35:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.848+0000 7f97c37fe640 1 -- 172.21.15.181:0/2235149764 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f97d000ea80 con 0x7f97dc131c00 2024-09-15T03:35:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.848+0000 7f97c37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.848+0000 7f97c37fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:17.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f97c80386f0 msgr2=0x7f97c803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f97c80386f0 0x7f97c803aba0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f97cc006fd0 tx=0x7f97cc006d30 comp rx=0 tx=0).stop 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 10 monclient: shutdown 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 msgr2=0x7f97dc134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 0x7f97dc134ba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c610 tx=0x7f97d0004960 comp rx=0 tx=0).stop 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 shutdown_connections 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f97c80386f0 0x7f97c803aba0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97dc1326e0 0x7f97dc1350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc131c00 0x7f97dc134ba0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 --2- 172.21.15.181:0/2235149764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97dc0a0b00 0x7f97dc13a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 >> 172.21.15.181:0/2235149764 conn(0x7f97dc071370 msgr2=0x7f97dc0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:17.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 shutdown_connections 2024-09-15T03:35:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.850+0000 7f97e208b640 1 -- 172.21.15.181:0/2235149764 wait complete. 2024-09-15T03:35:17.861 INFO:tasks.workunit.client.0.smithi181.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:17.861 INFO:tasks.workunit.client.0.smithi181.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:17.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:17.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:17.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep fish 2024-09-15T03:35:17.862 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '12\.' 2024-09-15T03:35:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.933+0000 7f75c8b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.933+0000 7f75c8b5d640 10 monclient: build_initial_monmap 2024-09-15T03:35:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.933+0000 7f75c8b5d640 10 monclient: monmap: 2024-09-15T03:35:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:17.933903+0000 2024-09-15T03:35:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:17.933903+0000 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:17.936 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:17.937 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.933+0000 7f75c8b5d640 1 Processor -- start 2024-09-15T03:35:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.933+0000 7f75c8b5d640 1 -- start start 2024-09-15T03:35:17.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.933+0000 7f75c8b5d640 10 monclient: init 2024-09-15T03:35:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T03:35:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75c409ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f75c409f020 0x7f75c4069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f75c409f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75c4136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c4067420 con 0x7f75c409ec50 2024-09-15T03:35:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c40a0480 con 0x7f75c4136d60 2024-09-15T03:35:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c40a0200 con 0x7f75c409f020 2024-09-15T03:35:17.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c8b5d640 10 monclient(hunting): authenticate will time out at 1122.415353s 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c2d9d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f75c4136d60 legacy=0x7f75c4139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c1d9b640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f75c409f020 msgr2=0x7f75c4069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c1d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f75c409f020 0x7f75c4069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f75c409ec50 0x7f75c4131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:58408/0 (socket says 172.21.15.181:58408) 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.934+0000 7f75c259c640 1 -- 172.21.15.181:0/2678358064 learned_addr learned my addr 172.21.15.181:0/2678358064 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.936+0000 7f75c259c640 1 -- 172.21.15.181:0/2678358064 >> v1:172.21.15.153:6789/0 conn(0x7f75c409ec50 legacy=0x7f75c4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.936+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127158224 0 0) 0x7f75c4067420 con 0x7f75c409ec50 2024-09-15T03:35:17.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.936+0000 7f75c159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.936+0000 7f75c159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.936+0000 7f75c159a640 10 monclient(hunting): my global_id is 84438 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.936+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ac0034e0 con 0x7f75c409ec50 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232199586 0 0) 0x7f75ac0034e0 con 0x7f75c409ec50 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 >> v2:172.21.15.181:3300/0 conn(0x7f75c409f020 msgr2=0x7f75c4069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 --2- 172.21.15.181:0/2678358064 >> v2:172.21.15.181:3300/0 conn(0x7f75c409f020 0x7f75c4069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 >> v1:172.21.15.181:6789/0 conn(0x7f75c4136d60 legacy=0x7f75c4139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:35:17.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c409fd80 con 0x7f75c409ec50 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f75bc0017e0 con 0x7f75c409ec50 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c8b5d640 5 monclient: authenticate success, global_id 84438 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c8b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: dump: 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.961 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 >> v1:172.21.15.153:6789/0 conn(0x7f75c409ec50 legacy=0x7f75c4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac0039d0 0x7f75ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): picked mon.c con 0x7f75ac0039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac004780 0x7f75ac004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): picked mon.b con 0x7f75ac004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 0x7f75ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): picked mon.a con 0x7f75ac005280 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40a0200 con 0x7f75ac005280 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac00c240 con 0x7f75ac004780 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac00c400 con 0x7f75ac0039d0 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c159a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c8b5d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c1d9b640 1 -- 172.21.15.181:0/2678358064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac004780 msgr2=0x7f75ac004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c1d9b640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac004780 0x7f75ac004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 0x7f75ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac0039d0 0x7f75ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): get_auth_request con 0x7f75ac005280 auth_method 0 2024-09-15T03:35:17.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): get_auth_request con 0x7f75ac0039d0 auth_method 0 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:17.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_done global_id 84438 payload 293 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 -- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac0039d0 msgr2=0x7f75ac003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac0039d0 0x7f75ac003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 -- 172.21.15.181:0/2678358064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac004780 msgr2=0x7f75ac004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac004780 0x7f75ac004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 monclient: found mon.a 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 1 -- 172.21.15.181:0/2678358064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0034e0 con 0x7f75ac005280 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.937+0000 7f75c2d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.938+0000 7f75c2d9d640 1 -- 172.21.15.181:0/2678358064 reap_dead start 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c2d9d640 1 -- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 msgr2=0x7f75ac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 0x7f75ac00b9d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009f90 tx=0x7f75b40095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a090938ee54f4494 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4032430 con 0x7f75ac005280 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: dump: 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.967 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c8b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b402d070 con 0x7f75ac005280 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 1 -- 172.21.15.181:0/2678358064 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4032800 con 0x7f75ac005280 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c159a640 10 monclient: dump: 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.940+0000 7f75c8b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 4 set_mon_vals no callback set 2024-09-15T03:35:17.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 10 monclient: shutdown 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 -- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 msgr2=0x7f75ac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 0x7f75ac00b9d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009f90 tx=0x7f75b40095d0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 -- 172.21.15.181:0/2678358064 shutdown_connections 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/2678358064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac005280 0x7f75ac00b9d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 -- 172.21.15.181:0/2678358064 >> 172.21.15.181:0/2678358064 conn(0x7f75c40a5cf0 msgr2=0x7f75c40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 -- 172.21.15.181:0/2678358064 shutdown_connections 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.941+0000 7f75c8b5d640 1 -- 172.21.15.181:0/2678358064 wait complete. 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient: build_initial_monmap 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient: monmap: 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:17.970 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 Processor -- start 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 -- start start 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient: init 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac005280 0x7f75c4141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f75ac005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75c411d990 0x7f75c4141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f75c411d990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c259c640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac005280 msgr2=0x7f75c4141400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 0x7f75c40999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac005280 0x7f75c4141400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f75c4099560 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75b27fc640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c409c020 con 0x7f75c4099560 2024-09-15T03:35:17.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c409d630 con 0x7f75ac005280 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c411caa0 con 0x7f75c411d990 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.942+0000 7f75c8b5d640 10 monclient(hunting): authenticate will time out at 1122.423979s 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75c411d990 0x7f75c4141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 0x7f75c40999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75c411d990 0x7f75c4141940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:60988/0 (socket says 172.21.15.181:60988) 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 1 -- 172.21.15.181:0/743742744 learned_addr learned my addr 172.21.15.181:0/743742744 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): get_auth_request con 0x7f75c411d990 auth_method 0 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): get_auth_request con 0x7f75c4099560 auth_method 0 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient(hunting): handle_auth_done global_id 84441 payload 824 2024-09-15T03:35:17.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 -- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75c411d990 msgr2=0x7f75c4141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75c411d990 0x7f75c4141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 -- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac005280 msgr2=0x7f75c4141400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac005280 0x7f75c4141400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 monclient: found mon.noname-a 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.943+0000 7f75c2d9d640 1 -- 172.21.15.181:0/743742744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c409b120 con 0x7f75c4099560 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.945+0000 7f75c2d9d640 1 -- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 msgr2=0x7f75c40999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.945+0000 7f75c2d9d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 0x7f75c40999a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f75c409cb90 tx=0x7f75b4035000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40053b0 con 0x7f75c4099560 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: dump: 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.976 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b4005530 con 0x7f75c4099560 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40058f0 con 0x7f75c4099560 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75b27fc640 10 monclient: dump: 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 5 monclient: authenticate success, global_id 84441 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 10 monclient: _renew_subs 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c0d99640 4 set_mon_vals no callback set 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c411c7c0 con 0x7f75c4099560 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 10 monclient: _renew_subs 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.946+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c4141e80 con 0x7f75c4099560 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75c8b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75c8b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c40008d0 con 0x7f75c4099560 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f75b4039070 con 0x7f75c4099560 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75b27fc640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f75a40386f0 0x7f75a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(154..154 src has 106..154) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f75b406e100 con 0x7f75c4099560 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75c259c640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f75a40386f0 0x7f75a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.947+0000 7f75c259c640 10 monclient: get_auth_request con 0x7f75a40386f0 auth_method 0 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.950+0000 7f75c259c640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f75a40386f0 0x7f75a403aba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f75bc001990 tx=0x7f75bc007b30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.952+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f75b4046590 con 0x7f75c4099560 2024-09-15T03:35:17.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.952+0000 7f75b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:17.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:17.952+0000 7f75b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:18.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.112+0000 7f75c8b5d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:18.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.112+0000 7f75c8b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:18.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.112+0000 7f75c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.112+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f75c40a13e0 con 0x7f75c4099560 2024-09-15T03:35:18.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.113+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f75b4041120 con 0x7f75c4099560 2024-09-15T03:35:18.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.113+0000 7f75b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:18.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.113+0000 7f75b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:18.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.209+0000 7f75c8b5d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:18.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.209+0000 7f75c8b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:18.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.209+0000 7f75c8b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.209+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f75c411b880 con 0x7f75c4099560 2024-09-15T03:35:18.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.210+0000 7f75b27fc640 1 -- 172.21.15.181:0/743742744 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f75b4047bb0 con 0x7f75c4099560 2024-09-15T03:35:18.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.210+0000 7f75b27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:18.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.210+0000 7f75b27fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f75a40386f0 msgr2=0x7f75a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f75a40386f0 0x7f75a403aba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f75bc001990 tx=0x7f75bc007b30 comp rx=0 tx=0).stop 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 10 monclient: shutdown 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 msgr2=0x7f75c40999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 0x7f75c40999a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f75c409cb90 tx=0x7f75b4035000 comp rx=0 tx=0).stop 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 shutdown_connections 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f75a40386f0 0x7f75a403aba0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75c4099560 0x7f75c40999a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75c411d990 0x7f75c4141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 --2- 172.21.15.181:0/743742744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75ac005280 0x7f75c4141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 >> 172.21.15.181:0/743742744 conn(0x7f75c40a5cf0 msgr2=0x7f75c40a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:18.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 shutdown_connections 2024-09-15T03:35:18.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.212+0000 7f75c8b5d640 1 -- 172.21.15.181:0/743742744 wait complete. 2024-09-15T03:35:18.222 INFO:tasks.workunit.client.0.smithi181.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:18.222 INFO:tasks.workunit.client.0.smithi181.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:18.222 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-15T03:35:18.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.294+0000 7f07bd95d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:18.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.294+0000 7f07bd95d640 10 monclient: build_initial_monmap 2024-09-15T03:35:18.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.294+0000 7f07bd95d640 10 monclient: monmap: 2024-09-15T03:35:18.295 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:18.295 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:18.295 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:18.295283+0000 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:18.295283+0000 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:18.296 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 Processor -- start 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 -- start start 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient: init 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f07b809ec50 0x7f07b8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): picked mon.noname-c con 0x7f07b809ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07b81321a0 0x7f07b8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): picked mon.noname-a con 0x7f07b81321a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07b809f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:18.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b8067420 con 0x7f07b809f020 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80a0480 con 0x7f07b81321a0 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80a0200 con 0x7f07b809ec50 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07bd95d640 10 monclient(hunting): authenticate will time out at 1122.776603s 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07b6ffd640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f07b809ec50 msgr2=0x7f07b8131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07b6ffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f07b809ec50 0x7f07b8131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07b77fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f07b809f020 legacy=0x7f07b8069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.295+0000 7f07b67fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07b81321a0 0x7f07b8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07b81321a0 0x7f07b8068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58644/0 (socket says 172.21.15.181:58644) 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 -- 172.21.15.181:0/2706876710 learned_addr learned my addr 172.21.15.181:0/2706876710 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:18.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): get_auth_request con 0x7f07b81321a0 auth_method 0 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient(hunting): handle_auth_done global_id 84444 payload 274 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 -- 172.21.15.181:0/2706876710 >> v2:172.21.15.181:3300/0 conn(0x7f07b809ec50 msgr2=0x7f07b8131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 --2- 172.21.15.181:0/2706876710 >> v2:172.21.15.181:3300/0 conn(0x7f07b809ec50 0x7f07b8131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 -- 172.21.15.181:0/2706876710 >> v1:172.21.15.181:6789/0 conn(0x7f07b809f020 legacy=0x7f07b8069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient: _finish_hunting 0 2024-09-15T03:35:18.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 monclient: found mon.noname-a 2024-09-15T03:35:18.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:18.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.296+0000 7f07b67fc640 1 -- 172.21.15.181:0/2706876710 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b809fd80 con 0x7f07b81321a0 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b67fc640 1 -- 172.21.15.181:0/2706876710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 msgr2=0x7f07b8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b67fc640 1 --2- 172.21.15.181:0/2706876710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8068d00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07b80719d0 tx=0x7f07ac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7cfec47c9f59 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 1 -- 172.21.15.181:0/2706876710 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac032440 con 0x7f07b81321a0 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: dump: 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.308 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 1 -- 172.21.15.181:0/2706876710 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac0325a0 con 0x7f07b81321a0 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07bd95d640 5 monclient: authenticate success, global_id 84444 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07bd95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07bd95d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:18.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07bd95d640 4 set_mon_vals no callback set 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 1 -- 172.21.15.181:0/2706876710 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac032940 con 0x7f07b81321a0 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.299+0000 7f07b5ffb640 10 monclient: dump: 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:18.310 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 10 monclient: shutdown 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 -- 172.21.15.181:0/2706876710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 msgr2=0x7f07b8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 --2- 172.21.15.181:0/2706876710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8068d00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07b80719d0 tx=0x7f07ac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 -- 172.21.15.181:0/2706876710 shutdown_connections 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 --2- 172.21.15.181:0/2706876710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8068d00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 --2- 172.21.15.181:0/2706876710 >> v2:172.21.15.181:3300/0 conn(0x7f07b809ec50 0x7f07b8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 -- 172.21.15.181:0/2706876710 >> 172.21.15.181:0/2706876710 conn(0x7f07b80a5cf0 msgr2=0x7f07b80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 -- 172.21.15.181:0/2706876710 shutdown_connections 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 1 -- 172.21.15.181:0/2706876710 wait complete. 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 10 monclient: build_initial_monmap 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.300+0000 7f07bd95d640 10 monclient: monmap: 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 Processor -- start 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 -- start start 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient: init 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07b809ec50 0x7f07b8141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): picked mon.noname-c con 0x7f07b809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07b809f020 0x7f07b812fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): picked mon.noname-b con 0x7f07b809f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:18.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8099ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): picked mon.noname-a con 0x7f07b81321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b811cf40 con 0x7f07b81321a0 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b809c020 con 0x7f07b809f020 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8134bc0 con 0x7f07b809ec50 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07bd95d640 10 monclient(hunting): authenticate will time out at 1122.782589s 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b67fc640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07b809f020 msgr2=0x7f07b812fb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b67fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07b809f020 0x7f07b812fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07b809ec50 0x7f07b8141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:18.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8099ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8099ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58658/0 (socket says 172.21.15.181:58658) 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 1 -- 172.21.15.181:0/146350126 learned_addr learned my addr 172.21.15.181:0/146350126 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): get_auth_request con 0x7f07b81321a0 auth_method 0 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 10 monclient(hunting): get_auth_request con 0x7f07b809ec50 auth_method 0 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:18.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.301+0000 7f07b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 10 monclient(hunting): handle_auth_done global_id 84447 payload 824 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 1 -- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07b809ec50 msgr2=0x7f07b8141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07b809ec50 0x7f07b8141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 1 -- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07b809f020 msgr2=0x7f07b812fb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07b809f020 0x7f07b812fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 1 monclient: found mon.noname-a 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.302+0000 7f07b77fe640 1 -- 172.21.15.181:0/146350126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac009450 con 0x7f07b81321a0 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07b77fe640 1 -- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 msgr2=0x7f07b8099ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07b77fe640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8099ad0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f07b000bf80 tx=0x7f07b0009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0016430 con 0x7f07b81321a0 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: dump: 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:18.317 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: _finish_auth 0 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 5 monclient: authenticate success, global_id 84447 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b00165d0 con 0x7f07b81321a0 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 10 monclient: _renew_subs 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b00167c0 con 0x7f07b81321a0 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b811ccc0 con 0x7f07b81321a0 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:18.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f079effd640 10 monclient: dump: 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07b57fa640 4 set_mon_vals no callback set 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 10 monclient: _renew_subs 2024-09-15T03:35:18.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.304+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b8141ec0 con 0x7f07b81321a0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f07b0012020 con 0x7f07b81321a0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f07bd95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f07bd95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f07bd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b80008d0 con 0x7f07b81321a0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f079effd640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07a40386f0 0x7f07a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(154..154 src has 106..154) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f07b004d4e0 con 0x7f07b81321a0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f07b6ffd640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07a40386f0 0x7f07a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.305+0000 7f07b6ffd640 10 monclient: get_auth_request con 0x7f07a40386f0 auth_method 0 2024-09-15T03:35:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.308+0000 7f07b6ffd640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07a40386f0 0x7f07a403aba0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f07a8003940 tx=0x7f07a80090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.310+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f07b00078f0 con 0x7f07b81321a0 2024-09-15T03:35:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.310+0000 7f079effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.310+0000 7f079effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:18.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.469+0000 7f07bd95d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:18.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.469+0000 7f07bd95d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:18.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.469+0000 7f07bd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:18.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:18.469+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f07b80a13e0 con 0x7f07b81321a0 2024-09-15T03:35:19.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.385+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v155) v1 ==== 123+0+0 (secure 0 0 0) 0x7f07b001d300 con 0x7f07b81321a0 2024-09-15T03:35:19.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.385+0000 7f079effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:19.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.385+0000 7f079effd640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-15T03:35:19.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.484+0000 7f07bd95d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:19.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.484+0000 7f07bd95d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:19.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.484+0000 7f07bd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:19.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:19.484+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f07b81416e0 con 0x7f07b81321a0 2024-09-15T03:35:20.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.390+0000 7f079effd640 1 -- 172.21.15.181:0/146350126 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v156) v1 ==== 149+0+0 (secure 0 0 0) 0x7f07b00072f0 con 0x7f07b81321a0 2024-09-15T03:35:20.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.390+0000 7f079effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:20.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.390+0000 7f079effd640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T03:35:20.392 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.393+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07a40386f0 msgr2=0x7f07a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.393+0000 7f07bd95d640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07a40386f0 0x7f07a403aba0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f07a8003940 tx=0x7f07a80090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.393+0000 7f07bd95d640 10 monclient: shutdown 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.393+0000 7f07bd95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.393+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 msgr2=0x7f07b8099ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.393+0000 7f07bd95d640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8099ad0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f07b000bf80 tx=0x7f07b0009730 comp rx=0 tx=0).stop 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 shutdown_connections 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f07a40386f0 0x7f07a403aba0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b81321a0 0x7f07b8099ad0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07b809f020 0x7f07b812fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 --2- 172.21.15.181:0/146350126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07b809ec50 0x7f07b8141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 >> 172.21.15.181:0/146350126 conn(0x7f07b80a5cf0 msgr2=0x7f07b80a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 shutdown_connections 2024-09-15T03:35:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.394+0000 7f07bd95d640 1 -- 172.21.15.181:0/146350126 wait complete. 2024-09-15T03:35:20.406 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-15T03:35:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.484+0000 7f9a026c9640 10 monclient: get_monmap_and_config 2024-09-15T03:35:20.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.484+0000 7f9a026c9640 10 monclient: build_initial_monmap 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.484+0000 7f9a026c9640 10 monclient: monmap: 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:20.485562+0000 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:20.485562+0000 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:20.486 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:20.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 1 Processor -- start 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 1 -- start start 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient: init 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 0x7f99fc09d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient(hunting): picked mon.noname-b con 0x7f99fc09d140 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99fc06ebc0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99fc09da50 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc09eb90 con 0x7f99fc09da50 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc094ea0 con 0x7f99fc06ebc0 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.485+0000 7f9a026c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc09ef50 con 0x7f99fc09d140 2024-09-15T03:35:20.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f9a026c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f9a026c9640 10 monclient(hunting): authenticate will time out at 1124.967008s 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fb7fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f99fc06ebc0 legacy=0x7f99fc094080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 0x7f99fc09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f9a00c66640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f99fc09da50 0x7f99fc094790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:34628/0 (socket says 172.21.15.181:34628) 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f9a00c66640 1 -- 172.21.15.181:0/3561527535 learned_addr learned my addr 172.21.15.181:0/3561527535 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): get_auth_request con 0x7f99fc09d140 auth_method 0 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:20.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient(hunting): handle_auth_done global_id 74543 payload 274 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 1 -- 172.21.15.181:0/3561527535 >> v1:172.21.15.181:6789/0 conn(0x7f99fc06ebc0 legacy=0x7f99fc094080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 1 -- 172.21.15.181:0/3561527535 >> v1:172.21.15.153:6789/0 conn(0x7f99fc09da50 legacy=0x7f99fc094790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 1 monclient: found mon.noname-b 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:20.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.486+0000 7f99fbfff640 1 -- 172.21.15.181:0/3561527535 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc067420 con 0x7f99fc09d140 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.489+0000 7f99fbfff640 1 -- 172.21.15.181:0/3561527535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 msgr2=0x7f99fc09d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.489+0000 7f99fbfff640 1 --2- 172.21.15.181:0/3561527535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 0x7f99fc09d510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0098b0 tx=0x7f99ec0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5409100f6dd7efc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 1 -- 172.21.15.181:0/3561527535 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec032440 con 0x7f99fc09d140 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: dump: 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:20.495 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 1 -- 172.21.15.181:0/3561527535 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec0325a0 con 0x7f99fc09d140 2024-09-15T03:35:20.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 5 monclient: authenticate success, global_id 74543 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 4 set_mon_vals no callback set 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 1 -- 172.21.15.181:0/3561527535 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec032940 con 0x7f99fc09d140 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f99faffd640 10 monclient: dump: 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.497 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 10 monclient: shutdown 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 -- 172.21.15.181:0/3561527535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 msgr2=0x7f99fc09d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 --2- 172.21.15.181:0/3561527535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 0x7f99fc09d510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0098b0 tx=0x7f99ec0090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 -- 172.21.15.181:0/3561527535 shutdown_connections 2024-09-15T03:35:20.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 --2- 172.21.15.181:0/3561527535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09d140 0x7f99fc09d510 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 -- 172.21.15.181:0/3561527535 >> 172.21.15.181:0/3561527535 conn(0x7f99fc0a2190 msgr2=0x7f99fc09e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 -- 172.21.15.181:0/3561527535 shutdown_connections 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.490+0000 7f9a026c9640 1 -- 172.21.15.181:0/3561527535 wait complete. 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.491+0000 7f9a026c9640 10 monclient: build_initial_monmap 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.491+0000 7f9a026c9640 10 monclient: monmap: 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:20.499 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.491+0000 7f9a026c9640 1 Processor -- start 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.491+0000 7f9a026c9640 1 -- start start 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.491+0000 7f9a026c9640 10 monclient: init 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99fc06ebc0 0x7f99fc12aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): picked mon.noname-b con 0x7f99fc06ebc0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99fc09d140 0x7f99fc129140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): picked mon.noname-a con 0x7f99fc09d140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 0x7f99fc129680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): picked mon.noname-c con 0x7f99fc09da50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc12c890 con 0x7f99fc09d140 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc12ad70 con 0x7f99fc06ebc0 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc0888c0 con 0x7f99fc09da50 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a026c9640 10 monclient(hunting): authenticate will time out at 1124.973227s 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fbfff640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99fc06ebc0 msgr2=0x7f99fc12aff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99fc06ebc0 0x7f99fc12aff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99fc09d140 0x7f99fc129140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:20.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 0x7f99fc129680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99fc09d140 0x7f99fc129140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58670/0 (socket says 172.21.15.181:58670) 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): get_auth_request con 0x7f99fc09da50 auth_method 0 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 1 -- 172.21.15.181:0/2716667809 learned_addr learned my addr 172.21.15.181:0/2716667809 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): get_auth_request con 0x7f99fc09d140 auth_method 0 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:20.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f99fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient(hunting): handle_auth_done global_id 74546 payload 824 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 -- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99fc06ebc0 msgr2=0x7f99fc12aff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99fc06ebc0 0x7f99fc12aff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 -- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99fc09d140 msgr2=0x7f99fc129140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99fc09d140 0x7f99fc129140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient: _finish_hunting 0 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 monclient: found mon.noname-c 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:20.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.492+0000 7f9a00c66640 1 -- 172.21.15.181:0/2716667809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec009390 con 0x7f99fc09da50 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a00c66640 1 -- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 msgr2=0x7f99fc129680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a00c66640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 0x7f99fc129680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99f400cb60 tx=0x7f99f40052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f400a850 con 0x7f99fc09da50 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: dump: 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:20.506 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f400a9d0 con 0x7f99fc09da50 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 5 monclient: authenticate success, global_id 74546 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 10 monclient: _renew_subs 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc12b5f0 con 0x7f99fc09da50 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:20.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f400ad90 con 0x7f99fc09da50 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 10 monclient: _renew_subs 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc094a50 con 0x7f99fc09da50 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99e3fff640 10 monclient: dump: 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.495+0000 7f99fa7fc640 4 set_mon_vals no callback set 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f9a026c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f9a026c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f9a026c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f99f4007300 con 0x7f99fc09da50 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc094cd0 con 0x7f99fc09da50 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f99e3fff640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99e80386f0 0x7f99e803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(156..156 src has 106..156) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f99f404cd20 con 0x7f99fc09da50 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f99fbfff640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99e80386f0 0x7f99e803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.496+0000 7f99fbfff640 10 monclient: get_auth_request con 0x7f99e80386f0 auth_method 0 2024-09-15T03:35:20.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.499+0000 7f99fbfff640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99e80386f0 0x7f99e803aba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f99ec034000 tx=0x7f99ec032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:20.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.501+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f99f40075b0 con 0x7f99fc09da50 2024-09-15T03:35:20.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.501+0000 7f99e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:20.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.501+0000 7f99e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:20.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.657+0000 7f9a026c9640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T03:35:20.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.657+0000 7f9a026c9640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T03:35:20.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.657+0000 7f9a026c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:20.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:20.657+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f99fc0008d0 con 0x7f99fc09da50 2024-09-15T03:35:21.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.396+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v157) v1 ==== 145+0+0 (secure 0 0 0) 0x7f99f4010af0 con 0x7f99fc09da50 2024-09-15T03:35:21.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.396+0000 7f99e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T03:35:21.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.396+0000 7f99e3fff640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-15T03:35:21.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.491+0000 7f9a026c9640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T03:35:21.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.491+0000 7f9a026c9640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T03:35:21.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.491+0000 7f9a026c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:21.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:21.491+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f99fc06f4a0 con 0x7f99fc09da50 2024-09-15T03:35:22.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.118+0000 7f99e3fff640 1 -- 172.21.15.181:0/2716667809 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v158) v1 ==== 257+0+0 (secure 0 0 0) 0x7f99f4010cd0 con 0x7f99fc09da50 2024-09-15T03:35:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.118+0000 7f99e3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-15T03:35:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.118+0000 7f99e3fff640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-15T03:35:22.120 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-15T03:35:22.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.120+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99e80386f0 msgr2=0x7f99e803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99e80386f0 0x7f99e803aba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f99ec034000 tx=0x7f99ec032000 comp rx=0 tx=0).stop 2024-09-15T03:35:22.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 10 monclient: shutdown 2024-09-15T03:35:22.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:22.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 msgr2=0x7f99fc129680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 0x7f99fc129680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99f400cb60 tx=0x7f99f40052b0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 shutdown_connections 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f99e80386f0 0x7f99e803aba0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99fc09da50 0x7f99fc129680 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99fc09d140 0x7f99fc129140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 --2- 172.21.15.181:0/2716667809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99fc06ebc0 0x7f99fc12aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 >> 172.21.15.181:0/2716667809 conn(0x7f99fc0a2190 msgr2=0x7f99fc09f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 shutdown_connections 2024-09-15T03:35:22.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.121+0000 7f9a026c9640 1 -- 172.21.15.181:0/2716667809 wait complete. 2024-09-15T03:35:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:22.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:22.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep globster 2024-09-15T03:35:22.134 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '10\.' 2024-09-15T03:35:22.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.208+0000 7fcbc6817640 10 monclient: get_monmap_and_config 2024-09-15T03:35:22.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.208+0000 7fcbc6817640 10 monclient: build_initial_monmap 2024-09-15T03:35:22.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.208+0000 7fcbc6817640 10 monclient: monmap: 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:22.209584+0000 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:22.209584+0000 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:22.210 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.209+0000 7fcbc6817640 1 Processor -- start 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.209+0000 7fcbc6817640 1 -- start start 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.209+0000 7fcbc6817640 10 monclient: init 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcbc0096a30 0x7fcbc009d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc0096a30 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 0x7fcbc0096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc009b0b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbc0097340 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc009c470 con 0x7fcbc0097340 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc009c190 con 0x7fcbc0096a30 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc009c750 con 0x7fcbc009b0b0 2024-09-15T03:35:22.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbc6817640 10 monclient(hunting): authenticate will time out at 1126.691155s 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbffff640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fcbc0096a30 msgr2=0x7fcbc009d320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcbc0096a30 0x7fcbc009d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 0x7fcbc0096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 0x7fcbc0096e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59030/0 (socket says 172.21.15.181:59030) 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 -- 172.21.15.181:0/500956002 learned_addr learned my addr 172.21.15.181:0/500956002 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): get_auth_request con 0x7fcbc009b0b0 auth_method 0 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_done global_id 74549 payload 274 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 -- 172.21.15.181:0/500956002 >> v2:172.21.15.181:3300/0 conn(0x7fcbc0096a30 msgr2=0x7fcbc009d320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 --2- 172.21.15.181:0/500956002 >> v2:172.21.15.181:3300/0 conn(0x7fcbc0096a30 0x7fcbc009d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 -- 172.21.15.181:0/500956002 >> v1:172.21.15.153:6789/0 conn(0x7fcbc0097340 legacy=0x7fcbc012c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 monclient: found mon.noname-b 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.210+0000 7fcbbf7fe640 1 -- 172.21.15.181:0/500956002 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00977a0 con 0x7fcbc009b0b0 2024-09-15T03:35:22.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.213+0000 7fcbbf7fe640 1 -- 172.21.15.181:0/500956002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 msgr2=0x7fcbc0096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:22.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.213+0000 7fcbbf7fe640 1 --2- 172.21.15.181:0/500956002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 0x7fcbc0096e00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0068b80 tx=0x7fcbb40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=752df0b26174ab8a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 1 -- 172.21.15.181:0/500956002 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402ae00 con 0x7fcbc009b0b0 2024-09-15T03:35:22.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:22.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: dump: 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 1 -- 172.21.15.181:0/500956002 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb402f4b0 con 0x7fcbc009b0b0 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 1 -- 172.21.15.181:0/500956002 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402f850 con 0x7fcbc009b0b0 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 5 monclient: authenticate success, global_id 74549 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 4 set_mon_vals no callback set 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbbeffd640 10 monclient: dump: 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 10 monclient: shutdown 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 -- 172.21.15.181:0/500956002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 msgr2=0x7fcbc0096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 --2- 172.21.15.181:0/500956002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 0x7fcbc0096e00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0068b80 tx=0x7fcbb40091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 -- 172.21.15.181:0/500956002 shutdown_connections 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 --2- 172.21.15.181:0/500956002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc009b0b0 0x7fcbc0096e00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 --2- 172.21.15.181:0/500956002 >> v2:172.21.15.181:3300/0 conn(0x7fcbc0096a30 0x7fcbc009d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 -- 172.21.15.181:0/500956002 >> 172.21.15.181:0/500956002 conn(0x7fcbc0071100 msgr2=0x7fcbc00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 -- 172.21.15.181:0/500956002 shutdown_connections 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.214+0000 7fcbc6817640 1 -- 172.21.15.181:0/500956002 wait complete. 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.215+0000 7fcbc6817640 10 monclient: build_initial_monmap 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.215+0000 7fcbc6817640 10 monclient: monmap: 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:22.225 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.215+0000 7fcbc6817640 1 Processor -- start 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.215+0000 7fcbc6817640 1 -- start start 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.215+0000 7fcbc6817640 10 monclient: init 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 0x7fcbc012b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc0096a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc0097340 0x7fcbc0131d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc0097340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbc009b0b0 0x7fcbc01322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc009b0b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc009bd40 con 0x7fcbc0097340 2024-09-15T03:35:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0069c00 con 0x7fcbc009b0b0 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0134590 con 0x7fcbc0096a30 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc6817640 10 monclient(hunting): authenticate will time out at 1126.697272s 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc4db4640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbc009b0b0 msgr2=0x7fcbc01322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbc4db4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbc009b0b0 0x7fcbc01322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 0x7fcbc012b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc0097340 0x7fcbc0131d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 0x7fcbc012b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59044/0 (socket says 172.21.15.181:59044) 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 -- 172.21.15.181:0/1638774359 learned_addr learned my addr 172.21.15.181:0/1638774359 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): get_auth_request con 0x7fcbc0096a30 auth_method 0 2024-09-15T03:35:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): get_auth_request con 0x7fcbc0097340 auth_method 0 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient(hunting): handle_auth_done global_id 74552 payload 824 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 -- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbc009b0b0 msgr2=0x7fcbc01322a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbc009b0b0 0x7fcbc01322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 -- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc0097340 msgr2=0x7fcbc0131d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc0097340 0x7fcbc0131d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 monclient: found mon.noname-c 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbffff640 1 -- 172.21.15.181:0/1638774359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4009450 con 0x7fcbc0096a30 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.216+0000 7fcbbf7fe640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc0097340 0x7fcbc0131d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:22.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcbbffff640 1 -- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 msgr2=0x7fcbc012b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcbbffff640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 0x7fcbc012b6f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000b600 tx=0x7fcbb00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0015070 con 0x7fcbc0096a30 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcba7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcba7fff640 10 monclient: dump: 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcba7fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.219+0000 7fcba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb0003d90 con 0x7fcbc0096a30 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcba7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 5 monclient: authenticate success, global_id 74552 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0006380 con 0x7fcbc0096a30 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcba7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcba7fff640 10 monclient: dump: 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 10 monclient: _renew_subs 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbbe7fc640 4 set_mon_vals no callback set 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc0118dd0 con 0x7fcbc0096a30 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 10 monclient: _renew_subs 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.220+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc01335f0 con 0x7fcbc0096a30 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcbc6817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcbc6817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcbc6817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc00008d0 con 0x7fcbc0096a30 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcbb0010440 con 0x7fcbc0096a30 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcba7fff640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcbac038740 0x7fcbac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(158..158 src has 106..158) v4 ==== 9849+0+0 (secure 0 0 0) 0x7fcbb004c910 con 0x7fcbc0096a30 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcbbf7fe640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcbac038740 0x7fcbac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.221+0000 7fcbbf7fe640 10 monclient: get_auth_request con 0x7fcbac038740 auth_method 0 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.224+0000 7fcbbf7fe640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcbac038740 0x7fcbac03abf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcbc009d860 tx=0x7fcbb4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.226+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcbb0018af0 con 0x7fcbc0096a30 2024-09-15T03:35:22.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.226+0000 7fcba7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.226+0000 7fcba7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:22.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.386+0000 7fcbc6817640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.386+0000 7fcbc6817640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.386+0000 7fcbc6817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.386+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcbc00a1a30 con 0x7fcbc0096a30 2024-09-15T03:35:22.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.387+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcbb0018310 con 0x7fcbc0096a30 2024-09-15T03:35:22.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.387+0000 7fcba7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.387+0000 7fcba7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.485+0000 7fcbc6817640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.485+0000 7fcbc6817640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.485+0000 7fcbc6817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.485+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcbc0072640 con 0x7fcbc0096a30 2024-09-15T03:35:22.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.486+0000 7fcba7fff640 1 -- 172.21.15.181:0/1638774359 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcbb00184f0 con 0x7fcbc0096a30 2024-09-15T03:35:22.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.486+0000 7fcba7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.486+0000 7fcba7fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcbac038740 msgr2=0x7fcbac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcbac038740 0x7fcbac03abf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcbc009d860 tx=0x7fcbb4032000 comp rx=0 tx=0).stop 2024-09-15T03:35:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 10 monclient: shutdown 2024-09-15T03:35:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:22.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 msgr2=0x7fcbc012b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 0x7fcbc012b6f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000b600 tx=0x7fcbb00091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 shutdown_connections 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcbac038740 0x7fcbac03abf0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbc009b0b0 0x7fcbc01322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc0097340 0x7fcbc0131d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 --2- 172.21.15.181:0/1638774359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc0096a30 0x7fcbc012b6f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 >> 172.21.15.181:0/1638774359 conn(0x7fcbc0071100 msgr2=0x7fcbc006d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 shutdown_connections 2024-09-15T03:35:22.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.488+0000 7fcbc6817640 1 -- 172.21.15.181:0/1638774359 wait complete. 2024-09-15T03:35:22.498 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:22.498 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:22.499 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:22.499 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:22.499 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep globster 2024-09-15T03:35:22.499 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '11\.' 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.570+0000 7f726c35d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.570+0000 7f726c35d640 10 monclient: build_initial_monmap 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.570+0000 7f726c35d640 10 monclient: monmap: 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:22.571053+0000 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:22.571053+0000 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.572 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:22.572 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:22.572 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:22.572 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:22.572 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:22.572 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.570+0000 7f726c35d640 1 Processor -- start 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.570+0000 7f726c35d640 1 -- start start 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.570+0000 7f726c35d640 10 monclient: init 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f726409ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f726409f020 0x7f7264069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): picked mon.noname-a con 0x7f726409f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 0x7f7264139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): picked mon.noname-b con 0x7f7264136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264067420 con 0x7f726409ec50 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72640a0480 con 0x7f726409f020 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72640a0200 con 0x7f7264136d60 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726c35d640 10 monclient(hunting): authenticate will time out at 1127.052657s 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.571+0000 7f726a0f9640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f726409ec50 legacy=0x7f7264131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 0x7f7264139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f726409f020 0x7f7264069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 0x7f7264139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59058/0 (socket says 172.21.15.181:59058) 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f726409f020 0x7f7264069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58686/0 (socket says 172.21.15.181:58686) 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 1 -- 172.21.15.181:0/531287297 learned_addr learned my addr 172.21.15.181:0/531287297 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): get_auth_request con 0x7f7264136d60 auth_method 0 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): get_auth_request con 0x7f726409f020 auth_method 0 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient(hunting): handle_auth_done global_id 84456 payload 274 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 -- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 msgr2=0x7f7264139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 --2- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 0x7f7264139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 -- 172.21.15.181:0/531287297 >> v1:172.21.15.181:6789/0 conn(0x7f726409ec50 legacy=0x7f7264131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient: _finish_hunting 0 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 monclient: found mon.noname-a 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f72698f8640 1 -- 172.21.15.181:0/531287297 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726409fd80 con 0x7f726409f020 2024-09-15T03:35:22.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.572+0000 7f726a8fa640 1 --2- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 0x7f7264139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72698f8640 1 -- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409f020 msgr2=0x7f7264069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72698f8640 1 --2- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409f020 0x7f7264069d10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7254009850 tx=0x7f72540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2cc6e7acbc01c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 1 -- 172.21.15.181:0/531287297 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725402adf0 con 0x7f726409f020 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: dump: 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.584 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: _finish_auth 0 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 5 monclient: authenticate success, global_id 84456 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 1 -- 172.21.15.181:0/531287297 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7254032440 con 0x7f726409f020 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 4 set_mon_vals no callback set 2024-09-15T03:35:22.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 1 -- 172.21.15.181:0/531287297 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72540327e0 con 0x7f726409f020 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f72690f7640 10 monclient: dump: 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 10 monclient: shutdown 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 1 -- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409f020 msgr2=0x7f7264069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.575+0000 7f726c35d640 1 --2- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409f020 0x7f7264069d10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7254009850 tx=0x7f72540090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 -- 172.21.15.181:0/531287297 shutdown_connections 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 --2- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7264136d60 0x7f7264139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 --2- 172.21.15.181:0/531287297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409f020 0x7f7264069d10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 -- 172.21.15.181:0/531287297 >> 172.21.15.181:0/531287297 conn(0x7f72640a5cf0 msgr2=0x7f72640a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 -- 172.21.15.181:0/531287297 shutdown_connections 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 -- 172.21.15.181:0/531287297 wait complete. 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 10 monclient: build_initial_monmap 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 10 monclient: monmap: 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:22.587 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 Processor -- start 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 1 -- start start 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.576+0000 7f726c35d640 10 monclient: init 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:22.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 0x7f72640973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): picked mon.noname-a con 0x7f726409ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f726409f020 0x7f72640978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): picked mon.noname-c con 0x7f726409f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7264136d60 0x7f7264097e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): picked mon.noname-b con 0x7f7264136d60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264098ee0 con 0x7f726409ec50 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726409c020 con 0x7f7264136d60 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72640a4010 con 0x7f726409f020 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726c35d640 10 monclient(hunting): authenticate will time out at 1127.058366s 2024-09-15T03:35:22.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a8fa640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7264136d60 msgr2=0x7f7264097e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a8fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7264136d60 0x7f7264097e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 0x7f72640973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f726409f020 0x7f72640978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 0x7f72640973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58694/0 (socket says 172.21.15.181:58694) 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 -- 172.21.15.181:0/1394444467 learned_addr learned my addr 172.21.15.181:0/1394444467 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f726409f020 0x7f72640978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59064/0 (socket says 172.21.15.181:59064) 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): get_auth_request con 0x7f726409ec50 auth_method 0 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): get_auth_request con 0x7f726409f020 auth_method 0 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f72698f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient(hunting): handle_auth_done global_id 84459 payload 824 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 -- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f726409f020 msgr2=0x7f72640978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f726409f020 0x7f72640978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 -- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7264136d60 msgr2=0x7f7264097e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7264136d60 0x7f7264097e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient: _finish_hunting 0 2024-09-15T03:35:22.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 monclient: found mon.noname-a 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.577+0000 7f726a0f9640 1 -- 172.21.15.181:0/1394444467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c009ae0 con 0x7f726409ec50 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726a0f9640 1 -- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 msgr2=0x7f72640973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726a0f9640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 0x7f72640973a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f726000b600 tx=0x7f72600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260015070 con 0x7f726409ec50 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: dump: 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.592 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 5 monclient: authenticate success, global_id 84459 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7260003d90 con 0x7f726409ec50 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 10 monclient: _renew_subs 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7254009390 con 0x7f726409ec50 2024-09-15T03:35:22.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260006380 con 0x7f726409ec50 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f725a7fc640 10 monclient: dump: 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.594 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 10 monclient: _renew_subs 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726409b640 con 0x7f726409ec50 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.580+0000 7f72688f6640 4 set_mon_vals no callback set 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f726c35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f726c35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f726c35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726409b060 con 0x7f726409ec50 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7260010440 con 0x7f726409ec50 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f725a7fc640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72480386f0 0x7f724803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(158..158 src has 106..158) v4 ==== 9849+0+0 (secure 0 0 0) 0x7f726004c9e0 con 0x7f726409ec50 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f72698f8640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72480386f0 0x7f724803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.581+0000 7f72698f8640 10 monclient: get_auth_request con 0x7f72480386f0 auth_method 0 2024-09-15T03:35:22.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.584+0000 7f72698f8640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72480386f0 0x7f724803aba0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f72640719d0 tx=0x7f7254032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.586+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7260018d60 con 0x7f726409ec50 2024-09-15T03:35:22.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.586+0000 7f725a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.586+0000 7f725a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:22.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.746+0000 7f726c35d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.746+0000 7f726c35d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.746+0000 7f726c35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.746+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f726409aa90 con 0x7f726409ec50 2024-09-15T03:35:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.747+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f7260018780 con 0x7f726409ec50 2024-09-15T03:35:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.747+0000 7f725a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.747+0000 7f725a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:22.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.844+0000 7f726c35d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.844+0000 7f726c35d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.844+0000 7f726c35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.844+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f72640008d0 con 0x7f726409ec50 2024-09-15T03:35:22.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.845+0000 7f725a7fc640 1 -- 172.21.15.181:0/1394444467 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f7260018d60 con 0x7f726409ec50 2024-09-15T03:35:22.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.845+0000 7f725a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:22.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.845+0000 7f725a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:22.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72480386f0 msgr2=0x7f724803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72480386f0 0x7f724803aba0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f72640719d0 tx=0x7f7254032000 comp rx=0 tx=0).stop 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 10 monclient: shutdown 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 msgr2=0x7f72640973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 0x7f72640973a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f726000b600 tx=0x7f72600091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 shutdown_connections 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72480386f0 0x7f724803aba0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7264136d60 0x7f7264097e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f726409f020 0x7f72640978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 --2- 172.21.15.181:0/1394444467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f726409ec50 0x7f72640973a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 >> 172.21.15.181:0/1394444467 conn(0x7f72640a5cf0 msgr2=0x7f726409d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 shutdown_connections 2024-09-15T03:35:22.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.847+0000 7f726c35d640 1 -- 172.21.15.181:0/1394444467 wait complete. 2024-09-15T03:35:22.857 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:22.857 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:22.858 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:22.858 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:22.858 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep globster 2024-09-15T03:35:22.858 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '12\.' 2024-09-15T03:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.930+0000 7f88d0754640 10 monclient: get_monmap_and_config 2024-09-15T03:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.930+0000 7f88d0754640 10 monclient: build_initial_monmap 2024-09-15T03:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.930+0000 7f88d0754640 10 monclient: monmap: 2024-09-15T03:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:22.931301+0000 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:22.931301+0000 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:22.932 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 Processor -- start 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 -- start start 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient: init 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8069bd0 0x7f88c81320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): picked mon.noname-b con 0x7f88c8069bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88c8098330 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f88c8136d50 0x7f88c8139130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): picked mon.noname-a con 0x7f88c8136d50 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c8096bd0 con 0x7f88c8098330 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c8131db0 con 0x7f88c8136d50 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c8098e50 con 0x7f88c8069bd0 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.931+0000 7f88d0754640 10 monclient(hunting): authenticate will time out at 1127.412736s 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f88c8136d50 0x7f88c8139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8069bd0 0x7f88c81320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f88c8136d50 0x7f88c8139130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58700/0 (socket says 172.21.15.181:58700) 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 -- 172.21.15.181:0/2379454779 learned_addr learned my addr 172.21.15.181:0/2379454779 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): get_auth_request con 0x7f88c8136d50 auth_method 0 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): get_auth_request con 0x7f88c8069bd0 auth_method 0 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient(hunting): handle_auth_done global_id 84462 payload 274 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 -- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8069bd0 msgr2=0x7f88c81320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 --2- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8069bd0 0x7f88c81320a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 -- 172.21.15.181:0/2379454779 >> v1:172.21.15.153:6789/0 conn(0x7f88c8098330 legacy=0x7f88c8098740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient: _finish_hunting 0 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 monclient: found mon.noname-a 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88cecf1640 1 -- 172.21.15.181:0/2379454779 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c809d7f0 con 0x7f88c8136d50 2024-09-15T03:35:22.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.932+0000 7f88ce4f0640 1 --2- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8069bd0 0x7f88c81320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cecf1640 1 -- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8136d50 msgr2=0x7f88c8139130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cecf1640 1 --2- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8136d50 0x7f88c8139130 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009f20 tx=0x7f88c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7196d36f05279f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 1 -- 172.21.15.181:0/2379454779 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0007610 con 0x7f88c8136d50 2024-09-15T03:35:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: dump: 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: _finish_auth 0 2024-09-15T03:35:22.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:22.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 1 -- 172.21.15.181:0/2379454779 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c002d050 con 0x7f88c8136d50 2024-09-15T03:35:22.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:22.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 1 -- 172.21.15.181:0/2379454779 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00079a0 con 0x7f88c8136d50 2024-09-15T03:35:22.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88d0754640 5 monclient: authenticate success, global_id 84462 2024-09-15T03:35:22.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88d0754640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88d0754640 4 set_mon_vals no callback set 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.935+0000 7f88cd4ee640 10 monclient: dump: 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.950 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 10 monclient: shutdown 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 -- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8136d50 msgr2=0x7f88c8139130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 --2- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8136d50 0x7f88c8139130 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009f20 tx=0x7f88c0009840 comp rx=0 tx=0).stop 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 -- 172.21.15.181:0/2379454779 shutdown_connections 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 --2- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8136d50 0x7f88c8139130 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 --2- 172.21.15.181:0/2379454779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8069bd0 0x7f88c81320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 -- 172.21.15.181:0/2379454779 >> 172.21.15.181:0/2379454779 conn(0x7f88c809e9f0 msgr2=0x7f88c809edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 -- 172.21.15.181:0/2379454779 shutdown_connections 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 1 -- 172.21.15.181:0/2379454779 wait complete. 2024-09-15T03:35:22.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 10 monclient: build_initial_monmap 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.936+0000 7f88d0754640 10 monclient: monmap: 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:22.952 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 Processor -- start 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 -- start start 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient: init 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 0x7f88c8118880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): picked mon.noname-a con 0x7f88c8069bd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8098330 0x7f88c8118dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): picked mon.noname-b con 0x7f88c8098330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8136d50 0x7f88c8119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): picked mon.noname-c con 0x7f88c8136d50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c809fe00 con 0x7f88c8069bd0 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c811c1d0 con 0x7f88c8098330 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c811bc10 con 0x7f88c8136d50 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88d0754640 10 monclient(hunting): authenticate will time out at 1127.418796s 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88cdcef640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8098330 msgr2=0x7f88c8118dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:22.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.937+0000 7f88cdcef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8098330 0x7f88c8118dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 0x7f88c8118880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 0x7f88c8118880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58716/0 (socket says 172.21.15.181:58716) 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 -- 172.21.15.181:0/1386349579 learned_addr learned my addr 172.21.15.181:0/1386349579 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): get_auth_request con 0x7f88c8069bd0 auth_method 0 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8136d50 0x7f88c8119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): get_auth_request con 0x7f88c8136d50 auth_method 0 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:22.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient(hunting): handle_auth_done global_id 84465 payload 824 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 -- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8136d50 msgr2=0x7f88c8119300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8136d50 0x7f88c8119300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 -- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8098330 msgr2=0x7f88c8118dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8098330 0x7f88c8118dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient: _finish_hunting 0 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 monclient: found mon.noname-a 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88ce4f0640 1 -- 172.21.15.181:0/1386349579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c4009390 con 0x7f88c8069bd0 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.938+0000 7f88cecf1640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8136d50 0x7f88c8119300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88ce4f0640 1 -- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 msgr2=0x7f88c8118880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:22.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88ce4f0640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 0x7f88c8118880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f88c400ea70 tx=0x7f88c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c400cb50 con 0x7f88c8069bd0 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88be7fc640 10 monclient: dump: 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.940+0000 7f88d0754640 5 monclient: authenticate success, global_id 84465 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 monclient: _renew_subs 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c0009ae0 con 0x7f88c8069bd0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 monclient: _renew_subs 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8119840 con 0x7f88c8069bd0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c400ccb0 con 0x7f88c8069bd0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4013470 con 0x7f88c8069bd0 2024-09-15T03:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 10 monclient: dump: 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:22.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88ccced640 4 set_mon_vals no callback set 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c80008d0 con 0x7f88c8069bd0 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.941+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f88c4013820 con 0x7f88c8069bd0 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.942+0000 7f88be7fc640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88ac0386f0 0x7f88ac03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.942+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(158..158 src has 106..158) v4 ==== 9849+0+0 (secure 0 0 0) 0x7f88c404d4d0 con 0x7f88c8069bd0 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.942+0000 7f88cdcef640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88ac0386f0 0x7f88ac03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:22.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.942+0000 7f88cdcef640 10 monclient: get_auth_request con 0x7f88ac0386f0 auth_method 0 2024-09-15T03:35:22.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.944+0000 7f88cdcef640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88ac0386f0 0x7f88ac03aba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f88b8007fd0 tx=0x7f88b8007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:22.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.946+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f88c401b770 con 0x7f88c8069bd0 2024-09-15T03:35:22.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.946+0000 7f88be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:22.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:22.946+0000 7f88be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:23.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.106+0000 7f88d0754640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:23.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.106+0000 7f88d0754640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:23.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.106+0000 7f88d0754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:23.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.106+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f88c811bd50 con 0x7f88c8069bd0 2024-09-15T03:35:23.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.107+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f88c401b3b0 con 0x7f88c8069bd0 2024-09-15T03:35:23.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.107+0000 7f88be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:23.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.107+0000 7f88be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:23.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.205+0000 7f88d0754640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:23.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.205+0000 7f88d0754640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:23.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.205+0000 7f88d0754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:23.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.205+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f88c8064b00 con 0x7f88c8069bd0 2024-09-15T03:35:23.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.206+0000 7f88be7fc640 1 -- 172.21.15.181:0/1386349579 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f88c401b590 con 0x7f88c8069bd0 2024-09-15T03:35:23.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.206+0000 7f88be7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:23.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.206+0000 7f88be7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:23.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88ac0386f0 msgr2=0x7f88ac03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:23.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88ac0386f0 0x7f88ac03aba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f88b8007fd0 tx=0x7f88b8007a70 comp rx=0 tx=0).stop 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 10 monclient: shutdown 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 msgr2=0x7f88c8118880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 0x7f88c8118880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f88c400ea70 tx=0x7f88c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 shutdown_connections 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f88ac0386f0 0x7f88ac03aba0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88c8136d50 0x7f88c8119300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8098330 0x7f88c8118dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 --2- 172.21.15.181:0/1386349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c8069bd0 0x7f88c8118880 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 >> 172.21.15.181:0/1386349579 conn(0x7f88c809e9f0 msgr2=0x7f88c8097080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 shutdown_connections 2024-09-15T03:35:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.208+0000 7f88d0754640 1 -- 172.21.15.181:0/1386349579 wait complete. 2024-09-15T03:35:23.218 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:23.218 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-15T03:35:23.218 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-15T03:35:23.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.291+0000 7f3f28a94640 10 monclient: get_monmap_and_config 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.291+0000 7f3f28a94640 10 monclient: build_initial_monmap 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.291+0000 7f3f28a94640 10 monclient: monmap: 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:23.292693+0000 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:23.292693+0000 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:23.293 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:23.294 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.292+0000 7f3f28a94640 1 Processor -- start 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.292+0000 7f3f28a94640 1 -- start start 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.292+0000 7f3f28a94640 10 monclient: init 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f24096a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f24096e00 0x7f3f24097200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): picked mon.noname-a con 0x7f3f24096e00 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 0x7f3f24137200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): picked mon.noname-b con 0x7f3f24134e20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2409c400 con 0x7f3f24096a30 2024-09-15T03:35:23.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2409b970 con 0x7f3f24096e00 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2409d7a0 con 0x7f3f24134e20 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f2259c640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f3f24096a30 legacy=0x7f3f2409d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f28a94640 10 monclient(hunting): authenticate will time out at 1127.774235s 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 0x7f3f24137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f24096e00 0x7f3f24097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f24096e00 0x7f3f24097200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58718/0 (socket says 172.21.15.181:58718) 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 -- 172.21.15.181:0/2477583225 learned_addr learned my addr 172.21.15.181:0/2477583225 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): get_auth_request con 0x7f3f24096e00 auth_method 0 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 0x7f3f24137200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59084/0 (socket says 172.21.15.181:59084) 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:23.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): get_auth_request con 0x7f3f24134e20 auth_method 0 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient(hunting): handle_auth_done global_id 84468 payload 274 2024-09-15T03:35:23.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 -- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 msgr2=0x7f3f24137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 --2- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 0x7f3f24137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 -- 172.21.15.181:0/2477583225 >> v1:172.21.15.181:6789/0 conn(0x7f3f24096a30 legacy=0x7f3f2409d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 monclient: found mon.noname-a 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f21d9b640 1 -- 172.21.15.181:0/2477583225 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2409c120 con 0x7f3f24096e00 2024-09-15T03:35:23.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.293+0000 7f3f22d9d640 1 --2- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 0x7f3f24137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.296+0000 7f3f21d9b640 1 -- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24096e00 msgr2=0x7f3f24097200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.296+0000 7f3f21d9b640 1 --2- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24096e00 0x7f3f24097200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009910 tx=0x7f3f0c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99411f0c9fb90981 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 1 -- 172.21.15.181:0/2477583225 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c032440 con 0x7f3f24096e00 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: dump: 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.308 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: _finish_auth 0 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 1 -- 172.21.15.181:0/2477583225 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c0325a0 con 0x7f3f24096e00 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 5 monclient: authenticate success, global_id 84468 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:23.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 1 -- 172.21.15.181:0/2477583225 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c032940 con 0x7f3f24096e00 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 4 set_mon_vals no callback set 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f2159a640 10 monclient: dump: 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:23.310 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 10 monclient: shutdown 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 -- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24096e00 msgr2=0x7f3f24097200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 --2- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24096e00 0x7f3f24097200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009910 tx=0x7f3f0c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 -- 172.21.15.181:0/2477583225 shutdown_connections 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 --2- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24134e20 0x7f3f24137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 --2- 172.21.15.181:0/2477583225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24096e00 0x7f3f24097200 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 -- 172.21.15.181:0/2477583225 >> 172.21.15.181:0/2477583225 conn(0x7f3f240711a0 msgr2=0x7f3f240a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 -- 172.21.15.181:0/2477583225 shutdown_connections 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.297+0000 7f3f28a94640 1 -- 172.21.15.181:0/2477583225 wait complete. 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient: build_initial_monmap 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient: monmap: 2024-09-15T03:35:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 1 Processor -- start 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 1 -- start start 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient: init 2024-09-15T03:35:23.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 0x7f3f2411c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient(hunting): picked mon.noname-c con 0x7f3f24096a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24096e00 0x7f3f2411c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 10 monclient(hunting): picked mon.noname-b con 0x7f3f24096e00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.298+0000 7f3f28a94640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24134e20 0x7f3f2411ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 10 monclient(hunting): picked mon.noname-a con 0x7f3f24134e20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2409bba0 con 0x7f3f24134e20 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24069870 con 0x7f3f24096e00 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:23.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24120a50 con 0x7f3f24096a30 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f28a94640 10 monclient(hunting): authenticate will time out at 1127.780031s 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f21d9b640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24096e00 msgr2=0x7f3f2411c8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f21d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24096e00 0x7f3f2411c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 0x7f3f2411c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 0x7f3f2411c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59090/0 (socket says 172.21.15.181:59090) 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 -- 172.21.15.181:0/3702945860 learned_addr learned my addr 172.21.15.181:0/3702945860 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): get_auth_request con 0x7f3f24096a30 auth_method 0 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:23.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24134e20 0x7f3f2411ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): get_auth_request con 0x7f3f24134e20 auth_method 0 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:23.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient(hunting): handle_auth_done global_id 74570 payload 824 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 -- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24096e00 msgr2=0x7f3f2411c8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24096e00 0x7f3f2411c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 -- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24134e20 msgr2=0x7f3f2411ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24134e20 0x7f3f2411ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient: _finish_hunting 0 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 monclient: found mon.noname-c 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:23.316 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.299+0000 7f3f2259c640 1 -- 172.21.15.181:0/3702945860 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c009a20 con 0x7f3f24096a30 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f2259c640 1 -- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 msgr2=0x7f3f2411c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f2259c640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 0x7f3f2411c3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400b5c0 tx=0x7f3f14009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14003d90 con 0x7f3f24096a30 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f1a7fc640 10 monclient: dump: 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:23.317 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f1a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f28a94640 5 monclient: authenticate success, global_id 74570 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f28a94640 10 monclient: _renew_subs 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f28a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.301+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c009450 con 0x7f3f24096a30 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f14006150 con 0x7f3f24096a30 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14006550 con 0x7f3f24096a30 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 10 monclient: dump: 2024-09-15T03:35:23.318 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f20d99640 4 set_mon_vals no callback set 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 10 monclient: _renew_subs 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2411e830 con 0x7f3f24096a30 2024-09-15T03:35:23.319 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f240008d0 con 0x7f3f24096a30 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.302+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f14010570 con 0x7f3f24096a30 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.303+0000 7f3f1a7fc640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f040386f0 0x7f3f0403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.303+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(158..158 src has 106..158) v4 ==== 9849+0+0 (secure 0 0 0) 0x7f3f1404c3e0 con 0x7f3f24096a30 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.303+0000 7f3f21d9b640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f040386f0 0x7f3f0403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.303+0000 7f3f21d9b640 10 monclient: get_auth_request con 0x7f3f040386f0 auth_method 0 2024-09-15T03:35:23.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.305+0000 7f3f21d9b640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f040386f0 0x7f3f0403aba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c031000 tx=0x7f3f0c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:23.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.307+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3f14010820 con 0x7f3f24096a30 2024-09-15T03:35:23.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.307+0000 7f3f1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:23.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.307+0000 7f3f1a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.474+0000 7f3f28a94640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T03:35:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.474+0000 7f3f28a94640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T03:35:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.474+0000 7f3f28a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:23.474+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f3f2411dc60 con 0x7f3f24096a30 2024-09-15T03:35:24.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.414+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v159) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3f14017020 con 0x7f3f24096a30 2024-09-15T03:35:24.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.414+0000 7f3f1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T03:35:24.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.414+0000 7f3f1a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:24.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.518+0000 7f3f28a94640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T03:35:24.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.518+0000 7f3f28a94640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T03:35:24.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.518+0000 7f3f28a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:24.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:24.518+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f3f24118c20 con 0x7f3f24096a30 2024-09-15T03:35:25.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.419+0000 7f3f1a7fc640 1 -- 172.21.15.181:0/3702945860 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v160) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3f1400d290 con 0x7f3f24096a30 2024-09-15T03:35:25.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.419+0000 7f3f1a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-15T03:35:25.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.419+0000 7f3f1a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f040386f0 msgr2=0x7f3f0403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f040386f0 0x7f3f0403aba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c031000 tx=0x7f3f0c032000 comp rx=0 tx=0).stop 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 10 monclient: shutdown 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 msgr2=0x7f3f2411c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 0x7f3f2411c3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400b5c0 tx=0x7f3f14009120 comp rx=0 tx=0).stop 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 shutdown_connections 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3f040386f0 0x7f3f0403aba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f24134e20 0x7f3f2411ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24096e00 0x7f3f2411c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 --2- 172.21.15.181:0/3702945860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f24096a30 0x7f3f2411c3a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 >> 172.21.15.181:0/3702945860 conn(0x7f3f240711a0 msgr2=0x7f3f24069a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:25.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 shutdown_connections 2024-09-15T03:35:25.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.422+0000 7f3f28a94640 1 -- 172.21.15.181:0/3702945860 wait complete. 2024-09-15T03:35:25.435 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.515+0000 7f1da4ae8640 10 monclient: get_monmap_and_config 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.515+0000 7f1da4ae8640 10 monclient: build_initial_monmap 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.515+0000 7f1da4ae8640 10 monclient: monmap: 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:25.515890+0000 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:25.515890+0000 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:25.516 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:25.517 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:25.517 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:25.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.515+0000 7f1da4ae8640 1 Processor -- start 2024-09-15T03:35:25.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.515+0000 7f1da4ae8640 1 -- start start 2024-09-15T03:35:25.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.515+0000 7f1da4ae8640 10 monclient: init 2024-09-15T03:35:25.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:25.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1da009ec50 0x7f1da0131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): picked mon.noname-c con 0x7f1da009ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): picked mon.noname-b con 0x7f1da01321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1da009f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da0067420 con 0x7f1da009f020 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da009fdc0 con 0x7f1da009ec50 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da00a0510 con 0x7f1da01321a0 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:25.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1da4ae8640 10 monclient(hunting): authenticate will time out at 1129.997262s 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d9e59c640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f1da009ec50 msgr2=0x7f1da0131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d9e59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1da009ec50 0x7f1da0131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d9ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1da009f020 0x7f1da0069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:34648/0 (socket says 172.21.15.181:34648) 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d9ed9d640 1 -- 172.21.15.181:0/1226415815 learned_addr learned my addr 172.21.15.181:0/1226415815 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 --2- 172.21.15.181:0/1226415815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59092/0 (socket says 172.21.15.181:59092) 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): get_auth_request con 0x7f1da01321a0 auth_method 0 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient(hunting): handle_auth_done global_id 74573 payload 274 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 -- 172.21.15.181:0/1226415815 >> v2:172.21.15.181:3300/0 conn(0x7f1da009ec50 msgr2=0x7f1da0131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 --2- 172.21.15.181:0/1226415815 >> v2:172.21.15.181:3300/0 conn(0x7f1da009ec50 0x7f1da0131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 -- 172.21.15.181:0/1226415815 >> v1:172.21.15.153:6789/0 conn(0x7f1da009f020 legacy=0x7f1da0069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 monclient: found mon.noname-b 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.516+0000 7f1d95d9b640 1 -- 172.21.15.181:0/1226415815 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da00a0200 con 0x7f1da01321a0 2024-09-15T03:35:25.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.519+0000 7f1d95d9b640 1 -- 172.21.15.181:0/1226415815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 msgr2=0x7f1da0068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:25.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.519+0000 7f1d95d9b640 1 --2- 172.21.15.181:0/1226415815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0068d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009850 tx=0x7f1d880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481ae24e1994a51b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:25.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 1 -- 172.21.15.181:0/1226415815 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88032440 con 0x7f1da01321a0 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: dump: 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:25.532 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 1 -- 172.21.15.181:0/1226415815 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d880325a0 con 0x7f1da01321a0 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 1 -- 172.21.15.181:0/1226415815 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88032940 con 0x7f1da01321a0 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 5 monclient: authenticate success, global_id 74573 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 4 set_mon_vals no callback set 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1d9dd9b640 10 monclient: dump: 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:25.533 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 10 monclient: shutdown 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 -- 172.21.15.181:0/1226415815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 msgr2=0x7f1da0068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/1226415815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0068d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009850 tx=0x7f1d880090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 -- 172.21.15.181:0/1226415815 shutdown_connections 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/1226415815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0068d00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/1226415815 >> v2:172.21.15.181:3300/0 conn(0x7f1da009ec50 0x7f1da0131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 -- 172.21.15.181:0/1226415815 >> 172.21.15.181:0/1226415815 conn(0x7f1da00a5d40 msgr2=0x7f1da00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 -- 172.21.15.181:0/1226415815 shutdown_connections 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.520+0000 7f1da4ae8640 1 -- 172.21.15.181:0/1226415815 wait complete. 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.521+0000 7f1da4ae8640 10 monclient: build_initial_monmap 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.521+0000 7f1da4ae8640 10 monclient: monmap: 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:25.535 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.521+0000 7f1da4ae8640 1 Processor -- start 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.521+0000 7f1da4ae8640 1 -- start start 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.521+0000 7f1da4ae8640 10 monclient: init 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 0x7f1da009ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): picked mon.noname-a con 0x7f1da009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1da009f020 0x7f1da0130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:25.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): picked mon.noname-b con 0x7f1da009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0131180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): picked mon.noname-c con 0x7f1da01321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da009c000 con 0x7f1da009ec50 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da011c7b0 con 0x7f1da009f020 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da00a41b0 con 0x7f1da01321a0 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1da4ae8640 10 monclient(hunting): authenticate will time out at 1130.003207s 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d95d9b640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1da009f020 msgr2=0x7f1da0130c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d95d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1da009f020 0x7f1da0130c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 0x7f1da009ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:25.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0131180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 0x7f1da009ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58738/0 (socket says 172.21.15.181:58738) 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0131180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59094/0 (socket says 172.21.15.181:59094) 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 1 -- 172.21.15.181:0/3279097195 learned_addr learned my addr 172.21.15.181:0/3279097195 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): get_auth_request con 0x7f1da009ec50 auth_method 0 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): get_auth_request con 0x7f1da01321a0 auth_method 0 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:25.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient(hunting): handle_auth_done global_id 74576 payload 824 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 -- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1da009f020 msgr2=0x7f1da0130c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1da009f020 0x7f1da0130c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 -- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 msgr2=0x7f1da009ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 0x7f1da009ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:25.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9ed9d640 1 -- 172.21.15.181:0/3279097195 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d88009390 con 0x7f1da01321a0 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.522+0000 7f1d9e59c640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 0x7f1da009ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d9ed9d640 1 -- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 msgr2=0x7f1da0131180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d9ed9d640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0131180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800cb60 tx=0x7f1d980052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9800a850 con 0x7f1da01321a0 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: dump: 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.540 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9800a9d0 con 0x7f1da01321a0 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 5 monclient: authenticate success, global_id 74576 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9800ad90 con 0x7f1da01321a0 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 10 monclient: _renew_subs 2024-09-15T03:35:25.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da011dd60 con 0x7f1da01321a0 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d96ffd640 10 monclient: dump: 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:25.542 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1d9d59a640 4 set_mon_vals no callback set 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 10 monclient: _renew_subs 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.525+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da011ce70 con 0x7f1da01321a0 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1da4ae8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1da4ae8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1da4ae8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da00008d0 con 0x7f1da01321a0 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1d98007300 con 0x7f1da01321a0 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1d96ffd640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d80038740 0x7f1d8003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(160..160 src has 106..160) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f1d9804cf00 con 0x7f1da01321a0 2024-09-15T03:35:25.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1d9e59c640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d80038740 0x7f1d8003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.526+0000 7f1d9e59c640 10 monclient: get_auth_request con 0x7f1d80038740 auth_method 0 2024-09-15T03:35:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.529+0000 7f1d9e59c640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d80038740 0x7f1d8003abf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009a00 tx=0x7f1d900091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.531+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1d980075b0 con 0x7f1da01321a0 2024-09-15T03:35:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.531+0000 7f1d96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.531+0000 7f1d96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:25.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.694+0000 7f1da4ae8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T03:35:25.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.694+0000 7f1da4ae8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T03:35:25.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.694+0000 7f1da4ae8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:25.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:25.694+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f1da0064900 con 0x7f1da01321a0 2024-09-15T03:35:26.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.428+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v161) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1d98011e90 con 0x7f1da01321a0 2024-09-15T03:35:26.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.428+0000 7f1d96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T03:35:26.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.428+0000 7f1d96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:26.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.528+0000 7f1da4ae8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T03:35:26.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.528+0000 7f1da4ae8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T03:35:26.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.528+0000 7f1da4ae8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:26.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:26.528+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f1da0141700 con 0x7f1da01321a0 2024-09-15T03:35:27.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.122+0000 7f1d96ffd640 1 -- 172.21.15.181:0/3279097195 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v162) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1d98029070 con 0x7f1da01321a0 2024-09-15T03:35:27.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.122+0000 7f1d96ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-15T03:35:27.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.122+0000 7f1d96ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:27.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d80038740 msgr2=0x7f1d8003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d80038740 0x7f1d8003abf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009a00 tx=0x7f1d900091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 10 monclient: shutdown 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 msgr2=0x7f1da0131180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0131180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800cb60 tx=0x7f1d980052b0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 shutdown_connections 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1d80038740 0x7f1d8003abf0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da01321a0 0x7f1da0131180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1da009f020 0x7f1da0130c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 --2- 172.21.15.181:0/3279097195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da009ec50 0x7f1da009ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 >> 172.21.15.181:0/3279097195 conn(0x7f1da00a5d40 msgr2=0x7f1da00a44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 shutdown_connections 2024-09-15T03:35:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.125+0000 7f1da4ae8640 1 -- 172.21.15.181:0/3279097195 wait complete. 2024-09-15T03:35:27.137 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.219+0000 7f5ef9d1d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.219+0000 7f5ef9d1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.219+0000 7f5ef9d1d640 10 monclient: monmap: 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:27.220153+0000 2024-09-15T03:35:27.220 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:27.220153+0000 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:27.221 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:27.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 Processor -- start 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 -- start start 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient: init 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5ef40a19a0 0x7f5ef409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef40a19a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 0x7f5ef409c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef409be20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5ef4099250 0x7f5ef40996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef4099250 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef409c730 con 0x7f5ef4099250 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef40a1d70 con 0x7f5ef40a19a0 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4098600 con 0x7f5ef409be20 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef9d1d640 10 monclient(hunting): authenticate will time out at 1131.701711s 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef37fe640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f5ef40a19a0 msgr2=0x7f5ef409b8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.220+0000 7f5ef37fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5ef40a19a0 0x7f5ef409b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 0x7f5ef409c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5ef4099250 0x7f5ef40996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 0x7f5ef409c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59110/0 (socket says 172.21.15.181:59110) 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5ef4099250 0x7f5ef40996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:58740/0 (socket says 172.21.15.181:58740) 2024-09-15T03:35:27.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 1 -- 172.21.15.181:0/1164501182 learned_addr learned my addr 172.21.15.181:0/1164501182 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f5ef409be20 auth_method 0 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): get_auth_request con 0x7f5ef4099250 auth_method 0 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_done global_id 84477 payload 274 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 -- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 msgr2=0x7f5ef409c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 --2- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 0x7f5ef409c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 -- 172.21.15.181:0/1164501182 >> v2:172.21.15.181:3300/0 conn(0x7f5ef40a19a0 msgr2=0x7f5ef409b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 --2- 172.21.15.181:0/1164501182 >> v2:172.21.15.181:3300/0 conn(0x7f5ef40a19a0 0x7f5ef409b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 monclient: found mon.noname-a 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef3fff640 1 -- 172.21.15.181:0/1164501182 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4097a70 con 0x7f5ef4099250 2024-09-15T03:35:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.221+0000 7f5ef2ffd640 1 --2- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 0x7f5ef409c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:27.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef3fff640 1 -- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef4099250 msgr2=0x7f5ef40996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:27.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef3fff640 1 --2- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef4099250 0x7f5ef40996d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4068b30 tx=0x7f5eec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45395f375e4ff880 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:27.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 1 -- 172.21.15.181:0/1164501182 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec007760 con 0x7f5ef4099250 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: dump: 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:27.234 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 1 -- 172.21.15.181:0/1164501182 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec02d050 con 0x7f5ef4099250 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef9d1d640 5 monclient: authenticate success, global_id 84477 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef9d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef9d1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef9d1d640 4 set_mon_vals no callback set 2024-09-15T03:35:27.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 1 -- 172.21.15.181:0/1164501182 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec007af0 con 0x7f5ef4099250 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.224+0000 7f5ef27fc640 10 monclient: dump: 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:27.236 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 10 monclient: shutdown 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef4099250 msgr2=0x7f5ef40996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef4099250 0x7f5ef40996d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4068b30 tx=0x7f5eec009780 comp rx=0 tx=0).stop 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/1164501182 shutdown_connections 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef4099250 0x7f5ef40996d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/1164501182 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef409be20 0x7f5ef409c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/1164501182 >> v2:172.21.15.181:3300/0 conn(0x7f5ef40a19a0 0x7f5ef409b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/1164501182 >> 172.21.15.181:0/1164501182 conn(0x7f5ef40711a0 msgr2=0x7f5ef40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/1164501182 shutdown_connections 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/1164501182 wait complete. 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.225+0000 7f5ef9d1d640 10 monclient: monmap: 2024-09-15T03:35:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 Processor -- start 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 -- start start 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient: init 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 0x7f5ef413ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef4099250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef409be20 0x7f5ef4132900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef409be20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef40a19a0 0x7f5ef411c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef40a19a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef411df80 con 0x7f5ef40a19a0 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef409cc10 con 0x7f5ef409be20 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef409f0b0 con 0x7f5ef4099250 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef9d1d640 10 monclient(hunting): authenticate will time out at 1131.707712s 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef2ffd640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef409be20 msgr2=0x7f5ef4132900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef409be20 0x7f5ef4132900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 0x7f5ef413ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef40a19a0 0x7f5ef411c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 0x7f5ef413ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:59126/0 (socket says 172.21.15.181:59126) 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 1 -- 172.21.15.181:0/2814054807 learned_addr learned my addr 172.21.15.181:0/2814054807 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 10 monclient(hunting): get_auth_request con 0x7f5ef4099250 auth_method 0 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:27.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef3fff640 10 monclient(hunting): get_auth_request con 0x7f5ef40a19a0 auth_method 0 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.226+0000 7f5ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:27.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 10 monclient(hunting): handle_auth_done global_id 74582 payload 824 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 1 -- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef409be20 msgr2=0x7f5ef4132900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef409be20 0x7f5ef4132900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 1 -- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef40a19a0 msgr2=0x7f5ef411c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef40a19a0 0x7f5ef411c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 1 monclient: found mon.noname-c 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:27.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef37fe640 1 -- 172.21.15.181:0/2814054807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009450 con 0x7f5ef4099250 2024-09-15T03:35:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.227+0000 7f5ef3fff640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef40a19a0 0x7f5ef411c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5ef37fe640 1 -- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 msgr2=0x7f5ef413ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5ef37fe640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 0x7f5ef413ea40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400b660 tx=0x7f5ee40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4014030 con 0x7f5ef4099250 2024-09-15T03:35:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 10 monclient: dump: 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4003d90 con 0x7f5ef4099250 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5ef9d1d640 5 monclient: authenticate success, global_id 74582 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5ef9d1d640 10 monclient: _renew_subs 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5ef9d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.229+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec009a20 con 0x7f5ef4099250 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5edb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4006380 con 0x7f5ef4099250 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5edb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5edb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5edb7fe640 10 monclient: dump: 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5ef9d1d640 10 monclient: _renew_subs 2024-09-15T03:35:27.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5ef9d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef413ef80 con 0x7f5ef4099250 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.230+0000 7f5ef1ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5ee4017070 con 0x7f5ef4099250 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5ef9d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5ef9d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5ef9d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef411fe20 con 0x7f5ef4099250 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5edb7fe640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5ee00386f0 0x7f5ee003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(162..162 src has 106..162) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f5ee4015070 con 0x7f5ef4099250 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5ef2ffd640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5ee00386f0 0x7f5ee003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.231+0000 7f5ef2ffd640 10 monclient: get_auth_request con 0x7f5ee00386f0 auth_method 0 2024-09-15T03:35:27.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.233+0000 7f5ef2ffd640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5ee00386f0 0x7f5ee003aba0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8006c10 tx=0x7f5ee80091d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:27.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.235+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5ee4006730 con 0x7f5ef4099250 2024-09-15T03:35:27.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.236+0000 7f5edb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:27.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.236+0000 7f5edb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:27.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.395+0000 7f5ef9d1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T03:35:27.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.396+0000 7f5ef9d1d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T03:35:27.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.396+0000 7f5ef9d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:27.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:27.396+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7f5ef4120100 con 0x7f5ef4099250 2024-09-15T03:35:28.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.129+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v163) v1 ==== 132+0+0 (secure 0 0 0) 0x7f5ee40193d0 con 0x7f5ef4099250 2024-09-15T03:35:28.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.130+0000 7f5edb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T03:35:28.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.130+0000 7f5edb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:28.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.228+0000 7f5ef9d1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T03:35:28.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.228+0000 7f5ef9d1d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T03:35:28.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.228+0000 7f5ef9d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:28.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:28.228+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7f5ef40008d0 con 0x7f5ef4099250 2024-09-15T03:35:29.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.132+0000 7f5edb7fe640 1 -- 172.21.15.181:0/2814054807 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v164) v1 ==== 132+0+0 (secure 0 0 0) 0x7f5ee400e360 con 0x7f5ef4099250 2024-09-15T03:35:29.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.132+0000 7f5edb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-15T03:35:29.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.132+0000 7f5edb7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5ee00386f0 msgr2=0x7f5ee003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5ee00386f0 0x7f5ee003aba0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8006c10 tx=0x7f5ee80091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 10 monclient: shutdown 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 msgr2=0x7f5ef413ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 0x7f5ef413ea40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400b660 tx=0x7f5ee40091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 shutdown_connections 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5ee00386f0 0x7f5ee003aba0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ef40a19a0 0x7f5ef411c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef409be20 0x7f5ef4132900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 --2- 172.21.15.181:0/2814054807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ef4099250 0x7f5ef413ea40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.135+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 >> 172.21.15.181:0/2814054807 conn(0x7f5ef40711a0 msgr2=0x7f5ef406d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:29.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.136+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 shutdown_connections 2024-09-15T03:35:29.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.136+0000 7f5ef9d1d640 1 -- 172.21.15.181:0/2814054807 wait complete. 2024-09-15T03:35:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep globster 2024-09-15T03:35:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '7\.' 2024-09-15T03:35:29.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.225+0000 7f027191d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.225+0000 7f027191d640 10 monclient: build_initial_monmap 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.225+0000 7f027191d640 10 monclient: monmap: 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:29.226274+0000 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:29.226274+0000 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:29.227 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:29.228 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 1 Processor -- start 2024-09-15T03:35:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 1 -- start start 2024-09-15T03:35:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient: init 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f026c09ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f026c09f020 0x7f026c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient(hunting): picked mon.noname-c con 0x7f026c09f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f026c136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c067420 con 0x7f026c09ec50 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0a0480 con 0x7f026c136d60 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0a0200 con 0x7f026c09f020 2024-09-15T03:35:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f027191d640 10 monclient(hunting): authenticate will time out at 1133.707601s 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f026a7fc640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f026c09f020 msgr2=0x7f026c069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f026b7fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f026c136d60 legacy=0x7f026c139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f026a7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f026c09f020 0x7f026c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f026affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f026c09ec50 0x7f026c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:48442/0 (socket says 172.21.15.181:48442) 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.226+0000 7f026affd640 1 -- 172.21.15.181:0/3634851494 learned_addr learned my addr 172.21.15.181:0/3634851494 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:29.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f026affd640 1 -- 172.21.15.181:0/3634851494 >> v1:172.21.15.153:6789/0 conn(0x7f026c09ec50 legacy=0x7f026c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180429118 0 0) 0x7f026c067420 con 0x7f026c09ec50 2024-09-15T03:35:29.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 10 monclient(hunting): my global_id is 84483 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02540034e0 con 0x7f026c09ec50 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481571637 0 0) 0x7f02540034e0 con 0x7f026c09ec50 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 >> v2:172.21.15.181:3300/0 conn(0x7f026c09f020 msgr2=0x7f026c069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 --2- 172.21.15.181:0/3634851494 >> v2:172.21.15.181:3300/0 conn(0x7f026c09f020 0x7f026c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 >> v1:172.21.15.181:6789/0 conn(0x7f026c136d60 legacy=0x7f026c139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.229+0000 7f0269ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:35:29.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c09fd80 con 0x7f026c09ec50 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f025c001720 con 0x7f026c09ec50 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: dump: 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.243 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:35:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 >> v1:172.21.15.153:6789/0 conn(0x7f026c09ec50 legacy=0x7f026c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0254003a50 0x7f0254003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): picked mon.c con 0x7f0254003a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0254004780 0x7f0254004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): picked mon.b con 0x7f0254004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f025400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): picked mon.a con 0x7f02540099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0a0200 con 0x7f02540099c0 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025400c360 con 0x7f0254004780 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025400c520 con 0x7f0254003a50 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f0269ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026a7fc640 1 -- 172.21.15.181:0/3634851494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0254004780 msgr2=0x7f0254004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026a7fc640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0254004780 0x7f0254004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0254003a50 0x7f0254003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): get_auth_request con 0x7f0254003a50 auth_method 0 2024-09-15T03:35:29.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f025400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): get_auth_request con 0x7f02540099c0 auth_method 0 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient(hunting): handle_auth_done global_id 84483 payload 293 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 -- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0254003a50 msgr2=0x7f0254003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0254003a50 0x7f0254003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 -- 172.21.15.181:0/3634851494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0254004780 msgr2=0x7f0254004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0254004780 0x7f0254004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 monclient: found mon.a 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 1 -- 172.21.15.181:0/3634851494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02540034e0 con 0x7f02540099c0 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f026affd640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0254003a50 0x7f0254003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f027191d640 5 monclient: authenticate success, global_id 84483 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.230+0000 7f027191d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.231+0000 7f026b7fe640 1 -- 172.21.15.181:0/3634851494 reap_dead start 2024-09-15T03:35:29.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.233+0000 7f026b7fe640 1 -- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 msgr2=0x7f025400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.233+0000 7f026b7fe640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f025400bda0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0264009f90 tx=0x7f02640095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8969f806f65dd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.233+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02640050d0 con 0x7f02540099c0 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.233+0000 7f0269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.233+0000 7f0269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.233+0000 7f0269ffb640 10 monclient: dump: 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.250 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026402d070 con 0x7f02540099c0 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f0269ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 4 set_mon_vals no callback set 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f0269ffb640 1 -- 172.21.15.181:0/3634851494 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02640054a0 con 0x7f02540099c0 2024-09-15T03:35:29.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f0269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f0269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f0269ffb640 10 monclient: dump: 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.252 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 10 monclient: shutdown 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 -- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 msgr2=0x7f025400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f025400bda0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0264009f90 tx=0x7f02640095d0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 -- 172.21.15.181:0/3634851494 shutdown_connections 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 --2- 172.21.15.181:0/3634851494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f025400bda0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 -- 172.21.15.181:0/3634851494 >> 172.21.15.181:0/3634851494 conn(0x7f026c0a5cf0 msgr2=0x7f026c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 -- 172.21.15.181:0/3634851494 shutdown_connections 2024-09-15T03:35:29.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.234+0000 7f027191d640 1 -- 172.21.15.181:0/3634851494 wait complete. 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient: build_initial_monmap 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient: monmap: 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:29.254 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 Processor -- start 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 -- start start 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient: init 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f026c099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient(hunting): picked mon.noname-a con 0x7f02540099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026c099d50 0x7f026c09a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient(hunting): picked mon.noname-c con 0x7f026c099d50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:29.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026c09a6e0 0x7f026c147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient(hunting): picked mon.noname-b con 0x7f026c09a6e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c09d640 con 0x7f02540099c0 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c09b6b0 con 0x7f026c09a6e0 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.235+0000 7f027191d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c09c020 con 0x7f026c099d50 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f027191d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f027191d640 10 monclient(hunting): authenticate will time out at 1133.717007s 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026b7fe640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026c09a6e0 msgr2=0x7f026c147200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026c09a6e0 0x7f026c147200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f026c099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026c099d50 0x7f026c09a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f026c099810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:59604/0 (socket says 172.21.15.181:59604) 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): get_auth_request con 0x7f026c099d50 auth_method 0 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 -- 172.21.15.181:0/2713505100 learned_addr learned my addr 172.21.15.181:0/2713505100 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): get_auth_request con 0x7f02540099c0 auth_method 0 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient(hunting): handle_auth_done global_id 84486 payload 824 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 -- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026c099d50 msgr2=0x7f026c09a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026c099d50 0x7f026c09a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 -- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026c09a6e0 msgr2=0x7f026c147200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026c09a6e0 0x7f026c147200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 monclient: found mon.noname-a 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026affd640 1 -- 172.21.15.181:0/2713505100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c11d0f0 con 0x7f02540099c0 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.236+0000 7f026a7fc640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026c099d50 0x7f026c09a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f026affd640 1 -- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 msgr2=0x7f026c099810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f026affd640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f026c099810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f025c001350 tx=0x7f025c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c018030 con 0x7f02540099c0 2024-09-15T03:35:29.259 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: dump: 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.260 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c00ca90 con 0x7f02540099c0 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 5 monclient: authenticate success, global_id 84486 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 10 monclient: _renew_subs 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c11c850 con 0x7f02540099c0 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c00ce10 con 0x7f02540099c0 2024-09-15T03:35:29.261 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f0252ffd640 10 monclient: dump: 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.262 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 10 monclient: _renew_subs 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c11baf0 con 0x7f02540099c0 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.239+0000 7f02697fa640 4 set_mon_vals no callback set 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f027191d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f027191d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f027191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c0008d0 con 0x7f02540099c0 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f025c01b070 con 0x7f02540099c0 2024-09-15T03:35:29.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f0252ffd640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02580386f0 0x7f025803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(164..164 src has 106..164) v4 ==== 9845+0+0 (secure 0 0 0) 0x7f025c019070 con 0x7f02540099c0 2024-09-15T03:35:29.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f026a7fc640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02580386f0 0x7f025803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.240+0000 7f026a7fc640 10 monclient: get_auth_request con 0x7f02580386f0 auth_method 0 2024-09-15T03:35:29.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.243+0000 7f026a7fc640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02580386f0 0x7f025803aba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0260003940 tx=0x7f0260009120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.245+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f025c01b6e0 con 0x7f02540099c0 2024-09-15T03:35:29.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.245+0000 7f0252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.245+0000 7f0252ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:29.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.408+0000 7f027191d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.408+0000 7f027191d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.408+0000 7f027191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.408+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f026c0a13e0 con 0x7f02540099c0 2024-09-15T03:35:29.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.409+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f025c01b320 con 0x7f02540099c0 2024-09-15T03:35:29.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.409+0000 7f0252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.409+0000 7f0252ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:29.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.509+0000 7f027191d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.509+0000 7f027191d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.509+0000 7f027191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.509+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f026c072640 con 0x7f02540099c0 2024-09-15T03:35:29.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.510+0000 7f0252ffd640 1 -- 172.21.15.181:0/2713505100 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f025c02c090 con 0x7f02540099c0 2024-09-15T03:35:29.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.510+0000 7f0252ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.510+0000 7f0252ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02580386f0 msgr2=0x7f025803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02580386f0 0x7f025803aba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0260003940 tx=0x7f0260009120 comp rx=0 tx=0).stop 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 10 monclient: shutdown 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 msgr2=0x7f026c099810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f026c099810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f025c001350 tx=0x7f025c007a90 comp rx=0 tx=0).stop 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 shutdown_connections 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02580386f0 0x7f025803aba0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f026c09a6e0 0x7f026c147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f026c099d50 0x7f026c09a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 --2- 172.21.15.181:0/2713505100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02540099c0 0x7f026c099810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 >> 172.21.15.181:0/2713505100 conn(0x7f026c0a5cf0 msgr2=0x7f026c0a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:29.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 shutdown_connections 2024-09-15T03:35:29.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.512+0000 7f027191d640 1 -- 172.21.15.181:0/2713505100 wait complete. 2024-09-15T03:35:29.522 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T03:35:29.522 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T03:35:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep globster 2024-09-15T03:35:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '8\.' 2024-09-15T03:35:29.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.596+0000 7fc21a11d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:29.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.596+0000 7fc21a11d640 10 monclient: build_initial_monmap 2024-09-15T03:35:29.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.596+0000 7fc21a11d640 10 monclient: monmap: 2024-09-15T03:35:29.599 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:29.596822+0000 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:29.596822+0000 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.596+0000 7fc21a11d640 1 Processor -- start 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.596+0000 7fc21a11d640 1 -- start start 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.596+0000 7fc21a11d640 10 monclient: init 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc214131c00 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2141326e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc2140a0b00 0x7fc214138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2140a0b00 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140a1b70 con 0x7fc214131c00 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2140a08c0 con 0x7fc2140a0b00 2024-09-15T03:35:29.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a0200 con 0x7fc2141326e0 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc21a11d640 10 monclient(hunting): authenticate will time out at 1134.078297s 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc2137fe640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fc214131c00 legacy=0x7fc214131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc2140a0b00 0x7fc214138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35310/0 (socket says 172.21.15.181:35310) 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc2140a0b00 0x7fc214138fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:59616/0 (socket says 172.21.15.181:59616) 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 1 -- 172.21.15.181:0/3142059768 learned_addr learned my addr 172.21.15.181:0/3142059768 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): get_auth_request con 0x7fc2141326e0 auth_method 0 2024-09-15T03:35:29.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): get_auth_request con 0x7fc2140a0b00 auth_method 0 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 10 monclient(hunting): handle_auth_done global_id 74588 payload 274 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.597+0000 7fc212ffd640 1 -- 172.21.15.181:0/3142059768 >> v2:172.21.15.153:3300/0 conn(0x7fc2140a0b00 msgr2=0x7fc214138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 1 --2- 172.21.15.181:0/3142059768 >> v2:172.21.15.153:3300/0 conn(0x7fc2140a0b00 0x7fc214138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 1 -- 172.21.15.181:0/3142059768 >> v1:172.21.15.181:6789/0 conn(0x7fc214131c00 legacy=0x7fc214131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 1 monclient: found mon.noname-b 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.598+0000 7fc212ffd640 1 -- 172.21.15.181:0/3142059768 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2140a1890 con 0x7fc2141326e0 2024-09-15T03:35:29.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.600+0000 7fc212ffd640 1 -- 172.21.15.181:0/3142059768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 msgr2=0x7fc214069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.600+0000 7fc212ffd640 1 --2- 172.21.15.181:0/3142059768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214069d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc208009910 tx=0x7fc2080091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fa9273a95fb6a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 1 -- 172.21.15.181:0/3142059768 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20802aea0 con 0x7fc2141326e0 2024-09-15T03:35:29.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: dump: 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 1 -- 172.21.15.181:0/3142059768 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc208032440 con 0x7fc2141326e0 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 5 monclient: authenticate success, global_id 74588 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 4 set_mon_vals no callback set 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 1 -- 172.21.15.181:0/3142059768 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2080327e0 con 0x7fc2141326e0 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc2127fc640 10 monclient: dump: 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.612 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 10 monclient: shutdown 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 -- 172.21.15.181:0/3142059768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 msgr2=0x7fc214069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3142059768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214069d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc208009910 tx=0x7fc2080091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 -- 172.21.15.181:0/3142059768 shutdown_connections 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3142059768 >> v2:172.21.15.153:3300/0 conn(0x7fc2140a0b00 0x7fc214138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3142059768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214069d10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 -- 172.21.15.181:0/3142059768 >> 172.21.15.181:0/3142059768 conn(0x7fc214071370 msgr2=0x7fc2140a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 -- 172.21.15.181:0/3142059768 shutdown_connections 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.601+0000 7fc21a11d640 1 -- 172.21.15.181:0/3142059768 wait complete. 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.602+0000 7fc21a11d640 10 monclient: build_initial_monmap 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.602+0000 7fc21a11d640 10 monclient: monmap: 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.602+0000 7fc21a11d640 1 Processor -- start 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.602+0000 7fc21a11d640 1 -- start start 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.602+0000 7fc21a11d640 10 monclient: init 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2140a0b00 0x7fc21409a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2140a0b00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc214131c00 0x7fc214096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): picked mon.noname-b con 0x7fc214131c00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2141326e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a1040 con 0x7fc2140a0b00 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21409e840 con 0x7fc214131c00 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc214098a40 con 0x7fc2141326e0 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc21a11d640 10 monclient(hunting): authenticate will time out at 1134.084182s 2024-09-15T03:35:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc212ffd640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc214131c00 msgr2=0x7fc214096a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc212ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc214131c00 0x7fc214096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2140a0b00 0x7fc21409a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35316/0 (socket says 172.21.15.181:35316) 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 -- 172.21.15.181:0/3816544113 learned_addr learned my addr 172.21.15.181:0/3816544113 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): get_auth_request con 0x7fc2141326e0 auth_method 0 2024-09-15T03:35:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): get_auth_request con 0x7fc2140a0b00 auth_method 0 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc2137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient(hunting): handle_auth_done global_id 74591 payload 824 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 -- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc214131c00 msgr2=0x7fc214096a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc214131c00 0x7fc214096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 -- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2140a0b00 msgr2=0x7fc21409a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2140a0b00 0x7fc21409a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 monclient: found mon.noname-c 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.603+0000 7fc213fff640 1 -- 172.21.15.181:0/3816544113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208009450 con 0x7fc2141326e0 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc213fff640 1 -- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 msgr2=0x7fc214096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc213fff640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214096f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc20c009e90 tx=0x7fc20c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c014480 con 0x7fc2141326e0 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: dump: 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c00f040 con 0x7fc2141326e0 2024-09-15T03:35:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 5 monclient: authenticate success, global_id 74591 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c014830 con 0x7fc2141326e0 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 10 monclient: _renew_subs 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21409a240 con 0x7fc2141326e0 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc211ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc1fb7fe640 10 monclient: dump: 2024-09-15T03:35:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 10 monclient: _renew_subs 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.606+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21409aba0 con 0x7fc2141326e0 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc21a11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc21a11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc21a11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2140008d0 con 0x7fc2141326e0 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc20c014be0 con 0x7fc2141326e0 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc1fb7fe640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2000386f0 0x7fc20003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(164..164 src has 106..164) v4 ==== 9845+0+0 (secure 0 0 0) 0x7fc20c04daf0 con 0x7fc2141326e0 2024-09-15T03:35:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc2137fe640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2000386f0 0x7fc20003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.607+0000 7fc2137fe640 10 monclient: get_auth_request con 0x7fc2000386f0 auth_method 0 2024-09-15T03:35:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.609+0000 7fc2137fe640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2000386f0 0x7fc20003aba0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc204003850 tx=0x7fc2040090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.612+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc20c01c3b0 con 0x7fc2141326e0 2024-09-15T03:35:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.612+0000 7fc1fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.612+0000 7fc1fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:29.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.772+0000 7fc21a11d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.772+0000 7fc21a11d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.772+0000 7fc21a11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.772+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc2140a0440 con 0x7fc2141326e0 2024-09-15T03:35:29.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.773+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fc20c03ce60 con 0x7fc2141326e0 2024-09-15T03:35:29.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.773+0000 7fc1fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.773+0000 7fc1fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:29.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.871+0000 7fc21a11d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.871+0000 7fc21a11d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.871+0000 7fc21a11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.871+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc21409ae20 con 0x7fc2141326e0 2024-09-15T03:35:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.872+0000 7fc1fb7fe640 1 -- 172.21.15.181:0/3816544113 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fc20c041090 con 0x7fc2141326e0 2024-09-15T03:35:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.872+0000 7fc1fb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.872+0000 7fc1fb7fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:29.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2000386f0 msgr2=0x7fc20003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2000386f0 0x7fc20003aba0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc204003850 tx=0x7fc2040090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 10 monclient: shutdown 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 msgr2=0x7fc214096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214096f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc20c009e90 tx=0x7fc20c00b040 comp rx=0 tx=0).stop 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 shutdown_connections 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc2000386f0 0x7fc20003aba0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2141326e0 0x7fc214096f70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc214131c00 0x7fc214096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 --2- 172.21.15.181:0/3816544113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2140a0b00 0x7fc21409a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 >> 172.21.15.181:0/3816544113 conn(0x7fc214071370 msgr2=0x7fc21409d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 shutdown_connections 2024-09-15T03:35:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.874+0000 7fc21a11d640 1 -- 172.21.15.181:0/3816544113 wait complete. 2024-09-15T03:35:29.884 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T03:35:29.884 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T03:35:29.885 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush tree --show-shadow 2024-09-15T03:35:29.885 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep 'osd\.0' 2024-09-15T03:35:29.885 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep globster 2024-09-15T03:35:29.885 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep '6\.' 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.957+0000 7fe6b504f640 10 monclient: get_monmap_and_config 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.957+0000 7fe6b504f640 10 monclient: build_initial_monmap 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.957+0000 7fe6b504f640 10 monclient: monmap: 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:29.957943+0000 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:29.957943+0000 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:29.958 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:29.959 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.957+0000 7fe6b504f640 1 Processor -- start 2024-09-15T03:35:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.957+0000 7fe6b504f640 1 -- start start 2024-09-15T03:35:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.957+0000 7fe6b504f640 10 monclient: init 2024-09-15T03:35:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe6b009ec50 0x7fe6b0131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b009ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6b01321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe6b009f020 0x7fe6b0069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b009f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b0067420 con 0x7fe6b01321a0 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b00a0480 con 0x7fe6b009ec50 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b00a0200 con 0x7fe6b009f020 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6b504f640 10 monclient(hunting): authenticate will time out at 1134.439337s 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6af59e640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fe6b009f020 msgr2=0x7fe6b0069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6a659c640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7fe6b01321a0 legacy=0x7fe6b0068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6af59e640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe6b009f020 0x7fe6b0069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe6b009ec50 0x7fe6b0131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe6b009ec50 0x7fe6b0131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:59624/0 (socket says 172.21.15.181:59624) 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 -- 172.21.15.181:0/3731628165 learned_addr learned my addr 172.21.15.181:0/3731628165 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request con 0x7fe6b009ec50 auth_method 0 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_done global_id 84495 payload 274 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 -- 172.21.15.181:0/3731628165 >> v2:172.21.15.181:3300/0 conn(0x7fe6b009f020 msgr2=0x7fe6b0069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/3731628165 >> v2:172.21.15.181:3300/0 conn(0x7fe6b009f020 0x7fe6b0069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 -- 172.21.15.181:0/3731628165 >> v1:172.21.15.181:6789/0 conn(0x7fe6b01321a0 legacy=0x7fe6b0068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 monclient: found mon.noname-a 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.958+0000 7fe6aed9d640 1 -- 172.21.15.181:0/3731628165 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b009fd80 con 0x7fe6b009ec50 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.961+0000 7fe6aed9d640 1 -- 172.21.15.181:0/3731628165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 msgr2=0x7fe6b0131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.961+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/3731628165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b0131c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe6b009bf50 tx=0x7fe6a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa98014038983154 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 1 -- 172.21.15.181:0/3731628165 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0032440 con 0x7fe6b009ec50 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: dump: 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.975 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 1 -- 172.21.15.181:0/3731628165 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a00325a0 con 0x7fe6b009ec50 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 5 monclient: authenticate success, global_id 84495 2024-09-15T03:35:29.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 1 -- 172.21.15.181:0/3731628165 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0032940 con 0x7fe6b009ec50 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 4 set_mon_vals no callback set 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6ae59c640 10 monclient: dump: 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.977 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 10 monclient: shutdown 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 -- 172.21.15.181:0/3731628165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 msgr2=0x7fe6b0131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 --2- 172.21.15.181:0/3731628165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b0131c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe6b009bf50 tx=0x7fe6a00091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 -- 172.21.15.181:0/3731628165 shutdown_connections 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 --2- 172.21.15.181:0/3731628165 >> v2:172.21.15.181:3300/0 conn(0x7fe6b009f020 0x7fe6b0069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 --2- 172.21.15.181:0/3731628165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b0131c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 -- 172.21.15.181:0/3731628165 >> 172.21.15.181:0/3731628165 conn(0x7fe6b00a5cf0 msgr2=0x7fe6b00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 -- 172.21.15.181:0/3731628165 shutdown_connections 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.962+0000 7fe6b504f640 1 -- 172.21.15.181:0/3731628165 wait complete. 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.963+0000 7fe6b504f640 10 monclient: build_initial_monmap 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.963+0000 7fe6b504f640 10 monclient: monmap: 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:29.979 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.963+0000 7fe6b504f640 1 Processor -- start 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.963+0000 7fe6b504f640 1 -- start start 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.963+0000 7fe6b504f640 10 monclient: init 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6b011e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe6b011ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b01321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b009c020 con 0x7fe6b009ec50 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00a4500 con 0x7fe6b009f020 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0121980 con 0x7fe6b01321a0 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6b504f640 10 monclient(hunting): authenticate will time out at 1134.445119s 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6a659c640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 msgr2=0x7fe6b011e7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6a659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6b011e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe6b011ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:59638/0 (socket says 172.21.15.181:59638) 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe6b011ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35330/0 (socket says 172.21.15.181:35330) 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 learned_addr learned my addr 172.21.15.181:0/1846023660 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request con 0x7fe6b009ec50 auth_method 0 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): get_auth_request con 0x7fe6b01321a0 auth_method 0 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_done global_id 84498 payload 824 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 msgr2=0x7fe6b011ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe6b011ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 msgr2=0x7fe6b011e7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6b011e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 monclient: found mon.noname-a 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a0009450 con 0x7fe6b009ec50 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.964+0000 7fe6af59e640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe6b011ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.966+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 msgr2=0x7fe6b011e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.966+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035040 tx=0x7fe6a0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a003c960 con 0x7fe6b009ec50 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: dump: 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a003cac0 con 0x7fe6b009ec50 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 5 monclient: authenticate success, global_id 84498 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 10 monclient: _renew_subs 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a003ce40 con 0x7fe6b009ec50 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b01228c0 con 0x7fe6b009ec50 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6a77fe640 10 monclient: dump: 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.986 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6add9b640 4 set_mon_vals no callback set 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 10 monclient: _renew_subs 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.967+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b0120740 con 0x7fe6b009ec50 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6b504f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6b504f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6b504f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b00008d0 con 0x7fe6b009ec50 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe6a0039040 con 0x7fe6b009ec50 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 msgr2=0x7fe6b011e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 msgr2=0x7fe6b011e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035040 tx=0x7fe6a0032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035040 tx=0x7fe6a0032000 comp rx=0 tx=0).stop 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe690038740 0x7fe69003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 msgr2=0x7fe6b011e260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b009ec50 0x7fe6b011e260 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 0x7fe69003d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): picked mon.a con 0x7fe69003b300 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6900400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): picked mon.b con 0x7fe6b009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe690042a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): picked mon.c con 0x7fe6b01321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900433f0 con 0x7fe69003b300 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690043530 con 0x7fe6b009f020 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe690043670 con 0x7fe6b01321a0 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a77fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe690038740 0x7fe69003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6af59e640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 msgr2=0x7fe6900400c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request con 0x7fe690038740 auth_method 0 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6af59e640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6900400c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 0x7fe69003d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): get_auth_request con 0x7fe69003b300 auth_method 0 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe690042a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request con 0x7fe6b01321a0 auth_method 0 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient(hunting): handle_auth_done global_id 84498 payload 293 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 msgr2=0x7fe690042a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:29.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe690042a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 msgr2=0x7fe6900400c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6900400c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient: _finish_hunting 0 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 monclient: found mon.a 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fe6900437b0 con 0x7fe69003b300 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe698009990 con 0x7fe69003b300 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 10 monclient: _finish_auth 0 2024-09-15T03:35:29.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.968+0000 7fe6a659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.971+0000 7fe6aed9d640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe690038740 0x7fe69003abf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035040 tx=0x7fe6a0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.971+0000 7fe6a659c640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 msgr2=0x7fe69003d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.971+0000 7fe6a659c640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 0x7fe69003d6e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe698009df0 tx=0x7fe698009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698003bd0 con 0x7fe69003b300 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 10 monclient: dump: 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:29.993 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe698003d50 con 0x7fe69003b300 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6ad59a640 4 set_mon_vals no callback set 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe69800b8e0 con 0x7fe69003b300 2024-09-15T03:35:29.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.973+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 4 ==== osd_map(164..164 src has 106..164) v4 ==== 9845+0+0 (secure 0 0 0) 0x7fe69804a1d0 con 0x7fe69003b300 2024-09-15T03:35:29.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.978+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe69800d1c0 con 0x7fe69003b300 2024-09-15T03:35:29.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.978+0000 7fe6a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:29.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:29.978+0000 7fe6a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:30.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.137+0000 7fe6b504f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:30.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.137+0000 7fe6b504f640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:30.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.137+0000 7fe6b504f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:30.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.137+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fe6b0121ac0 con 0x7fe69003b300 2024-09-15T03:35:30.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.138+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fe69800d1c0 con 0x7fe69003b300 2024-09-15T03:35:30.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.138+0000 7fe6a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:30.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.138+0000 7fe6a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.235+0000 7fe6b504f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.235+0000 7fe6b504f640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.235+0000 7fe6b504f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.235+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fe6b00a13e0 con 0x7fe69003b300 2024-09-15T03:35:30.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.236+0000 7fe6a77fe640 1 -- 172.21.15.181:0/1846023660 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fe69800d1c0 con 0x7fe69003b300 2024-09-15T03:35:30.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.236+0000 7fe6a77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-15T03:35:30.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.236+0000 7fe6a77fe640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe690038740 msgr2=0x7fe69003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe690038740 0x7fe69003abf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035040 tx=0x7fe6a0032000 comp rx=0 tx=0).stop 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 10 monclient: shutdown 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 msgr2=0x7fe69003d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 0x7fe69003d6e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe698009df0 tx=0x7fe698009e20 comp rx=0 tx=0).stop 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 shutdown_connections 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6b01321a0 0x7fe690042a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b009f020 0x7fe6900400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe69003b300 0x7fe69003d6e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 --2- 172.21.15.181:0/1846023660 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe690038740 0x7fe69003abf0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 >> 172.21.15.181:0/1846023660 conn(0x7fe6b00a5cf0 msgr2=0x7fe6b009b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:30.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.237+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 shutdown_connections 2024-09-15T03:35:30.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.238+0000 7fe6b504f640 1 -- 172.21.15.181:0/1846023660 wait complete. 2024-09-15T03:35:30.247 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T03:35:30.247 INFO:tasks.workunit.client.0.smithi181.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-15T03:35:30.247 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.321+0000 7f3f0775d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.321+0000 7f3f0775d640 10 monclient: build_initial_monmap 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.321+0000 7f3f0775d640 10 monclient: monmap: 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:30.322054+0000 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:30.322054+0000 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:30.322 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:30.323 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:30.323 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:30.323 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:30.323 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:30.323 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:30.323 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:30.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.321+0000 7f3f0775d640 1 Processor -- start 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 1 -- start start 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient: init 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 0x7f3f00131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f0009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f001321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f0009f020 0x7f3f00069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0009f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f00067420 con 0x7f3f001321a0 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:30.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000a0480 con 0x7f3f0009f020 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000a0200 con 0x7f3f0009ec50 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f0775d640 10 monclient(hunting): authenticate will time out at 1134.803630s 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f054f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 0x7f3f00131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f05cfa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f0009f020 0x7f3f00069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f04cf8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3f001321a0 0x7f3f00068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:48450/0 (socket says 172.21.15.181:48450) 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f054f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 0x7f3f00131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35356/0 (socket says 172.21.15.181:35356) 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f04cf8640 1 -- 172.21.15.181:0/637686499 learned_addr learned my addr 172.21.15.181:0/637686499 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f054f9640 10 monclient(hunting): get_auth_request con 0x7f3f0009ec50 auth_method 0 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f054f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f054f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:30.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f054f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f05cfa640 10 monclient(hunting): get_auth_request con 0x7f3f0009f020 auth_method 0 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f05cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f05cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.322+0000 7f3f05cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f054f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f054f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f054f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_done global_id 84501 payload 274 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 1 -- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 msgr2=0x7f3f00131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 1 --2- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 0x7f3f00131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 1 -- 172.21.15.181:0/637686499 >> v1:172.21.15.153:6789/0 conn(0x7f3f001321a0 legacy=0x7f3f00068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:30.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:35:30.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 1 monclient: found mon.noname-a 2024-09-15T03:35:30.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:30.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:30.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f05cfa640 1 -- 172.21.15.181:0/637686499 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0009fd80 con 0x7f3f0009f020 2024-09-15T03:35:30.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.323+0000 7f3f054f9640 1 --2- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 0x7f3f00131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3f05cfa640 1 -- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009f020 msgr2=0x7f3f00069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3f05cfa640 1 --2- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009f020 0x7f3f00069410 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8009e60 tx=0x7f3ef8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3dc3d09f0e92cb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 1 -- 172.21.15.181:0/637686499 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8007760 con 0x7f3f0009f020 2024-09-15T03:35:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: dump: 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:30.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 1 -- 172.21.15.181:0/637686499 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef802d050 con 0x7f3f0009f020 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3f0775d640 5 monclient: authenticate success, global_id 84501 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 1 -- 172.21.15.181:0/637686499 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8007af0 con 0x7f3f0009f020 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3ef7fff640 10 monclient: dump: 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:30.337 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3f0775d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.326+0000 7f3f0775d640 4 set_mon_vals no callback set 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 10 monclient: shutdown 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 -- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009f020 msgr2=0x7f3f00069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 --2- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009f020 0x7f3f00069410 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8009e60 tx=0x7f3ef8009780 comp rx=0 tx=0).stop 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 -- 172.21.15.181:0/637686499 shutdown_connections 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 --2- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009f020 0x7f3f00069410 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 --2- 172.21.15.181:0/637686499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0009ec50 0x7f3f00131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 -- 172.21.15.181:0/637686499 >> 172.21.15.181:0/637686499 conn(0x7f3f000a5cf0 msgr2=0x7f3f000a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 -- 172.21.15.181:0/637686499 shutdown_connections 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 1 -- 172.21.15.181:0/637686499 wait complete. 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 10 monclient: build_initial_monmap 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.327+0000 7f3f0775d640 10 monclient: monmap: 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 Processor -- start 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 -- start start 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient: init 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 0x7f3f0009a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0009f020 0x7f3f00130880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f0009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 0x7f3f0011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f001321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0011d9a0 con 0x7f3f0009ec50 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0011d740 con 0x7f3f0009f020 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00134ad0 con 0x7f3f001321a0 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f0775d640 10 monclient(hunting): authenticate will time out at 1134.809757s 2024-09-15T03:35:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f04cf8640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0009f020 msgr2=0x7f3f00130880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.328+0000 7f3f04cf8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0009f020 0x7f3f00130880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 0x7f3f0009a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 0x7f3f0011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 0x7f3f0009a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:59672/0 (socket says 172.21.15.181:59672) 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 1 -- 172.21.15.181:0/3679602733 learned_addr learned my addr 172.21.15.181:0/3679602733 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): get_auth_request con 0x7f3f001321a0 auth_method 0 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): get_auth_request con 0x7f3f0009ec50 auth_method 0 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient(hunting): handle_auth_done global_id 74600 payload 824 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 -- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0009f020 msgr2=0x7f3f00130880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0009f020 0x7f3f00130880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 -- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 msgr2=0x7f3f0009a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 0x7f3f0009a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 monclient: found mon.noname-c 2024-09-15T03:35:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f05cfa640 1 -- 172.21.15.181:0/3679602733 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc009390 con 0x7f3f001321a0 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.329+0000 7f3f054f9640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 0x7f3f0009a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.331+0000 7f3f05cfa640 1 -- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 msgr2=0x7f3f0011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.331+0000 7f3f05cfa640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 0x7f3f0011b760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8031000 tx=0x7f3ef8037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8031ac0 con 0x7f3f001321a0 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: dump: 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef8040070 con 0x7f3f001321a0 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 5 monclient: authenticate success, global_id 74600 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 10 monclient: _renew_subs 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef8009a20 con 0x7f3f001321a0 2024-09-15T03:35:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8031e70 con 0x7f3f001321a0 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef57fa640 10 monclient: dump: 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:30.346 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3ef77fe640 4 set_mon_vals no callback set 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 10 monclient: _renew_subs 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.332+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0011ca40 con 0x7f3f001321a0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3ef8042030 con 0x7f3f001321a0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3f0775d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3f0775d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3f0775d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0011dcb0 con 0x7f3f001321a0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3ef57fa640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ee4038740 0x7f3ee403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(164..164 src has 106..164) v4 ==== 9845+0+0 (secure 0 0 0) 0x7f3ef806e780 con 0x7f3f001321a0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3f054f9640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ee4038740 0x7f3ee403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:30.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.333+0000 7f3f054f9640 10 monclient: get_auth_request con 0x7f3ee4038740 auth_method 0 2024-09-15T03:35:30.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.335+0000 7f3f054f9640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ee4038740 0x7f3ee403abf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fd0 tx=0x7f3efc0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:30.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.337+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3ef8044bb0 con 0x7f3f001321a0 2024-09-15T03:35:30.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.337+0000 7f3ef57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:30.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.337+0000 7f3ef57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:30.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.495+0000 7f3f0775d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:30.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.495+0000 7f3f0775d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:30.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.495+0000 7f3f0775d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:30.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:30.495+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f3f000008d0 con 0x7f3f001321a0 2024-09-15T03:35:31.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.140+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v165) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3ef80443d0 con 0x7f3f001321a0 2024-09-15T03:35:31.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.140+0000 7f3ef57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:31.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.140+0000 7f3ef57fa640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-15T03:35:31.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.237+0000 7f3f0775d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:31.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.237+0000 7f3f0775d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:31.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.237+0000 7f3f0775d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:31.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:31.237+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f3f000008d0 con 0x7f3f001321a0 2024-09-15T03:35:32.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.121+0000 7f3ef57fa640 1 -- 172.21.15.181:0/3679602733 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v166) v1 ==== 149+0+0 (secure 0 0 0) 0x7f3ef80445b0 con 0x7f3f001321a0 2024-09-15T03:35:32.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.121+0000 7f3ef57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-15T03:35:32.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.121+0000 7f3ef57fa640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T03:35:32.123 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-15T03:35:32.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ee4038740 msgr2=0x7f3ee403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:32.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ee4038740 0x7f3ee403abf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fd0 tx=0x7f3efc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 10 monclient: shutdown 2024-09-15T03:35:32.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 msgr2=0x7f3f0011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 0x7f3f0011b760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8031000 tx=0x7f3ef8037000 comp rx=0 tx=0).stop 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 shutdown_connections 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ee4038740 0x7f3ee403abf0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f001321a0 0x7f3f0011b760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0009f020 0x7f3f00130880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 --2- 172.21.15.181:0/3679602733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0009ec50 0x7f3f0009a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 >> 172.21.15.181:0/3679602733 conn(0x7f3f000a5cf0 msgr2=0x7f3f00134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 shutdown_connections 2024-09-15T03:35:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.124+0000 7f3f0775d640 1 -- 172.21.15.181:0/3679602733 wait complete. 2024-09-15T03:35:32.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-15T03:35:32.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.214+0000 7f4cab31d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:32.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.214+0000 7f4cab31d640 10 monclient: build_initial_monmap 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.214+0000 7f4cab31d640 10 monclient: monmap: 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:32.215515+0000 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:32.215515+0000 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:32.216 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:32.217 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:32.217 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:32.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 1 Processor -- start 2024-09-15T03:35:32.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 1 -- start start 2024-09-15T03:35:32.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 10 monclient: init 2024-09-15T03:35:32.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:32.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:35:32.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ca4096a30 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 0x7f4ca4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca4099f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4ca409d4e0 0x7f4ca41360e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.215+0000 7f4cab31d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca409d4e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca409cb80 con 0x7f4ca4096a30 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca4098c50 con 0x7f4ca409d4e0 2024-09-15T03:35:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca4098f30 con 0x7f4ca4099f00 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4cab31d640 10 monclient(hunting): authenticate will time out at 1136.697039s 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca98ba640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f4ca409d4e0 msgr2=0x7f4ca41360e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca98ba640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4ca409d4e0 0x7f4ca41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 0x7f4ca4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca90b9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4ca4096a30 0x7f4ca4096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:48458/0 (socket says 172.21.15.181:48458) 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca90b9640 1 -- 172.21.15.181:0/3094880183 learned_addr learned my addr 172.21.15.181:0/3094880183 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): get_auth_request con 0x7f4ca4099f00 auth_method 0 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_done global_id 74603 payload 274 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 1 -- 172.21.15.181:0/3094880183 >> v2:172.21.15.181:3300/0 conn(0x7f4ca409d4e0 msgr2=0x7f4ca41360e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 1 --2- 172.21.15.181:0/3094880183 >> v2:172.21.15.181:3300/0 conn(0x7f4ca409d4e0 0x7f4ca41360e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 1 -- 172.21.15.181:0/3094880183 >> v1:172.21.15.153:6789/0 conn(0x7f4ca4096a30 legacy=0x7f4ca4096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient: _finish_hunting 0 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 1 monclient: found mon.noname-b 2024-09-15T03:35:32.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:32.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:32.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.216+0000 7f4ca88b8640 1 -- 172.21.15.181:0/3094880183 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca40a1ba0 con 0x7f4ca4099f00 2024-09-15T03:35:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4ca88b8640 1 -- 172.21.15.181:0/3094880183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 msgr2=0x7f4ca4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4ca88b8640 1 --2- 172.21.15.181:0/3094880183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 0x7f4ca4133b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009850 tx=0x7f4ca00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1d9a84cca0dd779 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 1 -- 172.21.15.181:0/3094880183 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0032440 con 0x7f4ca4099f00 2024-09-15T03:35:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: dump: 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:32.230 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: _finish_auth 0 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 1 -- 172.21.15.181:0/3094880183 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca00325a0 con 0x7f4ca4099f00 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4cab31d640 5 monclient: authenticate success, global_id 74603 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4cab31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 1 -- 172.21.15.181:0/3094880183 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0032940 con 0x7f4ca4099f00 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4cab31d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4cab31d640 4 set_mon_vals no callback set 2024-09-15T03:35:32.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.219+0000 7f4c9bfff640 10 monclient: dump: 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:32.232 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 10 monclient: shutdown 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 -- 172.21.15.181:0/3094880183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 msgr2=0x7f4ca4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3094880183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 0x7f4ca4133b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009850 tx=0x7f4ca00090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 -- 172.21.15.181:0/3094880183 shutdown_connections 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3094880183 >> v2:172.21.15.181:3300/0 conn(0x7f4ca409d4e0 0x7f4ca41360e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3094880183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca4099f00 0x7f4ca4133b90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 -- 172.21.15.181:0/3094880183 >> 172.21.15.181:0/3094880183 conn(0x7f4ca40711a0 msgr2=0x7f4ca40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 -- 172.21.15.181:0/3094880183 shutdown_connections 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 1 -- 172.21.15.181:0/3094880183 wait complete. 2024-09-15T03:35:32.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 10 monclient: build_initial_monmap 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.220+0000 7f4cab31d640 10 monclient: monmap: 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:32.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 Processor -- start 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 -- start start 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient: init 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca4096a30 0x7f4ca413f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca4096a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 0x7f4ca413fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca4099f00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 0x7f4ca4140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca409d4e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:32.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca406d9e0 con 0x7f4ca4099f00 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca411eac0 con 0x7f4ca4096a30 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca411e470 con 0x7f4ca409d4e0 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4cab31d640 10 monclient(hunting): authenticate will time out at 1136.702928s 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4ca90b9640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca4096a30 msgr2=0x7f4ca413f8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.221+0000 7f4ca90b9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca4096a30 0x7f4ca413f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 0x7f4ca4140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 0x7f4ca413fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:32.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 0x7f4ca4140320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35384/0 (socket says 172.21.15.181:35384) 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 -- 172.21.15.181:0/3835182080 learned_addr learned my addr 172.21.15.181:0/3835182080 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 0x7f4ca413fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:59674/0 (socket says 172.21.15.181:59674) 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): get_auth_request con 0x7f4ca409d4e0 auth_method 0 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): get_auth_request con 0x7f4ca4099f00 auth_method 0 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient(hunting): handle_auth_done global_id 74606 payload 824 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 -- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca4096a30 msgr2=0x7f4ca413f8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca4096a30 0x7f4ca413f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 -- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 msgr2=0x7f4ca413fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 0x7f4ca413fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:32.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient: _finish_hunting 0 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 monclient: found mon.noname-c 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca98ba640 1 -- 172.21.15.181:0/3835182080 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0009390 con 0x7f4ca409d4e0 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.222+0000 7f4ca88b8640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 0x7f4ca413fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4ca98ba640 1 -- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 msgr2=0x7f4ca4140320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4ca98ba640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 0x7f4ca4140320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009fc0 tx=0x7f4c9c009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c006150 con 0x7f4ca409d4e0 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: dump: 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c0062f0 con 0x7f4ca409d4e0 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0064e0 con 0x7f4ca409d4e0 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4c997fa640 10 monclient: dump: 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:32.241 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.224+0000 7f4cab31d640 5 monclient: authenticate success, global_id 74606 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4cab31d640 10 monclient: _renew_subs 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4cab31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:32.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca411e750 con 0x7f4ca409d4e0 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4c9b7fe640 4 set_mon_vals no callback set 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4cab31d640 10 monclient: _renew_subs 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4cab31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.225+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca411d580 con 0x7f4ca409d4e0 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4c9c012020 con 0x7f4ca409d4e0 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4cab31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4cab31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4cab31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca411d800 con 0x7f4ca409d4e0 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4c997fa640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c88038740 0x7f4c8803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:32.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(166..166 src has 106..166) v4 ==== 9156+0+0 (secure 0 0 0) 0x7f4c9c04cdd0 con 0x7f4ca409d4e0 2024-09-15T03:35:32.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4ca90b9640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c88038740 0x7f4c8803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:32.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.226+0000 7f4ca90b9640 10 monclient: get_auth_request con 0x7f4c88038740 auth_method 0 2024-09-15T03:35:32.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.228+0000 7f4ca90b9640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c88038740 0x7f4c8803abf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4c940047d0 tx=0x7f4c94007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:32.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.231+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4ca411d800 con 0x7f4ca409d4e0 2024-09-15T03:35:32.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.231+0000 7f4c997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:32.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.231+0000 7f4c997fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:32.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.384+0000 7f4cab31d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:32.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.384+0000 7f4cab31d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:32.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.384+0000 7f4cab31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:32.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:32.384+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ca411d800 con 0x7f4ca409d4e0 2024-09-15T03:35:33.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.149+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v167) v1 ==== 148+0+0 (secure 0 0 0) 0x7f4ca411d800 con 0x7f4ca409d4e0 2024-09-15T03:35:33.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.149+0000 7f4c997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:33.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.149+0000 7f4c997fa640 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-15T03:35:33.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.244+0000 7f4cab31d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:33.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.244+0000 7f4cab31d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:33.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.244+0000 7f4cab31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.244+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ca411d800 con 0x7f4ca409d4e0 2024-09-15T03:35:33.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.246+0000 7f4c997fa640 1 -- 172.21.15.181:0/3835182080 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v167) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4ca411d800 con 0x7f4ca409d4e0 2024-09-15T03:35:33.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.246+0000 7f4c997fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:33.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.246+0000 7f4c997fa640 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-15T03:35:33.247 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'cool' does not exist 2024-09-15T03:35:33.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c88038740 msgr2=0x7f4c8803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:33.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c88038740 0x7f4c8803abf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4c940047d0 tx=0x7f4c94007b80 comp rx=0 tx=0).stop 2024-09-15T03:35:33.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 10 monclient: shutdown 2024-09-15T03:35:33.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:33.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 msgr2=0x7f4ca4140320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:33.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 0x7f4ca4140320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009fc0 tx=0x7f4c9c009730 comp rx=0 tx=0).stop 2024-09-15T03:35:33.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 shutdown_connections 2024-09-15T03:35:33.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4c88038740 0x7f4c8803abf0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ca409d4e0 0x7f4ca4140320 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca4099f00 0x7f4ca413fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 --2- 172.21.15.181:0/3835182080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca4096a30 0x7f4ca413f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 >> 172.21.15.181:0/3835182080 conn(0x7f4ca40711a0 msgr2=0x7f4ca406cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:33.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 shutdown_connections 2024-09-15T03:35:33.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.247+0000 7f4cab31d640 1 -- 172.21.15.181:0/3835182080 wait complete. 2024-09-15T03:35:33.258 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.333+0000 7f15f3a2b640 10 monclient: get_monmap_and_config 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.333+0000 7f15f3a2b640 10 monclient: build_initial_monmap 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.333+0000 7f15f3a2b640 10 monclient: monmap: 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:33.333951+0000 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:33.333951+0000 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:33.335 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:33.335 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:33.335 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:33.335 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:33.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.333+0000 7f15f3a2b640 1 Processor -- start 2024-09-15T03:35:33.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.333+0000 7f15f3a2b640 1 -- start start 2024-09-15T03:35:33.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient: init 2024-09-15T03:35:33.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15ec09ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 0x7f15ec069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f15ec09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15ec136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec067420 con 0x7f15ec09ec50 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec09fdc0 con 0x7f15ec136d60 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:33.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0a0510 con 0x7f15ec09f020 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f3a2b640 10 monclient(hunting): authenticate will time out at 1137.815573s 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f1fc8640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f15ec136d60 legacy=0x7f15ec139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f0fc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 0x7f15ec069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f17c7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f15ec09ec50 0x7f15ec131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:48460/0 (socket says 172.21.15.181:48460) 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f17c7640 1 -- 172.21.15.181:0/550020701 learned_addr learned my addr 172.21.15.181:0/550020701 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f0fc6640 10 monclient(hunting): get_auth_request con 0x7f15ec09f020 auth_method 0 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f0fc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f0fc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.334+0000 7f15f0fc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:33.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_done global_id 74609 payload 274 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 1 -- 172.21.15.181:0/550020701 >> v1:172.21.15.181:6789/0 conn(0x7f15ec136d60 legacy=0x7f15ec139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 1 -- 172.21.15.181:0/550020701 >> v1:172.21.15.153:6789/0 conn(0x7f15ec09ec50 legacy=0x7f15ec131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 10 monclient: _finish_hunting 0 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 1 monclient: found mon.noname-b 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.335+0000 7f15f0fc6640 1 -- 172.21.15.181:0/550020701 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec0a0200 con 0x7f15ec09f020 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f0fc6640 1 -- 172.21.15.181:0/550020701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 msgr2=0x7f15ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f0fc6640 1 --2- 172.21.15.181:0/550020701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 0x7f15ec069d10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009880 tx=0x7f15dc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3df2f080394c097d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 1 -- 172.21.15.181:0/550020701 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc032440 con 0x7f15ec09f020 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: dump: 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:33.347 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 5 monclient: authenticate success, global_id 74609 2024-09-15T03:35:33.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 1 -- 172.21.15.181:0/550020701 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc0325a0 con 0x7f15ec09f020 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 4 set_mon_vals no callback set 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 1 -- 172.21.15.181:0/550020701 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc032940 con 0x7f15ec09f020 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:33.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15e3fff640 10 monclient: dump: 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 10 monclient: shutdown 2024-09-15T03:35:33.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 1 -- 172.21.15.181:0/550020701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 msgr2=0x7f15ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/550020701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 0x7f15ec069d10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009880 tx=0x7f15dc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 1 -- 172.21.15.181:0/550020701 shutdown_connections 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/550020701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec09f020 0x7f15ec069d10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.338+0000 7f15f3a2b640 1 -- 172.21.15.181:0/550020701 >> 172.21.15.181:0/550020701 conn(0x7f15ec0a5d40 msgr2=0x7f15ec0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 1 -- 172.21.15.181:0/550020701 shutdown_connections 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 1 -- 172.21.15.181:0/550020701 wait complete. 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 10 monclient: build_initial_monmap 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 10 monclient: monmap: 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:33.351 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 1 Processor -- start 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 1 -- start start 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.339+0000 7f15f3a2b640 10 monclient: init 2024-09-15T03:35:33.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:33.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:33.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15ec09ec50 0x7f15ec136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:33.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f15ec09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:33.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ec09f020 0x7f15ec096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:33.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f15ec09f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 0x7f15ec096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f15ec136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec097d20 con 0x7f15ec09f020 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec09c000 con 0x7f15ec09ec50 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0a3ce0 con 0x7f15ec136d60 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f17c7640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15ec09ec50 msgr2=0x7f15ec136310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f3a2b640 10 monclient(hunting): authenticate will time out at 1137.821523s 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f17c7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15ec09ec50 0x7f15ec136310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:33.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 0x7f15ec096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 0x7f15ec096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35404/0 (socket says 172.21.15.181:35404) 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 1 -- 172.21.15.181:0/3182210828 learned_addr learned my addr 172.21.15.181:0/3182210828 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): get_auth_request con 0x7f15ec136d60 auth_method 0 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ec09f020 0x7f15ec096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): get_auth_request con 0x7f15ec09f020 auth_method 0 2024-09-15T03:35:33.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f1fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:33.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.340+0000 7f15f0fc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:33.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 10 monclient(hunting): handle_auth_done global_id 74612 payload 824 2024-09-15T03:35:33.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 1 -- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15ec09ec50 msgr2=0x7f15ec136310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:33.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15ec09ec50 0x7f15ec136310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 1 -- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ec09f020 msgr2=0x7f15ec096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:33.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ec09f020 0x7f15ec096a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:33.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 10 monclient: _finish_hunting 0 2024-09-15T03:35:33.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 1 monclient: found mon.noname-c 2024-09-15T03:35:33.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:33.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.341+0000 7f15f1fc8640 1 -- 172.21.15.181:0/3182210828 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc009390 con 0x7f15ec136d60 2024-09-15T03:35:33.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f1fc8640 1 -- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 msgr2=0x7f15ec096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:33.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f1fc8640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 0x7f15ec096f70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15e400bfb0 tx=0x7f15e4009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:33.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e400cc00 con 0x7f15ec136d60 2024-09-15T03:35:33.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:33.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:33.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: dump: 2024-09-15T03:35:33.360 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:33.360 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:33.360 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.361 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.361 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:33.361 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:33.361 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:33.361 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:33.361 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:33.362 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:33.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:33.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:33.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 5 monclient: authenticate success, global_id 74612 2024-09-15T03:35:33.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 10 monclient: _renew_subs 2024-09-15T03:35:33.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e400cd60 con 0x7f15ec136d60 2024-09-15T03:35:33.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec0997c0 con 0x7f15ec136d60 2024-09-15T03:35:33.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:33.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e2ffd640 4 set_mon_vals no callback set 2024-09-15T03:35:33.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 10 monclient: _renew_subs 2024-09-15T03:35:33.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec098a00 con 0x7f15ec136d60 2024-09-15T03:35:33.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4016430 con 0x7f15ec136d60 2024-09-15T03:35:33.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:33.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:33.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.343+0000 7f15e17fa640 10 monclient: dump: 2024-09-15T03:35:33.365 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:33.365 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:33.365 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:33.367 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15f3a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15f3a2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15f3a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec099ba0 con 0x7f15ec136d60 2024-09-15T03:35:33.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f15e4016860 con 0x7f15ec136d60 2024-09-15T03:35:33.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15e17fa640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15d00386f0 0x7f15d003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:33.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(167..167 src has 106..167) v4 ==== 8485+0+0 (secure 0 0 0) 0x7f15e404ca80 con 0x7f15ec136d60 2024-09-15T03:35:33.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.344+0000 7f15f17c7640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15d00386f0 0x7f15d003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:33.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.345+0000 7f15f17c7640 10 monclient: get_auth_request con 0x7f15d00386f0 auth_method 0 2024-09-15T03:35:33.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.347+0000 7f15f17c7640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15d00386f0 0x7f15d003aba0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f15e80047d0 tx=0x7f15e8007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:33.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.349+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f15e4013510 con 0x7f15ec136d60 2024-09-15T03:35:33.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.349+0000 7f15e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:33.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.349+0000 7f15e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:33.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.501+0000 7f15f3a2b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:33.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.501+0000 7f15f3a2b640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:33.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.501+0000 7f15f3a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:33.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:33.501+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f15ec0008d0 con 0x7f15ec136d60 2024-09-15T03:35:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.157+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v168) v1 ==== 148+0+0 (secure 0 0 0) 0x7f15e4014b30 con 0x7f15ec136d60 2024-09-15T03:35:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.157+0000 7f15e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.157+0000 7f15e17fa640 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-15T03:35:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.251+0000 7f15f3a2b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.251+0000 7f15f3a2b640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.251+0000 7f15f3a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.251+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f15ec099e10 con 0x7f15ec136d60 2024-09-15T03:35:34.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.252+0000 7f15e17fa640 1 -- 172.21.15.181:0/3182210828 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v168) v1 ==== 155+0+0 (secure 0 0 0) 0x7f15e40132e0 con 0x7f15ec136d60 2024-09-15T03:35:34.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.252+0000 7f15e17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-15T03:35:34.253 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.252+0000 7f15e17fa640 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-15T03:35:34.253 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'cold' does not exist 2024-09-15T03:35:34.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15d00386f0 msgr2=0x7f15d003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:34.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15d00386f0 0x7f15d003aba0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f15e80047d0 tx=0x7f15e8007b80 comp rx=0 tx=0).stop 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 10 monclient: shutdown 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 msgr2=0x7f15ec096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 0x7f15ec096f70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15e400bfb0 tx=0x7f15e4009700 comp rx=0 tx=0).stop 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 shutdown_connections 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f15d00386f0 0x7f15d003aba0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ec136d60 0x7f15ec096f70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ec09f020 0x7f15ec096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 --2- 172.21.15.181:0/3182210828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15ec09ec50 0x7f15ec136310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 >> 172.21.15.181:0/3182210828 conn(0x7f15ec0a5d40 msgr2=0x7f15ec09e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 shutdown_connections 2024-09-15T03:35:34.256 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.254+0000 7f15f3a2b640 1 -- 172.21.15.181:0/3182210828 wait complete. 2024-09-15T03:35:34.264 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T03:35:34.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.335+0000 7f21b00a8640 10 monclient: get_monmap_and_config 2024-09-15T03:35:34.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.335+0000 7f21b00a8640 10 monclient: build_initial_monmap 2024-09-15T03:35:34.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.335+0000 7f21b00a8640 10 monclient: monmap: 2024-09-15T03:35:34.337 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:34.337 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:34.336757+0000 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:34.336757+0000 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.336+0000 7f21b00a8640 1 Processor -- start 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.336+0000 7f21b00a8640 1 -- start start 2024-09-15T03:35:34.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.336+0000 7f21b00a8640 10 monclient: init 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21a80677e0 0x7f21a8095800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): picked mon.noname-c con 0x7f21a80677e0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a809c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): picked mon.noname-b con 0x7f21a809c090 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21a809c9a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a809d570 con 0x7f21a809c9a0 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a809b560 con 0x7f21a80677e0 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:34.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a809d800 con 0x7f21a809c090 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ade44640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f21a80677e0 msgr2=0x7f21a8095800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21b00a8640 10 monclient(hunting): authenticate will time out at 1138.818329s 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ae645640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f21a809c9a0 legacy=0x7f21a8094080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ade44640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f21a80677e0 0x7f21a8095800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a809c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a809c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35408/0 (socket says 172.21.15.181:35408) 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 -- 172.21.15.181:0/1945841017 learned_addr learned my addr 172.21.15.181:0/1945841017 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): get_auth_request con 0x7f21a809c090 auth_method 0 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:34.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient(hunting): handle_auth_done global_id 74615 payload 274 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 -- 172.21.15.181:0/1945841017 >> v2:172.21.15.181:3300/0 conn(0x7f21a80677e0 msgr2=0x7f21a8095800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 --2- 172.21.15.181:0/1945841017 >> v2:172.21.15.181:3300/0 conn(0x7f21a80677e0 0x7f21a8095800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 -- 172.21.15.181:0/1945841017 >> v1:172.21.15.181:6789/0 conn(0x7f21a809c9a0 legacy=0x7f21a8094080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient: _finish_hunting 0 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 monclient: found mon.noname-b 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.337+0000 7f21ad643640 1 -- 172.21.15.181:0/1945841017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a809cdc0 con 0x7f21a809c090 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.340+0000 7f21ad643640 1 -- 172.21.15.181:0/1945841017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 msgr2=0x7f21a809c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.340+0000 7f21ad643640 1 --2- 172.21.15.181:0/1945841017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a809c460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2198009940 tx=0x7f21980091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=894b0fc518c6b651 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 1 -- 172.21.15.181:0/1945841017 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198032440 con 0x7f21a809c090 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: dump: 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:34.351 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: _finish_auth 0 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 1 -- 172.21.15.181:0/1945841017 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21980325a0 con 0x7f21a809c090 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 5 monclient: authenticate success, global_id 74615 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 1 -- 172.21.15.181:0/1945841017 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198032940 con 0x7f21a809c090 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 4 set_mon_vals no callback set 2024-09-15T03:35:34.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21ace42640 10 monclient: dump: 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:34.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 10 monclient: shutdown 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 -- 172.21.15.181:0/1945841017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 msgr2=0x7f21a809c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 --2- 172.21.15.181:0/1945841017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a809c460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2198009940 tx=0x7f21980091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 -- 172.21.15.181:0/1945841017 shutdown_connections 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 --2- 172.21.15.181:0/1945841017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a809c460 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 --2- 172.21.15.181:0/1945841017 >> v2:172.21.15.181:3300/0 conn(0x7f21a80677e0 0x7f21a8095800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 -- 172.21.15.181:0/1945841017 >> 172.21.15.181:0/1945841017 conn(0x7f21a80a16a0 msgr2=0x7f21a80a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 -- 172.21.15.181:0/1945841017 shutdown_connections 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.341+0000 7f21b00a8640 1 -- 172.21.15.181:0/1945841017 wait complete. 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 10 monclient: build_initial_monmap 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 10 monclient: monmap: 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:34.354 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 1 Processor -- start 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 1 -- start start 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 10 monclient: init 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.342+0000 7f21b00a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:34.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a80677e0 0x7f21a813efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): picked mon.noname-a con 0x7f21a80677e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a8093640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): picked mon.noname-c con 0x7f21a809c090 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a809c9a0 0x7f21a8093b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): picked mon.noname-b con 0x7f21a809c9a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a813ff20 con 0x7f21a80677e0 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a808d0c0 con 0x7f21a809c9a0 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a808cde0 con 0x7f21a809c090 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21b00a8640 10 monclient(hunting): authenticate will time out at 1138.824081s 2024-09-15T03:35:34.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ae645640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a809c9a0 msgr2=0x7f21a8093b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ae645640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a809c9a0 0x7f21a8093b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a8093640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a80677e0 0x7f21a813efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a8093640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:35424/0 (socket says 172.21.15.181:35424) 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 -- 172.21.15.181:0/2746497560 learned_addr learned my addr 172.21.15.181:0/2746497560 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): get_auth_request con 0x7f21a809c090 auth_method 0 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): get_auth_request con 0x7f21a80677e0 auth_method 0 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:34.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient(hunting): handle_auth_done global_id 74618 payload 824 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 -- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a809c9a0 msgr2=0x7f21a8093b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a809c9a0 0x7f21a8093b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 -- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a80677e0 msgr2=0x7f21a813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a80677e0 0x7f21a813efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient: _finish_hunting 0 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 monclient: found mon.noname-c 2024-09-15T03:35:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ad643640 1 -- 172.21.15.181:0/2746497560 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198009450 con 0x7f21a809c090 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.343+0000 7f21ade44640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a80677e0 0x7f21a813efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.345+0000 7f21ad643640 1 -- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 msgr2=0x7f21a8093640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.345+0000 7f21ad643640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a8093640 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f21980099a0 tx=0x7f2198032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198031940 con 0x7f21a809c090 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: dump: 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:34.359 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2198031aa0 con 0x7f21a809c090 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 5 monclient: authenticate success, global_id 74618 2024-09-15T03:35:34.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:34.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198031e40 con 0x7f21a809c090 2024-09-15T03:35:34.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:34.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:34.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219dffb640 10 monclient: dump: 2024-09-15T03:35:34.361 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 10 monclient: _renew_subs 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a808cb00 con 0x7f21a809c090 2024-09-15T03:35:34.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f219ffff640 4 set_mon_vals no callback set 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 10 monclient: _renew_subs 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.346+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a808b3b0 con 0x7f21a809c090 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f21b00a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f21b00a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f21b00a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a80008d0 con 0x7f21a809c090 2024-09-15T03:35:34.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2198039070 con 0x7f21a809c090 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f219dffb640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f218c0386f0 0x7f218c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(168..168 src has 106..168) v4 ==== 7678+0+0 (secure 0 0 0) 0x7f219806ccf0 con 0x7f21a809c090 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f21ade44640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f218c0386f0 0x7f218c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.347+0000 7f21ade44640 10 monclient: get_auth_request con 0x7f218c0386f0 auth_method 0 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.350+0000 7f21ade44640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f218c0386f0 0x7f218c03aba0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f21a4003940 tx=0x7f21a40090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.352+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f219803c620 con 0x7f21a809c090 2024-09-15T03:35:34.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.352+0000 7f219dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:34.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.352+0000 7f219dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:34.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.515+0000 7f21b00a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:35:34.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.515+0000 7f21b00a8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:35:34.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.515+0000 7f21b00a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:34.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:34.515+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f21a8095d40 con 0x7f21a809c090 2024-09-15T03:35:35.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.167+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v169) v1 ==== 78+0+0 (secure 0 0 0) 0x7f219804a070 con 0x7f21a809c090 2024-09-15T03:35:35.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.168+0000 7f219dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:35:35.169 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.168+0000 7f219dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:35.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.274+0000 7f21b00a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:35:35.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.274+0000 7f21b00a8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:35:35.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.274+0000 7f21b00a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:35.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:35.274+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f21a8095d40 con 0x7f21a809c090 2024-09-15T03:35:36.115 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:35:36.115 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:35:36.115 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:35:36.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:35:36.119 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:35:36.119 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:35:36.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.171+0000 7f219dffb640 1 -- 172.21.15.181:0/2746497560 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v170) v1 ==== 78+0+0 (secure 0 0 0) 0x7f219804a070 con 0x7f21a809c090 2024-09-15T03:35:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.171+0000 7f219dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:35:36.172 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.171+0000 7f219dffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:35:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f218c0386f0 msgr2=0x7f218c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f218c0386f0 0x7f218c03aba0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f21a4003940 tx=0x7f21a40090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 10 monclient: shutdown 2024-09-15T03:35:36.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 msgr2=0x7f21a8093640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a8093640 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f21980099a0 tx=0x7f2198032000 comp rx=0 tx=0).stop 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 shutdown_connections 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f218c0386f0 0x7f218c03aba0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a809c9a0 0x7f21a8093b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a809c090 0x7f21a8093640 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 --2- 172.21.15.181:0/2746497560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a80677e0 0x7f21a813efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 >> 172.21.15.181:0/2746497560 conn(0x7f21a80a16a0 msgr2=0x7f21a8132ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 shutdown_connections 2024-09-15T03:35:36.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.174+0000 7f21b00a8640 1 -- 172.21.15.181:0/2746497560 wait complete. 2024-09-15T03:35:36.192 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T03:35:36.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.230+0000 7fa570e4f640 10 monclient: get_monmap_and_config 2024-09-15T03:35:36.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.230+0000 7fa570e4f640 10 monclient: build_initial_monmap 2024-09-15T03:35:36.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.230+0000 7fa570e4f640 10 monclient: monmap: 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:36.231110+0000 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:36.231110+0000 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.230+0000 7fa570e4f640 1 Processor -- start 2024-09-15T03:35:36.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.230+0000 7fa570e4f640 1 -- start start 2024-09-15T03:35:36.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.230+0000 7fa570e4f640 10 monclient: init 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa56c06e6f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 0x7fa56c06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c06eac0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa56c134aa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c09c610 con 0x7fa56c06e6f0 2024-09-15T03:35:36.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c09b090 con 0x7fa56c134aa0 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c09c890 con 0x7fa56c06eac0 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa570e4f640 10 monclient(hunting): authenticate will time out at 1133.870622s 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa56a59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa56c06e6f0 0x7fa56c12fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55422/0 (socket says 172.21.15.153:55422) 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa56a59c640 1 -- 172.21.15.153:0/3941461334 learned_addr learned my addr 172.21.15.153:0/3941461334 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 0x7fa56c06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 10 monclient(hunting): get_auth_request con 0x7fa56c06eac0 auth_method 0 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 1 -- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 msgr2=0x7fa56c06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:36.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 1 -- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 msgr2=0x7fa56c06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.231+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 0x7fa56c06eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.232+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 0x7fa56c06eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.233+0000 7fa56a59c640 1 -- 172.21.15.153:0/3941461334 >> v1:172.21.15.153:6789/0 conn(0x7fa56c06e6f0 legacy=0x7fa56c12fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 1 -- 172.21.15.153:0/3941461334 >> v1:172.21.15.181:6789/0 conn(0x7fa56c134aa0 legacy=0x7fa56c136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976085106 0 0) 0x7fa56c09c610 con 0x7fa56c06e6f0 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): my global_id is 84516 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5540034e0 con 0x7fa56c06e6f0 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 381903802 0 0) 0x7fa5540034e0 con 0x7fa56c06e6f0 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 msgr2=0x7fa56c06eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:36.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 --2- 172.21.15.153:0/3941461334 >> v2:172.21.15.181:3300/0 conn(0x7fa56c06eac0 0x7fa56c06eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 >> v1:172.21.15.181:6789/0 conn(0x7fa56c134aa0 legacy=0x7fa56c136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c09cb70 con 0x7fa56c06e6f0 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: _finish_auth 0 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fa55c001720 con 0x7fa56c06e6f0 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa570e4f640 5 monclient: authenticate success, global_id 84516 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa570e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: dump: 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:36.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 >> v1:172.21.15.153:6789/0 conn(0x7fa56c06e6f0 legacy=0x7fa56c12fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa554003b50 0x7fa554003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): picked mon.c con 0x7fa554003b50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa554004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): picked mon.a con 0x7fa554004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5540099c0 0x7fa55400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): picked mon.b con 0x7fa5540099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c09c890 con 0x7fa554004780 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55400c3c0 con 0x7fa5540099c0 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55400c580 con 0x7fa554003b50 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa570e4f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa554004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): get_auth_request con 0x7fa554004780 auth_method 0 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa554003b50 0x7fa554003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): get_auth_request con 0x7fa554003b50 auth_method 0 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5540099c0 0x7fa55400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 10 monclient(hunting): get_auth_request con 0x7fa5540099c0 auth_method 0 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:36.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient(hunting): handle_auth_done global_id 84516 payload 293 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 -- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa554003b50 msgr2=0x7fa554003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa554003b50 0x7fa554003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 -- 172.21.15.153:0/3941461334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5540099c0 msgr2=0x7fa55400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5540099c0 0x7fa55400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 monclient: found mon.a 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 1 -- 172.21.15.153:0/3941461334 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540034e0 con 0x7fa554004780 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 10 monclient: _finish_auth 0 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa569d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5540099c0 0x7fa55400bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:36.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.234+0000 7fa56ad9d640 1 -- 172.21.15.153:0/3941461334 reap_dead start 2024-09-15T03:35:36.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa569d9b640 1 -- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 msgr2=0x7fa554004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:36.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa569d9b640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa554004bf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa560006fd0 tx=0x7fa560008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358b728c55edbc56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56002add0 con 0x7fa554004780 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: dump: 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa570e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56002e070 con 0x7fa554004780 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa570e4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa570e4f640 4 set_mon_vals no callback set 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 1 -- 172.21.15.153:0/3941461334 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560034660 con 0x7fa554004780 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.237+0000 7fa56959a640 10 monclient: dump: 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 10 monclient: shutdown 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 -- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 msgr2=0x7fa554004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa554004bf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa560006fd0 tx=0x7fa560008040 comp rx=0 tx=0).stop 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 -- 172.21.15.153:0/3941461334 shutdown_connections 2024-09-15T03:35:36.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 --2- 172.21.15.153:0/3941461334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa554004bf0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 -- 172.21.15.153:0/3941461334 >> 172.21.15.153:0/3941461334 conn(0x7fa56c0672b0 msgr2=0x7fa56c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 -- 172.21.15.153:0/3941461334 shutdown_connections 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 1 -- 172.21.15.153:0/3941461334 wait complete. 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 10 monclient: build_initial_monmap 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.238+0000 7fa570e4f640 10 monclient: monmap: 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:36.255 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 Processor -- start 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 -- start start 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient: init 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa56c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): picked mon.noname-a con 0x7fa554004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 0x7fa56c12d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c135340 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 0x7fa56c117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c12dc30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:36.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c06e000 con 0x7fa554004780 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0a0110 con 0x7fa56c135340 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c06f6a0 con 0x7fa56c12dc30 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:36.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa570e4f640 10 monclient(hunting): authenticate will time out at 1133.878590s 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa56c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 0x7fa56c117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa56c06d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52694/0 (socket says 172.21.15.153:52694) 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 0x7fa56c117480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52802/0 (socket says 172.21.15.153:52802) 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 1 -- 172.21.15.153:0/1996127948 learned_addr learned my addr 172.21.15.153:0/1996127948 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): get_auth_request con 0x7fa554004780 auth_method 0 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): get_auth_request con 0x7fa56c12dc30 auth_method 0 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 0x7fa56c12d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:36.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 10 monclient(hunting): get_auth_request con 0x7fa56c135340 auth_method 0 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 msgr2=0x7fa56c12d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 msgr2=0x7fa56c12d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 0x7fa56c12d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa569d9b640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 0x7fa56c12d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient(hunting): handle_auth_done global_id 74621 payload 824 2024-09-15T03:35:36.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 msgr2=0x7fa56c12d6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 0x7fa56c12d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 msgr2=0x7fa56c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa56c06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 monclient: found mon.noname-c 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56ad9d640 1 -- 172.21.15.153:0/1996127948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c13fd50 con 0x7fa56c12dc30 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.239+0000 7fa56a59c640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa56c06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa56ad9d640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 msgr2=0x7fa56c117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa56ad9d640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 0x7fa56c117480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa56c09d790 tx=0x7fa564009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400e8e0 con 0x7fa56c12dc30 2024-09-15T03:35:36.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: dump: 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 5 monclient: authenticate success, global_id 74621 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56400ea60 con 0x7fa56c12dc30 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400ee20 con 0x7fa56c12dc30 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 10 monclient: _renew_subs 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c119800 con 0x7fa56c12dc30 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa568d99640 4 set_mon_vals no callback set 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa55a7fc640 10 monclient: dump: 2024-09-15T03:35:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 10 monclient: _renew_subs 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.242+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c118910 con 0x7fa56c12dc30 2024-09-15T03:35:36.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa5640072e0 con 0x7fa56c12dc30 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa570e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa570e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa570e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c04bf60 con 0x7fa56c12dc30 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa55a7fc640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa54c038770 0x7fa54c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(170..170 src has 106..170) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fa56404b4e0 con 0x7fa56c12dc30 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa56a59c640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa54c038770 0x7fa54c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.243+0000 7fa56a59c640 10 monclient: get_auth_request con 0x7fa54c038770 auth_method 0 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.246+0000 7fa56a59c640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa54c038770 0x7fa54c03ac20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa55c007fd0 tx=0x7fa55c007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.247+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa564011d20 con 0x7fa56c12dc30 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.247+0000 7fa55a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:36.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.247+0000 7fa55a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:36.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.266+0000 7f53a98dd640 10 monclient: get_monmap_and_config 2024-09-15T03:35:36.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.266+0000 7f53a98dd640 10 monclient: build_initial_monmap 2024-09-15T03:35:36.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.266+0000 7f53a98dd640 10 monclient: monmap: 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:36.267706+0000 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:36.267706+0000 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:36.268 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.267+0000 7f53a98dd640 1 Processor -- start 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.267+0000 7f53a98dd640 1 -- start start 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.267+0000 7f53a98dd640 10 monclient: init 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): picked mon.noname-b con 0x7f53a4131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): picked mon.noname-c con 0x7f53a40a0b00 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:36.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53a4068d00 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a409d230 con 0x7f53a4068d00 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40a08c0 con 0x7f53a40a0b00 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40a0200 con 0x7f53a4131c00 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a98dd640 10 monclient(hunting): authenticate will time out at 1140.749133s 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55748/0 (socket says 172.21.15.181:55748) 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 -- 172.21.15.181:0/3267270252 learned_addr learned my addr 172.21.15.181:0/3267270252 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 10 monclient(hunting): get_auth_request con 0x7f53a40a0b00 auth_method 0 2024-09-15T03:35:36.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 1 --2- 172.21.15.181:0/3267270252 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request con 0x7f53a4131c00 auth_method 0 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 -- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 msgr2=0x7f53a4132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 -- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 msgr2=0x7f53a4132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 --2- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a27fc640 1 --2- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_done global_id 74624 payload 274 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 1 -- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 msgr2=0x7f53a4132510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 1 --2- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 1 -- 172.21.15.181:0/3267270252 >> v1:172.21.15.153:6789/0 conn(0x7f53a4068d00 legacy=0x7f53a4069100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.268+0000 7f53a2ffd640 1 -- 172.21.15.181:0/3267270252 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40a1b70 con 0x7f53a4131c00 2024-09-15T03:35:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a2ffd640 1 -- 172.21.15.181:0/3267270252 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 msgr2=0x7f53a4131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a2ffd640 1 --2- 172.21.15.181:0/3267270252 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4131fd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5394009850 tx=0x7f53940090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aca268138236ce68 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 1 -- 172.21.15.181:0/3267270252 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539402adf0 con 0x7f53a4131c00 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: dump: 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 1 -- 172.21.15.181:0/3267270252 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394032440 con 0x7f53a4131c00 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 1 -- 172.21.15.181:0/3267270252 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53940327e0 con 0x7f53a4131c00 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a98dd640 5 monclient: authenticate success, global_id 74624 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a98dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a98dd640 4 set_mon_vals no callback set 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:36.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.271+0000 7f53a1ffb640 10 monclient: dump: 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 10 monclient: shutdown 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:36.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 -- 172.21.15.181:0/3267270252 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 msgr2=0x7f53a4131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 --2- 172.21.15.181:0/3267270252 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4131fd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5394009850 tx=0x7f53940090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 -- 172.21.15.181:0/3267270252 shutdown_connections 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 --2- 172.21.15.181:0/3267270252 >> v2:172.21.15.181:3300/0 conn(0x7f53a40a0b00 0x7f53a4132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 --2- 172.21.15.181:0/3267270252 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4131fd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 -- 172.21.15.181:0/3267270252 >> 172.21.15.181:0/3267270252 conn(0x7f53a4071370 msgr2=0x7f53a40a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 -- 172.21.15.181:0/3267270252 shutdown_connections 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 1 -- 172.21.15.181:0/3267270252 wait complete. 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 10 monclient: build_initial_monmap 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.272+0000 7f53a98dd640 10 monclient: monmap: 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 Processor -- start 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 -- start start 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient: init 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a4068d00 0x7f53a4130f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): picked mon.noname-a con 0x7f53a4068d00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): picked mon.noname-b con 0x7f53a40a0b00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): picked mon.noname-c con 0x7f53a4131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40a0fd0 con 0x7f53a4068d00 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a411f670 con 0x7f53a40a0b00 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a411cf90 con 0x7f53a4131c00 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a98dd640 10 monclient(hunting): authenticate will time out at 1140.754587s 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55762/0 (socket says 172.21.15.181:55762) 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 -- 172.21.15.181:0/2944247864 learned_addr learned my addr 172.21.15.181:0/2944247864 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 10 monclient(hunting): get_auth_request con 0x7f53a40a0b00 auth_method 0 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a4068d00 0x7f53a4130f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request con 0x7f53a4068d00 auth_method 0 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 msgr2=0x7f53a41314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 msgr2=0x7f53a41314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:36.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): get_auth_request con 0x7f53a4131c00 auth_method 0 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a27fc640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:36.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.273+0000 7f53a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 10 monclient(hunting): handle_auth_done global_id 74627 payload 824 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 msgr2=0x7f53a41314c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a4068d00 msgr2=0x7f53a4130f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a4068d00 0x7f53a4130f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 1 monclient: found mon.noname-c 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.274+0000 7f53a37fe640 1 -- 172.21.15.181:0/2944247864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394009390 con 0x7f53a4131c00 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a37fe640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 msgr2=0x7f53a4141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a37fe640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4141260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f539c00b810 tx=0x7f539c0092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00c940 con 0x7f53a4131c00 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: dump: 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: _finish_auth 0 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c00cae0 con 0x7f53a4131c00 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 5 monclient: authenticate success, global_id 74627 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 10 monclient: _renew_subs 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a411f0b0 con 0x7f53a4131c00 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c014410 con 0x7f53a4131c00 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:36.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f538affd640 10 monclient: dump: 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 10 monclient: _renew_subs 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a41417a0 con 0x7f53a4131c00 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.276+0000 7f53a17fa640 4 set_mon_vals no callback set 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f539c014840 con 0x7f53a4131c00 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f53a98dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f53a98dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f53a98dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a411f820 con 0x7f53a4131c00 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f538affd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5390038740 0x7f539003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(170..170 src has 106..170) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f539c04be90 con 0x7f53a4131c00 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f53a2ffd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5390038740 0x7f539003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.277+0000 7f53a2ffd640 10 monclient: get_auth_request con 0x7f5390038740 auth_method 0 2024-09-15T03:35:36.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.280+0000 7f53a2ffd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5390038740 0x7f539003abf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f53a409c7c0 tx=0x7f5394032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:36.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.281+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f53a411f820 con 0x7f53a4131c00 2024-09-15T03:35:36.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.281+0000 7f538affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:36.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.281+0000 7f538affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:36.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.443+0000 7f53a98dd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:35:36.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.443+0000 7f53a98dd640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:35:36.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.443+0000 7f53a98dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:36.443+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f53a411f820 con 0x7f53a4131c00 2024-09-15T03:35:36.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.464+0000 7fa570e4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.464+0000 7fa570e4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.464+0000 7fa570e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.464+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa56c1305d0 con 0x7fa56c12dc30 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.465+0000 7fa55a7fc640 1 -- 172.21.15.153:0/1996127948 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa564012e60 con 0x7fa56c12dc30 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.465+0000 7fa55a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.465+0000 7fa55a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:36.466 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":72,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.466+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa54c038770 msgr2=0x7fa54c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.466+0000 7fa570e4f640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa54c038770 0x7fa54c03ac20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa55c007fd0 tx=0x7fa55c007a50 comp rx=0 tx=0).stop 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.466+0000 7fa570e4f640 10 monclient: shutdown 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.466+0000 7fa570e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.466+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 msgr2=0x7fa56c117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.466+0000 7fa570e4f640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 0x7fa56c117480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa56c09d790 tx=0x7fa564009040 comp rx=0 tx=0).stop 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 shutdown_connections 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa54c038770 0x7fa54c03ac20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa56c12dc30 0x7fa56c117480 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa56c135340 0x7fa56c12d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 --2- 172.21.15.153:0/1996127948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa554004780 0x7fa56c06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:36.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 >> 172.21.15.153:0/1996127948 conn(0x7fa56c0672b0 msgr2=0x7fa56c1334a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:36.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 shutdown_connections 2024-09-15T03:35:36.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:36.467+0000 7fa570e4f640 1 -- 172.21.15.153:0/1996127948 wait complete. 2024-09-15T03:35:39.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:39.274+0000 7f5389ffb640 10 monclient: tick 2024-09-15T03:35:39.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:39.274+0000 7f5389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:39.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:39.274+0000 7f5389ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:39.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:39.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.585+0000 7fe17211d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:39.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.585+0000 7fe17211d640 10 monclient: build_initial_monmap 2024-09-15T03:35:39.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.585+0000 7fe17211d640 10 monclient: monmap: 2024-09-15T03:35:39.586 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:39.587 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:39.587 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:39.585872+0000 2024-09-15T03:35:39.587 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:39.585872+0000 2024-09-15T03:35:39.587 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:39.587 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:39.587 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:39.588 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:39.588 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:39.588 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:39.588 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:39.588 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.585+0000 7fe17211d640 1 Processor -- start 2024-09-15T03:35:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 1 -- start start 2024-09-15T03:35:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient: init 2024-09-15T03:35:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:39.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-15T03:35:39.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe16c131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:39.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:39.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient(hunting): picked mon.noname-a con 0x7fe16c09e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:39.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe16c09f250 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:39.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c09faf0 con 0x7fe16c09f250 2024-09-15T03:35:39.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c09fd30 con 0x7fe16c131ab0 2024-09-15T03:35:39.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:39.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c132590 con 0x7fe16c09e910 2024-09-15T03:35:39.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:39.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe17211d640 10 monclient(hunting): authenticate will time out at 1137.225603s 2024-09-15T03:35:39.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:39.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe16c09f250 0x7fe16c138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55430/0 (socket says 172.21.15.153:55430) 2024-09-15T03:35:39.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16bfff640 1 -- 172.21.15.153:0/18278172 learned_addr learned my addr 172.21.15.153:0/18278172 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:39.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c09e910 auth_method 0 2024-09-15T03:35:39.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:39.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:39.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:39.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:39.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:39.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:39.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.586+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:39.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.588+0000 7fe16bfff640 1 -- 172.21.15.153:0/18278172 >> v1:172.21.15.153:6789/0 conn(0x7fe16c09f250 legacy=0x7fe16c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:39.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.589+0000 7fe16b7fe640 1 -- 172.21.15.153:0/18278172 >> v1:172.21.15.181:6789/0 conn(0x7fe16c131ab0 legacy=0x7fe16c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c09e910 auth_method 0 2024-09-15T03:35:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:39.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:39.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:39.787+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c09e910 auth_method 0 2024-09-15T03:35:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:40.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:40.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.188+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:40.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:40.274+0000 7f5389ffb640 10 monclient: tick 2024-09-15T03:35:40.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:40.274+0000 7f5389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c09e910 auth_method 0 2024-09-15T03:35:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:40.989+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:35:41.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:41.274+0000 7f5389ffb640 10 monclient: tick 2024-09-15T03:35:41.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:41.274+0000 7f5389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:41.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.865+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183804768 0 0) 0x7fe16c09faf0 con 0x7fe16c09f250 2024-09-15T03:35:41.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.865+0000 7fe16a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.865+0000 7fe16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:41.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.865+0000 7fe16a7fc640 10 monclient(hunting): my global_id is 84525 2024-09-15T03:35:41.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.865+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1540034e0 con 0x7fe16c09f250 2024-09-15T03:35:41.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2610544599 0 0) 0x7fe1540034e0 con 0x7fe16c09f250 2024-09-15T03:35:41.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 msgr2=0x7fe16c09ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:41.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 --2- 172.21.15.153:0/18278172 >> v2:172.21.15.153:3300/0 conn(0x7fe16c09e910 0x7fe16c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 >> v1:172.21.15.181:6789/0 conn(0x7fe16c131ab0 legacy=0x7fe16c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0a1110 con 0x7fe16c09f250 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fe164001aa0 con 0x7fe16c09f250 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: dump: 2024-09-15T03:35:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 >> v1:172.21.15.153:6789/0 conn(0x7fe16c09f250 legacy=0x7fe16c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:41.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe154003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): picked mon.b con 0x7fe154003b20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe154004780 0x7fe154004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): picked mon.a con 0x7fe154004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1540053b0 0x7fe15400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): picked mon.c con 0x7fe1540053b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c132590 con 0x7fe154004780 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15400c380 con 0x7fe154003b20 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15400c540 con 0x7fe1540053b0 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe154004780 0x7fe154004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1540053b0 0x7fe15400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe154003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe154004780 auth_method 0 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request con 0x7fe154003b20 auth_method 0 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): get_auth_request con 0x7fe1540053b0 auth_method 0 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_done global_id 84525 payload 293 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 1 -- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1540053b0 msgr2=0x7fe15400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1540053b0 0x7fe15400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.867+0000 7fe16b7fe640 1 -- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe154004780 msgr2=0x7fe154004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe154004780 0x7fe154004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 1 monclient: found mon.b 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 1 -- 172.21.15.153:0/18278172 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1540034e0 con 0x7fe154003b20 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe17211d640 5 monclient: authenticate success, global_id 84525 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe17211d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.868+0000 7fe16bfff640 1 -- 172.21.15.153:0/18278172 reap_dead start 2024-09-15T03:35:41.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16b7fe640 1 -- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 msgr2=0x7fe154003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:41.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe154003f30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe15c001b00 tx=0x7fe15c007c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a467db412b3a599e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:41.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c02ac50 con 0x7fe154003b20 2024-09-15T03:35:41.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:41.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: dump: 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c02adb0 con 0x7fe154003b20 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe17211d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:41.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe17211d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe17211d640 4 set_mon_vals no callback set 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 1 -- 172.21.15.153:0/18278172 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c007340 con 0x7fe154003b20 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.871+0000 7fe16a7fc640 10 monclient: dump: 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:41.884 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 10 monclient: shutdown 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 -- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 msgr2=0x7fe154003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe154003f30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe15c001b00 tx=0x7fe15c007c40 comp rx=0 tx=0).stop 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 -- 172.21.15.153:0/18278172 shutdown_connections 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 --2- 172.21.15.153:0/18278172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe154003f30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 -- 172.21.15.153:0/18278172 >> 172.21.15.153:0/18278172 conn(0x7fe16c067d30 msgr2=0x7fe16c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 -- 172.21.15.153:0/18278172 shutdown_connections 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 1 -- 172.21.15.153:0/18278172 wait complete. 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 10 monclient: build_initial_monmap 2024-09-15T03:35:41.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.872+0000 7fe17211d640 10 monclient: monmap: 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.873+0000 7fe17211d640 1 Processor -- start 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.873+0000 7fe17211d640 1 -- start start 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.873+0000 7fe17211d640 10 monclient: init 2024-09-15T03:35:41.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe16c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): picked mon.noname-b con 0x7fe154003b20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16c11bb80 0x7fe16c130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): picked mon.noname-c con 0x7fe16c11bb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 0x7fe16c140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): picked mon.noname-a con 0x7fe16c1310c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c11e080 con 0x7fe16c1310c0 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c09bde0 con 0x7fe154003b20 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0a41b0 con 0x7fe16c11bb80 2024-09-15T03:35:41.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe17211d640 10 monclient(hunting): authenticate will time out at 1139.513150s 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16c11bb80 0x7fe16c130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 0x7fe16c140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe16c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16c11bb80 0x7fe16c130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52824/0 (socket says 172.21.15.153:52824) 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): get_auth_request con 0x7fe16c1310c0 auth_method 0 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 1 -- 172.21.15.153:0/3145465961 learned_addr learned my addr 172.21.15.153:0/3145465961 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request con 0x7fe154003b20 auth_method 0 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:41.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c11bb80 auth_method 0 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:41.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient(hunting): handle_auth_done global_id 84528 payload 824 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 -- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16c11bb80 msgr2=0x7fe16c130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16c11bb80 0x7fe16c130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 -- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 msgr2=0x7fe16c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe16c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 monclient: found mon.noname-a 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16bfff640 1 -- 172.21.15.153:0/3145465961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c09b600 con 0x7fe16c1310c0 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.874+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe16c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.876+0000 7fe16bfff640 1 -- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 msgr2=0x7fe16c140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:41.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.876+0000 7fe16bfff640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 0x7fe16c140f10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0a4f90 tx=0x7fe16400a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16400ae00 con 0x7fe16c1310c0 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: dump: 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:41.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe164007350 con 0x7fe16c1310c0 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 5 monclient: authenticate success, global_id 84528 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 10 monclient: _renew_subs 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c11c610 con 0x7fe16c1310c0 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164007710 con 0x7fe16c1310c0 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe1537fe640 10 monclient: dump: 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:41.892 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 10 monclient: _renew_subs 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c11cd40 con 0x7fe16c1310c0 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.877+0000 7fe169ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe16401d460 con 0x7fe16c1310c0 2024-09-15T03:35:41.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe17211d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe17211d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe17211d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c04bf60 con 0x7fe16c1310c0 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe1537fe640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe158038740 0x7fe15803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(170..170 src has 106..170) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fe16404c270 con 0x7fe16c1310c0 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.878+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe158038740 0x7fe15803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.879+0000 7fe16b7fe640 10 monclient: get_auth_request con 0x7fe158038740 auth_method 0 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.881+0000 7fe16b7fe640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe158038740 0x7fe15803abf0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe15c02f000 tx=0x7fe15c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:41.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.883+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe16401a6d0 con 0x7fe16c1310c0 2024-09-15T03:35:41.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.883+0000 7fe1537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:41.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:41.883+0000 7fe1537fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:41.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:41.925+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v171) v1 ==== 82+0+0 (secure 0 0 0) 0x7f539c025070 con 0x7f53a4131c00 2024-09-15T03:35:41.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:41.925+0000 7f538affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:35:41.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:41.925+0000 7f538affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:42.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.028+0000 7f53a98dd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:35:42.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.028+0000 7f53a98dd640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:35:42.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.028+0000 7f53a98dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.028+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f53a40008d0 con 0x7f53a4131c00 2024-09-15T03:35:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.029+0000 7f538affd640 1 -- 172.21.15.181:0/2944247864 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v171) v1 ==== 115+0+0 (secure 0 0 0) 0x7f53a411f820 con 0x7f53a4131c00 2024-09-15T03:35:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.029+0000 7f538affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:35:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.029+0000 7f538affd640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T03:35:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:compat weight-set already created 2024-09-15T03:35:42.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5390038740 msgr2=0x7f539003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5390038740 0x7f539003abf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f53a409c7c0 tx=0x7f5394032000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 10 monclient: shutdown 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 msgr2=0x7f53a4141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4141260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f539c00b810 tx=0x7f539c0092b0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 shutdown_connections 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5390038740 0x7f539003abf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53a4131c00 0x7f53a4141260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53a40a0b00 0x7f53a41314c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 --2- 172.21.15.181:0/2944247864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a4068d00 0x7f53a4130f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 >> 172.21.15.181:0/2944247864 conn(0x7f53a4071370 msgr2=0x7f53a409de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 shutdown_connections 2024-09-15T03:35:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.031+0000 7f53a98dd640 1 -- 172.21.15.181:0/2944247864 wait complete. 2024-09-15T03:35:42.043 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket fooo host 2024-09-15T03:35:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.106+0000 7fe17211d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.106+0000 7fe17211d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.106+0000 7fe17211d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.106+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe16c066280 con 0x7fe16c1310c0 2024-09-15T03:35:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.107+0000 7fe1537fe640 1 -- 172.21.15.153:0/3145465961 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe16401a310 con 0x7fe16c1310c0 2024-09-15T03:35:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.107+0000 7fe1537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:42.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.107+0000 7fe1537fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:42.109 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe158038740 msgr2=0x7fe15803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe158038740 0x7fe15803abf0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe15c02f000 tx=0x7fe15c030000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 10 monclient: shutdown 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 msgr2=0x7fe16c140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 0x7fe16c140f10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0a4f90 tx=0x7fe16400a200 comp rx=0 tx=0).stop 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 shutdown_connections 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe158038740 0x7fe15803abf0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe16c1310c0 0x7fe16c140f10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16c11bb80 0x7fe16c130b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 --2- 172.21.15.153:0/3145465961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe154003b20 0x7fe16c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 >> 172.21.15.153:0/3145465961 conn(0x7fe16c067d30 msgr2=0x7fe16c09c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 shutdown_connections 2024-09-15T03:35:42.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.109+0000 7fe17211d640 1 -- 172.21.15.153:0/3145465961 wait complete. 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.114+0000 7f17b779d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.114+0000 7f17b779d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.114+0000 7f17b779d640 10 monclient: monmap: 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:42.116672+0000 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:42.116672+0000 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.120 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.115+0000 7f17b779d640 1 Processor -- start 2024-09-15T03:35:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.115+0000 7f17b779d640 1 -- start start 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.115+0000 7f17b779d640 10 monclient: init 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.115+0000 7f17b779d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17b009cad0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f17b009c1a0 0x7f17b01348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): picked mon.noname-a con 0x7f17b009c1a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 0x7f17b01371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): picked mon.noname-b con 0x7f17b0134e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b009d5b0 con 0x7f17b009cad0 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b009b710 con 0x7f17b009c1a0 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b009a650 con 0x7f17b0134e10 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b779d640 10 monclient(hunting): authenticate will time out at 1146.598073s 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5539640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f17b009cad0 0x7f17b009cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:33870/0 (socket says 172.21.15.181:33870) 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5539640 1 -- 172.21.15.181:0/2679065643 learned_addr learned my addr 172.21.15.181:0/2679065643 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 1 --2- 172.21.15.181:0/2679065643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 0x7f17b01371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): get_auth_request con 0x7f17b0134e10 auth_method 0 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 1 --2- 172.21.15.181:0/2679065643 >> v2:172.21.15.153:3300/0 conn(0x7f17b009c1a0 0x7f17b01348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): get_auth_request con 0x7f17b009c1a0 auth_method 0 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b4d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_done global_id 74633 payload 274 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 1 -- 172.21.15.181:0/2679065643 >> v2:172.21.15.153:3300/0 conn(0x7f17b009c1a0 msgr2=0x7f17b01348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 1 --2- 172.21.15.181:0/2679065643 >> v2:172.21.15.153:3300/0 conn(0x7f17b009c1a0 0x7f17b01348d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 1 -- 172.21.15.181:0/2679065643 >> v1:172.21.15.181:6789/0 conn(0x7f17b009cad0 legacy=0x7f17b009cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 1 monclient: found mon.noname-b 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.116+0000 7f17b5d3a640 1 -- 172.21.15.181:0/2679065643 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b009f660 con 0x7f17b0134e10 2024-09-15T03:35:42.128 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:35:42.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:35:42.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:35:42.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17b5d3a640 1 -- 172.21.15.181:0/2679065643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 msgr2=0x7f17b01371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17b5d3a640 1 --2- 172.21.15.181:0/2679065643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 0x7f17b01371f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009e90 tx=0x7f17a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d7a28412813e7c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 1 -- 172.21.15.181:0/2679065643 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a8007760 con 0x7f17b0134e10 2024-09-15T03:35:42.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: dump: 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 1 -- 172.21.15.181:0/2679065643 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a802d050 con 0x7f17b0134e10 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17b779d640 5 monclient: authenticate success, global_id 74633 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17b779d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17b779d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 1 -- 172.21.15.181:0/2679065643 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a8007af0 con 0x7f17b0134e10 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17b779d640 4 set_mon_vals no callback set 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.119+0000 7f17a7fff640 10 monclient: dump: 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.138 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 10 monclient: shutdown 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 -- 172.21.15.181:0/2679065643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 msgr2=0x7f17b01371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 --2- 172.21.15.181:0/2679065643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 0x7f17b01371f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009e90 tx=0x7f17a8009780 comp rx=0 tx=0).stop 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 -- 172.21.15.181:0/2679065643 shutdown_connections 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 --2- 172.21.15.181:0/2679065643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b0134e10 0x7f17b01371f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 --2- 172.21.15.181:0/2679065643 >> v2:172.21.15.153:3300/0 conn(0x7f17b009c1a0 0x7f17b01348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 -- 172.21.15.181:0/2679065643 >> 172.21.15.181:0/2679065643 conn(0x7f17b0070e90 msgr2=0x7f17b00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 -- 172.21.15.181:0/2679065643 shutdown_connections 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 -- 172.21.15.181:0/2679065643 wait complete. 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 10 monclient: monmap: 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:42.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.120+0000 7f17b779d640 1 Processor -- start 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 -- start start 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient: init 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 0x7f17b011e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): picked mon.noname-a con 0x7f17b009c1a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b009cad0 0x7f17b011e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): picked mon.noname-c con 0x7f17b009cad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): picked mon.noname-b con 0x7f17b0134e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b006eba0 con 0x7f17b009c1a0 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b00a2190 con 0x7f17b0134e10 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b011fee0 con 0x7f17b009cad0 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b779d640 10 monclient(hunting): authenticate will time out at 1146.603670s 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55470/0 (socket says 172.21.15.181:55470) 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 -- 172.21.15.181:0/3168522428 learned_addr learned my addr 172.21.15.181:0/3168522428 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): get_auth_request con 0x7f17b0134e10 auth_method 0 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b009cad0 0x7f17b011e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 0x7f17b011e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): get_auth_request con 0x7f17b009cad0 auth_method 0 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): get_auth_request con 0x7f17b009c1a0 auth_method 0 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 msgr2=0x7f17b011edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:35:42.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 msgr2=0x7f17b011edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b4d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.121+0000 7f17b5d3a640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 10 monclient(hunting): handle_auth_done global_id 84534 payload 824 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b009cad0 msgr2=0x7f17b011e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b009cad0 0x7f17b011e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 msgr2=0x7f17b011edb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.144 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b5539640 1 -- 172.21.15.181:0/3168522428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a8009a20 con 0x7f17b009c1a0 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.122+0000 7f17b4d38640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b009cad0 0x7f17b011e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b5539640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 msgr2=0x7f17b011e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b5539640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 0x7f17b011e330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00c490 tx=0x7f17ac007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac005cb0 con 0x7f17b009c1a0 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17a57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17a57fa640 10 monclient: dump: 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.145 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17a57fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 5 monclient: authenticate success, global_id 84534 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 10 monclient: _renew_subs 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b01219a0 con 0x7f17b009c1a0 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 10 monclient: _renew_subs 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.146 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.124+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b01349f0 con 0x7f17b009c1a0 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ac005e10 con 0x7f17b009c1a0 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac0072e0 con 0x7f17b009c1a0 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 10 monclient: dump: 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.147 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a77fe640 4 set_mon_vals no callback set 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f17ac007440 con 0x7f17b009c1a0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17b779d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17b779d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17b779d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1790005010 con 0x7f17b009c1a0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1794038740 0x7f179403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.125+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(172..172 src has 106..172) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f17ac04b7b0 con 0x7f17b009c1a0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.126+0000 7f17b4d38640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1794038740 0x7f179403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.126+0000 7f17b4d38640 10 monclient: get_auth_request con 0x7f1794038740 auth_method 0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.128+0000 7f17b4d38640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1794038740 0x7f179403abf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17b009e610 tx=0x7f17a00090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.148 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.130+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f17ac01c440 con 0x7f17b009c1a0 2024-09-15T03:35:42.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.130+0000 7f17a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.130+0000 7f17a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:42.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.227+0000 7f21a971d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.227+0000 7f21a971d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.227+0000 7f21a971d640 10 monclient: monmap: 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:42.228605+0000 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:42.228605+0000 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:42.229 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 1 Processor -- start 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 1 -- start start 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient: init 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f21a4131ab0 0x7f21a4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient(hunting): picked mon.noname-a con 0x7f21a4131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21a409e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21a4136bb0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a409faf0 con 0x7f21a409e910 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a409fd30 con 0x7f21a4136bb0 2024-09-15T03:35:42.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a40a1110 con 0x7f21a4131ab0 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.228+0000 7f21a971d640 10 monclient(hunting): authenticate will time out at 1139.867978s 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f21a4131ab0 0x7f21a4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a27fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f21a409e910 0x7f21a409ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55442/0 (socket says 172.21.15.153:55442) 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a27fc640 1 -- 172.21.15.153:0/1147679034 learned_addr learned my addr 172.21.15.153:0/1147679034 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): get_auth_request con 0x7f21a4131ab0 auth_method 0 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_done global_id 84537 payload 274 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 1 -- 172.21.15.153:0/1147679034 >> v1:172.21.15.181:6789/0 conn(0x7f21a4136bb0 legacy=0x7f21a4138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 1 -- 172.21.15.153:0/1147679034 >> v1:172.21.15.153:6789/0 conn(0x7f21a409e910 legacy=0x7f21a409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.229+0000 7f21a2ffd640 1 -- 172.21.15.153:0/1147679034 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a40a13f0 con 0x7f21a4131ab0 2024-09-15T03:35:42.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a2ffd640 1 -- 172.21.15.153:0/1147679034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4131ab0 msgr2=0x7f21a4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a2ffd640 1 --2- 172.21.15.153:0/1147679034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4131ab0 0x7f21a4131e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2194009e90 tx=0x7f2194009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac9ca7c60f390cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 1 -- 172.21.15.153:0/1147679034 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194007840 con 0x7f21a4131ab0 2024-09-15T03:35:42.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: dump: 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 5 monclient: authenticate success, global_id 84537 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 1 -- 172.21.15.153:0/1147679034 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219402d050 con 0x7f21a4131ab0 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 4 set_mon_vals no callback set 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 1 -- 172.21.15.153:0/1147679034 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194007bd0 con 0x7f21a4131ab0 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a1ffb640 10 monclient: dump: 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.242 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 10 monclient: shutdown 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 -- 172.21.15.153:0/1147679034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4131ab0 msgr2=0x7f21a4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 --2- 172.21.15.153:0/1147679034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4131ab0 0x7f21a4131e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2194009e90 tx=0x7f2194009780 comp rx=0 tx=0).stop 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 -- 172.21.15.153:0/1147679034 shutdown_connections 2024-09-15T03:35:42.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 --2- 172.21.15.153:0/1147679034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4131ab0 0x7f21a4131e80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 -- 172.21.15.153:0/1147679034 >> 172.21.15.153:0/1147679034 conn(0x7f21a4067d30 msgr2=0x7f21a40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 -- 172.21.15.153:0/1147679034 shutdown_connections 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.232+0000 7f21a971d640 1 -- 172.21.15.153:0/1147679034 wait complete. 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.233+0000 7f21a971d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.233+0000 7f21a971d640 10 monclient: monmap: 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:42.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.233+0000 7f21a971d640 1 Processor -- start 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.233+0000 7f21a971d640 1 -- start start 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.233+0000 7f21a971d640 10 monclient: init 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a409e910 0x7f21a41328c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): picked mon.noname-a con 0x7f21a409e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 0x7f21a4132e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): picked mon.noname-c con 0x7f21a4131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a4136bb0 0x7f21a4133340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): picked mon.noname-b con 0x7f21a4136bb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:42.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4135bd0 con 0x7f21a409e910 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4134e20 con 0x7f21a4136bb0 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4135e40 con 0x7f21a4131ab0 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a971d640 10 monclient(hunting): authenticate will time out at 1139.873448s 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a409e910 0x7f21a41328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 0x7f21a4132e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 0x7f21a4132e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52826/0 (socket says 172.21.15.153:52826) 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a409e910 0x7f21a41328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52764/0 (socket says 172.21.15.153:52764) 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 -- 172.21.15.153:0/112473732 learned_addr learned my addr 172.21.15.153:0/112473732 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): get_auth_request con 0x7f21a4131ab0 auth_method 0 2024-09-15T03:35:42.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): get_auth_request con 0x7f21a409e910 auth_method 0 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a4136bb0 0x7f21a4133340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): get_auth_request con 0x7f21a4136bb0 auth_method 0 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient(hunting): handle_auth_done global_id 74639 payload 824 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a4136bb0 msgr2=0x7f21a4133340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21a4136bb0 0x7f21a4133340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a409e910 msgr2=0x7f21a41328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a409e910 0x7f21a41328c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 monclient: found mon.noname-c 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a27fc640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2194009a20 con 0x7f21a4131ab0 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.234+0000 7f21a2ffd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a409e910 0x7f21a41328c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a27fc640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 msgr2=0x7f21a4132e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a27fc640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 0x7f21a4132e00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f219800c5e0 tx=0x7f2198004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198005cb0 con 0x7f21a4131ab0 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: dump: 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.249 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2198005e30 con 0x7f21a4131ab0 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 5 monclient: authenticate success, global_id 74639 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198007340 con 0x7f21a4131ab0 2024-09-15T03:35:42.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 10 monclient: _renew_subs 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a4134110 con 0x7f21a4131ab0 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f218affd640 10 monclient: dump: 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.251 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a17fa640 4 set_mon_vals no callback set 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 10 monclient: _renew_subs 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.237+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a412d340 con 0x7f21a4131ab0 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a971d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a971d640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 0x7f21a4133880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a971d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4063740 con 0x7f21a41344c0 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f219801b400 con 0x7f21a4131ab0 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 0x7f21a4133880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient: get_auth_request con 0x7f21a41344c0 auth_method 0 2024-09-15T03:35:42.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f218affd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2190038740 0x7f219003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(172..172 src has 106..172) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f219804c100 con 0x7f21a4131ab0 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_done global_id 84543 payload 824 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a2ffd640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21a404bf60 con 0x7f21a41344c0 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a37fe640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2190038740 0x7f219003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.238+0000 7f21a37fe640 10 monclient: get_auth_request con 0x7f2190038740 auth_method 0 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.240+0000 7f21a2ffd640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 msgr2=0x7f21a4133880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.240+0000 7f21a2ffd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 0x7f21a4133880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2194034000 tx=0x7f2194037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.240+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194041070 con 0x7f21a41344c0 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.240+0000 7f218affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.240+0000 7f218affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.240+0000 7f218affd640 10 monclient: dump: 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f21a37fe640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2190038740 0x7f219003abf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f219c007fd0 tx=0x7f219c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f219403c630 con 0x7f21a41344c0 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 msgr2=0x7f21a4133880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 0x7f21a4133880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2194034000 tx=0x7f2194037000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 msgr2=0x7f21a4133880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.241+0000 7f218affd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a41344c0 0x7f21a4133880 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2194034000 tx=0x7f2194037000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.246+0000 7f21a971d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.246+0000 7f21a971d640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 0x7f21a4137380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.246+0000 7f21a971d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.246+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4066280 con 0x7f21a4136f80 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 0x7f21a4137380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient: get_auth_request con 0x7f21a4136f80 auth_method 0 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 10 monclient(hunting): handle_auth_done global_id 84546 payload 824 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.247+0000 7f21a2ffd640 1 -- 172.21.15.153:0/112473732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21a41378c0 con 0x7f21a4136f80 2024-09-15T03:35:42.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f21a2ffd640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 msgr2=0x7f21a4137380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f21a2ffd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 0x7f21a4137380 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f21a4133e10 tx=0x7f2194041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194052420 con 0x7f21a4136f80 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 10 monclient: dump: 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f2194052580 con 0x7f21a4136f80 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 msgr2=0x7f21a4137380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 0x7f21a4137380 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f21a4133e10 tx=0x7f2194041000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 msgr2=0x7f21a4137380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.249+0000 7f218affd640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a4136f80 0x7f21a4137380 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f21a4133e10 tx=0x7f2194041000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 80, 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:42.272 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2190038740 msgr2=0x7f219003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2190038740 0x7f219003abf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f219c007fd0 tx=0x7f219c007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 10 monclient: shutdown 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 msgr2=0x7f21a4132e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 0x7f21a4132e00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f219800c5e0 tx=0x7f2198004960 comp rx=0 tx=0).stop 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a37fe640 1 -- 172.21.15.153:0/112473732 reap_dead start 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 shutdown_connections 2024-09-15T03:35:42.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 --2- 172.21.15.153:0/112473732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21a4131ab0 0x7f21a4132e00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 >> 172.21.15.153:0/112473732 conn(0x7f21a4067d30 msgr2=0x7f21a40a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 shutdown_connections 2024-09-15T03:35:42.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.250+0000 7f21a971d640 1 -- 172.21.15.153:0/112473732 wait complete. 2024-09-15T03:35:42.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:42.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:35:42.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:35:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.291+0000 7f17b779d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T03:35:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.291+0000 7f17b779d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T03:35:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.291+0000 7f17b779d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:42.291+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f1790005840 con 0x7f17b009c1a0 2024-09-15T03:35:42.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.368+0000 7f81e291d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:42.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.368+0000 7f81e291d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.368+0000 7f81e291d640 10 monclient: monmap: 2024-09-15T03:35:42.369 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.369 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:42.369359+0000 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:42.369359+0000 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:42.370 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 Processor -- start 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 -- start start 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient: init 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f81dc096c60 0x7f81dc09a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): picked mon.noname-c con 0x7f81dc096c60 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81dc097030 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81dc133b40 0x7f81dc135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): picked mon.noname-a con 0x7f81dc133b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:42.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc0696a0 con 0x7f81dc097030 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc09ac40 con 0x7f81dc133b40 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc09d6b0 con 0x7f81dc096c60 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e291d640 10 monclient(hunting): authenticate will time out at 1140.008880s 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.369+0000 7f81e0eba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81dc133b40 0x7f81dc135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81db7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f81dc097030 0x7f81dc097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55454/0 (socket says 172.21.15.153:55454) 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81db7fe640 1 -- 172.21.15.153:0/1402192319 learned_addr learned my addr 172.21.15.153:0/1402192319 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): get_auth_request con 0x7f81dc133b40 auth_method 0 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81dbfff640 1 --2- 172.21.15.153:0/1402192319 >> v2:172.21.15.181:3300/0 conn(0x7f81dc096c60 0x7f81dc09a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81dbfff640 10 monclient(hunting): get_auth_request con 0x7f81dc096c60 auth_method 0 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient(hunting): handle_auth_done global_id 84549 payload 274 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 1 -- 172.21.15.153:0/1402192319 >> v2:172.21.15.181:3300/0 conn(0x7f81dc096c60 msgr2=0x7f81dc09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 1 --2- 172.21.15.153:0/1402192319 >> v2:172.21.15.181:3300/0 conn(0x7f81dc096c60 0x7f81dc09a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 1 -- 172.21.15.153:0/1402192319 >> v1:172.21.15.153:6789/0 conn(0x7f81dc097030 legacy=0x7f81dc097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.370+0000 7f81e0eba640 1 -- 172.21.15.153:0/1402192319 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc09d2e0 con 0x7f81dc133b40 2024-09-15T03:35:42.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e0eba640 1 -- 172.21.15.153:0/1402192319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc133b40 msgr2=0x7f81dc135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e0eba640 1 --2- 172.21.15.153:0/1402192319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc133b40 0x7f81dc135f20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0637c0 tx=0x7f81d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c976764fdd44e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 1 -- 172.21.15.153:0/1402192319 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4007760 con 0x7f81dc133b40 2024-09-15T03:35:42.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: dump: 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 5 monclient: authenticate success, global_id 84549 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 1 -- 172.21.15.153:0/1402192319 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d402d050 con 0x7f81dc133b40 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 4 set_mon_vals no callback set 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 1 -- 172.21.15.153:0/1402192319 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4007b30 con 0x7f81dc133b40 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81daffd640 10 monclient: dump: 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.384 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 10 monclient: shutdown 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 -- 172.21.15.153:0/1402192319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc133b40 msgr2=0x7f81dc135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 --2- 172.21.15.153:0/1402192319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc133b40 0x7f81dc135f20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0637c0 tx=0x7f81d4009780 comp rx=0 tx=0).stop 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 -- 172.21.15.153:0/1402192319 shutdown_connections 2024-09-15T03:35:42.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 --2- 172.21.15.153:0/1402192319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc133b40 0x7f81dc135f20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 --2- 172.21.15.153:0/1402192319 >> v2:172.21.15.181:3300/0 conn(0x7f81dc096c60 0x7f81dc09a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 -- 172.21.15.153:0/1402192319 >> 172.21.15.153:0/1402192319 conn(0x7f81dc0672b0 msgr2=0x7f81dc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 -- 172.21.15.153:0/1402192319 shutdown_connections 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.373+0000 7f81e291d640 1 -- 172.21.15.153:0/1402192319 wait complete. 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.374+0000 7f81e291d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.374+0000 7f81e291d640 10 monclient: monmap: 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:42.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.374+0000 7f81e291d640 1 Processor -- start 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.374+0000 7f81e291d640 1 -- start start 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.374+0000 7f81e291d640 10 monclient: init 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81dc096c60 0x7f81dc09b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): picked mon.noname-b con 0x7f81dc096c60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc097030 0x7f81dc132040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): picked mon.noname-a con 0x7f81dc097030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc133b40 0x7f81dc132580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): picked mon.noname-c con 0x7f81dc133b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc135840 con 0x7f81dc097030 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc09add0 con 0x7f81dc096c60 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc133340 con 0x7f81dc133b40 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e291d640 10 monclient(hunting): authenticate will time out at 1140.014362s 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc097030 0x7f81dc132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc133b40 0x7f81dc132580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc133b40 0x7f81dc132580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52838/0 (socket says 172.21.15.153:52838) 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81dc096c60 0x7f81dc09b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 1 -- 172.21.15.153:0/1381510545 learned_addr learned my addr 172.21.15.153:0/1381510545 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): get_auth_request con 0x7f81dc133b40 auth_method 0 2024-09-15T03:35:42.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): get_auth_request con 0x7f81dc097030 auth_method 0 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): get_auth_request con 0x7f81dc096c60 auth_method 0 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient(hunting): handle_auth_done global_id 84552 payload 824 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc133b40 msgr2=0x7f81dc132580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc133b40 0x7f81dc132580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81dc096c60 msgr2=0x7f81dc09b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81dc096c60 0x7f81dc09b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81db7fe640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d4009a20 con 0x7f81dc097030 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.375+0000 7f81e0eba640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc133b40 0x7f81dc132580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81db7fe640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc097030 msgr2=0x7f81dc132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81db7fe640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc097030 0x7f81dc132040 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81d000c4c0 tx=0x7f81d0007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0005cb0 con 0x7f81dc097030 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: dump: 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d0005e30 con 0x7f81dc097030 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 5 monclient: authenticate success, global_id 84552 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 10 monclient: _renew_subs 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc09c400 con 0x7f81dc097030 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0007320 con 0x7f81dc097030 2024-09-15T03:35:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81c3fff640 10 monclient: dump: 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 10 monclient: _renew_subs 2024-09-15T03:35:42.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81da7fc640 4 set_mon_vals no callback set 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc06a440 con 0x7f81dc097030 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.378+0000 7f81e291d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81e291d640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 0x7f81dc12bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc065f20 con 0x7f81dc132ac0 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f81d001c440 con 0x7f81dc097030 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 0x7f81dc12bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient: get_auth_request con 0x7f81dc132ac0 auth_method 0 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81c3fff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81c8038790 0x7f81c803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 10 monclient(hunting): handle_auth_done global_id 74645 payload 824 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(172..172 src has 106..172) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f81d004ba60 con 0x7f81dc097030 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81dbfff640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81dc132e90 con 0x7f81dc132ac0 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81e0eba640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81c8038790 0x7f81c803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.379+0000 7f81e0eba640 10 monclient: get_auth_request con 0x7f81c8038790 auth_method 0 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81dbfff640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 msgr2=0x7f81dc12bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81dbfff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 0x7f81dc12bd30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0108b0 tx=0x7f81cc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00ed50 con 0x7f81dc132ac0 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81c3fff640 10 monclient: dump: 2024-09-15T03:35:42.395 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.381+0000 7f81e0eba640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81c8038790 0x7f81c803ac40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f81d4003400 tx=0x7f81d4037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f81cc00eeb0 con 0x7f81dc132ac0 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:42.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 msgr2=0x7f81dc12bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 0x7f81dc12bd30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0108b0 tx=0x7f81cc008040 comp rx=0 tx=0).stop 2024-09-15T03:35:42.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 msgr2=0x7f81dc12bd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.382+0000 7f81c3fff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc132ac0 0x7f81dc12bd30 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0108b0 tx=0x7f81cc008040 comp rx=0 tx=0).stop 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81e291d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81e291d640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 0x7f81dc12c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81e291d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc04bf60 con 0x7f81dc12c270 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 0x7f81dc12c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient: get_auth_request con 0x7f81dc12c270 auth_method 0 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.387+0000 7f81dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.388+0000 7f81dbfff640 10 monclient(hunting): handle_auth_done global_id 74648 payload 824 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.388+0000 7f81dbfff640 1 -- 172.21.15.153:0/1381510545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81dc0983b0 con 0x7f81dc12c270 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81dbfff640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 msgr2=0x7f81dc12c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81dbfff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 0x7f81dc12c670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f81cc019000 tx=0x7f81cc01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc02a070 con 0x7f81dc12c270 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 10 monclient: dump: 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f81cc022040 con 0x7f81dc12c270 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 msgr2=0x7f81dc12c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 0x7f81dc12c670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f81cc019000 tx=0x7f81cc01c000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 msgr2=0x7f81dc12c670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81c3fff640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81dc12c270 0x7f81dc12c670 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f81cc019000 tx=0x7f81cc01c000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.401 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 80, 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:42.414 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81c8038790 msgr2=0x7f81c803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81c8038790 0x7f81c803ac40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f81d4003400 tx=0x7f81d4037000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 10 monclient: shutdown 2024-09-15T03:35:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc097030 msgr2=0x7f81dc132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 1 --2- 172.21.15.153:0/1381510545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81dc097030 0x7f81dc132040 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81d000c4c0 tx=0x7f81d0007f00 comp rx=0 tx=0).stop 2024-09-15T03:35:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e0eba640 1 -- 172.21.15.153:0/1381510545 reap_dead start 2024-09-15T03:35:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e0eba640 1 -- 172.21.15.153:0/1381510545 reap_dead start 2024-09-15T03:35:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 shutdown_connections 2024-09-15T03:35:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.390+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 >> 172.21.15.153:0/1381510545 conn(0x7f81dc0672b0 msgr2=0x7f81dc135b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.391+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 shutdown_connections 2024-09-15T03:35:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.391+0000 7f81e291d640 1 -- 172.21.15.153:0/1381510545 wait complete. 2024-09-15T03:35:42.416 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:42.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:35:42.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:35:42.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.509+0000 7f7a4ff1d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:42.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.509+0000 7f7a4ff1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.509+0000 7f7a4ff1d640 10 monclient: monmap: 2024-09-15T03:35:42.510 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:42.510517+0000 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:42.510517+0000 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:42.511 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 Processor -- start 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 -- start start 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient: init 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 0x7f7a4809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a48096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7a48097030 0x7f7a48097400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a48097030 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:42.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7a4812e5f0 0x7f7a4812ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a4812e5f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4809d2e0 con 0x7f7a48097030 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4809ac00 con 0x7f7a4812e5f0 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4809d6b0 con 0x7f7a48096c60 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.510+0000 7f7a4ff1d640 10 monclient(hunting): authenticate will time out at 1140.149990s 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 0x7f7a4809a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7a48097030 0x7f7a48097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7a48097030 0x7f7a48097400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52804/0 (socket says 172.21.15.153:52804) 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 0x7f7a4809a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52866/0 (socket says 172.21.15.153:52866) 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1634503298 learned_addr learned my addr 172.21.15.153:0/1634503298 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): get_auth_request con 0x7f7a48097030 auth_method 0 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request con 0x7f7a48096c60 auth_method 0 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4e4ba640 1 --2- 172.21.15.153:0/1634503298 >> v2:172.21.15.181:3300/0 conn(0x7f7a4812e5f0 0x7f7a4812ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4e4ba640 10 monclient(hunting): get_auth_request con 0x7f7a4812e5f0 auth_method 0 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_done global_id 84555 payload 274 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 msgr2=0x7f7a4809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 --2- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 0x7f7a4809a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1634503298 >> v2:172.21.15.181:3300/0 conn(0x7f7a4812e5f0 msgr2=0x7f7a4812ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 --2- 172.21.15.153:0/1634503298 >> v2:172.21.15.181:3300/0 conn(0x7f7a4812e5f0 0x7f7a4812ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1634503298 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a480992f0 con 0x7f7a48097030 2024-09-15T03:35:42.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.511+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 0x7f7a4809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48097030 msgr2=0x7f7a48097400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4d4b8640 1 --2- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48097030 0x7f7a48097400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009910 tx=0x7f7a380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b060fcb49d7cbd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 1 -- 172.21.15.153:0/1634503298 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38032440 con 0x7f7a48097030 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: dump: 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.527 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 5 monclient: authenticate success, global_id 84555 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 1 -- 172.21.15.153:0/1634503298 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a380325a0 con 0x7f7a48097030 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:42.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 4 set_mon_vals no callback set 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 1 -- 172.21.15.153:0/1634503298 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38032940 con 0x7f7a48097030 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ccb7640 10 monclient: dump: 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.529 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 10 monclient: shutdown 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48097030 msgr2=0x7f7a48097400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48097030 0x7f7a48097400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009910 tx=0x7f7a380091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1634503298 shutdown_connections 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1634503298 >> v2:172.21.15.181:3300/0 conn(0x7f7a4812e5f0 0x7f7a4812ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48097030 0x7f7a48097400 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1634503298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48096c60 0x7f7a4809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.514+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1634503298 >> 172.21.15.153:0/1634503298 conn(0x7f7a480672b0 msgr2=0x7f7a480676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1634503298 shutdown_connections 2024-09-15T03:35:42.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1634503298 wait complete. 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 10 monclient: monmap: 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 1 Processor -- start 2024-09-15T03:35:42.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 1 -- start start 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.515+0000 7f7a4ff1d640 10 monclient: init 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48096c60 0x7f7a4809b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a48096c60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 0x7f7a4813f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a48097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812e5f0 0x7f7a4812d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4812e5f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4813f790 con 0x7f7a48096c60 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4813fd30 con 0x7f7a4812e5f0 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4811cd70 con 0x7f7a48097030 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4ff1d640 10 monclient(hunting): authenticate will time out at 1140.155232s 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48096c60 0x7f7a4809b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 0x7f7a4813f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48096c60 0x7f7a4809b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52816/0 (socket says 172.21.15.153:52816) 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 0x7f7a4813f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52872/0 (socket says 172.21.15.153:52872) 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 learned_addr learned my addr 172.21.15.153:0/1150320886 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request con 0x7f7a48096c60 auth_method 0 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): get_auth_request con 0x7f7a48097030 auth_method 0 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812e5f0 0x7f7a4812d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): get_auth_request con 0x7f7a4812e5f0 auth_method 0 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4e4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient(hunting): handle_auth_done global_id 74654 payload 824 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812e5f0 msgr2=0x7f7a4812d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812e5f0 0x7f7a4812d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48096c60 msgr2=0x7f7a4809b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48096c60 0x7f7a4809b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 monclient: found mon.noname-c 2024-09-15T03:35:42.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.516+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a44009390 con 0x7f7a48097030 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.518+0000 7f7a4d4b8640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 msgr2=0x7f7a4813f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.518+0000 7f7a4d4b8640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 0x7f7a4813f250 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7a38031000 tx=0x7f7a38032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.518+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38031b10 con 0x7f7a48097030 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.518+0000 7f7a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: dump: 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.536 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38031c90 con 0x7f7a48097030 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 5 monclient: authenticate success, global_id 74654 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 10 monclient: _renew_subs 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3803d690 con 0x7f7a48097030 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a38009450 con 0x7f7a48097030 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 10 monclient: dump: 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.539 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.539 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3ffff640 4 set_mon_vals no callback set 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 10 monclient: _renew_subs 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4809c140 con 0x7f7a48097030 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.519+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7a38039040 con 0x7f7a48097030 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4ff1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4811cc30 con 0x7f7a4812de90 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a2c038740 0x7f7a2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(172..172 src has 106..172) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f7a3806b3c0 con 0x7f7a48097030 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient: get_auth_request con 0x7f7a4812de90 auth_method 0 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4e4ba640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a2c038740 0x7f7a2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4e4ba640 10 monclient: get_auth_request con 0x7f7a2c038740 auth_method 0 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_done global_id 104119 payload 824 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.520+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a4811bc60 con 0x7f7a4812de90 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.522+0000 7f7a4e4ba640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a2c038740 0x7f7a2c03abf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7a40007640 tx=0x7f7a40009850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.522+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 msgr2=0x7f7a4811b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.522+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a44011cb0 tx=0x7f7a440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 msgr2=0x7f7a4811b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 msgr2=0x7f7a4811b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a44011cb0 tx=0x7f7a440090f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a44011cb0 tx=0x7f7a440090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 msgr2=0x7f7a4811b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 0x7f7a2c043780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 msgr2=0x7f7a4811b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a4812de90 0x7f7a4811b010 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c043dc0 con 0x7f7a2c041370 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 0x7f7a2c043780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient: get_auth_request con 0x7f7a2c041370 auth_method 0 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_done global_id 104122 payload 824 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.523+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a2c043f30 con 0x7f7a2c041370 2024-09-15T03:35:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.525+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 msgr2=0x7f7a2c043780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.525+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 0x7f7a2c043780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a44011c80 tx=0x7f7a4401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.525+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44020030 con 0x7f7a2c041370 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.525+0000 7f7a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.525+0000 7f7a3dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.525+0000 7f7a3dffb640 10 monclient: dump: 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7a44021040 con 0x7f7a2c041370 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 msgr2=0x7f7a2c043780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 0x7f7a2c043780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a44011c80 tx=0x7f7a4401c000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 msgr2=0x7f7a2c043780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.526+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a2c041370 0x7f7a2c043780 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a44011c80 tx=0x7f7a4401c000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.531+0000 7f7a4ff1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.531+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 0x7f7a48029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.531+0000 7f7a4ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.531+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48065f20 con 0x7f7a480298c0 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 0x7f7a48029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient: get_auth_request con 0x7f7a480298c0 auth_method 0 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 10 monclient(hunting): handle_auth_done global_id 104125 payload 824 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.532+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a4811cf00 con 0x7f7a480298c0 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.534+0000 7f7a4dcb9640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 msgr2=0x7f7a48029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.534+0000 7f7a4dcb9640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 0x7f7a48029cc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a44028000 tx=0x7f7a44025000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.534+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44034480 con 0x7f7a480298c0 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.534+0000 7f7a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.534+0000 7f7a3dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.534+0000 7f7a3dffb640 10 monclient: dump: 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.548 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f7a4401cc00 con 0x7f7a480298c0 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 msgr2=0x7f7a48029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 0x7f7a48029cc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a44028000 tx=0x7f7a44025000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 msgr2=0x7f7a48029cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:42.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a3dffb640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a480298c0 0x7f7a48029cc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4e4ba640 1 -- 172.21.15.153:0/1150320886 reap_dead start 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 80, 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:42.550 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.551 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.552 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:42.553 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:42.554 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:42.555 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:42.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:42.561 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a2c038740 msgr2=0x7f7a2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a2c038740 0x7f7a2c03abf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7a40007640 tx=0x7f7a40009850 comp rx=0 tx=0).stop 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 10 monclient: shutdown 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 msgr2=0x7f7a4813f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 0x7f7a4813f250 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7a38031000 tx=0x7f7a38032000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 shutdown_connections 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7a2c038740 0x7f7a2c03abf0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 --2- 172.21.15.153:0/1150320886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a48097030 0x7f7a4813f250 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.535+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 >> 172.21.15.153:0/1150320886 conn(0x7f7a480672b0 msgr2=0x7f7a48069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.536+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 shutdown_connections 2024-09-15T03:35:42.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.536+0000 7f7a4ff1d640 1 -- 172.21.15.153:0/1150320886 wait complete. 2024-09-15T03:35:42.564 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:42.564 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:35:42.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:35:42.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.656+0000 7fda5db1d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.656+0000 7fda5db1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.656+0000 7fda5db1d640 10 monclient: monmap: 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:42.657627+0000 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:42.657627+0000 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:42.658 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.657+0000 7fda5db1d640 1 Processor -- start 2024-09-15T03:35:42.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.657+0000 7fda5db1d640 1 -- start start 2024-09-15T03:35:42.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.657+0000 7fda5db1d640 10 monclient: init 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fda58131980 0x7fda58131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): picked mon.noname-c con 0x7fda58131980 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda5809fda0 0x7fda58132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): picked mon.noname-a con 0x7fda5809fda0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 0x7fda5809ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): picked mon.noname-b con 0x7fda5809e9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580a1180 con 0x7fda5809fda0 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580a1460 con 0x7fda58131980 2024-09-15T03:35:42.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5809f340 con 0x7fda5809e9c0 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda5db1d640 10 monclient(hunting): authenticate will time out at 1140.297228s 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 0x7fda5809ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda5809fda0 0x7fda58132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 0x7fda5809ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52888/0 (socket says 172.21.15.153:52888) 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda5809fda0 0x7fda58132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52818/0 (socket says 172.21.15.153:52818) 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 1 -- 172.21.15.153:0/1247324797 learned_addr learned my addr 172.21.15.153:0/1247324797 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): get_auth_request con 0x7fda5809e9c0 auth_method 0 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): get_auth_request con 0x7fda5809fda0 auth_method 0 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 1 --2- 172.21.15.153:0/1247324797 >> v2:172.21.15.181:3300/0 conn(0x7fda58131980 0x7fda58131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): get_auth_request con 0x7fda58131980 auth_method 0 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient(hunting): handle_auth_done global_id 84561 payload 274 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 -- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 msgr2=0x7fda5809ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 --2- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 0x7fda5809ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 -- 172.21.15.153:0/1247324797 >> v2:172.21.15.181:3300/0 conn(0x7fda58131980 msgr2=0x7fda58131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 --2- 172.21.15.153:0/1247324797 >> v2:172.21.15.181:3300/0 conn(0x7fda58131980 0x7fda58131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda56ffd640 1 -- 172.21.15.153:0/1247324797 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5809f4b0 con 0x7fda5809fda0 2024-09-15T03:35:42.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.658+0000 7fda57fff640 1 --2- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 0x7fda5809ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda56ffd640 1 -- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809fda0 msgr2=0x7fda58132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda56ffd640 1 --2- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809fda0 0x7fda58132290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda580a4fb0 tx=0x7fda4c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fc1aebb4a7a544 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 1 -- 172.21.15.153:0/1247324797 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c02adf0 con 0x7fda5809fda0 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: dump: 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda5db1d640 5 monclient: authenticate success, global_id 84561 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 1 -- 172.21.15.153:0/1247324797 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c032440 con 0x7fda5809fda0 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda5db1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda5db1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda5db1d640 4 set_mon_vals no callback set 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 1 -- 172.21.15.153:0/1247324797 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c032820 con 0x7fda5809fda0 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.661+0000 7fda567fc640 10 monclient: dump: 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 10 monclient: shutdown 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 -- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809fda0 msgr2=0x7fda58132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 --2- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809fda0 0x7fda58132290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda580a4fb0 tx=0x7fda4c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 -- 172.21.15.153:0/1247324797 shutdown_connections 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 --2- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809e9c0 0x7fda5809ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 --2- 172.21.15.153:0/1247324797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809fda0 0x7fda58132290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 --2- 172.21.15.153:0/1247324797 >> v2:172.21.15.181:3300/0 conn(0x7fda58131980 0x7fda58131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 -- 172.21.15.153:0/1247324797 >> 172.21.15.153:0/1247324797 conn(0x7fda58067bd0 msgr2=0x7fda580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 -- 172.21.15.153:0/1247324797 shutdown_connections 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 1 -- 172.21.15.153:0/1247324797 wait complete. 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 10 monclient: build_initial_monmap 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.662+0000 7fda5db1d640 10 monclient: monmap: 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:42.671 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 Processor -- start 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 -- start start 2024-09-15T03:35:42.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient: init 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 0x7fda58140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): picked mon.noname-a con 0x7fda5809e9c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809fda0 0x7fda58130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): picked mon.noname-c con 0x7fda5809fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda58131980 0x7fda5811e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): picked mon.noname-b con 0x7fda58131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:42.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5809ce70 con 0x7fda5809e9c0 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda58121ba0 con 0x7fda58131980 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda58121e80 con 0x7fda5809fda0 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda5db1d640 10 monclient(hunting): authenticate will time out at 1140.302633s 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 0x7fda58140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 0x7fda58140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52828/0 (socket says 172.21.15.153:52828) 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809fda0 0x7fda58130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 -- 172.21.15.153:0/3356487300 learned_addr learned my addr 172.21.15.153:0/3356487300 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): get_auth_request con 0x7fda5809e9c0 auth_method 0 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): get_auth_request con 0x7fda5809fda0 auth_method 0 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda57fff640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda58131980 0x7fda5811e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda57fff640 10 monclient(hunting): get_auth_request con 0x7fda58131980 auth_method 0 2024-09-15T03:35:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 10 monclient(hunting): handle_auth_done global_id 84564 payload 824 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 -- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809fda0 msgr2=0x7fda58130a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809fda0 0x7fda58130a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 -- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda58131980 msgr2=0x7fda5811e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.663+0000 7fda577fe640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda58131980 0x7fda5811e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda577fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda577fe640 1 monclient: found mon.noname-a 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda577fe640 1 -- 172.21.15.153:0/3356487300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda50009450 con 0x7fda5809e9c0 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda57fff640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda58131980 0x7fda5811e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.664+0000 7fda56ffd640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809fda0 0x7fda58130a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda577fe640 1 -- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 msgr2=0x7fda58140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda577fe640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 0x7fda58140f20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fda48009ef0 tx=0x7fda4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4800ce10 con 0x7fda5809e9c0 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: dump: 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.680 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 5 monclient: authenticate success, global_id 84564 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 10 monclient: _renew_subs 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4800f040 con 0x7fda5809e9c0 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c009390 con 0x7fda5809e9c0 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda480148c0 con 0x7fda5809e9c0 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:42.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda3f7fe640 10 monclient: dump: 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 10 monclient: _renew_subs 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda58141460 con 0x7fda5809e9c0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.666+0000 7fda55ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fda48014c30 con 0x7fda5809e9c0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda5db1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda5db1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda5db1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda581222f0 con 0x7fda5809e9c0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda3f7fe640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda44038740 0x7fda4403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.667+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(172..172 src has 106..172) v4 ==== 7525+0+0 (secure 0 0 0) 0x7fda4804d1c0 con 0x7fda5809e9c0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.668+0000 7fda56ffd640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda44038740 0x7fda4403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.668+0000 7fda56ffd640 10 monclient: get_auth_request con 0x7fda44038740 auth_method 0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.670+0000 7fda56ffd640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda44038740 0x7fda4403abf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fda4c034000 tx=0x7fda4c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.672+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fda480118c0 con 0x7fda5809e9c0 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.672+0000 7fda3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:42.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.672+0000 7fda3f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.888+0000 7fda5db1d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:35:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.888+0000 7fda5db1d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:35:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.888+0000 7fda5db1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.888+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fda5804bf60 con 0x7fda5809e9c0 2024-09-15T03:35:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.889+0000 7fda3f7fe640 1 -- 172.21.15.153:0/3356487300 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fda4801a2c0 con 0x7fda5809e9c0 2024-09-15T03:35:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.889+0000 7fda3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:35:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.889+0000 7fda3f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:42.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.889+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.889+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=8,logm=62} crc {auth=2661429552,config=3140514009,health=960440409,kv=3235807718,logm=3438833583}) 2024-09-15T03:35:42.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.889+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=8,logm=61} crc {auth=2453222631,config=3140514009,health=960440409,kv=852659224,logm=2319838639}) 2024-09-15T03:35:42.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.889+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.889+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=8,logm=62} crc {auth=2661429552,config=3140514009,health=960440409,kv=3235807718,logm=3438833583}) 2024-09-15T03:35:42.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.889+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=7,logm=63} crc {auth=1645069365,config=3140514009,health=1899157650,kv=2243821149,logm=4103456415}) 2024-09-15T03:35:42.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.890+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.890+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074088748}) 2024-09-15T03:35:42.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.890+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403877315}) 2024-09-15T03:35:42.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.890+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.890+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074088748}) 2024-09-15T03:35:42.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.890+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159927993}) 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda44038740 msgr2=0x7fda4403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda44038740 0x7fda4403abf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fda4c034000 tx=0x7fda4c032000 comp rx=0 tx=0).stop 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 10 monclient: shutdown 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 msgr2=0x7fda58140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 0x7fda58140f20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fda48009ef0 tx=0x7fda4800b040 comp rx=0 tx=0).stop 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 shutdown_connections 2024-09-15T03:35:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fda44038740 0x7fda4403abf0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda58131980 0x7fda5811e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda5809fda0 0x7fda58130a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 --2- 172.21.15.153:0/3356487300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5809e9c0 0x7fda58140f20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 >> 172.21.15.153:0/3356487300 conn(0x7fda58067bd0 msgr2=0x7fda580a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:42.894 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.891+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.894 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.891+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260600938}) 2024-09-15T03:35:42.894 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.891+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3237591321}) 2024-09-15T03:35:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 shutdown_connections 2024-09-15T03:35:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:42.891+0000 7fda5db1d640 1 -- 172.21.15.153:0/3356487300 wait complete. 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.891+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.891+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260600938}) 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.891+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1157896813}) 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.892+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.892+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373915624}) 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.892+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695073541}) 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.892+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.892+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373915624}) 2024-09-15T03:35:42.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.892+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988491755}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.893+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.893+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359456582}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.893+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192677271}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.893+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.893+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359456582}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.893+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914688743}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.894+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.894+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939413929}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.894+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779572282}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.894+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.894+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939413929}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.894+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1307698235}) 2024-09-15T03:35:42.898 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.895+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.895+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738395243}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.895+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398054418}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.895+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.895+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738395243}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.895+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275733548}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.896+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.896+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759922338}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.896+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522048786}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.896+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.896+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759922338}) 2024-09-15T03:35:42.899 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.896+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4239428371}) 2024-09-15T03:35:42.901 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.901 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955655737}) 2024-09-15T03:35:42.901 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346958812}) 2024-09-15T03:35:42.901 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.901 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955655737}) 2024-09-15T03:35:42.901 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638182403}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.897+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=781963594}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825791259}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=781963594}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833945949}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572825308}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651156659}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572825308}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.898+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=302399544}) 2024-09-15T03:35:42.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.899+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.899+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696051483}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.899+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330351284}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.899+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.899+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696051483}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.899+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186533841}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.900+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.900+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559186299}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.900+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960717120}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.900+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.900+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559186299}) 2024-09-15T03:35:42.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.900+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088864985}) 2024-09-15T03:35:42.905 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.901+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.905 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.901+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75953268}) 2024-09-15T03:35:42.905 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.901+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=391699288}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.901+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.901+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75953268}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.901+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=709340232}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.902+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.902+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858065988}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.902+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387099860}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.902+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.902+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858065988}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.902+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2330034175}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912975624}) 2024-09-15T03:35:42.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292793064}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912975624}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966885971}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964118889}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4046635666}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964118889}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.903+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299359457}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.904+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.904+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888789949}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.904+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514514368}) 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.904+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.904+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888789949}) 2024-09-15T03:35:42.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.904+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431030526}) 2024-09-15T03:35:42.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.905+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.905+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175419740}) 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.905+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996672541}) 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.905+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.905+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175419740}) 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.905+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371722720}) 2024-09-15T03:35:42.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355261210}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=248481106}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355261210}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421828991}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2124062990}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525568539}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2124062990}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.906+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337496990}) 2024-09-15T03:35:42.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242299639}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091673309}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242299639}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=298482565}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058510862}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531257982}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058510862}) 2024-09-15T03:35:42.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.907+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813415589}) 2024-09-15T03:35:42.930 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.929+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.930 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.929+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394026350}) 2024-09-15T03:35:42.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.929+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=686653429}) 2024-09-15T03:35:42.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.929+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.929+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394026350}) 2024-09-15T03:35:42.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.929+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372560236}) 2024-09-15T03:35:42.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.932+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.932+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4023293561}) 2024-09-15T03:35:42.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.933+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1576316607}) 2024-09-15T03:35:42.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.933+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.933+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4023293561}) 2024-09-15T03:35:42.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.933+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240433336}) 2024-09-15T03:35:42.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.938+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.938+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798431956}) 2024-09-15T03:35:42.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.938+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1383255550}) 2024-09-15T03:35:42.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.938+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:42.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.938+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798431956}) 2024-09-15T03:35:42.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:42.938+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920575568}) 2024-09-15T03:35:43.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.132+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.132+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699942954}) 2024-09-15T03:35:43.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.132+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=354388011}) 2024-09-15T03:35:43.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.132+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.132+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699942954}) 2024-09-15T03:35:43.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.132+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368485082}) 2024-09-15T03:35:43.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.134+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v173) v1 ==== 140+0+0 (secure 0 0 0) 0x7f17ac017070 con 0x7f17b009c1a0 2024-09-15T03:35:43.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.134+0000 7f17a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T03:35:43.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.134+0000 7f17a57fa640 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-15T03:35:43.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.136+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.136+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313842681}) 2024-09-15T03:35:43.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.136+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052548814}) 2024-09-15T03:35:43.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.136+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.136+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313842681}) 2024-09-15T03:35:43.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.136+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768282411}) 2024-09-15T03:35:43.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.236+0000 7f17b779d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.236+0000 7f17b779d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.236+0000 7f17b779d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.236+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f17900052c0 con 0x7f17b009c1a0 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.237+0000 7f17a57fa640 1 -- 172.21.15.181:0/3168522428 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v173) v1 ==== 128+0+0 (secure 0 0 0) 0x7f17ac011b50 con 0x7f17b009c1a0 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.237+0000 7f17a57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.237+0000 7f17a57fa640 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-15T03:35:43.239 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'fooo' already exists 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1794038740 msgr2=0x7f179403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1794038740 0x7f179403abf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17b009e610 tx=0x7f17a00090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 10 monclient: shutdown 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 msgr2=0x7f17b011e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 0x7f17b011e330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00c490 tx=0x7f17ac007f00 comp rx=0 tx=0).stop 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 shutdown_connections 2024-09-15T03:35:43.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1794038740 0x7f179403abf0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b0134e10 0x7f17b011edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17b009cad0 0x7f17b011e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 --2- 172.21.15.181:0/3168522428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b009c1a0 0x7f17b011e330 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 >> 172.21.15.181:0/3168522428 conn(0x7f17b0070e90 msgr2=0x7f17b00a0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:43.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 shutdown_connections 2024-09-15T03:35:43.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.239+0000 7f17b779d640 1 -- 172.21.15.181:0/3168522428 wait complete. 2024-09-15T03:35:43.250 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush move fooo root=default 2024-09-15T03:35:43.306 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724775507}) 2024-09-15T03:35:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604535058}) 2024-09-15T03:35:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724775507}) 2024-09-15T03:35:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2086381070}) 2024-09-15T03:35:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558738559}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2044807623}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558738559}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.306+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542210020}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.307+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.307+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2690921000,mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2596337278}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.307+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=82,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=4020344538,mdsmap=1531151339,mgr=2815566031,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3360099187}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.307+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.307+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2690921000,mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2596337278}) 2024-09-15T03:35:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.307+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2864998961,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2596337278}) 2024-09-15T03:35:43.310 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.309+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.310 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.309+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=82} crc {mgrstat=2647640722,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=582413037,osd_snap=2673976383,osdmap=1343978601}) 2024-09-15T03:35:43.310 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.309+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=81} crc {mgrstat=2647640722,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=582413037,osd_snap=454132730,osdmap=893740419}) 2024-09-15T03:35:43.310 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.309+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.310 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.309+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=82} crc {mgrstat=2647640722,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=582413037,osd_snap=2673976383,osdmap=1343978601}) 2024-09-15T03:35:43.310 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.309+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=2,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mgrstat=2647640722,mon_config_key=1321035168,monmap=244636553,osd_metadata=851455416,osd_pg_creating=582413037,osd_snap=454132730,osdmap=1373939382}) 2024-09-15T03:35:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.314+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.314+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=4042333206}) 2024-09-15T03:35:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.314+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=30} crc {osdmap=3271010525}) 2024-09-15T03:35:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.314+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:35:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.314+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=4042333206}) 2024-09-15T03:35:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:35:43.314+0000 7f64a4bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=2777250224}) 2024-09-15T03:35:43.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.324+0000 7f80f004f640 10 monclient: get_monmap_and_config 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.324+0000 7f80f004f640 10 monclient: build_initial_monmap 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.324+0000 7f80f004f640 10 monclient: monmap: 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:43.325900+0000 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:43.325900+0000 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:43.327 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.324+0000 7f80f004f640 1 Processor -- start 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.324+0000 7f80f004f640 1 -- start start 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.324+0000 7f80f004f640 10 monclient: init 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f80e809ec50 0x7f80e8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): picked mon.noname-c con 0x7f80e809ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80e81321a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f80e809f020 0x7f80e8069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): picked mon.noname-a con 0x7f80e809f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e8067420 con 0x7f80e81321a0 2024-09-15T03:35:43.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e809fdc0 con 0x7f80e809f020 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e80a0510 con 0x7f80e809ec50 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80f004f640 10 monclient(hunting): authenticate will time out at 1147.807485s 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f80e809ec50 0x7f80e8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f80e809ec50 0x7f80e8131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55484/0 (socket says 172.21.15.181:55484) 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 -- 172.21.15.181:0/189967268 learned_addr learned my addr 172.21.15.181:0/189967268 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): get_auth_request con 0x7f80e809ec50 auth_method 0 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80ee5ec640 1 --2- 172.21.15.181:0/189967268 >> v2:172.21.15.153:3300/0 conn(0x7f80e809f020 0x7f80e8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80ee5ec640 10 monclient(hunting): get_auth_request con 0x7f80e809f020 auth_method 0 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80ee5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80ee5ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80ee5ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient(hunting): handle_auth_done global_id 104134 payload 274 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 -- 172.21.15.181:0/189967268 >> v2:172.21.15.153:3300/0 conn(0x7f80e809f020 msgr2=0x7f80e8069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 --2- 172.21.15.181:0/189967268 >> v2:172.21.15.153:3300/0 conn(0x7f80e809f020 0x7f80e8069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 -- 172.21.15.181:0/189967268 >> v1:172.21.15.153:6789/0 conn(0x7f80e81321a0 legacy=0x7f80e8068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient: _finish_hunting 0 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 monclient: found mon.noname-c 2024-09-15T03:35:43.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:43.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:35:43.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80eddeb640 1 -- 172.21.15.181:0/189967268 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e80a0200 con 0x7f80e809ec50 2024-09-15T03:35:43.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.325+0000 7f80ee5ec640 1 --2- 172.21.15.181:0/189967268 >> v2:172.21.15.153:3300/0 conn(0x7f80e809f020 0x7f80e8069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:43.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80eddeb640 1 -- 172.21.15.181:0/189967268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809ec50 msgr2=0x7f80e8131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:43.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80eddeb640 1 --2- 172.21.15.181:0/189967268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809ec50 0x7f80e8131c60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f80e80719d0 tx=0x7f80e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e221239252cbb84a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 1 -- 172.21.15.181:0/189967268 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4007610 con 0x7f80e809ec50 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: dump: 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:43.338 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: _finish_auth 0 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 1 -- 172.21.15.181:0/189967268 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e402d050 con 0x7f80e809ec50 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80f004f640 5 monclient: authenticate success, global_id 104134 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80f004f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80f004f640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 1 -- 172.21.15.181:0/189967268 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40079a0 con 0x7f80e809ec50 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80f004f640 4 set_mon_vals no callback set 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.328+0000 7f80ecde9640 10 monclient: dump: 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:43.339 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 10 monclient: shutdown 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 -- 172.21.15.181:0/189967268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809ec50 msgr2=0x7f80e8131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 --2- 172.21.15.181:0/189967268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809ec50 0x7f80e8131c60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f80e80719d0 tx=0x7f80e4009840 comp rx=0 tx=0).stop 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 -- 172.21.15.181:0/189967268 shutdown_connections 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 --2- 172.21.15.181:0/189967268 >> v2:172.21.15.153:3300/0 conn(0x7f80e809f020 0x7f80e8069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 --2- 172.21.15.181:0/189967268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809ec50 0x7f80e8131c60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 -- 172.21.15.181:0/189967268 >> 172.21.15.181:0/189967268 conn(0x7f80e80a5d40 msgr2=0x7f80e80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 -- 172.21.15.181:0/189967268 shutdown_connections 2024-09-15T03:35:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.329+0000 7f80f004f640 1 -- 172.21.15.181:0/189967268 wait complete. 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient: build_initial_monmap 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient: monmap: 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:43.342 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 Processor -- start 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 -- start start 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient: init 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e809ec50 0x7f80e809ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): picked mon.noname-a con 0x7f80e809ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 0x7f80e811bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): picked mon.noname-b con 0x7f80e809f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80e81321a0 0x7f80e8130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): picked mon.noname-c con 0x7f80e81321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:43.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e811dfe0 con 0x7f80e809ec50 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e811dd00 con 0x7f80e809f020 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e809dda0 con 0x7f80e81321a0 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80f004f640 10 monclient(hunting): authenticate will time out at 1147.812819s 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 0x7f80e811bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 0x7f80e811bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55500/0 (socket says 172.21.15.181:55500) 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 1 -- 172.21.15.181:0/2717281538 learned_addr learned my addr 172.21.15.181:0/2717281538 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 10 monclient(hunting): get_auth_request con 0x7f80e809f020 auth_method 0 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:43.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ed5ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.330+0000 7f80ee5ec640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80e81321a0 0x7f80e8130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): get_auth_request con 0x7f80e81321a0 auth_method 0 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e809ec50 0x7f80e809ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): get_auth_request con 0x7f80e809ec50 auth_method 0 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ee5ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80eddeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 10 monclient(hunting): handle_auth_done global_id 104137 payload 824 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 1 -- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80e81321a0 msgr2=0x7f80e8130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80e81321a0 0x7f80e8130be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 1 -- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e809ec50 msgr2=0x7f80e809ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e809ec50 0x7f80e809ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 10 monclient: _finish_hunting 0 2024-09-15T03:35:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 1 monclient: found mon.noname-b 2024-09-15T03:35:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:35:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.331+0000 7f80ed5ea640 1 -- 172.21.15.181:0/2717281538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e4009ae0 con 0x7f80e809f020 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ed5ea640 1 -- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 msgr2=0x7f80e811bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ed5ea640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 0x7f80e811bc30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f80d800c5e0 tx=0x7f80d8007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8012840 con 0x7f80e809f020 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: dump: 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:43.348 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d80129a0 con 0x7f80e809f020 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 5 monclient: authenticate success, global_id 104137 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 10 monclient: _renew_subs 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e811c170 con 0x7f80e809f020 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8012da0 con 0x7f80e809f020 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:43.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80ddffb640 10 monclient: dump: 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80dffff640 4 set_mon_vals no callback set 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 10 monclient: _renew_subs 2024-09-15T03:35:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.333+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e81312c0 con 0x7f80e809f020 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.334+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f80d8011590 con 0x7f80e809f020 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.334+0000 7f80f004f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.334+0000 7f80f004f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.334+0000 7f80f004f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.334+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e809c070 con 0x7f80e809f020 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.335+0000 7f80ddffb640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f80cc038740 0x7f80cc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.335+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(173..173 src has 106..173) v4 ==== 7553+0+0 (secure 0 0 0) 0x7f80d804c6e0 con 0x7f80e809f020 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.335+0000 7f80eddeb640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f80cc038740 0x7f80cc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.335+0000 7f80eddeb640 10 monclient: get_auth_request con 0x7f80cc038740 auth_method 0 2024-09-15T03:35:43.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.337+0000 7f80eddeb640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f80cc038740 0x7f80cc03abf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f80e80a2b00 tx=0x7f80e4036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:43.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.339+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f80e809c070 con 0x7f80e809f020 2024-09-15T03:35:43.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.339+0000 7f80ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:43.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.339+0000 7f80ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:43.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.498+0000 7f80f004f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T03:35:43.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.498+0000 7f80f004f640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T03:35:43.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.498+0000 7f80f004f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:43.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:43.498+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7f80e809c070 con 0x7f80e809f020 2024-09-15T03:35:43.910 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:35:43.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:44.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.007+0000 7f0c01078640 10 monclient: get_monmap_and_config 2024-09-15T03:35:44.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.007+0000 7f0c01078640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.007+0000 7f0c01078640 10 monclient: monmap: 2024-09-15T03:35:44.008 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:44.008353+0000 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:44.008353+0000 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:44.009 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 1 Processor -- start 2024-09-15T03:35:44.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 1 -- start start 2024-09-15T03:35:44.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient: init 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bfc131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bfc09fda0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0bfc09f0d0 0x7f0bfc139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient(hunting): picked mon.noname-a con 0x7f0bfc09f0d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc09f510 con 0x7f0bfc09fda0 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc0a1180 con 0x7f0bfc131980 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bfc0a1460 con 0x7f0bfc09f0d0 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0c01078640 10 monclient(hunting): authenticate will time out at 1141.647690s 2024-09-15T03:35:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0bfc09f0d0 0x7f0bfc139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfa59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0bfc09fda0 0x7f0bfc09e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55464/0 (socket says 172.21.15.153:55464) 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0bfc09f0d0 0x7f0bfc139cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52834/0 (socket says 172.21.15.153:52834) 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfa59c640 1 -- 172.21.15.153:0/236974798 learned_addr learned my addr 172.21.15.153:0/236974798 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): get_auth_request con 0x7f0bfc09f0d0 auth_method 0 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.008+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_done global_id 84573 payload 274 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 1 -- 172.21.15.153:0/236974798 >> v1:172.21.15.181:6789/0 conn(0x7f0bfc131980 legacy=0x7f0bfc131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 1 -- 172.21.15.153:0/236974798 >> v1:172.21.15.153:6789/0 conn(0x7f0bfc09fda0 legacy=0x7f0bfc09e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 1 monclient: found mon.noname-a 2024-09-15T03:35:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:44.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.009+0000 7f0bfb59e640 1 -- 172.21.15.153:0/236974798 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc132460 con 0x7f0bfc09f0d0 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.011+0000 7f0bfb59e640 1 -- 172.21.15.153:0/236974798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc09f0d0 msgr2=0x7f0bfc139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.011+0000 7f0bfb59e640 1 --2- 172.21.15.153:0/236974798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc139cd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009e60 tx=0x7f0bf4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56eb4ac387484f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 1 -- 172.21.15.153:0/236974798 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4007630 con 0x7f0bfc09f0d0 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: dump: 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 1 -- 172.21.15.153:0/236974798 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf402d050 con 0x7f0bfc09f0d0 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 5 monclient: authenticate success, global_id 84573 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 1 -- 172.21.15.153:0/236974798 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40079c0 con 0x7f0bfc09f0d0 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 4 set_mon_vals no callback set 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0bf9d9b640 10 monclient: dump: 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.024 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.024 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 10 monclient: shutdown 2024-09-15T03:35:44.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 -- 172.21.15.153:0/236974798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc09f0d0 msgr2=0x7f0bfc139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 --2- 172.21.15.153:0/236974798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc139cd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009e60 tx=0x7f0bf4009780 comp rx=0 tx=0).stop 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 -- 172.21.15.153:0/236974798 shutdown_connections 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 --2- 172.21.15.153:0/236974798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc139cd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 -- 172.21.15.153:0/236974798 >> 172.21.15.153:0/236974798 conn(0x7f0bfc067bd0 msgr2=0x7f0bfc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 -- 172.21.15.153:0/236974798 shutdown_connections 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.012+0000 7f0c01078640 1 -- 172.21.15.153:0/236974798 wait complete. 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient: monmap: 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.025 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 1 Processor -- start 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 1 -- start start 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient: init 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:35:44.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient(hunting): picked mon.noname-b con 0x7f0bfc09f0d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 0x7f0bfc134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.013+0000 7f0c01078640 10 monclient(hunting): picked mon.noname-c con 0x7f0bfc09fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc131980 0x7f0bfc1345b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 10 monclient(hunting): picked mon.noname-a con 0x7f0bfc131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc09bf10 con 0x7f0bfc131980 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc139fd0 con 0x7f0bfc09f0d0 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc1379a0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0c01078640 10 monclient(hunting): authenticate will time out at 1141.653039s 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 0x7f0bfc134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc131980 0x7f0bfc1345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 0x7f0bfc134070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52896/0 (socket says 172.21.15.153:52896) 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 -- 172.21.15.153:0/131556971 learned_addr learned my addr 172.21.15.153:0/131556971 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): get_auth_request con 0x7f0bfc09fda0 auth_method 0 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): get_auth_request con 0x7f0bfc131980 auth_method 0 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): get_auth_request con 0x7f0bfc09f0d0 auth_method 0 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient(hunting): handle_auth_done global_id 74666 payload 824 2024-09-15T03:35:44.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 -- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc09f0d0 msgr2=0x7f0bfc138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc138380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 -- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc131980 msgr2=0x7f0bfc1345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc131980 0x7f0bfc1345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 monclient: found mon.noname-c 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfa59c640 1 -- 172.21.15.153:0/131556971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4009a20 con 0x7f0bfc09fda0 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.014+0000 7f0bfb59e640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc131980 0x7f0bfc1345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0bfa59c640 1 -- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 msgr2=0x7f0bfc134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0bfa59c640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 0x7f0bfc134070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c5e0 tx=0x7f0bf0004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0005cb0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: dump: 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 5 monclient: authenticate success, global_id 74666 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf0005e30 con 0x7f0bfc09fda0 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 10 monclient: _renew_subs 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc1376c0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0007340 con 0x7f0bfc09fda0 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0be2ffd640 10 monclient: dump: 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 10 monclient: _renew_subs 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bfc1388c0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.016+0000 7f0bf959a640 4 set_mon_vals no callback set 2024-09-15T03:35:44.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.017+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0bf001b400 con 0x7f0bfc09fda0 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.017+0000 7f0c01078640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.017+0000 7f0c01078640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.017+0000 7f0c01078640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.017+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bfc138ba0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.017+0000 7f0be2ffd640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0be8038790 0x7f0be803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.018+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(173..173 src has 106..173) v4 ==== 7553+0+0 (secure 0 0 0) 0x7f0bf004c0b0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.018+0000 7f0bfad9d640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0be8038790 0x7f0be803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.018+0000 7f0bfad9d640 10 monclient: get_auth_request con 0x7f0be8038790 auth_method 0 2024-09-15T03:35:44.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.020+0000 7f0bfad9d640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0be8038790 0x7f0be803ac40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0bec007fd0 tx=0x7f0bec007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.022+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0bf001b6b0 con 0x7f0bfc09fda0 2024-09-15T03:35:44.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.022+0000 7f0be2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.022+0000 7f0be2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:44.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.137+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v174) v1 ==== 173+0+0 (secure 0 0 0) 0x7f80e809c070 con 0x7f80e809f020 2024-09-15T03:35:44.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.137+0000 7f80ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T03:35:44.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.137+0000 7f80ddffb640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-15T03:35:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.236+0000 7f80f004f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.236+0000 7f80f004f640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.236+0000 7f80f004f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.236+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7f80e809c070 con 0x7f80e809f020 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.237+0000 7f80ddffb640 1 -- 172.21.15.181:0/2717281538 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v174) v1 ==== 183+0+0 (secure 0 0 0) 0x7f80e809c070 con 0x7f80e809f020 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.237+0000 7f80ddffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.237+0000 7f80ddffb640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-15T03:35:44.239 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f80cc038740 msgr2=0x7f80cc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f80cc038740 0x7f80cc03abf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f80e80a2b00 tx=0x7f80e4036000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 10 monclient: shutdown 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 msgr2=0x7f80e811bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 0x7f80e811bc30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f80d800c5e0 tx=0x7f80d8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 shutdown_connections 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f80cc038740 0x7f80cc03abf0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80e81321a0 0x7f80e8130be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80e809f020 0x7f80e811bc30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.239+0000 7f80f004f640 1 --2- 172.21.15.181:0/2717281538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e809ec50 0x7f80e809ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.240+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 >> 172.21.15.181:0/2717281538 conn(0x7f80e80a5d40 msgr2=0x7f80e809e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.240+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 shutdown_connections 2024-09-15T03:35:44.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.240+0000 7f80f004f640 1 -- 172.21.15.181:0/2717281538 wait complete. 2024-09-15T03:35:44.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0c01078640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:44.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0c01078640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:44.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0c01078640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0bfc04bf60 con 0x7f0bfc09fda0 2024-09-15T03:35:44.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0be2ffd640 1 -- 172.21.15.153:0/131556971 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0bf001a360 con 0x7f0bfc09fda0 2024-09-15T03:35:44.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0be2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:44.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.241+0000 7f0be2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:44.244 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:44.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0be8038790 msgr2=0x7f0be803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0be8038790 0x7f0be803ac40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0bec007fd0 tx=0x7f0bec007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:44.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 10 monclient: shutdown 2024-09-15T03:35:44.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 msgr2=0x7f0bfc134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 0x7f0bfc134070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c5e0 tx=0x7f0bf0004960 comp rx=0 tx=0).stop 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 shutdown_connections 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0be8038790 0x7f0be803ac40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bfc131980 0x7f0bfc1345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bfc09fda0 0x7f0bfc134070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 --2- 172.21.15.153:0/131556971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc09f0d0 0x7f0bfc138380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 >> 172.21.15.153:0/131556971 conn(0x7f0bfc067bd0 msgr2=0x7f0bfc0a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 shutdown_connections 2024-09-15T03:35:44.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.243+0000 7f0c01078640 1 -- 172.21.15.153:0/131556971 wait complete. 2024-09-15T03:35:44.251 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket barr rack 2024-09-15T03:35:44.255 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:35:44.255 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:35:44.255 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:35:44.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:35:44.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.322+0000 7f01e3434640 10 monclient: get_monmap_and_config 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.322+0000 7f01e3434640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.322+0000 7f01e3434640 10 monclient: monmap: 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:44.324725+0000 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:44.324725+0000 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:44.325 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:44.326 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:44.326 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:44.326 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.323+0000 7f01e3434640 1 Processor -- start 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.323+0000 7f01e3434640 1 -- start start 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.323+0000 7f01e3434640 10 monclient: init 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01dc098c70 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f01dc099c70 0x7f01dc136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): picked mon.noname-a con 0x7f01dc099c70 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f01dc137200 0x7f01dc1395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc137200 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc09f680 con 0x7f01dc098c70 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01dc09f420 con 0x7f01dc099c70 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01dc068a20 con 0x7f01dc137200 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e3434640 10 monclient(hunting): authenticate will time out at 1148.806326s 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f01dc137200 0x7f01dc1395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f01dc137200 0x7f01dc1395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55514/0 (socket says 172.21.15.181:55514) 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 -- 172.21.15.181:0/433970110 learned_addr learned my addr 172.21.15.181:0/433970110 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): get_auth_request con 0x7f01dc137200 auth_method 0 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 1 --2- 172.21.15.181:0/433970110 >> v2:172.21.15.153:3300/0 conn(0x7f01dc099c70 0x7f01dc136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): get_auth_request con 0x7f01dc099c70 auth_method 0 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e09cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient(hunting): handle_auth_done global_id 104143 payload 274 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 -- 172.21.15.181:0/433970110 >> v2:172.21.15.153:3300/0 conn(0x7f01dc099c70 msgr2=0x7f01dc136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 --2- 172.21.15.181:0/433970110 >> v2:172.21.15.153:3300/0 conn(0x7f01dc099c70 0x7f01dc136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 -- 172.21.15.181:0/433970110 >> v1:172.21.15.153:6789/0 conn(0x7f01dc098c70 legacy=0x7f01dc099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 monclient: found mon.noname-c 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.324+0000 7f01e19d1640 1 -- 172.21.15.181:0/433970110 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc096a30 con 0x7f01dc137200 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01e19d1640 1 -- 172.21.15.181:0/433970110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc137200 msgr2=0x7f01dc1395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01e19d1640 1 --2- 172.21.15.181:0/433970110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc137200 0x7f01dc1395e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01dc096c20 tx=0x7f01d4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d84d9fee3f4d03b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 1 -- 172.21.15.181:0/433970110 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d402aea0 con 0x7f01dc137200 2024-09-15T03:35:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: dump: 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:44.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 1 -- 172.21.15.181:0/433970110 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d402d050 con 0x7f01dc137200 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01e3434640 5 monclient: authenticate success, global_id 104143 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01e3434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01e3434640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 1 -- 172.21.15.181:0/433970110 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4007840 con 0x7f01dc137200 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01e3434640 4 set_mon_vals no callback set 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.327+0000 7f01d3fff640 10 monclient: dump: 2024-09-15T03:35:44.335 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 10 monclient: shutdown 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 -- 172.21.15.181:0/433970110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc137200 msgr2=0x7f01dc1395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 --2- 172.21.15.181:0/433970110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc137200 0x7f01dc1395e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01dc096c20 tx=0x7f01d4009840 comp rx=0 tx=0).stop 2024-09-15T03:35:44.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 -- 172.21.15.181:0/433970110 shutdown_connections 2024-09-15T03:35:44.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 --2- 172.21.15.181:0/433970110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc137200 0x7f01dc1395e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 --2- 172.21.15.181:0/433970110 >> v2:172.21.15.153:3300/0 conn(0x7f01dc099c70 0x7f01dc136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 -- 172.21.15.181:0/433970110 >> 172.21.15.181:0/433970110 conn(0x7f01dc070e90 msgr2=0x7f01dc0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 -- 172.21.15.181:0/433970110 shutdown_connections 2024-09-15T03:35:44.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 1 -- 172.21.15.181:0/433970110 wait complete. 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.328+0000 7f01e3434640 10 monclient: monmap: 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.342 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 Processor -- start 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 -- start start 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient: init 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01dc098c70 0x7f01dc135e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): picked mon.noname-a con 0x7f01dc098c70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 0x7f01dc11e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc099c70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01dc137200 0x7f01dc11eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc137200 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc1205c0 con 0x7f01dc098c70 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc122830 con 0x7f01dc099c70 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc06eba0 con 0x7f01dc137200 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.329+0000 7f01e3434640 10 monclient(hunting): authenticate will time out at 1148.811916s 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 0x7f01dc11e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 0x7f01dc11e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55520/0 (socket says 172.21.15.181:55520) 2024-09-15T03:35:44.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 -- 172.21.15.181:0/2124110689 learned_addr learned my addr 172.21.15.181:0/2124110689 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): get_auth_request con 0x7f01dc099c70 auth_method 0 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01dc098c70 0x7f01dc135e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01dc137200 0x7f01dc11eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): get_auth_request con 0x7f01dc098c70 auth_method 0 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): get_auth_request con 0x7f01dc137200 auth_method 0 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e11d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e19d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient(hunting): handle_auth_done global_id 104146 payload 824 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 -- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01dc137200 msgr2=0x7f01dc11eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01dc137200 0x7f01dc11eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 -- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01dc098c70 msgr2=0x7f01dc135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01dc098c70 0x7f01dc135e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 monclient: found mon.noname-b 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.330+0000 7f01e09cf640 1 -- 172.21.15.181:0/2124110689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4009ae0 con 0x7f01dc099c70 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e09cf640 1 -- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 msgr2=0x7f01dc11e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e09cf640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 0x7f01dc11e5a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00ea20 tx=0x7f01cc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc009600 con 0x7f01dc099c70 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:44.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: dump: 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc00cc20 con 0x7f01dc099c70 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 5 monclient: authenticate success, global_id 104146 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 10 monclient: _renew_subs 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01dc121b70 con 0x7f01dc099c70 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc013470 con 0x7f01dc099c70 2024-09-15T03:35:44.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d17fa640 10 monclient: dump: 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01d37fe640 4 set_mon_vals no callback set 2024-09-15T03:35:44.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 10 monclient: _renew_subs 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.332+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01dc136e10 con 0x7f01dc099c70 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f01cc013860 con 0x7f01dc099c70 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01e3434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01e3434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01e3434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01d17fa640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01c0038740 0x7f01c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01dc0a1ab0 con 0x7f01dc099c70 2024-09-15T03:35:44.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7f01cc04c9e0 con 0x7f01dc099c70 2024-09-15T03:35:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01e11d0640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01c0038740 0x7f01c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.333+0000 7f01e11d0640 10 monclient: get_auth_request con 0x7f01c0038740 auth_method 0 2024-09-15T03:35:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.336+0000 7f01e11d0640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01c0038740 0x7f01c003abf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f01d8007fd0 tx=0x7f01d8007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.338+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01cc0236b0 con 0x7f01dc099c70 2024-09-15T03:35:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.338+0000 7f01d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.338+0000 7f01d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:44.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.349+0000 7fe946d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.349+0000 7fe946d9d640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.349+0000 7fe946d9d640 10 monclient: monmap: 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:44.350108+0000 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:44.350108+0000 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:44.362 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 1 Processor -- start 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 1 -- start start 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient: init 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe94009caa0 0x7fe94009ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe94009caa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe94009a540 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9401346c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94009abd0 con 0x7fe94009a540 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94009b920 con 0x7fe9401346c0 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94009ed40 con 0x7fe94009caa0 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe946d9d640 10 monclient(hunting): authenticate will time out at 1141.989635s 2024-09-15T03:35:44.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe93ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe94009a540 0x7fe94006ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55474/0 (socket says 172.21.15.153:55474) 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe93ffff640 1 -- 172.21.15.153:0/764049720 learned_addr learned my addr 172.21.15.153:0/764049720 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 1 --2- 172.21.15.153:0/764049720 >> v2:172.21.15.181:3300/0 conn(0x7fe94009caa0 0x7fe94009ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): get_auth_request con 0x7fe94009caa0 auth_method 0 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.350+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 10 monclient(hunting): handle_auth_done global_id 104149 payload 274 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 1 -- 172.21.15.153:0/764049720 >> v1:172.21.15.181:6789/0 conn(0x7fe9401346c0 legacy=0x7fe940136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 1 -- 172.21.15.153:0/764049720 >> v1:172.21.15.153:6789/0 conn(0x7fe94009a540 legacy=0x7fe94006ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 1 monclient: found mon.noname-c 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.351+0000 7fe944b39640 1 -- 172.21.15.153:0/764049720 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94009c620 con 0x7fe94009caa0 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.353+0000 7fe944b39640 1 -- 172.21.15.153:0/764049720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 msgr2=0x7fe94009ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.353+0000 7fe944b39640 1 --2- 172.21.15.153:0/764049720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe94009ce70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe934009850 tx=0x7fe9340090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a6e300e46720eed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 1 -- 172.21.15.153:0/764049720 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934032440 con 0x7fe94009caa0 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:44.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: dump: 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 1 -- 172.21.15.153:0/764049720 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9340325a0 con 0x7fe94009caa0 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 1 -- 172.21.15.153:0/764049720 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934032940 con 0x7fe94009caa0 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 5 monclient: authenticate success, global_id 104149 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 4 set_mon_vals no callback set 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe93f7fe640 10 monclient: dump: 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.368 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 10 monclient: shutdown 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 -- 172.21.15.153:0/764049720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 msgr2=0x7fe94009ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 --2- 172.21.15.153:0/764049720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe94009ce70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe934009850 tx=0x7fe9340090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 -- 172.21.15.153:0/764049720 shutdown_connections 2024-09-15T03:35:44.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 --2- 172.21.15.153:0/764049720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe94009ce70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 -- 172.21.15.153:0/764049720 >> 172.21.15.153:0/764049720 conn(0x7fe940067cd0 msgr2=0x7fe9400a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 -- 172.21.15.153:0/764049720 shutdown_connections 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.354+0000 7fe946d9d640 1 -- 172.21.15.153:0/764049720 wait complete. 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.355+0000 7fe946d9d640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.355+0000 7fe946d9d640 10 monclient: monmap: 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:44.370 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.355+0000 7fe946d9d640 1 Processor -- start 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.355+0000 7fe946d9d640 1 -- start start 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.355+0000 7fe946d9d640 10 monclient: init 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94009a540 0x7fe940119530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe94009a540 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe940119a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe94009caa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9401346c0 0x7fe940119fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe9401346c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940099dc0 con 0x7fe94009a540 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94011c2f0 con 0x7fe94009caa0 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940137d60 con 0x7fe9401346c0 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe946d9d640 10 monclient(hunting): authenticate will time out at 1141.995144s 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94009a540 0x7fe940119530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94009a540 0x7fe940119530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52856/0 (socket says 172.21.15.153:52856) 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 1 -- 172.21.15.153:0/2261680436 learned_addr learned my addr 172.21.15.153:0/2261680436 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): get_auth_request con 0x7fe94009a540 auth_method 0 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe940119a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): get_auth_request con 0x7fe94009caa0 auth_method 0 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9401346c0 0x7fe940119fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): get_auth_request con 0x7fe9401346c0 auth_method 0 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe94533a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient(hunting): handle_auth_done global_id 104152 payload 824 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9401346c0 msgr2=0x7fe940119fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9401346c0 0x7fe940119fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94009a540 msgr2=0x7fe940119530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94009a540 0x7fe940119530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 monclient: found mon.noname-b 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe93ffff640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe934009390 con 0x7fe94009caa0 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.356+0000 7fe944b39640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94009a540 0x7fe940119530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93ffff640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 msgr2=0x7fe940119a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93ffff640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe940119a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00c4c0 tx=0x7fe92c007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c005cb0 con 0x7fe94009caa0 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93cff9640 10 monclient: dump: 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.376 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93cff9640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.358+0000 7fe93cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c005e30 con 0x7fe94009caa0 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c007320 con 0x7fe94009caa0 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 10 monclient: dump: 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.377 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 5 monclient: authenticate success, global_id 104152 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 10 monclient: _renew_subs 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93effd640 4 set_mon_vals no callback set 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94011b5d0 con 0x7fe94009caa0 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 10 monclient: _renew_subs 2024-09-15T03:35:44.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94011a840 con 0x7fe94009caa0 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe92c01b3b0 con 0x7fe94009caa0 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 0x7fe94006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940065e00 con 0x7fe94006bff0 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe944b39640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 0x7fe94006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe944b39640 10 monclient: get_auth_request con 0x7fe94006bff0 auth_method 0 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe944b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe944b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.359+0000 7fe944b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe93cff9640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe924038740 0x7fe92403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7fe92c04bb40 con 0x7fe94009caa0 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe944b39640 10 monclient(hunting): handle_auth_done global_id 84588 payload 824 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe944b39640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe94004bf60 con 0x7fe94006bff0 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe94533a640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe924038740 0x7fe92403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.360+0000 7fe94533a640 10 monclient: get_auth_request con 0x7fe924038740 auth_method 0 2024-09-15T03:35:44.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe944b39640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 msgr2=0x7fe94006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe944b39640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 0x7fe94006c3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe934035040 tx=0x7fe934032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe94533a640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe924038740 0x7fe92403abf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe938007fd0 tx=0x7fe9380025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93403c440 con 0x7fe94006bff0 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe93cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe93cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.362+0000 7fe93cff9640 10 monclient: dump: 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe93403c5a0 con 0x7fe94006bff0 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 msgr2=0x7fe94006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 0x7fe94006c3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe934035040 tx=0x7fe934032000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 msgr2=0x7fe94006c3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.363+0000 7fe93cff9640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006bff0 0x7fe94006c3c0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe934035040 tx=0x7fe934032000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe946d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe946d9d640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 0x7fe940149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe946d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94011b3b0 con 0x7fe94006c900 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 0x7fe940149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient: get_auth_request con 0x7fe94006c900 auth_method 0 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.368+0000 7fe944b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.369+0000 7fe944b39640 10 monclient(hunting): handle_auth_done global_id 84591 payload 824 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.369+0000 7fe944b39640 1 -- 172.21.15.153:0/2261680436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe94006e7c0 con 0x7fe94006c900 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe944b39640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 msgr2=0x7fe940149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe944b39640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 0x7fe940149640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe934003080 tx=0x7fe93403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340468f0 con 0x7fe94006c900 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 10 monclient: dump: 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7fe93403b3d0 con 0x7fe94006c900 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 msgr2=0x7fe940149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 0x7fe940149640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe934003080 tx=0x7fe93403d000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 msgr2=0x7fe940149640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe93cff9640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94006c900 0x7fe940149640 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe934003080 tx=0x7fe93403d000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.388 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:44.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:44.388 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:44.388 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 80, 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:35:44.389 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.390 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.391 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:44.393 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:44.394 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.395 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.397 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.401 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.402 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:44.403 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:44.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe924038740 msgr2=0x7fe92403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe946d9d640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe924038740 0x7fe92403abf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe938007fd0 tx=0x7fe9380025b0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe946d9d640 10 monclient: shutdown 2024-09-15T03:35:44.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe946d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 msgr2=0x7fe940119a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe946d9d640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe940119a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00c4c0 tx=0x7fe92c007f00 comp rx=0 tx=0).stop 2024-09-15T03:35:44.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.371+0000 7fe94533a640 1 -- 172.21.15.153:0/2261680436 reap_dead start 2024-09-15T03:35:44.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.372+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 shutdown_connections 2024-09-15T03:35:44.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.372+0000 7fe946d9d640 1 --2- 172.21.15.153:0/2261680436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe94009caa0 0x7fe940119a70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.372+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 >> 172.21.15.153:0/2261680436 conn(0x7fe940067cd0 msgr2=0x7fe9400a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.372+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 shutdown_connections 2024-09-15T03:35:44.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.372+0000 7fe946d9d640 1 -- 172.21.15.153:0/2261680436 wait complete. 2024-09-15T03:35:44.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:44.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:35:44.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:35:44.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.495+0000 7f01e3434640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T03:35:44.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.495+0000 7f01e3434640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T03:35:44.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.495+0000 7f01e3434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:44.495+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7f01dc09a070 con 0x7f01dc099c70 2024-09-15T03:35:44.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.501+0000 7f9513d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:44.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.501+0000 7f9513d9d640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.501+0000 7f9513d9d640 10 monclient: monmap: 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:44.502633+0000 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:44.502633+0000 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:44.504 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:44.504 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:44.504 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:44.504 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.502+0000 7f9513d9d640 1 Processor -- start 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.502+0000 7f9513d9d640 1 -- start start 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.502+0000 7f9513d9d640 10 monclient: init 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f950c09c9c0 0x7f950c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f950c09c9c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:44.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f950c09e340 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f950c09e750 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c09bdd0 con 0x7f950c09e340 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c09a880 con 0x7f950c09e750 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c130590 con 0x7f950c09c9c0 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9513d9d640 10 monclient(hunting): authenticate will time out at 1142.142311s 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f950c09c9c0 0x7f950c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f950c09c9c0 0x7f950c09d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52878/0 (socket says 172.21.15.153:52878) 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511338640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f950c09e340 0x7f950c12f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55478/0 (socket says 172.21.15.153:55478) 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 -- 172.21.15.153:0/1681605094 learned_addr learned my addr 172.21.15.153:0/1681605094 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): get_auth_request con 0x7f950c09c9c0 auth_method 0 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient(hunting): handle_auth_done global_id 84594 payload 274 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 -- 172.21.15.153:0/1681605094 >> v1:172.21.15.181:6789/0 conn(0x7f950c09e750 legacy=0x7f950c12fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 -- 172.21.15.153:0/1681605094 >> v1:172.21.15.153:6789/0 conn(0x7f950c09e340 legacy=0x7f950c12f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 monclient: found mon.noname-a 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.503+0000 7f9511b39640 1 -- 172.21.15.153:0/1681605094 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c09a440 con 0x7f950c09c9c0 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9511b39640 1 -- 172.21.15.153:0/1681605094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09c9c0 msgr2=0x7f950c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9511b39640 1 --2- 172.21.15.153:0/1681605094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09c9c0 0x7f950c09d830 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9504009910 tx=0x7f95040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcba240ace42daf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 1 -- 172.21.15.153:0/1681605094 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504032440 con 0x7f950c09c9c0 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: dump: 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9513d9d640 5 monclient: authenticate success, global_id 84594 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9513d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 1 -- 172.21.15.153:0/1681605094 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95040325a0 con 0x7f950c09c9c0 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9513d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9513d9d640 4 set_mon_vals no callback set 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 1 -- 172.21.15.153:0/1681605094 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504032940 con 0x7f950c09c9c0 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.506+0000 7f9510b37640 10 monclient: dump: 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 10 monclient: shutdown 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 -- 172.21.15.153:0/1681605094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09c9c0 msgr2=0x7f950c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 --2- 172.21.15.153:0/1681605094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09c9c0 0x7f950c09d830 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9504009910 tx=0x7f95040091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 -- 172.21.15.153:0/1681605094 shutdown_connections 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 --2- 172.21.15.153:0/1681605094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09c9c0 0x7f950c09d830 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 -- 172.21.15.153:0/1681605094 >> 172.21.15.153:0/1681605094 conn(0x7f950c0a2e40 msgr2=0x7f950c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 -- 172.21.15.153:0/1681605094 shutdown_connections 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 -- 172.21.15.153:0/1681605094 wait complete. 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 10 monclient: monmap: 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.521 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 Processor -- start 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 1 -- start start 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.507+0000 7f9513d9d640 10 monclient: init 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c09c9c0 0x7f950c11ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f950c09c9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09e340 0x7f950c11bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f950c09e340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f950c09e750 0x7f950c11c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f950c09e750 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c099cc0 con 0x7f950c09e340 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c098d80 con 0x7f950c09e750 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c12e700 con 0x7f950c09c9c0 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9513d9d640 10 monclient(hunting): authenticate will time out at 1142.147402s 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c09c9c0 0x7f950c11ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09e340 0x7f950c11bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c09c9c0 0x7f950c11ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52920/0 (socket says 172.21.15.153:52920) 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 1 -- 172.21.15.153:0/693738599 learned_addr learned my addr 172.21.15.153:0/693738599 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): get_auth_request con 0x7f950c09c9c0 auth_method 0 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): get_auth_request con 0x7f950c09e340 auth_method 0 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f951233a640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f950c09e750 0x7f950c11c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f951233a640 10 monclient(hunting): get_auth_request con 0x7f950c09e750 auth_method 0 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f951233a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f951233a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f951233a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient(hunting): handle_auth_done global_id 84597 payload 824 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c09c9c0 msgr2=0x7f950c11ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c09c9c0 0x7f950c11ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f950c09e750 msgr2=0x7f950c11c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f950c09e750 0x7f950c11c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 monclient: found mon.noname-a 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511338640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9504009450 con 0x7f950c09e340 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f951233a640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f950c09e750 0x7f950c11c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.508+0000 7f9511b39640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c09c9c0 0x7f950c11ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:44.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9511338640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09e340 msgr2=0x7f950c11bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9511338640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09e340 0x7f950c11bfa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f950000c490 tx=0x7f9500007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000074f0 con 0x7f950c09e340 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: dump: 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9500007670 con 0x7f950c09e340 2024-09-15T03:35:44.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 5 monclient: authenticate success, global_id 84597 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 10 monclient: _renew_subs 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c11f0d0 con 0x7f950c09e340 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500005a20 con 0x7f950c09e340 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9509ffb640 10 monclient: dump: 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f950bfff640 4 set_mon_vals no callback set 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 10 monclient: _renew_subs 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.511+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c11e170 con 0x7f950c09e340 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f950001b3b0 con 0x7f950c09e340 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9513d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9513d9d640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 0x7f950c06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9513d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c065e00 con 0x7f950c11ca20 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 0x7f950c06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient: get_auth_request con 0x7f950c11ca20 auth_method 0 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9509ffb640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f94f8038770 0x7f94f803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7f950004bb50 con 0x7f950c09e340 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 10 monclient(hunting): handle_auth_done global_id 74678 payload 824 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f9511b39640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f950c11f550 con 0x7f950c11ca20 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f951233a640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f94f8038770 0x7f94f803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.512+0000 7f951233a640 10 monclient: get_auth_request con 0x7f94f8038770 auth_method 0 2024-09-15T03:35:44.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.514+0000 7f9511b39640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 msgr2=0x7f950c06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.514+0000 7f9511b39640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 0x7f950c06d8c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9504009420 tx=0x7f9504032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.514+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950403c440 con 0x7f950c11ca20 2024-09-15T03:35:44.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.514+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.514+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:44.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.514+0000 7f9509ffb640 10 monclient: dump: 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f951233a640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f94f8038770 0x7f94f803ac20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94fc002ee0 tx=0x7f94fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f950403c5a0 con 0x7f950c11ca20 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 msgr2=0x7f950c06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 0x7f950c06d8c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9504009420 tx=0x7f9504032000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 msgr2=0x7f950c06d8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.515+0000 7f9509ffb640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c11ca20 0x7f950c06d8c0 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9504009420 tx=0x7f9504032000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9513d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9513d9d640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 0x7f950c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9513d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c11ff30 con 0x7f950c06de00 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9511b39640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 0x7f950c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9511b39640 10 monclient: get_auth_request con 0x7f950c06de00 auth_method 0 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9511b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9511b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.520+0000 7f9511b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.521+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.521+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.521+0000 7f9511b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.521+0000 7f9511b39640 10 monclient(hunting): handle_auth_done global_id 74681 payload 824 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.521+0000 7f9511b39640 1 -- 172.21.15.153:0/693738599 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f950c11d600 con 0x7f950c06de00 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9511b39640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 msgr2=0x7f950c06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9511b39640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 0x7f950c06e200 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9504003080 tx=0x7f950403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504057070 con 0x7f950c06de00 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 10 monclient: dump: 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f9504053430 con 0x7f950c06de00 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:44.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 msgr2=0x7f950c06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 0x7f950c06e200 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9504003080 tx=0x7f950403d000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 msgr2=0x7f950c06e200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.523+0000 7f9509ffb640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f950c06de00 0x7f950c06e200 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9504003080 tx=0x7f950403d000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 80, 2024-09-15T03:35:44.539 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:44.542 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:44.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.548 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:44.549 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.550 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.551 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.552 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:35:44.552 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.552 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.552 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:44.552 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:44.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f94f8038770 msgr2=0x7f94f803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f94f8038770 0x7f94f803ac20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94fc002ee0 tx=0x7f94fc009040 comp rx=0 tx=0).stop 2024-09-15T03:35:44.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 10 monclient: shutdown 2024-09-15T03:35:44.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09e340 msgr2=0x7f950c11bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 --2- 172.21.15.153:0/693738599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f950c09e340 0x7f950c11bfa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f950000c490 tx=0x7f9500007e40 comp rx=0 tx=0).stop 2024-09-15T03:35:44.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f951233a640 1 -- 172.21.15.153:0/693738599 reap_dead start 2024-09-15T03:35:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f951233a640 1 -- 172.21.15.153:0/693738599 reap_dead start 2024-09-15T03:35:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 shutdown_connections 2024-09-15T03:35:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 >> 172.21.15.153:0/693738599 conn(0x7f950c0a2e40 msgr2=0x7f950c097420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 shutdown_connections 2024-09-15T03:35:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.524+0000 7f9513d9d640 1 -- 172.21.15.153:0/693738599 wait complete. 2024-09-15T03:35:44.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:44.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:35:44.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:35:44.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.649+0000 7f627215d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:44.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.649+0000 7f627215d640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.649+0000 7f627215d640 10 monclient: monmap: 2024-09-15T03:35:44.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:44.650348+0000 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:44.650348+0000 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:44.651 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 Processor -- start 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 -- start start 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient: init 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): picked mon.noname-b con 0x7f626c067110 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f626c09b010 0x7f626c09b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): picked mon.noname-a con 0x7f626c09b010 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f626c09c820 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:44.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c130540 con 0x7f626c09c820 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c09b940 con 0x7f626c09b010 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c067ed0 con 0x7f626c067110 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f627215d640 10 monclient(hunting): authenticate will time out at 1142.289818s 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f626affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f626c09b010 0x7f626c09b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52934/0 (socket says 172.21.15.153:52934) 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.650+0000 7f626b7fe640 1 -- 172.21.15.153:0/690950605 learned_addr learned my addr 172.21.15.153:0/690950605 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): get_auth_request con 0x7f626c067110 auth_method 0 2024-09-15T03:35:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): get_auth_request con 0x7f626c09b010 auth_method 0 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient(hunting): handle_auth_done global_id 74684 payload 274 2024-09-15T03:35:44.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 1 -- 172.21.15.153:0/690950605 >> v2:172.21.15.153:3300/0 conn(0x7f626c09b010 msgr2=0x7f626c09b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 1 --2- 172.21.15.153:0/690950605 >> v2:172.21.15.153:3300/0 conn(0x7f626c09b010 0x7f626c09b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 1 -- 172.21.15.153:0/690950605 >> v1:172.21.15.153:6789/0 conn(0x7f626c09c820 legacy=0x7f626c09cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626b7fe640 1 -- 172.21.15.153:0/690950605 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c099800 con 0x7f626c067110 2024-09-15T03:35:44.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.651+0000 7f626affd640 1 --2- 172.21.15.153:0/690950605 >> v2:172.21.15.153:3300/0 conn(0x7f626c09b010 0x7f626c09b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:44.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626b7fe640 1 -- 172.21.15.153:0/690950605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 msgr2=0x7f626c12f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626b7fe640 1 --2- 172.21.15.153:0/690950605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12f670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f625c009910 tx=0x7f625c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cdac45769b6391a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 1 -- 172.21.15.153:0/690950605 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c02aea0 con 0x7f626c067110 2024-09-15T03:35:44.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: dump: 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 1 -- 172.21.15.153:0/690950605 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c032440 con 0x7f626c067110 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 5 monclient: authenticate success, global_id 74684 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 1 -- 172.21.15.153:0/690950605 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c0327e0 con 0x7f626c067110 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 4 set_mon_vals no callback set 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f626a7fc640 10 monclient: dump: 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.666 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 10 monclient: shutdown 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 -- 172.21.15.153:0/690950605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 msgr2=0x7f626c12f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 --2- 172.21.15.153:0/690950605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12f670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f625c009910 tx=0x7f625c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 -- 172.21.15.153:0/690950605 shutdown_connections 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 --2- 172.21.15.153:0/690950605 >> v2:172.21.15.153:3300/0 conn(0x7f626c09b010 0x7f626c09b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 --2- 172.21.15.153:0/690950605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12f670 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 -- 172.21.15.153:0/690950605 >> 172.21.15.153:0/690950605 conn(0x7f626c0a1e30 msgr2=0x7f626c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 -- 172.21.15.153:0/690950605 shutdown_connections 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.654+0000 7f627215d640 1 -- 172.21.15.153:0/690950605 wait complete. 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.655+0000 7f627215d640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.655+0000 7f627215d640 10 monclient: monmap: 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:44.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.655+0000 7f627215d640 1 Processor -- start 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.655+0000 7f627215d640 1 -- start start 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.655+0000 7f627215d640 10 monclient: init 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): picked mon.noname-c con 0x7f626c067110 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c09b010 0x7f626c13ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): picked mon.noname-b con 0x7f626c09b010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626c09c820 0x7f626c13f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): picked mon.noname-a con 0x7f626c09c820 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c11aeb0 con 0x7f626c09c820 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c11cc50 con 0x7f626c09b010 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c11c970 con 0x7f626c067110 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f627215d640 10 monclient(hunting): authenticate will time out at 1142.295403s 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626c09c820 0x7f626c13f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52950/0 (socket says 172.21.15.153:52950) 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 -- 172.21.15.153:0/4200102028 learned_addr learned my addr 172.21.15.153:0/4200102028 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): get_auth_request con 0x7f626c067110 auth_method 0 2024-09-15T03:35:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): get_auth_request con 0x7f626c09c820 auth_method 0 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c09b010 0x7f626c13ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): get_auth_request con 0x7f626c09b010 auth_method 0 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient(hunting): handle_auth_done global_id 74687 payload 824 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c09b010 msgr2=0x7f626c13ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c09b010 0x7f626c13ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626c09c820 msgr2=0x7f626c13f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626c09c820 0x7f626c13f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626b7fe640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260009390 con 0x7f626c067110 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.656+0000 7f626bfff640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626c09c820 0x7f626c13f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f626b7fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 msgr2=0x7f626c12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f626b7fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12e840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f625c035040 tx=0x7f625c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c02aea0 con 0x7f626c067110 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: dump: 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.673 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c000d10 con 0x7f626c067110 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c03d440 con 0x7f626c067110 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 5 monclient: authenticate success, global_id 74687 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 10 monclient: _renew_subs 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c009450 con 0x7f626c067110 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 10 monclient: dump: 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f6269ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 10 monclient: _renew_subs 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c12ed80 con 0x7f626c067110 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f625c039040 con 0x7f626c067110 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.659+0000 7f627215d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f627215d640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 0x7f626c119360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c065f90 con 0x7f626c11ce10 2024-09-15T03:35:44.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f62537fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6258038740 0x7f625803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 0x7f626c119360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7f625c06c0f0 con 0x7f626c067110 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient: get_auth_request con 0x7f626c11ce10 auth_method 0 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626bfff640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6258038740 0x7f625803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626bfff640 10 monclient: get_auth_request con 0x7f6258038740 auth_method 0 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 10 monclient(hunting): handle_auth_done global_id 104161 payload 824 2024-09-15T03:35:44.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.660+0000 7f626affd640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f626c11bd00 con 0x7f626c11ce10 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f626affd640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 msgr2=0x7f626c119360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f626affd640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 0x7f626c119360 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6260011bf0 tx=0x7f62600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f626bfff640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6258038740 0x7f625803abf0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6264007fd0 tx=0x7f62640025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626000fe90 con 0x7f626c11ce10 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f62537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f62537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.662+0000 7f62537fe640 10 monclient: dump: 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.678 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6260014650 con 0x7f626c11ce10 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 msgr2=0x7f626c119360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 0x7f626c119360 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6260011bf0 tx=0x7f62600090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 msgr2=0x7f626c119360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:44.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.663+0000 7f62537fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c11ce10 0x7f626c119360 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6260011bf0 tx=0x7f62600090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.668+0000 7f627215d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:44.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.668+0000 7f627215d640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 0x7f626c119ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.668+0000 7f627215d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.668+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c11a1e0 con 0x7f626c1198a0 2024-09-15T03:35:44.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 0x7f626c119ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient: get_auth_request con 0x7f626c1198a0 auth_method 0 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 10 monclient(hunting): handle_auth_done global_id 104164 payload 824 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.669+0000 7f626affd640 1 -- 172.21.15.153:0/4200102028 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f626c04bf60 con 0x7f626c1198a0 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.671+0000 7f626affd640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 msgr2=0x7f626c119ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.671+0000 7f626affd640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 0x7f626c119ca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6260011bf0 tx=0x7f626001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6260025040 con 0x7f626c1198a0 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 10 monclient: dump: 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f626001c3e0 con 0x7f626c1198a0 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:44.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 msgr2=0x7f626c119ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 0x7f626c119ca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6260011bf0 tx=0x7f626001d000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 msgr2=0x7f626c119ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f62537fe640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f626c1198a0 0x7f626c119ca0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6260011bf0 tx=0x7f626001d000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:44.684 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 80, 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.685 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.686 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:44.687 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:44.688 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.689 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.691 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.692 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.694 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:44.696 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:35:44.696 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:44.696 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:44.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:44.696 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:44.696 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6258038740 msgr2=0x7f625803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6258038740 0x7f625803abf0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6264007fd0 tx=0x7f62640025b0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 10 monclient: shutdown 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 msgr2=0x7f626c12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 --2- 172.21.15.153:0/4200102028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f626c067110 0x7f626c12e840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f625c035040 tx=0x7f625c032000 comp rx=0 tx=0).stop 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f626bfff640 1 -- 172.21.15.153:0/4200102028 reap_dead start 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f626bfff640 1 -- 172.21.15.153:0/4200102028 reap_dead start 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 shutdown_connections 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 >> 172.21.15.153:0/4200102028 conn(0x7f626c0a1e30 msgr2=0x7f626c09cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.672+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 shutdown_connections 2024-09-15T03:35:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.673+0000 7f627215d640 1 -- 172.21.15.153:0/4200102028 wait complete. 2024-09-15T03:35:44.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:35:44.699 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:35:44.699 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:35:44.699 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:35:44.699 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:35:44.699 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:35:44.699 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:35:44.711 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:35:44.711 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:35:44.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.844+0000 7fab6c420640 10 monclient: get_monmap_and_config 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.844+0000 7fab6c420640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.844+0000 7fab6c420640 10 monclient: monmap: 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:44.845162+0000 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:44.845162+0000 2024-09-15T03:35:44.845 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:44.846 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.844+0000 7fab6c420640 1 Processor -- start 2024-09-15T03:35:44.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 1 -- start start 2024-09-15T03:35:44.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient: init 2024-09-15T03:35:44.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab6409ad90 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fab6409b160 0x7fab6409b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): picked mon.noname-c con 0x7fab6409b160 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fab64136c90 0x7fab64139070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): picked mon.noname-a con 0x7fab64136c90 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab640692c0 con 0x7fab6409ad90 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6409cce0 con 0x7fab64136c90 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6409db70 con 0x7fab6409b160 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6a1bc640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fab6409ad90 legacy=0x7fab6409c5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6c420640 10 monclient(hunting): authenticate will time out at 1142.484653s 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6a9bd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fab64136c90 msgr2=0x7fab64139070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab6a9bd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fab64136c90 0x7fab64139070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fab6409b160 0x7fab6409b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fab6409b160 0x7fab6409b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:39028/0 (socket says 172.21.15.153:39028) 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 1 -- 172.21.15.153:0/3619175248 learned_addr learned my addr 172.21.15.153:0/3619175248 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 10 monclient(hunting): get_auth_request con 0x7fab6409b160 auth_method 0 2024-09-15T03:35:44.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.845+0000 7fab699bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 10 monclient(hunting): handle_auth_done global_id 104167 payload 274 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 1 -- 172.21.15.153:0/3619175248 >> v2:172.21.15.153:3300/0 conn(0x7fab64136c90 msgr2=0x7fab64139070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 1 --2- 172.21.15.153:0/3619175248 >> v2:172.21.15.153:3300/0 conn(0x7fab64136c90 0x7fab64139070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 1 -- 172.21.15.153:0/3619175248 >> v1:172.21.15.153:6789/0 conn(0x7fab6409ad90 legacy=0x7fab6409c5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 1 monclient: found mon.noname-c 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:35:44.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.846+0000 7fab699bb640 1 -- 172.21.15.153:0/3619175248 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6409d5b0 con 0x7fab6409b160 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab699bb640 1 -- 172.21.15.153:0/3619175248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab6409b160 msgr2=0x7fab6409b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab699bb640 1 --2- 172.21.15.153:0/3619175248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab6409b160 0x7fab6409b530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab6406ed00 tx=0x7fab58009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc9376e6787c2758 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 1 -- 172.21.15.153:0/3619175248 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5802aea0 con 0x7fab6409b160 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: dump: 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 1 -- 172.21.15.153:0/3619175248 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5802d050 con 0x7fab6409b160 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 5 monclient: authenticate success, global_id 104167 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 4 set_mon_vals no callback set 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 1 -- 172.21.15.153:0/3619175248 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58007820 con 0x7fab6409b160 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab691ba640 10 monclient: dump: 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.859 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 10 monclient: shutdown 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 1 -- 172.21.15.153:0/3619175248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab6409b160 msgr2=0x7fab6409b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 1 --2- 172.21.15.153:0/3619175248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab6409b160 0x7fab6409b530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab6406ed00 tx=0x7fab58009840 comp rx=0 tx=0).stop 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 1 -- 172.21.15.153:0/3619175248 shutdown_connections 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 1 --2- 172.21.15.153:0/3619175248 >> v2:172.21.15.153:3300/0 conn(0x7fab64136c90 0x7fab64139070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 1 --2- 172.21.15.153:0/3619175248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab6409b160 0x7fab6409b530 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.849+0000 7fab6c420640 1 -- 172.21.15.153:0/3619175248 >> 172.21.15.153:0/3619175248 conn(0x7fab640678a0 msgr2=0x7fab640a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 1 -- 172.21.15.153:0/3619175248 shutdown_connections 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 1 -- 172.21.15.153:0/3619175248 wait complete. 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 10 monclient: build_initial_monmap 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 10 monclient: monmap: 2024-09-15T03:35:44.860 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 1 Processor -- start 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 1 -- start start 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.850+0000 7fab6c420640 10 monclient: init 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 0x7fab641359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): picked mon.noname-c con 0x7fab6409ad90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab6409b160 0x7fab64135f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): picked mon.noname-a con 0x7fab6409b160 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab64136c90 0x7fab6408bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): picked mon.noname-b con 0x7fab64136c90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6408dbb0 con 0x7fab6409b160 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6408f8c0 con 0x7fab64136c90 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6408f5e0 con 0x7fab6409ad90 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6c420640 10 monclient(hunting): authenticate will time out at 1142.490268s 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab699bb640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab6409b160 msgr2=0x7fab64135f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab699bb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab6409b160 0x7fab64135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 0x7fab641359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 0x7fab641359e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:52968/0 (socket says 172.21.15.153:52968) 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 -- 172.21.15.153:0/120456729 learned_addr learned my addr 172.21.15.153:0/120456729 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): get_auth_request con 0x7fab6409ad90 auth_method 0 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a9bd640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab64136c90 0x7fab6408bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a9bd640 10 monclient(hunting): get_auth_request con 0x7fab64136c90 auth_method 0 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a9bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a9bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a9bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient(hunting): handle_auth_done global_id 74690 payload 824 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 -- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab64136c90 msgr2=0x7fab6408bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab64136c90 0x7fab6408bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 -- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab6409b160 msgr2=0x7fab64135f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:44.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab6409b160 0x7fab64135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient: _finish_hunting 0 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 monclient: found mon.noname-c 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.851+0000 7fab6a1bc640 1 -- 172.21.15.153:0/120456729 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58009ae0 con 0x7fab6409ad90 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.853+0000 7fab6a1bc640 1 -- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 msgr2=0x7fab641359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.853+0000 7fab6a1bc640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 0x7fab641359e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00b630 tx=0x7fab5c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.853+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c015070 con 0x7fab6409ad90 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.853+0000 7fab627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: dump: 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 5 monclient: authenticate success, global_id 74690 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c003d90 con 0x7fab6409ad90 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c006380 con 0x7fab6409ad90 2024-09-15T03:35:44.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 monclient: _renew_subs 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6408fba0 con 0x7fab6409ad90 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab689b9640 4 set_mon_vals no callback set 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 10 monclient: dump: 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:44.867 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 monclient: _renew_subs 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab64136600 con 0x7fab6409ad90 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fab5c010440 con 0x7fab6409ad90 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.854+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6404bf60 con 0x7fab6409ad90 2024-09-15T03:35:44.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.855+0000 7fab627fc640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab500386f0 0x7fab5003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:44.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.855+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7fab5c04c120 con 0x7fab6409ad90 2024-09-15T03:35:44.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.855+0000 7fab699bb640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab500386f0 0x7fab5003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:44.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.855+0000 7fab699bb640 10 monclient: get_auth_request con 0x7fab500386f0 auth_method 0 2024-09-15T03:35:44.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.857+0000 7fab699bb640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab500386f0 0x7fab5003aba0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fab58033000 tx=0x7fab58036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:44.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.859+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fab5c018a70 con 0x7fab6409ad90 2024-09-15T03:35:44.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.859+0000 7fab627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:44.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:44.859+0000 7fab627fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab6c420640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab6c420640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab6c420640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab640638d0 con 0x7fab6409ad90 2024-09-15T03:35:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab627fc640 1 -- 172.21.15.153:0/120456729 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fab5c018a70 con 0x7fab6409ad90 2024-09-15T03:35:45.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:45.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.072+0000 7fab627fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:45.074 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:45.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab500386f0 msgr2=0x7fab5003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab500386f0 0x7fab5003aba0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fab58033000 tx=0x7fab58036000 comp rx=0 tx=0).stop 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 10 monclient: shutdown 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 msgr2=0x7fab641359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 0x7fab641359e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00b630 tx=0x7fab5c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 shutdown_connections 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab500386f0 0x7fab5003aba0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab64136c90 0x7fab6408bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab6409b160 0x7fab64135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 --2- 172.21.15.153:0/120456729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab6409ad90 0x7fab641359e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 >> 172.21.15.153:0/120456729 conn(0x7fab640678a0 msgr2=0x7fab640942c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:45.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 shutdown_connections 2024-09-15T03:35:45.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:45.074+0000 7fab6c420640 1 -- 172.21.15.153:0/120456729 wait complete. 2024-09-15T03:35:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:47.330+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:47.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:47.330+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:47.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:47.330+0000 7f01c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:48.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.190+0000 7f69b555d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.190+0000 7f69b555d640 10 monclient: build_initial_monmap 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.191+0000 7f69b555d640 10 monclient: monmap: 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:48.191734+0000 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:48.191734+0000 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:48.193 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:48.193 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:48.193 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:48.193 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:48.193 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:48.193 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.191+0000 7f69b555d640 1 Processor -- start 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.191+0000 7f69b555d640 1 -- start start 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.191+0000 7f69b555d640 10 monclient: init 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69b00974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f69b009a8a0 0x7f69b009ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): picked mon.noname-a con 0x7f69b009a8a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 0x7f69b0135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): picked mon.noname-b con 0x7f69b0133070 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b012b210 con 0x7f69b00974f0 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b00978c0 con 0x7f69b009a8a0 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b009b220 con 0x7f69b0133070 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69b555d640 10 monclient(hunting): authenticate will time out at 1145.831183s 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69aeffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f69b00974f0 legacy=0x7f69b012ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69ae7fc640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f69b009a8a0 msgr2=0x7f69b009ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69ae7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f69b009a8a0 0x7f69b009ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 0x7f69b0135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 0x7f69b0135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55214/0 (socket says 172.21.15.153:55214) 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 -- 172.21.15.153:0/3207768 learned_addr learned my addr 172.21.15.153:0/3207768 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): get_auth_request con 0x7f69b0133070 auth_method 0 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:48.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient(hunting): handle_auth_done global_id 74693 payload 274 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 -- 172.21.15.153:0/3207768 >> v2:172.21.15.153:3300/0 conn(0x7f69b009a8a0 msgr2=0x7f69b009ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 --2- 172.21.15.153:0/3207768 >> v2:172.21.15.153:3300/0 conn(0x7f69b009a8a0 0x7f69b009ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 -- 172.21.15.153:0/3207768 >> v1:172.21.15.153:6789/0 conn(0x7f69b00974f0 legacy=0x7f69b012ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 monclient: found mon.noname-b 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.192+0000 7f69af7fe640 1 -- 172.21.15.153:0/3207768 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b009ca20 con 0x7f69b0133070 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69af7fe640 1 -- 172.21.15.153:0/3207768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 msgr2=0x7f69b0135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69af7fe640 1 --2- 172.21.15.153:0/3207768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 0x7f69b0135450 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009910 tx=0x7f69a80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31fe13a5778014eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 1 -- 172.21.15.153:0/3207768 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8032440 con 0x7f69b0133070 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:48.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: dump: 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 1 -- 172.21.15.153:0/3207768 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a80325a0 con 0x7f69b0133070 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69b555d640 5 monclient: authenticate success, global_id 74693 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69b555d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69b555d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 1 -- 172.21.15.153:0/3207768 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8032940 con 0x7f69b0133070 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69b555d640 4 set_mon_vals no callback set 2024-09-15T03:35:48.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.195+0000 7f69adffb640 10 monclient: dump: 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 10 monclient: shutdown 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 -- 172.21.15.153:0/3207768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 msgr2=0x7f69b0135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 --2- 172.21.15.153:0/3207768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 0x7f69b0135450 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009910 tx=0x7f69a80091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 -- 172.21.15.153:0/3207768 shutdown_connections 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 --2- 172.21.15.153:0/3207768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b0133070 0x7f69b0135450 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 --2- 172.21.15.153:0/3207768 >> v2:172.21.15.153:3300/0 conn(0x7f69b009a8a0 0x7f69b009ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 -- 172.21.15.153:0/3207768 >> 172.21.15.153:0/3207768 conn(0x7f69b00672b0 msgr2=0x7f69b00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 -- 172.21.15.153:0/3207768 shutdown_connections 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 1 -- 172.21.15.153:0/3207768 wait complete. 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 10 monclient: build_initial_monmap 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.196+0000 7f69b555d640 10 monclient: monmap: 2024-09-15T03:35:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 Processor -- start 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 -- start start 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient: init 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 0x7f69b006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): picked mon.noname-c con 0x7f69b00974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b009a8a0 0x7f69b006e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): picked mon.noname-a con 0x7f69b009a8a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69b0133070 0x7f69b0069c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): picked mon.noname-b con 0x7f69b0133070 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b006e740 con 0x7f69b009a8a0 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b006d860 con 0x7f69b0133070 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b009a3b0 con 0x7f69b00974f0 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69b555d640 10 monclient(hunting): authenticate will time out at 1145.836755s 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69ae7fc640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b009a8a0 msgr2=0x7f69b006e200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69ae7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b009a8a0 0x7f69b006e200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 0x7f69b006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 0x7f69b006f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55226/0 (socket says 172.21.15.153:55226) 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 1 -- 172.21.15.153:0/3758735758 learned_addr learned my addr 172.21.15.153:0/3758735758 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): get_auth_request con 0x7f69b00974f0 auth_method 0 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.197+0000 7f69aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 10 monclient(hunting): handle_auth_done global_id 74696 payload 824 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 1 -- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69b0133070 msgr2=0x7f69b0069c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69b0133070 0x7f69b0069c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 1 -- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b009a8a0 msgr2=0x7f69b006e200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b009a8a0 0x7f69b006e200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 1 monclient: found mon.noname-c 2024-09-15T03:35:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.198+0000 7f69aeffd640 1 -- 172.21.15.153:0/3758735758 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009450 con 0x7f69b00974f0 2024-09-15T03:35:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69aeffd640 1 -- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 msgr2=0x7f69b006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69aeffd640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 0x7f69b006f1e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f69a000bb80 tx=0x7f69a00097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000cc80 con 0x7f69b00974f0 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: dump: 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a000cde0 con 0x7f69b00974f0 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 5 monclient: authenticate success, global_id 74696 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 10 monclient: _renew_subs 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b006c380 con 0x7f69b00974f0 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0016430 con 0x7f69b00974f0 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f6996ffd640 10 monclient: dump: 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 10 monclient: _renew_subs 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b006b750 con 0x7f69b00974f0 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.200+0000 7f69ad7fa640 4 set_mon_vals no callback set 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f69a0012020 con 0x7f69b00974f0 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f6996ffd640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f699c0386f0 0x7f699c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f69b555d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7f69a004c660 con 0x7f69b00974f0 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f69b555d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f69b555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b006bf00 con 0x7f69b00974f0 2024-09-15T03:35:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f69ae7fc640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f699c0386f0 0x7f699c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.201+0000 7f69ae7fc640 10 monclient: get_auth_request con 0x7f699c0386f0 auth_method 0 2024-09-15T03:35:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.204+0000 7f69ae7fc640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f699c0386f0 0x7f699c03aba0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f69a4006fd0 tx=0x7f69a4006d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.206+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f69a0006150 con 0x7f69b00974f0 2024-09-15T03:35:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.206+0000 7f6996ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.206+0000 7f6996ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:48.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:48.331+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:48.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:48.331+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:48.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.421+0000 7f69b555d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:48.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.421+0000 7f69b555d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:48.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.421+0000 7f69b555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:48.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.421+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f69b004bf60 con 0x7f69b00974f0 2024-09-15T03:35:48.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.422+0000 7f6996ffd640 1 -- 172.21.15.153:0/3758735758 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f69a0024080 con 0x7f69b00974f0 2024-09-15T03:35:48.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.422+0000 7f6996ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:48.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.422+0000 7f6996ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:48.423 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f699c0386f0 msgr2=0x7f699c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f699c0386f0 0x7f699c03aba0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f69a4006fd0 tx=0x7f69a4006d30 comp rx=0 tx=0).stop 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 10 monclient: shutdown 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 msgr2=0x7f69b006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 0x7f69b006f1e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f69a000bb80 tx=0x7f69a00097f0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 shutdown_connections 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f699c0386f0 0x7f699c03aba0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69b0133070 0x7f69b0069c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b009a8a0 0x7f69b006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 --2- 172.21.15.153:0/3758735758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69b00974f0 0x7f69b006f1e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:48.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 >> 172.21.15.153:0/3758735758 conn(0x7f69b00672b0 msgr2=0x7f69b0129540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:48.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 shutdown_connections 2024-09-15T03:35:48.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:48.424+0000 7f69b555d640 1 -- 172.21.15.153:0/3758735758 wait complete. 2024-09-15T03:35:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:49.331+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:49.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:49.331+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:50.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:50.330+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:50.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:50.330+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:51.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:51.331+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:51.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:51.331+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:51.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:51.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.539+0000 7fd1a8b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:51.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.539+0000 7fd1a8b9d640 10 monclient: build_initial_monmap 2024-09-15T03:35:51.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.539+0000 7fd1a8b9d640 10 monclient: monmap: 2024-09-15T03:35:51.540 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:51.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:51.540368+0000 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:51.540368+0000 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:51.541 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:51.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 1 Processor -- start 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 1 -- start start 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient: init 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1a4096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1a40971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 0x7fd1a4136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a4134b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a409d4a0 con 0x7fd1a4096e10 2024-09-15T03:35:51.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a4099620 con 0x7fd1a40971e0 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a4099340 con 0x7fd1a4134b40 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a8b9d640 10 monclient(hunting): authenticate will time out at 1149.179968s 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.540+0000 7fd1a259c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fd1a4096e10 legacy=0x7fd1a409a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 0x7fd1a4136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 0x7fd1a4136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55236/0 (socket says 172.21.15.153:55236) 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 -- 172.21.15.153:0/4280150064 learned_addr learned my addr 172.21.15.153:0/4280150064 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): get_auth_request con 0x7fd1a4134b40 auth_method 0 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:51.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient(hunting): handle_auth_done global_id 74699 payload 274 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 -- 172.21.15.153:0/4280150064 >> v1:172.21.15.181:6789/0 conn(0x7fd1a40971e0 legacy=0x7fd1a40975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 -- 172.21.15.153:0/4280150064 >> v1:172.21.15.153:6789/0 conn(0x7fd1a4096e10 legacy=0x7fd1a409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 monclient: found mon.noname-b 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:51.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.541+0000 7fd1a2d9d640 1 -- 172.21.15.153:0/4280150064 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a4099900 con 0x7fd1a4134b40 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a2d9d640 1 -- 172.21.15.153:0/4280150064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 msgr2=0x7fd1a4136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a2d9d640 1 --2- 172.21.15.153:0/4280150064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 0x7fd1a4136f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd1a409b020 tx=0x7fd19c00ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e989f76b1b8c0e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 1 -- 172.21.15.153:0/4280150064 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c009610 con 0x7fd1a4134b40 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: dump: 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.552 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 5 monclient: authenticate success, global_id 74699 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 1 -- 172.21.15.153:0/4280150064 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c009770 con 0x7fd1a4134b40 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 1 -- 172.21.15.153:0/4280150064 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c009b10 con 0x7fd1a4134b40 2024-09-15T03:35:51.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 4 set_mon_vals no callback set 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a1d9b640 10 monclient: dump: 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:51.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 10 monclient: shutdown 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/4280150064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 msgr2=0x7fd1a4136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/4280150064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 0x7fd1a4136f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd1a409b020 tx=0x7fd19c00ba50 comp rx=0 tx=0).stop 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/4280150064 shutdown_connections 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/4280150064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4134b40 0x7fd1a4136f20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/4280150064 >> 172.21.15.153:0/4280150064 conn(0x7fd1a40672b0 msgr2=0x7fd1a40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.544+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/4280150064 shutdown_connections 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.545+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/4280150064 wait complete. 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.545+0000 7fd1a8b9d640 10 monclient: build_initial_monmap 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.545+0000 7fd1a8b9d640 10 monclient: monmap: 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:51.555 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.545+0000 7fd1a8b9d640 1 Processor -- start 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.545+0000 7fd1a8b9d640 1 -- start start 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.545+0000 7fd1a8b9d640 10 monclient: init 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:51.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 0x7fd1a413d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a4096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a40971e0 0x7fd1a406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a40971e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a4134b40 0x7fd1a406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a4134b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a4118e40 con 0x7fd1a4134b40 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a4134250 con 0x7fd1a40971e0 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a409b050 con 0x7fd1a4096e10 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a8b9d640 10 monclient(hunting): authenticate will time out at 1149.185281s 2024-09-15T03:35:51.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a2d9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a4134b40 msgr2=0x7fd1a406d930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 0x7fd1a413d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a4134b40 0x7fd1a406d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 0x7fd1a413d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55252/0 (socket says 172.21.15.153:55252) 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 -- 172.21.15.153:0/3387502181 learned_addr learned my addr 172.21.15.153:0/3387502181 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): get_auth_request con 0x7fd1a4096e10 auth_method 0 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd19bfff640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a40971e0 0x7fd1a406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:51.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient(hunting): handle_auth_done global_id 74702 payload 824 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 -- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a40971e0 msgr2=0x7fd1a406d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a40971e0 0x7fd1a406d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 -- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a4134b40 msgr2=0x7fd1a406d930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a4134b40 0x7fd1a406d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient: _finish_hunting 0 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 monclient: found mon.noname-c 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd1a259c640 1 -- 172.21.15.153:0/3387502181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c00bcf0 con 0x7fd1a4096e10 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd19bfff640 10 monclient: get_auth_request con 0x7fd1a40971e0 auth_method 0 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.546+0000 7fd19bfff640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a40971e0 0x7fd1a406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a259c640 1 -- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 msgr2=0x7fd1a413d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:51.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a259c640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 0x7fd1a413d610 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd19400b5f0 tx=0x7fd194009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194015070 con 0x7fd1a4096e10 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: dump: 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:51.560 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd194003d90 con 0x7fd1a4096e10 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a8b9d640 5 monclient: authenticate success, global_id 74702 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a8b9d640 10 monclient: _renew_subs 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a8b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a406efe0 con 0x7fd1a4096e10 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1940063a0 con 0x7fd1a4096e10 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd19a7fc640 10 monclient: dump: 2024-09-15T03:35:51.561 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.548+0000 7fd1a159a640 4 set_mon_vals no callback set 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 10 monclient: _renew_subs 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a412ed00 con 0x7fd1a4096e10 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd194010440 con 0x7fd1a4096e10 2024-09-15T03:35:51.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.549+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a404bf60 con 0x7fd1a4096e10 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.550+0000 7fd19a7fc640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd184038720 0x7fd18403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.550+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7fd19404c200 con 0x7fd1a4096e10 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.550+0000 7fd19bfff640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd184038720 0x7fd18403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.550+0000 7fd19bfff640 10 monclient: get_auth_request con 0x7fd184038720 auth_method 0 2024-09-15T03:35:51.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.552+0000 7fd19bfff640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd184038720 0x7fd18403abd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd1a406eaf0 tx=0x7fd18c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:51.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.554+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd1940067d0 con 0x7fd1a4096e10 2024-09-15T03:35:51.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.554+0000 7fd19a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:51.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.554+0000 7fd19a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:51.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.766+0000 7fd1a8b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:51.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.766+0000 7fd1a8b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:51.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.766+0000 7fd1a8b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:51.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.766+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd1a40a07e0 con 0x7fd1a4096e10 2024-09-15T03:35:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.767+0000 7fd19a7fc640 1 -- 172.21.15.153:0/3387502181 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd194019370 con 0x7fd1a4096e10 2024-09-15T03:35:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.767+0000 7fd19a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:51.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.767+0000 7fd19a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:51.768 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd184038720 msgr2=0x7fd18403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd184038720 0x7fd18403abd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd1a406eaf0 tx=0x7fd18c009040 comp rx=0 tx=0).stop 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 10 monclient: shutdown 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 msgr2=0x7fd1a413d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 0x7fd1a413d610 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd19400b5f0 tx=0x7fd194009120 comp rx=0 tx=0).stop 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 shutdown_connections 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd184038720 0x7fd18403abd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a4134b40 0x7fd1a406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a40971e0 0x7fd1a406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 --2- 172.21.15.153:0/3387502181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a4096e10 0x7fd1a413d610 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:51.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 >> 172.21.15.153:0/3387502181 conn(0x7fd1a40672b0 msgr2=0x7fd1a4132000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:51.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 shutdown_connections 2024-09-15T03:35:51.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:51.769+0000 7fd1a8b9d640 1 -- 172.21.15.153:0/3387502181 wait complete. 2024-09-15T03:35:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:52.331+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:52.331+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:53.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:53.331+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:53.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:53.331+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:54.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:54.331+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:54.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:54.331+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:54.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.885+0000 7fcb15caf640 10 monclient: get_monmap_and_config 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.885+0000 7fcb15caf640 10 monclient: build_initial_monmap 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.885+0000 7fcb15caf640 10 monclient: monmap: 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:54.886050+0000 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:54.886050+0000 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:54.887 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:54.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 Processor -- start 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 -- start start 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient: init 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 0x7fcb10131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): picked mon.noname-a con 0x7fcb10131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): picked mon.noname-c con 0x7fcb1009e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): picked mon.noname-b con 0x7fcb1009f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1009faf0 con 0x7fcb10131ab0 2024-09-15T03:35:54.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1009fd30 con 0x7fcb1009e910 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100a1110 con 0x7fcb1009f250 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb15caf640 10 monclient(hunting): authenticate will time out at 1152.525674s 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0f7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 msgr2=0x7fcb10131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 0x7fcb10131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55266/0 (socket says 172.21.15.153:55266) 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 learned_addr learned my addr 172.21.15.153:0/2311985715 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request con 0x7fcb1009f250 auth_method 0 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.886+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request con 0x7fcb1009e910 auth_method 0 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:54.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:54.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:54.887+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:55.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.087+0000 7fcb0f7fe640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 msgr2=0x7fcb10131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:55.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.087+0000 7fcb0f7fe640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 0x7fcb10131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:55.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request con 0x7fcb1009f250 auth_method 0 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:55.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request con 0x7fcb1009e910 auth_method 0 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:55.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.088+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:35:55.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:55.332+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:55.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:55.332+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.488+0000 7fcb0f7fe640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 msgr2=0x7fcb10131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.488+0000 7fcb0f7fe640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 0x7fcb10131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request con 0x7fcb1009f250 auth_method 0 2024-09-15T03:35:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request con 0x7fcb1009e910 auth_method 0 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:55.489+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:35:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.289+0000 7fcb0f7fe640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 msgr2=0x7fcb10131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.289+0000 7fcb0f7fe640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 0x7fcb10131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request con 0x7fcb1009e910 auth_method 0 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request con 0x7fcb1009f250 auth_method 0 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:35:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:35:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:35:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:35:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:35:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:56.290+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:35:56.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:56.332+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:56.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:56.332+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:57.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:57.332+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:57.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:57.332+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:57.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:57.332+0000 7f01c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:57.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 10 monclient(hunting): tick 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 monclient(hunting): continuing hunt 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 msgr2=0x7fcb10068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb1009f250 0x7fcb10068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 msgr2=0x7fcb1009ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.181:3300/0 conn(0x7fcb1009e910 0x7fcb1009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 -- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 msgr2=0x7fcb10131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 --2- 172.21.15.153:0/2311985715 >> v2:172.21.15.153:3300/0 conn(0x7fcb10131ab0 0x7fcb10131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 10 monclient: _add_conns ranks=[1,0] 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcaf0003670 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:57.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcaf00072d0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:57.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 -- 172.21.15.153:0/2311985715 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100a13f0 con 0x7fcaf00072d0 2024-09-15T03:35:57.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 1 -- 172.21.15.153:0/2311985715 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf000c040 con 0x7fcaf0003670 2024-09-15T03:35:57.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb0dffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:57.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.887+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v1:172.21.15.153:6789/0 conn(0x7fcaf00072d0 legacy=0x7fcaf0009780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:57.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:57.889+0000 7fcb0f7fe640 1 -- 172.21.15.153:0/2311985715 >> v1:172.21.15.181:6789/0 conn(0x7fcaf0003670 legacy=0x7fcaf0005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:58.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:58.088+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v1:172.21.15.153:6789/0 conn(0x7fcaf00072d0 legacy=0x7fcaf0009780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:58.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:58.332+0000 7f01c7fff640 10 monclient: tick 2024-09-15T03:35:58.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:58.332+0000 7f01c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:58.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:58.489+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> v1:172.21.15.153:6789/0 conn(0x7fcaf00072d0 legacy=0x7fcaf0009780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:59.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.149+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021730911 0 0) 0x7fcaf000c040 con 0x7fcaf0003670 2024-09-15T03:35:59.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.149+0000 7fcb0effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.149+0000 7fcb0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.149+0000 7fcb0effd640 10 monclient(hunting): my global_id is 104173 2024-09-15T03:35:59.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.149+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcafc0034e0 con 0x7fcaf0003670 2024-09-15T03:35:59.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1795610449 0 0) 0x7fcafc0034e0 con 0x7fcaf0003670 2024-09-15T03:35:59.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 >> v1:172.21.15.153:6789/0 conn(0x7fcaf00072d0 legacy=0x7fcaf0009780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:35:59.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf000d240 con 0x7fcaf0003670 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fcb00004280 con 0x7fcaf0003670 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb15caf640 5 monclient: authenticate success, global_id 104173 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb15caf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: dump: 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 >> v1:172.21.15.181:6789/0 conn(0x7fcaf0003670 legacy=0x7fcaf0005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0ffff640 1 -- 172.21.15.153:0/2311985715 reap_dead start 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcafc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): picked mon.a con 0x7fcafc0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcafc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): picked mon.b con 0x7fcafc004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcafc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0f7fe640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 msgr2=0x7fcafc003da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): picked mon.c con 0x7fcafc005250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc00c1a0 con 0x7fcafc0039d0 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc00c310 con 0x7fcafc004780 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0f7fe640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcafc003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc00c4d0 con 0x7fcafc005250 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb0effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.151+0000 7fcb15caf640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcafc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcafc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request con 0x7fcafc005250 auth_method 0 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request con 0x7fcafc004780 auth_method 0 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient(hunting): handle_auth_done global_id 104173 payload 293 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 msgr2=0x7fcafc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcafc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 msgr2=0x7fcafc003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcafc003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 monclient: found mon.b 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc0034e0 con 0x7fcafc004780 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.152+0000 7fcb06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb06ffd640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 msgr2=0x7fcafc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcafc004bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006ea0 tx=0x7fcaf8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00d9860d8eb7769 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf802b850 con 0x7fcafc004780 2024-09-15T03:35:59.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: dump: 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb15caf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf802f070 con 0x7fcafc004780 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb15caf640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 1 -- 172.21.15.153:0/2311985715 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf802bc20 con 0x7fcafc004780 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb15caf640 4 set_mon_vals no callback set 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.155+0000 7fcb0effd640 10 monclient: dump: 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.214 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 10 monclient: shutdown 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 -- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 msgr2=0x7fcafc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcafc004bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006ea0 tx=0x7fcaf8008040 comp rx=0 tx=0).stop 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 -- 172.21.15.153:0/2311985715 shutdown_connections 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcafc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcafc004bf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 --2- 172.21.15.153:0/2311985715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcafc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 -- 172.21.15.153:0/2311985715 >> 172.21.15.153:0/2311985715 conn(0x7fcb10067d30 msgr2=0x7fcb100a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 -- 172.21.15.153:0/2311985715 shutdown_connections 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.156+0000 7fcb15caf640 1 -- 172.21.15.153:0/2311985715 wait complete. 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.157+0000 7fcb15caf640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.157+0000 7fcb15caf640 10 monclient: monmap: 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:59.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.157+0000 7fcb15caf640 1 Processor -- start 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 -- start start 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient: init 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcb1009a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): picked mon.noname-a con 0x7fcafc0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcb10130950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): picked mon.noname-b con 0x7fcafc004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcb1011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): picked mon.noname-c con 0x7fcafc005250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1009b3f0 con 0x7fcafc0039d0 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb10134130 con 0x7fcafc004780 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb10141470 con 0x7fcafc005250 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb15caf640 10 monclient(hunting): authenticate will time out at 1156.797556s 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0f7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 msgr2=0x7fcb1009a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcb1009a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcb1011b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcb1011b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49654/0 (socket says 172.21.15.153:49654) 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 1 -- 172.21.15.153:0/1725829558 learned_addr learned my addr 172.21.15.153:0/1725829558 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request con 0x7fcafc005250 auth_method 0 2024-09-15T03:35:59.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcb10130950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request con 0x7fcafc004780 auth_method 0 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.158+0000 7fcb06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 10 monclient(hunting): handle_auth_done global_id 74705 payload 824 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 1 -- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 msgr2=0x7fcb10130950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcb10130950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 1 -- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 msgr2=0x7fcb1009a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcb1009a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 1 monclient: found mon.noname-c 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb0ffff640 1 -- 172.21.15.153:0/1725829558 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1011da70 con 0x7fcafc005250 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.159+0000 7fcb06ffd640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcb10130950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb0ffff640 1 -- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 msgr2=0x7fcb1011b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb0ffff640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcb1011b6b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007240 tx=0x7fcb0800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08018660 con 0x7fcafc005250 2024-09-15T03:35:59.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: dump: 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0800f040 con 0x7fcafc005250 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 5 monclient: authenticate success, global_id 74705 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 10 monclient: _renew_subs 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1011dd50 con 0x7fcafc005250 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08018a30 con 0x7fcafc005250 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb07fff640 10 monclient: dump: 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 10 monclient: _renew_subs 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb0e7fc640 4 set_mon_vals no callback set 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.161+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb10135a30 con 0x7fcafc005250 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb08021450 con 0x7fcafc005250 2024-09-15T03:35:59.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb15caf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb15caf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb15caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1009be50 con 0x7fcafc005250 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb07fff640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaf0038f80 0x7fcaf003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.162+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(174..174 src has 106..174) v4 ==== 7741+0+0 (secure 0 0 0) 0x7fcb0804d500 con 0x7fcafc005250 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.163+0000 7fcb0f7fe640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaf0038f80 0x7fcaf003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.163+0000 7fcb0f7fe640 10 monclient: get_auth_request con 0x7fcaf0038f80 auth_method 0 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.165+0000 7fcb0f7fe640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaf0038f80 0x7fcaf003b430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcb00004430 tx=0x7fcb000042f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.167+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcb1009be50 con 0x7fcafc005250 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.167+0000 7fcb07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.167+0000 7fcb07fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:59.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.207+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v175) v1 ==== 140+0+0 (secure 0 0 0) 0x7f01cc0430a0 con 0x7f01dc099c70 2024-09-15T03:35:59.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.207+0000 7f01d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T03:35:59.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.207+0000 7f01d17fa640 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-15T03:35:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.305+0000 7f01e3434640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.305+0000 7f01e3434640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.305+0000 7f01e3434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.305+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7f01dc0008d0 con 0x7f01dc099c70 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.306+0000 7f01d17fa640 1 -- 172.21.15.181:0/2124110689 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v175) v1 ==== 128+0+0 (secure 0 0 0) 0x7f01cc0232f0 con 0x7f01dc099c70 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.306+0000 7f01d17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.306+0000 7f01d17fa640 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-15T03:35:59.307 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'barr' already exists 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01c0038740 msgr2=0x7f01c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01c0038740 0x7f01c003abf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f01d8007fd0 tx=0x7f01d8007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 10 monclient: shutdown 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 msgr2=0x7f01dc11e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 0x7f01dc11e5a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00ea20 tx=0x7f01cc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 shutdown_connections 2024-09-15T03:35:59.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01c0038740 0x7f01c003abf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01dc137200 0x7f01dc11eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01dc099c70 0x7f01dc11e5a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 --2- 172.21.15.181:0/2124110689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01dc098c70 0x7f01dc135e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 >> 172.21.15.181:0/2124110689 conn(0x7f01dc070e90 msgr2=0x7f01dc0971c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 shutdown_connections 2024-09-15T03:35:59.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.308+0000 7f01e3434640 1 -- 172.21.15.181:0/2124110689 wait complete. 2024-09-15T03:35:59.317 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush move barr root=default 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb15caf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb15caf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb15caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcb1009be50 con 0x7fcafc005250 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb07fff640 1 -- 172.21.15.153:0/1725829558 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fcb1009be50 con 0x7fcafc005250 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:35:59.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.385+0000 7fcb07fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:59.387 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:35:59.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaf0038f80 msgr2=0x7fcaf003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaf0038f80 0x7fcaf003b430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcb00004430 tx=0x7fcb000042f0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 10 monclient: shutdown 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 msgr2=0x7fcb1011b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcb1011b6b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007240 tx=0x7fcb0800b040 comp rx=0 tx=0).stop 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 shutdown_connections 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcaf0038f80 0x7fcaf003b430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcafc005250 0x7fcb1011b6b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcafc004780 0x7fcb10130950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 --2- 172.21.15.153:0/1725829558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcafc0039d0 0x7fcb1009a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 >> 172.21.15.153:0/1725829558 conn(0x7fcb10067d30 msgr2=0x7fcb10134340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 shutdown_connections 2024-09-15T03:35:59.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.387+0000 7fcb15caf640 1 -- 172.21.15.153:0/1725829558 wait complete. 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.392+0000 7ffb0db5d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.392+0000 7ffb0db5d640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.392+0000 7ffb0db5d640 10 monclient: monmap: 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:35:59.392953+0000 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:35:59.392953+0000 2024-09-15T03:35:59.393 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:59.394 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.392+0000 7ffb0db5d640 1 Processor -- start 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.392+0000 7ffb0db5d640 1 -- start start 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.392+0000 7ffb0db5d640 10 monclient: init 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb0809ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb0809f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffb08069d10 0x7ffb0813a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb08069d10 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0806a120 con 0x7ffb0809f020 2024-09-15T03:35:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb08067420 con 0x7ffb0809ec50 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0809fdc0 con 0x7ffb08069d10 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb0db5d640 10 monclient(hunting): authenticate will time out at 1163.874433s 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb06ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7ffb0809f020 legacy=0x7ffb08136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb077fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7ffb0809ec50 0x7ffb08131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:42836/0 (socket says 172.21.15.181:42836) 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb077fe640 1 -- 172.21.15.181:0/2784945428 learned_addr learned my addr 172.21.15.181:0/2784945428 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb07fff640 1 -- 172.21.15.181:0/2784945428 >> v2:172.21.15.153:3300/0 conn(0x7ffb08069d10 msgr2=0x7ffb0813a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.393+0000 7ffb07fff640 1 --2- 172.21.15.181:0/2784945428 >> v2:172.21.15.153:3300/0 conn(0x7ffb08069d10 0x7ffb0813a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.405 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:35:59.405 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:35:59.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:35:59.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.395+0000 7ffb077fe640 1 -- 172.21.15.181:0/2784945428 >> v1:172.21.15.181:6789/0 conn(0x7ffb0809ec50 legacy=0x7ffb08131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.395+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282993717 0 0) 0x7ffb08067420 con 0x7ffb0809ec50 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.395+0000 7ffb067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.395+0000 7ffb067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.395+0000 7ffb067fc640 10 monclient(hunting): my global_id is 104179 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.395+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf40034e0 con 0x7ffb0809ec50 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042804765 0 0) 0x7ffaf40034e0 con 0x7ffb0809ec50 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 >> v2:172.21.15.153:3300/0 conn(0x7ffb08069d10 msgr2=0x7ffb0813a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 --2- 172.21.15.181:0/2784945428 >> v2:172.21.15.153:3300/0 conn(0x7ffb08069d10 0x7ffb0813a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 >> v1:172.21.15.153:6789/0 conn(0x7ffb0809f020 legacy=0x7ffb08136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080a0510 con 0x7ffb0809ec50 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7ffaf8001720 con 0x7ffb0809ec50 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: dump: 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 >> v1:172.21.15.181:6789/0 conn(0x7ffb0809ec50 legacy=0x7ffb08131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaf40039d0 0x7ffaf4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): picked mon.c con 0x7ffaf40039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 0x7ffaf4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): picked mon.b con 0x7ffaf4004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4005330 0x7ffaf400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): picked mon.a con 0x7ffaf4005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0809fdc0 con 0x7ffaf4005330 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf400c290 con 0x7ffaf4004780 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf400c450 con 0x7ffaf40039d0 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb067fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb07fff640 1 -- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4005330 msgr2=0x7ffaf400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb07fff640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4005330 0x7ffaf400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 0x7ffaf4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): get_auth_request con 0x7ffaf4004780 auth_method 0 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaf40039d0 0x7ffaf4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): get_auth_request con 0x7ffaf40039d0 auth_method 0 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_done global_id 104179 payload 293 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 -- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaf40039d0 msgr2=0x7ffaf4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaf40039d0 0x7ffaf4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 -- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4005330 msgr2=0x7ffaf400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4005330 0x7ffaf400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 monclient: found mon.b 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb07fff640 1 -- 172.21.15.181:0/2784945428 reap_dead start 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 1 -- 172.21.15.181:0/2784945428 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40034e0 con 0x7ffaf4004780 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb0db5d640 5 monclient: authenticate success, global_id 104179 2024-09-15T03:35:59.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.396+0000 7ffb0db5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb06ffd640 1 -- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 msgr2=0x7ffaf4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 0x7ffaf4004bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00099c0 tx=0x7ffaf0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c0067d18da651 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0003b80 con 0x7ffaf4004780 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: dump: 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.416 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf0003d20 con 0x7ffaf4004780 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 4 set_mon_vals no callback set 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 1 -- 172.21.15.181:0/2784945428 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf002ab80 con 0x7ffaf4004780 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb067fc640 10 monclient: dump: 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 10 monclient: shutdown 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 -- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 msgr2=0x7ffaf4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 0x7ffaf4004bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00099c0 tx=0x7ffaf0009060 comp rx=0 tx=0).stop 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 -- 172.21.15.181:0/2784945428 shutdown_connections 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/2784945428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaf4004780 0x7ffaf4004bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 -- 172.21.15.181:0/2784945428 >> 172.21.15.181:0/2784945428 conn(0x7ffb080a5d40 msgr2=0x7ffb080a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 -- 172.21.15.181:0/2784945428 shutdown_connections 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.399+0000 7ffb0db5d640 1 -- 172.21.15.181:0/2784945428 wait complete. 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.400+0000 7ffb0db5d640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.400+0000 7ffb0db5d640 10 monclient: monmap: 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:35:59.419 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.400+0000 7ffb0db5d640 1 Processor -- start 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.400+0000 7ffb0db5d640 1 -- start start 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.400+0000 7ffb0db5d640 10 monclient: init 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4004780 0x7ffb08140c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf4004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:59.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 0x7ffb08130ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb08130ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb0811d640 0x7ffb0811dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb0811d640 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb081313e0 con 0x7ffaf4004780 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08120ee0 con 0x7ffb08130ab0 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0811f220 con 0x7ffb0811d640 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb0db5d640 10 monclient(hunting): authenticate will time out at 1163.882431s 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb077fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4004780 msgr2=0x7ffb08140c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb077fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4004780 0x7ffb08140c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 0x7ffb08130ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 0x7ffb08130ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46728/0 (socket says 172.21.15.181:46728) 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 -- 172.21.15.181:0/4001774552 learned_addr learned my addr 172.21.15.181:0/4001774552 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): get_auth_request con 0x7ffb08130ab0 auth_method 0 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb0811d640 0x7ffb0811dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): get_auth_request con 0x7ffb0811d640 auth_method 0 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient(hunting): handle_auth_done global_id 104182 payload 824 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 -- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb0811d640 msgr2=0x7ffb0811dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb0811d640 0x7ffb0811dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 -- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4004780 msgr2=0x7ffb08140c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4004780 0x7ffb08140c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 monclient: found mon.noname-b 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.401+0000 7ffb06ffd640 1 -- 172.21.15.181:0/4001774552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080a43e0 con 0x7ffb08130ab0 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.403+0000 7ffb06ffd640 1 -- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 msgr2=0x7ffb08130ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.403+0000 7ffb06ffd640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 0x7ffb08130ea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00099c0 tx=0x7ffaf0030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0003b80 con 0x7ffb08130ab0 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: dump: 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.424 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 5 monclient: authenticate success, global_id 104182 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf0038070 con 0x7ffb08130ab0 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 10 monclient: _renew_subs 2024-09-15T03:35:59.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb08120c00 con 0x7ffb08130ab0 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0005e30 con 0x7ffb08130ab0 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffaff7fe640 10 monclient: dump: 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb05ffb640 4 set_mon_vals no callback set 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 10 monclient: _renew_subs 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.406+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb08141350 con 0x7ffb08130ab0 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffaf0030070 con 0x7ffb08130ab0 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffb0db5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffb0db5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffb0db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb08121120 con 0x7ffb08130ab0 2024-09-15T03:35:59.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffaff7fe640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffae8038770 0x7ffae803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(175..175 src has 106..175) v4 ==== 7769+0+0 (secure 0 0 0) 0x7ffaf006c590 con 0x7ffb08130ab0 2024-09-15T03:35:59.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffb077fe640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffae8038770 0x7ffae803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.407+0000 7ffb077fe640 10 monclient: get_auth_request con 0x7ffae8038770 auth_method 0 2024-09-15T03:35:59.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.410+0000 7ffb077fe640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffae8038770 0x7ffae803ac20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8001900 tx=0x7ffb0809fca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.412+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffaf0032df0 con 0x7ffb08130ab0 2024-09-15T03:35:59.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.412+0000 7ffaff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.412+0000 7ffaff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.521+0000 7f4db595d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.521+0000 7f4db595d640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.521+0000 7f4db595d640 10 monclient: monmap: 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:59.521831+0000 2024-09-15T03:35:59.522 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:59.521831+0000 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.521+0000 7f4db595d640 1 Processor -- start 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.521+0000 7f4db595d640 1 -- start start 2024-09-15T03:35:59.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.521+0000 7f4db595d640 10 monclient: init 2024-09-15T03:35:59.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:35:59.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4db01318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:35:59.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient(hunting): picked mon.noname-b con 0x7f4db009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4db0136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0132640 con 0x7f4db01318f0 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0131d00 con 0x7f4db0136c20 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db009fc10 con 0x7f4db009ea50 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4db595d640 10 monclient(hunting): authenticate will time out at 1157.161205s 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4daeffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f4db01318f0 legacy=0x7f4db0131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:35:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49664/0 (socket says 172.21.15.153:49664) 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 -- 172.21.15.153:0/3103346627 learned_addr learned my addr 172.21.15.153:0/3103346627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): get_auth_request con 0x7f4db009ea50 auth_method 0 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_done global_id 74711 payload 274 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 -- 172.21.15.153:0/3103346627 >> v1:172.21.15.181:6789/0 conn(0x7f4db0136c20 legacy=0x7f4db0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 -- 172.21.15.153:0/3103346627 >> v1:172.21.15.153:6789/0 conn(0x7f4db01318f0 legacy=0x7f4db0131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 monclient: found mon.noname-b 2024-09-15T03:35:59.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.522+0000 7f4dae7fc640 1 -- 172.21.15.153:0/3103346627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db009fe30 con 0x7f4db009ea50 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dae7fc640 1 -- 172.21.15.153:0/3103346627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 msgr2=0x7f4db009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dae7fc640 1 --2- 172.21.15.153:0/3103346627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db009ee20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009940 tx=0x7f4da40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8458910393d108cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 1 -- 172.21.15.153:0/3103346627 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4032440 con 0x7f4db009ea50 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: dump: 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.534 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 1 -- 172.21.15.153:0/3103346627 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40325a0 con 0x7f4db009ea50 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4db595d640 5 monclient: authenticate success, global_id 74711 2024-09-15T03:35:59.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4db595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4db595d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 1 -- 172.21.15.153:0/3103346627 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4032940 con 0x7f4db009ea50 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4db595d640 4 set_mon_vals no callback set 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.525+0000 7f4dadffb640 10 monclient: dump: 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.536 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 10 monclient: shutdown 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 -- 172.21.15.153:0/3103346627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 msgr2=0x7f4db009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 --2- 172.21.15.153:0/3103346627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db009ee20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009940 tx=0x7f4da40091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 -- 172.21.15.153:0/3103346627 shutdown_connections 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 --2- 172.21.15.153:0/3103346627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db009ee20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 -- 172.21.15.153:0/3103346627 >> 172.21.15.153:0/3103346627 conn(0x7f4db0067110 msgr2=0x7f4db0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 -- 172.21.15.153:0/3103346627 shutdown_connections 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 1 -- 172.21.15.153:0/3103346627 wait complete. 2024-09-15T03:35:59.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.526+0000 7f4db595d640 10 monclient: monmap: 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 Processor -- start 2024-09-15T03:35:59.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 -- start start 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient: init 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db01327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): picked mon.noname-c con 0x7f4db009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01318f0 0x7f4db0132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): picked mon.noname-a con 0x7f4db01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4db0136c20 0x7f4db0133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): picked mon.noname-b con 0x7f4db0136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0130a30 con 0x7f4db01318f0 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db009bf40 con 0x7f4db0136c20 2024-09-15T03:35:59.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0136350 con 0x7f4db009ea50 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4dae7fc640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01318f0 msgr2=0x7f4db0132d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4db595d640 10 monclient(hunting): authenticate will time out at 1157.166630s 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4dae7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01318f0 0x7f4db0132d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db01327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db01327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49674/0 (socket says 172.21.15.153:49674) 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 1 -- 172.21.15.153:0/1429249285 learned_addr learned my addr 172.21.15.153:0/1429249285 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): get_auth_request con 0x7f4db009ea50 auth_method 0 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4db0136c20 0x7f4db0133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): get_auth_request con 0x7f4db0136c20 auth_method 0 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.527+0000 7f4daeffd640 10 monclient(hunting): handle_auth_done global_id 74714 payload 824 2024-09-15T03:35:59.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4db0136c20 msgr2=0x7f4db0133250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4db0136c20 0x7f4db0133250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01318f0 msgr2=0x7f4db0132d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01318f0 0x7f4db0132d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 1 monclient: found mon.noname-c 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.528+0000 7f4daeffd640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da4009450 con 0x7f4db009ea50 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4daeffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 msgr2=0x7f4db01327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4daeffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db01327d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4da000bb80 tx=0x7f4da00097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da000cc80 con 0x7f4db009ea50 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: dump: 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da000ce00 con 0x7f4db009ea50 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 5 monclient: authenticate success, global_id 74714 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 10 monclient: _renew_subs 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db0134350 con 0x7f4db009ea50 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0016430 con 0x7f4db009ea50 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4d96ffd640 10 monclient: dump: 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 10 monclient: _renew_subs 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4dad7fa640 4 set_mon_vals no callback set 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.530+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db0134ea0 con 0x7f4db009ea50 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4db595d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4db595d640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 0x7f4db009c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4db595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4da0012020 con 0x7f4db009ea50 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0065e00 con 0x7f4db0133790 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 0x7f4db009c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient: get_auth_request con 0x7f4db0133790 auth_method 0 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4d96ffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4d9c0386f0 0x7f4d9c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(175..175 src has 106..175) v4 ==== 7769+0+0 (secure 0 0 0) 0x7f4da004c670 con 0x7f4db009ea50 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_done global_id 74717 payload 824 2024-09-15T03:35:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4dae7fc640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4db004bf60 con 0x7f4db0133790 2024-09-15T03:35:59.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4daf7fe640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4d9c0386f0 0x7f4d9c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.531+0000 7f4daf7fe640 10 monclient: get_auth_request con 0x7f4d9c0386f0 auth_method 0 2024-09-15T03:35:59.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.533+0000 7f4dae7fc640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 msgr2=0x7f4db009c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.533+0000 7f4dae7fc640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 0x7f4db009c9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4da4034000 tx=0x7f4da4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4032440 con 0x7f4db0133790 2024-09-15T03:35:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 10 monclient: dump: 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4daf7fe640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4d9c0386f0 0x7f4d9c03aba0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4db012bb90 tx=0x7f4da8007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4da4032710 con 0x7f4db0133790 2024-09-15T03:35:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 msgr2=0x7f4db009c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 0x7f4db009c9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4da4034000 tx=0x7f4da4032000 comp rx=0 tx=0).stop 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 msgr2=0x7f4db009c9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.534+0000 7f4d96ffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db0133790 0x7f4db009c9f0 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4da4034000 tx=0x7f4da4032000 comp rx=0 tx=0).stop 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.539+0000 7f4db595d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4db595d640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 0x7f4db009d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4db595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0135120 con 0x7f4db009cf30 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 0x7f4db009d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient: get_auth_request con 0x7f4db009cf30 auth_method 0 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 10 monclient(hunting): handle_auth_done global_id 74720 payload 824 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.540+0000 7f4dae7fc640 1 -- 172.21.15.153:0/1429249285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4db0133b60 con 0x7f4db009cf30 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4dae7fc640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 msgr2=0x7f4db009d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4dae7fc640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 0x7f4db009d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009890 tx=0x7f4da403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4045840 con 0x7f4db009cf30 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 10 monclient: dump: 2024-09-15T03:35:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f4da40459a0 con 0x7f4db009cf30 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 msgr2=0x7f4db009d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 0x7f4db009d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009890 tx=0x7f4da403d000 comp rx=0 tx=0).stop 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 msgr2=0x7f4db009d330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.542+0000 7f4d96ffd640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009cf30 0x7f4db009d330 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009890 tx=0x7f4da403d000 comp rx=0 tx=0).stop 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 82, 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:59.554 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:59.555 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:59.556 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:59.557 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:59.558 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:59.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:59.560 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:59.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:59.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.564 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.564 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:59.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:59.568 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:59.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4d9c0386f0 msgr2=0x7f4d9c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4d9c0386f0 0x7f4d9c03aba0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4db012bb90 tx=0x7f4da8007e10 comp rx=0 tx=0).stop 2024-09-15T03:35:59.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 10 monclient: shutdown 2024-09-15T03:35:59.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 msgr2=0x7f4db01327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 --2- 172.21.15.153:0/1429249285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4db009ea50 0x7f4db01327d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4da000bb80 tx=0x7f4da00097f0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4daf7fe640 1 -- 172.21.15.153:0/1429249285 reap_dead start 2024-09-15T03:35:59.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4daf7fe640 1 -- 172.21.15.153:0/1429249285 reap_dead start 2024-09-15T03:35:59.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 shutdown_connections 2024-09-15T03:35:59.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 >> 172.21.15.153:0/1429249285 conn(0x7f4db0067110 msgr2=0x7f4db012bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 shutdown_connections 2024-09-15T03:35:59.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.543+0000 7f4db595d640 1 -- 172.21.15.153:0/1429249285 wait complete. 2024-09-15T03:35:59.571 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:35:59.571 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:35:59.572 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:35:59.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.571+0000 7ffb0db5d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T03:35:59.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.571+0000 7ffb0db5d640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T03:35:59.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.571+0000 7ffb0db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:35:59.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:35:59.571+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7ffb080008d0 con 0x7ffb08130ab0 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.666+0000 7f988c39d640 10 monclient: get_monmap_and_config 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.666+0000 7f988c39d640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.666+0000 7f988c39d640 10 monclient: monmap: 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:35:59.666885+0000 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:35:59.666885+0000 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:35:59.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:35:59.668 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:35:59.668 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:35:59.668 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:35:59.668 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.666+0000 7f988c39d640 1 Processor -- start 2024-09-15T03:35:59.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.666+0000 7f988c39d640 1 -- start start 2024-09-15T03:35:59.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.666+0000 7f988c39d640 10 monclient: init 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f98841318f0 0x7f9884131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f98841318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 0x7f988409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f988409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9884068d90 0x7f98840691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f9884068d90 addr v2:172.21.15.181:3300/0 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9884131cc0 con 0x7f98841318f0 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988409fbf0 con 0x7f9884068d90 2024-09-15T03:35:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988409fe30 con 0x7f988409ea50 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988c39d640 10 monclient(hunting): authenticate will time out at 1157.306485s 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988a139640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f98841318f0 msgr2=0x7f9884131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988a139640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f98841318f0 0x7f9884131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 0x7f988409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 0x7f988409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49702/0 (socket says 172.21.15.153:49702) 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 -- 172.21.15.153:0/3682761995 learned_addr learned my addr 172.21.15.153:0/3682761995 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:59.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): get_auth_request con 0x7f988409ea50 auth_method 0 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988a93a640 1 --2- 172.21.15.153:0/3682761995 >> v2:172.21.15.181:3300/0 conn(0x7f9884068d90 0x7f98840691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient(hunting): handle_auth_done global_id 74723 payload 274 2024-09-15T03:35:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 -- 172.21.15.153:0/3682761995 >> v2:172.21.15.181:3300/0 conn(0x7f9884068d90 msgr2=0x7f98840691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 --2- 172.21.15.153:0/3682761995 >> v2:172.21.15.181:3300/0 conn(0x7f9884068d90 0x7f98840691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 -- 172.21.15.153:0/3682761995 >> v2:172.21.15.153:3300/0 conn(0x7f98841318f0 msgr2=0x7f9884131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 --2- 172.21.15.153:0/3682761995 >> v2:172.21.15.153:3300/0 conn(0x7f98841318f0 0x7f9884131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 monclient: found mon.noname-b 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f9889938640 1 -- 172.21.15.153:0/3682761995 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98840a1210 con 0x7f988409ea50 2024-09-15T03:35:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988a93a640 10 monclient: get_auth_request con 0x7f9884068d90 auth_method 0 2024-09-15T03:35:59.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.667+0000 7f988a93a640 1 --2- 172.21.15.153:0/3682761995 >> v2:172.21.15.181:3300/0 conn(0x7f9884068d90 0x7f98840691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:35:59.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.670+0000 7f9889938640 1 -- 172.21.15.153:0/3682761995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 msgr2=0x7f988409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.670+0000 7f9889938640 1 --2- 172.21.15.153:0/3682761995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 0x7f988409ee20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9884064050 tx=0x7f98740091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a555c23dad32550f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 1 -- 172.21.15.153:0/3682761995 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9874032440 con 0x7f988409ea50 2024-09-15T03:35:59.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: dump: 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 1 -- 172.21.15.153:0/3682761995 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98740325a0 con 0x7f988409ea50 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 5 monclient: authenticate success, global_id 74723 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 1 -- 172.21.15.153:0/3682761995 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9874032940 con 0x7f988409ea50 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 4 set_mon_vals no callback set 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f9889137640 10 monclient: dump: 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 10 monclient: shutdown 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 -- 172.21.15.153:0/3682761995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 msgr2=0x7f988409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 --2- 172.21.15.153:0/3682761995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 0x7f988409ee20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9884064050 tx=0x7f98740091d0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 -- 172.21.15.153:0/3682761995 shutdown_connections 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 --2- 172.21.15.153:0/3682761995 >> v2:172.21.15.181:3300/0 conn(0x7f9884068d90 0x7f98840691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 --2- 172.21.15.153:0/3682761995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988409ea50 0x7f988409ee20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 --2- 172.21.15.153:0/3682761995 >> v2:172.21.15.153:3300/0 conn(0x7f98841318f0 0x7f9884131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 -- 172.21.15.153:0/3682761995 >> 172.21.15.153:0/3682761995 conn(0x7f9884067110 msgr2=0x7f9884067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 -- 172.21.15.153:0/3682761995 shutdown_connections 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.671+0000 7f988c39d640 1 -- 172.21.15.153:0/3682761995 wait complete. 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient: build_initial_monmap 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient: monmap: 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:35:59.685 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 Processor -- start 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 -- start start 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient: init 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9884068d90 0x7f988409ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f9884068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988409ea50 0x7f9884130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f988409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 0x7f9884140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f98841318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988409b740 con 0x7f9884068d90 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988409bf40 con 0x7f988409ea50 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988411dfc0 con 0x7f98841318f0 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988c39d640 10 monclient(hunting): authenticate will time out at 1157.311944s 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988a139640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9884068d90 msgr2=0x7f988409ab00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.672+0000 7f988a139640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9884068d90 0x7f988409ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:35:59.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 0x7f9884140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 0x7f9884140f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:49714/0 (socket says 172.21.15.153:49714) 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 learned_addr learned my addr 172.21.15.153:0/1620825686 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): get_auth_request con 0x7f98841318f0 auth_method 0 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f9889938640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988409ea50 0x7f9884130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient(hunting): handle_auth_done global_id 74726 payload 824 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988409ea50 msgr2=0x7f9884130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988409ea50 0x7f9884130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9884068d90 msgr2=0x7f988409ab00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9884068d90 0x7f988409ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient: _finish_hunting 0 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 monclient: found mon.noname-c 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9874009450 con 0x7f98841318f0 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f9889938640 10 monclient: get_auth_request con 0x7f988409ea50 auth_method 0 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.673+0000 7f9889938640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988409ea50 0x7f9884130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 msgr2=0x7f9884140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988a93a640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 0x7f9884140f30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f987c00cef0 tx=0x7f987c007860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00e600 con 0x7f98841318f0 2024-09-15T03:35:59.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: dump: 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:35:59.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c00e760 con 0x7f98841318f0 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 5 monclient: authenticate success, global_id 74726 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 10 monclient: _renew_subs 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988411dce0 con 0x7f98841318f0 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00eb20 con 0x7f98841318f0 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f987a7fc640 10 monclient: dump: 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.691 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f9888936640 4 set_mon_vals no callback set 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 10 monclient: _renew_subs 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.675+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988411c940 con 0x7f98841318f0 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f987c016420 con 0x7f98841318f0 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988c39d640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 0x7f9884141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9884065e00 con 0x7f988411bbf0 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f987a7fc640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f98680386f0 0x7f986803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(175..175 src has 106..175) v4 ==== 7769+0+0 (secure 0 0 0) 0x7f987c012070 con 0x7f98841318f0 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988a139640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 0x7f9884141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988a139640 10 monclient: get_auth_request con 0x7f988411bbf0 auth_method 0 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f988a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.676+0000 7f9889938640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f98680386f0 0x7f986803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.677+0000 7f988a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.677+0000 7f988a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.677+0000 7f988a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.677+0000 7f9889938640 10 monclient: get_auth_request con 0x7f98680386f0 auth_method 0 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.677+0000 7f988a139640 10 monclient(hunting): handle_auth_done global_id 104188 payload 824 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.677+0000 7f988a139640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f988404bf60 con 0x7f988411bbf0 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f988a139640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 msgr2=0x7f9884141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f988a139640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 0x7f9884141470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f988000b480 tx=0x7f98800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f9889938640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f98680386f0 0x7f986803aba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9884076600 tx=0x7f9874032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98800050f0 con 0x7f988411bbf0 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 10 monclient: dump: 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9880005250 con 0x7f988411bbf0 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 msgr2=0x7f9884141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 0x7f9884141470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f988000b480 tx=0x7f98800091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 msgr2=0x7f9884141470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:59.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.679+0000 7f987a7fc640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f988411bbf0 0x7f9884141470 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f988000b480 tx=0x7f98800091a0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.685+0000 7f988c39d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:35:59.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.685+0000 7f988c39d640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 0x7f98841419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:35:59.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.685+0000 7f988c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:35:59.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.685+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988411e780 con 0x7f9884131100 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.685+0000 7f988a139640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 0x7f98841419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.685+0000 7f988a139640 10 monclient: get_auth_request con 0x7f9884131100 auth_method 0 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 10 monclient(hunting): handle_auth_done global_id 104191 payload 824 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.686+0000 7f988a139640 1 -- 172.21.15.153:0/1620825686 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f98840638d0 con 0x7f9884131100 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.688+0000 7f988a139640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 msgr2=0x7f98841419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.688+0000 7f988a139640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 0x7f98841419b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f988000b480 tx=0x7f9880015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:35:59.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.688+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988001b650 con 0x7f9884131100 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.688+0000 7f987a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.688+0000 7f987a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.688+0000 7f987a7fc640 10 monclient: dump: 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:35:59.699 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f988001b7b0 con 0x7f9884131100 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 msgr2=0x7f98841419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 0x7f98841419b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f988000b480 tx=0x7f9880015000 comp rx=0 tx=0).stop 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 msgr2=0x7f98841419b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f987a7fc640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9884131100 0x7f98841419b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 82, 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:35:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:35:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:35:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:35:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:35:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:35:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:35:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:35:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:35:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:35:59.713 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:35:59.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f98680386f0 msgr2=0x7f986803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f98680386f0 0x7f986803aba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9884076600 tx=0x7f9874032000 comp rx=0 tx=0).stop 2024-09-15T03:35:59.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 10 monclient: shutdown 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 msgr2=0x7f9884140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 --2- 172.21.15.153:0/1620825686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98841318f0 0x7f9884140f30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f987c00cef0 tx=0x7f987c007860 comp rx=0 tx=0).stop 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 reap_dead start 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988a93a640 1 -- 172.21.15.153:0/1620825686 reap_dead start 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 shutdown_connections 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 >> 172.21.15.153:0/1620825686 conn(0x7f9884067110 msgr2=0x7f98840a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 shutdown_connections 2024-09-15T03:35:59.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:35:59.689+0000 7f988c39d640 1 -- 172.21.15.153:0/1620825686 wait complete. 2024-09-15T03:35:59.715 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:35:59.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:36:00.212 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.211+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v176) v1 ==== 173+0+0 (secure 0 0 0) 0x7ffaf0032840 con 0x7ffb08130ab0 2024-09-15T03:36:00.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.211+0000 7ffaff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T03:36:00.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.211+0000 7ffaff7fe640 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-15T03:36:00.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffb0db5d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T03:36:00.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffb0db5d640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T03:36:00.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffb0db5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:00.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7ffb080a0200 con 0x7ffb08130ab0 2024-09-15T03:36:00.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffaff7fe640 1 -- 172.21.15.181:0/4001774552 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v176) v1 ==== 183+0+0 (secure 0 0 0) 0x7ffaf006a3d0 con 0x7ffb08130ab0 2024-09-15T03:36:00.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffaff7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-15T03:36:00.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.313+0000 7ffaff7fe640 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-15T03:36:00.315 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffae8038770 msgr2=0x7ffae803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffae8038770 0x7ffae803ac20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8001900 tx=0x7ffb0809fca0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 10 monclient: shutdown 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 msgr2=0x7ffb08130ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 0x7ffb08130ea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00099c0 tx=0x7ffaf0030000 comp rx=0 tx=0).stop 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 shutdown_connections 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffae8038770 0x7ffae803ac20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb0811d640 0x7ffb0811dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb08130ab0 0x7ffb08130ea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.317 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 --2- 172.21.15.181:0/4001774552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaf4004780 0x7ffb08140c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 >> 172.21.15.181:0/4001774552 conn(0x7ffb080a5d40 msgr2=0x7ffb0809dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 shutdown_connections 2024-09-15T03:36:00.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.315+0000 7ffb0db5d640 1 -- 172.21.15.181:0/4001774552 wait complete. 2024-09-15T03:36:00.325 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush move fooo rack=barr 2024-09-15T03:36:00.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.398+0000 7fc8fcb1d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:00.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.398+0000 7fc8fcb1d640 10 monclient: build_initial_monmap 2024-09-15T03:36:00.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.398+0000 7fc8fcb1d640 10 monclient: monmap: 2024-09-15T03:36:00.399 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:00.398848+0000 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:00.398848+0000 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:00.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.398+0000 7fc8fcb1d640 1 Processor -- start 2024-09-15T03:36:00.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.398+0000 7fc8fcb1d640 1 -- start start 2024-09-15T03:36:00.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.398+0000 7fc8fcb1d640 10 monclient: init 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8f809ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 0x7fc8f8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc8f8136d60 0x7fc8f8139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f8136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f8067420 con 0x7fc8f809ec50 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f809fdc0 con 0x7fc8f8136d60 2024-09-15T03:36:00.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80a0510 con 0x7fc8f809f020 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8fcb1d640 10 monclient(hunting): authenticate will time out at 1164.880513s 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f659c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc8f809ec50 0x7fc8f8131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:42846/0 (socket says 172.21.15.181:42846) 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f659c640 1 -- 172.21.15.181:0/3762260020 learned_addr learned my addr 172.21.15.181:0/3762260020 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f6d9d640 1 -- 172.21.15.181:0/3762260020 >> v2:172.21.15.153:3300/0 conn(0x7fc8f8136d60 msgr2=0x7fc8f8139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f6d9d640 1 --2- 172.21.15.181:0/3762260020 >> v2:172.21.15.153:3300/0 conn(0x7fc8f8136d60 0x7fc8f8139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 1 --2- 172.21.15.181:0/3762260020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 0x7fc8f8069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:00.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): get_auth_request con 0x7fc8f809f020 auth_method 0 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_done global_id 74729 payload 274 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/3762260020 >> v2:172.21.15.153:3300/0 conn(0x7fc8f8136d60 msgr2=0x7fc8f8139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:00.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 1 --2- 172.21.15.181:0/3762260020 >> v2:172.21.15.153:3300/0 conn(0x7fc8f8136d60 0x7fc8f8139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/3762260020 >> v1:172.21.15.181:6789/0 conn(0x7fc8f809ec50 legacy=0x7fc8f8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 1 monclient: found mon.noname-b 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.399+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/3762260020 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f80a0200 con 0x7fc8f809f020 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.402+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/3762260020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 msgr2=0x7fc8f8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.402+0000 7fc8f5d9b640 1 --2- 172.21.15.181:0/3762260020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 0x7fc8f8069d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009850 tx=0x7fc8ec0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55c2fa6b90b548f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 1 -- 172.21.15.181:0/3762260020 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec032440 con 0x7fc8f809f020 2024-09-15T03:36:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: dump: 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: _finish_auth 0 2024-09-15T03:36:00.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 5 monclient: authenticate success, global_id 74729 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 1 -- 172.21.15.181:0/3762260020 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec0325a0 con 0x7fc8f809f020 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 4 set_mon_vals no callback set 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 1 -- 172.21.15.181:0/3762260020 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec032940 con 0x7fc8f809f020 2024-09-15T03:36:00.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8f559a640 10 monclient: dump: 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:00.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 10 monclient: shutdown 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/3762260020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 msgr2=0x7fc8f8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/3762260020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 0x7fc8f8069d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009850 tx=0x7fc8ec0090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/3762260020 shutdown_connections 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/3762260020 >> v2:172.21.15.153:3300/0 conn(0x7fc8f8136d60 0x7fc8f8139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/3762260020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809f020 0x7fc8f8069d10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/3762260020 >> 172.21.15.181:0/3762260020 conn(0x7fc8f80a5d40 msgr2=0x7fc8f80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/3762260020 shutdown_connections 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.403+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/3762260020 wait complete. 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.404+0000 7fc8fcb1d640 10 monclient: build_initial_monmap 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.404+0000 7fc8fcb1d640 10 monclient: monmap: 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:00.418 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.404+0000 7fc8fcb1d640 1 Processor -- start 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.404+0000 7fc8fcb1d640 1 -- start start 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.404+0000 7fc8fcb1d640 10 monclient: init 2024-09-15T03:36:00.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809ec50 0x7fc8f80973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 0x7fc8f8097920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f809f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8136d60 0x7fc8f8097e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f8136d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8098e70 con 0x7fc8f8136d60 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f809c000 con 0x7fc8f809f020 2024-09-15T03:36:00.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80a3da0 con 0x7fc8f809ec50 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8fcb1d640 10 monclient(hunting): authenticate will time out at 1164.886115s 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 0x7fc8f8097920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 0x7fc8f8097920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46740/0 (socket says 172.21.15.181:46740) 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f6d9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8136d60 msgr2=0x7fc8f8097e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/4055406200 learned_addr learned my addr 172.21.15.181:0/4055406200 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): get_auth_request con 0x7fc8f809f020 auth_method 0 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f6d9d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8136d60 0x7fc8f8097e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:00.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809ec50 0x7fc8f80973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): get_auth_request con 0x7fc8f809ec50 auth_method 0 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient(hunting): handle_auth_done global_id 104194 payload 824 2024-09-15T03:36:00.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809ec50 msgr2=0x7fc8f80973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809ec50 0x7fc8f80973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8136d60 msgr2=0x7fc8f8097e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8136d60 0x7fc8f8097e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 monclient: found mon.noname-b 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:00.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.405+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/4055406200 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009390 con 0x7fc8f809f020 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8f5d9b640 1 -- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 msgr2=0x7fc8f8097920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8f5d9b640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 0x7fc8f8097920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009850 tx=0x7fc8ec032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec031990 con 0x7fc8f809f020 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: dump: 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.424 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec031b10 con 0x7fc8f809f020 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 5 monclient: authenticate success, global_id 104194 2024-09-15T03:36:00.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 10 monclient: _renew_subs 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f809a970 con 0x7fc8f809f020 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec03b400 con 0x7fc8f809f020 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8e67fc640 10 monclient: dump: 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.426 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8f4d99640 4 set_mon_vals no callback set 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 10 monclient: _renew_subs 2024-09-15T03:36:00.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.407+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f809b5d0 con 0x7fc8f809f020 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc8ec039040 con 0x7fc8f809f020 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8e67fc640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(176..176 src has 106..176) v4 ==== 7957+0+0 (secure 0 0 0) 0x7fc8ec06c7c0 con 0x7fc8f809f020 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8fcb1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8fcb1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8fcb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8f659c640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f80008d0 con 0x7fc8f809f020 2024-09-15T03:36:00.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.408+0000 7fc8f659c640 10 monclient: get_auth_request con 0x7fc8d80386f0 auth_method 0 2024-09-15T03:36:00.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.411+0000 7fc8f659c640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8007fd0 tx=0x7fc8e8007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:00.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.413+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc8ec03f020 con 0x7fc8f809f020 2024-09-15T03:36:00.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.413+0000 7fc8e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:00.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.413+0000 7fc8e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:00.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.571+0000 7fc8fcb1d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T03:36:00.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.571+0000 7fc8fcb1d640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T03:36:00.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.571+0000 7fc8fcb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:00.571+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fc8f809af60 con 0x7fc8f809f020 2024-09-15T03:36:01.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.213+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v177) v1 ==== 167+0+0 (secure 0 0 0) 0x7fc8ec037070 con 0x7fc8f809f020 2024-09-15T03:36:01.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.213+0000 7fc8e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T03:36:01.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.213+0000 7fc8e67fc640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.316+0000 7fc8fcb1d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.316+0000 7fc8fcb1d640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.316+0000 7fc8fcb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.316+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fc8f809af60 con 0x7fc8f809f020 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.317+0000 7fc8e67fc640 1 -- 172.21.15.181:0/4055406200 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v178) v1 ==== 177+0+0 (secure 0 0 0) 0x7fc8ec039ad0 con 0x7fc8f809f020 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.317+0000 7fc8e67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-15T03:36:01.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.317+0000 7fc8e67fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-15T03:36:01.319 INFO:tasks.workunit.client.0.smithi181.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-15T03:36:01.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 msgr2=0x7fc8d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:01.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8007fd0 tx=0x7fc8e8007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 10 monclient: shutdown 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 msgr2=0x7fc8f8097920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 0x7fc8f8097920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009850 tx=0x7fc8ec032000 comp rx=0 tx=0).stop 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 shutdown_connections 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f8136d60 0x7fc8f8097e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f809f020 0x7fc8f8097920 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 --2- 172.21.15.181:0/4055406200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f809ec50 0x7fc8f80973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 >> 172.21.15.181:0/4055406200 conn(0x7fc8f80a5d40 msgr2=0x7fc8f809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 shutdown_connections 2024-09-15T03:36:01.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.319+0000 7fc8fcb1d640 1 -- 172.21.15.181:0/4055406200 wait complete. 2024-09-15T03:36:01.330 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm fooo 2024-09-15T03:36:01.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.401+0000 7fc7bb8dd640 10 monclient: get_monmap_and_config 2024-09-15T03:36:01.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.401+0000 7fc7bb8dd640 10 monclient: build_initial_monmap 2024-09-15T03:36:01.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.401+0000 7fc7bb8dd640 10 monclient: monmap: 2024-09-15T03:36:01.408 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:01.408 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:01.402409+0000 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:01.402409+0000 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 1 Processor -- start 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 1 -- start start 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient: init 2024-09-15T03:36:01.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc7b406e000 0x7fc7b406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b406e000 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7b406a710 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7b409e170 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40699c0 con 0x7fc7b406a710 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b406be20 con 0x7fc7b409e170 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b406a360 con 0x7fc7b406e000 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7bb8dd640 10 monclient(hunting): authenticate will time out at 1165.883847s 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7b9679640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc7b406e000 0x7fc7b406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:01.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7b8e78640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fc7b406a710 legacy=0x7fc7b409da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7b9679640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc7b406e000 0x7fc7b406e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46742/0 (socket says 172.21.15.181:46742) 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 1 -- 172.21.15.181:0/2029923043 learned_addr learned my addr 172.21.15.181:0/2029923043 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.402+0000 7fc7b9e7a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc7b409e170 0x7fc7b409e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:42860/0 (socket says 172.21.15.181:42860) 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): get_auth_request con 0x7fc7b406e000 auth_method 0 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient(hunting): handle_auth_done global_id 104197 payload 274 2024-09-15T03:36:01.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 1 -- 172.21.15.181:0/2029923043 >> v1:172.21.15.181:6789/0 conn(0x7fc7b409e170 legacy=0x7fc7b409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 1 -- 172.21.15.181:0/2029923043 >> v1:172.21.15.153:6789/0 conn(0x7fc7b406a710 legacy=0x7fc7b409da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient: _finish_hunting 0 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 1 monclient: found mon.noname-c 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.403+0000 7fc7b9679640 1 -- 172.21.15.181:0/2029923043 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b4069f80 con 0x7fc7b406e000 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7b9679640 1 -- 172.21.15.181:0/2029923043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406e000 msgr2=0x7fc7b406e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7b9679640 1 --2- 172.21.15.181:0/2029923043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406e000 0x7fc7b406e3d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4063870 tx=0x7fc7b0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=581fe13a49796b67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 1 -- 172.21.15.181:0/2029923043 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0007610 con 0x7fc7b406e000 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: dump: 2024-09-15T03:36:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: _finish_auth 0 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 5 monclient: authenticate success, global_id 104197 2024-09-15T03:36:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 1 -- 172.21.15.181:0/2029923043 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b002d050 con 0x7fc7b406e000 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 4 set_mon_vals no callback set 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 1 -- 172.21.15.181:0/2029923043 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00079a0 con 0x7fc7b406e000 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7abfff640 10 monclient: dump: 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:01.414 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 10 monclient: shutdown 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/2029923043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406e000 msgr2=0x7fc7b406e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/2029923043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406e000 0x7fc7b406e3d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4063870 tx=0x7fc7b0009840 comp rx=0 tx=0).stop 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/2029923043 shutdown_connections 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/2029923043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406e000 0x7fc7b406e3d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/2029923043 >> 172.21.15.181:0/2029923043 conn(0x7fc7b4071340 msgr2=0x7fc7b40a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/2029923043 shutdown_connections 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.406+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/2029923043 wait complete. 2024-09-15T03:36:01.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.407+0000 7fc7bb8dd640 10 monclient: build_initial_monmap 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.407+0000 7fc7bb8dd640 10 monclient: monmap: 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.407+0000 7fc7bb8dd640 1 Processor -- start 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.407+0000 7fc7bb8dd640 1 -- start start 2024-09-15T03:36:01.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.407+0000 7fc7bb8dd640 10 monclient: init 2024-09-15T03:36:01.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:01.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:01.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 0x7fc7b4133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b406a710 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b406e000 0x7fc7b411dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b406e000 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b409e170 0x7fc7b411e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b409e170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b411ff30 con 0x7fc7b406e000 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4141f30 con 0x7fc7b406a710 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4120db0 con 0x7fc7b409e170 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:01.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7bb8dd640 10 monclient(hunting): authenticate will time out at 1165.889437s 2024-09-15T03:36:01.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 0x7fc7b4133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:01.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 0x7fc7b4133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46754/0 (socket says 172.21.15.181:46754) 2024-09-15T03:36:01.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 -- 172.21.15.181:0/1391354727 learned_addr learned my addr 172.21.15.181:0/1391354727 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:01.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): get_auth_request con 0x7fc7b406a710 auth_method 0 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b8e78640 1 -- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b406e000 msgr2=0x7fc7b411dae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b8e78640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b406e000 0x7fc7b411dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient(hunting): handle_auth_done global_id 104200 payload 824 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 -- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b409e170 msgr2=0x7fc7b411e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b409e170 0x7fc7b411e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 -- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b406e000 msgr2=0x7fc7b411dae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:01.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b406e000 0x7fc7b411dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient: _finish_hunting 0 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 monclient: found mon.noname-b 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.408+0000 7fc7b9679640 1 -- 172.21.15.181:0/1391354727 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b0009ae0 con 0x7fc7b406a710 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.410+0000 7fc7b9679640 1 -- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 msgr2=0x7fc7b4133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.410+0000 7fc7b9679640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 0x7fc7b4133c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009ab0 tx=0x7fc7b0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0031b80 con 0x7fc7b406a710 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: dump: 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:01.424 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b0031ce0 con 0x7fc7b406a710 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 5 monclient: authenticate success, global_id 104200 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:01.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b003c670 con 0x7fc7b406a710 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 10 monclient: _renew_subs 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b4121090 con 0x7fc7b406a710 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 10 monclient: dump: 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:01.426 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7ab7fe640 4 set_mon_vals no callback set 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 10 monclient: _renew_subs 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b4136590 con 0x7fc7b406a710 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.411+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc7b003ca60 con 0x7fc7b406a710 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7bb8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7bb8dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7bb8dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b411f3d0 con 0x7fc7b406a710 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7a97fa640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7980386f0 0x7fc79803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:01.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(178..178 src has 106..178) v4 ==== 7957+0+0 (secure 0 0 0) 0x7fc7b006d1f0 con 0x7fc7b406a710 2024-09-15T03:36:01.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7b8e78640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7980386f0 0x7fc79803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:01.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.412+0000 7fc7b8e78640 10 monclient: get_auth_request con 0x7fc7980386f0 auth_method 0 2024-09-15T03:36:01.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.414+0000 7fc7b8e78640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7980386f0 0x7fc79803aba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4099a20 tx=0x7fc7a4006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:01.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.416+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc7b003aaf0 con 0x7fc7b406a710 2024-09-15T03:36:01.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.416+0000 7fc7a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:01.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.416+0000 7fc7a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:01.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.576+0000 7fc7bb8dd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T03:36:01.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.576+0000 7fc7bb8dd640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T03:36:01.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.576+0000 7fc7bb8dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:01.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:01.576+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7fc7b40008d0 con 0x7fc7b406a710 2024-09-15T03:36:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.223+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v179) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc7b0040070 con 0x7fc7b406a710 2024-09-15T03:36:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.223+0000 7fc7a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T03:36:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.223+0000 7fc7a97fa640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-15T03:36:02.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7bb8dd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T03:36:02.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7bb8dd640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T03:36:02.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7bb8dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:02.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7fc7b40008d0 con 0x7fc7b406a710 2024-09-15T03:36:02.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7a97fa640 1 -- 172.21.15.181:0/1391354727 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v179) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc7b003a310 con 0x7fc7b406a710 2024-09-15T03:36:02.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7a97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-15T03:36:02.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.321+0000 7fc7a97fa640 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-15T03:36:02.323 INFO:tasks.workunit.client.0.smithi181.stderr:device 'fooo' does not appear in the crush map 2024-09-15T03:36:02.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7980386f0 msgr2=0x7fc79803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:02.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7980386f0 0x7fc79803aba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4099a20 tx=0x7fc7a4006d80 comp rx=0 tx=0).stop 2024-09-15T03:36:02.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 10 monclient: shutdown 2024-09-15T03:36:02.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 msgr2=0x7fc7b4133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 0x7fc7b4133c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009ab0 tx=0x7fc7b0036000 comp rx=0 tx=0).stop 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 shutdown_connections 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7980386f0 0x7fc79803aba0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b409e170 0x7fc7b411e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b406e000 0x7fc7b411dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 --2- 172.21.15.181:0/1391354727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7b406a710 0x7fc7b4133c90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 >> 172.21.15.181:0/1391354727 conn(0x7fc7b4071340 msgr2=0x7fc7b40a20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 shutdown_connections 2024-09-15T03:36:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.323+0000 7fc7bb8dd640 1 -- 172.21.15.181:0/1391354727 wait complete. 2024-09-15T03:36:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm barr 2024-09-15T03:36:02.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.404+0000 7f7e5aaa8640 10 monclient: get_monmap_and_config 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.404+0000 7f7e5aaa8640 10 monclient: build_initial_monmap 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.405+0000 7f7e5aaa8640 10 monclient: monmap: 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:02.405778+0000 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:02.405778+0000 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:02.407 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:02.407 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:02.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.405+0000 7f7e5aaa8640 1 Processor -- start 2024-09-15T03:36:02.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.405+0000 7f7e5aaa8640 1 -- start start 2024-09-15T03:36:02.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.405+0000 7f7e5aaa8640 10 monclient: init 2024-09-15T03:36:02.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7e540a1800 0x7f7e5409aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f7e540a1800 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 0x7f7e5409afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f7e540983d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e5412ec70 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e540a1bd0 con 0x7f7e5412ec70 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5409cbf0 con 0x7f7e540a1800 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5412f600 con 0x7f7e540983d0 2024-09-15T03:36:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e5aaa8640 10 monclient(hunting): authenticate will time out at 1166.887324s 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e58844640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f7e540a1800 msgr2=0x7f7e5409aa90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e59045640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f7e5412ec70 0x7f7e54096a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:42866/0 (socket says 172.21.15.181:42866) 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e59045640 1 -- 172.21.15.181:0/33473756 learned_addr learned my addr 172.21.15.181:0/33473756 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e58844640 1 --2- 172.21.15.181:0/33473756 >> v2:172.21.15.153:3300/0 conn(0x7f7e540a1800 0x7f7e5409aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 1 --2- 172.21.15.181:0/33473756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 0x7f7e5409afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): get_auth_request con 0x7f7e540983d0 auth_method 0 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:02.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient(hunting): handle_auth_done global_id 74735 payload 274 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 1 -- 172.21.15.181:0/33473756 >> v2:172.21.15.153:3300/0 conn(0x7f7e540a1800 msgr2=0x7f7e5409aa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 1 --2- 172.21.15.181:0/33473756 >> v2:172.21.15.153:3300/0 conn(0x7f7e540a1800 0x7f7e5409aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 1 -- 172.21.15.181:0/33473756 >> v1:172.21.15.181:6789/0 conn(0x7f7e5412ec70 legacy=0x7f7e54096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 1 monclient: found mon.noname-b 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.406+0000 7f7e53fff640 1 -- 172.21.15.181:0/33473756 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e54098cc0 con 0x7f7e540983d0 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.409+0000 7f7e53fff640 1 -- 172.21.15.181:0/33473756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 msgr2=0x7f7e5409afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.409+0000 7f7e53fff640 1 --2- 172.21.15.181:0/33473756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 0x7f7e5409afd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009850 tx=0x7f7e480090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2b899289fdc82f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 1 -- 172.21.15.181:0/33473756 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48032440 con 0x7f7e540983d0 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: dump: 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:02.418 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 1 -- 172.21.15.181:0/33473756 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e480325a0 con 0x7f7e540983d0 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 1 -- 172.21.15.181:0/33473756 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48032940 con 0x7f7e540983d0 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 5 monclient: authenticate success, global_id 74735 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 4 set_mon_vals no callback set 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e537fe640 10 monclient: dump: 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 10 monclient: shutdown 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/33473756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 msgr2=0x7f7e5409afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/33473756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 0x7f7e5409afd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009850 tx=0x7f7e480090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/33473756 shutdown_connections 2024-09-15T03:36:02.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/33473756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540983d0 0x7f7e5409afd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/33473756 >> v2:172.21.15.153:3300/0 conn(0x7f7e540a1800 0x7f7e5409aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.410+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/33473756 >> 172.21.15.181:0/33473756 conn(0x7f7e54071100 msgr2=0x7f7e540a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:02.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.411+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/33473756 shutdown_connections 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.411+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/33473756 wait complete. 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.411+0000 7f7e5aaa8640 10 monclient: build_initial_monmap 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.411+0000 7f7e5aaa8640 10 monclient: monmap: 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:02.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.411+0000 7f7e5aaa8640 1 Processor -- start 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.411+0000 7f7e5aaa8640 1 -- start start 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient: init 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540a1800 0x7f7e5409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): picked mon.noname-c con 0x7f7e540a1800 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5412ec70 0x7f7e5412dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f7e5412ec70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 0x7f7e54141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f7e54117a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:02.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5411e030 con 0x7f7e5412ec70 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5409c310 con 0x7f7e54117a20 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e541318c0 con 0x7f7e540a1800 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e5aaa8640 10 monclient(hunting): authenticate will time out at 1166.893434s 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 0x7f7e54141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 0x7f7e54141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46768/0 (socket says 172.21.15.181:46768) 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 -- 172.21.15.181:0/3964454186 learned_addr learned my addr 172.21.15.181:0/3964454186 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): get_auth_request con 0x7f7e54117a20 auth_method 0 2024-09-15T03:36:02.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e53fff640 1 -- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5412ec70 msgr2=0x7f7e5412dc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e53fff640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5412ec70 0x7f7e5412dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e58844640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540a1800 0x7f7e5409b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient(hunting): handle_auth_done global_id 104203 payload 824 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 -- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540a1800 msgr2=0x7f7e5409b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540a1800 0x7f7e5409b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 -- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5412ec70 msgr2=0x7f7e5412dc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:02.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5412ec70 0x7f7e5412dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient: _finish_hunting 0 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 monclient: found mon.noname-b 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.412+0000 7f7e59045640 1 -- 172.21.15.181:0/3964454186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48009390 con 0x7f7e54117a20 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e59045640 1 -- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 msgr2=0x7f7e54141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e59045640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 0x7f7e54141150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00cbc0 tx=0x7f7e4c005310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c00a850 con 0x7f7e54117a20 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: dump: 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:02.429 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 5 monclient: authenticate success, global_id 104203 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c00a9d0 con 0x7f7e54117a20 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 10 monclient: _renew_subs 2024-09-15T03:36:02.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5411dd50 con 0x7f7e54117a20 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c00ad90 con 0x7f7e54117a20 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e50ff9640 10 monclient: dump: 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:02.431 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 10 monclient: _renew_subs 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e52ffd640 4 set_mon_vals no callback set 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.415+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e54118490 con 0x7f7e54117a20 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7e4c007300 con 0x7f7e54117a20 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e50ff9640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e400386f0 0x7f7e4003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(179..179 src has 106..179) v4 ==== 7741+0+0 (secure 0 0 0) 0x7f7e4c04c8d0 con 0x7f7e54117a20 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e5aaa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e5aaa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:02.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e5aaa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:02.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e58844640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e400386f0 0x7f7e4003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:02.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e540008d0 con 0x7f7e54117a20 2024-09-15T03:36:02.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.416+0000 7f7e58844640 10 monclient: get_auth_request con 0x7f7e400386f0 auth_method 0 2024-09-15T03:36:02.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.418+0000 7f7e58844640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e400386f0 0x7f7e4003aba0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7e44006fd0 tx=0x7f7e44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:02.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.420+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7e4c01b400 con 0x7f7e54117a20 2024-09-15T03:36:02.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.420+0000 7f7e50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:02.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.420+0000 7f7e50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:02.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.578+0000 7f7e5aaa8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T03:36:02.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.578+0000 7f7e5aaa8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T03:36:02.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.578+0000 7f7e5aaa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:02.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:02.578+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f7e54064a20 con 0x7f7e54117a20 2024-09-15T03:36:03.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.230+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v180) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7e4c010af0 con 0x7f7e54117a20 2024-09-15T03:36:03.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.230+0000 7f7e50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T03:36:03.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.230+0000 7f7e50ff9640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-15T03:36:03.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e5aaa8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T03:36:03.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e5aaa8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T03:36:03.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e5aaa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:03.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f7e540983d0 con 0x7f7e54117a20 2024-09-15T03:36:03.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e50ff9640 1 -- 172.21.15.181:0/3964454186 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v180) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7e4c010cd0 con 0x7f7e54117a20 2024-09-15T03:36:03.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e50ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-15T03:36:03.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.334+0000 7f7e50ff9640 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-15T03:36:03.336 INFO:tasks.workunit.client.0.smithi181.stderr:device 'barr' does not appear in the crush map 2024-09-15T03:36:03.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e400386f0 msgr2=0x7f7e4003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:03.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e400386f0 0x7f7e4003aba0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7e44006fd0 tx=0x7f7e44008040 comp rx=0 tx=0).stop 2024-09-15T03:36:03.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 10 monclient: shutdown 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 msgr2=0x7f7e54141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 0x7f7e54141150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00cbc0 tx=0x7f7e4c005310 comp rx=0 tx=0).stop 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 shutdown_connections 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e400386f0 0x7f7e4003aba0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e54117a20 0x7f7e54141150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5412ec70 0x7f7e5412dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 --2- 172.21.15.181:0/3964454186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e540a1800 0x7f7e5409b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 >> 172.21.15.181:0/3964454186 conn(0x7f7e54071100 msgr2=0x7f7e54131ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 shutdown_connections 2024-09-15T03:36:03.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.337+0000 7f7e5aaa8640 1 -- 172.21.15.181:0/3964454186 wait complete. 2024-09-15T03:36:03.347 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T03:36:03.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.420+0000 7fdff659d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:03.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.420+0000 7fdff659d640 10 monclient: build_initial_monmap 2024-09-15T03:36:03.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.420+0000 7fdff659d640 10 monclient: monmap: 2024-09-15T03:36:03.421 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:03.421 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:03.421 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:03.421130+0000 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:03.421130+0000 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:03.422 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 Processor -- start 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 -- start start 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient: init 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdff00a1800 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdff00983d0 0x7fdff0133ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): picked mon.noname-a con 0x7fdff00983d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 0x7fdff0136020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): picked mon.noname-b con 0x7fdff012ec70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff00a1bd0 con 0x7fdff00a1800 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff009cbf0 con 0x7fdff00983d0 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff012f600 con 0x7fdff012ec70 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff659d640 10 monclient(hunting): authenticate will time out at 1167.902619s 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdfeffff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fdff00a1800 0x7fdff009aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:42880/0 (socket says 172.21.15.181:42880) 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdfeffff640 1 -- 172.21.15.181:0/661377196 learned_addr learned my addr 172.21.15.181:0/661377196 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:03.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdfef7fe640 1 -- 172.21.15.181:0/661377196 >> v2:172.21.15.153:3300/0 conn(0x7fdff00983d0 msgr2=0x7fdff0133ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdfef7fe640 1 --2- 172.21.15.181:0/661377196 >> v2:172.21.15.153:3300/0 conn(0x7fdff00983d0 0x7fdff0133ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff4b3a640 1 --2- 172.21.15.181:0/661377196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 0x7fdff0136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff4b3a640 10 monclient(hunting): get_auth_request con 0x7fdff012ec70 auth_method 0 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff4b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff4b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.421+0000 7fdff4b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_done global_id 74738 payload 274 2024-09-15T03:36:03.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 1 -- 172.21.15.181:0/661377196 >> v2:172.21.15.153:3300/0 conn(0x7fdff00983d0 msgr2=0x7fdff0133ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 1 --2- 172.21.15.181:0/661377196 >> v2:172.21.15.153:3300/0 conn(0x7fdff00983d0 0x7fdff0133ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 1 -- 172.21.15.181:0/661377196 >> v1:172.21.15.181:6789/0 conn(0x7fdff00a1800 legacy=0x7fdff009aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 1 monclient: found mon.noname-b 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:03.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.422+0000 7fdff4b3a640 1 -- 172.21.15.181:0/661377196 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0098cc0 con 0x7fdff012ec70 2024-09-15T03:36:03.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff4b3a640 1 -- 172.21.15.181:0/661377196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 msgr2=0x7fdff0136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:03.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff4b3a640 1 --2- 172.21.15.181:0/661377196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 0x7fdff0136020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009e60 tx=0x7fdfe8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37ce21f69cf7b14b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:03.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 1 -- 172.21.15.181:0/661377196 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8007840 con 0x7fdff012ec70 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: dump: 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:03.434 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 1 -- 172.21.15.181:0/661377196 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe802d050 con 0x7fdff012ec70 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 1 -- 172.21.15.181:0/661377196 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8007bd0 con 0x7fdff012ec70 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdfeeffd640 10 monclient: dump: 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.435 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 5 monclient: authenticate success, global_id 74738 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 4 set_mon_vals no callback set 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 10 monclient: shutdown 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:03.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 -- 172.21.15.181:0/661377196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 msgr2=0x7fdff0136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 --2- 172.21.15.181:0/661377196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 0x7fdff0136020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009e60 tx=0x7fdfe8009780 comp rx=0 tx=0).stop 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 -- 172.21.15.181:0/661377196 shutdown_connections 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 --2- 172.21.15.181:0/661377196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff012ec70 0x7fdff0136020 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 --2- 172.21.15.181:0/661377196 >> v2:172.21.15.153:3300/0 conn(0x7fdff00983d0 0x7fdff0133ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 -- 172.21.15.181:0/661377196 >> 172.21.15.181:0/661377196 conn(0x7fdff0071100 msgr2=0x7fdff00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 -- 172.21.15.181:0/661377196 shutdown_connections 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.425+0000 7fdff659d640 1 -- 172.21.15.181:0/661377196 wait complete. 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.426+0000 7fdff659d640 10 monclient: build_initial_monmap 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.426+0000 7fdff659d640 10 monclient: monmap: 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:03.437 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.426+0000 7fdff659d640 1 Processor -- start 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.426+0000 7fdff659d640 1 -- start start 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.426+0000 7fdff659d640 10 monclient: init 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff00a1800 0x7fdff0140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:03.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): picked mon.noname-c con 0x7fdff00a1800 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:03.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff012ec70 0x7fdff012db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:03.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): picked mon.noname-a con 0x7fdff012ec70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:03.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 0x7fdff0117600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:03.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): picked mon.noname-b con 0x7fdff012e090 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:03.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:03.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0098830 con 0x7fdff012ec70 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff011ac70 con 0x7fdff012e090 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff009aa60 con 0x7fdff00a1800 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff659d640 10 monclient(hunting): authenticate will time out at 1167.908307s 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 0x7fdff0117600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfef7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff012ec70 msgr2=0x7fdff012db50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfef7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff012ec70 0x7fdff012db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 0x7fdff0117600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46772/0 (socket says 172.21.15.181:46772) 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 -- 172.21.15.181:0/1497983953 learned_addr learned my addr 172.21.15.181:0/1497983953 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:03.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): get_auth_request con 0x7fdff012e090 auth_method 0 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff00a1800 0x7fdff0140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): get_auth_request con 0x7fdff00a1800 auth_method 0 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:03.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdfeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient(hunting): handle_auth_done global_id 104206 payload 824 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 -- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff00a1800 msgr2=0x7fdff0140de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff00a1800 0x7fdff0140de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 -- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff012ec70 msgr2=0x7fdff012db50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff012ec70 0x7fdff012db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 monclient: found mon.noname-b 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.427+0000 7fdff4b3a640 1 -- 172.21.15.181:0/1497983953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe8009a20 con 0x7fdff012e090 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.429+0000 7fdff4b3a640 1 -- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 msgr2=0x7fdff0117600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.429+0000 7fdff4b3a640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 0x7fdff0117600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfe80099f0 tx=0x7fdfe8037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8031cd0 con 0x7fdff012e090 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: dump: 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe8040070 con 0x7fdff012e090 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8045640 con 0x7fdff012e090 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 5 monclient: authenticate success, global_id 104206 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 10 monclient: _renew_subs 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff0119650 con 0x7fdff012e090 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfee7fc640 4 set_mon_vals no callback set 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:03.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 10 monclient: dump: 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 10 monclient: _renew_subs 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff0141be0 con 0x7fdff012e090 2024-09-15T03:36:03.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.430+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdfe8042030 con 0x7fdff012e090 2024-09-15T03:36:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdfd7fff640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfdc038720 0x7fdfdc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:03.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(180..180 src has 106..180) v4 ==== 7525+0+0 (secure 0 0 0) 0x7fdfe806dff0 con 0x7fdff012e090 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdff659d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdff659d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdff659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdfeffff640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfdc038720 0x7fdfdc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff00008d0 con 0x7fdff012e090 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.431+0000 7fdfeffff640 10 monclient: get_auth_request con 0x7fdfdc038720 auth_method 0 2024-09-15T03:36:03.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.433+0000 7fdfeffff640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfdc038720 0x7fdfdc03abd0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0007fd0 tx=0x7fdfe0007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:03.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.435+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdfe803c420 con 0x7fdff012e090 2024-09-15T03:36:03.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.435+0000 7fdfd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:03.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.435+0000 7fdfd7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:03.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.595+0000 7fdff659d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:36:03.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.595+0000 7fdff659d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:36:03.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.595+0000 7fdff659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:03.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:03.595+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdff011b190 con 0x7fdff012e090 2024-09-15T03:36:04.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.239+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v181) v1 ==== 78+0+0 (secure 0 0 0) 0x7fdfe80393c0 con 0x7fdff012e090 2024-09-15T03:36:04.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.239+0000 7fdfd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:36:04.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.239+0000 7fdfd7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:04.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.339+0000 7fdff659d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:36:04.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.339+0000 7fdff659d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:36:04.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.339+0000 7fdff659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:04.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:04.339+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdff011b190 con 0x7fdff012e090 2024-09-15T03:36:05.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.238+0000 7fdfd7fff640 1 -- 172.21.15.181:0/1497983953 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v182) v1 ==== 78+0+0 (secure 0 0 0) 0x7fdfe803a3a0 con 0x7fdff012e090 2024-09-15T03:36:05.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.238+0000 7fdfd7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:36:05.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.238+0000 7fdfd7fff640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:36:05.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfdc038720 msgr2=0x7fdfdc03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfdc038720 0x7fdfdc03abd0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0007fd0 tx=0x7fdfe0007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 10 monclient: shutdown 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 msgr2=0x7fdff0117600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 0x7fdff0117600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfe80099f0 tx=0x7fdfe8037000 comp rx=0 tx=0).stop 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 shutdown_connections 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfdc038720 0x7fdfdc03abd0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdff012e090 0x7fdff0117600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff012ec70 0x7fdff012db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 --2- 172.21.15.181:0/1497983953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdff00a1800 0x7fdff0140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.241+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 >> 172.21.15.181:0/1497983953 conn(0x7fdff0071100 msgr2=0x7fdff009f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.242+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 shutdown_connections 2024-09-15T03:36:05.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.242+0000 7fdff659d640 1 -- 172.21.15.181:0/1497983953 wait complete. 2024-09-15T03:36:05.255 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set create-compat 2024-09-15T03:36:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.333+0000 7f5dfb909640 10 monclient: get_monmap_and_config 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.333+0000 7f5dfb909640 10 monclient: build_initial_monmap 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.333+0000 7f5dfb909640 10 monclient: monmap: 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:05.334740+0000 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:05.334740+0000 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:05.335 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:05.336 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:05.336 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:05.336 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:05.336 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:05.336 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:05.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.334+0000 7f5dfb909640 1 Processor -- start 2024-09-15T03:36:05.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.334+0000 7f5dfb909640 1 -- start start 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.334+0000 7f5dfb909640 10 monclient: init 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5df4131c00 0x7f5df4131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): picked mon.noname-a con 0x7f5df4131c00 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5df40a0b00 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5df4132510 0x7f5df4069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): picked mon.noname-c con 0x7f5df4132510 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:05.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df40a1b70 con 0x7f5df40a0b00 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df40a08c0 con 0x7f5df4131c00 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df4132910 con 0x7f5df4132510 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5dfb909640 10 monclient(hunting): authenticate will time out at 1169.816229s 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5df4132510 0x7f5df4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df96a5640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f5df4131c00 msgr2=0x7f5df4131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df96a5640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5df4131c00 0x7f5df4131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5df4132510 0x7f5df4069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46786/0 (socket says 172.21.15.181:46786) 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 -- 172.21.15.181:0/1744979118 learned_addr learned my addr 172.21.15.181:0/1744979118 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:05.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): get_auth_request con 0x7f5df4132510 auth_method 0 2024-09-15T03:36:05.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:05.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:05.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df8ea4640 1 --1- 172.21.15.181:0/1744979118 >> v1:172.21.15.181:6789/0 conn(0x7f5df40a0b00 0x7f5df4068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:42882/0 (socket says 172.21.15.181:42882) 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient(hunting): handle_auth_done global_id 104209 payload 274 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 -- 172.21.15.181:0/1744979118 >> v2:172.21.15.153:3300/0 conn(0x7f5df4131c00 msgr2=0x7f5df4131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 --2- 172.21.15.181:0/1744979118 >> v2:172.21.15.153:3300/0 conn(0x7f5df4131c00 0x7f5df4131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 -- 172.21.15.181:0/1744979118 >> v1:172.21.15.181:6789/0 conn(0x7f5df40a0b00 legacy=0x7f5df4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient: _finish_hunting 0 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 monclient: found mon.noname-c 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:05.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.335+0000 7f5df9ea6640 1 -- 172.21.15.181:0/1744979118 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df40a0200 con 0x7f5df4132510 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5df9ea6640 1 -- 172.21.15.181:0/1744979118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4132510 msgr2=0x7f5df4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5df9ea6640 1 --2- 172.21.15.181:0/1744979118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4132510 0x7f5df4069410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5df40a1310 tx=0x7f5dec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=522aac05165da353 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 1 -- 172.21.15.181:0/1744979118 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec02aea0 con 0x7f5df4132510 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: dump: 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:05.349 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: _finish_auth 0 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 1 -- 172.21.15.181:0/1744979118 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dec02d050 con 0x7f5df4132510 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5dfb909640 5 monclient: authenticate success, global_id 104209 2024-09-15T03:36:05.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5dfb909640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 1 -- 172.21.15.181:0/1744979118 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec007820 con 0x7f5df4132510 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5debfff640 10 monclient: dump: 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:05.351 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5dfb909640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.338+0000 7f5dfb909640 4 set_mon_vals no callback set 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 10 monclient: shutdown 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 -- 172.21.15.181:0/1744979118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4132510 msgr2=0x7f5df4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 --2- 172.21.15.181:0/1744979118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4132510 0x7f5df4069410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5df40a1310 tx=0x7f5dec009840 comp rx=0 tx=0).stop 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 -- 172.21.15.181:0/1744979118 shutdown_connections 2024-09-15T03:36:05.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 --2- 172.21.15.181:0/1744979118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4132510 0x7f5df4069410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 --2- 172.21.15.181:0/1744979118 >> v2:172.21.15.153:3300/0 conn(0x7f5df4131c00 0x7f5df4131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 -- 172.21.15.181:0/1744979118 >> 172.21.15.181:0/1744979118 conn(0x7f5df4071370 msgr2=0x7f5df40a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:05.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 -- 172.21.15.181:0/1744979118 shutdown_connections 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 1 -- 172.21.15.181:0/1744979118 wait complete. 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 10 monclient: build_initial_monmap 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.339+0000 7f5dfb909640 10 monclient: monmap: 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 Processor -- start 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 -- start start 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient: init 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df40a0b00 0x7f5df409aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): picked mon.noname-c con 0x7f5df40a0b00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 0x7f5df4130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): picked mon.noname-b con 0x7f5df4131c00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df4132510 0x7f5df4140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): picked mon.noname-a con 0x7f5df4132510 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df411e040 con 0x7f5df4132510 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df411dd60 con 0x7f5df4131c00 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df411e320 con 0x7f5df40a0b00 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5dfb909640 10 monclient(hunting): authenticate will time out at 1169.821853s 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 0x7f5df4130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 0x7f5df4130c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46802/0 (socket says 172.21.15.181:46802) 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 1 -- 172.21.15.181:0/4084445823 learned_addr learned my addr 172.21.15.181:0/4084445823 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 10 monclient(hunting): get_auth_request con 0x7f5df4131c00 auth_method 0 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df8ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.340+0000 7f5df9ea6640 1 -- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df4132510 msgr2=0x7f5df4140e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df9ea6640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df4132510 0x7f5df4140e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:05.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df96a5640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df40a0b00 0x7f5df409aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 10 monclient(hunting): handle_auth_done global_id 104212 payload 824 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 1 -- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df40a0b00 msgr2=0x7f5df409aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df40a0b00 0x7f5df409aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 1 -- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df4132510 msgr2=0x7f5df4140e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df4132510 0x7f5df4140e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 10 monclient: _finish_hunting 0 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 1 monclient: found mon.noname-b 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df8ea4640 1 -- 172.21.15.181:0/4084445823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec009ae0 con 0x7f5df4131c00 2024-09-15T03:36:05.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df96a5640 10 monclient: get_auth_request con 0x7f5df40a0b00 auth_method 0 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.341+0000 7f5df96a5640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df40a0b00 0x7f5df409aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5df8ea4640 1 -- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 msgr2=0x7f5df4130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5df8ea4640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 0x7f5df4130c40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5de400c5b0 tx=0x7f5de4007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de40075f0 con 0x7f5df4131c00 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: dump: 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:05.359 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de40057d0 con 0x7f5df4131c00 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 5 monclient: authenticate success, global_id 104212 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 10 monclient: _renew_subs 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df411c940 con 0x7f5df4131c00 2024-09-15T03:36:05.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4005b90 con 0x7f5df4131c00 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5de97fa640 10 monclient: dump: 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:05.361 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5deb7fe640 4 set_mon_vals no callback set 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 10 monclient: _renew_subs 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.343+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df411cfa0 con 0x7f5df4131c00 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.344+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5de401b400 con 0x7f5df4131c00 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.344+0000 7f5dfb909640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.344+0000 7f5dfb909640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.344+0000 7f5dfb909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.344+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df409b600 con 0x7f5df4131c00 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.345+0000 7f5de97fa640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd80386f0 0x7f5dd803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.345+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(182..182 src has 106..182) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f5de404c120 con 0x7f5df4131c00 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.345+0000 7f5df96a5640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd80386f0 0x7f5dd803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:05.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.345+0000 7f5df96a5640 10 monclient: get_auth_request con 0x7f5dd80386f0 auth_method 0 2024-09-15T03:36:05.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.347+0000 7f5df96a5640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd80386f0 0x7f5dd803aba0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5df00038f0 tx=0x7f5df0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:05.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.349+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5de401ab40 con 0x7f5df4131c00 2024-09-15T03:36:05.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.349+0000 7f5de97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:05.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.349+0000 7f5de97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:05.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.511+0000 7f5dfb909640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:36:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.511+0000 7f5dfb909640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:36:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.511+0000 7f5dfb909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:05.511+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f5df40008d0 con 0x7f5df4131c00 2024-09-15T03:36:06.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.256+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v183) v1 ==== 82+0+0 (secure 0 0 0) 0x7f5de401a360 con 0x7f5df4131c00 2024-09-15T03:36:06.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.256+0000 7f5de97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:36:06.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.256+0000 7f5de97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:06.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5dfb909640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:36:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5dfb909640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:36:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5dfb909640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f5df40a1890 con 0x7f5df4131c00 2024-09-15T03:36:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5de97fa640 1 -- 172.21.15.181:0/4084445823 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v183) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5de401a540 con 0x7f5df4131c00 2024-09-15T03:36:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5de97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-15T03:36:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.355+0000 7f5de97fa640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-15T03:36:06.358 INFO:tasks.workunit.client.0.smithi181.stderr:compat weight-set already created 2024-09-15T03:36:06.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd80386f0 msgr2=0x7f5dd803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:06.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd80386f0 0x7f5dd803aba0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5df00038f0 tx=0x7f5df0008040 comp rx=0 tx=0).stop 2024-09-15T03:36:06.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 10 monclient: shutdown 2024-09-15T03:36:06.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:06.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 msgr2=0x7f5df4130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 0x7f5df4130c40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5de400c5b0 tx=0x7f5de4007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 shutdown_connections 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd80386f0 0x7f5dd803aba0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df4132510 0x7f5df4140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df4131c00 0x7f5df4130c40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 --2- 172.21.15.181:0/4084445823 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df40a0b00 0x7f5df409aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 >> 172.21.15.181:0/4084445823 conn(0x7f5df4071370 msgr2=0x7f5df409d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 shutdown_connections 2024-09-15T03:36:06.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.357+0000 7f5dfb909640 1 -- 172.21.15.181:0/4084445823 wait complete. 2024-09-15T03:36:06.367 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-15T03:36:06.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.441+0000 7f6ea331d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:06.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.441+0000 7f6ea331d640 10 monclient: build_initial_monmap 2024-09-15T03:36:06.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.441+0000 7f6ea331d640 10 monclient: monmap: 2024-09-15T03:36:06.443 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:06.444 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:06.444 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:06.442580+0000 2024-09-15T03:36:06.444 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:06.442580+0000 2024-09-15T03:36:06.444 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:06.444 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:06.444 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:06.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:06.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:06.445 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:06.445 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:06.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:06.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 Processor -- start 2024-09-15T03:36:06.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 -- start start 2024-09-15T03:36:06.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient: init 2024-09-15T03:36:06.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:06.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T03:36:06.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6e9c0a19a0 0x7f6e9c09b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:06.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e9c0a19a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:06.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c09c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e9c09be20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e9c099250 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c09c730 con 0x7f6e9c099250 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c0a1d70 con 0x7f6e9c0a19a0 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c098600 con 0x7f6e9c09be20 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.442+0000 7f6ea331d640 10 monclient(hunting): authenticate will time out at 1170.923971s 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea10b9640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f6e9c0a19a0 msgr2=0x7f6e9c09b8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea18ba640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f6e9c099250 legacy=0x7f6e9c0996d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea10b9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6e9c0a19a0 0x7f6e9c09b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c09c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c09c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:48588/0 (socket says 172.21.15.181:48588) 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 -- 172.21.15.181:0/3591307316 learned_addr learned my addr 172.21.15.181:0/3591307316 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): get_auth_request con 0x7f6e9c09be20 auth_method 0 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_done global_id 74744 payload 274 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 -- 172.21.15.181:0/3591307316 >> v2:172.21.15.153:3300/0 conn(0x7f6e9c0a19a0 msgr2=0x7f6e9c09b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 --2- 172.21.15.181:0/3591307316 >> v2:172.21.15.153:3300/0 conn(0x7f6e9c0a19a0 0x7f6e9c09b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 -- 172.21.15.181:0/3591307316 >> v1:172.21.15.153:6789/0 conn(0x7f6e9c099250 legacy=0x7f6e9c0996d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient: _finish_hunting 0 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 monclient: found mon.noname-b 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.443+0000 7f6ea08b8640 1 -- 172.21.15.181:0/3591307316 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c097a70 con 0x7f6e9c09be20 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.446+0000 7f6ea08b8640 1 -- 172.21.15.181:0/3591307316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 msgr2=0x7f6e9c09c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.446+0000 7f6ea08b8640 1 --2- 172.21.15.181:0/3591307316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c09c1f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c09c8c0 tx=0x7f6e8c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39374e4fd710511d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.446+0000 7f6e93fff640 1 -- 172.21.15.181:0/3591307316 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c032440 con 0x7f6e9c09be20 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.446+0000 7f6e93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: dump: 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 5 monclient: authenticate success, global_id 74744 2024-09-15T03:36:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 1 -- 172.21.15.181:0/3591307316 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c0325a0 con 0x7f6e9c09be20 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 4 set_mon_vals no callback set 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 1 -- 172.21.15.181:0/3591307316 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c032940 con 0x7f6e9c09be20 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6e93fff640 10 monclient: dump: 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 10 monclient: shutdown 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 -- 172.21.15.181:0/3591307316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 msgr2=0x7f6e9c09c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 --2- 172.21.15.181:0/3591307316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c09c1f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c09c8c0 tx=0x7f6e8c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 -- 172.21.15.181:0/3591307316 shutdown_connections 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 --2- 172.21.15.181:0/3591307316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c09c1f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 --2- 172.21.15.181:0/3591307316 >> v2:172.21.15.153:3300/0 conn(0x7f6e9c0a19a0 0x7f6e9c09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 -- 172.21.15.181:0/3591307316 >> 172.21.15.181:0/3591307316 conn(0x7f6e9c0711a0 msgr2=0x7f6e9c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 -- 172.21.15.181:0/3591307316 shutdown_connections 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.447+0000 7f6ea331d640 1 -- 172.21.15.181:0/3591307316 wait complete. 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.448+0000 7f6ea331d640 10 monclient: build_initial_monmap 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.448+0000 7f6ea331d640 10 monclient: monmap: 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:06.463 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:06.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.448+0000 7f6ea331d640 1 Processor -- start 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.448+0000 7f6ea331d640 1 -- start start 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.448+0000 7f6ea331d640 10 monclient: init 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e9c099250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c12eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e9c09be20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e9c0a19a0 0x7f6e9c118630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e9c0a19a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c09cc80 con 0x7f6e9c0a19a0 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c130840 con 0x7f6e9c099250 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c120040 con 0x7f6e9c09be20 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea331d640 10 monclient(hunting): authenticate will time out at 1170.930191s 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea18ba640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e9c0a19a0 msgr2=0x7f6e9c118630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea18ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e9c0a19a0 0x7f6e9c118630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:46820/0 (socket says 172.21.15.181:46820) 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 -- 172.21.15.181:0/2879225909 learned_addr learned my addr 172.21.15.181:0/2879225909 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): get_auth_request con 0x7f6e9c099250 auth_method 0 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:06.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c12eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): get_auth_request con 0x7f6e9c09be20 auth_method 0 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea08b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:06.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient(hunting): handle_auth_done global_id 104215 payload 824 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 -- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 msgr2=0x7f6e9c12eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c12eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 -- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e9c0a19a0 msgr2=0x7f6e9c118630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e9c0a19a0 0x7f6e9c118630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient: _finish_hunting 0 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 monclient: found mon.noname-b 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:06.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.449+0000 7f6ea10b9640 1 -- 172.21.15.181:0/2879225909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c009450 con 0x7f6e9c099250 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6ea10b9640 1 -- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 msgr2=0x7f6e9c141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6ea10b9640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b600 tx=0x7f6e980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98015070 con 0x7f6e9c099250 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 10 monclient: dump: 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:06.470 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:06.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98003d90 con 0x7f6e9c099250 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6ea331d640 5 monclient: authenticate success, global_id 104215 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6ea331d640 10 monclient: _renew_subs 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6ea331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.451+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c120320 con 0x7f6e9c099250 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e917fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98006380 con 0x7f6e9c099250 2024-09-15T03:36:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e917fa640 10 monclient: dump: 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:06.473 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6ea331d640 10 monclient: _renew_subs 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e937fe640 4 set_mon_vals no callback set 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6ea331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c11f160 con 0x7f6e9c099250 2024-09-15T03:36:06.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6e98010440 con 0x7f6e9c099250 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6ea331d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.452+0000 7f6ea331d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.453+0000 7f6ea331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.453+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c0008d0 con 0x7f6e9c099250 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.453+0000 7f6e917fa640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6e800386f0 0x7f6e8003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.453+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(183..183 src has 106..183) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f6e9804c0f0 con 0x7f6e9c099250 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.453+0000 7f6ea08b8640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6e800386f0 0x7f6e8003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.453+0000 7f6ea08b8640 10 monclient: get_auth_request con 0x7f6e800386f0 auth_method 0 2024-09-15T03:36:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.455+0000 7f6ea08b8640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6e800386f0 0x7f6e8003aba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c09cf50 tx=0x7f6e8c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:06.457+0000 7f6ea10b9640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b600 tx=0x7f6e980091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f6e98018a70 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:36:07.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.285+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6e98018a70 con 0x7f6e9c099250 2024-09-15T03:36:07.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.285+0000 7f6e917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:07.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.285+0000 7f6e917fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.446+0000 7f6ea331d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T03:36:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.446+0000 7f6ea331d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T03:36:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.446+0000 7f6ea331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:07.446+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f6e9c097cb0 con 0x7f6e9c099250 2024-09-15T03:36:08.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.264+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v184) v1 ==== 176+0+0 (secure 0 0 0) 0x7f6e98018a70 con 0x7f6e9c099250 2024-09-15T03:36:08.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.264+0000 7f6e917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T03:36:08.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.264+0000 7f6e917fa640 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-15T03:36:08.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.363+0000 7f6ea331d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T03:36:08.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.363+0000 7f6ea331d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T03:36:08.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.363+0000 7f6ea331d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.363+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f6e9c064a20 con 0x7f6e9c099250 2024-09-15T03:36:08.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.364+0000 7f6e917fa640 1 -- 172.21.15.181:0/2879225909 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v184) v1 ==== 150+0+0 (secure 0 0 0) 0x7f6e98018a70 con 0x7f6e9c099250 2024-09-15T03:36:08.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.364+0000 7f6e917fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-15T03:36:08.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.364+0000 7f6e917fa640 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-15T03:36:08.368 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'r1' already exists 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.365+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6e800386f0 msgr2=0x7f6e8003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.365+0000 7f6ea331d640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6e800386f0 0x7f6e8003aba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c09cf50 tx=0x7f6e8c032000 comp rx=0 tx=0).stop 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.365+0000 7f6ea331d640 10 monclient: shutdown 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.365+0000 7f6ea331d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.365+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 msgr2=0x7f6e9c141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.365+0000 7f6ea331d640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b600 tx=0x7f6e980091a0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 shutdown_connections 2024-09-15T03:36:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6e800386f0 0x7f6e8003aba0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e9c0a19a0 0x7f6e9c118630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6e9c09be20 0x7f6e9c12eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 --2- 172.21.15.181:0/2879225909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e9c099250 0x7f6e9c141c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 >> 172.21.15.181:0/2879225909 conn(0x7f6e9c0711a0 msgr2=0x7f6e9c12fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 shutdown_connections 2024-09-15T03:36:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.366+0000 7f6ea331d640 1 -- 172.21.15.181:0/2879225909 wait complete. 2024-09-15T03:36:08.376 INFO:tasks.workunit.client.0.smithi181.stderr:++ seq 1 32 2024-09-15T03:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-15T03:36:08.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.535+0000 7fb74e99d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.535+0000 7fb74e99d640 10 monclient: build_initial_monmap 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.535+0000 7fb74e99d640 10 monclient: monmap: 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:08.536566+0000 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:08.536566+0000 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:08.537 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:08.538 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.536+0000 7fb74e99d640 1 Processor -- start 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.536+0000 7fb74e99d640 1 -- start start 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.536+0000 7fb74e99d640 10 monclient: init 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.536+0000 7fb74e99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.536+0000 7fb74e99d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb748069a30 0x7fb7481320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): picked mon.noname-c con 0x7fb748069a30 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7480983e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb748136e90 0x7fb748139270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): picked mon.noname-a con 0x7fb748136e90 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74809cd00 con 0x7fb7480983e0 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb748069e00 con 0x7fb748136e90 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74809d140 con 0x7fb748069a30 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74e99d640 10 monclient(hunting): authenticate will time out at 1173.018088s 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb7477fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fb7480983e0 legacy=0x7fb748098840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb748069a30 0x7fb7481320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb748069a30 0x7fb7481320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48784/0 (socket says 172.21.15.181:48784) 2024-09-15T03:36:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74cf3a640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fb748136e90 msgr2=0x7fb748139270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 -- 172.21.15.181:0/3677206162 learned_addr learned my addr 172.21.15.181:0/3677206162 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): get_auth_request con 0x7fb748069a30 auth_method 0 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb74cf3a640 1 --2- 172.21.15.181:0/3677206162 >> v2:172.21.15.153:3300/0 conn(0x7fb748136e90 0x7fb748139270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:08.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient(hunting): handle_auth_done global_id 104218 payload 274 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 -- 172.21.15.181:0/3677206162 >> v2:172.21.15.153:3300/0 conn(0x7fb748136e90 msgr2=0x7fb748139270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 --2- 172.21.15.181:0/3677206162 >> v2:172.21.15.153:3300/0 conn(0x7fb748136e90 0x7fb748139270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 -- 172.21.15.181:0/3677206162 >> v1:172.21.15.153:6789/0 conn(0x7fb7480983e0 legacy=0x7fb748098840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 monclient: found mon.noname-c 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.537+0000 7fb747fff640 1 -- 172.21.15.181:0/3677206162 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74809e320 con 0x7fb748069a30 2024-09-15T03:36:08.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb747fff640 1 -- 172.21.15.181:0/3677206162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 msgr2=0x7fb7481320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb747fff640 1 --2- 172.21.15.181:0/3677206162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb7481320e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb738009910 tx=0x7fb7380091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c10459cd055d199 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 1 -- 172.21.15.181:0/3677206162 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738032440 con 0x7fb748069a30 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: dump: 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.550 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 1 -- 172.21.15.181:0/3677206162 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7380325a0 con 0x7fb748069a30 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 1 -- 172.21.15.181:0/3677206162 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738032940 con 0x7fb748069a30 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:08.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb746ffd640 10 monclient: dump: 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 5 monclient: authenticate success, global_id 104218 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:08.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 4 set_mon_vals no callback set 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 10 monclient: shutdown 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 -- 172.21.15.181:0/3677206162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 msgr2=0x7fb7481320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3677206162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb7481320e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb738009910 tx=0x7fb7380091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 -- 172.21.15.181:0/3677206162 shutdown_connections 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3677206162 >> v2:172.21.15.153:3300/0 conn(0x7fb748136e90 0x7fb748139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3677206162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb7481320e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 -- 172.21.15.181:0/3677206162 >> 172.21.15.181:0/3677206162 conn(0x7fb74809e9f0 msgr2=0x7fb74809edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 -- 172.21.15.181:0/3677206162 shutdown_connections 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.540+0000 7fb74e99d640 1 -- 172.21.15.181:0/3677206162 wait complete. 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.541+0000 7fb74e99d640 10 monclient: build_initial_monmap 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.541+0000 7fb74e99d640 10 monclient: monmap: 2024-09-15T03:36:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.541+0000 7fb74e99d640 1 Processor -- start 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.541+0000 7fb74e99d640 1 -- start start 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.541+0000 7fb74e99d640 10 monclient: init 2024-09-15T03:36:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): picked mon.noname-b con 0x7fb748069a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7480983e0 0x7fb74812d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7480983e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb748136e90 0x7fb748134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): picked mon.noname-a con 0x7fb748136e90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7480a0850 con 0x7fb748136e90 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74809d8a0 con 0x7fb748069a30 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb748118b00 con 0x7fb7480983e0 2024-09-15T03:36:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74e99d640 10 monclient(hunting): authenticate will time out at 1173.023467s 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48792/0 (socket says 172.21.15.181:48792) 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 -- 172.21.15.181:0/3277777942 learned_addr learned my addr 172.21.15.181:0/3277777942 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): get_auth_request con 0x7fb748069a30 auth_method 0 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74cf3a640 1 -- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb748136e90 msgr2=0x7fb748134bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb74cf3a640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb748136e90 0x7fb748134bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient(hunting): handle_auth_done global_id 104221 payload 824 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 -- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7480983e0 msgr2=0x7fb74812d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7480983e0 0x7fb74812d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 -- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb748136e90 msgr2=0x7fb748134bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb748136e90 0x7fb748134bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 monclient: found mon.noname-b 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.542+0000 7fb747fff640 1 -- 172.21.15.181:0/3277777942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738009450 con 0x7fb748069a30 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb747fff640 1 -- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 msgr2=0x7fb748141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb747fff640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb738035040 tx=0x7fb738032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738031d50 con 0x7fb748069a30 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: dump: 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: _finish_auth 0 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73803c470 con 0x7fb748069a30 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 5 monclient: authenticate success, global_id 104221 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 10 monclient: _renew_subs 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb748118680 con 0x7fb748069a30 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73803c830 con 0x7fb748069a30 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb72ffff640 10 monclient: dump: 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb7467fc640 4 set_mon_vals no callback set 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 10 monclient: _renew_subs 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.545+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb748142230 con 0x7fb748069a30 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb738039040 con 0x7fb748069a30 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb74e99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb74e99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb74e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74804bf60 con 0x7fb748069a30 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb72ffff640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb734038720 0x7fb73403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(184..184 src has 106..184) v4 ==== 7735+0+0 (secure 0 0 0) 0x7fb73806d390 con 0x7fb748069a30 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb7477fe640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb734038720 0x7fb73403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.546+0000 7fb7477fe640 10 monclient: get_auth_request con 0x7fb734038720 auth_method 0 2024-09-15T03:36:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.548+0000 7fb7477fe640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb734038720 0x7fb73403abd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb73c006fd0 tx=0x7fb73c006cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.550+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb738044720 con 0x7fb748069a30 2024-09-15T03:36:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.550+0000 7fb72ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.550+0000 7fb72ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:08.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.708+0000 7fb74e99d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:08.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.708+0000 7fb74e99d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:08.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.708+0000 7fb74e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:08.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:08.708+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb74809e600 con 0x7fb748069a30 2024-09-15T03:36:09.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.264+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v185) v1 ==== 166+0+0 (secure 0 0 0) 0x7fb73804a020 con 0x7fb748069a30 2024-09-15T03:36:09.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.264+0000 7fb72ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:09.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.264+0000 7fb72ffff640 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-15T03:36:09.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.362+0000 7fb74e99d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:09.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.362+0000 7fb74e99d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:09.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.362+0000 7fb74e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:09.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.362+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb74809e600 con 0x7fb748069a30 2024-09-15T03:36:09.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:09.362+0000 7fb747fff640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb738035040 tx=0x7fb738032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb738040020 mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v185) v1 2024-09-15T03:36:10.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.190+0000 7fb72ffff640 1 -- 172.21.15.181:0/3277777942 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v185) v1 ==== 145+0+0 (secure 0 0 0) 0x7fb738040020 con 0x7fb748069a30 2024-09-15T03:36:10.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.190+0000 7fb72ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:10.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.190+0000 7fb72ffff640 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-15T03:36:10.192 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h1' already exists 2024-09-15T03:36:10.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb734038720 msgr2=0x7fb73403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:10.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb734038720 0x7fb73403abd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb73c006fd0 tx=0x7fb73c006cf0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 10 monclient: shutdown 2024-09-15T03:36:10.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:10.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 msgr2=0x7fb748141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:10.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb738035040 tx=0x7fb738032000 comp rx=0 tx=0).stop 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 shutdown_connections 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb734038720 0x7fb73403abd0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb748136e90 0x7fb748134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7480983e0 0x7fb74812d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 --2- 172.21.15.181:0/3277777942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb748069a30 0x7fb748141470 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 >> 172.21.15.181:0/3277777942 conn(0x7fb74809e9f0 msgr2=0x7fb7480a0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 shutdown_connections 2024-09-15T03:36:10.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.193+0000 7fb74e99d640 1 -- 172.21.15.181:0/3277777942 wait complete. 2024-09-15T03:36:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-15T03:36:10.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.284+0000 7f063760f640 10 monclient: get_monmap_and_config 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.284+0000 7f063760f640 10 monclient: build_initial_monmap 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.285+0000 7f063760f640 10 monclient: monmap: 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:10.285777+0000 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:10.285777+0000 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:10.286 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:10.287 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:10.287 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.285+0000 7f063760f640 1 Processor -- start 2024-09-15T03:36:10.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.285+0000 7f063760f640 1 -- start start 2024-09-15T03:36:10.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.285+0000 7f063760f640 10 monclient: init 2024-09-15T03:36:10.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:10.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0630131bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0630069e10 0x7f0630131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient(hunting): picked mon.noname-a con 0x7f0630069e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0630136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300a06d0 con 0x7f0630131bd0 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630071780 con 0x7f0630136d60 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f063006a210 con 0x7f0630069e10 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f063760f640 10 monclient(hunting): authenticate will time out at 1174.767275s 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f0635bac640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f0630136d60 0x7f0630139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:53620/0 (socket says 172.21.15.181:53620) 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f0635bac640 1 -- 172.21.15.181:0/351297464 learned_addr learned my addr 172.21.15.181:0/351297464 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f06353ab640 1 -- 172.21.15.181:0/351297464 >> v1:172.21.15.153:6789/0 conn(0x7f0630131bd0 legacy=0x7f063009ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:10.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f0634baa640 1 -- 172.21.15.181:0/351297464 >> v2:172.21.15.153:3300/0 conn(0x7f0630069e10 msgr2=0x7f0630131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:10.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.286+0000 7f0634baa640 1 --2- 172.21.15.181:0/351297464 >> v2:172.21.15.153:3300/0 conn(0x7f0630069e10 0x7f0630131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f0635bac640 1 -- 172.21.15.181:0/351297464 >> v1:172.21.15.181:6789/0 conn(0x7f0630136d60 legacy=0x7f0630139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731049110 0 0) 0x7f0630071780 con 0x7f0630136d60 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient(hunting): my global_id is 104224 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06180034e0 con 0x7f0630136d60 2024-09-15T03:36:10.294 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2758419150 0 0) 0x7f06180034e0 con 0x7f0630136d60 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 >> v2:172.21.15.153:3300/0 conn(0x7f0630069e10 msgr2=0x7f0630131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 --2- 172.21.15.181:0/351297464 >> v2:172.21.15.153:3300/0 conn(0x7f0630069e10 0x7f0630131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 >> v1:172.21.15.153:6789/0 conn(0x7f0630131bd0 legacy=0x7f063009ebe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06300a0010 con 0x7f0630136d60 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: _finish_auth 0 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f0620001b60 con 0x7f0630136d60 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f063760f640 5 monclient: authenticate success, global_id 104224 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f063760f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:10.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: dump: 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 >> v1:172.21.15.181:6789/0 conn(0x7f0630136d60 legacy=0x7f0630139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.288+0000 7f062ffff640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06180039d0 0x7f0618003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): picked mon.a con 0x7f06180039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 0x7f0618004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): picked mon.b con 0x7f0618004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618005330 0x7f061800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): picked mon.c con 0x7f0618005330 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063006a210 con 0x7f06180039d0 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061800c290 con 0x7f0618004780 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061800c450 con 0x7f0618005330 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f062ffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f063760f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f06353ab640 1 -- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06180039d0 msgr2=0x7f0618003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 0x7f0618004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f06353ab640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06180039d0 0x7f0618003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): get_auth_request con 0x7f0618004780 auth_method 0 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient(hunting): handle_auth_done global_id 104224 payload 293 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 -- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618005330 msgr2=0x7f061800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618005330 0x7f061800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 -- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06180039d0 msgr2=0x7f0618003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06180039d0 0x7f0618003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient: _finish_hunting 0 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 monclient: found mon.b 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 1 -- 172.21.15.181:0/351297464 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180034e0 con 0x7f0618004780 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0635bac640 1 -- 172.21.15.181:0/351297464 reap_dead start 2024-09-15T03:36:10.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 10 monclient: _finish_auth 0 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.289+0000 7f0634baa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f0634baa640 1 -- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 msgr2=0x7f0618004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f0634baa640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 0x7f0618004bf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f06240098f0 tx=0x7f0624006ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=736a0d6fa189af5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624003b80 con 0x7f0618004780 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: dump: 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:10.300 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624003ce0 con 0x7f0618004780 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 1 -- 172.21.15.181:0/351297464 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062402aac0 con 0x7f0618004780 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 4 set_mon_vals no callback set 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f062ffff640 10 monclient: dump: 2024-09-15T03:36:10.301 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 10 monclient: shutdown 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 1 -- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 msgr2=0x7f0618004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 0x7f0618004bf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f06240098f0 tx=0x7f0624006ed0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 1 -- 172.21.15.181:0/351297464 shutdown_connections 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 1 --2- 172.21.15.181:0/351297464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0618004780 0x7f0618004bf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 1 -- 172.21.15.181:0/351297464 >> 172.21.15.181:0/351297464 conn(0x7f06300713b0 msgr2=0x7f06300a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.292+0000 7f063760f640 1 -- 172.21.15.181:0/351297464 shutdown_connections 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.293+0000 7f063760f640 1 -- 172.21.15.181:0/351297464 wait complete. 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.293+0000 7f063760f640 10 monclient: build_initial_monmap 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.293+0000 7f063760f640 10 monclient: monmap: 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:10.303 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:10.304 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:10.304 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.293+0000 7f063760f640 1 Processor -- start 2024-09-15T03:36:10.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.293+0000 7f063760f640 1 -- start start 2024-09-15T03:36:10.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.293+0000 7f063760f640 10 monclient: init 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618004780 0x7f0630130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): picked mon.noname-c con 0x7f0618004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630131020 0x7f0630099960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): picked mon.noname-a con 0x7f0630131020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 0x7f063009a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): picked mon.noname-b con 0x7f0630099ea0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063011cbf0 con 0x7f0630131020 2024-09-15T03:36:10.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063009bf30 con 0x7f0630099ea0 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0630142030 con 0x7f0618004780 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f063760f640 10 monclient(hunting): authenticate will time out at 1174.775476s 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 0x7f063009a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0634baa640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630131020 msgr2=0x7f0630099960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 0x7f063009a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48812/0 (socket says 172.21.15.181:48812) 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 -- 172.21.15.181:0/3589669647 learned_addr learned my addr 172.21.15.181:0/3589669647 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): get_auth_request con 0x7f0630099ea0 auth_method 0 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0634baa640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630131020 0x7f0630099960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:10.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f06353ab640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618004780 0x7f0630130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f06353ab640 10 monclient(hunting): get_auth_request con 0x7f0618004780 auth_method 0 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f06353ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f06353ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f06353ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient(hunting): handle_auth_done global_id 104227 payload 824 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 -- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618004780 msgr2=0x7f0630130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618004780 0x7f0630130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 -- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630131020 msgr2=0x7f0630099960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630131020 0x7f0630099960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient: _finish_hunting 0 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 monclient: found mon.noname-b 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.294+0000 7f0635bac640 1 -- 172.21.15.181:0/3589669647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630131810 con 0x7f0630099ea0 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f0635bac640 1 -- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 msgr2=0x7f063009a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f0635bac640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 0x7f063009a2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f06200017a0 tx=0x7f063006a350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0620004d30 con 0x7f0630099ea0 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: dump: 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:10.311 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0620007190 con 0x7f0630099ea0 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 5 monclient: authenticate success, global_id 104227 2024-09-15T03:36:10.312 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 10 monclient: _renew_subs 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06300a4960 con 0x7f0630099ea0 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0620007550 con 0x7f0630099ea0 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062d7fa640 10 monclient: dump: 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:10.313 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 10 monclient: _renew_subs 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063009b4f0 con 0x7f0630099ea0 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.297+0000 7f062f7fe640 4 set_mon_vals no callback set 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0620014460 con 0x7f0630099ea0 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f063760f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f063760f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f063760f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f062d7fa640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f061c0386f0 0x7f061c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06301369d0 con 0x7f0630099ea0 2024-09-15T03:36:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(185..185 src has 106..185) v4 ==== 7945+0+0 (secure 0 0 0) 0x7f062004b9f0 con 0x7f0630099ea0 2024-09-15T03:36:10.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f06353ab640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f061c0386f0 0x7f061c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:10.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.298+0000 7f06353ab640 10 monclient: get_auth_request con 0x7f061c0386f0 auth_method 0 2024-09-15T03:36:10.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.301+0000 7f06353ab640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f061c0386f0 0x7f061c03aba0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f06280038f0 tx=0x7f0628008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:10.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.302+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f06301369d0 con 0x7f0630099ea0 2024-09-15T03:36:10.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.302+0000 7f062d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:10.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.302+0000 7f062d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:10.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.458+0000 7f063760f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:10.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.458+0000 7f063760f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:10.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.458+0000 7f063760f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:10.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:10.458+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f06301369d0 con 0x7f0630099ea0 2024-09-15T03:36:11.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.219+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v186) v1 ==== 166+0+0 (secure 0 0 0) 0x7f06301369d0 con 0x7f0630099ea0 2024-09-15T03:36:11.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.219+0000 7f062d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:11.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.219+0000 7f062d7fa640 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.318+0000 7f063760f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.318+0000 7f063760f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.318+0000 7f063760f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.318+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f06301369d0 con 0x7f0630099ea0 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.319+0000 7f062d7fa640 1 -- 172.21.15.181:0/3589669647 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v186) v1 ==== 145+0+0 (secure 0 0 0) 0x7f06301369d0 con 0x7f0630099ea0 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.319+0000 7f062d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.319+0000 7f062d7fa640 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-15T03:36:11.320 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h2' already exists 2024-09-15T03:36:11.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f061c0386f0 msgr2=0x7f061c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:11.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f061c0386f0 0x7f061c03aba0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f06280038f0 tx=0x7f0628008040 comp rx=0 tx=0).stop 2024-09-15T03:36:11.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 10 monclient: shutdown 2024-09-15T03:36:11.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:11.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 msgr2=0x7f063009a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 0x7f063009a2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f06200017a0 tx=0x7f063006a350 comp rx=0 tx=0).stop 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 shutdown_connections 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f061c0386f0 0x7f061c03aba0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0630099ea0 0x7f063009a2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630131020 0x7f0630099960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 --2- 172.21.15.181:0/3589669647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0618004780 0x7f0630130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 >> 172.21.15.181:0/3589669647 conn(0x7f06300713b0 msgr2=0x7f063009e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 shutdown_connections 2024-09-15T03:36:11.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.321+0000 7f063760f640 1 -- 172.21.15.181:0/3589669647 wait complete. 2024-09-15T03:36:11.330 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:11.330 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-15T03:36:11.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.404+0000 7f4ce34c9640 10 monclient: get_monmap_and_config 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.404+0000 7f4ce34c9640 10 monclient: build_initial_monmap 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.404+0000 7f4ce34c9640 10 monclient: monmap: 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:11.405684+0000 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:11.405684+0000 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:11.406 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:11.407 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:11.407 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:11.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.405+0000 7f4ce34c9640 1 Processor -- start 2024-09-15T03:36:11.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.405+0000 7f4ce34c9640 1 -- start start 2024-09-15T03:36:11.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.405+0000 7f4ce34c9640 10 monclient: init 2024-09-15T03:36:11.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:11.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T03:36:11.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 0x7f4cdc06a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4cdc098f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4cdc0992d0 0x7f4cdc09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4cdc0992d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cdc09e010 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cdc09a060 con 0x7f4cdc09e010 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cdc09f750 con 0x7f4cdc0992d0 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc069f00 con 0x7f4cdc098f00 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce34c9640 10 monclient(hunting): authenticate will time out at 1175.887187s 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce0a64640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f4cdc0992d0 msgr2=0x7f4cdc09dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1a66640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f4cdc09e010 legacy=0x7f4cdc09e470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce0a64640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4cdc0992d0 0x7f4cdc09dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 0x7f4cdc06a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 0x7f4cdc06a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:47058/0 (socket says 172.21.15.181:47058) 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 -- 172.21.15.181:0/284829005 learned_addr learned my addr 172.21.15.181:0/284829005 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): get_auth_request con 0x7f4cdc098f00 auth_method 0 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient(hunting): handle_auth_done global_id 74753 payload 274 2024-09-15T03:36:11.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 -- 172.21.15.181:0/284829005 >> v2:172.21.15.153:3300/0 conn(0x7f4cdc0992d0 msgr2=0x7f4cdc09dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 --2- 172.21.15.181:0/284829005 >> v2:172.21.15.153:3300/0 conn(0x7f4cdc0992d0 0x7f4cdc09dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 -- 172.21.15.181:0/284829005 >> v1:172.21.15.153:6789/0 conn(0x7f4cdc09e010 legacy=0x7f4cdc09e470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient: _finish_hunting 0 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 monclient: found mon.noname-b 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:11.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.406+0000 7f4ce1265640 1 -- 172.21.15.181:0/284829005 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc09f430 con 0x7f4cdc098f00 2024-09-15T03:36:11.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.409+0000 7f4ce1265640 1 -- 172.21.15.181:0/284829005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 msgr2=0x7f4cdc06a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:11.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.409+0000 7f4ce1265640 1 --2- 172.21.15.181:0/284829005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 0x7f4cdc06a600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009f20 tx=0x7f4cd8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a69663351c4a49a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:11.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 1 -- 172.21.15.181:0/284829005 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8007610 con 0x7f4cdc098f00 2024-09-15T03:36:11.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: dump: 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:11.419 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 1 -- 172.21.15.181:0/284829005 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd802d050 con 0x7f4cdc098f00 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 5 monclient: authenticate success, global_id 74753 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 1 -- 172.21.15.181:0/284829005 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80079a0 con 0x7f4cdc098f00 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 4 set_mon_vals no callback set 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4cd3fff640 10 monclient: dump: 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:11.420 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 10 monclient: shutdown 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 1 -- 172.21.15.181:0/284829005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 msgr2=0x7f4cdc06a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:11.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/284829005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 0x7f4cdc06a600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009f20 tx=0x7f4cd8009840 comp rx=0 tx=0).stop 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 1 -- 172.21.15.181:0/284829005 shutdown_connections 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/284829005 >> v2:172.21.15.153:3300/0 conn(0x7f4cdc0992d0 0x7f4cdc09dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/284829005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc098f00 0x7f4cdc06a600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.410+0000 7f4ce34c9640 1 -- 172.21.15.181:0/284829005 >> 172.21.15.181:0/284829005 conn(0x7f4cdc0a3b30 msgr2=0x7f4cdc09fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 1 -- 172.21.15.181:0/284829005 shutdown_connections 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 1 -- 172.21.15.181:0/284829005 wait complete. 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 10 monclient: build_initial_monmap 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 10 monclient: monmap: 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:11.422 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 1 Processor -- start 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 1 -- start start 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.411+0000 7f4ce34c9640 10 monclient: init 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 0x7f4cdc13db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4cdc098f00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cdc0992d0 0x7f4cdc13d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:11.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4cdc0992d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc09e010 0x7f4cdc141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4cdc09e010 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc122350 con 0x7f4cdc0992d0 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc13e610 con 0x7f4cdc098f00 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc1225c0 con 0x7f4cdc09e010 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce34c9640 10 monclient(hunting): authenticate will time out at 1175.893391s 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce0a64640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cdc0992d0 msgr2=0x7f4cdc13d240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 0x7f4cdc13db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:11.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce0a64640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cdc0992d0 0x7f4cdc13d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 0x7f4cdc13db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48818/0 (socket says 172.21.15.181:48818) 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 -- 172.21.15.181:0/725657089 learned_addr learned my addr 172.21.15.181:0/725657089 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): get_auth_request con 0x7f4cdc098f00 auth_method 0 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc09e010 0x7f4cdc141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): get_auth_request con 0x7f4cdc09e010 auth_method 0 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:11.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient(hunting): handle_auth_done global_id 104230 payload 824 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 -- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc09e010 msgr2=0x7f4cdc141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc09e010 0x7f4cdc141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 -- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cdc0992d0 msgr2=0x7f4cdc13d240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cdc0992d0 0x7f4cdc13d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient: _finish_hunting 0 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 monclient: found mon.noname-b 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:11.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.412+0000 7f4ce1265640 1 -- 172.21.15.181:0/725657089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8009ae0 con 0x7f4cdc098f00 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.414+0000 7f4ce1265640 1 -- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 msgr2=0x7f4cdc13db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.414+0000 7f4ce1265640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 0x7f4cdc13db90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009ab0 tx=0x7f4cd8036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8031aa0 con 0x7f4cdc098f00 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: dump: 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 5 monclient: authenticate success, global_id 104230 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd8031c00 con 0x7f4cdc098f00 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd803c420 con 0x7f4cdc098f00 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd17fa640 10 monclient: dump: 2024-09-15T03:36:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 10 monclient: _renew_subs 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4cd2ffd640 4 set_mon_vals no callback set 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc122b80 con 0x7f4cdc098f00 2024-09-15T03:36:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 10 monclient: _renew_subs 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.415+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc121410 con 0x7f4cdc098f00 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4cd803c7d0 con 0x7f4cdc098f00 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4ce34c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4ce34c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4ce34c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc121600 con 0x7f4cdc098f00 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4cd17fa640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc00386f0 0x7f4cc003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(186..186 src has 106..186) v4 ==== 8155+0+0 (secure 0 0 0) 0x7f4cd806d390 con 0x7f4cdc098f00 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4ce0a64640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc00386f0 0x7f4cc003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.416+0000 7f4ce0a64640 10 monclient: get_auth_request con 0x7f4cc00386f0 auth_method 0 2024-09-15T03:36:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.418+0000 7f4ce0a64640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc00386f0 0x7f4cc003aba0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0038f0 tx=0x7f4ccc006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.420+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4cd8039c90 con 0x7f4cdc098f00 2024-09-15T03:36:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.420+0000 7f4cd17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.420+0000 7f4cd17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:11.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.575+0000 7f4ce34c9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:11.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.576+0000 7f4ce34c9640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:11.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.576+0000 7f4ce34c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:11.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:11.576+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4cdc06eb30 con 0x7f4cdc098f00 2024-09-15T03:36:12.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.278+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v187) v1 ==== 166+0+0 (secure 0 0 0) 0x7f4cd80400b0 con 0x7f4cdc098f00 2024-09-15T03:36:12.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.278+0000 7f4cd17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:12.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.278+0000 7f4cd17fa640 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-15T03:36:12.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4ce34c9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:12.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4ce34c9640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:12.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4ce34c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:12.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4cdc04bf60 con 0x7f4cdc098f00 2024-09-15T03:36:12.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4cd17fa640 1 -- 172.21.15.181:0/725657089 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v187) v1 ==== 145+0+0 (secure 0 0 0) 0x7f4cd8049020 con 0x7f4cdc098f00 2024-09-15T03:36:12.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4cd17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:12.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.377+0000 7f4cd17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-15T03:36:12.379 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h3' already exists 2024-09-15T03:36:12.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc00386f0 msgr2=0x7f4cc003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc00386f0 0x7f4cc003aba0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0038f0 tx=0x7f4ccc006d80 comp rx=0 tx=0).stop 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 10 monclient: shutdown 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 msgr2=0x7f4cdc13db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 0x7f4cdc13db90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009ab0 tx=0x7f4cd8036000 comp rx=0 tx=0).stop 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 shutdown_connections 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4cc00386f0 0x7f4cc003aba0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cdc09e010 0x7f4cdc141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cdc0992d0 0x7f4cdc13d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 --2- 172.21.15.181:0/725657089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cdc098f00 0x7f4cdc13db90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 >> 172.21.15.181:0/725657089 conn(0x7f4cdc0a3b30 msgr2=0x7f4cdc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:12.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 shutdown_connections 2024-09-15T03:36:12.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.379+0000 7f4ce34c9640 1 -- 172.21.15.181:0/725657089 wait complete. 2024-09-15T03:36:12.389 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:12.390 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.463+0000 7f6d4ec8b640 10 monclient: get_monmap_and_config 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.463+0000 7f6d4ec8b640 10 monclient: build_initial_monmap 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.463+0000 7f6d4ec8b640 10 monclient: monmap: 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:12.463930+0000 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:12.463930+0000 2024-09-15T03:36:12.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:12.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.463+0000 7f6d4ec8b640 1 Processor -- start 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.463+0000 7f6d4ec8b640 1 -- start start 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.463+0000 7f6d4ec8b640 10 monclient: init 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 0x7f6d4809a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4809c2b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6d4809c680 0x7f6d4809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4809c680 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d4809d980 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4809bb40 con 0x7f6d4809d980 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:12.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d48099f60 con 0x7f6d4809c680 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4809f700 con 0x7f6d4809c2b0 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ec8b640 10 monclient(hunting): authenticate will time out at 1176.945395s 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4d228640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f6d4809d980 legacy=0x7f6d4809dd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d3ffff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f6d4809c680 msgr2=0x7f6d4809ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d3ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6d4809c680 0x7f6d4809ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 0x7f6d4809a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:12.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 0x7f6d4809a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:47080/0 (socket says 172.21.15.181:47080) 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 1 -- 172.21.15.181:0/2825306335 learned_addr learned my addr 172.21.15.181:0/2825306335 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): get_auth_request con 0x7f6d4809c2b0 auth_method 0 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.464+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_done global_id 74759 payload 274 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 1 -- 172.21.15.181:0/2825306335 >> v2:172.21.15.153:3300/0 conn(0x7f6d4809c680 msgr2=0x7f6d4809ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 1 --2- 172.21.15.181:0/2825306335 >> v2:172.21.15.153:3300/0 conn(0x7f6d4809c680 0x7f6d4809ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 1 -- 172.21.15.181:0/2825306335 >> v1:172.21.15.153:6789/0 conn(0x7f6d4809d980 legacy=0x7f6d4809dd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 10 monclient: _finish_hunting 0 2024-09-15T03:36:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 1 monclient: found mon.noname-b 2024-09-15T03:36:12.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:12.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:12.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.465+0000 7f6d4ca27640 1 -- 172.21.15.181:0/2825306335 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4809b220 con 0x7f6d4809c2b0 2024-09-15T03:36:12.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ca27640 1 -- 172.21.15.181:0/2825306335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 msgr2=0x7f6d4809a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:12.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ca27640 1 --2- 172.21.15.181:0/2825306335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 0x7f6d4809a7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009910 tx=0x7f6d440091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c41befa8fbd6d42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:12.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 1 -- 172.21.15.181:0/2825306335 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44032440 con 0x7f6d4809c2b0 2024-09-15T03:36:12.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:12.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:12.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: dump: 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 5 monclient: authenticate success, global_id 74759 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 1 -- 172.21.15.181:0/2825306335 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d440325a0 con 0x7f6d4809c2b0 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 4 set_mon_vals no callback set 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 1 -- 172.21.15.181:0/2825306335 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44032980 con 0x7f6d4809c2b0 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d3f7fe640 10 monclient: dump: 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:12.483 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 10 monclient: shutdown 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/2825306335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 msgr2=0x7f6d4809a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/2825306335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 0x7f6d4809a7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009910 tx=0x7f6d440091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/2825306335 shutdown_connections 2024-09-15T03:36:12.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/2825306335 >> v2:172.21.15.153:3300/0 conn(0x7f6d4809c680 0x7f6d4809ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/2825306335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809c2b0 0x7f6d4809a7a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/2825306335 >> 172.21.15.181:0/2825306335 conn(0x7f6d480713b0 msgr2=0x7f6d480a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/2825306335 shutdown_connections 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.468+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/2825306335 wait complete. 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.469+0000 7f6d4ec8b640 10 monclient: build_initial_monmap 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.469+0000 7f6d4ec8b640 10 monclient: monmap: 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:12.485 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.469+0000 7f6d4ec8b640 1 Processor -- start 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.469+0000 7f6d4ec8b640 1 -- start start 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.469+0000 7f6d4ec8b640 10 monclient: init 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 0x7f6d48141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4809c2b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4809c680 0x7f6d4812fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4809c680 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809d980 0x7f6d48130200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4809d980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4806e9e0 con 0x7f6d4809c680 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4806aa10 con 0x7f6d4809c2b0 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4811dea0 con 0x7f6d4809d980 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ec8b640 10 monclient(hunting): authenticate will time out at 1176.951371s 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d3ffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4809c680 msgr2=0x7f6d4812fcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d3ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4809c680 0x7f6d4812fcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 0x7f6d48141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 0x7f6d48141340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48828/0 (socket says 172.21.15.181:48828) 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 -- 172.21.15.181:0/454579444 learned_addr learned my addr 172.21.15.181:0/454579444 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): get_auth_request con 0x7f6d4809c2b0 auth_method 0 2024-09-15T03:36:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809d980 0x7f6d48130200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): get_auth_request con 0x7f6d4809d980 auth_method 0 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:12.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4d228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient(hunting): handle_auth_done global_id 104233 payload 824 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 -- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809d980 msgr2=0x7f6d48130200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809d980 0x7f6d48130200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 -- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4809c680 msgr2=0x7f6d4812fcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4809c680 0x7f6d4812fcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient: _finish_hunting 0 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 monclient: found mon.noname-b 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.470+0000 7f6d4ca27640 1 -- 172.21.15.181:0/454579444 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44009450 con 0x7f6d4809c2b0 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ca27640 1 -- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 msgr2=0x7f6d48141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ca27640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 0x7f6d48141340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d44035040 tx=0x7f6d44032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403c960 con 0x7f6d4809c2b0 2024-09-15T03:36:12.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: dump: 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:12.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4403cae0 con 0x7f6d4809c2b0 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 5 monclient: authenticate success, global_id 104233 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 10 monclient: _renew_subs 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4811ee00 con 0x7f6d4809c2b0 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403cea0 con 0x7f6d4809c2b0 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3cff9640 10 monclient: dump: 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d3effd640 4 set_mon_vals no callback set 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 10 monclient: _renew_subs 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.473+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d48142100 con 0x7f6d4809c2b0 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6d44039040 con 0x7f6d4809c2b0 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d4ec8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d4ec8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:12.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d4ec8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d48098d30 con 0x7f6d4809c2b0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d3cff9640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d340386f0 0x7f6d3403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(187..187 src has 106..187) v4 ==== 8365+0+0 (secure 0 0 0) 0x7f6d44042030 con 0x7f6d4809c2b0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d3ffff640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d340386f0 0x7f6d3403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.474+0000 7f6d3ffff640 10 monclient: get_auth_request con 0x7f6d340386f0 auth_method 0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.476+0000 7f6d3ffff640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d340386f0 0x7f6d3403aba0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6d380038f0 tx=0x7f6d38006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.478+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6d48098d30 con 0x7f6d4809c2b0 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.478+0000 7f6d3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:12.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.478+0000 7f6d3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:12.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.634+0000 7f6d4ec8b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:12.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.634+0000 7f6d4ec8b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:12.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.634+0000 7f6d4ec8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:12.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:12.634+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6d48098d30 con 0x7f6d4809c2b0 2024-09-15T03:36:13.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.286+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v188) v1 ==== 166+0+0 (secure 0 0 0) 0x7f6d48098d30 con 0x7f6d4809c2b0 2024-09-15T03:36:13.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.287+0000 7f6d3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:13.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.287+0000 7f6d3cff9640 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-15T03:36:13.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.384+0000 7f6d4ec8b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.384+0000 7f6d4ec8b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.384+0000 7f6d4ec8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.384+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6d48098d30 con 0x7f6d4809c2b0 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.385+0000 7f6d3cff9640 1 -- 172.21.15.181:0/454579444 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v188) v1 ==== 145+0+0 (secure 0 0 0) 0x7f6d48098d30 con 0x7f6d4809c2b0 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.385+0000 7f6d3cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.385+0000 7f6d3cff9640 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-15T03:36:13.386 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h4' already exists 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d340386f0 msgr2=0x7f6d3403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d340386f0 0x7f6d3403aba0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6d380038f0 tx=0x7f6d38006d80 comp rx=0 tx=0).stop 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 10 monclient: shutdown 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 msgr2=0x7f6d48141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 0x7f6d48141340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d44035040 tx=0x7f6d44032000 comp rx=0 tx=0).stop 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 shutdown_connections 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d340386f0 0x7f6d3403aba0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4809d980 0x7f6d48130200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4809c680 0x7f6d4812fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 --2- 172.21.15.181:0/454579444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d4809c2b0 0x7f6d48141340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 >> 172.21.15.181:0/454579444 conn(0x7f6d480713b0 msgr2=0x7f6d480a0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 shutdown_connections 2024-09-15T03:36:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.387+0000 7f6d4ec8b640 1 -- 172.21.15.181:0/454579444 wait complete. 2024-09-15T03:36:13.396 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:13.397 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-15T03:36:13.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.471+0000 7f1310d09640 10 monclient: get_monmap_and_config 2024-09-15T03:36:13.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.471+0000 7f1310d09640 10 monclient: build_initial_monmap 2024-09-15T03:36:13.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.471+0000 7f1310d09640 10 monclient: monmap: 2024-09-15T03:36:13.472 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:13.472 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:13.472025+0000 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:13.472025+0000 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.471+0000 7f1310d09640 1 Processor -- start 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.471+0000 7f1310d09640 1 -- start start 2024-09-15T03:36:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.471+0000 7f1310d09640 10 monclient: init 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f130c131bd0 0x7f130c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): picked mon.noname-c con 0x7f130c131bd0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 0x7f130c068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): picked mon.noname-b con 0x7f130c09ebe0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f130c09f000 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0a06d0 con 0x7f130c09f000 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c071780 con 0x7f130c131bd0 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c0a0010 con 0x7f130c09ebe0 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1310d09640 10 monclient(hunting): authenticate will time out at 1177.953512s 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f130c131bd0 0x7f130c131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f130c131bd0 0x7f130c131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48830/0 (socket says 172.21.15.181:48830) 2024-09-15T03:36:13.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 1 -- 172.21.15.181:0/4256203739 learned_addr learned my addr 172.21.15.181:0/4256203739 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): get_auth_request con 0x7f130c131bd0 auth_method 0 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f130a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 1 --2- 172.21.15.181:0/4256203739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 0x7f130c068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): get_auth_request con 0x7f130c09ebe0 auth_method 0 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.472+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 10 monclient(hunting): handle_auth_done global_id 74765 payload 274 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 1 -- 172.21.15.181:0/4256203739 >> v2:172.21.15.181:3300/0 conn(0x7f130c131bd0 msgr2=0x7f130c131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 1 --2- 172.21.15.181:0/4256203739 >> v2:172.21.15.181:3300/0 conn(0x7f130c131bd0 0x7f130c131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 1 -- 172.21.15.181:0/4256203739 >> v1:172.21.15.181:6789/0 conn(0x7f130c09f000 legacy=0x7f130c069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 1 monclient: found mon.noname-b 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.473+0000 7f1309d9b640 1 -- 172.21.15.181:0/4256203739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0a1910 con 0x7f130c09ebe0 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.475+0000 7f1309d9b640 1 -- 172.21.15.181:0/4256203739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 msgr2=0x7f130c068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.475+0000 7f1309d9b640 1 --2- 172.21.15.181:0/4256203739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 0x7f130c068e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1300009850 tx=0x7f13000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=886d12093fc42915 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 1 -- 172.21.15.181:0/4256203739 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130002ae80 con 0x7f130c09ebe0 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: dump: 2024-09-15T03:36:13.485 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: _finish_auth 0 2024-09-15T03:36:13.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 5 monclient: authenticate success, global_id 74765 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 1 -- 172.21.15.181:0/4256203739 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1300032440 con 0x7f130c09ebe0 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 4 set_mon_vals no callback set 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 1 -- 172.21.15.181:0/4256203739 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13000327c0 con 0x7f130c09ebe0 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f130959a640 10 monclient: dump: 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 10 monclient: shutdown 2024-09-15T03:36:13.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 -- 172.21.15.181:0/4256203739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 msgr2=0x7f130c068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 --2- 172.21.15.181:0/4256203739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 0x7f130c068e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1300009850 tx=0x7f13000090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 -- 172.21.15.181:0/4256203739 shutdown_connections 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 --2- 172.21.15.181:0/4256203739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c09ebe0 0x7f130c068e00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 --2- 172.21.15.181:0/4256203739 >> v2:172.21.15.181:3300/0 conn(0x7f130c131bd0 0x7f130c131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 -- 172.21.15.181:0/4256203739 >> 172.21.15.181:0/4256203739 conn(0x7f130c0713b0 msgr2=0x7f130c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 -- 172.21.15.181:0/4256203739 shutdown_connections 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.476+0000 7f1310d09640 1 -- 172.21.15.181:0/4256203739 wait complete. 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.477+0000 7f1310d09640 10 monclient: build_initial_monmap 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.477+0000 7f1310d09640 10 monclient: monmap: 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.477+0000 7f1310d09640 1 Processor -- start 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.477+0000 7f1310d09640 1 -- start start 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.477+0000 7f1310d09640 10 monclient: init 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c09ebe0 0x7f130c09a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): picked mon.noname-a con 0x7f130c09ebe0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 0x7f130c130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): picked mon.noname-b con 0x7f130c09f000 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c131bd0 0x7f130c140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): picked mon.noname-c con 0x7f130c131bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c09b490 con 0x7f130c09ebe0 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c11dc70 con 0x7f130c09f000 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c11df50 con 0x7f130c131bd0 2024-09-15T03:36:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1310d09640 10 monclient(hunting): authenticate will time out at 1177.959087s 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130a59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c09ebe0 msgr2=0x7f130c09a970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c09ebe0 0x7f130c09a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 0x7f130c130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 0x7f130c130b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48832/0 (socket says 172.21.15.181:48832) 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 -- 172.21.15.181:0/527342223 learned_addr learned my addr 172.21.15.181:0/527342223 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): get_auth_request con 0x7f130c09f000 auth_method 0 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c131bd0 0x7f130c140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): get_auth_request con 0x7f130c131bd0 auth_method 0 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f130ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient(hunting): handle_auth_done global_id 104239 payload 824 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 -- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c131bd0 msgr2=0x7f130c140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c131bd0 0x7f130c140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 -- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c09ebe0 msgr2=0x7f130c09a970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c09ebe0 0x7f130c09a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 monclient: found mon.noname-b 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.478+0000 7f1309d9b640 1 -- 172.21.15.181:0/527342223 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300009390 con 0x7f130c09f000 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1309d9b640 1 -- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 msgr2=0x7f130c130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1309d9b640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 0x7f130c130b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1300009850 tx=0x7f1300032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300031c70 con 0x7f130c09f000 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: dump: 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 5 monclient: authenticate success, global_id 104239 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 10 monclient: _renew_subs 2024-09-15T03:36:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1300031dd0 con 0x7f130c09f000 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c11e230 con 0x7f130c09f000 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130003d650 con 0x7f130c09f000 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f12fa7fc640 10 monclient: dump: 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 10 monclient: _renew_subs 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1308d99640 4 set_mon_vals no callback set 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.480+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c11c550 con 0x7f130c09f000 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1300039040 con 0x7f130c09f000 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f1310d09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f1310d09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f1310d09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c04bf60 con 0x7f130c09f000 2024-09-15T03:36:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f12fa7fc640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f12ec0386f0 0x7f12ec03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(188..188 src has 106..188) v4 ==== 8575+0+0 (secure 0 0 0) 0x7f130006c780 con 0x7f130c09f000 2024-09-15T03:36:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f130a59c640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f12ec0386f0 0x7f12ec03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.481+0000 7f130a59c640 10 monclient: get_auth_request con 0x7f12ec0386f0 auth_method 0 2024-09-15T03:36:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.484+0000 7f130a59c640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f12ec0386f0 0x7f12ec03aba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f12fc006c10 tx=0x7f12fc0091d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.486+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1300044b00 con 0x7f130c09f000 2024-09-15T03:36:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.486+0000 7f12fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.486+0000 7f12fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:13.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.642+0000 7f1310d09640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:13.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.642+0000 7f1310d09640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:13.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.642+0000 7f1310d09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:13.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:13.642+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f130c11e3c0 con 0x7f130c09f000 2024-09-15T03:36:14.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.294+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v189) v1 ==== 166+0+0 (secure 0 0 0) 0x7f13000370b0 con 0x7f130c09f000 2024-09-15T03:36:14.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.294+0000 7f12fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:14.295 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.294+0000 7f12fa7fc640 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.391+0000 7f1310d09640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.391+0000 7f1310d09640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.391+0000 7f1310d09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.391+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f130c11e3c0 con 0x7f130c09f000 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.392+0000 7f12fa7fc640 1 -- 172.21.15.181:0/527342223 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v189) v1 ==== 145+0+0 (secure 0 0 0) 0x7f1300042030 con 0x7f130c09f000 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.392+0000 7f12fa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.392+0000 7f12fa7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-15T03:36:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h5' already exists 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.393+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f12ec0386f0 msgr2=0x7f12ec03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.393+0000 7f1310d09640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f12ec0386f0 0x7f12ec03aba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f12fc006c10 tx=0x7f12fc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.393+0000 7f1310d09640 10 monclient: shutdown 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.393+0000 7f1310d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.393+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 msgr2=0x7f130c130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.393+0000 7f1310d09640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 0x7f130c130b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1300009850 tx=0x7f1300032000 comp rx=0 tx=0).stop 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 shutdown_connections 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f12ec0386f0 0x7f12ec03aba0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f130c131bd0 0x7f130c140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f130c09f000 0x7f130c130b50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 --2- 172.21.15.181:0/527342223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f130c09ebe0 0x7f130c09a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 >> 172.21.15.181:0/527342223 conn(0x7f130c0713b0 msgr2=0x7f130c134ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 shutdown_connections 2024-09-15T03:36:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.394+0000 7f1310d09640 1 -- 172.21.15.181:0/527342223 wait complete. 2024-09-15T03:36:14.403 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:14.404 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-15T03:36:14.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.479+0000 7f72cf0a8640 10 monclient: get_monmap_and_config 2024-09-15T03:36:14.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.479+0000 7f72cf0a8640 10 monclient: build_initial_monmap 2024-09-15T03:36:14.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.479+0000 7f72cf0a8640 10 monclient: monmap: 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:14.480215+0000 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:14.480215+0000 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:14.483 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 1 Processor -- start 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 1 -- start start 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient: init 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f72c806df30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72c806e300 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72c809e1e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c80696e0 con 0x7f72c806e300 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c809f600 con 0x7f72c809e1e0 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c806a290 con 0x7f72c806df30 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cf0a8640 10 monclient(hunting): authenticate will time out at 1178.961721s 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72c7fff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f72c806e300 legacy=0x7f72c809dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cd645640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f72c809e1e0 0x7f72c809e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:53640/0 (socket says 172.21.15.181:53640) 2024-09-15T03:36:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cd645640 1 -- 172.21.15.181:0/2326639707 learned_addr learned my addr 172.21.15.181:0/2326639707 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.480+0000 7f72cce44640 1 --2- 172.21.15.181:0/2326639707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): get_auth_request con 0x7f72c806df30 auth_method 0 2024-09-15T03:36:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient(hunting): handle_auth_done global_id 74771 payload 274 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 1 -- 172.21.15.181:0/2326639707 >> v1:172.21.15.181:6789/0 conn(0x7f72c809e1e0 legacy=0x7f72c809e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 1 -- 172.21.15.181:0/2326639707 >> v1:172.21.15.153:6789/0 conn(0x7f72c806e300 legacy=0x7f72c809dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient: _finish_hunting 0 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 1 monclient: found mon.noname-b 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:14.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:14.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.481+0000 7f72cce44640 1 -- 172.21.15.181:0/2326639707 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c8069f40 con 0x7f72c806df30 2024-09-15T03:36:14.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cce44640 1 -- 172.21.15.181:0/2326639707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 msgr2=0x7f72c806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:14.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cce44640 1 --2- 172.21.15.181:0/2326639707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c806a6b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009880 tx=0x7f72bc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f29ce6cd6b79aae6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:14.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 1 -- 172.21.15.181:0/2326639707 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc032440 con 0x7f72c806df30 2024-09-15T03:36:14.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:14.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: dump: 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:14.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 1 -- 172.21.15.181:0/2326639707 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc0325a0 con 0x7f72c806df30 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 1 -- 172.21.15.181:0/2326639707 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc032940 con 0x7f72c806df30 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 5 monclient: authenticate success, global_id 74771 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 4 set_mon_vals no callback set 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72c77fe640 10 monclient: dump: 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:14.494 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 10 monclient: shutdown 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:14.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 -- 172.21.15.181:0/2326639707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 msgr2=0x7f72c806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/2326639707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c806a6b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009880 tx=0x7f72bc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 -- 172.21.15.181:0/2326639707 shutdown_connections 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/2326639707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c806a6b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 -- 172.21.15.181:0/2326639707 >> 172.21.15.181:0/2326639707 conn(0x7f72c80a3b30 msgr2=0x7f72c809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 -- 172.21.15.181:0/2326639707 shutdown_connections 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.484+0000 7f72cf0a8640 1 -- 172.21.15.181:0/2326639707 wait complete. 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.485+0000 7f72cf0a8640 10 monclient: build_initial_monmap 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.485+0000 7f72cf0a8640 10 monclient: monmap: 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:14.496 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:14.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.485+0000 7f72cf0a8640 1 Processor -- start 2024-09-15T03:36:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.485+0000 7f72cf0a8640 1 -- start start 2024-09-15T03:36:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.485+0000 7f72cf0a8640 10 monclient: init 2024-09-15T03:36:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c8141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f72c806df30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f72c806e300 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72c809e1e0 0x7f72c809b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f72c809e1e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c8099ea0 con 0x7f72c809e1e0 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c806d9f0 con 0x7f72c806e300 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c811f0e0 con 0x7f72c806df30 2024-09-15T03:36:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cf0a8640 10 monclient(hunting): authenticate will time out at 1178.967452s 2024-09-15T03:36:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48848/0 (socket says 172.21.15.181:48848) 2024-09-15T03:36:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 -- 172.21.15.181:0/4044655493 learned_addr learned my addr 172.21.15.181:0/4044655493 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): get_auth_request con 0x7f72c806e300 auth_method 0 2024-09-15T03:36:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:14.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:14.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:14.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cd645640 1 -- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72c809e1e0 msgr2=0x7f72c809b550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:14.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cd645640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72c809e1e0 0x7f72c809b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:14.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cce44640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c8141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cce44640 10 monclient(hunting): get_auth_request con 0x7f72c806df30 auth_method 0 2024-09-15T03:36:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cce44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cce44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72cce44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient(hunting): handle_auth_done global_id 104242 payload 824 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 -- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 msgr2=0x7f72c8141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c8141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 -- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72c809e1e0 msgr2=0x7f72c809b550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72c809e1e0 0x7f72c809b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 monclient: found mon.noname-b 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.486+0000 7f72c7fff640 1 -- 172.21.15.181:0/4044655493 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc009390 con 0x7f72c806e300 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c7fff640 1 -- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 msgr2=0x7f72c8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c7fff640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f72b400b5c0 tx=0x7f72b4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4003d40 con 0x7f72c806e300 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: dump: 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.513 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 5 monclient: authenticate success, global_id 104242 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 10 monclient: _renew_subs 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c81418a0 con 0x7f72c806e300 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b4006150 con 0x7f72c806e300 2024-09-15T03:36:14.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4006510 con 0x7f72c806e300 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 10 monclient: dump: 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:14.515 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c6ffd640 4 set_mon_vals no callback set 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 10 monclient: _renew_subs 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c81421a0 con 0x7f72c806e300 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.489+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f72b40105d0 con 0x7f72c806e300 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72cf0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72cf0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72cf0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:14.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c811dcd0 con 0x7f72c806e300 2024-09-15T03:36:14.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72c4ff9640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72ac038720 0x7f72ac03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:14.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(189..189 src has 106..189) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f72b404c270 con 0x7f72c806e300 2024-09-15T03:36:14.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72cce44640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72ac038720 0x7f72ac03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:14.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.490+0000 7f72cce44640 10 monclient: get_auth_request con 0x7f72ac038720 auth_method 0 2024-09-15T03:36:14.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.492+0000 7f72cce44640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72ac038720 0x7f72ac03abd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f72bc031000 tx=0x7f72bc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:14.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:14.494+0000 7f72c7fff640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f72b400b5c0 tx=0x7f72b4009120 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f72b401b3b0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:36:15.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.322+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f72b401b3b0 con 0x7f72c806e300 2024-09-15T03:36:15.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.322+0000 7f72c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:15.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.322+0000 7f72c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:15.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.481+0000 7f72cf0a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:15.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.481+0000 7f72cf0a8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:15.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.481+0000 7f72cf0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:15.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:15.481+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f72c804bf60 con 0x7f72c806e300 2024-09-15T03:36:16.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.221+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v190) v1 ==== 166+0+0 (secure 0 0 0) 0x7f72b400f7e0 con 0x7f72c806e300 2024-09-15T03:36:16.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.221+0000 7f72c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:16.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.221+0000 7f72c4ff9640 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.326+0000 7f72cf0a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.326+0000 7f72cf0a8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.326+0000 7f72cf0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.326+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f72c8068a20 con 0x7f72c806e300 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.327+0000 7f72c4ff9640 1 -- 172.21.15.181:0/4044655493 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v190) v1 ==== 145+0+0 (secure 0 0 0) 0x7f72b400f9d0 con 0x7f72c806e300 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.327+0000 7f72c4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.327+0000 7f72c4ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-15T03:36:16.328 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h6' already exists 2024-09-15T03:36:16.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.328+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72ac038720 msgr2=0x7f72ac03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:16.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.328+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72ac038720 0x7f72ac03abd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f72bc031000 tx=0x7f72bc032000 comp rx=0 tx=0).stop 2024-09-15T03:36:16.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.328+0000 7f72cf0a8640 10 monclient: shutdown 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.328+0000 7f72cf0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.328+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 msgr2=0x7f72c8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.328+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f72b400b5c0 tx=0x7f72b4009120 comp rx=0 tx=0).stop 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 shutdown_connections 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f72ac038720 0x7f72ac03abd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72c809e1e0 0x7f72c809b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72c806e300 0x7f72c8131f30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 --2- 172.21.15.181:0/4044655493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72c806df30 0x7f72c8141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 >> 172.21.15.181:0/4044655493 conn(0x7f72c80a3b30 msgr2=0x7f72c8134e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 shutdown_connections 2024-09-15T03:36:16.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.329+0000 7f72cf0a8640 1 -- 172.21.15.181:0/4044655493 wait complete. 2024-09-15T03:36:16.338 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:16.339 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.412+0000 7fcd2339d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.412+0000 7fcd2339d640 10 monclient: build_initial_monmap 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.412+0000 7fcd2339d640 10 monclient: monmap: 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:16.413587+0000 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:16.413587+0000 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 1 Processor -- start 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 1 -- start start 2024-09-15T03:36:16.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient: init 2024-09-15T03:36:16.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:16.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T03:36:16.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd1c131ac0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:16.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd1c09ff50 0x7fcd1c098150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd1c09ff50 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd1c098690 0x7fcd1c138e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd1c098690 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0a1330 con 0x7fcd1c131ac0 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.413+0000 7fcd2339d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c1325a0 con 0x7fcd1c09ff50 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2339d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c1326e0 con 0x7fcd1c098690 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2339d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2339d640 10 monclient(hunting): authenticate will time out at 1180.895016s 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd21139640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fcd1c131ac0 legacy=0x7fcd1c131e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd20938640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fcd1c09ff50 msgr2=0x7fcd1c098150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:16.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd20938640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd1c09ff50 0x7fcd1c098150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd1c098690 0x7fcd1c138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd1c098690 0x7fcd1c138e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48864/0 (socket says 172.21.15.181:48864) 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 -- 172.21.15.181:0/679759565 learned_addr learned my addr 172.21.15.181:0/679759565 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): get_auth_request con 0x7fcd1c098690 auth_method 0 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient(hunting): handle_auth_done global_id 104245 payload 274 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 -- 172.21.15.181:0/679759565 >> v2:172.21.15.153:3300/0 conn(0x7fcd1c09ff50 msgr2=0x7fcd1c098150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 --2- 172.21.15.181:0/679759565 >> v2:172.21.15.153:3300/0 conn(0x7fcd1c09ff50 0x7fcd1c098150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 -- 172.21.15.181:0/679759565 >> v1:172.21.15.153:6789/0 conn(0x7fcd1c131ac0 legacy=0x7fcd1c131e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 monclient: found mon.noname-c 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.414+0000 7fcd2193a640 1 -- 172.21.15.181:0/679759565 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c098a60 con 0x7fcd1c098690 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd2193a640 1 -- 172.21.15.181:0/679759565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c098690 msgr2=0x7fcd1c138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:16.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd2193a640 1 --2- 172.21.15.181:0/679759565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c098690 0x7fcd1c138e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c09feb0 tx=0x7fcd14009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f14b3c983fc5573a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 1 -- 172.21.15.181:0/679759565 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1402aea0 con 0x7fcd1c098690 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: dump: 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd2339d640 5 monclient: authenticate success, global_id 104245 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 1 -- 172.21.15.181:0/679759565 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1402d050 con 0x7fcd1c098690 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd2339d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd2339d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 1 -- 172.21.15.181:0/679759565 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14007820 con 0x7fcd1c098690 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd2339d640 4 set_mon_vals no callback set 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.417+0000 7fcd13fff640 10 monclient: dump: 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 10 monclient: shutdown 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 -- 172.21.15.181:0/679759565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c098690 msgr2=0x7fcd1c138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 --2- 172.21.15.181:0/679759565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c098690 0x7fcd1c138e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c09feb0 tx=0x7fcd14009840 comp rx=0 tx=0).stop 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 -- 172.21.15.181:0/679759565 shutdown_connections 2024-09-15T03:36:16.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 --2- 172.21.15.181:0/679759565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c098690 0x7fcd1c138e00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 --2- 172.21.15.181:0/679759565 >> v2:172.21.15.153:3300/0 conn(0x7fcd1c09ff50 0x7fcd1c098150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 -- 172.21.15.181:0/679759565 >> 172.21.15.181:0/679759565 conn(0x7fcd1c0a5190 msgr2=0x7fcd1c0a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 -- 172.21.15.181:0/679759565 shutdown_connections 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 1 -- 172.21.15.181:0/679759565 wait complete. 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 10 monclient: build_initial_monmap 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.418+0000 7fcd2339d640 10 monclient: monmap: 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 Processor -- start 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 -- start start 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient: init 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd1c098690 0x7fcd1c134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd1c098690 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd1c09ff50 0x7fcd1c134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd1c09ff50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 0x7fcd1c134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:16.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): picked mon.noname-b con 0x7fcd1c131ac0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c136380 con 0x7fcd1c09ff50 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c098e70 con 0x7fcd1c131ac0 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0a3350 con 0x7fcd1c098690 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2339d640 10 monclient(hunting): authenticate will time out at 1180.900811s 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd20938640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd1c09ff50 msgr2=0x7fcd1c134760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd20938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd1c09ff50 0x7fcd1c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 0x7fcd1c134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 0x7fcd1c134ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48870/0 (socket says 172.21.15.181:48870) 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 1 -- 172.21.15.181:0/3946523855 learned_addr learned my addr 172.21.15.181:0/3946523855 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:16.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 10 monclient(hunting): get_auth_request con 0x7fcd1c131ac0 auth_method 0 2024-09-15T03:36:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.419+0000 7fcd2193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd1c098690 0x7fcd1c134220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): get_auth_request con 0x7fcd1c098690 auth_method 0 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd21139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 10 monclient(hunting): handle_auth_done global_id 104248 payload 824 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 1 -- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd1c098690 msgr2=0x7fcd1c134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd1c098690 0x7fcd1c134220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 1 -- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd1c09ff50 msgr2=0x7fcd1c134760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd1c09ff50 0x7fcd1c134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 1 monclient: found mon.noname-b 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.420+0000 7fcd2193a640 1 -- 172.21.15.181:0/3946523855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd14009ae0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2193a640 1 -- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 msgr2=0x7fcd1c134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2193a640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 0x7fcd1c134ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd14009ab0 tx=0x7fcd14036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14031da0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: dump: 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1403c420 con 0x7fcd1c131ac0 2024-09-15T03:36:16.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2339d640 5 monclient: authenticate success, global_id 104248 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2339d640 10 monclient: _renew_subs 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c1378c0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1403c7e0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd117fa640 10 monclient: dump: 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:16.445 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd137fe640 4 set_mon_vals no callback set 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.422+0000 7fcd2339d640 10 monclient: _renew_subs 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd2339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c1369d0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcd1403cbd0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd2339d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd2339d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd2339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.423+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c04bf60 con 0x7fcd1c131ac0 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.424+0000 7fcd117fa640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd000386f0 0x7fcd0003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.424+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(190..190 src has 106..190) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fcd1406d7f0 con 0x7fcd1c131ac0 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.424+0000 7fcd21139640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd000386f0 0x7fcd0003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.424+0000 7fcd21139640 10 monclient: get_auth_request con 0x7fcd000386f0 auth_method 0 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.426+0000 7fcd21139640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd000386f0 0x7fcd0003aba0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcd180038f0 tx=0x7fcd180090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.428+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcd1402fd70 con 0x7fcd1c131ac0 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.428+0000 7fcd117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:16.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.428+0000 7fcd117fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:16.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.585+0000 7fcd2339d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:16.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.585+0000 7fcd2339d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:16.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.585+0000 7fcd2339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:16.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:16.585+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcd1c064a70 con 0x7fcd1c131ac0 2024-09-15T03:36:17.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.226+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v191) v1 ==== 166+0+0 (secure 0 0 0) 0x7fcd14040020 con 0x7fcd1c131ac0 2024-09-15T03:36:17.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.226+0000 7fcd117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:17.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.226+0000 7fcd117fa640 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.321+0000 7fcd2339d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.321+0000 7fcd2339d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.321+0000 7fcd2339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.321+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcd1c137ab0 con 0x7fcd1c131ac0 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.322+0000 7fcd117fa640 1 -- 172.21.15.181:0/3946523855 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v191) v1 ==== 145+0+0 (secure 0 0 0) 0x7fcd14040070 con 0x7fcd1c131ac0 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.322+0000 7fcd117fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.322+0000 7fcd117fa640 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-15T03:36:17.323 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h7' already exists 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd000386f0 msgr2=0x7fcd0003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd000386f0 0x7fcd0003aba0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcd180038f0 tx=0x7fcd180090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 10 monclient: shutdown 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 msgr2=0x7fcd1c134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 0x7fcd1c134ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd14009ab0 tx=0x7fcd14036000 comp rx=0 tx=0).stop 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 shutdown_connections 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd000386f0 0x7fcd0003aba0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd1c131ac0 0x7fcd1c134ca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd1c09ff50 0x7fcd1c134760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 --2- 172.21.15.181:0/3946523855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd1c098690 0x7fcd1c134220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 >> 172.21.15.181:0/3946523855 conn(0x7fcd1c0a5190 msgr2=0x7fcd1c09b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 shutdown_connections 2024-09-15T03:36:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.324+0000 7fcd2339d640 1 -- 172.21.15.181:0/3946523855 wait complete. 2024-09-15T03:36:17.334 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:17.334 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.408+0000 7f8dcdd9d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.408+0000 7f8dcdd9d640 10 monclient: build_initial_monmap 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.408+0000 7f8dcdd9d640 10 monclient: monmap: 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:17.408839+0000 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:17.408839+0000 2024-09-15T03:36:17.409 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:17.410 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.408+0000 7f8dcdd9d640 1 Processor -- start 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.408+0000 7f8dcdd9d640 1 -- start start 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.408+0000 7f8dcdd9d640 10 monclient: init 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 0x7f8dc8099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc80972c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:17.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8dc8097cb0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dc809c0e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc8097690 con 0x7f8dc809c0e0 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc809d8d0 con 0x7f8dc8097cb0 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8097a70 con 0x7f8dc80972c0 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dcdd9d640 10 monclient(hunting): authenticate will time out at 1181.890293s 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dbeffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f8dc8097cb0 0x7f8dc809b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:53652/0 (socket says 172.21.15.181:53652) 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dbeffd640 1 -- 172.21.15.181:0/3907316427 learned_addr learned my addr 172.21.15.181:0/3907316427 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc7fff640 1 -- 172.21.15.181:0/3907316427 >> v1:172.21.15.153:6789/0 conn(0x7f8dc809c0e0 legacy=0x7f8dc81357f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:17.414 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 1 --2- 172.21.15.181:0/3907316427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 0x7f8dc8099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request con 0x7f8dc80972c0 auth_method 0 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_done global_id 74780 payload 274 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3907316427 >> v1:172.21.15.181:6789/0 conn(0x7f8dc8097cb0 legacy=0x7f8dc809b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:17.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3907316427 >> v1:172.21.15.153:6789/0 conn(0x7f8dc809c0e0 legacy=0x7f8dc81357f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:17.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:36:17.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 1 monclient: found mon.noname-b 2024-09-15T03:36:17.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:17.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:17.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.409+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3907316427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc806ebf0 con 0x7f8dc80972c0 2024-09-15T03:36:17.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.412+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3907316427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 msgr2=0x7f8dc8099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:17.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.412+0000 7f8dc77fe640 1 --2- 172.21.15.181:0/3907316427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 0x7f8dc8099040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8db800aa20 tx=0x7f8db800a2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19766806f37602cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:17.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 1 -- 172.21.15.181:0/3907316427 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8030800 con 0x7f8dc80972c0 2024-09-15T03:36:17.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:17.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: dump: 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:17.423 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 1 -- 172.21.15.181:0/3907316427 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db802e050 con 0x7f8dc80972c0 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 1 -- 172.21.15.181:0/3907316427 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8030b90 con 0x7f8dc80972c0 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 5 monclient: authenticate success, global_id 74780 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 4 set_mon_vals no callback set 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dc6ffd640 10 monclient: dump: 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:17.425 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 10 monclient: shutdown 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3907316427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 msgr2=0x7f8dc8099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3907316427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 0x7f8dc8099040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8db800aa20 tx=0x7f8db800a2b0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3907316427 shutdown_connections 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3907316427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc80972c0 0x7f8dc8099040 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3907316427 >> 172.21.15.181:0/3907316427 conn(0x7f8dc80a0fe0 msgr2=0x7f8dc80a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3907316427 shutdown_connections 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.413+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3907316427 wait complete. 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.414+0000 7f8dcdd9d640 10 monclient: build_initial_monmap 2024-09-15T03:36:17.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.414+0000 7f8dcdd9d640 10 monclient: monmap: 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.414+0000 7f8dcdd9d640 1 Processor -- start 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.414+0000 7f8dcdd9d640 1 -- start start 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.414+0000 7f8dcdd9d640 10 monclient: init 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 0x7f8dc8130090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc80972c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc8097cb0 0x7f8dc811ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc8097cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:17.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dc809c0e0 0x7f8dc811bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc809c0e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc811d5a0 con 0x7f8dc809c0e0 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc811dcc0 con 0x7f8dc80972c0 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc809cc90 con 0x7f8dc8097cb0 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dcdd9d640 10 monclient(hunting): authenticate will time out at 1181.896372s 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 0x7f8dc8130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:17.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc7fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dc809c0e0 msgr2=0x7f8dc811bfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dc809c0e0 0x7f8dc811bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 0x7f8dc8130090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48872/0 (socket says 172.21.15.181:48872) 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3118240256 learned_addr learned my addr 172.21.15.181:0/3118240256 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request con 0x7f8dc80972c0 auth_method 0 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc8097cb0 0x7f8dc811ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): get_auth_request con 0x7f8dc8097cb0 auth_method 0 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient(hunting): handle_auth_done global_id 104251 payload 824 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc8097cb0 msgr2=0x7f8dc811ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc8097cb0 0x7f8dc811ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dc809c0e0 msgr2=0x7f8dc811bfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dc809c0e0 0x7f8dc811bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 monclient: found mon.noname-b 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.415+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3118240256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db800a530 con 0x7f8dc80972c0 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.417+0000 7f8dc77fe640 1 -- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 msgr2=0x7f8dc8130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.417+0000 7f8dc77fe640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 0x7f8dc8130090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8db800aa20 tx=0x7f8db8038000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8032cb0 con 0x7f8dc80972c0 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:17.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: dump: 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:17.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: _finish_auth 0 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 5 monclient: authenticate success, global_id 104251 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8032e10 con 0x7f8dc80972c0 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 10 monclient: _renew_subs 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc811f060 con 0x7f8dc80972c0 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db803c660 con 0x7f8dc80972c0 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:17.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 10 monclient: dump: 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 10 monclient: _renew_subs 2024-09-15T03:36:17.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dc67fc640 4 set_mon_vals no callback set 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc812f510 con 0x7f8dc80972c0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.418+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8db803ca90 con 0x7f8dc80972c0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dcdd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dcdd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dcdd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dbffff640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8da80386f0 0x7f8da803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc804bf60 con 0x7f8dc80972c0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(191..191 src has 106..191) v4 ==== 9333+0+0 (secure 0 0 0) 0x7f8db806ddf0 con 0x7f8dc80972c0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dbeffd640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8da80386f0 0x7f8da803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:17.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.419+0000 7f8dbeffd640 10 monclient: get_auth_request con 0x7f8da80386f0 auth_method 0 2024-09-15T03:36:17.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.421+0000 7f8dbeffd640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8da80386f0 0x7f8da803aba0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8db0007fd0 tx=0x7f8db0007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:17.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.423+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8db804db30 con 0x7f8dc80972c0 2024-09-15T03:36:17.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.423+0000 7f8dbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:17.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.423+0000 7f8dbffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.579+0000 7f8dcdd9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.579+0000 7f8dcdd9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.579+0000 7f8dcdd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:17.579+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8dc8099580 con 0x7f8dc80972c0 2024-09-15T03:36:18.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.228+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v192) v1 ==== 166+0+0 (secure 0 0 0) 0x7f8db804d350 con 0x7f8dc80972c0 2024-09-15T03:36:18.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.228+0000 7f8dbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:18.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.228+0000 7f8dbffff640 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.323+0000 7f8dcdd9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.323+0000 7f8dcdd9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.323+0000 7f8dcdd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.323+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8dc8099580 con 0x7f8dc80972c0 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.324+0000 7f8dbffff640 1 -- 172.21.15.181:0/3118240256 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v192) v1 ==== 145+0+0 (secure 0 0 0) 0x7f8db804d530 con 0x7f8dc80972c0 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.324+0000 7f8dbffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.324+0000 7f8dbffff640 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-15T03:36:18.325 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h8' already exists 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8da80386f0 msgr2=0x7f8da803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8da80386f0 0x7f8da803aba0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8db0007fd0 tx=0x7f8db0007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 10 monclient: shutdown 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 msgr2=0x7f8dc8130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 0x7f8dc8130090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8db800aa20 tx=0x7f8db8038000 comp rx=0 tx=0).stop 2024-09-15T03:36:18.327 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 shutdown_connections 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8da80386f0 0x7f8da803aba0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dc809c0e0 0x7f8dc811bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8dc8097cb0 0x7f8dc811ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 --2- 172.21.15.181:0/3118240256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc80972c0 0x7f8dc8130090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 >> 172.21.15.181:0/3118240256 conn(0x7f8dc80a0fe0 msgr2=0x7f8dc809da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 shutdown_connections 2024-09-15T03:36:18.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.326+0000 7f8dcdd9d640 1 -- 172.21.15.181:0/3118240256 wait complete. 2024-09-15T03:36:18.335 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:18.336 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-15T03:36:18.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.410+0000 7f46f34af640 10 monclient: get_monmap_and_config 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.410+0000 7f46f34af640 10 monclient: build_initial_monmap 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.410+0000 7f46f34af640 10 monclient: monmap: 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:18.411521+0000 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:18.411521+0000 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:18.416 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.411+0000 7f46f34af640 1 Processor -- start 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.411+0000 7f46f34af640 1 -- start start 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.411+0000 7f46f34af640 10 monclient: init 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.411+0000 7f46f34af640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.411+0000 7f46f34af640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:36:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f46ec096a30 0x7f46ec099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): picked mon.noname-a con 0x7f46ec096a30 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46ec09d530 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec135c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): picked mon.noname-b con 0x7f46ec097510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec097e80 con 0x7f46ec09d530 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec098be0 con 0x7f46ec096a30 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec098ec0 con 0x7f46ec097510 2024-09-15T03:36:18.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f34af640 10 monclient(hunting): authenticate will time out at 1182.893097s 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f124b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f46ec096a30 msgr2=0x7f46ec099d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f124b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f46ec096a30 0x7f46ec099d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f0a4a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f46ec09d530 0x7f46ec096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56516/0 (socket says 172.21.15.181:56516) 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f0a4a640 1 -- 172.21.15.181:0/2676066280 learned_addr learned my addr 172.21.15.181:0/2676066280 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 1 --2- 172.21.15.181:0/2676066280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): get_auth_request con 0x7f46ec097510 auth_method 0 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:18.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_done global_id 74786 payload 274 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 1 -- 172.21.15.181:0/2676066280 >> v2:172.21.15.153:3300/0 conn(0x7f46ec096a30 msgr2=0x7f46ec099d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 1 --2- 172.21.15.181:0/2676066280 >> v2:172.21.15.153:3300/0 conn(0x7f46ec096a30 0x7f46ec099d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 1 -- 172.21.15.181:0/2676066280 >> v1:172.21.15.181:6789/0 conn(0x7f46ec09d530 legacy=0x7f46ec096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 1 monclient: found mon.noname-b 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:18.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.412+0000 7f46f1a4c640 1 -- 172.21.15.181:0/2676066280 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec097990 con 0x7f46ec097510 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.415+0000 7f46f1a4c640 1 -- 172.21.15.181:0/2676066280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 msgr2=0x7f46ec135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.415+0000 7f46f1a4c640 1 --2- 172.21.15.181:0/2676066280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec135c10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009e60 tx=0x7f46e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc0431e47ec12799 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 1 -- 172.21.15.181:0/2676066280 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4007840 con 0x7f46ec097510 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: dump: 2024-09-15T03:36:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 1 -- 172.21.15.181:0/2676066280 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e402d050 con 0x7f46ec097510 2024-09-15T03:36:18.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 1 -- 172.21.15.181:0/2676066280 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4007c10 con 0x7f46ec097510 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 5 monclient: authenticate success, global_id 74786 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 4 set_mon_vals no callback set 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46e3fff640 10 monclient: dump: 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.428 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 10 monclient: shutdown 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 -- 172.21.15.181:0/2676066280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 msgr2=0x7f46ec135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 --2- 172.21.15.181:0/2676066280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec135c10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009e60 tx=0x7f46e4009780 comp rx=0 tx=0).stop 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 -- 172.21.15.181:0/2676066280 shutdown_connections 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 --2- 172.21.15.181:0/2676066280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec135c10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 --2- 172.21.15.181:0/2676066280 >> v2:172.21.15.153:3300/0 conn(0x7f46ec096a30 0x7f46ec099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 -- 172.21.15.181:0/2676066280 >> 172.21.15.181:0/2676066280 conn(0x7f46ec0a0fe0 msgr2=0x7f46ec0a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 -- 172.21.15.181:0/2676066280 shutdown_connections 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.416+0000 7f46f34af640 1 -- 172.21.15.181:0/2676066280 wait complete. 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.417+0000 7f46f34af640 10 monclient: build_initial_monmap 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.417+0000 7f46f34af640 10 monclient: monmap: 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:18.430 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.417+0000 7f46f34af640 1 Processor -- start 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.417+0000 7f46f34af640 1 -- start start 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.417+0000 7f46f34af640 10 monclient: init 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ec096a30 0x7f46ec13cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): picked mon.noname-a con 0x7f46ec096a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): picked mon.noname-c con 0x7f46ec097510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:18.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 0x7f46ec11a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:18.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): picked mon.noname-b con 0x7f46ec09d530 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:18.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:18.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec13dc00 con 0x7f46ec096a30 2024-09-15T03:36:18.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:18.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec098530 con 0x7f46ec09d530 2024-09-15T03:36:18.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:18.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec09f210 con 0x7f46ec097510 2024-09-15T03:36:18.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:18.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f34af640 10 monclient(hunting): authenticate will time out at 1182.899165s 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f124b640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ec096a30 msgr2=0x7f46ec13cd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f124b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ec096a30 0x7f46ec13cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 0x7f46ec11a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 0x7f46ec11a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36220/0 (socket says 172.21.15.181:36220) 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 -- 172.21.15.181:0/3109229996 learned_addr learned my addr 172.21.15.181:0/3109229996 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): get_auth_request con 0x7f46ec09d530 auth_method 0 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f0a4a640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f0a4a640 10 monclient(hunting): get_auth_request con 0x7f46ec097510 auth_method 0 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f0a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f0a4a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f0a4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient(hunting): handle_auth_done global_id 104254 payload 824 2024-09-15T03:36:18.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 -- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 msgr2=0x7f46ec130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:18.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec130be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 -- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ec096a30 msgr2=0x7f46ec13cd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ec096a30 0x7f46ec13cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 monclient: found mon.noname-b 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.418+0000 7f46f1a4c640 1 -- 172.21.15.181:0/3109229996 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4009a20 con 0x7f46ec09d530 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46f1a4c640 1 -- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 msgr2=0x7f46ec11a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46f1a4c640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 0x7f46ec11a7c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009e60 tx=0x7f46e4037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4031cd0 con 0x7f46ec09d530 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:18.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: dump: 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:18.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46f34af640 5 monclient: authenticate success, global_id 104254 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e4040070 con 0x7f46ec09d530 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46f34af640 10 monclient: _renew_subs 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46f34af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec11dd00 con 0x7f46ec09d530 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4045640 con 0x7f46ec09d530 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e17fa640 10 monclient: dump: 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:18.439 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.421+0000 7f46f34af640 10 monclient: _renew_subs 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.421+0000 7f46f34af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.421+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec11dfe0 con 0x7f46ec09d530 2024-09-15T03:36:18.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.420+0000 7f46e37fe640 4 set_mon_vals no callback set 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.421+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f46e4003020 con 0x7f46ec09d530 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.421+0000 7f46e17fa640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46d00386f0 0x7f46d003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.421+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(192..192 src has 106..192) v4 ==== 9543+0+0 (secure 0 0 0) 0x7f46e406ec60 con 0x7f46ec09d530 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.422+0000 7f46f34af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.422+0000 7f46f34af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.422+0000 7f46f34af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.422+0000 7f46f124b640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46d00386f0 0x7f46d003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.422+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec04bf60 con 0x7f46ec09d530 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.422+0000 7f46f124b640 10 monclient: get_auth_request con 0x7f46d00386f0 auth_method 0 2024-09-15T03:36:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.424+0000 7f46f124b640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46d00386f0 0x7f46d003aba0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f46e80038f0 tx=0x7f46e8006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:18.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.426+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f46e403b420 con 0x7f46ec09d530 2024-09-15T03:36:18.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.426+0000 7f46e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:18.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.426+0000 7f46e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:18.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.583+0000 7f46f34af640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:18.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.583+0000 7f46f34af640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:18.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.583+0000 7f46f34af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:18.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:18.583+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f46ec09cd10 con 0x7f46ec09d530 2024-09-15T03:36:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.230+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v193) v1 ==== 166+0+0 (secure 0 0 0) 0x7f46e40393c0 con 0x7f46ec09d530 2024-09-15T03:36:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.230+0000 7f46e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:19.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.230+0000 7f46e17fa640 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.326+0000 7f46f34af640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.326+0000 7f46f34af640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.326+0000 7f46f34af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.326+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f46ec09cd10 con 0x7f46ec09d530 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.327+0000 7f46e17fa640 1 -- 172.21.15.181:0/3109229996 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v193) v1 ==== 145+0+0 (secure 0 0 0) 0x7f46e403a3a0 con 0x7f46ec09d530 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.327+0000 7f46e17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.327+0000 7f46e17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-15T03:36:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h9' already exists 2024-09-15T03:36:19.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46d00386f0 msgr2=0x7f46d003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46d00386f0 0x7f46d003aba0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f46e80038f0 tx=0x7f46e8006d80 comp rx=0 tx=0).stop 2024-09-15T03:36:19.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 10 monclient: shutdown 2024-09-15T03:36:19.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:19.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 msgr2=0x7f46ec11a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 0x7f46ec11a7c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009e60 tx=0x7f46e4037000 comp rx=0 tx=0).stop 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 shutdown_connections 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46d00386f0 0x7f46d003aba0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ec09d530 0x7f46ec11a7c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ec097510 0x7f46ec130be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 --2- 172.21.15.181:0/3109229996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ec096a30 0x7f46ec13cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 >> 172.21.15.181:0/3109229996 conn(0x7f46ec0a0fe0 msgr2=0x7f46ec09a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 shutdown_connections 2024-09-15T03:36:19.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.329+0000 7f46f34af640 1 -- 172.21.15.181:0/3109229996 wait complete. 2024-09-15T03:36:19.339 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:19.339 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.412+0000 7f863dcc9640 10 monclient: get_monmap_and_config 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.412+0000 7f863dcc9640 10 monclient: build_initial_monmap 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.412+0000 7f863dcc9640 10 monclient: monmap: 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:19.413005+0000 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:19.413005+0000 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:19.413 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:19.414 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:19.414 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:19.414 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:19.414 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:19.414 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:19.414 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.412+0000 7f863dcc9640 1 Processor -- start 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 1 -- start start 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient: init 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8638069a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86380983e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8638137ea0 0x7f863813a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient(hunting): picked mon.noname-c con 0x7f8638137ea0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863809cd00 con 0x7f86380983e0 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86381327f0 con 0x7f8638069a30 2024-09-15T03:36:19.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8638069e00 con 0x7f8638137ea0 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f863dcc9640 10 monclient(hunting): authenticate will time out at 1183.894560s 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8638137ea0 0x7f863813a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8636ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f86380983e0 legacy=0x7f8638098840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8638137ea0 0x7f863813a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36234/0 (socket says 172.21.15.181:36234) 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 1 -- 172.21.15.181:0/3475343676 learned_addr learned my addr 172.21.15.181:0/3475343676 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): get_auth_request con 0x7f8638137ea0 auth_method 0 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.413+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.415+0000 7f86377fe640 1 -- 172.21.15.181:0/3475343676 >> v1:172.21.15.181:6789/0 conn(0x7f8638069a30 legacy=0x7f86381320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 10 monclient(hunting): handle_auth_done global_id 104257 payload 274 2024-09-15T03:36:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 1 -- 172.21.15.181:0/3475343676 >> v1:172.21.15.181:6789/0 conn(0x7f8638069a30 legacy=0x7f86381320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 1 -- 172.21.15.181:0/3475343676 >> v1:172.21.15.153:6789/0 conn(0x7f86380983e0 legacy=0x7f8638098840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 1 monclient: found mon.noname-c 2024-09-15T03:36:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.416+0000 7f8637fff640 1 -- 172.21.15.181:0/3475343676 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863809d140 con 0x7f8638137ea0 2024-09-15T03:36:19.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f8637fff640 1 -- 172.21.15.181:0/3475343676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 msgr2=0x7f863813a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f8637fff640 1 --2- 172.21.15.181:0/3475343676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f863813a280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8630009fd0 tx=0x7f8630009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76919ede4a6589af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 1 -- 172.21.15.181:0/3475343676 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630007610 con 0x7f8638137ea0 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: dump: 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 5 monclient: authenticate success, global_id 104257 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 1 -- 172.21.15.181:0/3475343676 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8630007770 con 0x7f8638137ea0 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 4 set_mon_vals no callback set 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 1 -- 172.21.15.181:0/3475343676 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630007b10 con 0x7f8638137ea0 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f86367fc640 10 monclient: dump: 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 10 monclient: shutdown 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 -- 172.21.15.181:0/3475343676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 msgr2=0x7f863813a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 --2- 172.21.15.181:0/3475343676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f863813a280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8630009fd0 tx=0x7f8630009840 comp rx=0 tx=0).stop 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 -- 172.21.15.181:0/3475343676 shutdown_connections 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 --2- 172.21.15.181:0/3475343676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f863813a280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 -- 172.21.15.181:0/3475343676 >> 172.21.15.181:0/3475343676 conn(0x7f863809e9f0 msgr2=0x7f863809edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 -- 172.21.15.181:0/3475343676 shutdown_connections 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.419+0000 7f863dcc9640 1 -- 172.21.15.181:0/3475343676 wait complete. 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.420+0000 7f863dcc9640 10 monclient: build_initial_monmap 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.420+0000 7f863dcc9640 10 monclient: monmap: 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:19.425 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:19.425 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:19.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.420+0000 7f863dcc9640 1 Processor -- start 2024-09-15T03:36:19.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.420+0000 7f863dcc9640 1 -- start start 2024-09-15T03:36:19.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.420+0000 7f863dcc9640 10 monclient: init 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8638069a30 0x7f86381344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): picked mon.noname-a con 0x7f8638069a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86380983e0 0x7f86381349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): picked mon.noname-c con 0x7f86380983e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f8638134f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): picked mon.noname-b con 0x7f8638137ea0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638136590 con 0x7f8638069a30 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86381190e0 con 0x7f8638137ea0 2024-09-15T03:36:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863809d8a0 con 0x7f86380983e0 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f863dcc9640 10 monclient(hunting): authenticate will time out at 1183.902268s 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f8638134f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f8638134f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36246/0 (socket says 172.21.15.181:36246) 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f86377fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8638069a30 msgr2=0x7f86381344a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 -- 172.21.15.181:0/706230808 learned_addr learned my addr 172.21.15.181:0/706230808 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): get_auth_request con 0x7f8638137ea0 auth_method 0 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f86377fe640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8638069a30 0x7f86381344a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8636ffd640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86380983e0 0x7f86381349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8636ffd640 10 monclient(hunting): get_auth_request con 0x7f86380983e0 auth_method 0 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient(hunting): handle_auth_done global_id 104260 payload 824 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 -- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86380983e0 msgr2=0x7f86381349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86380983e0 0x7f86381349e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 -- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8638069a30 msgr2=0x7f86381344a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8638069a30 0x7f86381344a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 monclient: found mon.noname-b 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8637fff640 1 -- 172.21.15.181:0/706230808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630009ae0 con 0x7f8638137ea0 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.421+0000 7f8636ffd640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86380983e0 0x7f86381349e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.423+0000 7f8637fff640 1 -- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 msgr2=0x7f8638134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.423+0000 7f8637fff640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f8638134f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8630009ab0 tx=0x7f8630034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630032e10 con 0x7f8638137ea0 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:19.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: dump: 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 5 monclient: authenticate success, global_id 104260 2024-09-15T03:36:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863003d430 con 0x7f8638137ea0 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 monclient: _renew_subs 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863003d7d0 con 0x7f8638137ea0 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863813a370 con 0x7f8638137ea0 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 10 monclient: dump: 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.435 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f8635ffb640 4 set_mon_vals no callback set 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 monclient: _renew_subs 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8638136010 con 0x7f8638137ea0 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f863003db80 con 0x7f8638137ea0 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.424+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863804bf60 con 0x7f8638137ea0 2024-09-15T03:36:19.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.425+0000 7f862f7fe640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f86180386f0 0x7f861803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.425+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(193..193 src has 106..193) v4 ==== 9753+0+0 (secure 0 0 0) 0x7f863006e030 con 0x7f8638137ea0 2024-09-15T03:36:19.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.425+0000 7f86377fe640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f86180386f0 0x7f861803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.425+0000 7f86377fe640 10 monclient: get_auth_request con 0x7f86180386f0 auth_method 0 2024-09-15T03:36:19.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.427+0000 7f86377fe640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f86180386f0 0x7f861803aba0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f86280014b0 tx=0x7f8628007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:19.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.429+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f863009fe40 con 0x7f8638137ea0 2024-09-15T03:36:19.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.429+0000 7f862f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:19.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.429+0000 7f862f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:19.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.587+0000 7f863dcc9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:19.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.587+0000 7f863dcc9640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:19.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.587+0000 7f863dcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:19.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:19.587+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f863809e320 con 0x7f8638137ea0 2024-09-15T03:36:19.716 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:36:19.717 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:36:19.717 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:36:19.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:36:19.721 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:36:19.721 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:36:19.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.824+0000 7f902a8af640 10 monclient: get_monmap_and_config 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.824+0000 7f902a8af640 10 monclient: build_initial_monmap 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.824+0000 7f902a8af640 10 monclient: monmap: 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:19.825636+0000 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:19.825636+0000 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:19.827 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:19.827 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:19.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:19.827 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:19.827 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:19.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.825+0000 7f902a8af640 1 Processor -- start 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.825+0000 7f902a8af640 1 -- start start 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.825+0000 7f902a8af640 10 monclient: init 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90241318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f902409ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f902413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient(hunting): picked mon.noname-b con 0x7f9024137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9024132640 con 0x7f902409ea50 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9024131cc0 con 0x7f90241318f0 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902409f530 con 0x7f9024137c30 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f902a8af640 10 monclient(hunting): authenticate will time out at 1177.465232s 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f90237fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f902409ea50 0x7f902409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:57014/0 (socket says 172.21.15.153:57014) 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f902413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f90237fe640 1 -- 172.21.15.153:0/4025733021 learned_addr learned my addr 172.21.15.153:0/4025733021 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): get_auth_request con 0x7f9024137c30 auth_method 0 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient(hunting): handle_auth_done global_id 74795 payload 274 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 1 -- 172.21.15.153:0/4025733021 >> v1:172.21.15.181:6789/0 conn(0x7f90241318f0 legacy=0x7f9024131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 1 -- 172.21.15.153:0/4025733021 >> v1:172.21.15.153:6789/0 conn(0x7f902409ea50 legacy=0x7f902409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 1 monclient: found mon.noname-b 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.826+0000 7f9028e4c640 1 -- 172.21.15.153:0/4025733021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902409fbf0 con 0x7f9024137c30 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9028e4c640 1 -- 172.21.15.153:0/4025733021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 msgr2=0x7f902413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9028e4c640 1 --2- 172.21.15.153:0/4025733021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f902413a010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f901c009e90 tx=0x7f901c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13d75d14f11581e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:19.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 1 -- 172.21.15.153:0/4025733021 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c007840 con 0x7f9024137c30 2024-09-15T03:36:19.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:19.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: dump: 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 1 -- 172.21.15.153:0/4025733021 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c02d050 con 0x7f9024137c30 2024-09-15T03:36:19.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f902a8af640 5 monclient: authenticate success, global_id 74795 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f902a8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f902a8af640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 1 -- 172.21.15.153:0/4025733021 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c007bd0 con 0x7f9024137c30 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f902a8af640 4 set_mon_vals no callback set 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.829+0000 7f9022ffd640 10 monclient: dump: 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 10 monclient: shutdown 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 -- 172.21.15.153:0/4025733021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 msgr2=0x7f902413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 --2- 172.21.15.153:0/4025733021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f902413a010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f901c009e90 tx=0x7f901c009780 comp rx=0 tx=0).stop 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 -- 172.21.15.153:0/4025733021 shutdown_connections 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 --2- 172.21.15.153:0/4025733021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f902413a010 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 -- 172.21.15.153:0/4025733021 >> 172.21.15.153:0/4025733021 conn(0x7f9024067110 msgr2=0x7f9024067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 -- 172.21.15.153:0/4025733021 shutdown_connections 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 1 -- 172.21.15.153:0/4025733021 wait complete. 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 10 monclient: build_initial_monmap 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.830+0000 7f902a8af640 10 monclient: monmap: 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 Processor -- start 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 -- start start 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient: init 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f902409ea50 0x7f9024136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): picked mon.noname-b con 0x7f902409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): picked mon.noname-a con 0x7f90241318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f9024130410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): picked mon.noname-c con 0x7f9024137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90241347f0 con 0x7f90241318f0 2024-09-15T03:36:19.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902413a2c0 con 0x7f902409ea50 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9024134f20 con 0x7f9024137c30 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f902a8af640 10 monclient(hunting): authenticate will time out at 1177.470905s 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f90237fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.831+0000 7f9028e4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f9024130410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f9024130410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50972/0 (socket says 172.21.15.153:50972) 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 -- 172.21.15.153:0/198064400 learned_addr learned my addr 172.21.15.153:0/198064400 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38474/0 (socket says 172.21.15.153:38474) 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): get_auth_request con 0x7f9024137c30 auth_method 0 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 10 monclient(hunting): get_auth_request con 0x7f90241318f0 auth_method 0 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 msgr2=0x7f902412fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 msgr2=0x7f902412fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f90237fe640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:19.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9023fff640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f902409ea50 0x7f9024136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9023fff640 10 monclient(hunting): get_auth_request con 0x7f902409ea50 auth_method 0 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9023fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient(hunting): handle_auth_done global_id 74798 payload 824 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f902409ea50 msgr2=0x7f9024136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f902409ea50 0x7f9024136310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 msgr2=0x7f902412fed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:19.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 monclient: found mon.noname-c 2024-09-15T03:36:19.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:19.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:19.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.832+0000 7f9028e4c640 1 -- 172.21.15.153:0/198064400 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c009a20 con 0x7f9024137c30 2024-09-15T03:36:19.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f9028e4c640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 msgr2=0x7f9024130410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f9028e4c640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f9024130410 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f901c009de0 tx=0x7f901c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c007ec0 con 0x7f9024137c30 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: dump: 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: _finish_auth 0 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 5 monclient: authenticate success, global_id 74798 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c040070 con 0x7f9024137c30 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c045480 con 0x7f9024137c30 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 10 monclient: _renew_subs 2024-09-15T03:36:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9024134510 con 0x7f9024137c30 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f900bfff640 10 monclient: dump: 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:19.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f90227fc640 4 set_mon_vals no callback set 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 10 monclient: _renew_subs 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.834+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9024137300 con 0x7f9024137c30 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.835+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f901c042030 con 0x7f9024137c30 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.835+0000 7f902a8af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.835+0000 7f902a8af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.835+0000 7f902a8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.835+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90241374f0 con 0x7f9024137c30 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.836+0000 7f900bfff640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9010038740 0x7f901003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.836+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(193..193 src has 106..193) v4 ==== 9753+0+0 (secure 0 0 0) 0x7f901c06e710 con 0x7f9024137c30 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.836+0000 7f9023fff640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9010038740 0x7f901003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.836+0000 7f9023fff640 10 monclient: get_auth_request con 0x7f9010038740 auth_method 0 2024-09-15T03:36:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.838+0000 7f9023fff640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9010038740 0x7f901003abf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f90140037d0 tx=0x7f9014009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.840+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f901c04edf0 con 0x7f9024137c30 2024-09-15T03:36:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.840+0000 7f900bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:19.840+0000 7f900bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f902a8af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f902a8af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f902a8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f902404bf60 con 0x7f9024137c30 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f900bfff640 1 -- 172.21.15.153:0/198064400 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f901c04a070 con 0x7f9024137c30 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f900bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:20.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.059+0000 7f900bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:20.061 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9010038740 msgr2=0x7f901003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9010038740 0x7f901003abf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f90140037d0 tx=0x7f9014009040 comp rx=0 tx=0).stop 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 10 monclient: shutdown 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 msgr2=0x7f9024130410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f9024130410 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f901c009de0 tx=0x7f901c037000 comp rx=0 tx=0).stop 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 shutdown_connections 2024-09-15T03:36:20.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9010038740 0x7f901003abf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9024137c30 0x7f9024130410 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90241318f0 0x7f902412fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 --2- 172.21.15.153:0/198064400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f902409ea50 0x7f9024136310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 >> 172.21.15.153:0/198064400 conn(0x7f9024067110 msgr2=0x7f902412b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:20.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 shutdown_connections 2024-09-15T03:36:20.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:20.061+0000 7f902a8af640 1 -- 172.21.15.153:0/198064400 wait complete. 2024-09-15T03:36:20.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.234+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v194) v1 ==== 168+0+0 (secure 0 0 0) 0x7f86300398c0 con 0x7f8638137ea0 2024-09-15T03:36:20.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.234+0000 7f862f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:20.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.234+0000 7f862f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-15T03:36:20.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.330+0000 7f863dcc9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:20.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.330+0000 7f863dcc9640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:20.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.330+0000 7f863dcc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:20.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.330+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f863809e320 con 0x7f8638137ea0 2024-09-15T03:36:20.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.331+0000 7f862f7fe640 1 -- 172.21.15.181:0/706230808 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v194) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8630030300 con 0x7f8638137ea0 2024-09-15T03:36:20.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.331+0000 7f862f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.331+0000 7f862f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-15T03:36:20.334 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h10' already exists 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.332+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f86180386f0 msgr2=0x7f861803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.332+0000 7f863dcc9640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f86180386f0 0x7f861803aba0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f86280014b0 tx=0x7f8628007a10 comp rx=0 tx=0).stop 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.332+0000 7f863dcc9640 10 monclient: shutdown 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.332+0000 7f863dcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.332+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 msgr2=0x7f8638134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.332+0000 7f863dcc9640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f8638134f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8630009ab0 tx=0x7f8630034000 comp rx=0 tx=0).stop 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 shutdown_connections 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f86180386f0 0x7f861803aba0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8638137ea0 0x7f8638134f20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86380983e0 0x7f86381349e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 --2- 172.21.15.181:0/706230808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8638069a30 0x7f86381344a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 >> 172.21.15.181:0/706230808 conn(0x7f863809e9f0 msgr2=0x7f86380a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 shutdown_connections 2024-09-15T03:36:20.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.333+0000 7f863dcc9640 1 -- 172.21.15.181:0/706230808 wait complete. 2024-09-15T03:36:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:20.343 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-15T03:36:20.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.416+0000 7f07b06dd640 10 monclient: get_monmap_and_config 2024-09-15T03:36:20.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.416+0000 7f07b06dd640 10 monclient: build_initial_monmap 2024-09-15T03:36:20.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.416+0000 7f07b06dd640 10 monclient: monmap: 2024-09-15T03:36:20.417 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:20.417445+0000 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:20.417445+0000 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:20.418 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 Processor -- start 2024-09-15T03:36:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 -- start start 2024-09-15T03:36:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient: init 2024-09-15T03:36:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07a806df30 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 0x7f07a806e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): picked mon.noname-a con 0x7f07a806a6b0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 0x7f07a809e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): picked mon.noname-b con 0x7f07a809e1e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a806e840 con 0x7f07a806df30 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a806aab0 con 0x7f07a806a6b0 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a806a290 con 0x7f07a809e1e0 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.417+0000 7f07b06dd640 10 monclient(hunting): authenticate will time out at 1184.898901s 2024-09-15T03:36:20.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07ae479640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f07a806df30 0x7f07a809dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56538/0 (socket says 172.21.15.181:56538) 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07ae479640 1 -- 172.21.15.181:0/2696469978 learned_addr learned my addr 172.21.15.181:0/2696469978 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 1 --2- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 0x7f07a806e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 10 monclient(hunting): get_auth_request con 0x7f07a806a6b0 auth_method 0 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 1 -- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 msgr2=0x7f07a806e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 1 -- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 msgr2=0x7f07a806e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 1 --2- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 0x7f07a806e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.418+0000 7f07adc78640 1 --2- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 0x7f07a806e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 1 --2- 172.21.15.181:0/2696469978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 0x7f07a809e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): get_auth_request con 0x7f07a809e1e0 auth_method 0 2024-09-15T03:36:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient(hunting): handle_auth_done global_id 74801 payload 274 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 1 -- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 msgr2=0x7f07a806e300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 1 --2- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 0x7f07a806e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 1 -- 172.21.15.181:0/2696469978 >> v1:172.21.15.181:6789/0 conn(0x7f07a806df30 legacy=0x7f07a809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 1 monclient: found mon.noname-b 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:20.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:20.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.419+0000 7f07aec7a640 1 -- 172.21.15.181:0/2696469978 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80696e0 con 0x7f07a809e1e0 2024-09-15T03:36:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07aec7a640 1 -- 172.21.15.181:0/2696469978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 msgr2=0x7f07a809e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07aec7a640 1 --2- 172.21.15.181:0/2696469978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 0x7f07a809e640 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009e60 tx=0x7f07a0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b704e827ea3a5c7c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 1 -- 172.21.15.181:0/2696469978 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0007760 con 0x7f07a809e1e0 2024-09-15T03:36:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: dump: 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: _finish_auth 0 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 1 -- 172.21.15.181:0/2696469978 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a002d050 con 0x7f07a809e1e0 2024-09-15T03:36:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07b06dd640 5 monclient: authenticate success, global_id 74801 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07b06dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07b06dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 1 -- 172.21.15.181:0/2696469978 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0007af0 con 0x7f07a809e1e0 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07b06dd640 4 set_mon_vals no callback set 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.422+0000 7f07ad477640 10 monclient: dump: 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 10 monclient: shutdown 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 -- 172.21.15.181:0/2696469978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 msgr2=0x7f07a809e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2696469978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 0x7f07a809e640 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009e60 tx=0x7f07a0009780 comp rx=0 tx=0).stop 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 -- 172.21.15.181:0/2696469978 shutdown_connections 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2696469978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a809e1e0 0x7f07a809e640 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2696469978 >> v2:172.21.15.153:3300/0 conn(0x7f07a806a6b0 0x7f07a806e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 -- 172.21.15.181:0/2696469978 >> 172.21.15.181:0/2696469978 conn(0x7f07a80a3b30 msgr2=0x7f07a809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 -- 172.21.15.181:0/2696469978 shutdown_connections 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 1 -- 172.21.15.181:0/2696469978 wait complete. 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 10 monclient: build_initial_monmap 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.423+0000 7f07b06dd640 10 monclient: monmap: 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:20.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 Processor -- start 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 -- start start 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient: init 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 0x7f07a813d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): picked mon.noname-a con 0x7f07a806a6b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a806df30 0x7f07a813cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): picked mon.noname-c con 0x7f07a806df30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 0x7f07a8140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): picked mon.noname-b con 0x7f07a809e1e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a8122890 con 0x7f07a806a6b0 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81222d0 con 0x7f07a809e1e0 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a811fdf0 con 0x7f07a806df30 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07b06dd640 10 monclient(hunting): authenticate will time out at 1184.905733s 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 0x7f07a8140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 0x7f07a8140e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36256/0 (socket says 172.21.15.181:36256) 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 1 -- 172.21.15.181:0/2634290074 learned_addr learned my addr 172.21.15.181:0/2634290074 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): get_auth_request con 0x7f07a809e1e0 auth_method 0 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07aec7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07ae479640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 0x7f07a813d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.424+0000 7f07adc78640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a806df30 0x7f07a813cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): get_auth_request con 0x7f07a806df30 auth_method 0 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 10 monclient(hunting): get_auth_request con 0x7f07a806a6b0 auth_method 0 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07adc78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:20.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 msgr2=0x7f07a813d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 msgr2=0x7f07a813d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 0x7f07a813d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 10 monclient(hunting): handle_auth_done global_id 104266 payload 824 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07ae479640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 0x7f07a813d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a806df30 msgr2=0x7f07a813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a806df30 0x7f07a813cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 msgr2=0x7f07a813d8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 0x7f07a813d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 1 monclient: found mon.noname-b 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.425+0000 7f07aec7a640 1 -- 172.21.15.181:0/2634290074 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0009a20 con 0x7f07a809e1e0 2024-09-15T03:36:20.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07aec7a640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 msgr2=0x7f07a8140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07aec7a640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 0x7f07a8140e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07a00099f0 tx=0x7f07a0037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0031c10 con 0x7f07a809e1e0 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: dump: 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:20.439 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0040070 con 0x7f07a809e1e0 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 5 monclient: authenticate success, global_id 104266 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 10 monclient: _renew_subs 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a8121ff0 con 0x7f07a809e1e0 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0045430 con 0x7f07a809e1e0 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f079e7fc640 10 monclient: dump: 2024-09-15T03:36:20.440 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07acc76640 4 set_mon_vals no callback set 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 10 monclient: _renew_subs 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:20.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.427+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a813d630 con 0x7f07a809e1e0 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f07a0042030 con 0x7f07a809e1e0 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f07b06dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f07b06dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f07b06dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a804bf60 con 0x7f07a809e1e0 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f079e7fc640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f078c038740 0x7f078c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(194..194 src has 106..194) v4 ==== 9966+0+0 (secure 0 0 0) 0x7f07a006e6e0 con 0x7f07a809e1e0 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f07ae479640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f078c038740 0x7f078c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:20.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.428+0000 7f07ae479640 10 monclient: get_auth_request con 0x7f078c038740 auth_method 0 2024-09-15T03:36:20.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.431+0000 7f07ae479640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f078c038740 0x7f078c03abf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f07a4007d10 tx=0x7f07a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:20.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.432+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f07a003b420 con 0x7f07a809e1e0 2024-09-15T03:36:20.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.432+0000 7f079e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:20.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.432+0000 7f079e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:20.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.590+0000 7f07b06dd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:20.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.590+0000 7f07b06dd640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:20.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.590+0000 7f07b06dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:20.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:20.590+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f07a806e840 con 0x7f07a809e1e0 2024-09-15T03:36:21.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.238+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v195) v1 ==== 168+0+0 (secure 0 0 0) 0x7f07a806e840 con 0x7f07a809e1e0 2024-09-15T03:36:21.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.238+0000 7f079e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:21.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.238+0000 7f079e7fc640 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.333+0000 7f07b06dd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.333+0000 7f07b06dd640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.333+0000 7f07b06dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.333+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f07a806e840 con 0x7f07a809e1e0 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.334+0000 7f079e7fc640 1 -- 172.21.15.181:0/2634290074 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v195) v1 ==== 147+0+0 (secure 0 0 0) 0x7f07a806e840 con 0x7f07a809e1e0 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.334+0000 7f079e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.334+0000 7f079e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-15T03:36:21.335 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h11' already exists 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f078c038740 msgr2=0x7f078c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f078c038740 0x7f078c03abf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f07a4007d10 tx=0x7f07a4009040 comp rx=0 tx=0).stop 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 10 monclient: shutdown 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 msgr2=0x7f07a8140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 0x7f07a8140e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07a00099f0 tx=0x7f07a0037000 comp rx=0 tx=0).stop 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 shutdown_connections 2024-09-15T03:36:21.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f078c038740 0x7f078c03abf0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07a809e1e0 0x7f07a8140e10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a806df30 0x7f07a813cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 --2- 172.21.15.181:0/2634290074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07a806a6b0 0x7f07a813d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 >> 172.21.15.181:0/2634290074 conn(0x7f07a80a3b30 msgr2=0x7f07a806c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:21.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 shutdown_connections 2024-09-15T03:36:21.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.336+0000 7f07b06dd640 1 -- 172.21.15.181:0/2634290074 wait complete. 2024-09-15T03:36:21.345 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:21.345 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-15T03:36:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.418+0000 7f9ac3b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.418+0000 7f9ac3b5d640 10 monclient: build_initial_monmap 2024-09-15T03:36:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.418+0000 7f9ac3b5d640 10 monclient: monmap: 2024-09-15T03:36:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:21.420 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:21.419753+0000 2024-09-15T03:36:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:21.419753+0000 2024-09-15T03:36:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:21.421 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:21.422 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:21.422 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:21.422 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:21.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.419+0000 7f9ac3b5d640 1 Processor -- start 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.419+0000 7f9ac3b5d640 1 -- start start 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.419+0000 7f9ac3b5d640 10 monclient: init 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9abc099e10 0x7f9abc09e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f9abc099e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:21.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9abc09dfa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9abc09eb90 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc09f620 con 0x7f9abc09dfa0 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc09f860 con 0x7f9abc09eb90 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc09b7c0 con 0x7f9abc099e10 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac3b5d640 10 monclient(hunting): authenticate will time out at 1185.901474s 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9abc099e10 0x7f9abc09e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:21.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9abc099e10 0x7f9abc09e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36268/0 (socket says 172.21.15.181:36268) 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 1 -- 172.21.15.181:0/4042470498 learned_addr learned my addr 172.21.15.181:0/4042470498 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): get_auth_request con 0x7f9abc099e10 auth_method 0 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:21.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.420+0000 7f9ac18f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac20fa640 1 -- 172.21.15.181:0/4042470498 >> v1:172.21.15.181:6789/0 conn(0x7f9abc09eb90 legacy=0x7f9abc09c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 10 monclient(hunting): handle_auth_done global_id 104269 payload 274 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 1 -- 172.21.15.181:0/4042470498 >> v1:172.21.15.181:6789/0 conn(0x7f9abc09eb90 legacy=0x7f9abc09c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 1 -- 172.21.15.181:0/4042470498 >> v1:172.21.15.153:6789/0 conn(0x7f9abc09dfa0 legacy=0x7f9abc09c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 10 monclient: _finish_hunting 0 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 1 monclient: found mon.noname-c 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:21.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.422+0000 7f9ac18f9640 1 -- 172.21.15.181:0/4042470498 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc09b4e0 con 0x7f9abc099e10 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac18f9640 1 -- 172.21.15.181:0/4042470498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc099e10 msgr2=0x7f9abc09e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac18f9640 1 --2- 172.21.15.181:0/4042470498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc099e10 0x7f9abc09e650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0648a0 tx=0x7f9ab80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8895bc6a9ab54dee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 1 -- 172.21.15.181:0/4042470498 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802f4b0 con 0x7f9abc099e10 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: dump: 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: _finish_auth 0 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac3b5d640 5 monclient: authenticate success, global_id 104269 2024-09-15T03:36:21.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 1 -- 172.21.15.181:0/4042470498 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab802f610 con 0x7f9abc099e10 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac3b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac3b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac3b5d640 4 set_mon_vals no callback set 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 1 -- 172.21.15.181:0/4042470498 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802f9b0 con 0x7f9abc099e10 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.425+0000 7f9ac08f7640 10 monclient: dump: 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.442 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 10 monclient: shutdown 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/4042470498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc099e10 msgr2=0x7f9abc09e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/4042470498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc099e10 0x7f9abc09e650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0648a0 tx=0x7f9ab80091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/4042470498 shutdown_connections 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/4042470498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc099e10 0x7f9abc09e650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/4042470498 >> 172.21.15.181:0/4042470498 conn(0x7f9abc0a3b30 msgr2=0x7f9abc09fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/4042470498 shutdown_connections 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.426+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/4042470498 wait complete. 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient: build_initial_monmap 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient: monmap: 2024-09-15T03:36:21.444 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 Processor -- start 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 -- start start 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient: init 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:21.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9abc099e10 0x7f9abc1411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f9abc099e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9abc09eb90 0x7f9abc134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f9abc09eb90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 0x7f9abc11eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f9abc135510 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc122740 con 0x7f9abc09eb90 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc06eb80 con 0x7f9abc135510 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc069d40 con 0x7f9abc099e10 2024-09-15T03:36:21.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac3b5d640 10 monclient(hunting): authenticate will time out at 1185.908893s 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.427+0000 7f9ac20fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 0x7f9abc11eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 0x7f9abc11eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36272/0 (socket says 172.21.15.181:36272) 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 -- 172.21.15.181:0/3326781180 learned_addr learned my addr 172.21.15.181:0/3326781180 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): get_auth_request con 0x7f9abc135510 auth_method 0 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:21.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac18f9640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9abc099e10 0x7f9abc1411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient(hunting): handle_auth_done global_id 104272 payload 824 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac10f8640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9abc09eb90 0x7f9abc134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 -- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9abc099e10 msgr2=0x7f9abc1411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9abc099e10 0x7f9abc1411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 -- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9abc09eb90 msgr2=0x7f9abc134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9abc09eb90 0x7f9abc134fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:21.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient: _finish_hunting 0 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 monclient: found mon.noname-b 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.428+0000 7f9ac20fa640 1 -- 172.21.15.181:0/3326781180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009450 con 0x7f9abc135510 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac20fa640 1 -- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 msgr2=0x7f9abc11eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac20fa640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 0x7f9abc11eb60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c840 tx=0x7f9ab40018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4001950 con 0x7f9abc135510 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: dump: 2024-09-15T03:36:21.449 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:21.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:36:21.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:21.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 5 monclient: authenticate success, global_id 104272 2024-09-15T03:36:21.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab400a640 con 0x7f9abc135510 2024-09-15T03:36:21.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:21.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 10 monclient: _renew_subs 2024-09-15T03:36:21.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab400aa00 con 0x7f9abc135510 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc120ad0 con 0x7f9abc135510 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab1ffb640 10 monclient: dump: 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:21.452 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:21.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ab3fff640 4 set_mon_vals no callback set 2024-09-15T03:36:21.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 10 monclient: _renew_subs 2024-09-15T03:36:21.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.430+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc142020 con 0x7f9abc135510 2024-09-15T03:36:21.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ab40070f0 con 0x7f9abc135510 2024-09-15T03:36:21.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ab1ffb640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9aa0038740 0x7f9aa003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:21.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(195..195 src has 106..195) v4 ==== 10179+0+0 (secure 0 0 0) 0x7f9ab404ce40 con 0x7f9abc135510 2024-09-15T03:36:21.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ac3b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:21.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ac3b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:21.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ac3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc04bf60 con 0x7f9abc135510 2024-09-15T03:36:21.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ac18f9640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9aa0038740 0x7f9aa003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:21.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.431+0000 7f9ac18f9640 10 monclient: get_auth_request con 0x7f9aa0038740 auth_method 0 2024-09-15T03:36:21.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.434+0000 7f9ac18f9640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9aa0038740 0x7f9aa003abf0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0a0f90 tx=0x7f9ab8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:21.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.436+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ab400ae30 con 0x7f9abc135510 2024-09-15T03:36:21.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.436+0000 7f9ab1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:21.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.436+0000 7f9ab1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:21.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.594+0000 7f9ac3b5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:21.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.594+0000 7f9ac3b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:21.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.594+0000 7f9ac3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:21.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:21.594+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9abc064a90 con 0x7f9abc135510 2024-09-15T03:36:22.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.242+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v196) v1 ==== 168+0+0 (secure 0 0 0) 0x7f9ab4010770 con 0x7f9abc135510 2024-09-15T03:36:22.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.242+0000 7f9ab1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:22.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.242+0000 7f9ab1ffb640 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-15T03:36:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ac3b5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ac3b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ac3b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9abc121190 con 0x7f9abc135510 2024-09-15T03:36:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ab1ffb640 1 -- 172.21.15.181:0/3326781180 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v196) v1 ==== 147+0+0 (secure 0 0 0) 0x7f9ab4010950 con 0x7f9abc135510 2024-09-15T03:36:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ab1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:22.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.344+0000 7f9ab1ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-15T03:36:22.346 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h12' already exists 2024-09-15T03:36:22.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9aa0038740 msgr2=0x7f9aa003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9aa0038740 0x7f9aa003abf0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0a0f90 tx=0x7f9ab8032000 comp rx=0 tx=0).stop 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 10 monclient: shutdown 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 msgr2=0x7f9abc11eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 0x7f9abc11eb60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c840 tx=0x7f9ab40018c0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 shutdown_connections 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9aa0038740 0x7f9aa003abf0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9abc135510 0x7f9abc11eb60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9abc09eb90 0x7f9abc134fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 --2- 172.21.15.181:0/3326781180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9abc099e10 0x7f9abc1411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 >> 172.21.15.181:0/3326781180 conn(0x7f9abc0a3b30 msgr2=0x7f9abc0a0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:22.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 shutdown_connections 2024-09-15T03:36:22.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.346+0000 7f9ac3b5d640 1 -- 172.21.15.181:0/3326781180 wait complete. 2024-09-15T03:36:22.357 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:22.357 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.431+0000 7ff11e554640 10 monclient: get_monmap_and_config 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.431+0000 7ff11e554640 10 monclient: build_initial_monmap 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.431+0000 7ff11e554640 10 monclient: monmap: 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:22.432002+0000 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:22.432002+0000 2024-09-15T03:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:22.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.431+0000 7ff11e554640 1 Processor -- start 2024-09-15T03:36:22.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.431+0000 7ff11e554640 1 -- start start 2024-09-15T03:36:22.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.431+0000 7ff11e554640 10 monclient: init 2024-09-15T03:36:22.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:22.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:36:22.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff118131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient(hunting): picked mon.noname-b con 0x7ff118131bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff11809ebe0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff11809f000 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180a06d0 con 0x7ff11809f000 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff118071780 con 0x7ff11809ebe0 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180a0010 con 0x7ff118131bd0 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff11e554640 10 monclient(hunting): authenticate will time out at 1186.913485s 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff1177fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7ff11809ebe0 0x7ff118068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56562/0 (socket says 172.21.15.181:56562) 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff1177fe640 1 -- 172.21.15.181:0/2643991516 learned_addr learned my addr 172.21.15.181:0/2643991516 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 1 --2- 172.21.15.181:0/2643991516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff118131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): get_auth_request con 0x7ff118131bd0 auth_method 0 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.432+0000 7ff117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 10 monclient(hunting): handle_auth_done global_id 74807 payload 274 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 1 -- 172.21.15.181:0/2643991516 >> v1:172.21.15.181:6789/0 conn(0x7ff11809ebe0 legacy=0x7ff118068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 1 -- 172.21.15.181:0/2643991516 >> v1:172.21.15.153:6789/0 conn(0x7ff11809f000 legacy=0x7ff118069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 1 monclient: found mon.noname-b 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:22.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.433+0000 7ff117fff640 1 -- 172.21.15.181:0/2643991516 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180a1910 con 0x7ff118131bd0 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff117fff640 1 -- 172.21.15.181:0/2643991516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 msgr2=0x7ff118131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff117fff640 1 --2- 172.21.15.181:0/2643991516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff118131fa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff108009880 tx=0x7ff1080090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d993604f505e28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 1 -- 172.21.15.181:0/2643991516 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108032440 con 0x7ff118131bd0 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: dump: 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 1 -- 172.21.15.181:0/2643991516 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1080325a0 con 0x7ff118131bd0 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 1 -- 172.21.15.181:0/2643991516 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108032940 con 0x7ff118131bd0 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff116ffd640 10 monclient: dump: 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff11e554640 5 monclient: authenticate success, global_id 74807 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff11e554640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.436+0000 7ff11e554640 4 set_mon_vals no callback set 2024-09-15T03:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 10 monclient: shutdown 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 -- 172.21.15.181:0/2643991516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 msgr2=0x7ff118131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 --2- 172.21.15.181:0/2643991516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff118131fa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff108009880 tx=0x7ff1080090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 -- 172.21.15.181:0/2643991516 shutdown_connections 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 --2- 172.21.15.181:0/2643991516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff118131fa0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 -- 172.21.15.181:0/2643991516 >> 172.21.15.181:0/2643991516 conn(0x7ff1180713b0 msgr2=0x7ff1180a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 -- 172.21.15.181:0/2643991516 shutdown_connections 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 -- 172.21.15.181:0/2643991516 wait complete. 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 10 monclient: build_initial_monmap 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 10 monclient: monmap: 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.437+0000 7ff11e554640 1 Processor -- start 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 -- start start 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient: init 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 0x7ff11809ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): picked mon.noname-a con 0x7ff11809ebe0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:22.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 0x7ff118130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): picked mon.noname-b con 0x7ff11809f000 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff11811b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): picked mon.noname-c con 0x7ff118131bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff118142050 con 0x7ff11809ebe0 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11809e660 con 0x7ff11809f000 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11811e0f0 con 0x7ff118131bd0 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11e554640 10 monclient(hunting): authenticate will time out at 1186.919482s 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 0x7ff118130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 0x7ff118130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36288/0 (socket says 172.21.15.181:36288) 2024-09-15T03:36:22.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 -- 172.21.15.181:0/2230971574 learned_addr learned my addr 172.21.15.181:0/2230971574 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): get_auth_request con 0x7ff11809f000 auth_method 0 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff11811b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): get_auth_request con 0x7ff118131bd0 auth_method 0 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 0x7ff11809ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 10 monclient(hunting): get_auth_request con 0x7ff11809ebe0 auth_method 0 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff11caf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 msgr2=0x7ff11809ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 msgr2=0x7ff11809ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 0x7ff11809ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff117fff640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 0x7ff11809ab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient(hunting): handle_auth_done global_id 104275 payload 824 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 msgr2=0x7ff11811b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff11811b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 msgr2=0x7ff11809ab10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 0x7ff11809ab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient: _finish_hunting 0 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 monclient: found mon.noname-b 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.438+0000 7ff1177fe640 1 -- 172.21.15.181:0/2230971574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108009390 con 0x7ff11809f000 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff1177fe640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 msgr2=0x7ff118130c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff1177fe640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 0x7ff118130c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c5e0 tx=0x7ff100004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100005cb0 con 0x7ff11809f000 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: dump: 2024-09-15T03:36:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: _finish_auth 0 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 5 monclient: authenticate success, global_id 104275 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 10 monclient: _renew_subs 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:22.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11811dd90 con 0x7ff11809f000 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff100005e30 con 0x7ff11809f000 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100007340 con 0x7ff11809f000 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff10ffff640 10 monclient: dump: 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 10 monclient: _renew_subs 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff1167fc640 4 set_mon_vals no callback set 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.441+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11811e400 con 0x7ff11809f000 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff10001b400 con 0x7ff11809f000 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff11e554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff11e554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff11e554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11804bf60 con 0x7ff11809f000 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff10ffff640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0f8038740 0x7ff0f803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(196..196 src has 106..196) v4 ==== 10392+0+0 (secure 0 0 0) 0x7ff10004cc80 con 0x7ff11809f000 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff117fff640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0f8038740 0x7ff0f803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.442+0000 7ff117fff640 10 monclient: get_auth_request con 0x7ff0f8038740 auth_method 0 2024-09-15T03:36:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.445+0000 7ff117fff640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0f8038740 0x7ff0f803abf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff1180a1240 tx=0x7ff108032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.446+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff10001b6b0 con 0x7ff11809f000 2024-09-15T03:36:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.446+0000 7ff10ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.446+0000 7ff10ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:22.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.604+0000 7ff11e554640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:22.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.604+0000 7ff11e554640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:22.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.604+0000 7ff11e554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:22.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:22.604+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff118064990 con 0x7ff11809f000 2024-09-15T03:36:23.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.178+0000 7f3bfa35d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.178+0000 7f3bfa35d640 10 monclient: build_initial_monmap 2024-09-15T03:36:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.178+0000 7f3bfa35d640 10 monclient: monmap: 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:23.179543+0000 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:23.179543+0000 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:23.180 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.179+0000 7f3bfa35d640 1 Processor -- start 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.179+0000 7f3bfa35d640 1 -- start start 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.179+0000 7f3bfa35d640 10 monclient: init 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 0x7f3bf4096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bf409a510 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bf40971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bf40975e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf409cb00 con 0x7f3bf40975e0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf409a8e0 con 0x7f3bf40971e0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf409d590 con 0x7f3bf409a510 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bfa35d640 10 monclient(hunting): authenticate will time out at 1180.819265s 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 0x7f3bf4096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf88fa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3bf40975e0 0x7f3bf412fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:57022/0 (socket says 172.21.15.153:57022) 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf88fa640 1 -- 172.21.15.153:0/1459666681 learned_addr learned my addr 172.21.15.153:0/1459666681 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 10 monclient(hunting): get_auth_request con 0x7f3bf409a510 auth_method 0 2024-09-15T03:36:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 1 -- 172.21.15.153:0/1459666681 >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 msgr2=0x7f3bf4096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 1 -- 172.21.15.153:0/1459666681 >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 msgr2=0x7f3bf4096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/1459666681 >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 0x7f3bf4096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.180+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/1459666681 >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 0x7f3bf4096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.182+0000 7f3bf88fa640 1 -- 172.21.15.153:0/1459666681 >> v1:172.21.15.153:6789/0 conn(0x7f3bf40975e0 legacy=0x7f3bf412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.182+0000 7f3bf37fe640 1 -- 172.21.15.153:0/1459666681 >> v1:172.21.15.181:6789/0 conn(0x7f3bf40971e0 legacy=0x7f3bf412f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737863063 0 0) 0x7f3bf409a8e0 con 0x7f3bf40971e0 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): my global_id is 104278 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bdc0034e0 con 0x7f3bf40971e0 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2294810478 0 0) 0x7f3bdc0034e0 con 0x7f3bf40971e0 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 msgr2=0x7f3bf4096ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 --2- 172.21.15.153:0/1459666681 >> v2:172.21.15.153:3300/0 conn(0x7f3bf409a510 0x7f3bf4096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 >> v1:172.21.15.153:6789/0 conn(0x7f3bf40975e0 legacy=0x7f3bf412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf409c0e0 con 0x7f3bf40971e0 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bfa35d640 5 monclient: authenticate success, global_id 104278 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f3be8001770 con 0x7f3bf40971e0 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bfa35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: dump: 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 >> v1:172.21.15.181:6789/0 conn(0x7f3bf40971e0 legacy=0x7f3bf412f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bdc003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): picked mon.a con 0x7f3bdc003b20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bdc004780 0x7f3bdc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): picked mon.c con 0x7f3bdc004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bdc005310 0x7f3bdc00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): picked mon.b con 0x7f3bdc005310 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf409d590 con 0x7f3bdc003b20 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc00c2d0 con 0x7f3bdc005310 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc00c490 con 0x7f3bdc004780 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bfa35d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bdc003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): get_auth_request con 0x7f3bdc003b20 auth_method 0 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bdc004780 0x7f3bdc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bdc005310 0x7f3bdc00b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): get_auth_request con 0x7f3bdc004780 auth_method 0 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:23.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): get_auth_request con 0x7f3bdc005310 auth_method 0 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient(hunting): handle_auth_done global_id 104278 payload 293 2024-09-15T03:36:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 -- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bdc004780 msgr2=0x7f3bdc004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bdc004780 0x7f3bdc004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 -- 172.21.15.153:0/1459666681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bdc005310 msgr2=0x7f3bdc00b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bdc005310 0x7f3bdc00b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 monclient: found mon.a 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf88fa640 1 -- 172.21.15.153:0/1459666681 reap_dead start 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 1 -- 172.21.15.153:0/1459666681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc0034e0 con 0x7f3bdc003b20 2024-09-15T03:36:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.183+0000 7f3bf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf3fff640 1 -- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 msgr2=0x7f3bdc003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bdc003f30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3be4006fd0 tx=0x7f3be4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706bc526a50b6c30 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be402add0 con 0x7f3bdc003b20 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: dump: 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:23.204 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bfa35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be402e070 con 0x7f3bdc003b20 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 1 -- 172.21.15.153:0/1459666681 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4034660 con 0x7f3bdc003b20 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bfa35d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bfa35d640 4 set_mon_vals no callback set 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.186+0000 7f3bf2ffd640 10 monclient: dump: 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 10 monclient: shutdown 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 -- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 msgr2=0x7f3bdc003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bdc003f30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3be4006fd0 tx=0x7f3be4008040 comp rx=0 tx=0).stop 2024-09-15T03:36:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 -- 172.21.15.153:0/1459666681 shutdown_connections 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/1459666681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bdc003f30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 -- 172.21.15.153:0/1459666681 >> 172.21.15.153:0/1459666681 conn(0x7f3bf40672b0 msgr2=0x7f3bf40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 -- 172.21.15.153:0/1459666681 shutdown_connections 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 1 -- 172.21.15.153:0/1459666681 wait complete. 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 10 monclient: build_initial_monmap 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.187+0000 7f3bfa35d640 10 monclient: monmap: 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 1 Processor -- start 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 1 -- start start 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 10 monclient: init 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bdc003b20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bf41319c0 0x7f3bf4131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf41319c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.188+0000 7f3bfa35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 0x7f3bf40987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf4098320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf411f800 con 0x7f3bdc003b20 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf4141d20 con 0x7f3bf41319c0 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf409c370 con 0x7f3bf4098320 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bfa35d640 10 monclient(hunting): authenticate will time out at 1180.828036s 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38504/0 (socket says 172.21.15.153:38504) 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 -- 172.21.15.153:0/2086260177 learned_addr learned my addr 172.21.15.153:0/2086260177 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 0x7f3bf40987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 10 monclient(hunting): get_auth_request con 0x7f3bdc003b20 auth_method 0 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): get_auth_request con 0x7f3bf4098320 auth_method 0 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 msgr2=0x7f3bf4140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:36:23.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 msgr2=0x7f3bf4140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf37fe640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bf41319c0 0x7f3bf4131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient(hunting): handle_auth_done global_id 74813 payload 824 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bf41319c0 msgr2=0x7f3bf4131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bf41319c0 0x7f3bf4131d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 msgr2=0x7f3bf4140e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient: _finish_hunting 0 2024-09-15T03:36:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 monclient: found mon.noname-c 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf88fa640 1 -- 172.21.15.153:0/2086260177 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4132780 con 0x7f3bf4098320 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf37fe640 10 monclient: get_auth_request con 0x7f3bf41319c0 auth_method 0 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.189+0000 7f3bf37fe640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bf41319c0 0x7f3bf4131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bf88fa640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 msgr2=0x7f3bf40987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bf88fa640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 0x7f3bf40987a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4068ad0 tx=0x7f3bec001ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec012770 con 0x7f3bf4098320 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: dump: 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:23.213 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: _finish_auth 0 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 5 monclient: authenticate success, global_id 74813 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec0128f0 con 0x7f3bf4098320 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 10 monclient: _renew_subs 2024-09-15T03:36:23.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf4120b00 con 0x7f3bf4098320 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec012cb0 con 0x7f3bf4098320 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bdbfff640 10 monclient: dump: 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 10 monclient: _renew_subs 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf411fc10 con 0x7f3bf4098320 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.191+0000 7f3bf27fc640 4 set_mon_vals no callback set 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.192+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3bec013600 con 0x7f3bf4098320 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.192+0000 7f3bfa35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.192+0000 7f3bfa35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.192+0000 7f3bfa35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:23.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.192+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf404bf60 con 0x7f3bf4098320 2024-09-15T03:36:23.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.192+0000 7f3bdbfff640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3be0038740 0x7f3be003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.193+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(196..196 src has 106..196) v4 ==== 10392+0+0 (secure 0 0 0) 0x7f3bec04cc80 con 0x7f3bf4098320 2024-09-15T03:36:23.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.193+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3be0038740 0x7f3be003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.193+0000 7f3bf3fff640 10 monclient: get_auth_request con 0x7f3be0038740 auth_method 0 2024-09-15T03:36:23.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.195+0000 7f3bf3fff640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3be0038740 0x7f3be003abf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3be402ab90 tx=0x7f3be4035000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:23.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.197+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3bec01d460 con 0x7f3bf4098320 2024-09-15T03:36:23.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.197+0000 7f3bdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:23.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.197+0000 7f3bdbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.244+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v197) v1 ==== 168+0+0 (secure 0 0 0) 0x7ff10001a360 con 0x7ff11809f000 2024-09-15T03:36:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.244+0000 7ff10ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:23.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.244+0000 7ff10ffff640 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff11e554640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff11e554640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff11e554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff1181415e0 con 0x7ff11809f000 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff10ffff640 1 -- 172.21.15.181:0/2230971574 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v197) v1 ==== 147+0+0 (secure 0 0 0) 0x7ff10001a540 con 0x7ff11809f000 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff10ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.341+0000 7ff10ffff640 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-15T03:36:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h13' already exists 2024-09-15T03:36:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0f8038740 msgr2=0x7ff0f803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0f8038740 0x7ff0f803abf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff1180a1240 tx=0x7ff108032000 comp rx=0 tx=0).stop 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 10 monclient: shutdown 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 msgr2=0x7ff118130c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 0x7ff118130c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c5e0 tx=0x7ff100004960 comp rx=0 tx=0).stop 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 shutdown_connections 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff0f8038740 0x7ff0f803abf0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff118131bd0 0x7ff11811b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff11809f000 0x7ff118130c70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 --2- 172.21.15.181:0/2230971574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11809ebe0 0x7ff11809ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 >> 172.21.15.181:0/2230971574 conn(0x7ff1180713b0 msgr2=0x7ff1180a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 shutdown_connections 2024-09-15T03:36:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.343+0000 7ff11e554640 1 -- 172.21.15.181:0/2230971574 wait complete. 2024-09-15T03:36:23.353 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:23.353 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-15T03:36:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bfa35d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bfa35d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bfa35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3bf40638d0 con 0x7f3bf4098320 2024-09-15T03:36:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bdbfff640 1 -- 172.21.15.153:0/2086260177 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f3bec0112b0 con 0x7f3bf4098320 2024-09-15T03:36:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.409+0000 7f3bdbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:23.411 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3be0038740 msgr2=0x7f3be003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3be0038740 0x7f3be003abf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3be402ab90 tx=0x7f3be4035000 comp rx=0 tx=0).stop 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 10 monclient: shutdown 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 msgr2=0x7f3bf40987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 0x7f3bf40987a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4068ad0 tx=0x7f3bec001ce0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 shutdown_connections 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3be0038740 0x7f3be003abf0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bf4098320 0x7f3bf40987a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bf41319c0 0x7f3bf4131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 --2- 172.21.15.153:0/2086260177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bdc003b20 0x7f3bf4140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.411+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 >> 172.21.15.153:0/2086260177 conn(0x7f3bf40672b0 msgr2=0x7f3bf4132990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.412+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 shutdown_connections 2024-09-15T03:36:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:23.412+0000 7f3bfa35d640 1 -- 172.21.15.153:0/2086260177 wait complete. 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.426+0000 7fbe5c314640 10 monclient: get_monmap_and_config 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.426+0000 7fbe5c314640 10 monclient: build_initial_monmap 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.426+0000 7fbe5c314640 10 monclient: monmap: 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:23.427492+0000 2024-09-15T03:36:23.428 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:23.427492+0000 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:23.429 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 1 Processor -- start 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 1 -- start start 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient: init 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe54069bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 0x7fbe540981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5409cf60 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe540986f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5409cd00 con 0x7fbe54069bd0 2024-09-15T03:36:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5409e320 con 0x7fbe540986f0 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe54132640 con 0x7fbe5409cf60 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.427+0000 7fbe5c314640 10 monclient(hunting): authenticate will time out at 1187.908935s 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe5a8b1640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fbe540986f0 0x7fbe54138ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:56566/0 (socket says 172.21.15.181:56566) 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 learned_addr learned my addr 172.21.15.181:0/2175647675 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 1 --2- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 0x7fbe540981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 10 monclient(hunting): get_auth_request con 0x7fbe5409cf60 auth_method 0 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 1 -- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 msgr2=0x7fbe540981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:23.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 1 -- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 msgr2=0x7fbe540981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 1 --2- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 0x7fbe540981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.428+0000 7fbe598af640 1 --2- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 0x7fbe540981b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 >> v1:172.21.15.181:6789/0 conn(0x7fbe540986f0 legacy=0x7fbe54138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe5a0b0640 1 -- 172.21.15.181:0/2175647675 >> v1:172.21.15.153:6789/0 conn(0x7fbe54069bd0 legacy=0x7fbe54131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10340110 0 0) 0x7fbe5409e320 con 0x7fbe540986f0 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): my global_id is 104281 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe400034e0 con 0x7fbe540986f0 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4144293303 0 0) 0x7fbe400034e0 con 0x7fbe540986f0 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 msgr2=0x7fbe540981b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 --2- 172.21.15.181:0/2175647675 >> v2:172.21.15.153:3300/0 conn(0x7fbe5409cf60 0x7fbe540981b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 >> v1:172.21.15.153:6789/0 conn(0x7fbe54069bd0 legacy=0x7fbe54131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: _finish_hunting 0 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54098b50 con 0x7fbe540986f0 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: _finish_auth 0 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fbe4c001aa0 con 0x7fbe540986f0 2024-09-15T03:36:23.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe5c314640 5 monclient: authenticate success, global_id 104281 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe5c314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: dump: 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 >> v1:172.21.15.181:6789/0 conn(0x7fbe540986f0 legacy=0x7fbe54138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe40003b20 0x7fbe40003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): picked mon.a con 0x7fbe40003b20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe40004780 0x7fbe40004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): picked mon.c con 0x7fbe40004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 0x7fbe4000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): picked mon.b con 0x7fbe400053b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4000c350 con 0x7fbe40003b20 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4000c4c0 con 0x7fbe400053b0 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4000c680 con 0x7fbe40004780 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe590ae640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.430+0000 7fbe5c314640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 0x7fbe4000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): get_auth_request con 0x7fbe400053b0 auth_method 0 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:23.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a0b0640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe40003b20 0x7fbe40003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a0b0640 10 monclient(hunting): get_auth_request con 0x7fbe40003b20 auth_method 0 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a0b0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a0b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient(hunting): handle_auth_done global_id 104281 payload 293 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe598af640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe40004780 0x7fbe40004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe40004780 msgr2=0x7fbe40004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe40004780 0x7fbe40004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe40003b20 msgr2=0x7fbe40003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe40003b20 0x7fbe40003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient: _finish_hunting 0 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 monclient: found mon.b 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400034e0 con 0x7fbe400053b0 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 10 monclient: _finish_auth 0 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe598af640 10 monclient: get_auth_request con 0x7fbe40004780 auth_method 0 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe598af640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe40004780 0x7fbe40004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:23.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.431+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 reap_dead start 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.433+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 msgr2=0x7fbe4000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.433+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 0x7fbe4000baa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c001810 tx=0x7fbe4c00a200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb1c684d7f9d295 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c02b670 con 0x7fbe400053b0 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: dump: 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.449 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4c02b830 con 0x7fbe400053b0 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 1 -- 172.21.15.181:0/2175647675 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c02bbf0 con 0x7fbe400053b0 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 4 set_mon_vals no callback set 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe590ae640 10 monclient: dump: 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 10 monclient: shutdown 2024-09-15T03:36:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 -- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 msgr2=0x7fbe4000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 0x7fbe4000baa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c001810 tx=0x7fbe4c00a200 comp rx=0 tx=0).stop 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 -- 172.21.15.181:0/2175647675 shutdown_connections 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 --2- 172.21.15.181:0/2175647675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe400053b0 0x7fbe4000baa0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 -- 172.21.15.181:0/2175647675 >> 172.21.15.181:0/2175647675 conn(0x7fbe5409e9f0 msgr2=0x7fbe5409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 -- 172.21.15.181:0/2175647675 shutdown_connections 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.434+0000 7fbe5c314640 1 -- 172.21.15.181:0/2175647675 wait complete. 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient: build_initial_monmap 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient: monmap: 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 Processor -- start 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 -- start start 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient: init 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe541410b0 0x7fbe54141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): picked mon.noname-c con 0x7fbe541410b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 0x7fbe5412ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5412ea40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 0x7fbe541189d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): picked mon.noname-a con 0x7fbe54118560 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5411a5d0 con 0x7fbe54118560 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5409d8a0 con 0x7fbe5412ea40 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe54141a80 con 0x7fbe541410b0 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe5c314640 10 monclient(hunting): authenticate will time out at 1187.916916s 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.435+0000 7fbe598af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 0x7fbe5412ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe541410b0 0x7fbe54141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 0x7fbe5412ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36298/0 (socket says 172.21.15.181:36298) 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe541410b0 0x7fbe54141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38762/0 (socket says 172.21.15.181:38762) 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 -- 172.21.15.181:0/4118053308 learned_addr learned my addr 172.21.15.181:0/4118053308 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): get_auth_request con 0x7fbe5412ea40 auth_method 0 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): get_auth_request con 0x7fbe541410b0 auth_method 0 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 0x7fbe541189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 10 monclient(hunting): get_auth_request con 0x7fbe54118560 auth_method 0 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 msgr2=0x7fbe541189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 msgr2=0x7fbe541189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 0x7fbe541189d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe5a8b1640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 0x7fbe541189d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient(hunting): handle_auth_done global_id 104284 payload 824 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe541410b0 msgr2=0x7fbe54141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe541410b0 0x7fbe54141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 msgr2=0x7fbe541189d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 0x7fbe541189d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient: _finish_hunting 0 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 monclient: found mon.noname-b 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.436+0000 7fbe598af640 1 -- 172.21.15.181:0/4118053308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5411bb60 con 0x7fbe5412ea40 2024-09-15T03:36:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe598af640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 msgr2=0x7fbe5412ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe598af640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 0x7fbe5412ee70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400e870 tx=0x7fbe44008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4400c6d0 con 0x7fbe5412ea40 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: dump: 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 5 monclient: authenticate success, global_id 104284 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 10 monclient: _renew_subs 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5411be40 con 0x7fbe5412ea40 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4400c830 con 0x7fbe5412ea40 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4400cbf0 con 0x7fbe5412ea40 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe4a7fc640 10 monclient: dump: 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 10 monclient: _renew_subs 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe588ad640 4 set_mon_vals no callback set 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.438+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5411a0b0 con 0x7fbe5412ea40 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbe44013620 con 0x7fbe5412ea40 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe5c314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe5c314640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe5c314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5404bf60 con 0x7fbe5412ea40 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe4a7fc640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbe38038740 0x7fbe3803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(197..197 src has 106..197) v4 ==== 10605+0+0 (secure 0 0 0) 0x7fbe4404cef0 con 0x7fbe5412ea40 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe5a0b0640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbe38038740 0x7fbe3803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.439+0000 7fbe5a0b0640 10 monclient: get_auth_request con 0x7fbe38038740 auth_method 0 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.442+0000 7fbe5a0b0640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbe38038740 0x7fbe3803abf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007fd0 tx=0x7fbe5000c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.443+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbe44022300 con 0x7fbe5412ea40 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.443+0000 7fbe4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:23.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.443+0000 7fbe4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:23.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.601+0000 7fbe5c314640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:23.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.601+0000 7fbe5c314640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:23.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.601+0000 7fbe5c314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:23.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:23.601+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbe54064100 con 0x7fbe5412ea40 2024-09-15T03:36:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.253+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v198) v1 ==== 168+0+0 (secure 0 0 0) 0x7fbe44012df0 con 0x7fbe5412ea40 2024-09-15T03:36:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.253+0000 7fbe4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:24.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.253+0000 7fbe4a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-15T03:36:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe5c314640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe5c314640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe5c314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:24.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbe5412f4b0 con 0x7fbe5412ea40 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe4a7fc640 1 -- 172.21.15.181:0/4118053308 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v198) v1 ==== 147+0+0 (secure 0 0 0) 0x7fbe44019310 con 0x7fbe5412ea40 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe4a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.352+0000 7fbe4a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h14' already exists 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbe38038740 msgr2=0x7fbe3803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbe38038740 0x7fbe3803abf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007fd0 tx=0x7fbe5000c040 comp rx=0 tx=0).stop 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 10 monclient: shutdown 2024-09-15T03:36:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 msgr2=0x7fbe5412ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 0x7fbe5412ee70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400e870 tx=0x7fbe44008040 comp rx=0 tx=0).stop 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 shutdown_connections 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbe38038740 0x7fbe3803abf0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe54118560 0x7fbe541189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe5412ea40 0x7fbe5412ee70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 --2- 172.21.15.181:0/4118053308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe541410b0 0x7fbe54141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 >> 172.21.15.181:0/4118053308 conn(0x7fbe5409e9f0 msgr2=0x7fbe540689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 shutdown_connections 2024-09-15T03:36:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.354+0000 7fbe5c314640 1 -- 172.21.15.181:0/4118053308 wait complete. 2024-09-15T03:36:24.364 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-15T03:36:24.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.439+0000 7ffb0b080640 10 monclient: get_monmap_and_config 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.439+0000 7ffb0b080640 10 monclient: build_initial_monmap 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.439+0000 7ffb0b080640 10 monclient: monmap: 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:24.440569+0000 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:24.440569+0000 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:24.441 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:24.442 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:24.442 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:24.442 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:24.442 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:24.442 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.440+0000 7ffb0b080640 1 Processor -- start 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.440+0000 7ffb0b080640 1 -- start start 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.440+0000 7ffb0b080640 10 monclient: init 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb0409bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): picked mon.noname-b con 0x7ffb04069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:24.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb0409d170 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): picked mon.noname-a con 0x7ffb04134cf0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb04069e00 con 0x7ffb0409d170 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0409e650 con 0x7ffb04134cf0 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0409e370 con 0x7ffb04069a30 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0b080640 10 monclient(hunting): authenticate will time out at 1188.922223s 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb0409bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb0409bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38768/0 (socket says 172.21.15.181:38768) 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb03fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ffb0409d170 0x7ffb0412fe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:39352/0 (socket says 172.21.15.181:39352) 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 -- 172.21.15.181:0/3831177458 learned_addr learned my addr 172.21.15.181:0/3831177458 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:38798/0 (socket says 172.21.15.181:38798) 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): get_auth_request con 0x7ffb04069a30 auth_method 0 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 10 monclient(hunting): get_auth_request con 0x7ffb04134cf0 auth_method 0 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:24.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 1 -- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 msgr2=0x7ffb041370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 1 -- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 msgr2=0x7ffb041370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 1 --2- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb0961d640 1 --2- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_done global_id 74819 payload 274 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 -- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 msgr2=0x7ffb041370d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:24.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 --2- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 -- 172.21.15.181:0/3831177458 >> v1:172.21.15.153:6789/0 conn(0x7ffb0409d170 legacy=0x7ffb0412fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 monclient: found mon.noname-b 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.441+0000 7ffb08e1c640 1 -- 172.21.15.181:0/3831177458 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04130530 con 0x7ffb04069a30 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb08e1c640 1 -- 172.21.15.181:0/3831177458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 msgr2=0x7ffb0409bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb08e1c640 1 --2- 172.21.15.181:0/3831177458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb0409bc00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009910 tx=0x7ffaf40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e867de1154aca470 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 1 -- 172.21.15.181:0/3831177458 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf402aea0 con 0x7ffb04069a30 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: dump: 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:24.451 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 1 -- 172.21.15.181:0/3831177458 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4032440 con 0x7ffb04069a30 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb0b080640 5 monclient: authenticate success, global_id 74819 2024-09-15T03:36:24.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb0b080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb0b080640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb0b080640 4 set_mon_vals no callback set 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 1 -- 172.21.15.181:0/3831177458 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40327e0 con 0x7ffb04069a30 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.444+0000 7ffb037fe640 10 monclient: dump: 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:24.453 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 10 monclient: shutdown 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 -- 172.21.15.181:0/3831177458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 msgr2=0x7ffb0409bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 --2- 172.21.15.181:0/3831177458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb0409bc00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009910 tx=0x7ffaf40091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 -- 172.21.15.181:0/3831177458 shutdown_connections 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 --2- 172.21.15.181:0/3831177458 >> v2:172.21.15.153:3300/0 conn(0x7ffb04134cf0 0x7ffb041370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 --2- 172.21.15.181:0/3831177458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb0409bc00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 -- 172.21.15.181:0/3831177458 >> 172.21.15.181:0/3831177458 conn(0x7ffb0409e9f0 msgr2=0x7ffb0409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:24.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 -- 172.21.15.181:0/3831177458 shutdown_connections 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 1 -- 172.21.15.181:0/3831177458 wait complete. 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 10 monclient: build_initial_monmap 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.445+0000 7ffb0b080640 10 monclient: monmap: 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:24.455 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:24.456 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:24.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 Processor -- start 2024-09-15T03:36:24.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 -- start start 2024-09-15T03:36:24.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient: init 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb04133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): picked mon.noname-c con 0x7ffb04069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 0x7ffb0412ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): picked mon.noname-a con 0x7ffb0409d170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 0x7ffb04118560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): picked mon.noname-b con 0x7ffb04134cf0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0409b0b0 con 0x7ffb0409d170 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0406eea0 con 0x7ffb04134cf0 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04135590 con 0x7ffb04069a30 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.446+0000 7ffb0b080640 10 monclient(hunting): authenticate will time out at 1188.927941s 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 0x7ffb04118560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 0x7ffb04118560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36314/0 (socket says 172.21.15.181:36314) 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 -- 172.21.15.181:0/2730788340 learned_addr learned my addr 172.21.15.181:0/2730788340 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): get_auth_request con 0x7ffb04134cf0 auth_method 0 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:24.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb04133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 0x7ffb0412ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): get_auth_request con 0x7ffb04069a30 auth_method 0 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 10 monclient(hunting): get_auth_request con 0x7ffb0409d170 auth_method 0 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:24.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 msgr2=0x7ffb0412ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb08e1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 msgr2=0x7ffb0412ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 0x7ffb0412ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb03fff640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 0x7ffb0412ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient(hunting): handle_auth_done global_id 104287 payload 824 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 msgr2=0x7ffb04133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb04133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 msgr2=0x7ffb0412ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 0x7ffb0412ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient: _finish_hunting 0 2024-09-15T03:36:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 monclient: found mon.noname-b 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.447+0000 7ffb0961d640 1 -- 172.21.15.181:0/2730788340 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf4009450 con 0x7ffb04134cf0 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0961d640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 msgr2=0x7ffb04118560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0961d640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 0x7ffb04118560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffafc007f50 tx=0x7ffafc009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc014420 con 0x7ffb04134cf0 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: dump: 2024-09-15T03:36:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 5 monclient: authenticate success, global_id 104287 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc014580 con 0x7ffb04134cf0 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 10 monclient: _renew_subs 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb04119bb0 con 0x7ffb04134cf0 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc014940 con 0x7ffb04134cf0 2024-09-15T03:36:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb00ff9640 10 monclient: dump: 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 10 monclient: _renew_subs 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb04134760 con 0x7ffb04134cf0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.449+0000 7ffb02ffd640 4 set_mon_vals no callback set 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffafc01e460 con 0x7ffb04134cf0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb0b080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb0b080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb0b080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0411b5f0 con 0x7ffb04134cf0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb00ff9640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffaf0038740 0x7ffaf003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(198..198 src has 106..198) v4 ==== 10818+0+0 (secure 0 0 0) 0x7ffafc04d880 con 0x7ffb04134cf0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.450+0000 7ffb08e1c640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffaf0038740 0x7ffaf003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.451+0000 7ffb08e1c640 10 monclient: get_auth_request con 0x7ffaf0038740 auth_method 0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.453+0000 7ffb08e1c640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffaf0038740 0x7ffaf003abf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffb04071990 tx=0x7ffaf4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:24.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.455+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffb0411b5f0 con 0x7ffb04134cf0 2024-09-15T03:36:24.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.455+0000 7ffb00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:24.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.455+0000 7ffb00ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:24.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.614+0000 7ffb0b080640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:24.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.614+0000 7ffb0b080640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:24.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.614+0000 7ffb0b080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:24.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:24.614+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ffb0411b5f0 con 0x7ffb04134cf0 2024-09-15T03:36:25.257 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.257+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v199) v1 ==== 168+0+0 (secure 0 0 0) 0x7ffb0411b5f0 con 0x7ffb04134cf0 2024-09-15T03:36:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.257+0000 7ffb00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:25.258 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.257+0000 7ffb00ff9640 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.355+0000 7ffb0b080640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.355+0000 7ffb0b080640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.355+0000 7ffb0b080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.355+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ffb0411b5f0 con 0x7ffb04134cf0 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.356+0000 7ffb00ff9640 1 -- 172.21.15.181:0/2730788340 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v199) v1 ==== 147+0+0 (secure 0 0 0) 0x7ffb0411b5f0 con 0x7ffb04134cf0 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.356+0000 7ffb00ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.356+0000 7ffb00ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-15T03:36:25.357 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h15' already exists 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffaf0038740 msgr2=0x7ffaf003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffaf0038740 0x7ffaf003abf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffb04071990 tx=0x7ffaf4032000 comp rx=0 tx=0).stop 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 10 monclient: shutdown 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 msgr2=0x7ffb04118560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 0x7ffb04118560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffafc007f50 tx=0x7ffafc009830 comp rx=0 tx=0).stop 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 shutdown_connections 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffaf0038740 0x7ffaf003abf0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb04134cf0 0x7ffb04118560 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb0409d170 0x7ffb0412ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 --2- 172.21.15.181:0/2730788340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb04069a30 0x7ffb04133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.357+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 >> 172.21.15.181:0/2730788340 conn(0x7ffb0409e9f0 msgr2=0x7ffb040a10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.358+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 shutdown_connections 2024-09-15T03:36:25.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.358+0000 7ffb0b080640 1 -- 172.21.15.181:0/2730788340 wait complete. 2024-09-15T03:36:25.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:25.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.442+0000 7ff14f234640 10 monclient: get_monmap_and_config 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.442+0000 7ff14f234640 10 monclient: build_initial_monmap 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.442+0000 7ff14f234640 10 monclient: monmap: 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:25.443279+0000 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:25.443279+0000 2024-09-15T03:36:25.443 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:25.444 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 Processor -- start 2024-09-15T03:36:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 -- start start 2024-09-15T03:36:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient: init 2024-09-15T03:36:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff148067640 0x7ff14809add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): picked mon.noname-c con 0x7ff148067640 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff148098b10 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff148136000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): picked mon.noname-b con 0x7ff148099620 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14809df40 con 0x7ff148098b10 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14809a010 con 0x7ff148067640 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff148067a10 con 0x7ff148099620 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14f234640 10 monclient(hunting): authenticate will time out at 1189.924889s 2024-09-15T03:36:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.443+0000 7ff14cfd0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff148067640 0x7ff14809add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff148067640 0x7ff14809add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36324/0 (socket says 172.21.15.181:36324) 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 -- 172.21.15.181:0/372189945 learned_addr learned my addr 172.21.15.181:0/372189945 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): get_auth_request con 0x7ff148067640 auth_method 0 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14d7d1640 1 --2- 172.21.15.181:0/372189945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff148136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14d7d1640 10 monclient(hunting): get_auth_request con 0x7ff148099620 auth_method 0 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14d7d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14d7d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14d7d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_done global_id 104290 payload 274 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 -- 172.21.15.181:0/372189945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 msgr2=0x7ff148136000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 --2- 172.21.15.181:0/372189945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff148136000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 -- 172.21.15.181:0/372189945 >> v1:172.21.15.153:6789/0 conn(0x7ff148098b10 legacy=0x7ff148098f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient: _finish_hunting 0 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 monclient: found mon.noname-c 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.444+0000 7ff14cfd0640 1 -- 172.21.15.181:0/372189945 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14809d340 con 0x7ff148067640 2024-09-15T03:36:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14cfd0640 1 -- 172.21.15.181:0/372189945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 msgr2=0x7ff14809add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14cfd0640 1 --2- 172.21.15.181:0/372189945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14809add0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1480769f0 tx=0x7ff13c00b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c013ff8e4190de8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 1 -- 172.21.15.181:0/372189945 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c031720 con 0x7ff148067640 2024-09-15T03:36:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: dump: 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 1 -- 172.21.15.181:0/372189945 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c02f050 con 0x7ff148067640 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 1 -- 172.21.15.181:0/372189945 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c031ab0 con 0x7ff148067640 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 5 monclient: authenticate success, global_id 104290 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 4 set_mon_vals no callback set 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff1477fe640 10 monclient: dump: 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 10 monclient: shutdown 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 -- 172.21.15.181:0/372189945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 msgr2=0x7ff14809add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 --2- 172.21.15.181:0/372189945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14809add0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1480769f0 tx=0x7ff13c00b460 comp rx=0 tx=0).stop 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 -- 172.21.15.181:0/372189945 shutdown_connections 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 --2- 172.21.15.181:0/372189945 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff148136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 --2- 172.21.15.181:0/372189945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14809add0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 -- 172.21.15.181:0/372189945 >> 172.21.15.181:0/372189945 conn(0x7ff1480a1630 msgr2=0x7ff1480a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:25.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 -- 172.21.15.181:0/372189945 shutdown_connections 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.447+0000 7ff14f234640 1 -- 172.21.15.181:0/372189945 wait complete. 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.448+0000 7ff14f234640 10 monclient: build_initial_monmap 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.448+0000 7ff14f234640 10 monclient: monmap: 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.448+0000 7ff14f234640 1 Processor -- start 2024-09-15T03:36:25.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.448+0000 7ff14f234640 1 -- start start 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.448+0000 7ff14f234640 10 monclient: init 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): picked mon.noname-b con 0x7ff148067640 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 0x7ff14813cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): picked mon.noname-a con 0x7ff148098b10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff14813d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): picked mon.noname-c con 0x7ff148099620 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14809b870 con 0x7ff148098b10 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480a05e0 con 0x7ff148067640 2024-09-15T03:36:25.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14806dcb0 con 0x7ff148099620 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14f234640 10 monclient(hunting): authenticate will time out at 1189.930242s 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14806eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36326/0 (socket says 172.21.15.181:36326) 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 -- 172.21.15.181:0/3336913377 learned_addr learned my addr 172.21.15.181:0/3336913377 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): get_auth_request con 0x7ff148067640 auth_method 0 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:25.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 0x7ff14813cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14d7d1640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff14813d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 10 monclient(hunting): get_auth_request con 0x7ff148098b10 auth_method 0 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14d7d1640 10 monclient(hunting): get_auth_request con 0x7ff148099620 auth_method 0 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14d7d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14d7d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14d7d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient(hunting): handle_auth_done global_id 104293 payload 824 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 msgr2=0x7ff14813cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:36:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 msgr2=0x7ff14813cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 0x7ff14813cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 msgr2=0x7ff14813d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff14813d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff147fff640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 0x7ff14813cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 msgr2=0x7ff14813cbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 0x7ff14813cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient: _finish_hunting 0 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 monclient: found mon.noname-b 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14cfd0640 1 -- 172.21.15.181:0/3336913377 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c00b700 con 0x7ff148067640 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.449+0000 7ff14d7d1640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff14813d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.451+0000 7ff14cfd0640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 msgr2=0x7ff14806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:25.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.451+0000 7ff14cfd0640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14806eca0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00b6d0 tx=0x7ff13c039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.451+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c033ac0 con 0x7ff148067640 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.451+0000 7ff144ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: dump: 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:25.463 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 5 monclient: authenticate success, global_id 104293 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 10 monclient: _renew_subs 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14806df90 con 0x7ff148067640 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c042070 con 0x7ff148067640 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c033e70 con 0x7ff148067640 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 10 monclient: dump: 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:25.464 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff146ffd640 4 set_mon_vals no callback set 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 10 monclient: _renew_subs 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14806cdc0 con 0x7ff148067640 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.452+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff13c044030 con 0x7ff148067640 2024-09-15T03:36:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff144ff9640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff12c038740 0x7ff12c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(199..199 src has 106..199) v4 ==== 11031+0+0 (secure 0 0 0) 0x7ff13c070b50 con 0x7ff148067640 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff147fff640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff12c038740 0x7ff12c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff14f234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff14f234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff14f234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14809a280 con 0x7ff148067640 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.453+0000 7ff147fff640 10 monclient: get_auth_request con 0x7ff12c038740 auth_method 0 2024-09-15T03:36:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.455+0000 7ff147fff640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff12c038740 0x7ff12c03abf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff134007c20 tx=0x7ff134009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:25.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.457+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff13c032b80 con 0x7ff148067640 2024-09-15T03:36:25.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.457+0000 7ff144ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:25.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.457+0000 7ff144ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.615+0000 7ff14f234640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.615+0000 7ff14f234640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.615+0000 7ff14f234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:25.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:25.615+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff14804bf60 con 0x7ff148067640 2024-09-15T03:36:26.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.527+0000 7f0a67268640 10 monclient: get_monmap_and_config 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.527+0000 7f0a67268640 10 monclient: build_initial_monmap 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.527+0000 7f0a67268640 10 monclient: monmap: 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:26.527795+0000 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:26.527795+0000 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:26.528 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:26.529 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:26.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:26.529 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:26.529 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:26.529 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:26.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.527+0000 7f0a67268640 1 Processor -- start 2024-09-15T03:36:26.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.527+0000 7f0a67268640 1 -- start start 2024-09-15T03:36:26.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.527+0000 7f0a67268640 10 monclient: init 2024-09-15T03:36:26.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient(hunting): picked mon.noname-a con 0x7f0a601318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a6009ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a60136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6009f530 con 0x7f0a6009ea50 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6009fbf0 con 0x7f0a60136c20 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:26.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a60131d50 con 0x7f0a601318f0 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a67268640 10 monclient(hunting): authenticate will time out at 1184.167499s 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a64803640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0a6009ea50 0x7f0a6009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53228/0 (socket says 172.21.15.153:53228) 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 learned_addr learned my addr 172.21.15.153:0/1637141214 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:26.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.528+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:26.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.530+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> v1:172.21.15.153:6789/0 conn(0x7f0a6009ea50 legacy=0x7f0a6009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:26.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.531+0000 7f0a65805640 1 -- 172.21.15.153:0/1637141214 >> v1:172.21.15.181:6789/0 conn(0x7f0a60136c20 legacy=0x7f0a60139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:26.729+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.130+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:27.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:27.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:27.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:27.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:27.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:27.931+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:36:28.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:28.449+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:28.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:28.449+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:28.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:28.449+0000 7ff1337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:29.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:29.450+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:29.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:29.450+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:29.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): tick 2024-09-15T03:36:29.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 msgr2=0x7f0a60131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a60131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 >> v1:172.21.15.181:6789/0 conn(0x7f0a60136c20 legacy=0x7f0a60139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 >> v1:172.21.15.153:6789/0 conn(0x7f0a6009ea50 legacy=0x7f0a6009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient: _add_conns ranks=[4,2] 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0a440037a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0a440062d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a44008e60 con 0x7f0a440037a0 2024-09-15T03:36:29.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44008fd0 con 0x7f0a440062d0 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a56ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a440062d0 auth_method 0 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.528+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a64803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a440037a0 auth_method 0 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:29.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:29.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:29.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:29.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:29.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:29.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:29.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.529+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a440062d0 auth_method 0 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a440037a0 auth_method 0 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:29.730+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a440062d0 auth_method 0 2024-09-15T03:36:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a440037a0 auth_method 0 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.131+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:30.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:30.450+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:30.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:30.450+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a440062d0 auth_method 0 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:30.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a440037a0 auth_method 0 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:30.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:30.932+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:36:31.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:31.450+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:31.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:31.450+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:32.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:32.449+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:32.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:32.449+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:32.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): tick 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 msgr2=0x7f0a44008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a440062d0 0x7f0a44008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 msgr2=0x7f0a44005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a440037a0 0x7f0a44005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 1 -- 172.21.15.153:0/1637141214 reap_dead start 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a4400b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0a601318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 0x7f0a44006750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0a44003be0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a440070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0a44006c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a44008e60 con 0x7f0a601318f0 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a44008fd0 con 0x7f0a44003be0 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 1 -- 172.21.15.153:0/1637141214 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440076d0 con 0x7f0a44006c90 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a56ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.153:3300/0 conn(0x7f0a601318f0 0x7f0a4400b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a440070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:32.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:32.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 10 monclient(hunting): get_auth_request con 0x7f0a44006c90 auth_method 0 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a440070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a440070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a440070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65805640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a440070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:32.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 0x7f0a44006750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a44003be0 auth_method 0 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient(hunting): handle_auth_done global_id 114099 payload 274 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a440070e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a440070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 msgr2=0x7f0a44006750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 msgr2=0x7f0a44006750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 1 -- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 msgr2=0x7f0a44006750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 0x7f0a44006750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a64803640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 0x7f0a44006750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:32.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 0x7f0a44006750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient: _finish_hunting 0 2024-09-15T03:36:32.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 monclient: found mon.noname-a 2024-09-15T03:36:32.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:32.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:32.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.529+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6009fe30 con 0x7f0a601318f0 2024-09-15T03:36:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a65004640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a4400b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a65004640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a601318f0 0x7f0a4400b440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0a60068a40 tx=0x7f0a5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4196a5768c38c1a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a57fff640 1 -- 172.21.15.153:0/1637141214 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c02e850 con 0x7f0a601318f0 2024-09-15T03:36:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a57fff640 10 monclient: dump: 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.532+0000 7f0a57fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 5 monclient: authenticate success, global_id 114099 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 1 -- 172.21.15.153:0/1637141214 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c02e9d0 con 0x7f0a601318f0 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 1 -- 172.21.15.153:0/1637141214 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c02ed90 con 0x7f0a601318f0 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 4 set_mon_vals no callback set 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a57fff640 10 monclient: dump: 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 10 monclient: shutdown 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 -- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a4400b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a601318f0 0x7f0a4400b440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0a60068a40 tx=0x7f0a5c00b040 comp rx=0 tx=0).stop 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 -- 172.21.15.153:0/1637141214 shutdown_connections 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a440070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 --2- 172.21.15.153:0/1637141214 >> v2:172.21.15.181:3300/0 conn(0x7f0a44003be0 0x7f0a44006750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 --2- 172.21.15.153:0/1637141214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a601318f0 0x7f0a4400b440 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.533+0000 7f0a67268640 1 -- 172.21.15.153:0/1637141214 >> 172.21.15.153:0/1637141214 conn(0x7f0a60067110 msgr2=0x7f0a60067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.534+0000 7f0a67268640 1 -- 172.21.15.153:0/1637141214 shutdown_connections 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.534+0000 7f0a67268640 1 -- 172.21.15.153:0/1637141214 wait complete. 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.534+0000 7f0a67268640 10 monclient: build_initial_monmap 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.534+0000 7f0a67268640 10 monclient: monmap: 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:32.546 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 Processor -- start 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 -- start start 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient: init 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): picked mon.noname-a con 0x7f0a44003be0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): picked mon.noname-c con 0x7f0a44006c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): picked mon.noname-b con 0x7f0a601318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6009bf10 con 0x7f0a44003be0 2024-09-15T03:36:32.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600a4690 con 0x7f0a601318f0 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6011c580 con 0x7f0a44006c90 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.535+0000 7f0a67268640 10 monclient(hunting): authenticate will time out at 1190.174970s 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51108/0 (socket says 172.21.15.153:51108) 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50744/0 (socket says 172.21.15.153:50744) 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 -- 172.21.15.153:0/1254253132 learned_addr learned my addr 172.21.15.153:0/1254253132 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a44006c90 auth_method 0 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a44003be0 auth_method 0 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a60099690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a60099690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 msgr2=0x7f0a60130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 msgr2=0x7f0a60130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65004640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a60099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a60099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.536+0000 7f0a65805640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 10 monclient(hunting): get_auth_request con 0x7f0a44006c90 auth_method 0 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): get_auth_request con 0x7f0a44003be0 auth_method 0 2024-09-15T03:36:32.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a60099690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a60099690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 10 monclient(hunting): get_auth_request con 0x7f0a601318f0 auth_method 0 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:32.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a60099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a60099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65805640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient(hunting): handle_auth_done global_id 114105 payload 824 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 msgr2=0x7f0a60099690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 msgr2=0x7f0a60099bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient: _finish_hunting 0 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 monclient: found mon.noname-a 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:32.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:32.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.737+0000 7f0a65004640 1 -- 172.21.15.153:0/1254253132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c009c50 con 0x7f0a44003be0 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.739+0000 7f0a65004640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 msgr2=0x7f0a60130780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.739+0000 7f0a65004640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c035000 tx=0x7f0a5c03b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c049070 con 0x7f0a44003be0 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: dump: 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.745 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c009990 con 0x7f0a44003be0 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 5 monclient: authenticate success, global_id 114105 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 10 monclient: _renew_subs 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6011c890 con 0x7f0a44003be0 2024-09-15T03:36:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0456a0 con 0x7f0a44003be0 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a557fa640 10 monclient: dump: 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:32.747 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a577fe640 4 set_mon_vals no callback set 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 10 monclient: _renew_subs 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.740+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6011cc20 con 0x7f0a44003be0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0a5c003040 con 0x7f0a44003be0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a557fa640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a4403a3a0 0x7f0a4403c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(199..199 src has 106..199) v4 ==== 11031+0+0 (secure 0 0 0) 0x7f0a5c035030 con 0x7f0a44003be0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a67268640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a67268640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a67268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6011ca20 con 0x7f0a44003be0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a4403a3a0 0x7f0a4403c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:32.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.741+0000 7f0a64803640 10 monclient: get_auth_request con 0x7f0a4403a3a0 auth_method 0 2024-09-15T03:36:32.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.744+0000 7f0a64803640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a4403a3a0 0x7f0a4403c850 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0a50007fd0 tx=0x7f0a50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:32.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.746+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0a5c02f5c0 con 0x7f0a44003be0 2024-09-15T03:36:32.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.746+0000 7f0a557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:32.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.746+0000 7f0a557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.965+0000 7f0a67268640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.965+0000 7f0a67268640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.965+0000 7f0a67268640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.965+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0a6004bf60 con 0x7f0a44003be0 2024-09-15T03:36:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.965+0000 7f0a557fa640 1 -- 172.21.15.153:0/1254253132 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f0a5c0443d0 con 0x7f0a44003be0 2024-09-15T03:36:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.966+0000 7f0a557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.966+0000 7f0a557fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:32.967 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a4403a3a0 msgr2=0x7f0a4403c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a4403a3a0 0x7f0a4403c850 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0a50007fd0 tx=0x7f0a50009040 comp rx=0 tx=0).stop 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 10 monclient: shutdown 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 msgr2=0x7f0a60130780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c035000 tx=0x7f0a5c03b000 comp rx=0 tx=0).stop 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 shutdown_connections 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a4403a3a0 0x7f0a4403c850 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a601318f0 0x7f0a60099bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a44006c90 0x7f0a60099690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 --2- 172.21.15.153:0/1254253132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a44003be0 0x7f0a60130780 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 >> 172.21.15.153:0/1254253132 conn(0x7f0a60067110 msgr2=0x7f0a6009ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 shutdown_connections 2024-09-15T03:36:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:32.967+0000 7f0a67268640 1 -- 172.21.15.153:0/1254253132 wait complete. 2024-09-15T03:36:33.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:33.449+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:33.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:33.449+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:34.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:34.449+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:34.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:34.449+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:35.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:35.449+0000 7ff1337fe640 10 monclient: tick 2024-09-15T03:36:35.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:35.449+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:35.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.084+0000 7f13506c0640 10 monclient: get_monmap_and_config 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.084+0000 7f13506c0640 10 monclient: build_initial_monmap 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.084+0000 7f13506c0640 10 monclient: monmap: 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:36.084971+0000 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:36.084971+0000 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:36.085 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:36.086 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:36.086 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:36.086 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:36.086 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.084+0000 7f13506c0640 1 Processor -- start 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.084+0000 7f13506c0640 1 -- start start 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.084+0000 7f13506c0640 10 monclient: init 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1348131ab0 0x7f1348131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1348131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:36.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f134809e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1348136bb0 0x7f1348138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1348136bb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134809faf0 con 0x7f134809e910 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134809fd30 con 0x7f1348136bb0 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13480a1110 con 0x7f1348131ab0 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f13506c0640 10 monclient(hunting): authenticate will time out at 1193.724550s 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1348136bb0 0x7f1348138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1348136bb0 0x7f1348138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42776/0 (socket says 172.21.15.153:42776) 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 1 -- 172.21.15.153:0/1512493097 learned_addr learned my addr 172.21.15.153:0/1512493097 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): get_auth_request con 0x7f1348136bb0 auth_method 0 2024-09-15T03:36:36.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134e45c640 1 --2- 172.21.15.153:0/1512493097 >> v2:172.21.15.181:3300/0 conn(0x7f1348131ab0 0x7f1348131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient(hunting): handle_auth_done global_id 114108 payload 274 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 1 -- 172.21.15.153:0/1512493097 >> v2:172.21.15.181:3300/0 conn(0x7f1348131ab0 msgr2=0x7f1348131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 1 --2- 172.21.15.153:0/1512493097 >> v2:172.21.15.181:3300/0 conn(0x7f1348131ab0 0x7f1348131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 1 -- 172.21.15.153:0/1512493097 >> v1:172.21.15.181:6789/0 conn(0x7f134809e910 legacy=0x7f134809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.085+0000 7f134ec5d640 10 monclient: _finish_hunting 0 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.086+0000 7f134ec5d640 1 monclient: found mon.noname-a 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.086+0000 7f134ec5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:36.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.086+0000 7f134ec5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:36.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.086+0000 7f134ec5d640 1 -- 172.21.15.153:0/1512493097 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13480a13f0 con 0x7f1348136bb0 2024-09-15T03:36:36.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.086+0000 7f134e45c640 10 monclient: get_auth_request con 0x7f1348131ab0 auth_method 0 2024-09-15T03:36:36.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.086+0000 7f134e45c640 1 --2- 172.21.15.153:0/1512493097 >> v2:172.21.15.181:3300/0 conn(0x7f1348131ab0 0x7f1348131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:36.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.088+0000 7f134ec5d640 1 -- 172.21.15.153:0/1512493097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 msgr2=0x7f1348138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:36.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.088+0000 7f134ec5d640 1 --2- 172.21.15.153:0/1512493097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f1348138f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1340009f20 tx=0x7f1340009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc3564e9ba3db91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:36.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 1 -- 172.21.15.153:0/1512493097 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134002aea0 con 0x7f1348136bb0 2024-09-15T03:36:36.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:36.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: dump: 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: _finish_auth 0 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 5 monclient: authenticate success, global_id 114108 2024-09-15T03:36:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 1 -- 172.21.15.153:0/1512493097 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134002d050 con 0x7f1348136bb0 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 4 set_mon_vals no callback set 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 1 -- 172.21.15.153:0/1512493097 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1340007840 con 0x7f1348136bb0 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f134d45a640 10 monclient: dump: 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 10 monclient: shutdown 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:36.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 -- 172.21.15.153:0/1512493097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 msgr2=0x7f1348138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 --2- 172.21.15.153:0/1512493097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f1348138f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1340009f20 tx=0x7f1340009840 comp rx=0 tx=0).stop 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 -- 172.21.15.153:0/1512493097 shutdown_connections 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 --2- 172.21.15.153:0/1512493097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f1348138f90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 --2- 172.21.15.153:0/1512493097 >> v2:172.21.15.181:3300/0 conn(0x7f1348131ab0 0x7f1348131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 -- 172.21.15.153:0/1512493097 >> 172.21.15.153:0/1512493097 conn(0x7f1348067d30 msgr2=0x7f13480a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 -- 172.21.15.153:0/1512493097 shutdown_connections 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.089+0000 7f13506c0640 1 -- 172.21.15.153:0/1512493097 wait complete. 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient: build_initial_monmap 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient: monmap: 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:36.106 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 Processor -- start 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 -- start start 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient: init 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:36.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 0x7f1348135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): picked mon.noname-c con 0x7f134809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1348131ab0 0x7f134812f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1348131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f134812f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1348136bb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13481337c0 con 0x7f1348136bb0 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134809db30 con 0x7f1348131ab0 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1348133460 con 0x7f134809e910 2024-09-15T03:36:36.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.090+0000 7f13506c0640 10 monclient(hunting): authenticate will time out at 1193.729988s 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 0x7f1348135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 0x7f1348135fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37890/0 (socket says 172.21.15.153:37890) 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f134812f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 -- 172.21.15.153:0/1261026864 learned_addr learned my addr 172.21.15.153:0/1261026864 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): get_auth_request con 0x7f134809e910 auth_method 0 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): get_auth_request con 0x7f1348136bb0 auth_method 0 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134dc5b640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1348131ab0 0x7f134812f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134dc5b640 10 monclient(hunting): get_auth_request con 0x7f1348131ab0 auth_method 0 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134dc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134dc5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134dc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient(hunting): handle_auth_done global_id 74831 payload 824 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 -- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1348131ab0 msgr2=0x7f134812f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1348131ab0 0x7f134812f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 -- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 msgr2=0x7f134812f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f134812f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 monclient: found mon.noname-c 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134e45c640 1 -- 172.21.15.153:0/1261026864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340009ae0 con 0x7f134809e910 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.091+0000 7f134ec5d640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f134812f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f134e45c640 1 -- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 msgr2=0x7f1348135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f134e45c640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 0x7f1348135fe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f133c00c880 tx=0x7f133c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00ce10 con 0x7f134809e910 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:36.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: dump: 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:36.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 5 monclient: authenticate success, global_id 74831 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c007aa0 con 0x7f134809e910 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c007e60 con 0x7f134809e910 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 10 monclient: _renew_subs 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348136520 con 0x7f134809e910 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f133a7fc640 10 monclient: dump: 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.113 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f134cc59640 4 set_mon_vals no callback set 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 10 monclient: _renew_subs 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.093+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348130e50 con 0x7f134809e910 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.094+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f133c011600 con 0x7f134809e910 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.094+0000 7f13506c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.094+0000 7f13506c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:36.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.094+0000 7f13506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.094+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348133900 con 0x7f134809e910 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.094+0000 7f133a7fc640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f132c038740 0x7f132c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.095+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(199..199 src has 106..199) v4 ==== 11031+0+0 (secure 0 0 0) 0x7f133c04d4d0 con 0x7f134809e910 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.095+0000 7f134dc5b640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f132c038740 0x7f132c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.095+0000 7f134dc5b640 10 monclient: get_auth_request con 0x7f132c038740 auth_method 0 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.097+0000 7f134dc5b640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f132c038740 0x7f132c03abf0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1348137e10 tx=0x7f1344009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.099+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1348133900 con 0x7f134809e910 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.099+0000 7f133a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:36.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.099+0000 7f133a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:36.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.222+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v200) v1 ==== 168+0+0 (secure 0 0 0) 0x7ff13c0350c0 con 0x7ff148067640 2024-09-15T03:36:36.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.222+0000 7ff144ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:36.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.222+0000 7ff144ff9640 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-15T03:36:36.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f13506c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:36.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f13506c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:36.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f13506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:36.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1348133900 con 0x7f134809e910 2024-09-15T03:36:36.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f133a7fc640 1 -- 172.21.15.153:0/1261026864 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f1348133900 con 0x7f134809e910 2024-09-15T03:36:36.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f133a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:36.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.315+0000 7f133a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:36.317 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":88,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f132c038740 msgr2=0x7f132c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f132c038740 0x7f132c03abf0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1348137e10 tx=0x7f1344009040 comp rx=0 tx=0).stop 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 10 monclient: shutdown 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 msgr2=0x7f1348135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 0x7f1348135fe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f133c00c880 tx=0x7f133c008040 comp rx=0 tx=0).stop 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 shutdown_connections 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f132c038740 0x7f132c03abf0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1348136bb0 0x7f134812f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1348131ab0 0x7f134812f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 --2- 172.21.15.153:0/1261026864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f134809e910 0x7f1348135fe0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 >> 172.21.15.153:0/1261026864 conn(0x7f1348067d30 msgr2=0x7f134809c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:36.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 shutdown_connections 2024-09-15T03:36:36.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:36.317+0000 7f13506c0640 1 -- 172.21.15.153:0/1261026864 wait complete. 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff14f234640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff14f234640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff14f234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff14804bf60 con 0x7ff148067640 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff144ff9640 1 -- 172.21.15.181:0/3336913377 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v200) v1 ==== 147+0+0 (secure 0 0 0) 0x7ff13c0323a0 con 0x7ff148067640 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff144ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.327+0000 7ff144ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-15T03:36:36.330 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h16' already exists 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff12c038740 msgr2=0x7ff12c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff12c038740 0x7ff12c03abf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff134007c20 tx=0x7ff134009040 comp rx=0 tx=0).stop 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 10 monclient: shutdown 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 msgr2=0x7ff14806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14806eca0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00b6d0 tx=0x7ff13c039000 comp rx=0 tx=0).stop 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 shutdown_connections 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff12c038740 0x7ff12c03abf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff148099620 0x7ff14813d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff148098b10 0x7ff14813cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 --2- 172.21.15.181:0/3336913377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff148067640 0x7ff14806eca0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 >> 172.21.15.181:0/3336913377 conn(0x7ff1480a1630 msgr2=0x7ff14809f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 shutdown_connections 2024-09-15T03:36:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.329+0000 7ff14f234640 1 -- 172.21.15.181:0/3336913377 wait complete. 2024-09-15T03:36:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.414+0000 7fb71d4ef640 10 monclient: get_monmap_and_config 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.414+0000 7fb71d4ef640 10 monclient: build_initial_monmap 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.414+0000 7fb71d4ef640 10 monclient: monmap: 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:36.416738+0000 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:36.416738+0000 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:36.418 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:36.418 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:36.418 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.415+0000 7fb71d4ef640 1 Processor -- start 2024-09-15T03:36:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.415+0000 7fb71d4ef640 1 -- start start 2024-09-15T03:36:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.415+0000 7fb71d4ef640 10 monclient: init 2024-09-15T03:36:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:36:36.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb71809cae0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:36.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb71806a7c0 0x7fb718099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): picked mon.noname-c con 0x7fb71806a7c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:36.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb71809ceb0 0x7fb71809dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): picked mon.noname-a con 0x7fb71809ceb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:36.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71809f680 con 0x7fb71809cae0 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb718099fa0 con 0x7fb71809ceb0 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71809ed00 con 0x7fb71806a7c0 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb71d4ef640 10 monclient(hunting): authenticate will time out at 1200.898109s 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb71806a7c0 0x7fb718099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb71806a7c0 0x7fb718099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:53260/0 (socket says 172.21.15.181:53260) 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 -- 172.21.15.181:0/1915033482 learned_addr learned my addr 172.21.15.181:0/1915033482 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): get_auth_request con 0x7fb71806a7c0 auth_method 0 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 1 --2- 172.21.15.181:0/1915033482 >> v2:172.21.15.153:3300/0 conn(0x7fb71809ceb0 0x7fb71809dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): get_auth_request con 0x7fb71809ceb0 auth_method 0 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_done global_id 104299 payload 274 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 -- 172.21.15.181:0/1915033482 >> v2:172.21.15.153:3300/0 conn(0x7fb71809ceb0 msgr2=0x7fb71809dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 --2- 172.21.15.181:0/1915033482 >> v2:172.21.15.153:3300/0 conn(0x7fb71809ceb0 0x7fb71809dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 -- 172.21.15.181:0/1915033482 >> v1:172.21.15.153:6789/0 conn(0x7fb71809cae0 legacy=0x7fb71809d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 monclient: found mon.noname-c 2024-09-15T03:36:36.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:36.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.416+0000 7fb70e7fc640 1 -- 172.21.15.181:0/1915033482 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71809f420 con 0x7fb71806a7c0 2024-09-15T03:36:36.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb70e7fc640 1 -- 172.21.15.181:0/1915033482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71806a7c0 msgr2=0x7fb718099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:36.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb70e7fc640 1 --2- 172.21.15.181:0/1915033482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71806a7c0 0x7fb718099a60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb718071880 tx=0x7fb7000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d487693eba8453 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:36.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 1 -- 172.21.15.181:0/1915033482 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700032440 con 0x7fb71806a7c0 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: dump: 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 1 -- 172.21.15.181:0/1915033482 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7000325a0 con 0x7fb71806a7c0 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb71d4ef640 5 monclient: authenticate success, global_id 104299 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb71d4ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb71d4ef640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 1 -- 172.21.15.181:0/1915033482 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700032940 con 0x7fb71806a7c0 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb71d4ef640 4 set_mon_vals no callback set 2024-09-15T03:36:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.419+0000 7fb7167fc640 10 monclient: dump: 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.435 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 10 monclient: shutdown 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 -- 172.21.15.181:0/1915033482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71806a7c0 msgr2=0x7fb718099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/1915033482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71806a7c0 0x7fb718099a60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb718071880 tx=0x7fb7000091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 -- 172.21.15.181:0/1915033482 shutdown_connections 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/1915033482 >> v2:172.21.15.153:3300/0 conn(0x7fb71809ceb0 0x7fb71809dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/1915033482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71806a7c0 0x7fb718099a60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 -- 172.21.15.181:0/1915033482 >> 172.21.15.181:0/1915033482 conn(0x7fb7180713a0 msgr2=0x7fb7180a2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 -- 172.21.15.181:0/1915033482 shutdown_connections 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 1 -- 172.21.15.181:0/1915033482 wait complete. 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 10 monclient: build_initial_monmap 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.420+0000 7fb71d4ef640 10 monclient: monmap: 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 Processor -- start 2024-09-15T03:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 -- start start 2024-09-15T03:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient: init 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71806a7c0 0x7fb718133b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): picked mon.noname-a con 0x7fb71806a7c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 0x7fb71811d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): picked mon.noname-b con 0x7fb71809cae0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71809ceb0 0x7fb71811de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): picked mon.noname-c con 0x7fb71809ceb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71806eb10 con 0x7fb71806a7c0 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71811f1d0 con 0x7fb71809cae0 2024-09-15T03:36:36.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb718136380 con 0x7fb71809ceb0 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb71d4ef640 10 monclient(hunting): authenticate will time out at 1200.903698s 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 0x7fb71811d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 0x7fb71811d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:53266/0 (socket says 172.21.15.181:53266) 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 1 -- 172.21.15.181:0/3441153265 learned_addr learned my addr 172.21.15.181:0/3441153265 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): get_auth_request con 0x7fb71809cae0 auth_method 0 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71809ceb0 0x7fb71811de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb716ffd640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71806a7c0 0x7fb718133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 10 monclient(hunting): get_auth_request con 0x7fb71809ceb0 auth_method 0 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb716ffd640 10 monclient(hunting): get_auth_request con 0x7fb71806a7c0 auth_method 0 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb716ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb716ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.421+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb716ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb716ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb716ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 10 monclient(hunting): handle_auth_done global_id 104302 payload 824 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 1 -- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71809ceb0 msgr2=0x7fb71811de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71809ceb0 0x7fb71811de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 1 -- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71806a7c0 msgr2=0x7fb718133b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71806a7c0 0x7fb718133b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 1 monclient: found mon.noname-b 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.422+0000 7fb70e7fc640 1 -- 172.21.15.181:0/3441153265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700009450 con 0x7fb71809cae0 2024-09-15T03:36:36.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70e7fc640 1 -- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 msgr2=0x7fb71811d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:36.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70e7fc640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 0x7fb71811d910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb700009210 tx=0x7fb700032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:36.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7000319c0 con 0x7fb71809cae0 2024-09-15T03:36:36.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:36.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: dump: 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:36.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb700031b40 con 0x7fb71809cae0 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 5 monclient: authenticate success, global_id 104302 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70003d440 con 0x7fb71809cae0 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 10 monclient: _renew_subs 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb718120c80 con 0x7fb71809cae0 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb70f7fe640 10 monclient: dump: 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.447 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb715ffb640 4 set_mon_vals no callback set 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 10 monclient: _renew_subs 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.424+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb718141c70 con 0x7fb71809cae0 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb700039040 con 0x7fb71809cae0 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb70f7fe640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6f8038740 0x7fb6f803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(200..200 src has 106..200) v4 ==== 11500+0+0 (secure 0 0 0) 0x7fb70006cfc0 con 0x7fb71809cae0 2024-09-15T03:36:36.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb716ffd640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6f8038740 0x7fb6f803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb71d4ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb71d4ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb71d4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71804bf60 con 0x7fb71809cae0 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.425+0000 7fb716ffd640 10 monclient: get_auth_request con 0x7fb6f8038740 auth_method 0 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.428+0000 7fb716ffd640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6f8038740 0x7fb6f803abf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb708007fd0 tx=0x7fb708007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.429+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb700037020 con 0x7fb71809cae0 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.429+0000 7fb70f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.429+0000 7fb70f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:36.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.585+0000 7fb71d4ef640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:36.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.586+0000 7fb71d4ef640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:36.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.586+0000 7fb71d4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:36.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:36.586+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb7181210b0 con 0x7fb71809cae0 2024-09-15T03:36:37.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.230+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v201) v1 ==== 168+0+0 (secure 0 0 0) 0x7fb700044b00 con 0x7fb71809cae0 2024-09-15T03:36:37.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.230+0000 7fb70f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:37.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.230+0000 7fb70f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-15T03:36:37.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb71d4ef640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:37.328 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb71d4ef640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:37.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb71d4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb7181210b0 con 0x7fb71809cae0 2024-09-15T03:36:37.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb70f7fe640 1 -- 172.21.15.181:0/3441153265 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v201) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb700030e60 con 0x7fb71809cae0 2024-09-15T03:36:37.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb70f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:37.329 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.326+0000 7fb70f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-15T03:36:37.329 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h17' already exists 2024-09-15T03:36:37.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6f8038740 msgr2=0x7fb6f803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:37.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6f8038740 0x7fb6f803abf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb708007fd0 tx=0x7fb708007e10 comp rx=0 tx=0).stop 2024-09-15T03:36:37.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 10 monclient: shutdown 2024-09-15T03:36:37.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:37.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 msgr2=0x7fb71811d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:37.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 0x7fb71811d910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb700009210 tx=0x7fb700032000 comp rx=0 tx=0).stop 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 shutdown_connections 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb6f8038740 0x7fb6f803abf0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71809ceb0 0x7fb71811de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb71809cae0 0x7fb71811d910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 --2- 172.21.15.181:0/3441153265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71806a7c0 0x7fb718133b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 >> 172.21.15.181:0/3441153265 conn(0x7fb7180713a0 msgr2=0x7fb7180a1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 shutdown_connections 2024-09-15T03:36:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.328+0000 7fb71d4ef640 1 -- 172.21.15.181:0/3441153265 wait complete. 2024-09-15T03:36:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:37.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-15T03:36:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.413+0000 7fe88c0d4640 10 monclient: get_monmap_and_config 2024-09-15T03:36:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.413+0000 7fe88c0d4640 10 monclient: build_initial_monmap 2024-09-15T03:36:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.413+0000 7fe88c0d4640 10 monclient: monmap: 2024-09-15T03:36:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:37.414950+0000 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:37.414950+0000 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.413+0000 7fe88c0d4640 1 Processor -- start 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.413+0000 7fe88c0d4640 1 -- start start 2024-09-15T03:36:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.413+0000 7fe88c0d4640 10 monclient: init 2024-09-15T03:36:37.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:37.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:36:37.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe884131bd0 0x7fe884131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:37.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): picked mon.noname-c con 0x7fe884131bd0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409ebe0 0x7fe884068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): picked mon.noname-b con 0x7fe88409ebe0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe88409f000 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8840a06d0 con 0x7fe88409f000 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe884071780 con 0x7fe884131bd0 2024-09-15T03:36:37.419 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840a0010 con 0x7fe88409ebe0 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88c0d4640 10 monclient(hunting): authenticate will time out at 1201.896427s 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe884131bd0 0x7fe884131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe884131bd0 0x7fe884131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:53282/0 (socket says 172.21.15.181:53282) 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 -- 172.21.15.181:0/1959239413 learned_addr learned my addr 172.21.15.181:0/1959239413 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): get_auth_request con 0x7fe884131bd0 auth_method 0 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:37.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88966f640 1 --2- 172.21.15.181:0/1959239413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409ebe0 0x7fe884068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88966f640 10 monclient(hunting): get_auth_request con 0x7fe88409ebe0 auth_method 0 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88966f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88966f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe88966f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient(hunting): handle_auth_done global_id 104305 payload 274 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 -- 172.21.15.181:0/1959239413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409ebe0 msgr2=0x7fe884068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 --2- 172.21.15.181:0/1959239413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409ebe0 0x7fe884068e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 -- 172.21.15.181:0/1959239413 >> v1:172.21.15.153:6789/0 conn(0x7fe88409f000 legacy=0x7fe884069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient: _finish_hunting 0 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 monclient: found mon.noname-c 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.414+0000 7fe889e70640 1 -- 172.21.15.181:0/1959239413 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840a1910 con 0x7fe884131bd0 2024-09-15T03:36:37.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe889e70640 1 -- 172.21.15.181:0/1959239413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 msgr2=0x7fe884131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:37.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe889e70640 1 --2- 172.21.15.181:0/1959239413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe884131fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe880009910 tx=0x7fe8800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33dc6784c6d7b52 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:37.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 1 -- 172.21.15.181:0/1959239413 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880032440 con 0x7fe884131bd0 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: dump: 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:37.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: _finish_auth 0 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe88c0d4640 5 monclient: authenticate success, global_id 104305 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 1 -- 172.21.15.181:0/1959239413 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8800325a0 con 0x7fe884131bd0 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe88c0d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe88c0d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe88c0d4640 4 set_mon_vals no callback set 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 1 -- 172.21.15.181:0/1959239413 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880032940 con 0x7fe884131bd0 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.417+0000 7fe888e6e640 10 monclient: dump: 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 10 monclient: shutdown 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 -- 172.21.15.181:0/1959239413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 msgr2=0x7fe884131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/1959239413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe884131fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe880009910 tx=0x7fe8800091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 -- 172.21.15.181:0/1959239413 shutdown_connections 2024-09-15T03:36:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/1959239413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409ebe0 0x7fe884068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/1959239413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe884131fa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 -- 172.21.15.181:0/1959239413 >> 172.21.15.181:0/1959239413 conn(0x7fe8840713b0 msgr2=0x7fe8840a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 -- 172.21.15.181:0/1959239413 shutdown_connections 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 1 -- 172.21.15.181:0/1959239413 wait complete. 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 10 monclient: build_initial_monmap 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.418+0000 7fe88c0d4640 10 monclient: monmap: 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:37.432 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:37.432 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:37.432 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:37.432 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:37.432 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 Processor -- start 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 -- start start 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient: init 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe88409ebe0 0x7fe88409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe88409ebe0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409f000 0x7fe884130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): picked mon.noname-c con 0x7fe88409f000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe88411b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): picked mon.noname-b con 0x7fe884131bd0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88409bec0 con 0x7fe88409ebe0 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88409e5a0 con 0x7fe884131bd0 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88411dfc0 con 0x7fe88409f000 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88c0d4640 10 monclient(hunting): authenticate will time out at 1201.901653s 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe88411b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe88411b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:53284/0 (socket says 172.21.15.181:53284) 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 1 -- 172.21.15.181:0/823432931 learned_addr learned my addr 172.21.15.181:0/823432931 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): get_auth_request con 0x7fe884131bd0 auth_method 0 2024-09-15T03:36:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88a671640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88966f640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409f000 0x7fe884130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88966f640 10 monclient(hunting): get_auth_request con 0x7fe88409f000 auth_method 0 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88966f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88966f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe88966f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe889e70640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe88409ebe0 0x7fe88409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe889e70640 10 monclient(hunting): get_auth_request con 0x7fe88409ebe0 auth_method 0 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe889e70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe889e70640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.419+0000 7fe889e70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88966f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88966f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88966f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe889e70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe889e70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe889e70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 10 monclient(hunting): handle_auth_done global_id 104308 payload 824 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 1 -- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409f000 msgr2=0x7fe884130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409f000 0x7fe884130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 1 -- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe88409ebe0 msgr2=0x7fe88409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe88409ebe0 0x7fe88409aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 10 monclient: _finish_hunting 0 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 1 monclient: found mon.noname-b 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.420+0000 7fe88a671640 1 -- 172.21.15.181:0/823432931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe880009450 con 0x7fe884131bd0 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88a671640 1 -- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 msgr2=0x7fe88411b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88a671640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe88411b6f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00c890 tx=0x7fe87c002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00ce80 con 0x7fe884131bd0 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: dump: 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:37.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: _finish_auth 0 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c00aa00 con 0x7fe884131bd0 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 5 monclient: authenticate success, global_id 104308 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 10 monclient: _renew_subs 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe884141990 con 0x7fe884131bd0 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00ae00 con 0x7fe884131bd0 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe879ffb640 10 monclient: dump: 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:37.439 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 10 monclient: _renew_subs 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe87bfff640 4 set_mon_vals no callback set 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.422+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8841414d0 con 0x7fe884131bd0 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe87c012630 con 0x7fe884131bd0 2024-09-15T03:36:37.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe88c0d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe88c0d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe88c0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88411e150 con 0x7fe884131bd0 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe879ffb640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe868038740 0x7fe86803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(201..201 src has 106..201) v4 ==== 11713+0+0 (secure 0 0 0) 0x7fe87c04d120 con 0x7fe884131bd0 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe889e70640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe868038740 0x7fe86803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:37.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.423+0000 7fe889e70640 10 monclient: get_auth_request con 0x7fe868038740 auth_method 0 2024-09-15T03:36:37.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.426+0000 7fe889e70640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe868038740 0x7fe86803abf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe8840a1240 tx=0x7fe880032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:37.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.428+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe87c0188a0 con 0x7fe884131bd0 2024-09-15T03:36:37.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.428+0000 7fe879ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:37.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.428+0000 7fe879ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:37.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.583+0000 7fe88c0d4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:37.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.583+0000 7fe88c0d4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:37.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.583+0000 7fe88c0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:37.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:37.583+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe884141990 con 0x7fe884131bd0 2024-09-15T03:36:38.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.232+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v202) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe884141990 con 0x7fe884131bd0 2024-09-15T03:36:38.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.232+0000 7fe879ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:38.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.232+0000 7fe879ffb640 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.329+0000 7fe88c0d4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.329+0000 7fe88c0d4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.329+0000 7fe88c0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.329+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe884141930 con 0x7fe884131bd0 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.330+0000 7fe879ffb640 1 -- 172.21.15.181:0/823432931 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v202) v1 ==== 147+0+0 (secure 0 0 0) 0x7fe87c0182c0 con 0x7fe884131bd0 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.330+0000 7fe879ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.330+0000 7fe879ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-15T03:36:38.332 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h18' already exists 2024-09-15T03:36:38.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe868038740 msgr2=0x7fe86803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe868038740 0x7fe86803abf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe8840a1240 tx=0x7fe880032000 comp rx=0 tx=0).stop 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 10 monclient: shutdown 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 msgr2=0x7fe88411b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe88411b6f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00c890 tx=0x7fe87c002640 comp rx=0 tx=0).stop 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 shutdown_connections 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe868038740 0x7fe86803abf0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe884131bd0 0x7fe88411b6f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe88409f000 0x7fe884130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 --2- 172.21.15.181:0/823432931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe88409ebe0 0x7fe88409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 >> 172.21.15.181:0/823432931 conn(0x7fe8840713b0 msgr2=0x7fe8840a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:38.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 shutdown_connections 2024-09-15T03:36:38.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.332+0000 7fe88c0d4640 1 -- 172.21.15.181:0/823432931 wait complete. 2024-09-15T03:36:38.343 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:38.343 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-15T03:36:38.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.416+0000 7f61586d4640 10 monclient: get_monmap_and_config 2024-09-15T03:36:38.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.416+0000 7f61586d4640 10 monclient: build_initial_monmap 2024-09-15T03:36:38.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.416+0000 7f61586d4640 10 monclient: monmap: 2024-09-15T03:36:38.418 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:38.418 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:38.418333+0000 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:38.418333+0000 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:38.419 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 Processor -- start 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 -- start start 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient: init 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6150069a30 0x7f61501320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6150069a30 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f61500983e0 0x7f6150098840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): picked mon.noname-a con 0x7f61500983e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150070da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): picked mon.noname-b con 0x7f6150070920 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615009cd00 con 0x7f61500983e0 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6150069e00 con 0x7f6150069a30 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615009d140 con 0x7f6150070920 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f61586d4640 10 monclient(hunting): authenticate will time out at 1202.899788s 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6150069a30 0x7f61501320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6150069a30 0x7f61501320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44204/0 (socket says 172.21.15.181:44204) 2024-09-15T03:36:38.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 1 -- 172.21.15.181:0/2823579490 learned_addr learned my addr 172.21.15.181:0/2823579490 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 10 monclient(hunting): get_auth_request con 0x7f6150069a30 auth_method 0 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.417+0000 7f6156470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156c71640 1 --2- 172.21.15.181:0/2823579490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150070da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6155c6f640 1 --2- 172.21.15.181:0/2823579490 >> v2:172.21.15.153:3300/0 conn(0x7f61500983e0 0x7f6150098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156c71640 10 monclient(hunting): get_auth_request con 0x7f6150070920 auth_method 0 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156c71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:38.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6155c6f640 10 monclient(hunting): get_auth_request con 0x7f61500983e0 auth_method 0 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6155c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6155c6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6155c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 10 monclient(hunting): handle_auth_done global_id 104311 payload 274 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 1 -- 172.21.15.181:0/2823579490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 msgr2=0x7f6150070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 1 --2- 172.21.15.181:0/2823579490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150070da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 1 -- 172.21.15.181:0/2823579490 >> v2:172.21.15.153:3300/0 conn(0x7f61500983e0 msgr2=0x7f6150098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 1 --2- 172.21.15.181:0/2823579490 >> v2:172.21.15.153:3300/0 conn(0x7f61500983e0 0x7f6150098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 10 monclient: _finish_hunting 0 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 1 monclient: found mon.noname-c 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.418+0000 7f6156470640 1 -- 172.21.15.181:0/2823579490 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615009e320 con 0x7f6150069a30 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f6156470640 1 -- 172.21.15.181:0/2823579490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 msgr2=0x7f61501320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f6156470640 1 --2- 172.21.15.181:0/2823579490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f61501320e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f614c009910 tx=0x7f614c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803c0d4bcc2c4cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 1 -- 172.21.15.181:0/2823579490 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c032440 con 0x7f6150069a30 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: dump: 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:38.433 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:38.434 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: _finish_auth 0 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 1 -- 172.21.15.181:0/2823579490 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c0325a0 con 0x7f6150069a30 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 1 -- 172.21.15.181:0/2823579490 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c032940 con 0x7f6150069a30 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 5 monclient: authenticate success, global_id 104311 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 4 set_mon_vals no callback set 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:38.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f615546e640 10 monclient: dump: 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 10 monclient: shutdown 2024-09-15T03:36:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 -- 172.21.15.181:0/2823579490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 msgr2=0x7f61501320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 --2- 172.21.15.181:0/2823579490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f61501320e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f614c009910 tx=0x7f614c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 -- 172.21.15.181:0/2823579490 shutdown_connections 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 --2- 172.21.15.181:0/2823579490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150070da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 --2- 172.21.15.181:0/2823579490 >> v2:172.21.15.153:3300/0 conn(0x7f61500983e0 0x7f6150098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 --2- 172.21.15.181:0/2823579490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f61501320e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 -- 172.21.15.181:0/2823579490 >> 172.21.15.181:0/2823579490 conn(0x7f615009e9f0 msgr2=0x7f615009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 -- 172.21.15.181:0/2823579490 shutdown_connections 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.421+0000 7f61586d4640 1 -- 172.21.15.181:0/2823579490 wait complete. 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.422+0000 7f61586d4640 10 monclient: build_initial_monmap 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.422+0000 7f61586d4640 10 monclient: monmap: 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:38.437 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.422+0000 7f61586d4640 1 Processor -- start 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.422+0000 7f61586d4640 1 -- start start 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.422+0000 7f61586d4640 10 monclient: init 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:38.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f615012e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): picked mon.noname-b con 0x7f6150069a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150118560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6150070920 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61500983e0 0x7f6150118aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): picked mon.noname-a con 0x7f61500983e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61501365c0 con 0x7f61500983e0 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615009d8a0 con 0x7f6150069a30 2024-09-15T03:36:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6150134c80 con 0x7f6150070920 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f61586d4640 10 monclient(hunting): authenticate will time out at 1202.905184s 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f615012e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f615012e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44214/0 (socket says 172.21.15.181:44214) 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 -- 172.21.15.181:0/3804462768 learned_addr learned my addr 172.21.15.181:0/3804462768 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): get_auth_request con 0x7f6150069a30 auth_method 0 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:38.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156c71640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61500983e0 0x7f6150118aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150118560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156c71640 10 monclient(hunting): get_auth_request con 0x7f61500983e0 auth_method 0 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156c71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): get_auth_request con 0x7f6150070920 auth_method 0 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:38.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6155c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient(hunting): handle_auth_done global_id 104314 payload 824 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 -- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 msgr2=0x7f6150118560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150118560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 -- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61500983e0 msgr2=0x7f6150118aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61500983e0 0x7f6150118aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient: _finish_hunting 0 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 monclient: found mon.noname-b 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:38.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156470640 1 -- 172.21.15.181:0/3804462768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c009450 con 0x7f6150069a30 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.423+0000 7f6156c71640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61500983e0 0x7f6150118aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f6156470640 1 -- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 msgr2=0x7f615012e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f6156470640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f615012e6e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f614c035040 tx=0x7f614c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c031a40 con 0x7f6150069a30 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: dump: 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 5 monclient: authenticate success, global_id 104314 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c031be0 con 0x7f6150069a30 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 10 monclient: _renew_subs 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615011be20 con 0x7f6150069a30 2024-09-15T03:36:38.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c03d440 con 0x7f6150069a30 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61467fc640 10 monclient: dump: 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:38.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f6154c6d640 4 set_mon_vals no callback set 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 10 monclient: _renew_subs 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.425+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6150119cd0 con 0x7f6150069a30 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f614c039040 con 0x7f6150069a30 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61586d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61586d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61586d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615004bf60 con 0x7f6150069a30 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61467fc640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6134038740 0x7f613403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(202..202 src has 106..202) v4 ==== 11926+0+0 (secure 0 0 0) 0x7f614c06d4d0 con 0x7f6150069a30 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.426+0000 7f6155c6f640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6134038740 0x7f613403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:38.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.427+0000 7f6155c6f640 10 monclient: get_auth_request con 0x7f6134038740 auth_method 0 2024-09-15T03:36:38.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.429+0000 7f6155c6f640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6134038740 0x7f613403abf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6140007ba0 tx=0x7f6140008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:38.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.431+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f614c030b20 con 0x7f6150069a30 2024-09-15T03:36:38.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.431+0000 7f61467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:38.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.431+0000 7f61467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:38.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.586+0000 7f61586d4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:38.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.586+0000 7f61586d4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:38.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.586+0000 7f61586d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:38.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:38.586+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f615009e600 con 0x7f6150069a30 2024-09-15T03:36:39.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.237+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v203) v1 ==== 168+0+0 (secure 0 0 0) 0x7f614c0370b0 con 0x7f6150069a30 2024-09-15T03:36:39.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.237+0000 7f61467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:39.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.237+0000 7f61467fc640 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-15T03:36:39.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:39.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61586d4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:39.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61586d4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:39.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61586d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f61501329c0 con 0x7f6150069a30 2024-09-15T03:36:39.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61467fc640 1 -- 172.21.15.181:0/3804462768 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v203) v1 ==== 147+0+0 (secure 0 0 0) 0x7f614c042070 con 0x7f6150069a30 2024-09-15T03:36:39.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61467fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:39.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.334+0000 7f61467fc640 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-15T03:36:39.337 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h19' already exists 2024-09-15T03:36:39.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6134038740 msgr2=0x7f613403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6134038740 0x7f613403abf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6140007ba0 tx=0x7f6140008040 comp rx=0 tx=0).stop 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 10 monclient: shutdown 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 msgr2=0x7f615012e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f615012e6e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f614c035040 tx=0x7f614c032000 comp rx=0 tx=0).stop 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 shutdown_connections 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6134038740 0x7f613403abf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61500983e0 0x7f6150118aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6150070920 0x7f6150118560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 --2- 172.21.15.181:0/3804462768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6150069a30 0x7f615012e6e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 >> 172.21.15.181:0/3804462768 conn(0x7f615009e9f0 msgr2=0x7f61500a0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 shutdown_connections 2024-09-15T03:36:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.336+0000 7f61586d4640 1 -- 172.21.15.181:0/3804462768 wait complete. 2024-09-15T03:36:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-15T03:36:39.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.419+0000 7fde082e8640 10 monclient: get_monmap_and_config 2024-09-15T03:36:39.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.419+0000 7fde082e8640 10 monclient: build_initial_monmap 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.419+0000 7fde082e8640 10 monclient: monmap: 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:39.421638+0000 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:39.421638+0000 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:39.423 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:39.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.420+0000 7fde082e8640 1 Processor -- start 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.420+0000 7fde082e8640 1 -- start start 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.420+0000 7fde082e8640 10 monclient: init 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.420+0000 7fde082e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fde00096a30 0x7fde0009d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): picked mon.noname-c con 0x7fde00096a30 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde00096e00 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): picked mon.noname-b con 0x7fde00134c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0009c410 con 0x7fde00096e00 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0009c6f0 con 0x7fde00096a30 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0009c130 con 0x7fde00134c90 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde082e8640 10 monclient(hunting): authenticate will time out at 1203.903084s 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fde00096a30 0x7fde0009d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fde00096a30 0x7fde0009d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44226/0 (socket says 172.21.15.181:44226) 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 1 -- 172.21.15.181:0/351296491 learned_addr learned my addr 172.21.15.181:0/351296491 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): get_auth_request con 0x7fde00096a30 auth_method 0 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06084640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 1 --2- 172.21.15.181:0/351296491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): get_auth_request con 0x7fde00134c90 auth_method 0 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient(hunting): handle_auth_done global_id 74849 payload 274 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 1 -- 172.21.15.181:0/351296491 >> v2:172.21.15.181:3300/0 conn(0x7fde00096a30 msgr2=0x7fde0009d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 1 --2- 172.21.15.181:0/351296491 >> v2:172.21.15.181:3300/0 conn(0x7fde00096a30 0x7fde0009d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 1 -- 172.21.15.181:0/351296491 >> v1:172.21.15.181:6789/0 conn(0x7fde00096e00 legacy=0x7fde00097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient: _finish_hunting 0 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 1 monclient: found mon.noname-b 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:39.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.421+0000 7fde06885640 1 -- 172.21.15.181:0/351296491 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0009ab30 con 0x7fde00134c90 2024-09-15T03:36:39.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde06885640 1 -- 172.21.15.181:0/351296491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 msgr2=0x7fde00137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:39.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde06885640 1 --2- 172.21.15.181:0/351296491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00137070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009e60 tx=0x7fddf8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fcb890c57f97291 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:39.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 1 -- 172.21.15.181:0/351296491 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8007760 con 0x7fde00134c90 2024-09-15T03:36:39.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: dump: 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: _finish_auth 0 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde082e8640 5 monclient: authenticate success, global_id 74849 2024-09-15T03:36:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 1 -- 172.21.15.181:0/351296491 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf802d050 con 0x7fde00134c90 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 1 -- 172.21.15.181:0/351296491 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8007af0 con 0x7fde00134c90 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde082e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde082e8640 4 set_mon_vals no callback set 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:39.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.424+0000 7fde05082640 10 monclient: dump: 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 10 monclient: shutdown 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:39.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 -- 172.21.15.181:0/351296491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 msgr2=0x7fde00137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 --2- 172.21.15.181:0/351296491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00137070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009e60 tx=0x7fddf8009780 comp rx=0 tx=0).stop 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 -- 172.21.15.181:0/351296491 shutdown_connections 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 --2- 172.21.15.181:0/351296491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00137070 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 --2- 172.21.15.181:0/351296491 >> v2:172.21.15.181:3300/0 conn(0x7fde00096a30 0x7fde0009d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 -- 172.21.15.181:0/351296491 >> 172.21.15.181:0/351296491 conn(0x7fde000a0fe0 msgr2=0x7fde000a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 -- 172.21.15.181:0/351296491 shutdown_connections 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 1 -- 172.21.15.181:0/351296491 wait complete. 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.425+0000 7fde082e8640 10 monclient: build_initial_monmap 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient: monmap: 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:39.441 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 Processor -- start 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 -- start start 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient: init 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 0x7fde00141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): picked mon.noname-b con 0x7fde00096a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:39.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde00096e00 0x7fde00133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): picked mon.noname-a con 0x7fde00096e00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00098540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): picked mon.noname-c con 0x7fde00134c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde00120ca0 con 0x7fde00096e00 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0009b9d0 con 0x7fde00096a30 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0006e390 con 0x7fde00134c90 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde082e8640 10 monclient(hunting): authenticate will time out at 1203.908908s 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 0x7fde00141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 0x7fde00141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44230/0 (socket says 172.21.15.181:44230) 2024-09-15T03:36:39.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 1 -- 172.21.15.181:0/2005663316 learned_addr learned my addr 172.21.15.181:0/2005663316 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 10 monclient(hunting): get_auth_request con 0x7fde00096a30 auth_method 0 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.426+0000 7fde06084640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06885640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00098540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06885640 10 monclient(hunting): get_auth_request con 0x7fde00134c90 auth_method 0 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06885640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06885640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde05883640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde00096e00 0x7fde00133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde05883640 10 monclient(hunting): get_auth_request con 0x7fde00096e00 auth_method 0 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde05883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde05883640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde05883640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 10 monclient(hunting): handle_auth_done global_id 104320 payload 824 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 1 -- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 msgr2=0x7fde00098540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00098540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 1 -- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde00096e00 msgr2=0x7fde00133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde00096e00 0x7fde00133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 10 monclient: _finish_hunting 0 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 1 monclient: found mon.noname-b 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06084640 1 -- 172.21.15.181:0/2005663316 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8009a20 con 0x7fde00096a30 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde06885640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00098540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.427+0000 7fde05883640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde00096e00 0x7fde00133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde06084640 1 -- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 msgr2=0x7fde00141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde06084640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 0x7fde00141190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00ea20 tx=0x7fddfc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0095a0 con 0x7fde00096a30 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: dump: 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 5 monclient: authenticate success, global_id 104320 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc00cb50 con 0x7fde00096a30 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 10 monclient: _renew_subs 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc013470 con 0x7fde00096a30 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde00121fa0 con 0x7fde00096a30 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fddf67fc640 10 monclient: dump: 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde04881640 4 set_mon_vals no callback set 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 10 monclient: _renew_subs 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.429+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde00141ef0 con 0x7fde00096a30 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fddfc013820 con 0x7fde00096a30 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fde082e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fde082e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fde082e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0004bf60 con 0x7fde00096a30 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fddf67fc640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdde4038740 0x7fdde403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(203..203 src has 106..203) v4 ==== 12139+0+0 (secure 0 0 0) 0x7fddfc04da50 con 0x7fde00096a30 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fde05883640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdde4038740 0x7fdde403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.430+0000 7fde05883640 10 monclient: get_auth_request con 0x7fdde4038740 auth_method 0 2024-09-15T03:36:39.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.433+0000 7fde05883640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdde4038740 0x7fdde403abf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fddf0002860 tx=0x7fddf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:39.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.435+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fddfc023ad0 con 0x7fde00096a30 2024-09-15T03:36:39.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.435+0000 7fddf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:39.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.435+0000 7fddf67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:39.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.433+0000 7f262c39d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:39.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.433+0000 7f262c39d640 10 monclient: build_initial_monmap 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.433+0000 7f262c39d640 10 monclient: monmap: 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:39.434507+0000 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:39.434507+0000 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:39.460 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.434+0000 7f262c39d640 1 Processor -- start 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.434+0000 7f262c39d640 1 -- start start 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.434+0000 7f262c39d640 10 monclient: init 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.434+0000 7f262c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.434+0000 7f262c39d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26240981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2624131da0 0x7f2624136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f2624131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2624132170 0x7f26241390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f2624132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262409ca60 con 0x7f26240981c0 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2624131bd0 con 0x7f2624132170 2024-09-15T03:36:39.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262409cca0 con 0x7f2624131da0 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262c39d640 10 monclient(hunting): authenticate will time out at 1197.074095s 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a139640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f26240981c0 0x7f2624098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37048/0 (socket says 172.21.15.153:37048) 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a139640 1 -- 172.21.15.153:0/339351881 learned_addr learned my addr 172.21.15.153:0/339351881 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2624131da0 0x7f2624136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): get_auth_request con 0x7f2624131da0 auth_method 0 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 1 --2- 172.21.15.153:0/339351881 >> v2:172.21.15.181:3300/0 conn(0x7f2624132170 0x7f26241390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): get_auth_request con 0x7f2624132170 auth_method 0 2024-09-15T03:36:39.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient(hunting): handle_auth_done global_id 104323 payload 274 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 1 -- 172.21.15.153:0/339351881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2624131da0 msgr2=0x7f2624136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2624131da0 0x7f2624136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 1 -- 172.21.15.153:0/339351881 >> v1:172.21.15.153:6789/0 conn(0x7f26240981c0 legacy=0x7f2624098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 1 monclient: found mon.noname-c 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f262a93a640 1 -- 172.21.15.153:0/339351881 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262409e010 con 0x7f2624132170 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.435+0000 7f2629938640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2624131da0 0x7f2624136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f262a93a640 1 -- 172.21.15.153:0/339351881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624132170 msgr2=0x7f26241390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f262a93a640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624132170 0x7f26241390c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f261c009e60 tx=0x7f261c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5e396d409fadd0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 1 -- 172.21.15.153:0/339351881 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c007760 con 0x7f2624132170 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: dump: 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: _finish_auth 0 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 1 -- 172.21.15.153:0/339351881 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c02d050 con 0x7f2624132170 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f262c39d640 5 monclient: authenticate success, global_id 104323 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:39.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f262c39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 1 -- 172.21.15.153:0/339351881 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c007af0 con 0x7f2624132170 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f262c39d640 4 set_mon_vals no callback set 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.438+0000 7f2629137640 10 monclient: dump: 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 10 monclient: shutdown 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 -- 172.21.15.153:0/339351881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624132170 msgr2=0x7f26241390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624132170 0x7f26241390c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f261c009e60 tx=0x7f261c009780 comp rx=0 tx=0).stop 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 -- 172.21.15.153:0/339351881 shutdown_connections 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624132170 0x7f26241390c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 --2- 172.21.15.153:0/339351881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2624131da0 0x7f2624136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 -- 172.21.15.153:0/339351881 >> 172.21.15.153:0/339351881 conn(0x7f26240a1e30 msgr2=0x7f26240a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 -- 172.21.15.153:0/339351881 shutdown_connections 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 -- 172.21.15.153:0/339351881 wait complete. 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 10 monclient: build_initial_monmap 2024-09-15T03:36:39.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 10 monclient: monmap: 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.439+0000 7f262c39d640 1 Processor -- start 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 -- start start 2024-09-15T03:36:39.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient: init 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 0x7f26241310f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f26240981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624131da0 0x7f2624140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f2624131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2624132170 0x7f2624137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f2624132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262409bba0 con 0x7f2624132170 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262411aa90 con 0x7f2624131da0 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:39.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26241194d0 con 0x7f26240981c0 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262c39d640 10 monclient(hunting): authenticate will time out at 1197.079506s 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 0x7f26241310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 0x7f26241310f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37914/0 (socket says 172.21.15.153:37914) 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 -- 172.21.15.153:0/1293792261 learned_addr learned my addr 172.21.15.153:0/1293792261 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): get_auth_request con 0x7f26240981c0 auth_method 0 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2624132170 0x7f2624137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): get_auth_request con 0x7f2624132170 auth_method 0 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624131da0 0x7f2624140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): get_auth_request con 0x7f2624131da0 auth_method 0 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:39.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f2629938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient(hunting): handle_auth_done global_id 74858 payload 824 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 -- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624131da0 msgr2=0x7f2624140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624131da0 0x7f2624140dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 -- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2624132170 msgr2=0x7f2624137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2624132170 0x7f2624137c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient: _finish_hunting 0 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 monclient: found mon.noname-c 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.440+0000 7f262a139640 1 -- 172.21.15.153:0/1293792261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2620009390 con 0x7f26240981c0 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262a139640 1 -- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 msgr2=0x7f26241310f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:39.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262a139640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 0x7f26241310f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f261400c5e0 tx=0x7f2614004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614005cb0 con 0x7f26240981c0 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: dump: 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.473 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2614005e30 con 0x7f26240981c0 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 5 monclient: authenticate success, global_id 74858 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614007340 con 0x7f26240981c0 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 10 monclient: _renew_subs 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c009a20 con 0x7f26240981c0 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f2628936640 4 set_mon_vals no callback set 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f261a7fc640 10 monclient: dump: 2024-09-15T03:36:39.474 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 10 monclient: _renew_subs 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.443+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26241186c0 con 0x7f26240981c0 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f261401b400 con 0x7f26240981c0 2024-09-15T03:36:39.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f262c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f262c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f262c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f261a7fc640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2608038740 0x7f260803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262411a310 con 0x7f26240981c0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(203..203 src has 106..203) v4 ==== 12139+0+0 (secure 0 0 0) 0x7f261404d2c0 con 0x7f26240981c0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f2629938640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2608038740 0x7f260803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.444+0000 7f2629938640 10 monclient: get_auth_request con 0x7f2608038740 auth_method 0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.447+0000 7f2629938640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2608038740 0x7f260803abf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2624139430 tx=0x7f26200090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.448+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f261401ab40 con 0x7f26240981c0 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.448+0000 7f261a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.448+0000 7f261a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:39.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.591+0000 7fde082e8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:39.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.591+0000 7fde082e8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:39.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.591+0000 7fde082e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:39.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:39.591+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fde00141730 con 0x7fde00096a30 2024-09-15T03:36:39.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.665+0000 7f262c39d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:39.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.665+0000 7f262c39d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:39.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.665+0000 7f262c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:39.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.665+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f262404bf60 con 0x7f26240981c0 2024-09-15T03:36:39.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.666+0000 7f261a7fc640 1 -- 172.21.15.153:0/1293792261 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f261401a360 con 0x7f26240981c0 2024-09-15T03:36:39.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.666+0000 7f261a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:39.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.666+0000 7f261a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:39.668 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":88,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2608038740 msgr2=0x7f260803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2608038740 0x7f260803abf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2624139430 tx=0x7f26200090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 10 monclient: shutdown 2024-09-15T03:36:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 msgr2=0x7f26241310f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 0x7f26241310f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f261400c5e0 tx=0x7f2614004960 comp rx=0 tx=0).stop 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 shutdown_connections 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2608038740 0x7f260803abf0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2624132170 0x7f2624137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2624131da0 0x7f2624140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 --2- 172.21.15.153:0/1293792261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26240981c0 0x7f26241310f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 >> 172.21.15.153:0/1293792261 conn(0x7f26240a1e30 msgr2=0x7f2624132570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 shutdown_connections 2024-09-15T03:36:39.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:39.668+0000 7f262c39d640 1 -- 172.21.15.153:0/1293792261 wait complete. 2024-09-15T03:36:40.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.247+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v204) v1 ==== 168+0+0 (secure 0 0 0) 0x7fddfc0232f0 con 0x7fde00096a30 2024-09-15T03:36:40.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.247+0000 7fddf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:40.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.247+0000 7fddf67fc640 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-15T03:36:40.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fde082e8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:40.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fde082e8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:40.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fde082e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fde0009b2f0 con 0x7fde00096a30 2024-09-15T03:36:40.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fddf67fc640 1 -- 172.21.15.181:0/2005663316 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v204) v1 ==== 147+0+0 (secure 0 0 0) 0x7fddfc0234d0 con 0x7fde00096a30 2024-09-15T03:36:40.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fddf67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:40.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.342+0000 7fddf67fc640 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-15T03:36:40.345 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h20' already exists 2024-09-15T03:36:40.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdde4038740 msgr2=0x7fdde403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdde4038740 0x7fdde403abf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fddf0002860 tx=0x7fddf0009040 comp rx=0 tx=0).stop 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 10 monclient: shutdown 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 msgr2=0x7fde00141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 0x7fde00141190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00ea20 tx=0x7fddfc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 shutdown_connections 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdde4038740 0x7fdde403abf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde00134c90 0x7fde00098540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde00096e00 0x7fde00133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 --2- 172.21.15.181:0/2005663316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde00096a30 0x7fde00141190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 >> 172.21.15.181:0/2005663316 conn(0x7fde000a0fe0 msgr2=0x7fde000a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 shutdown_connections 2024-09-15T03:36:40.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.344+0000 7fde082e8640 1 -- 172.21.15.181:0/2005663316 wait complete. 2024-09-15T03:36:40.355 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:40.355 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-15T03:36:40.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.428+0000 7f071acdd640 10 monclient: get_monmap_and_config 2024-09-15T03:36:40.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.428+0000 7f071acdd640 10 monclient: build_initial_monmap 2024-09-15T03:36:40.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.428+0000 7f071acdd640 10 monclient: monmap: 2024-09-15T03:36:40.430 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:40.430500+0000 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:40.430500+0000 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:40.431 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 Processor -- start 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 -- start start 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient: init 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0714131bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0714069e10 0x7f0714131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): picked mon.noname-c con 0x7f0714069e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0714136d60 0x7f0714139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): picked mon.noname-a con 0x7f0714136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07140a06d0 con 0x7f0714131bd0 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0714071780 con 0x7f0714136d60 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071406a210 con 0x7f0714069e10 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f071acdd640 10 monclient(hunting): authenticate will time out at 1204.911901s 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.429+0000 7f070bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0714069e10 0x7f0714131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0714069e10 0x7f0714131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44232/0 (socket says 172.21.15.181:44232) 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 -- 172.21.15.181:0/2583107149 learned_addr learned my addr 172.21.15.181:0/2583107149 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): get_auth_request con 0x7f0714069e10 auth_method 0 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:40.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f071927a640 1 --2- 172.21.15.181:0/2583107149 >> v2:172.21.15.153:3300/0 conn(0x7f0714136d60 0x7f0714139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient(hunting): handle_auth_done global_id 104329 payload 274 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 -- 172.21.15.181:0/2583107149 >> v2:172.21.15.153:3300/0 conn(0x7f0714136d60 msgr2=0x7f0714139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:40.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 --2- 172.21.15.181:0/2583107149 >> v2:172.21.15.153:3300/0 conn(0x7f0714136d60 0x7f0714139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 -- 172.21.15.181:0/2583107149 >> v1:172.21.15.153:6789/0 conn(0x7f0714131bd0 legacy=0x7f071409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 monclient: found mon.noname-c 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f070bfff640 1 -- 172.21.15.181:0/2583107149 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140a0010 con 0x7f0714069e10 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f071927a640 10 monclient: get_auth_request con 0x7f0714136d60 auth_method 0 2024-09-15T03:36:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.430+0000 7f071927a640 1 --2- 172.21.15.181:0/2583107149 >> v2:172.21.15.153:3300/0 conn(0x7f0714136d60 0x7f0714139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070bfff640 1 -- 172.21.15.181:0/2583107149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714069e10 msgr2=0x7f0714131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070bfff640 1 --2- 172.21.15.181:0/2583107149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714069e10 0x7f0714131fa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0714068d90 tx=0x7f07040091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2b3b4cf3d7b890a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 1 -- 172.21.15.181:0/2583107149 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704032440 con 0x7f0714069e10 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: dump: 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 1 -- 172.21.15.181:0/2583107149 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07040325a0 con 0x7f0714069e10 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 5 monclient: authenticate success, global_id 104329 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 4 set_mon_vals no callback set 2024-09-15T03:36:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 1 -- 172.21.15.181:0/2583107149 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704032940 con 0x7f0714069e10 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f070b7fe640 10 monclient: dump: 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:40.446 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 10 monclient: shutdown 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 -- 172.21.15.181:0/2583107149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714069e10 msgr2=0x7f0714131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 --2- 172.21.15.181:0/2583107149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714069e10 0x7f0714131fa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0714068d90 tx=0x7f07040091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 -- 172.21.15.181:0/2583107149 shutdown_connections 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 --2- 172.21.15.181:0/2583107149 >> v2:172.21.15.153:3300/0 conn(0x7f0714136d60 0x7f0714139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 --2- 172.21.15.181:0/2583107149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714069e10 0x7f0714131fa0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 -- 172.21.15.181:0/2583107149 >> 172.21.15.181:0/2583107149 conn(0x7f07140713b0 msgr2=0x7f07140a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 -- 172.21.15.181:0/2583107149 shutdown_connections 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.433+0000 7f071acdd640 1 -- 172.21.15.181:0/2583107149 wait complete. 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.434+0000 7f071acdd640 10 monclient: build_initial_monmap 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.434+0000 7f071acdd640 10 monclient: monmap: 2024-09-15T03:36:40.447 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.434+0000 7f071acdd640 1 Processor -- start 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.434+0000 7f071acdd640 1 -- start start 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.434+0000 7f071acdd640 10 monclient: init 2024-09-15T03:36:40.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0714069e10 0x7f0714141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): picked mon.noname-a con 0x7f0714069e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0714131bd0 0x7f0714141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): picked mon.noname-c con 0x7f0714131bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 0x7f07140995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): picked mon.noname-b con 0x7f0714136d60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071409b1f0 con 0x7f0714069e10 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071411c920 con 0x7f0714136d60 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:40.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071411cc00 con 0x7f0714131bd0 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071acdd640 10 monclient(hunting): authenticate will time out at 1204.917322s 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 0x7f07140995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 0x7f07140995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44246/0 (socket says 172.21.15.181:44246) 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 -- 172.21.15.181:0/2781423940 learned_addr learned my addr 172.21.15.181:0/2781423940 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): get_auth_request con 0x7f0714136d60 auth_method 0 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:40.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f070bfff640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0714131bd0 0x7f0714141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f0718a79640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0714069e10 0x7f0714141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient(hunting): handle_auth_done global_id 104332 payload 824 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 -- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0714131bd0 msgr2=0x7f0714141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0714131bd0 0x7f0714141940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 -- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0714069e10 msgr2=0x7f0714141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0714069e10 0x7f0714141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient: _finish_hunting 0 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 monclient: found mon.noname-b 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f071927a640 1 -- 172.21.15.181:0/2781423940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704009450 con 0x7f0714136d60 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f0718a79640 10 monclient: get_auth_request con 0x7f0714069e10 auth_method 0 2024-09-15T03:36:40.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.435+0000 7f0718a79640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0714069e10 0x7f0714141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.437+0000 7f071927a640 1 -- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 msgr2=0x7f07140995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.437+0000 7f071927a640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 0x7f07140995e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f070c00c880 tx=0x7f070c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.437+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.439+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c00ce10 con 0x7f0714136d60 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.439+0000 7f0708ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: dump: 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:40.452 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c007a40 con 0x7f0714136d60 2024-09-15T03:36:40.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 5 monclient: authenticate success, global_id 104332 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c007e00 con 0x7f0714136d60 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 10 monclient: dump: 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:40.454 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 10 monclient: _renew_subs 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f070affd640 4 set_mon_vals no callback set 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071411cee0 con 0x7f0714136d60 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 10 monclient: _renew_subs 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0714141e80 con 0x7f0714136d60 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.440+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f070c011600 con 0x7f0714136d60 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f071acdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f071acdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f071acdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071411d0d0 con 0x7f0714136d60 2024-09-15T03:36:40.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f0708ff9640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0700038790 0x7f070003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:40.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(204..204 src has 106..204) v4 ==== 12352+0+0 (secure 0 0 0) 0x7f070c04d8c0 con 0x7f0714136d60 2024-09-15T03:36:40.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f0718a79640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0700038790 0x7f070003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:40.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.441+0000 7f0718a79640 10 monclient: get_auth_request con 0x7f0700038790 auth_method 0 2024-09-15T03:36:40.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.443+0000 7f0718a79640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0700038790 0x7f070003ac40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0710007fd0 tx=0x7f0710009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:40.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.445+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f070c017ae0 con 0x7f0714136d60 2024-09-15T03:36:40.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.445+0000 7f0708ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:40.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.445+0000 7f0708ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:40.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.601+0000 7f071acdd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:40.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.601+0000 7f071acdd640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:40.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.601+0000 7f071acdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:40.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:40.601+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f071404bf60 con 0x7f0714136d60 2024-09-15T03:36:42.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.789+0000 7f152335d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.789+0000 7f152335d640 10 monclient: build_initial_monmap 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.789+0000 7f152335d640 10 monclient: monmap: 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:42.789855+0000 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:42.789855+0000 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:42.793 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.789+0000 7f152335d640 1 Processor -- start 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.789+0000 7f152335d640 1 -- start start 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.789+0000 7f152335d640 10 monclient: init 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f151c096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f151c0971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient(hunting): picked mon.noname-c con 0x7f151c134b40 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c09d4a0 con 0x7f151c096e10 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c099620 con 0x7f151c0971e0 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c099340 con 0x7f151c134b40 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f152335d640 10 monclient(hunting): authenticate will time out at 1200.429504s 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15210f9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f151c096e10 0x7f151c09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37052/0 (socket says 172.21.15.153:37052) 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15210f9640 1 -- 172.21.15.153:0/3752673711 learned_addr learned my addr 172.21.15.153:0/3752673711 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 10 monclient(hunting): get_auth_request con 0x7f151c134b40 auth_method 0 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:42.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:42.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:42.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.790+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:42.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.792+0000 7f15210f9640 1 -- 172.21.15.153:0/3752673711 >> v1:172.21.15.153:6789/0 conn(0x7f151c096e10 legacy=0x7f151c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:42.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.793+0000 7f15208f8640 1 -- 172.21.15.153:0/3752673711 >> v1:172.21.15.181:6789/0 conn(0x7f151c0971e0 legacy=0x7f151c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:42.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:42.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 10 monclient(hunting): get_auth_request con 0x7f151c134b40 auth_method 0 2024-09-15T03:36:42.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:42.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:42.991+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:43.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 10 monclient(hunting): get_auth_request con 0x7f151c134b40 auth_method 0 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:43.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:43.392+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:43.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:43.436+0000 7f06fb7fe640 10 monclient: tick 2024-09-15T03:36:43.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:43.436+0000 7f06fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:43.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:43.436+0000 7f06fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:44.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:44.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 10 monclient(hunting): get_auth_request con 0x7f151c134b40 auth_method 0 2024-09-15T03:36:44.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:44.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:44.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:44.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:44.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:44.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:44.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:44.194+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:36:44.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:44.436+0000 7f06fb7fe640 10 monclient: tick 2024-09-15T03:36:44.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:44.436+0000 7f06fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.254+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472412427 0 0) 0x7f151c099620 con 0x7f151c0971e0 2024-09-15T03:36:45.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.254+0000 7f1513fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.254+0000 7f1513fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.254+0000 7f1513fff640 10 monclient(hunting): my global_id is 104335 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15080034e0 con 0x7f151c0971e0 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2872295706 0 0) 0x7f15080034e0 con 0x7f151c0971e0 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 msgr2=0x7f151c136f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 --2- 172.21.15.153:0/3752673711 >> v2:172.21.15.181:3300/0 conn(0x7f151c134b40 0x7f151c136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 >> v1:172.21.15.153:6789/0 conn(0x7f151c096e10 legacy=0x7f151c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c099900 con 0x7f151c0971e0 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f150c001770 con 0x7f151c0971e0 2024-09-15T03:36:45.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:36:45.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: dump: 2024-09-15T03:36:45.260 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.260 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.261 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.261 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.261 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.261 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:36:45.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 >> v1:172.21.15.181:6789/0 conn(0x7f151c0971e0 legacy=0x7f151c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1508003b20 0x7f1508003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): picked mon.a con 0x7f1508003b20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1508004780 0x7f1508004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): picked mon.b con 0x7f1508004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 0x7f150800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): picked mon.c con 0x7f15080053b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:45.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c099340 con 0x7f1508003b20 2024-09-15T03:36:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150800c380 con 0x7f1508004780 2024-09-15T03:36:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150800c540 con 0x7f15080053b0 2024-09-15T03:36:45.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f1513fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 0x7f150800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1508004780 0x7f1508004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): get_auth_request con 0x7f15080053b0 auth_method 0 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): get_auth_request con 0x7f1508004780 auth_method 0 2024-09-15T03:36:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15208f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 10 monclient(hunting): handle_auth_done global_id 104335 payload 293 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1508004780 msgr2=0x7f1508004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.255+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1508004780 0x7f1508004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1508003b20 msgr2=0x7f1508003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1508003b20 0x7f1508003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 1 monclient: found mon.c 2024-09-15T03:36:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15080034e0 con 0x7f15080053b0 2024-09-15T03:36:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f152335d640 5 monclient: authenticate success, global_id 104335 2024-09-15T03:36:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f152335d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:45.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.256+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 reap_dead start 2024-09-15T03:36:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.258+0000 7f15218fa640 1 -- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 msgr2=0x7f150800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.258+0000 7f15218fa640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 0x7f150800baa0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1514015760 tx=0x7f1514009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db88e1d68127cc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151402ed90 con 0x7f15080053b0 2024-09-15T03:36:45.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: dump: 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.279 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514032070 con 0x7f15080053b0 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f152335d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f152335d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:45.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f152335d640 4 set_mon_vals no callback set 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 1 -- 172.21.15.153:0/3752673711 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514038650 con 0x7f15080053b0 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.261+0000 7f1513fff640 10 monclient: dump: 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.281 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 10 monclient: shutdown 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 -- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 msgr2=0x7f150800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 0x7f150800baa0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1514015760 tx=0x7f1514009860 comp rx=0 tx=0).stop 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 -- 172.21.15.153:0/3752673711 shutdown_connections 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 --2- 172.21.15.153:0/3752673711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15080053b0 0x7f150800baa0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 -- 172.21.15.153:0/3752673711 >> 172.21.15.153:0/3752673711 conn(0x7f151c0672b0 msgr2=0x7f151c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 -- 172.21.15.153:0/3752673711 shutdown_connections 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.262+0000 7f152335d640 1 -- 172.21.15.153:0/3752673711 wait complete. 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.263+0000 7f152335d640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.263+0000 7f152335d640 10 monclient: monmap: 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:45.284 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.263+0000 7f152335d640 1 Processor -- start 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.263+0000 7f152335d640 1 -- start start 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.263+0000 7f152335d640 10 monclient: init 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:45.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c12e360 0x7f151c12e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): picked mon.noname-b con 0x7f151c12e360 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 0x7f151c06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): picked mon.noname-a con 0x7f151c119790 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 0x7f151c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): picked mon.noname-c con 0x7f151c06d5f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c09f760 con 0x7f151c119790 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c117820 con 0x7f151c12e360 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c09b050 con 0x7f151c06d5f0 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f152335d640 10 monclient(hunting): authenticate will time out at 1202.903714s 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15218fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 0x7f151c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 0x7f151c06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 0x7f151c06d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60556/0 (socket says 172.21.15.153:60556) 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15218fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 0x7f151c06da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45364/0 (socket says 172.21.15.153:45364) 2024-09-15T03:36:45.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 1 -- 172.21.15.153:0/437547116 learned_addr learned my addr 172.21.15.153:0/437547116 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): get_auth_request con 0x7f151c119790 auth_method 0 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15218fa640 10 monclient(hunting): get_auth_request con 0x7f151c06d5f0 auth_method 0 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15218fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15218fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15218fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15210f9640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c12e360 0x7f151c12e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.264+0000 7f15208f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): get_auth_request con 0x7f151c12e360 auth_method 0 2024-09-15T03:36:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15210f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 10 monclient(hunting): handle_auth_done global_id 74861 payload 824 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 1 -- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c12e360 msgr2=0x7f151c12e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c12e360 0x7f151c12e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 1 -- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 msgr2=0x7f151c06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 0x7f151c06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 1 monclient: found mon.noname-c 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15218fa640 1 -- 172.21.15.153:0/437547116 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c118ac0 con 0x7f151c06d5f0 2024-09-15T03:36:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.265+0000 7f15208f8640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 0x7f151c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15218fa640 1 -- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 msgr2=0x7f151c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15218fa640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 0x7f151c06da60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f150800c6d0 tx=0x7f151403a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514042810 con 0x7f151c06d5f0 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: dump: 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 5 monclient: authenticate success, global_id 74861 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514042990 con 0x7f151c06d5f0 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 10 monclient: _renew_subs 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c118500 con 0x7f151c06d5f0 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514042d50 con 0x7f151c06d5f0 2024-09-15T03:36:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15117fa640 10 monclient: dump: 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.294 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 10 monclient: _renew_subs 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f15137fe640 4 set_mon_vals no callback set 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.267+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c06dfa0 con 0x7f151c06d5f0 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.268+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1514003040 con 0x7f151c06d5f0 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.268+0000 7f152335d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.268+0000 7f152335d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.268+0000 7f152335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.269+0000 7f15117fa640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1500038740 0x7f150003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.269+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c04bf60 con 0x7f151c06d5f0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.269+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(204..204 src has 106..204) v4 ==== 12352+0+0 (secure 0 0 0) 0x7f1514072040 con 0x7f151c06d5f0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.269+0000 7f15210f9640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1500038740 0x7f150003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.269+0000 7f15210f9640 10 monclient: get_auth_request con 0x7f1500038740 auth_method 0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.271+0000 7f15210f9640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1500038740 0x7f150003abf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1518000c00 tx=0x7f1518009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.273+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f151404ab40 con 0x7f151c06d5f0 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.273+0000 7f15117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.273+0000 7f15117fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:45.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.323+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v205) v1 ==== 168+0+0 (secure 0 0 0) 0x7f070c017300 con 0x7f0714136d60 2024-09-15T03:36:45.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.323+0000 7f0708ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:45.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.323+0000 7f0708ff9640 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-15T03:36:45.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.423+0000 7f071acdd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:45.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.423+0000 7f071acdd640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:45.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.423+0000 7f071acdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.423+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0714064990 con 0x7f0714136d60 2024-09-15T03:36:45.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.424+0000 7f0708ff9640 1 -- 172.21.15.181:0/2781423940 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v205) v1 ==== 147+0+0 (secure 0 0 0) 0x7f070c0174e0 con 0x7f0714136d60 2024-09-15T03:36:45.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.424+0000 7f0708ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:45.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.424+0000 7f0708ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-15T03:36:45.425 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h21' already exists 2024-09-15T03:36:45.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0700038790 msgr2=0x7f070003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0700038790 0x7f070003ac40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0710007fd0 tx=0x7f0710009040 comp rx=0 tx=0).stop 2024-09-15T03:36:45.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 10 monclient: shutdown 2024-09-15T03:36:45.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 msgr2=0x7f07140995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 0x7f07140995e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f070c00c880 tx=0x7f070c008040 comp rx=0 tx=0).stop 2024-09-15T03:36:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 shutdown_connections 2024-09-15T03:36:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0700038790 0x7f070003ac40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0714136d60 0x7f07140995e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0714131bd0 0x7f0714141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 --2- 172.21.15.181:0/2781423940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0714069e10 0x7f0714141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 >> 172.21.15.181:0/2781423940 conn(0x7f07140713b0 msgr2=0x7f07140a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 shutdown_connections 2024-09-15T03:36:45.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.426+0000 7f071acdd640 1 -- 172.21.15.181:0/2781423940 wait complete. 2024-09-15T03:36:45.436 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:45.436 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.495+0000 7f152335d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.495+0000 7f152335d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.495+0000 7f152335d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.495+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f151c118170 con 0x7f151c06d5f0 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.496+0000 7f15117fa640 1 -- 172.21.15.153:0/437547116 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f151404a360 con 0x7f151c06d5f0 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.496+0000 7f15117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.496+0000 7f15117fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:45.497 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1500038740 msgr2=0x7f150003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1500038740 0x7f150003abf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1518000c00 tx=0x7f1518009040 comp rx=0 tx=0).stop 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 10 monclient: shutdown 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 msgr2=0x7f151c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 0x7f151c06da60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f150800c6d0 tx=0x7f151403a000 comp rx=0 tx=0).stop 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 shutdown_connections 2024-09-15T03:36:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1500038740 0x7f150003abf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151c06d5f0 0x7f151c06da60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f151c119790 0x7f151c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 --2- 172.21.15.153:0/437547116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f151c12e360 0x7f151c12e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 >> 172.21.15.153:0/437547116 conn(0x7f151c0672b0 msgr2=0x7f151c09d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 shutdown_connections 2024-09-15T03:36:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.498+0000 7f152335d640 1 -- 172.21.15.153:0/437547116 wait complete. 2024-09-15T03:36:45.509 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:36:45.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:36:45.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:36:45.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.512+0000 7f558e694640 10 monclient: get_monmap_and_config 2024-09-15T03:36:45.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.512+0000 7f558e694640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.512+0000 7f558e694640 10 monclient: monmap: 2024-09-15T03:36:45.513 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:45.513 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.513 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:45.513242+0000 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:45.513242+0000 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:45.514 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 Processor -- start 2024-09-15T03:36:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 -- start start 2024-09-15T03:36:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient: init 2024-09-15T03:36:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f558806df30 0x7f558806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): picked mon.noname-c con 0x7f558806df30 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f558806e300 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f558809e1e0 0x7f558809e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): picked mon.noname-a con 0x7f558809e1e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880696e0 con 0x7f558806e300 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558806a290 con 0x7f558809e1e0 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558809f840 con 0x7f558806df30 2024-09-15T03:36:45.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558e694640 10 monclient(hunting): authenticate will time out at 1209.994680s 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f558806df30 0x7f558806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f558806df30 0x7f558806a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44252/0 (socket says 172.21.15.181:44252) 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 1 -- 172.21.15.181:0/250893160 learned_addr learned my addr 172.21.15.181:0/250893160 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): get_auth_request con 0x7f558806df30 auth_method 0 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558cc31640 1 --2- 172.21.15.181:0/250893160 >> v2:172.21.15.153:3300/0 conn(0x7f558809e1e0 0x7f558809e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f5587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558cc31640 10 monclient(hunting): get_auth_request con 0x7f558809e1e0 auth_method 0 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558cc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558cc31640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.513+0000 7f558cc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f558cc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f558cc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f558cc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 10 monclient(hunting): handle_auth_done global_id 104341 payload 274 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 1 -- 172.21.15.181:0/250893160 >> v2:172.21.15.153:3300/0 conn(0x7f558809e1e0 msgr2=0x7f558809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 1 --2- 172.21.15.181:0/250893160 >> v2:172.21.15.153:3300/0 conn(0x7f558809e1e0 0x7f558809e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 1 -- 172.21.15.181:0/250893160 >> v1:172.21.15.153:6789/0 conn(0x7f558806e300 legacy=0x7f558809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 1 monclient: found mon.noname-c 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.514+0000 7f5587fff640 1 -- 172.21.15.181:0/250893160 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558809f600 con 0x7f558806df30 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.516+0000 7f5587fff640 1 -- 172.21.15.181:0/250893160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558806df30 msgr2=0x7f558806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.516+0000 7f5587fff640 1 --2- 172.21.15.181:0/250893160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558806df30 0x7f558806a6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f558806a3d0 tx=0x7f55780091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a7314b50914df8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.516+0000 7f5586ffd640 1 -- 172.21.15.181:0/250893160 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578032440 con 0x7f558806df30 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: dump: 2024-09-15T03:36:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:45.531 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.531 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.531 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.531 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:45.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 1 -- 172.21.15.181:0/250893160 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55780325a0 con 0x7f558806df30 2024-09-15T03:36:45.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 5 monclient: authenticate success, global_id 104341 2024-09-15T03:36:45.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:45.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:45.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 1 -- 172.21.15.181:0/250893160 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578032940 con 0x7f558806df30 2024-09-15T03:36:45.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 4 set_mon_vals no callback set 2024-09-15T03:36:45.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:45.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f5586ffd640 10 monclient: dump: 2024-09-15T03:36:45.533 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:45.533 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.535 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.535 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:45.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 10 monclient: shutdown 2024-09-15T03:36:45.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 -- 172.21.15.181:0/250893160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558806df30 msgr2=0x7f558806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 --2- 172.21.15.181:0/250893160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558806df30 0x7f558806a6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f558806a3d0 tx=0x7f55780091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 -- 172.21.15.181:0/250893160 shutdown_connections 2024-09-15T03:36:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 --2- 172.21.15.181:0/250893160 >> v2:172.21.15.153:3300/0 conn(0x7f558809e1e0 0x7f558809e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 --2- 172.21.15.181:0/250893160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558806df30 0x7f558806a6b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 -- 172.21.15.181:0/250893160 >> 172.21.15.181:0/250893160 conn(0x7f55880a3b30 msgr2=0x7f558809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 -- 172.21.15.181:0/250893160 shutdown_connections 2024-09-15T03:36:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.517+0000 7f558e694640 1 -- 172.21.15.181:0/250893160 wait complete. 2024-09-15T03:36:45.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient: monmap: 2024-09-15T03:36:45.537 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:45.537 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:45.537 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:45.537 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:45.538 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.538 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:45.538 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:45.538 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:45.538 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:45.538 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 1 Processor -- start 2024-09-15T03:36:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 1 -- start start 2024-09-15T03:36:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient: init 2024-09-15T03:36:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f558806df30 0x7f5588141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient(hunting): picked mon.noname-c con 0x7f558806df30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:45.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f558806e300 0x7f5588135010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.518+0000 7f558e694640 10 monclient(hunting): picked mon.noname-a con 0x7f558806e300 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:45.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 0x7f558811ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 10 monclient(hunting): picked mon.noname-b con 0x7f558809e1e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:45.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588120bf0 con 0x7f558806e300 2024-09-15T03:36:45.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588099ea0 con 0x7f558809e1e0 2024-09-15T03:36:45.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558806d820 con 0x7f558806df30 2024-09-15T03:36:45.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558e694640 10 monclient(hunting): authenticate will time out at 1210.000042s 2024-09-15T03:36:45.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 0x7f558811ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 0x7f558811ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44262/0 (socket says 172.21.15.181:44262) 2024-09-15T03:36:45.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 -- 172.21.15.181:0/901632860 learned_addr learned my addr 172.21.15.181:0/901632860 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:45.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): get_auth_request con 0x7f558809e1e0 auth_method 0 2024-09-15T03:36:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f558806df30 0x7f5588141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f558806e300 0x7f5588135010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): get_auth_request con 0x7f558806e300 auth_method 0 2024-09-15T03:36:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): get_auth_request con 0x7f558806df30 auth_method 0 2024-09-15T03:36:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f55877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f5587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient(hunting): handle_auth_done global_id 104344 payload 824 2024-09-15T03:36:45.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 -- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f558806df30 msgr2=0x7f5588141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f558806df30 0x7f5588141170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 -- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f558806e300 msgr2=0x7f5588135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f558806e300 0x7f5588135010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 monclient: found mon.noname-b 2024-09-15T03:36:45.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.519+0000 7f558cc31640 1 -- 172.21.15.181:0/901632860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5578009450 con 0x7f558809e1e0 2024-09-15T03:36:45.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558cc31640 1 -- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 msgr2=0x7f558811ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558cc31640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 0x7f558811ea90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5580009ec0 tx=0x7f558000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580014480 con 0x7f558809e1e0 2024-09-15T03:36:45.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:45.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: dump: 2024-09-15T03:36:45.549 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:45.549 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.550 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.550 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.550 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.550 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:45.551 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.551 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.551 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.551 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:45.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558000f040 con 0x7f558809e1e0 2024-09-15T03:36:45.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 5 monclient: authenticate success, global_id 104344 2024-09-15T03:36:45.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 10 monclient: _renew_subs 2024-09-15T03:36:45.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5588122130 con 0x7f558809e1e0 2024-09-15T03:36:45.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580014810 con 0x7f558809e1e0 2024-09-15T03:36:45.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:45.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f556ffff640 10 monclient: dump: 2024-09-15T03:36:45.553 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.555 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 10 monclient: _renew_subs 2024-09-15T03:36:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5588120670 con 0x7f558809e1e0 2024-09-15T03:36:45.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.521+0000 7f55867fc640 4 set_mon_vals no callback set 2024-09-15T03:36:45.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5580014bc0 con 0x7f558809e1e0 2024-09-15T03:36:45.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f558e694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f558e694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f558e694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558804bf60 con 0x7f558809e1e0 2024-09-15T03:36:45.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f556ffff640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5574038740 0x7f557403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(205..205 src has 106..205) v4 ==== 12565+0+0 (secure 0 0 0) 0x7f558004e4f0 con 0x7f558809e1e0 2024-09-15T03:36:45.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.522+0000 7f5587fff640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5574038740 0x7f557403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.523+0000 7f5587fff640 10 monclient: get_auth_request con 0x7f5574038740 auth_method 0 2024-09-15T03:36:45.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.525+0000 7f5587fff640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5574038740 0x7f557403abf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5588071980 tx=0x7f5578032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.527+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f558001a2c0 con 0x7f558809e1e0 2024-09-15T03:36:45.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.527+0000 7f556ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.527+0000 7f556ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:45.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.603+0000 7f9d33a8b640 10 monclient: get_monmap_and_config 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.603+0000 7f9d33a8b640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.603+0000 7f9d33a8b640 10 monclient: monmap: 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:45.604177+0000 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:45.604177+0000 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:45.605 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:45.606 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:45.606 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:45.606 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.603+0000 7f9d33a8b640 1 Processor -- start 2024-09-15T03:36:45.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 1 -- start start 2024-09-15T03:36:45.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient: init 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9d2c0974f0 0x7f9d2c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d2c0974f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d2c09a8a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9d2c133070 0x7f9d2c135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2c133070 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2c0978c0 con 0x7f9d2c09a8a0 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d2c09ca20 con 0x7f9d2c133070 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d2c098ad0 con 0x7f9d2c0974f0 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d33a8b640 10 monclient(hunting): authenticate will time out at 1203.243660s 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9d2c133070 0x7f9d2c135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9d2c133070 0x7f9d2c135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60570/0 (socket says 172.21.15.153:60570) 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 1 -- 172.21.15.153:0/3058218793 learned_addr learned my addr 172.21.15.153:0/3058218793 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): get_auth_request con 0x7f9d2c133070 auth_method 0 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d31827640 1 --2- 172.21.15.153:0/3058218793 >> v2:172.21.15.181:3300/0 conn(0x7f9d2c0974f0 0x7f9d2c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d31827640 10 monclient(hunting): get_auth_request con 0x7f9d2c0974f0 auth_method 0 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d31827640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d31827640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.604+0000 7f9d31827640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d31827640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d31827640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d31827640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 10 monclient(hunting): handle_auth_done global_id 114144 payload 274 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 1 -- 172.21.15.153:0/3058218793 >> v2:172.21.15.181:3300/0 conn(0x7f9d2c0974f0 msgr2=0x7f9d2c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 1 --2- 172.21.15.153:0/3058218793 >> v2:172.21.15.181:3300/0 conn(0x7f9d2c0974f0 0x7f9d2c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 1 -- 172.21.15.153:0/3058218793 >> v1:172.21.15.181:6789/0 conn(0x7f9d2c09a8a0 legacy=0x7f9d2c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 1 monclient: found mon.noname-a 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.605+0000 7f9d32028640 1 -- 172.21.15.153:0/3058218793 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c098db0 con 0x7f9d2c133070 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.607+0000 7f9d32028640 1 -- 172.21.15.153:0/3058218793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c133070 msgr2=0x7f9d2c135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.607+0000 7f9d32028640 1 --2- 172.21.15.153:0/3058218793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c133070 0x7f9d2c135450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c0768c0 tx=0x7f9d24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5641b4ce0e850c9f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 1 -- 172.21.15.153:0/3058218793 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24007610 con 0x7f9d2c133070 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: dump: 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 1 -- 172.21.15.153:0/3058218793 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2402d050 con 0x7f9d2c133070 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 5 monclient: authenticate success, global_id 114144 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:45.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 1 -- 172.21.15.153:0/3058218793 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240079a0 con 0x7f9d2c133070 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 4 set_mon_vals no callback set 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d30825640 10 monclient: dump: 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.615 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 10 monclient: shutdown 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 -- 172.21.15.153:0/3058218793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c133070 msgr2=0x7f9d2c135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/3058218793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c133070 0x7f9d2c135450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c0768c0 tx=0x7f9d24009840 comp rx=0 tx=0).stop 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 -- 172.21.15.153:0/3058218793 shutdown_connections 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/3058218793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c133070 0x7f9d2c135450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/3058218793 >> v2:172.21.15.181:3300/0 conn(0x7f9d2c0974f0 0x7f9d2c12ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 -- 172.21.15.153:0/3058218793 >> 172.21.15.153:0/3058218793 conn(0x7f9d2c0672b0 msgr2=0x7f9d2c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 -- 172.21.15.153:0/3058218793 shutdown_connections 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.608+0000 7f9d33a8b640 1 -- 172.21.15.153:0/3058218793 wait complete. 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.609+0000 7f9d33a8b640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.609+0000 7f9d33a8b640 10 monclient: monmap: 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.609+0000 7f9d33a8b640 1 Processor -- start 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.609+0000 7f9d33a8b640 1 -- start start 2024-09-15T03:36:45.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.609+0000 7f9d33a8b640 10 monclient: init 2024-09-15T03:36:45.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:45.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 0x7f9d2c1318c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2c0974f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:45.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d2c09a8a0 0x7f9d2c132180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d2c09a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d2c133070 0x7f9d2c1326c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d2c133070 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c06f040 con 0x7f9d2c0974f0 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c098300 con 0x7f9d2c133070 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c09df20 con 0x7f9d2c09a8a0 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d33a8b640 10 monclient(hunting): authenticate will time out at 1203.249126s 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 0x7f9d2c1318c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 0x7f9d2c1318c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60584/0 (socket says 172.21.15.153:60584) 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 -- 172.21.15.153:0/4115566483 learned_addr learned my addr 172.21.15.153:0/4115566483 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d2c09a8a0 0x7f9d2c132180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): get_auth_request con 0x7f9d2c0974f0 auth_method 0 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): get_auth_request con 0x7f9d2c09a8a0 auth_method 0 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d2c133070 0x7f9d2c1326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): get_auth_request con 0x7f9d2c133070 auth_method 0 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31026640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient(hunting): handle_auth_done global_id 114147 payload 824 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d2c09a8a0 msgr2=0x7f9d2c132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d2c09a8a0 0x7f9d2c132180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d2c133070 msgr2=0x7f9d2c1326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d2c133070 0x7f9d2c1326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 monclient: found mon.noname-a 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.610+0000 7f9d31827640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d24009ae0 con 0x7f9d2c0974f0 2024-09-15T03:36:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d31827640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 msgr2=0x7f9d2c1318c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d31827640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 0x7f9d2c1318c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e950 tx=0x7f9d2000a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000a700 con 0x7f9d2c0974f0 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: dump: 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2000cc50 con 0x7f9d2c0974f0 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d33a8b640 5 monclient: authenticate success, global_id 114147 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d33a8b640 10 monclient: _renew_subs 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2001a430 con 0x7f9d2c0974f0 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d33a8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2c1308d0 con 0x7f9d2c0974f0 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:45.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1dffb640 10 monclient: dump: 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.612+0000 7f9d1ffff640 4 set_mon_vals no callback set 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 10 monclient: _renew_subs 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2c06e840 con 0x7f9d2c0974f0 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d200113f0 con 0x7f9d2c0974f0 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d1dffb640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d10038740 0x7f9d1003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(205..205 src has 106..205) v4 ==== 12565+0+0 (secure 0 0 0) 0x7f9d20015030 con 0x7f9d2c0974f0 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 0x7f9d2c06d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.613+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c131030 con 0x7f9d2c06cd10 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 0x7f9d2c06d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d31026640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d10038740 0x7f9d1003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient: get_auth_request con 0x7f9d2c06cd10 auth_method 0 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d31026640 10 monclient: get_auth_request con 0x7f9d10038740 auth_method 0 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 10 monclient(hunting): handle_auth_done global_id 114150 payload 824 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.614+0000 7f9d32028640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d2c130b10 con 0x7f9d2c06cd10 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d32028640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 msgr2=0x7f9d2c06d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d32028640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 0x7f9d2c06d0e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d24033000 tx=0x7f9d24036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d31026640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d10038740 0x7f9d1003abf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9d28007fd0 tx=0x7f9d28007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24031aa0 con 0x7f9d2c06cd10 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 10 monclient: dump: 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9d24040070 con 0x7f9d2c06cd10 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 msgr2=0x7f9d2c06d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 0x7f9d2c06d0e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d24033000 tx=0x7f9d24036000 comp rx=0 tx=0).stop 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 msgr2=0x7f9d2c06d0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.616+0000 7f9d1dffb640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06cd10 0x7f9d2c06d0e0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d24033000 tx=0x7f9d24036000 comp rx=0 tx=0).stop 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d33a8b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 0x7f9d2c0696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d33a8b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c065f20 con 0x7f9d2c06d620 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 0x7f9d2c0696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient: get_auth_request con 0x7f9d2c06d620 auth_method 0 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 10 monclient(hunting): handle_auth_done global_id 114153 payload 824 2024-09-15T03:36:45.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.622+0000 7f9d32028640 1 -- 172.21.15.153:0/4115566483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d2c04bf60 con 0x7f9d2c06d620 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d32028640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 msgr2=0x7f9d2c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d32028640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 0x7f9d2c0696a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009880 tx=0x7f9d2403e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2403c810 con 0x7f9d2c06d620 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 10 monclient: dump: 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.637 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f9d2403c970 con 0x7f9d2c06d620 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 msgr2=0x7f9d2c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 0x7f9d2c0696a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009880 tx=0x7f9d2403e070 comp rx=0 tx=0).stop 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 msgr2=0x7f9d2c0696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:45.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d1dffb640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c06d620 0x7f9d2c0696a0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009880 tx=0x7f9d2403e070 comp rx=0 tx=0).stop 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 92, 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:45.639 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:36:45.640 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:36:45.641 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:45.642 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:36:45.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:45.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:45.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:36:45.646 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:36:45.648 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:36:45.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.650 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:36:45.651 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d10038740 msgr2=0x7f9d1003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d10038740 0x7f9d1003abf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9d28007fd0 tx=0x7f9d28007e10 comp rx=0 tx=0).stop 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 10 monclient: shutdown 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 msgr2=0x7f9d2c1318c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d32028640 1 -- 172.21.15.153:0/4115566483 reap_dead start 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 0x7f9d2c1318c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e950 tx=0x7f9d2000a2b0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 shutdown_connections 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 --2- 172.21.15.153:0/4115566483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d2c0974f0 0x7f9d2c1318c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.625+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 >> 172.21.15.153:0/4115566483 conn(0x7f9d2c0672b0 msgr2=0x7f9d2c129390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.626+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 shutdown_connections 2024-09-15T03:36:45.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.626+0000 7f9d33a8b640 1 -- 172.21.15.153:0/4115566483 wait complete. 2024-09-15T03:36:45.653 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:36:45.653 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:36:45.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:36:45.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.686+0000 7f558e694640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:45.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.686+0000 7f558e694640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:45.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.686+0000 7f558e694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:45.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:45.686+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5588068a20 con 0x7f558809e1e0 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.747+0000 7fa318b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.747+0000 7fa318b9d640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.747+0000 7fa318b9d640 10 monclient: monmap: 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:45.747886+0000 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:45.747886+0000 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:45.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.747+0000 7fa318b9d640 1 Processor -- start 2024-09-15T03:36:45.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.747+0000 7fa318b9d640 1 -- start start 2024-09-15T03:36:45.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.747+0000 7fa318b9d640 10 monclient: init 2024-09-15T03:36:45.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:36:45.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa3141318f0 0x7fa314131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3141318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa31409ea50 0x7fa31409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa31409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa314068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31409fbf0 con 0x7fa314068d90 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa314131d50 con 0x7fa3141318f0 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31409fe30 con 0x7fa31409ea50 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa318b9d640 10 monclient(hunting): authenticate will time out at 1203.387349s 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa3141318f0 0x7fa314131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa3141318f0 0x7fa314131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60598/0 (socket says 172.21.15.153:60598) 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 -- 172.21.15.153:0/2966818340 learned_addr learned my addr 172.21.15.153:0/2966818340 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): get_auth_request con 0x7fa3141318f0 auth_method 0 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 1 --2- 172.21.15.153:0/2966818340 >> v2:172.21.15.181:3300/0 conn(0x7fa31409ea50 0x7fa31409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): get_auth_request con 0x7fa31409ea50 auth_method 0 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient(hunting): handle_auth_done global_id 114156 payload 274 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 -- 172.21.15.153:0/2966818340 >> v2:172.21.15.181:3300/0 conn(0x7fa31409ea50 msgr2=0x7fa31409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 --2- 172.21.15.153:0/2966818340 >> v2:172.21.15.181:3300/0 conn(0x7fa31409ea50 0x7fa31409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 -- 172.21.15.153:0/2966818340 >> v1:172.21.15.181:6789/0 conn(0x7fa314068d90 legacy=0x7fa3140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 monclient: found mon.noname-a 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.748+0000 7fa31259c640 1 -- 172.21.15.153:0/2966818340 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140a1210 con 0x7fa3141318f0 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa31259c640 1 -- 172.21.15.153:0/2966818340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 msgr2=0x7fa314131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa31259c640 1 --2- 172.21.15.153:0/2966818340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314131f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa31406f600 tx=0x7fa3040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7687da8a481bcd5d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 1 -- 172.21.15.153:0/2966818340 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304032440 con 0x7fa3141318f0 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: dump: 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.761 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 1 -- 172.21.15.153:0/2966818340 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3040325a0 con 0x7fa3141318f0 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa318b9d640 5 monclient: authenticate success, global_id 114156 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa318b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 1 -- 172.21.15.153:0/2966818340 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304032940 con 0x7fa3141318f0 2024-09-15T03:36:45.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa311d9b640 10 monclient: dump: 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa318b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:45.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.751+0000 7fa318b9d640 4 set_mon_vals no callback set 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 10 monclient: shutdown 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 -- 172.21.15.153:0/2966818340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 msgr2=0x7fa314131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 --2- 172.21.15.153:0/2966818340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314131f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa31406f600 tx=0x7fa3040091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 -- 172.21.15.153:0/2966818340 shutdown_connections 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 --2- 172.21.15.153:0/2966818340 >> v2:172.21.15.181:3300/0 conn(0x7fa31409ea50 0x7fa31409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 --2- 172.21.15.153:0/2966818340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314131f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 -- 172.21.15.153:0/2966818340 >> 172.21.15.153:0/2966818340 conn(0x7fa314067110 msgr2=0x7fa314067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 -- 172.21.15.153:0/2966818340 shutdown_connections 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 1 -- 172.21.15.153:0/2966818340 wait complete. 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 10 monclient: build_initial_monmap 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.752+0000 7fa318b9d640 10 monclient: monmap: 2024-09-15T03:36:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 Processor -- start 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 -- start start 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient: init 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:45.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa314068d90 0x7fa31409a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa314068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31409ea50 0x7fa314130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa31409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3141318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3140a4420 con 0x7fa3141318f0 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31411e260 con 0x7fa314068d90 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31411dfb0 con 0x7fa31409ea50 2024-09-15T03:36:45.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa318b9d640 10 monclient(hunting): authenticate will time out at 1203.392660s 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31409ea50 0x7fa314130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314140df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60600/0 (socket says 172.21.15.153:60600) 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 learned_addr learned my addr 172.21.15.153:0/763975483 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): get_auth_request con 0x7fa3141318f0 auth_method 0 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): get_auth_request con 0x7fa31409ea50 auth_method 0 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa314068d90 0x7fa31409a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.753+0000 7fa312d9d640 10 monclient(hunting): handle_auth_done global_id 114159 payload 824 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31409ea50 msgr2=0x7fa314130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31409ea50 0x7fa314130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa314068d90 msgr2=0x7fa31409a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa314068d90 0x7fa31409a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 1 monclient: found mon.noname-a 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304009450 con 0x7fa3141318f0 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa31259c640 10 monclient: get_auth_request con 0x7fa314068d90 auth_method 0 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.754+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa314068d90 0x7fa31409a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 msgr2=0x7fa314140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa312d9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314140df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00c890 tx=0x7fa30c002640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00ce80 con 0x7fa3141318f0 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: dump: 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: _finish_auth 0 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 5 monclient: authenticate success, global_id 114159 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00aa00 con 0x7fa3141318f0 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 10 monclient: _renew_subs 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31411c7f0 con 0x7fa3141318f0 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00ae00 con 0x7fa3141318f0 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa30affd640 10 monclient: dump: 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.771 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa31159a640 4 set_mon_vals no callback set 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 10 monclient: _renew_subs 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:45.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.756+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31411bdd0 con 0x7fa3141318f0 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa30c012630 con 0x7fa3141318f0 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa318b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa318b9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 0x7fa314141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa314065e00 con 0x7fa31411cda0 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa30affd640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f4038770 0x7fa2f403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 0x7fa314141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient: get_auth_request con 0x7fa31411cda0 auth_method 0 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(205..205 src has 106..205) v4 ==== 12565+0+0 (secure 0 0 0) 0x7fa30c04dc30 con 0x7fa3141318f0 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 10 monclient(hunting): handle_auth_done global_id 74873 payload 824 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa31259c640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa31411e3f0 con 0x7fa31411cda0 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa309d9b640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f4038770 0x7fa2f403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.757+0000 7fa309d9b640 10 monclient: get_auth_request con 0x7fa2f4038770 auth_method 0 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.759+0000 7fa31259c640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 msgr2=0x7fa314141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.759+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 0x7fa314141330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa304034000 tx=0x7fa304032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304031a80 con 0x7fa31411cda0 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 10 monclient: dump: 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.774 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa309d9b640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f4038770 0x7fa2f403ac20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009d20 tx=0x7fa2fc0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa304031be0 con 0x7fa31411cda0 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 msgr2=0x7fa314141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 0x7fa314141330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa304034000 tx=0x7fa304032000 comp rx=0 tx=0).stop 2024-09-15T03:36:45.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 msgr2=0x7fa314141330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:45.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.760+0000 7fa30affd640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31411cda0 0x7fa314141330 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa318b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa318b9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa318b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31411e0f0 con 0x7fa3141310c0 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient: get_auth_request con 0x7fa3141310c0 auth_method 0 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 msgr2=0x7fa314141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 msgr2=0x7fa314141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:45.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.766+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:45.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:45.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient: get_auth_request con 0x7fa3141310c0 auth_method 0 2024-09-15T03:36:45.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:45.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 10 monclient(hunting): handle_auth_done global_id 74879 payload 824 2024-09-15T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.967+0000 7fa31259c640 1 -- 172.21.15.153:0/763975483 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa31404bf60 con 0x7fa3141310c0 2024-09-15T03:36:45.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.969+0000 7fa31259c640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 msgr2=0x7fa314141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.969+0000 7fa31259c640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa304006c70 tx=0x7fa30403b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.969+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3040468a0 con 0x7fa3141310c0 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.969+0000 7fa30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.969+0000 7fa30affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.969+0000 7fa30affd640 10 monclient: dump: 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:45.971 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fa304031870 con 0x7fa3141310c0 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 msgr2=0x7fa314141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa304006c70 tx=0x7fa30403b000 comp rx=0 tx=0).stop 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 msgr2=0x7fa314141870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa30affd640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3141310c0 0x7fa314141870 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa304006c70 tx=0x7fa30403b000 comp rx=0 tx=0).stop 2024-09-15T03:36:45.972 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 92, 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:36:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:36:45.975 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:45.976 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.977 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:36:45.978 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:45.979 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.980 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:36:45.981 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:45.982 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.983 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:36:45.984 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f4038770 msgr2=0x7fa2f403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa2f4038770 0x7fa2f403ac20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009d20 tx=0x7fa2fc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 10 monclient: shutdown 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 msgr2=0x7fa314140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 --2- 172.21.15.153:0/763975483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3141318f0 0x7fa314140df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00c890 tx=0x7fa30c002640 comp rx=0 tx=0).stop 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 reap_dead start 2024-09-15T03:36:45.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa312d9d640 1 -- 172.21.15.153:0/763975483 reap_dead start 2024-09-15T03:36:45.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 shutdown_connections 2024-09-15T03:36:45.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 >> 172.21.15.153:0/763975483 conn(0x7fa314067110 msgr2=0x7fa3141349b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:45.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 shutdown_connections 2024-09-15T03:36:45.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:45.970+0000 7fa318b9d640 1 -- 172.21.15.153:0/763975483 wait complete. 2024-09-15T03:36:45.987 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:36:45.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:36:45.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.081+0000 7f523206f640 10 monclient: get_monmap_and_config 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.081+0000 7f523206f640 10 monclient: build_initial_monmap 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.081+0000 7f523206f640 10 monclient: monmap: 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:46.082060+0000 2024-09-15T03:36:46.082 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:46.082060+0000 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:46.083 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 Processor -- start 2024-09-15T03:36:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 -- start start 2024-09-15T03:36:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient: init 2024-09-15T03:36:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T03:36:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f522c131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f522c09e910 0x7f522c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): picked mon.noname-a con 0x7f522c09e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f522c09f250 0x7f522c138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): picked mon.noname-c con 0x7f522c09f250 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c09faf0 con 0x7f522c131ab0 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c132590 con 0x7f522c09e910 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c09fd30 con 0x7f522c09f250 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f523206f640 10 monclient(hunting): authenticate will time out at 1203.721705s 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f522c09e910 0x7f522c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f522b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f522c131ab0 0x7f522c131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35918/0 (socket says 172.21.15.153:35918) 2024-09-15T03:36:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): get_auth_request con 0x7f522c09e910 auth_method 0 2024-09-15T03:36:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f522b7fe640 1 -- 172.21.15.153:0/590177681 learned_addr learned my addr 172.21.15.153:0/590177681 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.082+0000 7f5222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f522bfff640 1 --2- 172.21.15.153:0/590177681 >> v2:172.21.15.181:3300/0 conn(0x7f522c09f250 0x7f522c138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 10 monclient(hunting): handle_auth_done global_id 114162 payload 274 2024-09-15T03:36:46.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 1 -- 172.21.15.153:0/590177681 >> v2:172.21.15.181:3300/0 conn(0x7f522c09f250 msgr2=0x7f522c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:46.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 1 --2- 172.21.15.153:0/590177681 >> v2:172.21.15.181:3300/0 conn(0x7f522c09f250 0x7f522c138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 1 -- 172.21.15.153:0/590177681 >> v1:172.21.15.153:6789/0 conn(0x7f522c131ab0 legacy=0x7f522c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:46.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:36:46.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 1 monclient: found mon.noname-a 2024-09-15T03:36:46.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:46.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:46.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.083+0000 7f5222ffd640 1 -- 172.21.15.153:0/590177681 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c0a1110 con 0x7f522c09e910 2024-09-15T03:36:46.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.085+0000 7f5222ffd640 1 -- 172.21.15.153:0/590177681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c09e910 msgr2=0x7f522c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.085+0000 7f5222ffd640 1 --2- 172.21.15.153:0/590177681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c09e910 0x7f522c09ed10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f522c06a250 tx=0x7f52140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67a508c76968a31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 1 -- 172.21.15.153:0/590177681 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214032440 con 0x7f522c09e910 2024-09-15T03:36:46.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: dump: 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: _finish_auth 0 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:46.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 5 monclient: authenticate success, global_id 114162 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 1 -- 172.21.15.153:0/590177681 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52140325a0 con 0x7f522c09e910 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 4 set_mon_vals no callback set 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 1 -- 172.21.15.153:0/590177681 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214032980 con 0x7f522c09e910 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f522affd640 10 monclient: dump: 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.102 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 10 monclient: shutdown 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 -- 172.21.15.153:0/590177681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c09e910 msgr2=0x7f522c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 --2- 172.21.15.153:0/590177681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c09e910 0x7f522c09ed10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f522c06a250 tx=0x7f52140090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 -- 172.21.15.153:0/590177681 shutdown_connections 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 --2- 172.21.15.153:0/590177681 >> v2:172.21.15.181:3300/0 conn(0x7f522c09f250 0x7f522c138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 --2- 172.21.15.153:0/590177681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c09e910 0x7f522c09ed10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 -- 172.21.15.153:0/590177681 >> 172.21.15.153:0/590177681 conn(0x7f522c067d30 msgr2=0x7f522c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 -- 172.21.15.153:0/590177681 shutdown_connections 2024-09-15T03:36:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.086+0000 7f523206f640 1 -- 172.21.15.153:0/590177681 wait complete. 2024-09-15T03:36:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 10 monclient: build_initial_monmap 2024-09-15T03:36:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 10 monclient: monmap: 2024-09-15T03:36:46.104 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 1 Processor -- start 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 1 -- start start 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 10 monclient: init 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c09e910 0x7f522c138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.087+0000 7f523206f640 10 monclient(hunting): picked mon.noname-b con 0x7f522c09e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f522c09f250 0x7f522c133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): picked mon.noname-c con 0x7f522c09f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c131ab0 0x7f522c134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): picked mon.noname-a con 0x7f522c131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c137680 con 0x7f522c131ab0 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c135240 con 0x7f522c09e910 2024-09-15T03:36:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c135dc0 con 0x7f522c09f250 2024-09-15T03:36:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f523206f640 10 monclient(hunting): authenticate will time out at 1203.727046s 2024-09-15T03:36:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c131ab0 0x7f522c134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c131ab0 0x7f522c134330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60626/0 (socket says 172.21.15.153:60626) 2024-09-15T03:36:46.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f522c09f250 0x7f522c133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 1 -- 172.21.15.153:0/3204232545 learned_addr learned my addr 172.21.15.153:0/3204232545 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:46.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): get_auth_request con 0x7f522c131ab0 auth_method 0 2024-09-15T03:36:46.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): get_auth_request con 0x7f522c09f250 auth_method 0 2024-09-15T03:36:46.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c09e910 0x7f522c138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): get_auth_request con 0x7f522c09e910 auth_method 0 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient(hunting): handle_auth_done global_id 74882 payload 824 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c09e910 msgr2=0x7f522c138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c09e910 0x7f522c138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c131ab0 msgr2=0x7f522c134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c131ab0 0x7f522c134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 monclient: found mon.noname-c 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f5222ffd640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214009390 con 0x7f522c09f250 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.088+0000 7f522bfff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f522c131ab0 0x7f522c134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5222ffd640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f522c09f250 msgr2=0x7f522c133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5222ffd640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f522c09f250 0x7f522c133df0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5214031000 tx=0x7f5214032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214031c70 con 0x7f522c09f250 2024-09-15T03:36:46.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: dump: 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:46.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214031df0 con 0x7f522c09f250 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 5 monclient: authenticate success, global_id 74882 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 10 monclient: _renew_subs 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c1373d0 con 0x7f522c09f250 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521403d670 con 0x7f522c09f250 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f5223fff640 10 monclient: dump: 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.113 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f522a7fc640 4 set_mon_vals no callback set 2024-09-15T03:36:46.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 10 monclient: _renew_subs 2024-09-15T03:36:46.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.090+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c136620 con 0x7f522c09f250 2024-09-15T03:36:46.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f523206f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5214039040 con 0x7f522c09f250 2024-09-15T03:36:46.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f523206f640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 0x7f522c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c063740 con 0x7f522c138570 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f5223fff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f520c038740 0x7f520c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.091+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(205..205 src has 106..205) v4 ==== 12565+0+0 (secure 0 0 0) 0x7f521406d8c0 con 0x7f522c09f250 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 0x7f522c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient: get_auth_request con 0x7f522c138570 auth_method 0 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522bfff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f520c038740 0x7f520c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522bfff640 10 monclient: get_auth_request con 0x7f520c038740 auth_method 0 2024-09-15T03:36:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 10 monclient(hunting): handle_auth_done global_id 104359 payload 824 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.092+0000 7f522b7fe640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f522c04bf60 con 0x7f522c138570 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f522b7fe640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 msgr2=0x7f522c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f522b7fe640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 0x7f522c02aa40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f521c00e710 tx=0x7f521c007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f522bfff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f520c038740 0x7f520c03abf0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f522400da60 tx=0x7f5224007870 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c004b50 con 0x7f522c138570 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f5223fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f5223fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.094+0000 7f5223fff640 10 monclient: dump: 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f521c004cb0 con 0x7f522c138570 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 msgr2=0x7f522c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 0x7f522c02aa40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f521c00e710 tx=0x7f521c007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 msgr2=0x7f522c02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.095+0000 7f5223fff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c138570 0x7f522c02aa40 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f521c00e710 tx=0x7f521c007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:46.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f523206f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:36:46.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f523206f640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 0x7f522c05e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f523206f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c134c70 con 0x7f522c134870 2024-09-15T03:36:46.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f522b7fe640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 0x7f522c05e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f522b7fe640 10 monclient: get_auth_request con 0x7f522c134870 auth_method 0 2024-09-15T03:36:46.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f522b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.100+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.101+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.101+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.101+0000 7f522b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.101+0000 7f522b7fe640 10 monclient(hunting): handle_auth_done global_id 104362 payload 824 2024-09-15T03:36:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.101+0000 7f522b7fe640 1 -- 172.21.15.153:0/3204232545 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f522c1377f0 con 0x7f522c134870 2024-09-15T03:36:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f522b7fe640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 msgr2=0x7f522c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f522b7fe640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 0x7f522c05e310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f521c00e710 tx=0x7f521c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c015d30 con 0x7f522c134870 2024-09-15T03:36:46.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:46.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 10 monclient: dump: 2024-09-15T03:36:46.125 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.126 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.126 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.126 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.126 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.126 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f521c015e90 con 0x7f522c134870 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:46.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 msgr2=0x7f522c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 0x7f522c05e310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f521c00e710 tx=0x7f521c019000 comp rx=0 tx=0).stop 2024-09-15T03:36:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 msgr2=0x7f522c05e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.103+0000 7f5223fff640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f522c134870 0x7f522c05e310 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f521c00e710 tx=0x7f521c019000 comp rx=0 tx=0).stop 2024-09-15T03:36:46.128 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:36:46.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:46.129 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:46.129 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:36:46.129 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 92, 2024-09-15T03:36:46.130 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:36:46.130 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:36:46.130 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:36:46.130 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:36:46.130 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:46.131 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:36:46.132 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:36:46.133 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:46.134 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.135 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:46.136 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:46.137 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:36:46.138 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:46.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:46.140 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:46.141 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:36:46.142 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f520c038740 msgr2=0x7f520c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f520c038740 0x7f520c03abf0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f522400da60 tx=0x7f5224007870 comp rx=0 tx=0).stop 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 10 monclient: shutdown 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f522c09f250 msgr2=0x7f522c133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 --2- 172.21.15.153:0/3204232545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f522c09f250 0x7f522c133df0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5214031000 tx=0x7f5214032000 comp rx=0 tx=0).stop 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f522bfff640 1 -- 172.21.15.153:0/3204232545 reap_dead start 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f522bfff640 1 -- 172.21.15.153:0/3204232545 reap_dead start 2024-09-15T03:36:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 shutdown_connections 2024-09-15T03:36:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 >> 172.21.15.153:0/3204232545 conn(0x7f522c067d30 msgr2=0x7f522c138940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 shutdown_connections 2024-09-15T03:36:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.104+0000 7f523206f640 1 -- 172.21.15.153:0/3204232545 wait complete. 2024-09-15T03:36:46.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:36:46.145 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:36:46.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:36:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.236+0000 7f5256a6f640 10 monclient: get_monmap_and_config 2024-09-15T03:36:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.236+0000 7f5256a6f640 10 monclient: build_initial_monmap 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.236+0000 7f5256a6f640 10 monclient: monmap: 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:46.237679+0000 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:46.237679+0000 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:46.238 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.237+0000 7f5256a6f640 1 Processor -- start 2024-09-15T03:36:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.237+0000 7f5256a6f640 1 -- start start 2024-09-15T03:36:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.237+0000 7f5256a6f640 10 monclient: init 2024-09-15T03:36:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f525009ad90 0x7f525009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f525009ad90 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f525009b160 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f5250132070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f525009b530 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500692c0 con 0x7f525009b160 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525009d890 con 0x7f525009ad90 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525009b990 con 0x7f525009b530 2024-09-15T03:36:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f5256a6f640 10 monclient(hunting): authenticate will time out at 1203.877254s 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f5250132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f5250132070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45448/0 (socket says 172.21.15.153:45448) 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 -- 172.21.15.153:0/3461015230 learned_addr learned my addr 172.21.15.153:0/3461015230 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): get_auth_request con 0x7f525009b530 auth_method 0 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 1 --2- 172.21.15.153:0/3461015230 >> v2:172.21.15.181:3300/0 conn(0x7f525009ad90 0x7f525009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): get_auth_request con 0x7f525009ad90 auth_method 0 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525480b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient(hunting): handle_auth_done global_id 74885 payload 274 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 -- 172.21.15.153:0/3461015230 >> v2:172.21.15.181:3300/0 conn(0x7f525009ad90 msgr2=0x7f525009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 --2- 172.21.15.153:0/3461015230 >> v2:172.21.15.181:3300/0 conn(0x7f525009ad90 0x7f525009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 -- 172.21.15.153:0/3461015230 >> v1:172.21.15.181:6789/0 conn(0x7f525009b160 legacy=0x7f5250131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 monclient: found mon.noname-b 2024-09-15T03:36:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.238+0000 7f525500c640 1 -- 172.21.15.153:0/3461015230 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525009db70 con 0x7f525009b530 2024-09-15T03:36:46.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f525500c640 1 -- 172.21.15.153:0/3461015230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 msgr2=0x7f5250132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f525500c640 1 --2- 172.21.15.153:0/3461015230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f5250132070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5248009910 tx=0x7f52480091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44618d382f77c6ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 1 -- 172.21.15.153:0/3461015230 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524802aea0 con 0x7f525009b530 2024-09-15T03:36:46.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:46.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: dump: 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f5256a6f640 5 monclient: authenticate success, global_id 74885 2024-09-15T03:36:46.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f5256a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 1 -- 172.21.15.153:0/3461015230 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5248032440 con 0x7f525009b530 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 1 -- 172.21.15.153:0/3461015230 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480327e0 con 0x7f525009b530 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f5256a6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f5256a6f640 4 set_mon_vals no callback set 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.241+0000 7f524f7fe640 10 monclient: dump: 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.255 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 10 monclient: shutdown 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 -- 172.21.15.153:0/3461015230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 msgr2=0x7f5250132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 --2- 172.21.15.153:0/3461015230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f5250132070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5248009910 tx=0x7f52480091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 -- 172.21.15.153:0/3461015230 shutdown_connections 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 --2- 172.21.15.153:0/3461015230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f5250132070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 --2- 172.21.15.153:0/3461015230 >> v2:172.21.15.181:3300/0 conn(0x7f525009ad90 0x7f525009c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 -- 172.21.15.153:0/3461015230 >> 172.21.15.153:0/3461015230 conn(0x7f52500678a0 msgr2=0x7f52500a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 -- 172.21.15.153:0/3461015230 shutdown_connections 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 -- 172.21.15.153:0/3461015230 wait complete. 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 10 monclient: build_initial_monmap 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 10 monclient: monmap: 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:46.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.242+0000 7f5256a6f640 1 Processor -- start 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 -- start start 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient: init 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f525009ad90 0x7f52500947d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): picked mon.noname-a con 0x7f525009ad90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525009b160 0x7f525008d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f525009b160 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f525008dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f525009b530 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250090d80 con 0x7f525009ad90 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525008efb0 con 0x7f525009b160 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5250090aa0 con 0x7f525009b530 2024-09-15T03:36:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f5256a6f640 10 monclient(hunting): authenticate will time out at 1203.882545s 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f525008dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f525009ad90 0x7f52500947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f525008dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45460/0 (socket says 172.21.15.153:45460) 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f525009ad90 0x7f52500947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60632/0 (socket says 172.21.15.153:60632) 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 -- 172.21.15.153:0/1496430547 learned_addr learned my addr 172.21.15.153:0/1496430547 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): get_auth_request con 0x7f525009b530 auth_method 0 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): get_auth_request con 0x7f525009ad90 auth_method 0 2024-09-15T03:36:46.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525480b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f524ffff640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525009b160 0x7f525008d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient(hunting): handle_auth_done global_id 74888 payload 824 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 -- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525009b160 msgr2=0x7f525008d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525009b160 0x7f525008d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 -- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f525009ad90 msgr2=0x7f52500947d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f525009ad90 0x7f52500947d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:46.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 monclient: found mon.noname-c 2024-09-15T03:36:46.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.243+0000 7f525500c640 1 -- 172.21.15.153:0/1496430547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248009450 con 0x7f525009b530 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f525500c640 1 -- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 msgr2=0x7f525008dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f525500c640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f525008dad0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5248009210 tx=0x7f5248032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524802aea0 con 0x7f525009b530 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: dump: 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: _finish_auth 0 2024-09-15T03:36:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5248031cf0 con 0x7f525009b530 2024-09-15T03:36:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 5 monclient: authenticate success, global_id 74888 2024-09-15T03:36:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 10 monclient: _renew_subs 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5250091060 con 0x7f525009b530 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524803d650 con 0x7f525009b530 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524cff9640 10 monclient: dump: 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:46.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 10 monclient: _renew_subs 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5250094d10 con 0x7f525009b530 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.246+0000 7f524effd640 4 set_mon_vals no callback set 2024-09-15T03:36:46.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5248039040 con 0x7f525009b530 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f5256a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f5256a6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f5256a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525004bf60 con 0x7f525009b530 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f524cff9640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f523c038740 0x7f523c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(205..205 src has 106..205) v4 ==== 12565+0+0 (secure 0 0 0) 0x7f524806d6e0 con 0x7f525009b530 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f525480b640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f523c038740 0x7f523c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.247+0000 7f525480b640 10 monclient: get_auth_request con 0x7f523c038740 auth_method 0 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.250+0000 7f525480b640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f523c038740 0x7f523c03abf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5240009c80 tx=0x7f52400090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.251+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5248031e60 con 0x7f525009b530 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.251+0000 7f524cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.251+0000 7f524cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:46.336 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.336+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v206) v1 ==== 168+0+0 (secure 0 0 0) 0x7f5580018020 con 0x7f558809e1e0 2024-09-15T03:36:46.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.336+0000 7f556ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:46.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.336+0000 7f556ffff640 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-15T03:36:46.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.431+0000 7f558e694640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:46.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.431+0000 7f558e694640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:46.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.431+0000 7f558e694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.431+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5588121240 con 0x7f558809e1e0 2024-09-15T03:36:46.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.432+0000 7f556ffff640 1 -- 172.21.15.181:0/901632860 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v206) v1 ==== 147+0+0 (secure 0 0 0) 0x7f5580018020 con 0x7f558809e1e0 2024-09-15T03:36:46.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.432+0000 7f556ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:46.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.432+0000 7f556ffff640 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-15T03:36:46.434 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h22' already exists 2024-09-15T03:36:46.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5574038740 msgr2=0x7f557403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5574038740 0x7f557403abf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5588071980 tx=0x7f5578032000 comp rx=0 tx=0).stop 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 10 monclient: shutdown 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 msgr2=0x7f558811ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 0x7f558811ea90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5580009ec0 tx=0x7f558000b040 comp rx=0 tx=0).stop 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 shutdown_connections 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5574038740 0x7f557403abf0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f558809e1e0 0x7f558811ea90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f558806e300 0x7f5588135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 --2- 172.21.15.181:0/901632860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f558806df30 0x7f5588141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 >> 172.21.15.181:0/901632860 conn(0x7f55880a3b30 msgr2=0x7f558806bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:46.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 shutdown_connections 2024-09-15T03:36:46.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.434+0000 7f558e694640 1 -- 172.21.15.181:0/901632860 wait complete. 2024-09-15T03:36:46.444 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:46.444 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-15T03:36:46.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.469+0000 7f5256a6f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:36:46.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.469+0000 7f5256a6f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:36:46.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.469+0000 7f5256a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:36:46.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.469+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f5250099c60 con 0x7f525009b530 2024-09-15T03:36:46.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=8,logm=60} crc {auth=1981543994,config=3140514009,health=410075256,kv=3235807718,logm=2136067787}) 2024-09-15T03:36:46.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=8,logm=60} crc {auth=1981543994,config=3140514009,health=70831016,kv=3235807718,logm=158092505}) 2024-09-15T03:36:46.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=8,logm=60} crc {auth=1981543994,config=3140514009,health=410075256,kv=3235807718,logm=2136067787}) 2024-09-15T03:36:46.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f524cff9640 1 -- 172.21.15.153:0/1496430547 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f5248042070 con 0x7f525009b530 2024-09-15T03:36:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f524cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:36:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f524cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.471+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=8,logm=60} crc {auth=1981543994,config=3140514009,health=410075256,kv=3235807718,logm=4091118574}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.472+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.472+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376332667}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.472+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021800494}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.472+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.472+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376332667}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.472+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828478835}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286397004}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1024265060}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286397004}) 2024-09-15T03:36:46.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=505589196}) 2024-09-15T03:36:46.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219032696}) 2024-09-15T03:36:46.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033566719}) 2024-09-15T03:36:46.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f523c038740 msgr2=0x7f523c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f523c038740 0x7f523c03abf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5240009c80 tx=0x7f52400090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 10 monclient: shutdown 2024-09-15T03:36:46.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219032696}) 2024-09-15T03:36:46.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202265783}) 2024-09-15T03:36:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 msgr2=0x7f525008dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f525008dad0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5248009210 tx=0x7f5248032000 comp rx=0 tx=0).stop 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 shutdown_connections 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f523c038740 0x7f523c03abf0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f525009b530 0x7f525008dad0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525009b160 0x7f525008d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 --2- 172.21.15.153:0/1496430547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f525009ad90 0x7f52500947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 >> 172.21.15.153:0/1496430547 conn(0x7f52500678a0 msgr2=0x7f5250097440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.473+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 shutdown_connections 2024-09-15T03:36:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5256a6f640 1 -- 172.21.15.153:0/1496430547 wait complete. 2024-09-15T03:36:46.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044305769}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170704825}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044305769}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.474+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3442868058}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.475+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.475+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=997979074}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.475+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940327046}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.475+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.475+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=997979074}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.475+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321465674}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214109801}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787015052}) 2024-09-15T03:36:46.486 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214109801}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716101982}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4189442377}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680543959}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4189442377}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.476+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077490175}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149747541}) 2024-09-15T03:36:46.487 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566657484}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149747541}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064525200}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166541814}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059835051}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166541814}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.477+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447272020}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3435637776}) 2024-09-15T03:36:46.488 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504240074}) 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3435637776}) 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2310541349}) 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698203970}) 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363559413}) 2024-09-15T03:36:46.489 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698203970}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.478+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766670326}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.479+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.479+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3416658670}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.479+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=207572951}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.479+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.479+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3416658670}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.479+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021175703}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495398044}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86818292}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495398044}) 2024-09-15T03:36:46.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103284575}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806487394}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561921265}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806487394}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.480+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6667367}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114392811}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809472569}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114392811}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122827363}) 2024-09-15T03:36:46.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740442453}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=706525546}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740442453}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.481+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992292467}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167274270}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515816434}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167274270}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047285363}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245047502}) 2024-09-15T03:36:46.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386338960}) 2024-09-15T03:36:46.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.482+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:46.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.483+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245047502}) 2024-09-15T03:36:46.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:46.483+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2274490256}) 2024-09-15T03:36:46.494 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.519+0000 7fdc8b660640 10 monclient: get_monmap_and_config 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.519+0000 7fdc8b660640 10 monclient: build_initial_monmap 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.519+0000 7fdc8b660640 10 monclient: monmap: 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:46.520599+0000 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:46.520599+0000 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:46.521 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.520+0000 7fdc8b660640 1 Processor -- start 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.520+0000 7fdc8b660640 1 -- start start 2024-09-15T03:36:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.520+0000 7fdc8b660640 10 monclient: init 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdc84099e10 0x7fdc8409e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): picked mon.noname-c con 0x7fdc84099e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc8409dfa0 0x7fdc8409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): picked mon.noname-b con 0x7fdc8409dfa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc8409eb90 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:46.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8409f620 con 0x7fdc8409eb90 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8409b7c0 con 0x7fdc84099e10 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8409f860 con 0x7fdc8409dfa0 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc8b660640 10 monclient(hunting): authenticate will time out at 1211.002092s 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdc84099e10 0x7fdc8409e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdc84099e10 0x7fdc8409e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44266/0 (socket says 172.21.15.181:44266) 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 -- 172.21.15.181:0/2291479261 learned_addr learned my addr 172.21.15.181:0/2291479261 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): get_auth_request con 0x7fdc84099e10 auth_method 0 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 1 --2- 172.21.15.181:0/2291479261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc8409dfa0 0x7fdc8409c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): get_auth_request con 0x7fdc8409dfa0 auth_method 0 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc88bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient(hunting): handle_auth_done global_id 104368 payload 274 2024-09-15T03:36:46.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 -- 172.21.15.181:0/2291479261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc8409dfa0 msgr2=0x7fdc8409c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 --2- 172.21.15.181:0/2291479261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc8409dfa0 0x7fdc8409c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 -- 172.21.15.181:0/2291479261 >> v1:172.21.15.153:6789/0 conn(0x7fdc8409eb90 legacy=0x7fdc8409c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 monclient: found mon.noname-c 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.521+0000 7fdc893fc640 1 -- 172.21.15.181:0/2291479261 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8409b4e0 con 0x7fdc84099e10 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc893fc640 1 -- 172.21.15.181:0/2291479261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84099e10 msgr2=0x7fdc8409e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc893fc640 1 --2- 172.21.15.181:0/2291479261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84099e10 0x7fdc8409e650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009910 tx=0x7fdc7c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a1985de7ae3e704 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 1 -- 172.21.15.181:0/2291479261 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c032440 con 0x7fdc84099e10 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:46.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: dump: 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 5 monclient: authenticate success, global_id 104368 2024-09-15T03:36:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 1 -- 172.21.15.181:0/2291479261 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c0325a0 con 0x7fdc84099e10 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 4 set_mon_vals no callback set 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 1 -- 172.21.15.181:0/2291479261 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c032940 con 0x7fdc84099e10 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc83fff640 10 monclient: dump: 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 10 monclient: shutdown 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 1 -- 172.21.15.181:0/2291479261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84099e10 msgr2=0x7fdc8409e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.524+0000 7fdc8b660640 1 --2- 172.21.15.181:0/2291479261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84099e10 0x7fdc8409e650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009910 tx=0x7fdc7c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 -- 172.21.15.181:0/2291479261 shutdown_connections 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 --2- 172.21.15.181:0/2291479261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc8409dfa0 0x7fdc8409c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 --2- 172.21.15.181:0/2291479261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84099e10 0x7fdc8409e650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 -- 172.21.15.181:0/2291479261 >> 172.21.15.181:0/2291479261 conn(0x7fdc840a3b30 msgr2=0x7fdc8409fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 -- 172.21.15.181:0/2291479261 shutdown_connections 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 -- 172.21.15.181:0/2291479261 wait complete. 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 10 monclient: build_initial_monmap 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 10 monclient: monmap: 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 Processor -- start 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 1 -- start start 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.525+0000 7fdc8b660640 10 monclient: init 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc84099e10 0x7fdc8413f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): picked mon.noname-c con 0x7fdc84099e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc8409eb90 0x7fdc8412fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): picked mon.noname-a con 0x7fdc8409eb90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 0x7fdc84119830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): picked mon.noname-b con 0x7fdc84130230 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8406eb80 con 0x7fdc8409eb90 2024-09-15T03:36:46.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8413ff50 con 0x7fdc84130230 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc84098d20 con 0x7fdc84099e10 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc8b660640 10 monclient(hunting): authenticate will time out at 1211.007542s 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 0x7fdc84119830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 0x7fdc84119830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44270/0 (socket says 172.21.15.181:44270) 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 -- 172.21.15.181:0/1717838731 learned_addr learned my addr 172.21.15.181:0/1717838731 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): get_auth_request con 0x7fdc84130230 auth_method 0 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc84099e10 0x7fdc8413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): get_auth_request con 0x7fdc84099e10 auth_method 0 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc8409eb90 0x7fdc8412fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): get_auth_request con 0x7fdc8409eb90 auth_method 0 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc893fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc88bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient(hunting): handle_auth_done global_id 104371 payload 824 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 -- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc84099e10 msgr2=0x7fdc8413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc84099e10 0x7fdc8413f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 -- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc8409eb90 msgr2=0x7fdc8412fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc8409eb90 0x7fdc8412fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient: _finish_hunting 0 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 monclient: found mon.noname-b 2024-09-15T03:36:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:46.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.526+0000 7fdc89bfd640 1 -- 172.21.15.181:0/1717838731 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c009450 con 0x7fdc84130230 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc89bfd640 1 -- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 msgr2=0x7fdc84119830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc89bfd640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 0x7fdc84119830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400c890 tx=0x7fdc74002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7400ce80 con 0x7fdc84130230 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: dump: 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.544 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: _finish_auth 0 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400aa00 con 0x7fdc84130230 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 5 monclient: authenticate success, global_id 104371 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:46.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 10 monclient: _renew_subs 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7400adc0 con 0x7fdc84130230 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8411ce10 con 0x7fdc84130230 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc817fa640 10 monclient: dump: 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc837fe640 4 set_mon_vals no callback set 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 10 monclient: _renew_subs 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.529+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8411beb0 con 0x7fdc84130230 2024-09-15T03:36:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdc74012630 con 0x7fdc84130230 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc817fa640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc700387a0 0x7fdc7003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc8b660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc8b660640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(206..206 src has 106..206) v4 ==== 12778+0+0 (secure 0 0 0) 0x7fdc7404dd00 con 0x7fdc84130230 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc8b660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8404bf60 con 0x7fdc84130230 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc893fc640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc700387a0 0x7fdc7003ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.530+0000 7fdc893fc640 10 monclient: get_auth_request con 0x7fdc700387a0 auth_method 0 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.532+0000 7fdc893fc640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc700387a0 0x7fdc7003ac50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdc8409a7e0 tx=0x7fdc7c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.535+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdc74016070 con 0x7fdc84130230 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.535+0000 7fdc817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.535+0000 7fdc817fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:46.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.692+0000 7fdc8b660640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:46.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.692+0000 7fdc8b660640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:46.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.692+0000 7fdc8b660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:46.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:46.692+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fdc8409c160 con 0x7fdc84130230 2024-09-15T03:36:47.176 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.175+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.177 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.175+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740614147}) 2024-09-15T03:36:47.177 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.175+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1532204054}) 2024-09-15T03:36:47.177 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.175+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.177 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.175+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740614147}) 2024-09-15T03:36:47.177 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.175+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631323341}) 2024-09-15T03:36:47.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.333+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.333+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047667554}) 2024-09-15T03:36:47.335 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.333+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2233159209}) 2024-09-15T03:36:47.335 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.333+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.335 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.333+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047667554}) 2024-09-15T03:36:47.335 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.333+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383099301}) 2024-09-15T03:36:47.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.337+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v207) v1 ==== 168+0+0 (secure 0 0 0) 0x7fdc74022770 con 0x7fdc84130230 2024-09-15T03:36:47.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.337+0000 7fdc817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:47.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.337+0000 7fdc817fa640 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-15T03:36:47.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.337+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.337+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2548259107}) 2024-09-15T03:36:47.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.337+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364289221}) 2024-09-15T03:36:47.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.337+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.337+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2548259107}) 2024-09-15T03:36:47.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.337+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245234111}) 2024-09-15T03:36:47.345 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.344+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.345 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.344+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062644722}) 2024-09-15T03:36:47.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.344+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549787030}) 2024-09-15T03:36:47.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.344+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.344+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062644722}) 2024-09-15T03:36:47.346 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.344+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810969118}) 2024-09-15T03:36:47.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.426+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.426+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511165871}) 2024-09-15T03:36:47.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.426+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546328333}) 2024-09-15T03:36:47.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.426+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.426+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511165871}) 2024-09-15T03:36:47.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.426+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2337711238}) 2024-09-15T03:36:47.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.427+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.427+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835637750}) 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.427+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3814638124}) 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.427+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.427+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835637750}) 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.427+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440637853}) 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879709277}) 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2498445336}) 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879709277}) 2024-09-15T03:36:47.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850691257}) 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812283617}) 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318627856}) 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812283617}) 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.428+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1340613848}) 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.429+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.429+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161302426}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.429+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775915019}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.429+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.429+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161302426}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.429+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112653071}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.430+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.430+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269448310}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.430+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1289557205}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.430+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.430+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269448310}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.430+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327554291}) 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.431+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.431+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824684313}) 2024-09-15T03:36:47.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.431+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3467525163}) 2024-09-15T03:36:47.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.431+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.431+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824684313}) 2024-09-15T03:36:47.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.431+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740045418}) 2024-09-15T03:36:47.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531361250}) 2024-09-15T03:36:47.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624659144}) 2024-09-15T03:36:47.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531361250}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680573245}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156554226}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1178517612}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.432+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156554226}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641821639}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714025700}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733532958}) 2024-09-15T03:36:47.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714025700}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.433+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251037397}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.434+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.434+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734339094}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.434+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472805219}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.434+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.434+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734339094}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.434+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231890008}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.435+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.435+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204055207}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.435+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=604768260}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.435+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.435+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204055207}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.435+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=800952571}) 2024-09-15T03:36:47.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861327820}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390107640}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861327820}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353615208}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772117666}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1601236964,mdsmap=1531151339,mgr=574286301}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772117666}) 2024-09-15T03:36:47.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.436+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3457942793}) 2024-09-15T03:36:47.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.438+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.438+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {logm=3967100284,mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2227287156,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2184309409,osd_snap=454132730,osdmap=661395336}) 2024-09-15T03:36:47.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.438+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {logm=3967100284,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2227287156,mon_config_key=3725707551,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2184309409,osd_snap=454132730,osdmap=1546789995}) 2024-09-15T03:36:47.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.438+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.438+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {logm=3967100284,mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2227287156,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=2184309409,osd_snap=454132730,osdmap=661395336}) 2024-09-15T03:36:47.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.438+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=60} crc {logm=3967100284,mdsmap=1531151339,mgr=3242484981,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=54008791,mon_config_key=4117641549,monmap=244636553,osd_metadata=4072215849,osd_pg_creating=2184309409,osd_snap=454132730,osdmap=4208038822}) 2024-09-15T03:36:47.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.437+0000 7fdc8b660640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:47.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.437+0000 7fdc8b660640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:47.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.437+0000 7fdc8b660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:47.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.437+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fdc84064a90 con 0x7fdc84130230 2024-09-15T03:36:47.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.438+0000 7fdc817fa640 1 -- 172.21.15.181:0/1717838731 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v207) v1 ==== 147+0+0 (secure 0 0 0) 0x7fdc7401c030 con 0x7fdc84130230 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.438+0000 7fdc817fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.438+0000 7fdc817fa640 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h23' already exists 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc700387a0 msgr2=0x7fdc7003ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc700387a0 0x7fdc7003ac50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdc8409a7e0 tx=0x7fdc7c032000 comp rx=0 tx=0).stop 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 10 monclient: shutdown 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 msgr2=0x7fdc84119830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 0x7fdc84119830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400c890 tx=0x7fdc74002640 comp rx=0 tx=0).stop 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 shutdown_connections 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc700387a0 0x7fdc7003ac50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc84130230 0x7fdc84119830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc8409eb90 0x7fdc8412fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 --2- 172.21.15.181:0/1717838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc84099e10 0x7fdc8413f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 >> 172.21.15.181:0/1717838731 conn(0x7fdc840a3b30 msgr2=0x7fdc84132a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:47.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 shutdown_connections 2024-09-15T03:36:47.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.440+0000 7fdc8b660640 1 -- 172.21.15.181:0/1717838731 wait complete. 2024-09-15T03:36:47.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.444+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.444+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1853658433}) 2024-09-15T03:36:47.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.444+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=95} crc {osdmap=3762871322}) 2024-09-15T03:36:47.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.444+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:36:47.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.444+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1853658433}) 2024-09-15T03:36:47.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:36:47.444+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=95} crc {osdmap=766713617}) 2024-09-15T03:36:47.451 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:36:47.451 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-15T03:36:47.494 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:36:47.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:47.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.524+0000 7feeed694640 10 monclient: get_monmap_and_config 2024-09-15T03:36:47.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.524+0000 7feeed694640 10 monclient: build_initial_monmap 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.524+0000 7feeed694640 10 monclient: monmap: 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:36:47.525756+0000 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:36:47.525756+0000 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:47.526 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.525+0000 7feeed694640 1 Processor -- start 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.525+0000 7feeed694640 1 -- start start 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.525+0000 7feeed694640 10 monclient: init 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feee8098f00 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feee809eae0 0x7feee806a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient(hunting): picked mon.noname-a con 0x7feee809eae0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:47.529 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feee8136cc0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee809a060 con 0x7feee8136cc0 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee8069f00 con 0x7feee8098f00 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee809f750 con 0x7feee809eae0 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feeed694640 10 monclient(hunting): authenticate will time out at 1212.007156s 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feee6ffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7feee8098f00 0x7feee80992d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:52452/0 (socket says 172.21.15.181:52452) 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feee6ffd640 1 -- 172.21.15.181:0/2234614881 learned_addr learned my addr 172.21.15.181:0/2234614881 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 1 --2- 172.21.15.181:0/2234614881 >> v2:172.21.15.153:3300/0 conn(0x7feee809eae0 0x7feee806a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): get_auth_request con 0x7feee809eae0 auth_method 0 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient(hunting): handle_auth_done global_id 114174 payload 274 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 1 -- 172.21.15.181:0/2234614881 >> v1:172.21.15.181:6789/0 conn(0x7feee8098f00 legacy=0x7feee80992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 1 -- 172.21.15.181:0/2234614881 >> v1:172.21.15.153:6789/0 conn(0x7feee8136cc0 legacy=0x7feee81390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 1 monclient: found mon.noname-a 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:47.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.526+0000 7feede7fc640 1 -- 172.21.15.181:0/2234614881 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee809f430 con 0x7feee809eae0 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.529+0000 7feede7fc640 1 -- 172.21.15.181:0/2234614881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee809eae0 msgr2=0x7feee806a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.529+0000 7feede7fc640 1 --2- 172.21.15.181:0/2234614881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee809eae0 0x7feee806a600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feed0009850 tx=0x7feed00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5663f9a621a77b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 1 -- 172.21.15.181:0/2234614881 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0032440 con 0x7feee809eae0 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: dump: 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 1 -- 172.21.15.181:0/2234614881 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed00325a0 con 0x7feee809eae0 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 5 monclient: authenticate success, global_id 114174 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 1 -- 172.21.15.181:0/2234614881 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0032940 con 0x7feee809eae0 2024-09-15T03:36:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 4 set_mon_vals no callback set 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feee67fc640 10 monclient: dump: 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.540 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 10 monclient: shutdown 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 1 -- 172.21.15.181:0/2234614881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee809eae0 msgr2=0x7feee806a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 1 --2- 172.21.15.181:0/2234614881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee809eae0 0x7feee806a600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feed0009850 tx=0x7feed00090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 1 -- 172.21.15.181:0/2234614881 shutdown_connections 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 1 --2- 172.21.15.181:0/2234614881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee809eae0 0x7feee806a600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 1 -- 172.21.15.181:0/2234614881 >> 172.21.15.181:0/2234614881 conn(0x7feee80a3b30 msgr2=0x7feee809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.530+0000 7feeed694640 1 -- 172.21.15.181:0/2234614881 shutdown_connections 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.531+0000 7feeed694640 1 -- 172.21.15.181:0/2234614881 wait complete. 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.531+0000 7feeed694640 10 monclient: build_initial_monmap 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.531+0000 7feeed694640 10 monclient: monmap: 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:36:47.541 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.531+0000 7feeed694640 1 Processor -- start 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 -- start start 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient: init 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee8098f00 0x7feee8135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): picked mon.noname-a con 0x7feee8098f00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 0x7feee811e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): picked mon.noname-b con 0x7feee809eae0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feee8136cc0 0x7feee811ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): picked mon.noname-c con 0x7feee8136cc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee8122180 con 0x7feee8098f00 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee806eb60 con 0x7feee809eae0 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee80a21e0 con 0x7feee8136cc0 2024-09-15T03:36:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feeed694640 10 monclient(hunting): authenticate will time out at 1212.013497s 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 0x7feee811e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 0x7feee811e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44282/0 (socket says 172.21.15.181:44282) 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 -- 172.21.15.181:0/3974097974 learned_addr learned my addr 172.21.15.181:0/3974097974 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): get_auth_request con 0x7feee809eae0 auth_method 0 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee8098f00 0x7feee8135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): get_auth_request con 0x7feee8098f00 auth_method 0 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee77fe640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feee8136cc0 0x7feee811ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee77fe640 10 monclient(hunting): get_auth_request con 0x7feee8136cc0 auth_method 0 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient(hunting): handle_auth_done global_id 104374 payload 824 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 -- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feee8136cc0 msgr2=0x7feee811ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feee8136cc0 0x7feee811ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 -- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee8098f00 msgr2=0x7feee8135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee8098f00 0x7feee8135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 monclient: found mon.noname-b 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feede7fc640 1 -- 172.21.15.181:0/3974097974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0009390 con 0x7feee809eae0 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.532+0000 7feee77fe640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feee8136cc0 0x7feee811ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feede7fc640 1 -- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 msgr2=0x7feee811e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feede7fc640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 0x7feee811e8c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feed0009850 tx=0x7feed0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0031940 con 0x7feee809eae0 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: dump: 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.547 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 5 monclient: authenticate success, global_id 104374 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 10 monclient: _renew_subs 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee8141fb0 con 0x7feee809eae0 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed0031aa0 con 0x7feee809eae0 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0031e60 con 0x7feee809eae0 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feedf7fe640 10 monclient: dump: 2024-09-15T03:36:47.548 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feee5ffb640 4 set_mon_vals no callback set 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 10 monclient: _renew_subs 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.535+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee8120370 con 0x7feee809eae0 2024-09-15T03:36:47.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7feed0039050 con 0x7feee809eae0 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feeed694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feeed694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feeed694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee804bf60 con 0x7feee809eae0 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feedf7fe640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feec8038740 0x7feec803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7feed006e200 con 0x7feee809eae0 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feee6ffd640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feec8038740 0x7feec803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.536+0000 7feee6ffd640 10 monclient: get_auth_request con 0x7feec8038740 auth_method 0 2024-09-15T03:36:47.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.539+0000 7feee6ffd640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feec8038740 0x7feec803abf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7feed8007fd0 tx=0x7feed8007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.541+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7feed0030570 con 0x7feee809eae0 2024-09-15T03:36:47.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.541+0000 7feedf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.541+0000 7feedf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.596+0000 7f92d92c0640 10 monclient: get_monmap_and_config 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.596+0000 7f92d92c0640 10 monclient: build_initial_monmap 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.596+0000 7f92d92c0640 10 monclient: monmap: 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:47.597066+0000 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:47.597066+0000 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.598 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:47.598 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:47.598 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:47.598 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:47.598 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:47.598 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.596+0000 7f92d92c0640 1 Processor -- start 2024-09-15T03:36:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 -- start start 2024-09-15T03:36:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient: init 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f92d4131ab0 0x7f92d4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): picked mon.noname-a con 0x7f92d4131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): picked mon.noname-b con 0x7f92d409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 0x7f92d4068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): picked mon.noname-c con 0x7f92d409f220 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d409faf0 con 0x7f92d4131ab0 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d409fd30 con 0x7f92d409f220 2024-09-15T03:36:47.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d40a1110 con 0x7f92d409e910 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d92c0640 10 monclient(hunting): authenticate will time out at 1205.236558s 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f92d4131ab0 0x7f92d4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45466/0 (socket says 172.21.15.153:45466) 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f92d4131ab0 0x7f92d4131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60652/0 (socket says 172.21.15.153:60652) 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 1 -- 172.21.15.153:0/3793179384 learned_addr learned my addr 172.21.15.153:0/3793179384 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): get_auth_request con 0x7f92d409e910 auth_method 0 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): get_auth_request con 0x7f92d4131ab0 auth_method 0 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 1 --2- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 0x7f92d4068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 10 monclient(hunting): get_auth_request con 0x7f92d409f220 auth_method 0 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 1 -- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 msgr2=0x7f92d4068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 1 -- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 msgr2=0x7f92d4068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 1 --2- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 0x7f92d4068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d37fe640 1 --2- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 0x7f92d4068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.597+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_done global_id 114180 payload 274 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 1 -- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 msgr2=0x7f92d409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 1 --2- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 1 -- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 msgr2=0x7f92d4068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 1 --2- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 0x7f92d4068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 1 monclient: found mon.noname-a 2024-09-15T03:36:47.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92d2ffd640 1 -- 172.21.15.153:0/3793179384 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40a13f0 con 0x7f92d4131ab0 2024-09-15T03:36:47.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.598+0000 7f92ca7fc640 1 --2- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d2ffd640 1 -- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d4131ab0 msgr2=0x7f92d4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d2ffd640 1 --2- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d4131ab0 0x7f92d4131e80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92d409bd80 tx=0x7f92c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdb9d24d6e82daf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d27fc640 1 -- 172.21.15.153:0/3793179384 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c402ae80 con 0x7f92d4131ab0 2024-09-15T03:36:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:47.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d27fc640 10 monclient: dump: 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.600+0000 7f92d27fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:47.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 1 -- 172.21.15.153:0/3793179384 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c4032440 con 0x7f92d4131ab0 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 5 monclient: authenticate success, global_id 114180 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 1 -- 172.21.15.153:0/3793179384 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c40327c0 con 0x7f92d4131ab0 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 4 set_mon_vals no callback set 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d27fc640 10 monclient: dump: 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.613 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 10 monclient: shutdown 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 -- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d4131ab0 msgr2=0x7f92d4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d4131ab0 0x7f92d4131e80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f92d409bd80 tx=0x7f92c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 -- 172.21.15.153:0/3793179384 shutdown_connections 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3793179384 >> v2:172.21.15.181:3300/0 conn(0x7f92d409f220 0x7f92d4068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3793179384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d4131ab0 0x7f92d4131e80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 -- 172.21.15.153:0/3793179384 >> 172.21.15.153:0/3793179384 conn(0x7f92d4067d30 msgr2=0x7f92d40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 -- 172.21.15.153:0/3793179384 shutdown_connections 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.601+0000 7f92d92c0640 1 -- 172.21.15.153:0/3793179384 wait complete. 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.602+0000 7f92d92c0640 10 monclient: build_initial_monmap 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.602+0000 7f92d92c0640 10 monclient: monmap: 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:47.615 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.602+0000 7f92d92c0640 1 Processor -- start 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.602+0000 7f92d92c0640 1 -- start start 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.602+0000 7f92d92c0640 10 monclient: init 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): picked mon.noname-c con 0x7f92d409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 0x7f92d4130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): picked mon.noname-a con 0x7f92d409f220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92d4131ab0 0x7f92d4140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): picked mon.noname-b con 0x7f92d4131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d411c690 con 0x7f92d409f220 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d411df10 con 0x7f92d4131ab0 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d411dc30 con 0x7f92d409e910 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d92c0640 10 monclient(hunting): authenticate will time out at 1205.242092s 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 0x7f92d4130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 0x7f92d4130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60666/0 (socket says 172.21.15.153:60666) 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 -- 172.21.15.153:0/3320085074 learned_addr learned my addr 172.21.15.153:0/3320085074 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): get_auth_request con 0x7f92d409e910 auth_method 0 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): get_auth_request con 0x7f92d409f220 auth_method 0 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92d4131ab0 0x7f92d4140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): get_auth_request con 0x7f92d4131ab0 auth_method 0 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient(hunting): handle_auth_done global_id 114183 payload 824 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 -- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 msgr2=0x7f92d409a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 -- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92d4131ab0 msgr2=0x7f92d4140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92d4131ab0 0x7f92d4140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 monclient: found mon.noname-a 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92ca7fc640 1 -- 172.21.15.153:0/3320085074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc009450 con 0x7f92d409f220 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.603+0000 7f92d2ffd640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92ca7fc640 1 -- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 msgr2=0x7f92d4130b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92ca7fc640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 0x7f92d4130b10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00ec80 tx=0x7f92bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc009730 con 0x7f92d409f220 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 10 monclient: dump: 2024-09-15T03:36:47.620 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92d92c0640 5 monclient: authenticate success, global_id 114183 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00eec0 con 0x7f92d409f220 2024-09-15T03:36:47.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc013470 con 0x7f92d409f220 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92d92c0640 10 monclient: _renew_subs 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92d92c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c4009390 con 0x7f92d409f220 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.605+0000 7f92d1ffb640 4 set_mon_vals no callback set 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92cb7fe640 10 monclient: dump: 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.622 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 10 monclient: _renew_subs 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d411cf50 con 0x7f92d409f220 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f92bc013820 con 0x7f92d409f220 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.606+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d411e0d0 con 0x7f92d409f220 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.607+0000 7f92cb7fe640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92b4038740 0x7f92b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.607+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f92bc04df50 con 0x7f92d409f220 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.607+0000 7f92d2ffd640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92b4038740 0x7f92b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.607+0000 7f92d2ffd640 10 monclient: get_auth_request con 0x7f92b4038740 auth_method 0 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.609+0000 7f92d2ffd640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92b4038740 0x7f92b403abf0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f92d4076760 tx=0x7f92c4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.611+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f92bc0236b0 con 0x7f92d409f220 2024-09-15T03:36:47.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.611+0000 7f92cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.611+0000 7f92cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:47.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.698+0000 7feeed694640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:47.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.698+0000 7feeed694640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:36:47.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.698+0000 7feeed694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:36:47.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:47.698+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7feee804bf60 con 0x7feee809eae0 2024-09-15T03:36:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92d92c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92d92c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92d92c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f92d411e360 con 0x7f92d409f220 2024-09-15T03:36:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92cb7fe640 1 -- 172.21.15.153:0/3320085074 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f92bc0232f0 con 0x7f92d409f220 2024-09-15T03:36:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.825+0000 7f92cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:47.827 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92b4038740 msgr2=0x7f92b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92b4038740 0x7f92b403abf0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f92d4076760 tx=0x7f92c4032000 comp rx=0 tx=0).stop 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 10 monclient: shutdown 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 msgr2=0x7f92d4130b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 0x7f92d4130b10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00ec80 tx=0x7f92bc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 shutdown_connections 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f92b4038740 0x7f92b403abf0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92d4131ab0 0x7f92d4140e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92d409f220 0x7f92d4130b10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 --2- 172.21.15.153:0/3320085074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92d409e910 0x7f92d409a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.827+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 >> 172.21.15.153:0/3320085074 conn(0x7f92d4067d30 msgr2=0x7f92d40a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.828+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 shutdown_connections 2024-09-15T03:36:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.828+0000 7f92d92c0640 1 -- 172.21.15.153:0/3320085074 wait complete. 2024-09-15T03:36:47.839 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:36:47.839 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:36:47.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:36:47.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:36:47.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.932+0000 7fbc44820640 10 monclient: get_monmap_and_config 2024-09-15T03:36:47.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.932+0000 7fbc44820640 10 monclient: build_initial_monmap 2024-09-15T03:36:47.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.932+0000 7fbc44820640 10 monclient: monmap: 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:47.933350+0000 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:47.933350+0000 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:47.934 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:47.935 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:47.935 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:47.935 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 1 Processor -- start 2024-09-15T03:36:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 1 -- start start 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient: init 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc40096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc400971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc40136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient(hunting): picked mon.noname-b con 0x7fbc40134b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4009d4a0 con 0x7fbc40096e10 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc40099620 con 0x7fbc400971e0 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc40099340 con 0x7fbc40134b40 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc44820640 10 monclient(hunting): authenticate will time out at 1205.572741s 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbc40096e10 0x7fbc4009a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35922/0 (socket says 172.21.15.153:35922) 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc40136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3e59c640 1 -- 172.21.15.153:0/3535110231 learned_addr learned my addr 172.21.15.153:0/3535110231 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): get_auth_request con 0x7fbc40134b40 auth_method 0 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.933+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_done global_id 74906 payload 274 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 1 -- 172.21.15.153:0/3535110231 >> v1:172.21.15.181:6789/0 conn(0x7fbc400971e0 legacy=0x7fbc400975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 1 -- 172.21.15.153:0/3535110231 >> v1:172.21.15.153:6789/0 conn(0x7fbc40096e10 legacy=0x7fbc4009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 1 monclient: found mon.noname-b 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.934+0000 7fbc3ed9d640 1 -- 172.21.15.153:0/3535110231 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc40099900 con 0x7fbc40134b40 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.936+0000 7fbc3ed9d640 1 -- 172.21.15.153:0/3535110231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 msgr2=0x7fbc40136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.936+0000 7fbc3ed9d640 1 --2- 172.21.15.153:0/3535110231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc40136f20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbc4009b020 tx=0x7fbc38009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9674b117b45cded0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.936+0000 7fbc3dd9b640 1 -- 172.21.15.153:0/3535110231 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38007840 con 0x7fbc40134b40 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: dump: 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.944 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 5 monclient: authenticate success, global_id 74906 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 1 -- 172.21.15.153:0/3535110231 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3802d050 con 0x7fbc40134b40 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 1 -- 172.21.15.153:0/3535110231 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38007bd0 con 0x7fbc40134b40 2024-09-15T03:36:47.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 4 set_mon_vals no callback set 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc3dd9b640 10 monclient: dump: 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.946 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 10 monclient: shutdown 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 -- 172.21.15.153:0/3535110231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 msgr2=0x7fbc40136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 --2- 172.21.15.153:0/3535110231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc40136f20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbc4009b020 tx=0x7fbc38009780 comp rx=0 tx=0).stop 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 -- 172.21.15.153:0/3535110231 shutdown_connections 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 --2- 172.21.15.153:0/3535110231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc40136f20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 -- 172.21.15.153:0/3535110231 >> 172.21.15.153:0/3535110231 conn(0x7fbc400672b0 msgr2=0x7fbc400676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 -- 172.21.15.153:0/3535110231 shutdown_connections 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.937+0000 7fbc44820640 1 -- 172.21.15.153:0/3535110231 wait complete. 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient: build_initial_monmap 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient: monmap: 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:47.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 1 Processor -- start 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 1 -- start start 2024-09-15T03:36:47.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient: init 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc40096e10 0x7fbc4009b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient(hunting): picked mon.noname-b con 0x7fbc40096e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 0x7fbc4009bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.938+0000 7fbc44820640 10 monclient(hunting): picked mon.noname-a con 0x7fbc400971e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc4009c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 10 monclient(hunting): picked mon.noname-c con 0x7fbc40134b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4009e660 con 0x7fbc400971e0 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc40133b10 con 0x7fbc40096e10 2024-09-15T03:36:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4012fb70 con 0x7fbc40134b40 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc44820640 10 monclient(hunting): authenticate will time out at 1205.578033s 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc4009c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 0x7fbc4009bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc4009c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45494/0 (socket says 172.21.15.153:45494) 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 1 -- 172.21.15.153:0/1194175736 learned_addr learned my addr 172.21.15.153:0/1194175736 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): get_auth_request con 0x7fbc40134b40 auth_method 0 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): get_auth_request con 0x7fbc400971e0 auth_method 0 2024-09-15T03:36:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc40096e10 0x7fbc4009b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): get_auth_request con 0x7fbc40096e10 auth_method 0 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient(hunting): handle_auth_done global_id 114186 payload 824 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 msgr2=0x7fbc4009c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc4009c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc40096e10 msgr2=0x7fbc4009b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc40096e10 0x7fbc4009b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 monclient: found mon.noname-a 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc35d9b640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38009a20 con 0x7fbc400971e0 2024-09-15T03:36:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.939+0000 7fbc3ed9d640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc40134b40 0x7fbc4009c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc35d9b640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 msgr2=0x7fbc4009bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc35d9b640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 0x7fbc4009bd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c4e0 tx=0x7fbc28007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc280117a0 con 0x7fbc400971e0 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: dump: 2024-09-15T03:36:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: _finish_auth 0 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28011920 con 0x7fbc400971e0 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28011ce0 con 0x7fbc400971e0 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 5 monclient: authenticate success, global_id 114186 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 10 monclient: _renew_subs 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4012f890 con 0x7fbc400971e0 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc36ffd640 10 monclient: dump: 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc3d59a640 4 set_mon_vals no callback set 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 10 monclient: _renew_subs 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.941+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc401344d0 con 0x7fbc400971e0 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc44820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc44820640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 0x7fbc4012ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc44820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc40065f20 con 0x7fbc4012aa60 2024-09-15T03:36:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbc28012630 con 0x7fbc400971e0 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 0x7fbc4012ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient: get_auth_request con 0x7fbc4012aa60 auth_method 0 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc36ffd640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc20038740 0x7fbc2003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.942+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7fbc2804d7f0 con 0x7fbc400971e0 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.943+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_done global_id 114189 payload 824 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.943+0000 7fbc3e59c640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc4004bf60 con 0x7fbc4012aa60 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.943+0000 7fbc3ed9d640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc20038740 0x7fbc2003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.943+0000 7fbc3ed9d640 10 monclient: get_auth_request con 0x7fbc20038740 auth_method 0 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc3e59c640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 msgr2=0x7fbc4012ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc3e59c640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 0x7fbc4012ae30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000e710 tx=0x7fbc30007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30004b50 con 0x7fbc4012aa60 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 10 monclient: dump: 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc3ed9d640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc20038740 0x7fbc2003abf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbc380097c0 tx=0x7fbc38037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbc30004cb0 con 0x7fbc4012aa60 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 msgr2=0x7fbc4012ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 0x7fbc4012ae30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000e710 tx=0x7fbc30007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 msgr2=0x7fbc4012ae30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.945+0000 7fbc36ffd640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012aa60 0x7fbc4012ae30 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000e710 tx=0x7fbc30007f00 comp rx=0 tx=0).stop 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc44820640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc44820640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 0x7fbc4014b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc44820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4009b0e0 con 0x7fbc4012b370 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 0x7fbc4014b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient: get_auth_request con 0x7fbc4012b370 auth_method 0 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 10 monclient(hunting): handle_auth_done global_id 114192 payload 824 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.951+0000 7fbc3e59c640 1 -- 172.21.15.153:0/1194175736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbc4006f700 con 0x7fbc4012b370 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc3e59c640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 msgr2=0x7fbc4014b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc3e59c640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 0x7fbc4014b660 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000e710 tx=0x7fbc30019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:47.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30015d30 con 0x7fbc4012b370 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 10 monclient: dump: 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:47.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7fbc30015e90 con 0x7fbc4012b370 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 msgr2=0x7fbc4014b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 0x7fbc4014b660 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000e710 tx=0x7fbc30019000 comp rx=0 tx=0).stop 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 msgr2=0x7fbc4014b660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc36ffd640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4012b370 0x7fbc4014b660 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000e710 tx=0x7fbc30019000 comp rx=0 tx=0).stop 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:47.966 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 92, 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:47.967 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:47.968 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:47.969 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:47.970 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:47.971 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.972 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:36:47.973 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:47.975 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:47.976 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:47.977 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:36:47.978 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:36:47.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc20038740 msgr2=0x7fbc2003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc20038740 0x7fbc2003abf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbc380097c0 tx=0x7fbc38037000 comp rx=0 tx=0).stop 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 10 monclient: shutdown 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 msgr2=0x7fbc4009bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 0x7fbc4009bd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c4e0 tx=0x7fbc28007e10 comp rx=0 tx=0).stop 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc3ed9d640 1 -- 172.21.15.153:0/1194175736 reap_dead start 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 shutdown_connections 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 --2- 172.21.15.153:0/1194175736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc400971e0 0x7fbc4009bd50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 >> 172.21.15.153:0/1194175736 conn(0x7fbc400672b0 msgr2=0x7fbc40135350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 shutdown_connections 2024-09-15T03:36:47.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:47.954+0000 7fbc44820640 1 -- 172.21.15.153:0/1194175736 wait complete. 2024-09-15T03:36:47.981 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:36:47.981 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:36:47.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.074+0000 7f0c42a3f640 10 monclient: get_monmap_and_config 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.074+0000 7f0c42a3f640 10 monclient: build_initial_monmap 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.074+0000 7f0c42a3f640 10 monclient: monmap: 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:48.075187+0000 2024-09-15T03:36:48.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:48.075187+0000 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:48.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 Processor -- start 2024-09-15T03:36:48.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 -- start start 2024-09-15T03:36:48.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient: init 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 0x7f0c3c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f0c3c0974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0c3c09a8a0 0x7f0c3c09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f0c3c09a8a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c3c12dd40 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3c0978c0 con 0x7f0c3c12dd40 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c3c09ca20 con 0x7f0c3c09a8a0 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c09b1f0 con 0x7f0c3c0974f0 2024-09-15T03:36:48.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c42a3f640 10 monclient(hunting): authenticate will time out at 1205.714688s 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 0x7f0c3c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c40fdc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0c3c12dd40 0x7f0c3c12e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35930/0 (socket says 172.21.15.153:35930) 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c40fdc640 1 -- 172.21.15.153:0/2569375167 learned_addr learned my addr 172.21.15.153:0/2569375167 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): get_auth_request con 0x7f0c3c0974f0 auth_method 0 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3b7fe640 1 --2- 172.21.15.153:0/2569375167 >> v2:172.21.15.181:3300/0 conn(0x7f0c3c09a8a0 0x7f0c3c09acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c3c09a8a0 auth_method 0 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.075+0000 7f0c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_done global_id 74912 payload 274 2024-09-15T03:36:48.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 1 -- 172.21.15.153:0/2569375167 >> v2:172.21.15.181:3300/0 conn(0x7f0c3c09a8a0 msgr2=0x7f0c3c09acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:48.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/2569375167 >> v2:172.21.15.181:3300/0 conn(0x7f0c3c09a8a0 0x7f0c3c09acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 1 -- 172.21.15.153:0/2569375167 >> v1:172.21.15.153:6789/0 conn(0x7f0c3c12dd40 legacy=0x7f0c3c12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:48.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:36:48.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 1 monclient: found mon.noname-b 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.076+0000 7f0c3bfff640 1 -- 172.21.15.153:0/2569375167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c098ad0 con 0x7f0c3c0974f0 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3bfff640 1 -- 172.21.15.153:0/2569375167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 msgr2=0x7f0c3c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/2569375167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 0x7f0c3c12ab00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009850 tx=0x7f0c2c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc697cc01157b1aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 1 -- 172.21.15.153:0/2569375167 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c02adf0 con 0x7f0c3c0974f0 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 10 monclient: dump: 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 10 monclient: _finish_auth 0 2024-09-15T03:36:48.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 1 -- 172.21.15.153:0/2569375167 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c032440 con 0x7f0c3c0974f0 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c42a3f640 5 monclient: authenticate success, global_id 74912 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c42a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.078+0000 7f0c3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c3affd640 1 -- 172.21.15.153:0/2569375167 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0327e0 con 0x7f0c3c0974f0 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 4 set_mon_vals no callback set 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c3affd640 10 monclient: dump: 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.087 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 10 monclient: shutdown 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 -- 172.21.15.153:0/2569375167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 msgr2=0x7f0c3c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/2569375167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 0x7f0c3c12ab00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009850 tx=0x7f0c2c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 -- 172.21.15.153:0/2569375167 shutdown_connections 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/2569375167 >> v2:172.21.15.181:3300/0 conn(0x7f0c3c09a8a0 0x7f0c3c09acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/2569375167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c0974f0 0x7f0c3c12ab00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 -- 172.21.15.153:0/2569375167 >> 172.21.15.153:0/2569375167 conn(0x7f0c3c0672b0 msgr2=0x7f0c3c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 -- 172.21.15.153:0/2569375167 shutdown_connections 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.079+0000 7f0c42a3f640 1 -- 172.21.15.153:0/2569375167 wait complete. 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient: build_initial_monmap 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient: monmap: 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:48.089 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:48.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 1 Processor -- start 2024-09-15T03:36:48.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 1 -- start start 2024-09-15T03:36:48.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient: init 2024-09-15T03:36:48.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:48.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0974f0 0x7f0c3c1176b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f0c3c0974f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c09a8a0 0x7f0c3c117bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f0c3c09a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 0x7f0c3c118130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient(hunting): picked mon.noname-a con 0x7f0c3c12dd40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.080+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c11b7b0 con 0x7f0c3c12dd40 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c098300 con 0x7f0c3c0974f0 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c09c0d0 con 0x7f0c3c09a8a0 2024-09-15T03:36:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c42a3f640 10 monclient(hunting): authenticate will time out at 1205.720022s 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c09a8a0 0x7f0c3c117bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 0x7f0c3c118130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c09a8a0 0x7f0c3c117bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45520/0 (socket says 172.21.15.153:45520) 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 1 -- 172.21.15.153:0/1217349365 learned_addr learned my addr 172.21.15.153:0/1217349365 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): get_auth_request con 0x7f0c3c12dd40 auth_method 0 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c3c09a8a0 auth_method 0 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0974f0 0x7f0c3c1176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): get_auth_request con 0x7f0c3c0974f0 auth_method 0 2024-09-15T03:36:48.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient(hunting): handle_auth_done global_id 114195 payload 824 2024-09-15T03:36:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c09a8a0 msgr2=0x7f0c3c117bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c09a8a0 0x7f0c3c117bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0974f0 msgr2=0x7f0c3c1176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0974f0 0x7f0c3c1176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient: _finish_hunting 0 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 monclient: found mon.noname-a 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c40fdc640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c009390 con 0x7f0c3c12dd40 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3b7fe640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c09a8a0 0x7f0c3c117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.081+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0974f0 0x7f0c3c1176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c40fdc640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 msgr2=0x7f0c3c118130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c40fdc640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 0x7f0c3c118130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400cb60 tx=0x7f0c340052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400a850 con 0x7f0c3c12dd40 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: dump: 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.107 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: _finish_auth 0 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3400a9d0 con 0x7f0c3c12dd40 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400ad90 con 0x7f0c3c12dd40 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 5 monclient: authenticate success, global_id 114195 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:48.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c23fff640 10 monclient: dump: 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.109 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 10 monclient: _renew_subs 2024-09-15T03:36:48.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c11a870 con 0x7f0c3c12dd40 2024-09-15T03:36:48.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c3a7fc640 4 set_mon_vals no callback set 2024-09-15T03:36:48.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 10 monclient: _renew_subs 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.083+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c118ad0 con 0x7f0c3c12dd40 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c42a3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 0x7f0c3c06b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c065f20 con 0x7f0c3c118670 2024-09-15T03:36:48.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0c34007300 con 0x7f0c3c12dd40 2024-09-15T03:36:48.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 0x7f0c3c06b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient: get_auth_request con 0x7f0c3c118670 auth_method 0 2024-09-15T03:36:48.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c23fff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c28038740 0x7f0c2803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.084+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f0c3404db70 con 0x7f0c3c12dd40 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.085+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_done global_id 74918 payload 824 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.085+0000 7f0c3bfff640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c3c04bf60 con 0x7f0c3c118670 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.085+0000 7f0c3b7fe640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c28038740 0x7f0c2803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.085+0000 7f0c3b7fe640 10 monclient: get_auth_request con 0x7f0c28038740 auth_method 0 2024-09-15T03:36:48.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c3bfff640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 msgr2=0x7f0c3c06b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 0x7f0c3c06b8b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c034000 tx=0x7f0c2c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c03c440 con 0x7f0c3c118670 2024-09-15T03:36:48.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:48.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 10 monclient: dump: 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.115 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.116 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.116 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.116 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0c2c03c5a0 con 0x7f0c3c118670 2024-09-15T03:36:48.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c3b7fe640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c28038740 0x7f0c2803abf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0c300114a0 tx=0x7f0c300090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 msgr2=0x7f0c3c06b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 0x7f0c3c06b8b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c034000 tx=0x7f0c2c032000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 msgr2=0x7f0c3c06b8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.087+0000 7f0c23fff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c118670 0x7f0c3c06b8b0 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c034000 tx=0x7f0c2c032000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c42a3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:36:48.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 0x7f0c3c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c42a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c11aa60 con 0x7f0c3c06bdf0 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 0x7f0c3c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient: get_auth_request con 0x7f0c3c06bdf0 auth_method 0 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 10 monclient(hunting): handle_auth_done global_id 74921 payload 824 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.093+0000 7f0c3bfff640 1 -- 172.21.15.153:0/1217349365 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0c3c1196a0 con 0x7f0c3c06bdf0 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c3bfff640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 msgr2=0x7f0c3c06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c3bfff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 0x7f0c3c06c1f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0686d0 tx=0x7f0c2c03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0468f0 con 0x7f0c3c06bdf0 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 10 monclient: dump: 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.119 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f0c2c046a50 con 0x7f0c3c06bdf0 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 msgr2=0x7f0c3c06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 0x7f0c3c06c1f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0686d0 tx=0x7f0c2c03d000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 msgr2=0x7f0c3c06c1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:48.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c23fff640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c3c06bdf0 0x7f0c3c06c1f0 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0686d0 tx=0x7f0c2c03d000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 92, 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:36:48.121 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:48.122 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:48.123 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:36:48.124 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:48.125 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:36:48.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:48.127 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:48.128 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:48.129 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:48.130 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.132 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.133 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:48.134 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:48.134 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:36:48.134 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.134 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.134 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:36:48.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.135 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:48.135 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:48.135 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:36:48.135 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.135 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.136 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:36:48.136 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:36:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c28038740 msgr2=0x7f0c2803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c28038740 0x7f0c2803abf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0c300114a0 tx=0x7f0c300090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 10 monclient: shutdown 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 msgr2=0x7f0c3c118130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c40fdc640 1 -- 172.21.15.153:0/1217349365 reap_dead start 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 0x7f0c3c118130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400cb60 tx=0x7f0c340052b0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 shutdown_connections 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 --2- 172.21.15.153:0/1217349365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c3c12dd40 0x7f0c3c118130 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 >> 172.21.15.153:0/1217349365 conn(0x7f0c3c0672b0 msgr2=0x7f0c3c09d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 shutdown_connections 2024-09-15T03:36:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.096+0000 7f0c42a3f640 1 -- 172.21.15.153:0/1217349365 wait complete. 2024-09-15T03:36:48.140 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:36:48.140 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:36:48.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.234+0000 7f27c111d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.234+0000 7f27c111d640 10 monclient: build_initial_monmap 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.234+0000 7f27c111d640 10 monclient: monmap: 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:48.234904+0000 2024-09-15T03:36:48.235 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:48.234904+0000 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.234+0000 7f27c111d640 1 Processor -- start 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.234+0000 7f27c111d640 1 -- start start 2024-09-15T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.234+0000 7f27c111d640 10 monclient: init 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27bc1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f27bc09ea50 0x7f27bc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc09ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f27bc136c20 0x7f27bc139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:48.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc131cc0 con 0x7f27bc1318f0 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc09fbf0 con 0x7f27bc136c20 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc09fe30 con 0x7f27bc09ea50 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27c111d640 10 monclient(hunting): authenticate will time out at 1205.874294s 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f27bc136c20 0x7f27bc139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f27bc1318f0 0x7f27bc131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35946/0 (socket says 172.21.15.153:35946) 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bad9d640 1 -- 172.21.15.153:0/3987799260 learned_addr learned my addr 172.21.15.153:0/3987799260 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): get_auth_request con 0x7f27bc136c20 auth_method 0 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 1 --2- 172.21.15.153:0/3987799260 >> v2:172.21.15.181:3300/0 conn(0x7f27bc09ea50 0x7f27bc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): get_auth_request con 0x7f27bc09ea50 auth_method 0 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient(hunting): handle_auth_done global_id 114198 payload 274 2024-09-15T03:36:48.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 1 -- 172.21.15.153:0/3987799260 >> v2:172.21.15.181:3300/0 conn(0x7f27bc09ea50 msgr2=0x7f27bc09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 1 --2- 172.21.15.153:0/3987799260 >> v2:172.21.15.181:3300/0 conn(0x7f27bc09ea50 0x7f27bc09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 1 -- 172.21.15.153:0/3987799260 >> v1:172.21.15.153:6789/0 conn(0x7f27bc1318f0 legacy=0x7f27bc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient: _finish_hunting 0 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 1 monclient: found mon.noname-a 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.235+0000 7f27bb59e640 1 -- 172.21.15.153:0/3987799260 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0a1210 con 0x7f27bc136c20 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27bb59e640 1 -- 172.21.15.153:0/3987799260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc136c20 msgr2=0x7f27bc139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27bb59e640 1 --2- 172.21.15.153:0/3987799260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc136c20 0x7f27bc139000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06f600 tx=0x7f27b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315674e800d138f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 1 -- 172.21.15.153:0/3987799260 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4007760 con 0x7f27bc136c20 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: dump: 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: _finish_auth 0 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27c111d640 5 monclient: authenticate success, global_id 114198 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27c111d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 1 -- 172.21.15.153:0/3987799260 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b402d050 con 0x7f27bc136c20 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 1 -- 172.21.15.153:0/3987799260 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4007af0 con 0x7f27bc136c20 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27c111d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27c111d640 4 set_mon_vals no callback set 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.238+0000 7f27ba59c640 10 monclient: dump: 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 10 monclient: shutdown 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 -- 172.21.15.153:0/3987799260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc136c20 msgr2=0x7f27bc139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 --2- 172.21.15.153:0/3987799260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc136c20 0x7f27bc139000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06f600 tx=0x7f27b4009780 comp rx=0 tx=0).stop 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 -- 172.21.15.153:0/3987799260 shutdown_connections 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 --2- 172.21.15.153:0/3987799260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc136c20 0x7f27bc139000 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 --2- 172.21.15.153:0/3987799260 >> v2:172.21.15.181:3300/0 conn(0x7f27bc09ea50 0x7f27bc09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 -- 172.21.15.153:0/3987799260 >> 172.21.15.153:0/3987799260 conn(0x7f27bc067110 msgr2=0x7f27bc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 -- 172.21.15.153:0/3987799260 shutdown_connections 2024-09-15T03:36:48.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 1 -- 172.21.15.153:0/3987799260 wait complete. 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 10 monclient: build_initial_monmap 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.239+0000 7f27c111d640 10 monclient: monmap: 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 Processor -- start 2024-09-15T03:36:48.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 -- start start 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient: init 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 0x7f27bc11daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc1318f0 0x7f27bc130620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): picked mon.noname-b con 0x7f27bc1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27bc136c20 0x7f27bc130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc09db60 con 0x7f27bc09ea50 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc09b9c0 con 0x7f27bc1318f0 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc09bf10 con 0x7f27bc136c20 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27c111d640 10 monclient(hunting): authenticate will time out at 1205.879641s 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 0x7f27bc11daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 0x7f27bc11daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60700/0 (socket says 172.21.15.153:60700) 2024-09-15T03:36:48.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 1 -- 172.21.15.153:0/3557462360 learned_addr learned my addr 172.21.15.153:0/3557462360 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): get_auth_request con 0x7f27bc09ea50 auth_method 0 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27bc136c20 0x7f27bc130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): get_auth_request con 0x7f27bc136c20 auth_method 0 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc1318f0 0x7f27bc130620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): get_auth_request con 0x7f27bc1318f0 auth_method 0 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27bb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.240+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 10 monclient(hunting): handle_auth_done global_id 114201 payload 824 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27bc136c20 msgr2=0x7f27bc130b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27bc136c20 0x7f27bc130b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc1318f0 msgr2=0x7f27bc130620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc1318f0 0x7f27bc130620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 1 monclient: found mon.noname-a 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bad9d640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4009a20 con 0x7f27bc09ea50 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.241+0000 7f27bb59e640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27bc136c20 0x7f27bc130b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27bad9d640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 msgr2=0x7f27bc11daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27bad9d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 0x7f27bc11daa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00c5e0 tx=0x7f27ac004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac005cb0 con 0x7f27bc09ea50 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: dump: 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.264 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: _finish_auth 0 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac005e30 con 0x7f27bc09ea50 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 5 monclient: authenticate success, global_id 114201 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 10 monclient: _renew_subs 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc09b200 con 0x7f27bc09ea50 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac007340 con 0x7f27bc09ea50 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b37fe640 10 monclient: dump: 2024-09-15T03:36:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27b9d9b640 4 set_mon_vals no callback set 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 10 monclient: _renew_subs 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.243+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc0997c0 con 0x7f27bc09ea50 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27c111d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27c111d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 0x7f27bc05e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc11c9b0 con 0x7f27bc05e310 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f27ac01b400 con 0x7f27bc09ea50 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b37fe640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f279c038740 0x7f279c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f27ac04d660 con 0x7f27bc09ea50 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 0x7f27bc05e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient: get_auth_request con 0x7f27bc05e310 auth_method 0 2024-09-15T03:36:48.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27bb59e640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f279c038740 0x7f279c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27bb59e640 10 monclient: get_auth_request con 0x7f279c038740 auth_method 0 2024-09-15T03:36:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 10 monclient(hunting): handle_auth_done global_id 104395 payload 824 2024-09-15T03:36:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.244+0000 7f27b259c640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27bc04bf60 con 0x7f27bc05e310 2024-09-15T03:36:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b259c640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 msgr2=0x7f27bc05e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b259c640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 0x7f27bc05e6e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f27a4011bf0 tx=0x7f27a40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27bb59e640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f279c038740 0x7f279c03abf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27bc076600 tx=0x7f27b4037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4014440 con 0x7f27bc05e310 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 10 monclient: dump: 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f27a40145a0 con 0x7f27bc05e310 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 msgr2=0x7f27bc05e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 0x7f27bc05e6e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f27a4011bf0 tx=0x7f27a40090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 msgr2=0x7f27bc05e6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:48.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.247+0000 7f27b37fe640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc05e310 0x7f27bc05e6e0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f27a4011bf0 tx=0x7f27a40090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27c111d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:36:48.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27c111d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 0x7f27bc029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27c111d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc065e00 con 0x7f27bc0298c0 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 0x7f27bc029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient: get_auth_request con 0x7f27bc0298c0 auth_method 0 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 10 monclient(hunting): handle_auth_done global_id 104398 payload 824 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.253+0000 7f27b259c640 1 -- 172.21.15.153:0/3557462360 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f27bc11caf0 con 0x7f27bc0298c0 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b259c640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 msgr2=0x7f27bc029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b259c640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 0x7f27bc029cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098b0 tx=0x7f27a401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a401de10 con 0x7f27bc0298c0 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 10 monclient: dump: 2024-09-15T03:36:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f27a401c3e0 con 0x7f27bc0298c0 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:36:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 msgr2=0x7f27bc029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 0x7f27bc029cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098b0 tx=0x7f27a401c000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 msgr2=0x7f27bc029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:36:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.256+0000 7f27b37fe640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27bc0298c0 0x7f27bc029cc0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098b0 tx=0x7f27a401c000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.279 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 92, 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:36:48.280 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:48.281 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:48.282 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:48.283 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:36:48.284 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:36:48.285 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:36:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:48.288 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:36:48.289 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:36:48.291 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:36:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:48.294 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:36:48.295 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:36:48.296 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f279c038740 msgr2=0x7f279c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f279c038740 0x7f279c03abf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27bc076600 tx=0x7f27b4037000 comp rx=0 tx=0).stop 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 10 monclient: shutdown 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 msgr2=0x7f27bc11daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 --2- 172.21.15.153:0/3557462360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc09ea50 0x7f27bc11daa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00c5e0 tx=0x7f27ac004960 comp rx=0 tx=0).stop 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27bb59e640 1 -- 172.21.15.153:0/3557462360 reap_dead start 2024-09-15T03:36:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27bb59e640 1 -- 172.21.15.153:0/3557462360 reap_dead start 2024-09-15T03:36:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 shutdown_connections 2024-09-15T03:36:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 >> 172.21.15.153:0/3557462360 conn(0x7f27bc067110 msgr2=0x7f27bc0a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 shutdown_connections 2024-09-15T03:36:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.257+0000 7f27c111d640 1 -- 172.21.15.153:0/3557462360 wait complete. 2024-09-15T03:36:48.298 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:36:48.298 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:36:48.299 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:36:48.299 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:36:48.299 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:36:48.299 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:36:48.299 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:36:48.312 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:36:48.313 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:36:48.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:48.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.444+0000 7f181875d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:48.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.444+0000 7f181875d640 10 monclient: build_initial_monmap 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.444+0000 7f181875d640 10 monclient: monmap: 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:48.445590+0000 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:48.445590+0000 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:48.446 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:48.447 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:48.447 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:48.447 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.445+0000 7f181875d640 1 Processor -- start 2024-09-15T03:36:48.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.445+0000 7f181875d640 1 -- start start 2024-09-15T03:36:48.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.445+0000 7f181875d640 10 monclient: init 2024-09-15T03:36:48.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:48.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f181009c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): picked mon.noname-b con 0x7f181006eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f181009ca80 0x7f181006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): picked mon.noname-c con 0x7f181009ca80 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f181006f490 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1810069b60 con 0x7f181006f490 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181006a780 con 0x7f181009ca80 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1810099ed0 con 0x7f181006eb80 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f18164f9640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 msgr2=0x7f181009c540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:36:48.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f181875d640 10 monclient(hunting): authenticate will time out at 1206.085297s 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f18164f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f181009c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1816cfa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f181006f490 0x7f181009d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35948/0 (socket says 172.21.15.153:35948) 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1816cfa640 1 -- 172.21.15.153:0/1742079188 learned_addr learned my addr 172.21.15.153:0/1742079188 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 1 --2- 172.21.15.153:0/1742079188 >> v2:172.21.15.181:3300/0 conn(0x7f181009ca80 0x7f181006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): get_auth_request con 0x7f181009ca80 auth_method 0 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient(hunting): handle_auth_done global_id 104401 payload 274 2024-09-15T03:36:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 1 -- 172.21.15.153:0/1742079188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 msgr2=0x7f181009c540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:48.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 1 --2- 172.21.15.153:0/1742079188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f181009c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 1 -- 172.21.15.153:0/1742079188 >> v1:172.21.15.153:6789/0 conn(0x7f181006f490 legacy=0x7f181009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:48.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient: _finish_hunting 0 2024-09-15T03:36:48.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 1 monclient: found mon.noname-c 2024-09-15T03:36:48.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:48.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:48.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.446+0000 7f1815cf8640 1 -- 172.21.15.153:0/1742079188 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1810068c00 con 0x7f181009ca80 2024-09-15T03:36:48.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.449+0000 7f1815cf8640 1 -- 172.21.15.153:0/1742079188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181009ca80 msgr2=0x7f181006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.449+0000 7f1815cf8640 1 --2- 172.21.15.153:0/1742079188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181009ca80 0x7f181006ef50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1800009850 tx=0x7f18000090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f364f01a020e8150 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 1 -- 172.21.15.153:0/1742079188 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800032440 con 0x7f181009ca80 2024-09-15T03:36:48.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:48.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: dump: 2024-09-15T03:36:48.467 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.469 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.469 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.469 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.469 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: _finish_auth 0 2024-09-15T03:36:48.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:48.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 1 -- 172.21.15.153:0/1742079188 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18000325a0 con 0x7f181009ca80 2024-09-15T03:36:48.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 5 monclient: authenticate success, global_id 104401 2024-09-15T03:36:48.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:36:48.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:48.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:48.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 4 set_mon_vals no callback set 2024-09-15T03:36:48.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 1 -- 172.21.15.153:0/1742079188 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800032940 con 0x7f181009ca80 2024-09-15T03:36:48.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:48.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f18154f7640 10 monclient: dump: 2024-09-15T03:36:48.472 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.472 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.472 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.472 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.472 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.473 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.473 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.473 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.473 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.473 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 10 monclient: shutdown 2024-09-15T03:36:48.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:48.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 -- 172.21.15.153:0/1742079188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181009ca80 msgr2=0x7f181006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 --2- 172.21.15.153:0/1742079188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181009ca80 0x7f181006ef50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1800009850 tx=0x7f18000090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 -- 172.21.15.153:0/1742079188 shutdown_connections 2024-09-15T03:36:48.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 --2- 172.21.15.153:0/1742079188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181009ca80 0x7f181006ef50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 --2- 172.21.15.153:0/1742079188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f181009c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 -- 172.21.15.153:0/1742079188 >> 172.21.15.153:0/1742079188 conn(0x7f1810067cd0 msgr2=0x7f18100a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:48.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 -- 172.21.15.153:0/1742079188 shutdown_connections 2024-09-15T03:36:48.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.450+0000 7f181875d640 1 -- 172.21.15.153:0/1742079188 wait complete. 2024-09-15T03:36:48.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.451+0000 7f181875d640 10 monclient: build_initial_monmap 2024-09-15T03:36:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.451+0000 7f181875d640 10 monclient: monmap: 2024-09-15T03:36:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:48.477 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.477 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:48.477 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:48.477 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:48.477 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.451+0000 7f181875d640 1 Processor -- start 2024-09-15T03:36:48.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.451+0000 7f181875d640 1 -- start start 2024-09-15T03:36:48.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.451+0000 7f181875d640 10 monclient: init 2024-09-15T03:36:48.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:48.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:36:48.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f1810140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): picked mon.noname-c con 0x7f181006eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:48.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181006f490 0x7f1810133d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): picked mon.noname-b con 0x7f181006f490 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:48.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 0x7f181011d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f18164f9640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 msgr2=0x7f1810140fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:36:48.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): picked mon.noname-a con 0x7f181009ca80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:48.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810121110 con 0x7f181009ca80 2024-09-15T03:36:48.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181011f3d0 con 0x7f181006f490 2024-09-15T03:36:48.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:48.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f18164f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f1810140fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:48.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1810120e30 con 0x7f181006eb80 2024-09-15T03:36:48.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:48.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f181875d640 10 monclient(hunting): authenticate will time out at 1206.091121s 2024-09-15T03:36:48.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 0x7f181011d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 0x7f181011d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60746/0 (socket says 172.21.15.153:60746) 2024-09-15T03:36:48.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181006f490 0x7f1810133d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 -- 172.21.15.153:0/1870857470 learned_addr learned my addr 172.21.15.153:0/1870857470 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:48.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): get_auth_request con 0x7f181009ca80 auth_method 0 2024-09-15T03:36:48.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): get_auth_request con 0x7f181006f490 auth_method 0 2024-09-15T03:36:48.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:48.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:48.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:48.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:48.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:48.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:48.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient(hunting): handle_auth_done global_id 114204 payload 824 2024-09-15T03:36:48.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 -- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 msgr2=0x7f1810140fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:36:48.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f1810140fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 -- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181006f490 msgr2=0x7f1810133d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181006f490 0x7f1810133d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:36:48.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 monclient: found mon.noname-a 2024-09-15T03:36:48.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:48.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1816cfa640 1 -- 172.21.15.153:0/1870857470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800009390 con 0x7f181009ca80 2024-09-15T03:36:48.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.452+0000 7f1815cf8640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181006f490 0x7f1810133d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:36:48.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.454+0000 7f1816cfa640 1 -- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 msgr2=0x7f181011d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:48.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.454+0000 7f1816cfa640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 0x7f181011d940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f180800ccc0 tx=0x7f1808005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.454+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180800ce80 con 0x7f181009ca80 2024-09-15T03:36:48.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.454+0000 7f18067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.454+0000 7f18067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:48.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.454+0000 7f18067fc640 10 monclient: dump: 2024-09-15T03:36:48.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.491 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 5 monclient: authenticate success, global_id 114204 2024-09-15T03:36:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 monclient: _renew_subs 2024-09-15T03:36:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180800aa60 con 0x7f181009ca80 2024-09-15T03:36:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18101213f0 con 0x7f181009ca80 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180800ae20 con 0x7f181009ca80 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 10 monclient: dump: 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f1814cf6640 4 set_mon_vals no callback set 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 monclient: _renew_subs 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1810141500 con 0x7f181009ca80 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1808007300 con 0x7f181009ca80 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.455+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181004bf60 con 0x7f181009ca80 2024-09-15T03:36:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.456+0000 7f18067fc640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17f40386f0 0x7f17f403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.456+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f180804dc20 con 0x7f181009ca80 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.456+0000 7f18164f9640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17f40386f0 0x7f17f403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.456+0000 7f18164f9640 10 monclient: get_auth_request con 0x7f17f40386f0 auth_method 0 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.458+0000 7f18164f9640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17f40386f0 0x7f17f403aba0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f180c003940 tx=0x7f180c006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.460+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f18080075b0 con 0x7f181009ca80 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.460+0000 7f18067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.460+0000 7f18067fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f181875d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f181875d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f181875d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f18100638d0 con 0x7f181009ca80 2024-09-15T03:36:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f18067fc640 1 -- 172.21.15.153:0/1870857470 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1808010af0 con 0x7f181009ca80 2024-09-15T03:36:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f18067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.672+0000 7f18067fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:48.674 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:48.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17f40386f0 msgr2=0x7f17f403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17f40386f0 0x7f17f403aba0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f180c003940 tx=0x7f180c006d80 comp rx=0 tx=0).stop 2024-09-15T03:36:48.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 10 monclient: shutdown 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 msgr2=0x7f181011d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 0x7f181011d940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f180800ccc0 tx=0x7f1808005370 comp rx=0 tx=0).stop 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 shutdown_connections 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17f40386f0 0x7f17f403aba0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f181009ca80 0x7f181011d940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f181006f490 0x7f1810133d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 --2- 172.21.15.153:0/1870857470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181006eb80 0x7f1810140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 >> 172.21.15.153:0/1870857470 conn(0x7f1810067cd0 msgr2=0x7f181006d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 shutdown_connections 2024-09-15T03:36:48.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:48.674+0000 7f181875d640 1 -- 172.21.15.153:0/1870857470 wait complete. 2024-09-15T03:36:50.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:50.532+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:50.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:50.532+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:50.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:50.532+0000 7feeddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:51.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:51.532+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:51.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:51.532+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:51.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.790+0000 7f3884d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.790+0000 7f3884d9d640 10 monclient: build_initial_monmap 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.790+0000 7f3884d9d640 10 monclient: monmap: 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:51.791210+0000 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:51.791210+0000 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:51.791 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:51.792 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:51.792 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:51.792 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:51.792 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:51.792 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:51.792 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:51.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 1 Processor -- start 2024-09-15T03:36:51.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 1 -- start start 2024-09-15T03:36:51.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient: init 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f388009a510 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3880097370 0x7f38801349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3880097370 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3880097770 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388009cb00 con 0x7f3880097770 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388009a8e0 con 0x7f388009a510 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388006ed80 con 0x7f3880097370 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:51.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f3884d9d640 10 monclient(hunting): authenticate will time out at 1209.430805s 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f387ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3880097770 0x7f3880136ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35962/0 (socket says 172.21.15.153:35962) 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.791+0000 7f387ed9d640 1 -- 172.21.15.153:0/4092006317 learned_addr learned my addr 172.21.15.153:0/4092006317 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 1 --2- 172.21.15.153:0/4092006317 >> v2:172.21.15.181:3300/0 conn(0x7f3880097370 0x7f38801349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): get_auth_request con 0x7f3880097370 auth_method 0 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient(hunting): handle_auth_done global_id 104407 payload 274 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 1 -- 172.21.15.153:0/4092006317 >> v1:172.21.15.181:6789/0 conn(0x7f388009a510 legacy=0x7f3880096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 1 -- 172.21.15.153:0/4092006317 >> v1:172.21.15.153:6789/0 conn(0x7f3880097770 legacy=0x7f3880136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:36:51.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:36:51.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 1 monclient: found mon.noname-c 2024-09-15T03:36:51.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:51.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:36:51.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.792+0000 7f387dd9b640 1 -- 172.21.15.153:0/4092006317 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388009c820 con 0x7f3880097370 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.795+0000 7f387dd9b640 1 -- 172.21.15.153:0/4092006317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3880097370 msgr2=0x7f38801349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.795+0000 7f387dd9b640 1 --2- 172.21.15.153:0/4092006317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3880097370 0x7f38801349a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3868009880 tx=0x7f38680090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8f24eb7545626fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.795+0000 7f387d59a640 1 -- 172.21.15.153:0/4092006317 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868032440 con 0x7f3880097370 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: dump: 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.801 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: _finish_auth 0 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 1 -- 172.21.15.153:0/4092006317 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38680325a0 con 0x7f3880097370 2024-09-15T03:36:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 1 -- 172.21.15.153:0/4092006317 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868032940 con 0x7f3880097370 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 5 monclient: authenticate success, global_id 104407 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 4 set_mon_vals no callback set 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f387d59a640 10 monclient: dump: 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 10 monclient: shutdown 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 -- 172.21.15.153:0/4092006317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3880097370 msgr2=0x7f38801349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 --2- 172.21.15.153:0/4092006317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3880097370 0x7f38801349a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3868009880 tx=0x7f38680090f0 comp rx=0 tx=0).stop 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 -- 172.21.15.153:0/4092006317 shutdown_connections 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 --2- 172.21.15.153:0/4092006317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3880097370 0x7f38801349a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 -- 172.21.15.153:0/4092006317 >> 172.21.15.153:0/4092006317 conn(0x7f38800672b0 msgr2=0x7f38800676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 -- 172.21.15.153:0/4092006317 shutdown_connections 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.796+0000 7f3884d9d640 1 -- 172.21.15.153:0/4092006317 wait complete. 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.797+0000 7f3884d9d640 10 monclient: build_initial_monmap 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.797+0000 7f3884d9d640 10 monclient: monmap: 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.797+0000 7f3884d9d640 1 Processor -- start 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.797+0000 7f3884d9d640 1 -- start start 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.797+0000 7f3884d9d640 10 monclient: init 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 0x7f388013ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3880097370 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3880097770 0x7f388012f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:51.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3880097770 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f388009a510 0x7f38801194e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f388009a510 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388009c370 con 0x7f3880097370 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388006da60 con 0x7f388009a510 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388011cdd0 con 0x7f3880097770 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f3884d9d640 10 monclient(hunting): authenticate will time out at 1209.437171s 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 0x7f388013ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 0x7f388013ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60762/0 (socket says 172.21.15.153:60762) 2024-09-15T03:36:51.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 -- 172.21.15.153:0/674482329 learned_addr learned my addr 172.21.15.153:0/674482329 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): get_auth_request con 0x7f3880097370 auth_method 0 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f388009a510 0x7f38801194e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): get_auth_request con 0x7f388009a510 auth_method 0 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:51.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient(hunting): handle_auth_done global_id 114207 payload 824 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 -- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3880097770 msgr2=0x7f388012f9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3880097770 0x7f388012f9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 -- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f388009a510 msgr2=0x7f38801194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f388009a510 0x7f38801194e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 monclient: found mon.noname-a 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:36:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.798+0000 7f387e59c640 1 -- 172.21.15.153:0/674482329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868009390 con 0x7f3880097370 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.800+0000 7f387e59c640 1 -- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 msgr2=0x7f388013ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.800+0000 7f387e59c640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 0x7f388013ef30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f387800c8b0 tx=0x7f38780026c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387800cea0 con 0x7f3880097370 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: dump: 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:51.813 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: _finish_auth 0 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387800aa00 con 0x7f3880097370 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 5 monclient: authenticate success, global_id 114207 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 10 monclient: _renew_subs 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388013f900 con 0x7f3880097370 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387800adc0 con 0x7f3880097370 2024-09-15T03:36:51.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 10 monclient: dump: 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f387cd99640 4 set_mon_vals no callback set 2024-09-15T03:36:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 10 monclient: _renew_subs 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388013f470 con 0x7f3880097370 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.801+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3878012630 con 0x7f3880097370 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f3884d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f3884d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f3884d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388004bf60 con 0x7f3880097370 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f38767fc640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3860038740 0x7f386003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f387804df50 con 0x7f3880097370 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f387dd9b640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3860038740 0x7f386003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.802+0000 7f387dd9b640 10 monclient: get_auth_request con 0x7f3860038740 auth_method 0 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.805+0000 7f387dd9b640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3860038740 0x7f386003abf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3880130680 tx=0x7f3868032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.806+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3878022d70 con 0x7f3880097370 2024-09-15T03:36:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.806+0000 7f38767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:36:51.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:51.806+0000 7f38767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:36:52.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.019+0000 7f3884d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:36:52.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.019+0000 7f3884d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:52.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.019+0000 7f3884d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:36:52.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.019+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f388011c760 con 0x7f3880097370 2024-09-15T03:36:52.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.020+0000 7f38767fc640 1 -- 172.21.15.153:0/674482329 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f3878016020 con 0x7f3880097370 2024-09-15T03:36:52.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.020+0000 7f38767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:36:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.020+0000 7f38767fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:36:52.022 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3860038740 msgr2=0x7f386003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3860038740 0x7f386003abf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3880130680 tx=0x7f3868032000 comp rx=0 tx=0).stop 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 10 monclient: shutdown 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 msgr2=0x7f388013ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 0x7f388013ef30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f387800c8b0 tx=0x7f38780026c0 comp rx=0 tx=0).stop 2024-09-15T03:36:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 shutdown_connections 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3860038740 0x7f386003abf0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f388009a510 0x7f38801194e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3880097770 0x7f388012f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 --2- 172.21.15.153:0/674482329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3880097370 0x7f388013ef30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 >> 172.21.15.153:0/674482329 conn(0x7f38800672b0 msgr2=0x7f388009fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 shutdown_connections 2024-09-15T03:36:52.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:52.022+0000 7f3884d9d640 1 -- 172.21.15.153:0/674482329 wait complete. 2024-09-15T03:36:52.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:52.532+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:52.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:52.532+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:53.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:53.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:54.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:54.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:54.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:54.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:55.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.141+0000 7fa3ece34640 10 monclient: get_monmap_and_config 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.141+0000 7fa3ece34640 10 monclient: build_initial_monmap 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.141+0000 7fa3ece34640 10 monclient: monmap: 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:36:55.141816+0000 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:36:55.141816+0000 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:36:55.142 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:36:55.143 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:36:55.143 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:36:55.143 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:36:55.143 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:36:55.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.141+0000 7fa3ece34640 1 Processor -- start 2024-09-15T03:36:55.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.141+0000 7fa3ece34640 1 -- start start 2024-09-15T03:36:55.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.141+0000 7fa3ece34640 10 monclient: init 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient: _reopen_session rank -1 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3e8096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e809a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3e80971e0 addr v1:172.21.15.153:6789/0 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3e80975b0 addr v1:172.21.15.181:6789/0 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e806ee10 con 0x7fa3e80971e0 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e809cb20 con 0x7fa3e80975b0 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e809a900 con 0x7fa3e809a510 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3ece34640 10 monclient(hunting): authenticate will time out at 1212.781426s 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3e659c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3e8096ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:36:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3e8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:55.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3e5d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa3e80971e0 0x7fa3e812f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37980/0 (socket says 172.21.15.153:37980) 2024-09-15T03:36:55.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.142+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 learned_addr learned my addr 172.21.15.153:0/3134885285 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:36:55.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.144+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v1:172.21.15.153:6789/0 conn(0x7fa3e80971e0 legacy=0x7fa3e812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:55.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.145+0000 7fa3e6d9d640 1 -- 172.21.15.153:0/3134885285 >> v1:172.21.15.181:6789/0 conn(0x7fa3e80975b0 legacy=0x7fa3e812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:36:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.343+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3e8096ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:36:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.343+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3e8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:55.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:55.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:55.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:55.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:55.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.744+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3e8096ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:36:55.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:55.744+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3e8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:56.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:56.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:56.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:56.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:56.545+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3e8096ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:36:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:56.545+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3e8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:36:57.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:57.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:57.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:57.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): tick 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 monclient(hunting): continuing hunt 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3e8096ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3e8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 >> v1:172.21.15.181:6789/0 conn(0x7fa3e80975b0 legacy=0x7fa3e812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 >> v1:172.21.15.153:6789/0 conn(0x7fa3e80971e0 legacy=0x7fa3e812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient: _add_conns ranks=[3,4] 2024-09-15T03:36:58.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c80037d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c8006290 addr v2:172.21.15.181:3300/0 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e809a900 con 0x7fa3c80037d0 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c8008df0 con 0x7fa3c8006290 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3dffff640 10 monclient(hunting): _renew_subs 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 10 monclient(hunting): get_auth_request con 0x7fa3c80037d0 auth_method 0 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:58.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:58.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.143+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 10 monclient(hunting): get_auth_request con 0x7fa3c80037d0 auth_method 0 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:58.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3c8006290 auth_method 0 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.344+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:36:58.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:58.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:58.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:58.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 10 monclient(hunting): get_auth_request con 0x7fa3c80037d0 auth_method 0 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3c8006290 auth_method 0 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:58.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:58.745+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:36:59.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:59.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:36:59.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:36:59.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:36:59.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:59.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 10 monclient(hunting): get_auth_request con 0x7fa3c80037d0 auth_method 0 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3c8006290 auth_method 0 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:36:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:36:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:36:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:36:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:36:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:36:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:36:59.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:36:59.545+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:00.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:00.533+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:37:00.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:00.533+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:00.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:00.533+0000 7feeddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:01.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): tick 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 monclient(hunting): continuing hunt 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 msgr2=0x7fa3c8008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8006290 0x7fa3c8008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 msgr2=0x7fa3c8005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.153:3300/0 conn(0x7fa3c80037d0 0x7fa3c8005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e6d9d640 1 -- 172.21.15.153:0/3134885285 reap_dead start 2024-09-15T03:37:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3c800b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e809a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c8003bd0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3c8006bf0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3c800b1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e809d590 con 0x7fa3c8006bf0 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e809a900 con 0x7fa3c8003bd0 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 1 -- 172.21.15.153:0/3134885285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c8008df0 con 0x7fa3e809a510 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3c800b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3dffff640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3c8003bd0 auth_method 0 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:01.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.142+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:01.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.144+0000 7fa3e6d9d640 1 -- 172.21.15.153:0/3134885285 >> v1:172.21.15.153:6789/0 conn(0x7fa3c8006bf0 legacy=0x7fa3c8007090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3c800b1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3c800b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3c8003bd0 auth_method 0 2024-09-15T03:37:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.343+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:01.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:01.534+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:37:01.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:01.534+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:01.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3c800b1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:01.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3c800b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:01.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:01.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3c8003bd0 auth_method 0 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:01.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:01.744+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:02.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.356+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331751102 0 0) 0x7fa3e809d590 con 0x7fa3c8006bf0 2024-09-15T03:37:02.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.356+0000 7fa3e559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.356+0000 7fa3e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.356+0000 7fa3e559a640 10 monclient(hunting): my global_id is 114210 2024-09-15T03:37:02.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.356+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d40034e0 con 0x7fa3c8006bf0 2024-09-15T03:37:02.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411598191 0 0) 0x7fa3d40034e0 con 0x7fa3c8006bf0 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3c800b1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3c800b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 msgr2=0x7fa3c80066b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 1 --2- 172.21.15.153:0/3134885285 >> v2:172.21.15.181:3300/0 conn(0x7fa3c8003bd0 0x7fa3c80066b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.357+0000 7fa3e559a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8008950 con 0x7fa3c8006bf0 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fa3e0001970 con 0x7fa3c8006bf0 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: dump: 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 >> v1:172.21.15.153:6789/0 conn(0x7fa3c8006bf0 legacy=0x7fa3c8007090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3d40039d0 0x7fa3d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): picked mon.a con 0x7fa3d40039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3d4004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): picked mon.c con 0x7fa3e809a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d4004cc0 0x7fa3d4005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): picked mon.b con 0x7fa3d4004cc0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e809a900 con 0x7fa3d40039d0 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c8008df0 con 0x7fa3d4004cc0 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d400bc10 con 0x7fa3e809a510 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e559a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3d4004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3d4004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3d40039d0 0x7fa3d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): get_auth_request con 0x7fa3d40039d0 auth_method 0 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d4004cc0 0x7fa3d4005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): get_auth_request con 0x7fa3d4004cc0 auth_method 0 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient(hunting): handle_auth_done global_id 114210 payload 293 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 msgr2=0x7fa3d4004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e809a510 0x7fa3d4004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d4004cc0 msgr2=0x7fa3d4005130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d4004cc0 0x7fa3d4005130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 monclient: found mon.a 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d40034e0 con 0x7fa3d40039d0 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3e659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3ece34640 5 monclient: authenticate success, global_id 114210 2024-09-15T03:37:02.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.358+0000 7fa3ece34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:02.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.361+0000 7fa3e659c640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3d40039d0 msgr2=0x7fa3d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.361+0000 7fa3e659c640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3d40039d0 0x7fa3d4003de0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8003940 tx=0x7fa3d8005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5276019938ef5e0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.361+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8032400 con 0x7fa3d40039d0 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: dump: 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.375 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d802e050 con 0x7fa3d40039d0 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 1 -- 172.21.15.153:0/3134885285 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80327d0 con 0x7fa3d40039d0 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 4 set_mon_vals no callback set 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e559a640 10 monclient: dump: 2024-09-15T03:37:02.376 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 10 monclient: shutdown 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:02.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 1 -- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3d40039d0 msgr2=0x7fa3d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 1 --2- 172.21.15.153:0/3134885285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3d40039d0 0x7fa3d4003de0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8003940 tx=0x7fa3d8005040 comp rx=0 tx=0).stop 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3e6d9d640 1 -- 172.21.15.153:0/3134885285 reap_dead start 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 1 -- 172.21.15.153:0/3134885285 shutdown_connections 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.362+0000 7fa3ece34640 1 -- 172.21.15.153:0/3134885285 >> 172.21.15.153:0/3134885285 conn(0x7fa3e80672b0 msgr2=0x7fa3e80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.363+0000 7fa3ece34640 1 -- 172.21.15.153:0/3134885285 shutdown_connections 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.363+0000 7fa3ece34640 1 -- 172.21.15.153:0/3134885285 wait complete. 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.363+0000 7fa3ece34640 10 monclient: build_initial_monmap 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.363+0000 7fa3ece34640 10 monclient: monmap: 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:02.378 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 Processor -- start 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 -- start start 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient: init 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:02.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d40039d0 0x7fa3e813ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): picked mon.noname-b con 0x7fa3d40039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 0x7fa3e8131d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e8131900 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e811b4f0 0x7fa3e811b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e811b4f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e811ed00 con 0x7fa3e8131900 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e809c370 con 0x7fa3d40039d0 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e813fb30 con 0x7fa3e811b4f0 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:02.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3ece34640 10 monclient(hunting): authenticate will time out at 1220.004686s 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e6d9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e811b4f0 msgr2=0x7fa3e811b990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e6d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e811b4f0 0x7fa3e811b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 0x7fa3e8131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 0x7fa3e8131d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54660/0 (socket says 172.21.15.153:54660) 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/656449078 learned_addr learned my addr 172.21.15.153:0/656449078 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request con 0x7fa3e8131900 auth_method 0 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e659c640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d40039d0 0x7fa3e813ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.364+0000 7fa3e659c640 10 monclient(hunting): get_auth_request con 0x7fa3d40039d0 auth_method 0 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 10 monclient(hunting): handle_auth_done global_id 114213 payload 824 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e811b4f0 msgr2=0x7fa3e811b990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e811b4f0 0x7fa3e811b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d40039d0 msgr2=0x7fa3e813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d40039d0 0x7fa3e813ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 1 monclient: found mon.noname-a 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.365+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/656449078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e811d410 con 0x7fa3e8131900 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3e5d9b640 1 -- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 msgr2=0x7fa3e8131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3e5d9b640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 0x7fa3e8131d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007880 tx=0x7fa3d0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0005470 con 0x7fa3e8131900 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: dump: 2024-09-15T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d000d070 con 0x7fa3e8131900 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 5 monclient: authenticate success, global_id 114213 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d00047a0 con 0x7fa3e8131900 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 10 monclient: _renew_subs 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e811dac0 con 0x7fa3e8131900 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3de7fc640 10 monclient: dump: 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.385 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3e4d99640 4 set_mon_vals no callback set 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 10 monclient: _renew_subs 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.367+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e813f1c0 con 0x7fa3e8131900 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.368+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa3d0004b50 con 0x7fa3e8131900 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3ece34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3ece34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3ece34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e811f030 con 0x7fa3e8131900 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3de7fc640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3c8039ab0 0x7fa3c803bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7fa3d004ee60 con 0x7fa3e8131900 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3e659c640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3c8039ab0 0x7fa3c803bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.369+0000 7fa3e659c640 10 monclient: get_auth_request con 0x7fa3c8039ab0 auth_method 0 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.372+0000 7fa3e659c640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3c8039ab0 0x7fa3c803bf60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4004700 tx=0x7fa3d8036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.374+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa3d0014570 con 0x7fa3e8131900 2024-09-15T03:37:02.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.374+0000 7fa3de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.374+0000 7fa3de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:02.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:02.534+0000 7feeddffb640 10 monclient: tick 2024-09-15T03:37:02.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:02.534+0000 7feeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3ece34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3ece34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3ece34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa3e804bf60 con 0x7fa3e8131900 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3de7fc640 1 -- 172.21.15.153:0/656449078 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fa3d0027020 con 0x7fa3e8131900 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:02.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.595+0000 7fa3de7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:02.598 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3c8039ab0 msgr2=0x7fa3c803bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3c8039ab0 0x7fa3c803bf60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4004700 tx=0x7fa3d8036000 comp rx=0 tx=0).stop 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 10 monclient: shutdown 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 msgr2=0x7fa3e8131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 0x7fa3e8131d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007880 tx=0x7fa3d0009040 comp rx=0 tx=0).stop 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 shutdown_connections 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3c8039ab0 0x7fa3c803bf60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3e811b4f0 0x7fa3e811b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3e8131900 0x7fa3e8131d70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 --2- 172.21.15.153:0/656449078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3d40039d0 0x7fa3e813ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 >> 172.21.15.153:0/656449078 conn(0x7fa3e80672b0 msgr2=0x7fa3e80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 shutdown_connections 2024-09-15T03:37:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.598+0000 7fa3ece34640 1 -- 172.21.15.153:0/656449078 wait complete. 2024-09-15T03:37:02.610 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:37:02.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:37:02.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:37:02.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.703+0000 7fd6ad2c9640 10 monclient: get_monmap_and_config 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.703+0000 7fd6ad2c9640 10 monclient: build_initial_monmap 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.703+0000 7fd6ad2c9640 10 monclient: monmap: 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:02.705497+0000 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:02.705497+0000 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:02.706 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:02.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 Processor -- start 2024-09-15T03:37:02.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 -- start start 2024-09-15T03:37:02.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient: init 2024-09-15T03:37:02.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:02.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:37:02.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6a809c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd6a809e840 0x7fd6a809aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a809e840 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a8139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a8136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a809ec40 con 0x7fd6a809c860 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a809a9d0 con 0x7fd6a809e840 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a806a320 con 0x7fd6a8136cb0 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.704+0000 7fd6ad2c9640 10 monclient(hunting): authenticate will time out at 1220.344987s 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd6a77fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 msgr2=0x7fd6a8139090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:02.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd6a77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a8139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd6a809e840 0x7fd6a809aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd6a809e840 0x7fd6a809aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54674/0 (socket says 172.21.15.153:54674) 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 -- 172.21.15.153:0/1890097257 learned_addr learned my addr 172.21.15.153:0/1890097257 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): get_auth_request con 0x7fd6a809e840 auth_method 0 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient(hunting): handle_auth_done global_id 114216 payload 274 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 -- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 msgr2=0x7fd6a8139090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:02.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 --2- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a8139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 -- 172.21.15.153:0/1890097257 >> v1:172.21.15.181:6789/0 conn(0x7fd6a809c860 legacy=0x7fd6a809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:02.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 monclient: found mon.noname-a 2024-09-15T03:37:02.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.705+0000 7fd69ffff640 1 -- 172.21.15.153:0/1890097257 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8068a20 con 0x7fd6a809e840 2024-09-15T03:37:02.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd69ffff640 1 -- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809e840 msgr2=0x7fd6a809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd69ffff640 1 --2- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809e840 0x7fd6a809aba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd690009910 tx=0x7fd6900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f6167650a4f136 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 1 -- 172.21.15.153:0/1890097257 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690032440 con 0x7fd6a809e840 2024-09-15T03:37:02.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:02.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: dump: 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.721 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 5 monclient: authenticate success, global_id 114216 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 1 -- 172.21.15.153:0/1890097257 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6900325a0 con 0x7fd6a809e840 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 1 -- 172.21.15.153:0/1890097257 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690032940 con 0x7fd6a809e840 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 4 set_mon_vals no callback set 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6a67fc640 10 monclient: dump: 2024-09-15T03:37:02.722 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 10 monclient: shutdown 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809e840 msgr2=0x7fd6a809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809e840 0x7fd6a809aba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd690009910 tx=0x7fd6900091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/1890097257 shutdown_connections 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a8139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/1890097257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809e840 0x7fd6a809aba0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/1890097257 >> 172.21.15.153:0/1890097257 conn(0x7fd6a80a2e40 msgr2=0x7fd6a80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/1890097257 shutdown_connections 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.708+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/1890097257 wait complete. 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.709+0000 7fd6ad2c9640 10 monclient: build_initial_monmap 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.709+0000 7fd6ad2c9640 10 monclient: monmap: 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:02.724 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.709+0000 7fd6ad2c9640 1 Processor -- start 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.709+0000 7fd6ad2c9640 1 -- start start 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.709+0000 7fd6ad2c9640 10 monclient: init 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 0x7fd6a806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a809c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a809e840 0x7fd6a806e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a809e840 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a8136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a811fba0 con 0x7fd6a809c860 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8099cc0 con 0x7fd6a809e840 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8139140 con 0x7fd6a8136cb0 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6ad2c9640 10 monclient(hunting): authenticate will time out at 1220.350190s 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a77fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 msgr2=0x7fd6a806ed50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a806ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 0x7fd6a806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 0x7fd6a806e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54690/0 (socket says 172.21.15.153:54690) 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 -- 172.21.15.153:0/691063664 learned_addr learned my addr 172.21.15.153:0/691063664 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6a809c860 auth_method 0 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd69ffff640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a809e840 0x7fd6a806e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd69ffff640 10 monclient(hunting): get_auth_request con 0x7fd6a809e840 auth_method 0 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd69ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_done global_id 114219 payload 824 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 msgr2=0x7fd6a806ed50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a8136cb0 0x7fd6a806ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a809e840 msgr2=0x7fd6a806e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6a809e840 0x7fd6a806e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.710+0000 7fd6a6ffd640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd690009450 con 0x7fd6a809c860 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6a6ffd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 msgr2=0x7fd6a806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6a6ffd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 0x7fd6a806e2d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c5e0 tx=0x7fd698007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698012840 con 0x7fd6a809c860 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: dump: 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.729 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 5 monclient: authenticate success, global_id 114219 2024-09-15T03:37:02.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6980129a0 con 0x7fd6a809c860 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 10 monclient: _renew_subs 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a811ea00 con 0x7fd6a809c860 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698012da0 con 0x7fd6a809c860 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd69effd640 10 monclient: dump: 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 10 monclient: _renew_subs 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a8136950 con 0x7fd6a809c860 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.712+0000 7fd6a5ffb640 4 set_mon_vals no callback set 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd698011590 con 0x7fd6a809c860 2024-09-15T03:37:02.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd6ad2c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 0x7fd6a8137080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8065e00 con 0x7fd6a806f290 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 0x7fd6a8137080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient: get_auth_request con 0x7fd6a806f290 auth_method 0 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69effd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd688038720 0x7fd68803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.713+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.714+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7fd69804dc50 con 0x7fd6a809c860 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.714+0000 7fd69ffff640 10 monclient(hunting): handle_auth_done global_id 114222 payload 824 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.714+0000 7fd69ffff640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6a811fde0 con 0x7fd6a806f290 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.714+0000 7fd6a77fe640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd688038720 0x7fd68803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.714+0000 7fd6a77fe640 10 monclient: get_auth_request con 0x7fd688038720 auth_method 0 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69ffff640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 msgr2=0x7fd6a8137080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69ffff640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 0x7fd6a8137080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd690034000 tx=0x7fd690032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69003c940 con 0x7fd6a806f290 2024-09-15T03:37:02.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 10 monclient: dump: 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.735 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd6a77fe640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd688038720 0x7fd68803abd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0003cc0 tx=0x7fd6a0009370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd690040070 con 0x7fd6a806f290 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 msgr2=0x7fd6a8137080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 0x7fd6a8137080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd690034000 tx=0x7fd690032000 comp rx=0 tx=0).stop 2024-09-15T03:37:02.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 msgr2=0x7fd6a8137080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:02.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.716+0000 7fd69effd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a806f290 0x7fd6a8137080 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd6ad2c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 0x7fd6a81379c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd6ad2c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a811fa00 con 0x7fd6a81375c0 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 0x7fd6a81379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient: get_auth_request con 0x7fd6a81375c0 auth_method 0 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 10 monclient(hunting): handle_auth_done global_id 114225 payload 824 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.722+0000 7fd69ffff640 1 -- 172.21.15.153:0/691063664 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6a804bf60 con 0x7fd6a81375c0 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69ffff640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 msgr2=0x7fd6a81379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69ffff640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 0x7fd6a81379c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6a811fd10 tx=0x7fd69003e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69003b8d0 con 0x7fd6a81375c0 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 10 monclient: dump: 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fd69003c730 con 0x7fd6a81375c0 2024-09-15T03:37:02.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:02.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:02.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 msgr2=0x7fd6a81379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 0x7fd6a81379c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6a811fd10 tx=0x7fd69003e070 comp rx=0 tx=0).stop 2024-09-15T03:37:02.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 msgr2=0x7fd6a81379c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:02.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd69effd640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a81375c0 0x7fd6a81379c0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6a811fd10 tx=0x7fd69003e070 comp rx=0 tx=0).stop 2024-09-15T03:37:02.744 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 96, 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:02.745 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:02.746 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.747 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:02.748 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:02.749 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:02.749 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.749 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:02.749 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:02.749 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:02.750 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:02.751 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:02.751 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:02.751 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:02.751 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:02.751 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:02.751 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:02.752 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:02.753 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:02.753 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:02.753 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:02.753 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.753 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:02.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.754 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:02.754 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.754 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:02.754 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:02.754 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:02.754 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:02.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.755 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:02.755 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:02.755 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.756 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.757 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:02.758 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.760 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:02.760 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:02.760 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.760 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.760 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.761 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:02.761 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:02.761 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:02.761 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:02.761 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.762 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:02.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:02.764 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd688038720 msgr2=0x7fd68803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd688038720 0x7fd68803abd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0003cc0 tx=0x7fd6a0009370 comp rx=0 tx=0).stop 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 10 monclient: shutdown 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 msgr2=0x7fd6a806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 --2- 172.21.15.153:0/691063664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a809c860 0x7fd6a806e2d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c5e0 tx=0x7fd698007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6a77fe640 1 -- 172.21.15.153:0/691063664 reap_dead start 2024-09-15T03:37:02.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6a77fe640 1 -- 172.21.15.153:0/691063664 reap_dead start 2024-09-15T03:37:02.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 shutdown_connections 2024-09-15T03:37:02.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 >> 172.21.15.153:0/691063664 conn(0x7fd6a80a2e40 msgr2=0x7fd6a809b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:02.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 shutdown_connections 2024-09-15T03:37:02.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.725+0000 7fd6ad2c9640 1 -- 172.21.15.153:0/691063664 wait complete. 2024-09-15T03:37:02.768 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:37:02.769 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:37:02.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.862+0000 7f6a5aa89640 10 monclient: get_monmap_and_config 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.862+0000 7f6a5aa89640 10 monclient: build_initial_monmap 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.862+0000 7f6a5aa89640 10 monclient: monmap: 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:02.864099+0000 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:02.864099+0000 2024-09-15T03:37:02.864 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:02.865 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 Processor -- start 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 -- start start 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient: init 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a540974f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a5409a8a0 0x7f6a5409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): picked mon.noname-b con 0x7f6a5409a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6a54133070 0x7f6a54135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): picked mon.noname-a con 0x7f6a54133070 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5412b210 con 0x7f6a540974f0 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a540978c0 con 0x7f6a54133070 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5409ca20 con 0x7f6a5409a8a0 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a5aa89640 10 monclient(hunting): authenticate will time out at 1220.503629s 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a53fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a5409a8a0 msgr2=0x7f6a5409acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:02.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a53fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a5409a8a0 0x7f6a5409acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6a54133070 0x7f6a54135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6a54133070 0x7f6a54135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54714/0 (socket says 172.21.15.153:54714) 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 1 -- 172.21.15.153:0/378762570 learned_addr learned my addr 172.21.15.153:0/378762570 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): get_auth_request con 0x7f6a54133070 auth_method 0 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.863+0000 7f6a59026640 10 monclient(hunting): handle_auth_done global_id 114228 payload 274 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 1 -- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a5409a8a0 msgr2=0x7f6a5409acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:02.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 1 --2- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a5409a8a0 0x7f6a5409acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 1 -- 172.21.15.153:0/378762570 >> v1:172.21.15.181:6789/0 conn(0x7f6a540974f0 legacy=0x7f6a5412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 1 monclient: found mon.noname-a 2024-09-15T03:37:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.864+0000 7f6a59026640 1 -- 172.21.15.153:0/378762570 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5409b1f0 con 0x7f6a54133070 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.866+0000 7f6a59026640 1 -- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a54133070 msgr2=0x7f6a54135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.866+0000 7f6a59026640 1 --2- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a54133070 0x7f6a54135450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009f20 tx=0x7f6a4c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4b672b439a8788 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 1 -- 172.21.15.153:0/378762570 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c007610 con 0x7f6a54133070 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: dump: 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.878 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 1 -- 172.21.15.153:0/378762570 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c02d050 con 0x7f6a54133070 2024-09-15T03:37:02.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 5 monclient: authenticate success, global_id 114228 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 1 -- 172.21.15.153:0/378762570 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0079e0 con 0x7f6a54133070 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 4 set_mon_vals no callback set 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a537fe640 10 monclient: dump: 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.880 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 10 monclient: shutdown 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 -- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a54133070 msgr2=0x7f6a54135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a54133070 0x7f6a54135450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009f20 tx=0x7f6a4c009840 comp rx=0 tx=0).stop 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 -- 172.21.15.153:0/378762570 shutdown_connections 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a54133070 0x7f6a54135450 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/378762570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a5409a8a0 0x7f6a5409acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 -- 172.21.15.153:0/378762570 >> 172.21.15.153:0/378762570 conn(0x7f6a540672b0 msgr2=0x7f6a540676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 -- 172.21.15.153:0/378762570 shutdown_connections 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.867+0000 7f6a5aa89640 1 -- 172.21.15.153:0/378762570 wait complete. 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.868+0000 7f6a5aa89640 10 monclient: build_initial_monmap 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.868+0000 7f6a5aa89640 10 monclient: monmap: 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:02.882 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.868+0000 7f6a5aa89640 1 Processor -- start 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.868+0000 7f6a5aa89640 1 -- start start 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.868+0000 7f6a5aa89640 10 monclient: init 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540974f0 0x7f6a5406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): picked mon.noname-b con 0x7f6a540974f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 0x7f6a5406a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): picked mon.noname-a con 0x7f6a5409a8a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:02.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a54133070 0x7f6a5406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): picked mon.noname-c con 0x7f6a54133070 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5406e890 con 0x7f6a5409a8a0 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5406d640 con 0x7f6a540974f0 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5406c490 con 0x7f6a54133070 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a5aa89640 10 monclient(hunting): authenticate will time out at 1220.509244s 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a59026640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a54133070 msgr2=0x7f6a5406abb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a59026640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a54133070 0x7f6a5406abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 0x7f6a5406a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 0x7f6a5406a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54722/0 (socket says 172.21.15.153:54722) 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 -- 172.21.15.153:0/3266894143 learned_addr learned my addr 172.21.15.153:0/3266894143 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): get_auth_request con 0x7f6a5409a8a0 auth_method 0 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540974f0 0x7f6a5406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): get_auth_request con 0x7f6a540974f0 auth_method 0 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient(hunting): handle_auth_done global_id 114231 payload 824 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a54133070 msgr2=0x7f6a5406abb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a54133070 0x7f6a5406abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540974f0 msgr2=0x7f6a5406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540974f0 0x7f6a5406e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:02.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 monclient: found mon.noname-a 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.869+0000 7f6a53fff640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c009ae0 con 0x7f6a5409a8a0 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a53fff640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 msgr2=0x7f6a5406a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a53fff640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 0x7f6a5406a670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000b5f0 tx=0x7f6a40009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40015070 con 0x7f6a5409a8a0 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 10 monclient: dump: 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.887 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40003d90 con 0x7f6a5409a8a0 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.871+0000 7f6a5aa89640 5 monclient: authenticate success, global_id 114231 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 10 monclient: _renew_subs 2024-09-15T03:37:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5406bd10 con 0x7f6a5409a8a0 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a50ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a400063a0 con 0x7f6a5409a8a0 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a50ff9640 10 monclient: dump: 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 10 monclient: _renew_subs 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5406f4d0 con 0x7f6a5409a8a0 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a52ffd640 4 set_mon_vals no callback set 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 0x7f6a5406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54065f20 con 0x7f6a540a05d0 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.872+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6a40010440 con 0x7f6a5409a8a0 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 0x7f6a5406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a50ff9640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6a380386f0 0x7f6a3803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient: get_auth_request con 0x7f6a540a05d0 auth_method 0 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 106..207) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f6a4004d550 con 0x7f6a5409a8a0 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a59026640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6a380386f0 0x7f6a3803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a59026640 10 monclient: get_auth_request con 0x7f6a380386f0 auth_method 0 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 10 monclient(hunting): handle_auth_done global_id 104422 payload 824 2024-09-15T03:37:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.873+0000 7f6a58825640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a5406da90 con 0x7f6a540a05d0 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a58825640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 msgr2=0x7f6a5406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a58825640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 0x7f6a5406b0f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800f780 tx=0x7f6a48007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a59026640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6a380386f0 0x7f6a3803aba0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c033000 tx=0x7f6a4c036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4800d8b0 con 0x7f6a540a05d0 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.875+0000 7f6a50ff9640 10 monclient: dump: 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.892 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6a4800da10 con 0x7f6a540a05d0 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 msgr2=0x7f6a5406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 0x7f6a5406b0f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800f780 tx=0x7f6a48007e10 comp rx=0 tx=0).stop 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 msgr2=0x7f6a5406b0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.876+0000 7f6a50ff9640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a540a05d0 0x7f6a5406b0f0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800f780 tx=0x7f6a48007e10 comp rx=0 tx=0).stop 2024-09-15T03:37:02.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.881+0000 7f6a5aa89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.881+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 0x7f6a54126e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a5aa89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5404bf60 con 0x7f6a54126a90 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 0x7f6a54126e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient: get_auth_request con 0x7f6a54126a90 auth_method 0 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:02.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 10 monclient(hunting): handle_auth_done global_id 104425 payload 824 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.882+0000 7f6a58825640 1 -- 172.21.15.153:0/3266894143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6a540638d0 con 0x7f6a54126a90 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.884+0000 7f6a58825640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 msgr2=0x7f6a54126e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.884+0000 7f6a58825640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 0x7f6a54126e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800f780 tx=0x7f6a48019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.884+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48014e20 con 0x7f6a54126a90 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 10 monclient: dump: 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.897 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f6a4801d610 con 0x7f6a54126a90 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 msgr2=0x7f6a54126e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 0x7f6a54126e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800f780 tx=0x7f6a48019000 comp rx=0 tx=0).stop 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 msgr2=0x7f6a54126e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a50ff9640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a54126a90 0x7f6a54126e90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 96, 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:37:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:02.901 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:02.902 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:02.903 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:02.904 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.905 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:02.906 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:02.908 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.909 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.910 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.911 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:02.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:02.913 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:02.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:02.913 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:02.913 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:02.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6a380386f0 msgr2=0x7f6a3803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6a380386f0 0x7f6a3803aba0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c033000 tx=0x7f6a4c036000 comp rx=0 tx=0).stop 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 10 monclient: shutdown 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 msgr2=0x7f6a5406a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 0x7f6a5406a670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000b5f0 tx=0x7f6a40009120 comp rx=0 tx=0).stop 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a59026640 1 -- 172.21.15.153:0/3266894143 reap_dead start 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 shutdown_connections 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 --2- 172.21.15.153:0/3266894143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a5409a8a0 0x7f6a5406a670 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.885+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 >> 172.21.15.153:0/3266894143 conn(0x7f6a540672b0 msgr2=0x7f6a54125760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.886+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 shutdown_connections 2024-09-15T03:37:02.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:02.886+0000 7f6a5aa89640 1 -- 172.21.15.153:0/3266894143 wait complete. 2024-09-15T03:37:02.916 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:37:02.916 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:37:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.359+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v208) v1 ==== 168+0+0 (secure 0 0 0) 0x7feed0040020 con 0x7feee809eae0 2024-09-15T03:37:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.359+0000 7feedf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.359+0000 7feedf7fe640 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-15T03:37:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.457+0000 7feeed694640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.457+0000 7feeed694640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.457+0000 7feeed694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.457+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7feee804bf60 con 0x7feee809eae0 2024-09-15T03:37:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.458+0000 7feedf7fe640 1 -- 172.21.15.181:0/3974097974 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v208) v1 ==== 147+0+0 (secure 0 0 0) 0x7feed0037070 con 0x7feee809eae0 2024-09-15T03:37:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.458+0000 7feedf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.458+0000 7feedf7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-15T03:37:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h24' already exists 2024-09-15T03:37:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feec8038740 msgr2=0x7feec803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feec8038740 0x7feec803abf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7feed8007fd0 tx=0x7feed8007f00 comp rx=0 tx=0).stop 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 10 monclient: shutdown 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 msgr2=0x7feee811e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 0x7feee811e8c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feed0009850 tx=0x7feed0032000 comp rx=0 tx=0).stop 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 shutdown_connections 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feec8038740 0x7feec803abf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feee8136cc0 0x7feee811ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feee809eae0 0x7feee811e8c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 --2- 172.21.15.181:0/3974097974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee8098f00 0x7feee8135d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.460+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 >> 172.21.15.181:0/3974097974 conn(0x7feee80a3b30 msgr2=0x7feee8097810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.461+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 shutdown_connections 2024-09-15T03:37:03.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.461+0000 7feeed694640 1 -- 172.21.15.181:0/3974097974 wait complete. 2024-09-15T03:37:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.546+0000 7fa1182a8640 10 monclient: get_monmap_and_config 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.546+0000 7fa1182a8640 10 monclient: build_initial_monmap 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.546+0000 7fa1182a8640 10 monclient: monmap: 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:03.547620+0000 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:03.547620+0000 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:03.548 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.547+0000 7fa1182a8640 1 Processor -- start 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.547+0000 7fa1182a8640 1 -- start start 2024-09-15T03:37:03.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.547+0000 7fa1182a8640 10 monclient: init 2024-09-15T03:37:03.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.547+0000 7fa1182a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:03.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.547+0000 7fa1182a8640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T03:37:03.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa110096a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:03.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa110096e00 0x7fa110097200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:03.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa110096e00 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:03.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa110134c90 0x7fa110137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa110134c90 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11009d770 con 0x7fa110096a30 2024-09-15T03:37:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:03.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11009c6f0 con 0x7fa110096e00 2024-09-15T03:37:03.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:03.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11009c130 con 0x7fa110134c90 2024-09-15T03:37:03.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:03.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa1182a8640 10 monclient(hunting): authenticate will time out at 1228.029095s 2024-09-15T03:37:03.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116044640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa110096a30 0x7fa11009d060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:54424/0 (socket says 172.21.15.181:54424) 2024-09-15T03:37:03.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa110134c90 0x7fa110137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:03.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116044640 1 -- 172.21.15.181:0/533590339 learned_addr learned my addr 172.21.15.181:0/533590339 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:03.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): get_auth_request con 0x7fa110134c90 auth_method 0 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa115843640 1 --2- 172.21.15.181:0/533590339 >> v2:172.21.15.153:3300/0 conn(0x7fa110096e00 0x7fa110097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa115843640 10 monclient(hunting): get_auth_request con 0x7fa110096e00 auth_method 0 2024-09-15T03:37:03.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa115843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:03.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa115843640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:03.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa115843640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:03.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient(hunting): handle_auth_done global_id 104428 payload 274 2024-09-15T03:37:03.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 1 -- 172.21.15.181:0/533590339 >> v2:172.21.15.153:3300/0 conn(0x7fa110096e00 msgr2=0x7fa110097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:03.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 1 --2- 172.21.15.181:0/533590339 >> v2:172.21.15.153:3300/0 conn(0x7fa110096e00 0x7fa110097200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 1 -- 172.21.15.181:0/533590339 >> v1:172.21.15.181:6789/0 conn(0x7fa110096a30 legacy=0x7fa11009d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient: _finish_hunting 0 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 1 monclient: found mon.noname-c 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa116845640 1 -- 172.21.15.181:0/533590339 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11009ab30 con 0x7fa110134c90 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.548+0000 7fa115843640 1 --2- 172.21.15.181:0/533590339 >> v2:172.21.15.153:3300/0 conn(0x7fa110096e00 0x7fa110097200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa116845640 1 -- 172.21.15.181:0/533590339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110134c90 msgr2=0x7fa110137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa116845640 1 --2- 172.21.15.181:0/533590339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110134c90 0x7fa110137070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa108009e60 tx=0x7fa108009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a88205e53e9c9d3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 1 -- 172.21.15.181:0/533590339 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108007550 con 0x7fa110134c90 2024-09-15T03:37:03.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: dump: 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:03.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: _finish_auth 0 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa1182a8640 5 monclient: authenticate success, global_id 104428 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 1 -- 172.21.15.181:0/533590339 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10802d050 con 0x7fa110134c90 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa1182a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa1182a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa1182a8640 4 set_mon_vals no callback set 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 1 -- 172.21.15.181:0/533590339 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1080078e0 con 0x7fa110134c90 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.551+0000 7fa115042640 10 monclient: dump: 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:03.569 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 10 monclient: shutdown 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 -- 172.21.15.181:0/533590339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110134c90 msgr2=0x7fa110137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 --2- 172.21.15.181:0/533590339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110134c90 0x7fa110137070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa108009e60 tx=0x7fa108009780 comp rx=0 tx=0).stop 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 -- 172.21.15.181:0/533590339 shutdown_connections 2024-09-15T03:37:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 --2- 172.21.15.181:0/533590339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110134c90 0x7fa110137070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 --2- 172.21.15.181:0/533590339 >> v2:172.21.15.153:3300/0 conn(0x7fa110096e00 0x7fa110097200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 -- 172.21.15.181:0/533590339 >> 172.21.15.181:0/533590339 conn(0x7fa1100a0fe0 msgr2=0x7fa1100a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 -- 172.21.15.181:0/533590339 shutdown_connections 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 1 -- 172.21.15.181:0/533590339 wait complete. 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 10 monclient: build_initial_monmap 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.552+0000 7fa1182a8640 10 monclient: monmap: 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:03.571 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 Processor -- start 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 -- start start 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient: init 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 0x7fa110133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa110096a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110096e00 0x7fa1100984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa110096e00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa110134c90 0x7fa110098a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:03.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa110134c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa110121dc0 con 0x7fa110096e00 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11009b9d0 con 0x7fa110096a30 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11006e300 con 0x7fa110134c90 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa1182a8640 10 monclient(hunting): authenticate will time out at 1228.034688s 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 0x7fa110133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116845640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa110134c90 msgr2=0x7fa110098a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 0x7fa110133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60982/0 (socket says 172.21.15.181:60982) 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 1 -- 172.21.15.181:0/948709948 learned_addr learned my addr 172.21.15.181:0/948709948 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:03.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): get_auth_request con 0x7fa110096a30 auth_method 0 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116845640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa110134c90 0x7fa110098a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa116044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110096e00 0x7fa1100984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): get_auth_request con 0x7fa110096e00 auth_method 0 2024-09-15T03:37:03.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.553+0000 7fa115843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 10 monclient(hunting): handle_auth_done global_id 104431 payload 824 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 1 -- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa110134c90 msgr2=0x7fa110098a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa110134c90 0x7fa110098a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 1 -- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110096e00 msgr2=0x7fa1100984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110096e00 0x7fa1100984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 10 monclient: _finish_hunting 0 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 1 monclient: found mon.noname-b 2024-09-15T03:37:03.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.554+0000 7fa116044640 1 -- 172.21.15.181:0/948709948 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108009a20 con 0x7fa110096a30 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa116044640 1 -- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 msgr2=0x7fa110133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa116044640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 0x7fa110133c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c5e0 tx=0x7fa10c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c005cb0 con 0x7fa110096a30 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: dump: 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:03.576 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c005e30 con 0x7fa110096a30 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 5 monclient: authenticate success, global_id 104431 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 10 monclient: _renew_subs 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa110120ac0 con 0x7fa110096a30 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c007340 con 0x7fa110096a30 2024-09-15T03:37:03.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1067fc640 10 monclient: dump: 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 10 monclient: _renew_subs 2024-09-15T03:37:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa114841640 4 set_mon_vals no callback set 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.556+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11009a070 con 0x7fa110096a30 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa10c01b400 con 0x7fa110096a30 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1182a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1182a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1182a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa110122230 con 0x7fa110096a30 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1067fc640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0f40386f0 0x7fa0f403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(208..208 src has 106..208) v4 ==== 13204+0+0 (secure 0 0 0) 0x7fa10c04d740 con 0x7fa110096a30 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa115843640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0f40386f0 0x7fa0f403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.557+0000 7fa115843640 10 monclient: get_auth_request con 0x7fa0f40386f0 auth_method 0 2024-09-15T03:37:03.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.560+0000 7fa115843640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0f40386f0 0x7fa0f403aba0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa110122870 tx=0x7fa100008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:03.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.561+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa10c01a540 con 0x7fa110096a30 2024-09-15T03:37:03.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.561+0000 7fa1067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:03.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.561+0000 7fa1067fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:03.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.720+0000 7fa1182a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:03.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.720+0000 7fa1182a8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:03.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.720+0000 7fa1182a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:03.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:03.720+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa11004bf60 con 0x7fa110096a30 2024-09-15T03:37:04.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.362+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v209) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa10c01a360 con 0x7fa110096a30 2024-09-15T03:37:04.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.362+0000 7fa1067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.362+0000 7fa1067fc640 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-15T03:37:04.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.459+0000 7fa1182a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.459+0000 7fa1182a8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.459+0000 7fa1182a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.459+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa110097740 con 0x7fa110096a30 2024-09-15T03:37:04.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.460+0000 7fa1067fc640 1 -- 172.21.15.181:0/948709948 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v209) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa10c01a360 con 0x7fa110096a30 2024-09-15T03:37:04.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.460+0000 7fa1067fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.460+0000 7fa1067fc640 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-15T03:37:04.461 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h25' already exists 2024-09-15T03:37:04.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0f40386f0 msgr2=0x7fa0f403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0f40386f0 0x7fa0f403aba0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa110122870 tx=0x7fa100008040 comp rx=0 tx=0).stop 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 10 monclient: shutdown 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 msgr2=0x7fa110133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 0x7fa110133c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c5e0 tx=0x7fa10c004960 comp rx=0 tx=0).stop 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 shutdown_connections 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0f40386f0 0x7fa0f403aba0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa110134c90 0x7fa110098a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110096e00 0x7fa1100984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 --2- 172.21.15.181:0/948709948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa110096a30 0x7fa110133c90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 >> 172.21.15.181:0/948709948 conn(0x7fa1100a0fe0 msgr2=0x7fa1100a0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:04.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 shutdown_connections 2024-09-15T03:37:04.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.462+0000 7fa1182a8640 1 -- 172.21.15.181:0/948709948 wait complete. 2024-09-15T03:37:04.472 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:04.472 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.546+0000 7fbbfe19d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.546+0000 7fbbfe19d640 10 monclient: build_initial_monmap 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.546+0000 7fbbfe19d640 10 monclient: monmap: 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:04.547230+0000 2024-09-15T03:37:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:04.547230+0000 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:04.548 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:04.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 Processor -- start 2024-09-15T03:37:04.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 -- start start 2024-09-15T03:37:04.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient: init 2024-09-15T03:37:04.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbf806df30 0x7fbbf806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf806df30 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf806e300 0x7fbbf809dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf806e300 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbbf809e010 0x7fbbf809e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf809e010 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf806a290 con 0x7fbbf809e010 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf809f840 con 0x7fbbf806df30 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf80696e0 con 0x7fbbf806e300 2024-09-15T03:37:04.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:04.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbfe19d640 10 monclient(hunting): authenticate will time out at 1229.028689s 2024-09-15T03:37:04.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbf806df30 0x7fbbf806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:04.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf6ffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf806e300 msgr2=0x7fbbf809dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:04.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbbf806df30 0x7fbbf806a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32782/0 (socket says 172.21.15.181:32782) 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 1 -- 172.21.15.181:0/524687361 learned_addr learned my addr 172.21.15.181:0/524687361 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): get_auth_request con 0x7fbbf806df30 auth_method 0 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf6ffd640 1 --2- 172.21.15.181:0/524687361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf806e300 0x7fbbf809dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf67fc640 1 -- 172.21.15.181:0/524687361 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.547+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:04.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf7fff640 1 --2- 172.21.15.181:0/524687361 >> v2:172.21.15.153:3300/0 conn(0x7fbbf809e010 0x7fbbf809e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:04.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf7fff640 10 monclient(hunting): get_auth_request con 0x7fbbf809e010 auth_method 0 2024-09-15T03:37:04.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:04.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:04.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:04.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_done global_id 104434 payload 274 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 1 -- 172.21.15.181:0/524687361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf806e300 msgr2=0x7fbbf809dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 1 --2- 172.21.15.181:0/524687361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf806e300 0x7fbbf809dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 1 -- 172.21.15.181:0/524687361 >> v2:172.21.15.153:3300/0 conn(0x7fbbf809e010 msgr2=0x7fbbf809e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 1 --2- 172.21.15.181:0/524687361 >> v2:172.21.15.153:3300/0 conn(0x7fbbf809e010 0x7fbbf809e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 1 monclient: found mon.noname-c 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.548+0000 7fbbf77fe640 1 -- 172.21.15.181:0/524687361 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf809f600 con 0x7fbbf806df30 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.550+0000 7fbbf77fe640 1 -- 172.21.15.181:0/524687361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 msgr2=0x7fbbf806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.550+0000 7fbbf77fe640 1 --2- 172.21.15.181:0/524687361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf806a6b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80670e0 tx=0x7fbbe80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e6b16bab79e65d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 1 -- 172.21.15.181:0/524687361 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8032440 con 0x7fbbf806df30 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: dump: 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 5 monclient: authenticate success, global_id 104434 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 1 -- 172.21.15.181:0/524687361 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe80325a0 con 0x7fbbf806df30 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 1 -- 172.21.15.181:0/524687361 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8032940 con 0x7fbbf806df30 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 4 set_mon_vals no callback set 2024-09-15T03:37:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbf67fc640 10 monclient: dump: 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:04.570 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 10 monclient: shutdown 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 -- 172.21.15.181:0/524687361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 msgr2=0x7fbbf806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/524687361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf806a6b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80670e0 tx=0x7fbbe80091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 -- 172.21.15.181:0/524687361 shutdown_connections 2024-09-15T03:37:04.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/524687361 >> v2:172.21.15.153:3300/0 conn(0x7fbbf809e010 0x7fbbf809e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/524687361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf806e300 0x7fbbf809dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/524687361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf806a6b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 -- 172.21.15.181:0/524687361 >> 172.21.15.181:0/524687361 conn(0x7fbbf80a3b30 msgr2=0x7fbbf809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 -- 172.21.15.181:0/524687361 shutdown_connections 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.551+0000 7fbbfe19d640 1 -- 172.21.15.181:0/524687361 wait complete. 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.552+0000 7fbbfe19d640 10 monclient: build_initial_monmap 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.552+0000 7fbbfe19d640 10 monclient: monmap: 2024-09-15T03:37:04.572 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.552+0000 7fbbfe19d640 1 Processor -- start 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.552+0000 7fbbfe19d640 1 -- start start 2024-09-15T03:37:04.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.552+0000 7fbbfe19d640 10 monclient: init 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf8141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf806df30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf806e300 0x7fbbf8134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf806e300 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809e010 0x7fbbf811ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf809e010 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf80a21c0 con 0x7fbbf806e300 2024-09-15T03:37:04.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf81204b0 con 0x7fbbf806df30 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8121fe0 con 0x7fbbf809e010 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbfe19d640 10 monclient(hunting): authenticate will time out at 1229.034103s 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf8141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf7fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809e010 msgr2=0x7fbbf811ea00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809e010 0x7fbbf811ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf8141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32796/0 (socket says 172.21.15.181:32796) 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 -- 172.21.15.181:0/4136987186 learned_addr learned my addr 172.21.15.181:0/4136987186 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): get_auth_request con 0x7fbbf806df30 auth_method 0 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:04.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf806e300 0x7fbbf8134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): get_auth_request con 0x7fbbf806e300 auth_method 0 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:04.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient(hunting): handle_auth_done global_id 104437 payload 824 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 -- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809e010 msgr2=0x7fbbf811ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809e010 0x7fbbf811ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 -- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf806e300 msgr2=0x7fbbf8134ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf806e300 0x7fbbf8134ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 monclient: found mon.noname-b 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.553+0000 7fbbf77fe640 1 -- 172.21.15.181:0/4136987186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8009450 con 0x7fbbf806df30 2024-09-15T03:37:04.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbf77fe640 1 -- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 msgr2=0x7fbbf8141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbf77fe640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf8141080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009940 tx=0x7fbbe8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8031a40 con 0x7fbbf806df30 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: dump: 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:04.578 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 5 monclient: authenticate success, global_id 104437 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe8031bc0 con 0x7fbbf806df30 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 10 monclient: _renew_subs 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf81222c0 con 0x7fbbf806df30 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe803d440 con 0x7fbbf806df30 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:04.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbdf7fe640 10 monclient: dump: 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 10 monclient: _renew_subs 2024-09-15T03:37:04.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbf5ffb640 4 set_mon_vals no callback set 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.555+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf81415c0 con 0x7fbbf806df30 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbbe8039040 con 0x7fbbf806df30 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbdf7fe640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbe40386f0 0x7fbbe403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(209..209 src has 106..209) v4 ==== 13417+0+0 (secure 0 0 0) 0x7fbbe806d8b0 con 0x7fbbf806df30 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbfe19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbfe19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbf6ffd640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbe40386f0 0x7fbbe403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbfe19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf8122450 con 0x7fbbf806df30 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.556+0000 7fbbf6ffd640 10 monclient: get_auth_request con 0x7fbbe40386f0 auth_method 0 2024-09-15T03:37:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.559+0000 7fbbf6ffd640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbe40386f0 0x7fbbe403aba0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0038f0 tx=0x7fbbec0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:04.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.561+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbbe8037070 con 0x7fbbf806df30 2024-09-15T03:37:04.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.561+0000 7fbbdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:04.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.561+0000 7fbbdf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:04.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.715+0000 7fbbfe19d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.715+0000 7fbbfe19d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.715+0000 7fbbfe19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.715+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbbf804bf60 con 0x7fbbf806df30 2024-09-15T03:37:04.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.817+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v210) v1 ==== 168+0+0 (secure 0 0 0) 0x7fbbe802f2a0 con 0x7fbbf806df30 2024-09-15T03:37:04.818 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.817+0000 7fbbdf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.819 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.817+0000 7fbbdf7fe640 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-15T03:37:04.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.914+0000 7fbbfe19d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.914+0000 7fbbfe19d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.914+0000 7fbbfe19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:04.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.914+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbbf8068a20 con 0x7fbbf806df30 2024-09-15T03:37:04.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.915+0000 7fbbdf7fe640 1 -- 172.21.15.181:0/4136987186 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v210) v1 ==== 147+0+0 (secure 0 0 0) 0x7fbbe8030e60 con 0x7fbbf806df30 2024-09-15T03:37:04.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.915+0000 7fbbdf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:04.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.915+0000 7fbbdf7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-15T03:37:04.916 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h26' already exists 2024-09-15T03:37:04.918 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbe40386f0 msgr2=0x7fbbe403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbe40386f0 0x7fbbe403aba0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0038f0 tx=0x7fbbec0090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 10 monclient: shutdown 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 msgr2=0x7fbbf8141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf8141080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009940 tx=0x7fbbe8032000 comp rx=0 tx=0).stop 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 shutdown_connections 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbbe40386f0 0x7fbbe403aba0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbbf809e010 0x7fbbf811ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf806e300 0x7fbbf8134ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 --2- 172.21.15.181:0/4136987186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf806df30 0x7fbbf8141080 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 >> 172.21.15.181:0/4136987186 conn(0x7fbbf80a3b30 msgr2=0x7fbbf80a23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 shutdown_connections 2024-09-15T03:37:04.919 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:04.917+0000 7fbbfe19d640 1 -- 172.21.15.181:0/4136987186 wait complete. 2024-09-15T03:37:04.927 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:04.927 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.001+0000 7eff7515d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.001+0000 7eff7515d640 10 monclient: build_initial_monmap 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.002+0000 7eff7515d640 10 monclient: monmap: 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:05.002779+0000 2024-09-15T03:37:05.003 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:05.002779+0000 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.002+0000 7eff7515d640 1 Processor -- start 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.002+0000 7eff7515d640 1 -- start start 2024-09-15T03:37:05.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.002+0000 7eff7515d640 10 monclient: init 2024-09-15T03:37:05.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:05.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:37:05.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:05.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): picked mon.noname-b con 0x7eff70069bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:05.006 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff7009cf60 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff700988c0 0x7eff70138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): picked mon.noname-a con 0x7eff700988c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7009cd00 con 0x7eff7009cf60 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7009d6d0 con 0x7eff700988c0 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff70098d40 con 0x7eff70069bd0 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff7515d640 10 monclient(hunting): authenticate will time out at 1229.484084s 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6ed9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 msgr2=0x7eff70131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff700988c0 0x7eff70138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:05.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff700988c0 0x7eff70138ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.181:47938/0 (socket says 172.21.15.181:47938) 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 -- 172.21.15.181:0/3711932277 learned_addr learned my addr 172.21.15.181:0/3711932277 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): get_auth_request con 0x7eff700988c0 auth_method 0 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6659c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7eff7009cf60 0x7eff700981b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:50102/0 (socket says 172.21.15.181:50102) 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient(hunting): handle_auth_done global_id 114246 payload 274 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 -- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 msgr2=0x7eff70131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 --2- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 -- 172.21.15.181:0/3711932277 >> v1:172.21.15.153:6789/0 conn(0x7eff7009cf60 legacy=0x7eff700981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:05.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:37:05.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 monclient: found mon.noname-a 2024-09-15T03:37:05.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:05.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:05.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.003+0000 7eff6f59e640 1 -- 172.21.15.181:0/3711932277 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7009e320 con 0x7eff700988c0 2024-09-15T03:37:05.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6f59e640 1 -- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff700988c0 msgr2=0x7eff70138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6f59e640 1 --2- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff700988c0 0x7eff70138ea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7eff68009f20 tx=0x7eff68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206a899d6337c2ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 1 -- 172.21.15.181:0/3711932277 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68007610 con 0x7eff700988c0 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: dump: 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:05.015 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: _finish_auth 0 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff7515d640 5 monclient: authenticate success, global_id 114246 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff7515d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 1 -- 172.21.15.181:0/3711932277 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6802d050 con 0x7eff700988c0 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff7515d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff7515d640 4 set_mon_vals no callback set 2024-09-15T03:37:05.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 1 -- 172.21.15.181:0/3711932277 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff680079e0 con 0x7eff700988c0 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.006+0000 7eff6e59c640 10 monclient: dump: 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:05.017 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 10 monclient: shutdown 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 -- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff700988c0 msgr2=0x7eff70138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 --2- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff700988c0 0x7eff70138ea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7eff68009f20 tx=0x7eff68009840 comp rx=0 tx=0).stop 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 -- 172.21.15.181:0/3711932277 shutdown_connections 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 --2- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff700988c0 0x7eff70138ea0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 --2- 172.21.15.181:0/3711932277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 -- 172.21.15.181:0/3711932277 >> 172.21.15.181:0/3711932277 conn(0x7eff7009e9f0 msgr2=0x7eff7009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 -- 172.21.15.181:0/3711932277 shutdown_connections 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 1 -- 172.21.15.181:0/3711932277 wait complete. 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 10 monclient: build_initial_monmap 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.007+0000 7eff7515d640 10 monclient: monmap: 2024-09-15T03:37:05.018 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 Processor -- start 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 -- start start 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient: init 2024-09-15T03:37:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): picked mon.noname-c con 0x7eff70069bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 0x7eff70135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): picked mon.noname-b con 0x7eff700988c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7009cf60 0x7eff701363b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): picked mon.noname-a con 0x7eff7009cf60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7012ed40 con 0x7eff7009cf60 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7009d8a0 con 0x7eff700988c0 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff700a0110 con 0x7eff70069bd0 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff7515d640 10 monclient(hunting): authenticate will time out at 1229.489875s 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff6659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 0x7eff70135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.008+0000 7eff6ed9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 msgr2=0x7eff70135930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70135930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 0x7eff70135e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32804/0 (socket says 172.21.15.181:32804) 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 -- 172.21.15.181:0/131993921 learned_addr learned my addr 172.21.15.181:0/131993921 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): get_auth_request con 0x7eff700988c0 auth_method 0 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:05.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6f59e640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7009cf60 0x7eff701363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6f59e640 10 monclient(hunting): get_auth_request con 0x7eff7009cf60 auth_method 0 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient(hunting): handle_auth_done global_id 104440 payload 824 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 -- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 msgr2=0x7eff70135930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70135930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 -- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7009cf60 msgr2=0x7eff701363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7009cf60 0x7eff701363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient: _finish_hunting 0 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 monclient: found mon.noname-b 2024-09-15T03:37:05.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.009+0000 7eff6659c640 1 -- 172.21.15.181:0/131993921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff68009ae0 con 0x7eff700988c0 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff6659c640 1 -- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 msgr2=0x7eff70135e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff6659c640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 0x7eff70135e70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c560 tx=0x7eff58007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580074f0 con 0x7eff700988c0 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: dump: 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.023 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 5 monclient: authenticate success, global_id 104440 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58007670 con 0x7eff700988c0 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58005a00 con 0x7eff700988c0 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 10 monclient: _renew_subs 2024-09-15T03:37:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff70118600 con 0x7eff700988c0 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff6dd9b640 4 set_mon_vals no callback set 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff677fe640 10 monclient: dump: 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 10 monclient: _renew_subs 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.011+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7012f690 con 0x7eff700988c0 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7eff5801b3b0 con 0x7eff700988c0 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff7515d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff7515d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff7515d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7004bf60 con 0x7eff700988c0 2024-09-15T03:37:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff677fe640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff500386f0 0x7eff5003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(210..210 src has 106..210) v4 ==== 13630+0+0 (secure 0 0 0) 0x7eff5804d290 con 0x7eff700988c0 2024-09-15T03:37:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff6ed9d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff500386f0 0x7eff5003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.012+0000 7eff6ed9d640 10 monclient: get_auth_request con 0x7eff500386f0 auth_method 0 2024-09-15T03:37:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.015+0000 7eff6ed9d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff500386f0 0x7eff5003aba0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7eff600038f0 tx=0x7eff60006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.016+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7eff58018630 con 0x7eff700988c0 2024-09-15T03:37:05.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.016+0000 7eff677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:05.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.016+0000 7eff677fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:05.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.173+0000 7eff7515d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:05.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.173+0000 7eff7515d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:05.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.173+0000 7eff7515d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:05.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.173+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7eff70132470 con 0x7eff700988c0 2024-09-15T03:37:05.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.826+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v211) v1 ==== 168+0+0 (secure 0 0 0) 0x7eff580170b0 con 0x7eff700988c0 2024-09-15T03:37:05.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.826+0000 7eff677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:05.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.826+0000 7eff677fe640 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-15T03:37:05.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.923+0000 7eff7515d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:05.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.923+0000 7eff7515d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:05.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.923+0000 7eff7515d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:05.924 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.923+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7eff701329c0 con 0x7eff700988c0 2024-09-15T03:37:05.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.924+0000 7eff677fe640 1 -- 172.21.15.181:0/131993921 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v211) v1 ==== 147+0+0 (secure 0 0 0) 0x7eff58005df0 con 0x7eff700988c0 2024-09-15T03:37:05.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.924+0000 7eff677fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:05.925 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.924+0000 7eff677fe640 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-15T03:37:05.925 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h27' already exists 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff500386f0 msgr2=0x7eff5003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff500386f0 0x7eff5003aba0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7eff600038f0 tx=0x7eff60006d80 comp rx=0 tx=0).stop 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 10 monclient: shutdown 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 msgr2=0x7eff70135e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 0x7eff70135e70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c560 tx=0x7eff58007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 shutdown_connections 2024-09-15T03:37:05.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff500386f0 0x7eff5003aba0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7009cf60 0x7eff701363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff700988c0 0x7eff70135e70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 --2- 172.21.15.181:0/131993921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff70069bd0 0x7eff70135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:05.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 >> 172.21.15.181:0/131993921 conn(0x7eff7009e9f0 msgr2=0x7eff700689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:05.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 shutdown_connections 2024-09-15T03:37:05.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:05.926+0000 7eff7515d640 1 -- 172.21.15.181:0/131993921 wait complete. 2024-09-15T03:37:05.936 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:05.936 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.011+0000 7f1f9891d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.011+0000 7f1f9891d640 10 monclient: build_initial_monmap 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.011+0000 7f1f9891d640 10 monclient: monmap: 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:06.011915+0000 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:06.011915+0000 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:06.012 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:06.013 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:06.013 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:06.013 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:06.013 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:06.013 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.011+0000 7f1f9891d640 1 Processor -- start 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.011+0000 7f1f9891d640 1 -- start start 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.011+0000 7f1f9891d640 10 monclient: init 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f94096a30 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:06.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9409d530 0x7f1f94099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f9409d530 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f94097510 0x7f1f94135c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f94097510 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f94097e80 con 0x7f1f94096a30 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f94098be0 con 0x7f1f94097510 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f94098ec0 con 0x7f1f9409d530 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f9891d640 10 monclient(hunting): authenticate will time out at 1230.493446s 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f94097510 0x7f1f94135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f91d9b640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9409d530 msgr2=0x7f1f94099d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1f94097510 0x7f1f94135c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32810/0 (socket says 172.21.15.181:32810) 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f91d9b640 1 --2- 172.21.15.181:0/1434305965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9409d530 0x7f1f94099d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 -- 172.21.15.181:0/1434305965 learned_addr learned my addr 172.21.15.181:0/1434305965 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): get_auth_request con 0x7f1f94097510 auth_method 0 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_done global_id 104443 payload 274 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 -- 172.21.15.181:0/1434305965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9409d530 msgr2=0x7f1f94099d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 --2- 172.21.15.181:0/1434305965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9409d530 0x7f1f94099d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 -- 172.21.15.181:0/1434305965 >> v1:172.21.15.153:6789/0 conn(0x7f1f94096a30 legacy=0x7f1f94096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:06.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:06.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 monclient: found mon.noname-c 2024-09-15T03:37:06.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:06.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.018 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.012+0000 7f1f92d9d640 1 -- 172.21.15.181:0/1434305965 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f94097990 con 0x7f1f94097510 2024-09-15T03:37:06.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f92d9d640 1 -- 172.21.15.181:0/1434305965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94097510 msgr2=0x7f1f94135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f92d9d640 1 --2- 172.21.15.181:0/1434305965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94097510 0x7f1f94135c10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009910 tx=0x7f1f8c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ede7642cad298087 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 1 -- 172.21.15.181:0/1434305965 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c032440 con 0x7f1f94097510 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: dump: 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:06.025 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: _finish_auth 0 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9891d640 5 monclient: authenticate success, global_id 104443 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9891d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 1 -- 172.21.15.181:0/1434305965 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c0325a0 con 0x7f1f94097510 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9891d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9891d640 4 set_mon_vals no callback set 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 1 -- 172.21.15.181:0/1434305965 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c032940 con 0x7f1f94097510 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:06.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.015+0000 7f1f9159a640 10 monclient: dump: 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 10 monclient: shutdown 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:06.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 -- 172.21.15.181:0/1434305965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94097510 msgr2=0x7f1f94135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 --2- 172.21.15.181:0/1434305965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94097510 0x7f1f94135c10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009910 tx=0x7f1f8c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 -- 172.21.15.181:0/1434305965 shutdown_connections 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 --2- 172.21.15.181:0/1434305965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f94097510 0x7f1f94135c10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 --2- 172.21.15.181:0/1434305965 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f9409d530 0x7f1f94099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 -- 172.21.15.181:0/1434305965 >> 172.21.15.181:0/1434305965 conn(0x7f1f940a0fe0 msgr2=0x7f1f940a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 -- 172.21.15.181:0/1434305965 shutdown_connections 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 1 -- 172.21.15.181:0/1434305965 wait complete. 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 10 monclient: build_initial_monmap 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.016+0000 7f1f9891d640 10 monclient: monmap: 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:06.028 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 Processor -- start 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 -- start start 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient: init 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f94096a30 0x7f1f9409bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f94096a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f94097510 0x7f1f9413f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f94097510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 0x7f1f9413f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f9409d530 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940984c0 con 0x7f1f94096a30 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940a0120 con 0x7f1f9409d530 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9411cca0 con 0x7f1f94097510 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9891d640 10 monclient(hunting): authenticate will time out at 1230.498614s 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f91d9b640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f94097510 msgr2=0x7f1f9413f370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 0x7f1f9413f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f91d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f94097510 0x7f1f9413f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 0x7f1f9413f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32824/0 (socket says 172.21.15.181:32824) 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 -- 172.21.15.181:0/2964047246 learned_addr learned my addr 172.21.15.181:0/2964047246 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): get_auth_request con 0x7f1f9409d530 auth_method 0 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f94096a30 0x7f1f9409bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): get_auth_request con 0x7f1f94096a30 auth_method 0 2024-09-15T03:37:06.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient(hunting): handle_auth_done global_id 104446 payload 824 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 -- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f94097510 msgr2=0x7f1f9413f370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f94097510 0x7f1f9413f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 -- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f94096a30 msgr2=0x7f1f9409bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f94096a30 0x7f1f9409bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 monclient: found mon.noname-b 2024-09-15T03:37:06.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.017+0000 7f1f92d9d640 1 -- 172.21.15.181:0/2964047246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c009450 con 0x7f1f9409d530 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f92d9d640 1 -- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 msgr2=0x7f1f9413f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f92d9d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 0x7f1f9413f8b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c035040 tx=0x7f1f8c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c031a90 con 0x7f1f9409d530 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: dump: 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:06.033 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 5 monclient: authenticate success, global_id 104446 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c031bf0 con 0x7f1f9409d530 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 10 monclient: _renew_subs 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c03c470 con 0x7f1f9409d530 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9409c780 con 0x7f1f9409d530 2024-09-15T03:37:06.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f90d99640 4 set_mon_vals no callback set 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f827fc640 10 monclient: dump: 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:06.035 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 10 monclient: _renew_subs 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.020+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9413fdf0 con 0x7f1f9409d530 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1f8c039040 con 0x7f1f9409d530 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f827fc640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f740386f0 0x7f1f7403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f9891d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f9891d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f9891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(211..211 src has 106..211) v4 ==== 13843+0+0 (secure 0 0 0) 0x7f1f8c042070 con 0x7f1f9409d530 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9404bf60 con 0x7f1f9409d530 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f9259c640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f740386f0 0x7f1f7403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.021+0000 7f1f9259c640 10 monclient: get_auth_request con 0x7f1f740386f0 auth_method 0 2024-09-15T03:37:06.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.023+0000 7f1f9259c640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f740386f0 0x7f1f7403aba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f84007fd0 tx=0x7f1f84002630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:06.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.025+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1f8c037020 con 0x7f1f9409d530 2024-09-15T03:37:06.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.025+0000 7f1f827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:06.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.025+0000 7f1f827fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:06.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.184+0000 7f1f9891d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:06.185 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.184+0000 7f1f9891d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:06.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.184+0000 7f1f9891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.184+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1f9409cd10 con 0x7f1f9409d530 2024-09-15T03:37:06.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.832+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v212) v1 ==== 168+0+0 (secure 0 0 0) 0x7f1f8c045950 con 0x7f1f9409d530 2024-09-15T03:37:06.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.832+0000 7f1f827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:06.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.832+0000 7f1f827fc640 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-15T03:37:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.926+0000 7f1f9891d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.926+0000 7f1f9891d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.926+0000 7f1f9891d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.926+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1f9409cd10 con 0x7f1f9409d530 2024-09-15T03:37:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.927+0000 7f1f827fc640 1 -- 172.21.15.181:0/2964047246 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v212) v1 ==== 147+0+0 (secure 0 0 0) 0x7f1f8c045380 con 0x7f1f9409d530 2024-09-15T03:37:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.927+0000 7f1f827fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:06.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.927+0000 7f1f827fc640 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-15T03:37:06.929 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h28' already exists 2024-09-15T03:37:06.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f740386f0 msgr2=0x7f1f7403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f740386f0 0x7f1f7403aba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f84007fd0 tx=0x7f1f84002630 comp rx=0 tx=0).stop 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 10 monclient: shutdown 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 msgr2=0x7f1f9413f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 0x7f1f9413f8b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c035040 tx=0x7f1f8c032000 comp rx=0 tx=0).stop 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 shutdown_connections 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1f740386f0 0x7f1f7403aba0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f9409d530 0x7f1f9413f8b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f94097510 0x7f1f9413f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 --2- 172.21.15.181:0/2964047246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f94096a30 0x7f1f9409bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:06.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 >> 172.21.15.181:0/2964047246 conn(0x7f1f940a0fe0 msgr2=0x7f1f9409eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:06.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 shutdown_connections 2024-09-15T03:37:06.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:06.929+0000 7f1f9891d640 1 -- 172.21.15.181:0/2964047246 wait complete. 2024-09-15T03:37:06.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:06.940 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.014+0000 7f198679d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.014+0000 7f198679d640 10 monclient: build_initial_monmap 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.014+0000 7f198679d640 10 monclient: monmap: 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:07.015048+0000 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:07.015048+0000 2024-09-15T03:37:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:07.016 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.014+0000 7f198679d640 1 Processor -- start 2024-09-15T03:37:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 1 -- start start 2024-09-15T03:37:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient: init 2024-09-15T03:37:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:37:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1980131bd0 0x7f1980131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): picked mon.noname-a con 0x7f1980131bd0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f198009ebe0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980069510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): picked mon.noname-b con 0x7f198009efe0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800a06d0 con 0x7f198009ebe0 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1980071780 con 0x7f1980131bd0 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19800a0010 con 0x7f198009efe0 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f198679d640 10 monclient(hunting): authenticate will time out at 1231.496645s 2024-09-15T03:37:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f1984d3a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 msgr2=0x7f1980069510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.015+0000 7f1984d3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980069510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1980131bd0 0x7f1980131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f198009ebe0 0x7f1980068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.181:50126/0 (socket says 172.21.15.181:50126) 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197f7fe640 1 -- 172.21.15.181:0/2650866028 learned_addr learned my addr 172.21.15.181:0/2650866028 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): get_auth_request con 0x7f1980131bd0 auth_method 0 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient(hunting): handle_auth_done global_id 114255 payload 274 2024-09-15T03:37:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 1 -- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 msgr2=0x7f1980069510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 1 --2- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980069510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 1 -- 172.21.15.181:0/2650866028 >> v1:172.21.15.153:6789/0 conn(0x7f198009ebe0 legacy=0x7f1980068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 1 monclient: found mon.noname-a 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.016+0000 7f197ffff640 1 -- 172.21.15.181:0/2650866028 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800a1910 con 0x7f1980131bd0 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197ffff640 1 -- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 msgr2=0x7f1980131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197ffff640 1 --2- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f1980131fa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1970009850 tx=0x7f19700090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b450d4817c3029e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 1 -- 172.21.15.181:0/2650866028 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970032440 con 0x7f1980131bd0 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: dump: 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: _finish_auth 0 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 1 -- 172.21.15.181:0/2650866028 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19700325a0 con 0x7f1980131bd0 2024-09-15T03:37:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 5 monclient: authenticate success, global_id 114255 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 1 -- 172.21.15.181:0/2650866028 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970032940 con 0x7f1980131bd0 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f197effd640 10 monclient: dump: 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:07.033 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 4 set_mon_vals no callback set 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 10 monclient: shutdown 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 1 -- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 msgr2=0x7f1980131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.019+0000 7f198679d640 1 --2- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f1980131fa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1970009850 tx=0x7f19700090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 -- 172.21.15.181:0/2650866028 shutdown_connections 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 --2- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 --2- 172.21.15.181:0/2650866028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f1980131fa0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 -- 172.21.15.181:0/2650866028 >> 172.21.15.181:0/2650866028 conn(0x7f19800713b0 msgr2=0x7f19800a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 -- 172.21.15.181:0/2650866028 shutdown_connections 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 -- 172.21.15.181:0/2650866028 wait complete. 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 10 monclient: build_initial_monmap 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 10 monclient: monmap: 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:07.035 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.020+0000 7f198679d640 1 Processor -- start 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 -- start start 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient: init 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 0x7f198009a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): picked mon.noname-b con 0x7f198009ebe0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): picked mon.noname-c con 0x7f198009efe0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f19801410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): picked mon.noname-a con 0x7f1980131bd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198009be40 con 0x7f1980131bd0 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198011dc70 con 0x7f198009ebe0 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198011df50 con 0x7f198009efe0 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f198679d640 10 monclient(hunting): authenticate will time out at 1231.502468s 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197f7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 msgr2=0x7f1980130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 0x7f198009a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 0x7f198009a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32830/0 (socket says 172.21.15.181:32830) 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 -- 172.21.15.181:0/3589727928 learned_addr learned my addr 172.21.15.181:0/3589727928 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): get_auth_request con 0x7f198009ebe0 auth_method 0 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f19801410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): get_auth_request con 0x7f1980131bd0 auth_method 0 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f1984d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient(hunting): handle_auth_done global_id 104449 payload 824 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 -- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 msgr2=0x7f1980130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 -- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 msgr2=0x7f19801410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f19801410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 monclient: found mon.noname-b 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.021+0000 7f197ffff640 1 -- 172.21.15.181:0/3589727928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970009390 con 0x7f198009ebe0 2024-09-15T03:37:07.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f197ffff640 1 -- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 msgr2=0x7f198009a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f197ffff640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 0x7f198009a8b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1970009850 tx=0x7f1970032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19700318c0 con 0x7f198009ebe0 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: dump: 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:07.040 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 5 monclient: authenticate success, global_id 104449 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 10 monclient: _renew_subs 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198011e230 con 0x7f198009ebe0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 10 monclient: _renew_subs 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198011c5e0 con 0x7f198009ebe0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970031a20 con 0x7f198009ebe0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970031da0 con 0x7f198009ebe0 2024-09-15T03:37:07.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f1977fff640 10 monclient: dump: 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.024+0000 7f197e7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:07.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1970039040 con 0x7f198009ebe0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f198679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f198679d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f198679d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198011e630 con 0x7f198009ebe0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f1977fff640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19600386f0 0x7f196003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(212..212 src has 106..212) v4 ==== 14056+0+0 (secure 0 0 0) 0x7f197006dc80 con 0x7f198009ebe0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f197f7fe640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19600386f0 0x7f196003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.025+0000 7f197f7fe640 10 monclient: get_auth_request con 0x7f19600386f0 auth_method 0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.028+0000 7f197f7fe640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19600386f0 0x7f196003aba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f198009e860 tx=0x7f1968007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.029+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1970039520 con 0x7f198009ebe0 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.029+0000 7f1977fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:07.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.029+0000 7f1977fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.186+0000 7f198679d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.186+0000 7f198679d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.186+0000 7f198679d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.186+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f198011e090 con 0x7f198009ebe0 2024-09-15T03:37:07.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.832+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v213) v1 ==== 168+0+0 (secure 0 0 0) 0x7f198011e090 con 0x7f198009ebe0 2024-09-15T03:37:07.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.832+0000 7f1977fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:07.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.832+0000 7f1977fff640 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-15T03:37:07.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.931+0000 7f198679d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:07.932 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.931+0000 7f198679d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:07.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.931+0000 7f198679d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:07.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.931+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f198011e090 con 0x7f198009ebe0 2024-09-15T03:37:07.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.933+0000 7f1977fff640 1 -- 172.21.15.181:0/3589727928 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v213) v1 ==== 147+0+0 (secure 0 0 0) 0x7f198011e090 con 0x7f198009ebe0 2024-09-15T03:37:07.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.933+0000 7f1977fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:07.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.933+0000 7f1977fff640 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-15T03:37:07.935 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h29' already exists 2024-09-15T03:37:07.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19600386f0 msgr2=0x7f196003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:07.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19600386f0 0x7f196003aba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f198009e860 tx=0x7f1968007b80 comp rx=0 tx=0).stop 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 10 monclient: shutdown 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 msgr2=0x7f198009a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 0x7f198009a8b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1970009850 tx=0x7f1970032000 comp rx=0 tx=0).stop 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 shutdown_connections 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19600386f0 0x7f196003aba0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1980131bd0 0x7f19801410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f198009efe0 0x7f1980130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 --2- 172.21.15.181:0/3589727928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f198009ebe0 0x7f198009a8b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 >> 172.21.15.181:0/3589727928 conn(0x7f19800713b0 msgr2=0x7f198009d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 shutdown_connections 2024-09-15T03:37:07.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:07.935+0000 7f198679d640 1 -- 172.21.15.181:0/3589727928 wait complete. 2024-09-15T03:37:07.946 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:07.946 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-15T03:37:08.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.021+0000 7fcb3c4a8640 10 monclient: get_monmap_and_config 2024-09-15T03:37:08.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.021+0000 7fcb3c4a8640 10 monclient: build_initial_monmap 2024-09-15T03:37:08.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.021+0000 7fcb3c4a8640 10 monclient: monmap: 2024-09-15T03:37:08.022 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:08.022 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:08.022073+0000 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:08.022073+0000 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.021+0000 7fcb3c4a8640 1 Processor -- start 2024-09-15T03:37:08.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.021+0000 7fcb3c4a8640 1 -- start start 2024-09-15T03:37:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.021+0000 7fcb3c4a8640 10 monclient: init 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb34131bd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 0x7fcb34131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb34069e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb34136d60 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb340a06d0 con 0x7fcb34131bd0 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3406a230 con 0x7fcb34136d60 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34071780 con 0x7fcb34069e10 2024-09-15T03:37:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3c4a8640 10 monclient(hunting): authenticate will time out at 1232.503589s 2024-09-15T03:37:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb39a43640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 msgr2=0x7fcb34131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3aa45640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fcb34136d60 0x7fcb34139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:54430/0 (socket says 172.21.15.181:54430) 2024-09-15T03:37:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 learned_addr learned my addr 172.21.15.181:0/2267055979 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.022+0000 7fcb39a43640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 0x7fcb34131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:08.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.024+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 >> v1:172.21.15.181:6789/0 conn(0x7fcb34136d60 legacy=0x7fcb34139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:08.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206746029 0 0) 0x7fcb3406a230 con 0x7fcb34136d60 2024-09-15T03:37:08.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:08.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): my global_id is 104452 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb200034e0 con 0x7fcb34136d60 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 1 -- 172.21.15.181:0/2267055979 >> v1:172.21.15.153:6789/0 conn(0x7fcb34131bd0 legacy=0x7fcb3409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3450405054 0 0) 0x7fcb200034e0 con 0x7fcb34136d60 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 msgr2=0x7fcb34131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 0x7fcb34131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 >> v1:172.21.15.153:6789/0 conn(0x7fcb34131bd0 legacy=0x7fcb3409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:08.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: _finish_hunting 0 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340a0010 con 0x7fcb34136d60 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: _finish_auth 0 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fcb2c001aa0 con 0x7fcb34136d60 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: dump: 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:08.040 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:08.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 >> v1:172.21.15.181:6789/0 conn(0x7fcb34136d60 legacy=0x7fcb34139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb20003a50 0x7fcb20003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): picked mon.a con 0x7fcb20003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 0x7fcb20004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): picked mon.c con 0x7fcb34069e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb2000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): picked mon.b con 0x7fcb20004ec0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34071780 con 0x7fcb20003a50 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20005560 con 0x7fcb20004ec0 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2000bf10 con 0x7fcb34069e10 2024-09-15T03:37:08.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39242640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39a43640 1 -- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 msgr2=0x7fcb20004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb39a43640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 0x7fcb20004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb2000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb20003a50 0x7fcb20003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): get_auth_request con 0x7fcb20004ec0 auth_method 0 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): get_auth_request con 0x7fcb20003a50 auth_method 0 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:08.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3a244640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient(hunting): handle_auth_done global_id 104452 payload 293 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 msgr2=0x7fcb20004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34069e10 0x7fcb20004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb20003a50 msgr2=0x7fcb20003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb20003a50 0x7fcb20003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient: _finish_hunting 0 2024-09-15T03:37:08.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 monclient: found mon.b 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200034e0 con 0x7fcb20004ec0 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 10 monclient: _finish_auth 0 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3aa45640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3c4a8640 5 monclient: authenticate success, global_id 104452 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.025+0000 7fcb3c4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 msgr2=0x7fcb2000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb3aa45640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb2000b9d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c001810 tx=0x7fcb340718c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772196ac1fd15429 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c02b660 con 0x7fcb20004ec0 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: dump: 2024-09-15T03:37:08.045 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb3c4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c02b7e0 con 0x7fcb20004ec0 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:08.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 1 -- 172.21.15.181:0/2267055979 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c02bba0 con 0x7fcb20004ec0 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb3c4a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb3c4a8640 4 set_mon_vals no callback set 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.028+0000 7fcb39242640 10 monclient: dump: 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:08.047 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 10 monclient: shutdown 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/2267055979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 msgr2=0x7fcb2000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/2267055979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb2000b9d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c001810 tx=0x7fcb340718c0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3aa45640 1 -- 172.21.15.181:0/2267055979 reap_dead start 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/2267055979 shutdown_connections 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/2267055979 >> 172.21.15.181:0/2267055979 conn(0x7fcb340713b0 msgr2=0x7fcb340a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/2267055979 shutdown_connections 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/2267055979 wait complete. 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 10 monclient: build_initial_monmap 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.029+0000 7fcb3c4a8640 10 monclient: monmap: 2024-09-15T03:37:08.048 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 Processor -- start 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 -- start start 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient: init 2024-09-15T03:37:08.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb34141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb20004ec0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34141d70 0x7fcb34098900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb34141d70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb34098e40 0x7fcb340992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb34098e40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3409bec0 con 0x7fcb34098e40 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3409edc0 con 0x7fcb20004ec0 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:08.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3411ba60 con 0x7fcb34141d70 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb34141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3c4a8640 10 monclient(hunting): authenticate will time out at 1232.511602s 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb34141830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:32848/0 (socket says 172.21.15.181:32848) 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 -- 172.21.15.181:0/385807286 learned_addr learned my addr 172.21.15.181:0/385807286 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): get_auth_request con 0x7fcb20004ec0 auth_method 0 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:08.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb39a43640 1 -- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34141d70 msgr2=0x7fcb34098900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb39a43640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34141d70 0x7fcb34098900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3aa45640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb34098e40 0x7fcb340992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3aa45640 10 monclient(hunting): get_auth_request con 0x7fcb34098e40 auth_method 0 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3aa45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3aa45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3aa45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient(hunting): handle_auth_done global_id 104455 payload 824 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 -- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34141d70 msgr2=0x7fcb34098900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:08.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34141d70 0x7fcb34098900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 -- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb34098e40 msgr2=0x7fcb340992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb34098e40 0x7fcb340992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient: _finish_hunting 0 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 monclient: found mon.noname-b 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.030+0000 7fcb3a244640 1 -- 172.21.15.181:0/385807286 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3411bd40 con 0x7fcb20004ec0 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3a244640 1 -- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 msgr2=0x7fcb34141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3a244640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb34141830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400f880 tx=0x7fcb24009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240072e0 con 0x7fcb20004ec0 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: dump: 2024-09-15T03:37:08.053 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 5 monclient: authenticate success, global_id 104455 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24007440 con 0x7fcb20004ec0 2024-09-15T03:37:08.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 10 monclient: _renew_subs 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3409a360 con 0x7fcb20004ec0 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240059d0 con 0x7fcb20004ec0 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb2a7fc640 10 monclient: dump: 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:08.055 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 10 monclient: _renew_subs 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3411b6f0 con 0x7fcb20004ec0 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.033+0000 7fcb38a41640 4 set_mon_vals no callback set 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb3c4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb3c4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb3c4a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb24005d40 con 0x7fcb20004ec0 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3404bf60 con 0x7fcb20004ec0 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb2a7fc640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb18038720 0x7fcb1803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(213..213 src has 106..213) v4 ==== 14269+0+0 (secure 0 0 0) 0x7fcb2404e030 con 0x7fcb20004ec0 2024-09-15T03:37:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb39a43640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb18038720 0x7fcb1803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.034+0000 7fcb39a43640 10 monclient: get_auth_request con 0x7fcb18038720 auth_method 0 2024-09-15T03:37:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.036+0000 7fcb39a43640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb18038720 0x7fcb1803abd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcb30006fd0 tx=0x7fcb30006e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.038+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcb2407fe40 con 0x7fcb20004ec0 2024-09-15T03:37:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.038+0000 7fcb2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:08.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.038+0000 7fcb2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:08.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.194+0000 7fcb3c4a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:08.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.194+0000 7fcb3c4a8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:08.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.194+0000 7fcb3c4a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.194+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcb34064990 con 0x7fcb20004ec0 2024-09-15T03:37:08.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.841+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v214) v1 ==== 168+0+0 (secure 0 0 0) 0x7fcb24012b30 con 0x7fcb20004ec0 2024-09-15T03:37:08.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.841+0000 7fcb2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:08.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.841+0000 7fcb2a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-15T03:37:08.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.938+0000 7fcb3c4a8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:08.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.938+0000 7fcb3c4a8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:08.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.938+0000 7fcb3c4a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:08.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.938+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcb3412ff50 con 0x7fcb20004ec0 2024-09-15T03:37:08.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.939+0000 7fcb2a7fc640 1 -- 172.21.15.181:0/385807286 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v214) v1 ==== 147+0+0 (secure 0 0 0) 0x7fcb24012350 con 0x7fcb20004ec0 2024-09-15T03:37:08.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.939+0000 7fcb2a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:08.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.939+0000 7fcb2a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-15T03:37:08.940 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h30' already exists 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb18038720 msgr2=0x7fcb1803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb18038720 0x7fcb1803abd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcb30006fd0 tx=0x7fcb30006e50 comp rx=0 tx=0).stop 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 10 monclient: shutdown 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 msgr2=0x7fcb34141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb34141830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400f880 tx=0x7fcb24009040 comp rx=0 tx=0).stop 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 shutdown_connections 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb18038720 0x7fcb1803abd0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb34098e40 0x7fcb340992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb34141d70 0x7fcb34098900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 --2- 172.21.15.181:0/385807286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb20004ec0 0x7fcb34141830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:08.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 >> 172.21.15.181:0/385807286 conn(0x7fcb340713b0 msgr2=0x7fcb340a4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:08.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 shutdown_connections 2024-09-15T03:37:08.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:08.941+0000 7fcb3c4a8640 1 -- 172.21.15.181:0/385807286 wait complete. 2024-09-15T03:37:08.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:08.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.026+0000 7f439d59d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.026+0000 7f439d59d640 10 monclient: build_initial_monmap 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.026+0000 7f439d59d640 10 monclient: monmap: 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:09.026850+0000 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:09.026850+0000 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:09.029 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.026+0000 7f439d59d640 1 Processor -- start 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.026+0000 7f439d59d640 1 -- start start 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.026+0000 7f439d59d640 10 monclient: init 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4398131bd0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4398069e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 0x7f439813a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4398131fa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43980a06d0 con 0x7f4398069e10 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4398071780 con 0x7f4398131bd0 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4398132420 con 0x7f4398131fa0 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f439d59d640 10 monclient(hunting): authenticate will time out at 1233.508199s 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f4396ffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4398131bd0 0x7f439809ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35916/0 (socket says 172.21.15.181:35916) 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f4396ffd640 1 -- 172.21.15.181:0/4250613487 learned_addr learned my addr 172.21.15.181:0/4250613487 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f43977fe640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 msgr2=0x7f439813a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.027+0000 7f43977fe640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 0x7f439813a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4396ffd640 1 -- 172.21.15.181:0/4250613487 >> v1:172.21.15.181:6789/0 conn(0x7f4398131bd0 legacy=0x7f439809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f43967fc640 1 -- 172.21.15.181:0/4250613487 >> v1:172.21.15.153:6789/0 conn(0x7f4398069e10 legacy=0x7f4398136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835008839 0 0) 0x7f4398071780 con 0x7f4398131bd0 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 10 monclient(hunting): my global_id is 104458 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43800034e0 con 0x7f4398131bd0 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617019765 0 0) 0x7f43980a06d0 con 0x7f4398069e10 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:09.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 10 monclient(hunting): my global_id is 114267 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.029+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4398071780 con 0x7f4398069e10 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632180623 0 0) 0x7f43800034e0 con 0x7f4398131bd0 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 msgr2=0x7f439813a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 0x7f439813a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 >> v1:172.21.15.153:6789/0 conn(0x7f4398069e10 legacy=0x7f4398136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439806a210 con 0x7f4398131bd0 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f4388001720 con 0x7f4398131bd0 2024-09-15T03:37:09.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f439d59d640 5 monclient: authenticate success, global_id 104458 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f439d59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: dump: 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:09.043 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 >> v1:172.21.15.181:6789/0 conn(0x7f4398131bd0 legacy=0x7f439809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 0x7f4380003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): picked mon.c con 0x7f4398131fa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43800040e0 0x7f4380004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): picked mon.b con 0x7f43800040e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380004de0 0x7f4380005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): picked mon.a con 0x7f4380004de0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438000bc60 con 0x7f4380004de0 2024-09-15T03:37:09.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438000bdd0 con 0x7f43800040e0 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438000bf90 con 0x7f4398131fa0 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4395ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f439d59d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4396ffd640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 msgr2=0x7f4380003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f4396ffd640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 0x7f4380003b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43800040e0 0x7f4380004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): get_auth_request con 0x7f43800040e0 auth_method 0 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:09.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380004de0 0x7f4380005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): get_auth_request con 0x7f4380004de0 auth_method 0 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43977fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient(hunting): handle_auth_done global_id 104458 payload 293 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 msgr2=0x7f4380003b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398131fa0 0x7f4380003b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380004de0 msgr2=0x7f4380005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380004de0 0x7f4380005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 monclient: found mon.b 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 1 -- 172.21.15.181:0/4250613487 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43800034e0 con 0x7f43800040e0 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.030+0000 7f43967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f43967fc640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43800040e0 msgr2=0x7f4380004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f43967fc640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43800040e0 0x7f4380004780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f438c007da0 tx=0x7f438c001b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e1958b50bb99a07 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c02ade0 con 0x7f43800040e0 2024-09-15T03:37:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: dump: 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c0070f0 con 0x7f43800040e0 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 4 set_mon_vals no callback set 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 1 -- 172.21.15.181:0/4250613487 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c0074b0 con 0x7f43800040e0 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f4395ffb640 10 monclient: dump: 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 10 monclient: shutdown 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 1 -- 172.21.15.181:0/4250613487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43800040e0 msgr2=0x7f4380004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f439d59d640 1 --2- 172.21.15.181:0/4250613487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43800040e0 0x7f4380004780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f438c007da0 tx=0x7f438c001b70 comp rx=0 tx=0).stop 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.033+0000 7f43977fe640 1 -- 172.21.15.181:0/4250613487 reap_dead start 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 1 -- 172.21.15.181:0/4250613487 shutdown_connections 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 1 -- 172.21.15.181:0/4250613487 >> 172.21.15.181:0/4250613487 conn(0x7f43980713b0 msgr2=0x7f43980a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 1 -- 172.21.15.181:0/4250613487 shutdown_connections 2024-09-15T03:37:09.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 1 -- 172.21.15.181:0/4250613487 wait complete. 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 10 monclient: build_initial_monmap 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 10 monclient: monmap: 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.034+0000 7f439d59d640 1 Processor -- start 2024-09-15T03:37:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 -- start start 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient: init 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43800040e0 0x7f4398130950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): picked mon.noname-c con 0x7f43800040e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398130e90 0x7f439811d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4398130e90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 0x7f439811dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): picked mon.noname-b con 0x7f439811db40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4398131280 con 0x7f4398130e90 2024-09-15T03:37:09.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439809bec0 con 0x7f439811db40 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43980a4a30 con 0x7f43800040e0 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f439d59d640 10 monclient(hunting): authenticate will time out at 1233.516444s 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 0x7f439811dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 0x7f439811dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51464/0 (socket says 172.21.15.181:51464) 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f4396ffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43800040e0 msgr2=0x7f4398130950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 -- 172.21.15.181:0/224138701 learned_addr learned my addr 172.21.15.181:0/224138701 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): get_auth_request con 0x7f439811db40 auth_method 0 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:09.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f4396ffd640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43800040e0 0x7f4398130950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient(hunting): handle_auth_done global_id 104461 payload 824 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43967fc640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398130e90 0x7f439811d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 -- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43800040e0 msgr2=0x7f4398130950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43800040e0 0x7f4398130950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 -- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398130e90 msgr2=0x7f439811d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398130e90 0x7f439811d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 monclient: found mon.noname-b 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.035+0000 7f43977fe640 1 -- 172.21.15.181:0/224138701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4398141a40 con 0x7f439811db40 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f43977fe640 1 -- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 msgr2=0x7f439811dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f43977fe640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 0x7f439811dfe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f43980a1240 tx=0x7f43900095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390011070 con 0x7f439811db40 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: dump: 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: _finish_auth 0 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439000b970 con 0x7f439811db40 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 5 monclient: authenticate success, global_id 104461 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 10 monclient: _renew_subs 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4398120b00 con 0x7f439811db40 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439000bd30 con 0x7f439811db40 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 10 monclient: dump: 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 10 monclient: _renew_subs 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f43957fa640 4 set_mon_vals no callback set 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439811fb40 con 0x7f439811db40 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.038+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4390016630 con 0x7f439811db40 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f439d59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f439d59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f439d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439804bf60 con 0x7f439811db40 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f437effd640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43840386f0 0x7f438403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(214..214 src has 106..214) v4 ==== 14482+0+0 (secure 0 0 0) 0x7f4390006e40 con 0x7f439811db40 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f4396ffd640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43840386f0 0x7f438403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.039+0000 7f4396ffd640 10 monclient: get_auth_request con 0x7f43840386f0 auth_method 0 2024-09-15T03:37:09.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.041+0000 7f4396ffd640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43840386f0 0x7f438403aba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4388001ed0 tx=0x7f4398067390 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:09.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.043+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f43900153c0 con 0x7f439811db40 2024-09-15T03:37:09.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.043+0000 7f437effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:09.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.043+0000 7f437effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:09.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.202+0000 7f439d59d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:09.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.202+0000 7f439d59d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:09.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.202+0000 7f439d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.202+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f439809a110 con 0x7f439811db40 2024-09-15T03:37:09.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.839+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v215) v1 ==== 168+0+0 (secure 0 0 0) 0x7f439001d340 con 0x7f439811db40 2024-09-15T03:37:09.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.839+0000 7f437effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:09.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.839+0000 7f437effd640 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-15T03:37:09.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.937+0000 7f439d59d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:09.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.937+0000 7f439d59d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:09.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.937+0000 7f439d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:09.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.937+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4398064990 con 0x7f439811db40 2024-09-15T03:37:09.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.938+0000 7f437effd640 1 -- 172.21.15.181:0/224138701 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v215) v1 ==== 147+0+0 (secure 0 0 0) 0x7f439001d990 con 0x7f439811db40 2024-09-15T03:37:09.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.938+0000 7f437effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:09.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.938+0000 7f437effd640 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-15T03:37:09.940 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h31' already exists 2024-09-15T03:37:09.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43840386f0 msgr2=0x7f438403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:09.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43840386f0 0x7f438403aba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4388001ed0 tx=0x7f4398067390 comp rx=0 tx=0).stop 2024-09-15T03:37:09.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 10 monclient: shutdown 2024-09-15T03:37:09.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 msgr2=0x7f439811dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 0x7f439811dfe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f43980a1240 tx=0x7f43900095d0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 shutdown_connections 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43840386f0 0x7f438403aba0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439811db40 0x7f439811dfe0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398130e90 0x7f439811d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 --2- 172.21.15.181:0/224138701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43800040e0 0x7f4398130950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 >> 172.21.15.181:0/224138701 conn(0x7f43980713b0 msgr2=0x7f43980a3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 shutdown_connections 2024-09-15T03:37:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:09.941+0000 7f439d59d640 1 -- 172.21.15.181:0/224138701 wait complete. 2024-09-15T03:37:09.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:09.951 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-15T03:37:10.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.026+0000 7f099239d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:10.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.026+0000 7f099239d640 10 monclient: build_initial_monmap 2024-09-15T03:37:10.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.026+0000 7f099239d640 10 monclient: monmap: 2024-09-15T03:37:10.027 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:10.027234+0000 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:10.027234+0000 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:10.028 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:10.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 Processor -- start 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 -- start start 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient: init 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f098c06e2f0 0x7f098c06bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): picked mon.noname-c con 0x7f098c06e2f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f098c09d980 0x7f098c09dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): picked mon.noname-a con 0x7f098c09d980 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c09e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): picked mon.noname-b con 0x7f098c09e2c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:10.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098c06a2c0 con 0x7f098c09d980 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098c099ed0 con 0x7f098c06e2f0 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c09f6e0 con 0x7f098c09e2c0 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099239d640 10 monclient(hunting): authenticate will time out at 1234.508718s 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f098c06e2f0 0x7f098c06bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099093a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 msgr2=0x7f098c09e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f098c06e2f0 0x7f098c06bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51474/0 (socket says 172.21.15.181:51474) 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 1 -- 172.21.15.181:0/1430869670 learned_addr learned my addr 172.21.15.181:0/1430869670 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): get_auth_request con 0x7f098c06e2f0 auth_method 0 2024-09-15T03:37:10.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f099093a640 1 --2- 172.21.15.181:0/1430869670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c09e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.027+0000 7f098bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098b7fe640 1 --2- 172.21.15.181:0/1430869670 >> v2:172.21.15.153:3300/0 conn(0x7f098c09d980 0x7f098c09dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098b7fe640 10 monclient(hunting): get_auth_request con 0x7f098c09d980 auth_method 0 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 10 monclient(hunting): handle_auth_done global_id 104464 payload 274 2024-09-15T03:37:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 1 -- 172.21.15.181:0/1430869670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 msgr2=0x7f098c09e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 1 --2- 172.21.15.181:0/1430869670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c09e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 1 -- 172.21.15.181:0/1430869670 >> v2:172.21.15.153:3300/0 conn(0x7f098c09d980 msgr2=0x7f098c09dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 1 --2- 172.21.15.181:0/1430869670 >> v2:172.21.15.153:3300/0 conn(0x7f098c09d980 0x7f098c09dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 1 monclient: found mon.noname-c 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.028+0000 7f098bfff640 1 -- 172.21.15.181:0/1430869670 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c0699b0 con 0x7f098c06e2f0 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.030+0000 7f098bfff640 1 -- 172.21.15.181:0/1430869670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 msgr2=0x7f098c06bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.030+0000 7f098bfff640 1 --2- 172.21.15.181:0/1430869670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c06bd80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f098c0670d0 tx=0x7f097c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=139400fce54a15de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 1 -- 172.21.15.181:0/1430869670 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c032440 con 0x7f098c06e2f0 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: dump: 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: _finish_auth 0 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:10.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 5 monclient: authenticate success, global_id 104464 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 1 -- 172.21.15.181:0/1430869670 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c0325a0 con 0x7f098c06e2f0 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 1 -- 172.21.15.181:0/1430869670 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c032940 con 0x7f098c06e2f0 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 4 set_mon_vals no callback set 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f098affd640 10 monclient: dump: 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.045 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 10 monclient: shutdown 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:10.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 -- 172.21.15.181:0/1430869670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 msgr2=0x7f098c06bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 --2- 172.21.15.181:0/1430869670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c06bd80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f098c0670d0 tx=0x7f097c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 -- 172.21.15.181:0/1430869670 shutdown_connections 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 --2- 172.21.15.181:0/1430869670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c09e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 --2- 172.21.15.181:0/1430869670 >> v2:172.21.15.153:3300/0 conn(0x7f098c09d980 0x7f098c09dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 --2- 172.21.15.181:0/1430869670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c06bd80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 -- 172.21.15.181:0/1430869670 >> 172.21.15.181:0/1430869670 conn(0x7f098c0713b0 msgr2=0x7f098c0a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 -- 172.21.15.181:0/1430869670 shutdown_connections 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.031+0000 7f099239d640 1 -- 172.21.15.181:0/1430869670 wait complete. 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient: build_initial_monmap 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient: monmap: 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:10.049 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 1 Processor -- start 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 1 -- start start 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient: init 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient(hunting): picked mon.noname-b con 0x7f098c06e2f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c09d980 0x7f098c11da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 10 monclient(hunting): picked mon.noname-a con 0x7f098c09d980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.032+0000 7f099239d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c11df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 10 monclient(hunting): picked mon.noname-c con 0x7f098c09e2c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c121100 con 0x7f098c09d980 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c1213e0 con 0x7f098c06e2f0 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c120e20 con 0x7f098c09e2c0 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099239d640 10 monclient(hunting): authenticate will time out at 1234.514045s 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099093a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 msgr2=0x7f098c11df40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c133d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51486/0 (socket says 172.21.15.181:51486) 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f099093a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c11df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 -- 172.21.15.181:0/518454650 learned_addr learned my addr 172.21.15.181:0/518454650 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): get_auth_request con 0x7f098c06e2f0 auth_method 0 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c09d980 0x7f098c11da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:10.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): get_auth_request con 0x7f098c09d980 auth_method 0 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient(hunting): handle_auth_done global_id 104467 payload 824 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 -- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 msgr2=0x7f098c11df40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c11df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 -- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c09d980 msgr2=0x7f098c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c09d980 0x7f098c11da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 monclient: found mon.noname-b 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.033+0000 7f098bfff640 1 -- 172.21.15.181:0/518454650 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c009450 con 0x7f098c06e2f0 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.035+0000 7f098bfff640 1 -- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 msgr2=0x7f098c133d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.035+0000 7f098bfff640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c133d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f097c009420 tx=0x7f097c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.035+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c031a90 con 0x7f098c06e2f0 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: dump: 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:10.054 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c031bf0 con 0x7f098c06e2f0 2024-09-15T03:37:10.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 5 monclient: authenticate success, global_id 104467 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c03d420 con 0x7f098c06e2f0 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f0973fff640 10 monclient: dump: 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:10.056 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 10 monclient: _renew_subs 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f098a7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c141e30 con 0x7f098c06e2f0 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 10 monclient: _renew_subs 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.037+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c11fec0 con 0x7f098c06e2f0 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f097c039040 con 0x7f098c06e2f0 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f099239d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f099239d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:10.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f099239d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c11f250 con 0x7f098c06e2f0 2024-09-15T03:37:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f0973fff640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0978038740 0x7f097803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(215..215 src has 189..215) v4 ==== 14695+0+0 (secure 0 0 0) 0x7f097c06dd90 con 0x7f098c06e2f0 2024-09-15T03:37:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.038+0000 7f098b7fe640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0978038740 0x7f097803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.039+0000 7f098b7fe640 10 monclient: get_auth_request con 0x7f0978038740 auth_method 0 2024-09-15T03:37:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.041+0000 7f098b7fe640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0978038740 0x7f097803abf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0980007ba0 tx=0x7f0980008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:10.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.043+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f097c072070 con 0x7f098c06e2f0 2024-09-15T03:37:10.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.043+0000 7f0973fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:10.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.043+0000 7f0973fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:10.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.200+0000 7f099239d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:10.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.200+0000 7f099239d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:10.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.200+0000 7f099239d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.202 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.200+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f098c04bf60 con 0x7f098c06e2f0 2024-09-15T03:37:10.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.852+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v216) v1 ==== 168+0+0 (secure 0 0 0) 0x7f097c0370b0 con 0x7f098c06e2f0 2024-09-15T03:37:10.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.852+0000 7f0973fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:10.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.852+0000 7f0973fff640 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-15T03:37:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.947+0000 7f099239d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.947+0000 7f099239d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.947+0000 7f099239d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.947+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f098c072640 con 0x7f098c06e2f0 2024-09-15T03:37:10.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.948+0000 7f0973fff640 1 -- 172.21.15.181:0/518454650 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v216) v1 ==== 147+0+0 (secure 0 0 0) 0x7f097c030e60 con 0x7f098c06e2f0 2024-09-15T03:37:10.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.948+0000 7f0973fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-15T03:37:10.949 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.948+0000 7f0973fff640 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-15T03:37:10.950 INFO:tasks.workunit.client.0.smithi181.stderr:bucket 'h32' already exists 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0978038740 msgr2=0x7f097803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0978038740 0x7f097803abf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0980007ba0 tx=0x7f0980008040 comp rx=0 tx=0).stop 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 10 monclient: shutdown 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 msgr2=0x7f098c133d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c133d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f097c009420 tx=0x7f097c032000 comp rx=0 tx=0).stop 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 shutdown_connections 2024-09-15T03:37:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0978038740 0x7f097803abf0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f098c09e2c0 0x7f098c11df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c09d980 0x7f098c11da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 --2- 172.21.15.181:0/518454650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f098c06e2f0 0x7f098c133d00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 >> 172.21.15.181:0/518454650 conn(0x7f098c0713b0 msgr2=0x7f098c0690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 shutdown_connections 2024-09-15T03:37:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:10.950+0000 7f099239d640 1 -- 172.21.15.181:0/518454650 wait complete. 2024-09-15T03:37:10.960 INFO:tasks.workunit.client.0.smithi181.stderr:++ seq 1 32 2024-09-15T03:37:10.961 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:10.961 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h1 2024-09-15T03:37:11.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.034+0000 7fdeaf294640 10 monclient: get_monmap_and_config 2024-09-15T03:37:11.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.034+0000 7fdeaf294640 10 monclient: build_initial_monmap 2024-09-15T03:37:11.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.034+0000 7fdeaf294640 10 monclient: monmap: 2024-09-15T03:37:11.036 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:11.036 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:11.036 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:11.034998+0000 2024-09-15T03:37:11.036 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:11.034998+0000 2024-09-15T03:37:11.036 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:11.036 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:11.037 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:11.037 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:11.037 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:11.037 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:11.037 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:11.037 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.034+0000 7fdeaf294640 1 Processor -- start 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 1 -- start start 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient: init 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806e130 0x7fdea809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): picked mon.noname-b con 0x7fdea806e130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdea806a790 0x7fdea809dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): picked mon.noname-a con 0x7fdea806a790 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdea806e500 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea806a0b0 con 0x7fdea806e500 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea8068c30 con 0x7fdea806a790 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea8069750 con 0x7fdea806e130 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeaf294640 10 monclient(hunting): authenticate will time out at 1235.516573s 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdead831640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fdea806e500 0x7fdea809e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35926/0 (socket says 172.21.15.181:35926) 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdead831640 1 -- 172.21.15.181:0/2240918191 learned_addr learned my addr 172.21.15.181:0/2240918191 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdead030640 1 -- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806e130 msgr2=0x7fdea809da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdead030640 1 --2- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806e130 0x7fdea809da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeac82f640 1 --2- 172.21.15.181:0/2240918191 >> v2:172.21.15.153:3300/0 conn(0x7fdea806a790 0x7fdea809dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:11.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeac82f640 10 monclient(hunting): get_auth_request con 0x7fdea806a790 auth_method 0 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeac82f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeac82f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.035+0000 7fdeac82f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 10 monclient(hunting): handle_auth_done global_id 114276 payload 274 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 1 -- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806e130 msgr2=0x7fdea809da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 1 --2- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806e130 0x7fdea809da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 1 -- 172.21.15.181:0/2240918191 >> v1:172.21.15.181:6789/0 conn(0x7fdea806e500 legacy=0x7fdea809e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 10 monclient: _finish_hunting 0 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 1 monclient: found mon.noname-a 2024-09-15T03:37:11.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.036+0000 7fdeac82f640 1 -- 172.21.15.181:0/2240918191 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8069a60 con 0x7fdea806a790 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fdeac82f640 1 -- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806a790 msgr2=0x7fdea809dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fdeac82f640 1 --2- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806a790 0x7fdea809dfa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fde98009850 tx=0x7fde980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0041d5d3c3a79be server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 1 -- 172.21.15.181:0/2240918191 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98032440 con 0x7fdea806a790 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: dump: 2024-09-15T03:37:11.052 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:11.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: _finish_auth 0 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fdeaf294640 5 monclient: authenticate success, global_id 114276 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fdeaf294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 1 -- 172.21.15.181:0/2240918191 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde980325a0 con 0x7fdea806a790 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fdeaf294640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fdeaf294640 4 set_mon_vals no callback set 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 1 -- 172.21.15.181:0/2240918191 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98032980 con 0x7fdea806a790 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.039+0000 7fde9ffff640 10 monclient: dump: 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:11.054 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 10 monclient: shutdown 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:11.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 -- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806a790 msgr2=0x7fdea809dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806a790 0x7fdea809dfa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fde98009850 tx=0x7fde980090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 -- 172.21.15.181:0/2240918191 shutdown_connections 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806a790 0x7fdea809dfa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2240918191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806e130 0x7fdea809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 -- 172.21.15.181:0/2240918191 >> 172.21.15.181:0/2240918191 conn(0x7fdea8071340 msgr2=0x7fdea80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 -- 172.21.15.181:0/2240918191 shutdown_connections 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 1 -- 172.21.15.181:0/2240918191 wait complete. 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 10 monclient: build_initial_monmap 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.040+0000 7fdeaf294640 10 monclient: monmap: 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:11.056 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 Processor -- start 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 -- start start 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient: init 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806a790 0x7fdea8141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): picked mon.noname-c con 0x7fdea806a790 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:11.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806e130 0x7fdea8133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): picked mon.noname-a con 0x7fdea806e130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 0x7fdea811e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): picked mon.noname-b con 0x7fdea806e500 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea806e9a0 con 0x7fdea806e130 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea8121fd0 con 0x7fdea806e500 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea8122280 con 0x7fdea806a790 2024-09-15T03:37:11.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdeaf294640 10 monclient(hunting): authenticate will time out at 1235.522759s 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 0x7fdea811e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 0x7fdea811e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51494/0 (socket says 172.21.15.181:51494) 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 1 -- 172.21.15.181:0/2554719734 learned_addr learned my addr 172.21.15.181:0/2554719734 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): get_auth_request con 0x7fdea806e500 auth_method 0 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead030640 1 -- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806a790 msgr2=0x7fdea8141090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead030640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806a790 0x7fdea8141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:11.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.041+0000 7fdead831640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdeac82f640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806e130 0x7fdea8133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 10 monclient(hunting): handle_auth_done global_id 104470 payload 824 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 1 -- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806a790 msgr2=0x7fdea8141090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806a790 0x7fdea8141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 1 -- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806e130 msgr2=0x7fdea8133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806e130 0x7fdea8133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 10 monclient: _finish_hunting 0 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 1 monclient: found mon.noname-b 2024-09-15T03:37:11.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdead831640 1 -- 172.21.15.181:0/2554719734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde98009390 con 0x7fdea806e500 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdeac82f640 10 monclient: get_auth_request con 0x7fdea806e130 auth_method 0 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.042+0000 7fdeac82f640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806e130 0x7fdea8133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdead831640 1 -- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 msgr2=0x7fdea811e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdead831640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 0x7fdea811e990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdea000ccc0 tx=0x7fdea0005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000ce80 con 0x7fdea806e500 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: dump: 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:11.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:11.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 5 monclient: authenticate success, global_id 104470 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 10 monclient: _renew_subs 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea8122560 con 0x7fdea806e500 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea000aa60 con 0x7fdea806e500 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000ae20 con 0x7fdea806e500 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9d7fa640 10 monclient: dump: 2024-09-15T03:37:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:11.064 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:11.064 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.064 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:11.064 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:11.064 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:11.065 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:11.065 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:11.065 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:11.065 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:11.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 10 monclient: _renew_subs 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fde9f7fe640 4 set_mon_vals no callback set 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.044+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea81415d0 con 0x7fdea806e500 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdea0007300 con 0x7fdea806e500 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fde9d7fa640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde8c0386f0 0x7fde8c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fdeaf294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:11.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fdeaf294640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:11.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fdeaf294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:11.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(216..216 src has 189..216) v4 ==== 14908+0+0 (secure 0 0 0) 0x7fdea004e510 con 0x7fdea806e500 2024-09-15T03:37:11.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea8141850 con 0x7fdea806e500 2024-09-15T03:37:11.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fdead030640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde8c0386f0 0x7fde8c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:11.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.045+0000 7fdead030640 10 monclient: get_auth_request con 0x7fde8c0386f0 auth_method 0 2024-09-15T03:37:11.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.048+0000 7fdead030640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde8c0386f0 0x7fde8c03aba0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdea40038f0 tx=0x7fdea4006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:11.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.050+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdea001b400 con 0x7fdea806e500 2024-09-15T03:37:11.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.050+0000 7fde9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:11.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.050+0000 7fde9d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:11.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.208+0000 7fdeaf294640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T03:37:11.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.208+0000 7fdeaf294640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T03:37:11.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.208+0000 7fdeaf294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:11.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.208+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7fdea81226f0 con 0x7fdea806e500 2024-09-15T03:37:11.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.852+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v217) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdea0010af0 con 0x7fdea806e500 2024-09-15T03:37:11.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.852+0000 7fde9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T03:37:11.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.852+0000 7fde9d7fa640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-15T03:37:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.952+0000 7fdeaf294640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T03:37:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.952+0000 7fdeaf294640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T03:37:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.952+0000 7fdeaf294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.952+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7fdea80008d0 con 0x7fdea806e500 2024-09-15T03:37:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.954+0000 7fde9d7fa640 1 -- 172.21.15.181:0/2554719734 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v217) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdea0010cd0 con 0x7fdea806e500 2024-09-15T03:37:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.954+0000 7fde9d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-15T03:37:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.954+0000 7fde9d7fa640 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-15T03:37:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h1' does not appear in the crush map 2024-09-15T03:37:11.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde8c0386f0 msgr2=0x7fde8c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde8c0386f0 0x7fde8c03aba0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdea40038f0 tx=0x7fdea4006d80 comp rx=0 tx=0).stop 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 10 monclient: shutdown 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 msgr2=0x7fdea811e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 0x7fdea811e990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdea000ccc0 tx=0x7fdea0005370 comp rx=0 tx=0).stop 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 shutdown_connections 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde8c0386f0 0x7fde8c03aba0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdea806e500 0x7fdea811e990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdea806e130 0x7fdea8133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 --2- 172.21.15.181:0/2554719734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdea806a790 0x7fdea8141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:11.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 >> 172.21.15.181:0/2554719734 conn(0x7fdea8071340 msgr2=0x7fdea8098a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:11.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 shutdown_connections 2024-09-15T03:37:11.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:11.956+0000 7fdeaf294640 1 -- 172.21.15.181:0/2554719734 wait complete. 2024-09-15T03:37:11.966 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:11.966 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h2 2024-09-15T03:37:12.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.039+0000 7fe52264f640 10 monclient: get_monmap_and_config 2024-09-15T03:37:12.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.039+0000 7fe52264f640 10 monclient: build_initial_monmap 2024-09-15T03:37:12.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.040+0000 7fe52264f640 10 monclient: monmap: 2024-09-15T03:37:12.041 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:12.041 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:12.041 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:12.040794+0000 2024-09-15T03:37:12.041 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:12.040794+0000 2024-09-15T03:37:12.042 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:12.042 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:12.042 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:12.042 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:12.042 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:12.042 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:12.043 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:12.043 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:12.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.040+0000 7fe52264f640 1 Processor -- start 2024-09-15T03:37:12.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.040+0000 7fe52264f640 1 -- start start 2024-09-15T03:37:12.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.040+0000 7fe52264f640 10 monclient: init 2024-09-15T03:37:12.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:12.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe51c06e000 0x7fe51c06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): picked mon.noname-c con 0x7fe51c06e000 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c09da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): picked mon.noname-b con 0x7fe51c06a710 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe51c09dfa0 0x7fe51c09e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): picked mon.noname-a con 0x7fe51c09dfa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51c06a360 con 0x7fe51c09dfa0 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:12.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51c0699c0 con 0x7fe51c06e000 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c06be20 con 0x7fe51c06a710 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe52264f640 10 monclient(hunting): authenticate will time out at 1236.522323s 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe51c06e000 0x7fe51c06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51b7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 msgr2=0x7fe51c09da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe51c06e000 0x7fe51c06e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51500/0 (socket says 172.21.15.181:51500) 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 -- 172.21.15.181:0/3370854549 learned_addr learned my addr 172.21.15.181:0/3370854549 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): get_auth_request con 0x7fe51c06e000 auth_method 0 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:12.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51b7fe640 1 --2- 172.21.15.181:0/3370854549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c09da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51affd640 1 -- 172.21.15.181:0/3370854549 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 1 --2- 172.21.15.181:0/3370854549 >> v2:172.21.15.153:3300/0 conn(0x7fe51c09dfa0 0x7fe51c09e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): get_auth_request con 0x7fe51c09dfa0 auth_method 0 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe520bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient(hunting): handle_auth_done global_id 104473 payload 274 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 -- 172.21.15.181:0/3370854549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 msgr2=0x7fe51c09da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 --2- 172.21.15.181:0/3370854549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c09da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 -- 172.21.15.181:0/3370854549 >> v2:172.21.15.153:3300/0 conn(0x7fe51c09dfa0 msgr2=0x7fe51c09e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 --2- 172.21.15.181:0/3370854549 >> v2:172.21.15.153:3300/0 conn(0x7fe51c09dfa0 0x7fe51c09e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 monclient: found mon.noname-c 2024-09-15T03:37:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.041+0000 7fe51bfff640 1 -- 172.21.15.181:0/3370854549 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c069f80 con 0x7fe51c06e000 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51bfff640 1 -- 172.21.15.181:0/3370854549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c06e000 msgr2=0x7fe51c06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51bfff640 1 --2- 172.21.15.181:0/3370854549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c06e000 0x7fe51c06e3d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe51c063870 tx=0x7fe50c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20432defb3ed7a1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 1 -- 172.21.15.181:0/3370854549 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c032440 con 0x7fe51c06e000 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: dump: 2024-09-15T03:37:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: _finish_auth 0 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe52264f640 5 monclient: authenticate success, global_id 104473 2024-09-15T03:37:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe52264f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 1 -- 172.21.15.181:0/3370854549 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50c0325a0 con 0x7fe51c06e000 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe52264f640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 1 -- 172.21.15.181:0/3370854549 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c032940 con 0x7fe51c06e000 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe52264f640 4 set_mon_vals no callback set 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.044+0000 7fe51affd640 10 monclient: dump: 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 10 monclient: shutdown 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:12.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 -- 172.21.15.181:0/3370854549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c06e000 msgr2=0x7fe51c06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 --2- 172.21.15.181:0/3370854549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c06e000 0x7fe51c06e3d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe51c063870 tx=0x7fe50c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 -- 172.21.15.181:0/3370854549 shutdown_connections 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 --2- 172.21.15.181:0/3370854549 >> v2:172.21.15.153:3300/0 conn(0x7fe51c09dfa0 0x7fe51c09e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 --2- 172.21.15.181:0/3370854549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c09da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 --2- 172.21.15.181:0/3370854549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c06e000 0x7fe51c06e3d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 -- 172.21.15.181:0/3370854549 >> 172.21.15.181:0/3370854549 conn(0x7fe51c071340 msgr2=0x7fe51c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 -- 172.21.15.181:0/3370854549 shutdown_connections 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 1 -- 172.21.15.181:0/3370854549 wait complete. 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 10 monclient: build_initial_monmap 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.045+0000 7fe52264f640 10 monclient: monmap: 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:12.061 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 Processor -- start 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 -- start start 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient: init 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:12.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): picked mon.noname-c con 0x7fe51c06a710 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe51c06e000 0x7fe51c133d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): picked mon.noname-a con 0x7fe51c06e000 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 0x7fe51c11e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): picked mon.noname-b con 0x7fe51c09dfa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c06e970 con 0x7fe51c06e000 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c122130 con 0x7fe51c09dfa0 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c120f60 con 0x7fe51c06a710 2024-09-15T03:37:12.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe52264f640 10 monclient(hunting): authenticate will time out at 1236.527760s 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 0x7fe51c11e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe51bfff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 msgr2=0x7fe51c134f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe51bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c134f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 0x7fe51c11e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51504/0 (socket says 172.21.15.181:51504) 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 1 -- 172.21.15.181:0/2273171088 learned_addr learned my addr 172.21.15.181:0/2273171088 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): get_auth_request con 0x7fe51c09dfa0 auth_method 0 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:12.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.046+0000 7fe520bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 10 monclient(hunting): handle_auth_done global_id 104476 payload 824 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe51b7fe640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe51c06e000 0x7fe51c133d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 1 -- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 msgr2=0x7fe51c134f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c134f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 1 -- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe51c06e000 msgr2=0x7fe51c133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe51c06e000 0x7fe51c133d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 10 monclient: _finish_hunting 0 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 1 monclient: found mon.noname-b 2024-09-15T03:37:12.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe520bec640 1 -- 172.21.15.181:0/2273171088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c009450 con 0x7fe51c09dfa0 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe51b7fe640 10 monclient: get_auth_request con 0x7fe51c06e000 auth_method 0 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.047+0000 7fe51b7fe640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe51c06e000 0x7fe51c133d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe520bec640 1 -- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 msgr2=0x7fe51c11e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe520bec640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 0x7fe51c11e890 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe514009e90 tx=0x7fe51400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514014480 con 0x7fe51c09dfa0 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: dump: 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:12.066 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 5 monclient: authenticate success, global_id 104476 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 10 monclient: _renew_subs 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51400f040 con 0x7fe51c09dfa0 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51c121e50 con 0x7fe51c09dfa0 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514014830 con 0x7fe51c09dfa0 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe503fff640 10 monclient: dump: 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:12.068 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 10 monclient: _renew_subs 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51c134260 con 0x7fe51c09dfa0 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.049+0000 7fe51a7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe514014be0 con 0x7fe51c09dfa0 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe52264f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe52264f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe52264f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51c122520 con 0x7fe51c09dfa0 2024-09-15T03:37:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe503fff640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5080386f0 0x7fe50803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(217..217 src has 189..217) v4 ==== 14698+0+0 (secure 0 0 0) 0x7fe51404ee70 con 0x7fe51c09dfa0 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe51bfff640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5080386f0 0x7fe50803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.050+0000 7fe51bfff640 10 monclient: get_auth_request con 0x7fe5080386f0 auth_method 0 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.053+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.053+0000 7fe51bfff640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5080386f0 0x7fe50803aba0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe51c06e940 tx=0x7fe50c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.055+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe5140118c0 con 0x7fe51c09dfa0 2024-09-15T03:37:12.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.055+0000 7fe503fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:12.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.055+0000 7fe503fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:12.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.215+0000 7fe52264f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T03:37:12.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.215+0000 7fe52264f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T03:37:12.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.215+0000 7fe52264f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.216 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.215+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7fe51c0008d0 con 0x7fe51c09dfa0 2024-09-15T03:37:12.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.862+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v218) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe514018030 con 0x7fe51c09dfa0 2024-09-15T03:37:12.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.862+0000 7fe503fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T03:37:12.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.862+0000 7fe503fff640 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-15T03:37:12.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.961+0000 7fe52264f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T03:37:12.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.961+0000 7fe52264f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T03:37:12.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.961+0000 7fe52264f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:12.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.961+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7fe51c1222a0 con 0x7fe51c09dfa0 2024-09-15T03:37:12.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.962+0000 7fe503fff640 1 -- 172.21.15.181:0/2273171088 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v218) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe514018020 con 0x7fe51c09dfa0 2024-09-15T03:37:12.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.962+0000 7fe503fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-15T03:37:12.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.962+0000 7fe503fff640 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-15T03:37:12.964 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h2' does not appear in the crush map 2024-09-15T03:37:12.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5080386f0 msgr2=0x7fe50803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:12.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5080386f0 0x7fe50803aba0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe51c06e940 tx=0x7fe50c032000 comp rx=0 tx=0).stop 2024-09-15T03:37:12.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 10 monclient: shutdown 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 msgr2=0x7fe51c11e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 0x7fe51c11e890 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe514009e90 tx=0x7fe51400b040 comp rx=0 tx=0).stop 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 shutdown_connections 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe5080386f0 0x7fe50803aba0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c09dfa0 0x7fe51c11e890 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe51c06e000 0x7fe51c133d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.964+0000 7fe52264f640 1 --2- 172.21.15.181:0/2273171088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe51c06a710 0x7fe51c134f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:12.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.965+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 >> 172.21.15.181:0/2273171088 conn(0x7fe51c071340 msgr2=0x7fe51c0a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:12.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.965+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 shutdown_connections 2024-09-15T03:37:12.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:12.965+0000 7fe52264f640 1 -- 172.21.15.181:0/2273171088 wait complete. 2024-09-15T03:37:12.974 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:12.975 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h3 2024-09-15T03:37:13.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.047+0000 7f4dff860640 10 monclient: get_monmap_and_config 2024-09-15T03:37:13.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.047+0000 7f4dff860640 10 monclient: build_initial_monmap 2024-09-15T03:37:13.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.047+0000 7f4dff860640 10 monclient: monmap: 2024-09-15T03:37:13.050 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:13.047877+0000 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:13.047877+0000 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.047+0000 7f4dff860640 1 Processor -- start 2024-09-15T03:37:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.047+0000 7f4dff860640 1 -- start start 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.047+0000 7f4dff860640 10 monclient: init 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4df809eaa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4df8069d10 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4df813c2e0 0x7f4df813e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient(hunting): picked mon.noname-a con 0x7f4df813c2e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df809c850 con 0x7f4df809eaa0 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80a17a0 con 0x7f4df8069d10 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:13.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df809f580 con 0x7f4df813c2e0 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dff860640 10 monclient(hunting): authenticate will time out at 1237.529396s 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfcdfb640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4df8069d10 0x7f4df809d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35934/0 (socket says 172.21.15.181:35934) 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfcdfb640 1 -- 172.21.15.181:0/17563084 learned_addr learned my addr 172.21.15.181:0/17563084 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 1 --2- 172.21.15.181:0/17563084 >> v2:172.21.15.153:3300/0 conn(0x7f4df813c2e0 0x7f4df813e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): get_auth_request con 0x7f4df813c2e0 auth_method 0 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_done global_id 114282 payload 274 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 1 -- 172.21.15.181:0/17563084 >> v1:172.21.15.181:6789/0 conn(0x7f4df8069d10 legacy=0x7f4df809d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 1 -- 172.21.15.181:0/17563084 >> v1:172.21.15.153:6789/0 conn(0x7f4df809eaa0 legacy=0x7f4df809ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 1 monclient: found mon.noname-a 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:13.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.048+0000 7f4dfddfd640 1 -- 172.21.15.181:0/17563084 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df806a0e0 con 0x7f4df813c2e0 2024-09-15T03:37:13.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.051+0000 7f4dfddfd640 1 -- 172.21.15.181:0/17563084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 msgr2=0x7f4df813e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:13.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.051+0000 7f4dfddfd640 1 --2- 172.21.15.181:0/17563084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df813e6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009e60 tx=0x7f4df0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdf453c56fb5eb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:13.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 1 -- 172.21.15.181:0/17563084 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0007630 con 0x7f4df813c2e0 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: dump: 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: _finish_auth 0 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 1 -- 172.21.15.181:0/17563084 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df002d050 con 0x7f4df813c2e0 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 1 -- 172.21.15.181:0/17563084 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df00079c0 con 0x7f4df813c2e0 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 5 monclient: authenticate success, global_id 114282 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 4 set_mon_vals no callback set 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:13.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4deffff640 10 monclient: dump: 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:13.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 10 monclient: shutdown 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 -- 172.21.15.181:0/17563084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 msgr2=0x7f4df813e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 --2- 172.21.15.181:0/17563084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df813e6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009e60 tx=0x7f4df0009780 comp rx=0 tx=0).stop 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 -- 172.21.15.181:0/17563084 shutdown_connections 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 --2- 172.21.15.181:0/17563084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df813e6c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 -- 172.21.15.181:0/17563084 >> 172.21.15.181:0/17563084 conn(0x7f4df80a1bf0 msgr2=0x7f4df80a1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 -- 172.21.15.181:0/17563084 shutdown_connections 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.052+0000 7f4dff860640 1 -- 172.21.15.181:0/17563084 wait complete. 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.053+0000 7f4dff860640 10 monclient: build_initial_monmap 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.053+0000 7f4dff860640 10 monclient: monmap: 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:13.062 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.053+0000 7f4dff860640 1 Processor -- start 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.053+0000 7f4dff860640 1 -- start start 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.053+0000 7f4dff860640 10 monclient: init 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:13.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 0x7f4df813bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): picked mon.noname-b con 0x7f4df8069d10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df809eaa0 0x7f4df8145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): picked mon.noname-c con 0x7f4df809eaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df8145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): picked mon.noname-a con 0x7f4df813c2e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8139de0 con 0x7f4df813c2e0 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df813acd0 con 0x7f4df8069d10 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df813b290 con 0x7f4df809eaa0 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:13.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dff860640 10 monclient(hunting): authenticate will time out at 1237.535236s 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 0x7f4df813bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfcdfb640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df809eaa0 msgr2=0x7f4df8145210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfcdfb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df809eaa0 0x7f4df8145210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 0x7f4df813bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51510/0 (socket says 172.21.15.181:51510) 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 -- 172.21.15.181:0/4223094941 learned_addr learned my addr 172.21.15.181:0/4223094941 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): get_auth_request con 0x7f4df8069d10 auth_method 0 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:13.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df8145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): get_auth_request con 0x7f4df813c2e0 auth_method 0 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfddfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient(hunting): handle_auth_done global_id 104479 payload 824 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 -- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df809eaa0 msgr2=0x7f4df8145210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df809eaa0 0x7f4df8145210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 -- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 msgr2=0x7f4df8145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df8145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 monclient: found mon.noname-b 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.054+0000 7f4dfd5fc640 1 -- 172.21.15.181:0/4223094941 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0009a20 con 0x7f4df8069d10 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.056+0000 7f4dfd5fc640 1 -- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 msgr2=0x7f4df813bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.056+0000 7f4dfd5fc640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 0x7f4df813bcc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4df400c5e0 tx=0x7f4df4007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:13.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.056+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40117a0 con 0x7f4df8069d10 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: dump: 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.069 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4011900 con 0x7f4df8069d10 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 5 monclient: authenticate success, global_id 104479 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 10 monclient: _renew_subs 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:13.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df8139210 con 0x7f4df8069d10 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4011d00 con 0x7f4df8069d10 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4ded7fa640 10 monclient: dump: 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:13.071 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4def7fe640 4 set_mon_vals no callback set 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 10 monclient: _renew_subs 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.058+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df8145e30 con 0x7f4df8069d10 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.059+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4df4012630 con 0x7f4df8069d10 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.059+0000 7f4dff860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.059+0000 7f4dff860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.059+0000 7f4dff860640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.059+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df80008d0 con 0x7f4df8069d10 2024-09-15T03:37:13.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.060+0000 7f4ded7fa640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4ddc038740 0x7f4ddc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.060+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(218..218 src has 189..218) v4 ==== 14488+0+0 (secure 0 0 0) 0x7f4df404dda0 con 0x7f4df8069d10 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.060+0000 7f4dfcdfb640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4ddc038740 0x7f4ddc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.060+0000 7f4dfcdfb640 10 monclient: get_auth_request con 0x7f4ddc038740 auth_method 0 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.062+0000 7f4dfcdfb640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4ddc038740 0x7f4ddc03abf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4de80047d0 tx=0x7f4de8007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.064+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4df4002090 con 0x7f4df8069d10 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.064+0000 7f4ded7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.064+0000 7f4ded7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:13.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.223+0000 7f4dff860640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T03:37:13.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.223+0000 7f4dff860640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T03:37:13.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.223+0000 7f4dff860640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:13.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.223+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f4df813b070 con 0x7f4df8069d10 2024-09-15T03:37:13.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.862+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v219) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4df40102e0 con 0x7f4df8069d10 2024-09-15T03:37:13.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.862+0000 7f4ded7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T03:37:13.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.862+0000 7f4ded7fa640 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-15T03:37:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.962+0000 7f4dff860640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T03:37:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.962+0000 7f4dff860640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T03:37:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.962+0000 7f4dff860640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.962+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f4df8146050 con 0x7f4df8069d10 2024-09-15T03:37:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.963+0000 7f4ded7fa640 1 -- 172.21.15.181:0/4223094941 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v219) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4df40108a0 con 0x7f4df8069d10 2024-09-15T03:37:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.963+0000 7f4ded7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-15T03:37:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.963+0000 7f4ded7fa640 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-15T03:37:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h3' does not appear in the crush map 2024-09-15T03:37:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4ddc038740 msgr2=0x7f4ddc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4ddc038740 0x7f4ddc03abf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4de80047d0 tx=0x7f4de8007b80 comp rx=0 tx=0).stop 2024-09-15T03:37:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 10 monclient: shutdown 2024-09-15T03:37:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 msgr2=0x7f4df813bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 0x7f4df813bcc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4df400c5e0 tx=0x7f4df4007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 shutdown_connections 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4ddc038740 0x7f4ddc03abf0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df813c2e0 0x7f4df8145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df809eaa0 0x7f4df8145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 --2- 172.21.15.181:0/4223094941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df8069d10 0x7f4df813bcc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 >> 172.21.15.181:0/4223094941 conn(0x7f4df80a1bf0 msgr2=0x7f4df80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 shutdown_connections 2024-09-15T03:37:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:13.965+0000 7f4dff860640 1 -- 172.21.15.181:0/4223094941 wait complete. 2024-09-15T03:37:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h4 2024-09-15T03:37:14.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.047+0000 7fc8fb2e8640 10 monclient: get_monmap_and_config 2024-09-15T03:37:14.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.047+0000 7fc8fb2e8640 10 monclient: build_initial_monmap 2024-09-15T03:37:14.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.047+0000 7fc8fb2e8640 10 monclient: monmap: 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:14.048412+0000 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:14.048412+0000 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:14.050 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 Processor -- start 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 -- start start 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient: init 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f40677e0 0x7fc8f4095800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f40677e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc8f409c090 0x7fc8f409c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f409c090 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc8f409c9a0 0x7fc8f4094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f409c9a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f409d570 con 0x7fc8f409c9a0 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f409b560 con 0x7fc8f409c090 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f409d800 con 0x7fc8f40677e0 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.048+0000 7fc8fb2e8640 10 monclient(hunting): authenticate will time out at 1238.529959s 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc8f409c090 0x7fc8f409c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f9084640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f40677e0 msgr2=0x7fc8f4095800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f9084640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f40677e0 0x7fc8f4095800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc8f409c090 0x7fc8f409c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51514/0 (socket says 172.21.15.181:51514) 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 -- 172.21.15.181:0/1538598897 learned_addr learned my addr 172.21.15.181:0/1538598897 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): get_auth_request con 0x7fc8f409c090 auth_method 0 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:14.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f9885640 1 --2- 172.21.15.181:0/1538598897 >> v2:172.21.15.153:3300/0 conn(0x7fc8f409c9a0 0x7fc8f4094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient(hunting): handle_auth_done global_id 104482 payload 274 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 -- 172.21.15.181:0/1538598897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f40677e0 msgr2=0x7fc8f4095800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 --2- 172.21.15.181:0/1538598897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f40677e0 0x7fc8f4095800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 -- 172.21.15.181:0/1538598897 >> v2:172.21.15.153:3300/0 conn(0x7fc8f409c9a0 msgr2=0x7fc8f4094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 --2- 172.21.15.181:0/1538598897 >> v2:172.21.15.153:3300/0 conn(0x7fc8f409c9a0 0x7fc8f4094080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient: _finish_hunting 0 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 monclient: found mon.noname-c 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f8883640 1 -- 172.21.15.181:0/1538598897 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f409cdc0 con 0x7fc8f409c090 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f9885640 10 monclient: get_auth_request con 0x7fc8f409c9a0 auth_method 0 2024-09-15T03:37:14.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.049+0000 7fc8f9885640 1 --2- 172.21.15.181:0/1538598897 >> v2:172.21.15.153:3300/0 conn(0x7fc8f409c9a0 0x7fc8f4094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:14.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8f8883640 1 -- 172.21.15.181:0/1538598897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 msgr2=0x7fc8f409c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:14.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8f8883640 1 --2- 172.21.15.181:0/1538598897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f409c460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009910 tx=0x7fc8e40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b93d7aac818e8690 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:14.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 1 -- 172.21.15.181:0/1538598897 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4032440 con 0x7fc8f409c090 2024-09-15T03:37:14.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:14.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:14.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: dump: 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: _finish_auth 0 2024-09-15T03:37:14.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 5 monclient: authenticate success, global_id 104482 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 1 -- 172.21.15.181:0/1538598897 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e40325a0 con 0x7fc8f409c090 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 1 -- 172.21.15.181:0/1538598897 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4032940 con 0x7fc8f409c090 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 4 set_mon_vals no callback set 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:14.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8ebfff640 10 monclient: dump: 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 10 monclient: shutdown 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:14.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/1538598897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 msgr2=0x7fc8f409c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/1538598897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f409c460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009910 tx=0x7fc8e40091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/1538598897 shutdown_connections 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.052+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/1538598897 >> v2:172.21.15.153:3300/0 conn(0x7fc8f409c9a0 0x7fc8f4094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/1538598897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f409c460 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/1538598897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f40677e0 0x7fc8f4095800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/1538598897 >> 172.21.15.181:0/1538598897 conn(0x7fc8f40a16a0 msgr2=0x7fc8f40a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/1538598897 shutdown_connections 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/1538598897 wait complete. 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 10 monclient: build_initial_monmap 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 10 monclient: monmap: 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:14.065 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 Processor -- start 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 1 -- start start 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.053+0000 7fc8fb2e8640 10 monclient: init 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f40677e0 0x7fc8f412b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f40677e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:14.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f412a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f409c090 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f409c9a0 0x7fc8f4131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f409c9a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f412d520 con 0x7fc8f40677e0 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f412d770 con 0x7fc8f409c090 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f412b4a0 con 0x7fc8f409c9a0 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8fb2e8640 10 monclient(hunting): authenticate will time out at 1238.535502s 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f412a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f412a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51516/0 (socket says 172.21.15.181:51516) 2024-09-15T03:37:14.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 -- 172.21.15.181:0/3261128149 learned_addr learned my addr 172.21.15.181:0/3261128149 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): get_auth_request con 0x7fc8f409c090 auth_method 0 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9885640 1 -- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f409c9a0 msgr2=0x7fc8f4131cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9885640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f409c9a0 0x7fc8f4131cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9084640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f40677e0 0x7fc8f412b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9084640 10 monclient(hunting): get_auth_request con 0x7fc8f40677e0 auth_method 0 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9084640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:14.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9084640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f9084640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient(hunting): handle_auth_done global_id 104485 payload 824 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 -- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f409c9a0 msgr2=0x7fc8f4131cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f409c9a0 0x7fc8f4131cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 -- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f40677e0 msgr2=0x7fc8f412b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f40677e0 0x7fc8f412b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient: _finish_hunting 0 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 monclient: found mon.noname-b 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.054+0000 7fc8f8883640 1 -- 172.21.15.181:0/3261128149 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009450 con 0x7fc8f409c090 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8f8883640 1 -- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 msgr2=0x7fc8f412a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:14.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8f8883640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f412a230 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4035040 tx=0x7fc8e4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4031b10 con 0x7fc8f409c090 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: dump: 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4031c90 con 0x7fc8f409c090 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 5 monclient: authenticate success, global_id 104485 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 10 monclient: _renew_subs 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f412a770 con 0x7fc8f409c090 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e403b650 con 0x7fc8f409c090 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:14.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8e97fa640 10 monclient: dump: 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8eb7fe640 4 set_mon_vals no callback set 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 10 monclient: _renew_subs 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.057+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f4129dd0 con 0x7fc8f409c090 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc8e4039040 con 0x7fc8f409c090 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8fb2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8fb2e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8fb2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40008d0 con 0x7fc8f409c090 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8e97fa640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(219..219 src has 189..219) v4 ==== 14278+0+0 (secure 0 0 0) 0x7fc8e406def0 con 0x7fc8f409c090 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8f9084640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.058+0000 7fc8f9084640 10 monclient: get_auth_request con 0x7fc8d80386f0 auth_method 0 2024-09-15T03:37:14.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.060+0000 7fc8f9084640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00038f0 tx=0x7fc8f0006ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:14.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.062+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc8e403bac0 con 0x7fc8f409c090 2024-09-15T03:37:14.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.062+0000 7fc8e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:14.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.062+0000 7fc8e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:14.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.221+0000 7fc8fb2e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T03:37:14.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.221+0000 7fc8fb2e8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T03:37:14.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.221+0000 7fc8fb2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.222 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.221+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fc8f4129f60 con 0x7fc8f409c090 2024-09-15T03:37:14.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.871+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v220) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc8e403f090 con 0x7fc8f409c090 2024-09-15T03:37:14.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.871+0000 7fc8e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T03:37:14.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.871+0000 7fc8e97fa640 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-15T03:37:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.970+0000 7fc8fb2e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T03:37:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.970+0000 7fc8fb2e8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T03:37:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.970+0000 7fc8fb2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.970+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fc8f4095d40 con 0x7fc8f409c090 2024-09-15T03:37:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.971+0000 7fc8e97fa640 1 -- 172.21.15.181:0/3261128149 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v220) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc8e4037020 con 0x7fc8f409c090 2024-09-15T03:37:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.971+0000 7fc8e97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-15T03:37:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.971+0000 7fc8e97fa640 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-15T03:37:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h4' does not appear in the crush map 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.973+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 msgr2=0x7fc8d803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00038f0 tx=0x7fc8f0006ef0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 10 monclient: shutdown 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 msgr2=0x7fc8f412a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f412a230 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4035040 tx=0x7fc8e4032000 comp rx=0 tx=0).stop 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 shutdown_connections 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d80386f0 0x7fc8d803aba0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f409c9a0 0x7fc8f4131cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f409c090 0x7fc8f412a230 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 --2- 172.21.15.181:0/3261128149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f40677e0 0x7fc8f412b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 >> 172.21.15.181:0/3261128149 conn(0x7fc8f40a16a0 msgr2=0x7fc8f40a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 shutdown_connections 2024-09-15T03:37:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:14.974+0000 7fc8fb2e8640 1 -- 172.21.15.181:0/3261128149 wait complete. 2024-09-15T03:37:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h5 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.056+0000 7fb6870d4640 10 monclient: get_monmap_and_config 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.056+0000 7fb6870d4640 10 monclient: build_initial_monmap 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.056+0000 7fb6870d4640 10 monclient: monmap: 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:15.057059+0000 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:15.057059+0000 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.058 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:15.058 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:15.058 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:15.058 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:15.058 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:15.058 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 1 Processor -- start 2024-09-15T03:37:15.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 1 -- start start 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient: init 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb68009ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb68009f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 0x7fb68013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient(hunting): picked mon.noname-b con 0x7fb680069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb680067420 con 0x7fb68009f020 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6800a0480 con 0x7fb68009ec50 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68006a170 con 0x7fb680069d10 2024-09-15T03:37:15.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:15.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb6870d4640 10 monclient(hunting): authenticate will time out at 1239.538727s 2024-09-15T03:37:15.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb684e70640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fb68009ec50 0x7fb680131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35946/0 (socket says 172.21.15.181:35946) 2024-09-15T03:37:15.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb685671640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 msgr2=0x7fb68013a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:15.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb684e70640 1 -- 172.21.15.181:0/2861310874 learned_addr learned my addr 172.21.15.181:0/2861310874 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:15.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.057+0000 7fb685671640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 0x7fb68013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.059+0000 7fb684e70640 1 -- 172.21.15.181:0/2861310874 >> v1:172.21.15.181:6789/0 conn(0x7fb68009ec50 legacy=0x7fb680131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550729218 0 0) 0x7fb6800a0480 con 0x7fb68009ec50 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): my global_id is 104488 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6700034e0 con 0x7fb68009ec50 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67ffff640 1 -- 172.21.15.181:0/2861310874 >> v1:172.21.15.153:6789/0 conn(0x7fb68009f020 legacy=0x7fb680136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139343120 0 0) 0x7fb6700034e0 con 0x7fb68009ec50 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 msgr2=0x7fb68013a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 0x7fb68013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 >> v1:172.21.15.153:6789/0 conn(0x7fb68009f020 legacy=0x7fb680136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6800a0200 con 0x7fb68009ec50 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fb674001720 con 0x7fb68009ec50 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb6870d4640 5 monclient: authenticate success, global_id 104488 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb6870d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: dump: 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:37:15.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 >> v1:172.21.15.181:6789/0 conn(0x7fb68009ec50 legacy=0x7fb680131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb670003a50 0x7fb670003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): picked mon.a con 0x7fb670003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 0x7fb670004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): picked mon.c con 0x7fb680069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb67000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): picked mon.b con 0x7fb670004ec0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670005640 con 0x7fb670003a50 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000bf10 con 0x7fb670004ec0 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000c080 con 0x7fb680069d10 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb6870d4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67ffff640 1 -- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 msgr2=0x7fb670004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb67000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb67ffff640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 0x7fb670004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): get_auth_request con 0x7fb670004ec0 auth_method 0 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb670003a50 0x7fb670003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): get_auth_request con 0x7fb670003a50 auth_method 0 2024-09-15T03:37:15.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb684e70640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient(hunting): handle_auth_done global_id 104488 payload 293 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 -- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 msgr2=0x7fb670004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb680069d10 0x7fb670004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 -- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb670003a50 msgr2=0x7fb670003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:15.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb670003a50 0x7fb670003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient: _finish_hunting 0 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 monclient: found mon.b 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 1 -- 172.21.15.181:0/2861310874 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700034e0 con 0x7fb670004ec0 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 10 monclient: _finish_auth 0 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.060+0000 7fb685671640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb685671640 1 -- 172.21.15.181:0/2861310874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 msgr2=0x7fb67000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb685671640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb67000b9d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb678009f90 tx=0x7fb6780095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeba0a2034f91442 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678032430 con 0x7fb670004ec0 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: dump: 2024-09-15T03:37:15.072 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb6870d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67802d070 con 0x7fb670004ec0 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb6870d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb6870d4640 4 set_mon_vals no callback set 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 1 -- 172.21.15.181:0/2861310874 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678032820 con 0x7fb670004ec0 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.063+0000 7fb67f7fe640 10 monclient: dump: 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 10 monclient: shutdown 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 1 -- 172.21.15.181:0/2861310874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 msgr2=0x7fb67000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 1 --2- 172.21.15.181:0/2861310874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb67000b9d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb678009f90 tx=0x7fb6780095d0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb685671640 1 -- 172.21.15.181:0/2861310874 reap_dead start 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 1 -- 172.21.15.181:0/2861310874 shutdown_connections 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 1 -- 172.21.15.181:0/2861310874 >> 172.21.15.181:0/2861310874 conn(0x7fb6800a5cf0 msgr2=0x7fb6800a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 1 -- 172.21.15.181:0/2861310874 shutdown_connections 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 1 -- 172.21.15.181:0/2861310874 wait complete. 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 10 monclient: build_initial_monmap 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.064+0000 7fb6870d4640 10 monclient: monmap: 2024-09-15T03:37:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 Processor -- start 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 -- start start 2024-09-15T03:37:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient: init 2024-09-15T03:37:15.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:15.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:15.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb68009ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): picked mon.noname-b con 0x7fb670004ec0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680130cb0 0x7fb680131120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): picked mon.noname-a con 0x7fb680130cb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb68011b760 0x7fb68011bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): picked mon.noname-c con 0x7fb68011b760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6800a4310 con 0x7fb680130cb0 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68011ce70 con 0x7fb670004ec0 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb680141ab0 con 0x7fb68011b760 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb6870d4640 10 monclient(hunting): authenticate will time out at 1239.546818s 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb685671640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb68011b760 msgr2=0x7fb68011bbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb68009ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb685671640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb68011b760 0x7fb68011bbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb68009ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51536/0 (socket says 172.21.15.181:51536) 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 1 -- 172.21.15.181:0/1678598327 learned_addr learned my addr 172.21.15.181:0/1678598327 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): get_auth_request con 0x7fb670004ec0 auth_method 0 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.065+0000 7fb684e70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb67ffff640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680130cb0 0x7fb680131120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb67ffff640 10 monclient(hunting): get_auth_request con 0x7fb680130cb0 auth_method 0 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb67ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb67ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb67ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 10 monclient(hunting): handle_auth_done global_id 104491 payload 824 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 1 -- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb68011b760 msgr2=0x7fb68011bbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb68011b760 0x7fb68011bbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 1 -- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680130cb0 msgr2=0x7fb680131120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680130cb0 0x7fb680131120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 10 monclient: _finish_hunting 0 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 1 monclient: found mon.noname-b 2024-09-15T03:37:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb684e70640 1 -- 172.21.15.181:0/1678598327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68011ddd0 con 0x7fb670004ec0 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.066+0000 7fb67ffff640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680130cb0 0x7fb680131120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.068+0000 7fb684e70640 1 -- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 msgr2=0x7fb68009ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.068+0000 7fb684e70640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb68009ab40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb674007110 tx=0x7fb674007a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.068+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674018030 con 0x7fb670004ec0 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: dump: 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:15.084 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67400f9c0 con 0x7fb670004ec0 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400fd80 con 0x7fb670004ec0 2024-09-15T03:37:15.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67cff9640 10 monclient: dump: 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:15.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 5 monclient: authenticate success, global_id 104491 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 10 monclient: _renew_subs 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68011e0b0 con 0x7fb670004ec0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb67effd640 4 set_mon_vals no callback set 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 10 monclient: _renew_subs 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.070+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb680141140 con 0x7fb670004ec0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb67401b070 con 0x7fb670004ec0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb67cff9640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb664038770 0x7fb66403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb6870d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb6870d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb6870d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68009c090 con 0x7fb670004ec0 2024-09-15T03:37:15.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(220..220 src has 194..220) v4 ==== 14068+0+0 (secure 0 0 0) 0x7fb674019070 con 0x7fb670004ec0 2024-09-15T03:37:15.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb67ffff640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb664038770 0x7fb66403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:15.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.071+0000 7fb67ffff640 10 monclient: get_auth_request con 0x7fb664038770 auth_method 0 2024-09-15T03:37:15.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.074+0000 7fb67ffff640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb664038770 0x7fb66403ac20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb68009bbc0 tx=0x7fb66c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:15.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.076+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb68009c090 con 0x7fb670004ec0 2024-09-15T03:37:15.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.076+0000 7fb67cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:15.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.076+0000 7fb67cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:15.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.235+0000 7fb6870d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T03:37:15.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.235+0000 7fb6870d4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T03:37:15.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.235+0000 7fb6870d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.235+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7fb68009c090 con 0x7fb670004ec0 2024-09-15T03:37:15.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.872+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v221) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb68009c090 con 0x7fb670004ec0 2024-09-15T03:37:15.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.872+0000 7fb67cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T03:37:15.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.872+0000 7fb67cff9640 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-15T03:37:15.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.972+0000 7fb6870d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T03:37:15.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.972+0000 7fb6870d4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T03:37:15.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.972+0000 7fb6870d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:15.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.972+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7fb68009c090 con 0x7fb670004ec0 2024-09-15T03:37:15.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.973+0000 7fb67cff9640 1 -- 172.21.15.181:0/1678598327 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v221) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb68009c090 con 0x7fb670004ec0 2024-09-15T03:37:15.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.973+0000 7fb67cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-15T03:37:15.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.973+0000 7fb67cff9640 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-15T03:37:15.974 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h5' does not appear in the crush map 2024-09-15T03:37:15.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb664038770 msgr2=0x7fb66403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:15.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb664038770 0x7fb66403ac20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb68009bbc0 tx=0x7fb66c009040 comp rx=0 tx=0).stop 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 10 monclient: shutdown 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 msgr2=0x7fb68009ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb68009ab40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb674007110 tx=0x7fb674007a70 comp rx=0 tx=0).stop 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 shutdown_connections 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb664038770 0x7fb66403ac20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb68011b760 0x7fb68011bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680130cb0 0x7fb680131120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 --2- 172.21.15.181:0/1678598327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb670004ec0 0x7fb68009ab40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 >> 172.21.15.181:0/1678598327 conn(0x7fb6800a5cf0 msgr2=0x7fb680068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:15.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 shutdown_connections 2024-09-15T03:37:15.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:15.975+0000 7fb6870d4640 1 -- 172.21.15.181:0/1678598327 wait complete. 2024-09-15T03:37:15.985 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:15.985 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h6 2024-09-15T03:37:16.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.057+0000 7f7e74f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.057+0000 7f7e74f9d640 10 monclient: build_initial_monmap 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.057+0000 7f7e74f9d640 10 monclient: monmap: 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:16.058704+0000 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:16.058704+0000 2024-09-15T03:37:16.059 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:16.060 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:16.060 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:16.060 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:16.060 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:16.060 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:16.061 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:16.061 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.058+0000 7f7e74f9d640 1 Processor -- start 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.058+0000 7f7e74f9d640 1 -- start start 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.058+0000 7f7e74f9d640 10 monclient: init 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e70131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e70131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7e700a0b00 0x7f7e70132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e700a0b00 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e70068d00 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:16.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7009d230 con 0x7f7e70068d00 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700a08c0 con 0x7f7e700a0b00 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700a0200 con 0x7f7e70131c00 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e74f9d640 10 monclient(hunting): authenticate will time out at 1240.540193s 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6e59c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 msgr2=0x7f7e70131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6ed9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f7e70068d00 0x7f7e70069100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:35952/0 (socket says 172.21.15.181:35952) 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6e59c640 1 --2- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e70131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6ed9d640 1 -- 172.21.15.181:0/3436167625 learned_addr learned my addr 172.21.15.181:0/3436167625 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 1 --2- 172.21.15.181:0/3436167625 >> v2:172.21.15.153:3300/0 conn(0x7f7e700a0b00 0x7f7e70132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:16.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): get_auth_request con 0x7f7e700a0b00 auth_method 0 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient(hunting): handle_auth_done global_id 114297 payload 274 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 1 -- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 msgr2=0x7f7e70131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 1 --2- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e70131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 1 -- 172.21.15.181:0/3436167625 >> v1:172.21.15.181:6789/0 conn(0x7f7e70068d00 legacy=0x7f7e70069100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:16.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 1 monclient: found mon.noname-a 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.059+0000 7f7e6dd9b640 1 -- 172.21.15.181:0/3436167625 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700a1b70 con 0x7f7e700a0b00 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.062+0000 7f7e6dd9b640 1 -- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 msgr2=0x7f7e70132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.062+0000 7f7e6dd9b640 1 --2- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70132510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009850 tx=0x7f7e640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b950e290f5547c31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 1 -- 172.21.15.181:0/3436167625 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64032440 con 0x7f7e700a0b00 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: dump: 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:16.072 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: _finish_auth 0 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 1 -- 172.21.15.181:0/3436167625 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e640325a0 con 0x7f7e700a0b00 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 5 monclient: authenticate success, global_id 114297 2024-09-15T03:37:16.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 4 set_mon_vals no callback set 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 1 -- 172.21.15.181:0/3436167625 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64032940 con 0x7f7e700a0b00 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e6d59a640 10 monclient: dump: 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 10 monclient: shutdown 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 -- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 msgr2=0x7f7e70132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70132510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009850 tx=0x7f7e640090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 -- 172.21.15.181:0/3436167625 shutdown_connections 2024-09-15T03:37:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70132510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/3436167625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e70131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 -- 172.21.15.181:0/3436167625 >> 172.21.15.181:0/3436167625 conn(0x7f7e70071370 msgr2=0x7f7e700a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 -- 172.21.15.181:0/3436167625 shutdown_connections 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.063+0000 7f7e74f9d640 1 -- 172.21.15.181:0/3436167625 wait complete. 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.064+0000 7f7e74f9d640 10 monclient: build_initial_monmap 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.064+0000 7f7e74f9d640 10 monclient: monmap: 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.064+0000 7f7e74f9d640 1 Processor -- start 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.064+0000 7f7e74f9d640 1 -- start start 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.064+0000 7f7e74f9d640 10 monclient: init 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e70068d00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e700a0b00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e701411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e70131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7011cb00 con 0x7f7e700a0b00 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70134c10 con 0x7f7e70068d00 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7011ceb0 con 0x7f7e70131c00 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e74f9d640 10 monclient(hunting): authenticate will time out at 1240.546096s 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6ed9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 msgr2=0x7f7e701411d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e701411d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:16.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51552/0 (socket says 172.21.15.181:51552) 2024-09-15T03:37:16.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 -- 172.21.15.181:0/2346359036 learned_addr learned my addr 172.21.15.181:0/2346359036 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:16.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): get_auth_request con 0x7f7e70068d00 auth_method 0 2024-09-15T03:37:16.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6dd9b640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6dd9b640 10 monclient(hunting): get_auth_request con 0x7f7e700a0b00 auth_method 0 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient(hunting): handle_auth_done global_id 104494 payload 824 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 -- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 msgr2=0x7f7e701411d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e701411d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 -- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 msgr2=0x7f7e70130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:16.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:37:16.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 monclient: found mon.noname-b 2024-09-15T03:37:16.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:16.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:16.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.065+0000 7f7e6e59c640 1 -- 172.21.15.181:0/2346359036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64009390 con 0x7f7e70068d00 2024-09-15T03:37:16.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e6e59c640 1 -- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 msgr2=0x7f7e7009aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:16.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e6e59c640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000b600 tx=0x7f7e600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:16.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e6e59c640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000b600 tx=0x7f7e600091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f7e60015070 mon_map magic: 0 v1 2024-09-15T03:37:16.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60015070 con 0x7f7e70068d00 2024-09-15T03:37:16.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:16.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:16.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 10 monclient: dump: 2024-09-15T03:37:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:16.085 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60003d90 con 0x7f7e70068d00 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 5 monclient: authenticate success, global_id 104494 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 10 monclient: _renew_subs 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7011de70 con 0x7f7e70068d00 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e6cd99640 4 set_mon_vals no callback set 2024-09-15T03:37:16.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 10 monclient: _renew_subs 2024-09-15T03:37:16.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:16.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.067+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7011bdc0 con 0x7f7e70068d00 2024-09-15T03:37:16.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e74f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:16.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e74f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e74f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7011c0c0 con 0x7f7e70068d00 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60006380 con 0x7f7e70068d00 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e5e7fc640 10 monclient: dump: 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:16.090 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.068+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7e60010440 con 0x7f7e70068d00 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.069+0000 7f7e5e7fc640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e50038740 0x7f7e5003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.069+0000 7f7e6dd9b640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e50038740 0x7f7e5003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.069+0000 7f7e6dd9b640 10 monclient: get_auth_request con 0x7f7e50038740 auth_method 0 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.071+0000 7f7e6dd9b640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e50038740 0x7f7e5003abf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7e7009c7c0 tx=0x7f7e64032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:16.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.073+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(221..221 src has 194..221) v4 ==== 13858+0+0 (secure 0 0 0) 0x7f7e6004d9b0 con 0x7f7e70068d00 2024-09-15T03:37:16.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.690+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7e60006770 con 0x7f7e70068d00 2024-09-15T03:37:16.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.690+0000 7f7e5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:16.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.690+0000 7f7e5e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:16.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.847+0000 7f7e74f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T03:37:16.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.847+0000 7f7e74f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T03:37:16.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.847+0000 7f7e74f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:16.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.847+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f7e7011e7b0 con 0x7f7e70068d00 2024-09-15T03:37:16.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.880+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v222) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7e60018780 con 0x7f7e70068d00 2024-09-15T03:37:16.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.880+0000 7f7e5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T03:37:16.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.880+0000 7f7e5e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-15T03:37:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.978+0000 7f7e74f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T03:37:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.978+0000 7f7e74f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T03:37:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.978+0000 7f7e74f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:16.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.978+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f7e700008d0 con 0x7f7e70068d00 2024-09-15T03:37:16.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.979+0000 7f7e5e7fc640 1 -- 172.21.15.181:0/2346359036 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v222) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7e60018d60 con 0x7f7e70068d00 2024-09-15T03:37:16.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.979+0000 7f7e5e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-15T03:37:16.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.979+0000 7f7e5e7fc640 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-15T03:37:16.981 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h6' does not appear in the crush map 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e50038740 msgr2=0x7f7e5003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e50038740 0x7f7e5003abf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7e7009c7c0 tx=0x7f7e64032000 comp rx=0 tx=0).stop 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 10 monclient: shutdown 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 msgr2=0x7f7e7009aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000b600 tx=0x7f7e600091a0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 shutdown_connections 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7e50038740 0x7f7e5003abf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e70131c00 0x7f7e701411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e700a0b00 0x7f7e70130cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 --2- 172.21.15.181:0/2346359036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e70068d00 0x7f7e7009aa80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:16.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.981+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 >> 172.21.15.181:0/2346359036 conn(0x7f7e70071370 msgr2=0x7f7e70134e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:16.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.982+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 shutdown_connections 2024-09-15T03:37:16.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:16.982+0000 7f7e74f9d640 1 -- 172.21.15.181:0/2346359036 wait complete. 2024-09-15T03:37:16.991 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:16.991 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h7 2024-09-15T03:37:17.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.063+0000 7f7db5e46640 10 monclient: get_monmap_and_config 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.063+0000 7f7db5e46640 10 monclient: build_initial_monmap 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.064+0000 7f7db5e46640 10 monclient: monmap: 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:17.064797+0000 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:17.064797+0000 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:17.066 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:17.066 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:17.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.064+0000 7f7db5e46640 1 Processor -- start 2024-09-15T03:37:17.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.064+0000 7f7db5e46640 1 -- start start 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.064+0000 7f7db5e46640 10 monclient: init 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7db009ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7db009f020 0x7f7db0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): picked mon.noname-a con 0x7f7db009f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7db0136d60 0x7f7db0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): picked mon.noname-c con 0x7f7db0136d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db0067420 con 0x7f7db009ec50 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db00a0480 con 0x7f7db009f020 2024-09-15T03:37:17.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db00a0200 con 0x7f7db0136d60 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7db5e46640 10 monclient(hunting): authenticate will time out at 1241.546243s 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7db0136d60 0x7f7db0139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f7db0136d60 0x7f7db0139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51562/0 (socket says 172.21.15.181:51562) 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 -- 172.21.15.181:0/3722841458 learned_addr learned my addr 172.21.15.181:0/3722841458 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7db0136d60 auth_method 0 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7da6ffd640 1 --2- 172.21.15.181:0/3722841458 >> v2:172.21.15.153:3300/0 conn(0x7f7db009f020 0x7f7db0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7da6ffd640 10 monclient(hunting): get_auth_request con 0x7f7db009f020 auth_method 0 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7da6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient(hunting): handle_auth_done global_id 104497 payload 274 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 -- 172.21.15.181:0/3722841458 >> v2:172.21.15.153:3300/0 conn(0x7f7db009f020 msgr2=0x7f7db0069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 --2- 172.21.15.181:0/3722841458 >> v2:172.21.15.153:3300/0 conn(0x7f7db009f020 0x7f7db0069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 -- 172.21.15.181:0/3722841458 >> v1:172.21.15.153:6789/0 conn(0x7f7db009ec50 legacy=0x7f7db0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 monclient: found mon.noname-c 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.065+0000 7f7daffff640 1 -- 172.21.15.181:0/3722841458 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db009fd80 con 0x7f7db0136d60 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daffff640 1 -- 172.21.15.181:0/3722841458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db0136d60 msgr2=0x7f7db0139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daffff640 1 --2- 172.21.15.181:0/3722841458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db0136d60 0x7f7db0139140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7db009bf50 tx=0x7f7da8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bf64e8fff68c0bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 1 -- 172.21.15.181:0/3722841458 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8007610 con 0x7f7db0136d60 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: dump: 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7db5e46640 5 monclient: authenticate success, global_id 104497 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7db5e46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 1 -- 172.21.15.181:0/3722841458 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da802d050 con 0x7f7db0136d60 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7db5e46640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:17.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 1 -- 172.21.15.181:0/3722841458 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da80079a0 con 0x7f7db0136d60 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7db5e46640 4 set_mon_vals no callback set 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.068+0000 7f7daeffd640 10 monclient: dump: 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:17.078 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 10 monclient: shutdown 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 -- 172.21.15.181:0/3722841458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db0136d60 msgr2=0x7f7db0139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 --2- 172.21.15.181:0/3722841458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db0136d60 0x7f7db0139140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7db009bf50 tx=0x7f7da8009840 comp rx=0 tx=0).stop 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 -- 172.21.15.181:0/3722841458 shutdown_connections 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 --2- 172.21.15.181:0/3722841458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db0136d60 0x7f7db0139140 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 --2- 172.21.15.181:0/3722841458 >> v2:172.21.15.153:3300/0 conn(0x7f7db009f020 0x7f7db0069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 -- 172.21.15.181:0/3722841458 >> 172.21.15.181:0/3722841458 conn(0x7f7db00a5cf0 msgr2=0x7f7db00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 -- 172.21.15.181:0/3722841458 shutdown_connections 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 -- 172.21.15.181:0/3722841458 wait complete. 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 10 monclient: build_initial_monmap 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 10 monclient: monmap: 2024-09-15T03:37:17.079 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.069+0000 7f7db5e46640 1 Processor -- start 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 -- start start 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient: init 2024-09-15T03:37:17.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 0x7f7db01308a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): picked mon.noname-b con 0x7f7db009ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db009f020 0x7f7db0099790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): picked mon.noname-a con 0x7f7db009f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7db0136d60 0x7f7db0099cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): picked mon.noname-c con 0x7f7db0136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db011c730 con 0x7f7db009f020 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db009c020 con 0x7f7db009ec50 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00a3b90 con 0x7f7db0136d60 2024-09-15T03:37:17.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7db5e46640 10 monclient(hunting): authenticate will time out at 1241.551613s 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 0x7f7db01308a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 0x7f7db01308a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51566/0 (socket says 172.21.15.181:51566) 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 -- 172.21.15.181:0/1807980690 learned_addr learned my addr 172.21.15.181:0/1807980690 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): get_auth_request con 0x7f7db009ec50 auth_method 0 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daffff640 1 -- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7db0136d60 msgr2=0x7f7db0099cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daffff640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7db0136d60 0x7f7db0099cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:17.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7da6ffd640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db009f020 0x7f7db0099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7da6ffd640 10 monclient(hunting): get_auth_request con 0x7f7db009f020 auth_method 0 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7da6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient(hunting): handle_auth_done global_id 104500 payload 824 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 -- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7db0136d60 msgr2=0x7f7db0099cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7db0136d60 0x7f7db0099cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 -- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db009f020 msgr2=0x7f7db0099790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db009f020 0x7f7db0099790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:17.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 monclient: found mon.noname-b 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.070+0000 7f7daf7fe640 1 -- 172.21.15.181:0/1807980690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da8009ae0 con 0x7f7db009ec50 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7daf7fe640 1 -- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 msgr2=0x7f7db01308a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7daf7fe640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 0x7f7db01308a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7da000c610 tx=0x7f7da0007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0012840 con 0x7f7db009ec50 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: dump: 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da00129e0 con 0x7f7db009ec50 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 5 monclient: authenticate success, global_id 104500 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0012da0 con 0x7f7db009ec50 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7da7fff640 10 monclient: dump: 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7dae7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 10 monclient: _renew_subs 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db011bac0 con 0x7f7db009ec50 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 10 monclient: _renew_subs 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.073+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db009b370 con 0x7f7db009ec50 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7da0011590 con 0x7f7db009ec50 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7db5e46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7db5e46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7db5e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db0137660 con 0x7f7db009ec50 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7da7fff640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7d900386f0 0x7f7d9003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(222..222 src has 194..222) v4 ==== 13648+0+0 (secure 0 0 0) 0x7f7da004df50 con 0x7f7db009ec50 2024-09-15T03:37:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7da6ffd640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7d900386f0 0x7f7d9003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:17.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.074+0000 7f7da6ffd640 10 monclient: get_auth_request con 0x7f7d900386f0 auth_method 0 2024-09-15T03:37:17.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.076+0000 7f7da6ffd640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7d900386f0 0x7f7d9003aba0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7db009b7a0 tx=0x7f7d98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:17.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.078+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7da00108c0 con 0x7f7db009ec50 2024-09-15T03:37:17.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.078+0000 7f7da7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:17.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.078+0000 7f7da7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.236+0000 7f7db5e46640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T03:37:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.236+0000 7f7db5e46640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T03:37:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.236+0000 7f7db5e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.236+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f7db00008d0 con 0x7f7db009ec50 2024-09-15T03:37:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.881+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v223) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7da00102e0 con 0x7f7db009ec50 2024-09-15T03:37:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.881+0000 7f7da7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T03:37:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.881+0000 7f7da7fff640 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-15T03:37:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.978+0000 7f7db5e46640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T03:37:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.978+0000 7f7db5e46640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T03:37:17.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.978+0000 7f7db5e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:17.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.978+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f7db00a13e0 con 0x7f7db009ec50 2024-09-15T03:37:17.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.980+0000 7f7da7fff640 1 -- 172.21.15.181:0/1807980690 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v223) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7da004a6c0 con 0x7f7db009ec50 2024-09-15T03:37:17.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.980+0000 7f7da7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-15T03:37:17.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.980+0000 7f7da7fff640 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-15T03:37:17.981 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h7' does not appear in the crush map 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7d900386f0 msgr2=0x7f7d9003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7d900386f0 0x7f7d9003aba0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7db009b7a0 tx=0x7f7d98008040 comp rx=0 tx=0).stop 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 10 monclient: shutdown 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 msgr2=0x7f7db01308a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 0x7f7db01308a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7da000c610 tx=0x7f7da0007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 shutdown_connections 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7d900386f0 0x7f7d9003aba0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7db0136d60 0x7f7db0099cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7db009f020 0x7f7db0099790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 --2- 172.21.15.181:0/1807980690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7db009ec50 0x7f7db01308a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 >> 172.21.15.181:0/1807980690 conn(0x7f7db00a5cf0 msgr2=0x7f7db009d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 shutdown_connections 2024-09-15T03:37:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:17.982+0000 7f7db5e46640 1 -- 172.21.15.181:0/1807980690 wait complete. 2024-09-15T03:37:17.991 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:17.991 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h8 2024-09-15T03:37:18.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.064+0000 7fe2f5709640 10 monclient: get_monmap_and_config 2024-09-15T03:37:18.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.064+0000 7fe2f5709640 10 monclient: build_initial_monmap 2024-09-15T03:37:18.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.064+0000 7fe2f5709640 10 monclient: monmap: 2024-09-15T03:37:18.065 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:18.065 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:18.065203+0000 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:18.065203+0000 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:18.066 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:18.068 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 Processor -- start 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 -- start start 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient: init 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2f009ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe2f009f020 0x7fe2f0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f009f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe2f0136d60 0x7fe2f0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:18.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f0136d60 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f0067420 con 0x7fe2f009ec50 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f00a0480 con 0x7fe2f0136d60 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f00a0200 con 0x7fe2f009f020 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2f5709640 10 monclient(hunting): authenticate will time out at 1242.546639s 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe2f009f020 0x7fe2f0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe2f009f020 0x7fe2f0069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51576/0 (socket says 172.21.15.181:51576) 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 1 -- 172.21.15.181:0/2560104250 learned_addr learned my addr 172.21.15.181:0/2560104250 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): get_auth_request con 0x7fe2f009f020 auth_method 0 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2ef7fe640 1 --2- 172.21.15.181:0/2560104250 >> v2:172.21.15.153:3300/0 conn(0x7fe2f0136d60 0x7fe2f0139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2ef7fe640 10 monclient(hunting): get_auth_request con 0x7fe2f0136d60 auth_method 0 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.065+0000 7fe2ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:18.071 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 10 monclient(hunting): handle_auth_done global_id 104503 payload 274 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 1 -- 172.21.15.181:0/2560104250 >> v2:172.21.15.153:3300/0 conn(0x7fe2f0136d60 msgr2=0x7fe2f0139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 1 --2- 172.21.15.181:0/2560104250 >> v2:172.21.15.153:3300/0 conn(0x7fe2f0136d60 0x7fe2f0139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 1 -- 172.21.15.181:0/2560104250 >> v1:172.21.15.153:6789/0 conn(0x7fe2f009ec50 legacy=0x7fe2f0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 1 monclient: found mon.noname-c 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.072 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.066+0000 7fe2e7fff640 1 -- 172.21.15.181:0/2560104250 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f009fd80 con 0x7fe2f009f020 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.068+0000 7fe2e7fff640 1 -- 172.21.15.181:0/2560104250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009f020 msgr2=0x7fe2f0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.068+0000 7fe2e7fff640 1 --2- 172.21.15.181:0/2560104250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009f020 0x7fe2f0069d10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe2f006a250 tx=0x7fe2d80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee428d166d731289 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 1 -- 172.21.15.181:0/2560104250 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8032440 con 0x7fe2f009f020 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: dump: 2024-09-15T03:37:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 1 -- 172.21.15.181:0/2560104250 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d80325a0 con 0x7fe2f009f020 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 5 monclient: authenticate success, global_id 104503 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 1 -- 172.21.15.181:0/2560104250 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8032940 con 0x7fe2f009f020 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 4 set_mon_vals no callback set 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2ee7fc640 10 monclient: dump: 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 10 monclient: shutdown 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 -- 172.21.15.181:0/2560104250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009f020 msgr2=0x7fe2f0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 --2- 172.21.15.181:0/2560104250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009f020 0x7fe2f0069d10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe2f006a250 tx=0x7fe2d80091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 -- 172.21.15.181:0/2560104250 shutdown_connections 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 --2- 172.21.15.181:0/2560104250 >> v2:172.21.15.153:3300/0 conn(0x7fe2f0136d60 0x7fe2f0139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 --2- 172.21.15.181:0/2560104250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009f020 0x7fe2f0069d10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 -- 172.21.15.181:0/2560104250 >> 172.21.15.181:0/2560104250 conn(0x7fe2f00a5cf0 msgr2=0x7fe2f00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 -- 172.21.15.181:0/2560104250 shutdown_connections 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.069+0000 7fe2f5709640 1 -- 172.21.15.181:0/2560104250 wait complete. 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient: build_initial_monmap 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient: monmap: 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:18.083 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 Processor -- start 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 -- start start 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient: init 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 0x7fe2f0130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f009ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f009f020 0x7fe2f0099880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f009f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2f0136d60 0x7fe2f0099dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:18.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f0136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f009c020 con 0x7fe2f009f020 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00a4590 con 0x7fe2f009ec50 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f011ca90 con 0x7fe2f0136d60 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.070+0000 7fe2f5709640 10 monclient(hunting): authenticate will time out at 1242.551953s 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 0x7fe2f0130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2ef7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2f0136d60 msgr2=0x7fe2f0099dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2f0136d60 0x7fe2f0099dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 0x7fe2f0130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51584/0 (socket says 172.21.15.181:51584) 2024-09-15T03:37:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 -- 172.21.15.181:0/874687127 learned_addr learned my addr 172.21.15.181:0/874687127 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): get_auth_request con 0x7fe2f009ec50 auth_method 0 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2e7fff640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f009f020 0x7fe2f0099880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2e7fff640 10 monclient(hunting): get_auth_request con 0x7fe2f009f020 auth_method 0 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient(hunting): handle_auth_done global_id 104506 payload 824 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 -- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2f0136d60 msgr2=0x7fe2f0099dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2f0136d60 0x7fe2f0099dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 -- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f009f020 msgr2=0x7fe2f0099880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f009f020 0x7fe2f0099880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 monclient: found mon.noname-b 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.071+0000 7fe2eeffd640 1 -- 172.21.15.181:0/874687127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d8009450 con 0x7fe2f009ec50 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2eeffd640 1 -- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 msgr2=0x7fe2f0130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2eeffd640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 0x7fe2f0130980 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000c610 tx=0x7fe2e0007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0012840 con 0x7fe2f009ec50 2024-09-15T03:37:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: dump: 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:18.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e00129e0 con 0x7fe2f009ec50 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 5 monclient: authenticate success, global_id 104506 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0012da0 con 0x7fe2f009ec50 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 10 monclient: _renew_subs 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f011c7f0 con 0x7fe2f009ec50 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2e6ffd640 10 monclient: dump: 2024-09-15T03:37:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2edffb640 4 set_mon_vals no callback set 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 10 monclient: _renew_subs 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.073+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f009b410 con 0x7fe2f009ec50 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.074+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe2e0011590 con 0x7fe2f009ec50 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.074+0000 7fe2f5709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.074+0000 7fe2f5709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.074+0000 7fe2f5709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.074+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f00008d0 con 0x7fe2f009ec50 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.074+0000 7fe2e6ffd640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2d00386f0 0x7fe2d003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.075+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(223..223 src has 194..223) v4 ==== 13438+0+0 (secure 0 0 0) 0x7fe2e004de40 con 0x7fe2f009ec50 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.075+0000 7fe2e7fff640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2d00386f0 0x7fe2d003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.075+0000 7fe2e7fff640 10 monclient: get_auth_request con 0x7fe2d00386f0 auth_method 0 2024-09-15T03:37:18.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.077+0000 7fe2e7fff640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2d00386f0 0x7fe2d003aba0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00719d0 tx=0x7fe2d8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:18.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.079+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe2e00108e0 con 0x7fe2f009ec50 2024-09-15T03:37:18.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.079+0000 7fe2e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:18.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.079+0000 7fe2e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:18.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.239+0000 7fe2f5709640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T03:37:18.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.239+0000 7fe2f5709640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T03:37:18.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.239+0000 7fe2f5709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.239+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7fe2f00a13e0 con 0x7fe2f009ec50 2024-09-15T03:37:18.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.892+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v224) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe2e00102e0 con 0x7fe2f009ec50 2024-09-15T03:37:18.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.892+0000 7fe2e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T03:37:18.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.892+0000 7fe2e6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-15T03:37:18.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.993+0000 7fe2f5709640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T03:37:18.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.993+0000 7fe2f5709640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T03:37:18.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.993+0000 7fe2f5709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:18.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.993+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7fe2f011bc00 con 0x7fe2f009ec50 2024-09-15T03:37:18.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.995+0000 7fe2e6ffd640 1 -- 172.21.15.181:0/874687127 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v224) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe2e00104c0 con 0x7fe2f009ec50 2024-09-15T03:37:18.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.995+0000 7fe2e6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-15T03:37:18.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.995+0000 7fe2e6ffd640 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-15T03:37:18.996 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h8' does not appear in the crush map 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.996+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2d00386f0 msgr2=0x7fe2d003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.996+0000 7fe2f5709640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2d00386f0 0x7fe2d003aba0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00719d0 tx=0x7fe2d8032000 comp rx=0 tx=0).stop 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.996+0000 7fe2f5709640 10 monclient: shutdown 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.996+0000 7fe2f5709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.996+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 msgr2=0x7fe2f0130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.996+0000 7fe2f5709640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 0x7fe2f0130980 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000c610 tx=0x7fe2e0007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 shutdown_connections 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe2d00386f0 0x7fe2d003aba0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2f0136d60 0x7fe2f0099dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2f009f020 0x7fe2f0099880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 --2- 172.21.15.181:0/874687127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f009ec50 0x7fe2f0130980 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:18.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 >> 172.21.15.181:0/874687127 conn(0x7fe2f00a5cf0 msgr2=0x7fe2f00a47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:18.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 shutdown_connections 2024-09-15T03:37:18.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:18.997+0000 7fe2f5709640 1 -- 172.21.15.181:0/874687127 wait complete. 2024-09-15T03:37:19.006 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:19.006 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h9 2024-09-15T03:37:19.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.078+0000 7f630319d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.078+0000 7f630319d640 10 monclient: build_initial_monmap 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.079+0000 7f630319d640 10 monclient: monmap: 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:19.079792+0000 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:19.079792+0000 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:19.080 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:19.081 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:19.081 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.079+0000 7f630319d640 1 Processor -- start 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.079+0000 7f630319d640 1 -- start start 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.079+0000 7f630319d640 10 monclient: init 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f62fc06e000 0x7f62fc06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): picked mon.noname-c con 0x7f62fc06e000 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62fc06a710 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f62fc09e170 0x7f62fc09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): picked mon.noname-a con 0x7f62fc09e170 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc0699c0 con 0x7f62fc06a710 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc06a360 con 0x7f62fc09e170 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc06be20 con 0x7f62fc06e000 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630319d640 10 monclient(hunting): authenticate will time out at 1243.561233s 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f62fc06e000 0x7f62fc06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f62fc06e000 0x7f62fc06e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45358/0 (socket says 172.21.15.181:45358) 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 -- 172.21.15.181:0/1906498617 learned_addr learned my addr 172.21.15.181:0/1906498617 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:19.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): get_auth_request con 0x7f62fc06e000 auth_method 0 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630173a640 1 --2- 172.21.15.181:0/1906498617 >> v2:172.21.15.153:3300/0 conn(0x7f62fc09e170 0x7f62fc09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630173a640 10 monclient(hunting): get_auth_request con 0x7f62fc09e170 auth_method 0 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630173a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630173a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f630173a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient(hunting): handle_auth_done global_id 104509 payload 274 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 -- 172.21.15.181:0/1906498617 >> v2:172.21.15.153:3300/0 conn(0x7f62fc09e170 msgr2=0x7f62fc09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:19.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 --2- 172.21.15.181:0/1906498617 >> v2:172.21.15.153:3300/0 conn(0x7f62fc09e170 0x7f62fc09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 -- 172.21.15.181:0/1906498617 >> v1:172.21.15.153:6789/0 conn(0x7f62fc06a710 legacy=0x7f62fc09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:19.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient: _finish_hunting 0 2024-09-15T03:37:19.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 monclient: found mon.noname-c 2024-09-15T03:37:19.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:19.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.080+0000 7f6300f39640 1 -- 172.21.15.181:0/1906498617 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc069f80 con 0x7f62fc06e000 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f6300f39640 1 -- 172.21.15.181:0/1906498617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc06e000 msgr2=0x7f62fc06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f6300f39640 1 --2- 172.21.15.181:0/1906498617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc06e000 0x7f62fc06e3d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62fc063870 tx=0x7f62f80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f8dcf4c902cce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 1 -- 172.21.15.181:0/1906498617 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8032440 con 0x7f62fc06e000 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: dump: 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:19.090 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 1 -- 172.21.15.181:0/1906498617 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f80325a0 con 0x7f62fc06e000 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 1 -- 172.21.15.181:0/1906498617 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8032940 con 0x7f62fc06e000 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:19.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f62f37fe640 10 monclient: dump: 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f630319d640 5 monclient: authenticate success, global_id 104509 2024-09-15T03:37:19.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f630319d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.083+0000 7f630319d640 4 set_mon_vals no callback set 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 10 monclient: shutdown 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 -- 172.21.15.181:0/1906498617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc06e000 msgr2=0x7f62fc06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 --2- 172.21.15.181:0/1906498617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc06e000 0x7f62fc06e3d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62fc063870 tx=0x7f62f80091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 -- 172.21.15.181:0/1906498617 shutdown_connections 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 --2- 172.21.15.181:0/1906498617 >> v2:172.21.15.153:3300/0 conn(0x7f62fc09e170 0x7f62fc09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 --2- 172.21.15.181:0/1906498617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc06e000 0x7f62fc06e3d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 -- 172.21.15.181:0/1906498617 >> 172.21.15.181:0/1906498617 conn(0x7f62fc071340 msgr2=0x7f62fc0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 -- 172.21.15.181:0/1906498617 shutdown_connections 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 -- 172.21.15.181:0/1906498617 wait complete. 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 10 monclient: build_initial_monmap 2024-09-15T03:37:19.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 10 monclient: monmap: 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.084+0000 7f630319d640 1 Processor -- start 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 -- start start 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient: init 2024-09-15T03:37:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62fc06a710 0x7f62fc141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): picked mon.noname-a con 0x7f62fc06a710 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62fc06e000 0x7f62fc134ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): picked mon.noname-c con 0x7f62fc06e000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 0x7f62fc11e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): picked mon.noname-b con 0x7f62fc09e170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc120ab0 con 0x7f62fc06a710 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc09eb10 con 0x7f62fc09e170 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc06e970 con 0x7f62fc06e000 2024-09-15T03:37:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630319d640 10 monclient(hunting): authenticate will time out at 1243.566567s 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 0x7f62fc11e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f62f3fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62fc06e000 msgr2=0x7f62fc134ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 0x7f62fc11e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45360/0 (socket says 172.21.15.181:45360) 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 -- 172.21.15.181:0/968540280 learned_addr learned my addr 172.21.15.181:0/968540280 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): get_auth_request con 0x7f62fc09e170 auth_method 0 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f62f3fff640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62fc06e000 0x7f62fc134ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:19.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f6300f39640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62fc06a710 0x7f62fc141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f6300f39640 10 monclient(hunting): get_auth_request con 0x7f62fc06a710 auth_method 0 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f6300f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f6300f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f6300f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient(hunting): handle_auth_done global_id 104512 payload 824 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 -- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62fc06e000 msgr2=0x7f62fc134ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62fc06e000 0x7f62fc134ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 -- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62fc06a710 msgr2=0x7f62fc141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62fc06a710 0x7f62fc141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 monclient: found mon.noname-b 2024-09-15T03:37:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:19.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f630173a640 1 -- 172.21.15.181:0/968540280 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8009450 con 0x7f62fc09e170 2024-09-15T03:37:19.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.085+0000 7f6300f39640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62fc06a710 0x7f62fc141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:19.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.088+0000 7f630173a640 1 -- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 msgr2=0x7f62fc11e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:19.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.088+0000 7f630173a640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 0x7f62fc11e9e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009fa0 tx=0x7f62f4009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:19.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.088+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f400ee60 con 0x7f62fc09e170 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: dump: 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.100 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4013590 con 0x7f62fc09e170 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 5 monclient: authenticate success, global_id 104512 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4013950 con 0x7f62fc09e170 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 10 monclient: _renew_subs 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc121ff0 con 0x7f62fc09e170 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f0ff9640 10 monclient: dump: 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:19.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f62f2ffd640 4 set_mon_vals no callback set 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 10 monclient: _renew_subs 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.090+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc121100 con 0x7f62fc09e170 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f62f4013cc0 con 0x7f62fc09e170 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f630319d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f630319d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f630319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc0008d0 con 0x7f62fc09e170 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f62f0ff9640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62e8038740 0x7f62e803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(224..224 src has 194..224) v4 ==== 13228+0+0 (secure 0 0 0) 0x7f62f404e3f0 con 0x7f62fc09e170 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f6300f39640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62e8038740 0x7f62e803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.091+0000 7f6300f39640 10 monclient: get_auth_request con 0x7f62e8038740 auth_method 0 2024-09-15T03:37:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.093+0000 7f6300f39640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62e8038740 0x7f62e803abf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f62fc067100 tx=0x7f62f8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:19.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.095+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f62f4009990 con 0x7f62fc09e170 2024-09-15T03:37:19.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.095+0000 7f62f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:19.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.095+0000 7f62f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:19.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.253+0000 7f630319d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T03:37:19.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.253+0000 7f630319d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T03:37:19.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.253+0000 7f630319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.253+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f62fc064a20 con 0x7f62fc09e170 2024-09-15T03:37:19.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.893+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v225) v1 ==== 118+0+0 (secure 0 0 0) 0x7f62f401b310 con 0x7f62fc09e170 2024-09-15T03:37:19.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.893+0000 7f62f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T03:37:19.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.893+0000 7f62f0ff9640 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-15T03:37:19.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.991+0000 7f630319d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T03:37:19.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.991+0000 7f630319d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T03:37:19.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.991+0000 7f630319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:19.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.991+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f62fc141e30 con 0x7f62fc09e170 2024-09-15T03:37:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.992+0000 7f62f0ff9640 1 -- 172.21.15.181:0/968540280 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v225) v1 ==== 118+0+0 (secure 0 0 0) 0x7f62f40122e0 con 0x7f62fc09e170 2024-09-15T03:37:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.992+0000 7f62f0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-15T03:37:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.992+0000 7f62f0ff9640 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-15T03:37:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h9' does not appear in the crush map 2024-09-15T03:37:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62e8038740 msgr2=0x7f62e803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62e8038740 0x7f62e803abf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f62fc067100 tx=0x7f62f8032000 comp rx=0 tx=0).stop 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 10 monclient: shutdown 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 msgr2=0x7f62fc11e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 0x7f62fc11e9e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009fa0 tx=0x7f62f4009900 comp rx=0 tx=0).stop 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 shutdown_connections 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f62e8038740 0x7f62e803abf0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62fc09e170 0x7f62fc11e9e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62fc06e000 0x7f62fc134ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 --2- 172.21.15.181:0/968540280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62fc06a710 0x7f62fc141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 >> 172.21.15.181:0/968540280 conn(0x7f62fc071340 msgr2=0x7f62fc0a2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 shutdown_connections 2024-09-15T03:37:19.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:19.994+0000 7f630319d640 1 -- 172.21.15.181:0/968540280 wait complete. 2024-09-15T03:37:20.004 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:20.004 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h10 2024-09-15T03:37:20.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.077+0000 7f6047ed4640 10 monclient: get_monmap_and_config 2024-09-15T03:37:20.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.077+0000 7f6047ed4640 10 monclient: build_initial_monmap 2024-09-15T03:37:20.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.077+0000 7f6047ed4640 10 monclient: monmap: 2024-09-15T03:37:20.078 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:20.078 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:20.077842+0000 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:20.077842+0000 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:20.079 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:20.080 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:20.080 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:20.080 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.077+0000 7f6047ed4640 1 Processor -- start 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.077+0000 7f6047ed4640 1 -- start start 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.077+0000 7f6047ed4640 10 monclient: init 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f604009ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f604009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6040069d10 0x7f604013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f6040069d10 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6040067420 con 0x7f604009ec50 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60400a0480 con 0x7f604009f020 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604006a170 con 0x7f6040069d10 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6047ed4640 10 monclient(hunting): authenticate will time out at 1244.559282s 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6040069d10 0x7f604013a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f604546f640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f604009f020 0x7f6040136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55332/0 (socket says 172.21.15.181:55332) 2024-09-15T03:37:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6040069d10 0x7f604013a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45374/0 (socket says 172.21.15.181:45374) 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f604546f640 1 -- 172.21.15.181:0/3877258066 learned_addr learned my addr 172.21.15.181:0/3877258066 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): get_auth_request con 0x7f6040069d10 auth_method 0 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient(hunting): handle_auth_done global_id 104515 payload 274 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 1 -- 172.21.15.181:0/3877258066 >> v1:172.21.15.181:6789/0 conn(0x7f604009f020 legacy=0x7f6040136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 1 -- 172.21.15.181:0/3877258066 >> v1:172.21.15.153:6789/0 conn(0x7f604009ec50 legacy=0x7f6040131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient: _finish_hunting 0 2024-09-15T03:37:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 1 monclient: found mon.noname-c 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.078+0000 7f6046471640 1 -- 172.21.15.181:0/3877258066 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400a0200 con 0x7f6040069d10 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6046471640 1 -- 172.21.15.181:0/3877258066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6040069d10 msgr2=0x7f604013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6046471640 1 --2- 172.21.15.181:0/3877258066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6040069d10 0x7f604013a490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6030009e60 tx=0x7f6030009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e443a61c999b916 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 1 -- 172.21.15.181:0/3877258066 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030007840 con 0x7f6040069d10 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: dump: 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: _finish_auth 0 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6047ed4640 5 monclient: authenticate success, global_id 104515 2024-09-15T03:37:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 1 -- 172.21.15.181:0/3877258066 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603002d050 con 0x7f6040069d10 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6047ed4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6047ed4640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 1 -- 172.21.15.181:0/3877258066 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030007bd0 con 0x7f6040069d10 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6047ed4640 4 set_mon_vals no callback set 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.081+0000 7f6044c6e640 10 monclient: dump: 2024-09-15T03:37:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 10 monclient: shutdown 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 -- 172.21.15.181:0/3877258066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6040069d10 msgr2=0x7f604013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 --2- 172.21.15.181:0/3877258066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6040069d10 0x7f604013a490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6030009e60 tx=0x7f6030009780 comp rx=0 tx=0).stop 2024-09-15T03:37:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 -- 172.21.15.181:0/3877258066 shutdown_connections 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 --2- 172.21.15.181:0/3877258066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6040069d10 0x7f604013a490 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 -- 172.21.15.181:0/3877258066 >> 172.21.15.181:0/3877258066 conn(0x7f60400a5cf0 msgr2=0x7f60400a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 -- 172.21.15.181:0/3877258066 shutdown_connections 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 -- 172.21.15.181:0/3877258066 wait complete. 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 10 monclient: build_initial_monmap 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 10 monclient: monmap: 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.082+0000 7f6047ed4640 1 Processor -- start 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 -- start start 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient: init 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6040069d10 0x7f6040140e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f6040069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604009ec50 0x7f6040130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f604009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 0x7f604011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f604009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604011ef00 con 0x7f604009ec50 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604011cbd0 con 0x7f604009f020 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604011ec20 con 0x7f6040069d10 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6047ed4640 10 monclient(hunting): authenticate will time out at 1244.564449s 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 0x7f604011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6045c70640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6040069d10 msgr2=0x7f6040140e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 0x7f604011b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45384/0 (socket says 172.21.15.181:45384) 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 -- 172.21.15.181:0/2780416181 learned_addr learned my addr 172.21.15.181:0/2780416181 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): get_auth_request con 0x7f604009f020 auth_method 0 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6045c70640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6040069d10 0x7f6040140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f604546f640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604009ec50 0x7f6040130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient(hunting): handle_auth_done global_id 104518 payload 824 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 -- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6040069d10 msgr2=0x7f6040140e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6040069d10 0x7f6040140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 -- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604009ec50 msgr2=0x7f6040130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604009ec50 0x7f6040130af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:20.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient: _finish_hunting 0 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 monclient: found mon.noname-b 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f6046471640 1 -- 172.21.15.181:0/2780416181 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030009a20 con 0x7f604009f020 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f604546f640 10 monclient: get_auth_request con 0x7f604009ec50 auth_method 0 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.083+0000 7f604546f640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604009ec50 0x7f6040130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6046471640 1 -- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 msgr2=0x7f604011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6046471640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 0x7f604011b760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f60300099f0 tx=0x7f6030037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030031cd0 con 0x7f604009f020 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: dump: 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6030040070 con 0x7f604009f020 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 5 monclient: authenticate success, global_id 104518 2024-09-15T03:37:20.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 10 monclient: _renew_subs 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604011f1e0 con 0x7f604009f020 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030045660 con 0x7f604009f020 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603dffb640 10 monclient: dump: 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:20.102 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 10 monclient: _renew_subs 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604011de60 con 0x7f604009f020 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.086+0000 7f603ffff640 4 set_mon_vals no callback set 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6030042030 con 0x7f604009f020 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f6047ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f6047ed4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f6047ed4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60400008d0 con 0x7f604009f020 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f603dffb640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f602c0386f0 0x7f602c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(225..225 src has 199..225) v4 ==== 13018+0+0 (secure 0 0 0) 0x7f603006f610 con 0x7f604009f020 2024-09-15T03:37:20.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f6045c70640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f602c0386f0 0x7f602c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:20.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.087+0000 7f6045c70640 10 monclient: get_auth_request con 0x7f602c0386f0 auth_method 0 2024-09-15T03:37:20.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.089+0000 7f6045c70640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f602c0386f0 0x7f602c03aba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6038007fd0 tx=0x7f6038007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:20.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.091+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f60300449d0 con 0x7f604009f020 2024-09-15T03:37:20.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.091+0000 7f603dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:20.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.091+0000 7f603dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:20.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.254+0000 7f6047ed4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T03:37:20.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.254+0000 7f6047ed4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T03:37:20.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.254+0000 7f6047ed4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:20.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.254+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f60400008d0 con 0x7f604009f020 2024-09-15T03:37:20.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.908+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v226) v1 ==== 120+0+0 (secure 0 0 0) 0x7f60300330c0 con 0x7f604009f020 2024-09-15T03:37:20.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.908+0000 7f603dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T03:37:20.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:20.908+0000 7f603dffb640 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-15T03:37:21.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.006+0000 7f6047ed4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T03:37:21.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.006+0000 7f6047ed4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T03:37:21.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.006+0000 7f6047ed4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.006+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f60400008d0 con 0x7f604009f020 2024-09-15T03:37:21.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.008+0000 7f603dffb640 1 -- 172.21.15.181:0/2780416181 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v226) v1 ==== 120+0+0 (secure 0 0 0) 0x7f603003a3a0 con 0x7f604009f020 2024-09-15T03:37:21.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.008+0000 7f603dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-15T03:37:21.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.008+0000 7f603dffb640 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-15T03:37:21.009 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h10' does not appear in the crush map 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.009+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f602c0386f0 msgr2=0x7f602c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.009+0000 7f6047ed4640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f602c0386f0 0x7f602c03aba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6038007fd0 tx=0x7f6038007a70 comp rx=0 tx=0).stop 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 10 monclient: shutdown 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 msgr2=0x7f604011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 0x7f604011b760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f60300099f0 tx=0x7f6030037000 comp rx=0 tx=0).stop 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 shutdown_connections 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f602c0386f0 0x7f602c03aba0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f604009f020 0x7f604011b760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604009ec50 0x7f6040130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 --2- 172.21.15.181:0/2780416181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6040069d10 0x7f6040140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 >> 172.21.15.181:0/2780416181 conn(0x7f60400a5cf0 msgr2=0x7f604009dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:21.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 shutdown_connections 2024-09-15T03:37:21.012 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.010+0000 7f6047ed4640 1 -- 172.21.15.181:0/2780416181 wait complete. 2024-09-15T03:37:21.019 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:21.019 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h11 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.092+0000 7fc76703f640 10 monclient: get_monmap_and_config 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.092+0000 7fc76703f640 10 monclient: build_initial_monmap 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.092+0000 7fc76703f640 10 monclient: monmap: 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:21.092957+0000 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:21.092957+0000 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:21.093 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:21.094 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:21.094 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:21.094 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:21.094 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:21.094 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:21.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.092+0000 7fc76703f640 1 Processor -- start 2024-09-15T03:37:21.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.092+0000 7fc76703f640 1 -- start start 2024-09-15T03:37:21.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.092+0000 7fc76703f640 10 monclient: init 2024-09-15T03:37:21.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:21.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc76009ec50 0x7fc760131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): picked mon.noname-c con 0x7fc76009ec50 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7601321a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009f020 0x7fc760069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): picked mon.noname-b con 0x7fc76009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc760067420 con 0x7fc7601321a0 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7600a0480 con 0x7fc76009ec50 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600a0200 con 0x7fc76009f020 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc76703f640 10 monclient(hunting): authenticate will time out at 1245.574413s 2024-09-15T03:37:21.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc76009ec50 0x7fc760131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc7655dc640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009f020 msgr2=0x7fc760069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc75ffff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc7601321a0 0x7fc760068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55342/0 (socket says 172.21.15.181:55342) 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc75ffff640 1 -- 172.21.15.181:0/1963306594 learned_addr learned my addr 172.21.15.181:0/1963306594 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc7655dc640 1 --2- 172.21.15.181:0/1963306594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009f020 0x7fc760069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): get_auth_request con 0x7fc76009ec50 auth_method 0 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient(hunting): handle_auth_done global_id 104521 payload 274 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 1 -- 172.21.15.181:0/1963306594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009f020 msgr2=0x7fc760069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:21.098 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 1 --2- 172.21.15.181:0/1963306594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009f020 0x7fc760069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 1 -- 172.21.15.181:0/1963306594 >> v1:172.21.15.181:6789/0 conn(0x7fc7601321a0 legacy=0x7fc760068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:21.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient: _finish_hunting 0 2024-09-15T03:37:21.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 1 monclient: found mon.noname-c 2024-09-15T03:37:21.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:21.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.093+0000 7fc764ddb640 1 -- 172.21.15.181:0/1963306594 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76009fd80 con 0x7fc76009ec50 2024-09-15T03:37:21.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc764ddb640 1 -- 172.21.15.181:0/1963306594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009ec50 msgr2=0x7fc760131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:21.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc764ddb640 1 --2- 172.21.15.181:0/1963306594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009ec50 0x7fc760131c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc750009850 tx=0x7fc7500090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1132f3728b4e49e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 1 -- 172.21.15.181:0/1963306594 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750032440 con 0x7fc76009ec50 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: dump: 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:21.105 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc76703f640 5 monclient: authenticate success, global_id 104521 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc76703f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 1 -- 172.21.15.181:0/1963306594 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7500325a0 con 0x7fc76009ec50 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc76703f640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 1 -- 172.21.15.181:0/1963306594 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750032940 con 0x7fc76009ec50 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc76703f640 4 set_mon_vals no callback set 2024-09-15T03:37:21.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.096+0000 7fc75f7fe640 10 monclient: dump: 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:21.107 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 10 monclient: shutdown 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 -- 172.21.15.181:0/1963306594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009ec50 msgr2=0x7fc760131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 --2- 172.21.15.181:0/1963306594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009ec50 0x7fc760131c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc750009850 tx=0x7fc7500090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 -- 172.21.15.181:0/1963306594 shutdown_connections 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 --2- 172.21.15.181:0/1963306594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009f020 0x7fc760069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 --2- 172.21.15.181:0/1963306594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009ec50 0x7fc760131c60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 -- 172.21.15.181:0/1963306594 >> 172.21.15.181:0/1963306594 conn(0x7fc7600a5cf0 msgr2=0x7fc7600a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 -- 172.21.15.181:0/1963306594 shutdown_connections 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 1 -- 172.21.15.181:0/1963306594 wait complete. 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 10 monclient: build_initial_monmap 2024-09-15T03:37:21.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.097+0000 7fc76703f640 10 monclient: monmap: 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 Processor -- start 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 -- start start 2024-09-15T03:37:21.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient: init 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009ec50 0x7fc7601416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): picked mon.noname-c con 0x7fc76009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 0x7fc760141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): picked mon.noname-b con 0x7fc76009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7601321a0 0x7fc760142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7601321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760120750 con 0x7fc7601321a0 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76009c020 con 0x7fc76009f020 2024-09-15T03:37:21.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc760120980 con 0x7fc76009ec50 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc76703f640 10 monclient(hunting): authenticate will time out at 1245.579848s 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 0x7fc760141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc764ddb640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009ec50 msgr2=0x7fc7601416c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 0x7fc760141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45410/0 (socket says 172.21.15.181:45410) 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 1 -- 172.21.15.181:0/2085099397 learned_addr learned my addr 172.21.15.181:0/2085099397 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 10 monclient(hunting): get_auth_request con 0x7fc76009f020 auth_method 0 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc764ddb640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009ec50 0x7fc7601416c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:21.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.098+0000 7fc75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc7655dc640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7601321a0 0x7fc760142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc7655dc640 10 monclient(hunting): get_auth_request con 0x7fc7601321a0 auth_method 0 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc7655dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc7655dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc7655dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 10 monclient(hunting): handle_auth_done global_id 104524 payload 824 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 1 -- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009ec50 msgr2=0x7fc7601416c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009ec50 0x7fc7601416c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 1 -- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7601321a0 msgr2=0x7fc760142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:21.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7601321a0 0x7fc760142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 1 monclient: found mon.noname-b 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.099+0000 7fc75ffff640 1 -- 172.21.15.181:0/2085099397 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750009390 con 0x7fc76009f020 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75ffff640 1 -- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 msgr2=0x7fc760141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75ffff640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 0x7fc760141c00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c5e0 tx=0x7fc754004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754005cb0 con 0x7fc76009f020 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:21.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: dump: 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:21.116 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 5 monclient: authenticate success, global_id 104524 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 10 monclient: _renew_subs 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76011fb80 con 0x7fc76009f020 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc754005e30 con 0x7fc76009f020 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754007340 con 0x7fc76009f020 2024-09-15T03:37:21.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75cff9640 10 monclient: dump: 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:21.118 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 10 monclient: _renew_subs 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc75effd640 4 set_mon_vals no callback set 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.101+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76009e140 con 0x7fc76009f020 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc75401b400 con 0x7fc76009f020 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc76703f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc76703f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc76703f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7600008d0 con 0x7fc76009f020 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc75cff9640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc74c0386f0 0x7fc74c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:21.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(226..226 src has 199..226) v4 ==== 12805+0+0 (secure 0 0 0) 0x7fc75404d5f0 con 0x7fc76009f020 2024-09-15T03:37:21.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc764ddb640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc74c0386f0 0x7fc74c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:21.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.102+0000 7fc764ddb640 10 monclient: get_auth_request con 0x7fc74c0386f0 auth_method 0 2024-09-15T03:37:21.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.105+0000 7fc764ddb640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc74c0386f0 0x7fc74c03aba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc7601325a0 tx=0x7fc750032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:21.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.107+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc75401ab40 con 0x7fc76009f020 2024-09-15T03:37:21.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.107+0000 7fc75cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:21.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.107+0000 7fc75cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:21.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.264+0000 7fc76703f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T03:37:21.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.264+0000 7fc76703f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T03:37:21.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.264+0000 7fc76703f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:21.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:21.264+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7fc7600a13e0 con 0x7fc76009f020 2024-09-15T03:37:22.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.262+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v227) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc75401a360 con 0x7fc76009f020 2024-09-15T03:37:22.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.262+0000 7fc75cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T03:37:22.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.262+0000 7fc75cff9640 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-15T03:37:22.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.362+0000 7fc76703f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T03:37:22.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.362+0000 7fc76703f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T03:37:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.362+0000 7fc76703f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.362+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7fc7601204a0 con 0x7fc76009f020 2024-09-15T03:37:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.364+0000 7fc75cff9640 1 -- 172.21.15.181:0/2085099397 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v227) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc75401a540 con 0x7fc76009f020 2024-09-15T03:37:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.364+0000 7fc75cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-15T03:37:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.364+0000 7fc75cff9640 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-15T03:37:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h11' does not appear in the crush map 2024-09-15T03:37:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc74c0386f0 msgr2=0x7fc74c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:22.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc74c0386f0 0x7fc74c03aba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc7601325a0 tx=0x7fc750032000 comp rx=0 tx=0).stop 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 10 monclient: shutdown 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 msgr2=0x7fc760141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 0x7fc760141c00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c5e0 tx=0x7fc754004960 comp rx=0 tx=0).stop 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 shutdown_connections 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc74c0386f0 0x7fc74c03aba0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7601321a0 0x7fc760142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc76009f020 0x7fc760141c00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 --2- 172.21.15.181:0/2085099397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc76009ec50 0x7fc7601416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 >> 172.21.15.181:0/2085099397 conn(0x7fc7600a5cf0 msgr2=0x7fc76009e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 shutdown_connections 2024-09-15T03:37:22.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.366+0000 7fc76703f640 1 -- 172.21.15.181:0/2085099397 wait complete. 2024-09-15T03:37:22.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:22.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h12 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.450+0000 7fd8f3700640 10 monclient: get_monmap_and_config 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.450+0000 7fd8f3700640 10 monclient: build_initial_monmap 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.450+0000 7fd8f3700640 10 monclient: monmap: 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:22.450974+0000 2024-09-15T03:37:22.451 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:22.450974+0000 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:22.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.450+0000 7fd8f3700640 1 Processor -- start 2024-09-15T03:37:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.450+0000 7fd8f3700640 1 -- start start 2024-09-15T03:37:22.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.450+0000 7fd8f3700640 10 monclient: init 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8ec09ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd8ec09f020 0x7fd8ec069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): picked mon.noname-c con 0x7fd8ec09f020 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): picked mon.noname-b con 0x7fd8ec136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec067420 con 0x7fd8ec09ec50 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0a0480 con 0x7fd8ec09f020 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:22.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec0a0200 con 0x7fd8ec136d60 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f3700640 10 monclient(hunting): authenticate will time out at 1246.932468s 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd8ec09f020 0x7fd8ec069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd8ec09f020 0x7fd8ec069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45422/0 (socket says 172.21.15.181:45422) 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2192709180 learned_addr learned my addr 172.21.15.181:0/2192709180 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): get_auth_request con 0x7fd8ec09f020 auth_method 0 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f149c640 1 --1- 172.21.15.181:0/2192709180 >> v1:172.21.15.181:6789/0 conn(0x7fd8ec09ec50 0x7fd8ec131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55350/0 (socket says 172.21.15.181:55350) 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f1c9d640 1 -- 172.21.15.181:0/2192709180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 msgr2=0x7fd8ec139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:22.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f1c9d640 1 --2- 172.21.15.181:0/2192709180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_done global_id 104527 payload 274 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2192709180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 msgr2=0x7fd8ec139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 --2- 172.21.15.181:0/2192709180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2192709180 >> v1:172.21.15.181:6789/0 conn(0x7fd8ec09ec50 legacy=0x7fd8ec131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient: _finish_hunting 0 2024-09-15T03:37:22.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 monclient: found mon.noname-c 2024-09-15T03:37:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.451+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2192709180 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec09fd80 con 0x7fd8ec09f020 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2192709180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 msgr2=0x7fd8ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8f0c9b640 1 --2- 172.21.15.181:0/2192709180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec069d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009910 tx=0x7fd8dc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2378712762b851f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 1 -- 172.21.15.181:0/2192709180 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc032440 con 0x7fd8ec09f020 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: dump: 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:22.464 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 1 -- 172.21.15.181:0/2192709180 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc0325a0 con 0x7fd8ec09f020 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8f3700640 5 monclient: authenticate success, global_id 104527 2024-09-15T03:37:22.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8f3700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8f3700640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 1 -- 172.21.15.181:0/2192709180 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc032940 con 0x7fd8ec09f020 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8f3700640 4 set_mon_vals no callback set 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.454+0000 7fd8e3fff640 10 monclient: dump: 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.466 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 10 monclient: shutdown 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 -- 172.21.15.181:0/2192709180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 msgr2=0x7fd8ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2192709180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec069d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009910 tx=0x7fd8dc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 -- 172.21.15.181:0/2192709180 shutdown_connections 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2192709180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2192709180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec069d10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 -- 172.21.15.181:0/2192709180 >> 172.21.15.181:0/2192709180 conn(0x7fd8ec0a5cf0 msgr2=0x7fd8ec0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 -- 172.21.15.181:0/2192709180 shutdown_connections 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 1 -- 172.21.15.181:0/2192709180 wait complete. 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 10 monclient: build_initial_monmap 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.455+0000 7fd8f3700640 10 monclient: monmap: 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 Processor -- start 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 -- start start 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient: init 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8ec09ec50 0x7fd8ec141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): picked mon.noname-a con 0x7fd8ec09ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec141e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): picked mon.noname-b con 0x7fd8ec09f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec097530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): picked mon.noname-c con 0x7fd8ec136d60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:22.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec0990f0 con 0x7fd8ec09ec50 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec09c020 con 0x7fd8ec09f020 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec09b1e0 con 0x7fd8ec136d60 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.456+0000 7fd8f3700640 10 monclient(hunting): authenticate will time out at 1246.937945s 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f1c9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 msgr2=0x7fd8ec097530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec141e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f1c9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec097530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec141e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45430/0 (socket says 172.21.15.181:45430) 2024-09-15T03:37:22.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2038457020 learned_addr learned my addr 172.21.15.181:0/2038457020 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): get_auth_request con 0x7fd8ec09f020 auth_method 0 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8ec09ec50 0x7fd8ec141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): get_auth_request con 0x7fd8ec09ec50 auth_method 0 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:22.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f149c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient(hunting): handle_auth_done global_id 104530 payload 824 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 msgr2=0x7fd8ec097530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec097530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8ec09ec50 msgr2=0x7fd8ec141920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8ec09ec50 0x7fd8ec141920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient: _finish_hunting 0 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 monclient: found mon.noname-b 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.457+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2038457020 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc009450 con 0x7fd8ec09f020 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f0c9b640 1 -- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 msgr2=0x7fd8ec141e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f0c9b640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec141e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009910 tx=0x7fd8dc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc03c960 con 0x7fd8ec09f020 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: dump: 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 5 monclient: authenticate success, global_id 104530 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc03cac0 con 0x7fd8ec09f020 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 10 monclient: _renew_subs 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec09b4c0 con 0x7fd8ec09f020 2024-09-15T03:37:22.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc03ce80 con 0x7fd8ec09f020 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e17fa640 10 monclient: dump: 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:22.475 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8e37fe640 4 set_mon_vals no callback set 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 10 monclient: _renew_subs 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.459+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec09af00 con 0x7fd8ec09f020 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd8dc039040 con 0x7fd8ec09f020 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8e17fa640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd8d00386f0 0x7fd8d003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(227..227 src has 199..227) v4 ==== 12592+0+0 (secure 0 0 0) 0x7fd8dc06d970 con 0x7fd8ec09f020 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8f3700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8f3700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8f3700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec0008d0 con 0x7fd8ec09f020 2024-09-15T03:37:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8f149c640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd8d00386f0 0x7fd8d003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.460+0000 7fd8f149c640 10 monclient: get_auth_request con 0x7fd8d00386f0 auth_method 0 2024-09-15T03:37:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.463+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.463+0000 7fd8f149c640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd8d00386f0 0x7fd8d003aba0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8007310 tx=0x7fd8e8007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.465+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd8dc040020 con 0x7fd8ec09f020 2024-09-15T03:37:22.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.465+0000 7fd8e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:22.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.465+0000 7fd8e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:22.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.623+0000 7fd8f3700640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T03:37:22.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.623+0000 7fd8f3700640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T03:37:22.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.623+0000 7fd8f3700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:22.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:22.623+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7fd8ec0a13e0 con 0x7fd8ec09f020 2024-09-15T03:37:22.917 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:37:22.917 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:37:22.917 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:37:22.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:37:22.920 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:37:22.921 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:37:22.921 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.024+0000 7f9a38c6f640 10 monclient: get_monmap_and_config 2024-09-15T03:37:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.024+0000 7f9a38c6f640 10 monclient: build_initial_monmap 2024-09-15T03:37:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.024+0000 7f9a38c6f640 10 monclient: monmap: 2024-09-15T03:37:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:23.025479+0000 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:23.025479+0000 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:23.026 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 1 Processor -- start 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 1 -- start start 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 10 monclient: init 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a34131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.025+0000 7f9a38c6f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9a3409f220 0x7f9a34138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a3409f220 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3409faf0 con 0x7f9a34131ab0 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a34132590 con 0x7f9a3409f220 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3409fd30 con 0x7f9a3409e910 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a38c6f640 10 monclient(hunting): authenticate will time out at 1240.665046s 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a3259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9a34131ab0 0x7f9a34131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50446/0 (socket says 172.21.15.153:50446) 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a3259c640 1 -- 172.21.15.153:0/3876102217 learned_addr learned my addr 172.21.15.153:0/3876102217 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 1 --2- 172.21.15.153:0/3876102217 >> v2:172.21.15.153:3300/0 conn(0x7f9a3409f220 0x7f9a34138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): get_auth_request con 0x7f9a3409f220 auth_method 0 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 10 monclient(hunting): get_auth_request con 0x7f9a3409e910 auth_method 0 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 1 -- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 msgr2=0x7f9a3409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 1 -- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 msgr2=0x7f9a3409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a31d9b640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_done global_id 114327 payload 274 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 1 -- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 msgr2=0x7f9a3409ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 1 -- 172.21.15.153:0/3876102217 >> v1:172.21.15.153:6789/0 conn(0x7f9a34131ab0 legacy=0x7f9a34131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 1 monclient: found mon.noname-a 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.026+0000 7f9a32d9d640 1 -- 172.21.15.153:0/3876102217 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a340a1110 con 0x7f9a3409f220 2024-09-15T03:37:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a32d9d640 1 -- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3409f220 msgr2=0x7f9a34138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a32d9d640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3409f220 0x7f9a34138bc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009e60 tx=0x7f9a2c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a713e97f7c4075ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 1 -- 172.21.15.153:0/3876102217 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c007760 con 0x7f9a3409f220 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: dump: 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: _finish_auth 0 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 1 -- 172.21.15.153:0/3876102217 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2c02d050 con 0x7f9a3409f220 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 5 monclient: authenticate success, global_id 114327 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 1 -- 172.21.15.153:0/3876102217 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c007af0 con 0x7f9a3409f220 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 4 set_mon_vals no callback set 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a3159a640 10 monclient: dump: 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 10 monclient: shutdown 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.029+0000 7f9a38c6f640 1 -- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3409f220 msgr2=0x7f9a34138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3409f220 0x7f9a34138bc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009e60 tx=0x7f9a2c009780 comp rx=0 tx=0).stop 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 -- 172.21.15.153:0/3876102217 shutdown_connections 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3409f220 0x7f9a34138bc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/3876102217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 -- 172.21.15.153:0/3876102217 >> 172.21.15.153:0/3876102217 conn(0x7f9a34067d30 msgr2=0x7f9a340a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 -- 172.21.15.153:0/3876102217 shutdown_connections 2024-09-15T03:37:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 -- 172.21.15.153:0/3876102217 wait complete. 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 10 monclient: build_initial_monmap 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 10 monclient: monmap: 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.030+0000 7f9a38c6f640 1 Processor -- start 2024-09-15T03:37:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 -- start start 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient: init 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a3409f220 0x7f9a34096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3409f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 0x7f9a34096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a34131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34098000 con 0x7f9a34131ab0 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34099870 con 0x7f9a3409f220 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3409a250 con 0x7f9a3409e910 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a38c6f640 10 monclient(hunting): authenticate will time out at 1240.670481s 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 0x7f9a34096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 0x7f9a34096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46504/0 (socket says 172.21.15.153:46504) 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 -- 172.21.15.153:0/1057259937 learned_addr learned my addr 172.21.15.153:0/1057259937 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): get_auth_request con 0x7f9a34131ab0 auth_method 0 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 10 monclient(hunting): get_auth_request con 0x7f9a3409e910 auth_method 0 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 msgr2=0x7f9a3409b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 msgr2=0x7f9a3409b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a3409f220 0x7f9a34096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a3259c640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409b540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): get_auth_request con 0x7f9a3409f220 auth_method 0 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient(hunting): handle_auth_done global_id 114330 payload 824 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 msgr2=0x7f9a3409b540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409b540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a3409f220 msgr2=0x7f9a34096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a3409f220 0x7f9a34096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 monclient: found mon.noname-a 2024-09-15T03:37:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.031+0000 7f9a32d9d640 1 -- 172.21.15.153:0/1057259937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c009a20 con 0x7f9a34131ab0 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a32d9d640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 msgr2=0x7f9a34096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a32d9d640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 0x7f9a34096fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0099f0 tx=0x7f9a2c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c031c10 con 0x7f9a34131ab0 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: dump: 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2c040070 con 0x7f9a34131ab0 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 5 monclient: authenticate success, global_id 114330 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 10 monclient: _renew_subs 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3409d300 con 0x7f9a34131ab0 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c045430 con 0x7f9a34131ab0 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a227fc640 10 monclient: dump: 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a30d99640 4 set_mon_vals no callback set 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 10 monclient: _renew_subs 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.034+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3409a8a0 con 0x7f9a34131ab0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9a2c042030 con 0x7f9a34131ab0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a38c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a38c6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a38c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3409aa90 con 0x7f9a34131ab0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a227fc640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a14038740 0x7f9a1403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(227..227 src has 199..227) v4 ==== 12592+0+0 (secure 0 0 0) 0x7f9a2c06f270 con 0x7f9a34131ab0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a3259c640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a14038740 0x7f9a1403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.035+0000 7f9a3259c640 10 monclient: get_auth_request con 0x7f9a14038740 auth_method 0 2024-09-15T03:37:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.038+0000 7f9a3259c640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a14038740 0x7f9a1403abf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9a24007d10 tx=0x7f9a24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.039+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9a3409aa90 con 0x7f9a34131ab0 2024-09-15T03:37:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.039+0000 7f9a227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.039+0000 7f9a227fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.257+0000 7f9a38c6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.257+0000 7f9a38c6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.257+0000 7f9a38c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.257+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9a3409aa90 con 0x7f9a34131ab0 2024-09-15T03:37:23.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.258+0000 7f9a227fc640 1 -- 172.21.15.153:0/1057259937 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9a3409aa90 con 0x7f9a34131ab0 2024-09-15T03:37:23.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.258+0000 7f9a227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:23.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.258+0000 7f9a227fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:23.259 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a14038740 msgr2=0x7f9a1403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a14038740 0x7f9a1403abf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9a24007d10 tx=0x7f9a24009040 comp rx=0 tx=0).stop 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 10 monclient: shutdown 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 msgr2=0x7f9a34096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 0x7f9a34096fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0099f0 tx=0x7f9a2c037000 comp rx=0 tx=0).stop 2024-09-15T03:37:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 shutdown_connections 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9a14038740 0x7f9a1403abf0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34131ab0 0x7f9a34096fa0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a3409f220 0x7f9a34096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 --2- 172.21.15.153:0/1057259937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a3409e910 0x7f9a3409b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 >> 172.21.15.153:0/1057259937 conn(0x7f9a34067d30 msgr2=0x7f9a3409d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 shutdown_connections 2024-09-15T03:37:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:23.260+0000 7f9a38c6f640 1 -- 172.21.15.153:0/1057259937 wait complete. 2024-09-15T03:37:23.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.269+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v228) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd8dc037070 con 0x7fd8ec09f020 2024-09-15T03:37:23.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.269+0000 7fd8e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T03:37:23.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.269+0000 7fd8e17fa640 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-15T03:37:23.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.366+0000 7fd8f3700640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T03:37:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.366+0000 7fd8f3700640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T03:37:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.366+0000 7fd8f3700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.366+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7fd8ec099ce0 con 0x7fd8ec09f020 2024-09-15T03:37:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.368+0000 7fd8e17fa640 1 -- 172.21.15.181:0/2038457020 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v228) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd8dc043360 con 0x7fd8ec09f020 2024-09-15T03:37:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.368+0000 7fd8e17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-15T03:37:23.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.368+0000 7fd8e17fa640 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-15T03:37:23.369 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h12' does not appear in the crush map 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.369+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd8d00386f0 msgr2=0x7fd8d003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.369+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd8d00386f0 0x7fd8d003aba0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8007310 tx=0x7fd8e8007e40 comp rx=0 tx=0).stop 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.369+0000 7fd8f3700640 10 monclient: shutdown 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.369+0000 7fd8f3700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.369+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 msgr2=0x7fd8ec141e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.369+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec141e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009910 tx=0x7fd8dc032000 comp rx=0 tx=0).stop 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 shutdown_connections 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd8d00386f0 0x7fd8d003aba0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8ec136d60 0x7fd8ec097530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8ec09f020 0x7fd8ec141e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 --2- 172.21.15.181:0/2038457020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8ec09ec50 0x7fd8ec141920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 >> 172.21.15.181:0/2038457020 conn(0x7fd8ec0a5cf0 msgr2=0x7fd8ec09d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:23.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 shutdown_connections 2024-09-15T03:37:23.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.370+0000 7fd8f3700640 1 -- 172.21.15.181:0/2038457020 wait complete. 2024-09-15T03:37:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:23.379 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h13 2024-09-15T03:37:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.451+0000 7f02a151d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:23.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.451+0000 7f02a151d640 10 monclient: build_initial_monmap 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.451+0000 7f02a151d640 10 monclient: monmap: 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:23.452312+0000 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:23.452312+0000 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:23.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 1 Processor -- start 2024-09-15T03:37:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 1 -- start start 2024-09-15T03:37:23.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient: init 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f029c09ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f029c09f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f029c069d10 0x7f029c13a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient(hunting): picked mon.noname-a con 0x7f029c069d10 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c067420 con 0x7f029c09f020 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c0a0480 con 0x7f029c09ec50 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c06a170 con 0x7f029c069d10 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f02a151d640 10 monclient(hunting): authenticate will time out at 1247.933640s 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029affd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f029c09ec50 0x7f029c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55356/0 (socket says 172.21.15.181:55356) 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029affd640 1 -- 172.21.15.181:0/4152279872 learned_addr learned my addr 172.21.15.181:0/4152279872 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029b7fe640 1 --2- 172.21.15.181:0/4152279872 >> v2:172.21.15.153:3300/0 conn(0x7f029c069d10 0x7f029c13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029b7fe640 10 monclient(hunting): get_auth_request con 0x7f029c069d10 auth_method 0 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.452+0000 7f029b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:23.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 10 monclient(hunting): handle_auth_done global_id 114333 payload 274 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 1 -- 172.21.15.181:0/4152279872 >> v1:172.21.15.181:6789/0 conn(0x7f029c09ec50 legacy=0x7f029c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 1 -- 172.21.15.181:0/4152279872 >> v1:172.21.15.153:6789/0 conn(0x7f029c09f020 legacy=0x7f029c136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 1 monclient: found mon.noname-a 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:23.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.453+0000 7f029b7fe640 1 -- 172.21.15.181:0/4152279872 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0a0200 con 0x7f029c069d10 2024-09-15T03:37:23.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f029b7fe640 1 -- 172.21.15.181:0/4152279872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c069d10 msgr2=0x7f029c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:23.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f029b7fe640 1 --2- 172.21.15.181:0/4152279872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c069d10 0x7f029c13a490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0294009e60 tx=0x7f0294009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f95ab05b81201 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:23.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 1 -- 172.21.15.181:0/4152279872 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294007840 con 0x7f029c069d10 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: dump: 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 1 -- 172.21.15.181:0/4152279872 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029402d050 con 0x7f029c069d10 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 5 monclient: authenticate success, global_id 114333 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 4 set_mon_vals no callback set 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 1 -- 172.21.15.181:0/4152279872 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294007bd0 con 0x7f029c069d10 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f0299ffb640 10 monclient: dump: 2024-09-15T03:37:23.466 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 10 monclient: shutdown 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 -- 172.21.15.181:0/4152279872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c069d10 msgr2=0x7f029c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 --2- 172.21.15.181:0/4152279872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c069d10 0x7f029c13a490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0294009e60 tx=0x7f0294009780 comp rx=0 tx=0).stop 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 -- 172.21.15.181:0/4152279872 shutdown_connections 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 --2- 172.21.15.181:0/4152279872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c069d10 0x7f029c13a490 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 -- 172.21.15.181:0/4152279872 >> 172.21.15.181:0/4152279872 conn(0x7f029c0a5cf0 msgr2=0x7f029c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 -- 172.21.15.181:0/4152279872 shutdown_connections 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.456+0000 7f02a151d640 1 -- 172.21.15.181:0/4152279872 wait complete. 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.457+0000 7f02a151d640 10 monclient: build_initial_monmap 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.457+0000 7f02a151d640 10 monclient: monmap: 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:23.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.457+0000 7f02a151d640 1 Processor -- start 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.457+0000 7f02a151d640 1 -- start start 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.457+0000 7f02a151d640 10 monclient: init 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 0x7f029c11de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): picked mon.noname-b con 0x7f029c069d10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 0x7f029c11e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): picked mon.noname-c con 0x7f029c09ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c09f020 0x7f029c11e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): picked mon.noname-a con 0x7f029c09f020 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c120400 con 0x7f029c09f020 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c09c020 con 0x7f029c069d10 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c09e180 con 0x7f029c09ec50 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f02a151d640 10 monclient(hunting): authenticate will time out at 1247.939196s 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 0x7f029c11de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 0x7f029c11de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45436/0 (socket says 172.21.15.181:45436) 2024-09-15T03:37:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 -- 172.21.15.181:0/2655420812 learned_addr learned my addr 172.21.15.181:0/2655420812 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): get_auth_request con 0x7f029c069d10 auth_method 0 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c09f020 0x7f029c11e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 0x7f029c11e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): get_auth_request con 0x7f029c09f020 auth_method 0 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 10 monclient(hunting): get_auth_request con 0x7f029c09ec50 auth_method 0 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 msgr2=0x7f029c11e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 msgr2=0x7f029c11e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 0x7f029c11e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient(hunting): handle_auth_done global_id 104536 payload 824 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029a7fc640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 0x7f029c11e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:23.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 msgr2=0x7f029c11e360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 0x7f029c11e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c09f020 msgr2=0x7f029c11e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c09f020 0x7f029c11e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 monclient: found mon.noname-b 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.458+0000 7f029affd640 1 -- 172.21.15.181:0/2655420812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294009a20 con 0x7f029c069d10 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f029affd640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 msgr2=0x7f029c11de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f029affd640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 0x7f029c11de20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c610 tx=0x7f028c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c005cb0 con 0x7f029c069d10 2024-09-15T03:37:23.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f0282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f0282ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f0282ffd640 10 monclient: dump: 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f0282ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:23.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.460+0000 7f0282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 5 monclient: authenticate success, global_id 104536 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 monclient: _renew_subs 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c11fa00 con 0x7f029c069d10 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c005e30 con 0x7f029c069d10 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c007340 con 0x7f029c069d10 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 10 monclient: dump: 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:23.475 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 monclient: _renew_subs 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c121540 con 0x7f029c069d10 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02997fa640 4 set_mon_vals no callback set 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:23.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f028c01b400 con 0x7f029c069d10 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.461+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c131670 con 0x7f029c069d10 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.462+0000 7f0282ffd640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0288038740 0x7f028803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.462+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(228..228 src has 199..228) v4 ==== 12379+0+0 (secure 0 0 0) 0x7f028c04d400 con 0x7f029c069d10 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.462+0000 7f029a7fc640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0288038740 0x7f028803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.462+0000 7f029a7fc640 10 monclient: get_auth_request con 0x7f0288038740 auth_method 0 2024-09-15T03:37:23.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.464+0000 7f029a7fc640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0288038740 0x7f028803abf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f029c11f600 tx=0x7f0290009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:23.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.466+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f028c01a720 con 0x7f029c069d10 2024-09-15T03:37:23.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.466+0000 7f0282ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:23.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.466+0000 7f0282ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:23.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.622+0000 7f02a151d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T03:37:23.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.622+0000 7f02a151d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T03:37:23.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.622+0000 7f02a151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:23.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:23.622+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f029c0008d0 con 0x7f029c069d10 2024-09-15T03:37:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.389+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v229) v1 ==== 120+0+0 (secure 0 0 0) 0x7f028c01a360 con 0x7f029c069d10 2024-09-15T03:37:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.389+0000 7f0282ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T03:37:24.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.389+0000 7f0282ffd640 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-15T03:37:24.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.488+0000 7f02a151d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T03:37:24.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.488+0000 7f02a151d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T03:37:24.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.488+0000 7f02a151d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:24.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.488+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f029c09fd80 con 0x7f029c069d10 2024-09-15T03:37:24.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.490+0000 7f0282ffd640 1 -- 172.21.15.181:0/2655420812 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v229) v1 ==== 120+0+0 (secure 0 0 0) 0x7f028c01a540 con 0x7f029c069d10 2024-09-15T03:37:24.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.490+0000 7f0282ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-15T03:37:24.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.490+0000 7f0282ffd640 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-15T03:37:24.492 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h13' does not appear in the crush map 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0288038740 msgr2=0x7f028803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0288038740 0x7f028803abf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f029c11f600 tx=0x7f0290009040 comp rx=0 tx=0).stop 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 10 monclient: shutdown 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 msgr2=0x7f029c11de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 0x7f029c11de20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c610 tx=0x7f028c004960 comp rx=0 tx=0).stop 2024-09-15T03:37:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 shutdown_connections 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0288038740 0x7f028803abf0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f029c09f020 0x7f029c11e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f029c09ec50 0x7f029c11e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 --2- 172.21.15.181:0/2655420812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f029c069d10 0x7f029c11de20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 >> 172.21.15.181:0/2655420812 conn(0x7f029c0a5cf0 msgr2=0x7f029c09d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 shutdown_connections 2024-09-15T03:37:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.492+0000 7f02a151d640 1 -- 172.21.15.181:0/2655420812 wait complete. 2024-09-15T03:37:24.502 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:24.503 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h14 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.573+0000 7f27d7d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.573+0000 7f27d7d5d640 10 monclient: build_initial_monmap 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.573+0000 7f27d7d5d640 10 monclient: monmap: 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:24.574874+0000 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:24.574874+0000 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:24.575 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:24.576 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:24.576 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:24.576 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.573+0000 7f27d7d5d640 1 Processor -- start 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.573+0000 7f27d7d5d640 1 -- start start 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.573+0000 7f27d7d5d640 10 monclient: init 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f27d0096a30 0x7f27d009d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f27d0096a30 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27d009b0b0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27d0097510 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d009b970 con 0x7f27d009b0b0 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d009c470 con 0x7f27d0097510 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d009c190 con 0x7f27d0096a30 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d7d5d640 10 monclient(hunting): authenticate will time out at 1249.056313s 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d62fa640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f27d0097510 0x7f27d0136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55370/0 (socket says 172.21.15.181:55370) 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d62fa640 1 -- 172.21.15.181:0/3344495618 learned_addr learned my addr 172.21.15.181:0/3344495618 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 1 --2- 172.21.15.181:0/3344495618 >> v2:172.21.15.153:3300/0 conn(0x7f27d0096a30 0x7f27d009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): get_auth_request con 0x7f27d0096a30 auth_method 0 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient(hunting): handle_auth_done global_id 114339 payload 274 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 1 -- 172.21.15.181:0/3344495618 >> v1:172.21.15.181:6789/0 conn(0x7f27d0097510 legacy=0x7f27d0136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 1 -- 172.21.15.181:0/3344495618 >> v1:172.21.15.153:6789/0 conn(0x7f27d009b0b0 legacy=0x7f27d0096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient: _finish_hunting 0 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 1 monclient: found mon.noname-a 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:24.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.574+0000 7f27d5af9640 1 -- 172.21.15.181:0/3344495618 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d009c750 con 0x7f27d0096a30 2024-09-15T03:37:24.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.577+0000 7f27d5af9640 1 -- 172.21.15.181:0/3344495618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d0096a30 msgr2=0x7f27d009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:24.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.577+0000 7f27d5af9640 1 --2- 172.21.15.181:0/3344495618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d0096a30 0x7f27d009d320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009850 tx=0x7f27c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a1bc61e227fbd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:24.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 1 -- 172.21.15.181:0/3344495618 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4032440 con 0x7f27d0096a30 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: dump: 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:24.585 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: _finish_auth 0 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 1 -- 172.21.15.181:0/3344495618 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c40325a0 con 0x7f27d0096a30 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 1 -- 172.21.15.181:0/3344495618 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4032940 con 0x7f27d0096a30 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d4af7640 10 monclient: dump: 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:24.586 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d7d5d640 5 monclient: authenticate success, global_id 114339 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d7d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.578+0000 7f27d7d5d640 4 set_mon_vals no callback set 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 10 monclient: shutdown 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3344495618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d0096a30 msgr2=0x7f27d009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3344495618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d0096a30 0x7f27d009d320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009850 tx=0x7f27c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3344495618 shutdown_connections 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3344495618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d0096a30 0x7f27d009d320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3344495618 >> 172.21.15.181:0/3344495618 conn(0x7f27d0071100 msgr2=0x7f27d00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3344495618 shutdown_connections 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3344495618 wait complete. 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 10 monclient: build_initial_monmap 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 10 monclient: monmap: 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:24.588 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 Processor -- start 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 1 -- start start 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.579+0000 7f27d7d5d640 10 monclient: init 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 0x7f27d0132450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f27d0096a30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27d0097510 0x7f27d0117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f27d0097510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:24.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d009b0b0 0x7f27d0117a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:24.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f27d009b0b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d009bd40 con 0x7f27d009b0b0 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d011a690 con 0x7f27d0096a30 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d011b070 con 0x7f27d0097510 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d7d5d640 10 monclient(hunting): authenticate will time out at 1249.062375s 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 0x7f27d0132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 0x7f27d0132450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45448/0 (socket says 172.21.15.181:45448) 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 -- 172.21.15.181:0/3617837654 learned_addr learned my addr 172.21.15.181:0/3617837654 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): get_auth_request con 0x7f27d0096a30 auth_method 0 2024-09-15T03:37:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d62fa640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d009b0b0 0x7f27d0117a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d52f8640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27d0097510 0x7f27d0117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient(hunting): handle_auth_done global_id 104539 payload 824 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 -- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27d0097510 msgr2=0x7f27d0117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27d0097510 0x7f27d0117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 -- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d009b0b0 msgr2=0x7f27d0117a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d009b0b0 0x7f27d0117a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient: _finish_hunting 0 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 monclient: found mon.noname-b 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d5af9640 1 -- 172.21.15.181:0/3617837654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4009390 con 0x7f27d0096a30 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d62fa640 10 monclient: get_auth_request con 0x7f27d009b0b0 auth_method 0 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d52f8640 10 monclient: get_auth_request con 0x7f27d0097510 auth_method 0 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d62fa640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d009b0b0 0x7f27d0117a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.580+0000 7f27d52f8640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27d0097510 0x7f27d0117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.582+0000 7f27d5af9640 1 -- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 msgr2=0x7f27d0132450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.582+0000 7f27d5af9640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 0x7f27d0132450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009850 tx=0x7f27c4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:24.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.582+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:24.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40319a0 con 0x7f27d0096a30 2024-09-15T03:37:24.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:24.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:24.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: dump: 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:24.597 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 5 monclient: authenticate success, global_id 104539 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 10 monclient: _renew_subs 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d011b2f0 con 0x7f27d0096a30 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4031b00 con 0x7f27d0096a30 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4031e80 con 0x7f27d0096a30 2024-09-15T03:37:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c1ffb640 10 monclient: dump: 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27c3fff640 4 set_mon_vals no callback set 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 10 monclient: _renew_subs 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.585+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d0119450 con 0x7f27d0096a30 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f27c4039040 con 0x7f27d0096a30 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27d7d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27d7d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27d7d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d00008d0 con 0x7f27d0096a30 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27c1ffb640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f27b4038740 0x7f27b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(229..229 src has 199..229) v4 ==== 12166+0+0 (secure 0 0 0) 0x7f27c406df90 con 0x7f27d0096a30 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27d52f8640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f27b4038740 0x7f27b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.586+0000 7f27d52f8640 10 monclient: get_auth_request con 0x7f27b4038740 auth_method 0 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.588+0000 7f27d52f8640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f27b4038740 0x7f27b403abf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f27d0118b20 tx=0x7f27cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:24.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.590+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f27c404c580 con 0x7f27d0096a30 2024-09-15T03:37:24.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.590+0000 7f27c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:24.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.590+0000 7f27c1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:24.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.749+0000 7f27d7d5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T03:37:24.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.749+0000 7f27d7d5d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T03:37:24.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.749+0000 7f27d7d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:24.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:24.749+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f27d00008d0 con 0x7f27d0096a30 2024-09-15T03:37:25.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.387+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v230) v1 ==== 120+0+0 (secure 0 0 0) 0x7f27c4040020 con 0x7f27d0096a30 2024-09-15T03:37:25.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.388+0000 7f27c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T03:37:25.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.388+0000 7f27c1ffb640 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-15T03:37:25.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.491+0000 7f27d7d5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T03:37:25.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.491+0000 7f27d7d5d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T03:37:25.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.491+0000 7f27d7d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.491+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f27d011a7d0 con 0x7f27d0096a30 2024-09-15T03:37:25.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.492+0000 7f27c1ffb640 1 -- 172.21.15.181:0/3617837654 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v230) v1 ==== 120+0+0 (secure 0 0 0) 0x7f27d011a7d0 con 0x7f27d0096a30 2024-09-15T03:37:25.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.492+0000 7f27c1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-15T03:37:25.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.492+0000 7f27c1ffb640 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-15T03:37:25.494 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h14' does not appear in the crush map 2024-09-15T03:37:25.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f27b4038740 msgr2=0x7f27b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:25.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f27b4038740 0x7f27b403abf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f27d0118b20 tx=0x7f27cc009040 comp rx=0 tx=0).stop 2024-09-15T03:37:25.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 10 monclient: shutdown 2024-09-15T03:37:25.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:25.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 msgr2=0x7f27d0132450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:25.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 0x7f27d0132450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009850 tx=0x7f27c4032000 comp rx=0 tx=0).stop 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 shutdown_connections 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f27b4038740 0x7f27b403abf0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d009b0b0 0x7f27d0117a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27d0097510 0x7f27d0117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 --2- 172.21.15.181:0/3617837654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d0096a30 0x7f27d0132450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 >> 172.21.15.181:0/3617837654 conn(0x7f27d0071100 msgr2=0x7f27d006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 shutdown_connections 2024-09-15T03:37:25.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.494+0000 7f27d7d5d640 1 -- 172.21.15.181:0/3617837654 wait complete. 2024-09-15T03:37:25.505 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:25.505 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h15 2024-09-15T03:37:25.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.575+0000 7f6d37268640 10 monclient: get_monmap_and_config 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.575+0000 7f6d37268640 10 monclient: build_initial_monmap 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.576+0000 7f6d37268640 10 monclient: monmap: 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:25.577794+0000 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:25.577794+0000 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:25.578 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:25.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.576+0000 7f6d37268640 1 Processor -- start 2024-09-15T03:37:25.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.576+0000 7f6d37268640 1 -- start start 2024-09-15T03:37:25.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.576+0000 7f6d37268640 10 monclient: init 2024-09-15T03:37:25.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:25.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:37:25.581 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d300677e0 0x7f6d30095800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): picked mon.noname-b con 0x7f6d300677e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6d3009c090 0x7f6d3009c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): picked mon.noname-c con 0x7f6d3009c090 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6d3009c9a0 0x7f6d30094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): picked mon.noname-a con 0x7f6d3009c9a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3009d570 con 0x7f6d3009c9a0 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3009b560 con 0x7f6d3009c090 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3009d800 con 0x7f6d300677e0 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d37268640 10 monclient(hunting): authenticate will time out at 1250.059253s 2024-09-15T03:37:25.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6d3009c090 0x7f6d3009c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6d3009c090 0x7f6d3009c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45456/0 (socket says 172.21.15.181:45456) 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 -- 172.21.15.181:0/3695734967 learned_addr learned my addr 172.21.15.181:0/3695734967 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): get_auth_request con 0x7f6d3009c090 auth_method 0 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 1 --2- 172.21.15.181:0/3695734967 >> v2:172.21.15.153:3300/0 conn(0x7f6d3009c9a0 0x7f6d30094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35004640 1 --2- 172.21.15.181:0/3695734967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d300677e0 0x7f6d30095800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): get_auth_request con 0x7f6d3009c9a0 auth_method 0 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:25.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35004640 10 monclient(hunting): get_auth_request con 0x7f6d300677e0 auth_method 0 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d35805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient(hunting): handle_auth_done global_id 104542 payload 274 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 -- 172.21.15.181:0/3695734967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d300677e0 msgr2=0x7f6d30095800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 --2- 172.21.15.181:0/3695734967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d300677e0 0x7f6d30095800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 -- 172.21.15.181:0/3695734967 >> v2:172.21.15.153:3300/0 conn(0x7f6d3009c9a0 msgr2=0x7f6d30094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 --2- 172.21.15.181:0/3695734967 >> v2:172.21.15.153:3300/0 conn(0x7f6d3009c9a0 0x7f6d30094080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient: _finish_hunting 0 2024-09-15T03:37:25.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 monclient: found mon.noname-c 2024-09-15T03:37:25.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:25.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.577+0000 7f6d34803640 1 -- 172.21.15.181:0/3695734967 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3009cdc0 con 0x7f6d3009c090 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d34803640 1 -- 172.21.15.181:0/3695734967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 msgr2=0x7f6d3009c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d34803640 1 --2- 172.21.15.181:0/3695734967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d3009c460 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009910 tx=0x7f6d200091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d2b7a394e1e7959 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 1 -- 172.21.15.181:0/3695734967 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002aea0 con 0x7f6d3009c090 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: dump: 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d37268640 5 monclient: authenticate success, global_id 104542 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d37268640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 1 -- 172.21.15.181:0/3695734967 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d20032440 con 0x7f6d3009c090 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d37268640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 1 -- 172.21.15.181:0/3695734967 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d200327e0 con 0x7f6d3009c090 2024-09-15T03:37:25.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d37268640 4 set_mon_vals no callback set 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.580+0000 7f6d27fff640 10 monclient: dump: 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 10 monclient: shutdown 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 -- 172.21.15.181:0/3695734967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 msgr2=0x7f6d3009c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 --2- 172.21.15.181:0/3695734967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d3009c460 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009910 tx=0x7f6d200091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 -- 172.21.15.181:0/3695734967 shutdown_connections 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 --2- 172.21.15.181:0/3695734967 >> v2:172.21.15.153:3300/0 conn(0x7f6d3009c9a0 0x7f6d30094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 --2- 172.21.15.181:0/3695734967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d3009c460 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 --2- 172.21.15.181:0/3695734967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d300677e0 0x7f6d30095800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 -- 172.21.15.181:0/3695734967 >> 172.21.15.181:0/3695734967 conn(0x7f6d300a16a0 msgr2=0x7f6d300a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 -- 172.21.15.181:0/3695734967 shutdown_connections 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.581+0000 7f6d37268640 1 -- 172.21.15.181:0/3695734967 wait complete. 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient: build_initial_monmap 2024-09-15T03:37:25.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient: monmap: 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 Processor -- start 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 -- start start 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient: init 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:25.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d300677e0 0x7f6d30094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient(hunting): picked mon.noname-a con 0x7f6d300677e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d30094880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3009c090 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d3009c9a0 0x7f6d30128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient(hunting): picked mon.noname-c con 0x7f6d3009c9a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3012c050 con 0x7f6d300677e0 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.582+0000 7f6d37268640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3012a320 con 0x7f6d3009c090 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d37268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d37268640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3012aa40 con 0x7f6d3009c9a0 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d37268640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d37268640 10 monclient(hunting): authenticate will time out at 1250.065015s 2024-09-15T03:37:25.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d30094880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d30094880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45466/0 (socket says 172.21.15.181:45466) 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 -- 172.21.15.181:0/3049902723 learned_addr learned my addr 172.21.15.181:0/3049902723 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): get_auth_request con 0x7f6d3009c090 auth_method 0 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35004640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d300677e0 0x7f6d30094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35805640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d3009c9a0 0x7f6d30128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35004640 10 monclient(hunting): get_auth_request con 0x7f6d300677e0 auth_method 0 2024-09-15T03:37:25.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35004640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35004640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35004640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35805640 10 monclient(hunting): get_auth_request con 0x7f6d3009c9a0 auth_method 0 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d35805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient(hunting): handle_auth_done global_id 104545 payload 824 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 -- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d3009c9a0 msgr2=0x7f6d30128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d3009c9a0 0x7f6d30128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 -- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d300677e0 msgr2=0x7f6d30094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d300677e0 0x7f6d30094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient: _finish_hunting 0 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 monclient: found mon.noname-b 2024-09-15T03:37:25.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.583+0000 7f6d34803640 1 -- 172.21.15.181:0/3049902723 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d20009450 con 0x7f6d3009c090 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d34803640 1 -- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 msgr2=0x7f6d30094880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d34803640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d30094880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6d20035040 tx=0x7f6d20032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002aea0 con 0x7f6d3009c090 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: dump: 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:25.598 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d20000c40 con 0x7f6d3009c090 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 5 monclient: authenticate success, global_id 104545 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 10 monclient: _renew_subs 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d30132b90 con 0x7f6d3009c090 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2003d440 con 0x7f6d3009c090 2024-09-15T03:37:25.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d257fa640 10 monclient: dump: 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 10 monclient: _renew_subs 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d30094dc0 con 0x7f6d3009c090 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.585+0000 7f6d277fe640 4 set_mon_vals no callback set 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6d20039040 con 0x7f6d3009c090 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d37268640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d37268640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d37268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d300008d0 con 0x7f6d3009c090 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d257fa640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d14038740 0x7f6d1403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(230..230 src has 199..230) v4 ==== 11953+0+0 (secure 0 0 0) 0x7f6d2006d4d0 con 0x7f6d3009c090 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.586+0000 7f6d35004640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d14038740 0x7f6d1403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.587+0000 7f6d35004640 10 monclient: get_auth_request con 0x7f6d14038740 auth_method 0 2024-09-15T03:37:25.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.589+0000 7f6d35004640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d14038740 0x7f6d1403abf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00bb40 tx=0x7f6d2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:25.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.591+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6d20030b20 con 0x7f6d3009c090 2024-09-15T03:37:25.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.591+0000 7f6d257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:25.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.591+0000 7f6d257fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:25.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.750+0000 7f6d37268640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T03:37:25.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.750+0000 7f6d37268640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T03:37:25.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.750+0000 7f6d37268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:25.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:25.750+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f6d30095d40 con 0x7f6d3009c090 2024-09-15T03:37:26.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:26.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.376+0000 7faaff28b640 10 monclient: get_monmap_and_config 2024-09-15T03:37:26.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.376+0000 7faaff28b640 10 monclient: build_initial_monmap 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.376+0000 7faaff28b640 10 monclient: monmap: 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:26.377550+0000 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:26.377550+0000 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:26.378 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:26.379 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:26.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.377+0000 7faaff28b640 1 Processor -- start 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.377+0000 7faaff28b640 1 -- start start 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.377+0000 7faaff28b640 10 monclient: init 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaf8096c60 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:26.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaf8097030 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7faaf8134b50 0x7faaf8136f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient(hunting): picked mon.noname-a con 0x7faaf8134b50 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80696a0 con 0x7faaf8097030 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf809d2e0 con 0x7faaf8096c60 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf809d710 con 0x7faaf8134b50 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faaff28b640 10 monclient(hunting): authenticate will time out at 1244.017281s 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7faaf8134b50 0x7faaf8136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafc826640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7faaf8097030 0x7faaf8097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34520/0 (socket says 172.21.15.153:34520) 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafc826640 1 -- 172.21.15.153:0/2832309964 learned_addr learned my addr 172.21.15.153:0/2832309964 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): get_auth_request con 0x7faaf8134b50 auth_method 0 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient(hunting): handle_auth_done global_id 114348 payload 274 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 1 -- 172.21.15.153:0/2832309964 >> v1:172.21.15.181:6789/0 conn(0x7faaf8096c60 legacy=0x7faaf809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 1 -- 172.21.15.153:0/2832309964 >> v1:172.21.15.153:6789/0 conn(0x7faaf8097030 legacy=0x7faaf8097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient: _finish_hunting 0 2024-09-15T03:37:26.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 1 monclient: found mon.noname-a 2024-09-15T03:37:26.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:26.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:26.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.378+0000 7faafd828640 1 -- 172.21.15.153:0/2832309964 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf80992f0 con 0x7faaf8134b50 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faafd828640 1 -- 172.21.15.153:0/2832309964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8134b50 msgr2=0x7faaf8136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faafd828640 1 --2- 172.21.15.153:0/2832309964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8134b50 0x7faaf8136f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faaf000d0d0 tx=0x7faaf000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e75b181c7db98f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 1 -- 172.21.15.153:0/2832309964 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf000b3c0 con 0x7faaf8134b50 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: dump: 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.390 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: _finish_auth 0 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 1 -- 172.21.15.153:0/2832309964 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf0009760 con 0x7faaf8134b50 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaff28b640 5 monclient: authenticate success, global_id 114348 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaff28b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:26.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaff28b640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaff28b640 4 set_mon_vals no callback set 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 1 -- 172.21.15.153:0/2832309964 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0009b00 con 0x7faaf8134b50 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.381+0000 7faaebfff640 10 monclient: dump: 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 10 monclient: shutdown 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 -- 172.21.15.153:0/2832309964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8134b50 msgr2=0x7faaf8136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 --2- 172.21.15.153:0/2832309964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8134b50 0x7faaf8136f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faaf000d0d0 tx=0x7faaf000b990 comp rx=0 tx=0).stop 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 -- 172.21.15.153:0/2832309964 shutdown_connections 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 --2- 172.21.15.153:0/2832309964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8134b50 0x7faaf8136f30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 -- 172.21.15.153:0/2832309964 >> 172.21.15.153:0/2832309964 conn(0x7faaf80672b0 msgr2=0x7faaf80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 -- 172.21.15.153:0/2832309964 shutdown_connections 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.382+0000 7faaff28b640 1 -- 172.21.15.153:0/2832309964 wait complete. 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.383+0000 7faaff28b640 10 monclient: build_initial_monmap 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.383+0000 7faaff28b640 10 monclient: monmap: 2024-09-15T03:37:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.383+0000 7faaff28b640 1 Processor -- start 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.383+0000 7faaff28b640 1 -- start start 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.383+0000 7faaff28b640 10 monclient: init 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 0x7faaf812dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): picked mon.noname-a con 0x7faaf8096c60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf8097030 0x7faaf812e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): picked mon.noname-b con 0x7faaf8097030 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 0x7faaf806d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): picked mon.noname-c con 0x7faaf8134b50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf809add0 con 0x7faaf8096c60 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8134260 con 0x7faaf8097030 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8118940 con 0x7faaf8134b50 2024-09-15T03:37:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faaff28b640 10 monclient(hunting): authenticate will time out at 1244.023173s 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 0x7faaf812dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 0x7faaf812dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41246/0 (socket says 172.21.15.153:41246) 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 -- 172.21.15.153:0/3468584904 learned_addr learned my addr 172.21.15.153:0/3468584904 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): get_auth_request con 0x7faaf8096c60 auth_method 0 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 0x7faaf806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf8097030 0x7faaf812e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 10 monclient(hunting): get_auth_request con 0x7faaf8134b50 auth_method 0 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): get_auth_request con 0x7faaf8097030 auth_method 0 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 msgr2=0x7faaf806d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 msgr2=0x7faaf806d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 0x7faaf806d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafc826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd828640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 0x7faaf806d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient(hunting): handle_auth_done global_id 114351 payload 824 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 msgr2=0x7faaf806d580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 0x7faaf806d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf8097030 msgr2=0x7faaf812e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf8097030 0x7faaf812e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient: _finish_hunting 0 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 monclient: found mon.noname-a 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.384+0000 7faafd027640 1 -- 172.21.15.153:0/3468584904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf000bc30 con 0x7faaf8096c60 2024-09-15T03:37:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faafd027640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 msgr2=0x7faaf812dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faafd027640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 0x7faaf812dcf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faaec00c530 tx=0x7faaec007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec0117a0 con 0x7faaf8096c60 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 10 monclient: dump: 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaec011940 con 0x7faaf8096c60 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faaff28b640 5 monclient: authenticate success, global_id 114351 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.386+0000 7faae97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec011d00 con 0x7faaf8096c60 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaff28b640 10 monclient: _renew_subs 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaff28b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf8118660 con 0x7faaf8096c60 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faae97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faae97fa640 10 monclient: dump: 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaeb7fe640 4 set_mon_vals no callback set 2024-09-15T03:37:26.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaff28b640 10 monclient: _renew_subs 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaff28b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8117700 con 0x7faaf8096c60 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.387+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7faaec012630 con 0x7faaf8096c60 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faaff28b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faaff28b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faaff28b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf81183f0 con 0x7faaf8096c60 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faae97fa640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faadc038740 0x7faadc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(230..230 src has 199..230) v4 ==== 11953+0+0 (secure 0 0 0) 0x7faaec04d420 con 0x7faaf8096c60 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faafc826640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faadc038740 0x7faadc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.388+0000 7faafc826640 10 monclient: get_auth_request con 0x7faadc038740 auth_method 0 2024-09-15T03:37:26.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.397+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v231) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6d20037090 con 0x7f6d3009c090 2024-09-15T03:37:26.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.397+0000 7f6d257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T03:37:26.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.397+0000 7f6d257fa640 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-15T03:37:26.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.390+0000 7faafc826640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faadc038740 0x7faadc03abf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7faaf809b250 tx=0x7faaf4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:26.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.392+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7faaf81183f0 con 0x7faaf8096c60 2024-09-15T03:37:26.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.392+0000 7faae97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:26.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.392+0000 7faae97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:26.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.495+0000 7f6d37268640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T03:37:26.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.495+0000 7f6d37268640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T03:37:26.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.495+0000 7f6d37268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.495+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f6d3009ee10 con 0x7f6d3009c090 2024-09-15T03:37:26.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.496+0000 7f6d257fa640 1 -- 172.21.15.181:0/3049902723 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v231) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6d20042030 con 0x7f6d3009c090 2024-09-15T03:37:26.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.496+0000 7f6d257fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-15T03:37:26.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.496+0000 7f6d257fa640 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-15T03:37:26.498 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h15' does not appear in the crush map 2024-09-15T03:37:26.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d14038740 msgr2=0x7f6d1403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d14038740 0x7f6d1403abf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00bb40 tx=0x7f6d2c008040 comp rx=0 tx=0).stop 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 10 monclient: shutdown 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 msgr2=0x7f6d30094880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d30094880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6d20035040 tx=0x7f6d20032000 comp rx=0 tx=0).stop 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 shutdown_connections 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6d14038740 0x7f6d1403abf0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d3009c9a0 0x7f6d30128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d3009c090 0x7f6d30094880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 --2- 172.21.15.181:0/3049902723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d300677e0 0x7f6d30094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 >> 172.21.15.181:0/3049902723 conn(0x7f6d300a16a0 msgr2=0x7f6d300a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:26.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 shutdown_connections 2024-09-15T03:37:26.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.498+0000 7f6d37268640 1 -- 172.21.15.181:0/3049902723 wait complete. 2024-09-15T03:37:26.509 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:26.509 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h16 2024-09-15T03:37:26.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.580+0000 7fe7556ef640 10 monclient: get_monmap_and_config 2024-09-15T03:37:26.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.580+0000 7fe7556ef640 10 monclient: build_initial_monmap 2024-09-15T03:37:26.582 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.580+0000 7fe7556ef640 10 monclient: monmap: 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:26.582597+0000 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:26.582597+0000 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:26.583 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:26.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.581+0000 7fe7556ef640 1 Processor -- start 2024-09-15T03:37:26.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.581+0000 7fe7556ef640 1 -- start start 2024-09-15T03:37:26.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.581+0000 7fe7556ef640 10 monclient: init 2024-09-15T03:37:26.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.581+0000 7fe7556ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.581+0000 7fe7556ef640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.581+0000 7fe7556ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe750069bd0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe750098330 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe750137d60 0x7fe75013a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe750137d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe750096bd0 con 0x7fe750098330 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7501327b0 con 0x7fe750069bd0 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe750131db0 con 0x7fe750137d60 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe7556ef640 10 monclient(hunting): authenticate will time out at 1251.064114s 2024-09-15T03:37:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74effd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe750069bd0 0x7fe7501320a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55380/0 (socket says 172.21.15.181:55380) 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe750137d60 0x7fe75013a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74effd640 1 -- 172.21.15.181:0/1081079558 learned_addr learned my addr 172.21.15.181:0/1081079558 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request con 0x7fe750137d60 auth_method 0 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_done global_id 104551 payload 274 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 1 -- 172.21.15.181:0/1081079558 >> v1:172.21.15.181:6789/0 conn(0x7fe750069bd0 legacy=0x7fe7501320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 1 -- 172.21.15.181:0/1081079558 >> v1:172.21.15.153:6789/0 conn(0x7fe750098330 legacy=0x7fe750098770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:26.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:26.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 1 monclient: found mon.noname-c 2024-09-15T03:37:26.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:26.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.582+0000 7fe74f7fe640 1 -- 172.21.15.181:0/1081079558 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe750098e80 con 0x7fe750137d60 2024-09-15T03:37:26.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74f7fe640 1 -- 172.21.15.181:0/1081079558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750137d60 msgr2=0x7fe75013a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:26.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74f7fe640 1 --2- 172.21.15.181:0/1081079558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750137d60 0x7fe75013a140 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe748009e60 tx=0x7fe748009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4221b5e25e2ed6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:26.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 1 -- 172.21.15.181:0/1081079558 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748007630 con 0x7fe750137d60 2024-09-15T03:37:26.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: dump: 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:26.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 1 -- 172.21.15.181:0/1081079558 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74802d050 con 0x7fe750137d60 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 5 monclient: authenticate success, global_id 104551 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 1 -- 172.21.15.181:0/1081079558 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7480079c0 con 0x7fe750137d60 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe74e7fc640 10 monclient: dump: 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.597 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 4 set_mon_vals no callback set 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 10 monclient: shutdown 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 1 -- 172.21.15.181:0/1081079558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750137d60 msgr2=0x7fe75013a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 1 --2- 172.21.15.181:0/1081079558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750137d60 0x7fe75013a140 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe748009e60 tx=0x7fe748009780 comp rx=0 tx=0).stop 2024-09-15T03:37:26.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 1 -- 172.21.15.181:0/1081079558 shutdown_connections 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.585+0000 7fe7556ef640 1 --2- 172.21.15.181:0/1081079558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750137d60 0x7fe75013a140 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 1 -- 172.21.15.181:0/1081079558 >> 172.21.15.181:0/1081079558 conn(0x7fe75009e9f0 msgr2=0x7fe75009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 1 -- 172.21.15.181:0/1081079558 shutdown_connections 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 1 -- 172.21.15.181:0/1081079558 wait complete. 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 10 monclient: build_initial_monmap 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 10 monclient: monmap: 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:26.599 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 1 Processor -- start 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 1 -- start start 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.586+0000 7fe7556ef640 10 monclient: init 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 0x7fe750130770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe750069bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 0x7fe750134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe750098330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe750137d60 0x7fe750134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): picked mon.noname-a con 0x7fe750137d60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:26.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe750135a70 con 0x7fe750137d60 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe750118d00 con 0x7fe750098330 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe750118a20 con 0x7fe750069bd0 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe7556ef640 10 monclient(hunting): authenticate will time out at 1251.069245s 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 0x7fe750134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 0x7fe750134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45484/0 (socket says 172.21.15.181:45484) 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 -- 172.21.15.181:0/397286386 learned_addr learned my addr 172.21.15.181:0/397286386 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): get_auth_request con 0x7fe750098330 auth_method 0 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe750137d60 0x7fe750134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request con 0x7fe750137d60 auth_method 0 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 0x7fe750130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 10 monclient(hunting): get_auth_request con 0x7fe750069bd0 auth_method 0 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:26.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient(hunting): handle_auth_done global_id 104554 payload 824 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 msgr2=0x7fe750130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 msgr2=0x7fe750130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 msgr2=0x7fe750130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 0x7fe750130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe74effd640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 0x7fe750130770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 0x7fe750130770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe750137d60 msgr2=0x7fe750134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe750137d60 0x7fe750134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 monclient: found mon.noname-b 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.587+0000 7fe747fff640 1 -- 172.21.15.181:0/397286386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748009a20 con 0x7fe750098330 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.589+0000 7fe747fff640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 msgr2=0x7fe750134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.589+0000 7fe747fff640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 0x7fe750134560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe73800c5e0 tx=0x7fe738007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.589+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738012840 con 0x7fe750098330 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.589+0000 7fe746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.589+0000 7fe746ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.589+0000 7fe746ffd640 10 monclient: dump: 2024-09-15T03:37:26.604 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 5 monclient: authenticate success, global_id 104554 2024-09-15T03:37:26.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7380129a0 con 0x7fe750098330 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 monclient: _renew_subs 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe750136030 con 0x7fe750098330 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738012da0 con 0x7fe750098330 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 10 monclient: dump: 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:26.606 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 monclient: _renew_subs 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe74dffb640 4 set_mon_vals no callback set 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe750136da0 con 0x7fe750098330 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe738011590 con 0x7fe750098330 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.590+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe750118f40 con 0x7fe750098330 2024-09-15T03:37:26.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.591+0000 7fe746ffd640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe730038740 0x7fe73003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.591+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(231..231 src has 199..231) v4 ==== 11740+0+0 (secure 0 0 0) 0x7fe73804d6f0 con 0x7fe750098330 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.591+0000 7fe74effd640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe730038740 0x7fe73003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.591+0000 7fe74effd640 10 monclient: get_auth_request con 0x7fe730038740 auth_method 0 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.593+0000 7fe74effd640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe730038740 0x7fe73003abf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe740007d10 tx=0x7fe740009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.595+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe738010aa0 con 0x7fe750098330 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.595+0000 7fe746ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:26.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.595+0000 7fe746ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:26.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.611+0000 7faaff28b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:26.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.611+0000 7faaff28b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:26.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.611+0000 7faaff28b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:26.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.611+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faaf81183f0 con 0x7faaf8096c60 2024-09-15T03:37:26.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.612+0000 7faae97fa640 1 -- 172.21.15.153:0/3468584904 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7faaf81183f0 con 0x7faaf8096c60 2024-09-15T03:37:26.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.612+0000 7faae97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:26.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.612+0000 7faae97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:26.613 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:26.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faadc038740 msgr2=0x7faadc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faadc038740 0x7faadc03abf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7faaf809b250 tx=0x7faaf4007f00 comp rx=0 tx=0).stop 2024-09-15T03:37:26.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 10 monclient: shutdown 2024-09-15T03:37:26.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:26.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 msgr2=0x7faaf812dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:26.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 0x7faaf812dcf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faaec00c530 tx=0x7faaec007e40 comp rx=0 tx=0).stop 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 shutdown_connections 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faadc038740 0x7faadc03abf0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaf8134b50 0x7faaf806d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf8097030 0x7faaf812e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 --2- 172.21.15.153:0/3468584904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf8096c60 0x7faaf812dcf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 >> 172.21.15.153:0/3468584904 conn(0x7faaf80672b0 msgr2=0x7faaf81318a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 shutdown_connections 2024-09-15T03:37:26.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:26.614+0000 7faaff28b640 1 -- 172.21.15.153:0/3468584904 wait complete. 2024-09-15T03:37:26.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.755+0000 7fe7556ef640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T03:37:26.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.755+0000 7fe7556ef640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T03:37:26.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.755+0000 7fe7556ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:26.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:26.755+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7fe7500008d0 con 0x7fe750098330 2024-09-15T03:37:27.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.406+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v232) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe738016070 con 0x7fe750098330 2024-09-15T03:37:27.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.406+0000 7fe746ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T03:37:27.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.406+0000 7fe746ffd640 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-15T03:37:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.505+0000 7fe7556ef640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T03:37:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.505+0000 7fe7556ef640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T03:37:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.505+0000 7fe7556ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.505+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7fe75009ef10 con 0x7fe750098330 2024-09-15T03:37:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.506+0000 7fe746ffd640 1 -- 172.21.15.181:0/397286386 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v232) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe7380102e0 con 0x7fe750098330 2024-09-15T03:37:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.506+0000 7fe746ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-15T03:37:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.506+0000 7fe746ffd640 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-15T03:37:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h16' does not appear in the crush map 2024-09-15T03:37:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe730038740 msgr2=0x7fe73003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe730038740 0x7fe73003abf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe740007d10 tx=0x7fe740009040 comp rx=0 tx=0).stop 2024-09-15T03:37:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 10 monclient: shutdown 2024-09-15T03:37:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 msgr2=0x7fe750134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 0x7fe750134560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe73800c5e0 tx=0x7fe738007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 shutdown_connections 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe730038740 0x7fe73003abf0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe750137d60 0x7fe750134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe750098330 0x7fe750134560 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 --2- 172.21.15.181:0/397286386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe750069bd0 0x7fe750130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 >> 172.21.15.181:0/397286386 conn(0x7fe75009e9f0 msgr2=0x7fe750130fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 shutdown_connections 2024-09-15T03:37:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.508+0000 7fe7556ef640 1 -- 172.21.15.181:0/397286386 wait complete. 2024-09-15T03:37:27.519 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:27.519 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h17 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.590+0000 7f15a269f640 10 monclient: get_monmap_and_config 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.590+0000 7f15a269f640 10 monclient: build_initial_monmap 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.590+0000 7f15a269f640 10 monclient: monmap: 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:27.592042+0000 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:27.592042+0000 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:27.592 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:27.593 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.590+0000 7f15a269f640 1 Processor -- start 2024-09-15T03:37:27.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 1 -- start start 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient: init 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f159c06e000 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f159c06a710 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f159c09e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient(hunting): picked mon.noname-b con 0x7f159c06e3d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c0699c0 con 0x7f159c06e000 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c06be20 con 0x7f159c06a710 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c06a360 con 0x7f159c06e3d0 2024-09-15T03:37:27.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a269f640 10 monclient(hunting): authenticate will time out at 1252.073545s 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f159b7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f159c06a710 0x7f159c09e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55384/0 (socket says 172.21.15.181:55384) 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f159b7fe640 1 -- 172.21.15.181:0/3171787615 learned_addr learned my addr 172.21.15.181:0/3171787615 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.591+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f159c09e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 10 monclient(hunting): get_auth_request con 0x7f159c06e3d0 auth_method 0 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 msgr2=0x7f159c09e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 msgr2=0x7f159c09e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f159c09e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:27.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.592+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f159c09e880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.593+0000 7f159b7fe640 1 -- 172.21.15.181:0/3171787615 >> v1:172.21.15.181:6789/0 conn(0x7f159c06a710 legacy=0x7f159c09e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203717282 0 0) 0x7f159c06be20 con 0x7f159c06a710 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): my global_id is 104557 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 1 -- 172.21.15.181:0/3171787615 >> v1:172.21.15.153:6789/0 conn(0x7f159c06e000 legacy=0x7f159c09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15880034e0 con 0x7f159c06a710 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252467490 0 0) 0x7f159c0699c0 con 0x7f159c06e000 2024-09-15T03:37:27.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): my global_id is 114357 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c06be20 con 0x7f159c06e000 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721054376 0 0) 0x7f15880034e0 con 0x7f159c06a710 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 msgr2=0x7f159c09e880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f159c09e880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 >> v1:172.21.15.153:6789/0 conn(0x7f159c06e000 legacy=0x7f159c09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:37:27.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c069f80 con 0x7f159c06a710 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: _finish_auth 0 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f1584001720 con 0x7f159c06a710 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a269f640 5 monclient: authenticate success, global_id 104557 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a269f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: dump: 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 >> v1:172.21.15.181:6789/0 conn(0x7f159c06a710 legacy=0x7f159c09e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f1588003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): picked mon.c con 0x7f159c06e3d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15880041b0 0x7f1588004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): picked mon.a con 0x7f15880041b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1588004d90 0x7f1588005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): picked mon.b con 0x7f1588004d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c06a360 con 0x7f15880041b0 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158800bd20 con 0x7f1588004d90 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158800bee0 con 0x7f159c06e3d0 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a269f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1588004d90 0x7f1588005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): get_auth_request con 0x7f1588004d90 auth_method 0 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15880041b0 0x7f1588004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f1588003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): get_auth_request con 0x7f159c06e3d0 auth_method 0 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:27.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): get_auth_request con 0x7f15880041b0 auth_method 0 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_done global_id 104557 payload 293 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 msgr2=0x7f1588003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:27.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c06e3d0 0x7f1588003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15880041b0 msgr2=0x7f1588004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15880041b0 0x7f1588004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient: _finish_hunting 0 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 monclient: found mon.b 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15880034e0 con 0x7f1588004d90 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 10 monclient: _finish_auth 0 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.594+0000 7f15a0c3c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1588004d90 msgr2=0x7f1588005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1588004d90 0x7f1588005200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f15940073c0 tx=0x7f15940096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84ef94440621fb8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159402f410 con 0x7f1588004d90 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:27.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: dump: 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f15a269f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159402f5b0 con 0x7f1588004d90 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 1 -- 172.21.15.181:0/3171787615 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159402f970 con 0x7f1588004d90 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f15a269f640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f15a269f640 4 set_mon_vals no callback set 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.597+0000 7f159affd640 10 monclient: dump: 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:27.615 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 10 monclient: shutdown 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 1 -- 172.21.15.181:0/3171787615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1588004d90 msgr2=0x7f1588005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 1 --2- 172.21.15.181:0/3171787615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1588004d90 0x7f1588005200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f15940073c0 tx=0x7f15940096f0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3171787615 reap_dead start 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 1 -- 172.21.15.181:0/3171787615 shutdown_connections 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 1 -- 172.21.15.181:0/3171787615 >> 172.21.15.181:0/3171787615 conn(0x7f159c071340 msgr2=0x7f159c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 1 -- 172.21.15.181:0/3171787615 shutdown_connections 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 1 -- 172.21.15.181:0/3171787615 wait complete. 2024-09-15T03:37:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 10 monclient: build_initial_monmap 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.598+0000 7f15a269f640 10 monclient: monmap: 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 Processor -- start 2024-09-15T03:37:27.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 -- start start 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient: init 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1588004d90 0x7f159c13d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): picked mon.noname-a con 0x7f1588004d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 0x7f159c13cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): picked mon.noname-c con 0x7f159c11fb30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 0x7f159c140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): picked mon.noname-b con 0x7f159c120f70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c06e970 con 0x7f1588004d90 2024-09-15T03:37:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c06cff0 con 0x7f159c120f70 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c13e310 con 0x7f159c11fb30 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a269f640 10 monclient(hunting): authenticate will time out at 1252.081774s 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 0x7f159c140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 0x7f159c140c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:45498/0 (socket says 172.21.15.181:45498) 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3075650080 learned_addr learned my addr 172.21.15.181:0/3075650080 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): get_auth_request con 0x7f159c120f70 auth_method 0 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f159bfff640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1588004d90 0x7f159c13d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f159bfff640 10 monclient(hunting): get_auth_request con 0x7f1588004d90 auth_method 0 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f159bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f159bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f159bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f159b7fe640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 0x7f159c13cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.599+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 10 monclient(hunting): get_auth_request con 0x7f159c11fb30 auth_method 0 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:27.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 msgr2=0x7f159c13cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 msgr2=0x7f159c13cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 0x7f159c13cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 10 monclient(hunting): handle_auth_done global_id 104560 payload 824 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f159b7fe640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 0x7f159c13cdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 msgr2=0x7f159c13cdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 0x7f159c13cdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1588004d90 msgr2=0x7f159c13d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1588004d90 0x7f159c13d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 10 monclient: _finish_hunting 0 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 1 monclient: found mon.noname-b 2024-09-15T03:37:27.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.600+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3075650080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c121e90 con 0x7f159c120f70 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a0c3c640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 msgr2=0x7f159c140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a0c3c640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 0x7f159c140c90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f15940073c0 tx=0x7f1594036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594038970 con 0x7f159c120f70 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: dump: 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:27.622 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 5 monclient: authenticate success, global_id 104560 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159403e040 con 0x7f159c120f70 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 10 monclient: _renew_subs 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c122170 con 0x7f159c120f70 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594038d20 con 0x7f159c120f70 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f1593fff640 10 monclient: dump: 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f159a7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 10 monclient: _renew_subs 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.602+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c13d4d0 con 0x7f159c120f70 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f15940425b0 con 0x7f159c120f70 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f15a269f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f15a269f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f15a269f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c06cff0 con 0x7f159c120f70 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f1593fff640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f157c038740 0x7f157c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(232..232 src has 199..232) v4 ==== 11527+0+0 (secure 0 0 0) 0x7f159406d9d0 con 0x7f159c120f70 2024-09-15T03:37:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f159bfff640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f157c038740 0x7f157c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.603+0000 7f159bfff640 10 monclient: get_auth_request con 0x7f157c038740 auth_method 0 2024-09-15T03:37:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.606+0000 7f159bfff640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f157c038740 0x7f157c03abf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f158c001470 tx=0x7f158c001b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.607+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f159402cb00 con 0x7f159c120f70 2024-09-15T03:37:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.607+0000 7f1593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.607+0000 7f1593fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:27.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.765+0000 7f15a269f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T03:37:27.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.766+0000 7f15a269f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T03:37:27.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.766+0000 7f15a269f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:27.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:27.766+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f159c1224e0 con 0x7f159c120f70 2024-09-15T03:37:28.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.411+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v233) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1594048030 con 0x7f159c120f70 2024-09-15T03:37:28.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.411+0000 7f1593fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T03:37:28.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.411+0000 7f1593fff640 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-15T03:37:28.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.509+0000 7f15a269f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T03:37:28.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.509+0000 7f15a269f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T03:37:28.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.509+0000 7f15a269f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:28.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.509+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f159c0008d0 con 0x7f159c120f70 2024-09-15T03:37:28.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.780+0000 7f1593fff640 1 -- 172.21.15.181:0/3075650080 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v233) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1594048020 con 0x7f159c120f70 2024-09-15T03:37:28.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.780+0000 7f1593fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-15T03:37:28.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.780+0000 7f1593fff640 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-15T03:37:28.782 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h17' does not appear in the crush map 2024-09-15T03:37:28.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f157c038740 msgr2=0x7f157c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:28.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f157c038740 0x7f157c03abf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f158c001470 tx=0x7f158c001b50 comp rx=0 tx=0).stop 2024-09-15T03:37:28.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 10 monclient: shutdown 2024-09-15T03:37:28.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 msgr2=0x7f159c140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 0x7f159c140c90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f15940073c0 tx=0x7f1594036000 comp rx=0 tx=0).stop 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 shutdown_connections 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f157c038740 0x7f157c03abf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f159c120f70 0x7f159c140c90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f159c11fb30 0x7f159c13cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 --2- 172.21.15.181:0/3075650080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1588004d90 0x7f159c13d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 >> 172.21.15.181:0/3075650080 conn(0x7f159c071340 msgr2=0x7f159c0a1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 shutdown_connections 2024-09-15T03:37:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.782+0000 7f15a269f640 1 -- 172.21.15.181:0/3075650080 wait complete. 2024-09-15T03:37:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h18 2024-09-15T03:37:28.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.865+0000 7fbcfa4c9640 10 monclient: get_monmap_and_config 2024-09-15T03:37:28.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.865+0000 7fbcfa4c9640 10 monclient: build_initial_monmap 2024-09-15T03:37:28.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.865+0000 7fbcfa4c9640 10 monclient: monmap: 2024-09-15T03:37:28.867 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:28.867377+0000 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:28.867377+0000 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:28.869 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 Processor -- start 2024-09-15T03:37:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 -- start start 2024-09-15T03:37:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient: init 2024-09-15T03:37:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:37:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4099130 0x7fbcf4099500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf4099130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbcf406a790 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbcf409e170 0x7fbcf409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf409e170 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf4069770 con 0x7fbcf406a790 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf409a080 con 0x7fbcf409e170 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf406a040 con 0x7fbcf4099130 2024-09-15T03:37:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcfa4c9640 10 monclient(hunting): authenticate will time out at 1253.348821s 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf8a66640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbcf409e170 0x7fbcf409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf37fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fbcf406a790 0x7fbcf409da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:44114/0 (socket says 172.21.15.181:44114) 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf37fe640 1 -- 172.21.15.181:0/711629710 learned_addr learned my addr 172.21.15.181:0/711629710 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf8a66640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbcf409e170 0x7fbcf409e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36224/0 (socket says 172.21.15.181:36224) 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf8a66640 10 monclient(hunting): get_auth_request con 0x7fbcf409e170 auth_method 0 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf8a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf8a66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.866+0000 7fbcf8a66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf3fff640 1 --2- 172.21.15.181:0/711629710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4099130 0x7fbcf4099500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf3fff640 10 monclient(hunting): get_auth_request con 0x7fbcf4099130 auth_method 0 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_done global_id 104563 payload 274 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 1 -- 172.21.15.181:0/711629710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4099130 msgr2=0x7fbcf4099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 1 --2- 172.21.15.181:0/711629710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4099130 0x7fbcf4099500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 1 -- 172.21.15.181:0/711629710 >> v1:172.21.15.181:6789/0 conn(0x7fbcf406a790 legacy=0x7fbcf409da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 10 monclient: _finish_hunting 0 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 1 monclient: found mon.noname-c 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.867+0000 7fbcf8a66640 1 -- 172.21.15.181:0/711629710 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf409f5d0 con 0x7fbcf409e170 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.869+0000 7fbcf8a66640 1 -- 172.21.15.181:0/711629710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 msgr2=0x7fbcf409e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.869+0000 7fbcf8a66640 1 --2- 172.21.15.181:0/711629710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf409e5d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009e90 tx=0x7fbcec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1c4eb96c77d7440 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 1 -- 172.21.15.181:0/711629710 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec007760 con 0x7fbcf409e170 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: dump: 2024-09-15T03:37:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 5 monclient: authenticate success, global_id 104563 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 1 -- 172.21.15.181:0/711629710 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec02d050 con 0x7fbcf409e170 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 4 set_mon_vals no callback set 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 1 -- 172.21.15.181:0/711629710 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec007af0 con 0x7fbcf409e170 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcf2ffd640 10 monclient: dump: 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:28.884 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 10 monclient: shutdown 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:28.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/711629710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 msgr2=0x7fbcf409e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/711629710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf409e5d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009e90 tx=0x7fbcec009780 comp rx=0 tx=0).stop 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/711629710 shutdown_connections 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/711629710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf409e5d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/711629710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4099130 0x7fbcf4099500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/711629710 >> 172.21.15.181:0/711629710 conn(0x7fbcf4071340 msgr2=0x7fbcf40a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/711629710 shutdown_connections 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.870+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/711629710 wait complete. 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.871+0000 7fbcfa4c9640 10 monclient: build_initial_monmap 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.871+0000 7fbcfa4c9640 10 monclient: monmap: 2024-09-15T03:37:28.886 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.871+0000 7fbcfa4c9640 1 Processor -- start 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.871+0000 7fbcfa4c9640 1 -- start start 2024-09-15T03:37:28.887 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.871+0000 7fbcfa4c9640 10 monclient: init 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 0x7fbcf413d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf406a790 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 0x7fbcf413ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf4099130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf4140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf409e170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf4122770 con 0x7fbcf4099130 2024-09-15T03:37:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf41221b0 con 0x7fbcf409e170 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf411fc40 con 0x7fbcf406a790 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcfa4c9640 10 monclient(hunting): authenticate will time out at 1253.354128s 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf4140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf4140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36236/0 (socket says 172.21.15.181:36236) 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 -- 172.21.15.181:0/3417910632 learned_addr learned my addr 172.21.15.181:0/3417910632 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): get_auth_request con 0x7fbcf409e170 auth_method 0 2024-09-15T03:37:28.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:28.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:28.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:28.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:28.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:28.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:28.890 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 0x7fbcf413d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:28.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 0x7fbcf413ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:28.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 10 monclient(hunting): get_auth_request con 0x7fbcf406a790 auth_method 0 2024-09-15T03:37:28.891 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 10 monclient(hunting): get_auth_request con 0x7fbcf4099130 auth_method 0 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient(hunting): handle_auth_done global_id 104566 payload 824 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 msgr2=0x7fbcf413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 msgr2=0x7fbcf413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 0x7fbcf413d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 msgr2=0x7fbcf413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf3fff640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 0x7fbcf413d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 0x7fbcf413d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 msgr2=0x7fbcf413ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 msgr2=0x7fbcf413ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 msgr2=0x7fbcf413ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 0x7fbcf413ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf37fe640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 0x7fbcf413ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 0x7fbcf413ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:28.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient: _finish_hunting 0 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 monclient: found mon.noname-b 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.872+0000 7fbcf8a66640 1 -- 172.21.15.181:0/3417910632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec009a20 con 0x7fbcf409e170 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcf8a66640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 msgr2=0x7fbcf4140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcf8a66640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf4140cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009e90 tx=0x7fbcec037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec031ac0 con 0x7fbcf409e170 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: dump: 2024-09-15T03:37:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: _finish_auth 0 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec040070 con 0x7fbcf409e170 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec031e70 con 0x7fbcf409e170 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcfa4c9640 5 monclient: authenticate success, global_id 104566 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcdbfff640 10 monclient: dump: 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcfa4c9640 10 monclient: _renew_subs 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcfa4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf4121ed0 con 0x7fbcf409e170 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.874+0000 7fbcf27fc640 4 set_mon_vals no callback set 2024-09-15T03:37:28.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 10 monclient: _renew_subs 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf413d510 con 0x7fbcf409e170 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbcec042030 con 0x7fbcf409e170 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcdbfff640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbce0038740 0x7fbce003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.875+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf41225b0 con 0x7fbcf409e170 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.876+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(233..233 src has 199..233) v4 ==== 11314+0+0 (secure 0 0 0) 0x7fbcec06ef60 con 0x7fbcf409e170 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.876+0000 7fbcf3fff640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbce0038740 0x7fbce003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.876+0000 7fbcf3fff640 10 monclient: get_auth_request con 0x7fbce0038740 auth_method 0 2024-09-15T03:37:28.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.878+0000 7fbcf3fff640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbce0038740 0x7fbce003abf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbce4006fd0 tx=0x7fbce4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:28.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.880+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbcec03c420 con 0x7fbcf409e170 2024-09-15T03:37:28.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.880+0000 7fbcdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:28.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:28.880+0000 7fbcdbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:29.040 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.038+0000 7fbcfa4c9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T03:37:29.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.038+0000 7fbcfa4c9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T03:37:29.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.038+0000 7fbcfa4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.041 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.038+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7fbcf40008d0 con 0x7fbcf409e170 2024-09-15T03:37:29.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.420+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v234) v1 ==== 120+0+0 (secure 0 0 0) 0x7fbcec0303a0 con 0x7fbcf409e170 2024-09-15T03:37:29.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.420+0000 7fbcdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T03:37:29.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.420+0000 7fbcdbfff640 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-15T03:37:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.517+0000 7fbcfa4c9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T03:37:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.517+0000 7fbcfa4c9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T03:37:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.517+0000 7fbcfa4c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.517+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7fbcf40008d0 con 0x7fbcf409e170 2024-09-15T03:37:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.518+0000 7fbcdbfff640 1 -- 172.21.15.181:0/3417910632 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v234) v1 ==== 120+0+0 (secure 0 0 0) 0x7fbcec030580 con 0x7fbcf409e170 2024-09-15T03:37:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.518+0000 7fbcdbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-15T03:37:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.518+0000 7fbcdbfff640 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-15T03:37:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h18' does not appear in the crush map 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbce0038740 msgr2=0x7fbce003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbce0038740 0x7fbce003abf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbce4006fd0 tx=0x7fbce4008040 comp rx=0 tx=0).stop 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 10 monclient: shutdown 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 msgr2=0x7fbcf4140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf4140cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009e90 tx=0x7fbcec037000 comp rx=0 tx=0).stop 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 shutdown_connections 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbce0038740 0x7fbce003abf0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcf409e170 0x7fbcf4140cf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf4099130 0x7fbcf413ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 --2- 172.21.15.181:0/3417910632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf406a790 0x7fbcf413d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 >> 172.21.15.181:0/3417910632 conn(0x7fbcf4071340 msgr2=0x7fbcf406d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 shutdown_connections 2024-09-15T03:37:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.520+0000 7fbcfa4c9640 1 -- 172.21.15.181:0/3417910632 wait complete. 2024-09-15T03:37:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h19 2024-09-15T03:37:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.602+0000 7f100a554640 10 monclient: get_monmap_and_config 2024-09-15T03:37:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.602+0000 7f100a554640 10 monclient: build_initial_monmap 2024-09-15T03:37:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.602+0000 7f100a554640 10 monclient: monmap: 2024-09-15T03:37:29.604 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:29.604359+0000 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:29.604359+0000 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:29.605 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 Processor -- start 2024-09-15T03:37:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 -- start start 2024-09-15T03:37:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient: init 2024-09-15T03:37:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:37:29.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f100406e000 0x7f100406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): picked mon.noname-a con 0x7f100406e000 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f100406a710 0x7f100409da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): picked mon.noname-c con 0x7f100406a710 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100409dfa0 0x7f100409e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): picked mon.noname-b con 0x7f100409dfa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100406a360 con 0x7f100406e000 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10040699c0 con 0x7f100406a710 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100406be20 con 0x7f100409dfa0 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:29.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f100a554640 10 monclient(hunting): authenticate will time out at 1254.085776s 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f100406a710 0x7f100409da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f100406a710 0x7f100409da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36250/0 (socket says 172.21.15.181:36250) 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 1 -- 172.21.15.181:0/3564645710 learned_addr learned my addr 172.21.15.181:0/3564645710 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 10 monclient(hunting): get_auth_request con 0x7f100406a710 auth_method 0 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.603+0000 7f10037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1003fff640 1 --2- 172.21.15.181:0/3564645710 >> v2:172.21.15.153:3300/0 conn(0x7f100406e000 0x7f100406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1008af1640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100409dfa0 0x7f100409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1003fff640 10 monclient(hunting): get_auth_request con 0x7f100406e000 auth_method 0 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1003fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1003fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 10 monclient(hunting): handle_auth_done global_id 104569 payload 274 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 1 -- 172.21.15.181:0/3564645710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100409dfa0 msgr2=0x7f100409e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100409dfa0 0x7f100409e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 1 -- 172.21.15.181:0/3564645710 >> v2:172.21.15.153:3300/0 conn(0x7f100406e000 msgr2=0x7f100406e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 1 --2- 172.21.15.181:0/3564645710 >> v2:172.21.15.153:3300/0 conn(0x7f100406e000 0x7f100406e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 1 monclient: found mon.noname-c 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f10037fe640 1 -- 172.21.15.181:0/3564645710 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004069f80 con 0x7f100406a710 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1008af1640 10 monclient: get_auth_request con 0x7f100409dfa0 auth_method 0 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.604+0000 7f1008af1640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100409dfa0 0x7f100409e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f10037fe640 1 -- 172.21.15.181:0/3564645710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 msgr2=0x7f100409da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f10037fe640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f100409da60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1004067100 tx=0x7f0ff80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e3b536658eb250 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 1 -- 172.21.15.181:0/3564645710 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8032440 con 0x7f100406a710 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: dump: 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 1 -- 172.21.15.181:0/3564645710 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff80325a0 con 0x7f100406a710 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 5 monclient: authenticate success, global_id 104569 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 1 -- 172.21.15.181:0/3564645710 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8032980 con 0x7f100406a710 2024-09-15T03:37:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f1002ffd640 10 monclient: dump: 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 4 set_mon_vals no callback set 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 10 monclient: shutdown 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 -- 172.21.15.181:0/3564645710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 msgr2=0x7f100409da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f100409da60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1004067100 tx=0x7f0ff80091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 -- 172.21.15.181:0/3564645710 shutdown_connections 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100409dfa0 0x7f100409e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 --2- 172.21.15.181:0/3564645710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f100409da60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 --2- 172.21.15.181:0/3564645710 >> v2:172.21.15.153:3300/0 conn(0x7f100406e000 0x7f100406e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 -- 172.21.15.181:0/3564645710 >> 172.21.15.181:0/3564645710 conn(0x7f1004071340 msgr2=0x7f10040a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 -- 172.21.15.181:0/3564645710 shutdown_connections 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.607+0000 7f100a554640 1 -- 172.21.15.181:0/3564645710 wait complete. 2024-09-15T03:37:29.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.608+0000 7f100a554640 10 monclient: build_initial_monmap 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.608+0000 7f100a554640 10 monclient: monmap: 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.608+0000 7f100a554640 1 Processor -- start 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.608+0000 7f100a554640 1 -- start start 2024-09-15T03:37:29.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.608+0000 7f100a554640 10 monclient: init 2024-09-15T03:37:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f10041411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): picked mon.noname-b con 0x7f100406a710 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 0x7f1004133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): picked mon.noname-c con 0x7f100406e000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f100409dfa0 0x7f100411dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): picked mon.noname-a con 0x7f100409dfa0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1004121660 con 0x7f100409dfa0 2024-09-15T03:37:29.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100406e9a0 con 0x7f100406a710 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100411f540 con 0x7f100406e000 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f100a554640 10 monclient(hunting): authenticate will time out at 1254.091470s 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f10041411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f10041411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36262/0 (socket says 172.21.15.181:36262) 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 -- 172.21.15.181:0/1941399162 learned_addr learned my addr 172.21.15.181:0/1941399162 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): get_auth_request con 0x7f100406a710 auth_method 0 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 0x7f1004133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 10 monclient(hunting): get_auth_request con 0x7f100406e000 auth_method 0 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1008af1640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f100409dfa0 0x7f100411dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1008af1640 10 monclient(hunting): get_auth_request con 0x7f100409dfa0 auth_method 0 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1008af1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1008af1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1008af1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient(hunting): handle_auth_done global_id 104572 payload 824 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 msgr2=0x7f1004133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 msgr2=0x7f1004133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 msgr2=0x7f1004133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 0x7f1004133ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f10037fe640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 0x7f1004133ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 0x7f1004133ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f100409dfa0 msgr2=0x7f100411dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f100409dfa0 0x7f100411dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 monclient: found mon.noname-b 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:29.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1003fff640 1 -- 172.21.15.181:0/1941399162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8009450 con 0x7f100406a710 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.609+0000 7f1008af1640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f100409dfa0 0x7f100411dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f1003fff640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 msgr2=0x7f10041411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f1003fff640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f10041411f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400d890 tx=0x7f0ff4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400de20 con 0x7f100406a710 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: dump: 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.629 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: _finish_auth 0 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 5 monclient: authenticate success, global_id 104572 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff400bae0 con 0x7f100406a710 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 10 monclient: _renew_subs 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10041200f0 con 0x7f100406a710 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:29.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400be80 con 0x7f100406a710 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f0febfff640 10 monclient: dump: 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:29.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 10 monclient: _renew_subs 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1004120fe0 con 0x7f100406a710 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.612+0000 7f10027fc640 4 set_mon_vals no callback set 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0ff4014640 con 0x7f100406a710 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f100a554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f100a554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f100a554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10041213e0 con 0x7f100406a710 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f0febfff640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ff0038740 0x7f0ff003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(234..234 src has 199..234) v4 ==== 11101+0+0 (secure 0 0 0) 0x7f0ff404da80 con 0x7f100406a710 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f10037fe640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ff0038740 0x7f0ff003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.613+0000 7f10037fe640 10 monclient: get_auth_request con 0x7f0ff0038740 auth_method 0 2024-09-15T03:37:29.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:29.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.615+0000 7f10037fe640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ff0038740 0x7f0ff003abf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8034000 tx=0x7f0ff8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:29.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.617+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0ff401eb40 con 0x7f100406a710 2024-09-15T03:37:29.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.617+0000 7f0febfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:29.634 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.617+0000 7f0febfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.736+0000 7fc4f315d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.736+0000 7fc4f315d640 10 monclient: build_initial_monmap 2024-09-15T03:37:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.736+0000 7fc4f315d640 10 monclient: monmap: 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:29.737517+0000 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:29.737517+0000 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:29.739 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.737+0000 7fc4f315d640 1 Processor -- start 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.737+0000 7fc4f315d640 1 -- start start 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.737+0000 7fc4f315d640 10 monclient: init 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc4ec0974f0 0x7fc4ec12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ec0974f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc4ec09a8a0 0x7fc4ec09ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ec09a8a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4ec12dd40 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec0978c0 con 0x7fc4ec12dd40 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec09b220 con 0x7fc4ec09a8a0 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec09ca20 con 0x7fc4ec0974f0 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f315d640 10 monclient(hunting): authenticate will time out at 1247.377236s 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc4ec09a8a0 0x7fc4ec09ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f16fa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc4ec12dd40 0x7fc4ec12e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34530/0 (socket says 172.21.15.153:34530) 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f16fa640 1 -- 172.21.15.153:0/270385256 learned_addr learned my addr 172.21.15.153:0/270385256 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 --2- 172.21.15.153:0/270385256 >> v2:172.21.15.153:3300/0 conn(0x7fc4ec09a8a0 0x7fc4ec09ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41252/0 (socket says 172.21.15.153:41252) 2024-09-15T03:37:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request con 0x7fc4ec09a8a0 auth_method 0 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 1 --2- 172.21.15.153:0/270385256 >> v2:172.21.15.181:3300/0 conn(0x7fc4ec0974f0 0x7fc4ec12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): get_auth_request con 0x7fc4ec0974f0 auth_method 0 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_done global_id 114369 payload 274 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 -- 172.21.15.153:0/270385256 >> v2:172.21.15.181:3300/0 conn(0x7fc4ec0974f0 msgr2=0x7fc4ec12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 --2- 172.21.15.153:0/270385256 >> v2:172.21.15.181:3300/0 conn(0x7fc4ec0974f0 0x7fc4ec12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 -- 172.21.15.153:0/270385256 >> v1:172.21.15.153:6789/0 conn(0x7fc4ec12dd40 legacy=0x7fc4ec12e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 monclient: found mon.noname-a 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.738+0000 7fc4ebfff640 1 -- 172.21.15.153:0/270385256 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec098ad0 con 0x7fc4ec09a8a0 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4ebfff640 1 -- 172.21.15.153:0/270385256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec09a8a0 msgr2=0x7fc4ec09ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4ebfff640 1 --2- 172.21.15.153:0/270385256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec09ace0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0768c0 tx=0x7fc4e00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e4435da076055d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 1 -- 172.21.15.153:0/270385256 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e002ae00 con 0x7fc4ec09a8a0 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: dump: 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 1 -- 172.21.15.153:0/270385256 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e0032440 con 0x7fc4ec09a8a0 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4f315d640 5 monclient: authenticate success, global_id 114369 2024-09-15T03:37:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4f315d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 1 -- 172.21.15.153:0/270385256 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0032820 con 0x7fc4ec09a8a0 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4eb7fe640 10 monclient: dump: 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4f315d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.741+0000 7fc4f315d640 4 set_mon_vals no callback set 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 10 monclient: shutdown 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 -- 172.21.15.153:0/270385256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec09a8a0 msgr2=0x7fc4ec09ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 --2- 172.21.15.153:0/270385256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec09ace0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0768c0 tx=0x7fc4e00090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 -- 172.21.15.153:0/270385256 shutdown_connections 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 --2- 172.21.15.153:0/270385256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec09ace0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 --2- 172.21.15.153:0/270385256 >> v2:172.21.15.181:3300/0 conn(0x7fc4ec0974f0 0x7fc4ec12ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 -- 172.21.15.153:0/270385256 >> 172.21.15.153:0/270385256 conn(0x7fc4ec0672b0 msgr2=0x7fc4ec0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 -- 172.21.15.153:0/270385256 shutdown_connections 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 1 -- 172.21.15.153:0/270385256 wait complete. 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 10 monclient: build_initial_monmap 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.742+0000 7fc4f315d640 10 monclient: monmap: 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 Processor -- start 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 -- start start 2024-09-15T03:37:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient: init 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 0x7fc4ec13f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ec0974f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec13f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4ec09a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 0x7fc4ec117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ec12dd40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec098300 con 0x7fc4ec0974f0 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec09e020 con 0x7fc4ec09a8a0 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec1197f0 con 0x7fc4ec12dd40 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f315d640 10 monclient(hunting): authenticate will time out at 1247.382913s 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 0x7fc4ec13f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 0x7fc4ec13f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41258/0 (socket says 172.21.15.153:41258) 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 1 -- 172.21.15.153:0/2523667027 learned_addr learned my addr 172.21.15.153:0/2523667027 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 10 monclient(hunting): get_auth_request con 0x7fc4ec0974f0 auth_method 0 2024-09-15T03:37:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.743+0000 7fc4f0ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 0x7fc4ec117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec13f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 10 monclient(hunting): get_auth_request con 0x7fc4ec12dd40 auth_method 0 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request con 0x7fc4ec09a8a0 auth_method 0 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 msgr2=0x7fc4ec117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 msgr2=0x7fc4ec117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 0x7fc4ec117480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 10 monclient(hunting): handle_auth_done global_id 114372 payload 824 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f16fa640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 0x7fc4ec117480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 msgr2=0x7fc4ec117480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 0x7fc4ec117480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4ec09a8a0 msgr2=0x7fc4ec13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec13f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 1 monclient: found mon.noname-a 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.744+0000 7fc4f0ef9640 1 -- 172.21.15.153:0/2523667027 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0009390 con 0x7fc4ec0974f0 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f0ef9640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 msgr2=0x7fc4ec13f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f0ef9640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 0x7fc4ec13f3e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00ea50 tx=0x7fc4dc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc009600 con 0x7fc4ec0974f0 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: dump: 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc00cc20 con 0x7fc4ec0974f0 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 5 monclient: authenticate success, global_id 114372 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 10 monclient: _renew_subs 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec118c00 con 0x7fc4ec0974f0 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc013470 con 0x7fc4ec0974f0 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4e8ff9640 10 monclient: dump: 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 10 monclient: _renew_subs 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec11b970 con 0x7fc4ec0974f0 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.746+0000 7fc4eaffd640 4 set_mon_vals no callback set 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc4dc013860 con 0x7fc4ec0974f0 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4f315d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4f315d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4f315d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec04bf60 con 0x7fc4ec0974f0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4e8ff9640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4d8038740 0x7fc4d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.747+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(234..234 src has 199..234) v4 ==== 11101+0+0 (secure 0 0 0) 0x7fc4dc04d6c0 con 0x7fc4ec0974f0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.748+0000 7fc4ebfff640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4d8038740 0x7fc4d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.748+0000 7fc4ebfff640 10 monclient: get_auth_request con 0x7fc4d8038740 auth_method 0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.750+0000 7fc4ebfff640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4d8038740 0x7fc4d803abf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0034000 tx=0x7fc4e0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.752+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc4dc023ad0 con 0x7fc4ec0974f0 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.752+0000 7fc4e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.752+0000 7fc4e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:29.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.776+0000 7f100a554640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T03:37:29.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.776+0000 7f100a554640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T03:37:29.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.776+0000 7f100a554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:29.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:29.776+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f100406d070 con 0x7f100406a710 2024-09-15T03:37:29.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.970+0000 7fc4f315d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.970+0000 7fc4f315d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.970+0000 7fc4f315d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.970+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc4ec11a8f0 con 0x7fc4ec0974f0 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.971+0000 7fc4e8ff9640 1 -- 172.21.15.153:0/2523667027 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fc4dc0232f0 con 0x7fc4ec0974f0 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.971+0000 7fc4e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.971+0000 7fc4e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:29.972 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4d8038740 msgr2=0x7fc4d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4d8038740 0x7fc4d803abf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0034000 tx=0x7fc4e0032000 comp rx=0 tx=0).stop 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 10 monclient: shutdown 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 msgr2=0x7fc4ec13f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 0x7fc4ec13f3e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00ea50 tx=0x7fc4dc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 shutdown_connections 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc4d8038740 0x7fc4d803abf0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4ec12dd40 0x7fc4ec117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4ec09a8a0 0x7fc4ec13f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 --2- 172.21.15.153:0/2523667027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4ec0974f0 0x7fc4ec13f3e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:29.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 >> 172.21.15.153:0/2523667027 conn(0x7fc4ec0672b0 msgr2=0x7fc4ec128880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:29.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 shutdown_connections 2024-09-15T03:37:29.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:29.973+0000 7fc4f315d640 1 -- 172.21.15.153:0/2523667027 wait complete. 2024-09-15T03:37:29.984 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:37:29.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:37:29.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:37:30.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.077+0000 7eff66b00640 10 monclient: get_monmap_and_config 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.077+0000 7eff66b00640 10 monclient: build_initial_monmap 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.078+0000 7eff66b00640 10 monclient: monmap: 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:30.078736+0000 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:30.078736+0000 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:30.079 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:30.080 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:30.080 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:30.080 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.078+0000 7eff66b00640 1 Processor -- start 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.078+0000 7eff66b00640 1 -- start start 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.078+0000 7eff66b00640 10 monclient: init 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff6009a510 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff600973b0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff600977b0 0x7eff601380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient(hunting): picked mon.noname-a con 0x7eff600977b0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6006ee10 con 0x7eff6009a510 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6009cb00 con 0x7eff600973b0 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6009a8e0 con 0x7eff600977b0 2024-09-15T03:37:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff66b00640 10 monclient(hunting): authenticate will time out at 1247.718273s 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff600977b0 0x7eff601380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6489c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7eff6009a510 0x7eff60096ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34534/0 (socket says 172.21.15.153:34534) 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6489c640 1 -- 172.21.15.153:0/4230665335 learned_addr learned my addr 172.21.15.153:0/4230665335 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): get_auth_request con 0x7eff600977b0 auth_method 0 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient(hunting): handle_auth_done global_id 114375 payload 274 2024-09-15T03:37:30.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 1 -- 172.21.15.153:0/4230665335 >> v1:172.21.15.181:6789/0 conn(0x7eff600973b0 legacy=0x7eff601349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 1 -- 172.21.15.153:0/4230665335 >> v1:172.21.15.153:6789/0 conn(0x7eff6009a510 legacy=0x7eff60096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 1 monclient: found mon.noname-a 2024-09-15T03:37:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.079+0000 7eff6509d640 1 -- 172.21.15.153:0/4230665335 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6009d590 con 0x7eff600977b0 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff6509d640 1 -- 172.21.15.153:0/4230665335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff600977b0 msgr2=0x7eff601380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff6509d640 1 --2- 172.21.15.153:0/4230665335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff600977b0 0x7eff601380d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7eff58009e60 tx=0x7eff58009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d9fbb692d69e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 1 -- 172.21.15.153:0/4230665335 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58007840 con 0x7eff600977b0 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: dump: 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff66b00640 5 monclient: authenticate success, global_id 114375 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff66b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 1 -- 172.21.15.153:0/4230665335 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5802d050 con 0x7eff600977b0 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff66b00640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 1 -- 172.21.15.153:0/4230665335 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58007bd0 con 0x7eff600977b0 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff66b00640 4 set_mon_vals no callback set 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.082+0000 7eff577fe640 10 monclient: dump: 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.092 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 10 monclient: shutdown 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 -- 172.21.15.153:0/4230665335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff600977b0 msgr2=0x7eff601380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 --2- 172.21.15.153:0/4230665335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff600977b0 0x7eff601380d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7eff58009e60 tx=0x7eff58009780 comp rx=0 tx=0).stop 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 -- 172.21.15.153:0/4230665335 shutdown_connections 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 --2- 172.21.15.153:0/4230665335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff600977b0 0x7eff601380d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 -- 172.21.15.153:0/4230665335 >> 172.21.15.153:0/4230665335 conn(0x7eff600672b0 msgr2=0x7eff600676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 -- 172.21.15.153:0/4230665335 shutdown_connections 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 1 -- 172.21.15.153:0/4230665335 wait complete. 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 10 monclient: build_initial_monmap 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.083+0000 7eff66b00640 10 monclient: monmap: 2024-09-15T03:37:30.093 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 Processor -- start 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 -- start start 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient: init 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:30.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 0x7eff6013ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): picked mon.noname-c con 0x7eff600973b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff600977b0 0x7eff60131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): picked mon.noname-b con 0x7eff600977b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 0x7eff6011b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): picked mon.noname-a con 0x7eff6009a510 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6009c370 con 0x7eff6009a510 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6011f170 con 0x7eff600977b0 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6013fc30 con 0x7eff600973b0 2024-09-15T03:37:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff66b00640 10 monclient(hunting): authenticate will time out at 1247.723715s 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 0x7eff6011b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 0x7eff6013ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 0x7eff6011b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41278/0 (socket says 172.21.15.153:41278) 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 0x7eff6013ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51810/0 (socket says 172.21.15.153:51810) 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 1 -- 172.21.15.153:0/992884840 learned_addr learned my addr 172.21.15.153:0/992884840 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): get_auth_request con 0x7eff6009a510 auth_method 0 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 10 monclient(hunting): get_auth_request con 0x7eff600973b0 auth_method 0 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 msgr2=0x7eff6013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 msgr2=0x7eff6013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 0x7eff6013ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6509d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.084+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 0x7eff6013ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff57fff640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff600977b0 0x7eff60131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff57fff640 10 monclient(hunting): get_auth_request con 0x7eff600977b0 auth_method 0 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 10 monclient(hunting): handle_auth_done global_id 114378 payload 824 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 msgr2=0x7eff6013ede0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff600973b0 0x7eff6013ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff600977b0 msgr2=0x7eff60131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff600977b0 0x7eff60131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 1 monclient: found mon.noname-a 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff6509d640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58009a20 con 0x7eff6009a510 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.085+0000 7eff57fff640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff600977b0 0x7eff60131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff6509d640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 msgr2=0x7eff6011b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff6509d640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 0x7eff6011b550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7eff580099f0 tx=0x7eff58037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58007ec0 con 0x7eff6009a510 2024-09-15T03:37:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: dump: 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58040070 con 0x7eff6009a510 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 5 monclient: authenticate success, global_id 114378 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58045430 con 0x7eff6009a510 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 10 monclient: _renew_subs 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6011dbc0 con 0x7eff6009a510 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff54ff9640 10 monclient: dump: 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff56ffd640 4 set_mon_vals no callback set 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 10 monclient: _renew_subs 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.087+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff60132680 con 0x7eff6009a510 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff66b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff66b00640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff66b00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6011efa0 con 0x7eff60131f40 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7eff58042030 con 0x7eff6009a510 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient: get_auth_request con 0x7eff60131f40 auth_method 0 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff4c038740 0x7eff4c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(234..234 src has 199..234) v4 ==== 11101+0+0 (secure 0 0 0) 0x7eff5806ee30 con 0x7eff6009a510 2024-09-15T03:37:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 10 monclient(hunting): handle_auth_done global_id 114381 payload 824 2024-09-15T03:37:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.088+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff6009c500 con 0x7eff60131f40 2024-09-15T03:37:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.089+0000 7eff57fff640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff4c038740 0x7eff4c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.089+0000 7eff57fff640 10 monclient: get_auth_request con 0x7eff4c038740 auth_method 0 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.090+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 msgr2=0x7eff6011ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e710 tx=0x7eff5c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 msgr2=0x7eff6011ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 msgr2=0x7eff6011ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e710 tx=0x7eff5c009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e710 tx=0x7eff5c009040 comp rx=0 tx=0).stop 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 msgr2=0x7eff6011ba90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 0x7eff4c049220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 msgr2=0x7eff6011ba90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff60131f40 0x7eff6011ba90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c049860 con 0x7eff4c046e10 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 0x7eff4c049220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient: get_auth_request con 0x7eff4c046e10 auth_method 0 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 10 monclient(hunting): handle_auth_done global_id 114384 payload 824 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff4c0499d0 con 0x7eff4c046e10 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.091+0000 7eff57fff640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff4c038740 0x7eff4c03abf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7eff6011db50 tx=0x7eff50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 msgr2=0x7eff4c049220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 0x7eff4c049220 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7eff5c017000 tx=0x7eff5c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c027430 con 0x7eff4c046e10 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 10 monclient: dump: 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.109 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7eff5c022070 con 0x7eff4c046e10 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.093+0000 7eff54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.094+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 msgr2=0x7eff4c049220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.094+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 0x7eff4c049220 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7eff5c017000 tx=0x7eff5c01a000 comp rx=0 tx=0).stop 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.094+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 msgr2=0x7eff4c049220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.094+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff4c046e10 0x7eff4c049220 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7eff5c017000 tx=0x7eff5c01a000 comp rx=0 tx=0).stop 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff66b00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff66b00640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 0x7eff60117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff66b00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff60065f20 con 0x7eff6011bfd0 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 0x7eff60117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient: get_auth_request con 0x7eff6011bfd0 auth_method 0 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.099+0000 7eff6489c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.100+0000 7eff6489c640 10 monclient(hunting): handle_auth_done global_id 114387 payload 824 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.100+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7eff6011cff0 con 0x7eff6011bfd0 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff6489c640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 msgr2=0x7eff60117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff6489c640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 0x7eff60117480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00a090 tx=0x7eff5c020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c01dc70 con 0x7eff6011bfd0 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 10 monclient: dump: 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7eff5c01ddd0 con 0x7eff6011bfd0 2024-09-15T03:37:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 msgr2=0x7eff60117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 0x7eff60117480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00a090 tx=0x7eff5c020070 comp rx=0 tx=0).stop 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 msgr2=0x7eff60117480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff54ff9640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6011bfd0 0x7eff60117480 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.102+0000 7eff6509d640 1 -- 172.21.15.153:0/992884840 reap_dead start 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:30.113 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 98, 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:30.114 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:30.115 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:30.116 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.118 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:30.120 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.121 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:30.122 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:30.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:30.125 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:30.127 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:30.127 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:30.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:30.127 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:30.127 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff4c038740 msgr2=0x7eff4c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff4c038740 0x7eff4c03abf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7eff6011db50 tx=0x7eff50008040 comp rx=0 tx=0).stop 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 10 monclient: shutdown 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 msgr2=0x7eff6011b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 0x7eff6011b550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7eff580099f0 tx=0x7eff58037000 comp rx=0 tx=0).stop 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 shutdown_connections 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff4c038740 0x7eff4c03abf0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 --2- 172.21.15.153:0/992884840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff6009a510 0x7eff6011b550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 >> 172.21.15.153:0/992884840 conn(0x7eff600672b0 msgr2=0x7eff601329a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 shutdown_connections 2024-09-15T03:37:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.103+0000 7eff66b00640 1 -- 172.21.15.153:0/992884840 wait complete. 2024-09-15T03:37:30.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:37:30.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:37:30.130 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.223+0000 7ff00279d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.223+0000 7ff00279d640 10 monclient: build_initial_monmap 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.223+0000 7ff00279d640 10 monclient: monmap: 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:30.224042+0000 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:30.224042+0000 2024-09-15T03:37:30.224 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:30.225 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 Processor -- start 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 -- start start 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient: init 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feffc06eb80 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7feffc068a20 0x7feffc09c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): picked mon.noname-c con 0x7feffc068a20 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): picked mon.noname-b con 0x7feffc09ca80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:30.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc069b80 con 0x7feffc06eb80 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc099f90 con 0x7feffc068a20 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc069e00 con 0x7feffc09ca80 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff00279d640 10 monclient(hunting): authenticate will time out at 1247.863628s 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffbfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7feffc06eb80 0x7feffc06ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34548/0 (socket says 172.21.15.153:34548) 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffbfff640 1 -- 172.21.15.153:0/2862920389 learned_addr learned my addr 172.21.15.153:0/2862920389 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 10 monclient(hunting): get_auth_request con 0x7feffc09ca80 auth_method 0 2024-09-15T03:37:30.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 1 -- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 msgr2=0x7feffc138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 1 -- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 msgr2=0x7feffc138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc138ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7ff000d3a640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc138ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 1 --2- 172.21.15.153:0/2862920389 >> v2:172.21.15.181:3300/0 conn(0x7feffc068a20 0x7feffc09c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 10 monclient(hunting): get_auth_request con 0x7feffc068a20 auth_method 0 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.224+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 10 monclient(hunting): handle_auth_done global_id 104584 payload 274 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 1 -- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 msgr2=0x7feffc138ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc138ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 1 -- 172.21.15.153:0/2862920389 >> v1:172.21.15.153:6789/0 conn(0x7feffc06eb80 legacy=0x7feffc06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 1 monclient: found mon.noname-c 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.225+0000 7feffb7fe640 1 -- 172.21.15.153:0/2862920389 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc06f660 con 0x7feffc068a20 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffb7fe640 1 -- 172.21.15.153:0/2862920389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc068a20 msgr2=0x7feffc09c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffb7fe640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc068a20 0x7feffc09c540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fefec009850 tx=0x7fefec0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=182c863cc5905f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 1 -- 172.21.15.153:0/2862920389 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec02adf0 con 0x7feffc068a20 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: dump: 2024-09-15T03:37:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.239 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 1 -- 172.21.15.153:0/2862920389 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec032440 con 0x7feffc068a20 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 5 monclient: authenticate success, global_id 104584 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 1 -- 172.21.15.153:0/2862920389 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0327e0 con 0x7feffc068a20 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 4 set_mon_vals no callback set 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7feffaffd640 10 monclient: dump: 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.240 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 10 monclient: shutdown 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 -- 172.21.15.153:0/2862920389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc068a20 msgr2=0x7feffc09c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc068a20 0x7feffc09c540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fefec009850 tx=0x7fefec0090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 -- 172.21.15.153:0/2862920389 shutdown_connections 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 --2- 172.21.15.153:0/2862920389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc068a20 0x7feffc09c540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 -- 172.21.15.153:0/2862920389 >> 172.21.15.153:0/2862920389 conn(0x7feffc067cd0 msgr2=0x7feffc0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 -- 172.21.15.153:0/2862920389 shutdown_connections 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.228+0000 7ff00279d640 1 -- 172.21.15.153:0/2862920389 wait complete. 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.229+0000 7ff00279d640 10 monclient: build_initial_monmap 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.229+0000 7ff00279d640 10 monclient: monmap: 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.242 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.229+0000 7ff00279d640 1 Processor -- start 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.229+0000 7ff00279d640 1 -- start start 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.229+0000 7ff00279d640 10 monclient: init 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 0x7feffc136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): picked mon.noname-a con 0x7feffc068a20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc06eb80 0x7feffc136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): picked mon.noname-b con 0x7feffc06eb80 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc1374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): picked mon.noname-c con 0x7feffc09ca80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc09cf40 con 0x7feffc068a20 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc138440 con 0x7feffc06eb80 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc09b7a0 con 0x7feffc09ca80 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff00279d640 10 monclient(hunting): authenticate will time out at 1247.869349s 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 0x7feffc136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc1374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 0x7feffc136a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41318/0 (socket says 172.21.15.153:41318) 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 -- 172.21.15.153:0/3516324765 learned_addr learned my addr 172.21.15.153:0/3516324765 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:30.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): get_auth_request con 0x7feffc068a20 auth_method 0 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 10 monclient(hunting): get_auth_request con 0x7feffc09ca80 auth_method 0 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc06eb80 0x7feffc136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 msgr2=0x7feffc1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 msgr2=0x7feffc1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc1374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): get_auth_request con 0x7feffc06eb80 auth_method 0 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7ff000d3a640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc1374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient(hunting): handle_auth_done global_id 114390 payload 824 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 msgr2=0x7feffc1374d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc09ca80 0x7feffc1374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc06eb80 msgr2=0x7feffc136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feffc06eb80 0x7feffc136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 monclient: found mon.noname-a 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.230+0000 7feffbfff640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec009390 con 0x7feffc068a20 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.232+0000 7feffbfff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 msgr2=0x7feffc136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.232+0000 7feffbfff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 0x7feffc136a50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c610 tx=0x7fefe4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4005cb0 con 0x7feffc068a20 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: dump: 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe4005e30 con 0x7feffc068a20 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 5 monclient: authenticate success, global_id 114390 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 10 monclient: _renew_subs 2024-09-15T03:37:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc11ea20 con 0x7feffc068a20 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4007340 con 0x7feffc068a20 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 10 monclient: dump: 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.249 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 10 monclient: _renew_subs 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feffa7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc138a40 con 0x7feffc068a20 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.233+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fefe401b400 con 0x7feffc068a20 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7ff00279d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7ff00279d640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc065e00 con 0x7feffc137a10 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feff3fff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fefdc038740 0x7fefdc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(234..234 src has 199..234) v4 ==== 11101+0+0 (secure 0 0 0) 0x7fefe404cee0 con 0x7feffc068a20 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7feffc137a10 auth_method 0 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7ff000d3a640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fefdc038740 0x7fefdc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.234+0000 7ff000d3a640 10 monclient: get_auth_request con 0x7fefdc038740 auth_method 0 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.236+0000 7ff000d3a640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fefdc038740 0x7fefdc03abf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feff4009590 tx=0x7feff400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.416+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v235) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0ff401dbb0 con 0x7f100406a710 2024-09-15T03:37:30.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.416+0000 7f0febfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T03:37:30.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.416+0000 7f0febfff640 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7feffc137a10 auth_method 0 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.435+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:30.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.515+0000 7f100a554640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T03:37:30.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.515+0000 7f100a554640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T03:37:30.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.515+0000 7f100a554640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.515+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f10040008d0 con 0x7f100406a710 2024-09-15T03:37:30.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.516+0000 7f0febfff640 1 -- 172.21.15.181:0/1941399162 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v235) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0ff4011340 con 0x7f100406a710 2024-09-15T03:37:30.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.516+0000 7f0febfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-15T03:37:30.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.516+0000 7f0febfff640 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-15T03:37:30.518 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h19' does not appear in the crush map 2024-09-15T03:37:30.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ff0038740 msgr2=0x7f0ff003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ff0038740 0x7f0ff003abf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8034000 tx=0x7f0ff8032000 comp rx=0 tx=0).stop 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 10 monclient: shutdown 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 msgr2=0x7f10041411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f10041411f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400d890 tx=0x7f0ff4008040 comp rx=0 tx=0).stop 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 shutdown_connections 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ff0038740 0x7f0ff003abf0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f100409dfa0 0x7f100411dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f100406e000 0x7f1004133ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 --2- 172.21.15.181:0/1941399162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f100406a710 0x7f10041411f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 >> 172.21.15.181:0/1941399162 conn(0x7f1004071340 msgr2=0x7f10040a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 shutdown_connections 2024-09-15T03:37:30.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.518+0000 7f100a554640 1 -- 172.21.15.181:0/1941399162 wait complete. 2024-09-15T03:37:30.532 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:30.532 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h20 2024-09-15T03:37:30.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.600+0000 7f877ef9d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:30.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.600+0000 7f877ef9d640 10 monclient: build_initial_monmap 2024-09-15T03:37:30.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.600+0000 7f877ef9d640 10 monclient: monmap: 2024-09-15T03:37:30.602 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:30.602379+0000 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:30.602379+0000 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:30.603 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 Processor -- start 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 -- start start 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient: init 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8778131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:30.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87780a0b00 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8778132510 0x7f8778069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8778132510 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87780a1b70 con 0x7f87780a0b00 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87780a08c0 con 0x7f8778132510 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8778132910 con 0x7f8778131c00 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.601+0000 7f877ef9d640 10 monclient(hunting): authenticate will time out at 1255.083811s 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8778132510 0x7f8778069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:46180/0 (socket says 172.21.15.181:46180) 2024-09-15T03:37:30.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 -- 172.21.15.181:0/975173557 learned_addr learned my addr 172.21.15.181:0/975173557 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 10 monclient(hunting): get_auth_request con 0x7f8778131c00 auth_method 0 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): get_auth_request con 0x7f8778132510 auth_method 0 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 -- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 msgr2=0x7f8778131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 -- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 msgr2=0x7f8778131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877d53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.602+0000 7f877cd39640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:30.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 10 monclient(hunting): handle_auth_done global_id 114393 payload 274 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 1 -- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 msgr2=0x7f8778131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f876ffff640 1 -- 172.21.15.181:0/975173557 >> v1:172.21.15.153:6789/0 conn(0x7f87780a0b00 legacy=0x7f8778068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 1 -- 172.21.15.181:0/975173557 >> v1:172.21.15.153:6789/0 conn(0x7f87780a0b00 legacy=0x7f8778068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 1 monclient: found mon.noname-a 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:30.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.604+0000 7f877d53a640 1 -- 172.21.15.181:0/975173557 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87780a0200 con 0x7f8778132510 2024-09-15T03:37:30.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.607+0000 7f877d53a640 1 -- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 msgr2=0x7f8778069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.607+0000 7f877d53a640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f8778069410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8770009ee0 tx=0x7f8770009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4192cb91129b54c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.607+0000 7f876f7fe640 1 -- 172.21.15.181:0/975173557 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877002aea0 con 0x7f8778132510 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: dump: 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.617 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 1 -- 172.21.15.181:0/975173557 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877002d050 con 0x7f8778132510 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 5 monclient: authenticate success, global_id 114393 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 4 set_mon_vals no callback set 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 1 -- 172.21.15.181:0/975173557 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770007990 con 0x7f8778132510 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f876f7fe640 10 monclient: dump: 2024-09-15T03:37:30.618 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 10 monclient: shutdown 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 -- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 msgr2=0x7f8778069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f8778069410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8770009ee0 tx=0x7f8770009780 comp rx=0 tx=0).stop 2024-09-15T03:37:30.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 -- 172.21.15.181:0/975173557 shutdown_connections 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f8778069410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 --2- 172.21.15.181:0/975173557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 -- 172.21.15.181:0/975173557 >> 172.21.15.181:0/975173557 conn(0x7f8778071370 msgr2=0x7f87780a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 -- 172.21.15.181:0/975173557 shutdown_connections 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.608+0000 7f877ef9d640 1 -- 172.21.15.181:0/975173557 wait complete. 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.609+0000 7f877ef9d640 10 monclient: build_initial_monmap 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.609+0000 7f877ef9d640 10 monclient: monmap: 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:30.620 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.609+0000 7f877ef9d640 1 Processor -- start 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.609+0000 7f877ef9d640 1 -- start start 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.609+0000 7f877ef9d640 10 monclient: init 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 0x7f877809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): picked mon.noname-b con 0x7f87780a0b00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8778131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f877811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8778132510 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87780a4940 con 0x7f8778132510 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8778141c10 con 0x7f87780a0b00 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877811dc20 con 0x7f8778131c00 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877ef9d640 10 monclient(hunting): authenticate will time out at 1255.092171s 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 0x7f877809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 0x7f877809aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36276/0 (socket says 172.21.15.181:36276) 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 -- 172.21.15.181:0/2002772673 learned_addr learned my addr 172.21.15.181:0/2002772673 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): get_auth_request con 0x7f87780a0b00 auth_method 0 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f877811b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 10 monclient(hunting): get_auth_request con 0x7f8778131c00 auth_method 0 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.623 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): get_auth_request con 0x7f8778132510 auth_method 0 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 msgr2=0x7f8778130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 msgr2=0x7f8778130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f876ffff640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778130b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:30.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient(hunting): handle_auth_done global_id 104590 payload 824 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 msgr2=0x7f8778130b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778130b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 msgr2=0x7f877811b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f877811b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient: _finish_hunting 0 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 monclient: found mon.noname-b 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877cd39640 1 -- 172.21.15.181:0/2002772673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770009a20 con 0x7f87780a0b00 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.610+0000 7f877d53a640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f877811b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.612+0000 7f877cd39640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 msgr2=0x7f877809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.612+0000 7f877cd39640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 0x7f877809aa90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f877400e8a0 tx=0x7f8774008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.612+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877400c6d0 con 0x7f87780a0b00 2024-09-15T03:37:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.612+0000 7f876cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.612+0000 7f876cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.612+0000 7f876cff9640 10 monclient: dump: 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.626 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 5 monclient: authenticate success, global_id 104590 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877400c830 con 0x7f87780a0b00 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 monclient: _renew_subs 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877811df00 con 0x7f87780a0b00 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877400cbf0 con 0x7f87780a0b00 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 10 monclient: dump: 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:30.627 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 monclient: _renew_subs 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877811ccc0 con 0x7f87780a0b00 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876effd640 4 set_mon_vals no callback set 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8774013410 con 0x7f87780a0b00 2024-09-15T03:37:30.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.613+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87780008d0 con 0x7f87780a0b00 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.614+0000 7f876cff9640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8764038740 0x7f876403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.614+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(235..235 src has 199..235) v4 ==== 10888+0+0 (secure 0 0 0) 0x7f877404d250 con 0x7f87780a0b00 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.614+0000 7f876ffff640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8764038740 0x7f876403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.614+0000 7f876ffff640 10 monclient: get_auth_request con 0x7f8764038740 auth_method 0 2024-09-15T03:37:30.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.616+0000 7f876ffff640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8764038740 0x7f876403abf0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8768007a80 tx=0x7f8768009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:30.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.618+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8774023300 con 0x7f87780a0b00 2024-09-15T03:37:30.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.618+0000 7f876cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:30.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.618+0000 7f876cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:30.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.782+0000 7f877ef9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T03:37:30.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.782+0000 7f877ef9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T03:37:30.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.782+0000 7f877ef9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:30.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:30.782+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7f87780a1890 con 0x7f87780a0b00 2024-09-15T03:37:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7feffc137a10 auth_method 0 2024-09-15T03:37:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:30.836+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:31.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.435+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8774024350 con 0x7f87780a0b00 2024-09-15T03:37:31.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.435+0000 7f876cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T03:37:31.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.435+0000 7f876cff9640 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-15T03:37:31.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.537+0000 7f877ef9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T03:37:31.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.537+0000 7f877ef9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T03:37:31.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.537+0000 7f877ef9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:31.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.537+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7f8778072640 con 0x7f87780a0b00 2024-09-15T03:37:31.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7feffc137a10 auth_method 0 2024-09-15T03:37:31.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:31.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:31.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:31.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:31.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:31.637+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:31.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.728+0000 7f876cff9640 1 -- 172.21.15.181:0/2002772673 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8774024350 con 0x7f87780a0b00 2024-09-15T03:37:31.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.728+0000 7f876cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-15T03:37:31.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.728+0000 7f876cff9640 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-15T03:37:31.730 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h20' does not appear in the crush map 2024-09-15T03:37:31.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8764038740 msgr2=0x7f876403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:31.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8764038740 0x7f876403abf0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8768007a80 tx=0x7f8768009040 comp rx=0 tx=0).stop 2024-09-15T03:37:31.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 10 monclient: shutdown 2024-09-15T03:37:31.732 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 msgr2=0x7f877809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 0x7f877809aa90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f877400e8a0 tx=0x7f8774008040 comp rx=0 tx=0).stop 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 shutdown_connections 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8764038740 0x7f876403abf0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8778132510 0x7f877811b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8778131c00 0x7f8778130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 --2- 172.21.15.181:0/2002772673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87780a0b00 0x7f877809aa90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 >> 172.21.15.181:0/2002772673 conn(0x7f8778071370 msgr2=0x7f87780a3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 shutdown_connections 2024-09-15T03:37:31.733 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.730+0000 7f877ef9d640 1 -- 172.21.15.181:0/2002772673 wait complete. 2024-09-15T03:37:31.741 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:31.742 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h21 2024-09-15T03:37:31.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.813+0000 7f0165694640 10 monclient: get_monmap_and_config 2024-09-15T03:37:31.815 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.813+0000 7f0165694640 10 monclient: build_initial_monmap 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.813+0000 7f0165694640 10 monclient: monmap: 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:31.815621+0000 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:31.815621+0000 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:31.816 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:31.817 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.814+0000 7f0165694640 1 Processor -- start 2024-09-15T03:37:31.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.814+0000 7f0165694640 1 -- start start 2024-09-15T03:37:31.829 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.814+0000 7f0165694640 10 monclient: init 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.814+0000 7f0165694640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.814+0000 7f0165694640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f016009ec50 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f016009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f016013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 10 monclient(hunting): picked mon.noname-b con 0x7f0160069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0160067420 con 0x7f016009ec50 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01600a0480 con 0x7f016009f020 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016006a170 con 0x7f0160069d10 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f0165694640 10 monclient(hunting): authenticate will time out at 1256.297155s 2024-09-15T03:37:31.830 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015e7fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f016009f020 0x7f0160136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:44130/0 (socket says 172.21.15.181:44130) 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015e7fc640 1 -- 172.21.15.181:0/2109447894 learned_addr learned my addr 172.21.15.181:0/2109447894 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f016013a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 10 monclient(hunting): get_auth_request con 0x7f0160069d10 auth_method 0 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 msgr2=0x7f016013a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 msgr2=0x7f016013a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f016013a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.815+0000 7f015f7fe640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f016013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015e7fc640 1 -- 172.21.15.181:0/2109447894 >> v1:172.21.15.181:6789/0 conn(0x7f016009f020 legacy=0x7f0160136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015effd640 1 -- 172.21.15.181:0/2109447894 >> v1:172.21.15.153:6789/0 conn(0x7f016009ec50 legacy=0x7f0160131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176287678 0 0) 0x7f01600a0480 con 0x7f016009f020 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): my global_id is 104593 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01480034e0 con 0x7f016009f020 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646476863 0 0) 0x7f0160067420 con 0x7f016009ec50 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): my global_id is 114402 2024-09-15T03:37:31.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01600a0480 con 0x7f016009ec50 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243411692 0 0) 0x7f01480034e0 con 0x7f016009f020 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 msgr2=0x7f016013a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f016013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 >> v1:172.21.15.153:6789/0 conn(0x7f016009ec50 legacy=0x7f0160131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: _finish_hunting 0 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600a0200 con 0x7f016009f020 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f0154001720 con 0x7f016009f020 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f0165694640 5 monclient: authenticate success, global_id 104593 2024-09-15T03:37:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f0165694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: dump: 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 >> v1:172.21.15.181:6789/0 conn(0x7f016009f020 legacy=0x7f0160136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f0148003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): picked mon.c con 0x7f0160069d10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0148004190 0x7f0148004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): picked mon.b con 0x7f0148004190 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0148004de0 0x7f0148005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): picked mon.a con 0x7f0148004de0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014800bcd0 con 0x7f0148004de0 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014800be40 con 0x7f0148004190 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.817+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014800c000 con 0x7f0160069d10 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015dffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f0165694640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0148004190 0x7f0148004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): get_auth_request con 0x7f0148004190 auth_method 0 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f0148003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): get_auth_request con 0x7f0160069d10 auth_method 0 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0148004de0 0x7f0148005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): get_auth_request con 0x7f0148004de0 auth_method 0 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient(hunting): handle_auth_done global_id 104593 payload 293 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 msgr2=0x7f0148003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0160069d10 0x7f0148003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0148004de0 msgr2=0x7f0148005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0148004de0 0x7f0148005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 monclient: found mon.b 2024-09-15T03:37:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 1 -- 172.21.15.181:0/2109447894 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480034e0 con 0x7f0148004190 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.818+0000 7f015e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015e7fc640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0148004190 msgr2=0x7f0148004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015e7fc640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0148004190 0x7f0148004780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0154001490 tx=0x7f0154007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0df188dca1b094 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154005890 con 0x7f0148004190 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: dump: 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:31.839 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154005a10 con 0x7f0148004190 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 4 set_mon_vals no callback set 2024-09-15T03:37:31.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 1 -- 172.21.15.181:0/2109447894 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154005bc0 con 0x7f0148004190 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015dffb640 10 monclient: dump: 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.841 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 10 monclient: shutdown 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 1 -- 172.21.15.181:0/2109447894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0148004190 msgr2=0x7f0148004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 1 --2- 172.21.15.181:0/2109447894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0148004190 0x7f0148004780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0154001490 tx=0x7f0154007a50 comp rx=0 tx=0).stop 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f015f7fe640 1 -- 172.21.15.181:0/2109447894 reap_dead start 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 1 -- 172.21.15.181:0/2109447894 shutdown_connections 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 1 -- 172.21.15.181:0/2109447894 >> 172.21.15.181:0/2109447894 conn(0x7f01600a5cf0 msgr2=0x7f01600a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 1 -- 172.21.15.181:0/2109447894 shutdown_connections 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.821+0000 7f0165694640 1 -- 172.21.15.181:0/2109447894 wait complete. 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.822+0000 7f0165694640 10 monclient: build_initial_monmap 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.822+0000 7f0165694640 10 monclient: monmap: 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:31.843 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.822+0000 7f0165694640 1 Processor -- start 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.822+0000 7f0165694640 1 -- start start 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.822+0000 7f0165694640 10 monclient: init 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 0x7f0160140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): picked mon.noname-c con 0x7f0148004190 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0160141470 0x7f0160130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): picked mon.noname-a con 0x7f0160141470 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 0x7f016011e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): picked mon.noname-b con 0x7f01601311a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160120a90 con 0x7f0160141470 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016009c020 con 0x7f01601311a0 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016009dfc0 con 0x7f0148004190 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f0165694640 10 monclient(hunting): authenticate will time out at 1256.305197s 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 0x7f016011e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 0x7f016011e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36308/0 (socket says 172.21.15.181:36308) 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 -- 172.21.15.181:0/666293403 learned_addr learned my addr 172.21.15.181:0/666293403 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): get_auth_request con 0x7f01601311a0 auth_method 0 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:31.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 0x7f0160140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0160141470 0x7f0160130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 10 monclient(hunting): get_auth_request con 0x7f0148004190 auth_method 0 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): get_auth_request con 0x7f0160141470 auth_method 0 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 msgr2=0x7f0160140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:37:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 msgr2=0x7f0160140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 0x7f0160140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015effd640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 0x7f0160140f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient(hunting): handle_auth_done global_id 104596 payload 824 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 msgr2=0x7f0160140f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 0x7f0160140f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0160141470 msgr2=0x7f0160130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0160141470 0x7f0160130c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 monclient: found mon.noname-b 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.823+0000 7f015f7fe640 1 -- 172.21.15.181:0/666293403 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01601220a0 con 0x7f01601311a0 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.825+0000 7f015f7fe640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 msgr2=0x7f016011e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.825+0000 7f015f7fe640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 0x7f016011e830 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f01600a1310 tx=0x7f01580096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.825+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158007040 con 0x7f01601311a0 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.825+0000 7f0146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.825+0000 7f0146ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:31.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.825+0000 7f0146ffd640 10 monclient: dump: 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0158014810 con 0x7f01601311a0 2024-09-15T03:37:31.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 5 monclient: authenticate success, global_id 104596 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 10 monclient: _renew_subs 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0160120290 con 0x7f01601311a0 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158014bd0 con 0x7f01601311a0 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 10 monclient: dump: 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:31.850 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f015d7fa640 4 set_mon_vals no callback set 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 10 monclient: _renew_subs 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0160120e60 con 0x7f01601311a0 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.826+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f015801c3f0 con 0x7f01601311a0 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f0165694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:31.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f0165694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:31.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f0165694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:31.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f0146ffd640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f014c038770 0x7f014c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:31.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0160141df0 con 0x7f01601311a0 2024-09-15T03:37:31.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(236..236 src has 199..236) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f015804d220 con 0x7f01601311a0 2024-09-15T03:37:31.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f015effd640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f014c038770 0x7f014c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:31.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.827+0000 7f015effd640 10 monclient: get_auth_request con 0x7f014c038770 auth_method 0 2024-09-15T03:37:31.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.829+0000 7f015effd640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f014c038770 0x7f014c03ac20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f01500018a0 tx=0x7f0150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:31.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.831+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f015801a610 con 0x7f01601311a0 2024-09-15T03:37:31.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.831+0000 7f0146ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:31.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.831+0000 7f0146ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:31.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.991+0000 7f0165694640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T03:37:32.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.991+0000 7f0165694640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T03:37:32.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.991+0000 7f0165694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:31.991+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7f0160122230 con 0x7f01601311a0 2024-09-15T03:37:32.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.449+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7f015801a610 con 0x7f01601311a0 2024-09-15T03:37:32.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.449+0000 7f0146ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T03:37:32.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.449+0000 7f0146ffd640 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-15T03:37:32.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.546+0000 7f0165694640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T03:37:32.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.546+0000 7f0165694640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T03:37:32.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.546+0000 7f0165694640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.546+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7f01600008d0 con 0x7f01601311a0 2024-09-15T03:37:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.547+0000 7f0146ffd640 1 -- 172.21.15.181:0/666293403 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7f015804a500 con 0x7f01601311a0 2024-09-15T03:37:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.547+0000 7f0146ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-15T03:37:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.547+0000 7f0146ffd640 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-15T03:37:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h21' does not appear in the crush map 2024-09-15T03:37:32.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f014c038770 msgr2=0x7f014c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:32.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f014c038770 0x7f014c03ac20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f01500018a0 tx=0x7f0150009040 comp rx=0 tx=0).stop 2024-09-15T03:37:32.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 10 monclient: shutdown 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 msgr2=0x7f016011e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 0x7f016011e830 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f01600a1310 tx=0x7f01580096f0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 shutdown_connections 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f014c038770 0x7f014c03ac20 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01601311a0 0x7f016011e830 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0160141470 0x7f0160130c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 --2- 172.21.15.181:0/666293403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0148004190 0x7f0160140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 >> 172.21.15.181:0/666293403 conn(0x7f01600a5cf0 msgr2=0x7f01600a2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 shutdown_connections 2024-09-15T03:37:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.549+0000 7f0165694640 1 -- 172.21.15.181:0/666293403 wait complete. 2024-09-15T03:37:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h22 2024-09-15T03:37:32.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.631+0000 7f269048b640 10 monclient: get_monmap_and_config 2024-09-15T03:37:32.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.631+0000 7f269048b640 10 monclient: build_initial_monmap 2024-09-15T03:37:32.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.631+0000 7f269048b640 10 monclient: monmap: 2024-09-15T03:37:32.633 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:32.633324+0000 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:32.633324+0000 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:32.634 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:32.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 Processor -- start 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 -- start start 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient: init 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f268806e000 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806a710 0x7f268806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): picked mon.noname-b con 0x7f268806a710 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f268809e170 0x7f268809e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): picked mon.noname-c con 0x7f268809e170 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26880699c0 con 0x7f268806e000 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268806a360 con 0x7f268809e170 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268806be20 con 0x7f268806a710 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f269048b640 10 monclient(hunting): authenticate will time out at 1257.114824s 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f268809e170 0x7f268809e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f268809e170 0x7f268809e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36312/0 (socket says 172.21.15.181:36312) 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 1 -- 172.21.15.181:0/3388364386 learned_addr learned my addr 172.21.15.181:0/3388364386 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 10 monclient(hunting): get_auth_request con 0x7f268809e170 auth_method 0 2024-09-15T03:37:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.632+0000 7f268ea28640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268da26640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806a710 0x7f268806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 10 monclient(hunting): handle_auth_done global_id 104599 payload 274 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 1 -- 172.21.15.181:0/3388364386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806a710 msgr2=0x7f268806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806a710 0x7f268806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 1 -- 172.21.15.181:0/3388364386 >> v1:172.21.15.153:6789/0 conn(0x7f268806e000 legacy=0x7f268809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:32.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 10 monclient: _finish_hunting 0 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 1 monclient: found mon.noname-c 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268ea28640 1 -- 172.21.15.181:0/3388364386 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688069f80 con 0x7f268809e170 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268da26640 10 monclient: get_auth_request con 0x7f268806a710 auth_method 0 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.633+0000 7f268da26640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806a710 0x7f268806e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268ea28640 1 -- 172.21.15.181:0/3388364386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268809e170 msgr2=0x7f268809e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268ea28640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268809e170 0x7f268809e610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2680009f20 tx=0x7f2680009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8e7d502950dd34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 1 -- 172.21.15.181:0/3388364386 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680007610 con 0x7f268809e170 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:32.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: dump: 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:32.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: _finish_auth 0 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 5 monclient: authenticate success, global_id 104599 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 1 -- 172.21.15.181:0/3388364386 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268002d050 con 0x7f268809e170 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 1 -- 172.21.15.181:0/3388364386 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26800079a0 con 0x7f268809e170 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 4 set_mon_vals no callback set 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:32.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f268d225640 10 monclient: dump: 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:32.648 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 10 monclient: shutdown 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 -- 172.21.15.181:0/3388364386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268809e170 msgr2=0x7f268809e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268809e170 0x7f268809e610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2680009f20 tx=0x7f2680009840 comp rx=0 tx=0).stop 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 -- 172.21.15.181:0/3388364386 shutdown_connections 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268809e170 0x7f268809e610 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 --2- 172.21.15.181:0/3388364386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806a710 0x7f268806e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 -- 172.21.15.181:0/3388364386 >> 172.21.15.181:0/3388364386 conn(0x7f2688071340 msgr2=0x7f26880a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 -- 172.21.15.181:0/3388364386 shutdown_connections 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.636+0000 7f269048b640 1 -- 172.21.15.181:0/3388364386 wait complete. 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.637+0000 7f269048b640 10 monclient: build_initial_monmap 2024-09-15T03:37:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.637+0000 7f269048b640 10 monclient: monmap: 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:32.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.637+0000 7f269048b640 1 Processor -- start 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.637+0000 7f269048b640 1 -- start start 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.637+0000 7f269048b640 10 monclient: init 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.637+0000 7f269048b640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 0x7f268813d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): picked mon.noname-b con 0x7f268806a710 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 0x7f268813cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): picked mon.noname-c con 0x7f268806e000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268809e170 0x7f2688140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): picked mon.noname-a con 0x7f268809e170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:32.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26881203f0 con 0x7f268809e170 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26881220e0 con 0x7f268806a710 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268811fbc0 con 0x7f268806e000 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f269048b640 10 monclient(hunting): authenticate will time out at 1257.120063s 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 0x7f268813d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 0x7f268813d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36316/0 (socket says 172.21.15.181:36316) 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 -- 172.21.15.181:0/1187265021 learned_addr learned my addr 172.21.15.181:0/1187265021 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): get_auth_request con 0x7f268806a710 auth_method 0 2024-09-15T03:37:32.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 0x7f268813cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 10 monclient(hunting): get_auth_request con 0x7f268806e000 auth_method 0 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268809e170 0x7f2688140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:32.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): get_auth_request con 0x7f268809e170 auth_method 0 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 msgr2=0x7f268813cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 msgr2=0x7f268813cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 0x7f268813cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268ea28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient(hunting): handle_auth_done global_id 104602 payload 824 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268da26640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 0x7f268813cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 msgr2=0x7f268813cd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 0x7f268813cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268809e170 msgr2=0x7f2688140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268809e170 0x7f2688140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient: _finish_hunting 0 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 monclient: found mon.noname-b 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.638+0000 7f268e227640 1 -- 172.21.15.181:0/1187265021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009ae0 con 0x7f268806a710 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f268e227640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 msgr2=0x7f268813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f268e227640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 0x7f268813d6b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f268400c5e0 tx=0x7f2684007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684012840 con 0x7f268806a710 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: dump: 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:32.658 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26840129a0 con 0x7f268806a710 2024-09-15T03:37:32.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 5 monclient: authenticate success, global_id 104602 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 10 monclient: _renew_subs 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688121e00 con 0x7f268806a710 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684012de0 con 0x7f268806a710 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f267e7fc640 10 monclient: dump: 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:32.660 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 10 monclient: _renew_subs 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f268ca24640 4 set_mon_vals no callback set 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.640+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268813d2a0 con 0x7f268806a710 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f269048b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2684011590 con 0x7f268806a710 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f269048b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f269048b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268809deb0 con 0x7f268806a710 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f267e7fc640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f266c038740 0x7f266c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.641+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(237..237 src has 199..237) v4 ==== 10462+0+0 (secure 0 0 0) 0x7f268404d230 con 0x7f268806a710 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.642+0000 7f268da26640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f266c038740 0x7f266c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.642+0000 7f268da26640 10 monclient: get_auth_request con 0x7f266c038740 auth_method 0 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.644+0000 7f268da26640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f266c038740 0x7f266c03abf0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2678006fd0 tx=0x7f2678008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.646+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f268809deb0 con 0x7f268806a710 2024-09-15T03:37:32.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.646+0000 7f267e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:32.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.646+0000 7f267e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.804+0000 7f269048b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T03:37:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.804+0000 7f269048b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T03:37:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.804+0000 7f269048b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:32.804+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f268809deb0 con 0x7f268806a710 2024-09-15T03:37:33.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.230+0000 7feff2ffd640 10 monclient: tick 2024-09-15T03:37:33.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.230+0000 7feff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.230+0000 7feff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7feffc137a10 auth_method 0 2024-09-15T03:37:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 10 monclient(hunting): handle_auth_done global_id 124100 payload 824 2024-09-15T03:37:33.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.239+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feffc04bf60 con 0x7feffc137a10 2024-09-15T03:37:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefec034000 tx=0x7fefec032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec03cb90 con 0x7feffc137a10 2024-09-15T03:37:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 10 monclient: dump: 2024-09-15T03:37:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.248 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.248 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.248 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.248 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.248 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.249 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.249 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.249 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fefec03ccf0 con 0x7feffc137a10 2024-09-15T03:37:33.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:33.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefec034000 tx=0x7fefec032000 comp rx=0 tx=0).stop 2024-09-15T03:37:33.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 msgr2=0x7feffc06b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:33.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.242+0000 7feff3fff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc137a10 0x7feffc06b840 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefec034000 tx=0x7fefec032000 comp rx=0 tx=0).stop 2024-09-15T03:37:33.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7ff00279d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:33.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7ff00279d640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 0x7feffc06c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7ff00279d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc11ec60 con 0x7feffc06bd80 2024-09-15T03:37:33.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 0x7feffc06c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7feffc06bd80 auth_method 0 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 10 monclient(hunting): handle_auth_done global_id 124103 payload 824 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.249+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feffc0638d0 con 0x7feffc06bd80 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feffb7fe640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 msgr2=0x7feffc06c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feffb7fe640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 0x7feffc06c180 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feffc11e8c0 tx=0x7fefec040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0469e0 con 0x7feffc06bd80 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 10 monclient: dump: 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fefec046b40 con 0x7feffc06bd80 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 msgr2=0x7feffc06c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 0x7feffc06c180 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feffc11e8c0 tx=0x7fefec040040 comp rx=0 tx=0).stop 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 msgr2=0x7feffc06c180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.252+0000 7feff3fff640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feffc06bd80 0x7feffc06c180 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feffc11e8c0 tx=0x7fefec040040 comp rx=0 tx=0).stop 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 98, 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:33.272 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:33.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fefdc038740 msgr2=0x7fefdc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fefdc038740 0x7fefdc03abf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feff4009590 tx=0x7feff400b040 comp rx=0 tx=0).stop 2024-09-15T03:37:33.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 10 monclient: shutdown 2024-09-15T03:37:33.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 msgr2=0x7feffc136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 --2- 172.21.15.153:0/3516324765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc068a20 0x7feffc136a50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c610 tx=0x7fefe4004960 comp rx=0 tx=0).stop 2024-09-15T03:37:33.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff000d3a640 1 -- 172.21.15.153:0/3516324765 reap_dead start 2024-09-15T03:37:33.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff000d3a640 1 -- 172.21.15.153:0/3516324765 reap_dead start 2024-09-15T03:37:33.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 shutdown_connections 2024-09-15T03:37:33.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 >> 172.21.15.153:0/3516324765 conn(0x7feffc067cd0 msgr2=0x7feffc0a1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 shutdown_connections 2024-09-15T03:37:33.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.253+0000 7ff00279d640 1 -- 172.21.15.153:0/3516324765 wait complete. 2024-09-15T03:37:33.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:37:33.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:37:33.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:37:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.374+0000 7f47b7a20640 10 monclient: get_monmap_and_config 2024-09-15T03:37:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.374+0000 7f47b7a20640 10 monclient: build_initial_monmap 2024-09-15T03:37:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.374+0000 7f47b7a20640 10 monclient: monmap: 2024-09-15T03:37:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:33.375244+0000 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:33.375244+0000 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:33.376 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 1 Processor -- start 2024-09-15T03:37:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 1 -- start start 2024-09-15T03:37:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient: init 2024-09-15T03:37:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47b0131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47b009fda0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f47b009f0d0 0x7f47b0139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient(hunting): picked mon.noname-a con 0x7f47b009f0d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b009f510 con 0x7f47b0131980 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b00a1180 con 0x7f47b009fda0 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b00a1460 con 0x7f47b009f0d0 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b7a20640 10 monclient(hunting): authenticate will time out at 1251.014826s 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b57bc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f47b0131980 0x7f47b0131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34560/0 (socket says 172.21.15.153:34560) 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b57bc640 1 -- 172.21.15.153:0/3899222892 learned_addr learned my addr 172.21.15.153:0/3899222892 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.375+0000 7f47b5fbd640 1 --2- 172.21.15.153:0/3899222892 >> v2:172.21.15.153:3300/0 conn(0x7f47b009f0d0 0x7f47b0139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): get_auth_request con 0x7f47b009f0d0 auth_method 0 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_done global_id 114411 payload 274 2024-09-15T03:37:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 1 -- 172.21.15.153:0/3899222892 >> v1:172.21.15.181:6789/0 conn(0x7f47b009fda0 legacy=0x7f47b009e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 1 -- 172.21.15.153:0/3899222892 >> v1:172.21.15.153:6789/0 conn(0x7f47b0131980 legacy=0x7f47b0131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 1 monclient: found mon.noname-a 2024-09-15T03:37:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.376+0000 7f47b5fbd640 1 -- 172.21.15.153:0/3899222892 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b0132460 con 0x7f47b009f0d0 2024-09-15T03:37:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b5fbd640 1 -- 172.21.15.153:0/3899222892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b009f0d0 msgr2=0x7f47b0139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b5fbd640 1 --2- 172.21.15.153:0/3899222892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b009f0d0 0x7f47b0139cd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009e60 tx=0x7f47a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2776fcc94bc098e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 1 -- 172.21.15.153:0/3899222892 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8007630 con 0x7f47b009f0d0 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: dump: 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 1 -- 172.21.15.153:0/3899222892 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a802d050 con 0x7f47b009f0d0 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 5 monclient: authenticate success, global_id 114411 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 1 -- 172.21.15.153:0/3899222892 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a80079c0 con 0x7f47b009f0d0 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 4 set_mon_vals no callback set 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47a7fff640 10 monclient: dump: 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 10 monclient: shutdown 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 -- 172.21.15.153:0/3899222892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b009f0d0 msgr2=0x7f47b0139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 --2- 172.21.15.153:0/3899222892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b009f0d0 0x7f47b0139cd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009e60 tx=0x7f47a8009780 comp rx=0 tx=0).stop 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 -- 172.21.15.153:0/3899222892 shutdown_connections 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 --2- 172.21.15.153:0/3899222892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b009f0d0 0x7f47b0139cd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 -- 172.21.15.153:0/3899222892 >> 172.21.15.153:0/3899222892 conn(0x7f47b0067bd0 msgr2=0x7f47b00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 -- 172.21.15.153:0/3899222892 shutdown_connections 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.379+0000 7f47b7a20640 1 -- 172.21.15.153:0/3899222892 wait complete. 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.380+0000 7f47b7a20640 10 monclient: build_initial_monmap 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.380+0000 7f47b7a20640 10 monclient: monmap: 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.380+0000 7f47b7a20640 1 Processor -- start 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.380+0000 7f47b7a20640 1 -- start start 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.380+0000 7f47b7a20640 10 monclient: init 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 0x7f47b01383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): picked mon.noname-c con 0x7f47b009f0d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b009fda0 0x7f47b0133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): picked mon.noname-b con 0x7f47b009fda0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b0131980 0x7f47b01341c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): picked mon.noname-a con 0x7f47b0131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b009bf10 con 0x7f47b0131980 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0139fd0 con 0x7f47b009fda0 2024-09-15T03:37:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0137fe0 con 0x7f47b009f0d0 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b7a20640 10 monclient(hunting): authenticate will time out at 1251.020386s 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 0x7f47b01383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 0x7f47b01383d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51880/0 (socket says 172.21.15.153:51880) 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 -- 172.21.15.153:0/3286131227 learned_addr learned my addr 172.21.15.153:0/3286131227 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): get_auth_request con 0x7f47b009f0d0 auth_method 0 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b0131980 0x7f47b01341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): get_auth_request con 0x7f47b0131980 auth_method 0 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b009fda0 0x7f47b0133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): get_auth_request con 0x7f47b009fda0 auth_method 0 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b5fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient(hunting): handle_auth_done global_id 124106 payload 824 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b009fda0 msgr2=0x7f47b0133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b009fda0 0x7f47b0133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b0131980 msgr2=0x7f47b01341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b0131980 0x7f47b01341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 monclient: found mon.noname-c 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.381+0000 7f47b57bc640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009a20 con 0x7f47b009f0d0 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b57bc640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 msgr2=0x7f47b01383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b57bc640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 0x7f47b01383d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00c610 tx=0x7f47ac004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac005cb0 con 0x7f47b009f0d0 2024-09-15T03:37:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: dump: 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac005e30 con 0x7f47b009f0d0 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 5 monclient: authenticate success, global_id 124106 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac007340 con 0x7f47b009f0d0 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 10 monclient: _renew_subs 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b0135dd0 con 0x7f47b009f0d0 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a77fe640 4 set_mon_vals no callback set 2024-09-15T03:37:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47a57fa640 10 monclient: dump: 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 10 monclient: _renew_subs 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.384+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b0136430 con 0x7f47b009f0d0 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f47ac01b400 con 0x7f47b009f0d0 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b7a20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b7a20640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 0x7f47b0134700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b7a20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0065e00 con 0x7f47b01392f0 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47a57fa640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4794038740 0x7f479403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(237..237 src has 199..237) v4 ==== 10462+0+0 (secure 0 0 0) 0x7f47ac04cb90 con 0x7f47b009f0d0 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 0x7f47b0134700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient: get_auth_request con 0x7f47b01392f0 auth_method 0 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b5fbd640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4794038740 0x7f479403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b5fbd640 10 monclient: get_auth_request con 0x7f4794038740 auth_method 0 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_done global_id 104608 payload 824 2024-09-15T03:37:33.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.385+0000 7f47b4fbb640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47b0137720 con 0x7f47b01392f0 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47b4fbb640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 msgr2=0x7f47b0134700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47b4fbb640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 0x7f47b0134700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f47a000f780 tx=0x7f47a0007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47b5fbd640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4794038740 0x7f479403abf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f47a8034000 tx=0x7f47a8037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a000d8b0 con 0x7f47b01392f0 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 10 monclient: dump: 2024-09-15T03:37:33.409 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f47a000da10 con 0x7f47b01392f0 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 msgr2=0x7f47b0134700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 0x7f47b0134700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f47a000f780 tx=0x7f47a0007e10 comp rx=0 tx=0).stop 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 msgr2=0x7f47b0134700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.388+0000 7f47a57fa640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b01392f0 0x7f47b0134700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b7a20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b7a20640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 0x7f47b0098e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b7a20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0135d70 con 0x7f47b0098a70 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 0x7f47b0098e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient: get_auth_request con 0x7f47b0098a70 auth_method 0 2024-09-15T03:37:33.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.394+0000 7f47b4fbb640 10 monclient(hunting): handle_auth_done global_id 104611 payload 824 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.395+0000 7f47b4fbb640 1 -- 172.21.15.153:0/3286131227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47b004bf60 con 0x7f47b0098a70 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47b4fbb640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 msgr2=0x7f47b0098e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47b4fbb640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 0x7f47b0098e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f47a000f780 tx=0x7f47a0019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a001d820 con 0x7f47b0098a70 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 10 monclient: dump: 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f47a000fd20 con 0x7f47b0098a70 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 msgr2=0x7f47b0098e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 0x7f47b0098e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f47a000f780 tx=0x7f47a0019000 comp rx=0 tx=0).stop 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 msgr2=0x7f47b0098e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.397+0000 7f47a57fa640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b0098a70 0x7f47b0098e70 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f47a000f780 tx=0x7f47a0019000 comp rx=0 tx=0).stop 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:33.414 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 98, 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:33.415 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:33.416 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:33.417 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:33.418 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:33.419 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:33.422 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:33.425 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:33.426 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.428 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:33.429 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:33.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:33.430 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:33.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:33.430 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:33.430 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:33.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4794038740 msgr2=0x7f479403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4794038740 0x7f479403abf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f47a8034000 tx=0x7f47a8037000 comp rx=0 tx=0).stop 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 10 monclient: shutdown 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 msgr2=0x7f47b01383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 --2- 172.21.15.153:0/3286131227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47b009f0d0 0x7f47b01383d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00c610 tx=0x7f47ac004960 comp rx=0 tx=0).stop 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b5fbd640 1 -- 172.21.15.153:0/3286131227 reap_dead start 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b5fbd640 1 -- 172.21.15.153:0/3286131227 reap_dead start 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 shutdown_connections 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 >> 172.21.15.153:0/3286131227 conn(0x7f47b0067bd0 msgr2=0x7f47b00a3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 shutdown_connections 2024-09-15T03:37:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.398+0000 7f47b7a20640 1 -- 172.21.15.153:0/3286131227 wait complete. 2024-09-15T03:37:33.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:37:33.433 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:37:33.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:37:33.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.450+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f268809deb0 con 0x7f268806a710 2024-09-15T03:37:33.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.450+0000 7f267e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T03:37:33.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.450+0000 7f267e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.526+0000 7fd79fd1d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.526+0000 7fd79fd1d640 10 monclient: build_initial_monmap 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.526+0000 7fd79fd1d640 10 monclient: monmap: 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:33.527036+0000 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:33.527036+0000 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.527 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:33.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:33.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:33.528 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:33.528 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:33.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.526+0000 7fd79fd1d640 1 Processor -- start 2024-09-15T03:37:33.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 1 -- start start 2024-09-15T03:37:33.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient: init 2024-09-15T03:37:33.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd798131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd798131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd79809e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd798136bb0 0x7fd798138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd798136bb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79809faf0 con 0x7fd79809e910 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79809fd30 con 0x7fd798136bb0 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980a1110 con 0x7fd798131ab0 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:33.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79fd1d640 10 monclient(hunting): authenticate will time out at 1251.166544s 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd798131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd798131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51894/0 (socket says 172.21.15.153:51894) 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 -- 172.21.15.153:0/2733754083 learned_addr learned my addr 172.21.15.153:0/2733754083 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): get_auth_request con 0x7fd798131ab0 auth_method 0 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 1 --2- 172.21.15.153:0/2733754083 >> v2:172.21.15.181:3300/0 conn(0x7fd798136bb0 0x7fd798138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): get_auth_request con 0x7fd798136bb0 auth_method 0 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79e2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient(hunting): handle_auth_done global_id 124109 payload 274 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 -- 172.21.15.153:0/2733754083 >> v2:172.21.15.181:3300/0 conn(0x7fd798136bb0 msgr2=0x7fd798138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 --2- 172.21.15.153:0/2733754083 >> v2:172.21.15.181:3300/0 conn(0x7fd798136bb0 0x7fd798138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 -- 172.21.15.153:0/2733754083 >> v1:172.21.15.181:6789/0 conn(0x7fd79809e910 legacy=0x7fd79809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient: _finish_hunting 0 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 monclient: found mon.noname-b 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.527+0000 7fd79dab9640 1 -- 172.21.15.153:0/2733754083 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7980a13f0 con 0x7fd798131ab0 2024-09-15T03:37:33.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.530+0000 7fd79dab9640 1 -- 172.21.15.153:0/2733754083 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 msgr2=0x7fd798131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.530+0000 7fd79dab9640 1 --2- 172.21.15.153:0/2733754083 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd798131e80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bd67a14cc7a7477 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 1 -- 172.21.15.153:0/2733754083 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c032440 con 0x7fd798131ab0 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: dump: 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: _finish_auth 0 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 1 -- 172.21.15.153:0/2733754083 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0325a0 con 0x7fd798131ab0 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 5 monclient: authenticate success, global_id 124109 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 1 -- 172.21.15.153:0/2733754083 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c032940 con 0x7fd798131ab0 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 4 set_mon_vals no callback set 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79cab7640 10 monclient: dump: 2024-09-15T03:37:33.539 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 10 monclient: shutdown 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 -- 172.21.15.153:0/2733754083 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 msgr2=0x7fd798131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/2733754083 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd798131e80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009910 tx=0x7fd78c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 -- 172.21.15.153:0/2733754083 shutdown_connections 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/2733754083 >> v2:172.21.15.181:3300/0 conn(0x7fd798136bb0 0x7fd798138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/2733754083 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd798131e80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 -- 172.21.15.153:0/2733754083 >> 172.21.15.153:0/2733754083 conn(0x7fd798067d30 msgr2=0x7fd7980a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 -- 172.21.15.153:0/2733754083 shutdown_connections 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.533+0000 7fd79fd1d640 1 -- 172.21.15.153:0/2733754083 wait complete. 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.534+0000 7fd79fd1d640 10 monclient: build_initial_monmap 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.534+0000 7fd79fd1d640 10 monclient: monmap: 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:33.541 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.534+0000 7fd79fd1d640 1 Processor -- start 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.534+0000 7fd79fd1d640 1 -- start start 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.534+0000 7fd79fd1d640 10 monclient: init 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd79809e910 0x7fd79812f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd79809e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd79812f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd798131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd798136bb0 0x7fd79812fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd798136bb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd798132730 con 0x7fd798136bb0 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981347a0 con 0x7fd79809e910 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79809bde0 con 0x7fd798131ab0 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79fd1d640 10 monclient(hunting): authenticate will time out at 1251.174136s 2024-09-15T03:37:33.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd79812f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd79812f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:51908/0 (socket says 172.21.15.153:51908) 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd79809e910 0x7fd79812f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 -- 172.21.15.153:0/233491811 learned_addr learned my addr 172.21.15.153:0/233491811 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): get_auth_request con 0x7fd798131ab0 auth_method 0 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd798136bb0 0x7fd79812fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): get_auth_request con 0x7fd79809e910 auth_method 0 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): get_auth_request con 0x7fd798136bb0 auth_method 0 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79dab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79e2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient(hunting): handle_auth_done global_id 124112 payload 824 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 -- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd79809e910 msgr2=0x7fd79812f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd79809e910 0x7fd79812f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 -- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd798136bb0 msgr2=0x7fd79812fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd798136bb0 0x7fd79812fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient: _finish_hunting 0 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 monclient: found mon.noname-c 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.535+0000 7fd79d2b8640 1 -- 172.21.15.153:0/233491811 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c009450 con 0x7fd798131ab0 2024-09-15T03:37:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd79d2b8640 1 -- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 msgr2=0x7fd79812f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd79d2b8640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd79812f590 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c610 tx=0x7fd794007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794012840 con 0x7fd798131ab0 2024-09-15T03:37:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: dump: 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:33.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7940129e0 con 0x7fd798131ab0 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd79fd1d640 5 monclient: authenticate success, global_id 124112 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd79fd1d640 10 monclient: _renew_subs 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd79fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd798133850 con 0x7fd798131ab0 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794012da0 con 0x7fd798131ab0 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd789ffb640 10 monclient: dump: 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.550 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.537+0000 7fd78bfff640 4 set_mon_vals no callback set 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.538+0000 7fd79fd1d640 10 monclient: _renew_subs 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.538+0000 7fd79fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.538+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd798133570 con 0x7fd798131ab0 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.538+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd794011590 con 0x7fd798131ab0 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.538+0000 7fd789ffb640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd77c038740 0x7fd77c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.538+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(238..238 src has 199..238) v4 ==== 10249+0+0 (secure 0 0 0) 0x7fd79404d180 con 0x7fd798131ab0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.539+0000 7fd79fd1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.539+0000 7fd79fd1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.539+0000 7fd79fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.539+0000 7fd79dab9640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd77c038740 0x7fd77c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.539+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7981330e0 con 0x7fd798131ab0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.539+0000 7fd79dab9640 10 monclient: get_auth_request con 0x7fd77c038740 auth_method 0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.541+0000 7fd79dab9640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd77c038740 0x7fd77c03abf0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd78c031000 tx=0x7fd78c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.543+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd7940104c0 con 0x7fd798131ab0 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.543+0000 7fd789ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.543+0000 7fd789ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:33.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.551+0000 7f269048b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T03:37:33.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.551+0000 7f269048b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T03:37:33.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.551+0000 7f269048b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.551+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f268809deb0 con 0x7f268806a710 2024-09-15T03:37:33.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.552+0000 7f267e7fc640 1 -- 172.21.15.181:0/1187265021 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f268809deb0 con 0x7f268806a710 2024-09-15T03:37:33.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.552+0000 7f267e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-15T03:37:33.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.552+0000 7f267e7fc640 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-15T03:37:33.554 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h22' does not appear in the crush map 2024-09-15T03:37:33.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f266c038740 msgr2=0x7f266c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f266c038740 0x7f266c03abf0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2678006fd0 tx=0x7f2678008040 comp rx=0 tx=0).stop 2024-09-15T03:37:33.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 10 monclient: shutdown 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 msgr2=0x7f268813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 0x7f268813d6b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f268400c5e0 tx=0x7f2684007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 shutdown_connections 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f266c038740 0x7f266c03abf0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f268809e170 0x7f2688140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f268806e000 0x7f268813cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 --2- 172.21.15.181:0/1187265021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268806a710 0x7f268813d6b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 >> 172.21.15.181:0/1187265021 conn(0x7f2688071340 msgr2=0x7f2688098a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 shutdown_connections 2024-09-15T03:37:33.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.554+0000 7f269048b640 1 -- 172.21.15.181:0/1187265021 wait complete. 2024-09-15T03:37:33.565 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:33.565 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h23 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.636+0000 7f2c3d39d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.636+0000 7f2c3d39d640 10 monclient: build_initial_monmap 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.637+0000 7f2c3d39d640 10 monclient: monmap: 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:33.638791+0000 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:33.638791+0000 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:33.639 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:33.640 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:33.640 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:33.640 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:33.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.637+0000 7f2c3d39d640 1 Processor -- start 2024-09-15T03:37:33.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.637+0000 7f2c3d39d640 1 -- start start 2024-09-15T03:37:33.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.637+0000 7f2c3d39d640 10 monclient: init 2024-09-15T03:37:33.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:33.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:37:33.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c3806e130 0x7f2c3809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c3806e130 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c3806a790 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2c3806e500 0x7f2c3809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c3806e500 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3806a0b0 con 0x7f2c3806a790 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c38068c30 con 0x7f2c3806e130 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c38069750 con 0x7f2c3806e500 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c3d39d640 10 monclient(hunting): authenticate will time out at 1258.120218s 2024-09-15T03:37:33.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2c3806e500 0x7f2c3809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c2ffff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f2c3806a790 0x7f2c3809dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:44140/0 (socket says 172.21.15.181:44140) 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c2ffff640 1 -- 172.21.15.181:0/1666849577 learned_addr learned my addr 172.21.15.181:0/1666849577 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): get_auth_request con 0x7f2c3806e500 auth_method 0 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c36ffd640 1 --2- 172.21.15.181:0/1666849577 >> v2:172.21.15.153:3300/0 conn(0x7f2c3806e130 0x7f2c3809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c36ffd640 10 monclient(hunting): get_auth_request con 0x7f2c3806e130 auth_method 0 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient(hunting): handle_auth_done global_id 104620 payload 274 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 1 -- 172.21.15.181:0/1666849577 >> v2:172.21.15.153:3300/0 conn(0x7f2c3806e130 msgr2=0x7f2c3809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 1 --2- 172.21.15.181:0/1666849577 >> v2:172.21.15.153:3300/0 conn(0x7f2c3806e130 0x7f2c3809da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 1 -- 172.21.15.181:0/1666849577 >> v1:172.21.15.181:6789/0 conn(0x7f2c3806a790 legacy=0x7f2c3809dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient: _finish_hunting 0 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 1 monclient: found mon.noname-c 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c377fe640 1 -- 172.21.15.181:0/1666849577 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38069a60 con 0x7f2c3806e500 2024-09-15T03:37:33.645 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.638+0000 7f2c36ffd640 1 --2- 172.21.15.181:0/1666849577 >> v2:172.21.15.153:3300/0 conn(0x7f2c3806e130 0x7f2c3809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:33.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c377fe640 1 -- 172.21.15.181:0/1666849577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e500 msgr2=0x7f2c3809e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c377fe640 1 --2- 172.21.15.181:0/1666849577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e500 0x7f2c3809e6b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2c380a0eb0 tx=0x7f2c30009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c144edc9239dedd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 1 -- 172.21.15.181:0/1666849577 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30007760 con 0x7f2c3806e500 2024-09-15T03:37:33.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: dump: 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 1 -- 172.21.15.181:0/1666849577 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3002d050 con 0x7f2c3806e500 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c3d39d640 5 monclient: authenticate success, global_id 104620 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c3d39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c3d39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c3d39d640 4 set_mon_vals no callback set 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 1 -- 172.21.15.181:0/1666849577 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30007af0 con 0x7f2c3806e500 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.641+0000 7f2c367fc640 10 monclient: dump: 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:33.653 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 10 monclient: shutdown 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 -- 172.21.15.181:0/1666849577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e500 msgr2=0x7f2c3809e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/1666849577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e500 0x7f2c3809e6b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2c380a0eb0 tx=0x7f2c30009780 comp rx=0 tx=0).stop 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 -- 172.21.15.181:0/1666849577 shutdown_connections 2024-09-15T03:37:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/1666849577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e500 0x7f2c3809e6b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/1666849577 >> v2:172.21.15.153:3300/0 conn(0x7f2c3806e130 0x7f2c3809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 -- 172.21.15.181:0/1666849577 >> 172.21.15.181:0/1666849577 conn(0x7f2c38071340 msgr2=0x7f2c380a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 -- 172.21.15.181:0/1666849577 shutdown_connections 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 -- 172.21.15.181:0/1666849577 wait complete. 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 10 monclient: build_initial_monmap 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 10 monclient: monmap: 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:33.655 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.642+0000 7f2c3d39d640 1 Processor -- start 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 -- start start 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient: init 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c3806a790 0x7f2c3813d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c3806a790 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 0x7f2c3813cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): picked mon.noname-b con 0x7f2c3806e130 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:33.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3806e500 0x7f2c38140c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c3806e500 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38122410 con 0x7f2c3806e500 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38122130 con 0x7f2c3806e130 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3811fbc0 con 0x7f2c3806a790 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c3d39d640 10 monclient(hunting): authenticate will time out at 1258.125477s 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 0x7f2c3813cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 0x7f2c3813cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36332/0 (socket says 172.21.15.181:36332) 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 -- 172.21.15.181:0/790448591 learned_addr learned my addr 172.21.15.181:0/790448591 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:33.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): get_auth_request con 0x7f2c3806e130 auth_method 0 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c3806a790 0x7f2c3813d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3806e500 0x7f2c38140c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): get_auth_request con 0x7f2c3806a790 auth_method 0 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): get_auth_request con 0x7f2c3806e500 auth_method 0 2024-09-15T03:37:33.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient(hunting): handle_auth_done global_id 104623 payload 824 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 -- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c3806a790 msgr2=0x7f2c3813d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c3806a790 0x7f2c3813d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 -- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3806e500 msgr2=0x7f2c38140c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3806e500 0x7f2c38140c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 monclient: found mon.noname-b 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.643+0000 7f2c2ffff640 1 -- 172.21.15.181:0/790448591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c30009a20 con 0x7f2c3806e130 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2ffff640 1 -- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 msgr2=0x7f2c3813cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2ffff640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 0x7f2c3813cdb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000c530 tx=0x7f2c20007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20005cb0 con 0x7f2c3806e130 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: dump: 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.660 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: _finish_auth 0 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20005e30 con 0x7f2c3806e130 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20007320 con 0x7f2c3806e130 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 5 monclient: authenticate success, global_id 104623 2024-09-15T03:37:33.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c2effd640 10 monclient: dump: 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c35ffb640 4 set_mon_vals no callback set 2024-09-15T03:37:33.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 10 monclient: _renew_subs 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c38121e50 con 0x7f2c3806e130 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 10 monclient: _renew_subs 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.646+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3813e320 con 0x7f2c3806e130 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2c2001b3b0 con 0x7f2c3806e130 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c2effd640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c18038770 0x7f2c1803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(238..238 src has 199..238) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f2c2004c5b0 con 0x7f2c3806e130 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c3d39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c3d39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c3d39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c36ffd640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c18038770 0x7f2c1803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c380008d0 con 0x7f2c3806e130 2024-09-15T03:37:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.647+0000 7f2c36ffd640 10 monclient: get_auth_request con 0x7f2c18038770 auth_method 0 2024-09-15T03:37:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.649+0000 7f2c36ffd640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c18038770 0x7f2c1803ac20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006e70 tx=0x7f2c28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.651+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2c20017020 con 0x7f2c3806e130 2024-09-15T03:37:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.651+0000 7f2c2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.651+0000 7f2c2effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:33.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.763+0000 7fd79fd1d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:37:33.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.763+0000 7fd79fd1d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:37:33.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.763+0000 7fd79fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:33.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.763+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd79804bf60 con 0x7fd798131ab0 2024-09-15T03:37:33.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.764+0000 7fd789ffb640 1 -- 172.21.15.153:0/233491811 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fd7940102e0 con 0x7fd798131ab0 2024-09-15T03:37:33.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.764+0000 7fd789ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:37:33.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.764+0000 7fd789ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:33.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd77c038740 msgr2=0x7fd77c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd77c038740 0x7fd77c03abf0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd78c031000 tx=0x7fd78c032000 comp rx=0 tx=0).stop 2024-09-15T03:37:33.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 10 monclient: shutdown 2024-09-15T03:37:33.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:33.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 msgr2=0x7fd79812f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd79812f590 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c610 tx=0x7fd794007ed0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 shutdown_connections 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd77c038740 0x7fd77c03abf0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd798136bb0 0x7fd79812fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd798131ab0 0x7fd79812f590 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 --2- 172.21.15.153:0/233491811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd79809e910 0x7fd79812f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 >> 172.21.15.153:0/233491811 conn(0x7fd798067d30 msgr2=0x7fd79809c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 shutdown_connections 2024-09-15T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:33.766+0000 7fd79fd1d640 1 -- 172.21.15.153:0/233491811 wait complete. 2024-09-15T03:37:33.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:37:33.812 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.810+0000 7f2c3d39d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T03:37:33.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.810+0000 7f2c3d39d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T03:37:33.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.810+0000 7f2c3d39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:33.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:33.810+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7f2c38122750 con 0x7f2c3806e130 2024-09-15T03:37:33.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.843+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.843+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=1,health=12,kv=8,logm=62} crc {auth=1643308785,config=1228700967,health=724060797,kv=3235807718,logm=725412169}) 2024-09-15T03:37:33.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.843+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=7,logm=64} crc {auth=464177125,config=3140514009,health=1942745526,kv=3891702298,logm=1434453707}) 2024-09-15T03:37:33.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.843+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.843+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=1,health=12,kv=8,logm=62} crc {auth=1643308785,config=1228700967,health=724060797,kv=3235807718,logm=725412169}) 2024-09-15T03:37:33.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.843+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=1,health=11,kv=8,logm=61} crc {auth=3279793385,config=1228700967,health=883097184,kv=3235807718,logm=2659729445}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490411199}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099318022}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490411199}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3756701731}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779382378}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902906074}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779382378}) 2024-09-15T03:37:33.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.844+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3762171978}) 2024-09-15T03:37:33.849 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.845+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.849 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.845+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695220020}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.845+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227973937}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.845+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.845+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695220020}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.845+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134288694}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.846+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.846+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301794556}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.846+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=215078111}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.846+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.846+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301794556}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.846+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63818208}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51842018}) 2024-09-15T03:37:33.850 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=431808506}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51842018}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3054472254}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025487260}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2967127737}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025487260}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.847+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965317591}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289205611}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123598212}) 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.851 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289205611}) 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082282080}) 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493697581}) 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070426970}) 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493697581}) 2024-09-15T03:37:33.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.848+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448945710}) 2024-09-15T03:37:33.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.856+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.856+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027840869}) 2024-09-15T03:37:33.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.856+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2066455067}) 2024-09-15T03:37:33.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.856+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.856+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027840869}) 2024-09-15T03:37:33.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.856+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4135267913}) 2024-09-15T03:37:33.858 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604890693}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768472316}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604890693}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866951202}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179813394}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137504304}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179813394}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.857+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229857417}) 2024-09-15T03:37:33.859 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.860 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246014683}) 2024-09-15T03:37:33.860 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161718820}) 2024-09-15T03:37:33.860 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.860 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246014683}) 2024-09-15T03:37:33.860 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227709440}) 2024-09-15T03:37:33.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145457650}) 2024-09-15T03:37:33.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1140071993}) 2024-09-15T03:37:33.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145457650}) 2024-09-15T03:37:33.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.858+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948434395}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789033496}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325014649}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789033496}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655511196}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027195129}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2733199254}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.859+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027195129}) 2024-09-15T03:37:33.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350123513}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850943466}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412476492}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850943466}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.860+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737069113}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3903872326}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556119342}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3903872326}) 2024-09-15T03:37:33.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787092336}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813902509}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3510557015}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813902509}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.861+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=384392119}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095043278}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796117617}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095043278}) 2024-09-15T03:37:33.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=464436781}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.862+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615720769}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011073776}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615720769}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3592470570}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3182133925}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1508626541}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3182133925}) 2024-09-15T03:37:33.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.863+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3495969838}) 2024-09-15T03:37:33.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384488332}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11486737}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384488332}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284463273}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450575803}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2162637155}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450575803}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.864+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946061516}) 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791486563}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2708028681}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791486563}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106310548}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844758375}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977394308}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844758375}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.865+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3526740780}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.866+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.866+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693834139}) 2024-09-15T03:37:33.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.866+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1361412822}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.866+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.866+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693834139}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.866+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782326658}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960662144}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2479508047}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960662144}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053517132}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511838613}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765737637}) 2024-09-15T03:37:33.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511838613}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.867+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100307062}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.868+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.868+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348823550}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.868+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408577237}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.868+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.868+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348823550}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.868+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786837091}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886457175}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571586726}) 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886457175}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977808126}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203962925}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4121159246}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203962925}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.869+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=234268909}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493906692}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496650270}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493906692}) 2024-09-15T03:37:33.879 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290041735}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727909066}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484524430}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727909066}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.870+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226719926}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.871+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.871+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919420198}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.871+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328050090}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.871+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.871+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919420198}) 2024-09-15T03:37:33.880 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:33.871+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416417945}) 2024-09-15T03:37:34.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.115+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.115+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774660447}) 2024-09-15T03:37:34.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.115+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3040627223}) 2024-09-15T03:37:34.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.115+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.115+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774660447}) 2024-09-15T03:37:34.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.115+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=747407975}) 2024-09-15T03:37:34.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.116+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.116+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412691531}) 2024-09-15T03:37:34.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.116+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=250830878}) 2024-09-15T03:37:34.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.116+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.116+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412691531}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.116+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=970316149}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063323584}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442634219}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063323584}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849993412}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089784399}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507494871}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089784399}) 2024-09-15T03:37:34.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.117+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=854799344}) 2024-09-15T03:37:34.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018387843}) 2024-09-15T03:37:34.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763912541}) 2024-09-15T03:37:34.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018387843}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3533038186}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839962412}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817399195}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839962412}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.118+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213948847}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.119+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.119+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=9} crc {logm=2775070456,mdsmap=1531151339,mgr=1013486720,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2752824692,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=761132606,osd_snap=454132730,osdmap=3011159906}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.119+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=9} crc {logm=2712988689,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2752824692,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=761132606,osd_snap=454132730,osdmap=3310558105}) 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.119+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.119+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=9} crc {logm=2775070456,mdsmap=1531151339,mgr=1013486720,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2752824692,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=761132606,osd_snap=454132730,osdmap=3011159906}) 2024-09-15T03:37:34.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.119+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=50,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=2,osdmap=14} crc {logm=2383572535,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=938559951,mon_config_key=4117641549,monmap=244636553,osd_metadata=4072215849,osd_snap=454132730,osdmap=583185721}) 2024-09-15T03:37:34.125 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.125+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.125+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=68} crc {osdmap=210952994}) 2024-09-15T03:37:34.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.125+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=3121972244}) 2024-09-15T03:37:34.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.125+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:37:34.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.125+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=68} crc {osdmap=210952994}) 2024-09-15T03:37:34.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:37:34.125+0000 7f5adf7f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=68} crc {osdmap=2901115111}) 2024-09-15T03:37:34.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.567+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2c20018830 con 0x7f2c3806e130 2024-09-15T03:37:34.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.567+0000 7f2c2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T03:37:34.569 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.567+0000 7f2c2effd640 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-15T03:37:34.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.666+0000 7f2c3d39d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T03:37:34.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.666+0000 7f2c3d39d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T03:37:34.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.666+0000 7f2c3d39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.666+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7f2c38072640 con 0x7f2c3806e130 2024-09-15T03:37:34.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.667+0000 7f2c2effd640 1 -- 172.21.15.181:0/790448591 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2c20018260 con 0x7f2c3806e130 2024-09-15T03:37:34.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.667+0000 7f2c2effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-15T03:37:34.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.667+0000 7f2c2effd640 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-15T03:37:34.669 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h23' does not appear in the crush map 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.668+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c18038770 msgr2=0x7f2c1803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.668+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c18038770 0x7f2c1803ac20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006e70 tx=0x7f2c28008040 comp rx=0 tx=0).stop 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.668+0000 7f2c3d39d640 10 monclient: shutdown 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.668+0000 7f2c3d39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.668+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 msgr2=0x7f2c3813cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.668+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 0x7f2c3813cdb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000c530 tx=0x7f2c20007f90 comp rx=0 tx=0).stop 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 shutdown_connections 2024-09-15T03:37:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2c18038770 0x7f2c1803ac20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3806e500 0x7f2c38140c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2c3806e130 0x7f2c3813cdb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 --2- 172.21.15.181:0/790448591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c3806a790 0x7f2c3813d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 >> 172.21.15.181:0/790448591 conn(0x7f2c38071340 msgr2=0x7f2c380a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 shutdown_connections 2024-09-15T03:37:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.669+0000 7f2c3d39d640 1 -- 172.21.15.181:0/790448591 wait complete. 2024-09-15T03:37:34.679 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:34.680 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h24 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.750+0000 7fd27059d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.750+0000 7fd27059d640 10 monclient: build_initial_monmap 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.750+0000 7fd27059d640 10 monclient: monmap: 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:34.752759+0000 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:34.752759+0000 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.751+0000 7fd27059d640 1 Processor -- start 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.751+0000 7fd27059d640 1 -- start start 2024-09-15T03:37:34.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.751+0000 7fd27059d640 10 monclient: init 2024-09-15T03:37:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:37:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): picked mon.noname-b con 0x7fd26806e000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd26806a710 0x7fd26809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): picked mon.noname-c con 0x7fd26806a710 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd26809dfa0 0x7fd26809e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): picked mon.noname-a con 0x7fd26809dfa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26806a360 con 0x7fd26809dfa0 2024-09-15T03:37:34.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680699c0 con 0x7fd26806a710 2024-09-15T03:37:34.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26806be20 con 0x7fd26806e000 2024-09-15T03:37:34.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:34.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd27059d640 10 monclient(hunting): authenticate will time out at 1259.234162s 2024-09-15T03:37:34.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd26806a710 0x7fd26809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd26806a710 0x7fd26809da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36336/0 (socket says 172.21.15.181:36336) 2024-09-15T03:37:34.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 -- 172.21.15.181:0/2563562291 learned_addr learned my addr 172.21.15.181:0/2563562291 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:34.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): get_auth_request con 0x7fd26806a710 auth_method 0 2024-09-15T03:37:34.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.768 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 1 --2- 172.21.15.181:0/2563562291 >> v2:172.21.15.153:3300/0 conn(0x7fd26809dfa0 0x7fd26809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 1 --2- 172.21.15.181:0/2563562291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): get_auth_request con 0x7fd26809dfa0 auth_method 0 2024-09-15T03:37:34.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): get_auth_request con 0x7fd26806e000 auth_method 0 2024-09-15T03:37:34.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26eb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26e339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient(hunting): handle_auth_done global_id 104626 payload 274 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 -- 172.21.15.181:0/2563562291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 msgr2=0x7fd26806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 --2- 172.21.15.181:0/2563562291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 -- 172.21.15.181:0/2563562291 >> v2:172.21.15.153:3300/0 conn(0x7fd26809dfa0 msgr2=0x7fd26809e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 --2- 172.21.15.181:0/2563562291 >> v2:172.21.15.153:3300/0 conn(0x7fd26809dfa0 0x7fd26809e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient: _finish_hunting 0 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 monclient: found mon.noname-c 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.752+0000 7fd26db38640 1 -- 172.21.15.181:0/2563562291 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268069f80 con 0x7fd26806a710 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26db38640 1 -- 172.21.15.181:0/2563562291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 msgr2=0x7fd26809da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26db38640 1 --2- 172.21.15.181:0/2563562291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26809da60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd268067130 tx=0x7fd2580091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9848073ce0bfd8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 1 -- 172.21.15.181:0/2563562291 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258032440 con 0x7fd26806a710 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: dump: 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:34.773 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: _finish_auth 0 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 1 -- 172.21.15.181:0/2563562291 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580325a0 con 0x7fd26806a710 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd27059d640 5 monclient: authenticate success, global_id 104626 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd27059d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:34.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd27059d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 1 -- 172.21.15.181:0/2563562291 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258032940 con 0x7fd26806a710 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd27059d640 4 set_mon_vals no callback set 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.755+0000 7fd26d337640 10 monclient: dump: 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:34.775 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 10 monclient: shutdown 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 -- 172.21.15.181:0/2563562291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 msgr2=0x7fd26809da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 --2- 172.21.15.181:0/2563562291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26809da60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd268067130 tx=0x7fd2580091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 -- 172.21.15.181:0/2563562291 shutdown_connections 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 --2- 172.21.15.181:0/2563562291 >> v2:172.21.15.153:3300/0 conn(0x7fd26809dfa0 0x7fd26809e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 --2- 172.21.15.181:0/2563562291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26809da60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 --2- 172.21.15.181:0/2563562291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26806e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 -- 172.21.15.181:0/2563562291 >> 172.21.15.181:0/2563562291 conn(0x7fd268071340 msgr2=0x7fd2680a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:34.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 -- 172.21.15.181:0/2563562291 shutdown_connections 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 -- 172.21.15.181:0/2563562291 wait complete. 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 10 monclient: build_initial_monmap 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 10 monclient: monmap: 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:34.777 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 Processor -- start 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 1 -- start start 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.756+0000 7fd27059d640 10 monclient: init 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26811f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): picked mon.noname-b con 0x7fd26806a710 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26813cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): picked mon.noname-c con 0x7fd26806e000 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26809dfa0 0x7fd268140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): picked mon.noname-a con 0x7fd26809dfa0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26806e970 con 0x7fd26809dfa0 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26806d3e0 con 0x7fd26806a710 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd268121f80 con 0x7fd26806e000 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd27059d640 10 monclient(hunting): authenticate will time out at 1259.239443s 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26811f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26811f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36350/0 (socket says 172.21.15.181:36350) 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 -- 172.21.15.181:0/2796108122 learned_addr learned my addr 172.21.15.181:0/2796108122 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): get_auth_request con 0x7fd26806a710 auth_method 0 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26809dfa0 0x7fd268140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26813cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): get_auth_request con 0x7fd26809dfa0 auth_method 0 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): get_auth_request con 0x7fd26806e000 auth_method 0 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.780 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26db38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26eb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient(hunting): handle_auth_done global_id 104629 payload 824 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 -- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 msgr2=0x7fd26813cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26813cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 -- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26809dfa0 msgr2=0x7fd268140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26809dfa0 0x7fd268140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient: _finish_hunting 0 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 monclient: found mon.noname-b 2024-09-15T03:37:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.757+0000 7fd26e339640 1 -- 172.21.15.181:0/2796108122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009450 con 0x7fd26806a710 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.759+0000 7fd26e339640 1 -- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 msgr2=0x7fd26811f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd26e339640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26811f960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd26400ea20 tx=0x7fd2640090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264009600 con 0x7fd26806a710 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: dump: 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.782 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26400cc20 con 0x7fd26806a710 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 5 monclient: authenticate success, global_id 104629 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 10 monclient: _renew_subs 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd268122260 con 0x7fd26806a710 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264013470 con 0x7fd26806a710 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd25e7fc640 10 monclient: dump: 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.784 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 10 monclient: _renew_subs 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26813da60 con 0x7fd26806a710 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.760+0000 7fd26cb36640 4 set_mon_vals no callback set 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd264013860 con 0x7fd26806a710 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd27059d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd27059d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd27059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2681226d0 con 0x7fd26806a710 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd25e7fc640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd24c038740 0x7fd24c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(239..239 src has 199..239) v4 ==== 10036+0+0 (secure 0 0 0) 0x7fd26404d2e0 con 0x7fd26806a710 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd26db38640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd24c038740 0x7fd24c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.761+0000 7fd26db38640 10 monclient: get_auth_request con 0x7fd24c038740 auth_method 0 2024-09-15T03:37:34.786 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:37:34.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.763+0000 7fd26db38640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd24c038740 0x7fd24c03abf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd26806e940 tx=0x7fd258032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.765+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd2640236b0 con 0x7fd26806a710 2024-09-15T03:37:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.765+0000 7fd25e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.765+0000 7fd25e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.882+0000 7f316399d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.882+0000 7f316399d640 10 monclient: build_initial_monmap 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.882+0000 7f316399d640 10 monclient: monmap: 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:34.882996+0000 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:34.882996+0000 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:34.883 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:34.884 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:34.884 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:34.884 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:34.884 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:34.884 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.882+0000 7f316399d640 1 Processor -- start 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 1 -- start start 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient: init 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f315c131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 0x7f315c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient(hunting): picked mon.noname-b con 0x7f315c09e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f315c09f220 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c09faf0 con 0x7f315c09f220 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c09fd30 con 0x7f315c131ab0 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c132590 con 0x7f315c09e910 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f316399d640 10 monclient(hunting): authenticate will time out at 1252.522632s 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 0x7f315c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3161f3a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f315c09f220 0x7f315c138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34576/0 (socket says 172.21.15.153:34576) 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3161f3a640 1 -- 172.21.15.153:0/2022226874 learned_addr learned my addr 172.21.15.153:0/2022226874 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): get_auth_request con 0x7f315c09e910 auth_method 0 2024-09-15T03:37:34.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient(hunting): handle_auth_done global_id 124124 payload 274 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 1 -- 172.21.15.153:0/2022226874 >> v1:172.21.15.181:6789/0 conn(0x7f315c131ab0 legacy=0x7f315c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 1 -- 172.21.15.153:0/2022226874 >> v1:172.21.15.153:6789/0 conn(0x7f315c09f220 legacy=0x7f315c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient: _finish_hunting 0 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 1 monclient: found mon.noname-b 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:37:34.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.883+0000 7f3160f38640 1 -- 172.21.15.153:0/2022226874 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0a1110 con 0x7f315c09e910 2024-09-15T03:37:34.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3160f38640 1 -- 172.21.15.153:0/2022226874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 msgr2=0x7f315c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3160f38640 1 --2- 172.21.15.153:0/2022226874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 0x7f315c09ece0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f314c009880 tx=0x7f314c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb7e502dd58e8b67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 1 -- 172.21.15.153:0/2022226874 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c032440 con 0x7f315c09e910 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: dump: 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.895 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 1 -- 172.21.15.153:0/2022226874 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c0325a0 con 0x7f315c09e910 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f316399d640 5 monclient: authenticate success, global_id 124124 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f316399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f316399d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 1 -- 172.21.15.153:0/2022226874 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c032940 con 0x7f315c09e910 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f316399d640 4 set_mon_vals no callback set 2024-09-15T03:37:34.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.886+0000 7f3153fff640 10 monclient: dump: 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 10 monclient: shutdown 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 -- 172.21.15.153:0/2022226874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 msgr2=0x7f315c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 --2- 172.21.15.153:0/2022226874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 0x7f315c09ece0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f314c009880 tx=0x7f314c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 -- 172.21.15.153:0/2022226874 shutdown_connections 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 --2- 172.21.15.153:0/2022226874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c09e910 0x7f315c09ece0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 -- 172.21.15.153:0/2022226874 >> 172.21.15.153:0/2022226874 conn(0x7f315c067d30 msgr2=0x7f315c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 -- 172.21.15.153:0/2022226874 shutdown_connections 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.887+0000 7f316399d640 1 -- 172.21.15.153:0/2022226874 wait complete. 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient: build_initial_monmap 2024-09-15T03:37:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient: monmap: 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 1 Processor -- start 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 1 -- start start 2024-09-15T03:37:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient: init 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c09e910 0x7f315c12f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient(hunting): picked mon.noname-b con 0x7f315c09e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f315c09f220 0x7f315c12f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.888+0000 7f316399d640 10 monclient(hunting): picked mon.noname-a con 0x7f315c09f220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 0x7f315c12fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 10 monclient(hunting): picked mon.noname-c con 0x7f315c131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c0979f0 con 0x7f315c09f220 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c09ac80 con 0x7f315c09e910 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c096a60 con 0x7f315c131ab0 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f316399d640 10 monclient(hunting): authenticate will time out at 1252.528032s 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f315c09f220 0x7f315c12f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 0x7f315c12fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f315c09f220 0x7f315c12f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41366/0 (socket says 172.21.15.153:41366) 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): get_auth_request con 0x7f315c131ab0 auth_method 0 2024-09-15T03:37:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 1 -- 172.21.15.153:0/2462289566 learned_addr learned my addr 172.21.15.153:0/2462289566 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): get_auth_request con 0x7f315c09f220 auth_method 0 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c09e910 0x7f315c12f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): get_auth_request con 0x7f315c09e910 auth_method 0 2024-09-15T03:37:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3160f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient(hunting): handle_auth_done global_id 124127 payload 824 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 -- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c09e910 msgr2=0x7f315c12f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c09e910 0x7f315c12f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 -- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f315c09f220 msgr2=0x7f315c12f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f315c09f220 0x7f315c12f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 monclient: found mon.noname-c 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.889+0000 7f3161f3a640 1 -- 172.21.15.153:0/2462289566 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c009390 con 0x7f315c131ab0 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f3161f3a640 1 -- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 msgr2=0x7f315c12fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f3161f3a640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 0x7f315c12fcc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f315400cb60 tx=0x7f31540052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400a850 con 0x7f315c131ab0 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: dump: 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315400a9d0 con 0x7f315c131ab0 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 5 monclient: authenticate success, global_id 124127 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 10 monclient: _renew_subs 2024-09-15T03:37:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c1315f0 con 0x7f315c131ab0 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400ad90 con 0x7f315c131ab0 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31517fa640 10 monclient: dump: 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f31537fe640 4 set_mon_vals no callback set 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 10 monclient: _renew_subs 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.891+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c09b660 con 0x7f315c131ab0 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3154007300 con 0x7f315c131ab0 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f31517fa640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3140038740 0x7f314003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f315404d050 con 0x7f315c131ab0 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f316399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f316399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f316399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.892+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c04bf60 con 0x7f315c131ab0 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.893+0000 7f3161739640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3140038740 0x7f314003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.893+0000 7f3161739640 10 monclient: get_auth_request con 0x7f3140038740 auth_method 0 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.895+0000 7f3161739640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3140038740 0x7f314003abf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3158007fd0 tx=0x7f3158007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.897+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f315401bae0 con 0x7f315c131ab0 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.897+0000 7f31517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:34.897+0000 7f31517fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:34.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.926+0000 7fd27059d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T03:37:34.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.926+0000 7fd27059d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T03:37:34.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.926+0000 7fd27059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:34.929 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:34.926+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7fd2680008d0 con 0x7fd26806a710 2024-09-15T03:37:35.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.110+0000 7f316399d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:35.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.111+0000 7f316399d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:35.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.111+0000 7f316399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:35.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.111+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f315c066280 con 0x7f315c131ab0 2024-09-15T03:37:35.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.111+0000 7f31517fa640 1 -- 172.21.15.153:0/2462289566 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f3154029620 con 0x7f315c131ab0 2024-09-15T03:37:35.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.111+0000 7f31517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:35.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.111+0000 7f31517fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:35.113 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:35.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3140038740 msgr2=0x7f314003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3140038740 0x7f314003abf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3158007fd0 tx=0x7f3158007e10 comp rx=0 tx=0).stop 2024-09-15T03:37:35.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 10 monclient: shutdown 2024-09-15T03:37:35.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 msgr2=0x7f315c12fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 0x7f315c12fcc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f315400cb60 tx=0x7f31540052b0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 shutdown_connections 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3140038740 0x7f314003abf0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f315c131ab0 0x7f315c12fcc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f315c09f220 0x7f315c12f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 --2- 172.21.15.153:0/2462289566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c09e910 0x7f315c12f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 >> 172.21.15.153:0/2462289566 conn(0x7f315c067d30 msgr2=0x7f315c0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 shutdown_connections 2024-09-15T03:37:35.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.113+0000 7f316399d640 1 -- 172.21.15.153:0/2462289566 wait complete. 2024-09-15T03:37:35.125 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:37:35.125 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:37:35.125 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:37:35.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:37:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.219+0000 7f142295d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.219+0000 7f142295d640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.219+0000 7f142295d640 10 monclient: monmap: 2024-09-15T03:37:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:35.220273+0000 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:35.220273+0000 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:35.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 1 Processor -- start 2024-09-15T03:37:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 1 -- start start 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient: init 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f141c0a1ef0 0x7f141c09c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient(hunting): picked mon.noname-a con 0x7f141c0a1ef0 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f141c09abd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f141c136d00 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c06f010 con 0x7f141c09abd0 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c09dcc0 con 0x7f141c136d00 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141c09da00 con 0x7f141c0a1ef0 2024-09-15T03:37:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f142295d640 10 monclient(hunting): authenticate will time out at 1252.859752s 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f141c0a1ef0 0x7f141c09c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f141c09abd0 0x7f141c09afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40754/0 (socket says 172.21.15.153:40754) 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141b7fe640 1 -- 172.21.15.153:0/2128791156 learned_addr learned my addr 172.21.15.153:0/2128791156 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 10 monclient(hunting): get_auth_request con 0x7f141c0a1ef0 auth_method 0 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.220+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 10 monclient(hunting): handle_auth_done global_id 114435 payload 274 2024-09-15T03:37:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 1 -- 172.21.15.153:0/2128791156 >> v1:172.21.15.181:6789/0 conn(0x7f141c136d00 legacy=0x7f141c1390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 1 -- 172.21.15.153:0/2128791156 >> v1:172.21.15.153:6789/0 conn(0x7f141c09abd0 legacy=0x7f141c09afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 1 monclient: found mon.noname-a 2024-09-15T03:37:35.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.221+0000 7f141bfff640 1 -- 172.21.15.153:0/2128791156 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c0a22c0 con 0x7f141c0a1ef0 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.223+0000 7f141bfff640 1 -- 172.21.15.153:0/2128791156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c0a1ef0 msgr2=0x7f141c09c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.223+0000 7f141bfff640 1 --2- 172.21.15.153:0/2128791156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c0a1ef0 0x7f141c09c6a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f140c009850 tx=0x7f140c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd80c4512bb24a3a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 1 -- 172.21.15.153:0/2128791156 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c032440 con 0x7f141c0a1ef0 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: dump: 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.232 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 1 -- 172.21.15.153:0/2128791156 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c0325a0 con 0x7f141c0a1ef0 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 5 monclient: authenticate success, global_id 114435 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:35.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 4 set_mon_vals no callback set 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 1 -- 172.21.15.153:0/2128791156 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c032940 con 0x7f141c0a1ef0 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f141affd640 10 monclient: dump: 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 10 monclient: shutdown 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 -- 172.21.15.153:0/2128791156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c0a1ef0 msgr2=0x7f141c09c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 --2- 172.21.15.153:0/2128791156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c0a1ef0 0x7f141c09c6a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f140c009850 tx=0x7f140c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 -- 172.21.15.153:0/2128791156 shutdown_connections 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 --2- 172.21.15.153:0/2128791156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c0a1ef0 0x7f141c09c6a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 -- 172.21.15.153:0/2128791156 >> 172.21.15.153:0/2128791156 conn(0x7f141c0678a0 msgr2=0x7f141c0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 -- 172.21.15.153:0/2128791156 shutdown_connections 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.224+0000 7f142295d640 1 -- 172.21.15.153:0/2128791156 wait complete. 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.225+0000 7f142295d640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.225+0000 7f142295d640 10 monclient: monmap: 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.238 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.225+0000 7f142295d640 1 Processor -- start 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.225+0000 7f142295d640 1 -- start start 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.225+0000 7f142295d640 10 monclient: init 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 0x7f141c1354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): picked mon.noname-a con 0x7f141c09abd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f141c0a1ef0 0x7f141c131730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): picked mon.noname-c con 0x7f141c0a1ef0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141c136d00 0x7f141c131c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): picked mon.noname-b con 0x7f141c136d00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c1347e0 con 0x7f141c09abd0 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c134da0 con 0x7f141c136d00 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141c132c40 con 0x7f141c0a1ef0 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f142295d640 10 monclient(hunting): authenticate will time out at 1252.865257s 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 0x7f141c1354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 0x7f141c1354a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54192/0 (socket says 172.21.15.153:54192) 2024-09-15T03:37:35.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 -- 172.21.15.153:0/1678656423 learned_addr learned my addr 172.21.15.153:0/1678656423 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): get_auth_request con 0x7f141c09abd0 auth_method 0 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f141c0a1ef0 0x7f141c131730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141c136d00 0x7f141c131c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): get_auth_request con 0x7f141c0a1ef0 auth_method 0 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): get_auth_request con 0x7f141c136d00 auth_method 0 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f1420efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient(hunting): handle_auth_done global_id 114438 payload 824 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f141c0a1ef0 msgr2=0x7f141c131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f141c0a1ef0 0x7f141c131730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141c136d00 msgr2=0x7f141c131c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f141c136d00 0x7f141c131c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 monclient: found mon.noname-a 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.226+0000 7f141bfff640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c009390 con 0x7f141c09abd0 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.228+0000 7f141bfff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 msgr2=0x7f141c1354a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.228+0000 7f141bfff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 0x7f141c1354a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f140c009850 tx=0x7f140c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c0318c0 con 0x7f141c09abd0 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: dump: 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.244 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c031a40 con 0x7f141c09abd0 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 5 monclient: authenticate success, global_id 114438 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 10 monclient: _renew_subs 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c031e00 con 0x7f141c09abd0 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c133880 con 0x7f141c09abd0 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 10 monclient: dump: 2024-09-15T03:37:35.245 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f141a7fc640 4 set_mon_vals no callback set 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 10 monclient: _renew_subs 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141c135b80 con 0x7f141c09abd0 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f140c039040 con 0x7f141c09abd0 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.229+0000 7f142295d640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 0x7f141c095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f142295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c065e00 con 0x7f141c1321b0 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 0x7f141c095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient: get_auth_request con 0x7f141c1321b0 auth_method 0 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f1403fff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1408038740 0x7f140803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f140c06dad0 con 0x7f141c09abd0 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 10 monclient(hunting): handle_auth_done global_id 114441 payload 824 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f141b7fe640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f141c134950 con 0x7f141c1321b0 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f1420efa640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1408038740 0x7f140803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.230+0000 7f1420efa640 10 monclient: get_auth_request con 0x7f1408038740 auth_method 0 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f141b7fe640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 msgr2=0x7f141c095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f141b7fe640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 0x7f141c095430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f141000e7b0 tx=0x7f1410006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141000cac0 con 0x7f141c1321b0 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 10 monclient: dump: 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.251 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f141000cc20 con 0x7f141c1321b0 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 msgr2=0x7f141c095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.232+0000 7f1403fff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 0x7f141c095430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f141000e7b0 tx=0x7f1410006f50 comp rx=0 tx=0).stop 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.233+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 msgr2=0x7f141c095430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:35.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.233+0000 7f1403fff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1321b0 0x7f141c095430 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f141000e7b0 tx=0x7f1410006f50 comp rx=0 tx=0).stop 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.233+0000 7f1420efa640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1408038740 0x7f140803abf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1414007fd0 tx=0x7f14140025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f142295d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f142295d640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 0x7f141c095d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f142295d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c132580 con 0x7f141c095970 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 0x7f141c095d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient: get_auth_request con 0x7f141c095970 auth_method 0 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.238+0000 7f141b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.239+0000 7f141b7fe640 10 monclient(hunting): handle_auth_done global_id 114444 payload 824 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.239+0000 7f141b7fe640 1 -- 172.21.15.153:0/1678656423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f141c04bf60 con 0x7f141c095970 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f141b7fe640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 msgr2=0x7f141c095d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f141b7fe640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 0x7f141c095d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f141000f010 tx=0x7f141001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141001cc00 con 0x7f141c095970 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 10 monclient: dump: 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f1410022040 con 0x7f141c095970 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 msgr2=0x7f141c095d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 0x7f141c095d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f141000f010 tx=0x7f141001a000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 msgr2=0x7f141c095d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:35.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.241+0000 7f1403fff640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c095970 0x7f141c095d70 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f141000f010 tx=0x7f141001a000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 98, 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:35.256 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.257 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:35.259 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.260 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:35.263 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:35.264 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.267 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.268 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:35.268 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:35.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1408038740 msgr2=0x7f140803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1408038740 0x7f140803abf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1414007fd0 tx=0x7f14140025b0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 10 monclient: shutdown 2024-09-15T03:37:35.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 msgr2=0x7f141c1354a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 --2- 172.21.15.153:0/1678656423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c09abd0 0x7f141c1354a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f140c009850 tx=0x7f140c032000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f1420efa640 1 -- 172.21.15.153:0/1678656423 reap_dead start 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f1420efa640 1 -- 172.21.15.153:0/1678656423 reap_dead start 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 shutdown_connections 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 >> 172.21.15.153:0/1678656423 conn(0x7f141c0678a0 msgr2=0x7f141c1390e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 shutdown_connections 2024-09-15T03:37:35.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.242+0000 7f142295d640 1 -- 172.21.15.153:0/1678656423 wait complete. 2024-09-15T03:37:35.270 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:37:35.270 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:37:35.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:37:35.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.363+0000 7fea6895d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:35.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.363+0000 7fea6895d640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.363+0000 7fea6895d640 10 monclient: monmap: 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:35.364282+0000 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:35.364282+0000 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 Processor -- start 2024-09-15T03:37:35.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 -- start start 2024-09-15T03:37:35.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient: init 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fea640981c0 0x7fea64098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): picked mon.noname-c con 0x7fea640981c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea64131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fea64132170 0x7fea64067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): picked mon.noname-a con 0x7fea64132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:35.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea64098ad0 con 0x7fea64131da0 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea64131bd0 con 0x7fea64132170 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6409ca60 con 0x7fea640981c0 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea6895d640 10 monclient(hunting): authenticate will time out at 1253.003799s 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea62d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fea64132170 0x7fea64067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea61d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fea64131da0 0x7fea64067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40764/0 (socket says 172.21.15.153:40764) 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea61d9b640 1 -- 172.21.15.153:0/603233040 learned_addr learned my addr 172.21.15.153:0/603233040 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea62d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fea64132170 0x7fea64067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54224/0 (socket says 172.21.15.153:54224) 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea62d9d640 10 monclient(hunting): get_auth_request con 0x7fea64132170 auth_method 0 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea62d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.364+0000 7fea62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea6259c640 1 --2- 172.21.15.153:0/603233040 >> v2:172.21.15.181:3300/0 conn(0x7fea640981c0 0x7fea64098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 10 monclient(hunting): handle_auth_done global_id 114447 payload 274 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 1 -- 172.21.15.153:0/603233040 >> v2:172.21.15.181:3300/0 conn(0x7fea640981c0 msgr2=0x7fea64098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 1 --2- 172.21.15.153:0/603233040 >> v2:172.21.15.181:3300/0 conn(0x7fea640981c0 0x7fea64098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 1 -- 172.21.15.153:0/603233040 >> v1:172.21.15.153:6789/0 conn(0x7fea64131da0 legacy=0x7fea64067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 1 monclient: found mon.noname-a 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea62d9d640 1 -- 172.21.15.153:0/603233040 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6409cca0 con 0x7fea64132170 2024-09-15T03:37:35.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea6259c640 10 monclient: get_auth_request con 0x7fea640981c0 auth_method 0 2024-09-15T03:37:35.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.365+0000 7fea6259c640 1 --2- 172.21.15.153:0/603233040 >> v2:172.21.15.181:3300/0 conn(0x7fea640981c0 0x7fea64098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:35.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.367+0000 7fea62d9d640 1 -- 172.21.15.153:0/603233040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 msgr2=0x7fea64067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.367+0000 7fea62d9d640 1 --2- 172.21.15.153:0/603233040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea64067820 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea64066560 tx=0x7fea5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d382ed49a3489d31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 1 -- 172.21.15.153:0/603233040 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c007840 con 0x7fea64132170 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: dump: 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 5 monclient: authenticate success, global_id 114447 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 1 -- 172.21.15.153:0/603233040 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c02d050 con 0x7fea64132170 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 1 -- 172.21.15.153:0/603233040 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c007c10 con 0x7fea64132170 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 4 set_mon_vals no callback set 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6159a640 10 monclient: dump: 2024-09-15T03:37:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 10 monclient: shutdown 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 -- 172.21.15.153:0/603233040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 msgr2=0x7fea64067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 --2- 172.21.15.153:0/603233040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea64067820 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea64066560 tx=0x7fea5c009780 comp rx=0 tx=0).stop 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 -- 172.21.15.153:0/603233040 shutdown_connections 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 --2- 172.21.15.153:0/603233040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea64067820 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 --2- 172.21.15.153:0/603233040 >> v2:172.21.15.181:3300/0 conn(0x7fea640981c0 0x7fea64098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 -- 172.21.15.153:0/603233040 >> 172.21.15.153:0/603233040 conn(0x7fea640a1e30 msgr2=0x7fea640a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 -- 172.21.15.153:0/603233040 shutdown_connections 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.368+0000 7fea6895d640 1 -- 172.21.15.153:0/603233040 wait complete. 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient: monmap: 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:35.382 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 Processor -- start 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 -- start start 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient: init 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640981c0 0x7fea64140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): picked mon.noname-b con 0x7fea640981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:35.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 0x7fea64130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): picked mon.noname-c con 0x7fea64131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea6411a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): picked mon.noname-a con 0x7fea64132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6409b780 con 0x7fea64132170 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6411b960 con 0x7fea640981c0 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6411c080 con 0x7fea64131da0 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.369+0000 7fea6895d640 10 monclient(hunting): authenticate will time out at 1253.008968s 2024-09-15T03:37:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea6411a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea6411a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54234/0 (socket says 172.21.15.153:54234) 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 0x7fea64130a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 -- 172.21.15.153:0/2129951912 learned_addr learned my addr 172.21.15.153:0/2129951912 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): get_auth_request con 0x7fea64132170 auth_method 0 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): get_auth_request con 0x7fea64131da0 auth_method 0 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640981c0 0x7fea64140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): get_auth_request con 0x7fea640981c0 auth_method 0 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient(hunting): handle_auth_done global_id 114450 payload 824 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 msgr2=0x7fea64130a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 msgr2=0x7fea64130a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 0x7fea64130a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 msgr2=0x7fea64130a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea61d9b640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 0x7fea64130a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64131da0 0x7fea64130a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640981c0 msgr2=0x7fea64140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640981c0 0x7fea64140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 monclient: found mon.noname-a 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea62d9d640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c009a20 con 0x7fea64132170 2024-09-15T03:37:35.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.370+0000 7fea6259c640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea640981c0 0x7fea64140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea62d9d640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 msgr2=0x7fea6411a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea62d9d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea6411a4f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0097c0 tx=0x7fea5c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c031cd0 con 0x7fea64132170 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: dump: 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 5 monclient: authenticate success, global_id 114450 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 10 monclient: _renew_subs 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c040070 con 0x7fea64132170 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6411db40 con 0x7fea64132170 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c045690 con 0x7fea64132170 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea527fc640 10 monclient: dump: 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.390 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea60d99640 4 set_mon_vals no callback set 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 10 monclient: _renew_subs 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.372+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea641414e0 con 0x7fea64132170 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6895d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6895d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 0x7fea6411aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea64065f90 con 0x7fea64130f60 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fea5c042030 con 0x7fea64132170 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 0x7fea6411aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient: get_auth_request con 0x7fea64130f60 auth_method 0 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.373+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.374+0000 7fea6259c640 10 monclient(hunting): handle_auth_done global_id 124136 payload 824 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.374+0000 7fea527fc640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea44038740 0x7fea4403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.374+0000 7fea6259c640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea6404bf60 con 0x7fea64130f60 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.374+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7fea5c06e740 con 0x7fea64132170 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.374+0000 7fea61d9b640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea44038740 0x7fea4403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.374+0000 7fea61d9b640 10 monclient: get_auth_request con 0x7fea44038740 auth_method 0 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea6259c640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 msgr2=0x7fea6411aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea6259c640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 0x7fea6411aa30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea5400aeb0 tx=0x7fea54008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea54013650 con 0x7fea64130f60 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 10 monclient: dump: 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fea5400c040 con 0x7fea64130f60 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 msgr2=0x7fea6411aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 0x7fea6411aa30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea5400aeb0 tx=0x7fea54008040 comp rx=0 tx=0).stop 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 msgr2=0x7fea6411aa30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea527fc640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea64130f60 0x7fea6411aa30 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea5400aeb0 tx=0x7fea54008040 comp rx=0 tx=0).stop 2024-09-15T03:37:35.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.376+0000 7fea61d9b640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea44038740 0x7fea4403abf0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fea640a0200 tx=0x7fea58007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6895d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6895d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 0x7fea641338b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6895d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea64131330 con 0x7fea6411af70 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 0x7fea641338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient: get_auth_request con 0x7fea6411af70 auth_method 0 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 10 monclient(hunting): handle_auth_done global_id 124139 payload 824 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.382+0000 7fea6259c640 1 -- 172.21.15.153:0/2129951912 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fea6411b370 con 0x7fea6411af70 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6259c640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 msgr2=0x7fea641338b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6259c640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 0x7fea641338b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea5400aeb0 tx=0x7fea5401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5401bea0 con 0x7fea6411af70 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:35.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 10 monclient: dump: 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fea540123c0 con 0x7fea6411af70 2024-09-15T03:37:35.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 msgr2=0x7fea641338b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 0x7fea641338b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea5400aeb0 tx=0x7fea5401d000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 msgr2=0x7fea641338b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea527fc640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea6411af70 0x7fea641338b0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea5400aeb0 tx=0x7fea5401d000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:35.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 98, 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.401 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:35.402 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.403 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.404 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.405 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.406 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.410 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:35.411 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.411 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.411 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:35.411 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea44038740 msgr2=0x7fea4403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fea44038740 0x7fea4403abf0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fea640a0200 tx=0x7fea58007f00 comp rx=0 tx=0).stop 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 10 monclient: shutdown 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 msgr2=0x7fea6411a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea6411a4f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0097c0 tx=0x7fea5c037000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea62d9d640 1 -- 172.21.15.153:0/2129951912 reap_dead start 2024-09-15T03:37:35.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 shutdown_connections 2024-09-15T03:37:35.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 --2- 172.21.15.153:0/2129951912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea64132170 0x7fea6411a4f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 >> 172.21.15.153:0/2129951912 conn(0x7fea640a1e30 msgr2=0x7fea64096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 shutdown_connections 2024-09-15T03:37:35.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.385+0000 7fea6895d640 1 -- 172.21.15.153:0/2129951912 wait complete. 2024-09-15T03:37:35.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:37:35.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:37:35.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.506+0000 7f603ae17640 10 monclient: get_monmap_and_config 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.506+0000 7f603ae17640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.506+0000 7f603ae17640 10 monclient: monmap: 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:35.506914+0000 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:35.506914+0000 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.507 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.506+0000 7f603ae17640 1 Processor -- start 2024-09-15T03:37:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.506+0000 7f603ae17640 1 -- start start 2024-09-15T03:37:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.506+0000 7f603ae17640 10 monclient: init 2024-09-15T03:37:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:37:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6034096e10 0x7f603409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): picked mon.noname-a con 0x7f6034096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f603409aaa0 0x7f60340971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): picked mon.noname-c con 0x7f603409aaa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6034097720 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6034099620 con 0x7f6034097720 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6034099340 con 0x7f6034096e10 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6034099900 con 0x7f603409aaa0 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f603ae17640 10 monclient(hunting): authenticate will time out at 1253.146499s 2024-09-15T03:37:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6034096e10 0x7f603409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f60393b4640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6034097720 0x7f603412e5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40768/0 (socket says 172.21.15.153:40768) 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6034096e10 0x7f603409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54248/0 (socket says 172.21.15.153:54248) 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f60393b4640 1 -- 172.21.15.153:0/4139216563 learned_addr learned my addr 172.21.15.153:0/4139216563 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): get_auth_request con 0x7f6034096e10 auth_method 0 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6033fff640 1 --2- 172.21.15.153:0/4139216563 >> v2:172.21.15.181:3300/0 conn(0x7f603409aaa0 0x7f60340971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6033fff640 10 monclient(hunting): get_auth_request con 0x7f603409aaa0 auth_method 0 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6033fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient(hunting): handle_auth_done global_id 114453 payload 274 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 -- 172.21.15.153:0/4139216563 >> v2:172.21.15.181:3300/0 conn(0x7f603409aaa0 msgr2=0x7f60340971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 --2- 172.21.15.153:0/4139216563 >> v2:172.21.15.181:3300/0 conn(0x7f603409aaa0 0x7f60340971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 -- 172.21.15.153:0/4139216563 >> v1:172.21.15.153:6789/0 conn(0x7f6034097720 legacy=0x7f603412e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 monclient: found mon.noname-a 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.507+0000 7f6038bb3640 1 -- 172.21.15.153:0/4139216563 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6034097b20 con 0x7f6034096e10 2024-09-15T03:37:35.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f6038bb3640 1 -- 172.21.15.153:0/4139216563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 msgr2=0x7f603409a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f6038bb3640 1 --2- 172.21.15.153:0/4139216563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603409a560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f603409aff0 tx=0x7f60280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7be9a24e61068d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 1 -- 172.21.15.153:0/4139216563 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602802ae00 con 0x7f6034096e10 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: dump: 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f603ae17640 5 monclient: authenticate success, global_id 114453 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 1 -- 172.21.15.153:0/4139216563 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028032440 con 0x7f6034096e10 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f603ae17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f603ae17640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f603ae17640 4 set_mon_vals no callback set 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 1 -- 172.21.15.153:0/4139216563 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60280327e0 con 0x7f6034096e10 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.510+0000 7f60337fe640 10 monclient: dump: 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 10 monclient: shutdown 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 -- 172.21.15.153:0/4139216563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 msgr2=0x7f603409a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 --2- 172.21.15.153:0/4139216563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603409a560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f603409aff0 tx=0x7f60280090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 -- 172.21.15.153:0/4139216563 shutdown_connections 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 --2- 172.21.15.153:0/4139216563 >> v2:172.21.15.181:3300/0 conn(0x7f603409aaa0 0x7f60340971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 --2- 172.21.15.153:0/4139216563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603409a560 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 -- 172.21.15.153:0/4139216563 >> 172.21.15.153:0/4139216563 conn(0x7f60340672b0 msgr2=0x7f60340676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 -- 172.21.15.153:0/4139216563 shutdown_connections 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 1 -- 172.21.15.153:0/4139216563 wait complete. 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.511+0000 7f603ae17640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient: monmap: 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:35.523 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 Processor -- start 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 -- start start 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient: init 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603412d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): picked mon.noname-a con 0x7f6034096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:35.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6034097720 0x7f603411cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): picked mon.noname-b con 0x7f6034097720 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f603409aaa0 0x7f603411d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): picked mon.noname-c con 0x7f603409aaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603411ea90 con 0x7f6034096e10 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6034120300 con 0x7f6034097720 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603409b050 con 0x7f603409aaa0 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.512+0000 7f603ae17640 10 monclient(hunting): authenticate will time out at 1253.151984s 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603412d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603412d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54258/0 (socket says 172.21.15.153:54258) 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 -- 172.21.15.153:0/3073899077 learned_addr learned my addr 172.21.15.153:0/3073899077 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): get_auth_request con 0x7f6034096e10 auth_method 0 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f603409aaa0 0x7f603411d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): get_auth_request con 0x7f603409aaa0 auth_method 0 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6034097720 0x7f603411cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): get_auth_request con 0x7f6034097720 auth_method 0 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient(hunting): handle_auth_done global_id 114456 payload 824 2024-09-15T03:37:35.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f603409aaa0 msgr2=0x7f603411d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f603409aaa0 0x7f603411d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6034097720 msgr2=0x7f603411cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6034097720 0x7f603411cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 monclient: found mon.noname-a 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f6038bb3640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028009390 con 0x7f6034096e10 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.513+0000 7f60393b4640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f603409aaa0 0x7f603411d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6038bb3640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 msgr2=0x7f603412d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6038bb3640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603412d010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6028009360 tx=0x7f6028032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028031c70 con 0x7f6034096e10 2024-09-15T03:37:35.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: dump: 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f603ae17640 5 monclient: authenticate success, global_id 114456 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028031dd0 con 0x7f6034096e10 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f603ae17640 10 monclient: _renew_subs 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602803d650 con 0x7f6034096e10 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f603ae17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6034120020 con 0x7f6034096e10 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:37:35.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6030ff9640 10 monclient: dump: 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.515+0000 7f6032ffd640 4 set_mon_vals no callback set 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 10 monclient: _renew_subs 2024-09-15T03:37:35.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6034120ce0 con 0x7f6034096e10 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6028039040 con 0x7f6034096e10 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 0x7f603402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.516+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6034065f20 con 0x7f603411f0c0 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6030ff9640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6018038740 0x7f601803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f602806cd40 con 0x7f6034096e10 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 0x7f603402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient: get_auth_request con 0x7f603411f0c0 auth_method 0 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f60393b4640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6018038740 0x7f601803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f60393b4640 10 monclient: get_auth_request con 0x7f6018038740 auth_method 0 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 10 monclient(hunting): handle_auth_done global_id 104647 payload 824 2024-09-15T03:37:35.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.517+0000 7f6033fff640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6034120500 con 0x7f603411f0c0 2024-09-15T03:37:35.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.519+0000 7f6033fff640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 msgr2=0x7f603402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.519+0000 7f6033fff640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 0x7f603402aa40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60200108b0 tx=0x7f6020008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.519+0000 7f60393b4640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6018038740 0x7f601803abf0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f602c00d6a0 tx=0x7f602c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602000ed50 con 0x7f603411f0c0 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 10 monclient: dump: 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.536 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f602000eeb0 con 0x7f603411f0c0 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 msgr2=0x7f603402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 0x7f603402aa40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60200108b0 tx=0x7f6020008040 comp rx=0 tx=0).stop 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 msgr2=0x7f603402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.520+0000 7f6030ff9640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411f0c0 0x7f603402aa40 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60200108b0 tx=0x7f6020008040 comp rx=0 tx=0).stop 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.525+0000 7f603ae17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.525+0000 7f603ae17640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 0x7f603405e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.525+0000 7f603ae17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.525+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603411d9b0 con 0x7f603411d5b0 2024-09-15T03:37:35.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 0x7f603405e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient: get_auth_request con 0x7f603411d5b0 auth_method 0 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 10 monclient(hunting): handle_auth_done global_id 104650 payload 824 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.526+0000 7f6033fff640 1 -- 172.21.15.153:0/3073899077 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f603404bf60 con 0x7f603411d5b0 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.528+0000 7f6033fff640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 msgr2=0x7f603405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.528+0000 7f6033fff640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 0x7f603405e310 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6020019000 tx=0x7f602001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602002a070 con 0x7f603411d5b0 2024-09-15T03:37:35.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 10 monclient: dump: 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f6020022040 con 0x7f603411d5b0 2024-09-15T03:37:35.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 msgr2=0x7f603405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 0x7f603405e310 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6020019000 tx=0x7f602001c000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 msgr2=0x7f603405e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f6030ff9640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f603411d5b0 0x7f603405e310 secure :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6020019000 tx=0x7f602001c000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:37:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 98, 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:35.547 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:35.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:35.550 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.551 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.552 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:35.553 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6018038740 msgr2=0x7f601803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6018038740 0x7f601803abf0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f602c00d6a0 tx=0x7f602c009040 comp rx=0 tx=0).stop 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 10 monclient: shutdown 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 msgr2=0x7f603412d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 --2- 172.21.15.153:0/3073899077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034096e10 0x7f603412d010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6028009360 tx=0x7f6028032000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f60393b4640 1 -- 172.21.15.153:0/3073899077 reap_dead start 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f60393b4640 1 -- 172.21.15.153:0/3073899077 reap_dead start 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 shutdown_connections 2024-09-15T03:37:35.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 >> 172.21.15.153:0/3073899077 conn(0x7f60340672b0 msgr2=0x7f603412dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 shutdown_connections 2024-09-15T03:37:35.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.529+0000 7f603ae17640 1 -- 172.21.15.153:0/3073899077 wait complete. 2024-09-15T03:37:35.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:37:35.556 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:37:35.556 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:37:35.556 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:37:35.556 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:37:35.557 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:37:35.557 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:37:35.569 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:37:35.569 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:37:35.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.700+0000 7f29d555d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.700+0000 7f29d555d640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.701+0000 7f29d555d640 10 monclient: monmap: 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:35.701728+0000 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:35.701728+0000 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.702 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.701+0000 7f29d555d640 1 Processor -- start 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.701+0000 7f29d555d640 1 -- start start 2024-09-15T03:37:35.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.701+0000 7f29d555d640 10 monclient: init 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f29d01318f0 0x7f29d0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): picked mon.noname-c con 0x7f29d01318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29d009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29d0136c20 0x7f29d0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): picked mon.noname-a con 0x7f29d0136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d009f530 con 0x7f29d009ea50 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d0131d50 con 0x7f29d0136c20 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d009fbf0 con 0x7f29d01318f0 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29d555d640 10 monclient(hunting): authenticate will time out at 1253.341082s 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29cf7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f29d0136c20 msgr2=0x7f29d0139000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29cf7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29d0136c20 0x7f29d0139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f29d01318f0 0x7f29d0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ce7fc640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f29d009ea50 0x7f29d009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:36590/0 (socket says 172.21.15.153:36590) 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ce7fc640 1 -- 172.21.15.153:0/3274086022 learned_addr learned my addr 172.21.15.153:0/3274086022 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:35.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): get_auth_request con 0x7f29d01318f0 auth_method 0 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient(hunting): handle_auth_done global_id 104653 payload 274 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 1 -- 172.21.15.153:0/3274086022 >> v2:172.21.15.153:3300/0 conn(0x7f29d0136c20 msgr2=0x7f29d0139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 1 --2- 172.21.15.153:0/3274086022 >> v2:172.21.15.153:3300/0 conn(0x7f29d0136c20 0x7f29d0139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 1 -- 172.21.15.153:0/3274086022 >> v1:172.21.15.181:6789/0 conn(0x7f29d009ea50 legacy=0x7f29d009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 1 monclient: found mon.noname-c 2024-09-15T03:37:35.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:35.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.702+0000 7f29ceffd640 1 -- 172.21.15.153:0/3274086022 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d009fe30 con 0x7f29d01318f0 2024-09-15T03:37:35.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.705+0000 7f29ceffd640 1 -- 172.21.15.153:0/3274086022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 msgr2=0x7f29d0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.705+0000 7f29ceffd640 1 --2- 172.21.15.153:0/3274086022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d0131f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29d0076600 tx=0x7f29c80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dced32863d6ee17c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 1 -- 172.21.15.153:0/3274086022 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8032440 con 0x7f29d01318f0 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: dump: 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 1 -- 172.21.15.153:0/3274086022 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c80325a0 con 0x7f29d01318f0 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 1 -- 172.21.15.153:0/3274086022 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8032940 con 0x7f29d01318f0 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 5 monclient: authenticate success, global_id 104653 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 4 set_mon_vals no callback set 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:35.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29cdffb640 10 monclient: dump: 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.713 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 10 monclient: shutdown 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 -- 172.21.15.153:0/3274086022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 msgr2=0x7f29d0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 --2- 172.21.15.153:0/3274086022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d0131f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29d0076600 tx=0x7f29c80090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 -- 172.21.15.153:0/3274086022 shutdown_connections 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 --2- 172.21.15.153:0/3274086022 >> v2:172.21.15.153:3300/0 conn(0x7f29d0136c20 0x7f29d0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 --2- 172.21.15.153:0/3274086022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d0131f30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 -- 172.21.15.153:0/3274086022 >> 172.21.15.153:0/3274086022 conn(0x7f29d0067110 msgr2=0x7f29d0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 -- 172.21.15.153:0/3274086022 shutdown_connections 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.706+0000 7f29d555d640 1 -- 172.21.15.153:0/3274086022 wait complete. 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.707+0000 7f29d555d640 10 monclient: build_initial_monmap 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.707+0000 7f29d555d640 10 monclient: monmap: 2024-09-15T03:37:35.719 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.707+0000 7f29d555d640 1 Processor -- start 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.707+0000 7f29d555d640 1 -- start start 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.707+0000 7f29d555d640 10 monclient: init 2024-09-15T03:37:35.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 0x7f29d0130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): picked mon.noname-c con 0x7f29d009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d00998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): picked mon.noname-b con 0x7f29d01318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0136c20 0x7f29d0099df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): picked mon.noname-a con 0x7f29d0136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:35.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d009b9c0 con 0x7f29d0136c20 2024-09-15T03:37:35.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d009bf10 con 0x7f29d01318f0 2024-09-15T03:37:35.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:35.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d009e140 con 0x7f29d009ea50 2024-09-15T03:37:35.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:35.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29d555d640 10 monclient(hunting): authenticate will time out at 1253.347453s 2024-09-15T03:37:35.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29cf7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0136c20 msgr2=0x7f29d0099df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:35.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29cf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0136c20 0x7f29d0099df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:35.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 0x7f29d0130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 0x7f29d0130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:39860/0 (socket says 172.21.15.153:39860) 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 -- 172.21.15.153:0/1079587438 learned_addr learned my addr 172.21.15.153:0/1079587438 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): get_auth_request con 0x7f29d009ea50 auth_method 0 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:35.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ce7fc640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d00998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ce7fc640 10 monclient(hunting): get_auth_request con 0x7f29d01318f0 auth_method 0 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:35.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient(hunting): handle_auth_done global_id 124145 payload 824 2024-09-15T03:37:35.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 -- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 msgr2=0x7f29d00998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d00998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 -- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0136c20 msgr2=0x7f29d0099df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:35.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0136c20 0x7f29d0099df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:35.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 monclient: found mon.noname-c 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ceffd640 1 -- 172.21.15.153:0/1079587438 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8009390 con 0x7f29d009ea50 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.708+0000 7f29ce7fc640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d00998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29ceffd640 1 -- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 msgr2=0x7f29d0130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29ceffd640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 0x7f29d0130980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f29c802a650 tx=0x7f29c8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8031940 con 0x7f29d009ea50 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: dump: 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.728 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.729 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.729 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.729 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.729 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.729 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.729 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c8031aa0 con 0x7f29d009ea50 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 5 monclient: authenticate success, global_id 124145 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 10 monclient: _renew_subs 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d0141ed0 con 0x7f29d009ea50 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8031e20 con 0x7f29d009ea50 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29c6ffd640 10 monclient: dump: 2024-09-15T03:37:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 10 monclient: _renew_subs 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29cd7fa640 4 set_mon_vals no callback set 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.711+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d011ca90 con 0x7f29d009ea50 2024-09-15T03:37:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f29c8039070 con 0x7f29d009ea50 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29d555d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29d555d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29d555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d009b500 con 0x7f29d009ea50 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29c6ffd640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29b00386f0 0x7f29b003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f29c806d5d0 con 0x7f29d009ea50 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29ce7fc640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29b00386f0 0x7f29b003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.712+0000 7f29ce7fc640 10 monclient: get_auth_request con 0x7f29b00386f0 auth_method 0 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.715+0000 7f29ce7fc640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29b00386f0 0x7f29b003aba0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f29b8007fd0 tx=0x7f29b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.716+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f29c803c620 con 0x7f29d009ea50 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.716+0000 7f29c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:35.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.716+0000 7f29c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:35.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29d555d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:35.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29d555d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:35.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29d555d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:35.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f29d004bf60 con 0x7f29d009ea50 2024-09-15T03:37:35.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29c6ffd640 1 -- 172.21.15.153:0/1079587438 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f29c8040020 con 0x7f29d009ea50 2024-09-15T03:37:35.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:35.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.929+0000 7f29c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:35.931 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:35.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29b00386f0 msgr2=0x7f29b003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29b00386f0 0x7f29b003aba0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f29b8007fd0 tx=0x7f29b8009040 comp rx=0 tx=0).stop 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 10 monclient: shutdown 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 msgr2=0x7f29d0130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 0x7f29d0130980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f29c802a650 tx=0x7f29c8032000 comp rx=0 tx=0).stop 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 shutdown_connections 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f29b00386f0 0x7f29b003aba0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0136c20 0x7f29d0099df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d01318f0 0x7f29d00998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 --2- 172.21.15.153:0/1079587438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29d009ea50 0x7f29d0130980 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 >> 172.21.15.153:0/1079587438 conn(0x7f29d0067110 msgr2=0x7f29d00a4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 shutdown_connections 2024-09-15T03:37:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:35.931+0000 7f29d555d640 1 -- 172.21.15.153:0/1079587438 wait complete. 2024-09-15T03:37:37.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:37.758+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:37.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:37.758+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:37.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:37.758+0000 7fd25d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:38.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:38.758+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:38.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:38.758+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:38.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.047+0000 7f517b39d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.047+0000 7f517b39d640 10 monclient: build_initial_monmap 2024-09-15T03:37:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.047+0000 7f517b39d640 10 monclient: monmap: 2024-09-15T03:37:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:39.048316+0000 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:39.048316+0000 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:39.049 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:39.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 Processor -- start 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 -- start start 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient: init 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5174067110 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f517409b010 0x7f517409b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f517409b010 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:39.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f51741349a0 0x7f5174136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f51741349a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5174130540 con 0x7f5174067110 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5174067ed0 con 0x7f517409b010 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f5179139640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f5174067110 legacy=0x7f517412f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5174099800 con 0x7f51741349a0 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f517b39d640 10 monclient(hunting): authenticate will time out at 1256.687995s 2024-09-15T03:37:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.048+0000 7f5178938640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f517409b010 msgr2=0x7f517409b430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f5178938640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f517409b010 0x7f517409b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f51741349a0 0x7f5174136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f51741349a0 0x7f5174136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:34842/0 (socket says 172.21.15.153:34842) 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 -- 172.21.15.153:0/715653648 learned_addr learned my addr 172.21.15.153:0/715653648 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): get_auth_request con 0x7f51741349a0 auth_method 0 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:39.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient(hunting): handle_auth_done global_id 104659 payload 274 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 -- 172.21.15.153:0/715653648 >> v2:172.21.15.153:3300/0 conn(0x7f517409b010 msgr2=0x7f517409b430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 --2- 172.21.15.153:0/715653648 >> v2:172.21.15.153:3300/0 conn(0x7f517409b010 0x7f517409b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 -- 172.21.15.153:0/715653648 >> v1:172.21.15.153:6789/0 conn(0x7f5174067110 legacy=0x7f517412f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 monclient: found mon.noname-c 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.049+0000 7f517993a640 1 -- 172.21.15.153:0/715653648 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51740674e0 con 0x7f51741349a0 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f517993a640 1 -- 172.21.15.153:0/715653648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51741349a0 msgr2=0x7f5174136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f517993a640 1 --2- 172.21.15.153:0/715653648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51741349a0 0x7f5174136d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5174067de0 tx=0x7f516c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359df249d2a4a1b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 1 -- 172.21.15.153:0/715653648 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c007610 con 0x7f51741349a0 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: dump: 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.062 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: _finish_auth 0 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 1 -- 172.21.15.153:0/715653648 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c02d050 con 0x7f51741349a0 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f517b39d640 5 monclient: authenticate success, global_id 104659 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f517b39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:39.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f517b39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 1 -- 172.21.15.153:0/715653648 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c007980 con 0x7f51741349a0 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f517b39d640 4 set_mon_vals no callback set 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.052+0000 7f516bfff640 10 monclient: dump: 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:39.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 10 monclient: shutdown 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 -- 172.21.15.153:0/715653648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51741349a0 msgr2=0x7f5174136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 --2- 172.21.15.153:0/715653648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51741349a0 0x7f5174136d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5174067de0 tx=0x7f516c009840 comp rx=0 tx=0).stop 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 -- 172.21.15.153:0/715653648 shutdown_connections 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 --2- 172.21.15.153:0/715653648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51741349a0 0x7f5174136d80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 --2- 172.21.15.153:0/715653648 >> v2:172.21.15.153:3300/0 conn(0x7f517409b010 0x7f517409b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 -- 172.21.15.153:0/715653648 >> 172.21.15.153:0/715653648 conn(0x7f51740a1e30 msgr2=0x7f51740a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 -- 172.21.15.153:0/715653648 shutdown_connections 2024-09-15T03:37:39.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.053+0000 7f517b39d640 1 -- 172.21.15.153:0/715653648 wait complete. 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.054+0000 7f517b39d640 10 monclient: build_initial_monmap 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.054+0000 7f517b39d640 10 monclient: monmap: 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:39.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.054+0000 7f517b39d640 1 Processor -- start 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.054+0000 7f517b39d640 1 -- start start 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.054+0000 7f517b39d640 10 monclient: init 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 0x7f517406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f5174067110 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f517409b010 0x7f517406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f517409b010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:39.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741349a0 0x7f517406cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f51741349a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517409e330 con 0x7f51741349a0 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740a02c0 con 0x7f517409b010 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5174118750 con 0x7f5174067110 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517b39d640 10 monclient(hunting): authenticate will time out at 1256.694134s 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517993a640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741349a0 msgr2=0x7f517406cf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f517993a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741349a0 0x7f517406cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 0x7f517406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:39.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 0x7f517406c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:39874/0 (socket says 172.21.15.153:39874) 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 -- 172.21.15.153:0/3744049629 learned_addr learned my addr 172.21.15.153:0/3744049629 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): get_auth_request con 0x7f5174067110 auth_method 0 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f517409b010 0x7f517406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): get_auth_request con 0x7f517409b010 auth_method 0 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:39.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5178938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient(hunting): handle_auth_done global_id 124148 payload 824 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 -- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f517409b010 msgr2=0x7f517406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f517409b010 0x7f517406ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 -- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741349a0 msgr2=0x7f517406cf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741349a0 0x7f517406cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient: _finish_hunting 0 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 monclient: found mon.noname-c 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.055+0000 7f5179139640 1 -- 172.21.15.153:0/3744049629 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c009ae0 con 0x7f5174067110 2024-09-15T03:37:39.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f5179139640 1 -- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 msgr2=0x7f517406c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f5179139640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 0x7f517406c4f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f517000b600 tx=0x7f51700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5170015070 con 0x7f5174067110 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: dump: 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:39.071 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5170003d90 con 0x7f5174067110 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f517b39d640 5 monclient: authenticate success, global_id 124148 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f517b39d640 10 monclient: _renew_subs 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f517b39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5174118490 con 0x7f5174067110 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5170006380 con 0x7f5174067110 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f51697fa640 10 monclient: dump: 2024-09-15T03:37:39.072 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.057+0000 7f516b7fe640 4 set_mon_vals no callback set 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.058+0000 7f517b39d640 10 monclient: _renew_subs 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.058+0000 7f517b39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.058+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517406e080 con 0x7f5174067110 2024-09-15T03:37:39.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.058+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5170010440 con 0x7f5174067110 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.058+0000 7f51697fa640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f51580386f0 0x7f515803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f517004ca20 con 0x7f5174067110 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f517b39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f517b39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f517b39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517404bf60 con 0x7f5174067110 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f5178938640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f51580386f0 0x7f515803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:39.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.059+0000 7f5178938640 10 monclient: get_auth_request con 0x7f51580386f0 auth_method 0 2024-09-15T03:37:39.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.061+0000 7f5178938640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f51580386f0 0x7f515803aba0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f517406dc80 tx=0x7f51640090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:39.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.063+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5170018ab0 con 0x7f5174067110 2024-09-15T03:37:39.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.063+0000 7f51697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:39.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.063+0000 7f51697fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:39.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f517b39d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:39.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f517b39d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:39.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f517b39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:39.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5174118890 con 0x7f5174067110 2024-09-15T03:37:39.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f51697fa640 1 -- 172.21.15.153:0/3744049629 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f5170018690 con 0x7f5174067110 2024-09-15T03:37:39.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f51697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:39.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.278+0000 7f51697fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:39.280 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f51580386f0 msgr2=0x7f515803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f51580386f0 0x7f515803aba0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f517406dc80 tx=0x7f51640090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 10 monclient: shutdown 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 msgr2=0x7f517406c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 0x7f517406c4f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f517000b600 tx=0x7f51700091a0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 shutdown_connections 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f51580386f0 0x7f515803aba0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51741349a0 0x7f517406cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f517409b010 0x7f517406ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.280+0000 7f517b39d640 1 --2- 172.21.15.153:0/3744049629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5174067110 0x7f517406c4f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:39.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.281+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 >> 172.21.15.153:0/3744049629 conn(0x7f51740a1e30 msgr2=0x7f517409c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:39.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.281+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 shutdown_connections 2024-09-15T03:37:39.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:39.281+0000 7f517b39d640 1 -- 172.21.15.153:0/3744049629 wait complete. 2024-09-15T03:37:39.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:39.758+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:39.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:39.758+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:40.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:40.758+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:40.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:40.758+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:41.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:41.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:41.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:41.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:42.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:42.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.396+0000 7f4dcb46f640 10 monclient: get_monmap_and_config 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.396+0000 7f4dcb46f640 10 monclient: build_initial_monmap 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.396+0000 7f4dcb46f640 10 monclient: monmap: 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:42.397442+0000 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:42.397442+0000 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:42.398 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:42.399 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:42.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 1 Processor -- start 2024-09-15T03:37:42.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 1 -- start start 2024-09-15T03:37:42.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 10 monclient: init 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4dc41318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4dc409ea50 0x7f4dc409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4dc4136c20 0x7f4dc4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.397+0000 7f4dcb46f640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc4136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc4131cc0 con 0x7f4dc41318f0 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc409fbf0 con 0x7f4dc4136c20 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:42.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc409fe30 con 0x7f4dc409ea50 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dcb46f640 10 monclient(hunting): authenticate will time out at 1260.037025s 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc9a0c640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f4dc4136c20 msgr2=0x7f4dc4139000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc9a0c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4dc4136c20 0x7f4dc4139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4dc409ea50 0x7f4dc409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc920b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4dc41318f0 0x7f4dc4131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:36592/0 (socket says 172.21.15.153:36592) 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc920b640 1 -- 172.21.15.153:0/1218516318 learned_addr learned my addr 172.21.15.153:0/1218516318 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): get_auth_request con 0x7f4dc409ea50 auth_method 0 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_done global_id 104665 payload 274 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/1218516318 >> v2:172.21.15.153:3300/0 conn(0x7f4dc4136c20 msgr2=0x7f4dc4139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 1 --2- 172.21.15.153:0/1218516318 >> v2:172.21.15.153:3300/0 conn(0x7f4dc4136c20 0x7f4dc4139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/1218516318 >> v1:172.21.15.181:6789/0 conn(0x7f4dc41318f0 legacy=0x7f4dc4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 1 monclient: found mon.noname-c 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.398+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/1218516318 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40a1210 con 0x7f4dc409ea50 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/1218516318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc409ea50 msgr2=0x7f4dc409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:42.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dc8a0a640 1 --2- 172.21.15.153:0/1218516318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc409ea50 0x7f4dc409ee20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4dc406f600 tx=0x7f4db40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7556689522b1b8e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 1 -- 172.21.15.153:0/1218516318 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4032440 con 0x7f4dc409ea50 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: dump: 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:42.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: _finish_auth 0 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 1 -- 172.21.15.153:0/1218516318 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db40325a0 con 0x7f4dc409ea50 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dcb46f640 5 monclient: authenticate success, global_id 104665 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dcb46f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dcb46f640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:42.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 1 -- 172.21.15.153:0/1218516318 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4032940 con 0x7f4dc409ea50 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dcb46f640 4 set_mon_vals no callback set 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.401+0000 7f4dbbfff640 10 monclient: dump: 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 10 monclient: shutdown 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 -- 172.21.15.153:0/1218516318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc409ea50 msgr2=0x7f4dc409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/1218516318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc409ea50 0x7f4dc409ee20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4dc406f600 tx=0x7f4db40090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 -- 172.21.15.153:0/1218516318 shutdown_connections 2024-09-15T03:37:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/1218516318 >> v2:172.21.15.153:3300/0 conn(0x7f4dc4136c20 0x7f4dc4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/1218516318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc409ea50 0x7f4dc409ee20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 -- 172.21.15.153:0/1218516318 >> 172.21.15.153:0/1218516318 conn(0x7f4dc4067110 msgr2=0x7f4dc4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 -- 172.21.15.153:0/1218516318 shutdown_connections 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.402+0000 7f4dcb46f640 1 -- 172.21.15.153:0/1218516318 wait complete. 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.403+0000 7f4dcb46f640 10 monclient: build_initial_monmap 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.403+0000 7f4dcb46f640 10 monclient: monmap: 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:42.417 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.403+0000 7f4dcb46f640 1 Processor -- start 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.403+0000 7f4dcb46f640 1 -- start start 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.403+0000 7f4dcb46f640 10 monclient: init 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.403+0000 7f4dcb46f640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dc409ea50 0x7f4dc41309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 0x7f4dc4130f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc4136c20 0x7f4dc4099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc4136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc409b460 con 0x7f4dc409ea50 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc409e0d0 con 0x7f4dc4136c20 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4141e80 con 0x7f4dc41318f0 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc920b640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dc409ea50 msgr2=0x7f4dc41309d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dcb46f640 10 monclient(hunting): authenticate will time out at 1260.043084s 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc920b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dc409ea50 0x7f4dc41309d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 0x7f4dc4130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 0x7f4dc4130f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:39880/0 (socket says 172.21.15.153:39880) 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/3711315036 learned_addr learned my addr 172.21.15.153:0/3711315036 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): get_auth_request con 0x7f4dc41318f0 auth_method 0 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc4136c20 0x7f4dc4099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): get_auth_request con 0x7f4dc4136c20 auth_method 0 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc9a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient(hunting): handle_auth_done global_id 124151 payload 824 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc4136c20 msgr2=0x7f4dc4099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc4136c20 0x7f4dc4099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dc409ea50 msgr2=0x7f4dc41309d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dc409ea50 0x7f4dc41309d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 monclient: found mon.noname-c 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.404+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/3711315036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4009390 con 0x7f4dc41318f0 2024-09-15T03:37:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dc8a0a640 1 -- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 msgr2=0x7f4dc4130f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dc8a0a640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 0x7f4dc4130f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4db4035040 tx=0x7f4db4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4031990 con 0x7f4dc41318f0 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: dump: 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: _finish_auth 0 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4031af0 con 0x7f4dc41318f0 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4031e70 con 0x7f4dc41318f0 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dcb46f640 5 monclient: authenticate success, global_id 124151 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dcb46f640 10 monclient: _renew_subs 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dcb46f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc411cae0 con 0x7f4dc41318f0 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4db97fa640 10 monclient: dump: 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dbb7fe640 4 set_mon_vals no callback set 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dcb46f640 10 monclient: _renew_subs 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.406+0000 7f4dcb46f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.407+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc41374c0 con 0x7f4dc41318f0 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.407+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4db4039040 con 0x7f4dc41318f0 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4dcb46f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4dcb46f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4dcb46f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc404bf60 con 0x7f4dc41318f0 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4db97fa640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4da80386f0 0x7f4da803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f4db406cff0 con 0x7f4dc41318f0 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4dc920b640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4da80386f0 0x7f4da803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.408+0000 7f4dc920b640 10 monclient: get_auth_request con 0x7f4da80386f0 auth_method 0 2024-09-15T03:37:42.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.411+0000 7f4dc920b640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4da80386f0 0x7f4da803aba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00047d0 tx=0x7f4dc0007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:42.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.412+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4db40398f0 con 0x7f4dc41318f0 2024-09-15T03:37:42.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.412+0000 7f4db97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:42.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.412+0000 7f4db97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4dcb46f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4dcb46f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4dcb46f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4dc40638d0 con 0x7f4dc41318f0 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4db97fa640 1 -- 172.21.15.153:0/3711315036 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4db403f020 con 0x7f4dc41318f0 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4db97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.625+0000 7f4db97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:42.627 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4da80386f0 msgr2=0x7f4da803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4da80386f0 0x7f4da803aba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00047d0 tx=0x7f4dc0007b80 comp rx=0 tx=0).stop 2024-09-15T03:37:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 10 monclient: shutdown 2024-09-15T03:37:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 msgr2=0x7f4dc4130f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 0x7f4dc4130f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4db4035040 tx=0x7f4db4032000 comp rx=0 tx=0).stop 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 shutdown_connections 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4da80386f0 0x7f4da803aba0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4dc4136c20 0x7f4dc4099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4dc41318f0 0x7f4dc4130f10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 --2- 172.21.15.153:0/3711315036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dc409ea50 0x7f4dc41309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 >> 172.21.15.153:0/3711315036 conn(0x7f4dc4067110 msgr2=0x7f4dc40a4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 shutdown_connections 2024-09-15T03:37:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:42.627+0000 7f4dcb46f640 1 -- 172.21.15.153:0/3711315036 wait complete. 2024-09-15T03:37:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:42.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:42.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:43.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:43.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:43.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:43.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:44.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:44.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:44.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:44.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:45.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.743+0000 7fc328d54640 10 monclient: get_monmap_and_config 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.743+0000 7fc328d54640 10 monclient: build_initial_monmap 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.743+0000 7fc328d54640 10 monclient: monmap: 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:45.743950+0000 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:45.743950+0000 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:45.744 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:45.745 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:45.745 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:45.745 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:45.745 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:45.745 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:45.745 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:45.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.743+0000 7fc328d54640 1 Processor -- start 2024-09-15T03:37:45.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.743+0000 7fc328d54640 1 -- start start 2024-09-15T03:37:45.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.743+0000 7fc328d54640 10 monclient: init 2024-09-15T03:37:45.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc324131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 0x7fc32409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): picked mon.noname-a con 0x7fc32409e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): picked mon.noname-b con 0x7fc32409f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32409faf0 con 0x7fc324131ab0 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc324132590 con 0x7fc32409e910 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32409fd30 con 0x7fc32409f250 2024-09-15T03:37:45.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc328d54640 10 monclient(hunting): authenticate will time out at 1263.383636s 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc321d9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 msgr2=0x7fc32409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc321d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 0x7fc32409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53212/0 (socket says 172.21.15.153:53212) 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 learned_addr learned my addr 172.21.15.153:0/758987196 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:45.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.744+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:45.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.747+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.181:6789/0 conn(0x7fc324131ab0 legacy=0x7fc324131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:45.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:45.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:45.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:45.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 msgr2=0x7fc32409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 0x7fc32409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:45.945+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 msgr2=0x7fc32409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 0x7fc32409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:46.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:46.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:46.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:46.346+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:46.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:46.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:46.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:46.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 msgr2=0x7fc32409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 0x7fc32409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:47.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:47.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:47.147+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:47.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:47.759+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:47.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:47.759+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:47.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:47.759+0000 7fd25d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:48.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient(hunting): tick 2024-09-15T03:37:48.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:37:48.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:37:48.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc324138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc324138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 msgr2=0x7fc32409ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc32409e910 0x7fc32409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.181:6789/0 conn(0x7fc324131ab0 legacy=0x7fc324131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient: _add_conns ranks=[4,0] 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc304003720 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3040061c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3240a1110 con 0x7fc3040061c0 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc324132590 con 0x7fc304003720 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc313fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.744+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc3040061c0 legacy=0x7fc304008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc304003720 auth_method 0 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:48.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:48.760+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:48.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:48.760+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:48.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.945+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc3040061c0 legacy=0x7fc304008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc304003720 auth_method 0 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:48.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:48.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:48.946+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:49.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.346+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc3040061c0 legacy=0x7fc304008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc304003720 auth_method 0 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:49.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:49.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:49.347+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:49.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:49.760+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:49.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:49.760+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:50.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.147+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc3040061c0 legacy=0x7fc304008670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:50.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.147+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:37:50.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.148+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:50.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.148+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc304003720 auth_method 0 2024-09-15T03:37:50.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.148+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:50.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.148+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:50.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.148+0000 7fc32259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:50.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.149+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:50.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.149+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:50.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.149+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:50.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:50.149+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:50.760+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:50.760+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): tick 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc304003720 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc3040061c0 legacy=0x7fc304008670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:37:51.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 reap_dead start 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc32409f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc30400a440 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc30400a910 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30400af60 con 0x7fc30400a910 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc324132590 con 0x7fc30400a440 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32409fd30 con 0x7fc32409f250 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc313fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:51.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc30400a910 legacy=0x7fc3040085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 10 monclient(hunting): get_auth_request con 0x7fc30400a440 auth_method 0 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:51.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.745+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:51.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:51.760+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:51.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:51.760+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:51.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc30400a910 legacy=0x7fc3040085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 10 monclient(hunting): get_auth_request con 0x7fc30400a440 auth_method 0 2024-09-15T03:37:51.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:51.946+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:37:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc30400a910 legacy=0x7fc3040085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 10 monclient(hunting): get_auth_request con 0x7fc30400a440 auth_method 0 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:52.347+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:37:52.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:52.760+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:52.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:52.760+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:53.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc30400a910 legacy=0x7fc3040085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc32409f250 auth_method 0 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:53.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.148+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 10 monclient(hunting): get_auth_request con 0x7fc30400a440 auth_method 0 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:37:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:53.149+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:37:53.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:53.761+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:53.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:53.761+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:54.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient(hunting): tick 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 msgr2=0x7fc304009f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc32409f250 0x7fc304009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 msgr2=0x7fc304005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.181:3300/0 conn(0x7fc30400a440 0x7fc304005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.153:6789/0 conn(0x7fc30400a910 legacy=0x7fc3040085f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient: _add_conns ranks=[3,1] 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc30400d370 0x7fc30400d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc30400d370 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:54.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc30400de40 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3240a1110 con 0x7fc30400de40 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 1 -- 172.21.15.153:0/758987196 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc324132590 con 0x7fc30400d370 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc313fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc30400d370 msgr2=0x7fc30400d770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.745+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc30400d370 0x7fc30400d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:54.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.748+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.181:6789/0 conn(0x7fc30400de40 legacy=0x7fc3040124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:54.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.760+0000 7fd25d7fa640 10 monclient: tick 2024-09-15T03:37:54.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.760+0000 7fd25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.834+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146095315 0 0) 0x7fc3240a1110 con 0x7fc30400de40 2024-09-15T03:37:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.834+0000 7fc32159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.834+0000 7fc32159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:54.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.834+0000 7fc32159a640 10 monclient(hunting): my global_id is 104671 2024-09-15T03:37:54.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.834+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc30c0034e0 con 0x7fc30400de40 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628071954 0 0) 0x7fc30c0034e0 con 0x7fc30400de40 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc30400d370 msgr2=0x7fc30400d770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 --2- 172.21.15.153:0/758987196 >> v2:172.21.15.153:3300/0 conn(0x7fc30400d370 0x7fc30400d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: _finish_hunting 0 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30400af60 con 0x7fc30400de40 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: _finish_auth 0 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fc3140098c0 con 0x7fc30400de40 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: dump: 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 >> v1:172.21.15.181:6789/0 conn(0x7fc30400de40 legacy=0x7fc3040124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 1 -- 172.21.15.153:0/758987196 reap_dead start 2024-09-15T03:37:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c0039d0 0x7fc30c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): picked mon.a con 0x7fc30c0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc30c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): picked mon.c con 0x7fc30c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c005230 0x7fc30c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): picked mon.b con 0x7fc30c005230 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc324132590 con 0x7fc30c0039d0 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32259c640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c0039d0 msgr2=0x7fc30c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c00c1f0 con 0x7fc30c005230 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c00c3b0 con 0x7fc30c004780 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32159a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc32259c640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c0039d0 0x7fc30c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc30c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): get_auth_request con 0x7fc30c004780 auth_method 0 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c005230 0x7fc30c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): get_auth_request con 0x7fc30c005230 auth_method 0 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc322d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient(hunting): handle_auth_done global_id 104671 payload 293 2024-09-15T03:37:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c005230 msgr2=0x7fc30c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c005230 0x7fc30c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c0039d0 msgr2=0x7fc30c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c0039d0 0x7fc30c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 monclient: found mon.c 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c0034e0 con 0x7fc30c004780 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 10 monclient: _finish_auth 0 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc321d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc328d54640 5 monclient: authenticate success, global_id 104671 2024-09-15T03:37:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.837+0000 7fc328d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc321d9b640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 msgr2=0x7fc30c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc321d9b640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc30c004bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc314003940 tx=0x7fc314005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=380081087230b83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31400b120 con 0x7fc30c004780 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: dump: 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc328d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31402e030 con 0x7fc30c004780 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc328d54640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 1 -- 172.21.15.153:0/758987196 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314035440 con 0x7fc30c004780 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc328d54640 4 set_mon_vals no callback set 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.840+0000 7fc32159a640 10 monclient: dump: 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 10 monclient: shutdown 2024-09-15T03:37:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 -- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 msgr2=0x7fc30c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc30c004bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc314003940 tx=0x7fc314005040 comp rx=0 tx=0).stop 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 -- 172.21.15.153:0/758987196 shutdown_connections 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c005230 0x7fc30c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc30c004bf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 --2- 172.21.15.153:0/758987196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c0039d0 0x7fc30c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 -- 172.21.15.153:0/758987196 >> 172.21.15.153:0/758987196 conn(0x7fc324067d30 msgr2=0x7fc3240a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 -- 172.21.15.153:0/758987196 shutdown_connections 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.841+0000 7fc328d54640 1 -- 172.21.15.153:0/758987196 wait complete. 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.842+0000 7fc328d54640 10 monclient: build_initial_monmap 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.842+0000 7fc328d54640 10 monclient: monmap: 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.842+0000 7fc328d54640 1 Processor -- start 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 -- start start 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient: init 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c0039d0 0x7fc32409aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): picked mon.noname-b con 0x7fc30c0039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc3241308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): picked mon.noname-c con 0x7fc30c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c005230 0x7fc32411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): picked mon.noname-a con 0x7fc30c005230 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3241346f0 con 0x7fc30c005230 2024-09-15T03:37:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32411dcd0 con 0x7fc30c0039d0 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32411ca90 con 0x7fc30c004780 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc328d54640 10 monclient(hunting): authenticate will time out at 1272.482636s 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc322d9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c005230 msgr2=0x7fc32411b6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc322d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c005230 0x7fc32411b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc3241308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc3241308d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53314/0 (socket says 172.21.15.153:53314) 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 1 -- 172.21.15.153:0/2045726536 learned_addr learned my addr 172.21.15.153:0/2045726536 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): get_auth_request con 0x7fc30c004780 auth_method 0 2024-09-15T03:37:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc32259c640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c0039d0 0x7fc32409aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.843+0000 7fc321d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc32259c640 10 monclient(hunting): get_auth_request con 0x7fc30c0039d0 auth_method 0 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc32259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc32259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 10 monclient(hunting): handle_auth_done global_id 124154 payload 824 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 1 -- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c0039d0 msgr2=0x7fc32409aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c0039d0 0x7fc32409aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 1 -- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c005230 msgr2=0x7fc32411b6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c005230 0x7fc32411b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 1 monclient: found mon.noname-c 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc321d9b640 1 -- 172.21.15.153:0/2045726536 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3241419a0 con 0x7fc30c004780 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.844+0000 7fc32259c640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c0039d0 0x7fc32409aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc321d9b640 1 -- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 msgr2=0x7fc3241308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc321d9b640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc3241308d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc31400ba00 tx=0x7fc314038000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314044070 con 0x7fc30c004780 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: dump: 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31400b120 con 0x7fc30c004780 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 5 monclient: authenticate success, global_id 124154 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 10 monclient: _renew_subs 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32411d9f0 con 0x7fc30c004780 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3140357d0 con 0x7fc30c004780 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc3127fc640 10 monclient: dump: 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc313fff640 4 set_mon_vals no callback set 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 10 monclient: _renew_subs 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.846+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc324135ff0 con 0x7fc30c004780 2024-09-15T03:37:54.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.847+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc314035b40 con 0x7fc30c004780 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc328d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc328d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc328d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc3127fc640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc304039cb0 0x7fc30403c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32404bf60 con 0x7fc30c004780 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(240..240 src has 199..240) v4 ==== 10036+0+0 (secure 0 0 0) 0x7fc31406da50 con 0x7fc30c004780 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc32259c640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc304039cb0 0x7fc30403c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.848+0000 7fc32259c640 10 monclient: get_auth_request con 0x7fc304039cb0 auth_method 0 2024-09-15T03:37:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.851+0000 7fc32259c640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc304039cb0 0x7fc30403c160 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc318008fd0 tx=0x7fc318018040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:54.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.853+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc31403f5d0 con 0x7fc30c004780 2024-09-15T03:37:54.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.853+0000 7fc3127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:54.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:54.853+0000 7fc3127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.899+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd2640232f0 con 0x7fd26806a710 2024-09-15T03:37:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.899+0000 7fd25e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T03:37:54.900 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.899+0000 7fd25e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-15T03:37:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.998+0000 7fd27059d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T03:37:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.998+0000 7fd27059d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T03:37:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.998+0000 7fd27059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:54.999 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.998+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7fd268072640 con 0x7fd26806a710 2024-09-15T03:37:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.999+0000 7fd25e7fc640 1 -- 172.21.15.181:0/2796108122 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd2640234d0 con 0x7fd26806a710 2024-09-15T03:37:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.999+0000 7fd25e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-15T03:37:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:54.999+0000 7fd25e7fc640 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-15T03:37:55.000 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h24' does not appear in the crush map 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd24c038740 msgr2=0x7fd24c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd24c038740 0x7fd24c03abf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd26806e940 tx=0x7fd258032000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 10 monclient: shutdown 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 msgr2=0x7fd26811f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26811f960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd26400ea20 tx=0x7fd2640090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 shutdown_connections 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd24c038740 0x7fd24c03abf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26809dfa0 0x7fd268140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd26806e000 0x7fd26813cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 --2- 172.21.15.181:0/2796108122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26806a710 0x7fd26811f960 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 >> 172.21.15.181:0/2796108122 conn(0x7fd268071340 msgr2=0x7fd26809c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 shutdown_connections 2024-09-15T03:37:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.001+0000 7fd27059d640 1 -- 172.21.15.181:0/2796108122 wait complete. 2024-09-15T03:37:55.011 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:55.011 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h25 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.072+0000 7fc328d54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.073+0000 7fc328d54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.073+0000 7fc328d54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.073+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc324066280 con 0x7fc30c004780 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.073+0000 7fc3127fc640 1 -- 172.21.15.153:0/2045726536 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fc31403b260 con 0x7fc30c004780 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.073+0000 7fc3127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.073+0000 7fc3127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:55.074 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc304039cb0 msgr2=0x7fc30403c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc304039cb0 0x7fc30403c160 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc318008fd0 tx=0x7fc318018040 comp rx=0 tx=0).stop 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 10 monclient: shutdown 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 msgr2=0x7fc3241308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc3241308d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc31400ba00 tx=0x7fc314038000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 shutdown_connections 2024-09-15T03:37:55.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc304039cb0 0x7fc30403c160 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30c005230 0x7fc32411b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc30c004780 0x7fc3241308d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 --2- 172.21.15.153:0/2045726536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc30c0039d0 0x7fc32409aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 >> 172.21.15.153:0/2045726536 conn(0x7fc324067d30 msgr2=0x7fc32409d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 shutdown_connections 2024-09-15T03:37:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.075+0000 7fc328d54640 1 -- 172.21.15.153:0/2045726536 wait complete. 2024-09-15T03:37:55.087 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:37:55.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:37:55.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:37:55.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.086+0000 7f0dc4d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:55.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.086+0000 7f0dc4d9d640 10 monclient: build_initial_monmap 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.086+0000 7f0dc4d9d640 10 monclient: monmap: 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:55.086975+0000 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:55.086975+0000 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.086+0000 7f0dc4d9d640 1 Processor -- start 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.086+0000 7f0dc4d9d640 1 -- start start 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.086+0000 7f0dc4d9d640 10 monclient: init 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dc009ec50 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0dc009f020 0x7f0dc0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc009f020 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0dc0136d60 0x7f0dc0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc0136d60 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc0067420 con 0x7f0dc009ec50 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc00a0480 con 0x7f0dc009f020 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc00a0200 con 0x7f0dc0136d60 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dc4d9d640 10 monclient(hunting): authenticate will time out at 1279.568412s 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0dc0136d60 0x7f0dc0139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0dc0136d60 0x7f0dc0139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60142/0 (socket says 172.21.15.181:60142) 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 -- 172.21.15.181:0/2939295965 learned_addr learned my addr 172.21.15.181:0/2939295965 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbe59c640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f0dc009ec50 0x7f0dc0131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:41582/0 (socket says 172.21.15.181:41582) 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): get_auth_request con 0x7f0dc0136d60 auth_method 0 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbdd9b640 1 -- 172.21.15.181:0/2939295965 >> v2:172.21.15.153:3300/0 conn(0x7f0dc009f020 msgr2=0x7f0dc0069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbdd9b640 1 --2- 172.21.15.181:0/2939295965 >> v2:172.21.15.153:3300/0 conn(0x7f0dc009f020 0x7f0dc0069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_done global_id 104677 payload 274 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 -- 172.21.15.181:0/2939295965 >> v2:172.21.15.153:3300/0 conn(0x7f0dc009f020 msgr2=0x7f0dc0069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 --2- 172.21.15.181:0/2939295965 >> v2:172.21.15.153:3300/0 conn(0x7f0dc009f020 0x7f0dc0069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 -- 172.21.15.181:0/2939295965 >> v1:172.21.15.181:6789/0 conn(0x7f0dc009ec50 legacy=0x7f0dc0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 monclient: found mon.noname-c 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:55.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.087+0000 7f0dbed9d640 1 -- 172.21.15.181:0/2939295965 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc009fd80 con 0x7f0dc0136d60 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbed9d640 1 -- 172.21.15.181:0/2939295965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 msgr2=0x7f0dc0139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbed9d640 1 --2- 172.21.15.181:0/2939295965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0139140 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0dc009bf50 tx=0x7f0db8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be04f1e8753ab6cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 1 -- 172.21.15.181:0/2939295965 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db802aea0 con 0x7f0dc0136d60 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: dump: 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.099 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: _finish_auth 0 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 1 -- 172.21.15.181:0/2939295965 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db802d050 con 0x7f0dc0136d60 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dc4d9d640 5 monclient: authenticate success, global_id 104677 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dc4d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dc4d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dc4d9d640 4 set_mon_vals no callback set 2024-09-15T03:37:55.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 1 -- 172.21.15.181:0/2939295965 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8007840 con 0x7f0dc0136d60 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.090+0000 7f0dbd59a640 10 monclient: dump: 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.101 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 10 monclient: shutdown 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/2939295965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 msgr2=0x7f0dc0139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/2939295965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0139140 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0dc009bf50 tx=0x7f0db8009840 comp rx=0 tx=0).stop 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/2939295965 shutdown_connections 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/2939295965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0139140 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/2939295965 >> v2:172.21.15.153:3300/0 conn(0x7f0dc009f020 0x7f0dc0069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/2939295965 >> 172.21.15.181:0/2939295965 conn(0x7f0dc00a5cf0 msgr2=0x7f0dc00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/2939295965 shutdown_connections 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.091+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/2939295965 wait complete. 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient: build_initial_monmap 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient: monmap: 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:55.102 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 Processor -- start 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 -- start start 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient: init 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc009ec50 0x7f0dc01411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc009ec50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dc009f020 0x7f0dc0141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc009f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc0136d60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc009c020 con 0x7f0dc009ec50 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc00a3c90 con 0x7f0dc0136d60 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc009b7b0 con 0x7f0dc009f020 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dc4d9d640 10 monclient(hunting): authenticate will time out at 1279.573910s 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dbe59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc009ec50 msgr2=0x7f0dc01411d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:55.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dbed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.092+0000 7f0dbe59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc009ec50 0x7f0dc01411d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0141c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60150/0 (socket says 172.21.15.181:60150) 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 -- 172.21.15.181:0/468747300 learned_addr learned my addr 172.21.15.181:0/468747300 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): get_auth_request con 0x7f0dc0136d60 auth_method 0 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbdd9b640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dc009f020 0x7f0dc0141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbdd9b640 10 monclient(hunting): get_auth_request con 0x7f0dc009f020 auth_method 0 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient(hunting): handle_auth_done global_id 104680 payload 824 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 -- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dc009f020 msgr2=0x7f0dc0141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dc009f020 0x7f0dc0141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 -- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc009ec50 msgr2=0x7f0dc01411d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc009ec50 0x7f0dc01411d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 monclient: found mon.noname-b 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbed9d640 1 -- 172.21.15.181:0/468747300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8009ae0 con 0x7f0dc0136d60 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.093+0000 7f0dbdd9b640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dc009f020 0x7f0dc0141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dbed9d640 1 -- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 msgr2=0x7f0dc0141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dbed9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0141c50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009ab0 tx=0x7f0db8036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8031cb0 con 0x7f0dc0136d60 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: dump: 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.107 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 5 monclient: authenticate success, global_id 104680 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 10 monclient: _renew_subs 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8031e10 con 0x7f0dc0136d60 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc011dc80 con 0x7f0dc0136d60 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db803c670 con 0x7f0dc0136d60 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dae7fc640 10 monclient: dump: 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:55.108 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dbcd99640 4 set_mon_vals no callback set 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 10 monclient: _renew_subs 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.095+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc011b9e0 con 0x7f0dc0136d60 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0db803caa0 con 0x7f0dc0136d60 2024-09-15T03:37:55.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dc4d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dc4d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dae7fc640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0da00386f0 0x7f0da003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dc4d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc00008d0 con 0x7f0dc0136d60 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(241..241 src has 199..241) v4 ==== 9823+0+0 (secure 0 0 0) 0x7f0db806d7e0 con 0x7f0dc0136d60 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dbe59c640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0da00386f0 0x7f0da003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.096+0000 7f0dbe59c640 10 monclient: get_auth_request con 0x7f0da00386f0 auth_method 0 2024-09-15T03:37:55.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.099+0000 7f0dbe59c640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0da00386f0 0x7f0da003aba0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0db0004880 tx=0x7f0db0007ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.101+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0db8040080 con 0x7f0dc0136d60 2024-09-15T03:37:55.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.101+0000 7f0dae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.101+0000 7f0dae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.185+0000 7f0a7428b640 10 monclient: get_monmap_and_config 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.185+0000 7f0a7428b640 10 monclient: build_initial_monmap 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.185+0000 7f0a7428b640 10 monclient: monmap: 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:55.186686+0000 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:55.186686+0000 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:55.188 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.186+0000 7f0a7428b640 1 Processor -- start 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.186+0000 7f0a7428b640 1 -- start start 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.186+0000 7f0a7428b640 10 monclient: init 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0a6c09ad90 0x7f0a6c09c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a6c09ad90 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0a6c09b160 0x7f0a6c09b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6c09b160 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a6c131960 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c09d890 con 0x7f0a6c09ad90 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c0692c0 con 0x7f0a6c09b160 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c09db70 con 0x7f0a6c131960 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a7428b640 10 monclient(hunting): authenticate will time out at 1272.826251s 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72027640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0a6c09ad90 msgr2=0x7f0a6c09c5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72027640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0a6c09ad90 0x7f0a6c09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:55.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c131dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36822/0 (socket says 172.21.15.153:36822) 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 -- 172.21.15.153:0/1482023844 learned_addr learned my addr 172.21.15.153:0/1482023844 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): get_auth_request con 0x7f0a6c131960 auth_method 0 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 1 --2- 172.21.15.153:0/1482023844 >> v2:172.21.15.181:3300/0 conn(0x7f0a6c09b160 0x7f0a6c09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): get_auth_request con 0x7f0a6c09b160 auth_method 0 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a71826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient(hunting): handle_auth_done global_id 124160 payload 274 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 -- 172.21.15.153:0/1482023844 >> v2:172.21.15.181:3300/0 conn(0x7f0a6c09b160 msgr2=0x7f0a6c09b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 --2- 172.21.15.153:0/1482023844 >> v2:172.21.15.181:3300/0 conn(0x7f0a6c09b160 0x7f0a6c09b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 -- 172.21.15.153:0/1482023844 >> v2:172.21.15.153:3300/0 conn(0x7f0a6c09ad90 msgr2=0x7f0a6c09c5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 --2- 172.21.15.153:0/1482023844 >> v2:172.21.15.153:3300/0 conn(0x7f0a6c09ad90 0x7f0a6c09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient: _finish_hunting 0 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 monclient: found mon.noname-b 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:55.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.187+0000 7f0a72828640 1 -- 172.21.15.153:0/1482023844 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c09d5b0 con 0x7f0a6c131960 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a72828640 1 -- 172.21.15.153:0/1482023844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 msgr2=0x7f0a6c131dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a72828640 1 --2- 172.21.15.153:0/1482023844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c131dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c06ed00 tx=0x7f0a64009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9749729cd0e3df1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 1 -- 172.21.15.153:0/1482023844 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64007610 con 0x7f0a6c131960 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: dump: 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.196 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: _finish_auth 0 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 1 -- 172.21.15.153:0/1482023844 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6402d050 con 0x7f0a6c131960 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a7428b640 5 monclient: authenticate success, global_id 124160 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a7428b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 1 -- 172.21.15.153:0/1482023844 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a640079a0 con 0x7f0a6c131960 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a7428b640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a7428b640 4 set_mon_vals no callback set 2024-09-15T03:37:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.190+0000 7f0a71025640 10 monclient: dump: 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 10 monclient: shutdown 2024-09-15T03:37:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 -- 172.21.15.153:0/1482023844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 msgr2=0x7f0a6c131dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 --2- 172.21.15.153:0/1482023844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c131dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c06ed00 tx=0x7f0a64009840 comp rx=0 tx=0).stop 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 -- 172.21.15.153:0/1482023844 shutdown_connections 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 --2- 172.21.15.153:0/1482023844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c131dc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 --2- 172.21.15.153:0/1482023844 >> v2:172.21.15.181:3300/0 conn(0x7f0a6c09b160 0x7f0a6c09b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 --2- 172.21.15.153:0/1482023844 >> v2:172.21.15.153:3300/0 conn(0x7f0a6c09ad90 0x7f0a6c09c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 -- 172.21.15.153:0/1482023844 >> 172.21.15.153:0/1482023844 conn(0x7f0a6c0678a0 msgr2=0x7f0a6c0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 -- 172.21.15.153:0/1482023844 shutdown_connections 2024-09-15T03:37:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 1 -- 172.21.15.153:0/1482023844 wait complete. 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 10 monclient: build_initial_monmap 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.191+0000 7f0a7428b640 10 monclient: monmap: 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 Processor -- start 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 -- start start 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient: init 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a6c09ad90 0x7f0a6c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a6c09ad90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6c09b160 0x7f0a6c1415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a6c09b160 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c129600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6c131960 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c12b9e0 con 0x7f0a6c09b160 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0a0b50 con 0x7f0a6c09ad90 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c12bbb0 con 0x7f0a6c131960 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a7428b640 10 monclient(hunting): authenticate will time out at 1272.831841s 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a71826640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6c09b160 msgr2=0x7f0a6c1415c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a71826640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6c09b160 0x7f0a6c1415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c129600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c129600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36830/0 (socket says 172.21.15.153:36830) 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 learned_addr learned my addr 172.21.15.153:0/3415118424 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 10 monclient(hunting): get_auth_request con 0x7f0a6c131960 auth_method 0 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72828640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.192+0000 7f0a72027640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a6c09ad90 0x7f0a6c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): get_auth_request con 0x7f0a6c09ad90 auth_method 0 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 10 monclient(hunting): handle_auth_done global_id 124163 payload 824 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a6c09ad90 msgr2=0x7f0a6c141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a6c09ad90 0x7f0a6c141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6c09b160 msgr2=0x7f0a6c1415c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6c09b160 0x7f0a6c1415c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 10 monclient: _finish_hunting 0 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 1 monclient: found mon.noname-c 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.193+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a64009ae0 con 0x7f0a6c131960 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 msgr2=0x7f0a6c129600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a72828640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c129600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a6402abc0 tx=0x7f0a64036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64007170 con 0x7f0a6c131960 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: dump: 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.210 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a640094e0 con 0x7f0a6c131960 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 5 monclient: authenticate success, global_id 124163 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64007e10 con 0x7f0a6c131960 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 10 monclient: _renew_subs 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c06f250 con 0x7f0a6c131960 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a70824640 4 set_mon_vals no callback set 2024-09-15T03:37:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a627fc640 10 monclient: dump: 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 10 monclient: _renew_subs 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.195+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c12b020 con 0x7f0a6c131960 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0a64003020 con 0x7f0a6c131960 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a7428b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a7428b640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 0x7f0a6c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c065e00 con 0x7f0a6c12a300 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a627fc640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a500386f0 0x7f0a5003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(241..241 src has 199..241) v4 ==== 9823+0+0 (secure 0 0 0) 0x7f0a6406d940 con 0x7f0a6c131960 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 0x7f0a6c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient: get_auth_request con 0x7f0a6c12a300 auth_method 0 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a71826640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a500386f0 0x7f0a5003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a71826640 10 monclient: get_auth_request con 0x7f0a500386f0 auth_method 0 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 10 monclient(hunting): handle_auth_done global_id 124166 payload 824 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.196+0000 7f0a72027640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a6c04bf60 con 0x7f0a6c12a300 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a72027640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 msgr2=0x7f0a6c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a72027640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 0x7f0a6c0298c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a680095e0 tx=0x7f0a680090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800e9f0 con 0x7f0a6c12a300 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 10 monclient: dump: 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a71826640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a500386f0 0x7f0a5003aba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009480 tx=0x7f0a5c0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0a6800eb50 con 0x7f0a6c12a300 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 msgr2=0x7f0a6c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 0x7f0a6c0298c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a680095e0 tx=0x7f0a680090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 msgr2=0x7f0a6c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:55.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.199+0000 7f0a627fc640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a300 0x7f0a6c0298c0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a680095e0 tx=0x7f0a680090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a7428b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a7428b640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 0x7f0a6c029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a7428b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c098220 con 0x7f0a6c12a6d0 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 0x7f0a6c029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient: get_auth_request con 0x7f0a6c12a6d0 auth_method 0 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 10 monclient(hunting): handle_auth_done global_id 124169 payload 824 2024-09-15T03:37:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.205+0000 7f0a72027640 1 -- 172.21.15.153:0/3415118424 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a6c0638d0 con 0x7f0a6c12a6d0 2024-09-15T03:37:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a72027640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 msgr2=0x7f0a6c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a72027640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 0x7f0a6c029e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a680095e0 tx=0x7f0a68015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68025070 con 0x7f0a6c12a6d0 2024-09-15T03:37:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 10 monclient: dump: 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f0a6800e7e0 con 0x7f0a6c12a6d0 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 msgr2=0x7f0a6c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 0x7f0a6c029e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a680095e0 tx=0x7f0a68015000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 msgr2=0x7f0a6c029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.207+0000 7f0a627fc640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c12a6d0 0x7f0a6c029e00 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a680095e0 tx=0x7f0a68015000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 102, 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:55.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:55.228 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.229 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:55.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:55.231 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.232 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:55.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:55.234 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:55.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a500386f0 msgr2=0x7f0a5003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0a500386f0 0x7f0a5003aba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009480 tx=0x7f0a5c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 10 monclient: shutdown 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 msgr2=0x7f0a6c129600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 --2- 172.21.15.153:0/3415118424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a6c131960 0x7f0a6c129600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a6402abc0 tx=0x7f0a64036000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 reap_dead start 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a72828640 1 -- 172.21.15.153:0/3415118424 reap_dead start 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 shutdown_connections 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 >> 172.21.15.153:0/3415118424 conn(0x7f0a6c0678a0 msgr2=0x7f0a6c095430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 shutdown_connections 2024-09-15T03:37:55.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.208+0000 7f0a7428b640 1 -- 172.21.15.153:0/3415118424 wait complete. 2024-09-15T03:37:55.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:37:55.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:37:55.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:37:55.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.261+0000 7f0dc4d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T03:37:55.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.261+0000 7f0dc4d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T03:37:55.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.261+0000 7f0dc4d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:55.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.261+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7f0dc00a13e0 con 0x7f0dc0136d60 2024-09-15T03:37:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.328+0000 7fd17d95d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.328+0000 7fd17d95d640 10 monclient: build_initial_monmap 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.328+0000 7fd17d95d640 10 monclient: monmap: 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:37:55.330590+0000 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:37:55.330590+0000 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:55.331 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 Processor -- start 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 -- start start 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient: init 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): picked mon.noname-b con 0x7fd17806eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd178068a20 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd17806f490 0x7fd178138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): picked mon.noname-c con 0x7fd17806f490 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd178069b80 con 0x7fd178068a20 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd178099f90 con 0x7fd17806f490 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd178069e00 con 0x7fd17806eb80 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd17d95d640 10 monclient(hunting): authenticate will time out at 1272.969933s 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.329+0000 7fd1767fc640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fd178068a20 legacy=0x7fd17809c540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17806ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36858/0 (socket says 172.21.15.153:36858) 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 -- 172.21.15.153:0/2862704026 learned_addr learned my addr 172.21.15.153:0/2862704026 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): get_auth_request con 0x7fd17806eb80 auth_method 0 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient(hunting): handle_auth_done global_id 124172 payload 274 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd1777fe640 1 --2- 172.21.15.153:0/2862704026 >> v2:172.21.15.181:3300/0 conn(0x7fd17806f490 0x7fd178138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 -- 172.21.15.153:0/2862704026 >> v2:172.21.15.181:3300/0 conn(0x7fd17806f490 msgr2=0x7fd178138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 --2- 172.21.15.153:0/2862704026 >> v2:172.21.15.181:3300/0 conn(0x7fd17806f490 0x7fd178138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 -- 172.21.15.153:0/2862704026 >> v1:172.21.15.153:6789/0 conn(0x7fd178068a20 legacy=0x7fd17809c540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 monclient: found mon.noname-b 2024-09-15T03:37:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.330+0000 7fd176ffd640 1 -- 172.21.15.153:0/2862704026 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17809cc50 con 0x7fd17806eb80 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd176ffd640 1 -- 172.21.15.153:0/2862704026 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 msgr2=0x7fd17806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd176ffd640 1 --2- 172.21.15.153:0/2862704026 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17806ef50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd168009910 tx=0x7fd1680091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad7f5bc3d2c891d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 1 -- 172.21.15.153:0/2862704026 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168032440 con 0x7fd17806eb80 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: dump: 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 5 monclient: authenticate success, global_id 124172 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 1 -- 172.21.15.153:0/2862704026 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1680325a0 con 0x7fd17806eb80 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 4 set_mon_vals no callback set 2024-09-15T03:37:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 1 -- 172.21.15.153:0/2862704026 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168032980 con 0x7fd17806eb80 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd175ffb640 10 monclient: dump: 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.345 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 10 monclient: shutdown 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 -- 172.21.15.153:0/2862704026 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 msgr2=0x7fd17806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 --2- 172.21.15.153:0/2862704026 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17806ef50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd168009910 tx=0x7fd1680091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 -- 172.21.15.153:0/2862704026 shutdown_connections 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 --2- 172.21.15.153:0/2862704026 >> v2:172.21.15.181:3300/0 conn(0x7fd17806f490 0x7fd178138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 --2- 172.21.15.153:0/2862704026 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17806ef50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 -- 172.21.15.153:0/2862704026 >> 172.21.15.153:0/2862704026 conn(0x7fd178067cd0 msgr2=0x7fd1780a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 -- 172.21.15.153:0/2862704026 shutdown_connections 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.333+0000 7fd17d95d640 1 -- 172.21.15.153:0/2862704026 wait complete. 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.334+0000 7fd17d95d640 10 monclient: build_initial_monmap 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.334+0000 7fd17d95d640 10 monclient: monmap: 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:37:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.334+0000 7fd17d95d640 1 Processor -- start 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.334+0000 7fd17d95d640 1 -- start start 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.334+0000 7fd17d95d640 10 monclient: init 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178068a20 0x7fd17811e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): picked mon.noname-b con 0x7fd178068a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17811e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): picked mon.noname-c con 0x7fd17806eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd17806f490 0x7fd17811ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): picked mon.noname-a con 0x7fd17806f490 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17809cf40 con 0x7fd17806f490 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780a2460 con 0x7fd178068a20 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd178121cc0 con 0x7fd17806eb80 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd17d95d640 10 monclient(hunting): authenticate will time out at 1272.975343s 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1777fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd17806f490 msgr2=0x7fd17811ee10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17811e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1777fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd17806f490 0x7fd17811ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17811e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36862/0 (socket says 172.21.15.153:36862) 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 -- 172.21.15.153:0/3436564489 learned_addr learned my addr 172.21.15.153:0/3436564489 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): get_auth_request con 0x7fd17806eb80 auth_method 0 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178068a20 0x7fd17811e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): get_auth_request con 0x7fd178068a20 auth_method 0 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient(hunting): handle_auth_done global_id 124175 payload 824 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178068a20 msgr2=0x7fd17811e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178068a20 0x7fd17811e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd17806f490 msgr2=0x7fd17811ee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd17806f490 0x7fd17811ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:55.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 monclient: found mon.noname-c 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.335+0000 7fd1767fc640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168009450 con 0x7fd17806eb80 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.337+0000 7fd1767fc640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 msgr2=0x7fd17811e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.337+0000 7fd1767fc640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17811e8d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b630 tx=0x7fd16c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c015070 con 0x7fd17806eb80 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: dump: 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.351 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: _finish_auth 0 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c003d90 con 0x7fd17806eb80 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 5 monclient: authenticate success, global_id 124175 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 10 monclient: _renew_subs 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1781219e0 con 0x7fd17806eb80 2024-09-15T03:37:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c006380 con 0x7fd17806eb80 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 10 monclient: dump: 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.353 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 10 monclient: _renew_subs 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd1757fa640 4 set_mon_vals no callback set 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd178122730 con 0x7fd17806eb80 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd16c010440 con 0x7fd17806eb80 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 0x7fd17811f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.338+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd178065e00 con 0x7fd178120af0 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd15effd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1640386f0 0x7fd16403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 0x7fd17811f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(241..241 src has 199..241) v4 ==== 9823+0+0 (secure 0 0 0) 0x7fd16c04c940 con 0x7fd17806eb80 2024-09-15T03:37:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient: get_auth_request con 0x7fd178120af0 auth_method 0 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd1777fe640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1640386f0 0x7fd16403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd1777fe640 10 monclient: get_auth_request con 0x7fd1640386f0 auth_method 0 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 10 monclient(hunting): handle_auth_done global_id 104692 payload 824 2024-09-15T03:37:55.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.339+0000 7fd176ffd640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd17804bf60 con 0x7fd178120af0 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd176ffd640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 msgr2=0x7fd17811f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd176ffd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 0x7fd17811f350 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd1780a3da0 tx=0x7fd168032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd1777fe640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1640386f0 0x7fd16403aba0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd178120880 tx=0x7fd170008430 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16803ca10 con 0x7fd178120af0 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd15effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd15effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.341+0000 7fd15effd640 10 monclient: dump: 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.358 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd168040070 con 0x7fd178120af0 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 msgr2=0x7fd17811f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 0x7fd17811f350 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd1780a3da0 tx=0x7fd168032000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 msgr2=0x7fd17811f350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.342+0000 7fd15effd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178120af0 0x7fd17811f350 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd1780a3da0 tx=0x7fd168032000 comp rx=0 tx=0).stop 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.347+0000 7fd17d95d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.347+0000 7fd17d95d640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 0x7fd178096e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.347+0000 7fd17d95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.347+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd178121e00 con 0x7fd178096a60 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 0x7fd178096e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient: get_auth_request con 0x7fd178096a60 auth_method 0 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:55.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 10 monclient(hunting): handle_auth_done global_id 104695 payload 824 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.348+0000 7fd176ffd640 1 -- 172.21.15.153:0/3436564489 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd1780638d0 con 0x7fd178096a60 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.350+0000 7fd176ffd640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 msgr2=0x7fd178096e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.350+0000 7fd176ffd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 0x7fd178096e60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd17811f890 tx=0x7fd16803e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.350+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16803ce50 con 0x7fd178096a60 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.350+0000 7fd15effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 10 monclient: dump: 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:37:55.361 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7fd16803b5e0 con 0x7fd178096a60 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 msgr2=0x7fd178096e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 0x7fd178096e60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd17811f890 tx=0x7fd16803e070 comp rx=0 tx=0).stop 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 msgr2=0x7fd178096e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:37:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd15effd640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd178096a60 0x7fd178096e60 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd17811f890 tx=0x7fd16803e070 comp rx=0 tx=0).stop 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 102, 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:55.363 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:37:55.364 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:37:55.365 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:37:55.366 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:37:55.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:55.370 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.372 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.373 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.374 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:37:55.375 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:37:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:37:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:37:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:37:55.376 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:37:55.376 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:37:55.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1640386f0 msgr2=0x7fd16403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1640386f0 0x7fd16403aba0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd178120880 tx=0x7fd170008430 comp rx=0 tx=0).stop 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 10 monclient: shutdown 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 msgr2=0x7fd17811e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 --2- 172.21.15.153:0/3436564489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd17806eb80 0x7fd17811e8d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b630 tx=0x7fd16c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd1777fe640 1 -- 172.21.15.153:0/3436564489 reap_dead start 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd1777fe640 1 -- 172.21.15.153:0/3436564489 reap_dead start 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 shutdown_connections 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 >> 172.21.15.153:0/3436564489 conn(0x7fd178067cd0 msgr2=0x7fd1780a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 shutdown_connections 2024-09-15T03:37:55.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:37:55.351+0000 7fd17d95d640 1 -- 172.21.15.153:0/3436564489 wait complete. 2024-09-15T03:37:55.379 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:37:55.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:37:55.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.900+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0db806d070 con 0x7f0dc0136d60 2024-09-15T03:37:55.901 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.900+0000 7f0dae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T03:37:55.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.900+0000 7f0dae7fc640 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-15T03:37:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.999+0000 7f0dc4d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T03:37:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.999+0000 7f0dc4d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T03:37:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.999+0000 7f0dc4d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:55.999+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7f0dc0072640 con 0x7f0dc0136d60 2024-09-15T03:37:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.000+0000 7f0dae7fc640 1 -- 172.21.15.181:0/468747300 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0db803a910 con 0x7f0dc0136d60 2024-09-15T03:37:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.000+0000 7f0dae7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-15T03:37:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.000+0000 7f0dae7fc640 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-15T03:37:56.001 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h25' does not appear in the crush map 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.001+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0da00386f0 msgr2=0x7f0da003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.001+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0da00386f0 0x7f0da003aba0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0db0004880 tx=0x7f0db0007ac0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.001+0000 7f0dc4d9d640 10 monclient: shutdown 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.001+0000 7f0dc4d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.001+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 msgr2=0x7f0dc0141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.001+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0141c50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009ab0 tx=0x7f0db8036000 comp rx=0 tx=0).stop 2024-09-15T03:37:56.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 shutdown_connections 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0da00386f0 0x7f0da003aba0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dc0136d60 0x7f0dc0141c50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0dc009f020 0x7f0dc0141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 --2- 172.21.15.181:0/468747300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc009ec50 0x7f0dc01411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 >> 172.21.15.181:0/468747300 conn(0x7f0dc00a5cf0 msgr2=0x7f0dc009d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 shutdown_connections 2024-09-15T03:37:56.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.002+0000 7f0dc4d9d640 1 -- 172.21.15.181:0/468747300 wait complete. 2024-09-15T03:37:56.013 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:56.013 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h26 2024-09-15T03:37:56.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.084+0000 7fe62595d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:56.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.084+0000 7fe62595d640 10 monclient: build_initial_monmap 2024-09-15T03:37:56.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.084+0000 7fe62595d640 10 monclient: monmap: 2024-09-15T03:37:56.086 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:56.086 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:56.086 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:56.085162+0000 2024-09-15T03:37:56.086 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:56.085162+0000 2024-09-15T03:37:56.086 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:56.087 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:56.087 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:56.087 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:56.087 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:56.087 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:56.088 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:56.088 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:56.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.084+0000 7fe62595d640 1 Processor -- start 2024-09-15T03:37:56.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 1 -- start start 2024-09-15T03:37:56.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient: init 2024-09-15T03:37:56.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:56.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:37:56.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe62009ec50 0x7fe620131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:56.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): picked mon.noname-a con 0x7fe62009ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:56.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe6201321a0 0x7fe620068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:56.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): picked mon.noname-c con 0x7fe6201321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:56.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe62009f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:56.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe620067420 con 0x7fe62009f020 2024-09-15T03:37:56.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:56.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6200a0480 con 0x7fe62009ec50 2024-09-15T03:37:56.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:56.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6200a0200 con 0x7fe6201321a0 2024-09-15T03:37:56.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:56.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe62595d640 10 monclient(hunting): authenticate will time out at 1280.566562s 2024-09-15T03:37:56.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe6201321a0 0x7fe620068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:56.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61effd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fe62009ec50 msgr2=0x7fe620131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:56.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe62009ec50 0x7fe620131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:56.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe6201321a0 0x7fe620068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60156/0 (socket says 172.21.15.181:60156) 2024-09-15T03:37:56.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61f7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe62009f020 0x7fe620069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:41594/0 (socket says 172.21.15.181:41594) 2024-09-15T03:37:56.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 -- 172.21.15.181:0/386465760 learned_addr learned my addr 172.21.15.181:0/386465760 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:56.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): get_auth_request con 0x7fe6201321a0 auth_method 0 2024-09-15T03:37:56.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:56.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:56.109 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:56.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:56.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:56.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:56.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_done global_id 104698 payload 274 2024-09-15T03:37:56.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 -- 172.21.15.181:0/386465760 >> v2:172.21.15.153:3300/0 conn(0x7fe62009ec50 msgr2=0x7fe620131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:56.110 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 --2- 172.21.15.181:0/386465760 >> v2:172.21.15.153:3300/0 conn(0x7fe62009ec50 0x7fe620131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 -- 172.21.15.181:0/386465760 >> v1:172.21.15.181:6789/0 conn(0x7fe62009f020 legacy=0x7fe620069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:56.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:56.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 monclient: found mon.noname-c 2024-09-15T03:37:56.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:56.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.085+0000 7fe61e7fc640 1 -- 172.21.15.181:0/386465760 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62009fd80 con 0x7fe6201321a0 2024-09-15T03:37:56.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61e7fc640 1 -- 172.21.15.181:0/386465760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6201321a0 msgr2=0x7fe620068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:56.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61e7fc640 1 --2- 172.21.15.181:0/386465760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6201321a0 0x7fe620068d00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe6201325a0 tx=0x7fe6140091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd03f323a8938a13 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:56.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 1 -- 172.21.15.181:0/386465760 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614032440 con 0x7fe6201321a0 2024-09-15T03:37:56.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:56.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:56.113 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: dump: 2024-09-15T03:37:56.113 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:56.113 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:56.113 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.113 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.114 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:56.114 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:56.114 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:56.114 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:56.114 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:56.115 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:56.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:56.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:56.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe62595d640 5 monclient: authenticate success, global_id 104698 2024-09-15T03:37:56.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 1 -- 172.21.15.181:0/386465760 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6140325a0 con 0x7fe6201321a0 2024-09-15T03:37:56.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe62595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:56.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:56.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe62595d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:56.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe62595d640 4 set_mon_vals no callback set 2024-09-15T03:37:56.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 1 -- 172.21.15.181:0/386465760 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614032940 con 0x7fe6201321a0 2024-09-15T03:37:56.116 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:56.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:56.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.088+0000 7fe61dffb640 10 monclient: dump: 2024-09-15T03:37:56.117 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:56.117 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:56.117 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.118 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.118 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:56.118 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:56.118 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:56.118 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:56.119 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:56.119 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:56.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 10 monclient: shutdown 2024-09-15T03:37:56.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:56.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 -- 172.21.15.181:0/386465760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6201321a0 msgr2=0x7fe620068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:56.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 --2- 172.21.15.181:0/386465760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6201321a0 0x7fe620068d00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe6201325a0 tx=0x7fe6140091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 -- 172.21.15.181:0/386465760 shutdown_connections 2024-09-15T03:37:56.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 --2- 172.21.15.181:0/386465760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6201321a0 0x7fe620068d00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 --2- 172.21.15.181:0/386465760 >> v2:172.21.15.153:3300/0 conn(0x7fe62009ec50 0x7fe620131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 -- 172.21.15.181:0/386465760 >> 172.21.15.181:0/386465760 conn(0x7fe6200a5cf0 msgr2=0x7fe6200a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:56.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 -- 172.21.15.181:0/386465760 shutdown_connections 2024-09-15T03:37:56.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.089+0000 7fe62595d640 1 -- 172.21.15.181:0/386465760 wait complete. 2024-09-15T03:37:56.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient: build_initial_monmap 2024-09-15T03:37:56.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient: monmap: 2024-09-15T03:37:56.121 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:56.121 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:56.121 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:56.122 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:56.122 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:56.122 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:56.122 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:56.122 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:56.123 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:56.123 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:56.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 Processor -- start 2024-09-15T03:37:56.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 -- start start 2024-09-15T03:37:56.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient: init 2024-09-15T03:37:56.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:56.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:37:56.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe62009ec50 0x7fe62009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:56.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): picked mon.noname-c con 0x7fe62009ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:56.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 0x7fe620130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:56.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): picked mon.noname-b con 0x7fe62009f020 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:56.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6201321a0 0x7fe620141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:56.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): picked mon.noname-a con 0x7fe6201321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:56.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:56.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62009e140 con 0x7fe6201321a0 2024-09-15T03:37:56.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:56.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62011cdf0 con 0x7fe62009f020 2024-09-15T03:37:56.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:56.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62011c6c0 con 0x7fe62009ec50 2024-09-15T03:37:56.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:56.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.090+0000 7fe62595d640 10 monclient(hunting): authenticate will time out at 1280.571934s 2024-09-15T03:37:56.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 0x7fe620130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:56.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 0x7fe620130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60160/0 (socket says 172.21.15.181:60160) 2024-09-15T03:37:56.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 -- 172.21.15.181:0/2795380511 learned_addr learned my addr 172.21.15.181:0/2795380511 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:56.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): get_auth_request con 0x7fe62009f020 auth_method 0 2024-09-15T03:37:56.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:56.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:56.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:56.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61f7fe640 1 -- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6201321a0 msgr2=0x7fe620141050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:56.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61f7fe640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6201321a0 0x7fe620141050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:56.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:56.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:56.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:56.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient(hunting): handle_auth_done global_id 104701 payload 824 2024-09-15T03:37:56.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 -- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe62009ec50 msgr2=0x7fe62009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:56.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe62009ec50 0x7fe62009aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 -- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6201321a0 msgr2=0x7fe620141050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:56.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6201321a0 0x7fe620141050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:56.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:37:56.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 monclient: found mon.noname-b 2024-09-15T03:37:56.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:56.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.091+0000 7fe61e7fc640 1 -- 172.21.15.181:0/2795380511 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614009450 con 0x7fe62009f020 2024-09-15T03:37:56.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe61e7fc640 1 -- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 msgr2=0x7fe620130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:56.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe61e7fc640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 0x7fe620130bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe614009210 tx=0x7fe614032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:56.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61403ca40 con 0x7fe62009f020 2024-09-15T03:37:56.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:56.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:56.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: dump: 2024-09-15T03:37:56.132 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:56.132 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:56.132 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.132 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.133 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:56.133 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:56.133 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:56.133 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:56.133 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:56.134 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:56.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: _finish_auth 0 2024-09-15T03:37:56.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:56.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61403cba0 con 0x7fe62009f020 2024-09-15T03:37:56.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 5 monclient: authenticate success, global_id 104701 2024-09-15T03:37:56.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 10 monclient: _renew_subs 2024-09-15T03:37:56.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62011dce0 con 0x7fe62009f020 2024-09-15T03:37:56.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:56.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140313a0 con 0x7fe62009f020 2024-09-15T03:37:56.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:56.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:56.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe606ffd640 10 monclient: dump: 2024-09-15T03:37:56.136 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:56.136 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:56.136 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.136 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:56.138 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:56.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe61d7fa640 4 set_mon_vals no callback set 2024-09-15T03:37:56.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 10 monclient: _renew_subs 2024-09-15T03:37:56.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.093+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62011e2a0 con 0x7fe62009f020 2024-09-15T03:37:56.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.094+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe614039040 con 0x7fe62009f020 2024-09-15T03:37:56.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.094+0000 7fe62595d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:56.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.094+0000 7fe62595d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:56.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.094+0000 7fe62595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.094+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6200008d0 con 0x7fe62009f020 2024-09-15T03:37:56.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.095+0000 7fe606ffd640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe60c0386f0 0x7fe60c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:56.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.095+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(242..242 src has 199..242) v4 ==== 9610+0+0 (secure 0 0 0) 0x7fe61406d010 con 0x7fe62009f020 2024-09-15T03:37:56.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.095+0000 7fe61effd640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe60c0386f0 0x7fe60c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:56.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.095+0000 7fe61effd640 10 monclient: get_auth_request con 0x7fe60c0386f0 auth_method 0 2024-09-15T03:37:56.140 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.097+0000 7fe61effd640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe60c0386f0 0x7fe60c03aba0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe610006fd0 tx=0x7fe610006cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:56.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.099+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe61409ee40 con 0x7fe62009f020 2024-09-15T03:37:56.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.099+0000 7fe606ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:56.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.099+0000 7fe606ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:56.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.268+0000 7fe62595d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T03:37:56.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.268+0000 7fe62595d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T03:37:56.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.268+0000 7fe62595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:56.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:56.268+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7fe6200a13e0 con 0x7fe62009f020 2024-09-15T03:37:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.242+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v244) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe614040070 con 0x7fe62009f020 2024-09-15T03:37:57.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.242+0000 7fe606ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T03:37:57.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.242+0000 7fe606ffd640 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-15T03:37:57.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe62595d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T03:37:57.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe62595d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T03:37:57.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe62595d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:57.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7fe6200a13e0 con 0x7fe62009f020 2024-09-15T03:37:57.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe606ffd640 1 -- 172.21.15.181:0/2795380511 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v244) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe614037030 con 0x7fe62009f020 2024-09-15T03:37:57.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe606ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-15T03:37:57.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.344+0000 7fe606ffd640 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-15T03:37:57.346 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h26' does not appear in the crush map 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe60c0386f0 msgr2=0x7fe60c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe60c0386f0 0x7fe60c03aba0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe610006fd0 tx=0x7fe610006cf0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 10 monclient: shutdown 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 msgr2=0x7fe620130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 0x7fe620130bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe614009210 tx=0x7fe614032000 comp rx=0 tx=0).stop 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 shutdown_connections 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe60c0386f0 0x7fe60c03aba0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6201321a0 0x7fe620141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.346+0000 7fe62595d640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe62009f020 0x7fe620130bc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.347+0000 7fe62595d640 1 --2- 172.21.15.181:0/2795380511 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe62009ec50 0x7fe62009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.347+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 >> 172.21.15.181:0/2795380511 conn(0x7fe6200a5cf0 msgr2=0x7fe62009e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:57.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.347+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 shutdown_connections 2024-09-15T03:37:57.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.347+0000 7fe62595d640 1 -- 172.21.15.181:0/2795380511 wait complete. 2024-09-15T03:37:57.357 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:57.357 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h27 2024-09-15T03:37:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.430+0000 7f9de5f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.430+0000 7f9de5f1d640 10 monclient: build_initial_monmap 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.430+0000 7f9de5f1d640 10 monclient: monmap: 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:57.431691+0000 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:57.431691+0000 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:57.432 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:57.433 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:57.433 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.431+0000 7f9de5f1d640 1 Processor -- start 2024-09-15T03:37:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.431+0000 7f9de5f1d640 1 -- start start 2024-09-15T03:37:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.431+0000 7f9de5f1d640 10 monclient: init 2024-09-15T03:37:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:37:57.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9de0096a30 addr v1:172.21.15.181:6789/0 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9de009b0b0 0x7f9de009d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9de009b0b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9de0097510 0x7f9de0136e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9de0097510 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de009b970 con 0x7f9de0096a30 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de009c470 con 0x7f9de0097510 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de009c190 con 0x7f9de009b0b0 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9de5f1d640 10 monclient(hunting): authenticate will time out at 1281.913278s 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9de009b0b0 0x7f9de009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:57.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9de009b0b0 0x7f9de009d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60174/0 (socket says 172.21.15.181:60174) 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1215349526 learned_addr learned my addr 172.21.15.181:0/1215349526 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): get_auth_request con 0x7f9de009b0b0 auth_method 0 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddf7fe640 1 --1- 172.21.15.181:0/1215349526 >> v1:172.21.15.181:6789/0 conn(0x7f9de0096a30 0x7f9de0096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:41596/0 (socket says 172.21.15.181:41596) 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddffff640 1 -- 172.21.15.181:0/1215349526 >> v2:172.21.15.153:3300/0 conn(0x7f9de0097510 msgr2=0x7f9de0136e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddffff640 1 --2- 172.21.15.181:0/1215349526 >> v2:172.21.15.153:3300/0 conn(0x7f9de0097510 0x7f9de0136e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:57.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_done global_id 104704 payload 274 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1215349526 >> v2:172.21.15.153:3300/0 conn(0x7f9de0097510 msgr2=0x7f9de0136e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 --2- 172.21.15.181:0/1215349526 >> v2:172.21.15.153:3300/0 conn(0x7f9de0097510 0x7f9de0136e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1215349526 >> v1:172.21.15.181:6789/0 conn(0x7f9de0096a30 legacy=0x7f9de0096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 monclient: found mon.noname-c 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:37:57.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.432+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1215349526 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de009c750 con 0x7f9de009b0b0 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1215349526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de009b0b0 msgr2=0x7f9de009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9ddeffd640 1 --2- 172.21.15.181:0/1215349526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de009b0b0 0x7f9de009d320 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9de0098050 tx=0x7f9dd40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=705a6926bdf2d41e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 1 -- 172.21.15.181:0/1215349526 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402ae50 con 0x7f9de009b0b0 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: dump: 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.444 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: _finish_auth 0 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9de5f1d640 5 monclient: authenticate success, global_id 104704 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 1 -- 172.21.15.181:0/1215349526 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd402f4b0 con 0x7f9de009b0b0 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9de5f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:37:57.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9de5f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 1 -- 172.21.15.181:0/1215349526 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402f850 con 0x7f9de009b0b0 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9de5f1d640 4 set_mon_vals no callback set 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.435+0000 7f9dde7fc640 10 monclient: dump: 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:57.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 10 monclient: shutdown 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1215349526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de009b0b0 msgr2=0x7f9de009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1215349526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de009b0b0 0x7f9de009d320 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9de0098050 tx=0x7f9dd40091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1215349526 shutdown_connections 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1215349526 >> v2:172.21.15.153:3300/0 conn(0x7f9de0097510 0x7f9de0136e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1215349526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de009b0b0 0x7f9de009d320 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1215349526 >> 172.21.15.181:0/1215349526 conn(0x7f9de0071100 msgr2=0x7f9de00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1215349526 shutdown_connections 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1215349526 wait complete. 2024-09-15T03:37:57.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 10 monclient: build_initial_monmap 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.436+0000 7f9de5f1d640 10 monclient: monmap: 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 Processor -- start 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 -- start start 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient: init 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de0096a30 0x7f9de01416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9de0096a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9de0097510 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9de009b0b0 0x7f9de0135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9de009b0b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de011b9f0 con 0x7f9de0096a30 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:57.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de011c910 con 0x7f9de0097510 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de009bd40 con 0x7f9de009b0b0 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.437+0000 7f9de5f1d640 10 monclient(hunting): authenticate will time out at 1281.918937s 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddf7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de0096a30 msgr2=0x7f9de01416a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60182/0 (socket says 172.21.15.181:60182) 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1422287437 learned_addr learned my addr 172.21.15.181:0/1422287437 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): get_auth_request con 0x7f9de0097510 auth_method 0 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddf7fe640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de0096a30 0x7f9de01416a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:57.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient(hunting): handle_auth_done global_id 104707 payload 824 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddffff640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9de009b0b0 0x7f9de0135cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9de009b0b0 msgr2=0x7f9de0135cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9de009b0b0 0x7f9de0135cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de0096a30 msgr2=0x7f9de01416a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de0096a30 0x7f9de01416a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 monclient: found mon.noname-b 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:57.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1422287437 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd4009450 con 0x7f9de0097510 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddffff640 10 monclient: get_auth_request con 0x7f9de009b0b0 auth_method 0 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.438+0000 7f9ddffff640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9de009b0b0 0x7f9de0135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9ddeffd640 1 -- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 msgr2=0x7f9de0141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9ddeffd640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4035040 tx=0x7f9dd4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4032a90 con 0x7f9de0097510 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: dump: 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: _finish_auth 0 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd4032c10 con 0x7f9de0097510 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 5 monclient: authenticate success, global_id 104707 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 10 monclient: _renew_subs 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de009a2c0 con 0x7f9de0097510 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd403b400 con 0x7f9de0097510 2024-09-15T03:37:57.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dc77fe640 10 monclient: dump: 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:57.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 10 monclient: _renew_subs 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de0142120 con 0x7f9de0097510 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.440+0000 7f9dddffb640 4 set_mon_vals no callback set 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9dd4039040 con 0x7f9de0097510 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9de5f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9de5f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9de5f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de011bb60 con 0x7f9de0097510 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9dc77fe640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9dcc0386f0 0x7f9dcc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(244..244 src has 199..244) v4 ==== 9397+0+0 (secure 0 0 0) 0x7f9dd406d180 con 0x7f9de0097510 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9ddf7fe640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9dcc0386f0 0x7f9dcc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:57.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.441+0000 7f9ddf7fe640 10 monclient: get_auth_request con 0x7f9dcc0386f0 auth_method 0 2024-09-15T03:37:57.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.444+0000 7f9ddf7fe640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9dcc0386f0 0x7f9dcc03aba0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0004880 tx=0x7f9dd0007a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:57.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:57.446+0000 7f9ddeffd640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4035040 tx=0x7f9dd4032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f9dd403ab40 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:37:58.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.274+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9dd403ab40 con 0x7f9de0097510 2024-09-15T03:37:58.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.274+0000 7f9dc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:58.275 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.274+0000 7f9dc77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:37:58.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.437+0000 7f9de5f1d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T03:37:58.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.437+0000 7f9de5f1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T03:37:58.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.437+0000 7f9de5f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:58.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:58.437+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f9de00008d0 con 0x7f9de0097510 2024-09-15T03:37:59.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.250+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9dd403f020 con 0x7f9de0097510 2024-09-15T03:37:59.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.250+0000 7f9dc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T03:37:59.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.250+0000 7f9dc77fe640 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.352+0000 7f9de5f1d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.352+0000 7f9de5f1d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.352+0000 7f9de5f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.352+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f9de00008d0 con 0x7f9de0097510 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.353+0000 7f9dc77fe640 1 -- 172.21.15.181:0/1422287437 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9dd403f070 con 0x7f9de0097510 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.353+0000 7f9dc77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.353+0000 7f9dc77fe640 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-15T03:37:59.354 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h27' does not appear in the crush map 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9dcc0386f0 msgr2=0x7f9dcc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9dcc0386f0 0x7f9dcc03aba0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0004880 tx=0x7f9dd0007a30 comp rx=0 tx=0).stop 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 10 monclient: shutdown 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 msgr2=0x7f9de0141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4035040 tx=0x7f9dd4032000 comp rx=0 tx=0).stop 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 shutdown_connections 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9dcc0386f0 0x7f9dcc03aba0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9de009b0b0 0x7f9de0135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9de0097510 0x7f9de0141be0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 --2- 172.21.15.181:0/1422287437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9de0096a30 0x7f9de01416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 >> 172.21.15.181:0/1422287437 conn(0x7f9de0071100 msgr2=0x7f9de009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 shutdown_connections 2024-09-15T03:37:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.355+0000 7f9de5f1d640 1 -- 172.21.15.181:0/1422287437 wait complete. 2024-09-15T03:37:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:37:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h28 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.438+0000 7f83b3e57640 10 monclient: get_monmap_and_config 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.438+0000 7f83b3e57640 10 monclient: build_initial_monmap 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.438+0000 7f83b3e57640 10 monclient: monmap: 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:37:59.438898+0000 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:37:59.438898+0000 2024-09-15T03:37:59.439 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:59.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.438+0000 7f83b3e57640 1 Processor -- start 2024-09-15T03:37:59.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.438+0000 7f83b3e57640 1 -- start start 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.438+0000 7f83b3e57640 10 monclient: init 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f83ac09ec50 0x7f83ac131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): picked mon.noname-a con 0x7f83ac09ec50 addr v2:172.21.15.153:3300/0 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 0x7f83ac068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): picked mon.noname-b con 0x7f83ac1321a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83ac09f020 addr v1:172.21.15.153:6789/0 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac067420 con 0x7f83ac09f020 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac0a0480 con 0x7f83ac09ec50 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:59.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0a0200 con 0x7f83ac1321a0 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b3e57640 10 monclient(hunting): authenticate will time out at 1283.920514s 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b1bf3640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f83ac09ec50 msgr2=0x7f83ac131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b23f4640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f83ac09f020 legacy=0x7f83ac069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b1bf3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f83ac09ec50 0x7f83ac131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 0x7f83ac068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 0x7f83ac068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:44662/0 (socket says 172.21.15.181:44662) 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 1 -- 172.21.15.181:0/3074873916 learned_addr learned my addr 172.21.15.181:0/3074873916 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): get_auth_request con 0x7f83ac1321a0 auth_method 0 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:59.444 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.439+0000 7f83b13f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 10 monclient(hunting): handle_auth_done global_id 124178 payload 274 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 1 -- 172.21.15.181:0/3074873916 >> v2:172.21.15.153:3300/0 conn(0x7f83ac09ec50 msgr2=0x7f83ac131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 1 --2- 172.21.15.181:0/3074873916 >> v2:172.21.15.153:3300/0 conn(0x7f83ac09ec50 0x7f83ac131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 1 -- 172.21.15.181:0/3074873916 >> v1:172.21.15.153:6789/0 conn(0x7f83ac09f020 legacy=0x7f83ac069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 10 monclient: _finish_hunting 0 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 1 monclient: found mon.noname-b 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:37:59.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.440+0000 7f83b13f2640 1 -- 172.21.15.181:0/3074873916 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac09fd80 con 0x7f83ac1321a0 2024-09-15T03:37:59.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.442+0000 7f83b13f2640 1 -- 172.21.15.181:0/3074873916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 msgr2=0x7f83ac068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.442+0000 7f83b13f2640 1 --2- 172.21.15.181:0/3074873916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 0x7f83ac068d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f839c009910 tx=0x7f839c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9550243babb7920c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 1 -- 172.21.15.181:0/3074873916 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c02ae30 con 0x7f83ac1321a0 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: dump: 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:59.452 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: _finish_auth 0 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 5 monclient: authenticate success, global_id 124178 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 1 -- 172.21.15.181:0/3074873916 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c02f4b0 con 0x7f83ac1321a0 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 10 monclient: get_monmap_and_config success 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 4 set_mon_vals no callback set 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 1 -- 172.21.15.181:0/3074873916 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c02f850 con 0x7f83ac1321a0 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:37:59.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b0bf1640 10 monclient: dump: 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 10 monclient: shutdown 2024-09-15T03:37:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 -- 172.21.15.181:0/3074873916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 msgr2=0x7f83ac068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3074873916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 0x7f83ac068d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f839c009910 tx=0x7f839c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 -- 172.21.15.181:0/3074873916 shutdown_connections 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3074873916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac1321a0 0x7f83ac068d00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3074873916 >> v2:172.21.15.153:3300/0 conn(0x7f83ac09ec50 0x7f83ac131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 -- 172.21.15.181:0/3074873916 >> 172.21.15.181:0/3074873916 conn(0x7f83ac0a5cf0 msgr2=0x7f83ac0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 -- 172.21.15.181:0/3074873916 shutdown_connections 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.443+0000 7f83b3e57640 1 -- 172.21.15.181:0/3074873916 wait complete. 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.444+0000 7f83b3e57640 10 monclient: build_initial_monmap 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.444+0000 7f83b3e57640 10 monclient: monmap: 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:37:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.444+0000 7f83b3e57640 1 Processor -- start 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.444+0000 7f83b3e57640 1 -- start start 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.444+0000 7f83b3e57640 10 monclient: init 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient: _reopen_session rank -1 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:59.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): picked mon.noname-b con 0x7f83ac09ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac09f020 0x7f83ac130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): picked mon.noname-c con 0x7f83ac09f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac1321a0 0x7f83ac141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): picked mon.noname-a con 0x7f83ac1321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac11e330 con 0x7f83ac1321a0 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac11dce0 con 0x7f83ac09ec50 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac11dfc0 con 0x7f83ac09f020 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): _renew_subs 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b3e57640 10 monclient(hunting): authenticate will time out at 1283.926169s 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b23f4640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac1321a0 msgr2=0x7f83ac141210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:37:59.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b23f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac1321a0 0x7f83ac141210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49444/0 (socket says 172.21.15.181:49444) 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 -- 172.21.15.181:0/3118640860 learned_addr learned my addr 172.21.15.181:0/3118640860 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): get_auth_request con 0x7f83ac09ec50 auth_method 0 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac09f020 0x7f83ac130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): get_auth_request con 0x7f83ac09f020 auth_method 0 2024-09-15T03:37:59.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b13f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient(hunting): handle_auth_done global_id 104710 payload 824 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 -- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac09f020 msgr2=0x7f83ac130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac09f020 0x7f83ac130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 -- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac1321a0 msgr2=0x7f83ac141210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac1321a0 0x7f83ac141210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient: _finish_hunting 0 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 monclient: found mon.noname-b 2024-09-15T03:37:59.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.445+0000 7f83b1bf3640 1 -- 172.21.15.181:0/3118640860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c009450 con 0x7f83ac09ec50 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83b1bf3640 1 -- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 msgr2=0x7f83ac09a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83b1bf3640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83a800b600 tx=0x7f83a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8015070 con 0x7f83ac09ec50 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: dump: 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.460 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83b3e57640 5 monclient: authenticate success, global_id 104710 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83b3e57640 10 monclient: _renew_subs 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83b3e57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:59.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8003d90 con 0x7f83ac09ec50 2024-09-15T03:37:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac09b4c0 con 0x7f83ac09ec50 2024-09-15T03:37:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:37:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8006380 con 0x7f83ac09ec50 2024-09-15T03:37:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:37:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:37:59.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a1ffb640 10 monclient: dump: 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.447+0000 7f83a3fff640 4 set_mon_vals no callback set 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.448+0000 7f83b3e57640 10 monclient: _renew_subs 2024-09-15T03:37:59.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.448+0000 7f83b3e57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.448+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac11c7a0 con 0x7f83ac09ec50 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.448+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f83a8010440 con 0x7f83ac09ec50 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83a1ffb640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f83900386f0 0x7f839003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(245..245 src has 199..245) v4 ==== 9184+0+0 (secure 0 0 0) 0x7f83a804c640 con 0x7f83ac09ec50 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83b3e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83b3e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83b3e57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac0008d0 con 0x7f83ac09ec50 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83b13f2640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f83900386f0 0x7f839003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:37:59.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.449+0000 7f83b13f2640 10 monclient: get_auth_request con 0x7f83900386f0 auth_method 0 2024-09-15T03:37:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.451+0000 7f83b13f2640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f83900386f0 0x7f839003aba0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f83ac09bf50 tx=0x7f839c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:37:59.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:37:59.453+0000 7f83b1bf3640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83a800b600 tx=0x7f83a80091a0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f83a8018ab0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:38:00.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.281+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f83a8018ab0 con 0x7f83ac09ec50 2024-09-15T03:38:00.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.281+0000 7f83a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:00.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.281+0000 7f83a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:00.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.440+0000 7f83b3e57640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T03:38:00.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.440+0000 7f83b3e57640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T03:38:00.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.440+0000 7f83b3e57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:00.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:00.440+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f83ac0a13e0 con 0x7f83ac09ec50 2024-09-15T03:38:01.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.259+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v246) v1 ==== 121+0+0 (secure 0 0 0) 0x7f83a8029090 con 0x7f83ac09ec50 2024-09-15T03:38:01.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.259+0000 7f83a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T03:38:01.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.259+0000 7f83a1ffb640 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-15T03:38:01.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.359+0000 7f83b3e57640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T03:38:01.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.359+0000 7f83b3e57640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T03:38:01.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.359+0000 7f83b3e57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.359+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f83ac072640 con 0x7f83ac09ec50 2024-09-15T03:38:01.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.360+0000 7f83a1ffb640 1 -- 172.21.15.181:0/3118640860 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v246) v1 ==== 120+0+0 (secure 0 0 0) 0x7f83a8018690 con 0x7f83ac09ec50 2024-09-15T03:38:01.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.360+0000 7f83a1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-15T03:38:01.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.360+0000 7f83a1ffb640 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-15T03:38:01.361 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h28' does not appear in the crush map 2024-09-15T03:38:01.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.361+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f83900386f0 msgr2=0x7f839003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.361+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f83900386f0 0x7f839003aba0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f83ac09bf50 tx=0x7f839c032000 comp rx=0 tx=0).stop 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.361+0000 7f83b3e57640 10 monclient: shutdown 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.361+0000 7f83b3e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.361+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 msgr2=0x7f83ac09a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.361+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83a800b600 tx=0x7f83a80091a0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 shutdown_connections 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f83900386f0 0x7f839003aba0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac1321a0 0x7f83ac141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac09f020 0x7f83ac130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 --2- 172.21.15.181:0/3118640860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83ac09ec50 0x7f83ac09a990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 >> 172.21.15.181:0/3118640860 conn(0x7f83ac0a5cf0 msgr2=0x7f83ac09e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 shutdown_connections 2024-09-15T03:38:01.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.362+0000 7f83b3e57640 1 -- 172.21.15.181:0/3118640860 wait complete. 2024-09-15T03:38:01.372 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:38:01.372 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h29 2024-09-15T03:38:01.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.444+0000 7f328835d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:01.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.444+0000 7f328835d640 10 monclient: build_initial_monmap 2024-09-15T03:38:01.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.444+0000 7f328835d640 10 monclient: monmap: 2024-09-15T03:38:01.445 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:01.445273+0000 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:01.445273+0000 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:01.446 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 Processor -- start 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 -- start start 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient: init 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f328009cee0 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:01.449 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f32800a0010 0x7f3280134590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): picked mon.noname-c con 0x7f32800a0010 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f3280136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): picked mon.noname-b con 0x7f3280134ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328009cd50 con 0x7f328009cee0 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328009d2b0 con 0x7f32800a0010 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328009f430 con 0x7f3280134ad0 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f328835d640 10 monclient(hunting): authenticate will time out at 1285.926852s 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32858f8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f32800a0010 0x7f3280134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32860f9640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f328009cee0 0x7f328009c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57512/0 (socket says 172.21.15.181:57512) 2024-09-15T03:38:01.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32860f9640 1 -- 172.21.15.181:0/4205414059 learned_addr learned my addr 172.21.15.181:0/4205414059 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32858f8640 10 monclient(hunting): get_auth_request con 0x7f32800a0010 auth_method 0 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32858f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32858f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32858f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.445+0000 7f32858f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 10 monclient(hunting): handle_auth_done global_id 104713 payload 274 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 1 -- 172.21.15.181:0/4205414059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 msgr2=0x7f3280136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 1 --2- 172.21.15.181:0/4205414059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f3280136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 1 -- 172.21.15.181:0/4205414059 >> v1:172.21.15.181:6789/0 conn(0x7f328009cee0 legacy=0x7f328009c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:01.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 10 monclient: _finish_hunting 0 2024-09-15T03:38:01.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 1 monclient: found mon.noname-c 2024-09-15T03:38:01.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:01.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.452 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.446+0000 7f32858f8640 1 -- 172.21.15.181:0/4205414059 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32800a0410 con 0x7f32800a0010 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.448+0000 7f32858f8640 1 -- 172.21.15.181:0/4205414059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 msgr2=0x7f3280134590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.448+0000 7f32858f8640 1 --2- 172.21.15.181:0/4205414059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f3280134590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3270009850 tx=0x7f32700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d08a77fb242829 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 1 -- 172.21.15.181:0/4205414059 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270032440 con 0x7f32800a0010 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: dump: 2024-09-15T03:38:01.458 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:01.459 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:01.460 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:01.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: _finish_auth 0 2024-09-15T03:38:01.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:01.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 1 -- 172.21.15.181:0/4205414059 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32700325a0 con 0x7f32800a0010 2024-09-15T03:38:01.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 5 monclient: authenticate success, global_id 104713 2024-09-15T03:38:01.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 1 -- 172.21.15.181:0/4205414059 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270032980 con 0x7f32800a0010 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f32850f7640 10 monclient: dump: 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:01.462 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:01.462 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:01.462 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:01.462 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:01.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:01.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 4 set_mon_vals no callback set 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 10 monclient: shutdown 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 -- 172.21.15.181:0/4205414059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 msgr2=0x7f3280134590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 --2- 172.21.15.181:0/4205414059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f3280134590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3270009850 tx=0x7f32700090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 -- 172.21.15.181:0/4205414059 shutdown_connections 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 --2- 172.21.15.181:0/4205414059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f3280136eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 --2- 172.21.15.181:0/4205414059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f3280134590 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 -- 172.21.15.181:0/4205414059 >> 172.21.15.181:0/4205414059 conn(0x7f32800a3480 msgr2=0x7f328009f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 -- 172.21.15.181:0/4205414059 shutdown_connections 2024-09-15T03:38:01.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.449+0000 7f328835d640 1 -- 172.21.15.181:0/4205414059 wait complete. 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.450+0000 7f328835d640 10 monclient: build_initial_monmap 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.450+0000 7f328835d640 10 monclient: monmap: 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:01.464 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.450+0000 7f328835d640 1 Processor -- start 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.450+0000 7f328835d640 1 -- start start 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.450+0000 7f328835d640 10 monclient: init 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328009cee0 0x7f328011a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): picked mon.noname-a con 0x7f328009cee0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f328011a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): picked mon.noname-b con 0x7f32800a0010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f328011ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): picked mon.noname-c con 0x7f3280134ad0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:01.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328011bc60 con 0x7f328009cee0 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3280099ea0 con 0x7f32800a0010 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32800a2150 con 0x7f3280134ad0 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f328835d640 10 monclient(hunting): authenticate will time out at 1285.932230s 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32860f9640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328009cee0 msgr2=0x7f328011a090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f328011a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32860f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328009cee0 0x7f328011a090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f328011a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49460/0 (socket says 172.21.15.181:49460) 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 -- 172.21.15.181:0/2417184662 learned_addr learned my addr 172.21.15.181:0/2417184662 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:01.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): get_auth_request con 0x7f32800a0010 auth_method 0 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32868fa640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f328011ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32868fa640 10 monclient(hunting): get_auth_request con 0x7f3280134ad0 auth_method 0 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32868fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32868fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32868fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:01.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient(hunting): handle_auth_done global_id 104716 payload 824 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 -- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 msgr2=0x7f328011ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f328011ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 -- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328009cee0 msgr2=0x7f328011a090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328009cee0 0x7f328011a090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient: _finish_hunting 0 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 monclient: found mon.noname-b 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32858f8640 1 -- 172.21.15.181:0/2417184662 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3270009390 con 0x7f32800a0010 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.451+0000 7f32868fa640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f328011ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:01.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32858f8640 1 -- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 msgr2=0x7f328011a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32858f8640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f328011a5d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f32700098b0 tx=0x7f3270032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270000c40 con 0x7f32800a0010 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 10 monclient: dump: 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f328835d640 5 monclient: authenticate success, global_id 104716 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3270031c70 con 0x7f32800a0010 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f328835d640 10 monclient: _renew_subs 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f328835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328011c920 con 0x7f32800a0010 2024-09-15T03:38:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.453+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327003c400 con 0x7f32800a0010 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f32767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f32767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f32767fc640 10 monclient: dump: 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:01.471 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f328835d640 10 monclient: _renew_subs 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f32848f6640 4 set_mon_vals no callback set 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f328835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.454+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3280134790 con 0x7f32800a0010 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f328835d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f328835d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f328835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f327003c810 con 0x7f32800a0010 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328011dde0 con 0x7f32800a0010 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f32767fc640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f32640386f0 0x7f326403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(246..246 src has 199..246) v4 ==== 8971+0+0 (secure 0 0 0) 0x7f327006d4a0 con 0x7f32800a0010 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f32860f9640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f32640386f0 0x7f326403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:01.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.455+0000 7f32860f9640 10 monclient: get_auth_request con 0x7f32640386f0 auth_method 0 2024-09-15T03:38:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.457+0000 7f32860f9640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f32640386f0 0x7f326403aba0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f327c0047d0 tx=0x7f327c007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.459+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f32700426a0 con 0x7f32800a0010 2024-09-15T03:38:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.459+0000 7f32767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:01.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.459+0000 7f32767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:01.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.615+0000 7f328835d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T03:38:01.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.615+0000 7f328835d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T03:38:01.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.615+0000 7f328835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:01.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:01.615+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f32800008d0 con 0x7f32800a0010 2024-09-15T03:38:02.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.268+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v247) v1 ==== 121+0+0 (secure 0 0 0) 0x7f3270037020 con 0x7f32800a0010 2024-09-15T03:38:02.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.268+0000 7f32767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T03:38:02.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.268+0000 7f32767fc640 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-15T03:38:02.366 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.365+0000 7f328835d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.365+0000 7f328835d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.365+0000 7f328835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.365+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f32800008d0 con 0x7f32800a0010 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.366+0000 7f32767fc640 1 -- 172.21.15.181:0/2417184662 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v247) v1 ==== 120+0+0 (secure 0 0 0) 0x7f327006d090 con 0x7f32800a0010 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.366+0000 7f32767fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.366+0000 7f32767fc640 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-15T03:38:02.367 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h29' does not appear in the crush map 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f32640386f0 msgr2=0x7f326403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f32640386f0 0x7f326403aba0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f327c0047d0 tx=0x7f327c007b80 comp rx=0 tx=0).stop 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 10 monclient: shutdown 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 msgr2=0x7f328011a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f328011a5d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f32700098b0 tx=0x7f3270032000 comp rx=0 tx=0).stop 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 shutdown_connections 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f32640386f0 0x7f326403aba0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3280134ad0 0x7f328011ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.367+0000 7f328835d640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32800a0010 0x7f328011a5d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.368+0000 7f328835d640 1 --2- 172.21.15.181:0/2417184662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328009cee0 0x7f328011a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.368+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 >> 172.21.15.181:0/2417184662 conn(0x7f32800a3480 msgr2=0x7f3280135a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.368+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 shutdown_connections 2024-09-15T03:38:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.368+0000 7f328835d640 1 -- 172.21.15.181:0/2417184662 wait complete. 2024-09-15T03:38:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:38:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h30 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.449+0000 7ff95042b640 10 monclient: get_monmap_and_config 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.450+0000 7ff95042b640 10 monclient: build_initial_monmap 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.450+0000 7ff95042b640 10 monclient: monmap: 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:02.450809+0000 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:02.450809+0000 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:02.451 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:02.452 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:02.452 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:02.452 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:02.452 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:02.452 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:02.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.450+0000 7ff95042b640 1 Processor -- start 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.450+0000 7ff95042b640 1 -- start start 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.450+0000 7ff95042b640 10 monclient: init 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 0x7ff948131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient(hunting): picked mon.noname-b con 0x7ff948131c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9480a0b00 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff948132510 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9480a1b70 con 0x7ff9480a0b00 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9480a08c0 con 0x7ff948132510 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff948132910 con 0x7ff948131c00 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff95042b640 10 monclient(hunting): authenticate will time out at 1286.932297s 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e9c8640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7ff948132510 0x7ff948069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57528/0 (socket says 172.21.15.181:57528) 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e9c8640 1 -- 172.21.15.181:0/3588587991 learned_addr learned my addr 172.21.15.181:0/3588587991 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94d9c6640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7ff9480a0b00 legacy=0x7ff948068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 1 --2- 172.21.15.181:0/3588587991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 0x7ff948131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): get_auth_request con 0x7ff948131c00 auth_method 0 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.451+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_done global_id 124187 payload 274 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 1 -- 172.21.15.181:0/3588587991 >> v1:172.21.15.181:6789/0 conn(0x7ff948132510 legacy=0x7ff948069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 1 -- 172.21.15.181:0/3588587991 >> v1:172.21.15.153:6789/0 conn(0x7ff9480a0b00 legacy=0x7ff948068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 10 monclient: _finish_hunting 0 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 1 monclient: found mon.noname-b 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.452+0000 7ff94e1c7640 1 -- 172.21.15.181:0/3588587991 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9480a0200 con 0x7ff948131c00 2024-09-15T03:38:02.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.454+0000 7ff94e1c7640 1 -- 172.21.15.181:0/3588587991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 msgr2=0x7ff948131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:02.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.454+0000 7ff94e1c7640 1 --2- 172.21.15.181:0/3588587991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 0x7ff948131fd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff944009880 tx=0x7ff9440090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d7b8ecc7fe9f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:02.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 1 -- 172.21.15.181:0/3588587991 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944032440 con 0x7ff948131c00 2024-09-15T03:38:02.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:02.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:02.465 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: dump: 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: _finish_auth 0 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:02.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 1 -- 172.21.15.181:0/3588587991 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440325a0 con 0x7ff948131c00 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 1 -- 172.21.15.181:0/3588587991 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944032940 con 0x7ff948131c00 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff94d1c5640 10 monclient: dump: 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:02.467 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff95042b640 5 monclient: authenticate success, global_id 124187 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff95042b640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.455+0000 7ff95042b640 4 set_mon_vals no callback set 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 10 monclient: shutdown 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 -- 172.21.15.181:0/3588587991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 msgr2=0x7ff948131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:02.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 --2- 172.21.15.181:0/3588587991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 0x7ff948131fd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff944009880 tx=0x7ff9440090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 -- 172.21.15.181:0/3588587991 shutdown_connections 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 --2- 172.21.15.181:0/3588587991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948131c00 0x7ff948131fd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 -- 172.21.15.181:0/3588587991 >> 172.21.15.181:0/3588587991 conn(0x7ff948071370 msgr2=0x7ff9480a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 -- 172.21.15.181:0/3588587991 shutdown_connections 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 1 -- 172.21.15.181:0/3588587991 wait complete. 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 10 monclient: build_initial_monmap 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.456+0000 7ff95042b640 10 monclient: monmap: 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 Processor -- start 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 -- start start 2024-09-15T03:38:02.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient: init 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 0x7ff9481411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): picked mon.noname-b con 0x7ff9480a0b00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff948131c00 0x7ff94812ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): picked mon.noname-a con 0x7ff948131c00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948132510 0x7ff94812f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): picked mon.noname-c con 0x7ff948132510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94812f9e0 con 0x7ff948131c00 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94811d650 con 0x7ff9480a0b00 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94811c2a0 con 0x7ff948132510 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff95042b640 10 monclient(hunting): authenticate will time out at 1286.938848s 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94d9c6640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff948131c00 msgr2=0x7ff94812ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94d9c6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff948131c00 0x7ff94812ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 0x7ff9481411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 0x7ff9481411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49468/0 (socket says 172.21.15.181:49468) 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 1 -- 172.21.15.181:0/1832068674 learned_addr learned my addr 172.21.15.181:0/1832068674 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 10 monclient(hunting): get_auth_request con 0x7ff9480a0b00 auth_method 0 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.457+0000 7ff94e1c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948132510 0x7ff94812f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): get_auth_request con 0x7ff948132510 auth_method 0 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e9c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 10 monclient(hunting): handle_auth_done global_id 104719 payload 824 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 1 -- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948132510 msgr2=0x7ff94812f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948132510 0x7ff94812f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 1 -- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff948131c00 msgr2=0x7ff94812ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:02.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff948131c00 0x7ff94812ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 10 monclient: _finish_hunting 0 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 1 monclient: found mon.noname-b 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.458+0000 7ff94e1c7640 1 -- 172.21.15.181:0/1832068674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944009390 con 0x7ff9480a0b00 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff94e1c7640 1 -- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 msgr2=0x7ff9481411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff94e1c7640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 0x7ff9481411e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff944009880 tx=0x7ff944032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944031990 con 0x7ff9480a0b00 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: dump: 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:02.474 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 5 monclient: authenticate success, global_id 104719 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff944031b30 con 0x7ff9480a0b00 2024-09-15T03:38:02.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944044410 con 0x7ff9480a0b00 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 10 monclient: _renew_subs 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94811d370 con 0x7ff9480a0b00 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff93e7fc640 10 monclient: dump: 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:02.476 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff94c9c4640 4 set_mon_vals no callback set 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 10 monclient: _renew_subs 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.460+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff948141720 con 0x7ff9480a0b00 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff944039040 con 0x7ff9480a0b00 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff95042b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff95042b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff95042b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:02.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9480008d0 con 0x7ff9480a0b00 2024-09-15T03:38:02.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff93e7fc640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff92c038720 0x7ff92c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:02.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(247..247 src has 199..247) v4 ==== 8758+0+0 (secure 0 0 0) 0x7ff94406d100 con 0x7ff9480a0b00 2024-09-15T03:38:02.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff94d9c6640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff92c038720 0x7ff92c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:02.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.461+0000 7ff94d9c6640 10 monclient: get_auth_request con 0x7ff92c038720 auth_method 0 2024-09-15T03:38:02.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.464+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:38:02.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.464+0000 7ff94d9c6640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff92c038720 0x7ff92c03abd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff9380038f0 tx=0x7ff938006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:02.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.466+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff94403b400 con 0x7ff9480a0b00 2024-09-15T03:38:02.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.466+0000 7ff93e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:02.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.466+0000 7ff93e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:02.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.622+0000 7ff95042b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T03:38:02.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.622+0000 7ff95042b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T03:38:02.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.622+0000 7ff95042b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:02.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:02.622+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7ff9480008d0 con 0x7ff9480a0b00 2024-09-15T03:38:03.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.268+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v248) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff94403f020 con 0x7ff9480a0b00 2024-09-15T03:38:03.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.268+0000 7ff93e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T03:38:03.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.268+0000 7ff93e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-15T03:38:03.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.368+0000 7ff95042b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.368+0000 7ff95042b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.368+0000 7ff95042b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.368+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7ff9480008d0 con 0x7ff9480a0b00 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.369+0000 7ff93e7fc640 1 -- 172.21.15.181:0/1832068674 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v248) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff94403f030 con 0x7ff9480a0b00 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.369+0000 7ff93e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.369+0000 7ff93e7fc640 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-15T03:38:03.370 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h30' does not appear in the crush map 2024-09-15T03:38:03.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff92c038720 msgr2=0x7ff92c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:03.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff92c038720 0x7ff92c03abd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff9380038f0 tx=0x7ff938006d80 comp rx=0 tx=0).stop 2024-09-15T03:38:03.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 10 monclient: shutdown 2024-09-15T03:38:03.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 msgr2=0x7ff9481411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 0x7ff9481411e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff944009880 tx=0x7ff944032000 comp rx=0 tx=0).stop 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 shutdown_connections 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff92c038720 0x7ff92c03abd0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff948132510 0x7ff94812f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff948131c00 0x7ff94812ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 --2- 172.21.15.181:0/1832068674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480a0b00 0x7ff9481411e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 >> 172.21.15.181:0/1832068674 conn(0x7ff948071370 msgr2=0x7ff948130f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 shutdown_connections 2024-09-15T03:38:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.371+0000 7ff95042b640 1 -- 172.21.15.181:0/1832068674 wait complete. 2024-09-15T03:38:03.382 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:38:03.382 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h31 2024-09-15T03:38:03.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.454+0000 7f5f3799d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.454+0000 7f5f3799d640 10 monclient: build_initial_monmap 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.454+0000 7f5f3799d640 10 monclient: monmap: 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:03.455759+0000 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:03.455759+0000 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:03.456 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:03.457 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.455+0000 7f5f3799d640 1 Processor -- start 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.455+0000 7f5f3799d640 1 -- start start 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.455+0000 7f5f3799d640 10 monclient: init 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5f30131c00 0x7f5f30131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f30131c00 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5f300a0b00 0x7f5f30132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f300a0b00 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f30068d00 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3009d230 con 0x7f5f30068d00 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f300a08c0 con 0x7f5f300a0b00 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f300a0200 con 0x7f5f30131c00 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f3799d640 10 monclient(hunting): authenticate will time out at 1287.937288s 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5f30131c00 0x7f5f30131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f34f38640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f5f300a0b00 msgr2=0x7f5f30132510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35f3a640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f5f30068d00 0x7f5f30069100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57536/0 (socket says 172.21.15.181:57536) 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5f30131c00 0x7f5f30131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49474/0 (socket says 172.21.15.181:49474) 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35f3a640 1 -- 172.21.15.181:0/2455491972 learned_addr learned my addr 172.21.15.181:0/2455491972 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): get_auth_request con 0x7f5f30131c00 auth_method 0 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f34f38640 1 --2- 172.21.15.181:0/2455491972 >> v2:172.21.15.153:3300/0 conn(0x7f5f300a0b00 0x7f5f30132510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient(hunting): handle_auth_done global_id 104722 payload 274 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 -- 172.21.15.181:0/2455491972 >> v2:172.21.15.153:3300/0 conn(0x7f5f300a0b00 msgr2=0x7f5f30132510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:03.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 --2- 172.21.15.181:0/2455491972 >> v2:172.21.15.153:3300/0 conn(0x7f5f300a0b00 0x7f5f30132510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 -- 172.21.15.181:0/2455491972 >> v1:172.21.15.181:6789/0 conn(0x7f5f30068d00 legacy=0x7f5f30069100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient: _finish_hunting 0 2024-09-15T03:38:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 monclient: found mon.noname-c 2024-09-15T03:38:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.456+0000 7f5f35739640 1 -- 172.21.15.181:0/2455491972 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300a1b70 con 0x7f5f30131c00 2024-09-15T03:38:03.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f35739640 1 -- 172.21.15.181:0/2455491972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 msgr2=0x7f5f30131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f35739640 1 --2- 172.21.15.181:0/2455491972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f30131fd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009e60 tx=0x7f5f20009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ebae86cc6f8c0bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 1 -- 172.21.15.181:0/2455491972 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20007840 con 0x7f5f30131c00 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: dump: 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: _finish_auth 0 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 1 -- 172.21.15.181:0/2455491972 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2002d050 con 0x7f5f30131c00 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f3799d640 5 monclient: authenticate success, global_id 104722 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f3799d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f3799d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f3799d640 4 set_mon_vals no callback set 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 1 -- 172.21.15.181:0/2455491972 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20007bd0 con 0x7f5f30131c00 2024-09-15T03:38:03.469 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.459+0000 7f5f2ffff640 10 monclient: dump: 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 10 monclient: shutdown 2024-09-15T03:38:03.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 -- 172.21.15.181:0/2455491972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 msgr2=0x7f5f30131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2455491972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f30131fd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009e60 tx=0x7f5f20009780 comp rx=0 tx=0).stop 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 -- 172.21.15.181:0/2455491972 shutdown_connections 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2455491972 >> v2:172.21.15.153:3300/0 conn(0x7f5f300a0b00 0x7f5f30132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2455491972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f30131fd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 -- 172.21.15.181:0/2455491972 >> 172.21.15.181:0/2455491972 conn(0x7f5f30071370 msgr2=0x7f5f300a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 -- 172.21.15.181:0/2455491972 shutdown_connections 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 1 -- 172.21.15.181:0/2455491972 wait complete. 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 10 monclient: build_initial_monmap 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.460+0000 7f5f3799d640 10 monclient: monmap: 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:03.471 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 Processor -- start 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 -- start start 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient: init 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:38:03.472 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f30068d00 0x7f5f3009ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f30068d00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f300a0b00 0x7f5f30130c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f300a0b00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f30131c00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f30134b80 con 0x7f5f30068d00 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3011e080 con 0x7f5f30131c00 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f301415a0 con 0x7f5f300a0b00 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:03.473 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f3799d640 10 monclient(hunting): authenticate will time out at 1287.942666s 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49476/0 (socket says 172.21.15.181:49476) 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 1 -- 172.21.15.181:0/2398192861 learned_addr learned my addr 172.21.15.181:0/2398192861 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): get_auth_request con 0x7f5f30131c00 auth_method 0 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35739640 1 -- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f30068d00 msgr2=0x7f5f3009ab00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35739640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f30068d00 0x7f5f3009ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:03.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f34f38640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f300a0b00 0x7f5f30130c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f34f38640 10 monclient(hunting): get_auth_request con 0x7f5f300a0b00 auth_method 0 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f34f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f34f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f34f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.461+0000 7f5f35f3a640 10 monclient(hunting): handle_auth_done global_id 104725 payload 824 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 1 -- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f300a0b00 msgr2=0x7f5f30130c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f300a0b00 0x7f5f30130c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 1 -- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f30068d00 msgr2=0x7f5f3009ab00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f30068d00 0x7f5f3009ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 10 monclient: _finish_hunting 0 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 1 monclient: found mon.noname-b 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:03.475 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.462+0000 7f5f35f3a640 1 -- 172.21.15.181:0/2398192861 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f20009a20 con 0x7f5f30131c00 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f35f3a640 1 -- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 msgr2=0x7f5f301311c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f35f3a640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c540 tx=0x7f5f28004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f35f3a640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c540 tx=0x7f5f28004900 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f5f28005cb0 mon_map magic: 0 v1 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28005cb0 con 0x7f5f30131c00 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 10 monclient: dump: 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.476 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f28005e10 con 0x7f5f30131c00 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 5 monclient: authenticate success, global_id 104725 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 10 monclient: _renew_subs 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3011dda0 con 0x7f5f30131c00 2024-09-15T03:38:03.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 10 monclient: _renew_subs 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f30141ee0 con 0x7f5f30131c00 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.464+0000 7f5f2f7fe640 4 set_mon_vals no callback set 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28007320 con 0x7f5f30131c00 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f2d7fa640 10 monclient: dump: 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:03.478 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5f2801b3b0 con 0x7f5f30131c00 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f3799d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f3799d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f3799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3011e240 con 0x7f5f30131c00 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f2d7fa640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5f1c038740 0x7f5f1c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f35739640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5f1c038740 0x7f5f1c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:03.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.465+0000 7f5f35739640 10 monclient: get_auth_request con 0x7f5f1c038740 auth_method 0 2024-09-15T03:38:03.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.468+0000 7f5f35739640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5f1c038740 0x7f5f1c03abf0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5f300a0ad0 tx=0x7f5f20037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:03.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:03.470+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(248..248 src has 199..248) v4 ==== 8545+0+0 (secure 0 0 0) 0x7f5f2804bed0 con 0x7f5f30131c00 2024-09-15T03:38:04.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.087+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5f28018680 con 0x7f5f30131c00 2024-09-15T03:38:04.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.087+0000 7f5f2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:04.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.087+0000 7f5f2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:04.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.249+0000 7f5f3799d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T03:38:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.249+0000 7f5f3799d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T03:38:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.249+0000 7f5f3799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.249+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7f5f300008d0 con 0x7f5f30131c00 2024-09-15T03:38:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.285+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v249) v1 ==== 121+0+0 (secure 0 0 0) 0x7f5f28017070 con 0x7f5f30131c00 2024-09-15T03:38:04.286 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.285+0000 7f5f2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T03:38:04.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.285+0000 7f5f2d7fa640 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-15T03:38:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f3799d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T03:38:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f3799d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T03:38:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f3799d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7f5f300a1890 con 0x7f5f30131c00 2024-09-15T03:38:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f2d7fa640 1 -- 172.21.15.181:0/2398192861 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v249) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5f2800e290 con 0x7f5f30131c00 2024-09-15T03:38:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f2d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-15T03:38:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.381+0000 7f5f2d7fa640 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-15T03:38:04.383 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h31' does not appear in the crush map 2024-09-15T03:38:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5f1c038740 msgr2=0x7f5f1c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5f1c038740 0x7f5f1c03abf0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5f300a0ad0 tx=0x7f5f20037000 comp rx=0 tx=0).stop 2024-09-15T03:38:04.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 10 monclient: shutdown 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 msgr2=0x7f5f301311c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c540 tx=0x7f5f28004900 comp rx=0 tx=0).stop 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 shutdown_connections 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5f1c038740 0x7f5f1c03abf0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f30131c00 0x7f5f301311c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f300a0b00 0x7f5f30130c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 --2- 172.21.15.181:0/2398192861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f30068d00 0x7f5f3009ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 >> 172.21.15.181:0/2398192861 conn(0x7f5f30071370 msgr2=0x7f5f30134d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 shutdown_connections 2024-09-15T03:38:04.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.383+0000 7f5f3799d640 1 -- 172.21.15.181:0/2398192861 wait complete. 2024-09-15T03:38:04.393 INFO:tasks.workunit.client.0.smithi181.stderr:+ for f in `seq 1 32` 2024-09-15T03:38:04.393 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm h32 2024-09-15T03:38:04.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.465+0000 7f6792cd4640 10 monclient: get_monmap_and_config 2024-09-15T03:38:04.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.465+0000 7f6792cd4640 10 monclient: build_initial_monmap 2024-09-15T03:38:04.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.465+0000 7f6792cd4640 10 monclient: monmap: 2024-09-15T03:38:04.467 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:04.467 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:04.467 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:04.466273+0000 2024-09-15T03:38:04.467 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:04.466273+0000 2024-09-15T03:38:04.467 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:04.468 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:04.468 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:04.468 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:04.468 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:04.468 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:04.468 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:04.469 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:04.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 1 Processor -- start 2024-09-15T03:38:04.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 1 -- start start 2024-09-15T03:38:04.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient: init 2024-09-15T03:38:04.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:04.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:38:04.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f678c0a1800 0x7f678c09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f678c0a1800 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f678c0983d0 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f678c12ec70 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c0a1bd0 con 0x7f678c12ec70 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c097850 con 0x7f678c0983d0 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c09cbf0 con 0x7f678c0a1800 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6792cd4640 10 monclient(hunting): authenticate will time out at 1288.947757s 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f678c0a1800 0x7f678c09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6791271640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f678c12ec70 legacy=0x7f678c097140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:38:04.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f678c0a1800 0x7f678c09aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49492/0 (socket says 172.21.15.181:49492) 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 1 -- 172.21.15.181:0/1524753249 learned_addr learned my addr 172.21.15.181:0/1524753249 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): get_auth_request con 0x7f678c0a1800 auth_method 0 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.466+0000 7f6790a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:04.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 10 monclient(hunting): handle_auth_done global_id 104728 payload 274 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 1 -- 172.21.15.181:0/1524753249 >> v1:172.21.15.181:6789/0 conn(0x7f678c0983d0 legacy=0x7f678c096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 1 -- 172.21.15.181:0/1524753249 >> v1:172.21.15.153:6789/0 conn(0x7f678c12ec70 legacy=0x7f678c097140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 10 monclient: _finish_hunting 0 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 1 monclient: found mon.noname-c 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.467+0000 7f6790a70640 1 -- 172.21.15.181:0/1524753249 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c12f600 con 0x7f678c0a1800 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.469+0000 7f6790a70640 1 -- 172.21.15.181:0/1524753249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c0a1800 msgr2=0x7f678c09aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.469+0000 7f6790a70640 1 --2- 172.21.15.181:0/1524753249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c0a1800 0x7f678c09aa90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6780009910 tx=0x7f67800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7c1b69d3e98543 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 1 -- 172.21.15.181:0/1524753249 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780032440 con 0x7f678c0a1800 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: dump: 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:04.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 1 -- 172.21.15.181:0/1524753249 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67800325a0 con 0x7f678c0a1800 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 5 monclient: authenticate success, global_id 104728 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 1 -- 172.21.15.181:0/1524753249 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780032940 con 0x7f678c0a1800 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 4 set_mon_vals no callback set 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f678b7fe640 10 monclient: dump: 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:04.492 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 10 monclient: shutdown 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 -- 172.21.15.181:0/1524753249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c0a1800 msgr2=0x7f678c09aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 --2- 172.21.15.181:0/1524753249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c0a1800 0x7f678c09aa90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6780009910 tx=0x7f67800091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 -- 172.21.15.181:0/1524753249 shutdown_connections 2024-09-15T03:38:04.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 --2- 172.21.15.181:0/1524753249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c0a1800 0x7f678c09aa90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 -- 172.21.15.181:0/1524753249 >> 172.21.15.181:0/1524753249 conn(0x7f678c071100 msgr2=0x7f678c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 -- 172.21.15.181:0/1524753249 shutdown_connections 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.470+0000 7f6792cd4640 1 -- 172.21.15.181:0/1524753249 wait complete. 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient: build_initial_monmap 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient: monmap: 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:04.494 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 1 Processor -- start 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 1 -- start start 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient: init 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f678c0a1800 0x7f678c09b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f678c0a1800 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f678c12ec70 0x7f678c117a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.471+0000 7f6792cd4640 10 monclient(hunting): picked mon.noname-a con 0x7f678c12ec70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 0x7f678c12e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 10 monclient(hunting): picked mon.noname-b con 0x7f678c12dc00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c117f60 con 0x7f678c12ec70 2024-09-15T03:38:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c118630 con 0x7f678c12dc00 2024-09-15T03:38:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c1319a0 con 0x7f678c0a1800 2024-09-15T03:38:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6792cd4640 10 monclient(hunting): authenticate will time out at 1288.953039s 2024-09-15T03:38:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 0x7f678c12e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 0x7f678c12e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49500/0 (socket says 172.21.15.181:49500) 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 -- 172.21.15.181:0/3627811709 learned_addr learned my addr 172.21.15.181:0/3627811709 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): get_auth_request con 0x7f678c12dc00 auth_method 0 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f678bfff640 1 -- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f678c12ec70 msgr2=0x7f678c117a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f678bfff640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f678c12ec70 0x7f678c117a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient(hunting): handle_auth_done global_id 104731 payload 824 2024-09-15T03:38:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 -- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f678c0a1800 msgr2=0x7f678c09b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6790a70640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f678c0a1800 0x7f678c09b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f678c0a1800 0x7f678c09b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 -- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f678c12ec70 msgr2=0x7f678c117a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f678c12ec70 0x7f678c117a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient: _finish_hunting 0 2024-09-15T03:38:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 monclient: found mon.noname-b 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6791271640 1 -- 172.21.15.181:0/3627811709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780009450 con 0x7f678c12dc00 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6790a70640 10 monclient: get_auth_request con 0x7f678c0a1800 auth_method 0 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.472+0000 7f6790a70640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f678c0a1800 0x7f678c09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6791271640 1 -- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 msgr2=0x7f678c12e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6791271640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 0x7f678c12e060 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f678400bb40 tx=0x7f67840097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678400cbf0 con 0x7f678c12dc00 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: dump: 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:04.501 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: _finish_auth 0 2024-09-15T03:38:04.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 5 monclient: authenticate success, global_id 104731 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 10 monclient: _renew_subs 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678400cd70 con 0x7f678c12dc00 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c1189e0 con 0x7f678c12dc00 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784006150 con 0x7f678c12dc00 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6788ff9640 10 monclient: dump: 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.503 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 10 monclient: _renew_subs 2024-09-15T03:38:04.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f678affd640 4 set_mon_vals no callback set 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.474+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c11e030 con 0x7f678c12dc00 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.475+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6784012020 con 0x7f678c12dc00 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.475+0000 7f6792cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.475+0000 7f6792cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.475+0000 7f6792cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.475+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c0008d0 con 0x7f678c12dc00 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.475+0000 7f6788ff9640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6770038720 0x7f677003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.476+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(249..249 src has 199..249) v4 ==== 8332+0+0 (secure 0 0 0) 0x7f678404c870 con 0x7f678c12dc00 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.476+0000 7f6790a70640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6770038720 0x7f677003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.476+0000 7f6790a70640 10 monclient: get_auth_request con 0x7f6770038720 auth_method 0 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.478+0000 7f6790a70640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6770038720 0x7f677003abd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f678c099a10 tx=0x7f6780032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:04.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.480+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f678401da80 con 0x7f678c12dc00 2024-09-15T03:38:04.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.480+0000 7f6788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:04.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.480+0000 7f6788ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:04.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.638+0000 7f6792cd4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T03:38:04.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.638+0000 7f6792cd4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T03:38:04.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.638+0000 7f6792cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:04.640 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:04.638+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7f678c098cc0 con 0x7f678c12dc00 2024-09-15T03:38:05.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.291+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v250) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6784006580 con 0x7f678c12dc00 2024-09-15T03:38:05.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.291+0000 7f6788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T03:38:05.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.291+0000 7f6788ff9640 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.388+0000 7f6792cd4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.388+0000 7f6792cd4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.388+0000 7f6792cd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.388+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7f678c072640 con 0x7f678c12dc00 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.389+0000 7f6788ff9640 1 -- 172.21.15.181:0/3627811709 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v250) v1 ==== 120+0+0 (secure 0 0 0) 0x7f678401b020 con 0x7f678c12dc00 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.389+0000 7f6788ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.389+0000 7f6788ff9640 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-15T03:38:05.390 INFO:tasks.workunit.client.0.smithi181.stderr:device 'h32' does not appear in the crush map 2024-09-15T03:38:05.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6770038720 msgr2=0x7f677003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:05.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6770038720 0x7f677003abd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f678c099a10 tx=0x7f6780032000 comp rx=0 tx=0).stop 2024-09-15T03:38:05.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 10 monclient: shutdown 2024-09-15T03:38:05.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:05.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 msgr2=0x7f678c12e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:05.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 0x7f678c12e060 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f678400bb40 tx=0x7f67840097d0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 shutdown_connections 2024-09-15T03:38:05.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6770038720 0x7f677003abd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f678c12dc00 0x7f678c12e060 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f678c12ec70 0x7f678c117a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 --2- 172.21.15.181:0/3627811709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f678c0a1800 0x7f678c09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 >> 172.21.15.181:0/3627811709 conn(0x7f678c071100 msgr2=0x7f678c09d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:05.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 shutdown_connections 2024-09-15T03:38:05.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.391+0000 7f6792cd4640 1 -- 172.21.15.181:0/3627811709 wait complete. 2024-09-15T03:38:05.400 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush rm r1 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.473+0000 7f3cfd59d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.473+0000 7f3cfd59d640 10 monclient: build_initial_monmap 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.473+0000 7f3cfd59d640 10 monclient: monmap: 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:05.473886+0000 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:05.473886+0000 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:05.474 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:05.475 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:05.475 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:05.475 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.473+0000 7f3cfd59d640 1 Processor -- start 2024-09-15T03:38:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.473+0000 7f3cfd59d640 1 -- start start 2024-09-15T03:38:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.473+0000 7f3cfd59d640 10 monclient: init 2024-09-15T03:38:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809ec50 0x7f3cf8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf809ec50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3cf81321a0 0x7f3cf8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf81321a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cf809f020 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf8067420 con 0x7f3cf809f020 2024-09-15T03:38:05.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:05.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf80a0480 con 0x7f3cf81321a0 2024-09-15T03:38:05.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:05.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf80a0200 con 0x7f3cf809ec50 2024-09-15T03:38:05.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:05.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cfd59d640 10 monclient(hunting): authenticate will time out at 1289.955369s 2024-09-15T03:38:05.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3cf81321a0 0x7f3cf8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cf77fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f3cf809f020 0x7f3cf8069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57556/0 (socket says 172.21.15.181:57556) 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): get_auth_request con 0x7f3cf81321a0 auth_method 0 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cf77fe640 1 -- 172.21.15.181:0/3629808033 learned_addr learned my addr 172.21.15.181:0/3629808033 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cf6ffd640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809ec50 0x7f3cf8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:05.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient(hunting): handle_auth_done global_id 104734 payload 274 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 1 -- 172.21.15.181:0/3629808033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809ec50 msgr2=0x7f3cf8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809ec50 0x7f3cf8131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 1 -- 172.21.15.181:0/3629808033 >> v1:172.21.15.181:6789/0 conn(0x7f3cf809f020 legacy=0x7f3cf8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 1 monclient: found mon.noname-c 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cee7fc640 1 -- 172.21.15.181:0/3629808033 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf809fd80 con 0x7f3cf81321a0 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cf6ffd640 10 monclient: get_auth_request con 0x7f3cf809ec50 auth_method 0 2024-09-15T03:38:05.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.474+0000 7f3cf6ffd640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809ec50 0x7f3cf8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cee7fc640 1 -- 172.21.15.181:0/3629808033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf81321a0 msgr2=0x7f3cf8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cee7fc640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf81321a0 0x7f3cf8068d00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009850 tx=0x7f3ce00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3620b2efe1054432 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 1 -- 172.21.15.181:0/3629808033 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0032440 con 0x7f3cf81321a0 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: dump: 2024-09-15T03:38:05.488 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:05.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 1 -- 172.21.15.181:0/3629808033 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce00325a0 con 0x7f3cf81321a0 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cfd59d640 5 monclient: authenticate success, global_id 104734 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cfd59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cfd59d640 4 set_mon_vals no callback set 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 1 -- 172.21.15.181:0/3629808033 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0032980 con 0x7f3cf81321a0 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.477+0000 7f3cf67fc640 10 monclient: dump: 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 10 monclient: shutdown 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 -- 172.21.15.181:0/3629808033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf81321a0 msgr2=0x7f3cf8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf81321a0 0x7f3cf8068d00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009850 tx=0x7f3ce00090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 -- 172.21.15.181:0/3629808033 shutdown_connections 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf81321a0 0x7f3cf8068d00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/3629808033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809ec50 0x7f3cf8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 -- 172.21.15.181:0/3629808033 >> 172.21.15.181:0/3629808033 conn(0x7f3cf80a5cf0 msgr2=0x7f3cf80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:05.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 -- 172.21.15.181:0/3629808033 shutdown_connections 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.478+0000 7f3cfd59d640 1 -- 172.21.15.181:0/3629808033 wait complete. 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 10 monclient: build_initial_monmap 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 10 monclient: monmap: 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:05.492 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 1 Processor -- start 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 1 -- start start 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 10 monclient: init 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 0x7f3cf809ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:05.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf809ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.479+0000 7f3cfd59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809f020 0x7f3cf8130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf809f020 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cf81321a0 0x7f3cf81310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf81321a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf809c020 con 0x7f3cf81321a0 2024-09-15T03:38:05.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:05.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf8134b50 con 0x7f3cf809ec50 2024-09-15T03:38:05.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:05.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf811dfa0 con 0x7f3cf809f020 2024-09-15T03:38:05.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:05.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cfd59d640 10 monclient(hunting): authenticate will time out at 1289.961068s 2024-09-15T03:38:05.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 0x7f3cf809ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:05.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf77fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cf81321a0 msgr2=0x7f3cf81310e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:05.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cf81321a0 0x7f3cf81310e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:05.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 0x7f3cf809ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49514/0 (socket says 172.21.15.181:49514) 2024-09-15T03:38:05.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 -- 172.21.15.181:0/2129976391 learned_addr learned my addr 172.21.15.181:0/2129976391 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:05.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): get_auth_request con 0x7f3cf809ec50 auth_method 0 2024-09-15T03:38:05.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:05.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:05.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:05.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:05.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:05.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:05.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cee7fc640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809f020 0x7f3cf8130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:05.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cee7fc640 10 monclient(hunting): get_auth_request con 0x7f3cf809f020 auth_method 0 2024-09-15T03:38:05.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:05.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:05.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:05.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient(hunting): handle_auth_done global_id 104737 payload 824 2024-09-15T03:38:05.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 -- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809f020 msgr2=0x7f3cf8130ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809f020 0x7f3cf8130ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 -- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cf81321a0 msgr2=0x7f3cf81310e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cf81321a0 0x7f3cf81310e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:38:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 monclient: found mon.noname-b 2024-09-15T03:38:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cf6ffd640 1 -- 172.21.15.181:0/2129976391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce0009390 con 0x7f3cf809ec50 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.480+0000 7f3cee7fc640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809f020 0x7f3cf8130ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.482+0000 7f3cf6ffd640 1 -- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 msgr2=0x7f3cf809ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.482+0000 7f3cf6ffd640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 0x7f3cf809ab10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800c8b0 tx=0x7f3ce8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.482+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce800ce40 con 0x7f3cf809ec50 2024-09-15T03:38:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: dump: 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:05.503 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:05.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce8007a40 con 0x7f3cf809ec50 2024-09-15T03:38:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 5 monclient: authenticate success, global_id 104737 2024-09-15T03:38:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 10 monclient: _renew_subs 2024-09-15T03:38:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf811dcc0 con 0x7f3cf809ec50 2024-09-15T03:38:05.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:05.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8007de0 con 0x7f3cf809ec50 2024-09-15T03:38:05.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:05.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:05.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cef7fe640 10 monclient: dump: 2024-09-15T03:38:05.505 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:05.506 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:05.506 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.506 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:05.506 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:05.506 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:05.506 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:05.507 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:05.507 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:05.507 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:05.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 10 monclient: _renew_subs 2024-09-15T03:38:05.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cf5ffb640 4 set_mon_vals no callback set 2024-09-15T03:38:05.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.484+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf811bc30 con 0x7f3cf809ec50 2024-09-15T03:38:05.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.485+0000 7f3cfd59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:05.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.485+0000 7f3cfd59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:05.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.485+0000 7f3cfd59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.485+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3ce8011600 con 0x7f3cf809ec50 2024-09-15T03:38:05.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.485+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf811c0f0 con 0x7f3cf809ec50 2024-09-15T03:38:05.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.486+0000 7f3cef7fe640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3cd8038740 0x7f3cd803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:05.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.486+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(250..250 src has 199..250) v4 ==== 8119+0+0 (secure 0 0 0) 0x7f3ce804c970 con 0x7f3cf809ec50 2024-09-15T03:38:05.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.486+0000 7f3cee7fc640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3cd8038740 0x7f3cd803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:05.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.486+0000 7f3cee7fc640 10 monclient: get_auth_request con 0x7f3cd8038740 auth_method 0 2024-09-15T03:38:05.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.488+0000 7f3cee7fc640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3cd8038740 0x7f3cd803abf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0034000 tx=0x7f3ce0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:05.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.490+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3ce8019570 con 0x7f3cf809ec50 2024-09-15T03:38:05.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.490+0000 7f3cef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:05.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.490+0000 7f3cef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:05.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.647+0000 7f3cfd59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T03:38:05.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.647+0000 7f3cfd59d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T03:38:05.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.647+0000 7f3cfd59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:05.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:05.647+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f3cf811e160 con 0x7f3cf809ec50 2024-09-15T03:38:06.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.231+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v251) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3ce8021300 con 0x7f3cf809ec50 2024-09-15T03:38:06.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.231+0000 7f3cef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T03:38:06.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.231+0000 7f3cef7fe640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cfd59d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cfd59d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cfd59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f3cf80008d0 con 0x7f3cf809ec50 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cef7fe640 1 -- 172.21.15.181:0/2129976391 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v251) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3ce80214e0 con 0x7f3cf809ec50 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-15T03:38:06.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.331+0000 7f3cef7fe640 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-15T03:38:06.333 INFO:tasks.workunit.client.0.smithi181.stderr:device 'r1' does not appear in the crush map 2024-09-15T03:38:06.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3cd8038740 msgr2=0x7f3cd803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:06.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3cd8038740 0x7f3cd803abf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0034000 tx=0x7f3ce0032000 comp rx=0 tx=0).stop 2024-09-15T03:38:06.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 10 monclient: shutdown 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 msgr2=0x7f3cf809ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 0x7f3cf809ab10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800c8b0 tx=0x7f3ce8008040 comp rx=0 tx=0).stop 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 shutdown_connections 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3cd8038740 0x7f3cd803abf0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cf81321a0 0x7f3cf81310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cf809f020 0x7f3cf8130ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 --2- 172.21.15.181:0/2129976391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3cf809ec50 0x7f3cf809ab10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 >> 172.21.15.181:0/2129976391 conn(0x7f3cf80a5cf0 msgr2=0x7f3cf8134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 shutdown_connections 2024-09-15T03:38:06.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.333+0000 7f3cfd59d640 1 -- 172.21.15.181:0/2129976391 wait complete. 2024-09-15T03:38:06.343 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd crush weight-set rm-compat 2024-09-15T03:38:06.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.415+0000 7f0efcaef640 10 monclient: get_monmap_and_config 2024-09-15T03:38:06.416 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.415+0000 7f0efcaef640 10 monclient: build_initial_monmap 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.415+0000 7f0efcaef640 10 monclient: monmap: 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:06.416310+0000 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:06.416310+0000 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:06.417 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:06.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 Processor -- start 2024-09-15T03:38:06.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 -- start start 2024-09-15T03:38:06.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient: init 2024-09-15T03:38:06.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:06.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef80677e0 0x7f0ef8095800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef80677e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0ef809c090 0x7f0ef809c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef809c090 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ef809c9a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef809d570 con 0x7f0ef809c9a0 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef809b560 con 0x7f0ef809c090 2024-09-15T03:38:06.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef809d800 con 0x7f0ef80677e0 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0efcaef640 10 monclient(hunting): authenticate will time out at 1290.897761s 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0ef809c090 0x7f0ef809c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef6d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f0ef809c9a0 0x7f0ef8094080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57566/0 (socket says 172.21.15.181:57566) 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0ef809c090 0x7f0ef809c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49520/0 (socket says 172.21.15.181:49520) 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef6d9d640 1 -- 172.21.15.181:0/2691771338 learned_addr learned my addr 172.21.15.181:0/2691771338 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 10 monclient(hunting): get_auth_request con 0x7f0ef809c090 auth_method 0 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.416+0000 7f0ef5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 10 monclient(hunting): handle_auth_done global_id 104740 payload 274 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 1 -- 172.21.15.181:0/2691771338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef80677e0 msgr2=0x7f0ef8095800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef659c640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef80677e0 0x7f0ef8095800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef80677e0 0x7f0ef8095800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 1 -- 172.21.15.181:0/2691771338 >> v1:172.21.15.181:6789/0 conn(0x7f0ef809c9a0 legacy=0x7f0ef8094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 1 monclient: found mon.noname-c 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef5d9b640 1 -- 172.21.15.181:0/2691771338 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef809cdc0 con 0x7f0ef809c090 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef659c640 10 monclient: get_auth_request con 0x7f0ef80677e0 auth_method 0 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.417+0000 7f0ef659c640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef80677e0 0x7f0ef8095800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.419+0000 7f0ef5d9b640 1 -- 172.21.15.181:0/2691771338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c090 msgr2=0x7f0ef809c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.419+0000 7f0ef5d9b640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c090 0x7f0ef809c460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009880 tx=0x7f0ee80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f28928724b5c139 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 1 -- 172.21.15.181:0/2691771338 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8032440 con 0x7f0ef809c090 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: dump: 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: _finish_auth 0 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 1 -- 172.21.15.181:0/2691771338 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee80325a0 con 0x7f0ef809c090 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 5 monclient: authenticate success, global_id 104740 2024-09-15T03:38:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 1 -- 172.21.15.181:0/2691771338 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8032940 con 0x7f0ef809c090 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 4 set_mon_vals no callback set 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0ef559a640 10 monclient: dump: 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 10 monclient: shutdown 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 -- 172.21.15.181:0/2691771338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c090 msgr2=0x7f0ef809c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c090 0x7f0ef809c460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009880 tx=0x7f0ee80090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 -- 172.21.15.181:0/2691771338 shutdown_connections 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c090 0x7f0ef809c460 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 --2- 172.21.15.181:0/2691771338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef80677e0 0x7f0ef8095800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 -- 172.21.15.181:0/2691771338 >> 172.21.15.181:0/2691771338 conn(0x7f0ef80a16a0 msgr2=0x7f0ef80a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:06.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 -- 172.21.15.181:0/2691771338 shutdown_connections 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.420+0000 7f0efcaef640 1 -- 172.21.15.181:0/2691771338 wait complete. 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient: build_initial_monmap 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient: monmap: 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:06.429 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 Processor -- start 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 -- start start 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient: init 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 0x7f0ef806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef80677e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809c090 0x7f0ef812fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef809c090 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:06.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809c9a0 0x7f0ef8126770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef809c9a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8127c40 con 0x7f0ef809c9a0 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8095090 con 0x7f0ef80677e0 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8094220 con 0x7f0ef809c090 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0efcaef640 10 monclient(hunting): authenticate will time out at 1290.902908s 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0ef659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 0x7f0ef806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.421+0000 7f0ef659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 0x7f0ef806ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49526/0 (socket says 172.21.15.181:49526) 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 -- 172.21.15.181:0/1746306098 learned_addr learned my addr 172.21.15.181:0/1746306098 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): get_auth_request con 0x7f0ef80677e0 auth_method 0 2024-09-15T03:38:06.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef6d9d640 1 -- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809c9a0 msgr2=0x7f0ef8126770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef6d9d640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809c9a0 0x7f0ef8126770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef5d9b640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809c090 0x7f0ef812fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef5d9b640 10 monclient(hunting): get_auth_request con 0x7f0ef809c090 auth_method 0 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient(hunting): handle_auth_done global_id 104743 payload 824 2024-09-15T03:38:06.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 -- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809c090 msgr2=0x7f0ef812fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809c090 0x7f0ef812fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 -- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809c9a0 msgr2=0x7f0ef8126770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809c9a0 0x7f0ef8126770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient: _finish_hunting 0 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 monclient: found mon.noname-b 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef659c640 1 -- 172.21.15.181:0/1746306098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8009390 con 0x7f0ef80677e0 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.422+0000 7f0ef5d9b640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809c090 0x7f0ef812fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.424+0000 7f0ef659c640 1 -- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 msgr2=0x7f0ef806ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.424+0000 7f0ef659c640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 0x7f0ef806ea90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000c880 tx=0x7f0ee0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.424+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:38:06.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee000ce10 con 0x7f0ef80677e0 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: dump: 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:06.438 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee0007aa0 con 0x7f0ef80677e0 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 5 monclient: authenticate success, global_id 104743 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 10 monclient: _renew_subs 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef8128080 con 0x7f0ef80677e0 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0007e60 con 0x7f0ef80677e0 2024-09-15T03:38:06.439 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0eee7fc640 10 monclient: dump: 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 10 monclient: _renew_subs 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.440 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef806f530 con 0x7f0ef80677e0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.426+0000 7f0ef4d99640 4 set_mon_vals no callback set 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0ee0011600 con 0x7f0ef80677e0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0efcaef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0efcaef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0efcaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef80008d0 con 0x7f0ef80677e0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0eee7fc640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ed8038740 0x7f0ed803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(251..251 src has 199..251) v4 ==== 7909+0+0 (secure 0 0 0) 0x7f0ee004c7e0 con 0x7f0ef80677e0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0ef5d9b640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ed8038740 0x7f0ed803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.427+0000 7f0ef5d9b640 10 monclient: get_auth_request con 0x7f0ed8038740 auth_method 0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.430+0000 7f0ef5d9b640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ed8038740 0x7f0ed803abf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009850 tx=0x7f0ee8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.432+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0ee0017ae0 con 0x7f0ef80677e0 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.432+0000 7f0eee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:06.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.432+0000 7f0eee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:06.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.594+0000 7f0efcaef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:38:06.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.594+0000 7f0efcaef640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:38:06.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.594+0000 7f0efcaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:06.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:06.594+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f0ef8095d40 con 0x7f0ef80677e0 2024-09-15T03:38:07.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.297+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v252) v1 ==== 78+0+0 (secure 0 0 0) 0x7f0ee0017300 con 0x7f0ef80677e0 2024-09-15T03:38:07.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.297+0000 7f0eee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:38:07.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.297+0000 7f0eee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:07.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.398+0000 7f0efcaef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:38:07.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.398+0000 7f0efcaef640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:38:07.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.398+0000 7f0efcaef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:07.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:07.398+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f0ef8098300 con 0x7f0ef80677e0 2024-09-15T03:38:08.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.302+0000 7f0eee7fc640 1 -- 172.21.15.181:0/1746306098 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v253) v1 ==== 78+0+0 (secure 0 0 0) 0x7f0ee00174e0 con 0x7f0ef80677e0 2024-09-15T03:38:08.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.302+0000 7f0eee7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-15T03:38:08.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.302+0000 7f0eee7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:08.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ed8038740 msgr2=0x7f0ed803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:08.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ed8038740 0x7f0ed803abf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009850 tx=0x7f0ee8032000 comp rx=0 tx=0).stop 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 10 monclient: shutdown 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 msgr2=0x7f0ef806ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 0x7f0ef806ea90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000c880 tx=0x7f0ee0008040 comp rx=0 tx=0).stop 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 shutdown_connections 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ed8038740 0x7f0ed803abf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809c9a0 0x7f0ef8126770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809c090 0x7f0ef812fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 --2- 172.21.15.181:0/1746306098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef80677e0 0x7f0ef806ea90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 >> 172.21.15.181:0/1746306098 conn(0x7f0ef80a16a0 msgr2=0x7f0ef8132b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 shutdown_connections 2024-09-15T03:38:08.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.305+0000 7f0efcaef640 1 -- 172.21.15.181:0/1746306098 wait complete. 2024-09-15T03:38:08.316 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo OK 2024-09-15T03:38:08.317 INFO:tasks.workunit.client.0.smithi181.stdout:OK 2024-09-15T03:38:08.317 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:38:08.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:38:08.345 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-15T03:38:08.347 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-15T03:38:08.347 DEBUG:teuthology.orchestra.run.smithi181:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-15T03:38:08.405 INFO:tasks.workunit.client.0.smithi181.stderr:+ set -e 2024-09-15T03:38:08.406 INFO:tasks.workunit.client.0.smithi181.stderr:++ uuidgen 2024-09-15T03:38:08.466 INFO:tasks.workunit.client.0.smithi181.stderr:+ ua=ecab9f08-f922-45d9-8393-9f53fccdb802 2024-09-15T03:38:08.467 INFO:tasks.workunit.client.0.smithi181.stderr:++ uuidgen 2024-09-15T03:38:08.467 INFO:tasks.workunit.client.0.smithi181.stderr:+ ub=8639cfee-8b01-45a4-bdca-7d4262ff6c5b 2024-09-15T03:38:08.467 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create ecab9f08-f922-45d9-8393-9f53fccdb802 2024-09-15T03:38:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.539+0000 7fb0cf91d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.539+0000 7fb0cf91d640 10 monclient: build_initial_monmap 2024-09-15T03:38:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.539+0000 7fb0cf91d640 10 monclient: monmap: 2024-09-15T03:38:08.540 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:08.540467+0000 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:08.540467+0000 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:08.541 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 Processor -- start 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 -- start start 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient: init 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8069a30 0x7fb0c81320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): picked mon.noname-b con 0x7fb0c8069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb0c8070920 0x7fb0c8070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0c8070920 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb0c8071230 0x7fb0c812e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0c8071230 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c809e560 con 0x7fb0c8071230 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c809cc60 con 0x7fb0c8070920 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c809d0a0 con 0x7fb0c8069a30 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:08.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.540+0000 7fb0cf91d640 10 monclient(hunting): authenticate will time out at 1293.021944s 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb0c8070920 0x7fb0c8070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cdeba640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fb0c8071230 msgr2=0x7fb0c812e740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb0c8070920 0x7fb0c8070cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36170/0 (socket says 172.21.15.181:36170) 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 -- 172.21.15.181:0/1061583526 learned_addr learned my addr 172.21.15.181:0/1061583526 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): get_auth_request con 0x7fb0c8070920 auth_method 0 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cdeba640 1 --2- 172.21.15.181:0/1061583526 >> v2:172.21.15.153:3300/0 conn(0x7fb0c8071230 0x7fb0c812e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient(hunting): handle_auth_done global_id 104746 payload 274 2024-09-15T03:38:08.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 -- 172.21.15.181:0/1061583526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8069a30 msgr2=0x7fb0c81320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 --2- 172.21.15.181:0/1061583526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8069a30 0x7fb0c81320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 -- 172.21.15.181:0/1061583526 >> v2:172.21.15.153:3300/0 conn(0x7fb0c8071230 msgr2=0x7fb0c812e740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 --2- 172.21.15.181:0/1061583526 >> v2:172.21.15.153:3300/0 conn(0x7fb0c8071230 0x7fb0c812e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient: _finish_hunting 0 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 monclient: found mon.noname-c 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.541+0000 7fb0cceb8640 1 -- 172.21.15.181:0/1061583526 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c809e280 con 0x7fb0c8070920 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cceb8640 1 -- 172.21.15.181:0/1061583526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8070920 msgr2=0x7fb0c8070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cceb8640 1 --2- 172.21.15.181:0/1061583526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8070920 0x7fb0c8070cf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8071960 tx=0x7fb0b80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e9da74b4647602 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 1 -- 172.21.15.181:0/1061583526 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8032440 con 0x7fb0c8070920 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: dump: 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:08.553 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: _finish_auth 0 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 5 monclient: authenticate success, global_id 104746 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 1 -- 172.21.15.181:0/1061583526 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b80325a0 con 0x7fb0c8070920 2024-09-15T03:38:08.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 4 set_mon_vals no callback set 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 1 -- 172.21.15.181:0/1061583526 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8032940 con 0x7fb0c8070920 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0bffff640 10 monclient: dump: 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.555 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 10 monclient: shutdown 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 -- 172.21.15.181:0/1061583526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8070920 msgr2=0x7fb0c8070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/1061583526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8070920 0x7fb0c8070cf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8071960 tx=0x7fb0b80091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 -- 172.21.15.181:0/1061583526 shutdown_connections 2024-09-15T03:38:08.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/1061583526 >> v2:172.21.15.153:3300/0 conn(0x7fb0c8071230 0x7fb0c812e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/1061583526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8070920 0x7fb0c8070cf0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/1061583526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8069a30 0x7fb0c81320d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 -- 172.21.15.181:0/1061583526 >> 172.21.15.181:0/1061583526 conn(0x7fb0c809ea70 msgr2=0x7fb0c809ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 -- 172.21.15.181:0/1061583526 shutdown_connections 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.544+0000 7fb0cf91d640 1 -- 172.21.15.181:0/1061583526 wait complete. 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.545+0000 7fb0cf91d640 10 monclient: build_initial_monmap 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.545+0000 7fb0cf91d640 10 monclient: monmap: 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:08.557 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 Processor -- start 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 -- start start 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient: init 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c8069a30 0x7fb0c81308e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0c8069a30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8070920 0x7fb0c811a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0c8070920 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 0x7fb0c811a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.558 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): picked mon.noname-b con 0x7fb0c8071230 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c8134c80 con 0x7fb0c8069a30 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c8136580 con 0x7fb0c8071230 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c811c450 con 0x7fb0c8070920 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cf91d640 10 monclient(hunting): authenticate will time out at 1293.027823s 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 0x7fb0c811a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cd6b9640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c8069a30 msgr2=0x7fb0c81308e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cd6b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c8069a30 0x7fb0c81308e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 0x7fb0c811a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36176/0 (socket says 172.21.15.181:36176) 2024-09-15T03:38:08.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 1 -- 172.21.15.181:0/3054028460 learned_addr learned my addr 172.21.15.181:0/3054028460 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): get_auth_request con 0x7fb0c8071230 auth_method 0 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.546+0000 7fb0cdeba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cceb8640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8070920 0x7fb0c811a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cceb8640 10 monclient(hunting): get_auth_request con 0x7fb0c8070920 auth_method 0 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cceb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:08.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 10 monclient(hunting): handle_auth_done global_id 104749 payload 824 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 1 -- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8070920 msgr2=0x7fb0c811a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8070920 0x7fb0c811a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 1 -- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c8069a30 msgr2=0x7fb0c81308e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c8069a30 0x7fb0c81308e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 10 monclient: _finish_hunting 0 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 1 monclient: found mon.noname-b 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.547+0000 7fb0cdeba640 1 -- 172.21.15.181:0/3054028460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b8009450 con 0x7fb0c8071230 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cdeba640 1 -- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 msgr2=0x7fb0c811a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:08.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cdeba640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 0x7fb0c811a8f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000bb10 tx=0x7fb0c00097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c000cbc0 con 0x7fb0c8071230 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: dump: 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c000cd40 con 0x7fb0c8071230 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 5 monclient: authenticate success, global_id 104749 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 10 monclient: _renew_subs 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0006150 con 0x7fb0c8071230 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c811cb70 con 0x7fb0c8071230 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bf7fe640 4 set_mon_vals no callback set 2024-09-15T03:38:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0bd7fa640 10 monclient: dump: 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 10 monclient: _renew_subs 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:08.564 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.549+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c811bf40 con 0x7fb0c8071230 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb0c0012020 con 0x7fb0c8071230 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0cf91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0cf91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0cf91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c80636e0 con 0x7fb0c8071230 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0bd7fa640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb0ac0386f0 0x7fb0ac03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(253..253 src has 199..253) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb0c004c560 con 0x7fb0c8071230 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0cd6b9640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb0ac0386f0 0x7fb0ac03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:08.565 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.550+0000 7fb0cd6b9640 10 monclient: get_auth_request con 0x7fb0ac0386f0 auth_method 0 2024-09-15T03:38:08.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.553+0000 7fb0cd6b9640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb0ac0386f0 0x7fb0ac03aba0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40025d0 tx=0x7fb0c4002a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:08.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.555+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb0c00134e0 con 0x7fb0c8071230 2024-09-15T03:38:08.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.555+0000 7fb0bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:08.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.555+0000 7fb0bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:08.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.719+0000 7fb0cf91d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:08.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.719+0000 7fb0cf91d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:08.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.719+0000 7fb0cf91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:08.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:08.719+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"} v 0) v1 -- 0x7fb0c811dc00 con 0x7fb0c8071230 2024-09-15T03:38:09.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.303+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7fb0c0014e60 con 0x7fb0c8071230 2024-09-15T03:38:09.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.303+0000 7fb0bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.303+0000 7fb0bd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:09.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0cf91d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0cf91d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0cf91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"} v 0) v1 -- 0x7fb0c811e020 con 0x7fb0c8071230 2024-09-15T03:38:09.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0bd7fa640 1 -- 172.21.15.181:0/3054028460 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7fb0c001daf0 con 0x7fb0c8071230 2024-09-15T03:38:09.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0bd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.407+0000 7fb0bd7fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb0ac0386f0 msgr2=0x7fb0ac03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb0ac0386f0 0x7fb0ac03aba0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40025d0 tx=0x7fb0c4002a60 comp rx=0 tx=0).stop 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 10 monclient: shutdown 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 msgr2=0x7fb0c811a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 0x7fb0c811a8f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000bb10 tx=0x7fb0c00097d0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 shutdown_connections 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb0ac0386f0 0x7fb0ac03aba0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c8071230 0x7fb0c811a8f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c8070920 0x7fb0c811a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.409+0000 7fb0cf91d640 1 --2- 172.21.15.181:0/3054028460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c8069a30 0x7fb0c81308e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.410+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 >> 172.21.15.181:0/3054028460 conn(0x7fb0c809ea70 msgr2=0x7fb0c8134e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:09.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.410+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 shutdown_connections 2024-09-15T03:38:09.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.410+0000 7fb0cf91d640 1 -- 172.21.15.181:0/3054028460 wait complete. 2024-09-15T03:38:09.419 INFO:tasks.workunit.client.0.smithi181.stderr:+ na=12 2024-09-15T03:38:09.419 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create ecab9f08-f922-45d9-8393-9f53fccdb802 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.491+0000 7f8a516e8640 10 monclient: get_monmap_and_config 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.491+0000 7f8a516e8640 10 monclient: build_initial_monmap 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.491+0000 7f8a516e8640 10 monclient: monmap: 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:09.492166+0000 2024-09-15T03:38:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:09.492166+0000 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:09.493 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.491+0000 7f8a516e8640 1 Processor -- start 2024-09-15T03:38:09.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 1 -- start start 2024-09-15T03:38:09.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient: init 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8a4c131b40 0x7f8a4c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a4c131b40 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a4c09f240 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c136e70 0x7f8a4c139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4c136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c06a230 con 0x7f8a4c09f240 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c0a0300 con 0x7f8a4c131b40 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0a00c0 con 0x7f8a4c136e70 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a516e8640 10 monclient(hunting): authenticate will time out at 1293.973477s 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8a4c131b40 0x7f8a4c09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8a4c131b40 0x7f8a4c09ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36184/0 (socket says 172.21.15.181:36184) 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4a7fc640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f8a4c09f240 legacy=0x7f8a4c131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 -- 172.21.15.181:0/1801741783 learned_addr learned my addr 172.21.15.181:0/1801741783 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): get_auth_request con 0x7f8a4c131b40 auth_method 0 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:09.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4b7fe640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c136e70 0x7f8a4c139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient(hunting): handle_auth_done global_id 104752 payload 274 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 -- 172.21.15.181:0/1801741783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c136e70 msgr2=0x7f8a4c139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c136e70 0x7f8a4c139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 -- 172.21.15.181:0/1801741783 >> v1:172.21.15.153:6789/0 conn(0x7f8a4c09f240 legacy=0x7f8a4c131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient: _finish_hunting 0 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 monclient: found mon.noname-c 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4affd640 1 -- 172.21.15.181:0/1801741783 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c09bee0 con 0x7f8a4c131b40 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4b7fe640 10 monclient: get_auth_request con 0x7f8a4c136e70 auth_method 0 2024-09-15T03:38:09.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.492+0000 7f8a4b7fe640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c136e70 0x7f8a4c139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:09.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a4affd640 1 -- 172.21.15.181:0/1801741783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c131b40 msgr2=0x7f8a4c09ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:09.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a4affd640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c131b40 0x7f8a4c09ed00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009910 tx=0x7f8a3c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6412b7028143a39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:09.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 1 -- 172.21.15.181:0/1801741783 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c032440 con 0x7f8a4c131b40 2024-09-15T03:38:09.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.504 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: dump: 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: _finish_auth 0 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:09.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 1 -- 172.21.15.181:0/1801741783 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c0325a0 con 0x7f8a4c131b40 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a516e8640 5 monclient: authenticate success, global_id 104752 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a516e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a516e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 1 -- 172.21.15.181:0/1801741783 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c032940 con 0x7f8a4c131b40 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a516e8640 4 set_mon_vals no callback set 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.495+0000 7f8a49ffb640 10 monclient: dump: 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.506 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 10 monclient: shutdown 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 -- 172.21.15.181:0/1801741783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c131b40 msgr2=0x7f8a4c09ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c131b40 0x7f8a4c09ed00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009910 tx=0x7f8a3c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 -- 172.21.15.181:0/1801741783 shutdown_connections 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c136e70 0x7f8a4c139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 --2- 172.21.15.181:0/1801741783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c131b40 0x7f8a4c09ed00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 -- 172.21.15.181:0/1801741783 >> 172.21.15.181:0/1801741783 conn(0x7f8a4c0a5100 msgr2=0x7f8a4c0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 -- 172.21.15.181:0/1801741783 shutdown_connections 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.496+0000 7f8a516e8640 1 -- 172.21.15.181:0/1801741783 wait complete. 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient: build_initial_monmap 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient: monmap: 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:09.508 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 Processor -- start 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 -- start start 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient: init 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a4c09f240 0x7f8a4c1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4c09f240 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c131b40 0x7f8a4c0988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a4c131b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 0x7f8a4c098df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4c136e70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c131040 con 0x7f8a4c09f240 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c11bc00 con 0x7f8a4c136e70 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0a0a30 con 0x7f8a4c131b40 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a516e8640 10 monclient(hunting): authenticate will time out at 1293.978848s 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4affd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a4c09f240 msgr2=0x7f8a4c1419d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a4c09f240 0x7f8a4c1419d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:09.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 0x7f8a4c098df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 0x7f8a4c098df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36190/0 (socket says 172.21.15.181:36190) 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 1 -- 172.21.15.181:0/2562917803 learned_addr learned my addr 172.21.15.181:0/2562917803 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 10 monclient(hunting): get_auth_request con 0x7f8a4c136e70 auth_method 0 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.497+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c131b40 0x7f8a4c0988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): get_auth_request con 0x7f8a4c131b40 auth_method 0 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:09.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_done global_id 104755 payload 824 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 1 -- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c131b40 msgr2=0x7f8a4c0988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c131b40 0x7f8a4c0988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 1 -- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a4c09f240 msgr2=0x7f8a4c1419d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a4c09f240 0x7f8a4c1419d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:09.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.498+0000 7f8a4b7fe640 1 -- 172.21.15.181:0/2562917803 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c009450 con 0x7f8a4c136e70 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a4b7fe640 1 -- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 msgr2=0x7f8a4c098df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a4b7fe640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 0x7f8a4c098df0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400a380 tx=0x7f8a440078f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400d860 con 0x7f8a4c136e70 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: dump: 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: _finish_auth 0 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 5 monclient: authenticate success, global_id 104755 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4400d9e0 con 0x7f8a4c136e70 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 10 monclient: _renew_subs 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c11c130 con 0x7f8a4c136e70 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:09.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400dda0 con 0x7f8a4c136e70 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a32ffd640 10 monclient: dump: 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.515 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 10 monclient: _renew_subs 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c09a3f0 con 0x7f8a4c136e70 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.500+0000 7f8a497fa640 4 set_mon_vals no callback set 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8a4401e3b0 con 0x7f8a4c136e70 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a516e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a516e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a516e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c0636e0 con 0x7f8a4c136e70 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a32ffd640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a380386f0 0x7f8a3803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(254..254 src has 199..254) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f8a4404cfa0 con 0x7f8a4c136e70 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a4affd640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a380386f0 0x7f8a3803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.501+0000 7f8a4affd640 10 monclient: get_auth_request con 0x7f8a380386f0 auth_method 0 2024-09-15T03:38:09.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.504+0000 7f8a4affd640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a380386f0 0x7f8a3803aba0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c09f660 tx=0x7f8a3c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:09.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.506+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8a44012a80 con 0x7f8a4c136e70 2024-09-15T03:38:09.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.506+0000 7f8a32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:09.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.506+0000 7f8a32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a516e8640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a516e8640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a516e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"} v 0) v1 -- 0x7f8a4c064a70 con 0x7f8a4c136e70 2024-09-15T03:38:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7f8a4401a020 con 0x7f8a4c136e70 2024-09-15T03:38:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.671+0000 7f8a32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:09.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.772+0000 7f8a516e8640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.772+0000 7f8a516e8640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.772+0000 7f8a516e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.772+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"} v 0) v1 -- 0x7f8a4c09afc0 con 0x7f8a4c136e70 2024-09-15T03:38:09.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.773+0000 7f8a32ffd640 1 -- 172.21.15.181:0/2562917803 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}]=0 v254) v1 ==== 106+0+3 (secure 0 0 0) 0x7f8a440122a0 con 0x7f8a4c136e70 2024-09-15T03:38:09.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.773+0000 7f8a32ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:09.774 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.773+0000 7f8a32ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:09.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a380386f0 msgr2=0x7f8a3803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a380386f0 0x7f8a3803aba0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c09f660 tx=0x7f8a3c032000 comp rx=0 tx=0).stop 2024-09-15T03:38:09.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 10 monclient: shutdown 2024-09-15T03:38:09.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:09.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 msgr2=0x7f8a4c098df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 0x7f8a4c098df0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400a380 tx=0x7f8a440078f0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 shutdown_connections 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a380386f0 0x7f8a3803aba0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a4c136e70 0x7f8a4c098df0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a4c131b40 0x7f8a4c0988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 --2- 172.21.15.181:0/2562917803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a4c09f240 0x7f8a4c1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 >> 172.21.15.181:0/2562917803 conn(0x7f8a4c0a5100 msgr2=0x7f8a4c0a3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 shutdown_connections 2024-09-15T03:38:09.777 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.775+0000 7f8a516e8640 1 -- 172.21.15.181:0/2562917803 wait complete. 2024-09-15T03:38:09.785 INFO:tasks.workunit.client.0.smithi181.stderr:+ test 12 -eq 12 2024-09-15T03:38:09.785 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 8639cfee-8b01-45a4-bdca-7d4262ff6c5b 2024-09-15T03:38:09.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.855+0000 7fb16bf9d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:09.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.855+0000 7fb16bf9d640 10 monclient: build_initial_monmap 2024-09-15T03:38:09.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.855+0000 7fb16bf9d640 10 monclient: monmap: 2024-09-15T03:38:09.856 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:09.856470+0000 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:09.856470+0000 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:09.857 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 Processor -- start 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 -- start start 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient: init 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb16409cb20 0x7fb16409afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb16409cb20 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb16412e940 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb16412f490 0x7fb164135c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb16412f490 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16409d6f0 con 0x7fb16412e940 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb164097a40 con 0x7fb16409cb20 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb164097e90 con 0x7fb16412f490 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.856+0000 7fb16bf9d640 10 monclient(hunting): authenticate will time out at 1294.337907s 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb169d39640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fb16409cb20 msgr2=0x7fb16409afb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb16412f490 0x7fb164135c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb169d39640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb16409cb20 0x7fb16409afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb16412f490 0x7fb164135c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36202/0 (socket says 172.21.15.181:36202) 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 -- 172.21.15.181:0/3950143657 learned_addr learned my addr 172.21.15.181:0/3950143657 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): get_auth_request con 0x7fb16412f490 auth_method 0 2024-09-15T03:38:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb169538640 1 -- 172.21.15.181:0/3950143657 >> v1:172.21.15.153:6789/0 conn(0x7fb16412e940 legacy=0x7fb16412ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb168d37640 1 -- 172.21.15.181:0/3950143657 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient(hunting): handle_auth_done global_id 104758 payload 274 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 -- 172.21.15.181:0/3950143657 >> v2:172.21.15.153:3300/0 conn(0x7fb16409cb20 msgr2=0x7fb16409afb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 --2- 172.21.15.181:0/3950143657 >> v2:172.21.15.153:3300/0 conn(0x7fb16409cb20 0x7fb16409afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 -- 172.21.15.181:0/3950143657 >> v1:172.21.15.153:6789/0 conn(0x7fb16412e940 legacy=0x7fb16412ed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient: _finish_hunting 0 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 monclient: found mon.noname-c 2024-09-15T03:38:09.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:09.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.857+0000 7fb16a53a640 1 -- 172.21.15.181:0/3950143657 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164098da0 con 0x7fb16412f490 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16a53a640 1 -- 172.21.15.181:0/3950143657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 msgr2=0x7fb164135c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16a53a640 1 --2- 172.21.15.181:0/3950143657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164135c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb16409b4f0 tx=0x7fb15c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f78f981b8a5bda85 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 1 -- 172.21.15.181:0/3950143657 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c02aea0 con 0x7fb16412f490 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: dump: 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.868 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: _finish_auth 0 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 5 monclient: authenticate success, global_id 104758 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:09.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 1 -- 172.21.15.181:0/3950143657 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c02d050 con 0x7fb16412f490 2024-09-15T03:38:09.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:09.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:09.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 1 -- 172.21.15.181:0/3950143657 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c007840 con 0x7fb16412f490 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 4 set_mon_vals no callback set 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb168d37640 10 monclient: dump: 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.871 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 10 monclient: shutdown 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 -- 172.21.15.181:0/3950143657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 msgr2=0x7fb164135c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/3950143657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164135c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb16409b4f0 tx=0x7fb15c009840 comp rx=0 tx=0).stop 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 -- 172.21.15.181:0/3950143657 shutdown_connections 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/3950143657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164135c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/3950143657 >> v2:172.21.15.153:3300/0 conn(0x7fb16409cb20 0x7fb16409afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 -- 172.21.15.181:0/3950143657 >> 172.21.15.181:0/3950143657 conn(0x7fb1640714d0 msgr2=0x7fb1640a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 -- 172.21.15.181:0/3950143657 shutdown_connections 2024-09-15T03:38:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.860+0000 7fb16bf9d640 1 -- 172.21.15.181:0/3950143657 wait complete. 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient: build_initial_monmap 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient: monmap: 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 1 Processor -- start 2024-09-15T03:38:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 1 -- start start 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient: init 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb16409cb20 0x7fb164141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb16409cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16412e940 0x7fb1641418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.861+0000 7fb16bf9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb16412e940 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb16412f490 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1641346e0 con 0x7fb16412e940 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164134ca0 con 0x7fb16412f490 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb164098950 con 0x7fb16409cb20 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16bf9d640 10 monclient(hunting): authenticate will time out at 1294.343028s 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36218/0 (socket says 172.21.15.181:36218) 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 -- 172.21.15.181:0/1251849537 learned_addr learned my addr 172.21.15.181:0/1251849537 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): get_auth_request con 0x7fb16412f490 auth_method 0 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:09.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb169538640 1 -- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16412e940 msgr2=0x7fb1641418b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb169538640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16412e940 0x7fb1641418b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb169d39640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb16409cb20 0x7fb164141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient(hunting): handle_auth_done global_id 104761 payload 824 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 -- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb16409cb20 msgr2=0x7fb164141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb16409cb20 0x7fb164141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 -- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16412e940 msgr2=0x7fb1641418b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16412e940 0x7fb1641418b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient: _finish_hunting 0 2024-09-15T03:38:09.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 monclient: found mon.noname-b 2024-09-15T03:38:09.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:09.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb16a53a640 1 -- 172.21.15.181:0/1251849537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c009ae0 con 0x7fb16412f490 2024-09-15T03:38:09.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb169d39640 10 monclient: get_auth_request con 0x7fb16409cb20 auth_method 0 2024-09-15T03:38:09.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.862+0000 7fb169d39640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb16409cb20 0x7fb164141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:09.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb16a53a640 1 -- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 msgr2=0x7fb164141df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb16a53a640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009ab0 tx=0x7fb15c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c031e80 con 0x7fb16412f490 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: dump: 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: _finish_auth 0 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c03c420 con 0x7fb16412f490 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb16bf9d640 5 monclient: authenticate success, global_id 104761 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb16bf9d640 10 monclient: _renew_subs 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c03c7c0 con 0x7fb16412f490 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb16bf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1641337f0 con 0x7fb16412f490 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:09.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb159ffb640 10 monclient: dump: 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:09.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.864+0000 7fb15bfff640 4 set_mon_vals no callback set 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 10 monclient: _renew_subs 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16406b300 con 0x7fb16412f490 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb15c03cbf0 con 0x7fb16412f490 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb159ffb640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1480386f0 0x7fb14803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.865+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1640636e0 con 0x7fb16412f490 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.866+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(254..254 src has 199..254) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb15c06d340 con 0x7fb16412f490 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.866+0000 7fb169d39640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1480386f0 0x7fb14803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:09.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.866+0000 7fb169d39640 10 monclient: get_auth_request con 0x7fb1480386f0 auth_method 0 2024-09-15T03:38:09.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.868+0000 7fb169d39640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1480386f0 0x7fb14803aba0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb1600038f0 tx=0x7fb160008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:09.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.870+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb15c03b380 con 0x7fb16412f490 2024-09-15T03:38:09.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.870+0000 7fb159ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:09.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:09.870+0000 7fb159ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:10.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:10.032+0000 7fb16bf9d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:10.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:10.032+0000 7fb16bf9d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:10.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:10.032+0000 7fb16bf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:10.034 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:10.032+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"} v 0) v1 -- 0x7fb1640a0800 con 0x7fb16412f490 2024-09-15T03:38:10.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:10.313+0000 7fb16a53a640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009ab0 tx=0x7fb15c036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb15c0400b0 mon_command_ack([{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}]=0 v255) v1 2024-09-15T03:38:11.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.141+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}]=0 v255) v1 ==== 106+0+3 (secure 0 0 0) 0x7fb15c0400b0 con 0x7fb16412f490 2024-09-15T03:38:11.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.141+0000 7fb159ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.141+0000 7fb159ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:11.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb16bf9d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.246 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb16bf9d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb16bf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"} v 0) v1 -- 0x7fb164134e10 con 0x7fb16412f490 2024-09-15T03:38:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb159ffb640 1 -- 172.21.15.181:0/1251849537 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}]=0 v255) v1 ==== 106+0+3 (secure 0 0 0) 0x7fb15c03a780 con 0x7fb16412f490 2024-09-15T03:38:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb159ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.245+0000 7fb159ffb640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1480386f0 msgr2=0x7fb14803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1480386f0 0x7fb14803aba0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb1600038f0 tx=0x7fb160008040 comp rx=0 tx=0).stop 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 10 monclient: shutdown 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 msgr2=0x7fb164141df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009ab0 tx=0x7fb15c036000 comp rx=0 tx=0).stop 2024-09-15T03:38:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 shutdown_connections 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1480386f0 0x7fb14803aba0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16412f490 0x7fb164141df0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16412e940 0x7fb1641418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 --2- 172.21.15.181:0/1251849537 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb16409cb20 0x7fb164141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 >> 172.21.15.181:0/1251849537 conn(0x7fb1640714d0 msgr2=0x7fb1640a0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 shutdown_connections 2024-09-15T03:38:11.250 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.248+0000 7fb16bf9d640 1 -- 172.21.15.181:0/1251849537 wait complete. 2024-09-15T03:38:11.257 INFO:tasks.workunit.client.0.smithi181.stderr:+ nb=14 2024-09-15T03:38:11.258 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create 8639cfee-8b01-45a4-bdca-7d4262ff6c5b 2024-09-15T03:38:11.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.330+0000 7f453f55d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.330+0000 7f453f55d640 10 monclient: build_initial_monmap 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.330+0000 7f453f55d640 10 monclient: monmap: 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:11.331294+0000 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:11.331294+0000 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.332 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 Processor -- start 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 -- start start 2024-09-15T03:38:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient: init 2024-09-15T03:38:11.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:11.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:38:11.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f453809f140 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:11.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f453809cf20 0x7f4538136a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): picked mon.noname-b con 0x7f453809cf20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f45380a0430 0x7f4538138f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): picked mon.noname-c con 0x7f45380a0430 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45380a01f0 con 0x7f453809f140 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45380a1730 con 0x7f45380a0430 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45380a1b10 con 0x7f453809cf20 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453f55d640 10 monclient(hunting): authenticate will time out at 1295.812764s 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453d2f9640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f453809f140 0x7f453809ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57952/0 (socket says 172.21.15.181:57952) 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453d2f9640 1 -- 172.21.15.181:0/89344109 learned_addr learned my addr 172.21.15.181:0/89344109 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 1 --2- 172.21.15.181:0/89344109 >> v2:172.21.15.181:3300/0 conn(0x7f45380a0430 0x7f4538138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): get_auth_request con 0x7f45380a0430 auth_method 0 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.331+0000 7f453dafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 10 monclient(hunting): handle_auth_done global_id 104764 payload 274 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 1 -- 172.21.15.181:0/89344109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f453809cf20 msgr2=0x7f4538136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 1 --2- 172.21.15.181:0/89344109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f453809cf20 0x7f4538136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 1 -- 172.21.15.181:0/89344109 >> v1:172.21.15.181:6789/0 conn(0x7f453809f140 legacy=0x7f453809ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:11.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 10 monclient: _finish_hunting 0 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 1 monclient: found mon.noname-c 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.332+0000 7f453dafa640 1 -- 172.21.15.181:0/89344109 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453809f510 con 0x7f45380a0430 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.334+0000 7f453dafa640 1 -- 172.21.15.181:0/89344109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45380a0430 msgr2=0x7f4538138f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.334+0000 7f453dafa640 1 --2- 172.21.15.181:0/89344109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45380a0430 0x7f4538138f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4530009e60 tx=0x7f4530009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51f82bbf3020bce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 1 -- 172.21.15.181:0/89344109 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530007840 con 0x7f45380a0430 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: dump: 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: _finish_auth 0 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 1 -- 172.21.15.181:0/89344109 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453002d050 con 0x7f45380a0430 2024-09-15T03:38:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 5 monclient: authenticate success, global_id 104764 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 1 -- 172.21.15.181:0/89344109 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530007bd0 con 0x7f45380a0430 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 4 set_mon_vals no callback set 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f452ffff640 10 monclient: dump: 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.347 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 10 monclient: shutdown 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 -- 172.21.15.181:0/89344109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45380a0430 msgr2=0x7f4538138f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 --2- 172.21.15.181:0/89344109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45380a0430 0x7f4538138f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4530009e60 tx=0x7f4530009780 comp rx=0 tx=0).stop 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 -- 172.21.15.181:0/89344109 shutdown_connections 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 --2- 172.21.15.181:0/89344109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45380a0430 0x7f4538138f70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 --2- 172.21.15.181:0/89344109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f453809cf20 0x7f4538136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 -- 172.21.15.181:0/89344109 >> 172.21.15.181:0/89344109 conn(0x7f45380a5230 msgr2=0x7f45380a5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 -- 172.21.15.181:0/89344109 shutdown_connections 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.335+0000 7f453f55d640 1 -- 172.21.15.181:0/89344109 wait complete. 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 10 monclient: build_initial_monmap 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 10 monclient: monmap: 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:11.349 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 1 Processor -- start 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 1 -- start start 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 10 monclient: init 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:38:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.336+0000 7f453f55d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 0x7f4538135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): picked mon.noname-b con 0x7f453809cf20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f453809f140 0x7f453811b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): picked mon.noname-a con 0x7f453809f140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45380a0430 0x7f453811bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): picked mon.noname-c con 0x7f45380a0430 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45380a39c0 con 0x7f453809f140 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453811f010 con 0x7f453809cf20 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453811e910 con 0x7f45380a0430 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453f55d640 10 monclient(hunting): authenticate will time out at 1295.818050s 2024-09-15T03:38:11.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 0x7f4538135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453caf8640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f453809f140 msgr2=0x7f453811b890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453caf8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f453809f140 0x7f453811b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 0x7f4538135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36230/0 (socket says 172.21.15.181:36230) 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 -- 172.21.15.181:0/3626452763 learned_addr learned my addr 172.21.15.181:0/3626452763 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): get_auth_request con 0x7f453809cf20 auth_method 0 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:11.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453dafa640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45380a0430 0x7f453811bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453dafa640 10 monclient(hunting): get_auth_request con 0x7f45380a0430 auth_method 0 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453dafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453dafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453dafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient(hunting): handle_auth_done global_id 104767 payload 824 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 -- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45380a0430 msgr2=0x7f453811bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45380a0430 0x7f453811bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 -- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f453809f140 msgr2=0x7f453811b890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f453809f140 0x7f453811b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient: _finish_hunting 0 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 monclient: found mon.noname-b 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:11.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453d2f9640 1 -- 172.21.15.181:0/3626452763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530009a20 con 0x7f453809cf20 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.337+0000 7f453dafa640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45380a0430 0x7f453811bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453d2f9640 1 -- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 msgr2=0x7f4538135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453d2f9640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 0x7f4538135450 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f453400c5e0 tx=0x7f4534004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534005cb0 con 0x7f453809cf20 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: dump: 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.354 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:11.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 5 monclient: authenticate success, global_id 104767 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4534005e30 con 0x7f453809cf20 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534007340 con 0x7f453809cf20 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 10 monclient: _renew_subs 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453811e630 con 0x7f453809cf20 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452f7fe640 4 set_mon_vals no callback set 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f452d7fa640 10 monclient: dump: 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.356 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 10 monclient: _renew_subs 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.339+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453811e1f0 con 0x7f453809cf20 2024-09-15T03:38:11.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.340+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f453401b400 con 0x7f453809cf20 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.340+0000 7f453f55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.340+0000 7f453f55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.340+0000 7f453f55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.340+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453811e440 con 0x7f453809cf20 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.341+0000 7f452d7fa640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f451c0386f0 0x7f451c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.341+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(255..255 src has 199..255) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f453404c180 con 0x7f453809cf20 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.341+0000 7f453caf8640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f451c0386f0 0x7f451c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.341+0000 7f453caf8640 10 monclient: get_auth_request con 0x7f451c0386f0 auth_method 0 2024-09-15T03:38:11.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.343+0000 7f453caf8640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f451c0386f0 0x7f451c03aba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f45280025d0 tx=0x7f4528002a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:11.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.345+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f453401a720 con 0x7f453809cf20 2024-09-15T03:38:11.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.345+0000 7f452d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:11.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.345+0000 7f452d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:11.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f453f55d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f453f55d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f453f55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"} v 0) v1 -- 0x7f45380636e0 con 0x7f453809cf20 2024-09-15T03:38:11.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}]=0 v255) v1 ==== 106+0+3 (secure 0 0 0) 0x7f453401a360 con 0x7f453809cf20 2024-09-15T03:38:11.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f452d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.509+0000 7f452d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:11.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f453f55d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f453f55d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f453f55d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"} v 0) v1 -- 0x7f453806f270 con 0x7f453809cf20 2024-09-15T03:38:11.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f452d7fa640 1 -- 172.21.15.181:0/3626452763 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}]=0 v255) v1 ==== 106+0+3 (secure 0 0 0) 0x7f453401a540 con 0x7f453809cf20 2024-09-15T03:38:11.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f452d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "8639cfee-8b01-45a4-bdca-7d4262ff6c5b"}] 2024-09-15T03:38:11.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.611+0000 7f452d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:11.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f451c0386f0 msgr2=0x7f451c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f451c0386f0 0x7f451c03aba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f45280025d0 tx=0x7f4528002a60 comp rx=0 tx=0).stop 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 10 monclient: shutdown 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 msgr2=0x7f4538135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 0x7f4538135450 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f453400c5e0 tx=0x7f4534004960 comp rx=0 tx=0).stop 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 shutdown_connections 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f451c0386f0 0x7f451c03aba0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45380a0430 0x7f453811bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f453809f140 0x7f453811b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 --2- 172.21.15.181:0/3626452763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f453809cf20 0x7f4538135450 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 >> 172.21.15.181:0/3626452763 conn(0x7f45380a5230 msgr2=0x7f453809d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 shutdown_connections 2024-09-15T03:38:11.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.613+0000 7f453f55d640 1 -- 172.21.15.181:0/3626452763 wait complete. 2024-09-15T03:38:11.623 INFO:tasks.workunit.client.0.smithi181.stderr:+ test 14 -eq 14 2024-09-15T03:38:11.623 INFO:tasks.workunit.client.0.smithi181.stderr:+ test 14 -ne 12 2024-09-15T03:38:11.623 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm 12 2024-09-15T03:38:11.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.694+0000 7fd0230c9640 10 monclient: get_monmap_and_config 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.694+0000 7fd0230c9640 10 monclient: build_initial_monmap 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.694+0000 7fd0230c9640 10 monclient: monmap: 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:11.695729+0000 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:11.695729+0000 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.696 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:11.697 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:11.697 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.695+0000 7fd0230c9640 1 Processor -- start 2024-09-15T03:38:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.695+0000 7fd0230c9640 1 -- start start 2024-09-15T03:38:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.695+0000 7fd0230c9640 10 monclient: init 2024-09-15T03:38:11.699 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd01c09c230 0x7fd01c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c09c230 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd01c09c600 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd01c09d170 0x7fd01c09e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c09d170 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c09b1d0 con 0x7fd01c09c600 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c09d570 con 0x7fd01c09d170 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c09f5b0 con 0x7fd01c09c230 2024-09-15T03:38:11.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd0230c9640 10 monclient(hunting): authenticate will time out at 1296.177322s 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd01c09c230 0x7fd01c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd01bfff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd01c09c600 0x7fd01c09da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57962/0 (socket says 172.21.15.181:57962) 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd01c09c230 0x7fd01c09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36238/0 (socket says 172.21.15.181:36238) 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd01bfff640 1 -- 172.21.15.181:0/513445100 learned_addr learned my addr 172.21.15.181:0/513445100 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): get_auth_request con 0x7fd01c09c230 auth_method 0 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd021666640 1 -- 172.21.15.181:0/513445100 >> v2:172.21.15.153:3300/0 conn(0x7fd01c09d170 msgr2=0x7fd01c09e170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd021666640 1 --2- 172.21.15.181:0/513445100 >> v2:172.21.15.153:3300/0 conn(0x7fd01c09d170 0x7fd01c09e170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient(hunting): handle_auth_done global_id 104770 payload 274 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 -- 172.21.15.181:0/513445100 >> v2:172.21.15.153:3300/0 conn(0x7fd01c09d170 msgr2=0x7fd01c09e170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 --2- 172.21.15.181:0/513445100 >> v2:172.21.15.153:3300/0 conn(0x7fd01c09d170 0x7fd01c09e170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 -- 172.21.15.181:0/513445100 >> v1:172.21.15.181:6789/0 conn(0x7fd01c09c600 legacy=0x7fd01c09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient: _finish_hunting 0 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 monclient: found mon.noname-c 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.696+0000 7fd020e65640 1 -- 172.21.15.181:0/513445100 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c09b470 con 0x7fd01c09c230 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd020e65640 1 -- 172.21.15.181:0/513445100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c230 msgr2=0x7fd01c09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd020e65640 1 --2- 172.21.15.181:0/513445100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c230 0x7fd01c09cc30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd01c09fac0 tx=0x7fd00c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a3184f06959e8ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 1 -- 172.21.15.181:0/513445100 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c032440 con 0x7fd01c09c230 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: dump: 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.709 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd0230c9640 5 monclient: authenticate success, global_id 104770 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd0230c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:11.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 1 -- 172.21.15.181:0/513445100 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c0325a0 con 0x7fd01c09c230 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd0230c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd0230c9640 4 set_mon_vals no callback set 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 1 -- 172.21.15.181:0/513445100 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c032940 con 0x7fd01c09c230 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.699+0000 7fd01b7fe640 10 monclient: dump: 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.711 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 10 monclient: shutdown 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 -- 172.21.15.181:0/513445100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c230 msgr2=0x7fd01c09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 --2- 172.21.15.181:0/513445100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c230 0x7fd01c09cc30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd01c09fac0 tx=0x7fd00c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 -- 172.21.15.181:0/513445100 shutdown_connections 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 --2- 172.21.15.181:0/513445100 >> v2:172.21.15.153:3300/0 conn(0x7fd01c09d170 0x7fd01c09e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 --2- 172.21.15.181:0/513445100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c230 0x7fd01c09cc30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 -- 172.21.15.181:0/513445100 >> 172.21.15.181:0/513445100 conn(0x7fd01c071350 msgr2=0x7fd01c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 -- 172.21.15.181:0/513445100 shutdown_connections 2024-09-15T03:38:11.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 1 -- 172.21.15.181:0/513445100 wait complete. 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 10 monclient: build_initial_monmap 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.700+0000 7fd0230c9640 10 monclient: monmap: 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 Processor -- start 2024-09-15T03:38:11.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 -- start start 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient: init 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd01c09c230 0x7fd01c13eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c09c230 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c09c600 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd01c09d170 0x7fd01c11b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c09d170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c06ec10 con 0x7fd01c09c230 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c06de90 con 0x7fd01c09c600 2024-09-15T03:38:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c11f090 con 0x7fd01c09d170 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd0230c9640 10 monclient(hunting): authenticate will time out at 1296.182691s 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36244/0 (socket says 172.21.15.181:36244) 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 1 -- 172.21.15.181:0/842010880 learned_addr learned my addr 172.21.15.181:0/842010880 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): get_auth_request con 0x7fd01c09c600 auth_method 0 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd020e65640 1 -- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd01c09c230 msgr2=0x7fd01c13eee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd020e65640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd01c09c230 0x7fd01c13eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:11.715 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.701+0000 7fd01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 10 monclient(hunting): handle_auth_done global_id 104773 payload 824 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 1 -- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd01c09d170 msgr2=0x7fd01c11b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd021666640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd01c09d170 0x7fd01c11b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd01c09d170 0x7fd01c11b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 1 -- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd01c09c230 msgr2=0x7fd01c13eee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd01c09c230 0x7fd01c13eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 1 monclient: found mon.noname-b 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd01bfff640 1 -- 172.21.15.181:0/842010880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c009390 con 0x7fd01c09c600 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd021666640 10 monclient: get_auth_request con 0x7fd01c09d170 auth_method 0 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.702+0000 7fd021666640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd01c09d170 0x7fd01c11b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd01bfff640 1 -- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 msgr2=0x7fd01c131c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd01bfff640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd00400c610 tx=0x7fd004004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd004005cb0 con 0x7fd01c09c600 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: dump: 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: _finish_auth 0 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd004005e30 con 0x7fd01c09c600 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 5 monclient: authenticate success, global_id 104773 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 10 monclient: _renew_subs 2024-09-15T03:38:11.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01c13fda0 con 0x7fd01c09c600 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd004007340 con 0x7fd01c09c600 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd018ff9640 10 monclient: dump: 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:11.719 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd01affd640 4 set_mon_vals no callback set 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 10 monclient: _renew_subs 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.704+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c11ddc0 con 0x7fd01c09c600 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd00401b400 con 0x7fd01c09c600 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd0230c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd0230c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd0230c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01c0636e0 con 0x7fd01c09c600 2024-09-15T03:38:11.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd018ff9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcffc0386f0 0x7fcffc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:11.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(255..255 src has 199..255) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd00404c3a0 con 0x7fd01c09c600 2024-09-15T03:38:11.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd020e65640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcffc0386f0 0x7fcffc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:11.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.705+0000 7fd020e65640 10 monclient: get_auth_request con 0x7fcffc0386f0 auth_method 0 2024-09-15T03:38:11.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.708+0000 7fd020e65640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcffc0386f0 0x7fcffc03aba0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd01c09b7c0 tx=0x7fd00c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:11.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:11.709+0000 7fd01bfff640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd00400c610 tx=0x7fd004004960 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fd00401a720 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:38:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.538+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd00401a720 con 0x7fd01c09c600 2024-09-15T03:38:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.538+0000 7fd018ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.538+0000 7fd018ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:12.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.704+0000 7fd0230c9640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:12.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.704+0000 7fd0230c9640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:12.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.704+0000 7fd0230c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:12.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:12.704+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fd01c09e6b0 con 0x7fd01c09c600 2024-09-15T03:38:12.708 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:12.705+0000 7f0eedff1640 -1 mon.b@1(leader).osd e255 definitely_dead 0 2024-09-15T03:38:13.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.402+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v256) v1 ==== 83+0+0 (secure 0 0 0) 0x7fd00401a360 con 0x7fd01c09c600 2024-09-15T03:38:13.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.402+0000 7fd018ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.402+0000 7fd018ff9640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-15T03:38:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd0230c9640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd0230c9640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd0230c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fd01c064b00 con 0x7fd01c09c600 2024-09-15T03:38:13.506 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:13.504+0000 7f0eedff1640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-15T03:38:13.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd018ff9640 1 -- 172.21.15.181:0/842010880 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v256) v1 ==== 92+0+0 (secure 0 0 0) 0x7fd00401a540 con 0x7fd01c09c600 2024-09-15T03:38:13.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd018ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.504+0000 7fd018ff9640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-15T03:38:13.507 INFO:tasks.workunit.client.0.smithi181.stderr:osd.12 does not exist. 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcffc0386f0 msgr2=0x7fcffc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcffc0386f0 0x7fcffc03aba0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd01c09b7c0 tx=0x7fd00c032000 comp rx=0 tx=0).stop 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 10 monclient: shutdown 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 msgr2=0x7fd01c131c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd00400c610 tx=0x7fd004004960 comp rx=0 tx=0).stop 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 shutdown_connections 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcffc0386f0 0x7fcffc03aba0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd01c09d170 0x7fd01c11b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd01c09c600 0x7fd01c131c00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 --2- 172.21.15.181:0/842010880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd01c09c230 0x7fd01c13eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.506+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 >> 172.21.15.181:0/842010880 conn(0x7fd01c071350 msgr2=0x7fd01c098a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:13.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.507+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 shutdown_connections 2024-09-15T03:38:13.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.507+0000 7fd0230c9640 1 -- 172.21.15.181:0/842010880 wait complete. 2024-09-15T03:38:13.516 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm 12 2024-09-15T03:38:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.588+0000 7f9b3ff00640 10 monclient: get_monmap_and_config 2024-09-15T03:38:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.588+0000 7f9b3ff00640 10 monclient: build_initial_monmap 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.588+0000 7f9b3ff00640 10 monclient: monmap: 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:13.588866+0000 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:13.588866+0000 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.588+0000 7f9b3ff00640 1 Processor -- start 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.588+0000 7f9b3ff00640 1 -- start start 2024-09-15T03:38:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.588+0000 7f9b3ff00640 10 monclient: init 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9b38131b40 0x7f9b3809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): picked mon.noname-a con 0x7f9b38131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b3809f240 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 0x7f9b38139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): picked mon.noname-b con 0x7f9b38136e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3806a230 con 0x7f9b3809f240 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b380a0300 con 0x7f9b38131b40 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380a00c0 con 0x7f9b38136e70 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3ff00640 10 monclient(hunting): authenticate will time out at 1298.070342s 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3dc9c640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9b38131b40 msgr2=0x7f9b3809ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3d49b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f9b3809f240 0x7f9b38131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57970/0 (socket says 172.21.15.181:57970) 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3d49b640 1 -- 172.21.15.181:0/2767731118 learned_addr learned my addr 172.21.15.181:0/2767731118 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3dc9c640 1 --2- 172.21.15.181:0/2767731118 >> v2:172.21.15.153:3300/0 conn(0x7f9b38131b40 0x7f9b3809ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 1 --2- 172.21.15.181:0/2767731118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 0x7f9b38139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): get_auth_request con 0x7f9b38136e70 auth_method 0 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.589+0000 7f9b3e49d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 10 monclient(hunting): handle_auth_done global_id 124211 payload 274 2024-09-15T03:38:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 1 -- 172.21.15.181:0/2767731118 >> v2:172.21.15.153:3300/0 conn(0x7f9b38131b40 msgr2=0x7f9b3809ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 1 --2- 172.21.15.181:0/2767731118 >> v2:172.21.15.153:3300/0 conn(0x7f9b38131b40 0x7f9b3809ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 1 -- 172.21.15.181:0/2767731118 >> v1:172.21.15.181:6789/0 conn(0x7f9b3809f240 legacy=0x7f9b38131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 10 monclient: _finish_hunting 0 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 1 monclient: found mon.noname-b 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.590+0000 7f9b3e49d640 1 -- 172.21.15.181:0/2767731118 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3809bee0 con 0x7f9b38136e70 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3e49d640 1 -- 172.21.15.181:0/2767731118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 msgr2=0x7f9b38139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3e49d640 1 --2- 172.21.15.181:0/2767731118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 0x7f9b38139250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009e60 tx=0x7f9b30009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=166a182645070746 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 1 -- 172.21.15.181:0/2767731118 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007840 con 0x7f9b38136e70 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: dump: 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: _finish_auth 0 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 1 -- 172.21.15.181:0/2767731118 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3002d050 con 0x7f9b38136e70 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 1 -- 172.21.15.181:0/2767731118 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007bd0 con 0x7f9b38136e70 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3cc9a640 10 monclient: dump: 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 5 monclient: authenticate success, global_id 124211 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 4 set_mon_vals no callback set 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 10 monclient: shutdown 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 -- 172.21.15.181:0/2767731118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 msgr2=0x7f9b38139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/2767731118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 0x7f9b38139250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009e60 tx=0x7f9b30009780 comp rx=0 tx=0).stop 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 -- 172.21.15.181:0/2767731118 shutdown_connections 2024-09-15T03:38:13.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/2767731118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b38136e70 0x7f9b38139250 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/2767731118 >> v2:172.21.15.153:3300/0 conn(0x7f9b38131b40 0x7f9b3809ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 -- 172.21.15.181:0/2767731118 >> 172.21.15.181:0/2767731118 conn(0x7f9b380a5100 msgr2=0x7f9b380a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 -- 172.21.15.181:0/2767731118 shutdown_connections 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.593+0000 7f9b3ff00640 1 -- 172.21.15.181:0/2767731118 wait complete. 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.594+0000 7f9b3ff00640 10 monclient: build_initial_monmap 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.594+0000 7f9b3ff00640 10 monclient: monmap: 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:13.608 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.594+0000 7f9b3ff00640 1 Processor -- start 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.594+0000 7f9b3ff00640 1 -- start start 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.594+0000 7f9b3ff00640 10 monclient: init 2024-09-15T03:38:13.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b3809f240 0x7f9b38141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3809f240 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 0x7f9b38141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): picked mon.noname-b con 0x7f9b38131b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b38136e70 0x7f9b380987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): picked mon.noname-a con 0x7f9b38136e70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3811be50 con 0x7f9b38136e70 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b38130b60 con 0x7f9b38131b40 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3811bb70 con 0x7f9b3809f240 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3ff00640 10 monclient(hunting): authenticate will time out at 1298.076190s 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 0x7f9b38141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3e49d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b38136e70 msgr2=0x7f9b380987f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3e49d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b38136e70 0x7f9b380987f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 0x7f9b38141a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36256/0 (socket says 172.21.15.181:36256) 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 -- 172.21.15.181:0/3981329430 learned_addr learned my addr 172.21.15.181:0/3981329430 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): get_auth_request con 0x7f9b38131b40 auth_method 0 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3dc9c640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b3809f240 0x7f9b38141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3dc9c640 10 monclient(hunting): get_auth_request con 0x7f9b3809f240 auth_method 0 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3dc9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3dc9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3dc9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient(hunting): handle_auth_done global_id 104776 payload 824 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 -- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b3809f240 msgr2=0x7f9b38141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b3809f240 0x7f9b38141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 -- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b38136e70 msgr2=0x7f9b380987f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b38136e70 0x7f9b380987f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient: _finish_hunting 0 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 monclient: found mon.noname-b 2024-09-15T03:38:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3d49b640 1 -- 172.21.15.181:0/3981329430 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30009a20 con 0x7f9b38131b40 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.595+0000 7f9b3dc9c640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b3809f240 0x7f9b38141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b3d49b640 1 -- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 msgr2=0x7f9b38141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b3d49b640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 0x7f9b38141a50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800c5e0 tx=0x7f9b28004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28005cb0 con 0x7f9b38131b40 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: dump: 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.614 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: _finish_auth 0 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b3ff00640 5 monclient: authenticate success, global_id 104776 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b28005e30 con 0x7f9b38131b40 2024-09-15T03:38:13.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28007340 con 0x7f9b38131b40 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b3ff00640 10 monclient: _renew_subs 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b3ff00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3811b890 con 0x7f9b38131b40 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2dffb640 10 monclient: dump: 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.616 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.597+0000 7f9b2ffff640 4 set_mon_vals no callback set 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 10 monclient: _renew_subs 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b38141f90 con 0x7f9b38131b40 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9b2801b400 con 0x7f9b38131b40 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.598+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3809a220 con 0x7f9b38131b40 2024-09-15T03:38:13.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.599+0000 7f9b2dffb640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b1c0386f0 0x7f9b1c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.599+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(256..256 src has 199..256) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9b2804c1d0 con 0x7f9b38131b40 2024-09-15T03:38:13.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.599+0000 7f9b3dc9c640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b1c0386f0 0x7f9b1c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.599+0000 7f9b3dc9c640 10 monclient: get_auth_request con 0x7f9b1c0386f0 auth_method 0 2024-09-15T03:38:13.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.601+0000 7f9b3dc9c640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b1c0386f0 0x7f9b1c03aba0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9b34004fd0 tx=0x7f9b34004980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:13.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.601+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:38:13.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.603+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9b2801ab40 con 0x7f9b38131b40 2024-09-15T03:38:13.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.603+0000 7f9b2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:13.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.603+0000 7f9b2dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:13.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.765+0000 7f9b3ff00640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.765+0000 7f9b3ff00640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.765+0000 7f9b3ff00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.765+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f9b380636e0 con 0x7f9b38131b40 2024-09-15T03:38:13.767 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:13.765+0000 7f0eedff1640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-15T03:38:13.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.766+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v256) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9b2801a360 con 0x7f9b38131b40 2024-09-15T03:38:13.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.766+0000 7f9b2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.766+0000 7f9b2dffb640 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-15T03:38:13.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b3ff00640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b3ff00640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b3ff00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f9b3809adb0 con 0x7f9b38131b40 2024-09-15T03:38:13.867 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f0eedff1640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-15T03:38:13.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b2dffb640 1 -- 172.21.15.181:0/3981329430 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v256) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9b2801a540 con 0x7f9b38131b40 2024-09-15T03:38:13.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b2dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-15T03:38:13.868 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.866+0000 7f9b2dffb640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-15T03:38:13.868 INFO:tasks.workunit.client.0.smithi181.stderr:osd.12 does not exist. 2024-09-15T03:38:13.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b1c0386f0 msgr2=0x7f9b1c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b1c0386f0 0x7f9b1c03aba0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9b34004fd0 tx=0x7f9b34004980 comp rx=0 tx=0).stop 2024-09-15T03:38:13.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 10 monclient: shutdown 2024-09-15T03:38:13.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:13.869 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 msgr2=0x7f9b38141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 0x7f9b38141a50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800c5e0 tx=0x7f9b28004960 comp rx=0 tx=0).stop 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 shutdown_connections 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b1c0386f0 0x7f9b1c03aba0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b38136e70 0x7f9b380987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b38131b40 0x7f9b38141a50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 --2- 172.21.15.181:0/3981329430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b3809f240 0x7f9b38141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 >> 172.21.15.181:0/3981329430 conn(0x7f9b380a5100 msgr2=0x7f9b380a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 shutdown_connections 2024-09-15T03:38:13.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.868+0000 7f9b3ff00640 1 -- 172.21.15.181:0/3981329430 wait complete. 2024-09-15T03:38:13.878 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm 14 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.950+0000 7fb21362b640 10 monclient: get_monmap_and_config 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.950+0000 7fb21362b640 10 monclient: build_initial_monmap 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.950+0000 7fb21362b640 10 monclient: monmap: 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:13.950975+0000 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:13.950975+0000 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:13.952 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.950+0000 7fb21362b640 1 Processor -- start 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.950+0000 7fb21362b640 1 -- start start 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.950+0000 7fb21362b640 10 monclient: init 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb20c0a37b0 0x7fb20c06e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): picked mon.noname-c con 0x7fb20c0a37b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb20c06be40 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:13.954 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c06a6f0 0x7fb20c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): picked mon.noname-b con 0x7fb20c06a6f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20c099ff0 con 0x7fb20c06be40 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20c069d10 con 0x7fb20c0a37b0 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0a3b80 con 0x7fb20c06a6f0 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb21362b640 10 monclient(hunting): authenticate will time out at 1298.432474s 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb20c0a37b0 0x7fb20c06e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb20c0a37b0 0x7fb20c06e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36268/0 (socket says 172.21.15.181:36268) 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 -- 172.21.15.181:0/3585188443 learned_addr learned my addr 172.21.15.181:0/3585188443 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): get_auth_request con 0x7fb20c0a37b0 auth_method 0 2024-09-15T03:38:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb210bc6640 1 --1- 172.21.15.181:0/3585188443 >> v1:172.21.15.181:6789/0 conn(0x7fb20c06be40 0x7fb20c09d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57976/0 (socket says 172.21.15.181:57976) 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb211bc8640 1 --2- 172.21.15.181:0/3585188443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c06a6f0 0x7fb20c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb211bc8640 10 monclient(hunting): get_auth_request con 0x7fb20c06a6f0 auth_method 0 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb211bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb211bc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb211bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient(hunting): handle_auth_done global_id 104779 payload 274 2024-09-15T03:38:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 -- 172.21.15.181:0/3585188443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c06a6f0 msgr2=0x7fb20c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 --2- 172.21.15.181:0/3585188443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c06a6f0 0x7fb20c09e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 -- 172.21.15.181:0/3585188443 >> v1:172.21.15.181:6789/0 conn(0x7fb20c06be40 legacy=0x7fb20c09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient: _finish_hunting 0 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 monclient: found mon.noname-c 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.951+0000 7fb2113c7640 1 -- 172.21.15.181:0/3585188443 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c09f520 con 0x7fb20c0a37b0 2024-09-15T03:38:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb2113c7640 1 -- 172.21.15.181:0/3585188443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c0a37b0 msgr2=0x7fb20c06e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb2113c7640 1 --2- 172.21.15.181:0/3585188443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c0a37b0 0x7fb20c06e2d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb208009910 tx=0x7fb2080091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e83dba76b8a08f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 1 -- 172.21.15.181:0/3585188443 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20802aea0 con 0x7fb20c0a37b0 2024-09-15T03:38:13.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: dump: 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.963 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: _finish_auth 0 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 1 -- 172.21.15.181:0/3585188443 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208032440 con 0x7fb20c0a37b0 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb21362b640 5 monclient: authenticate success, global_id 104779 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb21362b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 1 -- 172.21.15.181:0/3585188443 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080327e0 con 0x7fb20c0a37b0 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb21362b640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb21362b640 4 set_mon_vals no callback set 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.954+0000 7fb1fffff640 10 monclient: dump: 2024-09-15T03:38:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 10 monclient: shutdown 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 -- 172.21.15.181:0/3585188443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c0a37b0 msgr2=0x7fb20c06e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 --2- 172.21.15.181:0/3585188443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c0a37b0 0x7fb20c06e2d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb208009910 tx=0x7fb2080091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 -- 172.21.15.181:0/3585188443 shutdown_connections 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 --2- 172.21.15.181:0/3585188443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c06a6f0 0x7fb20c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 --2- 172.21.15.181:0/3585188443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c0a37b0 0x7fb20c06e2d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 -- 172.21.15.181:0/3585188443 >> 172.21.15.181:0/3585188443 conn(0x7fb20c070fd0 msgr2=0x7fb20c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 -- 172.21.15.181:0/3585188443 shutdown_connections 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 1 -- 172.21.15.181:0/3585188443 wait complete. 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 10 monclient: build_initial_monmap 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.955+0000 7fb21362b640 10 monclient: monmap: 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 Processor -- start 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 -- start start 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient: init 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20c06a6f0 0x7fb20c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): picked mon.noname-a con 0x7fb20c06a6f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 0x7fb20c133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): picked mon.noname-b con 0x7fb20c06be40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c0a37b0 0x7fb20c11ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): picked mon.noname-c con 0x7fb20c0a37b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c06eaa0 con 0x7fb20c06a6f0 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c120530 con 0x7fb20c06be40 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c120a00 con 0x7fb20c0a37b0 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.956+0000 7fb21362b640 10 monclient(hunting): authenticate will time out at 1298.437991s 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb2113c7640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20c06a6f0 msgr2=0x7fb20c141090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 0x7fb20c133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb2113c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20c06a6f0 0x7fb20c141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 0x7fb20c133ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36272/0 (socket says 172.21.15.181:36272) 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 -- 172.21.15.181:0/1198471847 learned_addr learned my addr 172.21.15.181:0/1198471847 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): get_auth_request con 0x7fb20c06be40 auth_method 0 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c0a37b0 0x7fb20c11ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): get_auth_request con 0x7fb20c0a37b0 auth_method 0 2024-09-15T03:38:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb211bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient(hunting): handle_auth_done global_id 104782 payload 824 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 -- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c0a37b0 msgr2=0x7fb20c11ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c0a37b0 0x7fb20c11ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 -- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20c06a6f0 msgr2=0x7fb20c141090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20c06a6f0 0x7fb20c141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient: _finish_hunting 0 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 monclient: found mon.noname-b 2024-09-15T03:38:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.957+0000 7fb210bc6640 1 -- 172.21.15.181:0/1198471847 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb208009450 con 0x7fb20c06be40 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb210bc6640 1 -- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 msgr2=0x7fb20c133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb210bc6640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 0x7fb20c133ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c5e0 tx=0x7fb200007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200007510 con 0x7fb20c06be40 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: dump: 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2000057d0 con 0x7fb20c06be40 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 5 monclient: authenticate success, global_id 104782 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 10 monclient: _renew_subs 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c122050 con 0x7fb20c06be40 2024-09-15T03:38:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200007690 con 0x7fb20c06be40 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1fd7fa640 10 monclient: dump: 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb1ff7fe640 4 set_mon_vals no callback set 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 10 monclient: _renew_subs 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.959+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c141e60 con 0x7fb20c06be40 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.960+0000 7fb21362b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.960+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb20001b400 con 0x7fb20c06be40 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.960+0000 7fb21362b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.960+0000 7fb21362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.960+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c1226d0 con 0x7fb20c06be40 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.961+0000 7fb1fd7fa640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1f00386f0 0x7fb1f003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.961+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(256..256 src has 199..256) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb20004c170 con 0x7fb20c06be40 2024-09-15T03:38:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.961+0000 7fb2113c7640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1f00386f0 0x7fb1f003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.961+0000 7fb2113c7640 10 monclient: get_auth_request con 0x7fb1f00386f0 auth_method 0 2024-09-15T03:38:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.963+0000 7fb2113c7640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1f00386f0 0x7fb1f003aba0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb20c069ef0 tx=0x7fb208032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.965+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb200005e60 con 0x7fb20c06be40 2024-09-15T03:38:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.965+0000 7fb1fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:13.965+0000 7fb1fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:14.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.129+0000 7fb21362b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T03:38:14.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.129+0000 7fb21362b640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T03:38:14.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.129+0000 7fb21362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.129+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7fb20c0636e0 con 0x7fb20c06be40 2024-09-15T03:38:14.131 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:14.129+0000 7f0eedff1640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-15T03:38:14.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.406+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v257) v1 ==== 83+0+0 (secure 0 0 0) 0x7fb20001a360 con 0x7fb20c06be40 2024-09-15T03:38:14.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.406+0000 7fb1fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T03:38:14.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.406+0000 7fb1fd7fa640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-15T03:38:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb21362b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T03:38:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb21362b640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T03:38:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb21362b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7fb20c122330 con 0x7fb20c06be40 2024-09-15T03:38:14.509 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:14.507+0000 7f0eedff1640 -1 mon.b@1(leader).osd e257 definitely_dead 0 2024-09-15T03:38:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb1fd7fa640 1 -- 172.21.15.181:0/1198471847 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v257) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb20001a540 con 0x7fb20c06be40 2024-09-15T03:38:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb1fd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-15T03:38:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.507+0000 7fb1fd7fa640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-15T03:38:14.509 INFO:tasks.workunit.client.0.smithi181.stderr:osd.14 does not exist. 2024-09-15T03:38:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1f00386f0 msgr2=0x7fb1f003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1f00386f0 0x7fb1f003aba0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb20c069ef0 tx=0x7fb208032000 comp rx=0 tx=0).stop 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 10 monclient: shutdown 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 msgr2=0x7fb20c133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 0x7fb20c133ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c5e0 tx=0x7fb200007f00 comp rx=0 tx=0).stop 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 shutdown_connections 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1f00386f0 0x7fb1f003aba0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20c0a37b0 0x7fb20c11ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb20c06be40 0x7fb20c133ec0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 --2- 172.21.15.181:0/1198471847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20c06a6f0 0x7fb20c141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 >> 172.21.15.181:0/1198471847 conn(0x7fb20c070fd0 msgr2=0x7fb20c06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 shutdown_connections 2024-09-15T03:38:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.509+0000 7fb21362b640 1 -- 172.21.15.181:0/1198471847 wait complete. 2024-09-15T03:38:14.519 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph osd rm 1000 2024-09-15T03:38:14.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.590+0000 7fe0d68f8640 10 monclient: get_monmap_and_config 2024-09-15T03:38:14.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.590+0000 7fe0d68f8640 10 monclient: build_initial_monmap 2024-09-15T03:38:14.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.590+0000 7fe0d68f8640 10 monclient: monmap: 2024-09-15T03:38:14.591 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:14.591 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.591 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:14.591236+0000 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:14.591236+0000 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:14.592 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 Processor -- start 2024-09-15T03:38:14.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 -- start start 2024-09-15T03:38:14.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient: init 2024-09-15T03:38:14.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:14.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0132170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d0069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe0d0070920 0x7fe0d0070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d0070920 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0d0071230 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d009d110 con 0x7fe0d0071230 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d009e520 con 0x7fe0d0070920 2024-09-15T03:38:14.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0069e00 con 0x7fe0d0069a30 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d68f8640 10 monclient(hunting): authenticate will time out at 1299.072685s 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe0d0070920 0x7fe0d0070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d4e95640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fe0d0071230 0x7fe0d012e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:57984/0 (socket says 172.21.15.181:57984) 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0d4e95640 1 -- 172.21.15.181:0/1902141535 learned_addr learned my addr 172.21.15.181:0/1902141535 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): get_auth_request con 0x7fe0d0070920 auth_method 0 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.591+0000 7fe0cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cffff640 1 --2- 172.21.15.181:0/1902141535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cffff640 10 monclient(hunting): get_auth_request con 0x7fe0d0069a30 auth_method 0 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 10 monclient(hunting): handle_auth_done global_id 104785 payload 274 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 1 -- 172.21.15.181:0/1902141535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 msgr2=0x7fe0d0132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 1 --2- 172.21.15.181:0/1902141535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0132170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 1 -- 172.21.15.181:0/1902141535 >> v1:172.21.15.181:6789/0 conn(0x7fe0d0071230 legacy=0x7fe0d012e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:38:14.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 1 monclient: found mon.noname-c 2024-09-15T03:38:14.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:14.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.599 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.592+0000 7fe0cf7fe640 1 -- 172.21.15.181:0/1902141535 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d009ce60 con 0x7fe0d0070920 2024-09-15T03:38:14.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.594+0000 7fe0cf7fe640 1 -- 172.21.15.181:0/1902141535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0070920 msgr2=0x7fe0d0070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:14.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.594+0000 7fe0cf7fe640 1 --2- 172.21.15.181:0/1902141535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0070920 0x7fe0d0070cf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4009850 tx=0x7fe0c40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34584e6051a617e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:14.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 1 -- 172.21.15.181:0/1902141535 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c402ae80 con 0x7fe0d0070920 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: dump: 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: _finish_auth 0 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 1 -- 172.21.15.181:0/1902141535 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c4032440 con 0x7fe0d0070920 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 5 monclient: authenticate success, global_id 104785 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 1 -- 172.21.15.181:0/1902141535 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40327c0 con 0x7fe0d0070920 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 4 set_mon_vals no callback set 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0ceffd640 10 monclient: dump: 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.608 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 10 monclient: shutdown 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1902141535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0070920 msgr2=0x7fe0d0070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1902141535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0070920 0x7fe0d0070cf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4009850 tx=0x7fe0c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1902141535 shutdown_connections 2024-09-15T03:38:14.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1902141535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0070920 0x7fe0d0070cf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1902141535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1902141535 >> 172.21.15.181:0/1902141535 conn(0x7fe0d009ea70 msgr2=0x7fe0d009ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1902141535 shutdown_connections 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.595+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1902141535 wait complete. 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 10 monclient: build_initial_monmap 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 10 monclient: monmap: 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:14.610 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 1 Processor -- start 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 1 -- start start 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 10 monclient: init 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.596+0000 7fe0d68f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d0069a30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0d0070920 0x7fe0d0141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d0070920 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 0x7fe0d0141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d0071230 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:14.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d011f350 con 0x7fe0d0070920 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d009d620 con 0x7fe0d0071230 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d011f070 con 0x7fe0d0069a30 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d68f8640 10 monclient(hunting): authenticate will time out at 1299.078064s 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cf7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0d0070920 msgr2=0x7fe0d0141020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0d0070920 0x7fe0d0141020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 0x7fe0d0141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 0x7fe0d0141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36294/0 (socket says 172.21.15.181:36294) 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 -- 172.21.15.181:0/1891070108 learned_addr learned my addr 172.21.15.181:0/1891070108 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:14.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): get_auth_request con 0x7fe0d0071230 auth_method 0 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): get_auth_request con 0x7fe0d0069a30 auth_method 0 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:14.613 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient(hunting): handle_auth_done global_id 104788 payload 824 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 -- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 msgr2=0x7fe0d0130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0130c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 -- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0d0070920 msgr2=0x7fe0d0141020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0d0070920 0x7fe0d0141020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient: _finish_hunting 0 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 monclient: found mon.noname-b 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.597+0000 7fe0d4e95640 1 -- 172.21.15.181:0/1891070108 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4009390 con 0x7fe0d0071230 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d4e95640 1 -- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 msgr2=0x7fe0d0141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d4e95640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 0x7fe0d0141560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800ccc0 tx=0x7fe0c8005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:14.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c800ce80 con 0x7fe0d0071230 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: dump: 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: _finish_auth 0 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c800aa60 con 0x7fe0d0071230 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 5 monclient: authenticate success, global_id 104788 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 10 monclient: _renew_subs 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d011dea0 con 0x7fe0d0071230 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c800ae20 con 0x7fe0d0071230 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0b7fff640 10 monclient: dump: 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.616 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 10 monclient: _renew_subs 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d011ee40 con 0x7fe0d0071230 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.599+0000 7fe0ce7fc640 4 set_mon_vals no callback set 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0d68f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:14.617 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe0c8007300 con 0x7fe0d0071230 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0d68f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0d68f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d00636e0 con 0x7fe0d0071230 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0b7fff640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0bc0386f0 0x7fe0bc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.600+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(257..257 src has 199..257) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe0c804c700 con 0x7fe0d0071230 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.601+0000 7fe0cffff640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0bc0386f0 0x7fe0bc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.601+0000 7fe0cffff640 10 monclient: get_auth_request con 0x7fe0bc0386f0 auth_method 0 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.603+0000 7fe0cffff640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0bc0386f0 0x7fe0bc03aba0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0007ba0 tx=0x7fe0c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.605+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe0c80075b0 con 0x7fe0d0071230 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.605+0000 7fe0b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:14.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.605+0000 7fe0b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:14.771 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.770+0000 7fe0d68f8640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T03:38:14.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.770+0000 7fe0d68f8640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T03:38:14.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.770+0000 7fe0d68f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.770+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7fe0d0064c00 con 0x7fe0d0071230 2024-09-15T03:38:14.772 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:14.771+0000 7f0eedff1640 -1 mon.b@1(leader).osd e257 definitely_dead 0 2024-09-15T03:38:14.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.771+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v257) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe0c8010af0 con 0x7fe0d0071230 2024-09-15T03:38:14.772 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.771+0000 7fe0b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T03:38:14.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.771+0000 7fe0b7fff640 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-15T03:38:14.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0d68f8640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T03:38:14.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0d68f8640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T03:38:14.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0d68f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7fe0d0131260 con 0x7fe0d0071230 2024-09-15T03:38:14.874 INFO:tasks.ceph.mon.b.smithi181.stderr:2024-09-15T03:38:14.872+0000 7f0eedff1640 -1 mon.b@1(leader).osd e257 definitely_dead 0 2024-09-15T03:38:14.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0b7fff640 1 -- 172.21.15.181:0/1891070108 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v257) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe0c801a360 con 0x7fe0d0071230 2024-09-15T03:38:14.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0b7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-15T03:38:14.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.872+0000 7fe0b7fff640 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-15T03:38:14.876 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1000 does not exist. 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0bc0386f0 msgr2=0x7fe0bc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0bc0386f0 0x7fe0bc03aba0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0007ba0 tx=0x7fe0c0008040 comp rx=0 tx=0).stop 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 10 monclient: shutdown 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 msgr2=0x7fe0d0141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 0x7fe0d0141560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800ccc0 tx=0x7fe0c8005370 comp rx=0 tx=0).stop 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 shutdown_connections 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe0bc0386f0 0x7fe0bc03aba0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0d0071230 0x7fe0d0141560 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0d0070920 0x7fe0d0141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 --2- 172.21.15.181:0/1891070108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0d0069a30 0x7fe0d0130c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 >> 172.21.15.181:0/1891070108 conn(0x7fe0d009ea70 msgr2=0x7fe0d00a0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:14.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 shutdown_connections 2024-09-15T03:38:14.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.874+0000 7fe0d68f8640 1 -- 172.21.15.181:0/1891070108 wait complete. 2024-09-15T03:38:14.884 INFO:tasks.workunit.client.0.smithi181.stderr:++ ceph osd create ecab9f08-f922-45d9-8393-9f53fccdb802 2024-09-15T03:38:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.955+0000 7fa15ab5d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.955+0000 7fa15ab5d640 10 monclient: build_initial_monmap 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.955+0000 7fa15ab5d640 10 monclient: monmap: 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:14.956527+0000 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:14.956527+0000 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:14.958 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 Processor -- start 2024-09-15T03:38:14.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 -- start start 2024-09-15T03:38:14.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient: init 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa15409afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa15409cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa15412e940 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa15412f460 0x7fa154135c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa15412f460 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15409d6f0 con 0x7fa15412e940 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa154097a40 con 0x7fa15412f460 2024-09-15T03:38:14.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa154097e90 con 0x7fa15409cb20 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.956+0000 7fa15ab5d640 10 monclient(hunting): authenticate will time out at 1299.437951s 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa14bfff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa15412e940 legacy=0x7fa15412ed50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1590fa640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa15412f460 msgr2=0x7fa154135c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1590fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa15412f460 0x7fa154135c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa15409afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa15409afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.181:38208/0 (socket says 172.21.15.181:38208) 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 -- 172.21.15.181:0/4274801727 learned_addr learned my addr 172.21.15.181:0/4274801727 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): get_auth_request con 0x7fa15409cb20 auth_method 0 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient(hunting): handle_auth_done global_id 124229 payload 274 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 -- 172.21.15.181:0/4274801727 >> v2:172.21.15.153:3300/0 conn(0x7fa15412f460 msgr2=0x7fa154135c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 --2- 172.21.15.181:0/4274801727 >> v2:172.21.15.153:3300/0 conn(0x7fa15412f460 0x7fa154135c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 -- 172.21.15.181:0/4274801727 >> v1:172.21.15.153:6789/0 conn(0x7fa15412e940 legacy=0x7fa15412ed50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient: _finish_hunting 0 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 monclient: found mon.noname-b 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:14.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.957+0000 7fa1588f9640 1 -- 172.21.15.181:0/4274801727 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa154098da0 con 0x7fa15409cb20 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa1588f9640 1 -- 172.21.15.181:0/4274801727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 msgr2=0x7fa15409afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa1588f9640 1 --2- 172.21.15.181:0/4274801727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa15409afb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa150009910 tx=0x7fa1500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeb4c0c47ee80d4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 1 -- 172.21.15.181:0/4274801727 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150032440 con 0x7fa15409cb20 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: dump: 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.972 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 1 -- 172.21.15.181:0/4274801727 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1500325a0 con 0x7fa15409cb20 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa15ab5d640 5 monclient: authenticate success, global_id 124229 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa15ab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 1 -- 172.21.15.181:0/4274801727 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150032940 con 0x7fa15409cb20 2024-09-15T03:38:14.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa15ab5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa15ab5d640 4 set_mon_vals no callback set 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.960+0000 7fa14b7fe640 10 monclient: dump: 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.974 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 10 monclient: shutdown 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 -- 172.21.15.181:0/4274801727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 msgr2=0x7fa15409afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/4274801727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa15409afb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa150009910 tx=0x7fa1500091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 -- 172.21.15.181:0/4274801727 shutdown_connections 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/4274801727 >> v2:172.21.15.153:3300/0 conn(0x7fa15412f460 0x7fa154135c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/4274801727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa15409afb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 -- 172.21.15.181:0/4274801727 >> 172.21.15.181:0/4274801727 conn(0x7fa1540714d0 msgr2=0x7fa1540a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 -- 172.21.15.181:0/4274801727 shutdown_connections 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 1 -- 172.21.15.181:0/4274801727 wait complete. 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 10 monclient: build_initial_monmap 2024-09-15T03:38:14.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.961+0000 7fa15ab5d640 10 monclient: monmap: 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 Processor -- start 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 -- start start 2024-09-15T03:38:14.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient: init 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa154133b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa15409cb20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa15412e940 0x7fa154134050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa15412e940 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 0x7fa154134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa15412f460 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154117dc0 con 0x7fa15412e940 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154134ef0 con 0x7fa15412f460 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:14.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15409c3b0 con 0x7fa15409cb20 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa15ab5d640 10 monclient(hunting): authenticate will time out at 1299.443708s 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa14bfff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa15412e940 msgr2=0x7fa154134050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa14bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa15412e940 0x7fa154134050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 0x7fa154134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 0x7fa154134590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36306/0 (socket says 172.21.15.181:36306) 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 1 -- 172.21.15.181:0/488119143 learned_addr learned my addr 172.21.15.181:0/488119143 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): get_auth_request con 0x7fa15412f460 auth_method 0 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:14.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1590fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.962+0000 7fa1588f9640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa154133b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1588f9640 10 monclient(hunting): get_auth_request con 0x7fa15409cb20 auth_method 0 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1588f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1588f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1588f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 10 monclient(hunting): handle_auth_done global_id 104791 payload 824 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 1 -- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 msgr2=0x7fa154133b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa154133b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 1 -- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa15412e940 msgr2=0x7fa154134050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa15412e940 0x7fa154134050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:14.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 10 monclient: _finish_hunting 0 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 1 monclient: found mon.noname-b 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.963+0000 7fa1590fa640 1 -- 172.21.15.181:0/488119143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150009450 con 0x7fa15412f460 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa1590fa640 1 -- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 msgr2=0x7fa154134590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa1590fa640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 0x7fa154134590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00bbb0 tx=0x7fa14c0097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c00ccb0 con 0x7fa15412f460 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: dump: 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.980 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: _finish_auth 0 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c00ce10 con 0x7fa15412f460 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 5 monclient: authenticate success, global_id 104791 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 10 monclient: _renew_subs 2024-09-15T03:38:14.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa154098950 con 0x7fa15412f460 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c006150 con 0x7fa15412f460 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa148ff9640 10 monclient: dump: 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 10 monclient: _renew_subs 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15409baa0 con 0x7fa15412f460 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.965+0000 7fa14affd640 4 set_mon_vals no callback set 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa14c012020 con 0x7fa15412f460 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa15ab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa15ab5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa15ab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1540636e0 con 0x7fa15412f460 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa148ff9640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa1400386f0 0x7fa14003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:14.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(257..257 src has 199..257) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa14c04c5f0 con 0x7fa15412f460 2024-09-15T03:38:14.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa1588f9640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa1400386f0 0x7fa14003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:14.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.966+0000 7fa1588f9640 10 monclient: get_auth_request con 0x7fa1400386f0 auth_method 0 2024-09-15T03:38:14.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.969+0000 7fa1588f9640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa1400386f0 0x7fa14003aba0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa154068eb0 tx=0x7fa150032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:14.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.971+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa14c01dae0 con 0x7fa15412f460 2024-09-15T03:38:14.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.971+0000 7fa148ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:14.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:14.971+0000 7fa148ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:15.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.134+0000 7fa15ab5d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:15.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.134+0000 7fa15ab5d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:15.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.134+0000 7fa15ab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:15.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.134+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"} v 0) v1 -- 0x7fa154064b20 con 0x7fa15412f460 2024-09-15T03:38:15.380 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:38:15.380 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:38:15.380 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:38:15.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:38:15.383 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:38:15.383 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:38:15.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:38:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.410+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}]=0 v258) v1 ==== 106+0+3 (secure 0 0 0) 0x7fa14c01b0b0 con 0x7fa15412f460 2024-09-15T03:38:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.410+0000 7fa148ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.410+0000 7fa148ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:15.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.488+0000 7efe2dd1d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:15.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.488+0000 7efe2dd1d640 10 monclient: build_initial_monmap 2024-09-15T03:38:15.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.488+0000 7efe2dd1d640 10 monclient: monmap: 2024-09-15T03:38:15.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:15.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:15.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:15.489346+0000 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:15.489346+0000 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:15.490 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 Processor -- start 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 -- start start 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient: init 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): picked mon.noname-a con 0x7efe28131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe2809e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 0x7efe28138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): picked mon.noname-b con 0x7efe28136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2809faf0 con 0x7efe2809e910 2024-09-15T03:38:15.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2809fd30 con 0x7efe28131ab0 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe280a1110 con 0x7efe28136bb0 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.489+0000 7efe2dd1d640 10 monclient(hunting): authenticate will time out at 1293.128944s 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55806/0 (socket says 172.21.15.153:55806) 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe26ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7efe2809e910 0x7efe2809ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40508/0 (socket says 172.21.15.153:40508) 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 -- 172.21.15.153:0/539416819 learned_addr learned my addr 172.21.15.153:0/539416819 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 0x7efe28138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 10 monclient(hunting): get_auth_request con 0x7efe28131ab0 auth_method 0 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:15.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): get_auth_request con 0x7efe28136bb0 auth_method 0 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 -- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 msgr2=0x7efe28131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 -- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 msgr2=0x7efe28131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 --2- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:15.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe277fe640 1 --2- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient(hunting): handle_auth_done global_id 124235 payload 274 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 1 -- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 msgr2=0x7efe28131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 1 --2- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 1 -- 172.21.15.153:0/539416819 >> v1:172.21.15.153:6789/0 conn(0x7efe2809e910 legacy=0x7efe2809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient: _finish_hunting 0 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 1 monclient: found mon.noname-b 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:15.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.490+0000 7efe27fff640 1 -- 172.21.15.153:0/539416819 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe280a13f0 con 0x7efe28136bb0 2024-09-15T03:38:15.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe27fff640 1 -- 172.21.15.153:0/539416819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 msgr2=0x7efe28138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:15.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe27fff640 1 --2- 172.21.15.153:0/539416819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 0x7efe28138f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efe20009e60 tx=0x7efe20009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d4fcb3cee12af1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:15.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 1 -- 172.21.15.153:0/539416819 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20007760 con 0x7efe28136bb0 2024-09-15T03:38:15.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:15.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: dump: 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:15.504 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 1 -- 172.21.15.153:0/539416819 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2002d050 con 0x7efe28136bb0 2024-09-15T03:38:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe2dd1d640 5 monclient: authenticate success, global_id 124235 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe2dd1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe2dd1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 1 -- 172.21.15.153:0/539416819 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20007af0 con 0x7efe28136bb0 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe2dd1d640 4 set_mon_vals no callback set 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.493+0000 7efe267fc640 10 monclient: dump: 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:15.506 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 10 monclient: shutdown 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 -- 172.21.15.153:0/539416819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 msgr2=0x7efe28138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/539416819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 0x7efe28138f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efe20009e60 tx=0x7efe20009780 comp rx=0 tx=0).stop 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 -- 172.21.15.153:0/539416819 shutdown_connections 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/539416819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe28136bb0 0x7efe28138f90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/539416819 >> v2:172.21.15.153:3300/0 conn(0x7efe28131ab0 0x7efe28131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 -- 172.21.15.153:0/539416819 >> 172.21.15.153:0/539416819 conn(0x7efe28067d30 msgr2=0x7efe280a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:15.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 -- 172.21.15.153:0/539416819 shutdown_connections 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 1 -- 172.21.15.153:0/539416819 wait complete. 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 10 monclient: build_initial_monmap 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.494+0000 7efe2dd1d640 10 monclient: monmap: 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:15.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 Processor -- start 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 -- start start 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient: init 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 0x7efe281360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): picked mon.noname-c con 0x7efe2809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 0x7efe2812f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): picked mon.noname-a con 0x7efe28131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe28136bb0 0x7efe2812f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): picked mon.noname-b con 0x7efe28136bb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28133eb0 con 0x7efe28131ab0 2024-09-15T03:38:15.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2809bde0 con 0x7efe28136bb0 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2809da90 con 0x7efe2809e910 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe2dd1d640 10 monclient(hunting): authenticate will time out at 1293.134679s 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 0x7efe281360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 0x7efe281360b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56878/0 (socket says 172.21.15.153:56878) 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 1 -- 172.21.15.153:0/3020984739 learned_addr learned my addr 172.21.15.153:0/3020984739 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): get_auth_request con 0x7efe2809e910 auth_method 0 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:15.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 0x7efe2812f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 10 monclient(hunting): get_auth_request con 0x7efe28131ab0 auth_method 0 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe28136bb0 0x7efe2812f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 10 monclient(hunting): get_auth_request con 0x7efe28136bb0 auth_method 0 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 msgr2=0x7efe2812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:38:15.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 msgr2=0x7efe2812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 0x7efe2812f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe26ffd640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 0x7efe2812f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.495+0000 7efe27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 10 monclient(hunting): handle_auth_done global_id 124238 payload 824 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe28136bb0 msgr2=0x7efe2812f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe28136bb0 0x7efe2812f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 msgr2=0x7efe2812f0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 0x7efe2812f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 10 monclient: _finish_hunting 0 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 1 monclient: found mon.noname-c 2024-09-15T03:38:15.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.496+0000 7efe277fe640 1 -- 172.21.15.153:0/3020984739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20009a20 con 0x7efe2809e910 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe277fe640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 msgr2=0x7efe281360b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe277fe640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 0x7efe281360b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00e870 tx=0x7efe1c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c00c6d0 con 0x7efe2809e910 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: dump: 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00c830 con 0x7efe2809e910 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 5 monclient: authenticate success, global_id 124238 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c00cbf0 con 0x7efe2809e910 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 10 monclient: _renew_subs 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:15.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe281356a0 con 0x7efe2809e910 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe0f7fe640 10 monclient: dump: 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:15.515 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe25ffb640 4 set_mon_vals no callback set 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 10 monclient: _renew_subs 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.498+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281308f0 con 0x7efe2809e910 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.500+0000 7efe2dd1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.500+0000 7efe2dd1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.500+0000 7efe2dd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.500+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2804bf60 con 0x7efe2809e910 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.500+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efe1c013410 con 0x7efe2809e910 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.501+0000 7efe0f7fe640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe14038740 0x7efe1403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.501+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efe1c04c560 con 0x7efe2809e910 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.501+0000 7efe26ffd640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe14038740 0x7efe1403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:15.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.501+0000 7efe26ffd640 10 monclient: get_auth_request con 0x7efe14038740 auth_method 0 2024-09-15T03:38:15.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.503+0000 7efe26ffd640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe14038740 0x7efe1403abf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efe18007c50 tx=0x7efe18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:15.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.504+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efe1c023300 con 0x7efe2809e910 2024-09-15T03:38:15.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.504+0000 7efe0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:15.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.504+0000 7efe0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:15.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa15ab5d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:15.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa15ab5d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:15.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa15ab5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:15.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"} v 0) v1 -- 0x7fa1541350c0 con 0x7fa15412f460 2024-09-15T03:38:15.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa148ff9640 1 -- 172.21.15.181:0/488119143 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}]=0 v258) v1 ==== 106+0+3 (secure 0 0 0) 0x7fa14c01d300 con 0x7fa15412f460 2024-09-15T03:38:15.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa148ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "ecab9f08-f922-45d9-8393-9f53fccdb802"}] 2024-09-15T03:38:15.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.516+0000 7fa148ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:15.520 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa1400386f0 msgr2=0x7fa14003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa1400386f0 0x7fa14003aba0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa154068eb0 tx=0x7fa150032000 comp rx=0 tx=0).stop 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 10 monclient: shutdown 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 msgr2=0x7fa154134590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 0x7fa154134590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00bbb0 tx=0x7fa14c0097b0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 shutdown_connections 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa1400386f0 0x7fa14003aba0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa15412f460 0x7fa154134590 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa15412e940 0x7fa154134050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 --2- 172.21.15.181:0/488119143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa15409cb20 0x7fa154133b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 >> 172.21.15.181:0/488119143 conn(0x7fa1540714d0 msgr2=0x7fa15409d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.518+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 shutdown_connections 2024-09-15T03:38:15.521 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:15.519+0000 7fa15ab5d640 1 -- 172.21.15.181:0/488119143 wait complete. 2024-09-15T03:38:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:+ na2=12 2024-09-15T03:38:15.529 INFO:tasks.workunit.client.0.smithi181.stdout:OK 2024-09-15T03:38:15.530 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo OK 2024-09-15T03:38:15.530 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:38:15.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:38:15.594 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-15T03:38:15.596 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-15T03:38:15.596 DEBUG:teuthology.orchestra.run.smithi181:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-15T03:38:15.655 INFO:tasks.workunit.client.0.smithi181.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-15T03:38:15.655 INFO:tasks.workunit.client.0.smithi181.stderr:+ exit_on_error=1 2024-09-15T03:38:15.655 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ ! -z '' ]] 2024-09-15T03:38:15.655 INFO:tasks.workunit.client.0.smithi181.stderr:++ uname 2024-09-15T03:38:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' Linux = FreeBSD ']' 2024-09-15T03:38:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:+ ETIMEDOUT=110 2024-09-15T03:38:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-15T03:38:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-15T03:38:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=0 2024-09-15T03:38:15.656 INFO:tasks.workunit.client.0.smithi181.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-15T03:38:15.657 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-15T03:38:15.657 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-15T03:38:15.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe2dd1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:38:15.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe2dd1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:15.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe2dd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:15.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe28130fe0 con 0x7efe2809e910 2024-09-15T03:38:15.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe0f7fe640 1 -- 172.21.15.153:0/3020984739 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7efe1c024350 con 0x7efe2809e910 2024-09-15T03:38:15.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:15.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.718+0000 7efe0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:15.720 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.720+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe14038740 msgr2=0x7efe1403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.720+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe14038740 0x7efe1403abf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efe18007c50 tx=0x7efe18009040 comp rx=0 tx=0).stop 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.720+0000 7efe2dd1d640 10 monclient: shutdown 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.720+0000 7efe2dd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.720+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 msgr2=0x7efe281360b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.720+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 0x7efe281360b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00e870 tx=0x7efe1c008040 comp rx=0 tx=0).stop 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 shutdown_connections 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe14038740 0x7efe1403abf0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe28136bb0 0x7efe2812f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe28131ab0 0x7efe2812f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 --2- 172.21.15.153:0/3020984739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe2809e910 0x7efe281360b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 >> 172.21.15.153:0/3020984739 conn(0x7efe28067d30 msgr2=0x7efe2812cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:15.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 shutdown_connections 2024-09-15T03:38:15.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:15.721+0000 7efe2dd1d640 1 -- 172.21.15.153:0/3020984739 wait complete. 2024-09-15T03:38:16.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2024-09-15T03:38:16.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T03:38:16.135 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-15T03:38:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-15T03:38:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:38:16.136 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-15T03:38:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-15T03:38:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-15T03:38:16.212 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:38:16.212 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:38:16.212 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:16.212 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph auth del client.bazar 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.283+0000 7f46d995d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.283+0000 7f46d995d640 10 monclient: build_initial_monmap 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.283+0000 7f46d995d640 10 monclient: monmap: 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:38:16.285040+0000 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:38:16.285040+0000 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:16.285 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:16.286 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:16.286 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:16.286 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:16.286 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:16.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.283+0000 7f46d995d640 1 Processor -- start 2024-09-15T03:38:16.288 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.283+0000 7f46d995d640 1 -- start start 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.283+0000 7f46d995d640 10 monclient: init 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f46d4131b40 0x7f46d409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): picked mon.noname-a con 0x7f46d4131b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46d409f240 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f46d4136e70 0x7f46d4139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): picked mon.noname-c con 0x7f46d4136e70 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40a0300 con 0x7f46d409f240 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d40a00c0 con 0x7f46d4131b40 2024-09-15T03:38:16.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d409bee0 con 0x7f46d4136e70 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d995d640 10 monclient(hunting): authenticate will time out at 1300.766331s 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f46d4136e70 0x7f46d4139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f46d4136e70 0x7f46d4139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36338/0 (socket says 172.21.15.181:36338) 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 -- 172.21.15.181:0/1684731078 learned_addr learned my addr 172.21.15.181:0/1684731078 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): get_auth_request con 0x7f46d4136e70 auth_method 0 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d2ffd640 1 --2- 172.21.15.181:0/1684731078 >> v2:172.21.15.153:3300/0 conn(0x7f46d4131b40 0x7f46d409ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient(hunting): handle_auth_done global_id 104806 payload 274 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 -- 172.21.15.181:0/1684731078 >> v2:172.21.15.153:3300/0 conn(0x7f46d4131b40 msgr2=0x7f46d409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 --2- 172.21.15.181:0/1684731078 >> v2:172.21.15.153:3300/0 conn(0x7f46d4131b40 0x7f46d409ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 -- 172.21.15.181:0/1684731078 >> v1:172.21.15.153:6789/0 conn(0x7f46d409f240 legacy=0x7f46d4131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 monclient: found mon.noname-c 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d37fe640 1 -- 172.21.15.181:0/1684731078 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d40a14b0 con 0x7f46d4136e70 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d2ffd640 10 monclient: get_auth_request con 0x7f46d4131b40 auth_method 0 2024-09-15T03:38:16.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.284+0000 7f46d2ffd640 1 --2- 172.21.15.181:0/1684731078 >> v2:172.21.15.153:3300/0 conn(0x7f46d4131b40 0x7f46d409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:16.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d37fe640 1 -- 172.21.15.181:0/1684731078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d4136e70 msgr2=0x7f46d4139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:16.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d37fe640 1 --2- 172.21.15.181:0/1684731078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d4136e70 0x7f46d4139250 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f46d406ab50 tx=0x7f46c40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8529fa04f9152895 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:16.297 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 1 -- 172.21.15.181:0/1684731078 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4032440 con 0x7f46d4136e70 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: dump: 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:16.298 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 1 -- 172.21.15.181:0/1684731078 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c40325a0 con 0x7f46d4136e70 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d995d640 5 monclient: authenticate success, global_id 104806 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d995d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 1 -- 172.21.15.181:0/1684731078 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4032940 con 0x7f46d4136e70 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d995d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d995d640 4 set_mon_vals no callback set 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.287+0000 7f46d1ffb640 10 monclient: dump: 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:16.299 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 10 monclient: shutdown 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 -- 172.21.15.181:0/1684731078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d4136e70 msgr2=0x7f46d4139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 --2- 172.21.15.181:0/1684731078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d4136e70 0x7f46d4139250 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f46d406ab50 tx=0x7f46c40091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 -- 172.21.15.181:0/1684731078 shutdown_connections 2024-09-15T03:38:16.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 --2- 172.21.15.181:0/1684731078 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d4136e70 0x7f46d4139250 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 --2- 172.21.15.181:0/1684731078 >> v2:172.21.15.153:3300/0 conn(0x7f46d4131b40 0x7f46d409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 -- 172.21.15.181:0/1684731078 >> 172.21.15.181:0/1684731078 conn(0x7f46d40a5100 msgr2=0x7f46d40a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 -- 172.21.15.181:0/1684731078 shutdown_connections 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 1 -- 172.21.15.181:0/1684731078 wait complete. 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 10 monclient: build_initial_monmap 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.288+0000 7f46d995d640 10 monclient: monmap: 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:16.301 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 Processor -- start 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 -- start start 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient: init 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 0x7f46d4098800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): picked mon.noname-b con 0x7f46d409f240 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46d4131b40 0x7f46d4098d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): picked mon.noname-c con 0x7f46d4131b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46d4136e70 0x7f46d4099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): picked mon.noname-a con 0x7f46d4136e70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:16.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40a0a30 con 0x7f46d4136e70 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4130c60 con 0x7f46d409f240 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d409a190 con 0x7f46d4131b40 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d995d640 10 monclient(hunting): authenticate will time out at 1300.771784s 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 0x7f46d4098800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 0x7f46d4098800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36348/0 (socket says 172.21.15.181:36348) 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 1 -- 172.21.15.181:0/3061669899 learned_addr learned my addr 172.21.15.181:0/3061669899 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): get_auth_request con 0x7f46d409f240 auth_method 0 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:16.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d37fe640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46d4136e70 0x7f46d4099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d37fe640 10 monclient(hunting): get_auth_request con 0x7f46d4136e70 auth_method 0 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.289+0000 7f46d27fc640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46d4131b40 0x7f46d4098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d27fc640 10 monclient(hunting): get_auth_request con 0x7f46d4131b40 auth_method 0 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 10 monclient(hunting): handle_auth_done global_id 104809 payload 824 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 1 -- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46d4131b40 msgr2=0x7f46d4098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46d4131b40 0x7f46d4098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 1 -- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46d4136e70 msgr2=0x7f46d4099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46d4136e70 0x7f46d4099280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.290+0000 7f46d2ffd640 1 -- 172.21.15.181:0/3061669899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4009450 con 0x7f46d409f240 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d2ffd640 1 -- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 msgr2=0x7f46d4098800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d2ffd640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 0x7f46d4098800 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00a430 tx=0x7f46cc007910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00d860 con 0x7f46d409f240 2024-09-15T03:38:16.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: dump: 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: _finish_auth 0 2024-09-15T03:38:16.306 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 5 monclient: authenticate success, global_id 104809 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc00d9c0 con 0x7f46d409f240 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 10 monclient: _renew_subs 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00dd80 con 0x7f46d409f240 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d409aae0 con 0x7f46d409f240 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46baffd640 10 monclient: dump: 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d17fa640 4 set_mon_vals no callback set 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 10 monclient: _renew_subs 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.292+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d411cbd0 con 0x7f46d409f240 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f46cc015650 con 0x7f46d409f240 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46d995d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46d995d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46d995d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d411bd40 con 0x7f46d409f240 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46baffd640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46c0038770 0x7f46c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f46cc04cf10 con 0x7f46d409f240 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46d27fc640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46c0038770 0x7f46c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.293+0000 7f46d27fc640 10 monclient: get_auth_request con 0x7f46c0038770 auth_method 0 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.295+0000 7f46d27fc640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46c0038770 0x7f46c003ac20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f46c800b850 tx=0x7f46c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.297+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f46cc008b10 con 0x7f46d409f240 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.297+0000 7f46baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.297+0000 7f46baffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:16.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.511+0000 7f46d995d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T03:38:16.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.511+0000 7f46d995d640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T03:38:16.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.511+0000 7f46d995d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.511+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7f46d40636e0 con 0x7f46d409f240 2024-09-15T03:38:16.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.515+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v17) v1 ==== 82+0+0 (secure 0 0 0) 0x7f46cc01a030 con 0x7f46d409f240 2024-09-15T03:38:16.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.515+0000 7f46baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T03:38:16.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.515+0000 7f46baffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.663+0000 7f46d995d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.664+0000 7f46d995d640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.664+0000 7f46d995d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.664+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7f46d41352a0 con 0x7f46d409f240 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.665+0000 7f46baffd640 1 -- 172.21.15.181:0/3061669899 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v17) v1 ==== 82+0+0 (secure 0 0 0) 0x7f46cc01a030 con 0x7f46d409f240 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.665+0000 7f46baffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-15T03:38:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.665+0000 7f46baffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:38:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46c0038770 msgr2=0x7f46c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46c0038770 0x7f46c003ac20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f46c800b850 tx=0x7f46c8008040 comp rx=0 tx=0).stop 2024-09-15T03:38:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 10 monclient: shutdown 2024-09-15T03:38:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 msgr2=0x7f46d4098800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 0x7f46d4098800 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00a430 tx=0x7f46cc007910 comp rx=0 tx=0).stop 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 shutdown_connections 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f46c0038770 0x7f46c003ac20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46d4136e70 0x7f46d4099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46d4131b40 0x7f46d4098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 --2- 172.21.15.181:0/3061669899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46d409f240 0x7f46d4098800 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.666+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 >> 172.21.15.181:0/3061669899 conn(0x7f46d40a5100 msgr2=0x7f46d40a3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.667+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 shutdown_connections 2024-09-15T03:38:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:38:16.667+0000 7f46d995d640 1 -- 172.21.15.181:0/3061669899 wait complete. 2024-09-15T03:38:16.677 INFO:tasks.workunit.client.0.smithi181.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-15T03:38:16.677 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-15T03:38:16.678 INFO:tasks.workunit.client.0.smithi181.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-15T03:38:16.678 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-15T03:38:16.678 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=0 2024-09-15T03:38:16.678 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-15T03:38:16.678 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-15T03:38:17.141 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2024-09-15T03:38:17.142 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T03:38:17.142 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:17.142 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-15T03:38:17.142 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T03:38:17.142 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-15T03:38:17.142 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=0 2024-09-15T03:38:17.143 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T03:38:17.143 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T03:38:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2024-09-15T03:38:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T03:38:17.604 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:17.605 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-15T03:38:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-15T03:38:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-15T03:38:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=0 2024-09-15T03:38:17.605 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-15T03:38:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-15T03:38:18.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2024-09-15T03:38:18.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T03:38:18.067 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:18.068 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-15T03:38:18.068 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-15T03:38:18.068 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-15T03:38:18.068 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:38:18.068 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-15T03:38:18.068 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-15T03:38:18.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.836+0000 7f8a06f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.836+0000 7f8a06f1d640 10 monclient: build_initial_monmap 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.836+0000 7f8a06f1d640 10 monclient: monmap: 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:18.836997+0000 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:18.836997+0000 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:18.837 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:18.838 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:18.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 Processor -- start 2024-09-15T03:38:18.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 -- start start 2024-09-15T03:38:18.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient: init 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 0x7f8a0009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f8a00096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a00097030 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8a00133b40 0x7f8a00135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8a00133b40 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000696a0 con 0x7f8a00097030 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0009ac00 con 0x7f8a00133b40 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0009d6b0 con 0x7f8a00096c60 2024-09-15T03:38:18.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a06f1d640 10 monclient(hunting): authenticate will time out at 1296.476641s 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 0x7f8a0009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f89fffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8a00097030 0x7f8a00097400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40524/0 (socket says 172.21.15.153:40524) 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f89fffff640 1 -- 172.21.15.153:0/2935245601 learned_addr learned my addr 172.21.15.153:0/2935245601 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): get_auth_request con 0x7f8a00096c60 auth_method 0 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a054ba640 1 --2- 172.21.15.153:0/2935245601 >> v2:172.21.15.181:3300/0 conn(0x7f8a00133b40 0x7f8a00135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:18.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a054ba640 10 monclient(hunting): get_auth_request con 0x7f8a00133b40 auth_method 0 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a054ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a054ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.837+0000 7f8a054ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a054ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a054ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a054ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_done global_id 124268 payload 274 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 1 -- 172.21.15.153:0/2935245601 >> v2:172.21.15.181:3300/0 conn(0x7f8a00133b40 msgr2=0x7f8a00135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 1 --2- 172.21.15.153:0/2935245601 >> v2:172.21.15.181:3300/0 conn(0x7f8a00133b40 0x7f8a00135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 1 -- 172.21.15.153:0/2935245601 >> v1:172.21.15.153:6789/0 conn(0x7f8a00097030 legacy=0x7f8a00097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 1 monclient: found mon.noname-b 2024-09-15T03:38:18.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:18.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:18.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.838+0000 7f8a04cb9640 1 -- 172.21.15.153:0/2935245601 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0009d2e0 con 0x7f8a00096c60 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.840+0000 7f8a04cb9640 1 -- 172.21.15.153:0/2935245601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 msgr2=0x7f8a0009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.840+0000 7f8a04cb9640 1 --2- 172.21.15.153:0/2935245601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 0x7f8a0009a6c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f89f800baf0 tx=0x7f89f800b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d4ee27276aebe5f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 1 -- 172.21.15.153:0/2935245601 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f80316c0 con 0x7f8a00096c60 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: dump: 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:18.849 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 5 monclient: authenticate success, global_id 124268 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 1 -- 172.21.15.153:0/2935245601 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f8031820 con 0x7f8a00096c60 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:18.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 4 set_mon_vals no callback set 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 1 -- 172.21.15.153:0/2935245601 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8031bc0 con 0x7f8a00096c60 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f89ff7fe640 10 monclient: dump: 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:18.851 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 10 monclient: shutdown 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2935245601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 msgr2=0x7f8a0009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2935245601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 0x7f8a0009a6c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f89f800baf0 tx=0x7f89f800b3b0 comp rx=0 tx=0).stop 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2935245601 shutdown_connections 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2935245601 >> v2:172.21.15.181:3300/0 conn(0x7f8a00133b40 0x7f8a00135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2935245601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00096c60 0x7f8a0009a6c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2935245601 >> 172.21.15.153:0/2935245601 conn(0x7f8a000672b0 msgr2=0x7f8a000676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2935245601 shutdown_connections 2024-09-15T03:38:18.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.841+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2935245601 wait complete. 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.842+0000 7f8a06f1d640 10 monclient: build_initial_monmap 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.842+0000 7f8a06f1d640 10 monclient: monmap: 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:18.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.842+0000 7f8a06f1d640 1 Processor -- start 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.842+0000 7f8a06f1d640 1 -- start start 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.842+0000 7f8a06f1d640 10 monclient: init 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 0x7f8a0012e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f8a00096c60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a00097030 0x7f8a00118210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f8a00097030 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 0x7f8a00118750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8a00133b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0011a240 con 0x7f8a00096c60 2024-09-15T03:38:18.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0009add0 con 0x7f8a00097030 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a00134f20 con 0x7f8a00133b40 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a06f1d640 10 monclient(hunting): authenticate will time out at 1296.482441s 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 0x7f8a0012e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 0x7f8a00118750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 0x7f8a00118750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56944/0 (socket says 172.21.15.153:56944) 2024-09-15T03:38:18.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 0x7f8a0012e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55846/0 (socket says 172.21.15.153:55846) 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 -- 172.21.15.153:0/2094814021 learned_addr learned my addr 172.21.15.153:0/2094814021 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): get_auth_request con 0x7f8a00133b40 auth_method 0 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): get_auth_request con 0x7f8a00096c60 auth_method 0 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:18.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient(hunting): handle_auth_done global_id 124271 payload 824 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f89fffff640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a00097030 0x7f8a00118210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 -- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a00097030 msgr2=0x7f8a00118210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a00097030 0x7f8a00118210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 -- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 msgr2=0x7f8a0012e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 0x7f8a0012e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient: _finish_hunting 0 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 monclient: found mon.noname-c 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a054ba640 1 -- 172.21.15.153:0/2094814021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f800b630 con 0x7f8a00133b40 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.843+0000 7f8a04cb9640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 0x7f8a0012e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a054ba640 1 -- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 msgr2=0x7f8a00118750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a054ba640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 0x7f8a00118750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f89e800e960 tx=0x7f89e80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80095c0 con 0x7f8a00133b40 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: dump: 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:18.858 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: _finish_auth 0 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e800cb50 con 0x7f8a00133b40 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 5 monclient: authenticate success, global_id 124271 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 10 monclient: _renew_subs 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0011b7f0 con 0x7f8a00133b40 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8013470 con 0x7f8a00133b40 2024-09-15T03:38:18.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89fcff9640 10 monclient: dump: 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 10 monclient: _renew_subs 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f89feffd640 4 set_mon_vals no callback set 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.846+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a00119cc0 con 0x7f8a00133b40 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f89e8013860 con 0x7f8a00133b40 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f8a06f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f8a06f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f8a06f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0004bf60 con 0x7f8a00133b40 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f89fcff9640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f89e0038770 0x7f89e003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f89e804c8c0 con 0x7f8a00133b40 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f8a04cb9640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f89e0038770 0x7f89e003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:18.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.847+0000 7f8a04cb9640 10 monclient: get_auth_request con 0x7f89e0038770 auth_method 0 2024-09-15T03:38:18.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.850+0000 7f8a04cb9640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f89e0038770 0x7f89e003ac20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f89f8036000 tx=0x7f89f8037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:18.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.852+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f89e8023ad0 con 0x7f8a00133b40 2024-09-15T03:38:18.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.852+0000 7f89fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:18.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:18.852+0000 7f89fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:19.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.068+0000 7f8a06f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.068+0000 7f8a06f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.068+0000 7f8a06f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.068+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8a00098390 con 0x7f8a00133b40 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.069+0000 7f89fcff9640 1 -- 172.21.15.153:0/2094814021 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f89e80232f0 con 0x7f8a00133b40 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.069+0000 7f89fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.069+0000 7f89fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:19.070 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":106,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:38:19.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f89e0038770 msgr2=0x7f89e003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f89e0038770 0x7f89e003ac20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f89f8036000 tx=0x7f89f8037000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 10 monclient: shutdown 2024-09-15T03:38:19.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 msgr2=0x7f8a00118750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 0x7f8a00118750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f89e800e960 tx=0x7f89e80090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 shutdown_connections 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f89e0038770 0x7f89e003ac20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a00133b40 0x7f8a00118750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a00097030 0x7f8a00118210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 --2- 172.21.15.153:0/2094814021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a00096c60 0x7f8a0012e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 >> 172.21.15.153:0/2094814021 conn(0x7f8a000672b0 msgr2=0x7f8a00133f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 shutdown_connections 2024-09-15T03:38:19.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.071+0000 7f8a06f1d640 1 -- 172.21.15.153:0/2094814021 wait complete. 2024-09-15T03:38:19.082 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:38:19.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:38:19.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:38:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.176+0000 7f3f1382b640 10 monclient: get_monmap_and_config 2024-09-15T03:38:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.176+0000 7f3f1382b640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.176+0000 7f3f1382b640 10 monclient: monmap: 2024-09-15T03:38:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.177 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:19.177038+0000 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:19.177038+0000 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.176+0000 7f3f1382b640 1 Processor -- start 2024-09-15T03:38:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 1 -- start start 2024-09-15T03:38:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient: init 2024-09-15T03:38:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T03:38:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f0c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:19.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3f0c09ea50 0x7f3f0c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): picked mon.noname-c con 0x7f3f0c09ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c136c20 0x7f3f0c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): picked mon.noname-b con 0x7f3f0c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c132640 con 0x7f3f0c1318f0 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c131cc0 con 0x7f3f0c09ea50 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c09fbf0 con 0x7f3f0c136c20 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f1382b640 10 monclient(hunting): authenticate will time out at 1296.816672s 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c136c20 0x7f3f0c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3f0c09ea50 0x7f3f0c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3f0c09ea50 0x7f3f0c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:39800/0 (socket says 172.21.15.153:39800) 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 1 -- 172.21.15.153:0/871420040 learned_addr learned my addr 172.21.15.153:0/871420040 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): get_auth_request con 0x7f3f0c136c20 auth_method 0 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): get_auth_request con 0x7f3f0c09ea50 auth_method 0 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f115c7640 1 --1- 172.21.15.153:0/871420040 >> v1:172.21.15.181:6789/0 conn(0x7f3f0c1318f0 0x7f3f0c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:59170/0 (socket says 172.21.15.153:59170) 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.177+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_done global_id 104833 payload 274 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 1 -- 172.21.15.153:0/871420040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c136c20 msgr2=0x7f3f0c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 1 --2- 172.21.15.153:0/871420040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c136c20 0x7f3f0c139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 1 -- 172.21.15.153:0/871420040 >> v1:172.21.15.181:6789/0 conn(0x7f3f0c1318f0 legacy=0x7f3f0c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 1 monclient: found mon.noname-c 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f10dc6640 1 -- 172.21.15.153:0/871420040 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c09fe30 con 0x7f3f0c09ea50 2024-09-15T03:38:19.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.178+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/871420040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c136c20 0x7f3f0c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:19.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.180+0000 7f3f10dc6640 1 -- 172.21.15.153:0/871420040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c09ea50 msgr2=0x7f3f0c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.180+0000 7f3f10dc6640 1 --2- 172.21.15.153:0/871420040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c09ee20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009910 tx=0x7f3f000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8e8a77780f6607f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 1 -- 172.21.15.153:0/871420040 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f00032440 con 0x7f3f0c09ea50 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: dump: 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.190 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 1 -- 172.21.15.153:0/871420040 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f000325a0 con 0x7f3f0c09ea50 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 5 monclient: authenticate success, global_id 104833 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 4 set_mon_vals no callback set 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 1 -- 172.21.15.153:0/871420040 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f00032940 con 0x7f3f0c09ea50 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f0bfff640 10 monclient: dump: 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 10 monclient: shutdown 2024-09-15T03:38:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 1 -- 172.21.15.153:0/871420040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c09ea50 msgr2=0x7f3f0c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.181+0000 7f3f1382b640 1 --2- 172.21.15.153:0/871420040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c09ee20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009910 tx=0x7f3f000091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 -- 172.21.15.153:0/871420040 shutdown_connections 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 --2- 172.21.15.153:0/871420040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c136c20 0x7f3f0c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 --2- 172.21.15.153:0/871420040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c09ee20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 -- 172.21.15.153:0/871420040 >> 172.21.15.153:0/871420040 conn(0x7f3f0c067110 msgr2=0x7f3f0c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 -- 172.21.15.153:0/871420040 shutdown_connections 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 -- 172.21.15.153:0/871420040 wait complete. 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 10 monclient: monmap: 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:19.193 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.182+0000 7f3f1382b640 1 Processor -- start 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 -- start start 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient: init 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c1318f0 0x7f3f0c130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): picked mon.noname-c con 0x7f3f0c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c136c20 0x7f3f0c099780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): picked mon.noname-b con 0x7f3f0c136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c11ca50 con 0x7f3f0c09ea50 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c11d080 con 0x7f3f0c136c20 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c11b990 con 0x7f3f0c1318f0 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f1382b640 10 monclient(hunting): authenticate will time out at 1296.822678s 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c1318f0 0x7f3f0c130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c1318f0 0x7f3f0c130e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56968/0 (socket says 172.21.15.153:56968) 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c130940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55848/0 (socket says 172.21.15.153:55848) 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 1 -- 172.21.15.153:0/1626351455 learned_addr learned my addr 172.21.15.153:0/1626351455 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): get_auth_request con 0x7f3f0c1318f0 auth_method 0 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): get_auth_request con 0x7f3f0c09ea50 auth_method 0 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f115c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.183+0000 7f3f10dc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c136c20 0x7f3f0c099780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 10 monclient(hunting): handle_auth_done global_id 134115 payload 824 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c1318f0 msgr2=0x7f3f0c130e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c1318f0 0x7f3f0c130e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c136c20 msgr2=0x7f3f0c099780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c136c20 0x7f3f0c099780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 1 monclient: found mon.noname-a 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f115c7640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc009a20 con 0x7f3f0c09ea50 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f11dc8640 10 monclient: get_auth_request con 0x7f3f0c136c20 auth_method 0 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f10dc6640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f0c1318f0 0x7f3f0c130e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.184+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f0c136c20 0x7f3f0c099780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f115c7640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 msgr2=0x7f3f0c130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f115c7640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c130940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c4c0 tx=0x7f3f04007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040074f0 con 0x7f3f0c09ea50 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: dump: 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.198 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 5 monclient: authenticate success, global_id 134115 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04007670 con 0x7f3f0c09ea50 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 10 monclient: _renew_subs 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f00009450 con 0x7f3f0c09ea50 2024-09-15T03:38:19.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04005a20 con 0x7f3f0c09ea50 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f097fa640 10 monclient: dump: 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.200 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 10 monclient: _renew_subs 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c09b2c0 con 0x7f3f0c09ea50 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.186+0000 7f3f0b7fe640 4 set_mon_vals no callback set 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3f0401c440 con 0x7f3f0c09ea50 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f097fa640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ef8038740 0x7f3ef803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f1382b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3f0404ba90 con 0x7f3f0c09ea50 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f1382b640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 0x7f3f0c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c065e00 con 0x7f3f0c09a480 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 0x7f3f0c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient: get_auth_request con 0x7f3f0c09a480 auth_method 0 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f10dc6640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ef8038740 0x7f3ef803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_done global_id 134118 payload 824 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f11dc8640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f0c04bf60 con 0x7f3f0c09a480 2024-09-15T03:38:19.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.187+0000 7f3f10dc6640 10 monclient: get_auth_request con 0x7f3ef8038740 auth_method 0 2024-09-15T03:38:19.203 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:38:19.203 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:38:19.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:19.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-15T03:38:19.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-15T03:38:19.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:38:19.204 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-15T03:38:19.204 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-15T03:38:19.205 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f11dc8640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 msgr2=0x7f3f0c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 0x7f3f0c0298c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00f380 tx=0x7f3efc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc0148d0 con 0x7f3f0c09a480 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 10 monclient: dump: 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.207 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3efc014a30 con 0x7f3f0c09a480 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 msgr2=0x7f3f0c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 0x7f3f0c0298c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00f380 tx=0x7f3efc00b040 comp rx=0 tx=0).stop 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 msgr2=0x7f3f0c0298c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f097fa640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a480 0x7f3f0c0298c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.190+0000 7f3f10dc6640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ef8038740 0x7f3ef803abf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009210 tx=0x7f3f00032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f1382b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f1382b640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 0x7f3f0c029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f1382b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c135eb0 con 0x7f3f0c09a850 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 0x7f3f0c029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient: get_auth_request con 0x7f3f0c09a850 auth_method 0 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 10 monclient(hunting): handle_auth_done global_id 134121 payload 824 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.196+0000 7f3f11dc8640 1 -- 172.21.15.153:0/1626351455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3f0c0638d0 con 0x7f3f0c09a850 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f11dc8640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 msgr2=0x7f3f0c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f11dc8640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 0x7f3f0c029e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fa0 tx=0x7f3efc01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc016c40 con 0x7f3f0c09a850 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 10 monclient: dump: 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.210 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f3efc0146c0 con 0x7f3f0c09a850 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 msgr2=0x7f3f0c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 0x7f3f0c029e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fa0 tx=0x7f3efc01b000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 msgr2=0x7f3f0c029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.198+0000 7f3f097fa640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09a850 0x7f3f0c029e00 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009fa0 tx=0x7f3efc01b000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.211 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 106, 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:19.212 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.213 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:19.214 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:19.215 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.216 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:19.220 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:19.221 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:19.222 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.224 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.224 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.225 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:19.226 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ef8038740 msgr2=0x7f3ef803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ef8038740 0x7f3ef803abf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009210 tx=0x7f3f00032000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 10 monclient: shutdown 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 msgr2=0x7f3f0c130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f11dc8640 1 -- 172.21.15.153:0/1626351455 reap_dead start 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c130940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c4c0 tx=0x7f3f04007e40 comp rx=0 tx=0).stop 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 shutdown_connections 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 --2- 172.21.15.153:0/1626351455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0c09ea50 0x7f3f0c130940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 >> 172.21.15.153:0/1626351455 conn(0x7f3f0c067110 msgr2=0x7f3f0c0a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 shutdown_connections 2024-09-15T03:38:19.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.199+0000 7f3f1382b640 1 -- 172.21.15.153:0/1626351455 wait complete. 2024-09-15T03:38:19.229 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:38:19.229 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:38:19.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:38:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.322+0000 7fe16fa4f640 10 monclient: get_monmap_and_config 2024-09-15T03:38:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.322+0000 7fe16fa4f640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.322+0000 7fe16fa4f640 10 monclient: monmap: 2024-09-15T03:38:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:19.323415+0000 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:19.323415+0000 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:19.324 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 1 Processor -- start 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 1 -- start start 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 10 monclient: init 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1680975f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.323+0000 7fe16fa4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 10 monclient(hunting): picked mon.noname-b con 0x7fe1680979c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 0x7fe168136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 10 monclient(hunting): picked mon.noname-a con 0x7fe1681349f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16809d700 con 0x7fe1680975f0 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16809d1b0 con 0x7fe1681349f0 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680961e0 con 0x7fe1680979c0 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16fa4f640 10 monclient(hunting): authenticate will time out at 1296.963069s 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 0x7fe168136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168097dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56970/0 (socket says 172.21.15.153:56970) 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 -- 172.21.15.153:0/87441843 learned_addr learned my addr 172.21.15.153:0/87441843 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): get_auth_request con 0x7fe1680979c0 auth_method 0 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 1 --2- 172.21.15.153:0/87441843 >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 0x7fe168136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55874/0 (socket says 172.21.15.153:55874) 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): get_auth_request con 0x7fe1681349f0 auth_method 0 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient(hunting): handle_auth_done global_id 124286 payload 274 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 -- 172.21.15.153:0/87441843 >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 msgr2=0x7fe168136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 --2- 172.21.15.153:0/87441843 >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 0x7fe168136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 -- 172.21.15.153:0/87441843 >> v1:172.21.15.181:6789/0 conn(0x7fe1680975f0 legacy=0x7fe168095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 monclient: found mon.noname-b 2024-09-15T03:38:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16cfea640 1 -- 172.21.15.153:0/87441843 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16809d420 con 0x7fe1680979c0 2024-09-15T03:38:19.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.324+0000 7fe16dfec640 1 --2- 172.21.15.153:0/87441843 >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 0x7fe168136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16cfea640 1 -- 172.21.15.153:0/87441843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 msgr2=0x7fe168097dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16cfea640 1 --2- 172.21.15.153:0/87441843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168097dc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe158009910 tx=0x7fe1580091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835536389162e817 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 1 -- 172.21.15.153:0/87441843 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15802aea0 con 0x7fe1680979c0 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: dump: 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 1 -- 172.21.15.153:0/87441843 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe158032440 con 0x7fe1680979c0 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 1 -- 172.21.15.153:0/87441843 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580327e0 con 0x7fe1680979c0 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 5 monclient: authenticate success, global_id 124286 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:19.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 4 set_mon_vals no callback set 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe15ffff640 10 monclient: dump: 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 10 monclient: shutdown 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 1 -- 172.21.15.153:0/87441843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 msgr2=0x7fe168097dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.327+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/87441843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168097dc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe158009910 tx=0x7fe1580091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 -- 172.21.15.153:0/87441843 shutdown_connections 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/87441843 >> v2:172.21.15.153:3300/0 conn(0x7fe1681349f0 0x7fe168136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/87441843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168097dc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 -- 172.21.15.153:0/87441843 >> 172.21.15.153:0/87441843 conn(0x7fe1680678a0 msgr2=0x7fe1680a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 -- 172.21.15.153:0/87441843 shutdown_connections 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 -- 172.21.15.153:0/87441843 wait complete. 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 10 monclient: monmap: 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 Processor -- start 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 1 -- start start 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.328+0000 7fe16fa4f640 10 monclient: init 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1680975f0 0x7fe168088fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): picked mon.noname-b con 0x7fe1680975f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168089500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): picked mon.noname-c con 0x7fe1680979c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 0x7fe168089a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): picked mon.noname-a con 0x7fe1681349f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680983c0 con 0x7fe1681349f0 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16809f430 con 0x7fe1680975f0 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16808c3e0 con 0x7fe1680979c0 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16fa4f640 10 monclient(hunting): authenticate will time out at 1296.968635s 2024-09-15T03:38:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168089500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 0x7fe168089a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 0x7fe168089a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55880/0 (socket says 172.21.15.153:55880) 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168089500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56984/0 (socket says 172.21.15.153:56984) 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 1 -- 172.21.15.153:0/3428610962 learned_addr learned my addr 172.21.15.153:0/3428610962 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): get_auth_request con 0x7fe1681349f0 auth_method 0 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): get_auth_request con 0x7fe1680979c0 auth_method 0 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1680975f0 0x7fe168088fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): get_auth_request con 0x7fe1680975f0 auth_method 0 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16cfea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.329+0000 7fe16dfec640 10 monclient(hunting): handle_auth_done global_id 134130 payload 824 2024-09-15T03:38:19.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 msgr2=0x7fe168089500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168089500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1680975f0 msgr2=0x7fe168088fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1680975f0 0x7fe168088fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 1 monclient: found mon.noname-a 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16dfec640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160009a20 con 0x7fe1681349f0 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.330+0000 7fe16cfea640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1680979c0 0x7fe168089500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16dfec640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 msgr2=0x7fe168089a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16dfec640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 0x7fe168089a40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe16000ef90 tx=0x7fe16000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16000ce70 con 0x7fe1681349f0 2024-09-15T03:38:19.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: dump: 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16000f040 con 0x7fe1681349f0 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 5 monclient: authenticate success, global_id 134130 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 10 monclient: _renew_subs 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1600148e0 con 0x7fe1681349f0 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe158009450 con 0x7fe1681349f0 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15d7fa640 10 monclient: dump: 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.346 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe15f7fe640 4 set_mon_vals no callback set 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 10 monclient: _renew_subs 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.332+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16808a610 con 0x7fe1681349f0 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16fa4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 0x7fe16806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe160014d10 con 0x7fe1681349f0 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16808c2a0 con 0x7fe168089f80 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 0x7fe16806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient: get_auth_request con 0x7fe168089f80 auth_method 0 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe15d7fa640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c038740 0x7fe14c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe16004d740 con 0x7fe1681349f0 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_done global_id 124292 payload 824 2024-09-15T03:38:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16d7eb640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe16808c620 con 0x7fe168089f80 2024-09-15T03:38:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16cfea640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c038740 0x7fe14c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.333+0000 7fe16cfea640 10 monclient: get_auth_request con 0x7fe14c038740 auth_method 0 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.335+0000 7fe16d7eb640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 msgr2=0x7fe16806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.335+0000 7fe16d7eb640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 0x7fe16806c7e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe16400f780 tx=0x7fe164007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16400d8b0 con 0x7fe168089f80 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 10 monclient: dump: 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.351 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe16400da10 con 0x7fe168089f80 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 msgr2=0x7fe16806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 0x7fe16806c7e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe16400f780 tx=0x7fe164007e10 comp rx=0 tx=0).stop 2024-09-15T03:38:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 msgr2=0x7fe16806c7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe15d7fa640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe168089f80 0x7fe16806c7e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.336+0000 7fe16cfea640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c038740 0x7fe14c03abf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe16806a3f0 tx=0x7fe158032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.341+0000 7fe16fa4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.341+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 0x7fe16806d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.341+0000 7fe16fa4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.341+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe168065e00 con 0x7fe16806cd20 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 0x7fe16806d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient: get_auth_request con 0x7fe16806cd20 auth_method 0 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 10 monclient(hunting): handle_auth_done global_id 124295 payload 824 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.342+0000 7fe16d7eb640 1 -- 172.21.15.153:0/3428610962 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe16804bf60 con 0x7fe16806cd20 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe16d7eb640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 msgr2=0x7fe16806d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe16d7eb640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 0x7fe16806d120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe164003420 tx=0x7fe164019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164007450 con 0x7fe16806cd20 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 10 monclient: dump: 2024-09-15T03:38:19.354 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fe16400fd10 con 0x7fe16806cd20 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:19.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 msgr2=0x7fe16806d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 0x7fe16806d120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe164003420 tx=0x7fe164019000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 msgr2=0x7fe16806d120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.344+0000 7fe15d7fa640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe16806cd20 0x7fe16806d120 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe164003420 tx=0x7fe164019000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 106, 2024-09-15T03:38:19.356 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:19.358 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:19.359 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:19.360 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.361 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.364 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.365 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:19.366 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.367 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.368 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.369 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:19.369 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c038740 msgr2=0x7fe14c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe14c038740 0x7fe14c03abf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe16806a3f0 tx=0x7fe158032000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 10 monclient: shutdown 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 msgr2=0x7fe168089a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 0x7fe168089a40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe16000ef90 tx=0x7fe16000b040 comp rx=0 tx=0).stop 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16dfec640 1 -- 172.21.15.153:0/3428610962 reap_dead start 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 shutdown_connections 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 --2- 172.21.15.153:0/3428610962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1681349f0 0x7fe168089a40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 >> 172.21.15.153:0/3428610962 conn(0x7fe1680678a0 msgr2=0x7fe168135080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 shutdown_connections 2024-09-15T03:38:19.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.345+0000 7fe16fa4f640 1 -- 172.21.15.153:0/3428610962 wait complete. 2024-09-15T03:38:19.371 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:38:19.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:38:19.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:38:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.464+0000 7efd5a91d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.464+0000 7efd5a91d640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.464+0000 7efd5a91d640 10 monclient: monmap: 2024-09-15T03:38:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.465 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:19.464831+0000 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:19.464831+0000 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.464+0000 7efd5a91d640 1 Processor -- start 2024-09-15T03:38:19.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.464+0000 7efd5a91d640 1 -- start start 2024-09-15T03:38:19.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.464+0000 7efd5a91d640 10 monclient: init 2024-09-15T03:38:19.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:38:19.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efd541318f0 0x7efd54131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient(hunting): picked mon.noname-a con 0x7efd541318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd5409ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd54136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5409f530 con 0x7efd54136c20 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5409fbf0 con 0x7efd5409ea50 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd54131d50 con 0x7efd541318f0 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd5a91d640 10 monclient(hunting): authenticate will time out at 1297.104433s 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efd541318f0 0x7efd54131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd58eba640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7efd54136c20 0x7efd54139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40534/0 (socket says 172.21.15.153:40534) 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd58eba640 1 -- 172.21.15.153:0/1870154588 learned_addr learned my addr 172.21.15.153:0/1870154588 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): get_auth_request con 0x7efd541318f0 auth_method 0 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient(hunting): handle_auth_done global_id 134133 payload 274 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 1 -- 172.21.15.153:0/1870154588 >> v1:172.21.15.181:6789/0 conn(0x7efd5409ea50 legacy=0x7efd5409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 1 -- 172.21.15.153:0/1870154588 >> v1:172.21.15.153:6789/0 conn(0x7efd54136c20 legacy=0x7efd54139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 1 monclient: found mon.noname-a 2024-09-15T03:38:19.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.465+0000 7efd53fff640 1 -- 172.21.15.153:0/1870154588 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5409fe30 con 0x7efd541318f0 2024-09-15T03:38:19.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.468+0000 7efd53fff640 1 -- 172.21.15.153:0/1870154588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd541318f0 msgr2=0x7efd54131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.468+0000 7efd53fff640 1 --2- 172.21.15.153:0/1870154588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd541318f0 0x7efd54131f30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd44009850 tx=0x7efd440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b15c997bab733 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.468+0000 7efd52ffd640 1 -- 172.21.15.153:0/1870154588 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44032440 con 0x7efd541318f0 2024-09-15T03:38:19.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:19.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: dump: 2024-09-15T03:38:19.479 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.481 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.481 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.481 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 1 -- 172.21.15.153:0/1870154588 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd440325a0 con 0x7efd541318f0 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 5 monclient: authenticate success, global_id 134133 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 1 -- 172.21.15.153:0/1870154588 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44032940 con 0x7efd541318f0 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 4 set_mon_vals no callback set 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd52ffd640 10 monclient: dump: 2024-09-15T03:38:19.483 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.483 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.483 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.483 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.483 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.484 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.484 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.484 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.484 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.484 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 10 monclient: shutdown 2024-09-15T03:38:19.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 -- 172.21.15.153:0/1870154588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd541318f0 msgr2=0x7efd54131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 --2- 172.21.15.153:0/1870154588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd541318f0 0x7efd54131f30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd44009850 tx=0x7efd440090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 -- 172.21.15.153:0/1870154588 shutdown_connections 2024-09-15T03:38:19.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 --2- 172.21.15.153:0/1870154588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd541318f0 0x7efd54131f30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 -- 172.21.15.153:0/1870154588 >> 172.21.15.153:0/1870154588 conn(0x7efd54067110 msgr2=0x7efd54067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 -- 172.21.15.153:0/1870154588 shutdown_connections 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.469+0000 7efd5a91d640 1 -- 172.21.15.153:0/1870154588 wait complete. 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient: monmap: 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:19.486 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 1 Processor -- start 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 1 -- start start 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient: init 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:38:19.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 0x7efd541329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient(hunting): picked mon.noname-a con 0x7efd5409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd541318f0 0x7efd54132f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 10 monclient(hunting): picked mon.noname-c con 0x7efd541318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.470+0000 7efd5a91d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd54136c20 0x7efd54133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 10 monclient(hunting): picked mon.noname-b con 0x7efd54136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd54133ef0 con 0x7efd5409ea50 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd54136220 con 0x7efd54136c20 2024-09-15T03:38:19.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd54135f40 con 0x7efd541318f0 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd5a91d640 10 monclient(hunting): authenticate will time out at 1297.110028s 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 0x7efd541329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 0x7efd541329f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55900/0 (socket says 172.21.15.153:55900) 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 -- 172.21.15.153:0/839547901 learned_addr learned my addr 172.21.15.153:0/839547901 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): get_auth_request con 0x7efd5409ea50 auth_method 0 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd541318f0 0x7efd54132f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): get_auth_request con 0x7efd541318f0 auth_method 0 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd54136c20 0x7efd54133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): get_auth_request con 0x7efd54136c20 auth_method 0 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd58eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient(hunting): handle_auth_done global_id 134136 payload 824 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd541318f0 msgr2=0x7efd54132f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd541318f0 0x7efd54132f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd54136c20 msgr2=0x7efd54133470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd54136c20 0x7efd54133470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 monclient: found mon.noname-a 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.471+0000 7efd53fff640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd44009390 con 0x7efd5409ea50 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd53fff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 msgr2=0x7efd541329f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd53fff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 0x7efd541329f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efd44009360 tx=0x7efd44032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd440318c0 con 0x7efd5409ea50 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: dump: 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.492 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd44031a20 con 0x7efd5409ea50 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 5 monclient: authenticate success, global_id 134136 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44031de0 con 0x7efd5409ea50 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 10 monclient: _renew_subs 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd54136500 con 0x7efd5409ea50 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd3bfff640 10 monclient: dump: 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd527fc640 4 set_mon_vals no callback set 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 10 monclient: _renew_subs 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.473+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd54131430 con 0x7efd5409ea50 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efd44039040 con 0x7efd5409ea50 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd5a91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd5a91d640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 0x7efd540a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd5a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd54065e00 con 0x7efd5412d160 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd3bfff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd40038740 0x7efd4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efd4406c8f0 con 0x7efd5409ea50 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd537fe640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 0x7efd540a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd537fe640 10 monclient: get_auth_request con 0x7efd5412d160 auth_method 0 2024-09-15T03:38:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.474+0000 7efd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd58eba640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd40038740 0x7efd4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd58eba640 10 monclient: get_auth_request con 0x7efd40038740 auth_method 0 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd537fe640 10 monclient(hunting): handle_auth_done global_id 104848 payload 824 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.475+0000 7efd537fe640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd54134ac0 con 0x7efd5412d160 2024-09-15T03:38:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd537fe640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 msgr2=0x7efd540a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd537fe640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 0x7efd540a3d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f780 tx=0x7efd48007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd58eba640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd40038740 0x7efd4003abf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efd4c006d70 tx=0x7efd4c0052b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4800d8b0 con 0x7efd5412d160 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 10 monclient: dump: 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efd4800da10 con 0x7efd5412d160 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 msgr2=0x7efd540a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 0x7efd540a3d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f780 tx=0x7efd48007e10 comp rx=0 tx=0).stop 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 msgr2=0x7efd540a3d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.477+0000 7efd3bfff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd5412d160 0x7efd540a3d90 secure :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f780 tx=0x7efd48007e10 comp rx=0 tx=0).stop 2024-09-15T03:38:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd5a91d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd5a91d640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 0x7efd540a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd5a91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5412d530 con 0x7efd540a42d0 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 0x7efd540a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient: get_auth_request con 0x7efd540a42d0 auth_method 0 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.483+0000 7efd537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.484+0000 7efd537fe640 10 monclient(hunting): handle_auth_done global_id 104851 payload 824 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.484+0000 7efd537fe640 1 -- 172.21.15.153:0/839547901 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efd5404bf60 con 0x7efd540a42d0 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd537fe640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 msgr2=0x7efd540a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd537fe640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 0x7efd540a46d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f780 tx=0x7efd48019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48014ea0 con 0x7efd540a42d0 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 10 monclient: dump: 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7efd48007240 con 0x7efd540a42d0 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 msgr2=0x7efd540a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 0x7efd540a46d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f780 tx=0x7efd48019000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 msgr2=0x7efd540a46d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.486+0000 7efd3bfff640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd540a42d0 0x7efd540a46d0 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efd4800f780 tx=0x7efd48019000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 106, 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.506 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:19.508 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.510 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.511 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.512 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:19.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:19.515 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.516 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.517 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:38:19.518 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:19.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:19.520 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:19.520 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:19.522 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:38:19.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:38:19.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:19.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-15T03:38:19.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-15T03:38:19.523 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:38:19.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd40038740 msgr2=0x7efd4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd40038740 0x7efd4003abf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efd4c006d70 tx=0x7efd4c0052b0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 10 monclient: shutdown 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 msgr2=0x7efd541329f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 0x7efd541329f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efd44009360 tx=0x7efd44032000 comp rx=0 tx=0).stop 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd58eba640 1 -- 172.21.15.153:0/839547901 reap_dead start 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 shutdown_connections 2024-09-15T03:38:19.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 --2- 172.21.15.153:0/839547901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd5409ea50 0x7efd541329f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 >> 172.21.15.153:0/839547901 conn(0x7efd54067110 msgr2=0x7efd5412bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 shutdown_connections 2024-09-15T03:38:19.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.487+0000 7efd5a91d640 1 -- 172.21.15.153:0/839547901 wait complete. 2024-09-15T03:38:19.525 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-15T03:38:19.525 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:38:19.525 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:38:19.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:38:19.526 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-15T03:38:19.526 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-15T03:38:19.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.620+0000 7f28a091d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.620+0000 7f28a091d640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.620+0000 7f28a091d640 10 monclient: monmap: 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:19.620794+0000 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:19.620794+0000 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:19.624 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.620+0000 7f28a091d640 1 Processor -- start 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.620+0000 7f28a091d640 1 -- start start 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.620+0000 7f28a091d640 10 monclient: init 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f289c1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:19.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f289c09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient(hunting): picked mon.noname-b con 0x7f289c137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c132640 con 0x7f289c1318f0 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c131cc0 con 0x7f289c09ea50 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289c09f530 con 0x7f289c137c30 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f28a091d640 10 monclient(hunting): authenticate will time out at 1297.260172s 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289a59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f289c1318f0 0x7f289c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40546/0 (socket says 172.21.15.153:40546) 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289a59c640 1 -- 172.21.15.153:0/3935914133 learned_addr learned my addr 172.21.15.153:0/3935914133 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): get_auth_request con 0x7f289c137c30 auth_method 0 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient(hunting): handle_auth_done global_id 124307 payload 274 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 1 -- 172.21.15.153:0/3935914133 >> v1:172.21.15.181:6789/0 conn(0x7f289c09ea50 legacy=0x7f289c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 1 -- 172.21.15.153:0/3935914133 >> v1:172.21.15.153:6789/0 conn(0x7f289c1318f0 legacy=0x7f289c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 1 monclient: found mon.noname-b 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:19.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.621+0000 7f289ad9d640 1 -- 172.21.15.153:0/3935914133 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c09fbf0 con 0x7f289c137c30 2024-09-15T03:38:19.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f289ad9d640 1 -- 172.21.15.153:0/3935914133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 msgr2=0x7f289c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f289ad9d640 1 --2- 172.21.15.153:0/3935914133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c13a010 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2894009e90 tx=0x7f2894009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91baba152d34ea18 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 1 -- 172.21.15.153:0/3935914133 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894007840 con 0x7f289c137c30 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: dump: 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.636 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.637 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.637 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.637 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.637 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 1 -- 172.21.15.153:0/3935914133 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289402d050 con 0x7f289c137c30 2024-09-15T03:38:19.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f28a091d640 5 monclient: authenticate success, global_id 124307 2024-09-15T03:38:19.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f28a091d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:19.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f28a091d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:19.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 1 -- 172.21.15.153:0/3935914133 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894007bd0 con 0x7f289c137c30 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f28a091d640 4 set_mon_vals no callback set 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.624+0000 7f2899d9b640 10 monclient: dump: 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.639 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 10 monclient: shutdown 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 -- 172.21.15.153:0/3935914133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 msgr2=0x7f289c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 --2- 172.21.15.153:0/3935914133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c13a010 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2894009e90 tx=0x7f2894009780 comp rx=0 tx=0).stop 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 -- 172.21.15.153:0/3935914133 shutdown_connections 2024-09-15T03:38:19.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 --2- 172.21.15.153:0/3935914133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c13a010 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 -- 172.21.15.153:0/3935914133 >> 172.21.15.153:0/3935914133 conn(0x7f289c067110 msgr2=0x7f289c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 -- 172.21.15.153:0/3935914133 shutdown_connections 2024-09-15T03:38:19.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 1 -- 172.21.15.153:0/3935914133 wait complete. 2024-09-15T03:38:19.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 10 monclient: build_initial_monmap 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.625+0000 7f28a091d640 10 monclient: monmap: 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 Processor -- start 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 -- start start 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient: init 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f289c09ea50 0x7f289c137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): picked mon.noname-b con 0x7f289c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 0x7f289c12df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): picked mon.noname-a con 0x7f289c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c12e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): picked mon.noname-c con 0x7f289c137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c131340 con 0x7f289c1318f0 2024-09-15T03:38:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c132870 con 0x7f289c09ea50 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289c132ea0 con 0x7f289c137c30 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f28a091d640 10 monclient(hunting): authenticate will time out at 1297.265498s 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 0x7f289c12df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c12e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 0x7f289c12df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55904/0 (socket says 172.21.15.153:55904) 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 -- 172.21.15.153:0/1571145142 learned_addr learned my addr 172.21.15.153:0/1571145142 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c12e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57020/0 (socket says 172.21.15.153:57020) 2024-09-15T03:38:19.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): get_auth_request con 0x7f289c1318f0 auth_method 0 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): get_auth_request con 0x7f289c137c30 auth_method 0 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f289c09ea50 0x7f289c137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): get_auth_request con 0x7f289c09ea50 auth_method 0 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:19.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient(hunting): handle_auth_done global_id 134142 payload 824 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 -- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 msgr2=0x7f289c12e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c12e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 -- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f289c09ea50 msgr2=0x7f289c137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f289c09ea50 0x7f289c137060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:38:19.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 monclient: found mon.noname-a 2024-09-15T03:38:19.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:19.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f2891d9b640 1 -- 172.21.15.153:0/1571145142 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2894009a20 con 0x7f289c1318f0 2024-09-15T03:38:19.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.626+0000 7f289ad9d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c12e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:19.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2891d9b640 1 -- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 msgr2=0x7f289c12df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:19.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2891d9b640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 0x7f289c12df80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f288400c5e0 tx=0x7f2884007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884012840 con 0x7f289c1318f0 2024-09-15T03:38:19.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:19.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: dump: 2024-09-15T03:38:19.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.650 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.650 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.651 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.651 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.651 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.651 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.651 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.651 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.652 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: _finish_auth 0 2024-09-15T03:38:19.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:19.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28840129c0 con 0x7f289c1318f0 2024-09-15T03:38:19.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 5 monclient: authenticate success, global_id 134142 2024-09-15T03:38:19.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:19.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884012d80 con 0x7f289c1318f0 2024-09-15T03:38:19.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 10 monclient: _renew_subs 2024-09-15T03:38:19.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c130450 con 0x7f289c1318f0 2024-09-15T03:38:19.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f289959a640 4 set_mon_vals no callback set 2024-09-15T03:38:19.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:19.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:19.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f2892ffd640 10 monclient: dump: 2024-09-15T03:38:19.654 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:19.654 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:19.655 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.655 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:19.655 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:19.656 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:19.656 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:19.656 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:19.656 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:19.656 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 10 monclient: _renew_subs 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.629+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c131620 con 0x7f289c1318f0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2884011590 con 0x7f289c1318f0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f28a091d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f28a091d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f28a091d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289c13a270 con 0x7f289c1318f0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f2892ffd640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f287c038740 0x7f287c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(258..258 src has 199..258) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f288404c7a0 con 0x7f289c1318f0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f289a59c640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f287c038740 0x7f287c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.630+0000 7f289a59c640 10 monclient: get_auth_request con 0x7f287c038740 auth_method 0 2024-09-15T03:38:19.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.633+0000 7f289a59c640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f287c038740 0x7f287c03abf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f288c007fd0 tx=0x7f288c007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:19.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.634+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2884010900 con 0x7f289c1318f0 2024-09-15T03:38:19.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.634+0000 7f2892ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:19.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.634+0000 7f2892ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:19.753 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:38:19.753 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:38:19.753 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:38:19.753 INFO:tasks.workunit.client.0.smithi181.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T03:38:19.753 INFO:tasks.workunit.client.0.smithi181.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T03:38:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-15T03:38:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-15T03:38:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=110 2024-09-15T03:38:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-15T03:38:19.754 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-15T03:38:19.754 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-15T03:38:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.850+0000 7f28a091d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:38:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.850+0000 7f28a091d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:38:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.850+0000 7f28a091d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.850+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f289c04bf60 con 0x7f289c1318f0 2024-09-15T03:38:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.851+0000 7f2892ffd640 1 -- 172.21.15.153:0/1571145142 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f2884016020 con 0x7f289c1318f0 2024-09-15T03:38:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.851+0000 7f2892ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:38:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.851+0000 7f2892ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:19.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.851+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.851+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=6,logm=58} crc {auth=3443974811,config=3140514009,health=2950547158,kv=3567005442,logm=506392510}) 2024-09-15T03:38:19.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.851+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=7,logm=59} crc {auth=863345154,config=3140514009,health=668177788,kv=4015280610,logm=2470096869}) 2024-09-15T03:38:19.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.851+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.851+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=6,logm=58} crc {auth=3443974811,config=3140514009,health=2950547158,kv=3567005442,logm=506392510}) 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.851+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=1,health=11,kv=6,logm=61} crc {auth=3574062342,config=1228700967,health=1646222991,kv=2373971965,logm=3180204389}) 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.852+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.852+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861442451}) 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.852+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284298870}) 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.852+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.852+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861442451}) 2024-09-15T03:38:19.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.852+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591779510}) 2024-09-15T03:38:19.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.853+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.853+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117295761}) 2024-09-15T03:38:19.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.853+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=591763707}) 2024-09-15T03:38:19.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.853+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.853+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117295761}) 2024-09-15T03:38:19.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.853+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=277940079}) 2024-09-15T03:38:19.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.854+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.854+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749725195}) 2024-09-15T03:38:19.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.854+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2132485347}) 2024-09-15T03:38:19.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.854+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f287c038740 msgr2=0x7f287c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.854+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749725195}) 2024-09-15T03:38:19.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.854+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219106708}) 2024-09-15T03:38:19.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f287c038740 0x7f287c03abf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f288c007fd0 tx=0x7f288c007f00 comp rx=0 tx=0).stop 2024-09-15T03:38:19.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 10 monclient: shutdown 2024-09-15T03:38:19.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 msgr2=0x7f289c12df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 0x7f289c12df80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f288400c5e0 tx=0x7f2884007ed0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 shutdown_connections 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f287c038740 0x7f287c03abf0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f289c137c30 0x7f289c12e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f289c1318f0 0x7f289c12df80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 --2- 172.21.15.153:0/1571145142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f289c09ea50 0x7f289c137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 >> 172.21.15.153:0/1571145142 conn(0x7f289c067110 msgr2=0x7f289c09d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 shutdown_connections 2024-09-15T03:38:19.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:19.854+0000 7f28a091d640 1 -- 172.21.15.153:0/1571145142 wait complete. 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.855+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.855+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015825259}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.855+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805678806}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.855+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.855+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015825259}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.855+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2549254414}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664198090}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682404349}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664198090}) 2024-09-15T03:38:19.864 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751876454}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389455620}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1650085921}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389455620}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.856+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972598428}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.857+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.857+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419241134}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.857+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453318892}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.857+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.857+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419241134}) 2024-09-15T03:38:19.865 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.857+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483219423}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.858+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.858+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299314964}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.858+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277406312}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.858+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.858+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299314964}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.858+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266275515}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.859+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.859+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771540824}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.859+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712458289}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.859+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.859+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771540824}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.859+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693307149}) 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.860+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.866 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.860+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1514805824}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.860+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734291412}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.860+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.860+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1514805824}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.860+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399363878}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344921892}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377429530}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344921892}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389559860}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.861+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597957487}) 2024-09-15T03:38:19.867 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=679285479}) 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597957487}) 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2030989014}) 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460703598}) 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3845202861}) 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460703598}) 2024-09-15T03:38:19.868 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.862+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425969754}) 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.863+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.863+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512270761}) 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.863+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3004338413}) 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.863+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.863+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512270761}) 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.863+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1211786500}) 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.864+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.872 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.864+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398711490}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.864+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750257409}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.864+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.864+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398711490}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.864+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2893922516}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.865+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.865+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304675117}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.865+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=261751439}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.865+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.865+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304675117}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.865+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=788757758}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745139417}) 2024-09-15T03:38:19.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2425287374}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745139417}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690525124}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764622292}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2375601193}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764622292}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.866+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3705976907}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.867+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.867+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577814085}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.867+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216081765}) 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.867+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.874 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.867+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577814085}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.867+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3237930847}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.868+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.868+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560506563}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.868+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758944324}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.868+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.868+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560506563}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.868+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882539712}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007070921}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763141753}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007070921}) 2024-09-15T03:38:19.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=284461758}) 2024-09-15T03:38:19.876 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:38:19.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.869+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2551754839}) 2024-09-15T03:38:19.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3593075213}) 2024-09-15T03:38:19.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2551754839}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343500238}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633816519}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4152662852}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633816519}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.870+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236156780}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967591865}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565490227}) 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967591865}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972908579}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461683960}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088463168}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461683960}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.871+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2893503939}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018159118}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202015867}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018159118}) 2024-09-15T03:38:19.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323136862}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443230677}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1735158468}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443230677}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.872+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986781433}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.873+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.873+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849580107}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.873+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161346022}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.873+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.873+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849580107}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.873+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120242091}) 2024-09-15T03:38:19.885 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174585838}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009105137}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174585838}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393072147}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779991923}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2637647036}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779991923}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.874+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883500414}) 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181439780}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=296947396}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181439780}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3902295679}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13595014}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1579545256}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13595014}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.875+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699373269}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.876+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.876+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929368984}) 2024-09-15T03:38:19.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.876+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062431668}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.876+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.876+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929368984}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.876+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280695092}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815702485}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1410150223}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815702485}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248685426}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765167159}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779088062}) 2024-09-15T03:38:19.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765167159}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.877+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151474108}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474360707}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560836415}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474360707}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840907859}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487763912}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050625966}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487763912}) 2024-09-15T03:38:19.889 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.878+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1946094319}) 2024-09-15T03:38:19.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.955+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.955+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339704316}) 2024-09-15T03:38:19.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.955+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826020910}) 2024-09-15T03:38:19.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.955+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:19.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.955+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339704316}) 2024-09-15T03:38:19.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:19.955+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095927639}) 2024-09-15T03:38:20.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.002+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.002+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329559383}) 2024-09-15T03:38:20.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.002+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1898541780}) 2024-09-15T03:38:20.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.002+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.002+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329559383}) 2024-09-15T03:38:20.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.002+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2943387887}) 2024-09-15T03:38:20.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.092+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.092+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423076831}) 2024-09-15T03:38:20.095 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.092+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078147528}) 2024-09-15T03:38:20.095 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.093+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.095 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.093+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423076831}) 2024-09-15T03:38:20.095 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.093+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384866311}) 2024-09-15T03:38:20.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.142+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.142+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132109490}) 2024-09-15T03:38:20.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.142+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2486331440}) 2024-09-15T03:38:20.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.142+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.142+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132109490}) 2024-09-15T03:38:20.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.142+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=132747171}) 2024-09-15T03:38:20.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800284352}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355652854}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800284352}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94301094}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194067524}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1006783452}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194067524}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.143+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515530375}) 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769742339}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3556766278}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769742339}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837064580}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658346153}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263931973}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.144+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658346153}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304260150}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757362141}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303709386}) 2024-09-15T03:38:20.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757362141}) 2024-09-15T03:38:20.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.145+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125966205}) 2024-09-15T03:38:20.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.146+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.146+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {logm=501981414,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2676606252,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1290294456,osd_snap=454132730,osdmap=3670354817}) 2024-09-15T03:38:20.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.146+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {logm=501981414,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2863211919,mon_config_key=1879821035,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1290294456,osd_snap=454132730,osdmap=1113522213}) 2024-09-15T03:38:20.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.146+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.146+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {logm=501981414,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2676606252,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1290294456,osd_snap=454132730,osdmap=3670354817}) 2024-09-15T03:38:20.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.146+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=5,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {logm=2163507524,mdsmap=1531151339,mgr=2291727002,mgr_metadata=2280977772,mgrstat=2676606252,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1290294456,osd_snap=454132730,osdmap=2239390995}) 2024-09-15T03:38:20.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.151+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.151+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=3788812711}) 2024-09-15T03:38:20.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.151+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=1403731308}) 2024-09-15T03:38:20.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.151+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:38:20.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.151+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=3788812711}) 2024-09-15T03:38:20.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:38:20.151+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=1419414592}) 2024-09-15T03:38:20.877 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:38:20.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:38:20.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.972+0000 7ff96c39d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.972+0000 7ff96c39d640 10 monclient: build_initial_monmap 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.972+0000 7ff96c39d640 10 monclient: monmap: 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:20.972768+0000 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:20.972768+0000 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:20.973 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:20.974 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:20.974 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.972+0000 7ff96c39d640 1 Processor -- start 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.972+0000 7ff96c39d640 1 -- start start 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.972+0000 7ff96c39d640 10 monclient: init 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 0x7ff964131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): picked mon.noname-b con 0x7ff964131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff96409e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:20.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff964136bb0 0x7ff964138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): picked mon.noname-a con 0x7ff964136bb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff96409faf0 con 0x7ff96409e910 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff96409fd30 con 0x7ff964136bb0 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9640a1110 con 0x7ff964131ab0 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96c39d640 10 monclient(hunting): authenticate will time out at 1298.612332s 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff964136bb0 0x7ff964138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 0x7ff964131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff964136bb0 0x7ff964138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55908/0 (socket says 172.21.15.153:55908) 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 0x7ff964131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57024/0 (socket says 172.21.15.153:57024) 2024-09-15T03:38:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 1 -- 172.21.15.153:0/1690083455 learned_addr learned my addr 172.21.15.153:0/1690083455 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff969938640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff96409e910 0x7ff96409ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40550/0 (socket says 172.21.15.153:40550) 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): get_auth_request con 0x7ff964136bb0 auth_method 0 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): get_auth_request con 0x7ff964131ab0 auth_method 0 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:20.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient(hunting): handle_auth_done global_id 124316 payload 274 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 -- 172.21.15.153:0/1690083455 >> v2:172.21.15.153:3300/0 conn(0x7ff964136bb0 msgr2=0x7ff964138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 --2- 172.21.15.153:0/1690083455 >> v2:172.21.15.153:3300/0 conn(0x7ff964136bb0 0x7ff964138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 -- 172.21.15.153:0/1690083455 >> v1:172.21.15.153:6789/0 conn(0x7ff96409e910 legacy=0x7ff96409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient: _finish_hunting 0 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 monclient: found mon.noname-b 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:20.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.973+0000 7ff96a139640 1 -- 172.21.15.153:0/1690083455 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640a13f0 con 0x7ff964131ab0 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff96a139640 1 -- 172.21.15.153:0/1690083455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 msgr2=0x7ff964131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff96a139640 1 --2- 172.21.15.153:0/1690083455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 0x7ff964131e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff960009850 tx=0x7ff9600090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=474cdbf0f0c7c6f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 1 -- 172.21.15.153:0/1690083455 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96002ae80 con 0x7ff964131ab0 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: dump: 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:20.985 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: _finish_auth 0 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 1 -- 172.21.15.153:0/1690083455 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff960032440 con 0x7ff964131ab0 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff96c39d640 5 monclient: authenticate success, global_id 124316 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff96c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff96c39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff96c39d640 4 set_mon_vals no callback set 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 1 -- 172.21.15.153:0/1690083455 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9600327c0 con 0x7ff964131ab0 2024-09-15T03:38:20.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.976+0000 7ff969137640 10 monclient: dump: 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:20.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 10 monclient: shutdown 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 -- 172.21.15.153:0/1690083455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 msgr2=0x7ff964131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 --2- 172.21.15.153:0/1690083455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 0x7ff964131e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff960009850 tx=0x7ff9600090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 -- 172.21.15.153:0/1690083455 shutdown_connections 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 --2- 172.21.15.153:0/1690083455 >> v2:172.21.15.153:3300/0 conn(0x7ff964136bb0 0x7ff964138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 --2- 172.21.15.153:0/1690083455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964131ab0 0x7ff964131e80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 -- 172.21.15.153:0/1690083455 >> 172.21.15.153:0/1690083455 conn(0x7ff964067d30 msgr2=0x7ff9640a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 -- 172.21.15.153:0/1690083455 shutdown_connections 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 1 -- 172.21.15.153:0/1690083455 wait complete. 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 10 monclient: build_initial_monmap 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.977+0000 7ff96c39d640 10 monclient: monmap: 2024-09-15T03:38:20.988 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 Processor -- start 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 -- start start 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient: init 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:20.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff96409e910 0x7ff96412efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): picked mon.noname-b con 0x7ff96409e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 0x7ff96412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): picked mon.noname-a con 0x7ff964131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 0x7ff96412fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): picked mon.noname-c con 0x7ff964136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96409bde0 con 0x7ff964131ab0 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96409d900 con 0x7ff96409e910 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9641336a0 con 0x7ff964136bb0 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:20.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96c39d640 10 monclient(hunting): authenticate will time out at 1298.617529s 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 0x7ff96412fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 0x7ff96412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 0x7ff96412fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57026/0 (socket says 172.21.15.153:57026) 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 1 -- 172.21.15.153:0/87100431 learned_addr learned my addr 172.21.15.153:0/87100431 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): get_auth_request con 0x7ff964131ab0 auth_method 0 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): get_auth_request con 0x7ff964136bb0 auth_method 0 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a139640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff96409e910 0x7ff96412efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient(hunting): handle_auth_done global_id 134154 payload 824 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 -- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 msgr2=0x7ff96412fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 0x7ff96412fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 -- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff96409e910 msgr2=0x7ff96412efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff96409e910 0x7ff96412efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient: _finish_hunting 0 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 monclient: found mon.noname-a 2024-09-15T03:38:20.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff969938640 1 -- 172.21.15.153:0/87100431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff960009390 con 0x7ff964131ab0 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a139640 10 monclient: get_auth_request con 0x7ff96409e910 auth_method 0 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a93a640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 0x7ff96412fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.978+0000 7ff96a139640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff96409e910 0x7ff96412efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff969938640 1 -- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 msgr2=0x7ff96412f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff969938640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 0x7ff96412f4f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c490 tx=0x7ff954007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9540074f0 con 0x7ff964131ab0 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: dump: 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:20.993 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff954007670 con 0x7ff964131ab0 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 5 monclient: authenticate success, global_id 134154 2024-09-15T03:38:20.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 10 monclient: _renew_subs 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff964133c60 con 0x7ff964131ab0 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954005a20 con 0x7ff964131ab0 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff95a7fc640 10 monclient: dump: 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:20.995 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 10 monclient: _renew_subs 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff968936640 4 set_mon_vals no callback set 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.981+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9641326a0 con 0x7ff964131ab0 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff96c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff95401b3b0 con 0x7ff964131ab0 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff96c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff96c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96404bf60 con 0x7ff964131ab0 2024-09-15T03:38:20.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff95a7fc640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff948038740 0x7ff94803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.982+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff95404b9c0 con 0x7ff964131ab0 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.983+0000 7ff96a139640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff948038740 0x7ff94803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.983+0000 7ff96a139640 10 monclient: get_auth_request con 0x7ff948038740 auth_method 0 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.985+0000 7ff96a139640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff948038740 0x7ff94803abf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff9640a5ec0 tx=0x7ff960032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.987+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff95400f860 con 0x7ff964131ab0 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.987+0000 7ff95a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:20.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:20.987+0000 7ff95a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:21.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.199+0000 7ff96c39d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:38:21.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.199+0000 7ff96c39d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:21.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.199+0000 7ff96c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:21.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.199+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff964133860 con 0x7ff964131ab0 2024-09-15T03:38:21.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.200+0000 7ff95a7fc640 1 -- 172.21.15.153:0/87100431 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7ff954017080 con 0x7ff964131ab0 2024-09-15T03:38:21.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.200+0000 7ff95a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:21.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.200+0000 7ff95a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:21.201 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":106,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff948038740 msgr2=0x7ff94803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff948038740 0x7ff94803abf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff9640a5ec0 tx=0x7ff960032000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 10 monclient: shutdown 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 msgr2=0x7ff96412f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 0x7ff96412f4f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c490 tx=0x7ff954007e40 comp rx=0 tx=0).stop 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 shutdown_connections 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff948038740 0x7ff94803abf0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff964136bb0 0x7ff96412fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff964131ab0 0x7ff96412f4f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 --2- 172.21.15.153:0/87100431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff96409e910 0x7ff96412efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.201+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 >> 172.21.15.153:0/87100431 conn(0x7ff964067d30 msgr2=0x7ff96409c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.202+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 shutdown_connections 2024-09-15T03:38:21.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.202+0000 7ff96c39d640 1 -- 172.21.15.153:0/87100431 wait complete. 2024-09-15T03:38:21.213 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:38:21.213 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:38:21.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:38:21.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.307+0000 7fc383b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.307+0000 7fc383b5d640 10 monclient: build_initial_monmap 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.307+0000 7fc383b5d640 10 monclient: monmap: 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:21.307917+0000 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:21.307917+0000 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:21.308 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:21.309 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:21.309 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:21.309 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:21.309 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.307+0000 7fc383b5d640 1 Processor -- start 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 -- start start 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient: init 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc37c1318f0 0x7fc37c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc37c1318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 0x7fc37c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc37c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:21.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 0x7fc37c0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc37c068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37c131cc0 con 0x7fc37c068d90 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37c09fbf0 con 0x7fc37c1318f0 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c09fe30 con 0x7fc37c09ea50 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc383b5d640 10 monclient(hunting): authenticate will time out at 1298.947507s 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 0x7fc37c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 0x7fc37c0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 0x7fc37c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57036/0 (socket says 172.21.15.153:57036) 2024-09-15T03:38:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 0x7fc37c0691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55924/0 (socket says 172.21.15.153:55924) 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 -- 172.21.15.153:0/3045659478 learned_addr learned my addr 172.21.15.153:0/3045659478 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): get_auth_request con 0x7fc37c09ea50 auth_method 0 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): get_auth_request con 0x7fc37c068d90 auth_method 0 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3818f9640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.181:3300/0 conn(0x7fc37c1318f0 0x7fc37c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient(hunting): handle_auth_done global_id 124322 payload 274 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 -- 172.21.15.153:0/3045659478 >> v2:172.21.15.181:3300/0 conn(0x7fc37c1318f0 msgr2=0x7fc37c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.181:3300/0 conn(0x7fc37c1318f0 0x7fc37c131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 -- 172.21.15.153:0/3045659478 >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 msgr2=0x7fc37c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 0x7fc37c0691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient: _finish_hunting 0 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 monclient: found mon.noname-b 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3810f8640 1 -- 172.21.15.153:0/3045659478 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0a1210 con 0x7fc37c09ea50 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3818f9640 10 monclient: get_auth_request con 0x7fc37c1318f0 auth_method 0 2024-09-15T03:38:21.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3820fa640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 0x7fc37c0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:21.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.308+0000 7fc3818f9640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.181:3300/0 conn(0x7fc37c1318f0 0x7fc37c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.311+0000 7fc3810f8640 1 -- 172.21.15.153:0/3045659478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 msgr2=0x7fc37c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.311+0000 7fc3810f8640 1 --2- 172.21.15.153:0/3045659478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 0x7fc37c09ee20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc37c076600 tx=0x7fc36c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8102a0d5a338856 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 1 -- 172.21.15.153:0/3045659478 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c032440 con 0x7fc37c09ea50 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: dump: 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.322 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: _finish_auth 0 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 5 monclient: authenticate success, global_id 124322 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 1 -- 172.21.15.153:0/3045659478 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c0325a0 con 0x7fc37c09ea50 2024-09-15T03:38:21.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 4 set_mon_vals no callback set 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 1 -- 172.21.15.153:0/3045659478 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c032980 con 0x7fc37c09ea50 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc3808f7640 10 monclient: dump: 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 10 monclient: shutdown 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 -- 172.21.15.153:0/3045659478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 msgr2=0x7fc37c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 --2- 172.21.15.153:0/3045659478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 0x7fc37c09ee20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc37c076600 tx=0x7fc36c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 -- 172.21.15.153:0/3045659478 shutdown_connections 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.153:3300/0 conn(0x7fc37c068d90 0x7fc37c0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 --2- 172.21.15.153:0/3045659478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c09ea50 0x7fc37c09ee20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 --2- 172.21.15.153:0/3045659478 >> v2:172.21.15.181:3300/0 conn(0x7fc37c1318f0 0x7fc37c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 -- 172.21.15.153:0/3045659478 >> 172.21.15.153:0/3045659478 conn(0x7fc37c067110 msgr2=0x7fc37c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 -- 172.21.15.153:0/3045659478 shutdown_connections 2024-09-15T03:38:21.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.312+0000 7fc383b5d640 1 -- 172.21.15.153:0/3045659478 wait complete. 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.313+0000 7fc383b5d640 10 monclient: build_initial_monmap 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.313+0000 7fc383b5d640 10 monclient: monmap: 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.313+0000 7fc383b5d640 1 Processor -- start 2024-09-15T03:38:21.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.313+0000 7fc383b5d640 1 -- start start 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.313+0000 7fc383b5d640 10 monclient: init 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.313+0000 7fc383b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc37c068d90 0x7fc37c09a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc37c068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c09ea50 0x7fc37c130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc37c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 0x7fc37c140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc37c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c09bf10 con 0x7fc37c09ea50 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0a4390 con 0x7fc37c068d90 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c09b590 con 0x7fc37c1318f0 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc383b5d640 10 monclient(hunting): authenticate will time out at 1298.953086s 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c09ea50 0x7fc37c130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 0x7fc37c140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c09ea50 0x7fc37c130c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55940/0 (socket says 172.21.15.153:55940) 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 0x7fc37c140d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57042/0 (socket says 172.21.15.153:57042) 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 1 -- 172.21.15.153:0/1844448311 learned_addr learned my addr 172.21.15.153:0/1844448311 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): get_auth_request con 0x7fc37c09ea50 auth_method 0 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): get_auth_request con 0x7fc37c1318f0 auth_method 0 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc37c068d90 0x7fc37c09a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): get_auth_request con 0x7fc37c068d90 auth_method 0 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3810f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient(hunting): handle_auth_done global_id 124325 payload 824 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc37c068d90 msgr2=0x7fc37c09a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc37c068d90 0x7fc37c09a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c09ea50 msgr2=0x7fc37c130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c09ea50 0x7fc37c130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient: _finish_hunting 0 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 monclient: found mon.noname-c 2024-09-15T03:38:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.314+0000 7fc3820fa640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374009a20 con 0x7fc37c1318f0 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc3820fa640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 msgr2=0x7fc37c140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc3820fa640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 0x7fc37c140d90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc374009e70 tx=0x7fc37400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374014480 con 0x7fc37c1318f0 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: dump: 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: _finish_auth 0 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37400f040 con 0x7fc37c1318f0 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc383b5d640 5 monclient: authenticate success, global_id 124325 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc383b5d640 10 monclient: _renew_subs 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374014850 con 0x7fc37c1318f0 2024-09-15T03:38:21.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc383b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c009450 con 0x7fc37c1318f0 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc371ffb640 10 monclient: dump: 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.316+0000 7fc373fff640 4 set_mon_vals no callback set 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 10 monclient: _renew_subs 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c11bc00 con 0x7fc37c1318f0 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc374003020 con 0x7fc37c1318f0 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 0x7fc37c1412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c065e00 con 0x7fc37c11c7e0 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.317+0000 7fc371ffb640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc360038790 0x7fc36003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 0x7fc37c1412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc37404d040 con 0x7fc37c1318f0 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient: get_auth_request con 0x7fc37c11c7e0 auth_method 0 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 10 monclient(hunting): handle_auth_done global_id 134163 payload 824 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3818f9640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc37c11e1c0 con 0x7fc37c11c7e0 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3810f8640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc360038790 0x7fc36003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.318+0000 7fc3810f8640 10 monclient: get_auth_request con 0x7fc360038790 auth_method 0 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc3818f9640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 msgr2=0x7fc37c1412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc3818f9640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 0x7fc37c1412d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc37800f970 tx=0x7fc378008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37800dce0 con 0x7fc37c11c7e0 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 10 monclient: dump: 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc37800c040 con 0x7fc37c11c7e0 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 msgr2=0x7fc37c1412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 0x7fc37c1412d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc37800f970 tx=0x7fc378008040 comp rx=0 tx=0).stop 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 msgr2=0x7fc37c1412d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc371ffb640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c11c7e0 0x7fc37c1412d0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc37800f970 tx=0x7fc378008040 comp rx=0 tx=0).stop 2024-09-15T03:38:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.320+0000 7fc3810f8640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc360038790 0x7fc36003ac40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009420 tx=0x7fc36c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc383b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc383b5d640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 0x7fc37c141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc383b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c04bf60 con 0x7fc37c131140 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 0x7fc37c141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient: get_auth_request con 0x7fc37c131140 auth_method 0 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 10 monclient(hunting): handle_auth_done global_id 134166 payload 824 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.326+0000 7fc3818f9640 1 -- 172.21.15.153:0/1844448311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc37c11dd80 con 0x7fc37c131140 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc3818f9640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 msgr2=0x7fc37c141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc3818f9640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 0x7fc37c141810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc37801a000 tx=0x7fc37801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378006250 con 0x7fc37c131140 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 10 monclient: dump: 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fc3780063b0 con 0x7fc37c131140 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 msgr2=0x7fc37c141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 0x7fc37c141810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc37801a000 tx=0x7fc37801d000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 msgr2=0x7fc37c141810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:21.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc371ffb640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc37c131140 0x7fc37c141810 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc37801a000 tx=0x7fc37801d000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 106, 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:38:21.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:21.343 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:21.344 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.346 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:21.348 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.351 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.352 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:21.353 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.354 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.354 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:21.354 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc360038790 msgr2=0x7fc36003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc360038790 0x7fc36003ac40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009420 tx=0x7fc36c032000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 10 monclient: shutdown 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 msgr2=0x7fc37c140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 --2- 172.21.15.153:0/1844448311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc37c1318f0 0x7fc37c140d90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc374009e70 tx=0x7fc37400b040 comp rx=0 tx=0).stop 2024-09-15T03:38:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc3820fa640 1 -- 172.21.15.153:0/1844448311 reap_dead start 2024-09-15T03:38:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc3820fa640 1 -- 172.21.15.153:0/1844448311 reap_dead start 2024-09-15T03:38:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 shutdown_connections 2024-09-15T03:38:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 >> 172.21.15.153:0/1844448311 conn(0x7fc37c067110 msgr2=0x7fc37c09ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 shutdown_connections 2024-09-15T03:38:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.329+0000 7fc383b5d640 1 -- 172.21.15.153:0/1844448311 wait complete. 2024-09-15T03:38:21.356 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:38:21.356 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:38:21.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.449+0000 7f695459d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.449+0000 7f695459d640 10 monclient: build_initial_monmap 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.449+0000 7f695459d640 10 monclient: monmap: 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:21.450665+0000 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:21.450665+0000 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:21.451 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.450+0000 7f695459d640 1 Processor -- start 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.450+0000 7f695459d640 1 -- start start 2024-09-15T03:38:21.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.450+0000 7f695459d640 10 monclient: init 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f694c096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f694c0971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f694c134b40 0x7f694c136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient(hunting): picked mon.noname-c con 0x7f694c134b40 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:21.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c09d4a0 con 0x7f694c096e10 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c099620 con 0x7f694c0971e0 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f694c099340 con 0x7f694c134b40 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f695459d640 10 monclient(hunting): authenticate will time out at 1299.090156s 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952339640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f694c096e10 0x7f694c09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40552/0 (socket says 172.21.15.153:40552) 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952339640 1 -- 172.21.15.153:0/2081919239 learned_addr learned my addr 172.21.15.153:0/2081919239 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 1 --2- 172.21.15.153:0/2081919239 >> v2:172.21.15.181:3300/0 conn(0x7f694c134b40 0x7f694c136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): get_auth_request con 0x7f694c134b40 auth_method 0 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient(hunting): handle_auth_done global_id 104869 payload 274 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 1 -- 172.21.15.153:0/2081919239 >> v1:172.21.15.181:6789/0 conn(0x7f694c0971e0 legacy=0x7f694c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 1 -- 172.21.15.153:0/2081919239 >> v1:172.21.15.153:6789/0 conn(0x7f694c096e10 legacy=0x7f694c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 1 monclient: found mon.noname-c 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:38:21.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.451+0000 7f6952b3a640 1 -- 172.21.15.153:0/2081919239 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c099900 con 0x7f694c134b40 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6952b3a640 1 -- 172.21.15.153:0/2081919239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 msgr2=0x7f694c136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6952b3a640 1 --2- 172.21.15.153:0/2081919239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 0x7f694c136f20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f694c09b020 tx=0x7f6944009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=716c7acccc3422d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 1 -- 172.21.15.153:0/2081919239 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944007630 con 0x7f694c134b40 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: dump: 2024-09-15T03:38:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: _finish_auth 0 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 1 -- 172.21.15.153:0/2081919239 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694402d050 con 0x7f694c134b40 2024-09-15T03:38:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 1 -- 172.21.15.153:0/2081919239 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69440079c0 con 0x7f694c134b40 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f695459d640 5 monclient: authenticate success, global_id 104869 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f695459d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f695459d640 4 set_mon_vals no callback set 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.454+0000 7f6951337640 10 monclient: dump: 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 10 monclient: shutdown 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 -- 172.21.15.153:0/2081919239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 msgr2=0x7f694c136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 --2- 172.21.15.153:0/2081919239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 0x7f694c136f20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f694c09b020 tx=0x7f6944009780 comp rx=0 tx=0).stop 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 -- 172.21.15.153:0/2081919239 shutdown_connections 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 --2- 172.21.15.153:0/2081919239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 0x7f694c136f20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 -- 172.21.15.153:0/2081919239 >> 172.21.15.153:0/2081919239 conn(0x7f694c0672b0 msgr2=0x7f694c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 -- 172.21.15.153:0/2081919239 shutdown_connections 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.455+0000 7f695459d640 1 -- 172.21.15.153:0/2081919239 wait complete. 2024-09-15T03:38:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient: build_initial_monmap 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient: monmap: 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 Processor -- start 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 -- start start 2024-09-15T03:38:21.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient: init 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 0x7f694c06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): picked mon.noname-c con 0x7f694c096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f694c0971e0 0x7f694c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): picked mon.noname-a con 0x7f694c0971e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 0x7f694c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): picked mon.noname-b con 0x7f694c134b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c09c4f0 con 0x7f694c0971e0 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c06f170 con 0x7f694c134b40 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c09b050 con 0x7f694c096e10 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f695459d640 10 monclient(hunting): authenticate will time out at 1299.095861s 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 0x7f694c06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6951b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f694c0971e0 0x7f694c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 0x7f694c06dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57046/0 (socket says 172.21.15.153:57046) 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6951b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f694c0971e0 0x7f694c06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55976/0 (socket says 172.21.15.153:55976) 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 1 -- 172.21.15.153:0/3988784970 learned_addr learned my addr 172.21.15.153:0/3988784970 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 10 monclient(hunting): get_auth_request con 0x7f694c096e10 auth_method 0 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6952339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6951b38640 10 monclient(hunting): get_auth_request con 0x7f694c0971e0 auth_method 0 2024-09-15T03:38:21.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6951b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6951b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.456+0000 7f6951b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952b3a640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 0x7f694c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952b3a640 10 monclient(hunting): get_auth_request con 0x7f694c134b40 auth_method 0 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 10 monclient(hunting): handle_auth_done global_id 124328 payload 824 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 msgr2=0x7f694c06e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f694c134b40 0x7f694c06e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f694c0971e0 msgr2=0x7f694c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f694c0971e0 0x7f694c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 10 monclient: _finish_hunting 0 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 1 monclient: found mon.noname-c 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:21.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6952339640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944009a20 con 0x7f694c096e10 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.457+0000 7f6951b38640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f694c0971e0 0x7f694c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f6952339640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 msgr2=0x7f694c06dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f6952339640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 0x7f694c06dab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694800c610 tx=0x7f6948004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948005cb0 con 0x7f694c096e10 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: dump: 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.471 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948005e30 con 0x7f694c096e10 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 5 monclient: authenticate success, global_id 124328 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948007340 con 0x7f694c096e10 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 10 monclient: _renew_subs 2024-09-15T03:38:21.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c118970 con 0x7f694c096e10 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f6950b36640 4 set_mon_vals no callback set 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f69427fc640 10 monclient: dump: 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.473 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 10 monclient: _renew_subs 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.459+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c06f570 con 0x7f694c096e10 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f694801b400 con 0x7f694c096e10 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f695459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f695459d640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 0x7f694c05e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f695459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c065f20 con 0x7f694c02aa40 2024-09-15T03:38:21.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 0x7f694c05e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient: get_auth_request con 0x7f694c02aa40 auth_method 0 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f69427fc640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6930038740 0x7f693003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f694804c160 con 0x7f694c096e10 2024-09-15T03:38:21.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 10 monclient(hunting): handle_auth_done global_id 124331 payload 824 2024-09-15T03:38:21.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6951b38640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f694c02ae10 con 0x7f694c02aa40 2024-09-15T03:38:21.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.460+0000 7f6952b3a640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6930038740 0x7f693003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.461+0000 7f6952b3a640 10 monclient: get_auth_request con 0x7f6930038740 auth_method 0 2024-09-15T03:38:21.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f6951b38640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 msgr2=0x7f694c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f6951b38640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 0x7f694c05e310 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f693c00fa50 tx=0x7f693c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00dda0 con 0x7f694c02aa40 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 10 monclient: dump: 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.481 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f693c014470 con 0x7f694c02aa40 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 msgr2=0x7f694c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 0x7f694c05e310 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f693c00fa50 tx=0x7f693c007e10 comp rx=0 tx=0).stop 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 msgr2=0x7f694c05e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f69427fc640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c02aa40 0x7f694c05e310 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f693c00fa50 tx=0x7f693c007e10 comp rx=0 tx=0).stop 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.463+0000 7f6952b3a640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6930038740 0x7f693003abf0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f694c076be0 tx=0x7f6944037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.468+0000 7f695459d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.468+0000 7f695459d640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 0x7f694c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f695459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c118540 con 0x7f694c05e850 2024-09-15T03:38:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 0x7f694c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient: get_auth_request con 0x7f694c05e850 auth_method 0 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 10 monclient(hunting): handle_auth_done global_id 124334 payload 824 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.469+0000 7f6951b38640 1 -- 172.21.15.153:0/3988784970 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f694c04bf60 con 0x7f694c05e850 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f6951b38640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 msgr2=0x7f694c14b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f6951b38640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 0x7f694c14b550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f693c019000 tx=0x7f693c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c01ed20 con 0x7f694c05e850 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 10 monclient: dump: 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f693c020410 con 0x7f694c05e850 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 msgr2=0x7f694c14b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 0x7f694c14b550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f693c019000 tx=0x7f693c01a000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 msgr2=0x7f694c14b550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.471+0000 7f69427fc640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c05e850 0x7f694c14b550 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f693c019000 tx=0x7f693c01a000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:21.485 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 106, 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.487 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:21.488 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:21.489 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:21.491 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:21.493 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:21.494 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.495 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:21.496 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.497 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:38:21.498 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.498 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.498 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:21.498 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6930038740 msgr2=0x7f693003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6930038740 0x7f693003abf0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f694c076be0 tx=0x7f6944037000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 10 monclient: shutdown 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 msgr2=0x7f694c06dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 --2- 172.21.15.153:0/3988784970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f694c096e10 0x7f694c06dab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694800c610 tx=0x7f6948004960 comp rx=0 tx=0).stop 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f6952b3a640 1 -- 172.21.15.153:0/3988784970 reap_dead start 2024-09-15T03:38:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f6952b3a640 1 -- 172.21.15.153:0/3988784970 reap_dead start 2024-09-15T03:38:21.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 shutdown_connections 2024-09-15T03:38:21.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 >> 172.21.15.153:0/3988784970 conn(0x7f694c0672b0 msgr2=0x7f694c1388f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 shutdown_connections 2024-09-15T03:38:21.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.472+0000 7f695459d640 1 -- 172.21.15.153:0/3988784970 wait complete. 2024-09-15T03:38:21.502 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:38:21.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:38:21.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.595+0000 7eff41f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.595+0000 7eff41f9d640 10 monclient: build_initial_monmap 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.595+0000 7eff41f9d640 10 monclient: monmap: 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:21.595803+0000 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:21.595803+0000 2024-09-15T03:38:21.596 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:21.597 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.595+0000 7eff41f9d640 1 Processor -- start 2024-09-15T03:38:21.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.595+0000 7eff41f9d640 1 -- start start 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.595+0000 7eff41f9d640 10 monclient: init 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff3c1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7eff3c09ea50 0x7eff3c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): picked mon.noname-c con 0x7eff3c09ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff3c136c20 0x7eff3c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): picked mon.noname-a con 0x7eff3c136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3c131cc0 con 0x7eff3c1318f0 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3c09fbf0 con 0x7eff3c136c20 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3c09fe30 con 0x7eff3c09ea50 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff41f9d640 10 monclient(hunting): authenticate will time out at 1299.235370s 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7eff3c1318f0 0x7eff3c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40568/0 (socket says 172.21.15.153:40568) 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff3c136c20 0x7eff3c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3b7fe640 1 -- 172.21.15.153:0/1190835296 learned_addr learned my addr 172.21.15.153:0/1190835296 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): get_auth_request con 0x7eff3c136c20 auth_method 0 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3affd640 1 --2- 172.21.15.153:0/1190835296 >> v2:172.21.15.181:3300/0 conn(0x7eff3c09ea50 0x7eff3c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3affd640 10 monclient(hunting): get_auth_request con 0x7eff3c09ea50 auth_method 0 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient(hunting): handle_auth_done global_id 134172 payload 274 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 1 -- 172.21.15.153:0/1190835296 >> v2:172.21.15.181:3300/0 conn(0x7eff3c09ea50 msgr2=0x7eff3c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 1 --2- 172.21.15.153:0/1190835296 >> v2:172.21.15.181:3300/0 conn(0x7eff3c09ea50 0x7eff3c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 1 -- 172.21.15.153:0/1190835296 >> v1:172.21.15.153:6789/0 conn(0x7eff3c1318f0 legacy=0x7eff3c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 1 monclient: found mon.noname-a 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3bfff640 1 -- 172.21.15.153:0/1190835296 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0a1210 con 0x7eff3c136c20 2024-09-15T03:38:21.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.596+0000 7eff3affd640 1 --2- 172.21.15.153:0/1190835296 >> v2:172.21.15.181:3300/0 conn(0x7eff3c09ea50 0x7eff3c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:21.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3bfff640 1 -- 172.21.15.153:0/1190835296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 msgr2=0x7eff3c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3bfff640 1 --2- 172.21.15.153:0/1190835296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c139000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff3c06f600 tx=0x7eff34009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56fe8b2566fb476 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 1 -- 172.21.15.153:0/1190835296 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34007760 con 0x7eff3c136c20 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: dump: 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 1 -- 172.21.15.153:0/1190835296 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3402d050 con 0x7eff3c136c20 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 1 -- 172.21.15.153:0/1190835296 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34007af0 con 0x7eff3c136c20 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff3a7fc640 10 monclient: dump: 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.611 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff41f9d640 5 monclient: authenticate success, global_id 134172 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff41f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.599+0000 7eff41f9d640 4 set_mon_vals no callback set 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 10 monclient: shutdown 2024-09-15T03:38:21.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 -- 172.21.15.153:0/1190835296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 msgr2=0x7eff3c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1190835296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c139000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff3c06f600 tx=0x7eff34009780 comp rx=0 tx=0).stop 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 -- 172.21.15.153:0/1190835296 shutdown_connections 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1190835296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c139000 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1190835296 >> v2:172.21.15.181:3300/0 conn(0x7eff3c09ea50 0x7eff3c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 -- 172.21.15.153:0/1190835296 >> 172.21.15.153:0/1190835296 conn(0x7eff3c067110 msgr2=0x7eff3c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 -- 172.21.15.153:0/1190835296 shutdown_connections 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 1 -- 172.21.15.153:0/1190835296 wait complete. 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 10 monclient: build_initial_monmap 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.600+0000 7eff41f9d640 10 monclient: monmap: 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 Processor -- start 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 -- start start 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient: init 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:38:21.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09ea50 0x7eff3c1412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): picked mon.noname-b con 0x7eff3c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 0x7eff3c141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): picked mon.noname-c con 0x7eff3c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c0995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): picked mon.noname-a con 0x7eff3c136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c09bf10 con 0x7eff3c136c20 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c09da80 con 0x7eff3c09ea50 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c11c710 con 0x7eff3c1318f0 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff41f9d640 10 monclient(hunting): authenticate will time out at 1299.240843s 2024-09-15T03:38:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 0x7eff3c141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c0995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 0x7eff3c141800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57078/0 (socket says 172.21.15.153:57078) 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 1 -- 172.21.15.153:0/1631427096 learned_addr learned my addr 172.21.15.153:0/1631427096 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c0995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55988/0 (socket says 172.21.15.153:55988) 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 10 monclient(hunting): get_auth_request con 0x7eff3c1318f0 auth_method 0 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3bfff640 10 monclient(hunting): get_auth_request con 0x7eff3c136c20 auth_method 0 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.601+0000 7eff3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3b7fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09ea50 0x7eff3c1412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3b7fe640 10 monclient(hunting): get_auth_request con 0x7eff3c09ea50 auth_method 0 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 10 monclient(hunting): handle_auth_done global_id 124337 payload 824 2024-09-15T03:38:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09ea50 msgr2=0x7eff3c1412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09ea50 0x7eff3c1412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 msgr2=0x7eff3c0995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff3c136c20 0x7eff3c0995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 10 monclient: _finish_hunting 0 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 1 monclient: found mon.noname-c 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3affd640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34009a20 con 0x7eff3c1318f0 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.602+0000 7eff3b7fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09ea50 0x7eff3c1412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff3affd640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 msgr2=0x7eff3c141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff3affd640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 0x7eff3c141800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7eff3000d890 tx=0x7eff30008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000de80 con 0x7eff3c1318f0 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:21.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: dump: 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 5 monclient: authenticate success, global_id 124337 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3000bae0 con 0x7eff3c1318f0 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000bea0 con 0x7eff3c1318f0 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 10 monclient: _renew_subs 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c11ccd0 con 0x7eff3c1318f0 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff39ffb640 4 set_mon_vals no callback set 2024-09-15T03:38:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff237fe640 10 monclient: dump: 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 10 monclient: _renew_subs 2024-09-15T03:38:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.604+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c141d40 con 0x7eff3c1318f0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7eff30014640 con 0x7eff3c1318f0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff41f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 0x7eff3c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c11c520 con 0x7eff3c09a2f0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff237fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff28038740 0x7eff2803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7eff3004ccc0 con 0x7eff3c1318f0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff3b7fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 0x7eff3c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff3b7fe640 10 monclient: get_auth_request con 0x7eff3c09a2f0 auth_method 0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff3bfff640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff28038740 0x7eff2803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.605+0000 7eff3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.606+0000 7eff3bfff640 10 monclient: get_auth_request con 0x7eff28038740 auth_method 0 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.606+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.606+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.606+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.606+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_done global_id 104881 payload 824 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.606+0000 7eff3b7fe640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff3c04bf60 con 0x7eff3c09a2f0 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff3b7fe640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 msgr2=0x7eff3c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff3b7fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 0x7eff3c0298c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00e740 tx=0x7eff2c009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff3bfff640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff28038740 0x7eff2803abf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7eff3c076600 tx=0x7eff34037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c005cb0 con 0x7eff3c09a2f0 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 10 monclient: dump: 2024-09-15T03:38:21.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7eff2c005e10 con 0x7eff3c09a2f0 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 msgr2=0x7eff3c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 0x7eff3c0298c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00e740 tx=0x7eff2c009040 comp rx=0 tx=0).stop 2024-09-15T03:38:21.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 msgr2=0x7eff3c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:21.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.608+0000 7eff237fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a2f0 0x7eff3c0298c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:21.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff41f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:21.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 0x7eff3c029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff41f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c065e00 con 0x7eff3c09a6c0 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff3b7fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 0x7eff3c029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff3b7fe640 10 monclient: get_auth_request con 0x7eff3c09a6c0 auth_method 0 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.614+0000 7eff3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.615+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.615+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.615+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.615+0000 7eff3b7fe640 10 monclient(hunting): handle_auth_done global_id 104884 payload 824 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.615+0000 7eff3b7fe640 1 -- 172.21.15.153:0/1631427096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7eff3c11b7d0 con 0x7eff3c09a6c0 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff3b7fe640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 msgr2=0x7eff3c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff3b7fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 0x7eff3c029e00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00e740 tx=0x7eff2c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00c820 con 0x7eff3c09a6c0 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 10 monclient: dump: 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7eff2c005aa0 con 0x7eff3c09a6c0 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.617+0000 7eff237fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 msgr2=0x7eff3c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff237fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 0x7eff3c029e00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00e740 tx=0x7eff2c01a000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff237fe640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 msgr2=0x7eff3c029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff237fe640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff3c09a6c0 0x7eff3c029e00 secure :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00e740 tx=0x7eff2c01a000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:21.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 106, 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.631 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:21.632 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:21.635 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:21.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:21.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:21.639 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.640 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:38:21.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:21.642 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:21.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff28038740 msgr2=0x7eff2803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff28038740 0x7eff2803abf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7eff3c076600 tx=0x7eff34037000 comp rx=0 tx=0).stop 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 10 monclient: shutdown 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 msgr2=0x7eff3c141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 --2- 172.21.15.153:0/1631427096 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff3c1318f0 0x7eff3c141800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7eff3000d890 tx=0x7eff30008040 comp rx=0 tx=0).stop 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff3bfff640 1 -- 172.21.15.153:0/1631427096 reap_dead start 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff3bfff640 1 -- 172.21.15.153:0/1631427096 reap_dead start 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 shutdown_connections 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 >> 172.21.15.153:0/1631427096 conn(0x7eff3c067110 msgr2=0x7eff3c0a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 shutdown_connections 2024-09-15T03:38:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:21.618+0000 7eff41f9d640 1 -- 172.21.15.153:0/1631427096 wait complete. 2024-09-15T03:38:21.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:38:21.645 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:38:21.645 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-15T03:38:21.645 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:38:21.645 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:38:21.645 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:38:21.645 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:38:21.663 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:38:21.663 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T03:38:21.663 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T03:38:21.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:38:36.665 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T03:38:36.665 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T03:38:36.665 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:38:36.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:38:36.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.768+0000 7f9984f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:36.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.768+0000 7f9984f5d640 10 monclient: build_initial_monmap 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.768+0000 7f9984f5d640 10 monclient: monmap: 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:36.769646+0000 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:36.769646+0000 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:36.770 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 Processor -- start 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 -- start start 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient: init 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f99800975f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f99800979c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f998009bf50 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:36.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998009d700 con 0x7f998009bf50 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998009d1b0 con 0x7f99800979c0 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998009d420 con 0x7f99800975f0 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.769+0000 7f9984f5d640 10 monclient(hunting): authenticate will time out at 1314.408997s 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36082/0 (socket says 172.21.15.153:36082) 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 learned_addr learned my addr 172.21.15.153:0/873491973 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 10 monclient(hunting): get_auth_request con 0x7f99800975f0 auth_method 0 2024-09-15T03:38:36.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:36.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:36.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:36.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99800979c0 auth_method 0 2024-09-15T03:38:36.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:36.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:36.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.770+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.772+0000 7f997ed9d640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.153:6789/0 conn(0x7f998009bf50 legacy=0x7f998009c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99800979c0 auth_method 0 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:38:36.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 10 monclient(hunting): get_auth_request con 0x7f99800975f0 auth_method 0 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:38:36.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:36.971+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:38:37.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 10 monclient(hunting): get_auth_request con 0x7f99800975f0 auth_method 0 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99800979c0 auth_method 0 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:37.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:38:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:38:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:37.372+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:38:38.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 10 monclient(hunting): get_auth_request con 0x7f99800975f0 auth_method 0 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99800979c0 auth_method 0 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:38:38.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:38:38.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:38.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:38:38.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:38.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:38:38.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:38:38.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:38.173+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:38:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient(hunting): tick 2024-09-15T03:38:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 monclient(hunting): continuing hunt 2024-09-15T03:38:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:38:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9980095ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9980095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 msgr2=0x7f9980097d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.153:3300/0 conn(0x7f99800979c0 0x7f9980097d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.153:6789/0 conn(0x7f998009bf50 legacy=0x7f998009c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient: _add_conns ranks=[4,1] 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient(hunting): picked mon.noname-c con 0x7f9960003720 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9960006240 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998009d9e0 con 0x7f9960006240 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998009d1b0 con 0x7f9960003720 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997cd99640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:39.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.181:6789/0 conn(0x7f9960006240 legacy=0x7f99600086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:38:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:39.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.770+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:39.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.971+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.181:6789/0 conn(0x7f9960006240 legacy=0x7f99600086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:38:39.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.971+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:39.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:39.971+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:38:40.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:40.372+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.181:6789/0 conn(0x7f9960006240 legacy=0x7f99600086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:38:40.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:40.372+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:40.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:40.372+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:38:41.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:41.173+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.181:6789/0 conn(0x7f9960006240 legacy=0x7f99600086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:38:41.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:41.173+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:41.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:41.173+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:38:42.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): tick 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 monclient(hunting): continuing hunt 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.181:6789/0 conn(0x7f9960006240 legacy=0x7f99600086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): picked mon.noname-c con 0x7f9960003720 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997ed9d640 1 -- 172.21.15.153:0/873491973 reap_dead start 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9960009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:42.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): picked mon.noname-b con 0x7f99800975f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99600066c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996000afe0 con 0x7f99600066c0 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998009d1b0 con 0x7f9960003720 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 1 -- 172.21.15.153:0/873491973 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998009d420 con 0x7f99800975f0 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997cd99640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997e59c640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f997e59c640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9960009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99800975f0 auth_method 0 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.770+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 10 monclient(hunting): handle_auth_done global_id 124340 payload 274 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 msgr2=0x7f9960005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> v1:172.21.15.153:6789/0 conn(0x7f99600066c0 legacy=0x7f996000a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 1 monclient: found mon.noname-b 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:42.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:42.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.771+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998009d9e0 con 0x7f99800975f0 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f9975d9b640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9960009f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f9975d9b640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9960009f60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9968006fd0 tx=0x7f9968008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a395801149c26ce6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 1 -- 172.21.15.153:0/873491973 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996802bc80 con 0x7f99800975f0 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: dump: 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.780 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 1 -- 172.21.15.153:0/873491973 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996802f070 con 0x7f99800975f0 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f9984f5d640 5 monclient: authenticate success, global_id 124340 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f9984f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:42.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f9984f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 1 -- 172.21.15.153:0/873491973 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968035650 con 0x7f99800975f0 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f9984f5d640 4 set_mon_vals no callback set 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.774+0000 7f997dd9b640 10 monclient: dump: 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:42.782 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 10 monclient: shutdown 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 -- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 msgr2=0x7f9960009f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9960009f60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9968006fd0 tx=0x7f9968008040 comp rx=0 tx=0).stop 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 -- 172.21.15.153:0/873491973 shutdown_connections 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 --2- 172.21.15.153:0/873491973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99800975f0 0x7f9960009f60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 --2- 172.21.15.153:0/873491973 >> v2:172.21.15.181:3300/0 conn(0x7f9960003720 0x7f9960005b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 -- 172.21.15.153:0/873491973 >> 172.21.15.153:0/873491973 conn(0x7f99800678a0 msgr2=0x7f99800a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 -- 172.21.15.153:0/873491973 shutdown_connections 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.775+0000 7f9984f5d640 1 -- 172.21.15.153:0/873491973 wait complete. 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.776+0000 7f9984f5d640 10 monclient: build_initial_monmap 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.776+0000 7f9984f5d640 10 monclient: monmap: 2024-09-15T03:38:42.783 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.776+0000 7f9984f5d640 1 Processor -- start 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.776+0000 7f9984f5d640 1 -- start start 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.776+0000 7f9984f5d640 10 monclient: init 2024-09-15T03:38:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9960003720 0x7f9980141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f9960003720 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 0x7f9980131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f99600066c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99800975f0 0x7f9980132160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f99800975f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998009a5e0 con 0x7f99800975f0 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980098460 con 0x7f9960003720 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9980141f00 con 0x7f99600066c0 2024-09-15T03:38:42.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9984f5d640 10 monclient(hunting): authenticate will time out at 1320.416271s 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 0x7f9980131c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997e59c640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9960003720 msgr2=0x7f9980141060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 0x7f9980131c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36136/0 (socket says 172.21.15.153:36136) 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 -- 172.21.15.153:0/4065170154 learned_addr learned my addr 172.21.15.153:0/4065170154 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99600066c0 auth_method 0 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997e59c640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9960003720 0x7f9980141060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99800975f0 0x7f9980132160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): get_auth_request con 0x7f99800975f0 auth_method 0 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:42.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f997ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient(hunting): handle_auth_done global_id 124343 payload 824 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 -- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9960003720 msgr2=0x7f9980141060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9960003720 0x7f9980141060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 -- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99800975f0 msgr2=0x7f9980132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99800975f0 0x7f9980132160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:42.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:38:42.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 monclient: found mon.noname-c 2024-09-15T03:38:42.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:42.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:42.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.777+0000 7f9975d9b640 1 -- 172.21.15.153:0/4065170154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996800e040 con 0x7f99600066c0 2024-09-15T03:38:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9975d9b640 1 -- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 msgr2=0x7f9980131c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9975d9b640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 0x7f9980131c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9968000c00 tx=0x7f9968037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968035610 con 0x7f99600066c0 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: dump: 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:42.793 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: _finish_auth 0 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996803e600 con 0x7f99600066c0 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 5 monclient: authenticate success, global_id 124343 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 10 monclient: _renew_subs 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998012de30 con 0x7f99600066c0 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996803e9c0 con 0x7f99600066c0 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9976ffd640 10 monclient: dump: 2024-09-15T03:38:42.794 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f997d59a640 4 set_mon_vals no callback set 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 10 monclient: _renew_subs 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:42.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.780+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998012e3f0 con 0x7f99600066c0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f99680493b0 con 0x7f99600066c0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9976ffd640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9960039940 0x7f996003bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9968000e40 con 0x7f99600066c0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9984f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9984f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9984f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.781+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998004bf60 con 0x7f99600066c0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.782+0000 7f997e59c640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9960039940 0x7f996003bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.782+0000 7f997e59c640 10 monclient: get_auth_request con 0x7f9960039940 auth_method 0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.784+0000 7f997e59c640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9960039940 0x7f996003bdf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9970006c90 tx=0x7f9970008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.786+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9968049660 con 0x7f99600066c0 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.786+0000 7f9976ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:42.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:42.786+0000 7f9976ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.004+0000 7f9984f5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:38:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.004+0000 7f9984f5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.004+0000 7f9984f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.004+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f998012cfa0 con 0x7f99600066c0 2024-09-15T03:38:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.005+0000 7f9976ffd640 1 -- 172.21.15.153:0/4065170154 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9968045030 con 0x7f99600066c0 2024-09-15T03:38:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.005+0000 7f9976ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:38:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.005+0000 7f9976ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:43.006 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":108,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9960039940 msgr2=0x7f996003bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9960039940 0x7f996003bdf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9970006c90 tx=0x7f9970008040 comp rx=0 tx=0).stop 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 10 monclient: shutdown 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 msgr2=0x7f9980131c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 0x7f9980131c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9968000c00 tx=0x7f9968037000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 shutdown_connections 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9960039940 0x7f996003bdf0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99800975f0 0x7f9980132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99600066c0 0x7f9980131c20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.006+0000 7f9984f5d640 1 --2- 172.21.15.153:0/4065170154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9960003720 0x7f9980141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.007+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 >> 172.21.15.153:0/4065170154 conn(0x7f99800678a0 msgr2=0x7f99801329f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.007+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 shutdown_connections 2024-09-15T03:38:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.007+0000 7f9984f5d640 1 -- 172.21.15.153:0/4065170154 wait complete. 2024-09-15T03:38:43.018 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:38:43.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:38:43.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.111+0000 7fa7a324f640 10 monclient: get_monmap_and_config 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.111+0000 7fa7a324f640 10 monclient: build_initial_monmap 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.111+0000 7fa7a324f640 10 monclient: monmap: 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:43.112323+0000 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:43.112323+0000 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 Processor -- start 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 -- start start 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient: init 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa79c06eb80 0x7fa79c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): picked mon.noname-c con 0x7fa79c06eb80 addr v2:172.21.15.181:3300/0 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c09c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): picked mon.noname-b con 0x7fa79c068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa79c06f490 addr v1:172.21.15.153:6789/0 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa79c069b80 con 0x7fa79c06f490 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79c099f90 con 0x7fa79c06eb80 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c069e00 con 0x7fa79c068a20 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.112+0000 7fa7a324f640 10 monclient(hunting): authenticate will time out at 1320.751920s 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa7a0feb640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7fa79c06eb80 msgr2=0x7fa79c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa7a17ec640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa79c06f490 0x7fa79c09d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:58632/0 (socket says 172.21.15.153:58632) 2024-09-15T03:38:43.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa7a17ec640 1 -- 172.21.15.153:0/3027963789 learned_addr learned my addr 172.21.15.153:0/3027963789 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa7a0feb640 1 --2- 172.21.15.153:0/3027963789 >> v2:172.21.15.181:3300/0 conn(0x7fa79c06eb80 0x7fa79c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 1 --2- 172.21.15.153:0/3027963789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c09c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): get_auth_request con 0x7fa79c068a20 auth_method 0 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient(hunting): handle_auth_done global_id 124346 payload 274 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 1 -- 172.21.15.153:0/3027963789 >> v2:172.21.15.181:3300/0 conn(0x7fa79c06eb80 msgr2=0x7fa79c06ef50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 1 --2- 172.21.15.153:0/3027963789 >> v2:172.21.15.181:3300/0 conn(0x7fa79c06eb80 0x7fa79c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 1 -- 172.21.15.153:0/3027963789 >> v1:172.21.15.153:6789/0 conn(0x7fa79c06f490 legacy=0x7fa79c09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:43.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient: _finish_hunting 0 2024-09-15T03:38:43.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 1 monclient: found mon.noname-b 2024-09-15T03:38:43.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:43.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:43.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.113+0000 7fa793fff640 1 -- 172.21.15.153:0/3027963789 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c09ca80 con 0x7fa79c068a20 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa793fff640 1 -- 172.21.15.153:0/3027963789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 msgr2=0x7fa79c09c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa793fff640 1 --2- 172.21.15.153:0/3027963789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c09c540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009880 tx=0x7fa78c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89bb4953a6ab4575 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 1 -- 172.21.15.153:0/3027963789 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c032440 con 0x7fa79c068a20 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: dump: 2024-09-15T03:38:43.123 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 1 -- 172.21.15.153:0/3027963789 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c0325a0 con 0x7fa79c068a20 2024-09-15T03:38:43.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7a324f640 5 monclient: authenticate success, global_id 124346 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7a324f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 1 -- 172.21.15.153:0/3027963789 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c032940 con 0x7fa79c068a20 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7a324f640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7a324f640 4 set_mon_vals no callback set 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.116+0000 7fa7937fe640 10 monclient: dump: 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.125 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 10 monclient: shutdown 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 -- 172.21.15.153:0/3027963789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 msgr2=0x7fa79c09c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 --2- 172.21.15.153:0/3027963789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c09c540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009880 tx=0x7fa78c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 -- 172.21.15.153:0/3027963789 shutdown_connections 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 --2- 172.21.15.153:0/3027963789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c09c540 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 --2- 172.21.15.153:0/3027963789 >> v2:172.21.15.181:3300/0 conn(0x7fa79c06eb80 0x7fa79c06ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 -- 172.21.15.153:0/3027963789 >> 172.21.15.153:0/3027963789 conn(0x7fa79c067cd0 msgr2=0x7fa79c0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:43.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 -- 172.21.15.153:0/3027963789 shutdown_connections 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 -- 172.21.15.153:0/3027963789 wait complete. 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 10 monclient: build_initial_monmap 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 10 monclient: monmap: 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:43.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.117+0000 7fa7a324f640 1 Processor -- start 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 -- start start 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient: init 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c1410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): picked mon.noname-c con 0x7fa79c068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c06eb80 0x7fa79c133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): picked mon.noname-a con 0x7fa79c06eb80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c06f490 0x7fa79c11d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): picked mon.noname-b con 0x7fa79c06f490 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c121180 con 0x7fa79c06eb80 2024-09-15T03:38:43.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c11f3e0 con 0x7fa79c06f490 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c120ea0 con 0x7fa79c068a20 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a324f640 10 monclient(hunting): authenticate will time out at 1320.757584s 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c1410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c06eb80 0x7fa79c133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c06eb80 0x7fa79c133d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48534/0 (socket says 172.21.15.153:48534) 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c1410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36154/0 (socket says 172.21.15.153:36154) 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 1 -- 172.21.15.153:0/2872036107 learned_addr learned my addr 172.21.15.153:0/2872036107 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): get_auth_request con 0x7fa79c06eb80 auth_method 0 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): get_auth_request con 0x7fa79c068a20 auth_method 0 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a17ec640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c06f490 msgr2=0x7fa79c11d940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a17ec640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c06f490 0x7fa79c11d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient(hunting): handle_auth_done global_id 124349 payload 824 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c06f490 msgr2=0x7fa79c11d940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c06f490 0x7fa79c11d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c06eb80 msgr2=0x7fa79c133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c06eb80 0x7fa79c133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 10 monclient: _finish_hunting 0 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.118+0000 7fa7a0feb640 1 monclient: found mon.noname-c 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.119+0000 7fa7a0feb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.119+0000 7fa7a0feb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.119+0000 7fa7a0feb640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c009390 con 0x7fa79c068a20 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.119+0000 7fa793fff640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c06eb80 0x7fa79c133d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.121+0000 7fa7a0feb640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 msgr2=0x7fa79c1410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.121+0000 7fa7a0feb640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c1410c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa79800b5b0 tx=0x7fa7980090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.121+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:38:43.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798003d90 con 0x7fa79c068a20 2024-09-15T03:38:43.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:38:43.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: dump: 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: _finish_auth 0 2024-09-15T03:38:43.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 5 monclient: authenticate success, global_id 124349 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa798006150 con 0x7fa79c068a20 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 10 monclient: _renew_subs 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79c121460 con 0x7fa79c068a20 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798006510 con 0x7fa79c068a20 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa790ff9640 10 monclient: dump: 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.137 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 10 monclient: _renew_subs 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79c141ee0 con 0x7fa79c068a20 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.123+0000 7fa792ffd640 4 set_mon_vals no callback set 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa7a324f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa7a324f640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 0x7fa79c11de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa7980103e0 con 0x7fa79c068a20 2024-09-15T03:38:43.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c065e00 con 0x7fa79c11ffb0 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 0x7fa79c11de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient: get_auth_request con 0x7fa79c11ffb0 auth_method 0 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa790ff9640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa788038740 0x7fa78803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa79804bda0 con 0x7fa79c068a20 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 10 monclient(hunting): handle_auth_done global_id 134184 payload 824 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa793fff640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa79c04bf60 con 0x7fa79c11ffb0 2024-09-15T03:38:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa7a17ec640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa788038740 0x7fa78803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.124+0000 7fa7a17ec640 10 monclient: get_auth_request con 0x7fa788038740 auth_method 0 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.126+0000 7fa793fff640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 msgr2=0x7fa79c11de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.126+0000 7fa793fff640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 0x7fa79c11de80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa78c035040 tx=0x7fa78c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.126+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c03d480 con 0x7fa79c11ffb0 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.126+0000 7fa790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.126+0000 7fa790ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.126+0000 7fa790ff9640 10 monclient: dump: 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa78c03d5e0 con 0x7fa79c11ffb0 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 msgr2=0x7fa79c11de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 0x7fa79c11de80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa78c035040 tx=0x7fa78c032000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 msgr2=0x7fa79c11de80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa790ff9640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c11ffb0 0x7fa79c11de80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.127+0000 7fa7a17ec640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa788038740 0x7fa78803abf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa79c06f8d0 tx=0x7fa794004fc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.132+0000 7fa7a324f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.132+0000 7fa7a324f640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 0x7fa79c11e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.132+0000 7fa7a324f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.132+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c1215f0 con 0x7fa79c1342c0 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.132+0000 7fa793fff640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 0x7fa79c11e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient: get_auth_request con 0x7fa79c1342c0 auth_method 0 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 10 monclient(hunting): handle_auth_done global_id 134187 payload 824 2024-09-15T03:38:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.133+0000 7fa793fff640 1 -- 172.21.15.153:0/2872036107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa79c1212f0 con 0x7fa79c1342c0 2024-09-15T03:38:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa793fff640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 msgr2=0x7fa79c11e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa793fff640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 0x7fa79c11e3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa78c003080 tx=0x7fa78c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c049680 con 0x7fa79c1342c0 2024-09-15T03:38:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 10 monclient: dump: 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fa78c032d70 con 0x7fa79c1342c0 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 msgr2=0x7fa79c11e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.135+0000 7fa790ff9640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 0x7fa79c11e3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa78c003080 tx=0x7fa78c03d000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa790ff9640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 msgr2=0x7fa79c11e3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa790ff9640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c1342c0 0x7fa79c11e3c0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa78c003080 tx=0x7fa78c03d000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 108, 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:43.149 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:43.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:43.151 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.154 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.155 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:43.157 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:43.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa788038740 msgr2=0x7fa78803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa788038740 0x7fa78803abf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa79c06f8d0 tx=0x7fa794004fc0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 10 monclient: shutdown 2024-09-15T03:38:43.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 msgr2=0x7fa79c1410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 --2- 172.21.15.153:0/2872036107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c068a20 0x7fa79c1410c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa79800b5b0 tx=0x7fa7980090f0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a17ec640 1 -- 172.21.15.153:0/2872036107 reap_dead start 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a17ec640 1 -- 172.21.15.153:0/2872036107 reap_dead start 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 shutdown_connections 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 >> 172.21.15.153:0/2872036107 conn(0x7fa79c067cd0 msgr2=0x7fa79c0a2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 shutdown_connections 2024-09-15T03:38:43.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.136+0000 7fa7a324f640 1 -- 172.21.15.153:0/2872036107 wait complete. 2024-09-15T03:38:43.159 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:38:43.159 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:38:43.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:38:43.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.252+0000 7feca535d640 10 monclient: get_monmap_and_config 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.252+0000 7feca535d640 10 monclient: build_initial_monmap 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.252+0000 7feca535d640 10 monclient: monmap: 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:38:43.253402+0000 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:38:43.253402+0000 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 Processor -- start 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 -- start start 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient: init 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 0x7feca0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): picked mon.noname-a con 0x7feca01318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feca009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): picked mon.noname-b con 0x7feca0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca0131cc0 con 0x7feca009ea50 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca009f530 con 0x7feca01318f0 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca009fbf0 con 0x7feca0136c20 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7feca535d640 10 monclient(hunting): authenticate will time out at 1320.892809s 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 0x7feca0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec967fc640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7feca009ea50 legacy=0x7feca009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 0x7feca0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48570/0 (socket says 172.21.15.153:48570) 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36168/0 (socket says 172.21.15.153:36168) 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 1 -- 172.21.15.153:0/2143295261 learned_addr learned my addr 172.21.15.153:0/2143295261 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 10 monclient(hunting): get_auth_request con 0x7feca01318f0 auth_method 0 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9f7fe640 10 monclient(hunting): get_auth_request con 0x7feca0136c20 auth_method 0 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.253+0000 7fec9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_done global_id 124352 payload 274 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 1 -- 172.21.15.153:0/2143295261 >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 msgr2=0x7feca0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 1 --2- 172.21.15.153:0/2143295261 >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 0x7feca0131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 1 -- 172.21.15.153:0/2143295261 >> v1:172.21.15.181:6789/0 conn(0x7feca009ea50 legacy=0x7feca009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:38:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 1 monclient: found mon.noname-b 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9f7fe640 1 -- 172.21.15.153:0/2143295261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca009fe30 con 0x7feca0136c20 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.254+0000 7fec9effd640 1 --2- 172.21.15.153:0/2143295261 >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 0x7feca0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9f7fe640 1 -- 172.21.15.153:0/2143295261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 msgr2=0x7feca0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9f7fe640 1 --2- 172.21.15.153:0/2143295261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0139000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e60 tx=0x7fec98009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7db1f2a5649c9d84 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 1 -- 172.21.15.153:0/2143295261 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98007760 con 0x7feca0136c20 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: dump: 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 1 -- 172.21.15.153:0/2143295261 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9802d050 con 0x7feca0136c20 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 5 monclient: authenticate success, global_id 124352 2024-09-15T03:38:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 10 monclient: get_monmap_and_config success 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 4 set_mon_vals no callback set 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 1 -- 172.21.15.153:0/2143295261 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98007af0 con 0x7feca0136c20 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7fec9e7fc640 10 monclient: dump: 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 10 monclient: shutdown 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 -- 172.21.15.153:0/2143295261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 msgr2=0x7feca0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 --2- 172.21.15.153:0/2143295261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0139000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e60 tx=0x7fec98009780 comp rx=0 tx=0).stop 2024-09-15T03:38:43.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 -- 172.21.15.153:0/2143295261 shutdown_connections 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 --2- 172.21.15.153:0/2143295261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0139000 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 --2- 172.21.15.153:0/2143295261 >> v2:172.21.15.153:3300/0 conn(0x7feca01318f0 0x7feca0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 -- 172.21.15.153:0/2143295261 >> 172.21.15.153:0/2143295261 conn(0x7feca0067110 msgr2=0x7feca0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.257+0000 7feca535d640 1 -- 172.21.15.153:0/2143295261 shutdown_connections 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.258+0000 7feca535d640 1 -- 172.21.15.153:0/2143295261 wait complete. 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.258+0000 7feca535d640 10 monclient: build_initial_monmap 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.258+0000 7feca535d640 10 monclient: monmap: 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:38:43.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.258+0000 7feca535d640 1 Processor -- start 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.258+0000 7feca535d640 1 -- start start 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.258+0000 7feca535d640 10 monclient: init 2024-09-15T03:38:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient: _reopen_session rank -1 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 0x7feca011d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): picked mon.noname-a con 0x7feca009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feca01318f0 0x7feca01412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): picked mon.noname-b con 0x7feca01318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): picked mon.noname-c con 0x7feca0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca011cd60 con 0x7feca009ea50 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca009bf10 con 0x7feca01318f0 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca009dda0 con 0x7feca0136c20 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): _renew_subs 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7feca535d640 10 monclient(hunting): authenticate will time out at 1320.898386s 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 0x7feca011d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 0x7feca011d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48572/0 (socket says 172.21.15.153:48572) 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 -- 172.21.15.153:0/1435632180 learned_addr learned my addr 172.21.15.153:0/1435632180 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): get_auth_request con 0x7feca009ea50 auth_method 0 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec967fc640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feca01318f0 msgr2=0x7feca01412c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): get_auth_request con 0x7feca0136c20 auth_method 0 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec967fc640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feca01318f0 0x7feca01412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient(hunting): handle_auth_done global_id 134193 payload 824 2024-09-15T03:38:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 msgr2=0x7feca0141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca0136c20 0x7feca0141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feca01318f0 msgr2=0x7feca01412c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:38:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feca01318f0 0x7feca01412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient: _finish_hunting 0 2024-09-15T03:38:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 monclient: found mon.noname-a 2024-09-15T03:38:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:38:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:38:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.259+0000 7fec9effd640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec90009450 con 0x7feca009ea50 2024-09-15T03:38:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.261+0000 7fec9effd640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 msgr2=0x7feca011d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec9effd640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 0x7feca011d9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fec9000ece0 tx=0x7fec900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec90009730 con 0x7feca009ea50 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: dump: 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: _finish_auth 0 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9000cdd0 con 0x7feca009ea50 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 5 monclient: authenticate success, global_id 134193 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec900136a0 con 0x7feca009ea50 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:38:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec977fe640 10 monclient: dump: 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 10 monclient: _renew_subs 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec98009a20 con 0x7feca009ea50 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7fec9dffb640 4 set_mon_vals no callback set 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 10 monclient: _renew_subs 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.262+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca009b110 con 0x7feca009ea50 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fec90013a10 con 0x7feca009ea50 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7feca535d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7feca535d640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 0x7feca005e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca0065e00 con 0x7feca009a650 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 0x7feca005e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient: get_auth_request con 0x7feca009a650 auth_method 0 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec977fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec800386f0 0x7fec8003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fec9004c9f0 con 0x7feca009ea50 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 10 monclient(hunting): handle_auth_done global_id 124358 payload 824 2024-09-15T03:38:43.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec967fc640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feca004bf60 con 0x7feca009a650 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec9f7fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec800386f0 0x7fec8003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.263+0000 7fec9f7fe640 10 monclient: get_auth_request con 0x7fec800386f0 auth_method 0 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.265+0000 7fec967fc640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 msgr2=0x7feca005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.265+0000 7fec967fc640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 0x7feca005e310 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feca0132410 tx=0x7fec880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.265+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec880050f0 con 0x7feca009a650 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.265+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.265+0000 7fec977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.265+0000 7fec977fe640 10 monclient: dump: 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.285 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fec88005250 con 0x7feca009a650 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 msgr2=0x7feca005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 0x7feca005e310 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feca0132410 tx=0x7fec880091a0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 msgr2=0x7feca005e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec977fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca009a650 0x7feca005e310 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feca0132410 tx=0x7fec880091a0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.266+0000 7fec9f7fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec800386f0 0x7fec8003aba0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7feca006f600 tx=0x7fec98037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.271+0000 7feca535d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.271+0000 7feca535d640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 0x7feca00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.271+0000 7feca535d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.271+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca011b7d0 con 0x7feca005e850 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.271+0000 7fec967fc640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 0x7feca00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient: get_auth_request con 0x7feca005e850 auth_method 0 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:38:43.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 10 monclient(hunting): handle_auth_done global_id 124361 payload 824 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.272+0000 7fec967fc640 1 -- 172.21.15.153:0/1435632180 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feca0135e70 con 0x7feca005e850 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec967fc640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 msgr2=0x7feca00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec967fc640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 0x7feca00298c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feca0132410 tx=0x7fec88015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88010e40 con 0x7feca005e850 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 10 monclient: dump: 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:38:43.288 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7fec88019400 con 0x7feca005e850 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 msgr2=0x7feca00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 0x7feca00298c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feca0132410 tx=0x7fec88015000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 msgr2=0x7feca00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:38:43.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.274+0000 7fec977fe640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca005e850 0x7feca00298c0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feca0132410 tx=0x7fec88015000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 108, 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:38:43.290 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:38:43.291 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:38:43.292 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:38:43.293 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:38:43.294 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:38:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:38:43.297 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.298 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.299 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.300 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:38:43.301 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.302 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:38:43.303 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:38:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:38:43.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec800386f0 msgr2=0x7fec8003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fec800386f0 0x7fec8003aba0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7feca006f600 tx=0x7fec98037000 comp rx=0 tx=0).stop 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 10 monclient: shutdown 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 msgr2=0x7feca011d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 --2- 172.21.15.153:0/1435632180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca009ea50 0x7feca011d9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fec9000ece0 tx=0x7fec900091d0 comp rx=0 tx=0).stop 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7fec9f7fe640 1 -- 172.21.15.153:0/1435632180 reap_dead start 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7fec9f7fe640 1 -- 172.21.15.153:0/1435632180 reap_dead start 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 shutdown_connections 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 >> 172.21.15.153:0/1435632180 conn(0x7feca0067110 msgr2=0x7feca00a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 shutdown_connections 2024-09-15T03:38:43.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:38:43.275+0000 7feca535d640 1 -- 172.21.15.153:0/1435632180 wait complete. 2024-09-15T03:38:43.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:38:43.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:39:03.307 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:39:03.307 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:39:03.307 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:39:03.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:39:03.311 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:39:03.311 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T03:39:03.311 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T03:39:03.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:39:18.312 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T03:39:18.313 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T03:39:18.313 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:39:18.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:39:18.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.415+0000 7f45bac57640 10 monclient: get_monmap_and_config 2024-09-15T03:39:18.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.415+0000 7f45bac57640 10 monclient: build_initial_monmap 2024-09-15T03:39:18.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.415+0000 7f45bac57640 10 monclient: monmap: 2024-09-15T03:39:18.420 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:18.420 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:18.420 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:18.416313+0000 2024-09-15T03:39:18.420 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:18.416313+0000 2024-09-15T03:39:18.420 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 1 Processor -- start 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 1 -- start start 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 10 monclient: init 2024-09-15T03:39:18.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 10 monclient(hunting): picked mon.noname-a con 0x7f45b4096c60 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.416+0000 7f45bac57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 10 monclient(hunting): picked mon.noname-b con 0x7f45b4097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45b412e5f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b409d2e0 con 0x7f45b412e5f0 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b409ac00 con 0x7f45b4096c60 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:18.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b409d6b0 con 0x7f45b4097030 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34312/0 (socket says 172.21.15.153:34312) 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51798/0 (socket says 172.21.15.153:51798) 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 learned_addr learned my addr 172.21.15.153:0/2849286334 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45bac57640 10 monclient(hunting): authenticate will time out at 1356.056118s 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45b4097030 auth_method 0 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:18.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 10 monclient(hunting): get_auth_request con 0x7f45b4096c60 auth_method 0 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:18.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.417+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:18.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.419+0000 7f45b91f4640 1 -- 172.21.15.153:0/2849286334 >> v1:172.21.15.181:6789/0 conn(0x7f45b412e5f0 legacy=0x7f45b412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:18.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 10 monclient(hunting): get_auth_request con 0x7f45b4096c60 auth_method 0 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45b4097030 auth_method 0 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:18.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:18.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:18.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:18.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:18.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:18.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:18.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:18.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:18.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:39:18.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:18.618+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:39:19.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 10 monclient(hunting): get_auth_request con 0x7f45b4096c60 auth_method 0 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45b4097030 auth_method 0 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:19.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:19.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.019+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 10 monclient(hunting): get_auth_request con 0x7f45b4096c60 auth_method 0 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:19.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45b4097030 auth_method 0 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:19.820+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:39:21.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient(hunting): tick 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 msgr2=0x7f45b4097430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b4097030 0x7f45b4097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 msgr2=0x7f45b409a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.153:3300/0 conn(0x7f45b4096c60 0x7f45b409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 -- 172.21.15.153:0/2849286334 >> v1:172.21.15.181:6789/0 conn(0x7f45b412e5f0 legacy=0x7f45b412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient: _add_conns ranks=[0,4] 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45a0005930 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f45a0009630 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:21.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 -- 172.21.15.153:0/2849286334 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b40992f0 con 0x7f45a0005930 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 1 -- 172.21.15.153:0/2849286334 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b409ac00 con 0x7f45a0009630 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45aa7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45a0009630 auth_method 0 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:21.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.417+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:21.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.419+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> v1:172.21.15.153:6789/0 conn(0x7f45a0005930 legacy=0x7f45a0007d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45a0009630 auth_method 0 2024-09-15T03:39:21.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:21.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:21.618+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:39:22.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:22.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45a0009630 auth_method 0 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:22.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.019+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.820+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.820+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45a0009630 auth_method 0 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.820+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.820+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.820+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.821+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.821+0000 7f45abfff640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.821+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:22.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:22.821+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:39:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238530116 0 0) 0x7f45b40992f0 con 0x7f45a0005930 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): my global_id is 134196 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c0034e0 con 0x7f45a0005930 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097603359 0 0) 0x7f459c0034e0 con 0x7f45a0005930 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 msgr2=0x7f45a000bae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 --2- 172.21.15.153:0/2849286334 >> v2:172.21.15.181:3300/0 conn(0x7f45a0009630 0x7f45a000bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:39:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a000d1f0 con 0x7f45a0005930 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: _finish_auth 0 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f45b000d480 con 0x7f45a0005930 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: dump: 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 >> v1:172.21.15.153:6789/0 conn(0x7f45a0005930 legacy=0x7f45a0007d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:39:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45b91f4640 1 -- 172.21.15.153:0/2849286334 reap_dead start 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f459c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): picked mon.c con 0x7f459c003a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f459c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): picked mon.a con 0x7f459c004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f459c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): picked mon.b con 0x7f459c005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b409ac00 con 0x7f459c004780 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c00c260 con 0x7f459c005330 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c00c420 con 0x7f459c003a20 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45ab7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f459c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45b89f3640 10 monclient(hunting): get_auth_request con 0x7f459c003a20 auth_method 0 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45b89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45b89f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45b89f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f459c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f459c004780 auth_method 0 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.032+0000 7f45abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f459c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): get_auth_request con 0x7f459c005330 auth_method 0 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b91f4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient(hunting): handle_auth_done global_id 134196 payload 293 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 msgr2=0x7f459c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f459c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 msgr2=0x7f459c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f459c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient: _finish_hunting 0 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 1 monclient: found mon.c 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0034e0 con 0x7f459c003a20 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 10 monclient: _finish_auth 0 2024-09-15T03:39:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45b89f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45bac57640 5 monclient: authenticate success, global_id 134196 2024-09-15T03:39:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45bac57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:23.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.033+0000 7f45abfff640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f459c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45b89f3640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 msgr2=0x7f459c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45b89f3640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f459c003e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f45b0006fd0 tx=0x7f45b0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f5065f1d1bc9c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0034410 con 0x7f459c003a20 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: dump: 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45bac57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b002e030 con 0x7f459c003a20 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45bac57640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 1 -- 172.21.15.153:0/2849286334 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b00347c0 con 0x7f459c003a20 2024-09-15T03:39:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45bac57640 4 set_mon_vals no callback set 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.036+0000 7f45ab7fe640 10 monclient: dump: 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 10 monclient: shutdown 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 -- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 msgr2=0x7f459c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f459c003e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f45b0006fd0 tx=0x7f45b0008040 comp rx=0 tx=0).stop 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 -- 172.21.15.153:0/2849286334 shutdown_connections 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f459c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f459c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 --2- 172.21.15.153:0/2849286334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f459c003e10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 -- 172.21.15.153:0/2849286334 >> 172.21.15.153:0/2849286334 conn(0x7f45b40672b0 msgr2=0x7f45b40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 -- 172.21.15.153:0/2849286334 shutdown_connections 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.037+0000 7f45bac57640 1 -- 172.21.15.153:0/2849286334 wait complete. 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.038+0000 7f45bac57640 10 monclient: build_initial_monmap 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.038+0000 7f45bac57640 10 monclient: monmap: 2024-09-15T03:39:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.038+0000 7f45bac57640 1 Processor -- start 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 -- start start 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient: init 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): picked mon.noname-c con 0x7f459c003a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f45b4119570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): picked mon.noname-a con 0x7f459c004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f45b4119ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): picked mon.noname-b con 0x7f459c005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b409add0 con 0x7f459c004780 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b412ef20 con 0x7f459c005330 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b411b560 con 0x7f459c003a20 2024-09-15T03:39:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45bac57640 10 monclient(hunting): authenticate will time out at 1360.678599s 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f45b4119570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34374/0 (socket says 172.21.15.153:34374) 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f45b4119570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51824/0 (socket says 172.21.15.153:51824) 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 -- 172.21.15.153:0/1920587243 learned_addr learned my addr 172.21.15.153:0/1920587243 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 10 monclient(hunting): get_auth_request con 0x7f459c003a20 auth_method 0 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f459c004780 auth_method 0 2024-09-15T03:39:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 msgr2=0x7f45b4119030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 msgr2=0x7f45b4119030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b91f4640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f45b4119ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b91f4640 10 monclient(hunting): get_auth_request con 0x7f459c005330 auth_method 0 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b89f3640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b91f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b91f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.039+0000 7f45b91f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45b91f4640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 msgr2=0x7f45b4119ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45b91f4640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 msgr2=0x7f45b4119ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45b91f4640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f45b4119ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45b91f4640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f45b4119ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 10 monclient(hunting): handle_auth_done global_id 134199 payload 824 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 msgr2=0x7f45b4119030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 msgr2=0x7f45b4119ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f45b4119ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 1 monclient: found mon.noname-a 2024-09-15T03:39:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.040+0000 7f45abfff640 1 -- 172.21.15.153:0/1920587243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b411aea0 con 0x7f459c004780 2024-09-15T03:39:23.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45abfff640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 msgr2=0x7f45b4119570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:23.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45abfff640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f45b4119570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45a4006c90 tx=0x7f45a401b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:23.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a401e050 con 0x7f459c004780 2024-09-15T03:39:23.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:23.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: dump: 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 5 monclient: authenticate success, global_id 134199 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a401f040 con 0x7f459c004780 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 10 monclient: _renew_subs 2024-09-15T03:39:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b411c450 con 0x7f459c004780 2024-09-15T03:39:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4023630 con 0x7f459c004780 2024-09-15T03:39:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45a8ff9640 10 monclient: dump: 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45aaffd640 4 set_mon_vals no callback set 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 10 monclient: _renew_subs 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.042+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b4119ff0 con 0x7f459c004780 2024-09-15T03:39:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.043+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f45a4023a20 con 0x7f459c004780 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45a8ff9640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45a003b410 0x7f45a003d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45bac57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45bac57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45bac57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f45a404e450 con 0x7f459c004780 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b411c850 con 0x7f459c004780 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45b89f3640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45a003b410 0x7f45a003d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.044+0000 7f45b89f3640 10 monclient: get_auth_request con 0x7f45a003b410 auth_method 0 2024-09-15T03:39:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.047+0000 7f45b89f3640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45a003b410 0x7f45a003d8c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009010 tx=0x7f45b0039000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.049+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f45a40173e0 con 0x7f459c004780 2024-09-15T03:39:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.049+0000 7f45a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.049+0000 7f45a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45bac57640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:39:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45bac57640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45bac57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f45b411a2a0 con 0x7f459c004780 2024-09-15T03:39:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45a8ff9640 1 -- 172.21.15.153:0/1920587243 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f45a4025110 con 0x7f459c004780 2024-09-15T03:39:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.268+0000 7f45a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:23.270 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":110,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:39:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45a003b410 msgr2=0x7f45a003d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45a003b410 0x7f45a003d8c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009010 tx=0x7f45b0039000 comp rx=0 tx=0).stop 2024-09-15T03:39:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 10 monclient: shutdown 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 msgr2=0x7f45b4119570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f45b4119570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45a4006c90 tx=0x7f45a401b040 comp rx=0 tx=0).stop 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 shutdown_connections 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45a003b410 0x7f45a003d8c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f459c005330 0x7f45b4119ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459c004780 0x7f45b4119570 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 --2- 172.21.15.153:0/1920587243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459c003a20 0x7f45b4119030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 >> 172.21.15.153:0/1920587243 conn(0x7f45b40672b0 msgr2=0x7f45b409b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 shutdown_connections 2024-09-15T03:39:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:23.270+0000 7f45bac57640 1 -- 172.21.15.153:0/1920587243 wait complete. 2024-09-15T03:39:26.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.387+0000 7f78d1f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.387+0000 7f78d1f1d640 10 monclient: build_initial_monmap 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.387+0000 7f78d1f1d640 10 monclient: monmap: 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:26.388031+0000 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:26.388031+0000 2024-09-15T03:39:26.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:26.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 Processor -- start 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 -- start start 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient: init 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78cc131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f78cc09e910 0x7f78cc09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f78cc09e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78cc09f250 0x7f78cc138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f78cc09f250 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78cc09faf0 con 0x7f78cc131ab0 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc132590 con 0x7f78cc09f250 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc09fd30 con 0x7f78cc09e910 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78d1f1d640 10 monclient(hunting): authenticate will time out at 1364.027851s 2024-09-15T03:39:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.388+0000 7f78cbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78cc09f250 0x7f78cc138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78cc09f250 0x7f78cc138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38348/0 (socket says 172.21.15.153:38348) 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 -- 172.21.15.153:0/1229670213 learned_addr learned my addr 172.21.15.153:0/1229670213 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): get_auth_request con 0x7f78cc09f250 auth_method 0 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78c3fff640 1 --2- 172.21.15.153:0/1229670213 >> v2:172.21.15.181:3300/0 conn(0x7f78cc09e910 0x7f78cc09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78c3fff640 10 monclient(hunting): get_auth_request con 0x7f78cc09e910 auth_method 0 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient(hunting): handle_auth_done global_id 134202 payload 274 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 -- 172.21.15.153:0/1229670213 >> v2:172.21.15.181:3300/0 conn(0x7f78cc09e910 msgr2=0x7f78cc09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 --2- 172.21.15.153:0/1229670213 >> v2:172.21.15.181:3300/0 conn(0x7f78cc09e910 0x7f78cc09ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 -- 172.21.15.153:0/1229670213 >> v1:172.21.15.181:6789/0 conn(0x7f78cc131ab0 legacy=0x7f78cc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 monclient: found mon.noname-a 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.389+0000 7f78cbfff640 1 -- 172.21.15.153:0/1229670213 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc0a1110 con 0x7f78cc09f250 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78cbfff640 1 -- 172.21.15.153:0/1229670213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09f250 msgr2=0x7f78cc138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78cbfff640 1 --2- 172.21.15.153:0/1229670213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09f250 0x7f78cc138bc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78cc09bdb0 tx=0x7f78c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7249fc05e55ba936 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 1 -- 172.21.15.153:0/1229670213 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4007610 con 0x7f78cc09f250 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: dump: 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: _finish_auth 0 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:26.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 1 -- 172.21.15.153:0/1229670213 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c402d050 con 0x7f78cc09f250 2024-09-15T03:39:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 5 monclient: authenticate success, global_id 134202 2024-09-15T03:39:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 1 -- 172.21.15.153:0/1229670213 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c40079a0 con 0x7f78cc09f250 2024-09-15T03:39:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 4 set_mon_vals no callback set 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78caffd640 10 monclient: dump: 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 10 monclient: shutdown 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 -- 172.21.15.153:0/1229670213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09f250 msgr2=0x7f78cc138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/1229670213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09f250 0x7f78cc138bc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78cc09bdb0 tx=0x7f78c4009840 comp rx=0 tx=0).stop 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 -- 172.21.15.153:0/1229670213 shutdown_connections 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/1229670213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09f250 0x7f78cc138bc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/1229670213 >> v2:172.21.15.181:3300/0 conn(0x7f78cc09e910 0x7f78cc09ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 -- 172.21.15.153:0/1229670213 >> 172.21.15.153:0/1229670213 conn(0x7f78cc067d30 msgr2=0x7f78cc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.392+0000 7f78d1f1d640 1 -- 172.21.15.153:0/1229670213 shutdown_connections 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.393+0000 7f78d1f1d640 1 -- 172.21.15.153:0/1229670213 wait complete. 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.393+0000 7f78d1f1d640 10 monclient: build_initial_monmap 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.393+0000 7f78d1f1d640 10 monclient: monmap: 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.393+0000 7f78d1f1d640 1 Processor -- start 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.393+0000 7f78d1f1d640 1 -- start start 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.393+0000 7f78d1f1d640 10 monclient: init 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 0x7f78cc098340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f78cc09e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78cc09f250 0x7f78cc09a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f78cc09f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78cc131ab0 0x7f78cc096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f78cc131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc099a80 con 0x7f78cc09e910 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc099810 con 0x7f78cc131ab0 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc099d60 con 0x7f78cc09f250 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78d1f1d640 10 monclient(hunting): authenticate will time out at 1364.033274s 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 0x7f78cc098340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78cc09f250 0x7f78cc09a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 0x7f78cc098340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38362/0 (socket says 172.21.15.153:38362) 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 -- 172.21.15.153:0/4105254509 learned_addr learned my addr 172.21.15.153:0/4105254509 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): get_auth_request con 0x7f78cc09e910 auth_method 0 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): get_auth_request con 0x7f78cc09f250 auth_method 0 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cbfff640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78cc131ab0 0x7f78cc096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cbfff640 10 monclient(hunting): get_auth_request con 0x7f78cc131ab0 auth_method 0 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_done global_id 134205 payload 824 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 -- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78cc09f250 msgr2=0x7f78cc09a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78cc09f250 0x7f78cc09a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 -- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78cc131ab0 msgr2=0x7f78cc096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78cc131ab0 0x7f78cc096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 monclient: found mon.noname-a 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cb7fe640 1 -- 172.21.15.153:0/4105254509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c4009ae0 con 0x7f78cc09e910 2024-09-15T03:39:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.394+0000 7f78cbfff640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78cc131ab0 0x7f78cc096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.396+0000 7f78cb7fe640 1 -- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 msgr2=0x7f78cc098340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.396+0000 7f78cb7fe640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 0x7f78cc098340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00c5e0 tx=0x7f78bc007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc012840 con 0x7f78cc09e910 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: dump: 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: _finish_auth 0 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 5 monclient: authenticate success, global_id 134205 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc0129a0 con 0x7f78cc09e910 2024-09-15T03:39:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 10 monclient: _renew_subs 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc097b50 con 0x7f78cc09e910 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc012da0 con 0x7f78cc09e910 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 10 monclient: dump: 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.419 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 10 monclient: _renew_subs 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78ca7fc640 4 set_mon_vals no callback set 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc1311c0 con 0x7f78cc09e910 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.397+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f78bc011590 con 0x7f78cc09e910 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78d1f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78d1f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78d1f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc131410 con 0x7f78cc09e910 2024-09-15T03:39:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78c37fe640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78ac038740 0x7f78ac03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f78bc04c6f0 con 0x7f78cc09e910 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78c3fff640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78ac038740 0x7f78ac03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.398+0000 7f78c3fff640 10 monclient: get_auth_request con 0x7f78ac038740 auth_method 0 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.401+0000 7f78c3fff640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78ac038740 0x7f78ac03abf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0996d0 tx=0x7f78b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.402+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f78bc010ac0 con 0x7f78cc09e910 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.402+0000 7f78c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.402+0000 7f78c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78d1f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78d1f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78d1f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78cc04bf60 con 0x7f78cc09e910 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78c37fe640 1 -- 172.21.15.153:0/4105254509 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f78bc016080 con 0x7f78cc09e910 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:26.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.616+0000 7f78c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:26.618 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78ac038740 msgr2=0x7f78ac03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78ac038740 0x7f78ac03abf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0996d0 tx=0x7f78b4008040 comp rx=0 tx=0).stop 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 10 monclient: shutdown 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 msgr2=0x7f78cc098340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 0x7f78cc098340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00c5e0 tx=0x7f78bc007ed0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 shutdown_connections 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78ac038740 0x7f78ac03abf0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78cc131ab0 0x7f78cc096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78cc09f250 0x7f78cc09a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 --2- 172.21.15.153:0/4105254509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78cc09e910 0x7f78cc098340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 >> 172.21.15.153:0/4105254509 conn(0x7f78cc067d30 msgr2=0x7f78cc0a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:26.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 shutdown_connections 2024-09-15T03:39:26.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.618+0000 7f78d1f1d640 1 -- 172.21.15.153:0/4105254509 wait complete. 2024-09-15T03:39:26.630 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:39:26.631 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:39:26.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:39:26.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.724+0000 7f78a626f640 10 monclient: get_monmap_and_config 2024-09-15T03:39:26.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.724+0000 7f78a626f640 10 monclient: build_initial_monmap 2024-09-15T03:39:26.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.724+0000 7f78a626f640 10 monclient: monmap: 2024-09-15T03:39:26.725 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:26.725 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:26.725387+0000 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:26.725387+0000 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:26.726 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 1 Processor -- start 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 1 -- start start 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 10 monclient: init 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78a0096e10 0x7f78a009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.725+0000 7f78a626f640 10 monclient(hunting): picked mon.noname-a con 0x7f78a0096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f78a009aaa0 0x7f78a00971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 10 monclient(hunting): picked mon.noname-c con 0x7f78a009aaa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78a0097720 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a0099620 con 0x7f78a0097720 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a0099340 con 0x7f78a0096e10 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a0099900 con 0x7f78a009aaa0 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a626f640 10 monclient(hunting): authenticate will time out at 1364.365048s 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78a0096e10 0x7f78a009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78a0096e10 0x7f78a009a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38378/0 (socket says 172.21.15.153:38378) 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 1 -- 172.21.15.153:0/2709022249 learned_addr learned my addr 172.21.15.153:0/2709022249 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): get_auth_request con 0x7f78a0096e10 auth_method 0 2024-09-15T03:39:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f78a480c640 1 --1- 172.21.15.153:0/2709022249 >> v1:172.21.15.153:6789/0 conn(0x7f78a0097720 0x7f78a012e5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44862/0 (socket says 172.21.15.153:44862) 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f789ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 1 --2- 172.21.15.153:0/2709022249 >> v2:172.21.15.181:3300/0 conn(0x7f78a009aaa0 0x7f78a00971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): get_auth_request con 0x7f78a009aaa0 auth_method 0 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient(hunting): handle_auth_done global_id 144106 payload 274 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 1 -- 172.21.15.153:0/2709022249 >> v2:172.21.15.153:3300/0 conn(0x7f78a0096e10 msgr2=0x7f78a009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 1 --2- 172.21.15.153:0/2709022249 >> v2:172.21.15.153:3300/0 conn(0x7f78a0096e10 0x7f78a009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 1 -- 172.21.15.153:0/2709022249 >> v1:172.21.15.153:6789/0 conn(0x7f78a0097720 legacy=0x7f78a012e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 1 monclient: found mon.noname-c 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:39:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.726+0000 7f7897fff640 1 -- 172.21.15.153:0/2709022249 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0097b20 con 0x7f78a009aaa0 2024-09-15T03:39:26.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f7897fff640 1 -- 172.21.15.153:0/2709022249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 msgr2=0x7f78a00971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f7897fff640 1 --2- 172.21.15.153:0/2709022249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 0x7f78a00971e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78a009aff0 tx=0x7f78880090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c8b261b34b9d7da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 1 -- 172.21.15.153:0/2709022249 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788802ae00 con 0x7f78a009aaa0 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: dump: 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.742 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 1 -- 172.21.15.153:0/2709022249 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7888032440 con 0x7f78a009aaa0 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 1 -- 172.21.15.153:0/2709022249 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78880327e0 con 0x7f78a009aaa0 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f789f7fe640 10 monclient: dump: 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f78a626f640 5 monclient: authenticate success, global_id 144106 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f78a626f640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.729+0000 7f78a626f640 4 set_mon_vals no callback set 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 10 monclient: shutdown 2024-09-15T03:39:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 -- 172.21.15.153:0/2709022249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 msgr2=0x7f78a00971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 --2- 172.21.15.153:0/2709022249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 0x7f78a00971e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78a009aff0 tx=0x7f78880090f0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 -- 172.21.15.153:0/2709022249 shutdown_connections 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 --2- 172.21.15.153:0/2709022249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 0x7f78a00971e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 --2- 172.21.15.153:0/2709022249 >> v2:172.21.15.153:3300/0 conn(0x7f78a0096e10 0x7f78a009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 -- 172.21.15.153:0/2709022249 >> 172.21.15.153:0/2709022249 conn(0x7f78a00672b0 msgr2=0x7f78a00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 -- 172.21.15.153:0/2709022249 shutdown_connections 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 1 -- 172.21.15.153:0/2709022249 wait complete. 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 10 monclient: build_initial_monmap 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.730+0000 7f78a626f640 10 monclient: monmap: 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 Processor -- start 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 -- start start 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient: init 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 0x7f78a012d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): picked mon.noname-a con 0x7f78a0096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78a0097720 0x7f78a0117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): picked mon.noname-c con 0x7f78a0097720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 0x7f78a01179c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): picked mon.noname-b con 0x7f78a009aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a013da00 con 0x7f78a0096e10 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a011abc0 con 0x7f78a009aaa0 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a0118dd0 con 0x7f78a0097720 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f78a626f640 10 monclient(hunting): authenticate will time out at 1364.370768s 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 0x7f78a012d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78a0097720 0x7f78a0117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 0x7f78a012d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38394/0 (socket says 172.21.15.153:38394) 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 1 -- 172.21.15.153:0/2099165241 learned_addr learned my addr 172.21.15.153:0/2099165241 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): get_auth_request con 0x7f78a0096e10 auth_method 0 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 10 monclient(hunting): get_auth_request con 0x7f78a0097720 auth_method 0 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f789ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.731+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f78a480c640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 0x7f78a01179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f78a480c640 10 monclient(hunting): get_auth_request con 0x7f78a009aaa0 auth_method 0 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f78a480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f78a480c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f78a480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 10 monclient(hunting): handle_auth_done global_id 134211 payload 824 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78a0097720 msgr2=0x7f78a0117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78a0097720 0x7f78a0117480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 msgr2=0x7f78a01179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78a009aaa0 0x7f78a01179c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 1 monclient: found mon.noname-a 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f789ffff640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888009390 con 0x7f78a0096e10 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.732+0000 7f7897fff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78a0097720 0x7f78a0117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789ffff640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 msgr2=0x7f78a012d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789ffff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 0x7f78a012d7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7890010e40 tx=0x7f789000b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890011660 con 0x7f78a0096e10 2024-09-15T03:39:26.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: dump: 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: _finish_auth 0 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78900117c0 con 0x7f78a0096e10 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 5 monclient: authenticate success, global_id 134211 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 10 monclient: _renew_subs 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a011a8e0 con 0x7f78a0096e10 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890011b60 con 0x7f78a0096e10 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789cff9640 10 monclient: dump: 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.752 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 10 monclient: _renew_subs 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f789effd640 4 set_mon_vals no callback set 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.734+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a0119980 con 0x7f78a0096e10 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7890019440 con 0x7f78a0096e10 2024-09-15T03:39:26.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f78a626f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f78a626f640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 0x7f78a0117f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a0065f20 con 0x7f78a012dd00 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 0x7f78a0117f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient: get_auth_request con 0x7f78a012dd00 auth_method 0 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f789cff9640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7880038740 0x7f788003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f789004f2a0 con 0x7f78a0096e10 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 10 monclient(hunting): handle_auth_done global_id 134214 payload 824 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.735+0000 7f7897fff640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78a012e820 con 0x7f78a012dd00 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.736+0000 7f78a480c640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7880038740 0x7f788003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.736+0000 7f78a480c640 10 monclient: get_auth_request con 0x7f7880038740 auth_method 0 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f7897fff640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 msgr2=0x7f78a0117f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f7897fff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 0x7f78a0117f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78a009d0e0 tx=0x7f7888032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888031e80 con 0x7f78a012dd00 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 10 monclient: dump: 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f788803d600 con 0x7f78a012dd00 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 msgr2=0x7f78a0117f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 0x7f78a0117f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78a009d0e0 tx=0x7f7888032000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 msgr2=0x7f78a0117f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:26.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f789cff9640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a012dd00 0x7f78a0117f00 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78a009d0e0 tx=0x7f7888032000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.738+0000 7f78a480c640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7880038740 0x7f788003abf0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f789800abd0 tx=0x7f7898009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f78a626f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:26.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f78a626f640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 0x7f78a006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f78a626f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a012e0d0 con 0x7f78a0118440 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 0x7f78a006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient: get_auth_request con 0x7f78a0118440 auth_method 0 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 10 monclient(hunting): handle_auth_done global_id 134217 payload 824 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.744+0000 7f7897fff640 1 -- 172.21.15.153:0/2099165241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f78a0118840 con 0x7f78a0118440 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.746+0000 7f7897fff640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 msgr2=0x7f78a006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.746+0000 7f7897fff640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 0x7f78a006b6b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7888009130 tx=0x7f788803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888047c90 con 0x7f78a0118440 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 10 monclient: dump: 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.760 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f7888047df0 con 0x7f78a0118440 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 msgr2=0x7f78a006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 0x7f78a006b6b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7888009130 tx=0x7f788803e000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 msgr2=0x7f78a006b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f789cff9640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0118440 0x7f78a006b6b0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7888009130 tx=0x7f788803e000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.761 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 114, 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:26.762 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:26.764 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:26.765 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.766 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.767 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:26.768 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:26.770 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:26.771 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:39:26.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.773 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.773 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.773 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:39:26.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.773 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:26.774 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7880038740 msgr2=0x7f788003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7880038740 0x7f788003abf0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f789800abd0 tx=0x7f7898009040 comp rx=0 tx=0).stop 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 10 monclient: shutdown 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 msgr2=0x7f78a012d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 --2- 172.21.15.153:0/2099165241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a0096e10 0x7f78a012d7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7890010e40 tx=0x7f789000b460 comp rx=0 tx=0).stop 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a480c640 1 -- 172.21.15.153:0/2099165241 reap_dead start 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a480c640 1 -- 172.21.15.153:0/2099165241 reap_dead start 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 shutdown_connections 2024-09-15T03:39:26.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 >> 172.21.15.153:0/2099165241 conn(0x7f78a00672b0 msgr2=0x7f78a012f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:26.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 shutdown_connections 2024-09-15T03:39:26.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.747+0000 7f78a626f640 1 -- 172.21.15.153:0/2099165241 wait complete. 2024-09-15T03:39:26.777 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:39:26.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:39:26.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.872+0000 7fc714a80640 10 monclient: get_monmap_and_config 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.872+0000 7fc714a80640 10 monclient: build_initial_monmap 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.872+0000 7fc714a80640 10 monclient: monmap: 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:26.872876+0000 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:26.872876+0000 2024-09-15T03:39:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.872+0000 7fc714a80640 1 Processor -- start 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.872+0000 7fc714a80640 1 -- start start 2024-09-15T03:39:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.872+0000 7fc714a80640 10 monclient: init 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7101318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc71009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71013a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient(hunting): picked mon.noname-b con 0x7fc710137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc710132640 con 0x7fc7101318f0 2024-09-15T03:39:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc710131cc0 con 0x7fc71009ea50 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71009f530 con 0x7fc710137c30 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc714a80640 10 monclient(hunting): authenticate will time out at 1364.512423s 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc7101318f0 0x7fc710131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44878/0 (socket says 172.21.15.153:44878) 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70e59c640 1 -- 172.21.15.153:0/3868039064 learned_addr learned my addr 172.21.15.153:0/3868039064 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 1 --2- 172.21.15.153:0/3868039064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): get_auth_request con 0x7fc710137c30 auth_method 0 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_done global_id 124370 payload 274 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 1 -- 172.21.15.153:0/3868039064 >> v1:172.21.15.181:6789/0 conn(0x7fc71009ea50 legacy=0x7fc71009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 1 -- 172.21.15.153:0/3868039064 >> v1:172.21.15.153:6789/0 conn(0x7fc7101318f0 legacy=0x7fc710131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 1 monclient: found mon.noname-b 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:39:26.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.873+0000 7fc70ed9d640 1 -- 172.21.15.153:0/3868039064 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71009fbf0 con 0x7fc710137c30 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70ed9d640 1 -- 172.21.15.153:0/3868039064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 msgr2=0x7fc71013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70ed9d640 1 --2- 172.21.15.153:0/3868039064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71013a010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc700009880 tx=0x7fc7000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e83c031456efe021 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 1 -- 172.21.15.153:0/3868039064 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700032440 con 0x7fc710137c30 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: dump: 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.885 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: _finish_auth 0 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 1 -- 172.21.15.153:0/3868039064 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7000325a0 con 0x7fc710137c30 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 1 -- 172.21.15.153:0/3868039064 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700032940 con 0x7fc710137c30 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc714a80640 5 monclient: authenticate success, global_id 124370 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc714a80640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc714a80640 4 set_mon_vals no callback set 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.876+0000 7fc70d59a640 10 monclient: dump: 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 10 monclient: shutdown 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:26.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 -- 172.21.15.153:0/3868039064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 msgr2=0x7fc71013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 --2- 172.21.15.153:0/3868039064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71013a010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc700009880 tx=0x7fc7000090f0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 -- 172.21.15.153:0/3868039064 shutdown_connections 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 --2- 172.21.15.153:0/3868039064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71013a010 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 -- 172.21.15.153:0/3868039064 >> 172.21.15.153:0/3868039064 conn(0x7fc710067110 msgr2=0x7fc710067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 -- 172.21.15.153:0/3868039064 shutdown_connections 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 1 -- 172.21.15.153:0/3868039064 wait complete. 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 10 monclient: build_initial_monmap 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.877+0000 7fc714a80640 10 monclient: monmap: 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:26.888 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 Processor -- start 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 -- start start 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient: init 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 0x7fc710137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): picked mon.noname-a con 0x7fc71009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7101318f0 0x7fc71012de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): picked mon.noname-b con 0x7fc7101318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71012e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): picked mon.noname-c con 0x7fc710137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71012f8a0 con 0x7fc71009ea50 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71009bf10 con 0x7fc7101318f0 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71013a230 con 0x7fc710137c30 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc714a80640 10 monclient(hunting): authenticate will time out at 1364.517772s 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 0x7fc710137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71012e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 0x7fc710137110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38418/0 (socket says 172.21.15.153:38418) 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 1 -- 172.21.15.153:0/2039849156 learned_addr learned my addr 172.21.15.153:0/2039849156 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): get_auth_request con 0x7fc71009ea50 auth_method 0 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): get_auth_request con 0x7fc710137c30 auth_method 0 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.878+0000 7fc70ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 10 monclient(hunting): handle_auth_done global_id 134220 payload 824 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70dd9b640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7101318f0 0x7fc71012de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 msgr2=0x7fc71012e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71012e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7101318f0 msgr2=0x7fc71012de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7101318f0 0x7fc71012de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 1 monclient: found mon.noname-a 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70e59c640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700009390 con 0x7fc71009ea50 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70dd9b640 10 monclient: get_auth_request con 0x7fc7101318f0 auth_method 0 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70ed9d640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710137c30 0x7fc71012e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.879+0000 7fc70dd9b640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7101318f0 0x7fc71012de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc70e59c640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 msgr2=0x7fc710137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc70e59c640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 0x7fc710137110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc70800ccd0 tx=0x7fc7080053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800aa60 con 0x7fc71009ea50 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: dump: 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: _finish_auth 0 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70800abe0 con 0x7fc71009ea50 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 5 monclient: authenticate success, global_id 134220 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 10 monclient: _renew_subs 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7080070f0 con 0x7fc71009ea50 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc710131340 con 0x7fc71009ea50 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc7067fc640 10 monclient: dump: 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc70cd99640 4 set_mon_vals no callback set 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 10 monclient: _renew_subs 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.881+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7101303e0 con 0x7fc71009ea50 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc714a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc714a80640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 0x7fc71012e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc714a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc710065e00 con 0x7fc710136580 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc7080074e0 con 0x7fc71009ea50 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 0x7fc71012e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient: get_auth_request con 0x7fc710136580 auth_method 0 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc7067fc640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6f0038740 0x7fc6f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_done global_id 124376 payload 824 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc70dd9b640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc71012ffc0 con 0x7fc710136580 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.882+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc70804c7c0 con 0x7fc71009ea50 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.883+0000 7fc70ed9d640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6f0038740 0x7fc6f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.883+0000 7fc70ed9d640 10 monclient: get_auth_request con 0x7fc6f0038740 auth_method 0 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.884+0000 7fc70dd9b640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 msgr2=0x7fc71012e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.884+0000 7fc70dd9b640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 0x7fc71012e880 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800d500 tx=0x7fc6f8009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8007030 con 0x7fc710136580 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 10 monclient: dump: 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc6f8007190 con 0x7fc710136580 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 msgr2=0x7fc71012e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 0x7fc71012e880 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800d500 tx=0x7fc6f8009040 comp rx=0 tx=0).stop 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 msgr2=0x7fc71012e880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc7067fc640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc710136580 0x7fc71012e880 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.885+0000 7fc70ed9d640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6f0038740 0x7fc6f003abf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc710076600 tx=0x7fc700032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.890+0000 7fc714a80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.890+0000 7fc714a80640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 0x7fc7100a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.890+0000 7fc714a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.890+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71004bf60 con 0x7fc7100a3c30 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 0x7fc7100a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient: get_auth_request con 0x7fc7100a3c30 auth_method 0 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 10 monclient(hunting): handle_auth_done global_id 124379 payload 824 2024-09-15T03:39:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.891+0000 7fc70dd9b640 1 -- 172.21.15.153:0/2039849156 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc710132900 con 0x7fc7100a3c30 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.893+0000 7fc70dd9b640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 msgr2=0x7fc7100a4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.893+0000 7fc70dd9b640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 0x7fc7100a4030 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800d500 tx=0x7fc6f801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.893+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8020070 con 0x7fc7100a3c30 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.893+0000 7fc7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.893+0000 7fc7067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.893+0000 7fc7067fc640 10 monclient: dump: 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:26.903 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fc6f8007650 con 0x7fc7100a3c30 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 msgr2=0x7fc7100a4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 0x7fc7100a4030 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800d500 tx=0x7fc6f801a000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 msgr2=0x7fc7100a4030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc7067fc640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7100a3c30 0x7fc7100a4030 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800d500 tx=0x7fc6f801a000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.904 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 114, 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:26.905 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:26.906 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:26.907 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:26.908 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:26.909 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:26.910 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:26.910 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:26.910 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:26.911 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:26.912 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:26.917 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:39:26.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:26.919 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:26.920 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:26.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6f0038740 msgr2=0x7fc6f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6f0038740 0x7fc6f003abf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc710076600 tx=0x7fc700032000 comp rx=0 tx=0).stop 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 10 monclient: shutdown 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 msgr2=0x7fc710137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 0x7fc710137110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc70800ccd0 tx=0x7fc7080053f0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc70ed9d640 1 -- 172.21.15.153:0/2039849156 reap_dead start 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 shutdown_connections 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 --2- 172.21.15.153:0/2039849156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71009ea50 0x7fc710137110 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 >> 172.21.15.153:0/2039849156 conn(0x7fc710067110 msgr2=0x7fc71009d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 shutdown_connections 2024-09-15T03:39:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:26.894+0000 7fc714a80640 1 -- 172.21.15.153:0/2039849156 wait complete. 2024-09-15T03:39:26.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:39:26.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:39:26.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:39:27.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.016+0000 7ff94f300640 10 monclient: get_monmap_and_config 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.016+0000 7ff94f300640 10 monclient: build_initial_monmap 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.016+0000 7ff94f300640 10 monclient: monmap: 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:27.017245+0000 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:27.017245+0000 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:27.018 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:27.019 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 Processor -- start 2024-09-15T03:39:27.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 -- start start 2024-09-15T03:39:27.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient: init 2024-09-15T03:39:27.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff94809caa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff94809a540 0x7ff94809ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): picked mon.noname-c con 0x7ff94809a540 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff9481346c0 0x7ff948136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): picked mon.noname-a con 0x7ff9481346c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94809abd0 con 0x7ff94809caa0 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94809ed40 con 0x7ff9481346c0 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94809d3b0 con 0x7ff94809a540 2024-09-15T03:39:27.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94f300640 10 monclient(hunting): authenticate will time out at 1364.656876s 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.017+0000 7ff94d89d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff9481346c0 0x7ff948136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff9481346c0 0x7ff948136aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38434/0 (socket says 172.21.15.153:38434) 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 -- 172.21.15.153:0/2361108422 learned_addr learned my addr 172.21.15.153:0/2361108422 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): get_auth_request con 0x7ff9481346c0 auth_method 0 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94c89b640 1 --2- 172.21.15.153:0/2361108422 >> v2:172.21.15.181:3300/0 conn(0x7ff94809a540 0x7ff94809ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94c89b640 10 monclient(hunting): get_auth_request con 0x7ff94809a540 auth_method 0 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94c89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94c89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94c89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient(hunting): handle_auth_done global_id 134223 payload 274 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 -- 172.21.15.153:0/2361108422 >> v2:172.21.15.181:3300/0 conn(0x7ff94809a540 msgr2=0x7ff94809ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 --2- 172.21.15.153:0/2361108422 >> v2:172.21.15.181:3300/0 conn(0x7ff94809a540 0x7ff94809ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 -- 172.21.15.153:0/2361108422 >> v1:172.21.15.181:6789/0 conn(0x7ff94809caa0 legacy=0x7ff94806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient: _finish_hunting 0 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 monclient: found mon.noname-a 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.018+0000 7ff94d89d640 1 -- 172.21.15.153:0/2361108422 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94809b920 con 0x7ff9481346c0 2024-09-15T03:39:27.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94d89d640 1 -- 172.21.15.153:0/2361108422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9481346c0 msgr2=0x7ff948136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94d89d640 1 --2- 172.21.15.153:0/2361108422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9481346c0 0x7ff948136aa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff94809f550 tx=0x7ff938009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bff8ed55df6e896 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 1 -- 172.21.15.153:0/2361108422 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938007610 con 0x7ff9481346c0 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: dump: 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.026 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.027 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.027 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: _finish_auth 0 2024-09-15T03:39:27.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:27.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 1 -- 172.21.15.153:0/2361108422 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93802d050 con 0x7ff9481346c0 2024-09-15T03:39:27.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:27.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 1 -- 172.21.15.153:0/2361108422 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9380079a0 con 0x7ff9481346c0 2024-09-15T03:39:27.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 5 monclient: authenticate success, global_id 134223 2024-09-15T03:39:27.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:27.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 4 set_mon_vals no callback set 2024-09-15T03:39:27.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:27.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff947fff640 10 monclient: dump: 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.030 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 10 monclient: shutdown 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 -- 172.21.15.153:0/2361108422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9481346c0 msgr2=0x7ff948136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 --2- 172.21.15.153:0/2361108422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9481346c0 0x7ff948136aa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff94809f550 tx=0x7ff938009840 comp rx=0 tx=0).stop 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 -- 172.21.15.153:0/2361108422 shutdown_connections 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 --2- 172.21.15.153:0/2361108422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9481346c0 0x7ff948136aa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 --2- 172.21.15.153:0/2361108422 >> v2:172.21.15.181:3300/0 conn(0x7ff94809a540 0x7ff94809ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 -- 172.21.15.153:0/2361108422 >> 172.21.15.153:0/2361108422 conn(0x7ff948067cd0 msgr2=0x7ff9480a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 -- 172.21.15.153:0/2361108422 shutdown_connections 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.021+0000 7ff94f300640 1 -- 172.21.15.153:0/2361108422 wait complete. 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.022+0000 7ff94f300640 10 monclient: build_initial_monmap 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.022+0000 7ff94f300640 10 monclient: monmap: 2024-09-15T03:39:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:27.032 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.022+0000 7ff94f300640 1 Processor -- start 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.022+0000 7ff94f300640 1 -- start start 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.022+0000 7ff94f300640 10 monclient: init 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 0x7ff948097430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): picked mon.noname-a con 0x7ff94809a540 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94809caa0 0x7ff948097970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): picked mon.noname-b con 0x7ff94809caa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9481346c0 0x7ff948097eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): picked mon.noname-c con 0x7ff9481346c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff948099d50 con 0x7ff94809a540 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94811e980 con 0x7ff94809caa0 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94811e6a0 con 0x7ff9481346c0 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94f300640 10 monclient(hunting): authenticate will time out at 1364.662269s 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 0x7ff948097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9481346c0 0x7ff948097eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 0x7ff948097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38436/0 (socket says 172.21.15.153:38436) 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 -- 172.21.15.153:0/3502670253 learned_addr learned my addr 172.21.15.153:0/3502670253 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:27.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): get_auth_request con 0x7ff94809a540 auth_method 0 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): get_auth_request con 0x7ff9481346c0 auth_method 0 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94c89b640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94809caa0 0x7ff948097970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94c89b640 10 monclient(hunting): get_auth_request con 0x7ff94809caa0 auth_method 0 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94c89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94c89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94c89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient(hunting): handle_auth_done global_id 134226 payload 824 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9481346c0 msgr2=0x7ff948097eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9481346c0 0x7ff948097eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94809caa0 msgr2=0x7ff948097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94809caa0 0x7ff948097970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient: _finish_hunting 0 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 monclient: found mon.noname-a 2024-09-15T03:39:27.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d09c640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938009ae0 con 0x7ff94809a540 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94d89d640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9481346c0 0x7ff948097eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.023+0000 7ff94c89b640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94809caa0 0x7ff948097970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.025+0000 7ff94d09c640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 msgr2=0x7ff948097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.025+0000 7ff94d09c640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 0x7ff948097430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff94000c5e0 tx=0x7ff940007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940012840 con 0x7ff94809a540 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: dump: 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.043 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: _finish_auth 0 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9400129a0 con 0x7ff94809a540 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 5 monclient: authenticate success, global_id 134226 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 10 monclient: _renew_subs 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94811ec60 con 0x7ff94809a540 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940012d60 con 0x7ff94809a540 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 10 monclient: dump: 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.045 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 10 monclient: _renew_subs 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9477fe640 4 set_mon_vals no callback set 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94811d940 con 0x7ff94809a540 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 0x7ff94806b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9480a2a50 con 0x7ff9480983f0 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.026+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff940011590 con 0x7ff94809a540 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff9457fa640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff934038740 0x7ff93403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff94004c720 con 0x7ff94809a540 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 0x7ff94806b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient: get_auth_request con 0x7ff9480983f0 auth_method 0 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94d89d640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff934038740 0x7ff93403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94d89d640 10 monclient: get_auth_request con 0x7ff934038740 auth_method 0 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 10 monclient(hunting): handle_auth_done global_id 144118 payload 824 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.027+0000 7ff94c89b640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff94804bf60 con 0x7ff9480983f0 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.029+0000 7ff94c89b640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 msgr2=0x7ff94806b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.029+0000 7ff94c89b640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 0x7ff94806b920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00f7c0 tx=0x7ff93c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.029+0000 7ff94d89d640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff934038740 0x7ff93403abf0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff94809f920 tx=0x7ff938036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c00fe90 con 0x7ff9480983f0 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 10 monclient: dump: 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.048 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff93c00dad0 con 0x7ff9480983f0 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 msgr2=0x7ff94806b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 0x7ff94806b920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00f7c0 tx=0x7ff93c008040 comp rx=0 tx=0).stop 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 msgr2=0x7ff94806b920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.030+0000 7ff9457fa640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9480983f0 0x7ff94806b920 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00f7c0 tx=0x7ff93c008040 comp rx=0 tx=0).stop 2024-09-15T03:39:27.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.035+0000 7ff94f300640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.035+0000 7ff94f300640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 0x7ff94806c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94f300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff948065e00 con 0x7ff94806be60 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 0x7ff94806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient: get_auth_request con 0x7ff94806be60 auth_method 0 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 10 monclient(hunting): handle_auth_done global_id 144121 payload 824 2024-09-15T03:39:27.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.036+0000 7ff94c89b640 1 -- 172.21.15.153:0/3502670253 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff94811eaf0 con 0x7ff94806be60 2024-09-15T03:39:27.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff94c89b640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 msgr2=0x7ff94806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:27.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff94c89b640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 0x7ff94806c230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff93c019000 tx=0x7ff93c01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c01ea10 con 0x7ff94806be60 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 10 monclient: dump: 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.052 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7ff93c01eb70 con 0x7ff94806be60 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 msgr2=0x7ff94806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 0x7ff94806c230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff93c019000 tx=0x7ff93c01c000 comp rx=0 tx=0).stop 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 msgr2=0x7ff94806c230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:27.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.039+0000 7ff9457fa640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94806be60 0x7ff94806c230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 114, 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:27.054 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:27.055 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:27.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:27.057 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:27.058 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:27.059 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:27.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:27.062 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.064 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:27.064 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:27.065 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:27.066 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:27.066 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:27.067 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:27.068 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:27.068 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:27.068 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:27.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:27.068 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:27.068 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff934038740 msgr2=0x7ff93403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff934038740 0x7ff93403abf0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff94809f920 tx=0x7ff938036000 comp rx=0 tx=0).stop 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 10 monclient: shutdown 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 msgr2=0x7ff948097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 --2- 172.21.15.153:0/3502670253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94809a540 0x7ff948097430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff94000c5e0 tx=0x7ff940007ed0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94d89d640 1 -- 172.21.15.153:0/3502670253 reap_dead start 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94d89d640 1 -- 172.21.15.153:0/3502670253 reap_dead start 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 shutdown_connections 2024-09-15T03:39:27.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 >> 172.21.15.153:0/3502670253 conn(0x7ff948067cd0 msgr2=0x7ff948133d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:27.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 shutdown_connections 2024-09-15T03:39:27.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.040+0000 7ff94f300640 1 -- 172.21.15.153:0/3502670253 wait complete. 2024-09-15T03:39:27.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:39:27.071 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:39:27.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:39:27.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.162+0000 7f693ab5d640 10 monclient: get_monmap_and_config 2024-09-15T03:39:27.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.162+0000 7f693ab5d640 10 monclient: build_initial_monmap 2024-09-15T03:39:27.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.163+0000 7f693ab5d640 10 monclient: monmap: 2024-09-15T03:39:27.164 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:27.164 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:27.163735+0000 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:27.163735+0000 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:27.165 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.163+0000 7f693ab5d640 1 Processor -- start 2024-09-15T03:39:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.163+0000 7f693ab5d640 1 -- start start 2024-09-15T03:39:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.163+0000 7f693ab5d640 10 monclient: init 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f693409c860 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f693409e840 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6934137cc0 0x7f693413a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6934137cc0 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693409ec40 con 0x7f693409c860 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693409b2b0 con 0x7f693409e840 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693406a320 con 0x7f6934137cc0 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f693ab5d640 10 monclient(hunting): authenticate will time out at 1364.803388s 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69388f9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f693409c860 0x7f693409cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44886/0 (socket says 172.21.15.153:44886) 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69388f9640 1 -- 172.21.15.153:0/1529013687 learned_addr learned my addr 172.21.15.153:0/1529013687 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 1 --2- 172.21.15.153:0/1529013687 >> v2:172.21.15.181:3300/0 conn(0x7f6934137cc0 0x7f693413a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): get_auth_request con 0x7f6934137cc0 auth_method 0 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient(hunting): handle_auth_done global_id 144124 payload 274 2024-09-15T03:39:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 1 -- 172.21.15.153:0/1529013687 >> v1:172.21.15.181:6789/0 conn(0x7f693409e840 legacy=0x7f693409aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 1 -- 172.21.15.153:0/1529013687 >> v1:172.21.15.153:6789/0 conn(0x7f693409c860 legacy=0x7f693409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient: _finish_hunting 0 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 1 monclient: found mon.noname-c 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.164+0000 7f69390fa640 1 -- 172.21.15.153:0/1529013687 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934068a20 con 0x7f6934137cc0 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.167+0000 7f69390fa640 1 -- 172.21.15.153:0/1529013687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6934137cc0 msgr2=0x7f693413a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.167+0000 7f69390fa640 1 --2- 172.21.15.153:0/1529013687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6934137cc0 0x7f693413a0a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f691c009850 tx=0x7f691c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ebe744a18b9053 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 1 -- 172.21.15.153:0/1529013687 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c032440 con 0x7f6934137cc0 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: dump: 2024-09-15T03:39:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: _finish_auth 0 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 1 -- 172.21.15.153:0/1529013687 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c0325a0 con 0x7f6934137cc0 2024-09-15T03:39:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 5 monclient: authenticate success, global_id 144124 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 4 set_mon_vals no callback set 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 1 -- 172.21.15.153:0/1529013687 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c032940 con 0x7f6934137cc0 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f69337fe640 10 monclient: dump: 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 10 monclient: shutdown 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 -- 172.21.15.153:0/1529013687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6934137cc0 msgr2=0x7f693413a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 --2- 172.21.15.153:0/1529013687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6934137cc0 0x7f693413a0a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f691c009850 tx=0x7f691c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 -- 172.21.15.153:0/1529013687 shutdown_connections 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 --2- 172.21.15.153:0/1529013687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6934137cc0 0x7f693413a0a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 -- 172.21.15.153:0/1529013687 >> 172.21.15.153:0/1529013687 conn(0x7f69340a2e40 msgr2=0x7f69340a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 -- 172.21.15.153:0/1529013687 shutdown_connections 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.168+0000 7f693ab5d640 1 -- 172.21.15.153:0/1529013687 wait complete. 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.169+0000 7f693ab5d640 10 monclient: build_initial_monmap 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.169+0000 7f693ab5d640 10 monclient: monmap: 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.169+0000 7f693ab5d640 1 Processor -- start 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.169+0000 7f693ab5d640 1 -- start start 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.169+0000 7f693ab5d640 10 monclient: init 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f693409c860 0x7f69341375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f693409c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 0x7f693406b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7f693409e840 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 0x7f693406bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6934137cc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693406e270 con 0x7f693409e840 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693409e010 con 0x7f693409c860 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6934099c40 con 0x7f6934137cc0 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f693ab5d640 10 monclient(hunting): authenticate will time out at 1364.809269s 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 0x7f693406b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 0x7f693406bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 0x7f693406bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57650/0 (socket says 172.21.15.153:57650) 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 1 -- 172.21.15.153:0/2207132536 learned_addr learned my addr 172.21.15.153:0/2207132536 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 0x7f693406b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38452/0 (socket says 172.21.15.153:38452) 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): get_auth_request con 0x7f6934137cc0 auth_method 0 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): get_auth_request con 0x7f693409e840 auth_method 0 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69388f9640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f693409c860 0x7f69341375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69388f9640 10 monclient(hunting): get_auth_request con 0x7f693409c860 auth_method 0 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69388f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69388f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69388f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient(hunting): handle_auth_done global_id 134229 payload 824 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 -- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 msgr2=0x7f693406bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 0x7f693406bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 -- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f693409c860 msgr2=0x7f69341375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f693409c860 0x7f69341375f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 monclient: found mon.noname-a 2024-09-15T03:39:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f6933fff640 1 -- 172.21.15.153:0/2207132536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c009390 con 0x7f693409e840 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.170+0000 7f69390fa640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 0x7f693406bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.172+0000 7f6933fff640 1 -- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 msgr2=0x7f693406b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.172+0000 7f6933fff640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 0x7f693406b8b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f692400c860 tx=0x7f6924002630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692400a850 con 0x7f693409e840 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: dump: 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: _finish_auth 0 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692400a9d0 con 0x7f693409e840 2024-09-15T03:39:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 5 monclient: authenticate success, global_id 134229 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 10 monclient: _renew_subs 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693406d0a0 con 0x7f693409e840 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692400ad90 con 0x7f693409e840 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 10 monclient: dump: 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 10 monclient: _renew_subs 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693406e550 con 0x7f693409e840 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6932ffd640 4 set_mon_vals no callback set 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.173+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6924012630 con 0x7f693409e840 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f693ab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f693ab5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:27.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f693ab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f6930ff9640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6914038740 0x7f691403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:27.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693404bf60 con 0x7f693409e840 2024-09-15T03:39:27.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f692404c8c0 con 0x7f693409e840 2024-09-15T03:39:27.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f69388f9640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6914038740 0x7f691403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:27.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.174+0000 7f69388f9640 10 monclient: get_auth_request con 0x7f6914038740 auth_method 0 2024-09-15T03:39:27.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.176+0000 7f69388f9640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6914038740 0x7f691403abf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f692c007fd0 tx=0x7f692c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:27.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.178+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6924016030 con 0x7f693409e840 2024-09-15T03:39:27.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.178+0000 7f6930ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:27.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.178+0000 7f6930ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:27.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.391+0000 7f693ab5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:39:27.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.391+0000 7f693ab5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:39:27.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.391+0000 7f693ab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:27.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.391+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f69340a2b50 con 0x7f693409e840 2024-09-15T03:39:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.392+0000 7f6930ff9640 1 -- 172.21.15.153:0/2207132536 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f6924016080 con 0x7f693409e840 2024-09-15T03:39:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.392+0000 7f6930ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:39:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.392+0000 7f6930ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:27.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.392+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.392+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=8,logm=58} crc {auth=3897506974,config=3140514009,health=3128945527,kv=3235807718,logm=3089644374}) 2024-09-15T03:39:27.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.392+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=8,logm=57} crc {auth=656291879,config=3140514009,health=2560086508,kv=3235807718,logm=2092438517}) 2024-09-15T03:39:27.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.392+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.392+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=8,logm=58} crc {auth=3897506974,config=3140514009,health=3128945527,kv=3235807718,logm=3089644374}) 2024-09-15T03:39:27.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.392+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=11,kv=8,logm=56} crc {auth=1007431792,config=3140514009,health=3514463435,kv=3235807718,logm=2492300628}) 2024-09-15T03:39:27.395 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.393+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.395 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.393+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918241980}) 2024-09-15T03:39:27.395 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.393+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=382798525}) 2024-09-15T03:39:27.395 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.393+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.395 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.393+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918241980}) 2024-09-15T03:39:27.395 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.393+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2345118663}) 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6914038740 msgr2=0x7f691403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6914038740 0x7f691403abf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f692c007fd0 tx=0x7f692c009040 comp rx=0 tx=0).stop 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 10 monclient: shutdown 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 msgr2=0x7f693406b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 0x7f693406b8b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f692400c860 tx=0x7f6924002630 comp rx=0 tx=0).stop 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 shutdown_connections 2024-09-15T03:39:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6914038740 0x7f691403abf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6934137cc0 0x7f693406bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f693409e840 0x7f693406b8b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 --2- 172.21.15.153:0/2207132536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f693409c860 0x7f69341375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 >> 172.21.15.153:0/2207132536 conn(0x7f69340a2e40 msgr2=0x7f69340a2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 shutdown_connections 2024-09-15T03:39:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:27.394+0000 7f693ab5d640 1 -- 172.21.15.153:0/2207132536 wait complete. 2024-09-15T03:39:27.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.395+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.395+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557428917}) 2024-09-15T03:39:27.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.395+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886393251}) 2024-09-15T03:39:27.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.395+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.395+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557428917}) 2024-09-15T03:39:27.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.395+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3720959390}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134964078}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142244449}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134964078}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587840916}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288692606}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854576684}) 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.396+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.401 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288692606}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538969264}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804510623}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339207248}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804510623}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.397+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715869715}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.398+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.398+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673726609}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.398+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786096544}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.398+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.398+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673726609}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.398+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993376862}) 2024-09-15T03:39:27.402 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.399+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.403 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.399+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523946880}) 2024-09-15T03:39:27.403 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.399+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279747691}) 2024-09-15T03:39:27.403 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.399+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.403 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.399+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523946880}) 2024-09-15T03:39:27.403 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.399+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297662313}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513362452}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=429477424}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513362452}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3547184351}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.400+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281173356}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1149938422}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281173356}) 2024-09-15T03:39:27.407 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=427998512}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819505262}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=329120162}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819505262}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.401+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824697561}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.402+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.402+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283293337}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.402+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365565618}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.402+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.402+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283293337}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.402+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747885478}) 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.403+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.408 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.403+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105748573}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.403+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770074204}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.403+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.403+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105748573}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.403+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3759093995}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386885275}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321985156}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386885275}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191608606}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070532152}) 2024-09-15T03:39:27.409 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511193577}) 2024-09-15T03:39:27.410 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.410 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070532152}) 2024-09-15T03:39:27.410 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.404+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889121888}) 2024-09-15T03:39:27.411 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:39:27.421 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.405+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.405+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3623882364}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.405+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384406825}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.405+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.405+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3623882364}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.405+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3502440300}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.406+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.406+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200625358}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.406+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47245787}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.406+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.406+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200625358}) 2024-09-15T03:39:27.422 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.406+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511664340}) 2024-09-15T03:39:27.424 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.424 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572397738}) 2024-09-15T03:39:27.424 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220480814}) 2024-09-15T03:39:27.424 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.424 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572397738}) 2024-09-15T03:39:27.424 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406985854}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155768586}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2810193383}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155768586}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.407+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1253666202}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.408+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.408+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049186016}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.408+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=241986736}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.408+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.408+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049186016}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.408+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665636151}) 2024-09-15T03:39:27.425 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.409+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.409+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598643085}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.409+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4052172705}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.409+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.409+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598643085}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.409+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618605239}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676832672}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524800454}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676832672}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324911761}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.410+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400560381}) 2024-09-15T03:39:27.426 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951389009}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400560381}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2820719920}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824513605}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1724995283}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824513605}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.411+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658028310}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.412+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.412+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656477125}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.412+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1436668562}) 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.412+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.412+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656477125}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.412+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136980044}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.413+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.413+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113767900}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.413+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764580410}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.413+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.413+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113767900}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.413+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403828080}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.414+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.414+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662065486}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.414+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=179960879}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.414+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.414+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662065486}) 2024-09-15T03:39:27.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.414+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497379409}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418385689}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=919413684}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418385689}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041284119}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240386720}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2475872854}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240386720}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.415+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608464160}) 2024-09-15T03:39:27.429 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.416+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.416+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622642558}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.416+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3609505512}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.416+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.416+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622642558}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.416+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712403791}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211020019}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983301041}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211020019}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2422837844}) 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.430 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033550904}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.417+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3955601831}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033550904}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069752171}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556888258}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468340705}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556888258}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.418+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3158069419}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.419+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.419+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254124155}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.419+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669542436}) 2024-09-15T03:39:27.431 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.419+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.419+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254124155}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.419+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128186538}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251890371}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128018361}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251890371}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285176598}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.420+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3368010491}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526249970}) 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3368010491}) 2024-09-15T03:39:27.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1529913691}) 2024-09-15T03:39:27.441 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359352825}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=333522818}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359352825}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.421+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2136869389}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.422+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.422+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482342995}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.422+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=724881581}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.422+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.422+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482342995}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.422+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735382831}) 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407833006}) 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184343752}) 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407833006}) 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82948795}) 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.423+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409632661}) 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3885474319}) 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409632661}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241270554}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620973207}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3053871193}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620973207}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.424+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008825303}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.425+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.425+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561387583}) 2024-09-15T03:39:27.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.425+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3040796973}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.425+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.425+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561387583}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.425+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062137652}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.426+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.426+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445027552}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.426+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451831964}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.426+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.426+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445027552}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.426+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608565126}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213485714}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1973983256}) 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213485714}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=28682977}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940864032}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=710961828}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940864032}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.427+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344180119}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.428+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.428+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592749723}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.428+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306035868}) 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.428+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.428+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592749723}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.428+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643692241}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.429+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.429+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071685009}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.429+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604742421}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.429+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.429+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071685009}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.429+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711548153}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180232228}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2880223712}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180232228}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556688036}) 2024-09-15T03:39:27.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999805508}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795002778}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999805508}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.430+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870323817}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.431+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.431+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704670410}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.431+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4031179894}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.431+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.431+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704670410}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.431+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052789466}) 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.432+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.432+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=4257812538,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2208283521}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.432+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3407427686,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1713204979}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.432+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.432+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=4257812538,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2208283521}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.432+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3806000392,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=333693505}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.434+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.434+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mgrstat=1317663361,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1333603144}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.434+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=79} crc {mgrstat=2779266048,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3830688950}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.434+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.434+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mgrstat=1317663361,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1333603144}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.434+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=80} crc {mgrstat=1317663361,mon_config_key=1321035168,monmap=244636553,osd_metadata=2347033642,osd_pg_creating=1754493876,osd_snap=454132731,osdmap=1437015006}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.439+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.439+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=3881212375}) 2024-09-15T03:39:27.449 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.439+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=210189358}) 2024-09-15T03:39:27.450 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.439+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:39:27.450 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.439+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=3881212375}) 2024-09-15T03:39:27.450 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:39:27.439+0000 7fdad85f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=1651649569}) 2024-09-15T03:39:28.412 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:39:28.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:39:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.515+0000 7f2d96f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:39:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.515+0000 7f2d96f9d640 10 monclient: build_initial_monmap 2024-09-15T03:39:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.515+0000 7f2d96f9d640 10 monclient: monmap: 2024-09-15T03:39:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:28.516022+0000 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:28.516022+0000 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 Processor -- start 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 -- start start 2024-09-15T03:39:28.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient: init 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 0x7f2d90131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d901318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d9009ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d90136c20 0x7f2d90139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d90136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d90131cc0 con 0x7f2d9009ea50 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9009f530 con 0x7f2d90136c20 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9009fbf0 con 0x7f2d901318f0 2024-09-15T03:39:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d96f9d640 10 monclient(hunting): authenticate will time out at 1366.155598s 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d90136c20 0x7f2d90139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 0x7f2d90131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2d90136c20 0x7f2d90139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38466/0 (socket says 172.21.15.153:38466) 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 1 -- 172.21.15.153:0/1821901701 learned_addr learned my addr 172.21.15.153:0/1821901701 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): get_auth_request con 0x7f2d90136c20 auth_method 0 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 0x7f2d90131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57652/0 (socket says 172.21.15.153:57652) 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): get_auth_request con 0x7f2d901318f0 auth_method 0 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d9553a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.516+0000 7f2d94d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 10 monclient(hunting): handle_auth_done global_id 124388 payload 274 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 1 -- 172.21.15.153:0/1821901701 >> v2:172.21.15.153:3300/0 conn(0x7f2d90136c20 msgr2=0x7f2d90139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 1 --2- 172.21.15.153:0/1821901701 >> v2:172.21.15.153:3300/0 conn(0x7f2d90136c20 0x7f2d90139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 1 -- 172.21.15.153:0/1821901701 >> v1:172.21.15.153:6789/0 conn(0x7f2d9009ea50 legacy=0x7f2d9009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 10 monclient: _finish_hunting 0 2024-09-15T03:39:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 1 monclient: found mon.noname-b 2024-09-15T03:39:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:39:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.517+0000 7f2d94d39640 1 -- 172.21.15.153:0/1821901701 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9009fe30 con 0x7f2d901318f0 2024-09-15T03:39:28.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.519+0000 7f2d94d39640 1 -- 172.21.15.153:0/1821901701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 msgr2=0x7f2d90131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:28.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.519+0000 7f2d94d39640 1 --2- 172.21.15.153:0/1821901701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 0x7f2d90131f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009850 tx=0x7f2d840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb5fe420bb951f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 1 -- 172.21.15.153:0/1821901701 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8402adf0 con 0x7f2d901318f0 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: dump: 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 1 -- 172.21.15.153:0/1821901701 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d84032440 con 0x7f2d901318f0 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 5 monclient: authenticate success, global_id 124388 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 1 -- 172.21.15.153:0/1821901701 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d840327e0 con 0x7f2d901318f0 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 4 set_mon_vals no callback set 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d8f7fe640 10 monclient: dump: 2024-09-15T03:39:28.531 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 10 monclient: shutdown 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1821901701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 msgr2=0x7f2d90131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1821901701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 0x7f2d90131f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009850 tx=0x7f2d840090f0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1821901701 shutdown_connections 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1821901701 >> v2:172.21.15.153:3300/0 conn(0x7f2d90136c20 0x7f2d90139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1821901701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d901318f0 0x7f2d90131f30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1821901701 >> 172.21.15.153:0/1821901701 conn(0x7f2d90067110 msgr2=0x7f2d90067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1821901701 shutdown_connections 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.520+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1821901701 wait complete. 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.521+0000 7f2d96f9d640 10 monclient: build_initial_monmap 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.521+0000 7f2d96f9d640 10 monclient: monmap: 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:28.533 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.521+0000 7f2d96f9d640 1 Processor -- start 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.521+0000 7f2d96f9d640 1 -- start start 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.521+0000 7f2d96f9d640 10 monclient: init 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 0x7f2d90130960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d9009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d901318f0 0x7f2d90099840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d901318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d90136c20 0x7f2d90099d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d90136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9011cd50 con 0x7f2d901318f0 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9009bf10 con 0x7f2d90136c20 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9009dfe0 con 0x7f2d9009ea50 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d96f9d640 10 monclient(hunting): authenticate will time out at 1366.161243s 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 0x7f2d90130960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 0x7f2d90130960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57664/0 (socket says 172.21.15.153:57664) 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d901318f0 0x7f2d90099840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 -- 172.21.15.153:0/1079211117 learned_addr learned my addr 172.21.15.153:0/1079211117 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): get_auth_request con 0x7f2d9009ea50 auth_method 0 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): get_auth_request con 0x7f2d901318f0 auth_method 0 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient(hunting): handle_auth_done global_id 124391 payload 824 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 -- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d90136c20 msgr2=0x7f2d90099d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d90136c20 0x7f2d90099d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 -- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d901318f0 msgr2=0x7f2d90099840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d901318f0 0x7f2d90099840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient: _finish_hunting 0 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 monclient: found mon.noname-c 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d94d39640 1 -- 172.21.15.153:0/1079211117 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84009390 con 0x7f2d9009ea50 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.522+0000 7f2d8ffff640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d901318f0 0x7f2d90099840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:28.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.524+0000 7f2d94d39640 1 -- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 msgr2=0x7f2d90130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.524+0000 7f2d94d39640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 0x7f2d90130960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009360 tx=0x7f2d84032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.524+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d84031b10 con 0x7f2d9009ea50 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: dump: 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: _finish_auth 0 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d84031c70 con 0x7f2d9009ea50 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 5 monclient: authenticate success, global_id 124391 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 10 monclient: _renew_subs 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9011bb80 con 0x7f2d9009ea50 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8403c400 con 0x7f2d9009ea50 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:28.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 10 monclient: dump: 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 10 monclient: _renew_subs 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8effd640 4 set_mon_vals no callback set 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:28.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d90141e40 con 0x7f2d9009ea50 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.525+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2d8403c830 con 0x7f2d9009ea50 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d96f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d96f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d96f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d8cff9640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d74038740 0x7f2d7403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9004bf60 con 0x7f2d9009ea50 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2d8406cf60 con 0x7f2d9009ea50 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d8ffff640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d74038740 0x7f2d7403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.526+0000 7f2d8ffff640 10 monclient: get_auth_request con 0x7f2d74038740 auth_method 0 2024-09-15T03:39:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.528+0000 7f2d8ffff640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d74038740 0x7f2d7403abf0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c007fd0 tx=0x7f2d7c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.530+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2d840426c0 con 0x7f2d9009ea50 2024-09-15T03:39:28.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.530+0000 7f2d8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:28.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.530+0000 7f2d8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:28.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.743+0000 7f2d96f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.743+0000 7f2d96f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.743+0000 7f2d96f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.743+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2d9009b3a0 con 0x7f2d9009ea50 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.744+0000 7f2d8cff9640 1 -- 172.21.15.153:0/1079211117 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2d8406f080 con 0x7f2d9009ea50 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.744+0000 7f2d8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.744+0000 7f2d8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:28.745 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:39:28.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d74038740 msgr2=0x7f2d7403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d74038740 0x7f2d7403abf0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c007fd0 tx=0x7f2d7c007e10 comp rx=0 tx=0).stop 2024-09-15T03:39:28.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 10 monclient: shutdown 2024-09-15T03:39:28.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 msgr2=0x7f2d90130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 0x7f2d90130960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009360 tx=0x7f2d84032000 comp rx=0 tx=0).stop 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 shutdown_connections 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d74038740 0x7f2d7403abf0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d90136c20 0x7f2d90099d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d901318f0 0x7f2d90099840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 --2- 172.21.15.153:0/1079211117 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9009ea50 0x7f2d90130960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 >> 172.21.15.153:0/1079211117 conn(0x7f2d90067110 msgr2=0x7f2d900a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 shutdown_connections 2024-09-15T03:39:28.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.746+0000 7f2d96f9d640 1 -- 172.21.15.153:0/1079211117 wait complete. 2024-09-15T03:39:28.757 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:39:28.757 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:39:28.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:39:28.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.850+0000 7f02f259d640 10 monclient: get_monmap_and_config 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.850+0000 7f02f259d640 10 monclient: build_initial_monmap 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.850+0000 7f02f259d640 10 monclient: monmap: 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:28.851268+0000 2024-09-15T03:39:28.851 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:28.851268+0000 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:28.852 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 Processor -- start 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 -- start start 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient: init 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02ec06eb80 0x7f02ec06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): picked mon.noname-a con 0x7f02ec06eb80 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f02ec068a20 0x7f02ec09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): picked mon.noname-c con 0x7f02ec068a20 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec09d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): picked mon.noname-b con 0x7f02ec09caf0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ec06a7a0 con 0x7f02ec06eb80 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ec09cef0 con 0x7f02ec068a20 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec06a0e0 con 0x7f02ec09caf0 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f259d640 10 monclient(hunting): authenticate will time out at 1366.490817s 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02ec06eb80 0x7f02ec06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f0b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec09d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02ec06eb80 0x7f02ec06ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38484/0 (socket says 172.21.15.153:38484) 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 1 -- 172.21.15.153:0/1180026563 learned_addr learned my addr 172.21.15.153:0/1180026563 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 10 monclient(hunting): get_auth_request con 0x7f02ec06eb80 auth_method 0 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f0b3a640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec09d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57672/0 (socket says 172.21.15.153:57672) 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f0b3a640 10 monclient(hunting): get_auth_request con 0x7f02ec09caf0 auth_method 0 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f0b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.851+0000 7f02f0b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02f0b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02f0b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02f0b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02f0b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02eb7fe640 1 --2- 172.21.15.153:0/1180026563 >> v2:172.21.15.181:3300/0 conn(0x7f02ec068a20 0x7f02ec09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02eb7fe640 10 monclient(hunting): get_auth_request con 0x7f02ec068a20 auth_method 0 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 10 monclient(hunting): handle_auth_done global_id 134238 payload 274 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 1 -- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 msgr2=0x7f02ec09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec09d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 1 -- 172.21.15.153:0/1180026563 >> v2:172.21.15.181:3300/0 conn(0x7f02ec068a20 msgr2=0x7f02ec09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:28.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 1 --2- 172.21.15.153:0/1180026563 >> v2:172.21.15.181:3300/0 conn(0x7f02ec068a20 0x7f02ec09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:28.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 1 monclient: found mon.noname-a 2024-09-15T03:39:28.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:28.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:28.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02ebfff640 1 -- 172.21.15.153:0/1180026563 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec069e00 con 0x7f02ec06eb80 2024-09-15T03:39:28.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.852+0000 7f02f0b3a640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec09d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02ebfff640 1 -- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec06eb80 msgr2=0x7f02ec06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02ebfff640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec06eb80 0x7f02ec06ef50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02ec066500 tx=0x7f02dc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3884a8483c58b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 1 -- 172.21.15.153:0/1180026563 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc032440 con 0x7f02ec06eb80 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: dump: 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.871 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: _finish_auth 0 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 1 -- 172.21.15.153:0/1180026563 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc0325a0 con 0x7f02ec06eb80 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 5 monclient: authenticate success, global_id 134238 2024-09-15T03:39:28.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 4 set_mon_vals no callback set 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 1 -- 172.21.15.153:0/1180026563 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc032940 con 0x7f02ec06eb80 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02eaffd640 10 monclient: dump: 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.873 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 10 monclient: shutdown 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 -- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec06eb80 msgr2=0x7f02ec06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec06eb80 0x7f02ec06ef50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02ec066500 tx=0x7f02dc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 -- 172.21.15.153:0/1180026563 shutdown_connections 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec09d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 --2- 172.21.15.153:0/1180026563 >> v2:172.21.15.181:3300/0 conn(0x7f02ec068a20 0x7f02ec09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 --2- 172.21.15.153:0/1180026563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec06eb80 0x7f02ec06ef50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 -- 172.21.15.153:0/1180026563 >> 172.21.15.153:0/1180026563 conn(0x7f02ec067cd0 msgr2=0x7f02ec0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:28.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 -- 172.21.15.153:0/1180026563 shutdown_connections 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.855+0000 7f02f259d640 1 -- 172.21.15.153:0/1180026563 wait complete. 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.856+0000 7f02f259d640 10 monclient: build_initial_monmap 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.856+0000 7f02f259d640 10 monclient: monmap: 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.856+0000 7f02f259d640 1 Processor -- start 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.856+0000 7f02f259d640 1 -- start start 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.856+0000 7f02f259d640 10 monclient: init 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 0x7f02ec134c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): picked mon.noname-a con 0x7f02ec068a20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02ec06eb80 0x7f02ec11e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): picked mon.noname-b con 0x7f02ec06eb80 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec11ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): picked mon.noname-c con 0x7f02ec09caf0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec0a2650 con 0x7f02ec068a20 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec121d60 con 0x7f02ec06eb80 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec122040 con 0x7f02ec09caf0 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f259d640 10 monclient(hunting): authenticate will time out at 1366.496205s 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 0x7f02ec134c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec11ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 0x7f02ec134c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38490/0 (socket says 172.21.15.153:38490) 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 -- 172.21.15.153:0/144948091 learned_addr learned my addr 172.21.15.153:0/144948091 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): get_auth_request con 0x7f02ec068a20 auth_method 0 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec11ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:57684/0 (socket says 172.21.15.153:57684) 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): get_auth_request con 0x7f02ec09caf0 auth_method 0 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02eb7fe640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02ec06eb80 0x7f02ec11e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02eb7fe640 10 monclient(hunting): get_auth_request con 0x7f02ec06eb80 auth_method 0 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient(hunting): handle_auth_done global_id 134241 payload 824 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 msgr2=0x7f02ec11ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec11ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02ec06eb80 msgr2=0x7f02ec11e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02ec06eb80 0x7f02ec11e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 monclient: found mon.noname-a 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02ebfff640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4009a20 con 0x7f02ec068a20 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.857+0000 7f02f0b3a640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02ec09caf0 0x7f02ec11ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:28.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02ebfff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 msgr2=0x7f02ec134c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02ebfff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 0x7f02ec134c40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009210 tx=0x7f02dc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0319c0 con 0x7f02ec068a20 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02d3fff640 10 monclient: dump: 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.880 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02d3fff640 10 monclient: _finish_auth 0 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.859+0000 7f02d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 5 monclient: authenticate success, global_id 134241 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 10 monclient: _renew_subs 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc009450 con 0x7f02ec068a20 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc031b20 con 0x7f02ec068a20 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc03d440 con 0x7f02ec068a20 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 10 monclient: dump: 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 10 monclient: _renew_subs 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ec136220 con 0x7f02ec068a20 2024-09-15T03:39:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02ea7fc640 4 set_mon_vals no callback set 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f02dc039040 con 0x7f02ec068a20 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 0x7f02ec11f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.860+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec120150 con 0x7f02ec135180 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 0x7f02ec11f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient: get_auth_request con 0x7f02ec135180 auth_method 0 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02d3fff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02d8038740 0x7f02d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f02dc06c350 con 0x7f02ec068a20 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_done global_id 134244 payload 824 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02eb7fe640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02ec1202e0 con 0x7f02ec135180 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02f0b3a640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02d8038740 0x7f02d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.861+0000 7f02f0b3a640 10 monclient: get_auth_request con 0x7f02d8038740 auth_method 0 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02eb7fe640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 msgr2=0x7f02ec11f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02eb7fe640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 0x7f02ec11f2a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02e000f590 tx=0x7f02e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0014880 con 0x7f02ec135180 2024-09-15T03:39:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 10 monclient: dump: 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02f0b3a640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02d8038740 0x7f02d803abf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f02e400eb00 tx=0x7f02e400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f02e00149e0 con 0x7f02ec135180 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 msgr2=0x7f02ec11f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 0x7f02ec11f2a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02e000f590 tx=0x7f02e0008040 comp rx=0 tx=0).stop 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 msgr2=0x7f02ec11f2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.863+0000 7f02d3fff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec135180 0x7f02ec11f2a0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02e000f590 tx=0x7f02e0008040 comp rx=0 tx=0).stop 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02f259d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02f259d640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 0x7f02ec096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02f259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec065e00 con 0x7f02ec11f7e0 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 0x7f02ec096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient: get_auth_request con 0x7f02ec11f7e0 auth_method 0 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:28.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:28.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:28.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:28.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:28.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 10 monclient(hunting): handle_auth_done global_id 134247 payload 824 2024-09-15T03:39:28.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.869+0000 7f02eb7fe640 1 -- 172.21.15.153:0/144948091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02ec122280 con 0x7f02ec11f7e0 2024-09-15T03:39:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02eb7fe640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 msgr2=0x7f02ec096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02eb7fe640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 0x7f02ec096a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02e0019000 tx=0x7f02e001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e001e950 con 0x7f02ec11f7e0 2024-09-15T03:39:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:28.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 10 monclient: dump: 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:28.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f02e001eab0 con 0x7f02ec11f7e0 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 msgr2=0x7f02ec096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 0x7f02ec096a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02e0019000 tx=0x7f02e001c000 comp rx=0 tx=0).stop 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 msgr2=0x7f02ec096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02d3fff640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec11f7e0 0x7f02ec096a60 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02e0019000 tx=0x7f02e001c000 comp rx=0 tx=0).stop 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:28.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 114, 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:28.891 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:28.892 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:28.893 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:28.894 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.895 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:28.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:28.898 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.900 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:28.901 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:28.902 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02d8038740 msgr2=0x7f02d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02d8038740 0x7f02d803abf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f02e400eb00 tx=0x7f02e400b040 comp rx=0 tx=0).stop 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 10 monclient: shutdown 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 msgr2=0x7f02ec134c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 --2- 172.21.15.153:0/144948091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02ec068a20 0x7f02ec134c40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009210 tx=0x7f02dc032000 comp rx=0 tx=0).stop 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f0b3a640 1 -- 172.21.15.153:0/144948091 reap_dead start 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f0b3a640 1 -- 172.21.15.153:0/144948091 reap_dead start 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 shutdown_connections 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 >> 172.21.15.153:0/144948091 conn(0x7f02ec067cd0 msgr2=0x7f02ec0a1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 shutdown_connections 2024-09-15T03:39:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.872+0000 7f02f259d640 1 -- 172.21.15.153:0/144948091 wait complete. 2024-09-15T03:39:28.905 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:39:28.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:39:28.905 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:39:28.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.999+0000 7f21414c0640 10 monclient: get_monmap_and_config 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.999+0000 7f21414c0640 10 monclient: build_initial_monmap 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.999+0000 7f21414c0640 10 monclient: monmap: 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:28.999796+0000 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:28.999796+0000 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:29.000 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:29.001 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:29.001 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:29.001 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.999+0000 7f21414c0640 1 Processor -- start 2024-09-15T03:39:29.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.999+0000 7f21414c0640 1 -- start start 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:28.999+0000 7f21414c0640 10 monclient: init 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f213c1318f0 0x7f213c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient(hunting): picked mon.noname-a con 0x7f213c1318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f213c09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f213c136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213c09f530 con 0x7f213c09ea50 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213c09fbf0 con 0x7f213c136c20 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c131d50 con 0x7f213c1318f0 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f21414c0640 10 monclient(hunting): authenticate will time out at 1366.639241s 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f213c1318f0 0x7f213c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f213c1318f0 0x7f213c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38512/0 (socket says 172.21.15.153:38512) 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 1 -- 172.21.15.153:0/1762871029 learned_addr learned my addr 172.21.15.153:0/1762871029 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 10 monclient(hunting): get_auth_request con 0x7f213c1318f0 auth_method 0 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.000+0000 7f213a7fc640 1 --1- 172.21.15.153:0/1762871029 >> v1:172.21.15.153:6789/0 conn(0x7f213c09ea50 0x7f213c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44896/0 (socket says 172.21.15.153:44896) 2024-09-15T03:39:29.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213a7fc640 1 -- 172.21.15.153:0/1762871029 >> v1:172.21.15.153:6789/0 conn(0x7f213c09ea50 legacy=0x7f213c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 10 monclient(hunting): handle_auth_done global_id 134250 payload 274 2024-09-15T03:39:29.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 1 -- 172.21.15.153:0/1762871029 >> v1:172.21.15.181:6789/0 conn(0x7f213c136c20 legacy=0x7f213c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:29.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 1 -- 172.21.15.153:0/1762871029 >> v1:172.21.15.153:6789/0 conn(0x7f213c09ea50 legacy=0x7f213c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 10 monclient: _finish_hunting 0 2024-09-15T03:39:29.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 1 monclient: found mon.noname-a 2024-09-15T03:39:29.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:29.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.002+0000 7f213affd640 1 -- 172.21.15.153:0/1762871029 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c09fe30 con 0x7f213c1318f0 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f213affd640 1 -- 172.21.15.153:0/1762871029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c1318f0 msgr2=0x7f213c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f213affd640 1 --2- 172.21.15.153:0/1762871029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c1318f0 0x7f213c131f30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f212c009fa0 tx=0x7f212c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42eb34fcb41bd1ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 1 -- 172.21.15.153:0/1762871029 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c007610 con 0x7f213c1318f0 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: dump: 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.012 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: _finish_auth 0 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f21414c0640 5 monclient: authenticate success, global_id 134250 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 1 -- 172.21.15.153:0/1762871029 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c02d050 con 0x7f213c1318f0 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f21414c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f21414c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f21414c0640 4 set_mon_vals no callback set 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 1 -- 172.21.15.153:0/1762871029 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0079a0 con 0x7f213c1318f0 2024-09-15T03:39:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.005+0000 7f2139ffb640 10 monclient: dump: 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 10 monclient: shutdown 2024-09-15T03:39:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 -- 172.21.15.153:0/1762871029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c1318f0 msgr2=0x7f213c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 --2- 172.21.15.153:0/1762871029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c1318f0 0x7f213c131f30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f212c009fa0 tx=0x7f212c009840 comp rx=0 tx=0).stop 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 -- 172.21.15.153:0/1762871029 shutdown_connections 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 --2- 172.21.15.153:0/1762871029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c1318f0 0x7f213c131f30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 -- 172.21.15.153:0/1762871029 >> 172.21.15.153:0/1762871029 conn(0x7f213c067110 msgr2=0x7f213c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 -- 172.21.15.153:0/1762871029 shutdown_connections 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.006+0000 7f21414c0640 1 -- 172.21.15.153:0/1762871029 wait complete. 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient: build_initial_monmap 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient: monmap: 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 1 Processor -- start 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 1 -- start start 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient: init 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 0x7f213c132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient(hunting): picked mon.noname-a con 0x7f213c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c1318f0 0x7f213c1330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient(hunting): picked mon.noname-c con 0x7f213c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f213c136c20 0x7f213c1335e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.007+0000 7f21414c0640 10 monclient(hunting): picked mon.noname-b con 0x7f213c136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c134890 con 0x7f213c09ea50 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c134490 con 0x7f213c136c20 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c135ee0 con 0x7f213c1318f0 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f21414c0640 10 monclient(hunting): authenticate will time out at 1366.647022s 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 0x7f213c132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 0x7f213c132b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38516/0 (socket says 172.21.15.153:38516) 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 -- 172.21.15.153:0/1168609535 learned_addr learned my addr 172.21.15.153:0/1168609535 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): get_auth_request con 0x7f213c09ea50 auth_method 0 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c1318f0 0x7f213c1330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): get_auth_request con 0x7f213c1318f0 auth_method 0 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient(hunting): handle_auth_done global_id 134256 payload 824 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213b7fe640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f213c136c20 0x7f213c1335e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c1318f0 msgr2=0x7f213c1330a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c1318f0 0x7f213c1330a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f213c136c20 msgr2=0x7f213c1335e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f213c136c20 0x7f213c1335e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient: _finish_hunting 0 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 monclient: found mon.noname-a 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213affd640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c009ae0 con 0x7f213c09ea50 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213b7fe640 10 monclient: get_auth_request con 0x7f213c136c20 auth_method 0 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213b7fe640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f213c136c20 0x7f213c1335e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.008+0000 7f213a7fc640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c1318f0 0x7f213c1330a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f213affd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 msgr2=0x7f213c132b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f213affd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 0x7f213c132b60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f212c009ab0 tx=0x7f212c036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c032a30 con 0x7f213c09ea50 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: dump: 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: _finish_auth 0 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c032b90 con 0x7f213c09ea50 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 5 monclient: authenticate success, global_id 134256 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 10 monclient: _renew_subs 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213c1361c0 con 0x7f213c09ea50 2024-09-15T03:39:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c03b410 con 0x7f213c09ea50 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f2132ffd640 10 monclient: dump: 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21397fa640 4 set_mon_vals no callback set 2024-09-15T03:39:29.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 10 monclient: _renew_subs 2024-09-15T03:39:29.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.010+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213c134f80 con 0x7f213c09ea50 2024-09-15T03:39:29.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f212c038050 con 0x7f213c09ea50 2024-09-15T03:39:29.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f21414c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:29.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f21414c0640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 0x7f213c0a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f21414c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c065e00 con 0x7f213c133b20 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 0x7f213c0a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 10 monclient: get_auth_request con 0x7f213c133b20 auth_method 0 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.011+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f2132ffd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f211c038740 0x7f211c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f212c06d150 con 0x7f213c09ea50 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f213a7fc640 10 monclient(hunting): handle_auth_done global_id 124403 payload 824 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f213a7fc640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f213c131760 con 0x7f213c133b20 2024-09-15T03:39:29.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f213b7fe640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f211c038740 0x7f211c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.012+0000 7f213b7fe640 10 monclient: get_auth_request con 0x7f211c038740 auth_method 0 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f213a7fc640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 msgr2=0x7f213c0a3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f213a7fc640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 0x7f213c0a3dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2124004a00 tx=0x7f2124007cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212400fcb0 con 0x7f213c133b20 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 10 monclient: dump: 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.029 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f212400fe10 con 0x7f213c133b20 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 msgr2=0x7f213c0a3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 0x7f213c0a3dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2124004a00 tx=0x7f2124007cc0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 msgr2=0x7f213c0a3dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f2132ffd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c133b20 0x7f213c0a3dd0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2124004a00 tx=0x7f2124007cc0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.014+0000 7f213b7fe640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f211c038740 0x7f211c03abf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f213c12cf90 tx=0x7f2134009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f21414c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f21414c0640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 0x7f213c0a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f21414c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c136610 con 0x7f213c0a4310 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 0x7f213c0a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient: get_auth_request con 0x7f213c0a4310 auth_method 0 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 10 monclient(hunting): handle_auth_done global_id 124406 payload 824 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.020+0000 7f213a7fc640 1 -- 172.21.15.153:0/1168609535 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f213c135230 con 0x7f213c0a4310 2024-09-15T03:39:29.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.022+0000 7f213a7fc640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 msgr2=0x7f213c0a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f213a7fc640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 0x7f213c0a4710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2124004a00 tx=0x7f2124015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212401c830 con 0x7f213c0a4310 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 10 monclient: dump: 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.032 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f212401c990 con 0x7f213c0a4310 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 msgr2=0x7f213c0a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 0x7f213c0a4710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2124004a00 tx=0x7f2124015000 comp rx=0 tx=0).stop 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 msgr2=0x7f213c0a4710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f2132ffd640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213c0a4310 0x7f213c0a4710 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2124004a00 tx=0x7f2124015000 comp rx=0 tx=0).stop 2024-09-15T03:39:29.033 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 114, 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:29.034 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:29.035 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:29.036 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:29.037 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.038 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.039 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:29.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:29.041 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:29.043 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:29.044 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:39:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:29.046 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.047 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:29.047 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f211c038740 msgr2=0x7f211c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f211c038740 0x7f211c03abf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f213c12cf90 tx=0x7f2134009040 comp rx=0 tx=0).stop 2024-09-15T03:39:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 10 monclient: shutdown 2024-09-15T03:39:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 msgr2=0x7f213c132b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 0x7f213c132b60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f212c009ab0 tx=0x7f212c036000 comp rx=0 tx=0).stop 2024-09-15T03:39:29.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f213b7fe640 1 -- 172.21.15.153:0/1168609535 reap_dead start 2024-09-15T03:39:29.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 shutdown_connections 2024-09-15T03:39:29.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 --2- 172.21.15.153:0/1168609535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f213c09ea50 0x7f213c132b60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 >> 172.21.15.153:0/1168609535 conn(0x7f213c067110 msgr2=0x7f213c09d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:29.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 shutdown_connections 2024-09-15T03:39:29.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.023+0000 7f21414c0640 1 -- 172.21.15.153:0/1168609535 wait complete. 2024-09-15T03:39:29.049 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:39:29.049 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:39:29.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:39:29.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.143+0000 7f835259d640 10 monclient: get_monmap_and_config 2024-09-15T03:39:29.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.143+0000 7f835259d640 10 monclient: build_initial_monmap 2024-09-15T03:39:29.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.143+0000 7f835259d640 10 monclient: monmap: 2024-09-15T03:39:29.144 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:29.144426+0000 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:29.144426+0000 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:29.145 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 1 Processor -- start 2024-09-15T03:39:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 1 -- start start 2024-09-15T03:39:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient: init 2024-09-15T03:39:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T03:39:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f834c0974f0 0x7f834c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient(hunting): picked mon.noname-c con 0x7f834c0974f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f834c09a8a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f834c133070 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c0978c0 con 0x7f834c09a8a0 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c09ca20 con 0x7f834c133070 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834c098ad0 con 0x7f834c0974f0 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.144+0000 7f835259d640 10 monclient(hunting): authenticate will time out at 1366.783925s 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f834c09a8a0 0x7f834c09ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44902/0 (socket says 172.21.15.153:44902) 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834b7fe640 1 -- 172.21.15.153:0/3329326819 learned_addr learned my addr 172.21.15.153:0/3329326819 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 1 --2- 172.21.15.153:0/3329326819 >> v2:172.21.15.181:3300/0 conn(0x7f834c0974f0 0x7f834c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): get_auth_request con 0x7f834c0974f0 auth_method 0 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient(hunting): handle_auth_done global_id 144136 payload 274 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 1 -- 172.21.15.153:0/3329326819 >> v1:172.21.15.181:6789/0 conn(0x7f834c133070 legacy=0x7f834c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 1 -- 172.21.15.153:0/3329326819 >> v1:172.21.15.153:6789/0 conn(0x7f834c09a8a0 legacy=0x7f834c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 1 monclient: found mon.noname-c 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:39:29.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.145+0000 7f834bfff640 1 -- 172.21.15.153:0/3329326819 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c098db0 con 0x7f834c0974f0 2024-09-15T03:39:29.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834bfff640 1 -- 172.21.15.153:0/3329326819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0974f0 msgr2=0x7f834c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834bfff640 1 --2- 172.21.15.153:0/3329326819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0974f0 0x7f834c12ab00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f833c009850 tx=0x7f833c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a104c1cf7a8fa530 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 1 -- 172.21.15.153:0/3329326819 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c032440 con 0x7f834c0974f0 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: dump: 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: _finish_auth 0 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 1 -- 172.21.15.153:0/3329326819 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c0325a0 con 0x7f834c0974f0 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f835259d640 5 monclient: authenticate success, global_id 144136 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f835259d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f835259d640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 1 -- 172.21.15.153:0/3329326819 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c032940 con 0x7f834c0974f0 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f835259d640 4 set_mon_vals no callback set 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.148+0000 7f834affd640 10 monclient: dump: 2024-09-15T03:39:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.161 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 10 monclient: shutdown 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 -- 172.21.15.153:0/3329326819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0974f0 msgr2=0x7f834c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 --2- 172.21.15.153:0/3329326819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0974f0 0x7f834c12ab00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f833c009850 tx=0x7f833c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 -- 172.21.15.153:0/3329326819 shutdown_connections 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 --2- 172.21.15.153:0/3329326819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0974f0 0x7f834c12ab00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 -- 172.21.15.153:0/3329326819 >> 172.21.15.153:0/3329326819 conn(0x7f834c0672b0 msgr2=0x7f834c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 -- 172.21.15.153:0/3329326819 shutdown_connections 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 1 -- 172.21.15.153:0/3329326819 wait complete. 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 10 monclient: build_initial_monmap 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.149+0000 7f835259d640 10 monclient: monmap: 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:29.162 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 Processor -- start 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 -- start start 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient: init 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 0x7f834c1325c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): picked mon.noname-a con 0x7f834c0974f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c09a8a0 0x7f834c131600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): picked mon.noname-b con 0x7f834c09a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f834c133070 0x7f834c06bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): picked mon.noname-c con 0x7f834c133070 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c131b40 con 0x7f834c0974f0 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c098300 con 0x7f834c09a8a0 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834c06de90 con 0x7f834c133070 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f835259d640 10 monclient(hunting): authenticate will time out at 1366.789764s 2024-09-15T03:39:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 0x7f834c1325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 0x7f834c1325c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38518/0 (socket says 172.21.15.153:38518) 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 1 -- 172.21.15.153:0/3593545283 learned_addr learned my addr 172.21.15.153:0/3593545283 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 10 monclient(hunting): get_auth_request con 0x7f834c0974f0 auth_method 0 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f8350b3a640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f834c133070 0x7f834c06bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f8350b3a640 10 monclient(hunting): get_auth_request con 0x7f834c133070 auth_method 0 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f8350b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f8350b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f8350b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.150+0000 7f834bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f8350b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f8350b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f8350b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834b7fe640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c09a8a0 0x7f834c131600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834b7fe640 10 monclient(hunting): get_auth_request con 0x7f834c09a8a0 auth_method 0 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 10 monclient(hunting): handle_auth_done global_id 134259 payload 824 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f834c133070 msgr2=0x7f834c06bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f834c133070 0x7f834c06bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c09a8a0 msgr2=0x7f834c131600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c09a8a0 0x7f834c131600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 1 monclient: found mon.noname-a 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834bfff640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c009390 con 0x7f834c0974f0 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f8350b3a640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f834c133070 0x7f834c06bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.151+0000 7f834b7fe640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c09a8a0 0x7f834c131600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f834bfff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 msgr2=0x7f834c1325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f834bfff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 0x7f834c1325c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f833c009360 tx=0x7f833c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0318c0 con 0x7f834c0974f0 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:39:29.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: dump: 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: _finish_auth 0 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 5 monclient: authenticate success, global_id 134259 2024-09-15T03:39:29.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 10 monclient: _renew_subs 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c031a20 con 0x7f834c0974f0 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c138960 con 0x7f834c0974f0 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c031dc0 con 0x7f834c0974f0 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f8343fff640 10 monclient: dump: 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.169 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f834a7fc640 4 set_mon_vals no callback set 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 10 monclient: _renew_subs 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.153+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c06e280 con 0x7f834c0974f0 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f835259d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f835259d640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 0x7f834c029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f835259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c065f20 con 0x7f834c0298c0 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f833c039040 con 0x7f834c0974f0 2024-09-15T03:39:29.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 0x7f834c029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient: get_auth_request con 0x7f834c0298c0 auth_method 0 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f8343fff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f832c038740 0x7f832c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f834b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.154+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f833c06c8d0 con 0x7f834c0974f0 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.155+0000 7f8350b3a640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f832c038740 0x7f832c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.155+0000 7f8350b3a640 10 monclient: get_auth_request con 0x7f832c038740 auth_method 0 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.155+0000 7f834b7fe640 10 monclient(hunting): handle_auth_done global_id 144142 payload 824 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.155+0000 7f834b7fe640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f834c06f520 con 0x7f834c0298c0 2024-09-15T03:39:29.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f834b7fe640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 msgr2=0x7f834c029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f834b7fe640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 0x7f834c029c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f834c06ddb0 tx=0x7f8334009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334005cb0 con 0x7f834c0298c0 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8350b3a640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f832c038740 0x7f832c03abf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f834c12b970 tx=0x7f83440025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 10 monclient: dump: 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8334005e10 con 0x7f834c0298c0 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 msgr2=0x7f834c029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 0x7f834c029c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f834c06ddb0 tx=0x7f8334009040 comp rx=0 tx=0).stop 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 msgr2=0x7f834c029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.157+0000 7f8343fff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c0298c0 0x7f834c029c90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.163+0000 7f835259d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.163+0000 7f835259d640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 0x7f834c02a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.163+0000 7f835259d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.163+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c04bf60 con 0x7f834c02a1d0 2024-09-15T03:39:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.163+0000 7f834b7fe640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 0x7f834c02a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient: get_auth_request con 0x7f834c02a1d0 auth_method 0 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 10 monclient(hunting): handle_auth_done global_id 144145 payload 824 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.164+0000 7f834b7fe640 1 -- 172.21.15.153:0/3593545283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f834c128d80 con 0x7f834c02a1d0 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.166+0000 7f834b7fe640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 msgr2=0x7f834c02a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.166+0000 7f834b7fe640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 0x7f834c02a5d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f834c06ddb0 tx=0x7f833401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334020810 con 0x7f834c02a1d0 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 10 monclient: dump: 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f8334005aa0 con 0x7f834c02a1d0 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 msgr2=0x7f834c02a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 0x7f834c02a5d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f834c06ddb0 tx=0x7f833401a000 comp rx=0 tx=0).stop 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 msgr2=0x7f834c02a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8343fff640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f834c02a1d0 0x7f834c02a5d0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f834c06ddb0 tx=0x7f833401a000 comp rx=0 tx=0).stop 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 114, 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:29.190 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:29.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f832c038740 msgr2=0x7f832c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f832c038740 0x7f832c03abf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f834c12b970 tx=0x7f83440025b0 comp rx=0 tx=0).stop 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 10 monclient: shutdown 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 msgr2=0x7f834c1325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 --2- 172.21.15.153:0/3593545283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f834c0974f0 0x7f834c1325c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f833c009360 tx=0x7f833c032000 comp rx=0 tx=0).stop 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8350b3a640 1 -- 172.21.15.153:0/3593545283 reap_dead start 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f8350b3a640 1 -- 172.21.15.153:0/3593545283 reap_dead start 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 shutdown_connections 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 >> 172.21.15.153:0/3593545283 conn(0x7f834c0672b0 msgr2=0x7f834c129840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 shutdown_connections 2024-09-15T03:39:29.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:29.167+0000 7f835259d640 1 -- 172.21.15.153:0/3593545283 wait complete. 2024-09-15T03:39:29.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:39:29.193 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:39:29.193 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T03:39:29.193 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:39:29.193 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:39:29.193 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:39:29.193 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:39:29.205 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:39:29.205 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:39:29.205 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:39:29.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:39:44.207 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:39:44.208 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:39:44.208 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:39:44.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:39:44.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.312+0000 7f495928b640 10 monclient: get_monmap_and_config 2024-09-15T03:39:44.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.312+0000 7f495928b640 10 monclient: build_initial_monmap 2024-09-15T03:39:44.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.312+0000 7f495928b640 10 monclient: monmap: 2024-09-15T03:39:44.313 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:44.314 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:44.314 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:44.313153+0000 2024-09-15T03:39:44.314 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:44.313153+0000 2024-09-15T03:39:44.314 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:44.314 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:44.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:44.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:44.315 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:44.315 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:44.315 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:44.315 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 1 Processor -- start 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 1 -- start start 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient: init 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4954131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f495409e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 0x7f4954139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient(hunting): picked mon.noname-a con 0x7f4954137bc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495409faf0 con 0x7f495409e910 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495409fd30 con 0x7f4954131ab0 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4954132590 con 0x7f4954137bc0 2024-09-15T03:39:44.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f495928b640 10 monclient(hunting): authenticate will time out at 1381.952708s 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f49527fc640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f495409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f49537fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 msgr2=0x7f4954139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f49537fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 0x7f4954139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f4952ffd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4954131ab0 0x7f4954131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:39828/0 (socket says 172.21.15.153:39828) 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.313+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 learned_addr learned my addr 172.21.15.153:0/761658119 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:39:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.316+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.181:6789/0 conn(0x7f4954131ab0 legacy=0x7f4954131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.514+0000 7f49537fe640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 msgr2=0x7f4954139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.514+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f495409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:39:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.514+0000 7f49537fe640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 0x7f4954139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:39:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.915+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f495409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:39:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.915+0000 7f49537fe640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 msgr2=0x7f4954139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:44.915+0000 7f49537fe640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 0x7f4954139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:39:45.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:45.716+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f495409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:39:45.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:45.716+0000 7f49537fe640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 msgr2=0x7f4954139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:45.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:45.716+0000 7f49537fe640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 0x7f4954139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:39:47.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): tick 2024-09-15T03:39:47.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 monclient(hunting): continuing hunt 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 msgr2=0x7f4954139fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4954137bc0 0x7f4954139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.181:6789/0 conn(0x7f4954131ab0 legacy=0x7f4954131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f495409ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient: _add_conns ranks=[2,4] 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f4934003730 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f49340062a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:47.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4954132590 con 0x7f49340062a0 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4934008e10 con 0x7f4934003730 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4950ff9640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 10 monclient(hunting): get_auth_request con 0x7f4934003730 auth_method 0 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:47.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 10 monclient(hunting): get_auth_request con 0x7f49340062a0 auth_method 0 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:47.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.313+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:47.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 10 monclient(hunting): get_auth_request con 0x7f4934003730 auth_method 0 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 10 monclient(hunting): get_auth_request con 0x7f49340062a0 auth_method 0 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:47.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:47.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:47.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:47.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:47.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.514+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:39:47.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:47.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 10 monclient(hunting): get_auth_request con 0x7f4934003730 auth_method 0 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 10 monclient(hunting): get_auth_request con 0x7f49340062a0 auth_method 0 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:47.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:47.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:47.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:47.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:47.915+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:39:48.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:48.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 10 monclient(hunting): get_auth_request con 0x7f4934003730 auth_method 0 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 10 monclient(hunting): get_auth_request con 0x7f49340062a0 auth_method 0 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:48.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:48.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:39:48.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:39:48.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:48.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:48.716+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:39:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.312+0000 7f4950ff9640 10 monclient(hunting): tick 2024-09-15T03:39:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 monclient(hunting): continuing hunt 2024-09-15T03:39:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 msgr2=0x7f4934005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4934003730 0x7f4934005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 msgr2=0x7f4934008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.181:3300/0 conn(0x7f49340062a0 0x7f4934008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f495409e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f49537fe640 1 -- 172.21.15.153:0/761658119 reap_dead start 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4954131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4934003b60 0x7f4934006740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f4934003b60 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49540a1110 con 0x7f495409e910 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495409faf0 con 0x7f4954131ab0 2024-09-15T03:39:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f493400b1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:39:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 1 -- 172.21.15.153:0/761658119 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4954132590 con 0x7f4934003b60 2024-09-15T03:39:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f4950ff9640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f49537fe640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4934003b60 msgr2=0x7f4934006740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.313+0000 7f49537fe640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4934003b60 0x7f4934006740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.315+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.181:6789/0 conn(0x7f4954131ab0 legacy=0x7f493400e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.315+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792662396 0 0) 0x7f495409faf0 con 0x7f4954131ab0 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.315+0000 7f4951ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.315+0000 7f4951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.315+0000 7f4951ffb640 10 monclient(hunting): my global_id is 144148 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.315+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49400034e0 con 0x7f4954131ab0 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894622297 0 0) 0x7f49400034e0 con 0x7f4954131ab0 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4934003b60 msgr2=0x7f4934006740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 --2- 172.21.15.153:0/761658119 >> v2:172.21.15.153:3300/0 conn(0x7f4934003b60 0x7f4934006740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.153:6789/0 conn(0x7f495409e910 legacy=0x7f493400b1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:39:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934007cd0 con 0x7f4954131ab0 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f493c007460 con 0x7f4954131ab0 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f495928b640 5 monclient: authenticate success, global_id 144148 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f495928b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: dump: 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 >> v1:172.21.15.181:6789/0 conn(0x7f4954131ab0 legacy=0x7f493400e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49400039d0 0x7f4940003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): picked mon.a con 0x7f49400039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 0x7f4940004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): picked mon.b con 0x7f4940004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4940005230 0x7f494000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): picked mon.c con 0x7f4940005230 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4954132590 con 0x7f49400039d0 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494000c1f0 con 0x7f4940004780 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494000c3b0 con 0x7f4940005230 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4952ffd640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49400039d0 msgr2=0x7f4940003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:50.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4951ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f495928b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f4952ffd640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49400039d0 0x7f4940003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 0x7f4940004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4940005230 0x7f494000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): get_auth_request con 0x7f4940004780 auth_method 0 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): get_auth_request con 0x7f4940005230 auth_method 0 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:39:50.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient(hunting): handle_auth_done global_id 144148 payload 293 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4940005230 msgr2=0x7f494000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4940005230 0x7f494000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49400039d0 msgr2=0x7f4940003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49400039d0 0x7f4940003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 monclient: found mon.b 2024-09-15T03:39:50.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49400034e0 con 0x7f4940004780 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.316+0000 7f49537fe640 1 -- 172.21.15.153:0/761658119 reap_dead start 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.319+0000 7f49527fc640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 msgr2=0x7f4940004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.319+0000 7f49527fc640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 0x7f4940004bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f493c000c00 tx=0x7f493c007960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd761e3c55a1163b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c00b0d0 con 0x7f4940004780 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: dump: 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c02e050 con 0x7f4940004780 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 1 -- 172.21.15.153:0/761658119 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c009570 con 0x7f4940004780 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:50.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 4 set_mon_vals no callback set 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f4951ffb640 10 monclient: dump: 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.330 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 10 monclient: shutdown 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 1 -- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 msgr2=0x7f4940004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 0x7f4940004bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f493c000c00 tx=0x7f493c007960 comp rx=0 tx=0).stop 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 1 -- 172.21.15.153:0/761658119 shutdown_connections 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 1 --2- 172.21.15.153:0/761658119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4940004780 0x7f4940004bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.320+0000 7f495928b640 1 -- 172.21.15.153:0/761658119 >> 172.21.15.153:0/761658119 conn(0x7f4954067d30 msgr2=0x7f49540a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.321+0000 7f495928b640 1 -- 172.21.15.153:0/761658119 shutdown_connections 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.321+0000 7f495928b640 1 -- 172.21.15.153:0/761658119 wait complete. 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.321+0000 7f495928b640 10 monclient: build_initial_monmap 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.321+0000 7f495928b640 10 monclient: monmap: 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:50.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 Processor -- start 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 -- start start 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient: init 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4940004780 0x7f4954141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): picked mon.noname-a con 0x7f4940004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 0x7f4954141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): picked mon.noname-b con 0x7f4954141560 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 0x7f49540989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): picked mon.noname-c con 0x7f49540985b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495409acd0 con 0x7f4940004780 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495409bde0 con 0x7f4954141560 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49540a3e40 con 0x7f49540985b0 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f4952ffd640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4940004780 msgr2=0x7f4954141020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.322+0000 7f495928b640 10 monclient(hunting): authenticate will time out at 1387.962987s 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f4952ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4940004780 0x7f4954141020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49537fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 0x7f49540989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49537fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 msgr2=0x7f49540989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49537fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 msgr2=0x7f49540989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49537fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 0x7f49540989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:39:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49537fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 0x7f49540989c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 0x7f4954141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 0x7f4954141930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:34478/0 (socket says 172.21.15.153:34478) 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 -- 172.21.15.153:0/2357506305 learned_addr learned my addr 172.21.15.153:0/2357506305 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): get_auth_request con 0x7f4954141560 auth_method 0 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient(hunting): handle_auth_done global_id 144151 payload 824 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 -- 172.21.15.153:0/2357506305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 msgr2=0x7f49540989c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:39:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 0x7f49540989c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 -- 172.21.15.153:0/2357506305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4940004780 msgr2=0x7f4954141020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4940004780 0x7f4954141020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 monclient: found mon.noname-b 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.323+0000 7f49527fc640 1 -- 172.21.15.153:0/2357506305 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4954141e70 con 0x7f4954141560 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.325+0000 7f49527fc640 1 -- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 msgr2=0x7f4954141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.325+0000 7f49527fc640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 0x7f4954141930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f493c000c00 tx=0x7f493c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c0099a0 con 0x7f4954141560 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: dump: 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c00f040 con 0x7f4954141560 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c009d10 con 0x7f4954141560 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 5 monclient: authenticate success, global_id 144151 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f494affd640 10 monclient: dump: 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 10 monclient: _renew_subs 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f49517fa640 4 set_mon_vals no callback set 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495409a610 con 0x7f4954141560 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 10 monclient: _renew_subs 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.326+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495409a100 con 0x7f4954141560 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f493c032400 con 0x7f4954141560 2024-09-15T03:39:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f495928b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f495928b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f495928b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f494affd640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49340393d0 0x7f493403b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495404bf60 con 0x7f4954141560 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f493c06cf90 con 0x7f4954141560 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f4952ffd640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49340393d0 0x7f493403b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.327+0000 7f4952ffd640 10 monclient: get_auth_request con 0x7f49340393d0 auth_method 0 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.330+0000 7f4952ffd640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49340393d0 0x7f493403b880 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4944007c10 tx=0x7f4944009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.332+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f493c0326b0 con 0x7f4954141560 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.332+0000 7f494affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.332+0000 7f494affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:50.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.549+0000 7f495928b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:39:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.549+0000 7f495928b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.549+0000 7f495928b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:39:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.549+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f495411ba80 con 0x7f4954141560 2024-09-15T03:39:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.550+0000 7f494affd640 1 -- 172.21.15.153:0/2357506305 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f493c042030 con 0x7f4954141560 2024-09-15T03:39:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.550+0000 7f494affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:39:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.550+0000 7f494affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:50.553 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":116,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:39:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49340393d0 msgr2=0x7f493403b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49340393d0 0x7f493403b880 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4944007c10 tx=0x7f4944009040 comp rx=0 tx=0).stop 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 10 monclient: shutdown 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 msgr2=0x7f4954141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 0x7f4954141930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f493c000c00 tx=0x7f493c037000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 shutdown_connections 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f49340393d0 0x7f493403b880 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49540985b0 0x7f49540989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4954141560 0x7f4954141930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 --2- 172.21.15.153:0/2357506305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4940004780 0x7f4954141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 >> 172.21.15.153:0/2357506305 conn(0x7f4954067d30 msgr2=0x7f495409d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:50.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 shutdown_connections 2024-09-15T03:39:50.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.552+0000 7f495928b640 1 -- 172.21.15.153:0/2357506305 wait complete. 2024-09-15T03:39:50.565 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:39:50.565 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:39:50.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.656+0000 7f4384ef8640 10 monclient: get_monmap_and_config 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.656+0000 7f4384ef8640 10 monclient: build_initial_monmap 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.656+0000 7f4384ef8640 10 monclient: monmap: 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:50.658170+0000 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:50.658170+0000 2024-09-15T03:39:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:50.659 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.656+0000 7f4384ef8640 1 Processor -- start 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 -- start start 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient: init 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 0x7f438009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f4380096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f438009aaa0 0x7f43800971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f438009aaa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4380097720 0x7f438012e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f4380097720 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4380099620 con 0x7f438009aaa0 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4380099340 con 0x7f4380097720 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4380099900 con 0x7f4380096e10 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f4384ef8640 10 monclient(hunting): authenticate will time out at 1388.297643s 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437dd9b640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f438009aaa0 msgr2=0x7f43800971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437dd9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f438009aaa0 0x7f43800971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 0x7f438009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 0x7f438009a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38260/0 (socket says 172.21.15.153:38260) 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 1 -- 172.21.15.153:0/2378662749 learned_addr learned my addr 172.21.15.153:0/2378662749 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): get_auth_request con 0x7f4380096e10 auth_method 0 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.657+0000 7f437ed9d640 1 --2- 172.21.15.153:0/2378662749 >> v2:172.21.15.181:3300/0 conn(0x7f4380097720 0x7f438012e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 10 monclient(hunting): handle_auth_done global_id 124412 payload 274 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 1 -- 172.21.15.153:0/2378662749 >> v2:172.21.15.181:3300/0 conn(0x7f4380097720 msgr2=0x7f438012e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 1 --2- 172.21.15.153:0/2378662749 >> v2:172.21.15.181:3300/0 conn(0x7f4380097720 0x7f438012e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 1 -- 172.21.15.153:0/2378662749 >> v2:172.21.15.153:3300/0 conn(0x7f438009aaa0 msgr2=0x7f43800971e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 1 --2- 172.21.15.153:0/2378662749 >> v2:172.21.15.153:3300/0 conn(0x7f438009aaa0 0x7f43800971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 1 monclient: found mon.noname-b 2024-09-15T03:39:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.658+0000 7f437e59c640 1 -- 172.21.15.153:0/2378662749 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4380097b10 con 0x7f4380096e10 2024-09-15T03:39:50.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.660+0000 7f437e59c640 1 -- 172.21.15.153:0/2378662749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 msgr2=0x7f438009a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.660+0000 7f437e59c640 1 --2- 172.21.15.153:0/2378662749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 0x7f438009a560 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f438009aff0 tx=0x7f43680091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e13ede03c7cbb0b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.660+0000 7f437d59a640 1 -- 172.21.15.153:0/2378662749 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4368032440 con 0x7f4380096e10 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.660+0000 7f437d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.660+0000 7f437d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.660+0000 7f437d59a640 10 monclient: dump: 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 1 -- 172.21.15.153:0/2378662749 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43680325a0 con 0x7f4380096e10 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 5 monclient: authenticate success, global_id 124412 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 1 -- 172.21.15.153:0/2378662749 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4368032980 con 0x7f4380096e10 2024-09-15T03:39:50.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 4 set_mon_vals no callback set 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f437d59a640 10 monclient: dump: 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 10 monclient: shutdown 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 -- 172.21.15.153:0/2378662749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 msgr2=0x7f438009a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2378662749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 0x7f438009a560 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f438009aff0 tx=0x7f43680091d0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 -- 172.21.15.153:0/2378662749 shutdown_connections 2024-09-15T03:39:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2378662749 >> v2:172.21.15.181:3300/0 conn(0x7f4380097720 0x7f438012e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2378662749 >> v2:172.21.15.153:3300/0 conn(0x7f438009aaa0 0x7f43800971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2378662749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380096e10 0x7f438009a560 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 -- 172.21.15.153:0/2378662749 >> 172.21.15.153:0/2378662749 conn(0x7f43800672b0 msgr2=0x7f43800676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 -- 172.21.15.153:0/2378662749 shutdown_connections 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.661+0000 7f4384ef8640 1 -- 172.21.15.153:0/2378662749 wait complete. 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient: build_initial_monmap 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient: monmap: 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 Processor -- start 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 -- start start 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient: init 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380096e10 0x7f438012d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f4380096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4380097720 0x7f4380117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f4380097720 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f438009aaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801320f0 con 0x7f4380096e10 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438009b050 con 0x7f4380097720 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43801307f0 con 0x7f438009aaa0 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f4384ef8640 10 monclient(hunting): authenticate will time out at 1388.302924s 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f437e59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380096e10 msgr2=0x7f438012d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.662+0000 7f437e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380096e10 0x7f438012d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38274/0 (socket says 172.21.15.153:38274) 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 -- 172.21.15.153:0/2878181724 learned_addr learned my addr 172.21.15.153:0/2878181724 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): get_auth_request con 0x7f438009aaa0 auth_method 0 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437dd9b640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4380097720 0x7f4380117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient(hunting): handle_auth_done global_id 124415 payload 824 2024-09-15T03:39:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4380097720 msgr2=0x7f4380117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4380097720 0x7f4380117480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380096e10 msgr2=0x7f438012d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4380096e10 0x7f438012d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.663+0000 7f437ed9d640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4368009450 con 0x7f438009aaa0 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f437ed9d640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 msgr2=0x7f43801179c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f437ed9d640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f437800cee0 tx=0x7f4378007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437800e610 con 0x7f438009aaa0 2024-09-15T03:39:50.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f437ed9d640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f437800cee0 tx=0x7f4378007870 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f437800e770 config(0 keys) v1 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 10 monclient: dump: 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437800e770 con 0x7f438009aaa0 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 5 monclient: authenticate success, global_id 124415 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f43767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 10 monclient: _renew_subs 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438011ac50 con 0x7f438009aaa0 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f437cd99640 4 set_mon_vals no callback set 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 10 monclient: _renew_subs 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.665+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438013da90 con 0x7f438009aaa0 2024-09-15T03:39:50.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f4384ef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437800eb30 con 0x7f438009aaa0 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 0x7f4380117f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4380065f20 con 0x7f438012dab0 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f43767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f43767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f43767fc640 10 monclient: dump: 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.684 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4378009030 con 0x7f438009aaa0 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 0x7f4380117f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient: get_auth_request con 0x7f438012dab0 auth_method 0 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f43767fc640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43600386f0 0x7f436003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 10 monclient(hunting): handle_auth_done global_id 124418 payload 824 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.666+0000 7f437e59c640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f438004bf60 con 0x7f438012dab0 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.667+0000 7f437dd9b640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43600386f0 0x7f436003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.667+0000 7f437dd9b640 10 monclient: get_auth_request con 0x7f43600386f0 auth_method 0 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f437e59c640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 msgr2=0x7f4380117f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f437e59c640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 0x7f4380117f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43800a0d60 tx=0x7f4368032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4368032440 con 0x7f438012dab0 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:50.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 10 monclient: dump: 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4368032960 con 0x7f438012dab0 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:50.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 msgr2=0x7f4380117f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 0x7f4380117f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43800a0d60 tx=0x7f4368032000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 msgr2=0x7f4380117f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:50.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f43767fc640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438012dab0 0x7f4380117f00 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43800a0d60 tx=0x7f4368032000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.669+0000 7f437dd9b640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43600386f0 0x7f436003aba0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4370006fd0 tx=0x7f4370008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f4384ef8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 0x7f438006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f4384ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438012de80 con 0x7f4380118440 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 0x7f438006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient: get_auth_request con 0x7f4380118440 auth_method 0 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 10 monclient(hunting): handle_auth_done global_id 124421 payload 824 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.675+0000 7f437e59c640 1 -- 172.21.15.153:0/2878181724 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4380118840 con 0x7f4380118440 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f437e59c640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 msgr2=0x7f438006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f437e59c640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 0x7f438006b6b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4368009210 tx=0x7f436803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4368031f00 con 0x7f4380118440 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 10 monclient: dump: 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f436803ce10 con 0x7f4380118440 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 msgr2=0x7f438006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 0x7f438006b6b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4368009210 tx=0x7f436803c000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 msgr2=0x7f438006b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.677+0000 7f43767fc640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4380118440 0x7f438006b6b0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4368009210 tx=0x7f436803c000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 116, 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:50.693 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:50.694 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:50.695 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:50.696 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:50.697 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.698 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:50.700 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:50.701 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:50.703 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.704 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:39:50.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:50.706 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43600386f0 msgr2=0x7f436003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43600386f0 0x7f436003aba0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4370006fd0 tx=0x7f4370008040 comp rx=0 tx=0).stop 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f437ed9d640 1 -- 172.21.15.153:0/2878181724 reap_dead start 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 10 monclient: shutdown 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 msgr2=0x7f43801179c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f437800cee0 tx=0x7f4378007870 comp rx=0 tx=0).stop 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f43767fc640 1 -- 172.21.15.153:0/2878181724 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7f4378012070 con 0x7f438009aaa0 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 shutdown_connections 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 --2- 172.21.15.153:0/2878181724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f438009aaa0 0x7f43801179c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 >> 172.21.15.153:0/2878181724 conn(0x7f43800672b0 msgr2=0x7f438012f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 shutdown_connections 2024-09-15T03:39:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.678+0000 7f4384ef8640 1 -- 172.21.15.153:0/2878181724 wait complete. 2024-09-15T03:39:50.709 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:39:50.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:39:50.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.801+0000 7f6bbf60f640 10 monclient: get_monmap_and_config 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.801+0000 7f6bbf60f640 10 monclient: build_initial_monmap 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.802+0000 7f6bbf60f640 10 monclient: monmap: 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:39:50.803756+0000 2024-09-15T03:39:50.804 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:39:50.803756+0000 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.802+0000 7f6bbf60f640 1 Processor -- start 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.802+0000 7f6bbf60f640 1 -- start start 2024-09-15T03:39:50.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.802+0000 7f6bbf60f640 10 monclient: init 2024-09-15T03:39:50.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:50.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:39:50.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb809a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb8096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:50.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bb809aaa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:39:50.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f6bb80978f0 0x7f6bb8135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb80978f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:39:50.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb809d4a0 con 0x7f6bb809aaa0 2024-09-15T03:39:50.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb8099620 con 0x7f6bb80978f0 2024-09-15T03:39:50.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8099340 con 0x7f6bb8096e10 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbf60f640 10 monclient(hunting): authenticate will time out at 1388.443291s 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb809a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38298/0 (socket says 172.21.15.153:38298) 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/3215502052 learned_addr learned my addr 172.21.15.153:0/3215502052 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): get_auth_request con 0x7f6bb8096e10 auth_method 0 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbdbac640 1 --2- 172.21.15.153:0/3215502052 >> v2:172.21.15.181:3300/0 conn(0x7f6bb80978f0 0x7f6bb8135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbdbac640 10 monclient(hunting): get_auth_request con 0x7f6bb80978f0 auth_method 0 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbdbac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbdbac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbdbac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_done global_id 124424 payload 274 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/3215502052 >> v2:172.21.15.181:3300/0 conn(0x7f6bb80978f0 msgr2=0x7f6bb8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 --2- 172.21.15.153:0/3215502052 >> v2:172.21.15.181:3300/0 conn(0x7f6bb80978f0 0x7f6bb8135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/3215502052 >> v1:172.21.15.181:6789/0 conn(0x7f6bb809aaa0 legacy=0x7f6bb80971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 monclient: found mon.noname-b 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.803+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/3215502052 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8099900 con 0x7f6bb8096e10 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/3215502052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 msgr2=0x7f6bb809a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bbd3ab640 1 --2- 172.21.15.153:0/3215502052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb809a560 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009910 tx=0x7f6bb00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d376e7c5f8fcbaa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 1 -- 172.21.15.153:0/3215502052 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb002aea0 con 0x7f6bb8096e10 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: dump: 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.816 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bbf60f640 5 monclient: authenticate success, global_id 124424 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bbf60f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 1 -- 172.21.15.153:0/3215502052 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb0032440 con 0x7f6bb8096e10 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:50.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bbf60f640 10 monclient: get_monmap_and_config success 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bbf60f640 4 set_mon_vals no callback set 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 1 -- 172.21.15.153:0/3215502052 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00327e0 con 0x7f6bb8096e10 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.806+0000 7f6bb7fff640 10 monclient: dump: 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 10 monclient: shutdown 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 -- 172.21.15.153:0/3215502052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 msgr2=0x7f6bb809a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/3215502052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb809a560 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009910 tx=0x7f6bb00091d0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 -- 172.21.15.153:0/3215502052 shutdown_connections 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/3215502052 >> v2:172.21.15.181:3300/0 conn(0x7f6bb80978f0 0x7f6bb8135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/3215502052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb809a560 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 -- 172.21.15.153:0/3215502052 >> 172.21.15.153:0/3215502052 conn(0x7f6bb80672b0 msgr2=0x7f6bb80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 -- 172.21.15.153:0/3215502052 shutdown_connections 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 1 -- 172.21.15.153:0/3215502052 wait complete. 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 10 monclient: build_initial_monmap 2024-09-15T03:39:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.807+0000 7f6bbf60f640 10 monclient: monmap: 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 Processor -- start 2024-09-15T03:39:50.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 -- start start 2024-09-15T03:39:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient: init 2024-09-15T03:39:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient: _reopen_session rank -1 2024-09-15T03:39:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:39:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb8135330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb8096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:39:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb80978f0 0x7f6bb806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb80978f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:39:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb809aaa0 0x7f6bb806e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb809aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:39:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb806f090 con 0x7f6bb80978f0 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb809b050 con 0x7f6bb809aaa0 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8130450 con 0x7f6bb8096e10 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): _renew_subs 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbf60f640 10 monclient(hunting): authenticate will time out at 1388.448837s 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbcbaa640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb80978f0 msgr2=0x7f6bb806dc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbcbaa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb80978f0 0x7f6bb806dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb8135330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb8135330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38304/0 (socket says 172.21.15.153:38304) 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/1879184351 learned_addr learned my addr 172.21.15.153:0/1879184351 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 10 monclient(hunting): get_auth_request con 0x7f6bb8096e10 auth_method 0 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.808+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbdbac640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb809aaa0 0x7f6bb806e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbdbac640 10 monclient(hunting): get_auth_request con 0x7f6bb809aaa0 auth_method 0 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbdbac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbdbac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbdbac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 10 monclient(hunting): handle_auth_done global_id 124427 payload 824 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb809aaa0 msgr2=0x7f6bb806e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb809aaa0 0x7f6bb806e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb80978f0 msgr2=0x7f6bb806dc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb80978f0 0x7f6bb806dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 10 monclient: _finish_hunting 0 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 1 monclient: found mon.noname-c 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.809+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0009450 con 0x7f6bb8096e10 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.811+0000 7f6bbd3ab640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 msgr2=0x7f6bb8135330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.811+0000 7f6bbd3ab640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb8135330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0035040 tx=0x7f6bb0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.811+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb002aea0 con 0x7f6bb8096e10 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: dump: 2024-09-15T03:39:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: _finish_auth 0 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb0031c70 con 0x7f6bb8096e10 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 5 monclient: authenticate success, global_id 124427 2024-09-15T03:39:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 10 monclient: _renew_subs 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb8130730 con 0x7f6bb8096e10 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb003d440 con 0x7f6bb8096e10 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb57fa640 10 monclient: dump: 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 10 monclient: _renew_subs 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb812f670 con 0x7f6bb8096e10 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.813+0000 7f6bb77fe640 4 set_mon_vals no callback set 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbf60f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 0x7f6bb812be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8130af0 con 0x7f6bb806e680 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6bb0039040 con 0x7f6bb8096e10 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bb57fa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ba4038740 0x7f6ba403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbcbaa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 0x7f6bb812be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6bb006c1b0 con 0x7f6bb8096e10 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbcbaa640 10 monclient: get_auth_request con 0x7f6bb806e680 auth_method 0 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbcbaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbcbaa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.814+0000 7f6bbcbaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbdbac640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ba4038740 0x7f6ba403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbdbac640 10 monclient: get_auth_request con 0x7f6ba4038740 auth_method 0 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_done global_id 144160 payload 824 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.815+0000 7f6bbcbaa640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bb81305c0 con 0x7f6bb806e680 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bbcbaa640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 msgr2=0x7f6bb812be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bbcbaa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 0x7f6bb812be60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6bb806f8d0 tx=0x7f6bac002670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bbdbac640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ba4038740 0x7f6ba403abf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800bb40 tx=0x7f6ba8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac0070f0 con 0x7f6bb806e680 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.817+0000 7f6bb57fa640 10 monclient: dump: 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6bac007250 con 0x7f6bb806e680 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 msgr2=0x7f6bb812be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 0x7f6bb812be60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6bb806f8d0 tx=0x7f6bac002670 comp rx=0 tx=0).stop 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 msgr2=0x7f6bb812be60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.818+0000 7f6bb57fa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb806e680 0x7f6bb812be60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.823+0000 7f6bbf60f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.823+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 0x7f6bb812c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.823+0000 7f6bbf60f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.823+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8065f20 con 0x7f6bb812c3a0 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 0x7f6bb812c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient: get_auth_request con 0x7f6bb812c3a0 auth_method 0 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:39:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 10 monclient(hunting): handle_auth_done global_id 144163 payload 824 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.824+0000 7f6bbcbaa640 1 -- 172.21.15.153:0/1879184351 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6bb81308f0 con 0x7f6bb812c3a0 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.826+0000 7f6bbcbaa640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 msgr2=0x7f6bb812c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.826+0000 7f6bbcbaa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 0x7f6bb812c7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6bac002920 tx=0x7f6bac016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.826+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac021840 con 0x7f6bb812c3a0 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.826+0000 7f6bb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 10 monclient: dump: 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:39:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f6bac0219a0 con 0x7f6bb812c3a0 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 msgr2=0x7f6bb812c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 0x7f6bb812c7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6bac002920 tx=0x7f6bac016000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 msgr2=0x7f6bb812c7a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:39:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bb57fa640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bb812c3a0 0x7f6bb812c7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 116, 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:39:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:39:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:39:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:39:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:39:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:39:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:39:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:39:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:39:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:39:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:39:50.852 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:39:50.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ba4038740 msgr2=0x7f6ba403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ba4038740 0x7f6ba403abf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800bb40 tx=0x7f6ba8008040 comp rx=0 tx=0).stop 2024-09-15T03:39:50.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 10 monclient: shutdown 2024-09-15T03:39:50.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 msgr2=0x7f6bb8135330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb8135330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0035040 tx=0x7f6bb0032000 comp rx=0 tx=0).stop 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbdbac640 1 -- 172.21.15.153:0/1879184351 reap_dead start 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 shutdown_connections 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 --2- 172.21.15.153:0/1879184351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bb8096e10 0x7f6bb8135330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 >> 172.21.15.153:0/1879184351 conn(0x7f6bb80672b0 msgr2=0x7f6bb809fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 shutdown_connections 2024-09-15T03:39:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:39:50.827+0000 7f6bbf60f640 1 -- 172.21.15.153:0/1879184351 wait complete. 2024-09-15T03:39:50.854 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:39:50.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:40:10.855 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:40:10.855 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:40:10.856 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:40:10.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:40:10.859 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:40:10.859 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:40:10.859 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:40:10.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:40:25.860 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:40:25.861 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:40:25.861 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:40:25.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:40:25.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.965+0000 7eff7b02b640 10 monclient: get_monmap_and_config 2024-09-15T03:40:25.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.965+0000 7eff7b02b640 10 monclient: build_initial_monmap 2024-09-15T03:40:25.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.965+0000 7eff7b02b640 10 monclient: monmap: 2024-09-15T03:40:25.966 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:25.966 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:25.966 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:25.966205+0000 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:25.966205+0000 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:25.967 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:25.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 1 Processor -- start 2024-09-15T03:40:25.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 1 -- start start 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient: init 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff740975f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff740979c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient(hunting): picked mon.noname-a con 0x7eff74097dc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff74098200 con 0x7eff740979c0 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7409d1b0 con 0x7eff740975f0 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff740961e0 con 0x7eff74097dc0 2024-09-15T03:40:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff7b02b640 10 monclient(hunting): authenticate will time out at 1423.605764s 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37656/0 (socket says 172.21.15.153:37656) 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff6bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7eff740979c0 0x7eff741349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46730/0 (socket says 172.21.15.153:46730) 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 learned_addr learned my addr 172.21.15.153:0/3205002868 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 10 monclient(hunting): get_auth_request con 0x7eff74097dc0 auth_method 0 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.966+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.967+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.969+0000 7eff6bfff640 1 -- 172.21.15.153:0/3205002868 >> v1:172.21.15.153:6789/0 conn(0x7eff740979c0 legacy=0x7eff741349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:25.969+0000 7eff78dc7640 1 -- 172.21.15.153:0/3205002868 >> v1:172.21.15.181:6789/0 conn(0x7eff740975f0 legacy=0x7eff74095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 10 monclient(hunting): get_auth_request con 0x7eff74097dc0 auth_method 0 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:26.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.168+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 10 monclient(hunting): get_auth_request con 0x7eff74097dc0 auth_method 0 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:26.569+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.079+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980971533 0 0) 0x7eff74098200 con 0x7eff740979c0 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.079+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.079+0000 7eff6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.079+0000 7eff6b7fe640 10 monclient(hunting): my global_id is 154098 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.079+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c0034e0 con 0x7eff740979c0 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483756419 0 0) 0x7eff7409d1b0 con 0x7eff740975f0 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): my global_id is 144166 2024-09-15T03:40:27.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff74098200 con 0x7eff740975f0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3576347878 0 0) 0x7eff5c0034e0 con 0x7eff740979c0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 msgr2=0x7eff74138120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 --2- 172.21.15.153:0/3205002868 >> v2:172.21.15.153:3300/0 conn(0x7eff74097dc0 0x7eff74138120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 >> v1:172.21.15.181:6789/0 conn(0x7eff740975f0 legacy=0x7eff74095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7409d420 con 0x7eff740979c0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7eff64001710 con 0x7eff740979c0 2024-09-15T03:40:27.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff7b02b640 5 monclient: authenticate success, global_id 154098 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff7b02b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: dump: 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.086 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 >> v1:172.21.15.153:6789/0 conn(0x7eff740979c0 legacy=0x7eff741349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c003c30 0x7eff5c004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): picked mon.b con 0x7eff5c003c30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff5c004780 0x7eff5c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): picked mon.a con 0x7eff5c004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 0x7eff5c00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): picked mon.c con 0x7eff5c005310 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740961e0 con 0x7eff5c004780 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00c2d0 con 0x7eff5c003c30 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff5c004780 0x7eff5c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 0x7eff5c00b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00c490 con 0x7eff5c005310 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff7b02b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): get_auth_request con 0x7eff5c005310 auth_method 0 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): get_auth_request con 0x7eff5c004780 auth_method 0 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c003c30 0x7eff5c004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): get_auth_request con 0x7eff5c003c30 auth_method 0 2024-09-15T03:40:27.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff78dc7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient(hunting): handle_auth_done global_id 154098 payload 293 2024-09-15T03:40:27.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c003c30 msgr2=0x7eff5c004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c003c30 0x7eff5c004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff5c004780 msgr2=0x7eff5c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff5c004780 0x7eff5c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 monclient: found mon.c 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0034e0 con 0x7eff5c005310 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.080+0000 7eff795c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.081+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 reap_dead start 2024-09-15T03:40:27.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.083+0000 7eff795c8640 1 -- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 msgr2=0x7eff5c00b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.083+0000 7eff795c8640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 0x7eff5c00b9f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009cd0 tx=0x7eff6c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d662b9847fdb5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c02c750 con 0x7eff5c005310 2024-09-15T03:40:27.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: dump: 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.098 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c02c8b0 con 0x7eff5c005310 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 1 -- 172.21.15.153:0/3205002868 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c02ca60 con 0x7eff5c005310 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 4 set_mon_vals no callback set 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff6b7fe640 10 monclient: dump: 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 10 monclient: shutdown 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 1 -- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 msgr2=0x7eff5c00b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.084+0000 7eff7b02b640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 0x7eff5c00b9f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009cd0 tx=0x7eff6c009040 comp rx=0 tx=0).stop 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 1 -- 172.21.15.153:0/3205002868 shutdown_connections 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 1 --2- 172.21.15.153:0/3205002868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff5c005310 0x7eff5c00b9f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 1 -- 172.21.15.153:0/3205002868 >> 172.21.15.153:0/3205002868 conn(0x7eff740678a0 msgr2=0x7eff740a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 1 -- 172.21.15.153:0/3205002868 shutdown_connections 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 1 -- 172.21.15.153:0/3205002868 wait complete. 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.085+0000 7eff7b02b640 10 monclient: monmap: 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:27.102 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 1 Processor -- start 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 1 -- start start 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 10 monclient: init 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c005310 0x7eff740942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 10 monclient(hunting): picked mon.noname-b con 0x7eff5c005310 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.086+0000 7eff7b02b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 0x7eff74094820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): picked mon.noname-a con 0x7eff7412a390 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff74128500 0x7eff74128910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): picked mon.noname-c con 0x7eff74128500 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff74129f70 con 0x7eff7412a390 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7412bfa0 con 0x7eff5c005310 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7412ab80 con 0x7eff74128500 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff7b02b640 10 monclient(hunting): authenticate will time out at 1424.726135s 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 0x7eff74094820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c005310 0x7eff740942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 0x7eff74094820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37718/0 (socket says 172.21.15.153:37718) 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 -- 172.21.15.153:0/610180730 learned_addr learned my addr 172.21.15.153:0/610180730 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): get_auth_request con 0x7eff7412a390 auth_method 0 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): get_auth_request con 0x7eff5c005310 auth_method 0 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff74128500 0x7eff74128910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): get_auth_request con 0x7eff74128500 auth_method 0 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff795c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient(hunting): handle_auth_done global_id 154101 payload 824 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 -- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff74128500 msgr2=0x7eff74128910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff74128500 0x7eff74128910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 -- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c005310 msgr2=0x7eff740942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c005310 0x7eff740942e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff6bfff640 1 -- 172.21.15.153:0/610180730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7412bcc0 con 0x7eff7412a390 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.087+0000 7eff78dc7640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c005310 0x7eff740942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.089+0000 7eff6bfff640 1 -- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 msgr2=0x7eff74094820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.089+0000 7eff6bfff640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 0x7eff74094820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7eff64007790 tx=0x7eff64007990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64018030 con 0x7eff7412a390 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: dump: 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff640117a0 con 0x7eff7412a390 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 5 monclient: authenticate success, global_id 154101 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 10 monclient: _renew_subs 2024-09-15T03:40:27.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff741326e0 con 0x7eff7412a390 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64011b60 con 0x7eff7412a390 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff68ff9640 10 monclient: dump: 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.109 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff6affd640 4 set_mon_vals no callback set 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 10 monclient: _renew_subs 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.090+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7412cbd0 con 0x7eff7412a390 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7eff6401b070 con 0x7eff7412a390 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff7b02b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff7b02b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff7b02b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff68ff9640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff60038740 0x7eff6003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7412cdc0 con 0x7eff7412a390 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7eff64019070 con 0x7eff7412a390 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff78dc7640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff60038740 0x7eff6003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.091+0000 7eff78dc7640 10 monclient: get_auth_request con 0x7eff60038740 auth_method 0 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.094+0000 7eff78dc7640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff60038740 0x7eff6003abf0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7eff700014b0 tx=0x7eff70001b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.096+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7eff6401bb00 con 0x7eff7412a390 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.096+0000 7eff68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.096+0000 7eff68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.312+0000 7eff7b02b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.312+0000 7eff7b02b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.312+0000 7eff7b02b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.312+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7eff7404bf60 con 0x7eff7412a390 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.313+0000 7eff68ff9640 1 -- 172.21.15.153:0/610180730 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7eff6401b320 con 0x7eff7412a390 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.313+0000 7eff68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.313+0000 7eff68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:27.314 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:40:27.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff60038740 msgr2=0x7eff6003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff60038740 0x7eff6003abf0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7eff700014b0 tx=0x7eff70001b50 comp rx=0 tx=0).stop 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 10 monclient: shutdown 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 msgr2=0x7eff74094820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 0x7eff74094820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7eff64007790 tx=0x7eff64007990 comp rx=0 tx=0).stop 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 shutdown_connections 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7eff60038740 0x7eff6003abf0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff74128500 0x7eff74128910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff7412a390 0x7eff74094820 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 --2- 172.21.15.153:0/610180730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff5c005310 0x7eff740942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 >> 172.21.15.153:0/610180730 conn(0x7eff740678a0 msgr2=0x7eff74132900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 shutdown_connections 2024-09-15T03:40:27.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.315+0000 7eff7b02b640 1 -- 172.21.15.153:0/610180730 wait complete. 2024-09-15T03:40:27.327 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:40:27.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:40:27.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:40:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.422+0000 7ffa3062b640 10 monclient: get_monmap_and_config 2024-09-15T03:40:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.422+0000 7ffa3062b640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.422+0000 7ffa3062b640 10 monclient: monmap: 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:27.423599+0000 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:27.423599+0000 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.425 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:27.425 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:27.425 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:27.425 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:27.425 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:27.425 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.423+0000 7ffa3062b640 1 Processor -- start 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.423+0000 7ffa3062b640 1 -- start start 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.423+0000 7ffa3062b640 10 monclient: init 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffa281318f0 0x7ffa28131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): picked mon.noname-a con 0x7ffa281318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa2809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): picked mon.noname-b con 0x7ffa2809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ffa28068d90 0x7ffa280691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): picked mon.noname-c con 0x7ffa28068d90 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:27.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa28131cc0 con 0x7ffa281318f0 2024-09-15T03:40:27.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2809fbf0 con 0x7ffa28068d90 2024-09-15T03:40:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2809fe30 con 0x7ffa2809ea50 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa3062b640 10 monclient(hunting): authenticate will time out at 1425.063178s 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffa281318f0 0x7ffa28131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa2809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa2809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48340/0 (socket says 172.21.15.153:48340) 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffa281318f0 0x7ffa28131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37748/0 (socket says 172.21.15.153:37748) 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 1 -- 172.21.15.153:0/3676039183 learned_addr learned my addr 172.21.15.153:0/3676039183 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): get_auth_request con 0x7ffa2809ea50 auth_method 0 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): get_auth_request con 0x7ffa281318f0 auth_method 0 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 1 --2- 172.21.15.153:0/3676039183 >> v2:172.21.15.181:3300/0 conn(0x7ffa28068d90 0x7ffa280691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): get_auth_request con 0x7ffa28068d90 auth_method 0 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2ebc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_done global_id 154104 payload 274 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 msgr2=0x7ffa2809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa2809ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/3676039183 >> v2:172.21.15.181:3300/0 conn(0x7ffa28068d90 msgr2=0x7ffa280691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/3676039183 >> v2:172.21.15.181:3300/0 conn(0x7ffa28068d90 0x7ffa280691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/3676039183 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280a1210 con 0x7ffa281318f0 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.424+0000 7ffa2dbc6640 1 --2- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa2809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 msgr2=0x7ffa28131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 0x7ffa28131f30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffa28064050 tx=0x7ffa240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e995d05b317893 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 1 -- 172.21.15.153:0/3676039183 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2402adf0 con 0x7ffa281318f0 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: dump: 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 1 -- 172.21.15.153:0/3676039183 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24032440 con 0x7ffa281318f0 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa3062b640 5 monclient: authenticate success, global_id 154104 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa3062b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa3062b640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 1 -- 172.21.15.153:0/3676039183 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240327e0 con 0x7ffa281318f0 2024-09-15T03:40:27.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa3062b640 4 set_mon_vals no callback set 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.427+0000 7ffa2d3c5640 10 monclient: dump: 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.450 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 10 monclient: shutdown 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 -- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 msgr2=0x7ffa28131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 --2- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 0x7ffa28131f30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffa28064050 tx=0x7ffa240090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 -- 172.21.15.153:0/3676039183 shutdown_connections 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 --2- 172.21.15.153:0/3676039183 >> v2:172.21.15.181:3300/0 conn(0x7ffa28068d90 0x7ffa280691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 --2- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa2809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 --2- 172.21.15.153:0/3676039183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 0x7ffa28131f30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 -- 172.21.15.153:0/3676039183 >> 172.21.15.153:0/3676039183 conn(0x7ffa28067110 msgr2=0x7ffa28067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 -- 172.21.15.153:0/3676039183 shutdown_connections 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 1 -- 172.21.15.153:0/3676039183 wait complete. 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.428+0000 7ffa3062b640 10 monclient: monmap: 2024-09-15T03:40:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:27.453 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 Processor -- start 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 -- start start 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient: init 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:40:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa28068d90 0x7ffa2809a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): picked mon.noname-b con 0x7ffa28068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa28130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): picked mon.noname-c con 0x7ffa2809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 0x7ffa281310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): picked mon.noname-a con 0x7ffa281318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2809bf10 con 0x7ffa281318f0 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280a43e0 con 0x7ffa28068d90 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2811dcc0 con 0x7ffa2809ea50 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa3062b640 10 monclient(hunting): authenticate will time out at 1425.068570s 2024-09-15T03:40:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa28130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa28130ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48348/0 (socket says 172.21.15.153:48348) 2024-09-15T03:40:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 -- 172.21.15.153:0/2014773198 learned_addr learned my addr 172.21.15.153:0/2014773198 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): get_auth_request con 0x7ffa2809ea50 auth_method 0 2024-09-15T03:40:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 0x7ffa281310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): get_auth_request con 0x7ffa281318f0 auth_method 0 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2ebc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa28068d90 0x7ffa2809a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2e3c7640 10 monclient(hunting): get_auth_request con 0x7ffa28068d90 auth_method 0 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2e3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient(hunting): handle_auth_done global_id 124436 payload 824 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa28068d90 msgr2=0x7ffa2809a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa28068d90 0x7ffa2809a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 msgr2=0x7ffa281310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa281318f0 0x7ffa281310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 monclient: found mon.noname-c 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:27.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2dbc6640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18009450 con 0x7ffa2809ea50 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.429+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa28068d90 0x7ffa2809a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa2dbc6640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 msgr2=0x7ffa28130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa2dbc6640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa28130ba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800ec80 tx=0x7ffa180091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18009730 con 0x7ffa2809ea50 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:27.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: dump: 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1800eec0 con 0x7ffa2809ea50 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 5 monclient: authenticate success, global_id 124436 2024-09-15T03:40:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 10 monclient: _renew_subs 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa24009390 con 0x7ffa2809ea50 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18013470 con 0x7ffa2809ea50 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa1e7fc640 10 monclient: dump: 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa2cbc4640 4 set_mon_vals no callback set 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 10 monclient: _renew_subs 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.432+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2811c670 con 0x7ffa2809ea50 2024-09-15T03:40:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa3062b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa3062b640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 0x7ffa28140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28065e00 con 0x7ffa2811bc30 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 0x7ffa28140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffa18013820 con 0x7ffa2809ea50 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient: get_auth_request con 0x7ffa2811bc30 auth_method 0 2024-09-15T03:40:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_done global_id 154110 payload 824 2024-09-15T03:40:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa0c038740 0x7ffa0c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa2804bf60 con 0x7ffa2811bc30 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ffa1804ca40 con 0x7ffa2809ea50 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2ebc8640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa0c038740 0x7ffa0c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.433+0000 7ffa2ebc8640 10 monclient: get_auth_request con 0x7ffa0c038740 auth_method 0 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 msgr2=0x7ffa28140e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 0x7ffa28140e20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa24034000 tx=0x7ffa24032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2403c970 con 0x7ffa2811bc30 2024-09-15T03:40:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 10 monclient: dump: 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ffa2403cad0 con 0x7ffa2811bc30 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.435+0000 7ffa1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.436+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 msgr2=0x7ffa28140e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.436+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 0x7ffa28140e20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa24034000 tx=0x7ffa24032000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.436+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 msgr2=0x7ffa28140e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.436+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2811bc30 0x7ffa28140e20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.436+0000 7ffa2ebc8640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa0c038740 0x7ffa0c03abf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009fd0 tx=0x7ffa2000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa3062b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa3062b640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 0x7ffa28141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa3062b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28141ca0 con 0x7ffa28141360 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 0x7ffa28141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient: get_auth_request con 0x7ffa28141360 auth_method 0 2024-09-15T03:40:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 10 monclient(hunting): handle_auth_done global_id 154113 payload 824 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.441+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/2014773198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa280638d0 con 0x7ffa28141360 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa2e3c7640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 msgr2=0x7ffa28141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa2e3c7640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 0x7ffa28141760 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009130 tx=0x7ffa2403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2403bb10 con 0x7ffa28141360 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 10 monclient: dump: 2024-09-15T03:40:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.481 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7ffa2404b070 con 0x7ffa28141360 2024-09-15T03:40:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 msgr2=0x7ffa28141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 0x7ffa28141760 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009130 tx=0x7ffa2403c000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 msgr2=0x7ffa28141760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa28141360 0x7ffa28141760 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009130 tx=0x7ffa2403c000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.483 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:40:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 124, 2024-09-15T03:40:27.483 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.499 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:27.500 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.501 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:27.503 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa0c038740 msgr2=0x7ffa0c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa0c038740 0x7ffa0c03abf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009fd0 tx=0x7ffa2000b040 comp rx=0 tx=0).stop 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 10 monclient: shutdown 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 msgr2=0x7ffa28130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 --2- 172.21.15.153:0/2014773198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2809ea50 0x7ffa28130ba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800ec80 tx=0x7ffa180091d0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa2ebc8640 1 -- 172.21.15.153:0/2014773198 reap_dead start 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa2ebc8640 1 -- 172.21.15.153:0/2014773198 reap_dead start 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 shutdown_connections 2024-09-15T03:40:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 >> 172.21.15.153:0/2014773198 conn(0x7ffa28067110 msgr2=0x7ffa280a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 shutdown_connections 2024-09-15T03:40:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.444+0000 7ffa3062b640 1 -- 172.21.15.153:0/2014773198 wait complete. 2024-09-15T03:40:27.507 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:40:27.507 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:40:27.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.601+0000 7f45dd309640 10 monclient: get_monmap_and_config 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.601+0000 7f45dd309640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.601+0000 7f45dd309640 10 monclient: monmap: 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:27.602012+0000 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:27.602012+0000 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.602 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.603 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:27.603 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:27.603 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:27.603 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:27.603 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:27.603 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.601+0000 7f45dd309640 1 Processor -- start 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.601+0000 7f45dd309640 1 -- start start 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.601+0000 7f45dd309640 10 monclient: init 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45d8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient(hunting): picked mon.noname-b con 0x7f45d81318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45d809ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45d8136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d8131cc0 con 0x7f45d8136c20 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d809f530 con 0x7f45d809ea50 2024-09-15T03:40:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d809fbf0 con 0x7f45d81318f0 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45dd309640 10 monclient(hunting): authenticate will time out at 1425.241417s 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f45d8136c20 0x7f45d8139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46736/0 (socket says 172.21.15.153:46736) 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45d8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d77fe640 1 -- 172.21.15.153:0/289147475 learned_addr learned my addr 172.21.15.153:0/289147475 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request con 0x7f45d81318f0 auth_method 0 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 msgr2=0x7f45d8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 msgr2=0x7f45d8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45d8131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.602+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45d8131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45d77fe640 1 -- 172.21.15.153:0/289147475 >> v1:172.21.15.153:6789/0 conn(0x7f45d8136c20 legacy=0x7f45d8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45ce7fc640 1 -- 172.21.15.153:0/289147475 >> v1:172.21.15.181:6789/0 conn(0x7f45d809ea50 legacy=0x7f45d809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310848004 0 0) 0x7f45d8131cc0 con 0x7f45d8136c20 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45d67fc640 10 monclient(hunting): my global_id is 154116 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.604+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c40034e0 con 0x7f45d8136c20 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1240102789 0 0) 0x7f45c40034e0 con 0x7f45d8136c20 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 msgr2=0x7f45d8131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45d8131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 >> v1:172.21.15.181:6789/0 conn(0x7f45d809ea50 legacy=0x7f45d809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d809fe30 con 0x7f45d8136c20 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f45d0001aa0 con 0x7f45d8136c20 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45dd309640 5 monclient: authenticate success, global_id 154116 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45dd309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: dump: 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 >> v1:172.21.15.153:6789/0 conn(0x7f45d8136c20 legacy=0x7f45d8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4003b00 0x7f45c4003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): picked mon.b con 0x7f45c4003b00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c4004780 0x7f45c4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): picked mon.a con 0x7f45c4004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45c400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): picked mon.c con 0x7f45d81318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d809fbf0 con 0x7f45c4004780 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c40055f0 con 0x7f45c4003b00 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c400bf40 con 0x7f45d81318f0 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d67fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45dd309640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c4004780 0x7f45c4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45c400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): get_auth_request con 0x7f45c4004780 auth_method 0 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4003b00 0x7f45c4003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): get_auth_request con 0x7f45d81318f0 auth_method 0 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request con 0x7f45c4003b00 auth_method 0 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_done global_id 154116 payload 293 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 msgr2=0x7f45c400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d81318f0 0x7f45c400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4003b00 msgr2=0x7f45c4003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4003b00 0x7f45c4003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 monclient: found mon.a 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 1 -- 172.21.15.153:0/289147475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c40034e0 con 0x7f45c4004780 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.605+0000 7f45ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45ce7fc640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c4004780 msgr2=0x7f45c4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45ce7fc640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c4004780 0x7f45c4004bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f45c0001b00 tx=0x7f45c0007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2719e84f91d30a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c002ac60 con 0x7f45c4004780 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: dump: 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c002adc0 con 0x7f45c4004780 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 4 set_mon_vals no callback set 2024-09-15T03:40:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 1 -- 172.21.15.153:0/289147475 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0007340 con 0x7f45c4004780 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45d67fc640 10 monclient: dump: 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 10 monclient: shutdown 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 1 -- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c4004780 msgr2=0x7f45c4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.608+0000 7f45dd309640 1 --2- 172.21.15.153:0/289147475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c4004780 0x7f45c4004bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f45c0001b00 tx=0x7f45c0007c40 comp rx=0 tx=0).stop 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45d77fe640 1 -- 172.21.15.153:0/289147475 reap_dead start 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45dd309640 1 -- 172.21.15.153:0/289147475 shutdown_connections 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45dd309640 1 -- 172.21.15.153:0/289147475 >> 172.21.15.153:0/289147475 conn(0x7f45d8067110 msgr2=0x7f45d8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45dd309640 1 -- 172.21.15.153:0/289147475 shutdown_connections 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45dd309640 1 -- 172.21.15.153:0/289147475 wait complete. 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45dd309640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.609+0000 7f45dd309640 10 monclient: monmap: 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 Processor -- start 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 -- start start 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient: init 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4004780 0x7f45d8130890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): picked mon.noname-b con 0x7f45c4004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d8130dd0 0x7f45d8099680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): picked mon.noname-c con 0x7f45d8130dd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 0x7f45d809a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): picked mon.noname-a con 0x7f45d8099bc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d811cf60 con 0x7f45d8099bc0 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d8141e80 con 0x7f45c4004780 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d809b2e0 con 0x7f45d8130dd0 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45dd309640 10 monclient(hunting): authenticate will time out at 1425.249530s 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 0x7f45d809a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d8130dd0 0x7f45d8099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 0x7f45d809a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37800/0 (socket says 172.21.15.153:37800) 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 learned_addr learned my addr 172.21.15.153:0/3168877085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d8130dd0 0x7f45d8099680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48364/0 (socket says 172.21.15.153:48364) 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): get_auth_request con 0x7f45d8099bc0 auth_method 0 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): get_auth_request con 0x7f45d8130dd0 auth_method 0 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4004780 0x7f45d8130890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request con 0x7f45c4004780 auth_method 0 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient(hunting): handle_auth_done global_id 154119 payload 824 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d8130dd0 msgr2=0x7f45d8099680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d8130dd0 0x7f45d8099680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4004780 msgr2=0x7f45d8130890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45c4004780 0x7f45d8130890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.610+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d811c9a0 con 0x7f45d8099bc0 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 msgr2=0x7f45d809a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45d77fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 0x7f45d809a030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f45d0001c80 tx=0x7f45d000a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0004e00 con 0x7f45d8099bc0 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: dump: 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0007350 con 0x7f45d8099bc0 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 5 monclient: authenticate success, global_id 154119 2024-09-15T03:40:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 10 monclient: _renew_subs 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d811cc80 con 0x7f45d8099bc0 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d00076f0 con 0x7f45d8099bc0 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45cf7fe640 10 monclient: dump: 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 10 monclient: _renew_subs 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45d5ffb640 4 set_mon_vals no callback set 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.613+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d811b720 con 0x7f45d8099bc0 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45dd309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45dd309640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 0x7f45d809a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45dd309640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d811cb30 con 0x7f45d802aa40 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f45d001d460 con 0x7f45d8099bc0 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 0x7f45d809a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient: get_auth_request con 0x7f45d802aa40 auth_method 0 2024-09-15T03:40:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45cf7fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45b8038740 0x7f45b803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_done global_id 124442 payload 824 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45d6ffd640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45d802ae10 con 0x7f45d802aa40 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f45d004c4f0 con 0x7f45d8099bc0 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.614+0000 7f45ce7fc640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45b8038740 0x7f45b803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.615+0000 7f45ce7fc640 10 monclient: get_auth_request con 0x7f45b8038740 auth_method 0 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.616+0000 7f45d6ffd640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 msgr2=0x7f45d809a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.616+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 0x7f45d809a570 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f45c8006fd0 tx=0x7f45c8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.616+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8010e80 con 0x7f45d802aa40 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.616+0000 7f45cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.616+0000 7f45cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.616+0000 7f45cf7fe640 10 monclient: dump: 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.640 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f45c8015450 con 0x7f45d802aa40 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 msgr2=0x7f45d809a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 0x7f45d809a570 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f45c8006fd0 tx=0x7f45c8008040 comp rx=0 tx=0).stop 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 msgr2=0x7f45d809a570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45cf7fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d802aa40 0x7f45d809a570 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f45c8006fd0 tx=0x7f45c8008040 comp rx=0 tx=0).stop 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.617+0000 7f45ce7fc640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45b8038740 0x7f45b803abf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f45c002f000 tx=0x7f45c0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.622+0000 7f45dd309640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.622+0000 7f45dd309640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 0x7f45d805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.622+0000 7f45dd309640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.622+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d811ce10 con 0x7f45d805e310 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 0x7f45d805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient: get_auth_request con 0x7f45d805e310 auth_method 0 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 10 monclient(hunting): handle_auth_done global_id 124445 payload 824 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.623+0000 7f45d6ffd640 1 -- 172.21.15.153:0/3168877085 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45d809aab0 con 0x7f45d805e310 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45d6ffd640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 msgr2=0x7f45d805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45d6ffd640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 0x7f45d805e710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f45c801e000 tx=0x7f45c8021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8020db0 con 0x7f45d805e310 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 10 monclient: dump: 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f45c802d450 con 0x7f45d805e310 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 msgr2=0x7f45d805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 0x7f45d805e710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f45c801e000 tx=0x7f45c8021000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 msgr2=0x7f45d805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.625+0000 7f45cf7fe640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d805e310 0x7f45d805e710 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f45c801e000 tx=0x7f45c8021000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 124, 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:27.646 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.647 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.648 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:27.649 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.650 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:27.651 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.652 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.653 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.655 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:27.659 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45b8038740 msgr2=0x7f45b803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f45b8038740 0x7f45b803abf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f45c002f000 tx=0x7f45c0030000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 10 monclient: shutdown 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 msgr2=0x7f45d809a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 --2- 172.21.15.153:0/3168877085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d8099bc0 0x7f45d809a030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f45d0001c80 tx=0x7f45d000a200 comp rx=0 tx=0).stop 2024-09-15T03:40:27.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 reap_dead start 2024-09-15T03:40:27.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45d77fe640 1 -- 172.21.15.153:0/3168877085 reap_dead start 2024-09-15T03:40:27.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 shutdown_connections 2024-09-15T03:40:27.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 >> 172.21.15.153:0/3168877085 conn(0x7f45d8067110 msgr2=0x7f45d809d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 shutdown_connections 2024-09-15T03:40:27.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.626+0000 7f45dd309640 1 -- 172.21.15.153:0/3168877085 wait complete. 2024-09-15T03:40:27.661 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:40:27.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:40:27.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:40:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.754+0000 7fa757817640 10 monclient: get_monmap_and_config 2024-09-15T03:40:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.754+0000 7fa757817640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.754+0000 7fa757817640 10 monclient: monmap: 2024-09-15T03:40:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:27.755373+0000 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:27.755373+0000 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:27.756 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 Processor -- start 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 -- start start 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient: init 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa7500981c0 0x7fa750098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): picked mon.noname-a con 0x7fa7500981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa750131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa750132170 0x7fa750067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): picked mon.noname-c con 0x7fa750132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa750098ad0 con 0x7fa750131da0 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa750131bd0 con 0x7fa7500981c0 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75009ca60 con 0x7fa750132170 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa757817640 10 monclient(hunting): authenticate will time out at 1425.394856s 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa7555b3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa7500981c0 0x7fa750098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.755+0000 7fa7555b3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa7500981c0 0x7fa750098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37806/0 (socket says 172.21.15.153:37806) 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 1 -- 172.21.15.153:0/4189107055 learned_addr learned my addr 172.21.15.153:0/4189107055 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): get_auth_request con 0x7fa7500981c0 auth_method 0 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa754db2640 1 --1- 172.21.15.153:0/4189107055 >> v1:172.21.15.153:6789/0 conn(0x7fa750131da0 0x7fa750067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46742/0 (socket says 172.21.15.153:46742) 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa755db4640 1 --2- 172.21.15.153:0/4189107055 >> v2:172.21.15.181:3300/0 conn(0x7fa750132170 0x7fa750067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa755db4640 10 monclient(hunting): get_auth_request con 0x7fa750132170 auth_method 0 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa755db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa755db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa755db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient(hunting): handle_auth_done global_id 154122 payload 274 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 1 -- 172.21.15.153:0/4189107055 >> v2:172.21.15.181:3300/0 conn(0x7fa750132170 msgr2=0x7fa750067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 1 --2- 172.21.15.153:0/4189107055 >> v2:172.21.15.181:3300/0 conn(0x7fa750132170 0x7fa750067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 1 -- 172.21.15.153:0/4189107055 >> v1:172.21.15.153:6789/0 conn(0x7fa750131da0 legacy=0x7fa750067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.756+0000 7fa7555b3640 1 -- 172.21.15.153:0/4189107055 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75009cca0 con 0x7fa7500981c0 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa7555b3640 1 -- 172.21.15.153:0/4189107055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 msgr2=0x7fa750098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa7555b3640 1 --2- 172.21.15.153:0/4189107055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750098590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa75006f820 tx=0x7fa74c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9716e9c7664310b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 1 -- 172.21.15.153:0/4189107055 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c032440 con 0x7fa7500981c0 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: dump: 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 1 -- 172.21.15.153:0/4189107055 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c0325a0 con 0x7fa7500981c0 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 5 monclient: authenticate success, global_id 154122 2024-09-15T03:40:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 4 set_mon_vals no callback set 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 1 -- 172.21.15.153:0/4189107055 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c032940 con 0x7fa7500981c0 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa747fff640 10 monclient: dump: 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 10 monclient: shutdown 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 -- 172.21.15.153:0/4189107055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 msgr2=0x7fa750098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 --2- 172.21.15.153:0/4189107055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750098590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa75006f820 tx=0x7fa74c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 -- 172.21.15.153:0/4189107055 shutdown_connections 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 --2- 172.21.15.153:0/4189107055 >> v2:172.21.15.181:3300/0 conn(0x7fa750132170 0x7fa750067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 --2- 172.21.15.153:0/4189107055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750098590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 -- 172.21.15.153:0/4189107055 >> 172.21.15.153:0/4189107055 conn(0x7fa7500a1e30 msgr2=0x7fa7500a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 -- 172.21.15.153:0/4189107055 shutdown_connections 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.759+0000 7fa757817640 1 -- 172.21.15.153:0/4189107055 wait complete. 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.760+0000 7fa757817640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.760+0000 7fa757817640 10 monclient: monmap: 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.760+0000 7fa757817640 1 Processor -- start 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.760+0000 7fa757817640 1 -- start start 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.760+0000 7fa757817640 10 monclient: init 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): picked mon.noname-a con 0x7fa7500981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa750131da0 0x7fa750130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): picked mon.noname-c con 0x7fa750131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750132170 0x7fa750131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): picked mon.noname-b con 0x7fa750132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75011e330 con 0x7fa7500981c0 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa750142290 con 0x7fa750132170 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7500a0520 con 0x7fa750131da0 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa757817640 10 monclient(hunting): authenticate will time out at 1425.400411s 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750141360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37808/0 (socket says 172.21.15.153:37808) 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 -- 172.21.15.153:0/3679765907 learned_addr learned my addr 172.21.15.153:0/3679765907 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): get_auth_request con 0x7fa7500981c0 auth_method 0 2024-09-15T03:40:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa750131da0 0x7fa750130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): get_auth_request con 0x7fa750131da0 auth_method 0 2024-09-15T03:40:27.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750132170 0x7fa750131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): get_auth_request con 0x7fa750132170 auth_method 0 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa755db4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient(hunting): handle_auth_done global_id 154125 payload 824 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa750131da0 msgr2=0x7fa750130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa750131da0 0x7fa750130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750132170 msgr2=0x7fa750131090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750132170 0x7fa750131090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.761+0000 7fa7555b3640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c009450 con 0x7fa7500981c0 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7555b3640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 msgr2=0x7fa750141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7555b3640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750141360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa74c035040 tx=0x7fa74c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c031a40 con 0x7fa7500981c0 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: dump: 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 5 monclient: authenticate success, global_id 154125 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c031be0 con 0x7fa7500981c0 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 10 monclient: _renew_subs 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75011dc70 con 0x7fa7500981c0 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c03d420 con 0x7fa7500981c0 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7457fa640 10 monclient: dump: 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.781 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 10 monclient: _renew_subs 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75011cd80 con 0x7fa7500981c0 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.764+0000 7fa7477fe640 4 set_mon_vals no callback set 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa757817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa757817640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 0x7fa75011a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa750065f90 con 0x7fa750141900 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa74c039040 con 0x7fa7500981c0 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 0x7fa75011a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient: get_auth_request con 0x7fa750141900 auth_method 0 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa7457fa640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa734038770 0x7fa73403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa74c06c520 con 0x7fa7500981c0 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa755db4640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa734038770 0x7fa73403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 10 monclient(hunting): handle_auth_done global_id 144190 payload 824 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa754db2640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa75004bf60 con 0x7fa750141900 2024-09-15T03:40:27.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.765+0000 7fa755db4640 10 monclient: get_auth_request con 0x7fa734038770 auth_method 0 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa754db2640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 msgr2=0x7fa75011a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa754db2640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 0x7fa75011a5d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa74000e710 tx=0x7fa740007e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740005050 con 0x7fa750141900 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 10 monclient: dump: 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.787 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa755db4640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa734038770 0x7fa73403ac20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa75011bbd0 tx=0x7fa748009850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa7400051b0 con 0x7fa750141900 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 msgr2=0x7fa75011a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 0x7fa75011a5d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa74000e710 tx=0x7fa740007e40 comp rx=0 tx=0).stop 2024-09-15T03:40:27.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 msgr2=0x7fa75011a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.768+0000 7fa7457fa640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa750141900 0x7fa75011a5d0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa74000e710 tx=0x7fa740007e40 comp rx=0 tx=0).stop 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa757817640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa757817640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 0x7fa75011af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa757817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75011b450 con 0x7fa75011ab10 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 0x7fa75011af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient: get_auth_request con 0x7fa75011ab10 auth_method 0 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 10 monclient(hunting): handle_auth_done global_id 144193 payload 824 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.774+0000 7fa754db2640 1 -- 172.21.15.153:0/3679765907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa750098ee0 con 0x7fa75011ab10 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.777+0000 7fa754db2640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 msgr2=0x7fa75011af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.777+0000 7fa754db2640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 0x7fa75011af10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa74000e710 tx=0x7fa740019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.777+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74001d610 con 0x7fa75011ab10 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.777+0000 7fa7457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.777+0000 7fa7457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.777+0000 7fa7457fa640 10 monclient: dump: 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.790 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fa740025070 con 0x7fa75011ab10 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 msgr2=0x7fa75011af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 0x7fa75011af10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa74000e710 tx=0x7fa740019000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 msgr2=0x7fa75011af10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa7457fa640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa75011ab10 0x7fa75011af10 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa74000e710 tx=0x7fa740019000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 124, 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:40:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:27.795 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.798 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.799 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.800 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.801 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.802 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:27.803 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.804 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:27.806 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:27.807 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa734038770 msgr2=0x7fa73403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa734038770 0x7fa73403ac20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa75011bbd0 tx=0x7fa748009850 comp rx=0 tx=0).stop 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 10 monclient: shutdown 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 msgr2=0x7fa750141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 --2- 172.21.15.153:0/3679765907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7500981c0 0x7fa750141360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa74c035040 tx=0x7fa74c032000 comp rx=0 tx=0).stop 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa755db4640 1 -- 172.21.15.153:0/3679765907 reap_dead start 2024-09-15T03:40:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa755db4640 1 -- 172.21.15.153:0/3679765907 reap_dead start 2024-09-15T03:40:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 shutdown_connections 2024-09-15T03:40:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 >> 172.21.15.153:0/3679765907 conn(0x7fa7500a1e30 msgr2=0x7fa750096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 shutdown_connections 2024-09-15T03:40:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.778+0000 7fa757817640 1 -- 172.21.15.153:0/3679765907 wait complete. 2024-09-15T03:40:27.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:40:27.809 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:40:27.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:40:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.902+0000 7f4b48e17640 10 monclient: get_monmap_and_config 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.902+0000 7f4b48e17640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.903+0000 7f4b48e17640 10 monclient: monmap: 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:27.903735+0000 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:27.903735+0000 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:27.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.903+0000 7f4b48e17640 1 Processor -- start 2024-09-15T03:40:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.903+0000 7f4b48e17640 1 -- start start 2024-09-15T03:40:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.903+0000 7f4b48e17640 10 monclient: init 2024-09-15T03:40:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:40:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4b44096e10 0x7f4b4409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): picked mon.noname-a con 0x7f4b44096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b4409aaa0 0x7f4b440971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): picked mon.noname-b con 0x7f4b4409aaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4b44097720 0x7f4b4412e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): picked mon.noname-c con 0x7f4b44097720 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b44099620 con 0x7f4b44096e10 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b44099340 con 0x7f4b44097720 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b44099900 con 0x7f4b4409aaa0 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b48e17640 10 monclient(hunting): authenticate will time out at 1425.543313s 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4b44096e10 0x7f4b4409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b4409aaa0 0x7f4b440971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4b44096e10 0x7f4b4409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37820/0 (socket says 172.21.15.153:37820) 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 -- 172.21.15.153:0/1980709047 learned_addr learned my addr 172.21.15.153:0/1980709047 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): get_auth_request con 0x7f4b44096e10 auth_method 0 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): get_auth_request con 0x7f4b4409aaa0 auth_method 0 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b42d9d640 1 --2- 172.21.15.153:0/1980709047 >> v2:172.21.15.181:3300/0 conn(0x7f4b44097720 0x7f4b4412e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b42d9d640 10 monclient(hunting): get_auth_request con 0x7f4b44097720 auth_method 0 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient(hunting): handle_auth_done global_id 154128 payload 274 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 -- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b4409aaa0 msgr2=0x7f4b440971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 --2- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b4409aaa0 0x7f4b440971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 -- 172.21.15.153:0/1980709047 >> v2:172.21.15.181:3300/0 conn(0x7f4b44097720 msgr2=0x7f4b4412e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 --2- 172.21.15.153:0/1980709047 >> v2:172.21.15.181:3300/0 conn(0x7f4b44097720 0x7f4b4412e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b4259c640 1 -- 172.21.15.153:0/1980709047 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b44097b10 con 0x7f4b44096e10 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.904+0000 7f4b41d9b640 1 --2- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b4409aaa0 0x7f4b440971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4259c640 1 -- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44096e10 msgr2=0x7f4b4409a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4259c640 1 --2- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44096e10 0x7f4b4409a560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4b4409aff0 tx=0x7f4b340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca2694ae2d7437c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 1 -- 172.21.15.153:0/1980709047 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34032440 con 0x7f4b44096e10 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: dump: 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b48e17640 5 monclient: authenticate success, global_id 154128 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b48e17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 1 -- 172.21.15.153:0/1980709047 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b340325a0 con 0x7f4b44096e10 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b48e17640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b48e17640 4 set_mon_vals no callback set 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 1 -- 172.21.15.153:0/1980709047 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34032940 con 0x7f4b44096e10 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.907+0000 7f4b4159a640 10 monclient: dump: 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.918 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 10 monclient: shutdown 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 -- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44096e10 msgr2=0x7f4b4409a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 --2- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44096e10 0x7f4b4409a560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4b4409aff0 tx=0x7f4b340091d0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 -- 172.21.15.153:0/1980709047 shutdown_connections 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 --2- 172.21.15.153:0/1980709047 >> v2:172.21.15.181:3300/0 conn(0x7f4b44097720 0x7f4b4412e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 --2- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b4409aaa0 0x7f4b440971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 --2- 172.21.15.153:0/1980709047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44096e10 0x7f4b4409a560 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 -- 172.21.15.153:0/1980709047 >> 172.21.15.153:0/1980709047 conn(0x7f4b440672b0 msgr2=0x7f4b440676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 -- 172.21.15.153:0/1980709047 shutdown_connections 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 1 -- 172.21.15.153:0/1980709047 wait complete. 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 10 monclient: build_initial_monmap 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.908+0000 7f4b48e17640 10 monclient: monmap: 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:27.920 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 Processor -- start 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 -- start start 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient: init 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 0x7f4b4412d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): picked mon.noname-c con 0x7f4b44096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 0x7f4b44118130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): picked mon.noname-a con 0x7f4b44097720 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b4409aaa0 0x7f4b44118670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): picked mon.noname-b con 0x7f4b4409aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4411be10 con 0x7f4b44097720 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4409b050 con 0x7f4b4409aaa0 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4413d8a0 con 0x7f4b44096e10 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b48e17640 10 monclient(hunting): authenticate will time out at 1425.548765s 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 0x7f4b44118130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 0x7f4b4412d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 0x7f4b44118130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37828/0 (socket says 172.21.15.153:37828) 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 0x7f4b4412d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48430/0 (socket says 172.21.15.153:48430) 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 1 -- 172.21.15.153:0/3881666278 learned_addr learned my addr 172.21.15.153:0/3881666278 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:27.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): get_auth_request con 0x7f4b44097720 auth_method 0 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): get_auth_request con 0x7f4b44096e10 auth_method 0 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.909+0000 7f4b4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:27.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b42d9d640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b4409aaa0 0x7f4b44118670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b42d9d640 10 monclient(hunting): get_auth_request con 0x7f4b4409aaa0 auth_method 0 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 10 monclient(hunting): handle_auth_done global_id 154131 payload 824 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 1 -- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 msgr2=0x7f4b4412d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 0x7f4b4412d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 1 -- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b4409aaa0 msgr2=0x7f4b44118670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b4409aaa0 0x7f4b44118670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 1 monclient: found mon.noname-a 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b41d9b640 1 -- 172.21.15.153:0/3881666278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b38009390 con 0x7f4b44097720 2024-09-15T03:40:27.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.910+0000 7f4b4259c640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 0x7f4b4412d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b41d9b640 1 -- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 msgr2=0x7f4b44118130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b41d9b640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 0x7f4b44118130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800ec80 tx=0x7f4b380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3800eec0 con 0x7f4b44097720 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: dump: 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: _finish_auth 0 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3800cdd0 con 0x7f4b44097720 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 5 monclient: authenticate success, global_id 154131 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38013680 con 0x7f4b44097720 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 10 monclient: _renew_subs 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b34009450 con 0x7f4b44097720 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b327fc640 10 monclient: dump: 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b40d99640 4 set_mon_vals no callback set 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 10 monclient: _renew_subs 2024-09-15T03:40:27.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.912+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4411a960 con 0x7f4b44097720 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4b380139f0 con 0x7f4b44097720 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b327fc640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4b24038790 0x7f4b2403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f4b3804c9a0 con 0x7f4b44097720 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b48e17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b48e17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b48e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4404bf60 con 0x7f4b44097720 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b4259c640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4b24038790 0x7f4b2403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.913+0000 7f4b4259c640 10 monclient: get_auth_request con 0x7f4b24038790 auth_method 0 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.916+0000 7f4b4259c640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4b24038790 0x7f4b2403ac40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b44076be0 tx=0x7f4b34032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.918+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4b380236b0 con 0x7f4b44097720 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.918+0000 7f4b327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:27.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:27.918+0000 7f4b327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:28.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.134+0000 7f4b48e17640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:40:28.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.134+0000 7f4b48e17640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:40:28.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.134+0000 7f4b48e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:28.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.134+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f4b440638d0 con 0x7f4b44097720 2024-09-15T03:40:28.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.135+0000 7f4b327fc640 1 -- 172.21.15.153:0/3881666278 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f4b380232f0 con 0x7f4b44097720 2024-09-15T03:40:28.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.135+0000 7f4b327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:40:28.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.135+0000 7f4b327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.135+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.135+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=8,logm=55} crc {auth=1510673366,config=3140514009,health=1070742485,kv=3235807718,logm=3415319119}) 2024-09-15T03:40:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.135+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=1988775402,config=3140514009,health=1952436384,kv=3235807718,logm=3768133274}) 2024-09-15T03:40:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.135+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.135+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=8,logm=55} crc {auth=1510673366,config=3140514009,health=1070742485,kv=3235807718,logm=3415319119}) 2024-09-15T03:40:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.135+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=7,logm=55} crc {auth=2336855919,config=3140514009,health=2892279626,kv=2145712985,logm=110845805}) 2024-09-15T03:40:28.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.136+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.136+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153146155}) 2024-09-15T03:40:28.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.136+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2161994195}) 2024-09-15T03:40:28.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.137+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.137+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153146155}) 2024-09-15T03:40:28.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.137+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1736033251}) 2024-09-15T03:40:28.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4b24038790 msgr2=0x7f4b2403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:28.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4b24038790 0x7f4b2403ac40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b44076be0 tx=0x7f4b34032000 comp rx=0 tx=0).stop 2024-09-15T03:40:28.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 10 monclient: shutdown 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 msgr2=0x7f4b44118130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 0x7f4b44118130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800ec80 tx=0x7f4b380090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 shutdown_connections 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4b24038790 0x7f4b2403ac40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b4409aaa0 0x7f4b44118670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b44097720 0x7f4b44118130 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 --2- 172.21.15.153:0/3881666278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b44096e10 0x7f4b4412d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.137+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 >> 172.21.15.153:0/3881666278 conn(0x7f4b440672b0 msgr2=0x7f4b4409da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.138+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 shutdown_connections 2024-09-15T03:40:28.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:28.138+0000 7f4b48e17640 1 -- 172.21.15.153:0/3881666278 wait complete. 2024-09-15T03:40:28.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.138+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.138+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134123294}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.138+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2611667294}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.138+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.138+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134123294}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.138+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309956754}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.139+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.139+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401384306}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.139+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3246905712}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.139+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.139+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401384306}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.139+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=459739911}) 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.140+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.140+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77984757}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.140+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2427396103}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.140+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.140+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77984757}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.140+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2802596301}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.141+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.141+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187965466}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.141+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026809952}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.141+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.141+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187965466}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.141+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845335501}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.142+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.142+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350441471}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.142+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3842632048}) 2024-09-15T03:40:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.142+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.142+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350441471}) 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.142+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173705041}) 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.143+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.143+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890264286}) 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.143+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153478017}) 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.143+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.143+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890264286}) 2024-09-15T03:40:28.146 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.143+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890906585}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188220580}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394059680}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188220580}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301300662}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581129943}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4098910764}) 2024-09-15T03:40:28.150 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581129943}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.144+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286047523}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.145+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.145+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159733901}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.145+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646663737}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.145+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.145+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159733901}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.145+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3389731979}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.146+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.146+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193684051}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.146+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616777656}) 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.146+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.151 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.146+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193684051}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.146+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=482638915}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.147+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.147+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426979502}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.147+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1900426253}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.147+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.147+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426979502}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.147+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=335467514}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.148+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.148+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161499742}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.148+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2903167473}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.148+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.148+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161499742}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.148+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235584846}) 2024-09-15T03:40:28.152 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.149+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.149+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102326230}) 2024-09-15T03:40:28.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.149+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747945626}) 2024-09-15T03:40:28.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.149+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.149+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102326230}) 2024-09-15T03:40:28.153 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.149+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50368664}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.150+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.150+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873470453}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.150+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1284203001}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.150+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.150+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873470453}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.150+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412462890}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529872407}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=414229235}) 2024-09-15T03:40:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529872407}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392660232}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688431196}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234452102}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688431196}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.151+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=330480404}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.152+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.152+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480901340}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.152+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485062648}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.152+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.152+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480901340}) 2024-09-15T03:40:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.152+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883132313}) 2024-09-15T03:40:28.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.153+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.153+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798785519}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.153+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1541653108}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.153+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.153+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798785519}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.153+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2113158892}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946656789}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3576936914}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946656789}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2630974334}) 2024-09-15T03:40:28.165 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330409825}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2349072952}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330409825}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.154+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021095043}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.155+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.155+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631781509}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.155+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=421817206}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.155+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.155+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631781509}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.155+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226142477}) 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.156+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.166 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.156+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764842767}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.156+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354326919}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.156+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.156+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764842767}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.156+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168117615}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589670181}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450319840}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589670181}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576065687}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152877415}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1696334978}) 2024-09-15T03:40:28.167 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152877415}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.157+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618808349}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.158+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.158+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663681215}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.158+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2667987237}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.158+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.158+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663681215}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.158+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2995982970}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764481514}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970505636}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764481514}) 2024-09-15T03:40:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723919117}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3073441001}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1433005422}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3073441001}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.159+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2211656737}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394283777}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760460378}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394283777}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1891330675}) 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3580145120}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219809590}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3580145120}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.160+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866193180}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.161+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.161+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767794469}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.161+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454358434}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.161+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.161+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767794469}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.161+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2368601939}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519276541}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497992821}) 2024-09-15T03:40:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519276541}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439972939}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2554068213}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=755770034}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2554068213}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.162+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670163251}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.163+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.163+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673606265}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.163+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=46287964}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.163+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.163+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673606265}) 2024-09-15T03:40:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.163+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963145195}) 2024-09-15T03:40:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873380946}) 2024-09-15T03:40:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1724169276}) 2024-09-15T03:40:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873380946}) 2024-09-15T03:40:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4005669533}) 2024-09-15T03:40:28.181 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.164+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.181 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458176239}) 2024-09-15T03:40:28.181 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363744387}) 2024-09-15T03:40:28.181 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.181 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458176239}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224407254}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952554576}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4092421006}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952554576}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.165+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864538896}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361446253}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807889592}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361446253}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=163137910}) 2024-09-15T03:40:28.182 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.166+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298919129}) 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158472231}) 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298919129}) 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1350762497}) 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760726916}) 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=391842202}) 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.183 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760726916}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.167+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187162788}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4073716080}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526296049}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4073716080}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045482110}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381324204}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845198693}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381324204}) 2024-09-15T03:40:28.184 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.168+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2811355028}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.169+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.169+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333943880}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.169+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754736598}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.169+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.169+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333943880}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.169+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575778416}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.170+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.170+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453813591}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.170+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3594668814}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.170+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.170+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453813591}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.170+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216733024}) 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950913513}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753907657}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950913513}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122994485}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191482230}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169126852}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191482230}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.171+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030155035}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439677147}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3564242645}) 2024-09-15T03:40:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439677147}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3826118746}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525615545}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657700249}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525615545}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.172+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666517923}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368181120}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204127690}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368181120}) 2024-09-15T03:40:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640111731}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.173+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3842495361}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=770910441}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3842495361}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496656214}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988505683}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088326194}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988505683}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.174+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019694317}) 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134831260}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143731187}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134831260}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619762608}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471894619}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3368053855}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471894619}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.175+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575640508}) 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.176+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.176+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928070398}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.176+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088878842}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.176+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.176+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928070398}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.176+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869176483}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.177+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.177+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=3930251486,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3050086756,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3197044406}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.177+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=12,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=986222419,mdsmap=1531151339,mgr=1088511988,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=703716562,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1709651908}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.177+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.177+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=3930251486,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3050086756,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3197044406}) 2024-09-15T03:40:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.177+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=986222419,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1144707552,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=4024138144}) 2024-09-15T03:40:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.183+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.183+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=2075920919}) 2024-09-15T03:40:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.183+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=2446224317}) 2024-09-15T03:40:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.183+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:40:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.183+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=2075920919}) 2024-09-15T03:40:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:40:28.183+0000 7ff2607f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=539734824}) 2024-09-15T03:40:29.159 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:40:29.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:40:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.256+0000 7f4bf4680640 10 monclient: get_monmap_and_config 2024-09-15T03:40:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.256+0000 7f4bf4680640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.256+0000 7f4bf4680640 10 monclient: monmap: 2024-09-15T03:40:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:29.257334+0000 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:29.257334+0000 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:29.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 Processor -- start 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 -- start start 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient: init 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4bec131980 0x7f4bec131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): picked mon.noname-a con 0x7f4bec131980 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09fe10 0x7f4bec09ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): picked mon.noname-b con 0x7f4bec09fe10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4bec09ef70 0x7f4bec068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): picked mon.noname-c con 0x7f4bec09ef70 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec1325b0 con 0x7f4bec131980 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec0a11f0 con 0x7f4bec09ef70 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec132290 con 0x7f4bec09fe10 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.257+0000 7f4bf4680640 10 monclient(hunting): authenticate will time out at 1426.896942s 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4bec131980 0x7f4bec131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09fe10 0x7f4bec09ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4bec131980 0x7f4bec131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37830/0 (socket says 172.21.15.153:37830) 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 -- 172.21.15.153:0/338139122 learned_addr learned my addr 172.21.15.153:0/338139122 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): get_auth_request con 0x7f4bec131980 auth_method 0 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): get_auth_request con 0x7f4bec09fe10 auth_method 0 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf2c1d640 1 --2- 172.21.15.153:0/338139122 >> v2:172.21.15.181:3300/0 conn(0x7f4bec09ef70 0x7f4bec068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient(hunting): handle_auth_done global_id 154134 payload 274 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 -- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09fe10 msgr2=0x7f4bec09ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 --2- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09fe10 0x7f4bec09ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 -- 172.21.15.153:0/338139122 >> v2:172.21.15.181:3300/0 conn(0x7f4bec09ef70 msgr2=0x7f4bec068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 --2- 172.21.15.153:0/338139122 >> v2:172.21.15.181:3300/0 conn(0x7f4bec09ef70 0x7f4bec068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 monclient: found mon.noname-a 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf241c640 1 -- 172.21.15.153:0/338139122 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec09f390 con 0x7f4bec131980 2024-09-15T03:40:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf2c1d640 10 monclient: get_auth_request con 0x7f4bec09ef70 auth_method 0 2024-09-15T03:40:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf1c1b640 1 --2- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09fe10 0x7f4bec09ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.258+0000 7f4bf2c1d640 1 --2- 172.21.15.153:0/338139122 >> v2:172.21.15.181:3300/0 conn(0x7f4bec09ef70 0x7f4bec068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf241c640 1 -- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 msgr2=0x7f4bec131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf241c640 1 --2- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec131d50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4bec09c9c0 tx=0x7f4be80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b04842fdef48c4e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 1 -- 172.21.15.153:0/338139122 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8032440 con 0x7f4bec131980 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: dump: 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 1 -- 172.21.15.153:0/338139122 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be80325a0 con 0x7f4bec131980 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 5 monclient: authenticate success, global_id 154134 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 1 -- 172.21.15.153:0/338139122 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8032940 con 0x7f4bec131980 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 4 set_mon_vals no callback set 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf141a640 10 monclient: dump: 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 10 monclient: shutdown 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 1 -- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 msgr2=0x7f4bec131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 1 --2- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec131d50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4bec09c9c0 tx=0x7f4be80091d0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 1 -- 172.21.15.153:0/338139122 shutdown_connections 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 1 --2- 172.21.15.153:0/338139122 >> v2:172.21.15.181:3300/0 conn(0x7f4bec09ef70 0x7f4bec068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 1 --2- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09fe10 0x7f4bec09ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.261+0000 7f4bf4680640 1 --2- 172.21.15.153:0/338139122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec131d50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 1 -- 172.21.15.153:0/338139122 >> 172.21.15.153:0/338139122 conn(0x7f4bec0a5050 msgr2=0x7f4bec0a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 1 -- 172.21.15.153:0/338139122 shutdown_connections 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 1 -- 172.21.15.153:0/338139122 wait complete. 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 10 monclient: monmap: 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 1 Processor -- start 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 1 -- start start 2024-09-15T03:40:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.262+0000 7f4bf4680640 10 monclient: init 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 0x7f4bec09abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): picked mon.noname-c con 0x7f4bec09ef70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bec09fe10 0x7f4bec130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): picked mon.noname-b con 0x7f4bec09fe10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec11b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): picked mon.noname-a con 0x7f4bec131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec09b660 con 0x7f4bec131980 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec069eb0 con 0x7f4bec09fe10 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec0a2940 con 0x7f4bec09ef70 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf4680640 10 monclient(hunting): authenticate will time out at 1426.902362s 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 0x7f4bec09abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 0x7f4bec09abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48450/0 (socket says 172.21.15.153:48450) 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 -- 172.21.15.153:0/1485894869 learned_addr learned my addr 172.21.15.153:0/1485894869 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): get_auth_request con 0x7f4bec09ef70 auth_method 0 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec11b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): get_auth_request con 0x7f4bec131980 auth_method 0 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf2c1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient(hunting): handle_auth_done global_id 124460 payload 824 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 -- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bec09fe10 msgr2=0x7f4bec130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bec09fe10 0x7f4bec130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 -- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 msgr2=0x7f4bec11b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec11b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 monclient: found mon.noname-c 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.263+0000 7f4bf241c640 1 -- 172.21.15.153:0/1485894869 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc009390 con 0x7f4bec09ef70 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.265+0000 7f4bf241c640 1 -- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 msgr2=0x7f4bec09abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.265+0000 7f4bf241c640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 0x7f4bec09abb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009210 tx=0x7f4be8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be80319c0 con 0x7f4bec09ef70 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: dump: 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.278 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be8031b20 con 0x7f4bec09ef70 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 5 monclient: authenticate success, global_id 124460 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 10 monclient: _renew_subs 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be8009450 con 0x7f4bec09ef70 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be803d440 con 0x7f4bec09ef70 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 10 monclient: dump: 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf0c19640 4 set_mon_vals no callback set 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 10 monclient: _renew_subs 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec11e070 con 0x7f4bec09ef70 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.266+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4be8039040 con 0x7f4bec09ef70 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4be27fc640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bd0038740 0x7f4bd003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f4be806c360 con 0x7f4bec09ef70 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4bf4680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4bf4680640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4bf4680640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec11e470 con 0x7f4bec09ef70 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4bf1c1b640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bd0038740 0x7f4bd003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.267+0000 7f4bf1c1b640 10 monclient: get_auth_request con 0x7f4bd0038740 auth_method 0 2024-09-15T03:40:29.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.269+0000 7f4bf1c1b640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bd0038740 0x7f4bd003abf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4bec06a870 tx=0x7f4bdc0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.271+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4be8042080 con 0x7f4bec09ef70 2024-09-15T03:40:29.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.271+0000 7f4be27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.271+0000 7f4be27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.488+0000 7f4bf4680640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.488+0000 7f4bf4680640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.488+0000 7f4bf4680640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.488+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4bec067f00 con 0x7f4bec09ef70 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.489+0000 7f4be27fc640 1 -- 172.21.15.153:0/1485894869 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4be8037020 con 0x7f4bec09ef70 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.489+0000 7f4be27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.489+0000 7f4be27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:29.490 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:40:29.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bd0038740 msgr2=0x7f4bd003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bd0038740 0x7f4bd003abf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4bec06a870 tx=0x7f4bdc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 10 monclient: shutdown 2024-09-15T03:40:29.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 msgr2=0x7f4bec09abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 0x7f4bec09abb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009210 tx=0x7f4be8032000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 shutdown_connections 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bd0038740 0x7f4bd003abf0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec131980 0x7f4bec11b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bec09fe10 0x7f4bec130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 --2- 172.21.15.153:0/1485894869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bec09ef70 0x7f4bec09abb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 >> 172.21.15.153:0/1485894869 conn(0x7f4bec0a5050 msgr2=0x7f4bec0a4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 shutdown_connections 2024-09-15T03:40:29.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.491+0000 7f4bf4680640 1 -- 172.21.15.153:0/1485894869 wait complete. 2024-09-15T03:40:29.502 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:40:29.502 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:40:29.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:40:29.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:40:29.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.595+0000 7fcd7d22b640 10 monclient: get_monmap_and_config 2024-09-15T03:40:29.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.595+0000 7fcd7d22b640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.595+0000 7fcd7d22b640 10 monclient: monmap: 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:29.596450+0000 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:29.596450+0000 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:29.597 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 Processor -- start 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 -- start start 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient: init 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcd780981c0 0x7fcd78098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): picked mon.noname-c con 0x7fcd780981c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd78131da0 0x7fcd78067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): picked mon.noname-a con 0x7fcd78131da0 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 0x7fcd78067650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): picked mon.noname-b con 0x7fcd78132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd78131bd0 con 0x7fcd78131da0 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd78098ad0 con 0x7fcd780981c0 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7809ca60 con 0x7fcd78132170 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7d22b640 10 monclient(hunting): authenticate will time out at 1427.235798s 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 0x7fcd78067650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd78131da0 0x7fcd78067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 0x7fcd78067650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48458/0 (socket says 172.21.15.153:48458) 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd78131da0 0x7fcd78067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37848/0 (socket says 172.21.15.153:37848) 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 1 -- 172.21.15.153:0/3057404994 learned_addr learned my addr 172.21.15.153:0/3057404994 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd78132170 auth_method 0 2024-09-15T03:40:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd78131da0 auth_method 0 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.596+0000 7fcd7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd76d9d640 1 --2- 172.21.15.153:0/3057404994 >> v2:172.21.15.181:3300/0 conn(0x7fcd780981c0 0x7fcd78098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd780981c0 auth_method 0 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 10 monclient(hunting): handle_auth_done global_id 154140 payload 274 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 1 -- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 msgr2=0x7fcd78067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 1 --2- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 0x7fcd78067650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 1 -- 172.21.15.153:0/3057404994 >> v2:172.21.15.181:3300/0 conn(0x7fcd780981c0 msgr2=0x7fcd78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 1 --2- 172.21.15.153:0/3057404994 >> v2:172.21.15.181:3300/0 conn(0x7fcd780981c0 0x7fcd78098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 1 monclient: found mon.noname-a 2024-09-15T03:40:29.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7659c640 1 -- 172.21.15.153:0/3057404994 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7809cca0 con 0x7fcd78131da0 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.597+0000 7fcd7759e640 1 --2- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 0x7fcd78067650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7659c640 1 -- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78131da0 msgr2=0x7fcd78067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7659c640 1 --2- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78131da0 0x7fcd78067110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd78066560 tx=0x7fcd680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d8306b65485f0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 1 -- 172.21.15.153:0/3057404994 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6802adf0 con 0x7fcd78131da0 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: dump: 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.606 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 1 -- 172.21.15.153:0/3057404994 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68032440 con 0x7fcd78131da0 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 5 monclient: authenticate success, global_id 154140 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 1 -- 172.21.15.153:0/3057404994 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd680327e0 con 0x7fcd78131da0 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 4 set_mon_vals no callback set 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd75d9b640 10 monclient: dump: 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.608 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 10 monclient: shutdown 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 -- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78131da0 msgr2=0x7fcd78067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78131da0 0x7fcd78067110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd78066560 tx=0x7fcd680090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 -- 172.21.15.153:0/3057404994 shutdown_connections 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78132170 0x7fcd78067650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/3057404994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78131da0 0x7fcd78067110 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/3057404994 >> v2:172.21.15.181:3300/0 conn(0x7fcd780981c0 0x7fcd78098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 -- 172.21.15.153:0/3057404994 >> 172.21.15.153:0/3057404994 conn(0x7fcd780a1e30 msgr2=0x7fcd780a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 -- 172.21.15.153:0/3057404994 shutdown_connections 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.600+0000 7fcd7d22b640 1 -- 172.21.15.153:0/3057404994 wait complete. 2024-09-15T03:40:29.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.601+0000 7fcd7d22b640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.601+0000 7fcd7d22b640 10 monclient: monmap: 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:29.610 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.601+0000 7fcd7d22b640 1 Processor -- start 2024-09-15T03:40:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.601+0000 7fcd7d22b640 1 -- start start 2024-09-15T03:40:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.601+0000 7fcd7d22b640 10 monclient: init 2024-09-15T03:40:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd780981c0 0x7fcd78130770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): picked mon.noname-a con 0x7fcd780981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 0x7fcd7811a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): picked mon.noname-c con 0x7fcd78131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd78132170 0x7fcd7811a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): picked mon.noname-b con 0x7fcd78132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7811dec0 con 0x7fcd780981c0 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78098cb0 con 0x7fcd78132170 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7809b9e0 con 0x7fcd78131da0 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7d22b640 10 monclient(hunting): authenticate will time out at 1427.241221s 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd780981c0 0x7fcd78130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 0x7fcd7811a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd780981c0 0x7fcd78130770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37856/0 (socket says 172.21.15.153:37856) 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd78131da0 auth_method 0 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 1 -- 172.21.15.153:0/607386493 learned_addr learned my addr 172.21.15.153:0/607386493 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd780981c0 auth_method 0 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd78132170 0x7fcd7811a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd78132170 auth_method 0 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient(hunting): handle_auth_done global_id 124466 payload 824 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd78132170 msgr2=0x7fcd7811a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd78132170 0x7fcd7811a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd780981c0 msgr2=0x7fcd78130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd780981c0 0x7fcd78130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 monclient: found mon.noname-c 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd7659c640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd70009ae0 con 0x7fcd78131da0 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.602+0000 7fcd76d9d640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd780981c0 0x7fcd78130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd7659c640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 msgr2=0x7fcd7811a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd7659c640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 0x7fcd7811a310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcd780714b0 tx=0x7fcd68032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68031b10 con 0x7fcd78131da0 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 10 monclient: dump: 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.621 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68031c70 con 0x7fcd78131da0 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd7d22b640 5 monclient: authenticate success, global_id 124466 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.604+0000 7fcd7d22b640 10 monclient: _renew_subs 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd68009390 con 0x7fcd78131da0 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd5effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6803d440 con 0x7fcd78131da0 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd5effd640 10 monclient: dump: 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 10 monclient: _renew_subs 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7559a640 4 set_mon_vals no callback set 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7811d890 con 0x7fcd78131da0 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcd68039040 con 0x7fcd78131da0 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 0x7fcd7811ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78065f90 con 0x7fcd78130cb0 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd76d9d640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 0x7fcd7811ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd76d9d640 10 monclient: get_auth_request con 0x7fcd78130cb0 auth_method 0 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.605+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd5effd640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd64038740 0x7fcd6403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcd6806c3b0 con 0x7fcd78131da0 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_done global_id 154146 payload 824 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd76d9d640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd7811dbe0 con 0x7fcd78130cb0 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd7759e640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd64038740 0x7fcd6403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.606+0000 7fcd7759e640 10 monclient: get_auth_request con 0x7fcd64038740 auth_method 0 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd76d9d640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 msgr2=0x7fcd7811ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd76d9d640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 0x7fcd7811ad90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c010ce0 tx=0x7fcd6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c014640 con 0x7fcd78130cb0 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 10 monclient: dump: 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd7759e640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd64038740 0x7fcd6403abf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcd700072f0 tx=0x7fcd7000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcd6c0147a0 con 0x7fcd78130cb0 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 msgr2=0x7fcd7811ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 0x7fcd7811ad90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c010ce0 tx=0x7fcd6c008040 comp rx=0 tx=0).stop 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 msgr2=0x7fcd7811ad90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.608+0000 7fcd5effd640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd78130cb0 0x7fcd7811ad90 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c010ce0 tx=0x7fcd6c008040 comp rx=0 tx=0).stop 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd7d22b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 0x7fcd78099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd7d22b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78131080 con 0x7fcd7811b2d0 2024-09-15T03:40:29.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 0x7fcd78099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient: get_auth_request con 0x7fcd7811b2d0 auth_method 0 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 10 monclient(hunting): handle_auth_done global_id 154149 payload 824 2024-09-15T03:40:29.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.614+0000 7fcd76d9d640 1 -- 172.21.15.153:0/607386493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcd7811b6d0 con 0x7fcd7811b2d0 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd76d9d640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 msgr2=0x7fcd78099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd76d9d640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 0x7fcd78099310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c007b50 tx=0x7fcd6c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c01ce90 con 0x7fcd7811b2d0 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 10 monclient: dump: 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.629 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fcd6c0135e0 con 0x7fcd7811b2d0 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 msgr2=0x7fcd78099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 0x7fcd78099310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c007b50 tx=0x7fcd6c01c000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 msgr2=0x7fcd78099310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd5effd640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7811b2d0 0x7fcd78099310 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c007b50 tx=0x7fcd6c01c000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 124, 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:40:29.631 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.632 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.633 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:29.634 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.635 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:29.638 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:29.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.641 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:29.642 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.617+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd64038740 msgr2=0x7fcd6403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd64038740 0x7fcd6403abf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcd700072f0 tx=0x7fcd7000b040 comp rx=0 tx=0).stop 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 10 monclient: shutdown 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 msgr2=0x7fcd7811a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 0x7fcd7811a310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcd780714b0 tx=0x7fcd68032000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7759e640 1 -- 172.21.15.153:0/607386493 reap_dead start 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 shutdown_connections 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 --2- 172.21.15.153:0/607386493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd78131da0 0x7fcd7811a310 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 >> 172.21.15.153:0/607386493 conn(0x7fcd780a1e30 msgr2=0x7fcd780a0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 shutdown_connections 2024-09-15T03:40:29.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.618+0000 7fcd7d22b640 1 -- 172.21.15.153:0/607386493 wait complete. 2024-09-15T03:40:29.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:40:29.645 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:40:29.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.738+0000 7fb5eef9d640 10 monclient: get_monmap_and_config 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.738+0000 7fb5eef9d640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.738+0000 7fb5eef9d640 10 monclient: monmap: 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:29.738926+0000 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:29.738926+0000 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:29.740 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.738+0000 7fb5eef9d640 1 Processor -- start 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.738+0000 7fb5eef9d640 1 -- start start 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.738+0000 7fb5eef9d640 10 monclient: init 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb5e809a510 0x7fb5e8096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e809a510 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5e80971a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5e80975a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e809cb30 con 0x7fb5e80975a0 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e809a900 con 0x7fb5e80971a0 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e806ed80 con 0x7fb5e809a510 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5eef9d640 10 monclient(hunting): authenticate will time out at 1427.378471s 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb5e809a510 0x7fb5e8096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ed53a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb5e80975a0 0x7fb5e812fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46746/0 (socket says 172.21.15.153:46746) 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ed53a640 1 -- 172.21.15.153:0/1056734427 learned_addr learned my addr 172.21.15.153:0/1056734427 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): get_auth_request con 0x7fb5e809a510 auth_method 0 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_done global_id 154152 payload 274 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 1 -- 172.21.15.153:0/1056734427 >> v1:172.21.15.181:6789/0 conn(0x7fb5e80971a0 legacy=0x7fb5e812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 1 -- 172.21.15.153:0/1056734427 >> v1:172.21.15.153:6789/0 conn(0x7fb5e80975a0 legacy=0x7fb5e812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 1 monclient: found mon.noname-a 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.739+0000 7fb5ecd39640 1 -- 172.21.15.153:0/1056734427 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e809c820 con 0x7fb5e809a510 2024-09-15T03:40:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.742+0000 7fb5ecd39640 1 -- 172.21.15.153:0/1056734427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e809a510 msgr2=0x7fb5e8096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.742+0000 7fb5ecd39640 1 --2- 172.21.15.153:0/1056734427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e809a510 0x7fb5e8096c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009880 tx=0x7fb5e40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a257eaf3ed6205 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.742+0000 7fb5df7fe640 1 -- 172.21.15.153:0/1056734427 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4032440 con 0x7fb5e809a510 2024-09-15T03:40:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.742+0000 7fb5df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:29.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: dump: 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 1 -- 172.21.15.153:0/1056734427 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e40325a0 con 0x7fb5e809a510 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 5 monclient: authenticate success, global_id 154152 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 1 -- 172.21.15.153:0/1056734427 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4032940 con 0x7fb5e809a510 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 4 set_mon_vals no callback set 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5df7fe640 10 monclient: dump: 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 10 monclient: shutdown 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 -- 172.21.15.153:0/1056734427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e809a510 msgr2=0x7fb5e8096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 --2- 172.21.15.153:0/1056734427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e809a510 0x7fb5e8096c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009880 tx=0x7fb5e40090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 -- 172.21.15.153:0/1056734427 shutdown_connections 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 --2- 172.21.15.153:0/1056734427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e809a510 0x7fb5e8096c60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 -- 172.21.15.153:0/1056734427 >> 172.21.15.153:0/1056734427 conn(0x7fb5e80672b0 msgr2=0x7fb5e80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 -- 172.21.15.153:0/1056734427 shutdown_connections 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.743+0000 7fb5eef9d640 1 -- 172.21.15.153:0/1056734427 wait complete. 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 10 monclient: monmap: 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 1 Processor -- start 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 1 -- start start 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 10 monclient: init 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.744+0000 7fb5eef9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e80971a0 0x7fb5e8141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e80971a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e80975a0 0x7fb5e812ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e80975a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5e809a510 0x7fb5e8098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e809a510 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e81420a0 con 0x7fb5e80975a0 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e8099eb0 con 0x7fb5e809a510 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e811def0 con 0x7fb5e80971a0 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5eef9d640 10 monclient(hunting): authenticate will time out at 1427.384047s 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e80971a0 0x7fb5e8141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e80975a0 0x7fb5e812ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e80971a0 0x7fb5e8141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48488/0 (socket says 172.21.15.153:48488) 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 1 -- 172.21.15.153:0/3552554042 learned_addr learned my addr 172.21.15.153:0/3552554042 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): get_auth_request con 0x7fb5e80971a0 auth_method 0 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): get_auth_request con 0x7fb5e80975a0 auth_method 0 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ed53a640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5e809a510 0x7fb5e8098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ed53a640 10 monclient(hunting): get_auth_request con 0x7fb5e809a510 auth_method 0 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ed53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ed53a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ed53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient(hunting): handle_auth_done global_id 154155 payload 824 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e80971a0 msgr2=0x7fb5e8141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e80971a0 0x7fb5e8141130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5e809a510 msgr2=0x7fb5e8098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5e809a510 0x7fb5e8098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 monclient: found mon.noname-a 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5dffff640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009390 con 0x7fb5e80975a0 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.745+0000 7fb5ecd39640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e80971a0 0x7fb5e8141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dffff640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e80975a0 msgr2=0x7fb5e812ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dffff640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e80975a0 0x7fb5e812ead0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c510 tx=0x7fb5d8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80117a0 con 0x7fb5e80975a0 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: dump: 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8011920 con 0x7fb5e80975a0 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 5 monclient: authenticate success, global_id 154155 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 10 monclient: _renew_subs 2024-09-15T03:40:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e812f730 con 0x7fb5e80975a0 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011ce0 con 0x7fb5e80975a0 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5dcff9640 10 monclient: dump: 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 10 monclient: _renew_subs 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e811cd20 con 0x7fb5e80975a0 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.747+0000 7fb5deffd640 4 set_mon_vals no callback set 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5eef9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5eef9d640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 0x7fb5e8098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5eef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e8065f20 con 0x7fb5e812f0d0 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb5d8012630 con 0x7fb5e80975a0 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 0x7fb5e8098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient: get_auth_request con 0x7fb5e812f0d0 auth_method 0 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5dcff9640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb5d4038740 0x7fb5d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.748+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_done global_id 124472 payload 824 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.749+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb5d804c1b0 con 0x7fb5e80975a0 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.749+0000 7fb5ecd39640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5e809c3e0 con 0x7fb5e812f0d0 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.749+0000 7fb5ed53a640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb5d4038740 0x7fb5d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.749+0000 7fb5ed53a640 10 monclient: get_auth_request con 0x7fb5d4038740 auth_method 0 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5ecd39640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 msgr2=0x7fb5e8098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5ecd39640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 0x7fb5e8098840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009360 tx=0x7fb5e4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4031e30 con 0x7fb5e812f0d0 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 10 monclient: dump: 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.767 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5ed53a640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb5d4038740 0x7fb5d403abf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0007bb0 tx=0x7fb5e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb5e4041070 con 0x7fb5e812f0d0 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 msgr2=0x7fb5e8098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 0x7fb5e8098840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009360 tx=0x7fb5e4032000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 msgr2=0x7fb5e8098840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.751+0000 7fb5dcff9640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e812f0d0 0x7fb5e8098840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5eef9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5eef9d640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 0x7fb5e8098d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5eef9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e811e220 con 0x7fb5e8141670 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 0x7fb5e8098d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient: get_auth_request con 0x7fb5e8141670 auth_method 0 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 10 monclient(hunting): handle_auth_done global_id 124475 payload 824 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.757+0000 7fb5ecd39640 1 -- 172.21.15.153:0/3552554042 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb5e804bf60 con 0x7fb5e8141670 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5ecd39640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 msgr2=0x7fb5e8098d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5ecd39640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 0x7fb5e8098d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4044000 tx=0x7fb5e403f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4053410 con 0x7fb5e8141670 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 10 monclient: dump: 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.770 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fb5e4032d30 con 0x7fb5e8141670 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 msgr2=0x7fb5e8098d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 0x7fb5e8098d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4044000 tx=0x7fb5e403f070 comp rx=0 tx=0).stop 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 msgr2=0x7fb5e8098d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5dcff9640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5e8141670 0x7fb5e8098d80 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4044000 tx=0x7fb5e403f070 comp rx=0 tx=0).stop 2024-09-15T03:40:29.771 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 124, 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:40:29.772 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.773 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.774 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:29.775 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.776 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.777 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.778 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.780 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.781 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.782 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:29.783 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:40:29.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:40:29.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:29.786 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:29.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb5d4038740 msgr2=0x7fb5d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb5d4038740 0x7fb5d403abf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0007bb0 tx=0x7fb5e0009040 comp rx=0 tx=0).stop 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 10 monclient: shutdown 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e80975a0 msgr2=0x7fb5e812ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 1 --2- 172.21.15.153:0/3552554042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5e80975a0 0x7fb5e812ead0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c510 tx=0x7fb5d8007e10 comp rx=0 tx=0).stop 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5ed53a640 1 -- 172.21.15.153:0/3552554042 reap_dead start 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5ed53a640 1 -- 172.21.15.153:0/3552554042 reap_dead start 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 shutdown_connections 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.760+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 >> 172.21.15.153:0/3552554042 conn(0x7fb5e80672b0 msgr2=0x7fb5e809fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.761+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 shutdown_connections 2024-09-15T03:40:29.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.761+0000 7fb5eef9d640 1 -- 172.21.15.153:0/3552554042 wait complete. 2024-09-15T03:40:29.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:40:29.789 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:40:29.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:40:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.881+0000 7fb946ac9640 10 monclient: get_monmap_and_config 2024-09-15T03:40:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.881+0000 7fb946ac9640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.881+0000 7fb946ac9640 10 monclient: monmap: 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:29.882535+0000 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:29.882535+0000 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:29.883 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 1 Processor -- start 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 1 -- start start 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 10 monclient: init 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb94009c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb94009e840 0x7fb94009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fb94009e840 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.882+0000 7fb946ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb940136cb0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:29.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb946ac9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94009ec40 con 0x7fb940136cb0 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb946ac9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94006a320 con 0x7fb94009c860 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb946ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb946ac9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb940068a20 con 0x7fb94009e840 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb946ac9640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb946ac9640 10 monclient(hunting): authenticate will time out at 1427.522040s 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb945066640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb940136cb0 0x7fb940139090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46754/0 (socket says 172.21.15.153:46754) 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb945066640 1 -- 172.21.15.153:0/3425329187 learned_addr learned my addr 172.21.15.153:0/3425329187 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb937fff640 1 --2- 172.21.15.153:0/3425329187 >> v2:172.21.15.181:3300/0 conn(0x7fb94009e840 0x7fb94009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb937fff640 10 monclient(hunting): get_auth_request con 0x7fb94009e840 auth_method 0 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb937fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb937fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.883+0000 7fb937fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb945066640 1 -- 172.21.15.153:0/3425329187 >> v1:172.21.15.153:6789/0 conn(0x7fb940136cb0 legacy=0x7fb940139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb944865640 1 -- 172.21.15.153:0/3425329187 >> v1:172.21.15.181:6789/0 conn(0x7fb94009c860 legacy=0x7fb94009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 1 -- 172.21.15.153:0/3425329187 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518671495 0 0) 0x7fb94009ec40 con 0x7fb940136cb0 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 10 monclient(hunting): my global_id is 154158 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 1 -- 172.21.15.153:0/3425329187 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9280034e0 con 0x7fb940136cb0 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 10 monclient(hunting): handle_auth_done global_id 144211 payload 274 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 1 -- 172.21.15.153:0/3425329187 >> v1:172.21.15.181:6789/0 conn(0x7fb94009c860 legacy=0x7fb94009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 1 -- 172.21.15.153:0/3425329187 >> v1:172.21.15.153:6789/0 conn(0x7fb940136cb0 legacy=0x7fb940139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 1 -- 172.21.15.153:0/3425329187 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4113924982 0 0) 0x7fb9280034e0 con 0x7fb940136cb0 2024-09-15T03:40:29.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 1 monclient: found mon.noname-c 2024-09-15T03:40:29.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:40:29.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb937fff640 1 -- 172.21.15.153:0/3425329187 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb940068e60 con 0x7fb94009e840 2024-09-15T03:40:29.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.885+0000 7fb9377fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb937fff640 1 -- 172.21.15.153:0/3425329187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94009e840 msgr2=0x7fb94009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb937fff640 1 --2- 172.21.15.153:0/3425329187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94009e840 0x7fb94009aba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb930009950 tx=0x7fb9300090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e0551cf468211a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb9377fe640 1 -- 172.21.15.153:0/3425329187 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938001aa0 con 0x7fb94009e840 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb9377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb9377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb9377fe640 10 monclient: dump: 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb9377fe640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb9377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb946ac9640 5 monclient: authenticate success, global_id 144211 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.888+0000 7fb946ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb9377fe640 1 -- 172.21.15.153:0/3425329187 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93002acd0 con 0x7fb94009e840 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb9377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb9377fe640 1 -- 172.21.15.153:0/3425329187 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930032650 con 0x7fb94009e840 2024-09-15T03:40:29.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 4 set_mon_vals no callback set 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb9377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb9377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb9377fe640 10 monclient: dump: 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 10 monclient: shutdown 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 -- 172.21.15.153:0/3425329187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94009e840 msgr2=0x7fb94009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3425329187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94009e840 0x7fb94009aba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb930009950 tx=0x7fb9300090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 -- 172.21.15.153:0/3425329187 shutdown_connections 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3425329187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94009e840 0x7fb94009aba0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 -- 172.21.15.153:0/3425329187 >> 172.21.15.153:0/3425329187 conn(0x7fb9400a2e40 msgr2=0x7fb9400a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 -- 172.21.15.153:0/3425329187 shutdown_connections 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.889+0000 7fb946ac9640 1 -- 172.21.15.153:0/3425329187 wait complete. 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient: build_initial_monmap 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient: monmap: 2024-09-15T03:40:29.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 Processor -- start 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 -- start start 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient: init 2024-09-15T03:40:29.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94009c860 0x7fb94006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fb94009c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 0x7fb94006d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fb94009e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb940136cb0 0x7fb94006dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fb940136cb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb940099cc0 con 0x7fb94009c860 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400a0c40 con 0x7fb940136cb0 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94006ecd0 con 0x7fb94009e840 2024-09-15T03:40:29.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.890+0000 7fb946ac9640 10 monclient(hunting): authenticate will time out at 1427.529988s 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94009c860 0x7fb94006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 0x7fb94006d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94009c860 0x7fb94006d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37910/0 (socket says 172.21.15.153:37910) 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 1 -- 172.21.15.153:0/3507069266 learned_addr learned my addr 172.21.15.153:0/3507069266 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): get_auth_request con 0x7fb94009c860 auth_method 0 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): get_auth_request con 0x7fb94009e840 auth_method 0 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb945066640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb940136cb0 0x7fb94006dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb945066640 10 monclient(hunting): get_auth_request con 0x7fb940136cb0 auth_method 0 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb945066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb945066640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb945066640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient(hunting): handle_auth_done global_id 124478 payload 824 2024-09-15T03:40:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb940136cb0 msgr2=0x7fb94006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb940136cb0 0x7fb94006dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94009c860 msgr2=0x7fb94006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94009c860 0x7fb94006d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient: _finish_hunting 0 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 monclient: found mon.noname-c 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb937fff640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930009390 con 0x7fb94009e840 2024-09-15T03:40:29.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.891+0000 7fb944865640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94009c860 0x7fb94006d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb937fff640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 msgr2=0x7fb94006d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb937fff640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 0x7fb94006d710 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb930031000 tx=0x7fb930032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93003b820 con 0x7fb94009e840 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: dump: 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.910 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: _finish_auth 0 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93003b980 con 0x7fb94009e840 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93003bd20 con 0x7fb94009e840 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 5 monclient: authenticate success, global_id 124478 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb934ff9640 10 monclient: dump: 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 10 monclient: _renew_subs 2024-09-15T03:40:29.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb940135db0 con 0x7fb94009e840 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb936ffd640 4 set_mon_vals no callback set 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 10 monclient: _renew_subs 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.893+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9401367f0 con 0x7fb94009e840 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb930039040 con 0x7fb94009e840 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb946ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb934ff9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 0x7fb92c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 0x7fb94002aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb946ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94011e900 con 0x7fb94011d8b0 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.894+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb93002ae40 con 0x7fb94009e840 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 0x7fb94002aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient: get_auth_request con 0x7fb94011d8b0 auth_method 0 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb944865640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 0x7fb92c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb944865640 10 monclient: get_auth_request con 0x7fb92c038740 auth_method 0 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb944865640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 msgr2=0x7fb92c03abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:40:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb944865640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 msgr2=0x7fb92c03abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:40:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb944865640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 0x7fb92c03abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb944865640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 0x7fb92c03abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 10 monclient(hunting): handle_auth_done global_id 144220 payload 824 2024-09-15T03:40:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.895+0000 7fb945066640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb94011ea70 con 0x7fb94011d8b0 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.897+0000 7fb945066640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 msgr2=0x7fb94002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.897+0000 7fb945066640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 0x7fb94002aa40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb940099fc0 tx=0x7fb938005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.897+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93800ce60 con 0x7fb94011d8b0 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.897+0000 7fb934ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.897+0000 7fb934ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.897+0000 7fb934ff9640 10 monclient: dump: 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb938013420 con 0x7fb94011d8b0 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 msgr2=0x7fb94002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 0x7fb94002aa40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb940099fc0 tx=0x7fb938005bc0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 msgr2=0x7fb94002aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.898+0000 7fb934ff9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94011d8b0 0x7fb94002aa40 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb940099fc0 tx=0x7fb938005bc0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.903+0000 7fb946ac9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.903+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 0x7fb94005e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.903+0000 7fb946ac9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.903+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb940065e00 con 0x7fb94006e190 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 0x7fb94005e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient: get_auth_request con 0x7fb94006e190 auth_method 0 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:29.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 10 monclient(hunting): handle_auth_done global_id 144223 payload 824 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.904+0000 7fb945066640 1 -- 172.21.15.153:0/3507069266 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb94006e590 con 0x7fb94006e190 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.906+0000 7fb945066640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 msgr2=0x7fb94005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.906+0000 7fb945066640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 0x7fb94005e310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb938002250 tx=0x7fb938015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.906+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93801fdb0 con 0x7fb94006e190 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.906+0000 7fb934ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.906+0000 7fb934ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.906+0000 7fb934ff9640 10 monclient: dump: 2024-09-15T03:40:29.921 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fb93801c410 con 0x7fb94006e190 2024-09-15T03:40:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 msgr2=0x7fb94005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 0x7fb94005e310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb938002250 tx=0x7fb938015000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 msgr2=0x7fb94005e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb934ff9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb94006e190 0x7fb94005e310 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb938002250 tx=0x7fb938015000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 124, 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.932 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.933 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.933 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.933 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:29.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:29.933 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:29.934 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:29.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.934 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:29.934 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:29.934 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:29.935 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.936 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:29.937 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 msgr2=0x7fb92c03abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb92c038740 0x7fb92c03abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 10 monclient: shutdown 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 msgr2=0x7fb94006d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb945066640 1 -- 172.21.15.153:0/3507069266 reap_dead start 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 0x7fb94006d710 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb930031000 tx=0x7fb930032000 comp rx=0 tx=0).stop 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 shutdown_connections 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 --2- 172.21.15.153:0/3507069266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb94009e840 0x7fb94006d710 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:29.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 >> 172.21.15.153:0/3507069266 conn(0x7fb9400a2e40 msgr2=0x7fb940098b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:29.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 shutdown_connections 2024-09-15T03:40:29.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:29.907+0000 7fb946ac9640 1 -- 172.21.15.153:0/3507069266 wait complete. 2024-09-15T03:40:29.940 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:40:29.940 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:40:29.940 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:40:29.940 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:40:29.940 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:40:29.941 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:40:29.941 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:40:29.951 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:40:29.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:40:29.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.085+0000 7f387375d640 10 monclient: get_monmap_and_config 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.085+0000 7f387375d640 10 monclient: build_initial_monmap 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.085+0000 7f387375d640 10 monclient: monmap: 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:30.085833+0000 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:30.085833+0000 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:30.086 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:30.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.085+0000 7f387375d640 1 Processor -- start 2024-09-15T03:40:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.085+0000 7f387375d640 1 -- start start 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.085+0000 7f387375d640 10 monclient: init 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f386c1318f0 0x7f386c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): picked mon.noname-a con 0x7f386c1318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f386c09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): picked mon.noname-b con 0x7f386c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c131cc0 con 0x7f386c09ea50 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386c09f530 con 0x7f386c1318f0 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c09fbf0 con 0x7f386c136c20 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f387375d640 10 monclient(hunting): authenticate will time out at 1427.725447s 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f38714f9640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f386c1318f0 msgr2=0x7f386c131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f38714f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f386c1318f0 0x7f386c131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48532/0 (socket says 172.21.15.153:48532) 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 -- 172.21.15.153:0/944915084 learned_addr learned my addr 172.21.15.153:0/944915084 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): get_auth_request con 0x7f386c136c20 auth_method 0 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient(hunting): handle_auth_done global_id 124481 payload 274 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 -- 172.21.15.153:0/944915084 >> v2:172.21.15.153:3300/0 conn(0x7f386c1318f0 msgr2=0x7f386c131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 --2- 172.21.15.153:0/944915084 >> v2:172.21.15.153:3300/0 conn(0x7f386c1318f0 0x7f386c131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 -- 172.21.15.153:0/944915084 >> v1:172.21.15.181:6789/0 conn(0x7f386c09ea50 legacy=0x7f386c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 monclient: found mon.noname-b 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:40:30.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.086+0000 7f3871cfa640 1 -- 172.21.15.153:0/944915084 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c09fe30 con 0x7f386c136c20 2024-09-15T03:40:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3871cfa640 1 -- 172.21.15.153:0/944915084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 msgr2=0x7f386c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3871cfa640 1 --2- 172.21.15.153:0/944915084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c139000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3864009f20 tx=0x7f3864009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37748077a56b46df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 1 -- 172.21.15.153:0/944915084 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864007610 con 0x7f386c136c20 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: dump: 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: _finish_auth 0 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f387375d640 5 monclient: authenticate success, global_id 124481 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f387375d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 1 -- 172.21.15.153:0/944915084 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386402d050 con 0x7f386c136c20 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f387375d640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f387375d640 4 set_mon_vals no callback set 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 1 -- 172.21.15.153:0/944915084 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38640079a0 con 0x7f386c136c20 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.089+0000 7f3863fff640 10 monclient: dump: 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 10 monclient: shutdown 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 -- 172.21.15.153:0/944915084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 msgr2=0x7f386c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 --2- 172.21.15.153:0/944915084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c139000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3864009f20 tx=0x7f3864009840 comp rx=0 tx=0).stop 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 -- 172.21.15.153:0/944915084 shutdown_connections 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 --2- 172.21.15.153:0/944915084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c139000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 --2- 172.21.15.153:0/944915084 >> v2:172.21.15.153:3300/0 conn(0x7f386c1318f0 0x7f386c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 -- 172.21.15.153:0/944915084 >> 172.21.15.153:0/944915084 conn(0x7f386c067110 msgr2=0x7f386c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 -- 172.21.15.153:0/944915084 shutdown_connections 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 1 -- 172.21.15.153:0/944915084 wait complete. 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 10 monclient: build_initial_monmap 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.090+0000 7f387375d640 10 monclient: monmap: 2024-09-15T03:40:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 Processor -- start 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 -- start start 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient: init 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f386c09ea50 0x7f386c130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): picked mon.noname-b con 0x7f386c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f386c1318f0 0x7f386c0996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): picked mon.noname-a con 0x7f386c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c099be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): picked mon.noname-c con 0x7f386c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c11cdf0 con 0x7f386c1318f0 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c09b880 con 0x7f386c09ea50 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c09bf10 con 0x7f386c136c20 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f387375d640 10 monclient(hunting): authenticate will time out at 1427.730769s 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3870cf8640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f386c1318f0 msgr2=0x7f386c0996a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3870cf8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f386c1318f0 0x7f386c0996a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c099be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c099be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48540/0 (socket says 172.21.15.153:48540) 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 1 -- 172.21.15.153:0/3640337829 learned_addr learned my addr 172.21.15.153:0/3640337829 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): get_auth_request con 0x7f386c136c20 auth_method 0 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f38714f9640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f386c09ea50 0x7f386c130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f3871cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f38714f9640 10 monclient(hunting): get_auth_request con 0x7f386c09ea50 auth_method 0 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f38714f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f38714f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.091+0000 7f38714f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f38714f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f38714f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f38714f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 10 monclient(hunting): handle_auth_done global_id 124484 payload 824 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 1 -- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f386c09ea50 msgr2=0x7f386c130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f386c09ea50 0x7f386c130900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 1 -- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f386c1318f0 msgr2=0x7f386c0996a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f386c1318f0 0x7f386c0996a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 1 monclient: found mon.noname-c 2024-09-15T03:40:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.092+0000 7f3871cfa640 1 -- 172.21.15.153:0/3640337829 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3864009ae0 con 0x7f386c136c20 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f3871cfa640 1 -- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 msgr2=0x7f386c099be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f3871cfa640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c099be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3864009f20 tx=0x7f3864036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864031cb0 con 0x7f386c136c20 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: dump: 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: _finish_auth 0 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864031e10 con 0x7f386c136c20 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386403c650 con 0x7f386c136c20 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 5 monclient: authenticate success, global_id 124484 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38617fa640 10 monclient: dump: 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 10 monclient: _renew_subs 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f38637fe640 4 set_mon_vals no callback set 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c09de20 con 0x7f386c136c20 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 10 monclient: _renew_subs 2024-09-15T03:40:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.094+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c11d050 con 0x7f386c136c20 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f386403ca40 con 0x7f386c136c20 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f387375d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f387375d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f387375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c11ca10 con 0x7f386c136c20 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f38617fa640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f38500386f0 0x7f385003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.095+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f386406cee0 con 0x7f386c136c20 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.096+0000 7f38714f9640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f38500386f0 0x7f385003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.096+0000 7f38714f9640 10 monclient: get_auth_request con 0x7f38500386f0 auth_method 0 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.098+0000 7f38714f9640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f38500386f0 0x7f385003aba0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3868003940 tx=0x7f38680090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.100+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f386402ae60 con 0x7f386c136c20 2024-09-15T03:40:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.100+0000 7f38617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.100+0000 7f38617fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f387375d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f387375d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f387375d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f386c04bf60 con 0x7f386c136c20 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f38617fa640 1 -- 172.21.15.153:0/3640337829 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f3864040020 con 0x7f386c136c20 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f38617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.313+0000 7f38617fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:30.318 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:40:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f38500386f0 msgr2=0x7f385003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f38500386f0 0x7f385003aba0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3868003940 tx=0x7f38680090f0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 10 monclient: shutdown 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 msgr2=0x7f386c099be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c099be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3864009f20 tx=0x7f3864036000 comp rx=0 tx=0).stop 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 shutdown_connections 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f38500386f0 0x7f385003aba0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f386c136c20 0x7f386c099be0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f386c1318f0 0x7f386c0996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 --2- 172.21.15.153:0/3640337829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f386c09ea50 0x7f386c130900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 >> 172.21.15.153:0/3640337829 conn(0x7f386c067110 msgr2=0x7f386c0a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 shutdown_connections 2024-09-15T03:40:30.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:30.315+0000 7f387375d640 1 -- 172.21.15.153:0/3640337829 wait complete. 2024-09-15T03:40:33.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:40:33.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.432+0000 7fe220699640 10 monclient: get_monmap_and_config 2024-09-15T03:40:33.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.432+0000 7fe220699640 10 monclient: build_initial_monmap 2024-09-15T03:40:33.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.432+0000 7fe220699640 10 monclient: monmap: 2024-09-15T03:40:33.433 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:33.433490+0000 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:33.433490+0000 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:33.434 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:33.435 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:33.435 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:33.435 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 1 Processor -- start 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 1 -- start start 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 10 monclient: init 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2180981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 0x7fe218136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.433+0000 7fe220699640 10 monclient(hunting): picked mon.noname-b con 0x7fe218131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe218132170 0x7fe2181390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 10 monclient(hunting): picked mon.noname-c con 0x7fe218132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21809ca60 con 0x7fe2180981c0 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe218131bd0 con 0x7fe218132170 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21809cca0 con 0x7fe218131da0 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe220699640 10 monclient(hunting): authenticate will time out at 1431.073037s 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21e435640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fe2180981c0 legacy=0x7fe218098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 0x7fe218136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 0x7fe218136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48548/0 (socket says 172.21.15.153:48548) 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 -- 172.21.15.153:0/3291392698 learned_addr learned my addr 172.21.15.153:0/3291392698 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): get_auth_request con 0x7fe218131da0 auth_method 0 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21ec36640 1 --2- 172.21.15.153:0/3291392698 >> v2:172.21.15.181:3300/0 conn(0x7fe218132170 0x7fe2181390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21ec36640 10 monclient(hunting): get_auth_request con 0x7fe218132170 auth_method 0 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21ec36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21ec36640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21ec36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient(hunting): handle_auth_done global_id 124487 payload 274 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 -- 172.21.15.153:0/3291392698 >> v2:172.21.15.181:3300/0 conn(0x7fe218132170 msgr2=0x7fe2181390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 --2- 172.21.15.153:0/3291392698 >> v2:172.21.15.181:3300/0 conn(0x7fe218132170 0x7fe2181390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 -- 172.21.15.153:0/3291392698 >> v1:172.21.15.153:6789/0 conn(0x7fe2180981c0 legacy=0x7fe218098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient: _finish_hunting 0 2024-09-15T03:40:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 monclient: found mon.noname-b 2024-09-15T03:40:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:40:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.434+0000 7fe21dc34640 1 -- 172.21.15.153:0/3291392698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21809e010 con 0x7fe218131da0 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21dc34640 1 -- 172.21.15.153:0/3291392698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 msgr2=0x7fe218136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21dc34640 1 --2- 172.21.15.153:0/3291392698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 0x7fe218136b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe208009910 tx=0x7fe2080091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c66572f3433f805 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 1 -- 172.21.15.153:0/3291392698 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20802aea0 con 0x7fe218131da0 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: dump: 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: _finish_auth 0 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 5 monclient: authenticate success, global_id 124487 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 1 -- 172.21.15.153:0/3291392698 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe208032440 con 0x7fe218131da0 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 4 set_mon_vals no callback set 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 1 -- 172.21.15.153:0/3291392698 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2080327e0 con 0x7fe218131da0 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe21d433640 10 monclient: dump: 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 10 monclient: shutdown 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 1 -- 172.21.15.153:0/3291392698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 msgr2=0x7fe218136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.437+0000 7fe220699640 1 --2- 172.21.15.153:0/3291392698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 0x7fe218136b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe208009910 tx=0x7fe2080091d0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 -- 172.21.15.153:0/3291392698 shutdown_connections 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 --2- 172.21.15.153:0/3291392698 >> v2:172.21.15.181:3300/0 conn(0x7fe218132170 0x7fe2181390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 --2- 172.21.15.153:0/3291392698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218131da0 0x7fe218136b70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 -- 172.21.15.153:0/3291392698 >> 172.21.15.153:0/3291392698 conn(0x7fe2180a1e30 msgr2=0x7fe2180a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 -- 172.21.15.153:0/3291392698 shutdown_connections 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 -- 172.21.15.153:0/3291392698 wait complete. 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 10 monclient: build_initial_monmap 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 10 monclient: monmap: 2024-09-15T03:40:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 Processor -- start 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 1 -- start start 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.438+0000 7fe220699640 10 monclient: init 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2180981c0 0x7fe218137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): picked mon.noname-a con 0x7fe2180981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe218131da0 0x7fe2181384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): picked mon.noname-b con 0x7fe218131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 0x7fe218138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): picked mon.noname-c con 0x7fe218132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21811b1e0 con 0x7fe2180981c0 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe218130ba0 con 0x7fe218131da0 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe218118ee0 con 0x7fe218132170 2024-09-15T03:40:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe220699640 10 monclient(hunting): authenticate will time out at 1431.078415s 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21e435640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2180981c0 msgr2=0x7fe218137f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21e435640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2180981c0 0x7fe218137f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 0x7fe218138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 0x7fe218138a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:48562/0 (socket says 172.21.15.153:48562) 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 -- 172.21.15.153:0/1617983848 learned_addr learned my addr 172.21.15.153:0/1617983848 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): get_auth_request con 0x7fe218132170 auth_method 0 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe218131da0 0x7fe2181384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): get_auth_request con 0x7fe218131da0 auth_method 0 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:33.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21dc34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient(hunting): handle_auth_done global_id 124490 payload 824 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 -- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe218131da0 msgr2=0x7fe2181384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe218131da0 0x7fe2181384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 -- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2180981c0 msgr2=0x7fe218137f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2180981c0 0x7fe218137f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient: _finish_hunting 0 2024-09-15T03:40:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 monclient: found mon.noname-c 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.439+0000 7fe21ec36640 1 -- 172.21.15.153:0/1617983848 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208009450 con 0x7fe218132170 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe21ec36640 1 -- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 msgr2=0x7fe218138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe21ec36640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 0x7fe218138a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe210009fa0 tx=0x7fe210009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21000ee60 con 0x7fe218132170 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: dump: 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe210013590 con 0x7fe218132170 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 5 monclient: authenticate success, global_id 124490 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 10 monclient: _renew_subs 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210013950 con 0x7fe218132170 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe218098d10 con 0x7fe218132170 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe20e7fc640 10 monclient: dump: 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe21cc32640 4 set_mon_vals no callback set 2024-09-15T03:40:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 10 monclient: _renew_subs 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.442+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe218118960 con 0x7fe218132170 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe210013cc0 con 0x7fe218132170 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe220699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe220699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe220699640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21809be20 con 0x7fe218132170 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe20e7fc640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe1fc038740 0x7fe1fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe21004cd10 con 0x7fe218132170 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe21e435640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe1fc038740 0x7fe1fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.443+0000 7fe21e435640 10 monclient: get_auth_request con 0x7fe1fc038740 auth_method 0 2024-09-15T03:40:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.446+0000 7fe21e435640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe1fc038740 0x7fe1fc03abf0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe214003940 tx=0x7fe214006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.448+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe210014420 con 0x7fe218132170 2024-09-15T03:40:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.448+0000 7fe20e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.448+0000 7fe20e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:33.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe220699640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:40:33.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe220699640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:33.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe220699640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:33.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe21804bf60 con 0x7fe218132170 2024-09-15T03:40:33.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe20e7fc640 1 -- 172.21.15.153:0/1617983848 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fe21001b310 con 0x7fe218132170 2024-09-15T03:40:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe20e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.667+0000 7fe20e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:33.669 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:40:33.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe1fc038740 msgr2=0x7fe1fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:33.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe1fc038740 0x7fe1fc03abf0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe214003940 tx=0x7fe214006d80 comp rx=0 tx=0).stop 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 10 monclient: shutdown 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 msgr2=0x7fe218138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 0x7fe218138a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe210009fa0 tx=0x7fe210009900 comp rx=0 tx=0).stop 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 shutdown_connections 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe1fc038740 0x7fe1fc03abf0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe218132170 0x7fe218138a10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe218131da0 0x7fe2181384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 --2- 172.21.15.153:0/1617983848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2180981c0 0x7fe218137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 >> 172.21.15.153:0/1617983848 conn(0x7fe2180a1e30 msgr2=0x7fe2180977a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 shutdown_connections 2024-09-15T03:40:33.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:33.669+0000 7fe220699640 1 -- 172.21.15.153:0/1617983848 wait complete. 2024-09-15T03:40:36.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:40:36.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.784+0000 7f43fa59d640 10 monclient: get_monmap_and_config 2024-09-15T03:40:36.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.784+0000 7f43fa59d640 10 monclient: build_initial_monmap 2024-09-15T03:40:36.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.784+0000 7f43fa59d640 10 monclient: monmap: 2024-09-15T03:40:36.785 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:36.785 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:36.785432+0000 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:36.785432+0000 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:36.786 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.785+0000 7f43fa59d640 1 Processor -- start 2024-09-15T03:40:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.785+0000 7f43fa59d640 1 -- start start 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.785+0000 7f43fa59d640 10 monclient: init 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.785+0000 7f43fa59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.785+0000 7f43fa59d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43f40974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43f409a8a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f43f4134080 0x7f43f4136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 10 monclient(hunting): picked mon.noname-c con 0x7f43f4134080 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f412b210 con 0x7f43f40974f0 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f40978c0 con 0x7f43f409a8a0 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f409ca20 con 0x7f43f4134080 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:36.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43fa59d640 10 monclient(hunting): authenticate will time out at 1434.425080s 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f3fff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f43f40974f0 legacy=0x7f43f412ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f43f4134080 0x7f43f4136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f43f4134080 0x7f43f4136460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:35826/0 (socket says 172.21.15.153:35826) 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 1 -- 172.21.15.153:0/1834061427 learned_addr learned my addr 172.21.15.153:0/1834061427 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): get_auth_request con 0x7f43f4134080 auth_method 0 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:36.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.786+0000 7f43f8b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:36.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f37fe640 1 -- 172.21.15.153:0/1834061427 >> v1:172.21.15.181:6789/0 conn(0x7f43f409a8a0 legacy=0x7f43f409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:36.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 10 monclient(hunting): handle_auth_done global_id 144235 payload 274 2024-09-15T03:40:36.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 1 -- 172.21.15.153:0/1834061427 >> v1:172.21.15.181:6789/0 conn(0x7f43f409a8a0 legacy=0x7f43f409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 1 -- 172.21.15.153:0/1834061427 >> v1:172.21.15.153:6789/0 conn(0x7f43f40974f0 legacy=0x7f43f412ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:40:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 1 monclient: found mon.noname-c 2024-09-15T03:40:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:40:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.788+0000 7f43f8b3a640 1 -- 172.21.15.153:0/1834061427 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f4098ad0 con 0x7f43f4134080 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.791+0000 7f43f8b3a640 1 -- 172.21.15.153:0/1834061427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f4134080 msgr2=0x7f43f4136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.791+0000 7f43f8b3a640 1 --2- 172.21.15.153:0/1834061427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f4134080 0x7f43f4136460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009fa0 tx=0x7f43ec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1bb1429fba5d28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 1 -- 172.21.15.153:0/1834061427 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec007610 con 0x7f43f4134080 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: dump: 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 1 -- 172.21.15.153:0/1834061427 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ec02d050 con 0x7f43f4134080 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 1 -- 172.21.15.153:0/1834061427 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec0079a0 con 0x7f43f4134080 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 5 monclient: authenticate success, global_id 144235 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 4 set_mon_vals no callback set 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43f2ffd640 10 monclient: dump: 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:36.796 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 10 monclient: shutdown 2024-09-15T03:40:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 1 -- 172.21.15.153:0/1834061427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f4134080 msgr2=0x7f43f4136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.792+0000 7f43fa59d640 1 --2- 172.21.15.153:0/1834061427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f4134080 0x7f43f4136460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009fa0 tx=0x7f43ec009840 comp rx=0 tx=0).stop 2024-09-15T03:40:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 1 -- 172.21.15.153:0/1834061427 shutdown_connections 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 1 --2- 172.21.15.153:0/1834061427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f4134080 0x7f43f4136460 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 1 -- 172.21.15.153:0/1834061427 >> 172.21.15.153:0/1834061427 conn(0x7f43f40672b0 msgr2=0x7f43f40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 1 -- 172.21.15.153:0/1834061427 shutdown_connections 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 1 -- 172.21.15.153:0/1834061427 wait complete. 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 10 monclient: build_initial_monmap 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 10 monclient: monmap: 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.793+0000 7f43fa59d640 1 Processor -- start 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 -- start start 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient: init 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 0x7f43f41280d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): picked mon.noname-c con 0x7f43f40974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f409a8a0 0x7f43f406dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): picked mon.noname-b con 0x7f43f409a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f4134080 0x7f43f406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): picked mon.noname-a con 0x7f43f4134080 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f406ec30 con 0x7f43f4134080 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f412cb30 con 0x7f43f409a8a0 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f412c850 con 0x7f43f40974f0 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43fa59d640 10 monclient(hunting): authenticate will time out at 1434.433563s 2024-09-15T03:40:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f8b3a640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f4134080 msgr2=0x7f43f406dff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f8b3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f4134080 0x7f43f406dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 0x7f43f41280d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 0x7f43f41280d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53162/0 (socket says 172.21.15.153:53162) 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 1 -- 172.21.15.153:0/348233532 learned_addr learned my addr 172.21.15.153:0/348233532 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): get_auth_request con 0x7f43f40974f0 auth_method 0 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f409a8a0 0x7f43f406dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): get_auth_request con 0x7f43f409a8a0 auth_method 0 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.794+0000 7f43f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 10 monclient(hunting): handle_auth_done global_id 124493 payload 824 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f409a8a0 msgr2=0x7f43f406dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f409a8a0 0x7f43f406dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f4134080 msgr2=0x7f43f406dff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f4134080 0x7f43f406dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:40:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 1 monclient: found mon.noname-c 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.795+0000 7f43f3fff640 1 -- 172.21.15.153:0/348233532 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec009ae0 con 0x7f43f40974f0 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43f3fff640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 msgr2=0x7f43f41280d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43f3fff640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 0x7f43f41280d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f43e400b630 tx=0x7f43e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4015070 con 0x7f43f40974f0 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: dump: 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.807 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: _finish_auth 0 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 5 monclient: authenticate success, global_id 124493 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 10 monclient: _renew_subs 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4003d90 con 0x7f43f40974f0 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f412ce10 con 0x7f43f40974f0 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4006380 con 0x7f43f40974f0 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43dbfff640 10 monclient: dump: 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:36.809 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 10 monclient: _renew_subs 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f412bc10 con 0x7f43f40974f0 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.797+0000 7f43f1ffb640 4 set_mon_vals no callback set 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f43e4010440 con 0x7f43f40974f0 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43fa59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43fa59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43fa59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f404bf60 con 0x7f43f40974f0 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43dbfff640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:36.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f43e404c0d0 con 0x7f43f40974f0 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43f37fe640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.798+0000 7f43f37fe640 10 monclient: get_auth_request con 0x7f43e00386f0 auth_method 0 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.799+0000 7f43f37fe640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 msgr2=0x7f43e003aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.799+0000 7f43f37fe640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 msgr2=0x7f43e003aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.799+0000 7f43f37fe640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.799+0000 7f43f37fe640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.803+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f43e4018a70 con 0x7f43f40974f0 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.803+0000 7f43dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:36.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:36.803+0000 7f43dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:37.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.000+0000 7f43f37fe640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:37.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.000+0000 7f43f37fe640 10 monclient: get_auth_request con 0x7f43e00386f0 auth_method 0 2024-09-15T03:40:37.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.003+0000 7f43f37fe640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f43e8007fd0 tx=0x7f43e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:37.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43fa59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:40:37.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43fa59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:37.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43fa59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:37.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f43f40638d0 con 0x7f43f40974f0 2024-09-15T03:40:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43dbfff640 1 -- 172.21.15.153:0/348233532 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f43e4018a70 con 0x7f43f40974f0 2024-09-15T03:40:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.017+0000 7f43dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 msgr2=0x7f43e003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f43e8007fd0 tx=0x7f43e8009040 comp rx=0 tx=0).stop 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 10 monclient: shutdown 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 msgr2=0x7f43f41280d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 0x7f43f41280d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f43e400b630 tx=0x7f43e40091a0 comp rx=0 tx=0).stop 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 shutdown_connections 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f43e00386f0 0x7f43e003aba0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f4134080 0x7f43f406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43f409a8a0 0x7f43f406dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:37.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 --2- 172.21.15.153:0/348233532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f40974f0 0x7f43f41280d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:37.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.019+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 >> 172.21.15.153:0/348233532 conn(0x7f43f40672b0 msgr2=0x7f43f4124750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:37.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.020+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 shutdown_connections 2024-09-15T03:40:37.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:37.020+0000 7f43fa59d640 1 -- 172.21.15.153:0/348233532 wait complete. 2024-09-15T03:40:40.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:40:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.136+0000 7f945e75d640 10 monclient: get_monmap_and_config 2024-09-15T03:40:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.136+0000 7f945e75d640 10 monclient: build_initial_monmap 2024-09-15T03:40:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.136+0000 7f945e75d640 10 monclient: monmap: 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:40.137617+0000 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:40.137617+0000 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:40.138 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.137+0000 7f945e75d640 1 Processor -- start 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.137+0000 7f945e75d640 1 -- start start 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.137+0000 7f945e75d640 10 monclient: init 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): picked mon.noname-b con 0x7f9458131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): picked mon.noname-c con 0x7f945809e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f945809f220 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:40.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945809faf0 con 0x7f945809f220 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945809fd30 con 0x7f945809e910 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94580a1110 con 0x7f9458131ab0 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945e75d640 10 monclient(hunting): authenticate will time out at 1437.777430s 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f945ccfa640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f945809f220 legacy=0x7f9458068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53170/0 (socket says 172.21.15.153:53170) 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 learned_addr learned my addr 172.21.15.153:0/2993251274 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 10 monclient(hunting): get_auth_request con 0x7f9458131ab0 auth_method 0 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:40.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 10 monclient(hunting): get_auth_request con 0x7f945809e910 auth_method 0 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:40:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.138+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:40.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 >> v1:172.21.15.153:6789/0 conn(0x7f945809f220 legacy=0x7f9458068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:40.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:40.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 10 monclient(hunting): get_auth_request con 0x7f9458131ab0 auth_method 0 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:40.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 10 monclient(hunting): get_auth_request con 0x7f945809e910 auth_method 0 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.339+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:40:40.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 >> v1:172.21.15.153:6789/0 conn(0x7f945809f220 legacy=0x7f9458068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:40.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:40.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 10 monclient(hunting): get_auth_request con 0x7f9458131ab0 auth_method 0 2024-09-15T03:40:40.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 10 monclient(hunting): get_auth_request con 0x7f945809e910 auth_method 0 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:40.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:40.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:40.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:40:40.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:40.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:40.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:40.740+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:40:41.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 >> v1:172.21.15.153:6789/0 conn(0x7f945809f220 legacy=0x7f9458068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:41.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:41.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 10 monclient(hunting): get_auth_request con 0x7f9458131ab0 auth_method 0 2024-09-15T03:40:41.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:41.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:41.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 10 monclient(hunting): get_auth_request con 0x7f945809e910 auth_method 0 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:40:41.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:41.541+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:40:43.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient(hunting): tick 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 msgr2=0x7f9458131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9458131ab0 0x7f9458131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 msgr2=0x7f945809ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.181:3300/0 conn(0x7f945809e910 0x7f945809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 -- 172.21.15.153:0/2993251274 >> v1:172.21.15.153:6789/0 conn(0x7f945809f220 legacy=0x7f9458068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient: _add_conns ranks=[3,1] 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 0x7f9438005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient(hunting): picked mon.noname-a con 0x7f9438003680 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9438006100 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:43.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 -- 172.21.15.153:0/2993251274 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580a13f0 con 0x7f9438006100 2024-09-15T03:40:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 1 -- 172.21.15.153:0/2993251274 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945809fd30 con 0x7f9438003680 2024-09-15T03:40:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f94567fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 msgr2=0x7f9438005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.138+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 0x7f9438005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:43.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.140+0000 7f944ffff640 1 -- 172.21.15.153:0/2993251274 >> v1:172.21.15.181:6789/0 conn(0x7f9438006100 legacy=0x7f94380085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.339+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 msgr2=0x7f9438005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:43.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.339+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 0x7f9438005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:40:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.740+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 msgr2=0x7f9438005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:43.740+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 0x7f9438005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:40:44.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.541+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 msgr2=0x7f9438005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:44.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.541+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 0x7f9438005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:40:44.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.550+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954227498 0 0) 0x7f94580a13f0 con 0x7f9438006100 2024-09-15T03:40:44.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.550+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.550+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.550+0000 7f94577fe640 10 monclient(hunting): my global_id is 144241 2024-09-15T03:40:44.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.550+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94440034e0 con 0x7f9438006100 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1999973325 0 0) 0x7f94440034e0 con 0x7f9438006100 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 msgr2=0x7f9438005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 --2- 172.21.15.153:0/2993251274 >> v2:172.21.15.153:3300/0 conn(0x7f9438003680 0x7f9438005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: _finish_hunting 0 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:44.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945809faf0 con 0x7f9438006100 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: _finish_auth 0 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f944000d480 con 0x7f9438006100 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: dump: 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.556 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 >> v1:172.21.15.181:6789/0 conn(0x7f9438006100 legacy=0x7f94380085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 reap_dead start 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94440039d0 0x7f9444003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): picked mon.a con 0x7f94440039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9444004780 0x7f9444004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): picked mon.b con 0x7f9444004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f944400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): picked mon.c con 0x7f9444005230 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945809fd30 con 0x7f94440039d0 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944400c1f0 con 0x7f9444004780 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944400c3b0 con 0x7f9444005230 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f9457fff640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94440039d0 msgr2=0x7f9444003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f94577fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f9457fff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94440039d0 0x7f9444003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f944400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): get_auth_request con 0x7f9444005230 auth_method 0 2024-09-15T03:40:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9444004780 0x7f9444004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): get_auth_request con 0x7f9444004780 auth_method 0 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient(hunting): handle_auth_done global_id 144241 payload 293 2024-09-15T03:40:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9444004780 msgr2=0x7f9444004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9444004780 0x7f9444004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94440039d0 msgr2=0x7f9444003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94440039d0 0x7f9444003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient: _finish_hunting 0 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 monclient: found mon.c 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94440034e0 con 0x7f9444005230 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 10 monclient: _finish_auth 0 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945ccfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945e75d640 5 monclient: authenticate success, global_id 144241 2024-09-15T03:40:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f945e75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.551+0000 7f944ffff640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9444004780 0x7f9444004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:40:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.554+0000 7f945ccfa640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 msgr2=0x7f944400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.554+0000 7f945ccfa640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f944400b9d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f94580a5da0 tx=0x7f9450007530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec8c6a883b2fd9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945002aaf0 con 0x7f9444005230 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: dump: 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945002ac90 con 0x7f9444005230 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 4 set_mon_vals no callback set 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 1 -- 172.21.15.153:0/2993251274 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94500336c0 con 0x7f9444005230 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f94577fe640 10 monclient: dump: 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 10 monclient: shutdown 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 -- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 msgr2=0x7f944400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f944400b9d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f94580a5da0 tx=0x7f9450007530 comp rx=0 tx=0).stop 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 -- 172.21.15.153:0/2993251274 shutdown_connections 2024-09-15T03:40:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f944400b9d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9444004780 0x7f9444004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 --2- 172.21.15.153:0/2993251274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94440039d0 0x7f9444003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 -- 172.21.15.153:0/2993251274 >> 172.21.15.153:0/2993251274 conn(0x7f9458067d30 msgr2=0x7f94580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.555+0000 7f945e75d640 1 -- 172.21.15.153:0/2993251274 shutdown_connections 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.556+0000 7f945e75d640 1 -- 172.21.15.153:0/2993251274 wait complete. 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.556+0000 7f945e75d640 10 monclient: build_initial_monmap 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.556+0000 7f945e75d640 10 monclient: monmap: 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.556+0000 7f945e75d640 1 Processor -- start 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 -- start start 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient: init 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94440039d0 0x7f945811dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): picked mon.noname-b con 0x7f94440039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9444004780 0x7f945811e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): picked mon.noname-a con 0x7f9444004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f945811e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): picked mon.noname-c con 0x7f9444005230 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580a0620 con 0x7f9444004780 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580a43c0 con 0x7f94440039d0 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9458120570 con 0x7f9444005230 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945e75d640 10 monclient(hunting): authenticate will time out at 1442.197600s 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f944ffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9444004780 msgr2=0x7f945811e3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f944ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9444004780 0x7f945811e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f945811e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f945811e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53202/0 (socket says 172.21.15.153:53202) 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 1 -- 172.21.15.153:0/296196887 learned_addr learned my addr 172.21.15.153:0/296196887 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): get_auth_request con 0x7f9444005230 auth_method 0 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f945ccfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f9457fff640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94440039d0 0x7f945811dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f9457fff640 10 monclient(hunting): get_auth_request con 0x7f94440039d0 auth_method 0 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f9457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f9457fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.557+0000 7f9457fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 10 monclient(hunting): handle_auth_done global_id 124496 payload 824 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 1 -- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94440039d0 msgr2=0x7f945811dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94440039d0 0x7f945811dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 1 -- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9444004780 msgr2=0x7f945811e3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9444004780 0x7f945811e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 10 monclient: _finish_hunting 0 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 1 monclient: found mon.noname-c 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f945ccfa640 1 -- 172.21.15.153:0/296196887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945811f900 con 0x7f9444005230 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.558+0000 7f9457fff640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94440039d0 0x7f945811dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945ccfa640 1 -- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 msgr2=0x7f945811e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945ccfa640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f945811e920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9450001220 tx=0x7f9450033000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945003f430 con 0x7f9444005230 2024-09-15T03:40:44.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: dump: 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: _finish_auth 0 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945003f590 con 0x7f9444005230 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 5 monclient: authenticate success, global_id 124496 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 10 monclient: _renew_subs 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9458121430 con 0x7f9444005230 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945003f930 con 0x7f9444005230 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9454ff9640 10 monclient: dump: 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 10 monclient: _renew_subs 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94581315f0 con 0x7f9444005230 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.560+0000 7f9456ffd640 4 set_mon_vals no callback set 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9450030070 con 0x7f9444005230 2024-09-15T03:40:44.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f9454ff9640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9438038ce0 0x7f943803b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f945e75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f945e75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f945e75d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f945006c6d0 con 0x7f9444005230 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.561+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945804bf60 con 0x7f9444005230 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.562+0000 7f9457fff640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9438038ce0 0x7f943803b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.562+0000 7f9457fff640 10 monclient: get_auth_request con 0x7f9438038ce0 auth_method 0 2024-09-15T03:40:44.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.564+0000 7f9457fff640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9438038ce0 0x7f943803b190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9448006cb0 tx=0x7f9448008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.566+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f945003e3e0 con 0x7f9444005230 2024-09-15T03:40:44.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.566+0000 7f9454ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:44.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.566+0000 7f9454ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:44.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f945e75d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:40:44.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f945e75d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:44.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f945e75d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9458121840 con 0x7f9444005230 2024-09-15T03:40:44.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f9454ff9640 1 -- 172.21.15.153:0/296196887 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9450046350 con 0x7f9444005230 2024-09-15T03:40:44.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f9454ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:40:44.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.789+0000 7f9454ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:44.792 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":126,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9438038ce0 msgr2=0x7f943803b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9438038ce0 0x7f943803b190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9448006cb0 tx=0x7f9448008040 comp rx=0 tx=0).stop 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 10 monclient: shutdown 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 msgr2=0x7f945811e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f945811e920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9450001220 tx=0x7f9450033000 comp rx=0 tx=0).stop 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 shutdown_connections 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9438038ce0 0x7f943803b190 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9444005230 0x7f945811e920 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9444004780 0x7f945811e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 --2- 172.21.15.153:0/296196887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94440039d0 0x7f945811dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.791+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 >> 172.21.15.153:0/296196887 conn(0x7f9458067d30 msgr2=0x7f945809b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:44.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.792+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 shutdown_connections 2024-09-15T03:40:44.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.792+0000 7f945e75d640 1 -- 172.21.15.153:0/296196887 wait complete. 2024-09-15T03:40:44.804 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:40:44.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:40:44.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:40:44.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.897+0000 7f367255d640 10 monclient: get_monmap_and_config 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.897+0000 7f367255d640 10 monclient: build_initial_monmap 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.897+0000 7f367255d640 10 monclient: monmap: 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:44.898800+0000 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:44.898800+0000 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:44.899 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:44.900 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:44.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.897+0000 7f367255d640 1 Processor -- start 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.897+0000 7f367255d640 1 -- start start 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.897+0000 7f367255d640 10 monclient: init 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f366c1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f366c09ea50 0x7f366c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): picked mon.noname-a con 0x7f366c09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): picked mon.noname-b con 0x7f366c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c132640 con 0x7f366c1318f0 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c131cc0 con 0x7f366c09ea50 2024-09-15T03:40:44.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c09fbf0 con 0x7f366c136c20 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f366bfff640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f366c1318f0 legacy=0x7f366c131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f367255d640 10 monclient(hunting): authenticate will time out at 1442.538505s 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f366b7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f366c09ea50 msgr2=0x7f366c09ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f366b7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f366c09ea50 0x7f366c09ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53206/0 (socket says 172.21.15.153:53206) 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 -- 172.21.15.153:0/819851779 learned_addr learned my addr 172.21.15.153:0/819851779 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): get_auth_request con 0x7f366c136c20 auth_method 0 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient(hunting): handle_auth_done global_id 124499 payload 274 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 -- 172.21.15.153:0/819851779 >> v2:172.21.15.153:3300/0 conn(0x7f366c09ea50 msgr2=0x7f366c09ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 --2- 172.21.15.153:0/819851779 >> v2:172.21.15.153:3300/0 conn(0x7f366c09ea50 0x7f366c09ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 -- 172.21.15.153:0/819851779 >> v1:172.21.15.153:6789/0 conn(0x7f366c1318f0 legacy=0x7f366c131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient: _finish_hunting 0 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 monclient: found mon.noname-b 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.898+0000 7f3670afa640 1 -- 172.21.15.153:0/819851779 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c09fe30 con 0x7f366c136c20 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f3670afa640 1 -- 172.21.15.153:0/819851779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 msgr2=0x7f366c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f3670afa640 1 --2- 172.21.15.153:0/819851779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c139000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3664009f20 tx=0x7f3664009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a82e65485b524f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 1 -- 172.21.15.153:0/819851779 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664007610 con 0x7f366c136c20 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: dump: 2024-09-15T03:40:44.910 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: _finish_auth 0 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 1 -- 172.21.15.153:0/819851779 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366402d050 con 0x7f366c136c20 2024-09-15T03:40:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f367255d640 5 monclient: authenticate success, global_id 124499 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f367255d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f367255d640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 1 -- 172.21.15.153:0/819851779 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640079a0 con 0x7f366c136c20 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f367255d640 4 set_mon_vals no callback set 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.901+0000 7f366affd640 10 monclient: dump: 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.912 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 10 monclient: shutdown 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 -- 172.21.15.153:0/819851779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 msgr2=0x7f366c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 --2- 172.21.15.153:0/819851779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c139000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3664009f20 tx=0x7f3664009840 comp rx=0 tx=0).stop 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 -- 172.21.15.153:0/819851779 shutdown_connections 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 --2- 172.21.15.153:0/819851779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c139000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 --2- 172.21.15.153:0/819851779 >> v2:172.21.15.153:3300/0 conn(0x7f366c09ea50 0x7f366c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 -- 172.21.15.153:0/819851779 >> 172.21.15.153:0/819851779 conn(0x7f366c067110 msgr2=0x7f366c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 -- 172.21.15.153:0/819851779 shutdown_connections 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.902+0000 7f367255d640 1 -- 172.21.15.153:0/819851779 wait complete. 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 10 monclient: build_initial_monmap 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 10 monclient: monmap: 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:44.914 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 1 Processor -- start 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 1 -- start start 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 10 monclient: init 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c09ea50 0x7f366c1412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.903+0000 7f367255d640 10 monclient(hunting): picked mon.noname-a con 0x7f366c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f366c1318f0 0x7f366c141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): picked mon.noname-b con 0x7f366c1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c099740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f366bfff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c09ea50 msgr2=0x7f366c1412c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): picked mon.noname-c con 0x7f366c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c09bf10 con 0x7f366c09ea50 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f366bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c09ea50 0x7f366c1412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c09dd70 con 0x7f366c1318f0 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c11cc10 con 0x7f366c136c20 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f367255d640 10 monclient(hunting): authenticate will time out at 1442.544082s 2024-09-15T03:40:44.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c099740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c099740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53218/0 (socket says 172.21.15.153:53218) 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 learned_addr learned my addr 172.21.15.153:0/96618871 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): get_auth_request con 0x7f366c136c20 auth_method 0 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f366b7fe640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f366c1318f0 0x7f366c141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient(hunting): handle_auth_done global_id 124502 payload 824 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f366c1318f0 msgr2=0x7f366c141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f366c1318f0 0x7f366c141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c09ea50 msgr2=0x7f366c1412c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c09ea50 0x7f366c1412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient: _finish_hunting 0 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 monclient: found mon.noname-c 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664009ae0 con 0x7f366c136c20 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f366b7fe640 10 monclient: get_auth_request con 0x7f366c1318f0 auth_method 0 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.904+0000 7f366b7fe640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f366c1318f0 0x7f366c141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 msgr2=0x7f366c099740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:44.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3670afa640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c099740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3664009ab0 tx=0x7f3664036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664031cd0 con 0x7f366c136c20 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3670afa640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c099740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3664009ab0 tx=0x7f3664036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f366403c670 mon_map magic: 0 v1 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 10 monclient: dump: 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 10 monclient: _finish_auth 0 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 5 monclient: authenticate success, global_id 124502 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664031e30 con 0x7f366c136c20 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f3653fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 10 monclient: _renew_subs 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c11c900 con 0x7f366c136c20 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f366a7fc640 4 set_mon_vals no callback set 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 10 monclient: _renew_subs 2024-09-15T03:40:44.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.906+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c141d40 con 0x7f366c136c20 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366403c670 con 0x7f366c136c20 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f3653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f367255d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f3653fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f3653fff640 10 monclient: dump: 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f366403c9e0 con 0x7f366c136c20 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f367255d640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 0x7f366c09a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c065e00 con 0x7f366c11db30 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 0x7f366c09a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient: get_auth_request con 0x7f366c11db30 auth_method 0 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f3653fff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f36580386f0 0x7f365803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.907+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.908+0000 7f366bfff640 10 monclient(hunting): handle_auth_done global_id 124505 payload 824 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.908+0000 7f366bfff640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f366c09b300 con 0x7f366c11db30 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.908+0000 7f366b7fe640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f36580386f0 0x7f365803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.908+0000 7f366b7fe640 10 monclient: get_auth_request con 0x7f36580386f0 auth_method 0 2024-09-15T03:40:44.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f366bfff640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 msgr2=0x7f366c09a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f366bfff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 0x7f366c09a440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f365c00b480 tx=0x7f365c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c0050f0 con 0x7f366c11db30 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 10 monclient: dump: 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f365c005250 con 0x7f366c11db30 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 msgr2=0x7f366c09a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 0x7f366c09a440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f365c00b480 tx=0x7f365c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 msgr2=0x7f366c09a440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f3653fff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c11db30 0x7f366c09a440 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f365c00b480 tx=0x7f365c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.910+0000 7f366b7fe640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f36580386f0 0x7f365803aba0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3660003850 tx=0x7f3660008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f367255d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f367255d640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 0x7f366c029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f367255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c04bf60 con 0x7f366c0298c0 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 0x7f366c029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient: get_auth_request con 0x7f366c0298c0 auth_method 0 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 10 monclient(hunting): handle_auth_done global_id 124508 payload 824 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.916+0000 7f366bfff640 1 -- 172.21.15.153:0/96618871 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f366c135eb0 con 0x7f366c0298c0 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.918+0000 7f366bfff640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 msgr2=0x7f366c029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:44.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.918+0000 7f366bfff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 0x7f366c029cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f365c00b480 tx=0x7f365c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.918+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c005930 con 0x7f366c0298c0 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.918+0000 7f3653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.918+0000 7f3653fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.918+0000 7f3653fff640 10 monclient: dump: 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:44.929 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f365c01b650 con 0x7f366c0298c0 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 msgr2=0x7f366c029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 0x7f366c029cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f365c00b480 tx=0x7f365c015000 comp rx=0 tx=0).stop 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 msgr2=0x7f366c029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c0298c0 0x7f366c029cc0 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f365c00b480 tx=0x7f365c015000 comp rx=0 tx=0).stop 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:44.930 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 126, 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:44.931 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:44.932 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:44.933 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.935 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:44.936 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:44.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:44.938 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:44.939 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:44.940 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.941 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:44.942 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:44.943 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f36580386f0 msgr2=0x7f365803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f36580386f0 0x7f365803aba0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3660003850 tx=0x7f3660008040 comp rx=0 tx=0).stop 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 10 monclient: shutdown 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 msgr2=0x7f366c099740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 --2- 172.21.15.153:0/96618871 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f366c136c20 0x7f366c099740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3664009ab0 tx=0x7f3664036000 comp rx=0 tx=0).stop 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 reap_dead start 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3670afa640 1 -- 172.21.15.153:0/96618871 reap_dead start 2024-09-15T03:40:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f3653fff640 1 -- 172.21.15.153:0/96618871 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7f366406d090 con 0x7f366c136c20 2024-09-15T03:40:44.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 shutdown_connections 2024-09-15T03:40:44.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 >> 172.21.15.153:0/96618871 conn(0x7f366c067110 msgr2=0x7f366c0a40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:44.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 shutdown_connections 2024-09-15T03:40:44.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:44.919+0000 7f367255d640 1 -- 172.21.15.153:0/96618871 wait complete. 2024-09-15T03:40:44.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:40:44.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:40:44.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:40:45.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.038+0000 7fc8f7c3f640 10 monclient: get_monmap_and_config 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.038+0000 7fc8f7c3f640 10 monclient: build_initial_monmap 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.038+0000 7fc8f7c3f640 10 monclient: monmap: 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:40:45.039827+0000 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:40:45.039827+0000 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:40:45.040 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:40:45.041 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:40:45.041 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.038+0000 7fc8f7c3f640 1 Processor -- start 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.038+0000 7fc8f7c3f640 1 -- start start 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.038+0000 7fc8f7c3f640 10 monclient: init 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8f009ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8f0136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:40:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0131cc0 con 0x7fc8f009ea50 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f009f530 con 0x7fc8f0136c20 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f009fbf0 con 0x7fc8f01318f0 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f7c3f640 10 monclient(hunting): authenticate will time out at 1442.679391s 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f51da640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fc8f009ea50 legacy=0x7fc8f009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53246/0 (socket says 172.21.15.153:53246) 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 -- 172.21.15.153:0/3315321820 learned_addr learned my addr 172.21.15.153:0/3315321820 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): get_auth_request con 0x7fc8f01318f0 auth_method 0 2024-09-15T03:40:45.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient(hunting): handle_auth_done global_id 124511 payload 274 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 -- 172.21.15.153:0/3315321820 >> v1:172.21.15.181:6789/0 conn(0x7fc8f0136c20 legacy=0x7fc8f0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 -- 172.21.15.153:0/3315321820 >> v1:172.21.15.153:6789/0 conn(0x7fc8f009ea50 legacy=0x7fc8f009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient: _finish_hunting 0 2024-09-15T03:40:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 monclient: found mon.noname-b 2024-09-15T03:40:45.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:45.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:40:45.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.039+0000 7fc8f59db640 1 -- 172.21.15.153:0/3315321820 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f009fe30 con 0x7fc8f01318f0 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f59db640 1 -- 172.21.15.153:0/3315321820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 msgr2=0x7fc8f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f59db640 1 --2- 172.21.15.153:0/3315321820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0131f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009940 tx=0x7fc8ec0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45bf611518011d36 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 1 -- 172.21.15.153:0/3315321820 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec032440 con 0x7fc8f01318f0 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: dump: 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: _finish_auth 0 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 1 -- 172.21.15.153:0/3315321820 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec0325a0 con 0x7fc8f01318f0 2024-09-15T03:40:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f7c3f640 5 monclient: authenticate success, global_id 124511 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f7c3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f7c3f640 10 monclient: get_monmap_and_config success 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 1 -- 172.21.15.153:0/3315321820 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec032940 con 0x7fc8f01318f0 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f7c3f640 4 set_mon_vals no callback set 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.042+0000 7fc8f49d9640 10 monclient: dump: 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.054 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 10 monclient: shutdown 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/3315321820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 msgr2=0x7fc8f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 --2- 172.21.15.153:0/3315321820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0131f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009940 tx=0x7fc8ec0091d0 comp rx=0 tx=0).stop 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/3315321820 shutdown_connections 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 --2- 172.21.15.153:0/3315321820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0131f30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/3315321820 >> 172.21.15.153:0/3315321820 conn(0x7fc8f0067110 msgr2=0x7fc8f0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:45.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/3315321820 shutdown_connections 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/3315321820 wait complete. 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 10 monclient: build_initial_monmap 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.043+0000 7fc8f7c3f640 10 monclient: monmap: 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:40:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 Processor -- start 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 -- start start 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient: init 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f009ea50 0x7fc8f009b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f0136c20 0x7fc8f0096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f0136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f009acf0 con 0x7fc8f009ea50 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f009bf10 con 0x7fc8f0136c20 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f013a4f0 con 0x7fc8f01318f0 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f7c3f640 10 monclient(hunting): authenticate will time out at 1442.684747s 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f59db640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f009ea50 msgr2=0x7fc8f009b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f59db640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f009ea50 0x7fc8f009b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:40:45.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53254/0 (socket says 172.21.15.153:53254) 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 1 -- 172.21.15.153:0/1962929093 learned_addr learned my addr 172.21.15.153:0/1962929093 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): get_auth_request con 0x7fc8f01318f0 auth_method 0 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.044+0000 7fc8f51da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 10 monclient(hunting): handle_auth_done global_id 124514 payload 824 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f0136c20 msgr2=0x7fc8f0096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f0136c20 0x7fc8f0096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f009ea50 msgr2=0x7fc8f009b670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:40:45.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f009ea50 0x7fc8f009b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 10 monclient: _finish_hunting 0 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 1 monclient: found mon.noname-c 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.045+0000 7fc8f51da640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009450 con 0x7fc8f01318f0 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f51da640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 msgr2=0x7fc8f0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f51da640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0096a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b600 tx=0x7fc8e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0015070 con 0x7fc8f01318f0 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: dump: 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:45.060 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 5 monclient: authenticate success, global_id 124514 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0003d90 con 0x7fc8f01318f0 2024-09-15T03:40:45.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 10 monclient: _renew_subs 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f0097cd0 con 0x7fc8f01318f0 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0006380 con 0x7fc8f01318f0 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e5ffb640 10 monclient: dump: 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8e7fff640 4 set_mon_vals no callback set 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 10 monclient: _renew_subs 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.047+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f0098840 con 0x7fc8f01318f0 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f7c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f7c3f640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 0x7fc8f00a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f7c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0065e00 con 0x7fc8f00974e0 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc8e0010440 con 0x7fc8f01318f0 2024-09-15T03:40:45.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8e5ffb640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d40386f0 0x7fc8d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc8e004c190 con 0x7fc8f01318f0 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 0x7fc8f00a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient: get_auth_request con 0x7fc8f00974e0 auth_method 0 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f61dc640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d40386f0 0x7fc8d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f61dc640 10 monclient: get_auth_request con 0x7fc8d40386f0 auth_method 0 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.048+0000 7fc8f59db640 10 monclient(hunting): handle_auth_done global_id 144247 payload 824 2024-09-15T03:40:45.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.049+0000 7fc8f59db640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8f0098ac0 con 0x7fc8f00974e0 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8f59db640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 msgr2=0x7fc8f00a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8f59db640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 0x7fc8f00a3870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec034000 tx=0x7fc8ec032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8f61dc640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d40386f0 0x7fc8d403aba0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8007fd0 tx=0x7fc8e8007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec032440 con 0x7fc8f00974e0 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 10 monclient: dump: 2024-09-15T03:40:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc8ec032710 con 0x7fc8f00974e0 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:40:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 msgr2=0x7fc8f00a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 0x7fc8f00a3870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec034000 tx=0x7fc8ec032000 comp rx=0 tx=0).stop 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 msgr2=0x7fc8f00a3870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.051+0000 7fc8e5ffb640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00974e0 0x7fc8f00a3870 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec034000 tx=0x7fc8ec032000 comp rx=0 tx=0).stop 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f7c3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f7c3f640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 0x7fc8f00a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f7c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00978b0 con 0x7fc8f00a3db0 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f59db640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 0x7fc8f00a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f59db640 10 monclient: get_auth_request con 0x7fc8f00a3db0 auth_method 0 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f59db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f59db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.057+0000 7fc8f59db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:40:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.058+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.058+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.058+0000 7fc8f59db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.058+0000 7fc8f59db640 10 monclient(hunting): handle_auth_done global_id 144250 payload 824 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.058+0000 7fc8f59db640 1 -- 172.21.15.153:0/1962929093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc8f009a590 con 0x7fc8f00a3db0 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8f59db640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 msgr2=0x7fc8f00a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8f59db640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 0x7fc8f00a41b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009890 tx=0x7fc8ec03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec045840 con 0x7fc8f00a3db0 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 10 monclient: dump: 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:40:45.072 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fc8ec0459a0 con 0x7fc8f00a3db0 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 msgr2=0x7fc8f00a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 0x7fc8f00a41b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009890 tx=0x7fc8ec03d000 comp rx=0 tx=0).stop 2024-09-15T03:40:45.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 msgr2=0x7fc8f00a41b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.060+0000 7fc8e5ffb640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f00a3db0 0x7fc8f00a41b0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009890 tx=0x7fc8ec03d000 comp rx=0 tx=0).stop 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 126, 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:40:45.074 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:45.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:45.076 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:40:45.077 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:40:45.078 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:40:45.079 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:45.080 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:45.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:40:45.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:45.084 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:40:45.085 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:40:45.086 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:40:45.086 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:40:45.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:40:45.086 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:40:45.086 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:40:45.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d40386f0 msgr2=0x7fc8d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:45.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc8d40386f0 0x7fc8d403aba0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8007fd0 tx=0x7fc8e8007e10 comp rx=0 tx=0).stop 2024-09-15T03:40:45.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 10 monclient: shutdown 2024-09-15T03:40:45.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 msgr2=0x7fc8f0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 --2- 172.21.15.153:0/1962929093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8f01318f0 0x7fc8f0096a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b600 tx=0x7fc8e00091a0 comp rx=0 tx=0).stop 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f61dc640 1 -- 172.21.15.153:0/1962929093 reap_dead start 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f61dc640 1 -- 172.21.15.153:0/1962929093 reap_dead start 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 shutdown_connections 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 >> 172.21.15.153:0/1962929093 conn(0x7fc8f0067110 msgr2=0x7fc8f009e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 shutdown_connections 2024-09-15T03:40:45.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:40:45.061+0000 7fc8f7c3f640 1 -- 172.21.15.153:0/1962929093 wait complete. 2024-09-15T03:40:45.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:40:45.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:41:05.089 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:41:05.090 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:41:05.090 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:41:05.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:41:05.093 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:41:05.094 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:41:05.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.198+0000 7f01e5820640 10 monclient: get_monmap_and_config 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.198+0000 7f01e5820640 10 monclient: build_initial_monmap 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.198+0000 7f01e5820640 10 monclient: monmap: 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:05.199220+0000 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:05.199220+0000 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:05.202 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 Processor -- start 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 -- start start 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient: init 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01e01318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 0x7f01e009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:05.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): picked mon.noname-a con 0x7f01e009ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): picked mon.noname-b con 0x7f01e0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e0132640 con 0x7f01e01318f0 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e0131cc0 con 0x7f01e009ea50 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e009fbf0 con 0x7f01e0136c20 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01e5820640 10 monclient(hunting): authenticate will time out at 1462.838622s 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 0x7f01e009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01deffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f01e01318f0 0x7f01e0131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41334/0 (socket says 172.21.15.153:41334) 2024-09-15T03:41:05.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01deffd640 1 -- 172.21.15.153:0/1439517976 learned_addr learned my addr 172.21.15.153:0/1439517976 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 10 monclient(hunting): get_auth_request con 0x7f01e009ea50 auth_method 0 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): get_auth_request con 0x7f01e0136c20 auth_method 0 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 1 -- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 msgr2=0x7f01e009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:41:05.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 1 -- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 msgr2=0x7f01e009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 1 --2- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 0x7f01e009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.199+0000 7f01de7fc640 1 --2- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 0x7f01e009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 10 monclient(hunting): handle_auth_done global_id 124517 payload 274 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 1 -- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 msgr2=0x7f01e009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 1 --2- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 0x7f01e009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 1 -- 172.21.15.153:0/1439517976 >> v1:172.21.15.153:6789/0 conn(0x7f01e01318f0 legacy=0x7f01e0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:41:05.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 1 monclient: found mon.noname-b 2024-09-15T03:41:05.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:05.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:05.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.200+0000 7f01df7fe640 1 -- 172.21.15.153:0/1439517976 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e009fe30 con 0x7f01e0136c20 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.202+0000 7f01df7fe640 1 -- 172.21.15.153:0/1439517976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 msgr2=0x7f01e0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.202+0000 7f01df7fe640 1 --2- 172.21.15.153:0/1439517976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0139000 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009e60 tx=0x7f01d8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8af406850b3384 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 1 -- 172.21.15.153:0/1439517976 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8007760 con 0x7f01e0136c20 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: dump: 2024-09-15T03:41:05.214 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:05.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:05.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.215 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.215 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:05.215 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:05.216 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:05.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:05.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:05.216 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:05.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: _finish_auth 0 2024-09-15T03:41:05.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:05.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 5 monclient: authenticate success, global_id 124517 2024-09-15T03:41:05.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 1 -- 172.21.15.153:0/1439517976 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d802d050 con 0x7f01e0136c20 2024-09-15T03:41:05.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:05.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:05.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 4 set_mon_vals no callback set 2024-09-15T03:41:05.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 1 -- 172.21.15.153:0/1439517976 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8007af0 con 0x7f01e0136c20 2024-09-15T03:41:05.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:05.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:05.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01ddffb640 10 monclient: dump: 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:05.219 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:05.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:05.220 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:05.220 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:05.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 10 monclient: shutdown 2024-09-15T03:41:05.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:05.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 -- 172.21.15.153:0/1439517976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 msgr2=0x7f01e0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:05.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 --2- 172.21.15.153:0/1439517976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0139000 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009e60 tx=0x7f01d8009780 comp rx=0 tx=0).stop 2024-09-15T03:41:05.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 -- 172.21.15.153:0/1439517976 shutdown_connections 2024-09-15T03:41:05.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 --2- 172.21.15.153:0/1439517976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0139000 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 --2- 172.21.15.153:0/1439517976 >> v2:172.21.15.153:3300/0 conn(0x7f01e009ea50 0x7f01e009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 -- 172.21.15.153:0/1439517976 >> 172.21.15.153:0/1439517976 conn(0x7f01e0067110 msgr2=0x7f01e0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:05.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 -- 172.21.15.153:0/1439517976 shutdown_connections 2024-09-15T03:41:05.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.203+0000 7f01e5820640 1 -- 172.21.15.153:0/1439517976 wait complete. 2024-09-15T03:41:05.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.204+0000 7f01e5820640 10 monclient: build_initial_monmap 2024-09-15T03:41:05.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.204+0000 7f01e5820640 10 monclient: monmap: 2024-09-15T03:41:05.222 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:05.222 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:05.222 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.204+0000 7f01e5820640 1 Processor -- start 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.204+0000 7f01e5820640 1 -- start start 2024-09-15T03:41:05.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.204+0000 7f01e5820640 10 monclient: init 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e009ea50 0x7f01e011daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): picked mon.noname-b con 0x7f01e009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): picked mon.noname-a con 0x7f01e01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): picked mon.noname-c con 0x7f01e0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e009dc00 con 0x7f01e01318f0 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:05.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e009bf10 con 0x7f01e009ea50 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e009b2a0 con 0x7f01e0136c20 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01e5820640 10 monclient(hunting): authenticate will time out at 1462.844168s 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38910/0 (socket says 172.21.15.153:38910) 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 -- 172.21.15.153:0/4189751211 learned_addr learned my addr 172.21.15.153:0/4189751211 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 10 monclient(hunting): get_auth_request con 0x7f01e01318f0 auth_method 0 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0130c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36384/0 (socket says 172.21.15.153:36384) 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:05.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): get_auth_request con 0x7f01e0136c20 auth_method 0 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:05.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 msgr2=0x7f01e01306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 msgr2=0x7f01e01306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01de7fc640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01deffd640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e009ea50 0x7f01e011daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01deffd640 10 monclient(hunting): get_auth_request con 0x7f01e009ea50 auth_method 0 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:05.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01deffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient(hunting): handle_auth_done global_id 124520 payload 824 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e009ea50 msgr2=0x7f01e011daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e009ea50 0x7f01e011daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 msgr2=0x7f01e01306d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 monclient: found mon.noname-c 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01df7fe640 1 -- 172.21.15.153:0/4189751211 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8009a20 con 0x7f01e0136c20 2024-09-15T03:41:05.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.205+0000 7f01deffd640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e009ea50 0x7f01e011daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01df7fe640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 msgr2=0x7f01e0130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01df7fe640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0130c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f01e009fd30 tx=0x7f01d8037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8031c10 con 0x7f01e0136c20 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: dump: 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:05.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d8040070 con 0x7f01e0136c20 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 5 monclient: authenticate success, global_id 124520 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 10 monclient: _renew_subs 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8045430 con 0x7f01e0136c20 2024-09-15T03:41:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e011c810 con 0x7f01e0136c20 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01dd7fa640 4 set_mon_vals no callback set 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01c6ffd640 10 monclient: dump: 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:05.232 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.232 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:05.232 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:05.232 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:05.232 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:05.232 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 10 monclient: _renew_subs 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.208+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e0099960 con 0x7f01e0136c20 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f01d8042030 con 0x7f01e0136c20 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01c6ffd640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01cc038740 0x7f01cc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f01d806e0b0 con 0x7f01e0136c20 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01e5820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01e5820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01e5820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:05.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e004bf60 con 0x7f01e0136c20 2024-09-15T03:41:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01deffd640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01cc038740 0x7f01cc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.209+0000 7f01deffd640 10 monclient: get_auth_request con 0x7f01cc038740 auth_method 0 2024-09-15T03:41:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.212+0000 7f01deffd640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01cc038740 0x7f01cc03abf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f01d0007fd0 tx=0x7f01d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.213+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01d803b5e0 con 0x7f01e0136c20 2024-09-15T03:41:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.214+0000 7f01c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.214+0000 7f01c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:05.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01e5820640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:05.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01e5820640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:05.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01e5820640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:05.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f01e011ca50 con 0x7f01e0136c20 2024-09-15T03:41:05.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01c6ffd640 1 -- 172.21.15.153:0/4189751211 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f01d80303a0 con 0x7f01e0136c20 2024-09-15T03:41:05.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:05.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.430+0000 7f01c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:05.432 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":126,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01cc038740 msgr2=0x7f01cc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01cc038740 0x7f01cc03abf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f01d0007fd0 tx=0x7f01d0009040 comp rx=0 tx=0).stop 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 10 monclient: shutdown 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 msgr2=0x7f01e0130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0130c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f01e009fd30 tx=0x7f01d8037000 comp rx=0 tx=0).stop 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 shutdown_connections 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01cc038740 0x7f01cc03abf0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01e0136c20 0x7f01e0130c10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e01318f0 0x7f01e01306d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 --2- 172.21.15.153:0/4189751211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e009ea50 0x7f01e011daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:05.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.432+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 >> 172.21.15.153:0/4189751211 conn(0x7f01e0067110 msgr2=0x7f01e00a3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:05.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.433+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 shutdown_connections 2024-09-15T03:41:05.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:05.433+0000 7f01e5820640 1 -- 172.21.15.153:0/4189751211 wait complete. 2024-09-15T03:41:08.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.549+0000 7fa78fea8640 10 monclient: get_monmap_and_config 2024-09-15T03:41:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.549+0000 7fa78fea8640 10 monclient: build_initial_monmap 2024-09-15T03:41:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.549+0000 7fa78fea8640 10 monclient: monmap: 2024-09-15T03:41:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:08.550566+0000 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:08.550566+0000 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:08.551 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.550+0000 7fa78fea8640 1 Processor -- start 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.550+0000 7fa78fea8640 1 -- start start 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.550+0000 7fa78fea8640 10 monclient: init 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): picked mon.noname-b con 0x7fa7881318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa78809ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): picked mon.noname-c con 0x7fa788136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa788131cc0 con 0x7fa78809ea50 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78809f530 con 0x7fa788136c20 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78809fbf0 con 0x7fa7881318f0 2024-09-15T03:41:08.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:08.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78fea8640 10 monclient(hunting): authenticate will time out at 1466.190229s 2024-09-15T03:41:08.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78d443640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa78809ea50 0x7fa78809ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41342/0 (socket says 172.21.15.153:41342) 2024-09-15T03:41:08.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78d443640 1 -- 172.21.15.153:0/3065457718 learned_addr learned my addr 172.21.15.153:0/3065457718 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:08.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 10 monclient(hunting): get_auth_request con 0x7fa7881318f0 auth_method 0 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 10 monclient(hunting): get_auth_request con 0x7fa788136c20 auth_method 0 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:08.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:08.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:08.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:08.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.551+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:08.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.553+0000 7fa78d443640 1 -- 172.21.15.153:0/3065457718 >> v1:172.21.15.153:6789/0 conn(0x7fa78809ea50 legacy=0x7fa78809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 10 monclient(hunting): get_auth_request con 0x7fa7881318f0 auth_method 0 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 10 monclient(hunting): get_auth_request con 0x7fa788136c20 auth_method 0 2024-09-15T03:41:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:08.752+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 10 monclient(hunting): get_auth_request con 0x7fa7881318f0 auth_method 0 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:09.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 10 monclient(hunting): get_auth_request con 0x7fa788136c20 auth_method 0 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:09.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:09.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.153+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 10 monclient(hunting): get_auth_request con 0x7fa7881318f0 auth_method 0 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:09.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 10 monclient(hunting): get_auth_request con 0x7fa788136c20 auth_method 0 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:09.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:09.954+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.422+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386259629 0 0) 0x7fa788131cc0 con 0x7fa78809ea50 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.422+0000 7fa78cc42640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.422+0000 7fa78cc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.422+0000 7fa78cc42640 10 monclient(hunting): my global_id is 164097 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.422+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7740034e0 con 0x7fa78809ea50 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802222259 0 0) 0x7fa7740034e0 con 0x7fa78809ea50 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa788131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa788131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 msgr2=0x7fa788139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 --2- 172.21.15.153:0/3065457718 >> v2:172.21.15.181:3300/0 conn(0x7fa788136c20 0x7fa788139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78809fe30 con 0x7fa78809ea50 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fa778001720 con 0x7fa78809ea50 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: dump: 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 >> v1:172.21.15.153:6789/0 conn(0x7fa78809ea50 legacy=0x7fa78809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774003ad0 0x7fa774003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): picked mon.b con 0x7fa774003ad0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa774004780 0x7fa774004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): picked mon.a con 0x7fa774004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa77400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): picked mon.c con 0x7fa7881318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78809f530 con 0x7fa774004780 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78809fbf0 con 0x7fa774003ad0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa774005560 con 0x7fa7881318f0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78cc42640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa774004780 0x7fa774004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa77400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): get_auth_request con 0x7fa774004780 auth_method 0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774003ad0 0x7fa774003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): get_auth_request con 0x7fa7881318f0 auth_method 0 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): get_auth_request con 0x7fa774003ad0 auth_method 0 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78e445640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78dc44640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 10 monclient(hunting): handle_auth_done global_id 164097 payload 293 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 msgr2=0x7fa77400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa77400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.423+0000 7fa78d443640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774003ad0 msgr2=0x7fa774003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774003ad0 0x7fa774003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 1 monclient: found mon.a 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 1 -- 172.21.15.153:0/3065457718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7740034e0 con 0x7fa774004780 2024-09-15T03:41:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78d443640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78fea8640 5 monclient: authenticate success, global_id 164097 2024-09-15T03:41:11.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78fea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:11.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78e445640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7881318f0 0x7fa77400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:11.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.424+0000 7fa78dc44640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774003ad0 0x7fa774003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:11.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.426+0000 7fa78d443640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa774004780 msgr2=0x7fa774004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.426+0000 7fa78d443640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa774004780 0x7fa774004bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa778001490 tx=0x7fa778007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6b9dd85ee4cb3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778005890 con 0x7fa774004780 2024-09-15T03:41:11.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:11.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: dump: 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778005a30 con 0x7fa774004780 2024-09-15T03:41:11.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 1 -- 172.21.15.153:0/3065457718 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778005df0 con 0x7fa774004780 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 4 set_mon_vals no callback set 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78cc42640 10 monclient: dump: 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.441 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 10 monclient: shutdown 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 1 -- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa774004780 msgr2=0x7fa774004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.427+0000 7fa78fea8640 1 --2- 172.21.15.153:0/3065457718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa774004780 0x7fa774004bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa778001490 tx=0x7fa778007a90 comp rx=0 tx=0).stop 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78e445640 1 -- 172.21.15.153:0/3065457718 reap_dead start 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78fea8640 1 -- 172.21.15.153:0/3065457718 shutdown_connections 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78fea8640 1 -- 172.21.15.153:0/3065457718 >> 172.21.15.153:0/3065457718 conn(0x7fa788067110 msgr2=0x7fa788067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78fea8640 1 -- 172.21.15.153:0/3065457718 shutdown_connections 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78fea8640 1 -- 172.21.15.153:0/3065457718 wait complete. 2024-09-15T03:41:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78fea8640 10 monclient: build_initial_monmap 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.428+0000 7fa78fea8640 10 monclient: monmap: 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 Processor -- start 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 -- start start 2024-09-15T03:41:11.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient: init 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774004780 0x7fa7881412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): picked mon.noname-b con 0x7fa774004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 0x7fa7880995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): picked mon.noname-a con 0x7fa788141800 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 0x7fa788099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): picked mon.noname-c con 0x7fa788141bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78809bf10 con 0x7fa788141800 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78809de50 con 0x7fa774004780 2024-09-15T03:41:11.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78809b140 con 0x7fa788141bd0 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78fea8640 10 monclient(hunting): authenticate will time out at 1469.068824s 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78e445640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 0x7fa788099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78d443640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 0x7fa7880995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78e445640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 0x7fa788099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36476/0 (socket says 172.21.15.153:36476) 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.429+0000 7fa78e445640 1 -- 172.21.15.153:0/1325807961 learned_addr learned my addr 172.21.15.153:0/1325807961 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): get_auth_request con 0x7fa788141800 auth_method 0 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): get_auth_request con 0x7fa788141bd0 auth_method 0 2024-09-15T03:41:11.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774004780 0x7fa7881412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): get_auth_request con 0x7fa774004780 auth_method 0 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78dc44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient(hunting): handle_auth_done global_id 164100 payload 824 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 1 -- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 msgr2=0x7fa788099af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 0x7fa788099af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 1 -- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774004780 msgr2=0x7fa7881412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774004780 0x7fa7881412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 1 monclient: found mon.noname-a 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78d443640 1 -- 172.21.15.153:0/1325807961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78811c7a0 con 0x7fa788141800 2024-09-15T03:41:11.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.430+0000 7fa78e445640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 0x7fa788099af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa78d443640 1 -- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 msgr2=0x7fa7880995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa78d443640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 0x7fa7880995b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa7740054b0 tx=0x7fa778034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778007340 con 0x7fa788141800 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 10 monclient: dump: 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.448 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7780074c0 con 0x7fa788141800 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa78fea8640 5 monclient: authenticate success, global_id 164100 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77802f470 con 0x7fa788141800 2024-09-15T03:41:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa78fea8640 10 monclient: _renew_subs 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa78fea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78809b840 con 0x7fa788141800 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.433+0000 7fa77dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.433+0000 7fa77dffb640 10 monclient: dump: 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.432+0000 7fa77ffff640 4 set_mon_vals no callback set 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.433+0000 7fa78fea8640 10 monclient: _renew_subs 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.433+0000 7fa78fea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.433+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78811d9c0 con 0x7fa788141800 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa77802f7e0 con 0x7fa788141800 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa77dffb640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa76c038790 0x7fa76c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa78fea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa78fea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa78fea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78804bf60 con 0x7fa788141800 2024-09-15T03:41:11.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa77806d060 con 0x7fa788141800 2024-09-15T03:41:11.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa78dc44640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa76c038790 0x7fa76c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.434+0000 7fa78dc44640 10 monclient: get_auth_request con 0x7fa76c038790 auth_method 0 2024-09-15T03:41:11.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.437+0000 7fa78dc44640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa76c038790 0x7fa76c03ac40 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa784011730 tx=0x7fa784008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.439+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa778039430 con 0x7fa788141800 2024-09-15T03:41:11.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.439+0000 7fa77dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.439+0000 7fa77dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa78fea8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa78fea8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa78fea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa7880638d0 con 0x7fa788141800 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa77dffb640 1 -- 172.21.15.153:0/1325807961 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa77802cb50 con 0x7fa788141800 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa77dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.655+0000 7fa77dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa76c038790 msgr2=0x7fa76c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa76c038790 0x7fa76c03ac40 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa784011730 tx=0x7fa784008040 comp rx=0 tx=0).stop 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 10 monclient: shutdown 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 msgr2=0x7fa7880995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 0x7fa7880995b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa7740054b0 tx=0x7fa778034000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 shutdown_connections 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa76c038790 0x7fa76c03ac40 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa788141bd0 0x7fa788099af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa788141800 0x7fa7880995b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 --2- 172.21.15.153:0/1325807961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa774004780 0x7fa7881412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 >> 172.21.15.153:0/1325807961 conn(0x7fa788067110 msgr2=0x7fa7880a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 shutdown_connections 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.657+0000 7fa78fea8640 1 -- 172.21.15.153:0/1325807961 wait complete. 2024-09-15T03:41:11.669 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:41:11.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:41:11.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:41:11.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.761+0000 7f835e089640 10 monclient: get_monmap_and_config 2024-09-15T03:41:11.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.761+0000 7f835e089640 10 monclient: build_initial_monmap 2024-09-15T03:41:11.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.761+0000 7f835e089640 10 monclient: monmap: 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:11.762238+0000 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:11.762238+0000 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:11.763 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:11.764 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 Processor -- start 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 -- start start 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient: init 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f835809c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 0x7f835809aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): picked mon.noname-b con 0x7f835809e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8358136cb0 0x7f8358139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): picked mon.noname-c con 0x7f8358136cb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:11.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835809ec40 con 0x7f835809c860 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835809a9d0 con 0x7f8358136cb0 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835806a320 con 0x7f835809e840 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f835e089640 10 monclient(hunting): authenticate will time out at 1469.401801s 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 0x7f835809aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 0x7f835809aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36480/0 (socket says 172.21.15.153:36480) 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 1 -- 172.21.15.153:0/854887660 learned_addr learned my addr 172.21.15.153:0/854887660 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 10 monclient(hunting): get_auth_request con 0x7f835809e840 auth_method 0 2024-09-15T03:41:11.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.762+0000 7f8356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 10 monclient(hunting): handle_auth_done global_id 124526 payload 274 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 1 -- 172.21.15.153:0/854887660 >> v2:172.21.15.181:3300/0 conn(0x7f8358136cb0 msgr2=0x7f8358139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 1 --2- 172.21.15.153:0/854887660 >> v2:172.21.15.181:3300/0 conn(0x7f8358136cb0 0x7f8358139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 1 -- 172.21.15.153:0/854887660 >> v1:172.21.15.181:6789/0 conn(0x7f835809c860 legacy=0x7f835809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:11.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 1 monclient: found mon.noname-b 2024-09-15T03:41:11.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.763+0000 7f8356ffd640 1 -- 172.21.15.153:0/854887660 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8358068a20 con 0x7f835809e840 2024-09-15T03:41:11.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.765+0000 7f8356ffd640 1 -- 172.21.15.153:0/854887660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 msgr2=0x7f835809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.765+0000 7f8356ffd640 1 --2- 172.21.15.153:0/854887660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 0x7f835809aba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8340009910 tx=0x7f83400091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2f7d7558ed8ca93 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 1 -- 172.21.15.153:0/854887660 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8340032440 con 0x7f835809e840 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: dump: 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.779 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 1 -- 172.21.15.153:0/854887660 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83400325a0 con 0x7f835809e840 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 5 monclient: authenticate success, global_id 124526 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:11.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:11.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:11.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 1 -- 172.21.15.153:0/854887660 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8340032940 con 0x7f835809e840 2024-09-15T03:41:11.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 4 set_mon_vals no callback set 2024-09-15T03:41:11.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f83567fc640 10 monclient: dump: 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.782 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 10 monclient: shutdown 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 -- 172.21.15.153:0/854887660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 msgr2=0x7f835809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 --2- 172.21.15.153:0/854887660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 0x7f835809aba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8340009910 tx=0x7f83400091d0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 -- 172.21.15.153:0/854887660 shutdown_connections 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 --2- 172.21.15.153:0/854887660 >> v2:172.21.15.181:3300/0 conn(0x7f8358136cb0 0x7f8358139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 --2- 172.21.15.153:0/854887660 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f835809e840 0x7f835809aba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 -- 172.21.15.153:0/854887660 >> 172.21.15.153:0/854887660 conn(0x7f83580a2e40 msgr2=0x7f83580a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 -- 172.21.15.153:0/854887660 shutdown_connections 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.766+0000 7f835e089640 1 -- 172.21.15.153:0/854887660 wait complete. 2024-09-15T03:41:11.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.767+0000 7f835e089640 10 monclient: build_initial_monmap 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.767+0000 7f835e089640 10 monclient: monmap: 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.767+0000 7f835e089640 1 Processor -- start 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.767+0000 7f835e089640 1 -- start start 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.767+0000 7f835e089640 10 monclient: init 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 0x7f835806e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): picked mon.noname-a con 0x7f835809c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835809e840 0x7f835811d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): picked mon.noname-b con 0x7f835809e840 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:11.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 0x7f835811ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): picked mon.noname-c con 0x7f8358136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835811ff00 con 0x7f835809c860 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8358099cc0 con 0x7f835809e840 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8358139120 con 0x7f8358136cb0 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f835e089640 10 monclient(hunting): authenticate will time out at 1469.407136s 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 0x7f835811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 0x7f835806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 0x7f835811ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36488/0 (socket says 172.21.15.153:36488) 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 learned_addr learned my addr 172.21.15.153:0/4240013456 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:11.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): get_auth_request con 0x7f8358136cb0 auth_method 0 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 10 monclient(hunting): get_auth_request con 0x7f835809c860 auth_method 0 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 msgr2=0x7f835806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 msgr2=0x7f835806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 0x7f835806e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835809e840 0x7f835811d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f83577fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 0x7f835806e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:11.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): get_auth_request con 0x7f835809e840 auth_method 0 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8356ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient(hunting): handle_auth_done global_id 124529 payload 824 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835809e840 msgr2=0x7f835811d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835809e840 0x7f835811d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 msgr2=0x7f835806e430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835809c860 0x7f835806e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 monclient: found mon.noname-c 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.768+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340009450 con 0x7f8358136cb0 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 msgr2=0x7f835811ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f8357fff640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 0x7f835811ddf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f835000bb90 tx=0x7f8350009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835000cc80 con 0x7f8358136cb0 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: dump: 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835000ce00 con 0x7f8358136cb0 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f835e089640 5 monclient: authenticate success, global_id 124529 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f835e089640 10 monclient: _renew_subs 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f835e089640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8358136920 con 0x7f8358136cb0 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350014660 con 0x7f8358136cb0 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f834f7fe640 10 monclient: dump: 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.791 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f8355ffb640 4 set_mon_vals no callback set 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.770+0000 7f835e089640 10 monclient: _renew_subs 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f835e089640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835811f8b0 con 0x7f8358136cb0 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f835e089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8350014a50 con 0x7f8358136cb0 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f835e089640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 0x7f83581416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8358065e00 con 0x7f835811e330 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f83577fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 0x7f83581416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f83577fe640 10 monclient: get_auth_request con 0x7f835811e330 auth_method 0 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f83577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f83577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.771+0000 7f83577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f83577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f83577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f83577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f834f7fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8338038740 0x7f833803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f835004c730 con 0x7f8358136cb0 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f83577fe640 10 monclient(hunting): handle_auth_done global_id 164103 payload 824 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f83577fe640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f835811e700 con 0x7f835811e330 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f8356ffd640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8338038740 0x7f833803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.772+0000 7f8356ffd640 10 monclient: get_auth_request con 0x7f8338038740 auth_method 0 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f83577fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 msgr2=0x7f83581416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f83577fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 0x7f83581416d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f834800d710 tx=0x7f8348008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348012900 con 0x7f835811e330 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 10 monclient: dump: 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8348012a60 con 0x7f835811e330 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 msgr2=0x7f83581416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 0x7f83581416d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f834800d710 tx=0x7f8348008040 comp rx=0 tx=0).stop 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 msgr2=0x7f83581416d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f834f7fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f835811e330 0x7f83581416d0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f834800d710 tx=0x7f8348008040 comp rx=0 tx=0).stop 2024-09-15T03:41:11.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.774+0000 7f8356ffd640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8338038740 0x7f833803abf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8358071740 tx=0x7f8340032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f835e089640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f835e089640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 0x7f8358142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f835e089640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835811fc20 con 0x7f8358141c10 2024-09-15T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 0x7f8358142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient: get_auth_request con 0x7f8358141c10 auth_method 0 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 10 monclient(hunting): handle_auth_done global_id 164106 payload 824 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.780+0000 7f83577fe640 1 -- 172.21.15.153:0/4240013456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8358142550 con 0x7f8358141c10 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f83577fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 msgr2=0x7f8358142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f83577fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 0x7f8358142010 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8348017000 tx=0x7f834801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834801b870 con 0x7f8358141c10 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 10 monclient: dump: 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f834801b9d0 con 0x7f8358141c10 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:11.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 msgr2=0x7f8358142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 0x7f8358142010 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8348017000 tx=0x7f834801a000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 msgr2=0x7f8358142010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f834f7fe640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8358141c10 0x7f8358142010 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8348017000 tx=0x7f834801a000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 130, 2024-09-15T03:41:11.800 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:11.801 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:11.802 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:11.803 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:11.804 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:11.805 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:11.806 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:11.807 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:11.808 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.809 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:11.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:11.811 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8338038740 msgr2=0x7f833803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8338038740 0x7f833803abf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8358071740 tx=0x7f8340032000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 10 monclient: shutdown 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 msgr2=0x7f835811ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 --2- 172.21.15.153:0/4240013456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358136cb0 0x7f835811ddf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f835000bb90 tx=0x7f8350009810 comp rx=0 tx=0).stop 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 reap_dead start 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f8357fff640 1 -- 172.21.15.153:0/4240013456 reap_dead start 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 shutdown_connections 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 >> 172.21.15.153:0/4240013456 conn(0x7f83580a2e40 msgr2=0x7f835809b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 shutdown_connections 2024-09-15T03:41:11.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.783+0000 7f835e089640 1 -- 172.21.15.153:0/4240013456 wait complete. 2024-09-15T03:41:11.814 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:41:11.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:41:11.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:41:11.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.907+0000 7efe56e4f640 10 monclient: get_monmap_and_config 2024-09-15T03:41:11.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.907+0000 7efe56e4f640 10 monclient: build_initial_monmap 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.907+0000 7efe56e4f640 10 monclient: monmap: 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:11.908476+0000 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:11.908476+0000 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:11.909 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.908+0000 7efe56e4f640 1 Processor -- start 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.908+0000 7efe56e4f640 1 -- start start 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.908+0000 7efe56e4f640 10 monclient: init 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.908+0000 7efe56e4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe50131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efe5009fda0 0x7efe5009e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): picked mon.noname-c con 0x7efe5009fda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe5009ef00 0x7efe50138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): picked mon.noname-a con 0x7efe5009ef00 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500a1180 con 0x7efe50131980 2024-09-15T03:41:11.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe500a1460 con 0x7efe5009ef00 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe50132460 con 0x7efe5009fda0 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe56e4f640 10 monclient(hunting): authenticate will time out at 1469.548123s 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efe5009ef00 0x7efe50138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe54beb640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7efe50131980 0x7efe50131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41346/0 (socket says 172.21.15.153:41346) 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe54beb640 1 -- 172.21.15.153:0/3076454381 learned_addr learned my addr 172.21.15.153:0/3076454381 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): get_auth_request con 0x7efe5009ef00 auth_method 0 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3076454381 >> v2:172.21.15.181:3300/0 conn(0x7efe5009fda0 0x7efe5009e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe4ffff640 10 monclient(hunting): get_auth_request con 0x7efe5009fda0 auth_method 0 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient(hunting): handle_auth_done global_id 164109 payload 274 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 1 -- 172.21.15.153:0/3076454381 >> v2:172.21.15.181:3300/0 conn(0x7efe5009fda0 msgr2=0x7efe5009e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 1 --2- 172.21.15.153:0/3076454381 >> v2:172.21.15.181:3300/0 conn(0x7efe5009fda0 0x7efe5009e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 1 -- 172.21.15.153:0/3076454381 >> v1:172.21.15.153:6789/0 conn(0x7efe50131980 legacy=0x7efe50131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 1 monclient: found mon.noname-a 2024-09-15T03:41:11.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:41:11.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.909+0000 7efe553ec640 1 -- 172.21.15.153:0/3076454381 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe501325d0 con 0x7efe5009ef00 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe553ec640 1 -- 172.21.15.153:0/3076454381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 msgr2=0x7efe50138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe553ec640 1 --2- 172.21.15.153:0/3076454381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50138cc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe50064050 tx=0x7efe48009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c019eb68cb0fb49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 1 -- 172.21.15.153:0/3076454381 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48007760 con 0x7efe5009ef00 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: dump: 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.924 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 1 -- 172.21.15.153:0/3076454381 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4802d050 con 0x7efe5009ef00 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 1 -- 172.21.15.153:0/3076454381 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48007b30 con 0x7efe5009ef00 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 5 monclient: authenticate success, global_id 164109 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 4 set_mon_vals no callback set 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe4f7fe640 10 monclient: dump: 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 10 monclient: shutdown 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:11.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 1 -- 172.21.15.153:0/3076454381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 msgr2=0x7efe50138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3076454381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50138cc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe50064050 tx=0x7efe48009780 comp rx=0 tx=0).stop 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 1 -- 172.21.15.153:0/3076454381 shutdown_connections 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3076454381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50138cc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3076454381 >> v2:172.21.15.181:3300/0 conn(0x7efe5009fda0 0x7efe5009e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.912+0000 7efe56e4f640 1 -- 172.21.15.153:0/3076454381 >> 172.21.15.153:0/3076454381 conn(0x7efe50067bd0 msgr2=0x7efe500a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.913+0000 7efe56e4f640 1 -- 172.21.15.153:0/3076454381 shutdown_connections 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.913+0000 7efe56e4f640 1 -- 172.21.15.153:0/3076454381 wait complete. 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.913+0000 7efe56e4f640 10 monclient: build_initial_monmap 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.913+0000 7efe56e4f640 10 monclient: monmap: 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.913+0000 7efe56e4f640 1 Processor -- start 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 -- start start 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient: init 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50130f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): picked mon.noname-a con 0x7efe5009ef00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): picked mon.noname-c con 0x7efe5009fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 0x7efe50096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): picked mon.noname-b con 0x7efe50131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe500998b0 con 0x7efe5009ef00 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5009a290 con 0x7efe50131980 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe50097b90 con 0x7efe5009fda0 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe56e4f640 10 monclient(hunting): authenticate will time out at 1469.553527s 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36490/0 (socket says 172.21.15.153:36490) 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50130f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38984/0 (socket says 172.21.15.153:38984) 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 learned_addr learned my addr 172.21.15.153:0/3091832452 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): get_auth_request con 0x7efe5009fda0 auth_method 0 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): get_auth_request con 0x7efe5009ef00 auth_method 0 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe553ec640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 0x7efe50096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe553ec640 10 monclient(hunting): get_auth_request con 0x7efe50131980 auth_method 0 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe553ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe553ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe553ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient(hunting): handle_auth_done global_id 124532 payload 824 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 msgr2=0x7efe50096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 0x7efe50096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 msgr2=0x7efe50130f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50130f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 monclient: found mon.noname-c 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe48009a20 con 0x7efe5009fda0 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.914+0000 7efe54beb640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe50130f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 msgr2=0x7efe50096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00d890 tx=0x7efe3c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c00de20 con 0x7efe5009fda0 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 msgr2=0x7efe50096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 msgr2=0x7efe50096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: dump: 2024-09-15T03:41:11.932 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00d890 tx=0x7efe3c008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:11.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00d890 tx=0x7efe3c008040 comp rx=0 tx=0).stop 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe56e4f640 5 monclient: authenticate success, global_id 124532 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe56e4f640 10 monclient: _renew_subs 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe56e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe500995d0 con 0x7efe5009fda0 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 msgr2=0x7efe50096a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe5009fda0 0x7efe50096a60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 0x7efe340056b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): picked mon.b con 0x7efe50131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe34008060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): picked mon.a con 0x7efe5009ef00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe340089e0 0x7efe3400ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): picked mon.c con 0x7efe340089e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3400b910 con 0x7efe5009ef00 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3400ba50 con 0x7efe50131980 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3400bb90 con 0x7efe340089e0 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4cff9640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe56e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe34008060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe340089e0 0x7efe3400ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): get_auth_request con 0x7efe5009ef00 auth_method 0 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): get_auth_request con 0x7efe340089e0 auth_method 0 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe54beb640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 0x7efe340056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe54beb640 10 monclient(hunting): get_auth_request con 0x7efe50131980 auth_method 0 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe54beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe54beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe54beb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient(hunting): handle_auth_done global_id 124532 payload 293 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 msgr2=0x7efe340056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe50131980 0x7efe340056b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 msgr2=0x7efe34008060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe34008060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient: _finish_hunting 0 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 monclient: found mon.c 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:11.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7efe3400be20 con 0x7efe340089e0 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe500995d0 con 0x7efe340089e0 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 10 monclient: _finish_auth 0 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe553ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.917+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5009ef00 0x7efe34008060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.918+0000 7efe56e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.918+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 0x7efe50130f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.918+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.918+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe50065e00 con 0x7efe500986e0 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe340089e0 msgr2=0x7efe3400ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe553ec640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe340089e0 0x7efe3400ae60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efe50068070 tx=0x7efe48037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 0x7efe50130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient: get_auth_request con 0x7efe500986e0 auth_method 0 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48007a00 con 0x7efe340089e0 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 10 monclient: dump: 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.939 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe48007b60 con 0x7efe340089e0 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4effd640 4 set_mon_vals no callback set 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 10 monclient(hunting): handle_auth_done global_id 124535 payload 824 2024-09-15T03:41:11.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe54beb640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe500a4220 con 0x7efe500986e0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.920+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efe4803c420 con 0x7efe340089e0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.921+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe5009fda0 0x7efe34042ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.921+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 4 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efe4806c800 con 0x7efe340089e0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.921+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe5009fda0 0x7efe34042ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.921+0000 7efe4ffff640 10 monclient: get_auth_request con 0x7efe5009fda0 auth_method 0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe54beb640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 msgr2=0x7efe50130f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe54beb640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 0x7efe50130f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe44010ac0 tx=0x7efe44007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4400edf0 con 0x7efe500986e0 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 10 monclient: dump: 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.941 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efe44014460 con 0x7efe500986e0 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:11.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:11.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 msgr2=0x7efe50130f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 0x7efe50130f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe44010ac0 tx=0x7efe44007f00 comp rx=0 tx=0).stop 2024-09-15T03:41:11.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 msgr2=0x7efe50130f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:11.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.922+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe500986e0 0x7efe50130f10 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe44010ac0 tx=0x7efe44007f00 comp rx=0 tx=0).stop 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.923+0000 7efe4ffff640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe5009fda0 0x7efe34042ea0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000c00 tx=0x7efe3c01b000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe56e4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 0x7efe50096e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe56e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe50131450 con 0x7efe50096a60 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 0x7efe50096e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient: get_auth_request con 0x7efe50096a60 auth_method 0 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.928+0000 7efe54beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.929+0000 7efe54beb640 10 monclient(hunting): handle_auth_done global_id 124538 payload 824 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.929+0000 7efe54beb640 1 -- 172.21.15.153:0/3091832452 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efe5004bf60 con 0x7efe50096a60 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe54beb640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 msgr2=0x7efe50096e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe54beb640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 0x7efe50096e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efe44010ac0 tx=0x7efe4401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4401ee00 con 0x7efe50096a60 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 10 monclient: dump: 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7efe44020410 con 0x7efe50096a60 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 msgr2=0x7efe50096e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 0x7efe50096e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efe44010ac0 tx=0x7efe4401a000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 msgr2=0x7efe50096e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe4cff9640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe50096a60 0x7efe50096e60 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efe44010ac0 tx=0x7efe4401a000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 130, 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:41:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:11.949 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:11.950 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:11.953 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:11.954 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.955 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.956 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.957 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:11.958 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.959 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:11.960 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe5009fda0 msgr2=0x7efe34042ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efe5009fda0 0x7efe34042ea0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000c00 tx=0x7efe3c01b000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe56e4f640 10 monclient: shutdown 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe56e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe340089e0 msgr2=0x7efe3400ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe56e4f640 1 --2- 172.21.15.153:0/3091832452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe340089e0 0x7efe3400ae60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efe50068070 tx=0x7efe48037000 comp rx=0 tx=0).stop 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 reap_dead start 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.931+0000 7efe553ec640 1 -- 172.21.15.153:0/3091832452 reap_dead start 2024-09-15T03:41:11.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.932+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 shutdown_connections 2024-09-15T03:41:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.932+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 >> 172.21.15.153:0/3091832452 conn(0x7efe50067bd0 msgr2=0x7efe5012cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.932+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 shutdown_connections 2024-09-15T03:41:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:11.932+0000 7efe56e4f640 1 -- 172.21.15.153:0/3091832452 wait complete. 2024-09-15T03:41:11.963 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:41:11.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:41:11.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:41:12.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.057+0000 7f834459d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:12.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.057+0000 7f834459d640 10 monclient: build_initial_monmap 2024-09-15T03:41:12.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.057+0000 7f834459d640 10 monclient: monmap: 2024-09-15T03:41:12.058 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:12.058 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.058 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:12.057918+0000 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:12.057918+0000 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.057+0000 7f834459d640 1 Processor -- start 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.057+0000 7f834459d640 1 -- start start 2024-09-15T03:41:12.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.057+0000 7f834459d640 10 monclient: init 2024-09-15T03:41:12.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f833c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 0x7f833c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): picked mon.noname-b con 0x7f833c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f833c136c20 0x7f833c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): picked mon.noname-a con 0x7f833c136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c132640 con 0x7f833c1318f0 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c131d00 con 0x7f833c136c20 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c09fbf0 con 0x7f833c09ea50 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:12.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f834459d640 10 monclient(hunting): authenticate will time out at 1469.697421s 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f833c136c20 0x7f833c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 0x7f833c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f833c136c20 0x7f833c139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39012/0 (socket says 172.21.15.153:39012) 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 0x7f833c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36514/0 (socket says 172.21.15.153:36514) 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 -- 172.21.15.153:0/4091455191 learned_addr learned my addr 172.21.15.153:0/4091455191 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): get_auth_request con 0x7f833c136c20 auth_method 0 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): get_auth_request con 0x7f833c09ea50 auth_method 0 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient(hunting): handle_auth_done global_id 164115 payload 274 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 -- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 msgr2=0x7f833c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 --2- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 0x7f833c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 -- 172.21.15.153:0/4091455191 >> v1:172.21.15.181:6789/0 conn(0x7f833c1318f0 legacy=0x7f833c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 monclient: found mon.noname-a 2024-09-15T03:41:12.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:12.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:41:12.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8342b3a640 1 -- 172.21.15.153:0/4091455191 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c09fe30 con 0x7f833c136c20 2024-09-15T03:41:12.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.058+0000 7f8341b38640 1 --2- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 0x7f833c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8342b3a640 1 -- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c136c20 msgr2=0x7f833c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8342b3a640 1 --2- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c136c20 0x7f833c139000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8334009f20 tx=0x7f8334009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6101a76463243aa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 1 -- 172.21.15.153:0/4091455191 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334007610 con 0x7f833c136c20 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: dump: 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.072 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: _finish_auth 0 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 1 -- 172.21.15.153:0/4091455191 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833402d050 con 0x7f833c136c20 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f834459d640 5 monclient: authenticate success, global_id 164115 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f834459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 1 -- 172.21.15.153:0/4091455191 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340079a0 con 0x7f833c136c20 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f834459d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:12.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f834459d640 4 set_mon_vals no callback set 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.061+0000 7f8341337640 10 monclient: dump: 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 10 monclient: shutdown 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 -- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c136c20 msgr2=0x7f833c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 --2- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c136c20 0x7f833c139000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8334009f20 tx=0x7f8334009840 comp rx=0 tx=0).stop 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 -- 172.21.15.153:0/4091455191 shutdown_connections 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 --2- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c136c20 0x7f833c139000 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 --2- 172.21.15.153:0/4091455191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c09ea50 0x7f833c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 -- 172.21.15.153:0/4091455191 >> 172.21.15.153:0/4091455191 conn(0x7f833c067110 msgr2=0x7f833c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 -- 172.21.15.153:0/4091455191 shutdown_connections 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.062+0000 7f834459d640 1 -- 172.21.15.153:0/4091455191 wait complete. 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient: build_initial_monmap 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient: monmap: 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:12.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 Processor -- start 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 -- start start 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient: init 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:12.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:41:12.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c09ea50 0x7f833c1307f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): picked mon.noname-b con 0x7f833c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:12.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c1318f0 0x7f833c0995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): picked mon.noname-a con 0x7f833c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 0x7f833c099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): picked mon.noname-c con 0x7f833c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c09bf10 con 0x7f833c1318f0 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c09dc10 con 0x7f833c09ea50 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c09b180 con 0x7f833c136c20 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f834459d640 10 monclient(hunting): authenticate will time out at 1469.702931s 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.063+0000 7f8341b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c1318f0 0x7f833c0995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 0x7f833c099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 0x7f833c099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36526/0 (socket says 172.21.15.153:36526) 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 -- 172.21.15.153:0/1760238508 learned_addr learned my addr 172.21.15.153:0/1760238508 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c1318f0 0x7f833c0995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39018/0 (socket says 172.21.15.153:39018) 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): get_auth_request con 0x7f833c136c20 auth_method 0 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): get_auth_request con 0x7f833c1318f0 auth_method 0 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8341b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342339640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c09ea50 0x7f833c1307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342339640 10 monclient(hunting): get_auth_request con 0x7f833c09ea50 auth_method 0 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient(hunting): handle_auth_done global_id 124544 payload 824 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c09ea50 msgr2=0x7f833c1307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c09ea50 0x7f833c1307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c1318f0 msgr2=0x7f833c0995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c1318f0 0x7f833c0995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 monclient: found mon.noname-c 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.064+0000 7f8342b3a640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330009390 con 0x7f833c136c20 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f8342b3a640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 msgr2=0x7f833c099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f8342b3a640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 0x7f833c099af0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8334009ab0 tx=0x7f8334036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032cc0 con 0x7f833c136c20 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: dump: 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.084 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8334032e60 con 0x7f833c136c20 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 5 monclient: authenticate success, global_id 124544 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 10 monclient: _renew_subs 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403b660 con 0x7f833c136c20 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8334009ae0 con 0x7f833c136c20 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f832e7fc640 10 monclient: dump: 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f8340b36640 4 set_mon_vals no callback set 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 10 monclient: _renew_subs 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.066+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c11c9b0 con 0x7f833c136c20 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f834459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f834459d640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 0x7f833c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c065e00 con 0x7f833c130d30 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8334038050 con 0x7f833c136c20 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 0x7f833c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f832e7fc640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8320038740 0x7f832003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 10 monclient: get_auth_request con 0x7f833c130d30 auth_method 0 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f833402ae40 con 0x7f833c136c20 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.067+0000 7f8342339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.068+0000 7f8342339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.068+0000 7f8341b38640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8320038740 0x7f832003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.068+0000 7f8341b38640 10 monclient: get_auth_request con 0x7f8320038740 auth_method 0 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.068+0000 7f8342339640 10 monclient(hunting): handle_auth_done global_id 144271 payload 824 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.068+0000 7f8342339640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f833c136670 con 0x7f833c130d30 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f8342339640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 msgr2=0x7f833c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:12.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f8342339640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 0x7f833c0298c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f833800e510 tx=0x7f8338009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f8341b38640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8320038740 0x7f832003abf0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f833c0a04f0 tx=0x7f83300090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338007450 con 0x7f833c130d30 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 10 monclient: dump: 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f83380075b0 con 0x7f833c130d30 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 msgr2=0x7f833c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 0x7f833c0298c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f833800e510 tx=0x7f8338009040 comp rx=0 tx=0).stop 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 msgr2=0x7f833c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.070+0000 7f832e7fc640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c130d30 0x7f833c0298c0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f833800e510 tx=0x7f8338009040 comp rx=0 tx=0).stop 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f834459d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f834459d640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 0x7f833c02a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f834459d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c11bad0 con 0x7f833c029e00 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 0x7f833c02a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient: get_auth_request con 0x7f833c029e00 auth_method 0 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.076+0000 7f8342339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.077+0000 7f8342339640 10 monclient(hunting): handle_auth_done global_id 144274 payload 824 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.077+0000 7f8342339640 1 -- 172.21.15.153:0/1760238508 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f833c04bf60 con 0x7f833c029e00 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.079+0000 7f8342339640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 msgr2=0x7f833c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.079+0000 7f8342339640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 0x7f833c02a200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f833800e510 tx=0x7f833801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.079+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338013c40 con 0x7f833c029e00 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.079+0000 7f832e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.079+0000 7f832e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.079+0000 7f832e7fc640 10 monclient: dump: 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f8338013da0 con 0x7f833c029e00 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 msgr2=0x7f833c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 0x7f833c02a200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f833800e510 tx=0x7f833801b000 comp rx=0 tx=0).stop 2024-09-15T03:41:12.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 msgr2=0x7f833c02a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f832e7fc640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f833c029e00 0x7f833c02a200 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f833800e510 tx=0x7f833801b000 comp rx=0 tx=0).stop 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 130, 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:12.107 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8320038740 msgr2=0x7f832003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8320038740 0x7f832003abf0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f833c0a04f0 tx=0x7f83300090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 10 monclient: shutdown 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 msgr2=0x7f833c099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 0x7f833c099af0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8334009ab0 tx=0x7f8334036000 comp rx=0 tx=0).stop 2024-09-15T03:41:12.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f8342b3a640 1 -- 172.21.15.153:0/1760238508 reap_dead start 2024-09-15T03:41:12.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 shutdown_connections 2024-09-15T03:41:12.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 --2- 172.21.15.153:0/1760238508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f833c136c20 0x7f833c099af0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 >> 172.21.15.153:0/1760238508 conn(0x7f833c067110 msgr2=0x7f833c0a3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:12.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 shutdown_connections 2024-09-15T03:41:12.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.080+0000 7f834459d640 1 -- 172.21.15.153:0/1760238508 wait complete. 2024-09-15T03:41:12.109 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:41:12.109 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:41:12.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.202+0000 7f78da39d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.202+0000 7f78da39d640 10 monclient: build_initial_monmap 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.202+0000 7f78da39d640 10 monclient: monmap: 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:12.203413+0000 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:12.203413+0000 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 1 Processor -- start 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 1 -- start start 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient: init 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78d41318f0 0x7f78d4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient(hunting): picked mon.noname-a con 0x7f78d41318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78d409ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78d4136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d409f530 con 0x7f78d409ea50 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d409fbf0 con 0x7f78d4136c20 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d4131d50 con 0x7f78d41318f0 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.203+0000 7f78da39d640 10 monclient(hunting): authenticate will time out at 1469.842942s 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78d41318f0 0x7f78d4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f78d409ea50 0x7f78d409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41354/0 (socket says 172.21.15.153:41354) 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d37fe640 1 -- 172.21.15.153:0/2597072605 learned_addr learned my addr 172.21.15.153:0/2597072605 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): get_auth_request con 0x7f78d41318f0 auth_method 0 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient(hunting): handle_auth_done global_id 164121 payload 274 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 1 -- 172.21.15.153:0/2597072605 >> v1:172.21.15.181:6789/0 conn(0x7f78d4136c20 legacy=0x7f78d4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 1 -- 172.21.15.153:0/2597072605 >> v1:172.21.15.153:6789/0 conn(0x7f78d409ea50 legacy=0x7f78d409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 1 monclient: found mon.noname-a 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.204+0000 7f78d3fff640 1 -- 172.21.15.153:0/2597072605 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d409fe30 con 0x7f78d41318f0 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d3fff640 1 -- 172.21.15.153:0/2597072605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d41318f0 msgr2=0x7f78d4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d3fff640 1 --2- 172.21.15.153:0/2597072605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d41318f0 0x7f78d4131f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009850 tx=0x7f78c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7aff1a9c188f4d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 1 -- 172.21.15.153:0/2597072605 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4032440 con 0x7f78d41318f0 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: dump: 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 1 -- 172.21.15.153:0/2597072605 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c40325a0 con 0x7f78d41318f0 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 5 monclient: authenticate success, global_id 164121 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 1 -- 172.21.15.153:0/2597072605 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4032940 con 0x7f78d41318f0 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 4 set_mon_vals no callback set 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78d2ffd640 10 monclient: dump: 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 10 monclient: shutdown 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 -- 172.21.15.153:0/2597072605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d41318f0 msgr2=0x7f78d4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 --2- 172.21.15.153:0/2597072605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d41318f0 0x7f78d4131f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009850 tx=0x7f78c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 -- 172.21.15.153:0/2597072605 shutdown_connections 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 --2- 172.21.15.153:0/2597072605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d41318f0 0x7f78d4131f30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 -- 172.21.15.153:0/2597072605 >> 172.21.15.153:0/2597072605 conn(0x7f78d4067110 msgr2=0x7f78d4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 -- 172.21.15.153:0/2597072605 shutdown_connections 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.207+0000 7f78da39d640 1 -- 172.21.15.153:0/2597072605 wait complete. 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.208+0000 7f78da39d640 10 monclient: build_initial_monmap 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.208+0000 7f78da39d640 10 monclient: monmap: 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.208+0000 7f78da39d640 1 Processor -- start 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.208+0000 7f78da39d640 1 -- start start 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.208+0000 7f78da39d640 10 monclient: init 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 0x7f78d409b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): picked mon.noname-a con 0x7f78d409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 0x7f78d4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): picked mon.noname-c con 0x7f78d41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78d4136c20 0x7f78d4096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): picked mon.noname-b con 0x7f78d4136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d409ac40 con 0x7f78d409ea50 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d409bf10 con 0x7f78d4136c20 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d413a510 con 0x7f78d41318f0 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78da39d640 10 monclient(hunting): authenticate will time out at 1469.848375s 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 0x7f78d409b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 0x7f78d409b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39040/0 (socket says 172.21.15.153:39040) 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 1 -- 172.21.15.153:0/2524757885 learned_addr learned my addr 172.21.15.153:0/2524757885 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): get_auth_request con 0x7f78d409ea50 auth_method 0 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 0x7f78d4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): get_auth_request con 0x7f78d41318f0 auth_method 0 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d893a640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78d4136c20 0x7f78d4096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d893a640 10 monclient(hunting): get_auth_request con 0x7f78d4136c20 auth_method 0 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d893a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d893a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient(hunting): handle_auth_done global_id 124547 payload 824 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 -- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78d4136c20 msgr2=0x7f78d4096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78d4136c20 0x7f78d4096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 -- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 msgr2=0x7f78d409b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 0x7f78d409b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 monclient: found mon.noname-c 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d37fe640 1 -- 172.21.15.153:0/2524757885 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c4009390 con 0x7f78d41318f0 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d3fff640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 0x7f78d409b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.209+0000 7f78d893a640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78d4136c20 0x7f78d4096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.211+0000 7f78d37fe640 1 -- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 msgr2=0x7f78d4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.211+0000 7f78d37fe640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 0x7f78d4096a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78c800c490 tx=0x7f78c8007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8005cb0 con 0x7f78d41318f0 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: dump: 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: _finish_auth 0 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c8005e30 con 0x7f78d41318f0 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 5 monclient: authenticate success, global_id 124547 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 10 monclient: _renew_subs 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d4099750 con 0x7f78d41318f0 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8007320 con 0x7f78d41318f0 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:12.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 10 monclient: dump: 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78d27fc640 4 set_mon_vals no callback set 2024-09-15T03:41:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 10 monclient: _renew_subs 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d4098790 con 0x7f78d41318f0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.212+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f78c801b3b0 con 0x7f78d41318f0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78da39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78da39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78da39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d40989b0 con 0x7f78d41318f0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78bbfff640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78c0038740 0x7f78c003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f78c804baf0 con 0x7f78d41318f0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78d3fff640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78c0038740 0x7f78c003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.213+0000 7f78d3fff640 10 monclient: get_auth_request con 0x7f78c0038740 auth_method 0 2024-09-15T03:41:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.215+0000 7f78d3fff640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78c0038740 0x7f78c003abf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009130 tx=0x7f78c4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.217+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f78c80187f0 con 0x7f78d41318f0 2024-09-15T03:41:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.217+0000 7f78bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.217+0000 7f78bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:12.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.434+0000 7f78da39d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:41:12.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.434+0000 7f78da39d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:41:12.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.434+0000 7f78da39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:12.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.434+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f78d404bf60 con 0x7f78d41318f0 2024-09-15T03:41:12.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f78bbfff640 1 -- 172.21.15.153:0/2524757885 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f78c8017080 con 0x7f78d41318f0 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f78bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f78bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:12.436 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.436 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=7,logm=52} crc {auth=2516633421,config=3140514009,health=1701369507,kv=3235807718,logm=2986552538}) 2024-09-15T03:41:12.436 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=1,health=11,kv=8,logm=53} crc {auth=3025434033,config=1228700967,health=3478832896,kv=3235807718,logm=4181451806}) 2024-09-15T03:41:12.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=7,logm=52} crc {auth=2516633421,config=3140514009,health=1701369507,kv=3235807718,logm=2986552538}) 2024-09-15T03:41:12.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.435+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=10,kv=8,logm=53} crc {auth=3886694952,config=3140514009,health=1059887827,kv=3235807718,logm=2639201438}) 2024-09-15T03:41:12.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.436+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.436+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824598286}) 2024-09-15T03:41:12.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.436+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016935976}) 2024-09-15T03:41:12.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.436+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.436+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824598286}) 2024-09-15T03:41:12.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.436+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3489840235}) 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78c0038740 msgr2=0x7f78c003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78c0038740 0x7f78c003abf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009130 tx=0x7f78c4032000 comp rx=0 tx=0).stop 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 10 monclient: shutdown 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 msgr2=0x7f78d4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 0x7f78d4096a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78c800c490 tx=0x7f78c8007f00 comp rx=0 tx=0).stop 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 shutdown_connections 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78c0038740 0x7f78c003abf0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78d4136c20 0x7f78d4096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78d41318f0 0x7f78d4096a60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 --2- 172.21.15.153:0/2524757885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d409ea50 0x7f78d409b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:12.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 >> 172.21.15.153:0/2524757885 conn(0x7f78d4067110 msgr2=0x7f78d40a3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:12.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062551071}) 2024-09-15T03:41:12.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2622016897}) 2024-09-15T03:41:12.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 shutdown_connections 2024-09-15T03:41:12.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f78da39d640 1 -- 172.21.15.153:0/2524757885 wait complete. 2024-09-15T03:41:12.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062551071}) 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.437+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1847793086}) 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.438+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.438+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3424482462}) 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.438+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=549081868}) 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.438+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.438+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3424482462}) 2024-09-15T03:41:12.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.438+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=883023979}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.439+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.439+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396331938}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.439+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1776741866}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.439+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.439+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396331938}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.439+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804352871}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.440+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.440+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1379457394}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.440+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769327789}) 2024-09-15T03:41:12.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.440+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.440+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1379457394}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.440+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461850232}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120968841}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204800938}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120968841}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1583228711}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20061601}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761220195}) 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.446 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20061601}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.441+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541021877}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.442+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.442+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456614347}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.442+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343057539}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.442+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.442+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456614347}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.442+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228052132}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322016349}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429332371}) 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.447 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322016349}) 2024-09-15T03:41:12.448 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680919601}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675367758}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1585412124}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675367758}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.443+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116719824}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.444+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.444+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755875623}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.444+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598529470}) 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.444+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.451 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.444+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755875623}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.444+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943978348}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.445+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.445+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995652006}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.445+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632502131}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.445+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.445+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995652006}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.445+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=344328273}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279156225}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021625625}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279156225}) 2024-09-15T03:41:12.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1491084804}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967718995}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284822419}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967718995}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.446+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367085646}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.447+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.447+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1703904728}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.447+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170581359}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.447+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.447+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1703904728}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.447+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3547020589}) 2024-09-15T03:41:12.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.448+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.448+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318983593}) 2024-09-15T03:41:12.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.448+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3912943319}) 2024-09-15T03:41:12.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.448+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.448+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318983593}) 2024-09-15T03:41:12.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.448+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3000172555}) 2024-09-15T03:41:12.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673729692}) 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4098842880}) 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673729692}) 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487048541}) 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459459947}) 2024-09-15T03:41:12.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2141505871}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.449+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459459947}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756417250}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662581115}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=419132257}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662581115}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.450+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130040766}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006702135}) 2024-09-15T03:41:12.465 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495229974}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006702135}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367186762}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.451+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889666005}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958966727}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889666005}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107181044}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655754209}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743712666}) 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.452+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.466 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655754209}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703016046}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983959296}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093598911}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983959296}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.453+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756507484}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.454+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.454+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156806231}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.454+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049196470}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.454+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.454+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156806231}) 2024-09-15T03:41:12.467 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.454+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1059320194}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933173441}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075755204}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933173441}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3401352471}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096378646}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506661471}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096378646}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.455+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1083451484}) 2024-09-15T03:41:12.468 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.456+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.456+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124918256}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.456+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37103984}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.456+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.456+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124918256}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.456+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399062471}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.457+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.457+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064319852}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.457+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2363019600}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.457+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.457+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064319852}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.457+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518376201}) 2024-09-15T03:41:12.469 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276227476}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159092117}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276227476}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190017937}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636109674}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2137333592}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636109674}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.458+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738325037}) 2024-09-15T03:41:12.470 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.459+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.459+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692404631}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.459+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067894212}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.459+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.459+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692404631}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.459+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985440618}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431950075}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=924211126}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431950075}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274834040}) 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.471 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779227210}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=75089408}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779227210}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.460+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=782123577}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.461+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.461+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950999588}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.461+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820081493}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.461+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.461+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950999588}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.461+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556467354}) 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.472 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311336855}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=360785261}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311336855}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1806347117}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.462+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964187701}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064793232}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964187701}) 2024-09-15T03:41:12.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061283385}) 2024-09-15T03:41:12.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176776627}) 2024-09-15T03:41:12.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=561016035}) 2024-09-15T03:41:12.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.463+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.464+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176776627}) 2024-09-15T03:41:12.474 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.464+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1441313742}) 2024-09-15T03:41:12.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443187798}) 2024-09-15T03:41:12.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4171571469}) 2024-09-15T03:41:12.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443187798}) 2024-09-15T03:41:12.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53103538}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511125406}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.494+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390517869}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511125406}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425858676}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039302642}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220130114}) 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039302642}) 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.495+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184698092}) 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.496+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.496+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032020479}) 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.496+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3644720294}) 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.496+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.496+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032020479}) 2024-09-15T03:41:12.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.496+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431603299}) 2024-09-15T03:41:12.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251396334}) 2024-09-15T03:41:12.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002926897}) 2024-09-15T03:41:12.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251396334}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1049278138}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034470842}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610095259}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034470842}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.497+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018163086}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.498+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.498+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420359212}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.498+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2134884382}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.498+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.498+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420359212}) 2024-09-15T03:41:12.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.498+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=101286864}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646398210}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3145320873}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646398210}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2562228247}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4093913974}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036449613}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4093913974}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.499+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3075751344}) 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.500+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.500+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949638687}) 2024-09-15T03:41:12.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.500+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511427349}) 2024-09-15T03:41:12.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.500+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.500+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949638687}) 2024-09-15T03:41:12.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.500+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3915809778}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419296968}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807064718}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419296968}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=142036821}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.501+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56505772}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795902061}) 2024-09-15T03:41:12.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56505772}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253819862}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813485675}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371698996}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813485675}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.502+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3467862632}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.503+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.503+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992325535}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.503+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=659625599}) 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.503+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.503+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992325535}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.503+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111335833}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.504+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.504+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568482995}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.504+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284176433}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.504+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.504+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568482995}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.504+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515136023}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180907887}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2049968853}) 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180907887}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=300329690}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793830929}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418060306}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793830929}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.505+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648627648}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.506+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.506+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125684711}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.506+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340142084}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.506+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.506+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125684711}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.506+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978429639}) 2024-09-15T03:41:12.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853500617}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257756101}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853500617}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196758456}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658623968}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921305219}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.507+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658623968}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020316553}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090234411}) 2024-09-15T03:41:12.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274642069}) 2024-09-15T03:41:12.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090234411}) 2024-09-15T03:41:12.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.508+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859014663}) 2024-09-15T03:41:12.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.509+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.509+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=3258115298,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3758581048,mon_config_key=3725707551,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3539844690}) 2024-09-15T03:41:12.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.509+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=551313494,mdsmap=1531151339,mgr=3860624965,mgr_metadata=2280977772,mgrstat=1273426643,mon_config_key=2726416669,monmap=244636553,osd_metadata=4072215849,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1696860359}) 2024-09-15T03:41:12.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.509+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.509+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=3258115298,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3758581048,mon_config_key=3725707551,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3539844690}) 2024-09-15T03:41:12.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.510+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=32,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=1,osdmap=34} crc {logm=2943836154,mdsmap=1531151339,mgr=1013486720,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3405451607,mon_config_key=4117641549,monmap=3140514009,osd_metadata=1337438140,osd_pg_creating=1754493876,osd_snap=3823785925,osdmap=3575382440}) 2024-09-15T03:41:12.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.516+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.516+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=1812992213}) 2024-09-15T03:41:12.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.516+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=2600615150}) 2024-09-15T03:41:12.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.516+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:41:12.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.516+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=1812992213}) 2024-09-15T03:41:12.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:41:12.516+0000 7f4004df1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=79} crc {osdmap=622183386}) 2024-09-15T03:41:13.456 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:41:13.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.556+0000 7fb694d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.556+0000 7fb694d9d640 10 monclient: build_initial_monmap 2024-09-15T03:41:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.556+0000 7fb694d9d640 10 monclient: monmap: 2024-09-15T03:41:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:13.557139+0000 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:13.557139+0000 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:13.558 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.556+0000 7fb694d9d640 1 Processor -- start 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 1 -- start start 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient: init 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6900981c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb690131da0 0x7fb690136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb690131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb690132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69009ca60 con 0x7fb690132170 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69009cca0 con 0x7fb6900981c0 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb690131bd0 con 0x7fb690131da0 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb694d9d640 10 monclient(hunting): authenticate will time out at 1471.196802s 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb68ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb690132170 0x7fb6901390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41356/0 (socket says 172.21.15.153:41356) 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.557+0000 7fb68ed9d640 1 -- 172.21.15.153:0/4163683806 learned_addr learned my addr 172.21.15.153:0/4163683806 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 1 --2- 172.21.15.153:0/4163683806 >> v2:172.21.15.181:3300/0 conn(0x7fb690131da0 0x7fb690136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): get_auth_request con 0x7fb690131da0 auth_method 0 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_done global_id 144280 payload 274 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 1 -- 172.21.15.153:0/4163683806 >> v1:172.21.15.181:6789/0 conn(0x7fb6900981c0 legacy=0x7fb690098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 1 -- 172.21.15.153:0/4163683806 >> v1:172.21.15.153:6789/0 conn(0x7fb690132170 legacy=0x7fb6901390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 1 monclient: found mon.noname-c 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:41:13.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.558+0000 7fb68dd9b640 1 -- 172.21.15.153:0/4163683806 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69009e010 con 0x7fb690131da0 2024-09-15T03:41:13.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68dd9b640 1 -- 172.21.15.153:0/4163683806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb690131da0 msgr2=0x7fb690136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68dd9b640 1 --2- 172.21.15.153:0/4163683806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb690131da0 0x7fb690136b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb678009850 tx=0x7fb6780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a456c73de91749a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 1 -- 172.21.15.153:0/4163683806 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678032440 con 0x7fb690131da0 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: dump: 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.571 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: _finish_auth 0 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 5 monclient: authenticate success, global_id 144280 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 1 -- 172.21.15.153:0/4163683806 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6780325a0 con 0x7fb690131da0 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 1 -- 172.21.15.153:0/4163683806 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678032940 con 0x7fb690131da0 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 4 set_mon_vals no callback set 2024-09-15T03:41:13.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb68d59a640 10 monclient: dump: 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.573 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 10 monclient: shutdown 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 1 -- 172.21.15.153:0/4163683806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb690131da0 msgr2=0x7fb690136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.561+0000 7fb694d9d640 1 --2- 172.21.15.153:0/4163683806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb690131da0 0x7fb690136b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb678009850 tx=0x7fb6780090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 1 -- 172.21.15.153:0/4163683806 shutdown_connections 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 1 --2- 172.21.15.153:0/4163683806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb690131da0 0x7fb690136b70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 1 -- 172.21.15.153:0/4163683806 >> 172.21.15.153:0/4163683806 conn(0x7fb6900a1e30 msgr2=0x7fb6900a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:13.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 1 -- 172.21.15.153:0/4163683806 shutdown_connections 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 1 -- 172.21.15.153:0/4163683806 wait complete. 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 10 monclient: build_initial_monmap 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 10 monclient: monmap: 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:13.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.562+0000 7fb694d9d640 1 Processor -- start 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 -- start start 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient: init 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:13.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900981c0 0x7fb6901314b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6900981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 0x7fb6901380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb690131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 0x7fb6901385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb690132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:13.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb690098cb0 con 0x7fb690131da0 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900a0920 con 0x7fb6900981c0 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb690119810 con 0x7fb690132170 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb694d9d640 10 monclient(hunting): authenticate will time out at 1471.202477s 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 0x7fb6901380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 0x7fb6901385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 0x7fb6901380b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39042/0 (socket says 172.21.15.153:39042) 2024-09-15T03:41:13.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 1 -- 172.21.15.153:0/831504570 learned_addr learned my addr 172.21.15.153:0/831504570 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:13.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): get_auth_request con 0x7fb690131da0 auth_method 0 2024-09-15T03:41:13.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): get_auth_request con 0x7fb690132170 auth_method 0 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900981c0 0x7fb6901314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): get_auth_request con 0x7fb6900981c0 auth_method 0 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient(hunting): handle_auth_done global_id 124550 payload 824 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 -- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900981c0 msgr2=0x7fb6901314b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900981c0 0x7fb6901314b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 -- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 msgr2=0x7fb6901380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 0x7fb6901380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:13.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68ed9d640 1 -- 172.21.15.153:0/831504570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb678009390 con 0x7fb690132170 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.563+0000 7fb68dd9b640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 0x7fb6901380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb68ed9d640 1 -- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 msgr2=0x7fb6901385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb68ed9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 0x7fb6901385f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb68800cb60 tx=0x7fb6880052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800a850 con 0x7fb690132170 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: dump: 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: _finish_auth 0 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68800a9d0 con 0x7fb690132170 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 5 monclient: authenticate success, global_id 124550 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 10 monclient: _renew_subs 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69011a160 con 0x7fb690132170 2024-09-15T03:41:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800ad90 con 0x7fb690132170 2024-09-15T03:41:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb6867fc640 10 monclient: dump: 2024-09-15T03:41:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.586 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 10 monclient: _renew_subs 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb68cd99640 4 set_mon_vals no callback set 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.566+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb690139c70 con 0x7fb690132170 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb688007300 con 0x7fb690132170 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb694d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb694d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb694d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69004bf60 con 0x7fb690132170 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb6867fc640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb670038740 0x7fb67003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb68804c680 con 0x7fb690132170 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb68e59c640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb670038740 0x7fb67003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.567+0000 7fb68e59c640 10 monclient: get_auth_request con 0x7fb670038740 auth_method 0 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.570+0000 7fb68e59c640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb670038740 0x7fb67003abf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb680007fd0 tx=0x7fb680007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.572+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb6880075b0 con 0x7fb690132170 2024-09-15T03:41:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.572+0000 7fb6867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:13.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.572+0000 7fb6867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.785+0000 7fb694d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.786+0000 7fb694d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.786+0000 7fb694d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.786+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb690119950 con 0x7fb690132170 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.786+0000 7fb6867fc640 1 -- 172.21.15.153:0/831504570 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb688010af0 con 0x7fb690132170 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.786+0000 7fb6867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:13.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.786+0000 7fb6867fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:13.788 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:13.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb670038740 msgr2=0x7fb67003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb670038740 0x7fb67003abf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb680007fd0 tx=0x7fb680007e10 comp rx=0 tx=0).stop 2024-09-15T03:41:13.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 10 monclient: shutdown 2024-09-15T03:41:13.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 msgr2=0x7fb6901385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 0x7fb6901385f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb68800cb60 tx=0x7fb6880052b0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 shutdown_connections 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb670038740 0x7fb67003abf0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb690132170 0x7fb6901385f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb690131da0 0x7fb6901380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 --2- 172.21.15.153:0/831504570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900981c0 0x7fb6901314b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 >> 172.21.15.153:0/831504570 conn(0x7fb6900a1e30 msgr2=0x7fb6900a0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:13.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 shutdown_connections 2024-09-15T03:41:13.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.788+0000 7fb694d9d640 1 -- 172.21.15.153:0/831504570 wait complete. 2024-09-15T03:41:13.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:41:13.799 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:41:13.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:41:13.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.893+0000 7f8e8cb9d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.893+0000 7f8e8cb9d640 10 monclient: build_initial_monmap 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.893+0000 7f8e8cb9d640 10 monclient: monmap: 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:13.894520+0000 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:13.894520+0000 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 1 Processor -- start 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 1 -- start start 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 10 monclient: init 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e88098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e880981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.894+0000 7f8e8cb9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8e88131da0 0x7f8e88067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e88131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e88132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e88098ad0 con 0x7f8e88132170 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e88131bd0 con 0x7f8e88131da0 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8809ca60 con 0x7f8e880981c0 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8cb9d640 10 monclient(hunting): authenticate will time out at 1471.534061s 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e88098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e86d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8e88132170 0x7f8e88067650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41362/0 (socket says 172.21.15.153:41362) 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): get_auth_request con 0x7f8e880981c0 auth_method 0 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e86d9d640 1 -- 172.21.15.153:0/1840198302 learned_addr learned my addr 172.21.15.153:0/1840198302 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e85d9b640 1 --2- 172.21.15.153:0/1840198302 >> v2:172.21.15.181:3300/0 conn(0x7f8e88131da0 0x7f8e88067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e85d9b640 10 monclient(hunting): get_auth_request con 0x7f8e88131da0 auth_method 0 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient(hunting): handle_auth_done global_id 124553 payload 274 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 1 -- 172.21.15.153:0/1840198302 >> v2:172.21.15.181:3300/0 conn(0x7f8e88131da0 msgr2=0x7f8e88067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 1 --2- 172.21.15.153:0/1840198302 >> v2:172.21.15.181:3300/0 conn(0x7f8e88131da0 0x7f8e88067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 1 -- 172.21.15.153:0/1840198302 >> v1:172.21.15.153:6789/0 conn(0x7f8e88132170 legacy=0x7f8e88067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient: _finish_hunting 0 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 1 monclient: found mon.noname-b 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.895+0000 7f8e8659c640 1 -- 172.21.15.153:0/1840198302 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8809cca0 con 0x7f8e880981c0 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8659c640 1 -- 172.21.15.153:0/1840198302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 msgr2=0x7f8e88098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8659c640 1 --2- 172.21.15.153:0/1840198302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e88098590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009850 tx=0x7f8e780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32bad7fec68aa40c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 1 -- 172.21.15.153:0/1840198302 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7802adf0 con 0x7f8e880981c0 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: dump: 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: _finish_auth 0 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 5 monclient: authenticate success, global_id 124553 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 1 -- 172.21.15.153:0/1840198302 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e78032440 con 0x7f8e880981c0 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 1 -- 172.21.15.153:0/1840198302 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780327e0 con 0x7f8e880981c0 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 4 set_mon_vals no callback set 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8559a640 10 monclient: dump: 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 10 monclient: shutdown 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/1840198302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 msgr2=0x7f8e88098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/1840198302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e88098590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009850 tx=0x7f8e780090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/1840198302 shutdown_connections 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/1840198302 >> v2:172.21.15.181:3300/0 conn(0x7f8e88131da0 0x7f8e88067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/1840198302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e88098590 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.898+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/1840198302 >> 172.21.15.153:0/1840198302 conn(0x7f8e880a1e30 msgr2=0x7f8e880a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/1840198302 shutdown_connections 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/1840198302 wait complete. 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 10 monclient: build_initial_monmap 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 10 monclient: monmap: 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 1 Processor -- start 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 1 -- start start 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.899+0000 7f8e8cb9d640 10 monclient: init 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e8811def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e880981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 0x7f8e8811e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e88131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e88132170 0x7f8e8811e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e88132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881309a0 con 0x7f8e88131da0 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881202a0 con 0x7f8e88132170 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881313f0 con 0x7f8e880981c0 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8cb9d640 10 monclient(hunting): authenticate will time out at 1471.539277s 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 0x7f8e8811e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e8811def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 0x7f8e8811e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39050/0 (socket says 172.21.15.153:39050) 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e8811def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36552/0 (socket says 172.21.15.153:36552) 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 -- 172.21.15.153:0/707402249 learned_addr learned my addr 172.21.15.153:0/707402249 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): get_auth_request con 0x7f8e88131da0 auth_method 0 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): get_auth_request con 0x7f8e880981c0 auth_method 0 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e86d9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e88132170 0x7f8e8811e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e86d9d640 10 monclient(hunting): get_auth_request con 0x7f8e88132170 auth_method 0 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient(hunting): handle_auth_done global_id 164130 payload 824 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 msgr2=0x7f8e8811def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e8811def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e88132170 msgr2=0x7f8e8811e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e88132170 0x7f8e8811e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 monclient: found mon.noname-a 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e85d9b640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78009390 con 0x7f8e88131da0 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e86d9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e88132170 0x7f8e8811e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.900+0000 7f8e8659c640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e880981c0 0x7f8e8811def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.902+0000 7f8e85d9b640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 msgr2=0x7f8e8811e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.902+0000 7f8e85d9b640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 0x7f8e8811e430 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00d890 tx=0x7f8e7c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.902+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00de20 con 0x7f8e88131da0 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: dump: 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: _finish_auth 0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c00bae0 con 0x7f8e88131da0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 5 monclient: authenticate success, global_id 164130 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 10 monclient: _renew_subs 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e880a0b40 con 0x7f8e88131da0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00be80 con 0x7f8e88131da0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 10 monclient: dump: 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 10 monclient: _renew_subs 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8811f8a0 con 0x7f8e88131da0 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e84d99640 4 set_mon_vals no callback set 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8e7c014640 con 0x7f8e88131da0 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 0x7f8e8811a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.903+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88065f90 con 0x7f8e8811eeb0 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 0x7f8e8811a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient: get_auth_request con 0x7f8e8811eeb0 auth_method 0 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e767fc640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8e68038770 0x7f8e6803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f8e7c04cbe0 con 0x7f8e88131da0 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 10 monclient(hunting): handle_auth_done global_id 164133 payload 824 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e8659c640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e88098cb0 con 0x7f8e8811eeb0 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e86d9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8e68038770 0x7f8e6803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.904+0000 7f8e86d9d640 10 monclient: get_auth_request con 0x7f8e68038770 auth_method 0 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e8659c640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 msgr2=0x7f8e8811a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e8659c640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 0x7f8e8811a4a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e78034000 tx=0x7f8e78032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7803c440 con 0x7f8e8811eeb0 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 10 monclient: dump: 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e86d9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8e68038770 0x7f8e6803ac20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000e1b0 tx=0x7f8e80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8e7803c5a0 con 0x7f8e8811eeb0 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 msgr2=0x7f8e8811a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 0x7f8e8811a4a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e78034000 tx=0x7f8e78032000 comp rx=0 tx=0).stop 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 msgr2=0x7f8e8811a4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.906+0000 7f8e767fc640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811eeb0 0x7f8e8811a4a0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e78034000 tx=0x7f8e78032000 comp rx=0 tx=0).stop 2024-09-15T03:41:13.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8cb9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:13.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 0x7f8e8811ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:13.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8cb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:13.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8804bf60 con 0x7f8e8811a9e0 2024-09-15T03:41:13.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 0x7f8e8811ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient: get_auth_request con 0x7f8e8811a9e0 auth_method 0 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 10 monclient(hunting): handle_auth_done global_id 164136 payload 824 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.912+0000 7f8e8659c640 1 -- 172.21.15.153:0/707402249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8e88098ee0 con 0x7f8e8811a9e0 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8659c640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 msgr2=0x7f8e8811ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8659c640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 0x7f8e8811ade0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e88120180 tx=0x7f8e7803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780469c0 con 0x7f8e8811a9e0 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 10 monclient: dump: 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:13.931 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f8e78046b20 con 0x7f8e8811a9e0 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 msgr2=0x7f8e8811ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 0x7f8e8811ade0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e88120180 tx=0x7f8e7803d000 comp rx=0 tx=0).stop 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 msgr2=0x7f8e8811ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e767fc640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e8811a9e0 0x7f8e8811ade0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e88120180 tx=0x7f8e7803d000 comp rx=0 tx=0).stop 2024-09-15T03:41:13.932 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 130, 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:13.933 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:13.934 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:13.935 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:13.936 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:13.937 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:13.938 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:13.939 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:13.940 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:13.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:13.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:13.943 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:13.944 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:13.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:13.946 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:13.947 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8e68038770 msgr2=0x7f8e6803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8e68038770 0x7f8e6803ac20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000e1b0 tx=0x7f8e80009040 comp rx=0 tx=0).stop 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 10 monclient: shutdown 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 msgr2=0x7f8e8811e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 --2- 172.21.15.153:0/707402249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e88131da0 0x7f8e8811e430 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00d890 tx=0x7f8e7c008040 comp rx=0 tx=0).stop 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e86d9d640 1 -- 172.21.15.153:0/707402249 reap_dead start 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e86d9d640 1 -- 172.21.15.153:0/707402249 reap_dead start 2024-09-15T03:41:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 shutdown_connections 2024-09-15T03:41:13.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 >> 172.21.15.153:0/707402249 conn(0x7f8e880a1e30 msgr2=0x7f8e88096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:13.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 shutdown_connections 2024-09-15T03:41:13.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:13.915+0000 7f8e8cb9d640 1 -- 172.21.15.153:0/707402249 wait complete. 2024-09-15T03:41:13.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:41:13.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:41:13.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.044+0000 7f6cfab5d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.044+0000 7f6cfab5d640 10 monclient: build_initial_monmap 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.044+0000 7f6cfab5d640 10 monclient: monmap: 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:14.044851+0000 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:14.044851+0000 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:14.045 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.046 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:14.046 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:14.046 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:14.046 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:14.046 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:14.046 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.044+0000 7f6cfab5d640 1 Processor -- start 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.044+0000 7f6cfab5d640 1 -- start start 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.044+0000 7f6cfab5d640 10 monclient: init 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6cf41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cf409ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cf4136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf4131cc0 con 0x7f6cf4136c20 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf409f530 con 0x7f6cf409ea50 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf409fbf0 con 0x7f6cf41318f0 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cfab5d640 10 monclient(hunting): authenticate will time out at 1471.684466s 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf90fa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6cf4136c20 0x7f6cf4139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41374/0 (socket says 172.21.15.153:41374) 2024-09-15T03:41:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf90fa640 1 -- 172.21.15.153:0/332963500 learned_addr learned my addr 172.21.15.153:0/332963500 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): get_auth_request con 0x7f6cf41318f0 auth_method 0 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_done global_id 124559 payload 274 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 1 -- 172.21.15.153:0/332963500 >> v1:172.21.15.181:6789/0 conn(0x7f6cf409ea50 legacy=0x7f6cf409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:14.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 1 -- 172.21.15.153:0/332963500 >> v1:172.21.15.153:6789/0 conn(0x7f6cf4136c20 legacy=0x7f6cf4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:14.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient: _finish_hunting 0 2024-09-15T03:41:14.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 1 monclient: found mon.noname-b 2024-09-15T03:41:14.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:14.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.045+0000 7f6cf88f9640 1 -- 172.21.15.153:0/332963500 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf409fe30 con 0x7f6cf41318f0 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf88f9640 1 -- 172.21.15.153:0/332963500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 msgr2=0x7f6cf4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/332963500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4131f30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009880 tx=0x7f6ce40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861cdf968ac4fa1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 1 -- 172.21.15.153:0/332963500 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4032440 con 0x7f6cf41318f0 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: dump: 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.059 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: _finish_auth 0 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 1 -- 172.21.15.153:0/332963500 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce40325a0 con 0x7f6cf41318f0 2024-09-15T03:41:14.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cfab5d640 5 monclient: authenticate success, global_id 124559 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cfab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cfab5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 1 -- 172.21.15.153:0/332963500 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4032940 con 0x7f6cf41318f0 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cfab5d640 4 set_mon_vals no callback set 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.048+0000 7f6cf37fe640 10 monclient: dump: 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.061 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 10 monclient: shutdown 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 -- 172.21.15.153:0/332963500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 msgr2=0x7f6cf4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 --2- 172.21.15.153:0/332963500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4131f30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009880 tx=0x7f6ce40090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 -- 172.21.15.153:0/332963500 shutdown_connections 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 --2- 172.21.15.153:0/332963500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4131f30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 -- 172.21.15.153:0/332963500 >> 172.21.15.153:0/332963500 conn(0x7f6cf4067110 msgr2=0x7f6cf4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 -- 172.21.15.153:0/332963500 shutdown_connections 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 1 -- 172.21.15.153:0/332963500 wait complete. 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 10 monclient: build_initial_monmap 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.049+0000 7f6cfab5d640 10 monclient: monmap: 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:14.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 Processor -- start 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 -- start start 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient: init 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:41:14.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cf409ea50 0x7f6cf41361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6cf409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6cf41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cf4136c20 0x7f6cf4096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6cf4136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf409bf10 con 0x7f6cf4136c20 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf413a4f0 con 0x7f6cf409ea50 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40996f0 con 0x7f6cf41318f0 2024-09-15T03:41:14.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cfab5d640 10 monclient(hunting): authenticate will time out at 1471.689799s 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cf4136c20 0x7f6cf4096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cf4136c20 0x7f6cf4096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39078/0 (socket says 172.21.15.153:39078) 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 1 -- 172.21.15.153:0/4072140210 learned_addr learned my addr 172.21.15.153:0/4072140210 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 10 monclient(hunting): get_auth_request con 0x7f6cf4136c20 auth_method 0 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf90fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf3fff640 10 monclient(hunting): get_auth_request con 0x7f6cf41318f0 auth_method 0 2024-09-15T03:41:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.050+0000 7f6cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cf409ea50 0x7f6cf41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf88f9640 10 monclient(hunting): get_auth_request con 0x7f6cf409ea50 auth_method 0 2024-09-15T03:41:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 10 monclient(hunting): handle_auth_done global_id 124562 payload 824 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cf409ea50 msgr2=0x7f6cf41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cf409ea50 0x7f6cf41361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cf4136c20 msgr2=0x7f6cf4096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cf4136c20 0x7f6cf4096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 1 monclient: found mon.noname-c 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf3fff640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4009390 con 0x7f6cf41318f0 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf90fa640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cf4136c20 0x7f6cf4096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.051+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cf409ea50 0x7f6cf41361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf3fff640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 msgr2=0x7f6cf4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf3fff640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4096a60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800b490 tx=0x7f6ce8006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800f720 con 0x7f6cf41318f0 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: dump: 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.070 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce800f880 con 0x7f6cf41318f0 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800fc40 con 0x7f6cf41318f0 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cfab5d640 5 monclient: authenticate success, global_id 124562 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cfab5d640 10 monclient: _renew_subs 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cfab5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf4097e50 con 0x7f6cf41318f0 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf0ff9640 10 monclient: dump: 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.072 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.053+0000 7f6cf2ffd640 4 set_mon_vals no callback set 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 10 monclient: _renew_subs 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf4098a20 con 0x7f6cf41318f0 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 0x7f6cf40974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf4065e00 con 0x7f6cf4099b20 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6ce8011610 con 0x7f6cf41318f0 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.054+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 0x7f6cf40974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient: get_auth_request con 0x7f6cf4099b20 auth_method 0 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf0ff9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ce0038740 0x7f6ce003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_done global_id 124565 payload 824 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6ce804bb00 con 0x7f6cf41318f0 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf88f9640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cf4099ef0 con 0x7f6cf4099b20 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf90fa640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ce0038740 0x7f6ce003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.055+0000 7f6cf90fa640 10 monclient: get_auth_request con 0x7f6ce0038740 auth_method 0 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf88f9640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 msgr2=0x7f6cf40974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 0x7f6cf40974e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4034000 tx=0x7f6ce4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4031a80 con 0x7f6cf4099b20 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 10 monclient: dump: 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.075 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6ce4031be0 con 0x7f6cf4099b20 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 msgr2=0x7f6cf40974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 0x7f6cf40974e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4034000 tx=0x7f6ce4032000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 msgr2=0x7f6cf40974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.057+0000 7f6cf0ff9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf4099b20 0x7f6cf40974e0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4034000 tx=0x7f6ce4032000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.058+0000 7f6cf90fa640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ce0038740 0x7f6ce003abf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4136720 tx=0x7f6cec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cfab5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:14.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cfab5d640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 0x7f6cf4132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cfab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf404bf60 con 0x7f6cf41327d0 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 0x7f6cf4132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient: get_auth_request con 0x7f6cf41327d0 auth_method 0 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 10 monclient(hunting): handle_auth_done global_id 124568 payload 824 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.063+0000 7f6cf88f9640 1 -- 172.21.15.153:0/4072140210 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6cf40a32e0 con 0x7f6cf41327d0 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf88f9640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 msgr2=0x7f6cf4132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf88f9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 0x7f6cf4132bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097d0 tx=0x7f6ce403c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4048a50 con 0x7f6cf41327d0 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 10 monclient: dump: 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f6ce4048bb0 con 0x7f6cf41327d0 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:14.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 msgr2=0x7f6cf4132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 0x7f6cf4132bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097d0 tx=0x7f6ce403c000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 msgr2=0x7f6cf4132bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf0ff9640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41327d0 0x7f6cf4132bd0 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097d0 tx=0x7f6ce403c000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:14.082 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 130, 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:14.083 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:14.086 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:14.087 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:14.088 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.089 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.091 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:14.092 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:14.093 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:14.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.093 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:14.093 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.094 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:14.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:41:14.096 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:41:14.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.098 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.098 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.098 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:14.098 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.098 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.099 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:14.099 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ce0038740 msgr2=0x7f6ce003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6ce0038740 0x7f6ce003abf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4136720 tx=0x7f6cec009040 comp rx=0 tx=0).stop 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 10 monclient: shutdown 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 msgr2=0x7f6cf4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 1 --2- 172.21.15.153:0/4072140210 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cf41318f0 0x7f6cf4096a60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800b490 tx=0x7f6ce8006f50 comp rx=0 tx=0).stop 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf90fa640 1 -- 172.21.15.153:0/4072140210 reap_dead start 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cf90fa640 1 -- 172.21.15.153:0/4072140210 reap_dead start 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 shutdown_connections 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.066+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 >> 172.21.15.153:0/4072140210 conn(0x7f6cf4067110 msgr2=0x7f6cf40a4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.067+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 shutdown_connections 2024-09-15T03:41:14.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.067+0000 7f6cfab5d640 1 -- 172.21.15.153:0/4072140210 wait complete. 2024-09-15T03:41:14.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:41:14.103 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:41:14.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:41:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.197+0000 7fb013e0f640 10 monclient: get_monmap_and_config 2024-09-15T03:41:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.197+0000 7fb013e0f640 10 monclient: build_initial_monmap 2024-09-15T03:41:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.197+0000 7fb013e0f640 10 monclient: monmap: 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:14.198534+0000 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:14.198534+0000 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:14.199 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.198+0000 7fb013e0f640 1 Processor -- start 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.198+0000 7fb013e0f640 1 -- start start 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.198+0000 7fb013e0f640 10 monclient: init 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb00c0981c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb00c131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb00c132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c09ca60 con 0x7fb00c132170 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c09cca0 con 0x7fb00c0981c0 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c131bd0 con 0x7fb00c131da0 2024-09-15T03:41:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb013e0f640 10 monclient(hunting): authenticate will time out at 1471.838150s 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0123ac640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb00c132170 0x7fb00c1390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41376/0 (socket says 172.21.15.153:41376) 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0123ac640 1 -- 172.21.15.153:0/2905412482 learned_addr learned my addr 172.21.15.153:0/2905412482 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): get_auth_request con 0x7fb00c131da0 auth_method 0 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient(hunting): handle_auth_done global_id 124571 payload 274 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 1 -- 172.21.15.153:0/2905412482 >> v1:172.21.15.181:6789/0 conn(0x7fb00c0981c0 legacy=0x7fb00c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:14.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 1 -- 172.21.15.153:0/2905412482 >> v1:172.21.15.153:6789/0 conn(0x7fb00c132170 legacy=0x7fb00c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:14.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient: _finish_hunting 0 2024-09-15T03:41:14.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 1 monclient: found mon.noname-b 2024-09-15T03:41:14.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:14.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.199+0000 7fb0113aa640 1 -- 172.21.15.153:0/2905412482 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c09e010 con 0x7fb00c131da0 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb0113aa640 1 -- 172.21.15.153:0/2905412482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 msgr2=0x7fb00c136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb0113aa640 1 --2- 172.21.15.153:0/2905412482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c136b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb000009880 tx=0x7fb0000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d726a3bb76f7b3e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 1 -- 172.21.15.153:0/2905412482 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000032440 con 0x7fb00c131da0 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: dump: 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: _finish_auth 0 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb013e0f640 5 monclient: authenticate success, global_id 124571 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 1 -- 172.21.15.153:0/2905412482 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0000325a0 con 0x7fb00c131da0 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb013e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb013e0f640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb013e0f640 4 set_mon_vals no callback set 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 1 -- 172.21.15.153:0/2905412482 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000032940 con 0x7fb00c131da0 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.202+0000 7fb010ba9640 10 monclient: dump: 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 10 monclient: shutdown 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 -- 172.21.15.153:0/2905412482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 msgr2=0x7fb00c136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 --2- 172.21.15.153:0/2905412482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c136b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb000009880 tx=0x7fb0000090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 -- 172.21.15.153:0/2905412482 shutdown_connections 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 --2- 172.21.15.153:0/2905412482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c136b70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 -- 172.21.15.153:0/2905412482 >> 172.21.15.153:0/2905412482 conn(0x7fb00c0a1e30 msgr2=0x7fb00c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 -- 172.21.15.153:0/2905412482 shutdown_connections 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 -- 172.21.15.153:0/2905412482 wait complete. 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 10 monclient: build_initial_monmap 2024-09-15T03:41:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 10 monclient: monmap: 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.203+0000 7fb013e0f640 1 Processor -- start 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 -- start start 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient: init 2024-09-15T03:41:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c0981c0 0x7fb00c131490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb00c0981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): picked mon.noname-c con 0x7fb00c131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb00c132170 0x7fb00c1384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): picked mon.noname-a con 0x7fb00c132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c098cb0 con 0x7fb00c132170 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0a08f0 con 0x7fb00c0981c0 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c1196d0 con 0x7fb00c131da0 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb013e0f640 10 monclient(hunting): authenticate will time out at 1471.843540s 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c137f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36616/0 (socket says 172.21.15.153:36616) 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb00c132170 0x7fb00c1384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 1 -- 172.21.15.153:0/508956515 learned_addr learned my addr 172.21.15.153:0/508956515 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): get_auth_request con 0x7fb00c131da0 auth_method 0 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): get_auth_request con 0x7fb00c132170 auth_method 0 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb011bab640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c0981c0 0x7fb00c131490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb011bab640 10 monclient(hunting): get_auth_request con 0x7fb00c0981c0 auth_method 0 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb011bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb011bab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb011bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient(hunting): handle_auth_done global_id 164142 payload 824 2024-09-15T03:41:14.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 msgr2=0x7fb00c137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c0981c0 msgr2=0x7fb00c131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c0981c0 0x7fb00c131490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient: _finish_hunting 0 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 monclient: found mon.noname-a 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0123ac640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb000009390 con 0x7fb00c132170 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb011bab640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c0981c0 0x7fb00c131490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.204+0000 7fb0113aa640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb00c131da0 0x7fb00c137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb0123ac640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb00c132170 msgr2=0x7fb00c1384d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb0123ac640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb00c132170 0x7fb00c1384d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb00400cb60 tx=0x7fb0040052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00400a850 con 0x7fb00c132170 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: dump: 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.220 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: _finish_auth 0 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 5 monclient: authenticate success, global_id 164142 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00400a9d0 con 0x7fb00c132170 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00400ad90 con 0x7fb00c132170 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 10 monclient: _renew_subs 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c11a1f0 con 0x7fb00c132170 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffffff640 4 set_mon_vals no callback set 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7faffdffb640 10 monclient: dump: 2024-09-15T03:41:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 10 monclient: _renew_subs 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.207+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c1309d0 con 0x7fb00c132170 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb004007300 con 0x7fb00c132170 2024-09-15T03:41:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb013e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb013e0f640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 0x7fb00c138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb013e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c119590 con 0x7fb00c118490 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7faffdffb640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faff0038740 0x7faff003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb00404c680 con 0x7fb00c132170 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 0x7fb00c138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient: get_auth_request con 0x7fb00c118490 auth_method 0 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb0113aa640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faff0038740 0x7faff003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb0113aa640 10 monclient: get_auth_request con 0x7faff0038740 auth_method 0 2024-09-15T03:41:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 10 monclient(hunting): handle_auth_done global_id 144298 payload 824 2024-09-15T03:41:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.208+0000 7fb011bab640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb00c04bf60 con 0x7fb00c118490 2024-09-15T03:41:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7fb011bab640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 msgr2=0x7fb00c138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7fb011bab640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 0x7fb00c138a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e710 tx=0x7fb008009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7fb0113aa640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faff0038740 0x7faff003abf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb000035040 tx=0x7fb000032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800c820 con 0x7fb00c118490 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 10 monclient: dump: 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb00800c980 con 0x7fb00c118490 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 msgr2=0x7fb00c138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 0x7fb00c138a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e710 tx=0x7fb008009040 comp rx=0 tx=0).stop 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 msgr2=0x7fb00c138a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.211+0000 7faffdffb640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c118490 0x7fb00c138a10 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e710 tx=0x7fb008009040 comp rx=0 tx=0).stop 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb013e0f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb013e0f640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 0x7fb00c1349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb013e0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c065f90 con 0x7fb00c1345a0 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 0x7fb00c1349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient: get_auth_request con 0x7fb00c1345a0 auth_method 0 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.217+0000 7fb011bab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.218+0000 7fb011bab640 10 monclient(hunting): handle_auth_done global_id 144301 payload 824 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.218+0000 7fb011bab640 1 -- 172.21.15.153:0/508956515 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb00c098ee0 con 0x7fb00c1345a0 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.220+0000 7fb011bab640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 msgr2=0x7fb00c1349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.220+0000 7fb011bab640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 0x7fb00c1349a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb008002380 tx=0x7fb00801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.220+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008006c90 con 0x7fb00c1345a0 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.220+0000 7faffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.220+0000 7faffdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.220+0000 7faffdffb640 10 monclient: dump: 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fb008006df0 con 0x7fb00c1345a0 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 msgr2=0x7fb00c1349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 0x7fb00c1349a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb008002380 tx=0x7fb00801a000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 msgr2=0x7fb00c1349a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7faffdffb640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb00c1345a0 0x7fb00c1349a0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb008002380 tx=0x7fb00801a000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 130, 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:41:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:14.243 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:14.244 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:41:14.244 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:14.244 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:14.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:14.244 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:14.244 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faff0038740 msgr2=0x7faff003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7faff0038740 0x7faff003abf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb000035040 tx=0x7fb000032000 comp rx=0 tx=0).stop 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 10 monclient: shutdown 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb00c132170 msgr2=0x7fb00c1384d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 --2- 172.21.15.153:0/508956515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb00c132170 0x7fb00c1384d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb00400cb60 tx=0x7fb0040052b0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb0123ac640 1 -- 172.21.15.153:0/508956515 reap_dead start 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb0123ac640 1 -- 172.21.15.153:0/508956515 reap_dead start 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 shutdown_connections 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 >> 172.21.15.153:0/508956515 conn(0x7fb00c0a1e30 msgr2=0x7fb00c096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 shutdown_connections 2024-09-15T03:41:14.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.221+0000 7fb013e0f640 1 -- 172.21.15.153:0/508956515 wait complete. 2024-09-15T03:41:14.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:41:14.247 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:41:14.247 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:41:14.247 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:41:14.247 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:41:14.247 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:41:14.247 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:41:14.257 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:41:14.258 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:41:14.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.389+0000 7f043b29a640 10 monclient: get_monmap_and_config 2024-09-15T03:41:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.389+0000 7f043b29a640 10 monclient: build_initial_monmap 2024-09-15T03:41:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.389+0000 7f043b29a640 10 monclient: monmap: 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:14.389841+0000 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:14.389841+0000 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:14.392 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:14.392 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.389+0000 7f043b29a640 1 Processor -- start 2024-09-15T03:41:14.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.389+0000 7f043b29a640 1 -- start start 2024-09-15T03:41:14.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.389+0000 7f043b29a640 10 monclient: init 2024-09-15T03:41:14.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:14.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04341318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f043409ea50 0x7f043409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): picked mon.noname-c con 0x7f043409ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 0x7f0434139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): picked mon.noname-b con 0x7f0434136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0434132640 con 0x7f04341318f0 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0434131cc0 con 0x7f043409ea50 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043409fbf0 con 0x7f0434136c20 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f043b29a640 10 monclient(hunting): authenticate will time out at 1472.029432s 2024-09-15T03:41:14.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439036640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f04341318f0 legacy=0x7f0434131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 0x7f0434139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 0x7f0434139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36630/0 (socket says 172.21.15.153:36630) 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 -- 172.21.15.153:0/1219808411 learned_addr learned my addr 172.21.15.153:0/1219808411 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): get_auth_request con 0x7f0434136c20 auth_method 0 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 1 --2- 172.21.15.153:0/1219808411 >> v2:172.21.15.181:3300/0 conn(0x7f043409ea50 0x7f043409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): get_auth_request con 0x7f043409ea50 auth_method 0 2024-09-15T03:41:14.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0438835640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient(hunting): handle_auth_done global_id 124577 payload 274 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 -- 172.21.15.153:0/1219808411 >> v2:172.21.15.181:3300/0 conn(0x7f043409ea50 msgr2=0x7f043409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 --2- 172.21.15.153:0/1219808411 >> v2:172.21.15.181:3300/0 conn(0x7f043409ea50 0x7f043409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 -- 172.21.15.153:0/1219808411 >> v1:172.21.15.153:6789/0 conn(0x7f04341318f0 legacy=0x7f0434131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient: _finish_hunting 0 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 monclient: found mon.noname-b 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:14.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.390+0000 7f0439837640 1 -- 172.21.15.153:0/1219808411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043409fe30 con 0x7f0434136c20 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0439837640 1 -- 172.21.15.153:0/1219808411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 msgr2=0x7f0434139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0439837640 1 --2- 172.21.15.153:0/1219808411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 0x7f0434139000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f042c009910 tx=0x7f042c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22c9c2964ce5a857 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 1 -- 172.21.15.153:0/1219808411 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c02aea0 con 0x7f0434136c20 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: dump: 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: _finish_auth 0 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 1 -- 172.21.15.153:0/1219808411 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c032440 con 0x7f0434136c20 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f043b29a640 5 monclient: authenticate success, global_id 124577 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f043b29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f043b29a640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 1 -- 172.21.15.153:0/1219808411 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0327e0 con 0x7f0434136c20 2024-09-15T03:41:14.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f043b29a640 4 set_mon_vals no callback set 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.393+0000 7f0433fff640 10 monclient: dump: 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.406 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 10 monclient: shutdown 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 -- 172.21.15.153:0/1219808411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 msgr2=0x7f0434139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 --2- 172.21.15.153:0/1219808411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 0x7f0434139000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f042c009910 tx=0x7f042c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 -- 172.21.15.153:0/1219808411 shutdown_connections 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 --2- 172.21.15.153:0/1219808411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0434136c20 0x7f0434139000 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 --2- 172.21.15.153:0/1219808411 >> v2:172.21.15.181:3300/0 conn(0x7f043409ea50 0x7f043409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 -- 172.21.15.153:0/1219808411 >> 172.21.15.153:0/1219808411 conn(0x7f0434067110 msgr2=0x7f0434067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 -- 172.21.15.153:0/1219808411 shutdown_connections 2024-09-15T03:41:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 1 -- 172.21.15.153:0/1219808411 wait complete. 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 10 monclient: build_initial_monmap 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.394+0000 7f043b29a640 10 monclient: monmap: 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 Processor -- start 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 -- start start 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient: init 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f043409ea50 0x7f043411daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): picked mon.noname-a con 0x7f043409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 0x7f04341308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): picked mon.noname-c con 0x7f04341318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0434136c20 0x7f04340996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): picked mon.noname-b con 0x7f0434136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043411cdd0 con 0x7f043409ea50 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043409dce0 con 0x7f0434136c20 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043411c840 con 0x7f04341318f0 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f043b29a640 10 monclient(hunting): authenticate will time out at 1472.034800s 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0439036640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f043409ea50 msgr2=0x7f043411daa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:14.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0439036640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f043409ea50 0x7f043411daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 0x7f04341308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 0x7f04341308b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36634/0 (socket says 172.21.15.153:36634) 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 1 -- 172.21.15.153:0/3504349520 learned_addr learned my addr 172.21.15.153:0/3504349520 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): get_auth_request con 0x7f04341318f0 auth_method 0 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0438835640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.395+0000 7f0439837640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0434136c20 0x7f04340996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): get_auth_request con 0x7f0434136c20 auth_method 0 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0439837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 10 monclient(hunting): handle_auth_done global_id 124580 payload 824 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 1 -- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0434136c20 msgr2=0x7f04340996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0434136c20 0x7f04340996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 1 -- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f043409ea50 msgr2=0x7f043411daa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f043409ea50 0x7f043411daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 10 monclient: _finish_hunting 0 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 1 monclient: found mon.noname-c 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.396+0000 7f0438835640 1 -- 172.21.15.153:0/3504349520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c009450 con 0x7f04341318f0 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f0438835640 1 -- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 msgr2=0x7f04341308b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:14.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f0438835640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 0x7f04341308b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f042800ea20 tx=0x7f04280090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042800cc20 con 0x7f04341318f0 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: dump: 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: _finish_auth 0 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042800cd80 con 0x7f04341318f0 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 5 monclient: authenticate success, global_id 124580 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 10 monclient: _renew_subs 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043411dfe0 con 0x7f04341318f0 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428013680 con 0x7f04341318f0 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:14.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04317fa640 10 monclient: dump: 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:14.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 10 monclient: _renew_subs 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f04337fe640 4 set_mon_vals no callback set 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.398+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0434130f90 con 0x7f04341318f0 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0428013a70 con 0x7f04341318f0 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f043b29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f043b29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f043b29a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043404bf60 con 0x7f04341318f0 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f04317fa640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0420038720 0x7f042003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:14.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f042804ca30 con 0x7f04341318f0 2024-09-15T03:41:14.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f0439036640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0420038720 0x7f042003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:14.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.399+0000 7f0439036640 10 monclient: get_auth_request con 0x7f0420038720 auth_method 0 2024-09-15T03:41:14.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.402+0000 7f0439036640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0420038720 0x7f042003abd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0424003cc0 tx=0x7f0424009310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:14.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.404+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f04280123d0 con 0x7f04341318f0 2024-09-15T03:41:14.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.404+0000 7f04317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:14.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.404+0000 7f04317fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:14.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f043b29a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:14.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f043b29a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:14.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f043b29a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:14.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f04341316d0 con 0x7f04341318f0 2024-09-15T03:41:14.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f04317fa640 1 -- 172.21.15.153:0/3504349520 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f0428023760 con 0x7f04341318f0 2024-09-15T03:41:14.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f04317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:14.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.619+0000 7f04317fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:14.621 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:14.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0420038720 msgr2=0x7f042003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0420038720 0x7f042003abd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0424003cc0 tx=0x7f0424009310 comp rx=0 tx=0).stop 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 10 monclient: shutdown 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 msgr2=0x7f04341308b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 0x7f04341308b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f042800ea20 tx=0x7f04280090f0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 shutdown_connections 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0420038720 0x7f042003abd0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0434136c20 0x7f04340996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04341318f0 0x7f04341308b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 --2- 172.21.15.153:0/3504349520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f043409ea50 0x7f043411daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:14.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 >> 172.21.15.153:0/3504349520 conn(0x7f0434067110 msgr2=0x7f04340a3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:14.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 shutdown_connections 2024-09-15T03:41:14.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:14.621+0000 7f043b29a640 1 -- 172.21.15.153:0/3504349520 wait complete. 2024-09-15T03:41:17.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:17.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.737+0000 7f78bf69a640 10 monclient: get_monmap_and_config 2024-09-15T03:41:17.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.737+0000 7f78bf69a640 10 monclient: build_initial_monmap 2024-09-15T03:41:17.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.737+0000 7f78bf69a640 10 monclient: monmap: 2024-09-15T03:41:17.738 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:17.738609+0000 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:17.738609+0000 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:17.739 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:17.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.738+0000 7f78bf69a640 1 Processor -- start 2024-09-15T03:41:17.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.738+0000 7f78bf69a640 1 -- start start 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.738+0000 7f78bf69a640 10 monclient: init 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78b809c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f78b809e840 0x7f78b809aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): picked mon.noname-c con 0x7f78b809e840 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78b8136cb0 0x7f78b8139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): picked mon.noname-a con 0x7f78b8136cb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b809ec40 con 0x7f78b809c860 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b806a320 con 0x7f78b8136cb0 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:17.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b8068a20 con 0x7f78b809e840 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bf69a640 10 monclient(hunting): authenticate will time out at 1475.378322s 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bdc37640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f78b8136cb0 msgr2=0x7f78b8139090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bdc37640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f78b8136cb0 0x7f78b8139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f78b809e840 0x7f78b809aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f78b809e840 0x7f78b809aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:55554/0 (socket says 172.21.15.153:55554) 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 -- 172.21.15.153:0/606168832 learned_addr learned my addr 172.21.15.153:0/606168832 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bd436640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f78b809c860 0x7f78b809cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:50492/0 (socket says 172.21.15.153:50492) 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): get_auth_request con 0x7f78b809e840 auth_method 0 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:17.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient(hunting): handle_auth_done global_id 144310 payload 274 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 -- 172.21.15.153:0/606168832 >> v2:172.21.15.153:3300/0 conn(0x7f78b8136cb0 msgr2=0x7f78b8139090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 --2- 172.21.15.153:0/606168832 >> v2:172.21.15.153:3300/0 conn(0x7f78b8136cb0 0x7f78b8139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 -- 172.21.15.153:0/606168832 >> v1:172.21.15.181:6789/0 conn(0x7f78b809c860 legacy=0x7f78b809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient: _finish_hunting 0 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 monclient: found mon.noname-c 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:41:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.739+0000 7f78bcc35640 1 -- 172.21.15.153:0/606168832 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b8068e60 con 0x7f78b809e840 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bcc35640 1 -- 172.21.15.153:0/606168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 msgr2=0x7f78b809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bcc35640 1 --2- 172.21.15.153:0/606168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b809aba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78b806b700 tx=0x7f78a80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8194405351c76173 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 1 -- 172.21.15.153:0/606168832 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8032440 con 0x7f78b809e840 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: dump: 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: _finish_auth 0 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 1 -- 172.21.15.153:0/606168832 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a80325a0 con 0x7f78b809e840 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 5 monclient: authenticate success, global_id 144310 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 1 -- 172.21.15.153:0/606168832 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8032940 con 0x7f78b809e840 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 4 set_mon_vals no callback set 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78affff640 10 monclient: dump: 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 10 monclient: shutdown 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 1 -- 172.21.15.153:0/606168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 msgr2=0x7f78b809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 1 --2- 172.21.15.153:0/606168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b809aba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78b806b700 tx=0x7f78a80091d0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 1 -- 172.21.15.153:0/606168832 shutdown_connections 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 1 --2- 172.21.15.153:0/606168832 >> v2:172.21.15.153:3300/0 conn(0x7f78b8136cb0 0x7f78b8139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.743+0000 7f78bf69a640 1 --2- 172.21.15.153:0/606168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b809aba0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 1 -- 172.21.15.153:0/606168832 >> 172.21.15.153:0/606168832 conn(0x7f78b80a2e40 msgr2=0x7f78b80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 1 -- 172.21.15.153:0/606168832 shutdown_connections 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 1 -- 172.21.15.153:0/606168832 wait complete. 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 10 monclient: build_initial_monmap 2024-09-15T03:41:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 10 monclient: monmap: 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 1 Processor -- start 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 1 -- start start 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.744+0000 7f78bf69a640 10 monclient: init 2024-09-15T03:41:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b809c860 0x7f78b806e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): picked mon.noname-a con 0x7f78b809c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b811d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): picked mon.noname-b con 0x7f78b809e840 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 0x7f78b811ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): picked mon.noname-c con 0x7f78b8136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b811fb40 con 0x7f78b809c860 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80a1150 con 0x7f78b809e840 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b811ea10 con 0x7f78b8136cb0 2024-09-15T03:41:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bf69a640 10 monclient(hunting): authenticate will time out at 1475.384543s 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bd436640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b809c860 msgr2=0x7f78b806e2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bd436640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b809c860 0x7f78b806e2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 0x7f78b811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 0x7f78b811ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50808/0 (socket says 172.21.15.153:50808) 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 1 -- 172.21.15.153:0/2071176782 learned_addr learned my addr 172.21.15.153:0/2071176782 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): get_auth_request con 0x7f78b8136cb0 auth_method 0 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b811d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): get_auth_request con 0x7f78b809e840 auth_method 0 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bcc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.745+0000 7f78bdc37640 10 monclient(hunting): handle_auth_done global_id 124583 payload 824 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 1 -- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 msgr2=0x7f78b811d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b811d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 1 -- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b809c860 msgr2=0x7f78b806e2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b809c860 0x7f78b806e2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 10 monclient: _finish_hunting 0 2024-09-15T03:41:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 1 monclient: found mon.noname-c 2024-09-15T03:41:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bdc37640 1 -- 172.21.15.153:0/2071176782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8009450 con 0x7f78b8136cb0 2024-09-15T03:41:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.746+0000 7f78bcc35640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b811d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bdc37640 1 -- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 msgr2=0x7f78b811ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bdc37640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 0x7f78b811ddf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f78b000bf80 tx=0x7f78b0009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b000cba0 con 0x7f78b8136cb0 2024-09-15T03:41:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: dump: 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: _finish_auth 0 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 5 monclient: authenticate success, global_id 124583 2024-09-15T03:41:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b000cd20 con 0x7f78b8136cb0 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 10 monclient: _renew_subs 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b811f860 con 0x7f78b8136cb0 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0016430 con 0x7f78b8136cb0 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78ad7fa640 10 monclient: dump: 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78af7fe640 4 set_mon_vals no callback set 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 10 monclient: _renew_subs 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.748+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b8136980 con 0x7f78b8136cb0 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f78b0012020 con 0x7f78b8136cb0 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78bf69a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78bf69a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78bf69a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b804bf60 con 0x7f78b8136cb0 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78ad7fa640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f789c0386f0 0x7f789c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.749+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f78b004c360 con 0x7f78b8136cb0 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.750+0000 7f78bd436640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f789c0386f0 0x7f789c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.750+0000 7f78bd436640 10 monclient: get_auth_request con 0x7f789c0386f0 auth_method 0 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.752+0000 7f78bd436640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f789c0386f0 0x7f789c03aba0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f78b40048d0 tx=0x7f78b4007ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.754+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f78b0014940 con 0x7f78b8136cb0 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.754+0000 7f78ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.754+0000 7f78ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78bf69a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78bf69a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78bf69a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78b811fd80 con 0x7f78b8136cb0 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78ad7fa640 1 -- 172.21.15.153:0/2071176782 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f78b00132e0 con 0x7f78b8136cb0 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.968+0000 7f78ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:17.969 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:17.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f789c0386f0 msgr2=0x7f789c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:17.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f789c0386f0 0x7f789c03aba0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f78b40048d0 tx=0x7f78b4007ac0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 10 monclient: shutdown 2024-09-15T03:41:17.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:17.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 msgr2=0x7f78b811ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:17.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 0x7f78b811ddf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f78b000bf80 tx=0x7f78b0009700 comp rx=0 tx=0).stop 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 shutdown_connections 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f789c0386f0 0x7f789c03aba0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b8136cb0 0x7f78b811ddf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78b809e840 0x7f78b811d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 --2- 172.21.15.153:0/2071176782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b809c860 0x7f78b806e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 >> 172.21.15.153:0/2071176782 conn(0x7f78b80a2e40 msgr2=0x7f78b809f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 shutdown_connections 2024-09-15T03:41:17.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:17.970+0000 7f78bf69a640 1 -- 172.21.15.153:0/2071176782 wait complete. 2024-09-15T03:41:20.983 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.087+0000 7f644ea89640 10 monclient: get_monmap_and_config 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.087+0000 7f644ea89640 10 monclient: build_initial_monmap 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.087+0000 7f644ea89640 10 monclient: monmap: 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:21.087975+0000 2024-09-15T03:41:21.088 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:21.087975+0000 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:21.089 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 1 Processor -- start 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 1 -- start start 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient: init 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64480a5760 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f644809ce40 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f644809f010 0x7f644813a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient(hunting): picked mon.noname-c con 0x7f644809f010 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:21.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480a1080 con 0x7f64480a5760 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480a5b30 con 0x7f644809ce40 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644809d240 con 0x7f644809f010 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644ea89640 10 monclient(hunting): authenticate will time out at 1478.727595s 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644c825640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f64480a5760 legacy=0x7f644809e900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644d026640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f644809f010 0x7f644813a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f644d026640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f644809f010 0x7f644813a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:55572/0 (socket says 172.21.15.153:55572) 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 1 -- 172.21.15.153:0/3202667671 learned_addr learned my addr 172.21.15.153:0/3202667671 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.088+0000 7f6447fff640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f644809ce40 0x7f6448136b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:50504/0 (socket says 172.21.15.153:50504) 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): get_auth_request con 0x7f644809f010 auth_method 0 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient(hunting): handle_auth_done global_id 144316 payload 274 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 1 -- 172.21.15.153:0/3202667671 >> v1:172.21.15.181:6789/0 conn(0x7f644809ce40 legacy=0x7f6448136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 1 -- 172.21.15.153:0/3202667671 >> v1:172.21.15.153:6789/0 conn(0x7f64480a5760 legacy=0x7f644809e900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient: _finish_hunting 0 2024-09-15T03:41:21.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 1 monclient: found mon.noname-c 2024-09-15T03:41:21.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:21.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:41:21.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.089+0000 7f644d026640 1 -- 172.21.15.153:0/3202667671 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644809d4c0 con 0x7f644809f010 2024-09-15T03:41:21.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644d026640 1 -- 172.21.15.153:0/3202667671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 msgr2=0x7f644813a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:21.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644d026640 1 --2- 172.21.15.153:0/3202667671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f644813a290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f644809d6b0 tx=0x7f6440009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92d720d68802b78 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:21.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 1 -- 172.21.15.153:0/3202667671 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644002aea0 con 0x7f644809f010 2024-09-15T03:41:21.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:21.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: dump: 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: _finish_auth 0 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 1 -- 172.21.15.153:0/3202667671 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644002d050 con 0x7f644809f010 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 5 monclient: authenticate success, global_id 144316 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 1 -- 172.21.15.153:0/3202667671 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440007840 con 0x7f644809f010 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 4 set_mon_vals no callback set 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f64477fe640 10 monclient: dump: 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 10 monclient: shutdown 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 1 -- 172.21.15.153:0/3202667671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 msgr2=0x7f644813a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.092+0000 7f644ea89640 1 --2- 172.21.15.153:0/3202667671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f644813a290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f644809d6b0 tx=0x7f6440009840 comp rx=0 tx=0).stop 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 1 -- 172.21.15.153:0/3202667671 shutdown_connections 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 1 --2- 172.21.15.153:0/3202667671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f644813a290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 1 -- 172.21.15.153:0/3202667671 >> 172.21.15.153:0/3202667671 conn(0x7f6448067680 msgr2=0x7f64480a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 1 -- 172.21.15.153:0/3202667671 shutdown_connections 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 1 -- 172.21.15.153:0/3202667671 wait complete. 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 10 monclient: build_initial_monmap 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 10 monclient: monmap: 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:21.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:21.108 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:21.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.093+0000 7f644ea89640 1 Processor -- start 2024-09-15T03:41:21.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 -- start start 2024-09-15T03:41:21.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient: init 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f644809ce40 0x7f6448140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): picked mon.noname-a con 0x7f644809ce40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f6448131bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): picked mon.noname-b con 0x7f644809f010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 0x7f644811c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): picked mon.noname-c con 0x7f64480a5760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644c825640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f644809ce40 msgr2=0x7f6448140e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644811e240 con 0x7f644809ce40 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:21.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644811e730 con 0x7f644809f010 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644811ffc0 con 0x7f64480a5760 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644ea89640 10 monclient(hunting): authenticate will time out at 1478.733656s 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644c825640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f644809ce40 0x7f6448140e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 0x7f644811c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 0x7f644811c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50820/0 (socket says 172.21.15.153:50820) 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 1 -- 172.21.15.153:0/4138730000 learned_addr learned my addr 172.21.15.153:0/4138730000 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): get_auth_request con 0x7f64480a5760 auth_method 0 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f6448131bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:21.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f644d026640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): get_auth_request con 0x7f644809f010 auth_method 0 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.094+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 10 monclient(hunting): handle_auth_done global_id 124586 payload 824 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 1 -- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 msgr2=0x7f6448131bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f6448131bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 1 -- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f644809ce40 msgr2=0x7f6448140e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f644809ce40 0x7f6448140e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 10 monclient: _finish_hunting 0 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 1 monclient: found mon.noname-c 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.095+0000 7f644d026640 1 -- 172.21.15.153:0/4138730000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6440009ae0 con 0x7f64480a5760 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644d026640 1 -- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 msgr2=0x7f644811c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644d026640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 0x7f644811c740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f644002ac20 tx=0x7f6440036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440031cb0 con 0x7f64480a5760 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: dump: 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: _finish_auth 0 2024-09-15T03:41:21.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 5 monclient: authenticate success, global_id 124586 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6440031e10 con 0x7f64480a5760 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 10 monclient: _renew_subs 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6448141c60 con 0x7f64480a5760 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644003c670 con 0x7f64480a5760 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6444ff9640 10 monclient: dump: 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:21.114 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f6446ffd640 4 set_mon_vals no callback set 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 10 monclient: _renew_subs 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.097+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644811ed80 con 0x7f64480a5760 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f644003ca20 con 0x7f64480a5760 2024-09-15T03:41:21.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f644ea89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f644ea89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f644ea89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644804bf60 con 0x7f64480a5760 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f6444ff9640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f642c0386f0 0x7f642c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.098+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f644006d1e0 con 0x7f64480a5760 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.099+0000 7f644c825640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f642c0386f0 0x7f642c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.099+0000 7f644c825640 10 monclient: get_auth_request con 0x7f642c0386f0 auth_method 0 2024-09-15T03:41:21.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.101+0000 7f644c825640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f642c0386f0 0x7f642c03aba0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f643c003940 tx=0x7f643c006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:21.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.103+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6440044400 con 0x7f64480a5760 2024-09-15T03:41:21.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.103+0000 7f6444ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:21.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.103+0000 7f6444ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:21.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.315+0000 7f644ea89640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.315+0000 7f644ea89640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.315+0000 7f644ea89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.315+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f64480638d0 con 0x7f64480a5760 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.316+0000 7f6444ff9640 1 -- 172.21.15.153:0/4138730000 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f6440030e60 con 0x7f64480a5760 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.316+0000 7f6444ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.316+0000 7f6444ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:21.317 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f642c0386f0 msgr2=0x7f642c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f642c0386f0 0x7f642c03aba0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f643c003940 tx=0x7f643c006d80 comp rx=0 tx=0).stop 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 10 monclient: shutdown 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 msgr2=0x7f644811c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 0x7f644811c740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f644002ac20 tx=0x7f6440036000 comp rx=0 tx=0).stop 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 shutdown_connections 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f642c0386f0 0x7f642c03aba0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64480a5760 0x7f644811c740 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f644809f010 0x7f6448131bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 --2- 172.21.15.153:0/4138730000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f644809ce40 0x7f6448140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:21.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 >> 172.21.15.153:0/4138730000 conn(0x7f6448067680 msgr2=0x7f64480a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:21.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 shutdown_connections 2024-09-15T03:41:21.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:21.318+0000 7f644ea89640 1 -- 172.21.15.153:0/4138730000 wait complete. 2024-09-15T03:41:24.329 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:24.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.432+0000 7f0fcf434640 10 monclient: get_monmap_and_config 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.432+0000 7f0fcf434640 10 monclient: build_initial_monmap 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.433+0000 7f0fcf434640 10 monclient: monmap: 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:24.433756+0000 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:24.433756+0000 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:24.434 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:24.435 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:24.435 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.433+0000 7f0fcf434640 1 Processor -- start 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.433+0000 7f0fcf434640 1 -- start start 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.433+0000 7f0fcf434640 10 monclient: init 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc809c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fc809aba0 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fc809af70 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc806a320 con 0x7f0fc809aba0 2024-09-15T03:41:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc8068a20 con 0x7f0fc809af70 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc809a9d0 con 0x7f0fc809c860 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcf434640 10 monclient(hunting): authenticate will time out at 1482.073457s 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcc9cf640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0fc809aba0 legacy=0x7f0fc809d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50830/0 (socket says 172.21.15.153:50830) 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 learned_addr learned my addr 172.21.15.153:0/2382114257 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request con 0x7f0fc809c860 auth_method 0 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:24.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:24.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:24.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.434+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:24.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.437+0000 7f0fcd9d1640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.181:6789/0 conn(0x7f0fc809af70 legacy=0x7f0fc809df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:24.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fc809aba0 legacy=0x7f0fc809d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request con 0x7f0fc809c860 auth_method 0 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:24.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:24.635+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fc809aba0 legacy=0x7f0fc809d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request con 0x7f0fc809c860 auth_method 0 2024-09-15T03:41:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:41:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.036+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fc809aba0 legacy=0x7f0fc809d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request con 0x7f0fc809c860 auth_method 0 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:25.837+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): tick 2024-09-15T03:41:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:41:27.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 msgr2=0x7f0fc809cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc809c860 0x7f0fc809cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.181:6789/0 conn(0x7f0fc809af70 legacy=0x7f0fc809df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fc809aba0 legacy=0x7f0fc809d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient: _add_conns ranks=[3,4] 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 0x7f0fac005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0fac003730 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0fac006270 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:27.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc809a9d0 con 0x7f0fac003730 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fac008de0 con 0x7f0fac006270 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fbeffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 msgr2=0x7f0fac005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.434+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 0x7f0fac005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006270 auth_method 0 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:27.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:41:27.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:27.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:27.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.635+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 msgr2=0x7f0fac005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.635+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 0x7f0fac005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006270 auth_method 0 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:27.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:28.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.036+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 msgr2=0x7f0fac005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:28.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.036+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 0x7f0fac005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:28.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:28.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006270 auth_method 0 2024-09-15T03:41:28.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:28.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:28.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:28.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:28.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:28.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:28.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.037+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:28.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.837+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 msgr2=0x7f0fac005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:28.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.837+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 0x7f0fac005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:28.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.837+0000 7f0fbffff640 1 -- 172.21.15.153:0/2382114257 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:41:28.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:28.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006270 auth_method 0 2024-09-15T03:41:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:28.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:28.839+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 10 monclient(hunting): tick 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 msgr2=0x7f0fac008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006270 0x7f0fac008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 msgr2=0x7f0fac005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac003730 0x7f0fac005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.434+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 0x7f0fac00b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:30.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0fac009170 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcd9d1640 1 -- 172.21.15.153:0/2382114257 reap_dead start 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0fac006710 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fac0040b0 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc8068e60 con 0x7f0fac0040b0 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc809a9d0 con 0x7f0fac009170 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fac008de0 con 0x7f0fac006710 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fbeffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:30.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 msgr2=0x7f0fac00b550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 0x7f0fac00b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006710 auth_method 0 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:30.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:30.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.437+0000 7f0fcd9d1640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.181:6789/0 conn(0x7f0fac0040b0 legacy=0x7f0fac004550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:30.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 msgr2=0x7f0fac00b550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:30.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 0x7f0fac00b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006710 auth_method 0 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:30.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:30.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 msgr2=0x7f0fac00b550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 0x7f0fac00b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006710 auth_method 0 2024-09-15T03:41:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:31.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:31.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:31.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:31.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.037+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:31.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 msgr2=0x7f0fac00b550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:31.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 0x7f0fac00b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:31.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:31.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac006710 auth_method 0 2024-09-15T03:41:31.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:31.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:31.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:31.838+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient(hunting): tick 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 msgr2=0x7f0fac003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.181:3300/0 conn(0x7f0fac006710 0x7f0fac003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 msgr2=0x7f0fac00b550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> v2:172.21.15.153:3300/0 conn(0x7f0fac009170 0x7f0fac00b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.181:6789/0 conn(0x7f0fac0040b0 legacy=0x7f0fac004550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient: _add_conns ranks=[0,2] 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fac00db40 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0fac011670 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc806a320 con 0x7f0fac00db40 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 1 -- 172.21.15.153:0/2382114257 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc809a9d0 con 0x7f0fac011670 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fbeffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fac00db40 legacy=0x7f0fac00ff20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac011670 auth_method 0 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 msgr2=0x7f0fac013a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 msgr2=0x7f0fac013a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.435+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fac00db40 legacy=0x7f0fac00ff20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request con 0x7f0fac011670 auth_method 0 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient(hunting): handle_auth_done global_id 124589 payload 274 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> v1:172.21.15.153:6789/0 conn(0x7f0fac00db40 legacy=0x7f0fac00ff20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient: _finish_hunting 0 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 1 monclient: found mon.noname-b 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.636+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac015250 con 0x7f0fac011670 2024-09-15T03:41:33.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fcc9cf640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 msgr2=0x7f0fac013a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:33.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0fb801b450 tx=0x7f0fb8005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5326390a9868052 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:33.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 1 -- 172.21.15.153:0/2382114257 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb802d9b0 con 0x7f0fac011670 2024-09-15T03:41:33.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 10 monclient: dump: 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:33.642 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 10 monclient: _finish_auth 0 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 1 -- 172.21.15.153:0/2382114257 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8031070 con 0x7f0fac011670 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fcf434640 5 monclient: authenticate success, global_id 124589 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fcf434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.639+0000 7f0fbffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 4 set_mon_vals no callback set 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fbffff640 1 -- 172.21.15.153:0/2382114257 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb802dd40 con 0x7f0fac011670 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fbffff640 10 monclient: dump: 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:33.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 10 monclient: shutdown 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 1 -- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 msgr2=0x7f0fac013a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:33.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 1 --2- 172.21.15.153:0/2382114257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fac011670 0x7f0fac013a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0fb801b450 tx=0x7f0fb8005040 comp rx=0 tx=0).stop 2024-09-15T03:41:33.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcd9d1640 1 -- 172.21.15.153:0/2382114257 reap_dead start 2024-09-15T03:41:33.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 1 -- 172.21.15.153:0/2382114257 shutdown_connections 2024-09-15T03:41:33.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.640+0000 7f0fcf434640 1 -- 172.21.15.153:0/2382114257 >> 172.21.15.153:0/2382114257 conn(0x7f0fc80a2e40 msgr2=0x7f0fc80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:33.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.641+0000 7f0fcf434640 1 -- 172.21.15.153:0/2382114257 shutdown_connections 2024-09-15T03:41:33.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.641+0000 7f0fcf434640 1 -- 172.21.15.153:0/2382114257 wait complete. 2024-09-15T03:41:33.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.641+0000 7f0fcf434640 10 monclient: build_initial_monmap 2024-09-15T03:41:33.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.641+0000 7f0fcf434640 10 monclient: monmap: 2024-09-15T03:41:33.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 Processor -- start 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 -- start start 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient: init 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 0x7f0fc8131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc8140ff0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fc81320d0 0x7f0fc806d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc81320d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc806dc40 0x7f0fc806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc806dc40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc811f0b0 con 0x7f0fc806dc40 2024-09-15T03:41:33.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81324c0 con 0x7f0fc81320d0 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.642+0000 7f0fcf434640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc811f5e0 con 0x7f0fc8140ff0 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcf434640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcf434640 10 monclient(hunting): authenticate will time out at 1491.282011s 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd9d1640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc806dc40 msgr2=0x7f0fc806e0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 0x7f0fc8131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd9d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc806dc40 0x7f0fc806e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 0x7f0fc8131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:42884/0 (socket says 172.21.15.153:42884) 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/3004439628 learned_addr learned my addr 172.21.15.153:0/3004439628 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request con 0x7f0fc8140ff0 auth_method 0 2024-09-15T03:41:33.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fc81320d0 0x7f0fc806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient(hunting): handle_auth_done global_id 124592 payload 824 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fc81320d0 msgr2=0x7f0fc806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fc81320d0 0x7f0fc806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc806dc40 msgr2=0x7f0fc806e0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc806dc40 0x7f0fc806e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient: _finish_hunting 0 2024-09-15T03:41:33.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 monclient: found mon.noname-c 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.643+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/3004439628 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8009d50 con 0x7f0fc8140ff0 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fcd1d0640 1 -- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 msgr2=0x7f0fc8131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fcd1d0640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 0x7f0fc8131b90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40149a0 tx=0x7f0fc4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4017890 con 0x7f0fc8140ff0 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: dump: 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.655 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:33.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc400c040 con 0x7f0fc8140ff0 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fcf434640 5 monclient: authenticate success, global_id 124592 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fcf434640 10 monclient: _renew_subs 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fcf434640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc8134a70 con 0x7f0fc8140ff0 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4017c40 con 0x7f0fc8140ff0 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbd7fa640 10 monclient: dump: 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.657 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.645+0000 7f0fbf7fe640 4 set_mon_vals no callback set 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.646+0000 7f0fcf434640 10 monclient: _renew_subs 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.646+0000 7f0fcf434640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.646+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc8141d50 con 0x7f0fc8140ff0 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.646+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0fc4020450 con 0x7f0fc8140ff0 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fbd7fa640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fac03a550 0x7f0fac03ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0fc4000e40 con 0x7f0fc8140ff0 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fcf434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fcf434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fcf434640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc8099cc0 con 0x7f0fc8140ff0 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fac03a550 0x7f0fac03ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.647+0000 7f0fcc9cf640 10 monclient: get_auth_request con 0x7f0fac03a550 auth_method 0 2024-09-15T03:41:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.650+0000 7f0fcc9cf640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fac03a550 0x7f0fac03ca00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0fac006240 tx=0x7f0fb8038000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:33.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.652+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0fc401f9b0 con 0x7f0fc8140ff0 2024-09-15T03:41:33.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.652+0000 7f0fbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:33.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.652+0000 7f0fbd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:33.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.871+0000 7f0fcf434640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.871+0000 7f0fcf434640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.871+0000 7f0fcf434640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.871+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0fc804bf60 con 0x7f0fc8140ff0 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.872+0000 7f0fbd7fa640 1 -- 172.21.15.153:0/3004439628 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f0fc401e360 con 0x7f0fc8140ff0 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.872+0000 7f0fbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.872+0000 7f0fbd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:33.873 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":134,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fac03a550 msgr2=0x7f0fac03ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fac03a550 0x7f0fac03ca00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0fac006240 tx=0x7f0fb8038000 comp rx=0 tx=0).stop 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 10 monclient: shutdown 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 msgr2=0x7f0fc8131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 0x7f0fc8131b90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40149a0 tx=0x7f0fc4008040 comp rx=0 tx=0).stop 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 shutdown_connections 2024-09-15T03:41:33.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0fac03a550 0x7f0fac03ca00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc806dc40 0x7f0fc806e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0fc81320d0 0x7f0fc806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 --2- 172.21.15.153:0/3004439628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fc8140ff0 0x7f0fc8131b90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 >> 172.21.15.153:0/3004439628 conn(0x7f0fc80a2e40 msgr2=0x7f0fc8134cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:33.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 shutdown_connections 2024-09-15T03:41:33.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.874+0000 7f0fcf434640 1 -- 172.21.15.153:0/3004439628 wait complete. 2024-09-15T03:41:33.887 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:41:33.887 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:41:33.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.980+0000 7f575dc2b640 10 monclient: get_monmap_and_config 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.980+0000 7f575dc2b640 10 monclient: build_initial_monmap 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.980+0000 7f575dc2b640 10 monclient: monmap: 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:33.981360+0000 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:33.981360+0000 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 Processor -- start 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 -- start start 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient: init 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f57580981c0 0x7f5758098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): picked mon.noname-a con 0x7f57580981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5758131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 0x7f5758067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): picked mon.noname-b con 0x7f5758132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5758098ad0 con 0x7f5758131da0 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5758131bd0 con 0x7f57580981c0 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575809ca60 con 0x7f5758132170 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f575dc2b640 10 monclient(hunting): authenticate will time out at 1491.620860s 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f57577fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f57580981c0 msgr2=0x7f5758098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f5756ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f5758131da0 legacy=0x7f5758067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f57577fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f57580981c0 0x7f5758098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.981+0000 7f5757fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 0x7f5758067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 0x7f5758067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:42900/0 (socket says 172.21.15.153:42900) 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 -- 172.21.15.153:0/958739176 learned_addr learned my addr 172.21.15.153:0/958739176 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): get_auth_request con 0x7f5758132170 auth_method 0 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient(hunting): handle_auth_done global_id 124595 payload 274 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 -- 172.21.15.153:0/958739176 >> v2:172.21.15.153:3300/0 conn(0x7f57580981c0 msgr2=0x7f5758098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 --2- 172.21.15.153:0/958739176 >> v2:172.21.15.153:3300/0 conn(0x7f57580981c0 0x7f5758098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 -- 172.21.15.153:0/958739176 >> v1:172.21.15.153:6789/0 conn(0x7f5758131da0 legacy=0x7f5758067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient: _finish_hunting 0 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 monclient: found mon.noname-b 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.982+0000 7f5757fff640 1 -- 172.21.15.153:0/958739176 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575809cca0 con 0x7f5758132170 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.984+0000 7f5757fff640 1 -- 172.21.15.153:0/958739176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 msgr2=0x7f5758067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.984+0000 7f5757fff640 1 --2- 172.21.15.153:0/958739176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 0x7f5758067820 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5758066560 tx=0x7f5750009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cb44be10ef3b960 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 1 -- 172.21.15.153:0/958739176 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750007610 con 0x7f5758132170 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: dump: 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: _finish_auth 0 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 1 -- 172.21.15.153:0/958739176 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575002d050 con 0x7f5758132170 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 5 monclient: authenticate success, global_id 124595 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 1 -- 172.21.15.153:0/958739176 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57500079a0 con 0x7f5758132170 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 4 set_mon_vals no callback set 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f57567fc640 10 monclient: dump: 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 10 monclient: shutdown 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 -- 172.21.15.153:0/958739176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 msgr2=0x7f5758067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 --2- 172.21.15.153:0/958739176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 0x7f5758067820 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5758066560 tx=0x7f5750009840 comp rx=0 tx=0).stop 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 -- 172.21.15.153:0/958739176 shutdown_connections 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 --2- 172.21.15.153:0/958739176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758132170 0x7f5758067820 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 --2- 172.21.15.153:0/958739176 >> v2:172.21.15.153:3300/0 conn(0x7f57580981c0 0x7f5758098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 -- 172.21.15.153:0/958739176 >> 172.21.15.153:0/958739176 conn(0x7f57580a1e30 msgr2=0x7f57580a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 -- 172.21.15.153:0/958739176 shutdown_connections 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.985+0000 7f575dc2b640 1 -- 172.21.15.153:0/958739176 wait complete. 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.986+0000 7f575dc2b640 10 monclient: build_initial_monmap 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.986+0000 7f575dc2b640 10 monclient: monmap: 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.986+0000 7f575dc2b640 1 Processor -- start 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.986+0000 7f575dc2b640 1 -- start start 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.986+0000 7f575dc2b640 10 monclient: init 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57580981c0 0x7f5758140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): picked mon.noname-b con 0x7f57580981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 0x7f5758130960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): picked mon.noname-c con 0x7f5758131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5758132170 0x7f575811a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): picked mon.noname-a con 0x7f5758132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575811c4d0 con 0x7f5758132170 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57580a0b40 con 0x7f57580981c0 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5758098cb0 con 0x7f5758131da0 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f575dc2b640 10 monclient(hunting): authenticate will time out at 1491.626313s 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5757fff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5758132170 msgr2=0x7f575811a410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5757fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5758132170 0x7f575811a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 0x7f5758130960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 0x7f5758130960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:42912/0 (socket says 172.21.15.153:42912) 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 -- 172.21.15.153:0/3588063198 learned_addr learned my addr 172.21.15.153:0/3588063198 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): get_auth_request con 0x7f5758131da0 auth_method 0 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f57577fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57580981c0 0x7f5758140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient(hunting): handle_auth_done global_id 124598 payload 824 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57580981c0 msgr2=0x7f5758140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57580981c0 0x7f5758140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5758132170 msgr2=0x7f575811a410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5758132170 0x7f575811a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 monclient: found mon.noname-c 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f5756ffd640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750009ae0 con 0x7f5758131da0 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f57577fe640 10 monclient: get_auth_request con 0x7f57580981c0 auth_method 0 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.987+0000 7f57577fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57580981c0 0x7f5758140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.989+0000 7f5756ffd640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 msgr2=0x7f5758130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.989+0000 7f5756ffd640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 0x7f5758130960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f574c00b5f0 tx=0x7f574c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f5756ffd640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 0x7f5758130960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f574c00b5f0 tx=0x7f574c009120 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f574c005340 mon_map magic: 0 v1 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c003cc0 con 0x7f5758131da0 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 10 monclient: dump: 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c003e40 con 0x7f5758131da0 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 5 monclient: authenticate success, global_id 124598 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 10 monclient: _renew_subs 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575811da80 con 0x7f5758131da0 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f573f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 10 monclient: _renew_subs 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5758141e10 con 0x7f5758131da0 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f5755ffb640 4 set_mon_vals no callback set 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 0x7f575811a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.990+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5758065f90 con 0x7f5758130ea0 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c005340 con 0x7f5758131da0 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f573f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f573f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f573f7fe640 10 monclient: dump: 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f574c010480 con 0x7f5758131da0 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 0x7f575811a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient: get_auth_request con 0x7f5758130ea0 auth_method 0 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 10 monclient(hunting): handle_auth_done global_id 124601 payload 824 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f573f7fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5744038740 0x7f574403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f57577fe640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f575811ddf0 con 0x7f5758130ea0 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f5757fff640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5744038740 0x7f574403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.991+0000 7f5757fff640 10 monclient: get_auth_request con 0x7f5744038740 auth_method 0 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f57577fe640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 msgr2=0x7f575811a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f57577fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 0x7f575811a950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f574800c700 tx=0x7f5748008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748003d90 con 0x7f5758130ea0 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 10 monclient: dump: 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5748006150 con 0x7f5758130ea0 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 msgr2=0x7f575811a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 0x7f575811a950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f574800c700 tx=0x7f5748008040 comp rx=0 tx=0).stop 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 msgr2=0x7f575811a950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.993+0000 7f573f7fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758130ea0 0x7f575811a950 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f574800c700 tx=0x7f5748008040 comp rx=0 tx=0).stop 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.994+0000 7f5757fff640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5744038740 0x7f574403abf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5758132870 tx=0x7f5750036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f575dc2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f575dc2b640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 0x7f575809a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f575dc2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5758131270 con 0x7f575811ae90 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 0x7f575809a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient: get_auth_request con 0x7f575811ae90 auth_method 0 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:33.999+0000 7f57577fe640 10 monclient(hunting): handle_auth_done global_id 124604 payload 824 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.000+0000 7f57577fe640 1 -- 172.21.15.153:0/3588063198 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f575811b290 con 0x7f575811ae90 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f57577fe640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 msgr2=0x7f575809a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f57577fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 0x7f575809a320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f574800c700 tx=0x7f5748017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748024070 con 0x7f575811ae90 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 10 monclient: dump: 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f5748008670 con 0x7f575811ae90 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 msgr2=0x7f575809a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 0x7f575809a320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f574800c700 tx=0x7f5748017000 comp rx=0 tx=0).stop 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 msgr2=0x7f575809a320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f575811ae90 0x7f575809a320 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 134, 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5744038740 msgr2=0x7f574403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5744038740 0x7f574403abf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5758132870 tx=0x7f5750036000 comp rx=0 tx=0).stop 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 10 monclient: shutdown 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 msgr2=0x7f5758130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 --2- 172.21.15.153:0/3588063198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5758131da0 0x7f5758130960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f574c00b5f0 tx=0x7f574c009120 comp rx=0 tx=0).stop 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f5757fff640 1 -- 172.21.15.153:0/3588063198 reap_dead start 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f573f7fe640 1 -- 172.21.15.153:0/3588063198 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7f574c04c560 con 0x7f5758131da0 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f5757fff640 1 -- 172.21.15.153:0/3588063198 reap_dead start 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 shutdown_connections 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 >> 172.21.15.153:0/3588063198 conn(0x7f57580a1e30 msgr2=0x7f5758134a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 shutdown_connections 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.002+0000 7f575dc2b640 1 -- 172.21.15.153:0/3588063198 wait complete. 2024-09-15T03:41:34.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:41:34.031 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:41:34.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.125+0000 7fdfaab5d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.125+0000 7fdfaab5d640 10 monclient: build_initial_monmap 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.125+0000 7fdfaab5d640 10 monclient: monmap: 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:34.125962+0000 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:34.125962+0000 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:34.126 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.127 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:34.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:34.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:34.127 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:34.127 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:34.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.125+0000 7fdfaab5d640 1 Processor -- start 2024-09-15T03:41:34.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.125+0000 7fdfaab5d640 1 -- start start 2024-09-15T03:41:34.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.125+0000 7fdfaab5d640 10 monclient: init 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdfa40981c0 0x7fdfa4098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa40981c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 0x7fdfa4067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa4131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdfa4132170 0x7fdfa4067650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa4132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa4131bd0 con 0x7fdfa4132170 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa4098ad0 con 0x7fdfa40981c0 2024-09-15T03:41:34.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa409ca60 con 0x7fdfa4131da0 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfaab5d640 10 monclient(hunting): authenticate will time out at 1491.765453s 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa90fa640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fdfa4132170 msgr2=0x7fdfa4067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa90fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdfa4132170 0x7fdfa4067650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 0x7fdfa4067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 0x7fdfa4067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:42938/0 (socket says 172.21.15.153:42938) 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 -- 172.21.15.153:0/1548821810 learned_addr learned my addr 172.21.15.153:0/1548821810 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): get_auth_request con 0x7fdfa4131da0 auth_method 0 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa88f9640 1 --2- 172.21.15.153:0/1548821810 >> v2:172.21.15.181:3300/0 conn(0x7fdfa40981c0 0x7fdfa4098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient(hunting): handle_auth_done global_id 124607 payload 274 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 -- 172.21.15.153:0/1548821810 >> v2:172.21.15.181:3300/0 conn(0x7fdfa40981c0 msgr2=0x7fdfa4098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/1548821810 >> v2:172.21.15.181:3300/0 conn(0x7fdfa40981c0 0x7fdfa4098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 -- 172.21.15.153:0/1548821810 >> v2:172.21.15.153:3300/0 conn(0x7fdfa4132170 msgr2=0x7fdfa4067650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/1548821810 >> v2:172.21.15.153:3300/0 conn(0x7fdfa4132170 0x7fdfa4067650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 monclient: found mon.noname-b 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:34.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa3fff640 1 -- 172.21.15.153:0/1548821810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa409cca0 con 0x7fdfa4131da0 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa88f9640 10 monclient: get_auth_request con 0x7fdfa40981c0 auth_method 0 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.126+0000 7fdfa88f9640 1 --2- 172.21.15.153:0/1548821810 >> v2:172.21.15.181:3300/0 conn(0x7fdfa40981c0 0x7fdfa4098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa3fff640 1 -- 172.21.15.153:0/1548821810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 msgr2=0x7fdfa4067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/1548821810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 0x7fdfa4067110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4066560 tx=0x7fdf900091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42d65405fef4a6e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa37fe640 1 -- 172.21.15.153:0/1548821810 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90032440 con 0x7fdfa4131da0 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa37fe640 10 monclient: dump: 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.134 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa37fe640 10 monclient: _finish_auth 0 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.129+0000 7fdfa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 5 monclient: authenticate success, global_id 124607 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfa37fe640 1 -- 172.21.15.153:0/1548821810 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf900325a0 con 0x7fdfa4131da0 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfa37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:34.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfa37fe640 1 -- 172.21.15.153:0/1548821810 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90032940 con 0x7fdfa4131da0 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 4 set_mon_vals no callback set 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfa37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfa37fe640 10 monclient: dump: 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.137 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 10 monclient: shutdown 2024-09-15T03:41:34.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:34.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 -- 172.21.15.153:0/1548821810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 msgr2=0x7fdfa4067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/1548821810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 0x7fdfa4067110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4066560 tx=0x7fdf900091d0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 -- 172.21.15.153:0/1548821810 shutdown_connections 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/1548821810 >> v2:172.21.15.153:3300/0 conn(0x7fdfa4132170 0x7fdfa4067650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/1548821810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4131da0 0x7fdfa4067110 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/1548821810 >> v2:172.21.15.181:3300/0 conn(0x7fdfa40981c0 0x7fdfa4098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 -- 172.21.15.153:0/1548821810 >> 172.21.15.153:0/1548821810 conn(0x7fdfa40a1e30 msgr2=0x7fdfa40a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 -- 172.21.15.153:0/1548821810 shutdown_connections 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.130+0000 7fdfaab5d640 1 -- 172.21.15.153:0/1548821810 wait complete. 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient: build_initial_monmap 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient: monmap: 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 Processor -- start 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 -- start start 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient: init 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfa40981c0 0x7fdfa411cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa40981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa4131da0 0x7fdfa411d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa4131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 0x7fdfa411d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa4132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4098cb0 con 0x7fdfa40981c0 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40a0960 con 0x7fdfa4131da0 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa411f2f0 con 0x7fdfa4132170 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfaab5d640 10 monclient(hunting): authenticate will time out at 1491.770967s 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfa88f9640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfa40981c0 msgr2=0x7fdfa411cbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:41:34.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.131+0000 7fdfa88f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfa40981c0 0x7fdfa411cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 0x7fdfa411d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 0x7fdfa411d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:42946/0 (socket says 172.21.15.153:42946) 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 -- 172.21.15.153:0/3126920863 learned_addr learned my addr 172.21.15.153:0/3126920863 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): get_auth_request con 0x7fdfa4132170 auth_method 0 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa4131da0 0x7fdfa411d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient(hunting): handle_auth_done global_id 124610 payload 824 2024-09-15T03:41:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa4131da0 msgr2=0x7fdfa411d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa4131da0 0x7fdfa411d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfa40981c0 msgr2=0x7fdfa411cbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfa40981c0 0x7fdfa411cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient: _finish_hunting 0 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 monclient: found mon.noname-c 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:34.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa90fa640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf90009450 con 0x7fdfa4132170 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa3fff640 10 monclient: get_auth_request con 0x7fdfa4131da0 auth_method 0 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.132+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa4131da0 0x7fdfa411d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa90fa640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 msgr2=0x7fdfa411d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa90fa640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 0x7fdfa411d660 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00bf80 tx=0x7fdf9c009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c016430 con 0x7fdfa4132170 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: dump: 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.148 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.149 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c0165b0 con 0x7fdfa4132170 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 5 monclient: authenticate success, global_id 124610 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 10 monclient: _renew_subs 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa4120250 con 0x7fdfa4132170 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c016760 con 0x7fdfa4132170 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa0ff9640 10 monclient: dump: 2024-09-15T03:41:34.150 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfa2ffd640 4 set_mon_vals no callback set 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 10 monclient: _renew_subs 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:34.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.134+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa4130270 con 0x7fdfa4132170 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfaab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdf9c012020 con 0x7fdfa4132170 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 0x7fdfa411dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4065f90 con 0x7fdfa402aa40 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa0ff9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf880386f0 0x7fdf8803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 0x7fdfa411dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdf9c04c820 con 0x7fdfa4132170 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient: get_auth_request con 0x7fdfa402aa40 auth_method 0 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf880386f0 0x7fdf8803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.135+0000 7fdfa3fff640 10 monclient: get_auth_request con 0x7fdf880386f0 auth_method 0 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.136+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_done global_id 144322 payload 824 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.136+0000 7fdfa88f9640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfa402ae10 con 0x7fdfa402aa40 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa88f9640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 msgr2=0x7fdfa411dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa88f9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 0x7fdfa411dba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c700 tx=0x7fdf98008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa3fff640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf880386f0 0x7fdf8803aba0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdfa409c940 tx=0x7fdf90032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98007850 con 0x7fdfa402aa40 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:34.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 10 monclient: dump: 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdf980079b0 con 0x7fdfa402aa40 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:34.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 msgr2=0x7fdfa411dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 0x7fdfa411dba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c700 tx=0x7fdf98008040 comp rx=0 tx=0).stop 2024-09-15T03:41:34.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 msgr2=0x7fdfa411dba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:34.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.138+0000 7fdfa0ff9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa402aa40 0x7fdfa411dba0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfaab5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 0x7fdfa405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfaab5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa41304c0 con 0x7fdfa405e310 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 0x7fdfa405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient: get_auth_request con 0x7fdfa405e310 auth_method 0 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 10 monclient(hunting): handle_auth_done global_id 144325 payload 824 2024-09-15T03:41:34.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.144+0000 7fdfa88f9640 1 -- 172.21.15.153:0/3126920863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdfa41206d0 con 0x7fdfa405e310 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa88f9640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 msgr2=0x7fdfa405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa88f9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 0x7fdfa405e710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf98016000 tx=0x7fdf98019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9801f070 con 0x7fdfa405e310 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 10 monclient: dump: 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:34.158 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fdf98007640 con 0x7fdfa405e310 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 msgr2=0x7fdfa405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 0x7fdfa405e710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf98016000 tx=0x7fdf98019000 comp rx=0 tx=0).stop 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 msgr2=0x7fdfa405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.147+0000 7fdfa0ff9640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa405e310 0x7fdfa405e710 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:41:34.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 134, 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:34.160 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:34.161 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:41:34.162 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:41:34.163 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:41:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.165 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.166 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:41:34.171 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf880386f0 msgr2=0x7fdf8803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdf880386f0 0x7fdf8803aba0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdfa409c940 tx=0x7fdf90032000 comp rx=0 tx=0).stop 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 10 monclient: shutdown 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 msgr2=0x7fdfa411d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 0x7fdfa411d660 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00bf80 tx=0x7fdf9c009730 comp rx=0 tx=0).stop 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfa90fa640 1 -- 172.21.15.153:0/3126920863 reap_dead start 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 shutdown_connections 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 --2- 172.21.15.153:0/3126920863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfa4132170 0x7fdfa411d660 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:34.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 >> 172.21.15.153:0/3126920863 conn(0x7fdfa40a1e30 msgr2=0x7fdfa409b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:34.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 shutdown_connections 2024-09-15T03:41:34.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:34.148+0000 7fdfaab5d640 1 -- 172.21.15.153:0/3126920863 wait complete. 2024-09-15T03:41:34.174 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:41:34.174 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:41:54.175 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:41:54.176 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:41:54.176 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:41:54.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:41:54.179 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:41:54.179 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:41:54.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:54.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.284+0000 7f311579d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:54.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.284+0000 7f311579d640 10 monclient: build_initial_monmap 2024-09-15T03:41:54.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.284+0000 7f311579d640 10 monclient: monmap: 2024-09-15T03:41:54.285 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:54.285 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:54.284827+0000 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:54.284827+0000 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.284+0000 7f311579d640 1 Processor -- start 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.284+0000 7f311579d640 1 -- start start 2024-09-15T03:41:54.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.284+0000 7f311579d640 10 monclient: init 2024-09-15T03:41:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:41:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31101318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:41:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f311009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 0x7f311013a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient(hunting): picked mon.noname-a con 0x7f3110137c30 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3110131cc0 con 0x7f31101318f0 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311009f530 con 0x7f311009ea50 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311009fbf0 con 0x7f3110137c30 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f311579d640 10 monclient(hunting): authenticate will time out at 1511.924483s 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f31101318f0 0x7f3110131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39296/0 (socket says 172.21.15.153:39296) 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 0x7f311013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310effd640 1 -- 172.21.15.153:0/3856603503 learned_addr learned my addr 172.21.15.153:0/3856603503 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 10 monclient(hunting): get_auth_request con 0x7f3110137c30 auth_method 0 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 1 -- 172.21.15.153:0/3856603503 >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 msgr2=0x7f311013a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 1 -- 172.21.15.153:0/3856603503 >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 msgr2=0x7f311013a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 1 --2- 172.21.15.153:0/3856603503 >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 0x7f311013a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.285+0000 7f310f7fe640 1 --2- 172.21.15.153:0/3856603503 >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 0x7f311013a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.287+0000 7f310effd640 1 -- 172.21.15.153:0/3856603503 >> v1:172.21.15.153:6789/0 conn(0x7f31101318f0 legacy=0x7f3110131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.287+0000 7f310e7fc640 1 -- 172.21.15.153:0/3856603503 >> v1:172.21.15.181:6789/0 conn(0x7f311009ea50 legacy=0x7f311009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447146652 0 0) 0x7f311009f530 con 0x7f311009ea50 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:54.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): my global_id is 144328 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f80034e0 con 0x7f311009ea50 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3118572591 0 0) 0x7f30f80034e0 con 0x7f311009ea50 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 msgr2=0x7f311013a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 --2- 172.21.15.153:0/3856603503 >> v2:172.21.15.153:3300/0 conn(0x7f3110137c30 0x7f311013a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 >> v1:172.21.15.153:6789/0 conn(0x7f31101318f0 legacy=0x7f3110131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: _finish_hunting 0 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311009fe30 con 0x7f311009ea50 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: _finish_auth 0 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f3104001830 con 0x7f311009ea50 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f311579d640 5 monclient: authenticate success, global_id 144328 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: dump: 2024-09-15T03:41:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:54.307 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:54.307 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 >> v1:172.21.15.181:6789/0 conn(0x7f311009ea50 legacy=0x7f311009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f30f8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): picked mon.a con 0x7f30f8003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f8004780 0x7f30f8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): picked mon.b con 0x7f30f8004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f80052e0 0x7f30f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): picked mon.c con 0x7f30f80052e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311009fbf0 con 0x7f30f8003a50 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f800c2b0 con 0x7f30f8004780 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f800c470 con 0x7f30f80052e0 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:41:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f311579d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f30f8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f80052e0 0x7f30f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): get_auth_request con 0x7f30f8003a50 auth_method 0 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): get_auth_request con 0x7f30f80052e0 auth_method 0 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f8004780 0x7f30f8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): get_auth_request con 0x7f30f8004780 auth_method 0 2024-09-15T03:41:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient(hunting): handle_auth_done global_id 144328 payload 293 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 -- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f80052e0 msgr2=0x7f30f800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f80052e0 0x7f30f800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 -- 172.21.15.153:0/3856603503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f8004780 msgr2=0x7f30f8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:41:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f8004780 0x7f30f8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient: _finish_hunting 0 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 monclient: found mon.a 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 1 -- 172.21.15.153:0/3856603503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f80034e0 con 0x7f30f8003a50 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 10 monclient: _finish_auth 0 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f80052e0 0x7f30f800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.288+0000 7f310f7fe640 1 -- 172.21.15.153:0/3856603503 reap_dead start 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310effd640 1 -- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 msgr2=0x7f30f8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310effd640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f30f8003e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3100001bb0 tx=0x7f3100007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cd1623159dfec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310002ac60 con 0x7f30f8003a50 2024-09-15T03:41:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: dump: 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f311579d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310002ade0 con 0x7f30f8003a50 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f311579d640 10 monclient: get_monmap_and_config success 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f311579d640 4 set_mon_vals no callback set 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 1 -- 172.21.15.153:0/3856603503 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100004ab0 con 0x7f30f8003a50 2024-09-15T03:41:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.291+0000 7f310dffb640 10 monclient: dump: 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 10 monclient: shutdown 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 -- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 msgr2=0x7f30f8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f30f8003e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3100001bb0 tx=0x7f3100007d50 comp rx=0 tx=0).stop 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 -- 172.21.15.153:0/3856603503 shutdown_connections 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 --2- 172.21.15.153:0/3856603503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f30f8003e60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 -- 172.21.15.153:0/3856603503 >> 172.21.15.153:0/3856603503 conn(0x7f3110067110 msgr2=0x7f3110067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 -- 172.21.15.153:0/3856603503 shutdown_connections 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 1 -- 172.21.15.153:0/3856603503 wait complete. 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 10 monclient: build_initial_monmap 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.292+0000 7f311579d640 10 monclient: monmap: 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 Processor -- start 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 -- start start 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient: init 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:41:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f3110141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): picked mon.noname-a con 0x7f30f8003a50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 0x7f31100987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): picked mon.noname-c con 0x7f3110141bc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3110098ce0 0x7f3110099130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): picked mon.noname-b con 0x7f3110098ce0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311009ddd0 con 0x7f30f8003a50 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311009bf10 con 0x7f3110098ce0 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311009aa50 con 0x7f3110141bc0 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f311579d640 10 monclient(hunting): authenticate will time out at 1511.932822s 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 0x7f31100987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f3110141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 0x7f31100987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44664/0 (socket says 172.21.15.153:44664) 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310effd640 10 monclient(hunting): get_auth_request con 0x7f30f8003a50 auth_method 0 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 1 -- 172.21.15.153:0/420891399 learned_addr learned my addr 172.21.15.153:0/420891399 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 10 monclient(hunting): get_auth_request con 0x7f3110141bc0 auth_method 0 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.293+0000 7f310e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:41:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310effd640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 msgr2=0x7f3110141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310effd640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 msgr2=0x7f3110141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310effd640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f3110141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310effd640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f3110141680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310f7fe640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3110098ce0 0x7f3110099130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310f7fe640 10 monclient(hunting): get_auth_request con 0x7f3110098ce0 auth_method 0 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 10 monclient(hunting): handle_auth_done global_id 124613 payload 824 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3110098ce0 msgr2=0x7f3110099130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3110098ce0 0x7f3110099130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 msgr2=0x7f3110141680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f3110141680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 1 monclient: found mon.noname-c 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.294+0000 7f310e7fc640 1 -- 172.21.15.153:0/420891399 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311009a7a0 con 0x7f3110141bc0 2024-09-15T03:41:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f310e7fc640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 msgr2=0x7f31100987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f310e7fc640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 0x7f31100987a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f310400ffa0 tx=0x7f3104009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040018a0 con 0x7f3110141bc0 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: dump: 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104001350 con 0x7f3110141bc0 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 5 monclient: authenticate success, global_id 124613 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 10 monclient: _renew_subs 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311011bf10 con 0x7f3110141bc0 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040070d0 con 0x7f3110141bc0 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:41:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f30f6ffd640 10 monclient: dump: 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 10 monclient: _renew_subs 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311009a290 con 0x7f3110141bc0 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.296+0000 7f310d7fa640 4 set_mon_vals no callback set 2024-09-15T03:41:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3104007440 con 0x7f3110141bc0 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f30f6ffd640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30fc038740 0x7f30fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f310404c950 con 0x7f3110141bc0 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f311579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f311579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f311579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311004bf60 con 0x7f3110141bc0 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f310effd640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30fc038740 0x7f30fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.297+0000 7f310effd640 10 monclient: get_auth_request con 0x7f30fc038740 auth_method 0 2024-09-15T03:41:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.300+0000 7f310effd640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30fc038740 0x7f30fc03abf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3100001b80 tx=0x7f310002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:41:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.302+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f31040129a0 con 0x7f3110141bc0 2024-09-15T03:41:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.302+0000 7f30f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:41:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.302+0000 7f30f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.517+0000 7f311579d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.517+0000 7f311579d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.517+0000 7f311579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.517+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3110135e40 con 0x7f3110141bc0 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.518+0000 7f30f6ffd640 1 -- 172.21.15.153:0/420891399 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f310402b080 con 0x7f3110141bc0 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.518+0000 7f30f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.518+0000 7f30f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:41:54.519 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":134,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:41:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30fc038740 msgr2=0x7f30fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30fc038740 0x7f30fc03abf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3100001b80 tx=0x7f310002f000 comp rx=0 tx=0).stop 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 10 monclient: shutdown 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 msgr2=0x7f31100987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 0x7f31100987a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f310400ffa0 tx=0x7f3104009040 comp rx=0 tx=0).stop 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 shutdown_connections 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30fc038740 0x7f30fc03abf0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3110098ce0 0x7f3110099130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3110141bc0 0x7f31100987a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 --2- 172.21.15.153:0/420891399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f8003a50 0x7f3110141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 >> 172.21.15.153:0/420891399 conn(0x7f3110067110 msgr2=0x7f31100a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 shutdown_connections 2024-09-15T03:41:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:54.520+0000 7f311579d640 1 -- 172.21.15.153:0/420891399 wait complete. 2024-09-15T03:41:57.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.636+0000 7fb3ff11d640 10 monclient: get_monmap_and_config 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.636+0000 7fb3ff11d640 10 monclient: build_initial_monmap 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.636+0000 7fb3ff11d640 10 monclient: monmap: 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:41:57.637107+0000 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:41:57.637107+0000 2024-09-15T03:41:57.637 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:41:57.638 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 Processor -- start 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 -- start start 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient: init 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:57.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3f8096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3f809aaa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3f80978f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f809d4a0 con 0x7fb3f809aaa0 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f8099620 con 0x7fb3f8096e10 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8099340 con 0x7fb3f80978f0 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3ff11d640 10 monclient(hunting): authenticate will time out at 1515.276832s 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fceb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:57.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45922/0 (socket says 172.21.15.153:45922) 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fceb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59784/0 (socket says 172.21.15.153:59784) 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 learned_addr learned my addr 172.21.15.153:0/2427520014 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3f80978f0 auth_method 0 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.637+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3f8096e10 auth_method 0 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:57.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:57.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.638+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:41:57.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.640+0000 7fb3f7fff640 1 -- 172.21.15.153:0/2427520014 >> v1:172.21.15.181:6789/0 conn(0x7fb3f809aaa0 legacy=0x7fb3f80971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:41:57.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:57.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:57.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3f80978f0 auth_method 0 2024-09-15T03:41:57.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:57.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:57.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:57.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3f8096e10 auth_method 0 2024-09-15T03:41:57.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:57.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:57.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:57.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:57.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:57.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:57.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:57.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:57.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:57.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:57.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:57.839+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:41:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3f8096e10 auth_method 0 2024-09-15T03:41:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3f80978f0 auth_method 0 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:58.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:58.240+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3f8096e10 auth_method 0 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3f80978f0 auth_method 0 2024-09-15T03:41:59.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:41:59.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:41:59.041+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669725837 0 0) 0x7fb3f809d4a0 con 0x7fb3f809aaa0 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient(hunting): my global_id is 144334 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e80034e0 con 0x7fb3f809aaa0 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092603171 0 0) 0x7fb3e80034e0 con 0x7fb3f809aaa0 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3f8135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3f8135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 msgr2=0x7fb3f809a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 --2- 172.21.15.153:0/2427520014 >> v2:172.21.15.153:3300/0 conn(0x7fb3f8096e10 0x7fb3f809a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8099900 con 0x7fb3f809aaa0 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient: _finish_auth 0 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fb3e4001770 con 0x7fb3f809aaa0 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.503+0000 7fb3f77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient: dump: 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 >> v1:172.21.15.181:6789/0 conn(0x7fb3f809aaa0 legacy=0x7fb3f80971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3e8003ad0 0x7fb3e8003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): picked mon.a con 0x7fb3e8003ad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3e8004780 0x7fb3e8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): picked mon.b con 0x7fb3e8004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3e800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): picked mon.c con 0x7fb3f80978f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8099620 con 0x7fb3e8003ad0 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8099340 con 0x7fb3e8004780 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8005560 con 0x7fb3f80978f0 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3e8004780 0x7fb3e8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3e800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3e8003ad0 0x7fb3e8003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3e8003ad0 auth_method 0 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): get_auth_request con 0x7fb3e8004780 auth_method 0 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3f80978f0 auth_method 0 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3fceb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_done global_id 144334 payload 293 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 msgr2=0x7fb3e800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3f80978f0 0x7fb3e800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3e8003ad0 msgr2=0x7fb3e8003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3e8003ad0 0x7fb3e8003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 monclient: found mon.b 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 1 -- 172.21.15.153:0/2427520014 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e80034e0 con 0x7fb3e8004780 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 10 monclient: _finish_auth 0 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3ff11d640 5 monclient: authenticate success, global_id 144334 2024-09-15T03:42:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.504+0000 7fb3ff11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:00.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.507+0000 7fb3f7fff640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3e8004780 msgr2=0x7fb3e8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.507+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3e8004780 0x7fb3e8004bf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4001950 tx=0x7fb3e40079e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a2966507f548a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e402c040 con 0x7fb3e8004780 2024-09-15T03:42:00.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: dump: 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.524 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e40070f0 con 0x7fb3e8004780 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 1 -- 172.21.15.153:0/2427520014 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40074b0 con 0x7fb3e8004780 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3ff11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3ff11d640 4 set_mon_vals no callback set 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.508+0000 7fb3f77fe640 10 monclient: dump: 2024-09-15T03:42:00.525 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 10 monclient: shutdown 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 1 -- 172.21.15.153:0/2427520014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3e8004780 msgr2=0x7fb3e8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/2427520014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3e8004780 0x7fb3e8004bf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4001950 tx=0x7fb3e40079e0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/2427520014 reap_dead start 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 1 -- 172.21.15.153:0/2427520014 shutdown_connections 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 1 -- 172.21.15.153:0/2427520014 >> 172.21.15.153:0/2427520014 conn(0x7fb3f80672b0 msgr2=0x7fb3f80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 1 -- 172.21.15.153:0/2427520014 shutdown_connections 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.509+0000 7fb3ff11d640 1 -- 172.21.15.153:0/2427520014 wait complete. 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.510+0000 7fb3ff11d640 10 monclient: build_initial_monmap 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.510+0000 7fb3ff11d640 10 monclient: monmap: 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:00.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.510+0000 7fb3ff11d640 1 Processor -- start 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.510+0000 7fb3ff11d640 1 -- start start 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.510+0000 7fb3ff11d640 10 monclient: init 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:42:00.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3f812e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3e8004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3f8118030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3f812eac0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 0x7fb3f81189e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3f8118570 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f812ee90 con 0x7fb3f8118570 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f811bcf0 con 0x7fb3f812eac0 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f809fd30 con 0x7fb3e8004780 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:00.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3ff11d640 10 monclient(hunting): authenticate will time out at 1518.150556s 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3f812e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 0x7fb3f81189e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3f812e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45974/0 (socket says 172.21.15.153:45974) 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3f8118030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 1 -- 172.21.15.153:0/1578157953 learned_addr learned my addr 172.21.15.153:0/1578157953 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3f8118570 auth_method 0 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): get_auth_request con 0x7fb3f812eac0 auth_method 0 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3e8004780 auth_method 0 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.511+0000 7fb3fceb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_done global_id 174099 payload 824 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 msgr2=0x7fb3f812e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3f812e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 msgr2=0x7fb3f8118030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3f8118030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 10 monclient: _finish_hunting 0 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 1 monclient: found mon.noname-a 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.512+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f811a740 con 0x7fb3f8118570 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 msgr2=0x7fb3f81189e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 0x7fb3f81189e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb3f809ae70 tx=0x7fb3f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0018870 con 0x7fb3f8118570 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: dump: 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.533 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f000f040 con 0x7fb3f8118570 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 5 monclient: authenticate success, global_id 174099 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 10 monclient: _renew_subs 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f811a080 con 0x7fb3f8118570 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0018c40 con 0x7fb3f8118570 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f4ff9640 10 monclient: dump: 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.534 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3f6ffd640 4 set_mon_vals no callback set 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 10 monclient: _renew_subs 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.514+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f8119b70 con 0x7fb3f8118570 2024-09-15T03:42:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb3f00173b0 con 0x7fb3f8118570 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3f4ff9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb3dc038790 0x7fb3dc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3ff11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb3f004cae0 con 0x7fb3f8118570 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3ff11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3ff11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.515+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f81342e0 con 0x7fb3f8118570 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 msgr2=0x7fb3f81189e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 msgr2=0x7fb3f81189e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 0x7fb3f81189e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb3f809ae70 tx=0x7fb3f000b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 0x7fb3f81189e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb3f809ae70 tx=0x7fb3f000b040 comp rx=0 tx=0).stop 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb3dc038790 0x7fb3dc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 msgr2=0x7fb3f81189e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8118570 0x7fb3f81189e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3dc043010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): picked mon.c con 0x7fb3e8004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3dc0438a0 0x7fb3dc045d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): picked mon.a con 0x7fb3dc0438a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3dc048850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): picked mon.b con 0x7fb3f812eac0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0491d0 con 0x7fb3dc0438a0 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc049310 con 0x7fb3f812eac0 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0494d0 con 0x7fb3e8004780 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f4ff9640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3dc038790 auth_method 0 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3dc043010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3dc0438a0 0x7fb3dc045d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): get_auth_request con 0x7fb3e8004780 auth_method 0 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request con 0x7fb3dc0438a0 auth_method 0 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3dc048850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request con 0x7fb3f812eac0 auth_method 0 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient(hunting): handle_auth_done global_id 174099 payload 293 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 msgr2=0x7fb3dc048850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3dc048850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3dc0438a0 msgr2=0x7fb3dc045d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3dc0438a0 0x7fb3dc045d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 monclient: found mon.c 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fb3dc049750 con 0x7fb3e8004780 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e80034e0 con 0x7fb3e8004780 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 10 monclient: _finish_auth 0 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.516+0000 7fb3fd6ba640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3dc0438a0 0x7fb3dc045d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.519+0000 7fb3fceb9640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb3dc038790 0x7fb3dc03ac40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec017850 tx=0x7fb3ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.519+0000 7fb3f7fff640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 msgr2=0x7fb3dc043010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.519+0000 7fb3f7fff640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3dc043010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40010f0 tx=0x7fb3e402f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e404b070 con 0x7fb3e8004780 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:00.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f4ff9640 10 monclient: dump: 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.542 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4001120 con 0x7fb3e8004780 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.520+0000 7fb3f67fc640 4 set_mon_vals no callback set 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.521+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb3e4044070 con 0x7fb3e8004780 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.525+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb3e4047470 con 0x7fb3e8004780 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.525+0000 7fb3f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:00.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.525+0000 7fb3f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:00.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.747+0000 7fb3ff11d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:42:00.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.747+0000 7fb3ff11d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:42:00.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.747+0000 7fb3ff11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.747+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb3f811ba10 con 0x7fb3e8004780 2024-09-15T03:42:00.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.748+0000 7fb3f4ff9640 1 -- 172.21.15.153:0/1578157953 <== mon.2 v2:172.21.15.153:3301/0 5 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb3e4036070 con 0x7fb3e8004780 2024-09-15T03:42:00.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.748+0000 7fb3f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:42:00.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.748+0000 7fb3f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:00.749 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb3dc038790 msgr2=0x7fb3dc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb3dc038790 0x7fb3dc03ac40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec017850 tx=0x7fb3ec008040 comp rx=0 tx=0).stop 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 10 monclient: shutdown 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 msgr2=0x7fb3dc043010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3dc043010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40010f0 tx=0x7fb3e402f000 comp rx=0 tx=0).stop 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 shutdown_connections 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f812eac0 0x7fb3dc048850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3dc0438a0 0x7fb3dc045d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3e8004780 0x7fb3dc043010 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 --2- 172.21.15.153:0/1578157953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb3dc038790 0x7fb3dc03ac40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 >> 172.21.15.153:0/1578157953 conn(0x7fb3f80672b0 msgr2=0x7fb3f81320b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:00.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 shutdown_connections 2024-09-15T03:42:00.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.750+0000 7fb3ff11d640 1 -- 172.21.15.153:0/1578157953 wait complete. 2024-09-15T03:42:00.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:42:00.762 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:42:00.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:42:00.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.855+0000 7fd1d791d640 10 monclient: get_monmap_and_config 2024-09-15T03:42:00.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.855+0000 7fd1d791d640 10 monclient: build_initial_monmap 2024-09-15T03:42:00.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.855+0000 7fd1d791d640 10 monclient: monmap: 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:00.855991+0000 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:00.855991+0000 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.855+0000 7fd1d791d640 1 Processor -- start 2024-09-15T03:42:00.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.855+0000 7fd1d791d640 1 -- start start 2024-09-15T03:42:00.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.855+0000 7fd1d791d640 10 monclient: init 2024-09-15T03:42:00.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:00.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:42:00.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd1d01318f0 0x7fd1d0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d01318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:00.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1d009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d0131cc0 con 0x7fd1d009ea50 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d009f530 con 0x7fd1d01318f0 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d009fbf0 con 0x7fd1d0136c20 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:00.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d791d640 10 monclient(hunting): authenticate will time out at 1518.495492s 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45986/0 (socket says 172.21.15.153:45986) 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 -- 172.21.15.153:0/3014217867 learned_addr learned my addr 172.21.15.153:0/3014217867 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): get_auth_request con 0x7fd1d0136c20 auth_method 0 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d56b9640 1 --2- 172.21.15.153:0/3014217867 >> v2:172.21.15.181:3300/0 conn(0x7fd1d01318f0 0x7fd1d0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d56b9640 10 monclient(hunting): get_auth_request con 0x7fd1d01318f0 auth_method 0 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d56b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d56b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d56b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_done global_id 124619 payload 274 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 -- 172.21.15.153:0/3014217867 >> v2:172.21.15.181:3300/0 conn(0x7fd1d01318f0 msgr2=0x7fd1d0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 --2- 172.21.15.153:0/3014217867 >> v2:172.21.15.181:3300/0 conn(0x7fd1d01318f0 0x7fd1d0131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 -- 172.21.15.153:0/3014217867 >> v1:172.21.15.181:6789/0 conn(0x7fd1d009ea50 legacy=0x7fd1d009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient: _finish_hunting 0 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 monclient: found mon.noname-b 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.856+0000 7fd1d5eba640 1 -- 172.21.15.153:0/3014217867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d009fe30 con 0x7fd1d0136c20 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1d5eba640 1 -- 172.21.15.153:0/3014217867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 msgr2=0x7fd1d0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1d5eba640 1 --2- 172.21.15.153:0/3014217867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0139000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009f20 tx=0x7fd1c8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2904ab9d3f640a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 1 -- 172.21.15.153:0/3014217867 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8007610 con 0x7fd1d0136c20 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: dump: 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: _finish_auth 0 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 1 -- 172.21.15.153:0/3014217867 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c802d050 con 0x7fd1d0136c20 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 1 -- 172.21.15.153:0/3014217867 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80079e0 con 0x7fd1d0136c20 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1d791d640 5 monclient: authenticate success, global_id 124619 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1d791d640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1d791d640 4 set_mon_vals no callback set 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.859+0000 7fd1c3fff640 10 monclient: dump: 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.867 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 10 monclient: shutdown 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 -- 172.21.15.153:0/3014217867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 msgr2=0x7fd1d0139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 --2- 172.21.15.153:0/3014217867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0139000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009f20 tx=0x7fd1c8009840 comp rx=0 tx=0).stop 2024-09-15T03:42:00.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 -- 172.21.15.153:0/3014217867 shutdown_connections 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 --2- 172.21.15.153:0/3014217867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0139000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 --2- 172.21.15.153:0/3014217867 >> v2:172.21.15.181:3300/0 conn(0x7fd1d01318f0 0x7fd1d0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 -- 172.21.15.153:0/3014217867 >> 172.21.15.153:0/3014217867 conn(0x7fd1d0067110 msgr2=0x7fd1d0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 -- 172.21.15.153:0/3014217867 shutdown_connections 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 1 -- 172.21.15.153:0/3014217867 wait complete. 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 10 monclient: build_initial_monmap 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.860+0000 7fd1d791d640 10 monclient: monmap: 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:00.869 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 Processor -- start 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 -- start start 2024-09-15T03:42:00.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient: init 2024-09-15T03:42:00.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:00.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1d009ea50 0x7fd1d011daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 0x7fd1d0130760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0099590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d0136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d009dd10 con 0x7fd1d01318f0 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d009bf10 con 0x7fd1d009ea50 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d009b830 con 0x7fd1d0136c20 2024-09-15T03:42:00.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:00.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.861+0000 7fd1d791d640 10 monclient(hunting): authenticate will time out at 1518.500985s 2024-09-15T03:42:00.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0099590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0099590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45998/0 (socket says 172.21.15.153:45998) 2024-09-15T03:42:00.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 -- 172.21.15.153:0/1011197775 learned_addr learned my addr 172.21.15.153:0/1011197775 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): get_auth_request con 0x7fd1d0136c20 auth_method 0 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 0x7fd1d0130760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 10 monclient(hunting): get_auth_request con 0x7fd1d01318f0 auth_method 0 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 msgr2=0x7fd1d0130760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:42:00.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 msgr2=0x7fd1d0130760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 0x7fd1d0130760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d4eb8640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 0x7fd1d0130760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d56b9640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1d009ea50 0x7fd1d011daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d56b9640 10 monclient(hunting): get_auth_request con 0x7fd1d009ea50 auth_method 0 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d56b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d56b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d56b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient(hunting): handle_auth_done global_id 124622 payload 824 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1d009ea50 msgr2=0x7fd1d011daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1d009ea50 0x7fd1d011daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 msgr2=0x7fd1d0130760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01318f0 0x7fd1d0130760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient: _finish_hunting 0 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 monclient: found mon.noname-c 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.862+0000 7fd1d5eba640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c8009ae0 con 0x7fd1d0136c20 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d5eba640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 msgr2=0x7fd1d0099590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d5eba640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0099590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009f20 tx=0x7fd1c8036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8032cc0 con 0x7fd1d0136c20 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: dump: 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: _finish_auth 0 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c8032e40 con 0x7fd1d0136c20 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c803b660 con 0x7fd1d0136c20 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 5 monclient: authenticate success, global_id 124622 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 10 monclient: _renew_subs 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d009b070 con 0x7fd1d0136c20 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c17fa640 10 monclient: dump: 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1c37fe640 4 set_mon_vals no callback set 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 10 monclient: _renew_subs 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.864+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d0130ca0 con 0x7fd1d0136c20 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d791d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d791d640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 0x7fd1d00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d0065e00 con 0x7fd1d009a290 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd1c8038050 con 0x7fd1d0136c20 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 0x7fd1d00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient: get_auth_request con 0x7fd1d009a290 auth_method 0 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1c17fa640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1b4038740 0x7fd1b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd1c802ae40 con 0x7fd1d0136c20 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.865+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_done global_id 174102 payload 824 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.866+0000 7fd1d56b9640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1d004bf60 con 0x7fd1d009a290 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.866+0000 7fd1d4eb8640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1b4038740 0x7fd1b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.866+0000 7fd1d4eb8640 10 monclient: get_auth_request con 0x7fd1b4038740 auth_method 0 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1d56b9640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 msgr2=0x7fd1d00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1d56b9640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 0x7fd1d00298c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400f590 tx=0x7fd1c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4014880 con 0x7fd1d009a290 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 10 monclient: dump: 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd1c40149e0 con 0x7fd1d009a290 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 msgr2=0x7fd1d00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 0x7fd1d00298c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400f590 tx=0x7fd1c4008040 comp rx=0 tx=0).stop 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 msgr2=0x7fd1d00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1c17fa640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a290 0x7fd1d00298c0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400f590 tx=0x7fd1c4008040 comp rx=0 tx=0).stop 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.868+0000 7fd1d4eb8640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1b4038740 0x7fd1b403abf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00a0900 tx=0x7fd1cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d791d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d791d640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 0x7fd1d0029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d791d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d01316d0 con 0x7fd1d009a660 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 0x7fd1d0029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient: get_auth_request con 0x7fd1d009a660 auth_method 0 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 10 monclient(hunting): handle_auth_done global_id 174105 payload 824 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.874+0000 7fd1d56b9640 1 -- 172.21.15.153:0/1011197775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd1d0135e70 con 0x7fd1d009a660 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.876+0000 7fd1d56b9640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 msgr2=0x7fd1d0029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.876+0000 7fd1d56b9640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 0x7fd1d0029e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd1c401a000 tx=0x7fd1c401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400d8a0 con 0x7fd1d009a660 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 10 monclient: dump: 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fd1c4025070 con 0x7fd1d009a660 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 msgr2=0x7fd1d0029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 0x7fd1d0029e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd1c401a000 tx=0x7fd1c401d000 comp rx=0 tx=0).stop 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 msgr2=0x7fd1d0029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1c17fa640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d009a660 0x7fd1d0029e00 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd1c401a000 tx=0x7fd1c401d000 comp rx=0 tx=0).stop 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 138, 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1b4038740 msgr2=0x7fd1b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd1b4038740 0x7fd1b403abf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00a0900 tx=0x7fd1cc009040 comp rx=0 tx=0).stop 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 10 monclient: shutdown 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 msgr2=0x7fd1d0099590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d5eba640 1 -- 172.21.15.153:0/1011197775 reap_dead start 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0099590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009f20 tx=0x7fd1c8036000 comp rx=0 tx=0).stop 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 shutdown_connections 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 --2- 172.21.15.153:0/1011197775 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1d0136c20 0x7fd1d0099590 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 >> 172.21.15.153:0/1011197775 conn(0x7fd1d0067110 msgr2=0x7fd1d00a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 shutdown_connections 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.877+0000 7fd1d791d640 1 -- 172.21.15.153:0/1011197775 wait complete. 2024-09-15T03:42:00.904 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:42:00.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:42:00.905 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.998+0000 7f05ac4b8640 10 monclient: get_monmap_and_config 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.998+0000 7f05ac4b8640 10 monclient: build_initial_monmap 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.998+0000 7f05ac4b8640 10 monclient: monmap: 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:00.998979+0000 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:00.998979+0000 2024-09-15T03:42:00.999 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.998+0000 7f05ac4b8640 1 Processor -- start 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.998+0000 7f05ac4b8640 1 -- start start 2024-09-15T03:42:01.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.998+0000 7f05ac4b8640 10 monclient: init 2024-09-15T03:42:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a41318f0 0x7f05a4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): picked mon.noname-b con 0x7f05a41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f05a409ea50 0x7f05a409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): picked mon.noname-a con 0x7f05a409ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05a4068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a4131cc0 con 0x7f05a4068d90 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a409fbf0 con 0x7f05a409ea50 2024-09-15T03:42:01.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a409fe30 con 0x7f05a41318f0 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05ac4b8640 10 monclient(hunting): authenticate will time out at 1518.638598s 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f05a409ea50 0x7f05a409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a41318f0 0x7f05a4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f05a409ea50 0x7f05a409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59876/0 (socket says 172.21.15.153:59876) 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 -- 172.21.15.153:0/3479075331 learned_addr learned my addr 172.21.15.153:0/3479075331 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): get_auth_request con 0x7f05a409ea50 auth_method 0 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aaa55640 1 --1- 172.21.15.153:0/3479075331 >> v1:172.21.15.153:6789/0 conn(0x7f05a4068d90 0x7f05a40691f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50896/0 (socket says 172.21.15.153:50896) 2024-09-15T03:42:01.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): get_auth_request con 0x7f05a41318f0 auth_method 0 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient(hunting): handle_auth_done global_id 174108 payload 274 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 -- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a41318f0 msgr2=0x7f05a4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 --2- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a41318f0 0x7f05a4131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 -- 172.21.15.153:0/3479075331 >> v1:172.21.15.153:6789/0 conn(0x7f05a4068d90 legacy=0x7f05a40691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 monclient: found mon.noname-a 2024-09-15T03:42:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05a9a53640 1 -- 172.21.15.153:0/3479075331 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a40a1210 con 0x7f05a409ea50 2024-09-15T03:42:01.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:00.999+0000 7f05aa254640 1 --2- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a41318f0 0x7f05a4131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.002+0000 7f05a9a53640 1 -- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a409ea50 msgr2=0x7f05a409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.002+0000 7f05a9a53640 1 --2- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a409ea50 0x7f05a409ee20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0598009910 tx=0x7f05980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a3af6fdc1860b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.002+0000 7f05a9252640 1 -- 172.21.15.153:0/3479075331 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059802aea0 con 0x7f05a409ea50 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.002+0000 7f05a9252640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.002+0000 7f05a9252640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.002+0000 7f05a9252640 10 monclient: dump: 2024-09-15T03:42:01.015 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 5 monclient: authenticate success, global_id 174108 2024-09-15T03:42:01.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 1 -- 172.21.15.153:0/3479075331 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0598032440 con 0x7f05a409ea50 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 4 set_mon_vals no callback set 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 1 -- 172.21.15.153:0/3479075331 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05980327e0 con 0x7f05a409ea50 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05a9252640 10 monclient: dump: 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.017 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 10 monclient: shutdown 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 -- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a409ea50 msgr2=0x7f05a409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a409ea50 0x7f05a409ee20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0598009910 tx=0x7f05980091d0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 -- 172.21.15.153:0/3479075331 shutdown_connections 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a409ea50 0x7f05a409ee20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/3479075331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a41318f0 0x7f05a4131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 -- 172.21.15.153:0/3479075331 >> 172.21.15.153:0/3479075331 conn(0x7f05a4067110 msgr2=0x7f05a4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 -- 172.21.15.153:0/3479075331 shutdown_connections 2024-09-15T03:42:01.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.003+0000 7f05ac4b8640 1 -- 172.21.15.153:0/3479075331 wait complete. 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient: build_initial_monmap 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient: monmap: 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 Processor -- start 2024-09-15T03:42:01.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 -- start start 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient: init 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05a409aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): picked mon.noname-a con 0x7f05a4068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): picked mon.noname-c con 0x7f05a409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f05a4140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): picked mon.noname-b con 0x7f05a41318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:01.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a409bf10 con 0x7f05a4068d90 2024-09-15T03:42:01.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40a43e0 con 0x7f05a41318f0 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a411dd50 con 0x7f05a409ea50 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05ac4b8640 10 monclient(hunting): authenticate will time out at 1518.643900s 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05aa254640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05a409aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05a9a53640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05aa254640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05a409aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59880/0 (socket says 172.21.15.153:59880) 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05a9a53640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46026/0 (socket says 172.21.15.153:46026) 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.004+0000 7f05aa254640 1 -- 172.21.15.153:0/234909653 learned_addr learned my addr 172.21.15.153:0/234909653 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): get_auth_request con 0x7f05a409ea50 auth_method 0 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): get_auth_request con 0x7f05a4068d90 auth_method 0 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aaa55640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f05a4140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aaa55640 10 monclient(hunting): get_auth_request con 0x7f05a41318f0 auth_method 0 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aaa55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aaa55640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aaa55640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient(hunting): handle_auth_done global_id 124628 payload 824 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 msgr2=0x7f05a4140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f05a4140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 msgr2=0x7f05a409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05a409aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 1 monclient: found mon.noname-c 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0009390 con 0x7f05a409ea50 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.005+0000 7f05aa254640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05a409aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 msgr2=0x7f05a4130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0598031000 tx=0x7f0598032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 msgr2=0x7f05a4130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 msgr2=0x7f05a4130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059802aea0 con 0x7f05a409ea50 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0598031000 tx=0x7f0598032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0598031000 tx=0x7f0598032000 comp rx=0 tx=0).stop 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: dump: 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.029 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 msgr2=0x7f05a4130c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409ea50 0x7f05a4130c30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:42:01.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05880056b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): picked mon.a con 0x7f05a4068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f0588008060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): picked mon.b con 0x7f05a41318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05880089e0 0x7f058800ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): picked mon.c con 0x7f05880089e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058800b910 con 0x7f05a4068d90 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058800ba50 con 0x7f05a41318f0 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058800bb90 con 0x7f05880089e0 2024-09-15T03:42:01.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05967fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05880056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): get_auth_request con 0x7f05a4068d90 auth_method 0 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05880089e0 0x7f058800ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): get_auth_request con 0x7f05880089e0 auth_method 0 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aa254640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.007+0000 7f05aaa55640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 10 monclient(hunting): handle_auth_done global_id 124628 payload 293 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f0588008060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05880089e0 msgr2=0x7f058800ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05880089e0 0x7f058800ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 msgr2=0x7f0588008060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f0588008060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 1 monclient: found mon.a 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f058800bdc0 con 0x7f05a4068d90 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05aa254640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 5 monclient: authenticate success, global_id 124628 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 10 monclient: _renew_subs 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0598009450 con 0x7f05a4068d90 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05a9a53640 10 monclient: get_auth_request con 0x7f05a41318f0 auth_method 0 2024-09-15T03:42:01.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05a41318f0 0x7f0588008060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 10 monclient: _renew_subs 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a411e0e0 con 0x7f05a4068d90 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 0x7f05a411bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4065e00 con 0x7f05a409aa40 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 0x7f05a411bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.008+0000 7f05a9a53640 10 monclient: get_auth_request con 0x7f05a409aa40 auth_method 0 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 10 monclient(hunting): handle_auth_done global_id 124631 payload 824 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.009+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05a411df20 con 0x7f05a409aa40 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05aa254640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 msgr2=0x7f05880056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05aa254640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05880056b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05a0006f20 tx=0x7f05a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0013680 con 0x7f05a4068d90 2024-09-15T03:42:01.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05967fc640 10 monclient: dump: 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a00137e0 con 0x7f05a4068d90 2024-09-15T03:42:01.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.010+0000 7f05a8a51640 4 set_mon_vals no callback set 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f05a0013950 con 0x7f05a4068d90 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f05a409ea50 0x7f0588042f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.0 v2:172.21.15.153:3300/0 4 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f05a0018030 con 0x7f05a4068d90 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 msgr2=0x7f05a411bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 0x7f05a411bb60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0598009210 tx=0x7f059800b050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059803de10 con 0x7f05a409aa40 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05aaa55640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f05a409ea50 0x7f0588042f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 10 monclient: dump: 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.038 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05aaa55640 10 monclient: get_auth_request con 0x7f05a409ea50 auth_method 0 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0598031c70 con 0x7f05a409aa40 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 msgr2=0x7f05a411bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 0x7f05a411bb60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0598009210 tx=0x7f059800b050 comp rx=0 tx=0).stop 2024-09-15T03:42:01.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 msgr2=0x7f05a411bb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:01.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.011+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a409aa40 0x7f05a411bb60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.014+0000 7f05aaa55640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f05a409ea50 0x7f0588042f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f059c007fd0 tx=0x7f059c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05ac4b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:01.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 0x7f05a4131030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05ac4b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a411e4a0 con 0x7f05a4130c30 2024-09-15T03:42:01.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 0x7f05a4131030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient: get_auth_request con 0x7f05a4130c30 auth_method 0 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 10 monclient(hunting): handle_auth_done global_id 124634 payload 824 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.017+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f05a409ae10 con 0x7f05a4130c30 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05a9a53640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 msgr2=0x7f05a4131030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05a9a53640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 0x7f05a4131030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f05a4068700 tx=0x7f0598034050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598046df0 con 0x7f05a4130c30 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 10 monclient: dump: 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f059803b810 con 0x7f05a4130c30 2024-09-15T03:42:01.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 msgr2=0x7f05a4131030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 0x7f05a4131030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f05a4068700 tx=0x7f0598034050 comp rx=0 tx=0).stop 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 msgr2=0x7f05a4131030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05967fc640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a4130c30 0x7f05a4131030 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f05a4068700 tx=0x7f0598034050 comp rx=0 tx=0).stop 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:01.045 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 138, 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:01.046 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:01.047 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:01.048 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:01.049 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:01.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:01.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:01.053 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.055 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.056 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:01.057 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f05a409ea50 msgr2=0x7f0588042f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f05a409ea50 0x7f0588042f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f059c007fd0 tx=0x7f059c009040 comp rx=0 tx=0).stop 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 10 monclient: shutdown 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 msgr2=0x7f05880056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05880056b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05a0006f20 tx=0x7f05a00090f0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05aaa55640 1 -- 172.21.15.153:0/234909653 reap_dead start 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 shutdown_connections 2024-09-15T03:42:01.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 --2- 172.21.15.153:0/234909653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a4068d90 0x7f05880056b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 >> 172.21.15.153:0/234909653 conn(0x7f05a4067110 msgr2=0x7f05a40a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:01.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 shutdown_connections 2024-09-15T03:42:01.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.020+0000 7f05ac4b8640 1 -- 172.21.15.153:0/234909653 wait complete. 2024-09-15T03:42:01.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:42:01.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:42:01.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.153+0000 7fed78e99640 10 monclient: get_monmap_and_config 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.153+0000 7fed78e99640 10 monclient: build_initial_monmap 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.153+0000 7fed78e99640 10 monclient: monmap: 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:01.154068+0000 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:01.154068+0000 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.154 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:01.155 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:01.155 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:01.155 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:01.155 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:01.155 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 1 Processor -- start 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 1 -- start start 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient: init 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fed74131980 0x7fed74131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient(hunting): picked mon.noname-a con 0x7fed74131980 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed7409fe10 addr v1:172.21.15.181:6789/0 2024-09-15T03:42:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed7409f140 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740a14d0 con 0x7fed7409f140 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed741325b0 con 0x7fed7409fe10 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740a11f0 con 0x7fed74131980 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed78e99640 10 monclient(hunting): authenticate will time out at 1518.793717s 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fed74131980 0x7fed74131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed72d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fed7409f140 0x7fed74138cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50912/0 (socket says 172.21.15.153:50912) 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed72d9d640 1 -- 172.21.15.153:0/3147401419 learned_addr learned my addr 172.21.15.153:0/3147401419 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): get_auth_request con 0x7fed74131980 auth_method 0 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.154+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 10 monclient(hunting): handle_auth_done global_id 174114 payload 274 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 1 -- 172.21.15.153:0/3147401419 >> v1:172.21.15.181:6789/0 conn(0x7fed7409fe10 legacy=0x7fed7409ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 1 -- 172.21.15.153:0/3147401419 >> v1:172.21.15.153:6789/0 conn(0x7fed7409f140 legacy=0x7fed74138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 1 monclient: found mon.noname-a 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.155+0000 7fed7259c640 1 -- 172.21.15.153:0/3147401419 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed74132290 con 0x7fed74131980 2024-09-15T03:42:01.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.157+0000 7fed7259c640 1 -- 172.21.15.153:0/3147401419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 msgr2=0x7fed74131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.157+0000 7fed7259c640 1 --2- 172.21.15.153:0/3147401419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74131d50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fed64009850 tx=0x7fed640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624d361eed04ed64 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 1 -- 172.21.15.153:0/3147401419 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64032440 con 0x7fed74131980 2024-09-15T03:42:01.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:01.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: dump: 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 1 -- 172.21.15.153:0/3147401419 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed640325a0 con 0x7fed74131980 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 5 monclient: authenticate success, global_id 174114 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 1 -- 172.21.15.153:0/3147401419 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64032940 con 0x7fed74131980 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed71d9b640 10 monclient: dump: 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.167 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 4 set_mon_vals no callback set 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 10 monclient: shutdown 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 -- 172.21.15.153:0/3147401419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 msgr2=0x7fed74131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 --2- 172.21.15.153:0/3147401419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74131d50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fed64009850 tx=0x7fed640090f0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 -- 172.21.15.153:0/3147401419 shutdown_connections 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 --2- 172.21.15.153:0/3147401419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74131d50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 -- 172.21.15.153:0/3147401419 >> 172.21.15.153:0/3147401419 conn(0x7fed740a5050 msgr2=0x7fed740a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:01.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 -- 172.21.15.153:0/3147401419 shutdown_connections 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.158+0000 7fed78e99640 1 -- 172.21.15.153:0/3147401419 wait complete. 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient: build_initial_monmap 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient: monmap: 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 Processor -- start 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 -- start start 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient: init 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed7409f140 0x7fed74134500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): picked mon.noname-c con 0x7fed7409f140 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7409fe10 0x7fed74134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): picked mon.noname-b con 0x7fed7409fe10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74134f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): picked mon.noname-a con 0x7fed74131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed74137db0 con 0x7fed74131980 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed74136b70 con 0x7fed7409fe10 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed74137ad0 con 0x7fed7409f140 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.159+0000 7fed78e99640 10 monclient(hunting): authenticate will time out at 1518.798935s 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed7409f140 0x7fed74134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74134f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed7409f140 0x7fed74134500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46072/0 (socket says 172.21.15.153:46072) 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74134f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59912/0 (socket says 172.21.15.153:59912) 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 1 -- 172.21.15.153:0/3703435533 learned_addr learned my addr 172.21.15.153:0/3703435533 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): get_auth_request con 0x7fed7409f140 auth_method 0 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): get_auth_request con 0x7fed74131980 auth_method 0 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed6bfff640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7409fe10 0x7fed74134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed6bfff640 10 monclient(hunting): get_auth_request con 0x7fed7409fe10 auth_method 0 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient(hunting): handle_auth_done global_id 174117 payload 824 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed7409f140 msgr2=0x7fed74134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed7409f140 0x7fed74134500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7409fe10 msgr2=0x7fed74134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7409fe10 0x7fed74134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 monclient: found mon.noname-a 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed72d9d640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64009390 con 0x7fed74131980 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.160+0000 7fed6bfff640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7409fe10 0x7fed74134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed72d9d640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 msgr2=0x7fed74134f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed72d9d640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74134f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bf20 tx=0x7fed6c009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c014440 con 0x7fed74131980 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: dump: 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.174 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c0145a0 con 0x7fed74131980 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 5 monclient: authenticate success, global_id 174117 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 10 monclient: _renew_subs 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed74138090 con 0x7fed74131980 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0149a0 con 0x7fed74131980 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed6a7fc640 10 monclient: dump: 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.175 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 10 monclient: _renew_subs 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed7159a640 4 set_mon_vals no callback set 2024-09-15T03:42:01.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.162+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed74139a20 con 0x7fed74131980 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed78e99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed78e99640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 0x7fed7402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed78e99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed74065e00 con 0x7fed74138ac0 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fed6c0133a0 con 0x7fed74131980 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed6a7fc640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed54038740 0x7fed5403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fed6c04c980 con 0x7fed74131980 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 0x7fed7402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient: get_auth_request con 0x7fed74138ac0 auth_method 0 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed6bfff640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed54038740 0x7fed5403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.163+0000 7fed6bfff640 10 monclient: get_auth_request con 0x7fed54038740 auth_method 0 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.164+0000 7fed7259c640 10 monclient(hunting): handle_auth_done global_id 144352 payload 824 2024-09-15T03:42:01.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.164+0000 7fed7259c640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed74069f20 con 0x7fed74138ac0 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed7259c640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 msgr2=0x7fed7402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed7259c640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 0x7fed7402aa40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed64035040 tx=0x7fed64032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6bfff640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed54038740 0x7fed5403abf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009970 tx=0x7fed5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed640319f0 con 0x7fed74138ac0 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:01.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 10 monclient: dump: 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fed64031b50 con 0x7fed74138ac0 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 msgr2=0x7fed7402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 0x7fed7402aa40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed64035040 tx=0x7fed64032000 comp rx=0 tx=0).stop 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 msgr2=0x7fed7402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.166+0000 7fed6a7fc640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed74138ac0 0x7fed7402aa40 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed64035040 tx=0x7fed64032000 comp rx=0 tx=0).stop 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed78e99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed78e99640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 0x7fed7405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed78e99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed741378d0 con 0x7fed7405e310 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 0x7fed7405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient: get_auth_request con 0x7fed7405e310 auth_method 0 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.172+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.173+0000 7fed7259c640 10 monclient(hunting): handle_auth_done global_id 144355 payload 824 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.173+0000 7fed7259c640 1 -- 172.21.15.153:0/3703435533 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed74135fa0 con 0x7fed7405e310 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed7259c640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 msgr2=0x7fed7405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed7259c640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 0x7fed7405e710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed64003080 tx=0x7fed6403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64045a30 con 0x7fed7405e310 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 10 monclient: dump: 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.184 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fed64045b90 con 0x7fed7405e310 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 msgr2=0x7fed7405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 0x7fed7405e710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed64003080 tx=0x7fed6403d000 comp rx=0 tx=0).stop 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 msgr2=0x7fed7405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:01.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.175+0000 7fed6a7fc640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed7405e310 0x7fed7405e710 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed64003080 tx=0x7fed6403d000 comp rx=0 tx=0).stop 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 138, 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:42:01.186 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:01.187 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:01.188 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:01.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:01.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:01.195 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:42:01.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:42:01.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:01.198 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:01.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed54038740 msgr2=0x7fed5403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fed54038740 0x7fed5403abf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009970 tx=0x7fed5c008040 comp rx=0 tx=0).stop 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 10 monclient: shutdown 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 msgr2=0x7fed74134f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 --2- 172.21.15.153:0/3703435533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed74131980 0x7fed74134f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bf20 tx=0x7fed6c009720 comp rx=0 tx=0).stop 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed72d9d640 1 -- 172.21.15.153:0/3703435533 reap_dead start 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed72d9d640 1 -- 172.21.15.153:0/3703435533 reap_dead start 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 shutdown_connections 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 >> 172.21.15.153:0/3703435533 conn(0x7fed740a5050 msgr2=0x7fed7409dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 shutdown_connections 2024-09-15T03:42:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.176+0000 7fed78e99640 1 -- 172.21.15.153:0/3703435533 wait complete. 2024-09-15T03:42:01.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:42:01.201 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:42:01.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.292+0000 7f9e5bc34640 10 monclient: get_monmap_and_config 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.292+0000 7f9e5bc34640 10 monclient: build_initial_monmap 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.292+0000 7f9e5bc34640 10 monclient: monmap: 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:01.293089+0000 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:01.293089+0000 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:01.293 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:01.294 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:01.294 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.292+0000 7f9e5bc34640 1 Processor -- start 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 1 -- start start 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient: init 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9e540981c0 0x7f9e54098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): picked mon.noname-a con 0x7f9e540981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e54131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9e54132170 0x7f9e54067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): picked mon.noname-c con 0x7f9e54132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e54098ad0 con 0x7f9e54131da0 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e54131bd0 con 0x7f9e540981c0 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5409ca60 con 0x7f9e54132170 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5bc34640 10 monclient(hunting): authenticate will time out at 1518.932625s 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9e540981c0 0x7f9e54098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e591cf640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9e54131da0 0x7f9e54067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50916/0 (socket says 172.21.15.153:50916) 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e591cf640 1 -- 172.21.15.153:0/4179160186 learned_addr learned my addr 172.21.15.153:0/4179160186 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): get_auth_request con 0x7f9e540981c0 auth_method 0 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e599d0640 10 monclient(hunting): handle_auth_done global_id 174120 payload 274 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.293+0000 7f9e5a1d1640 1 --2- 172.21.15.153:0/4179160186 >> v2:172.21.15.181:3300/0 conn(0x7f9e54132170 0x7f9e54067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 1 -- 172.21.15.153:0/4179160186 >> v2:172.21.15.181:3300/0 conn(0x7f9e54132170 msgr2=0x7f9e54067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 1 --2- 172.21.15.153:0/4179160186 >> v2:172.21.15.181:3300/0 conn(0x7f9e54132170 0x7f9e54067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 1 -- 172.21.15.153:0/4179160186 >> v1:172.21.15.153:6789/0 conn(0x7f9e54131da0 legacy=0x7f9e54067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 1 monclient: found mon.noname-a 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e599d0640 1 -- 172.21.15.153:0/4179160186 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5409cca0 con 0x7f9e540981c0 2024-09-15T03:42:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e5a1d1640 10 monclient: get_auth_request con 0x7f9e54132170 auth_method 0 2024-09-15T03:42:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.294+0000 7f9e5a1d1640 1 --2- 172.21.15.153:0/4179160186 >> v2:172.21.15.181:3300/0 conn(0x7f9e54132170 0x7f9e54067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:42:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.296+0000 7f9e599d0640 1 -- 172.21.15.153:0/4179160186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e540981c0 msgr2=0x7f9e54098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.296+0000 7f9e599d0640 1 --2- 172.21.15.153:0/4179160186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e540981c0 0x7f9e54098590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e5406f820 tx=0x7f9e500090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a560c5cb5332c481 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 1 -- 172.21.15.153:0/4179160186 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e50032440 con 0x7f9e540981c0 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: dump: 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 5 monclient: authenticate success, global_id 174120 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 1 -- 172.21.15.153:0/4179160186 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e500325a0 con 0x7f9e540981c0 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 4 set_mon_vals no callback set 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 1 -- 172.21.15.153:0/4179160186 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e50032940 con 0x7f9e540981c0 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e589ce640 10 monclient: dump: 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 10 monclient: shutdown 2024-09-15T03:42:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 -- 172.21.15.153:0/4179160186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e540981c0 msgr2=0x7f9e54098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/4179160186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e540981c0 0x7f9e54098590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e5406f820 tx=0x7f9e500090f0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 -- 172.21.15.153:0/4179160186 shutdown_connections 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/4179160186 >> v2:172.21.15.181:3300/0 conn(0x7f9e54132170 0x7f9e54067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/4179160186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e540981c0 0x7f9e54098590 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 -- 172.21.15.153:0/4179160186 >> 172.21.15.153:0/4179160186 conn(0x7f9e540a1e30 msgr2=0x7f9e540a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 -- 172.21.15.153:0/4179160186 shutdown_connections 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.297+0000 7f9e5bc34640 1 -- 172.21.15.153:0/4179160186 wait complete. 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.298+0000 7f9e5bc34640 10 monclient: build_initial_monmap 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.298+0000 7f9e5bc34640 10 monclient: monmap: 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:01.308 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.298+0000 7f9e5bc34640 1 Processor -- start 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.298+0000 7f9e5bc34640 1 -- start start 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.298+0000 7f9e5bc34640 10 monclient: init 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e540981c0 0x7f9e5411db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): picked mon.noname-b con 0x7f9e540981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e54131da0 0x7f9e5411e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): picked mon.noname-c con 0x7f9e54131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 0x7f9e5411e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): picked mon.noname-a con 0x7f9e54132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e541200a0 con 0x7f9e54132170 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54121790 con 0x7f9e540981c0 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54121e20 con 0x7f9e54131da0 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5bc34640 10 monclient(hunting): authenticate will time out at 1518.938162s 2024-09-15T03:42:01.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 0x7f9e5411e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e54131da0 0x7f9e5411e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 0x7f9e5411e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59938/0 (socket says 172.21.15.153:59938) 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 -- 172.21.15.153:0/3297517939 learned_addr learned my addr 172.21.15.153:0/3297517939 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): get_auth_request con 0x7f9e54132170 auth_method 0 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): get_auth_request con 0x7f9e54131da0 auth_method 0 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e540981c0 0x7f9e5411db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): get_auth_request con 0x7f9e540981c0 auth_method 0 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e599d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient(hunting): handle_auth_done global_id 174123 payload 824 2024-09-15T03:42:01.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 -- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e54131da0 msgr2=0x7f9e5411e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e54131da0 0x7f9e5411e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 -- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e540981c0 msgr2=0x7f9e5411db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e540981c0 0x7f9e5411db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient: _finish_hunting 0 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 monclient: found mon.noname-a 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e5a1d1640 1 -- 172.21.15.153:0/3297517939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50009390 con 0x7f9e54132170 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.299+0000 7f9e591cf640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e54131da0 0x7f9e5411e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e5a1d1640 1 -- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 msgr2=0x7f9e5411e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e5a1d1640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 0x7f9e5411e590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00a380 tx=0x7f9e4c007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c00d860 con 0x7f9e54132170 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: dump: 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.314 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: _finish_auth 0 2024-09-15T03:42:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c00d9c0 con 0x7f9e54132170 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e5bc34640 5 monclient: authenticate success, global_id 174123 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e5bc34640 10 monclient: _renew_subs 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e5bc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5409ba90 con 0x7f9e54132170 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c00dd80 con 0x7f9e54132170 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e49ffb640 10 monclient: dump: 2024-09-15T03:42:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.302+0000 7f9e5bc34640 10 monclient: _renew_subs 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.302+0000 7f9e5bc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.302+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e54130670 con 0x7f9e54132170 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.301+0000 7f9e4bfff640 4 set_mon_vals no callback set 2024-09-15T03:42:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.302+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9e4c01e3b0 con 0x7f9e54132170 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e49ffb640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e38038740 0x7f9e3803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9e4c04cfd0 con 0x7f9e54132170 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e5bc34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e5bc34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e5bc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5404bf60 con 0x7f9e54132170 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e599d0640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e38038740 0x7f9e3803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.303+0000 7f9e599d0640 10 monclient: get_auth_request con 0x7f9e38038740 auth_method 0 2024-09-15T03:42:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.305+0000 7f9e599d0640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e38038740 0x7f9e3803abf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9e54132870 tx=0x7f9e50032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.307+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9e4c01a070 con 0x7f9e54132170 2024-09-15T03:42:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.307+0000 7f9e49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.307+0000 7f9e49ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.519+0000 7f9e5bc34640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:42:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.519+0000 7f9e5bc34640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:42:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.519+0000 7f9e5bc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:01.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.519+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9e54098ee0 con 0x7f9e54132170 2024-09-15T03:42:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f9e49ffb640 1 -- 172.21.15.153:0/3297517939 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f9e4c012a80 con 0x7f9e54132170 2024-09-15T03:42:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f9e49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:42:01.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f9e49ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:01.522 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.522 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=2135303686,config=3140514009,health=48896099,kv=3235807719,logm=2389882036}) 2024-09-15T03:42:01.522 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=1,health=11,kv=6,logm=56} crc {auth=363427127,config=1228700967,health=3462558291,kv=3561859953,logm=354548362}) 2024-09-15T03:42:01.522 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.522 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=2135303686,config=3140514009,health=48896099,kv=3235807719,logm=2389882036}) 2024-09-15T03:42:01.523 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.520+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=748562867,config=3140514009,health=48896099,kv=3235807718,logm=911743804}) 2024-09-15T03:42:01.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.521+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.521+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156576664}) 2024-09-15T03:42:01.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.521+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618366265}) 2024-09-15T03:42:01.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.521+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.521+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156576664}) 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.521+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186470806}) 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288758902}) 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430786378}) 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288758902}) 2024-09-15T03:42:01.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270560220}) 2024-09-15T03:42:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e38038740 msgr2=0x7f9e3803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e38038740 0x7f9e3803abf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9e54132870 tx=0x7f9e50032000 comp rx=0 tx=0).stop 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 10 monclient: shutdown 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 msgr2=0x7f9e5411e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 0x7f9e5411e590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00a380 tx=0x7f9e4c007900 comp rx=0 tx=0).stop 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 shutdown_connections 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9e38038740 0x7f9e3803abf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e54132170 0x7f9e5411e590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e54131da0 0x7f9e5411e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 --2- 172.21.15.153:0/3297517939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e540981c0 0x7f9e5411db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 >> 172.21.15.153:0/3297517939 conn(0x7f9e540a1e30 msgr2=0x7f9e5409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 shutdown_connections 2024-09-15T03:42:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:01.523+0000 7f9e5bc34640 1 -- 172.21.15.153:0/3297517939 wait complete. 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.524+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.524+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167021802}) 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.524+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=243440954}) 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.524+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.524+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167021802}) 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.524+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245101017}) 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.525+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.525+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862611574}) 2024-09-15T03:42:01.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.525+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=855396962}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.525+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.525+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862611574}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.525+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152143638}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259641079}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142112098}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259641079}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955890586}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112306022}) 2024-09-15T03:42:01.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966026178}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112306022}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.526+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3677467423}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.527+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.527+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2254648922}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.527+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062501476}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.527+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.527+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2254648922}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.527+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482544867}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.528+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.528+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100125546}) 2024-09-15T03:42:01.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.528+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4088224202}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.528+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.528+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100125546}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.528+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240111367}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.529+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.529+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080223087}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.529+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3625082263}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.529+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.529+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080223087}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.529+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1532115857}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.530+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.530+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479230594}) 2024-09-15T03:42:01.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.530+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=298013152}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.530+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.530+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479230594}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.530+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3170126890}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.531+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.531+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752528193}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.531+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184823717}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.531+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.531+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752528193}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.531+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323367335}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.532+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.532+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713887678}) 2024-09-15T03:42:01.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.532+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890240040}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.532+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.532+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713887678}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.532+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=219305006}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.533+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.533+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336476225}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.533+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162787751}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.533+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.533+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336476225}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.533+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619266749}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720210958}) 2024-09-15T03:42:01.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3012078402}) 2024-09-15T03:42:01.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720210958}) 2024-09-15T03:42:01.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656942345}) 2024-09-15T03:42:01.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839271418}) 2024-09-15T03:42:01.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=524459825}) 2024-09-15T03:42:01.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.534+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839271418}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2540371604}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702796416}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645656509}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702796416}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.535+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=255270385}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.536+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.536+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641604746}) 2024-09-15T03:42:01.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.536+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2207410854}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.536+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.536+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641604746}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.536+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3338204802}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.537+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.537+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260488010}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.537+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582857061}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.537+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.537+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260488010}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.537+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=938787725}) 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661328851}) 2024-09-15T03:42:01.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817245625}) 2024-09-15T03:42:01.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661328851}) 2024-09-15T03:42:01.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420639241}) 2024-09-15T03:42:01.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:42:01.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.538+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088496607}) 2024-09-15T03:42:01.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339109049}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088496607}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3070083440}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877218681}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2280594930}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877218681}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.539+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590291682}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.540+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.540+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086215866}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.540+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3761432743}) 2024-09-15T03:42:01.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.540+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.540+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086215866}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.540+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706203771}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478275238}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2036181105}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478275238}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3749775857}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441862441}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454042040}) 2024-09-15T03:42:01.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441862441}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.541+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372746010}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.542+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.542+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729913868}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.542+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=146146988}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.542+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.542+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729913868}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.542+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164084625}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555755894}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396962060}) 2024-09-15T03:42:01.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555755894}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033705732}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528448553}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220397494}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528448553}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.543+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360616173}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.544+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.544+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972399010}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.544+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1049179050}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.544+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.544+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972399010}) 2024-09-15T03:42:01.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.544+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508206403}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499337024}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018159594}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499337024}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255688632}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302917446}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1459337707}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302917446}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.545+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635667154}) 2024-09-15T03:42:01.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.546+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.546+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191303103}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.546+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2447553101}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.546+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.546+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191303103}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.546+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298624084}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.547+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.547+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329128298}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.547+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281957389}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.547+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.547+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329128298}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.547+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434468522}) 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821968747}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112694184}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821968747}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1471389475}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174242}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322130363}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174242}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.548+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2047653405}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.549+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.549+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606962920}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.549+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1860145052}) 2024-09-15T03:42:01.570 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.549+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.549+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606962920}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.549+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517438179}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405862203}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1862499125}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405862203}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3413880194}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.550+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20756907}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011636584}) 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.571 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20756907}) 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1634816961}) 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627876818}) 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1167699279}) 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627876818}) 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092703538}) 2024-09-15T03:42:01.572 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206652977}) 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407218782}) 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206652977}) 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529327424}) 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443564128}) 2024-09-15T03:42:01.573 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948185691}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443564128}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464181157}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841475108}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177965615}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841475108}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.553+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797244496}) 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.574 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781749806}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2712975823}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781749806}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1207455255}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390487991}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3317687386}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390487991}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2111444212}) 2024-09-15T03:42:01.575 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663536668}) 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520717992}) 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663536668}) 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861435577}) 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306039347}) 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273733345}) 2024-09-15T03:42:01.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306039347}) 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2521844603}) 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106990762}) 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753597536}) 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106990762}) 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139578573}) 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935792533}) 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=822257871}) 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935792533}) 2024-09-15T03:42:01.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892463694}) 2024-09-15T03:42:01.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068054572}) 2024-09-15T03:42:01.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3406504076}) 2024-09-15T03:42:01.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068054572}) 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=859967518}) 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804921112}) 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3337733226}) 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804921112}) 2024-09-15T03:42:01.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:01.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732149343}) 2024-09-15T03:42:02.005 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.004+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.004+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891696679}) 2024-09-15T03:42:02.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.004+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170609639}) 2024-09-15T03:42:02.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.004+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.004+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891696679}) 2024-09-15T03:42:02.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.004+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057856673}) 2024-09-15T03:42:02.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.069+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.069+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121108739}) 2024-09-15T03:42:02.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.069+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451525418}) 2024-09-15T03:42:02.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.069+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.069+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121108739}) 2024-09-15T03:42:02.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.069+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035082365}) 2024-09-15T03:42:02.107 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.106+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.106+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244693461}) 2024-09-15T03:42:02.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.106+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362423938}) 2024-09-15T03:42:02.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.106+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.106+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244693461}) 2024-09-15T03:42:02.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.106+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716849790}) 2024-09-15T03:42:02.110 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.110 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758708748}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941760761}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758708748}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196784736}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585876542}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843917782}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585876542}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282209435}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164221165}) 2024-09-15T03:42:02.111 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699094173}) 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.108+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164221165}) 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784783515}) 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150464529}) 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=773974669}) 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150464529}) 2024-09-15T03:42:02.112 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.109+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4245504854}) 2024-09-15T03:42:02.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.110+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.110+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20590407}) 2024-09-15T03:42:02.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.110+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829131806}) 2024-09-15T03:42:02.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.110+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.110+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20590407}) 2024-09-15T03:42:02.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.110+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817294138}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.111+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.111+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336811110}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.111+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936026967}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.111+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.111+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336811110}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.111+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414792437}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.112+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.112+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3142092404}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.112+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189257711}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.112+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.112+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3142092404}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.112+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381615964}) 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198358272}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054600824}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198358272}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742842639}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229902709}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984366968}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229902709}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.113+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2326372914}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.114+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.114+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821259419}) 2024-09-15T03:42:02.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.114+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108276899}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.114+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.114+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821259419}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.114+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122024918}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541319065}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=246297438}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541319065}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182641152}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388416938}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691720792}) 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388416938}) 2024-09-15T03:42:02.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.115+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453206944}) 2024-09-15T03:42:02.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.123+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.123+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115032546}) 2024-09-15T03:42:02.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.123+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390125671}) 2024-09-15T03:42:02.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.123+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.123+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115032546}) 2024-09-15T03:42:02.125 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.123+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3544540219}) 2024-09-15T03:42:02.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.124+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.124+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=38} crc {logm=1627352256,mdsmap=1531151339,mgr=1013486720,mgr_metadata=2280977772,mgrstat=2744806935,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=88794453}) 2024-09-15T03:42:02.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.124+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=1157390175,mdsmap=1531151339,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=598380946,mon_config_key=4117641549,monmap=3140514009,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1190647597}) 2024-09-15T03:42:02.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.124+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.124+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=38} crc {logm=1627352256,mdsmap=1531151339,mgr=1013486720,mgr_metadata=2280977772,mgrstat=2744806935,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=88794453}) 2024-09-15T03:42:02.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.124+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_snap=2,osdmap=38} crc {logm=2179693991,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1841444161,mon_config_key=3380917293,monmap=244636553,osd_metadata=851455416,osd_snap=454132730,osdmap=209467357}) 2024-09-15T03:42:02.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.131+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.131+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=1410219205}) 2024-09-15T03:42:02.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.131+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=65} crc {osdmap=1213523683}) 2024-09-15T03:42:02.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.131+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:42:02.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.131+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=1410219205}) 2024-09-15T03:42:02.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:42:02.131+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=614620653}) 2024-09-15T03:42:02.548 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:42:02.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.653+0000 7f1bc1c78640 10 monclient: get_monmap_and_config 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.653+0000 7f1bc1c78640 10 monclient: build_initial_monmap 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.653+0000 7f1bc1c78640 10 monclient: monmap: 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:02.653931+0000 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:02.653931+0000 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:02.655 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:02.655 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:02.655 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:02.655 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:02.655 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.653+0000 7f1bc1c78640 1 Processor -- start 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 -- start start 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient: init 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc1318f0 0x7f1bbc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): picked mon.noname-b con 0x7f1bbc1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1bbc09ea50 0x7f1bbc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): picked mon.noname-c con 0x7f1bbc09ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1bbc068d90 0x7f1bbc0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): picked mon.noname-a con 0x7f1bbc068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bbc131cc0 con 0x7f1bbc068d90 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bbc09fbf0 con 0x7f1bbc09ea50 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc09fe30 con 0x7f1bbc1318f0 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bc1c78640 10 monclient(hunting): authenticate will time out at 1520.293551s 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1bbc068d90 0x7f1bbc0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc1318f0 0x7f1bbc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1bbc068d90 0x7f1bbc0691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59954/0 (socket says 172.21.15.153:59954) 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 -- 172.21.15.153:0/1536555543 learned_addr learned my addr 172.21.15.153:0/1536555543 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): get_auth_request con 0x7f1bbc068d90 auth_method 0 2024-09-15T03:42:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): get_auth_request con 0x7f1bbc1318f0 auth_method 0 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_done global_id 174126 payload 274 2024-09-15T03:42:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbaffd640 1 --2- 172.21.15.153:0/1536555543 >> v2:172.21.15.181:3300/0 conn(0x7f1bbc09ea50 0x7f1bbc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 -- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc1318f0 msgr2=0x7f1bbc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 --2- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc1318f0 0x7f1bbc131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 -- 172.21.15.153:0/1536555543 >> v2:172.21.15.181:3300/0 conn(0x7f1bbc09ea50 msgr2=0x7f1bbc09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 --2- 172.21.15.153:0/1536555543 >> v2:172.21.15.181:3300/0 conn(0x7f1bbc09ea50 0x7f1bbc09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 monclient: found mon.noname-a 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.654+0000 7f1bbbfff640 1 -- 172.21.15.153:0/1536555543 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc0a1210 con 0x7f1bbc068d90 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.655+0000 7f1bbaffd640 10 monclient: get_auth_request con 0x7f1bbc09ea50 auth_method 0 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.655+0000 7f1bbaffd640 1 --2- 172.21.15.153:0/1536555543 >> v2:172.21.15.181:3300/0 conn(0x7f1bbc09ea50 0x7f1bbc09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bbbfff640 1 -- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc068d90 msgr2=0x7f1bbc0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:02.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bbbfff640 1 --2- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc068d90 0x7f1bbc0691f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc068160 tx=0x7f1bb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a237b9f6e8479856 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 1 -- 172.21.15.153:0/1536555543 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4007610 con 0x7f1bbc068d90 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: dump: 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:02.672 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: _finish_auth 0 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 1 -- 172.21.15.153:0/1536555543 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb402d050 con 0x7f1bbc068d90 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bc1c78640 5 monclient: authenticate success, global_id 174126 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bc1c78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 1 -- 172.21.15.153:0/1536555543 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb40079a0 con 0x7f1bbc068d90 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:02.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bba7fc640 10 monclient: dump: 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:02.674 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bc1c78640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.657+0000 7f1bc1c78640 4 set_mon_vals no callback set 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 10 monclient: shutdown 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 -- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc068d90 msgr2=0x7f1bbc0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc068d90 0x7f1bbc0691f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc068160 tx=0x7f1bb4009840 comp rx=0 tx=0).stop 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 -- 172.21.15.153:0/1536555543 shutdown_connections 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc068d90 0x7f1bbc0691f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/1536555543 >> v2:172.21.15.181:3300/0 conn(0x7f1bbc09ea50 0x7f1bbc09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/1536555543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc1318f0 0x7f1bbc131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 -- 172.21.15.153:0/1536555543 >> 172.21.15.153:0/1536555543 conn(0x7f1bbc067110 msgr2=0x7f1bbc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:02.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 -- 172.21.15.153:0/1536555543 shutdown_connections 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.658+0000 7f1bc1c78640 1 -- 172.21.15.153:0/1536555543 wait complete. 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.659+0000 7f1bc1c78640 10 monclient: build_initial_monmap 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.659+0000 7f1bc1c78640 10 monclient: monmap: 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:02.676 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.659+0000 7f1bc1c78640 1 Processor -- start 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.659+0000 7f1bc1c78640 1 -- start start 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.659+0000 7f1bc1c78640 10 monclient: init 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 0x7f1bbc09a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): picked mon.noname-c con 0x7f1bbc068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bbc09ea50 0x7f1bbc1309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): picked mon.noname-b con 0x7f1bbc09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc1318f0 0x7f1bbc11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): picked mon.noname-a con 0x7f1bbc1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc09b3f0 con 0x7f1bbc1318f0 2024-09-15T03:42:02.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc09bf10 con 0x7f1bbc09ea50 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc09e3f0 con 0x7f1bbc068d90 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bc1c78640 10 monclient(hunting): authenticate will time out at 1520.299249s 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 0x7f1bbc09a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 0x7f1bbc09a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46098/0 (socket says 172.21.15.153:46098) 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 -- 172.21.15.153:0/959481134 learned_addr learned my addr 172.21.15.153:0/959481134 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): get_auth_request con 0x7f1bbc068d90 auth_method 0 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:02.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc1318f0 0x7f1bbc11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): get_auth_request con 0x7f1bbc1318f0 auth_method 0 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bbc09ea50 0x7f1bbc1309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): get_auth_request con 0x7f1bbc09ea50 auth_method 0 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:02.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient(hunting): handle_auth_done global_id 124646 payload 824 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 -- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bbc09ea50 msgr2=0x7f1bbc1309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bbc09ea50 0x7f1bbc1309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 -- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc1318f0 msgr2=0x7f1bbc11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc1318f0 0x7f1bbc11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 monclient: found mon.noname-c 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:02.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbb7fe640 1 -- 172.21.15.153:0/959481134 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb4009ae0 con 0x7f1bbc068d90 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.660+0000 7f1bbbfff640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc1318f0 0x7f1bbc11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.662+0000 7f1bbb7fe640 1 -- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 msgr2=0x7f1bbc09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.662+0000 7f1bbb7fe640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 0x7f1bbc09a830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400ea50 tx=0x7f1ba40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.662+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4009600 con 0x7f1bbc068d90 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: dump: 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.681 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: _finish_auth 0 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba400cc20 con 0x7f1bbc068d90 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 5 monclient: authenticate success, global_id 124646 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 10 monclient: _renew_subs 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:02.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc141990 con 0x7f1bbc068d90 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4013470 con 0x7f1bbc068d90 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 10 monclient: dump: 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:02.683 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 10 monclient: _renew_subs 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb9ffb640 4 set_mon_vals no callback set 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc11dbc0 con 0x7f1bbc068d90 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.663+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1ba4013860 con 0x7f1bbc068d90 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bc1c78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bc1c78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bc1c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:02.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bbc11e1f0 con 0x7f1bbc068d90 2024-09-15T03:42:02.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bb37fe640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b9c038740 0x7f1b9c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:02.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1ba404c920 con 0x7f1bbc068d90 2024-09-15T03:42:02.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bbaffd640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b9c038740 0x7f1b9c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:02.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.664+0000 7f1bbaffd640 10 monclient: get_auth_request con 0x7f1b9c038740 auth_method 0 2024-09-15T03:42:02.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.667+0000 7f1bbaffd640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b9c038740 0x7f1b9c03abf0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00acc0 tx=0x7f1bac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:02.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.669+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1ba4023ad0 con 0x7f1bbc068d90 2024-09-15T03:42:02.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.669+0000 7f1bb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:02.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.669+0000 7f1bb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bc1c78640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:42:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bc1c78640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:42:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bc1c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1bbc04bf60 con 0x7f1bbc068d90 2024-09-15T03:42:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bb37fe640 1 -- 172.21.15.153:0/959481134 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f1ba40232f0 con 0x7f1bbc068d90 2024-09-15T03:42:02.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:42:02.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.887+0000 7f1bb37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:02.889 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:42:02.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b9c038740 msgr2=0x7f1b9c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b9c038740 0x7f1b9c03abf0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00acc0 tx=0x7f1bac008040 comp rx=0 tx=0).stop 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 10 monclient: shutdown 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 msgr2=0x7f1bbc09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 0x7f1bbc09a830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400ea50 tx=0x7f1ba40090f0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 shutdown_connections 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1b9c038740 0x7f1b9c03abf0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bbc1318f0 0x7f1bbc11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bbc09ea50 0x7f1bbc1309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 --2- 172.21.15.153:0/959481134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bbc068d90 0x7f1bbc09a830 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 >> 172.21.15.153:0/959481134 conn(0x7f1bbc067110 msgr2=0x7f1bbc1347e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 shutdown_connections 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.889+0000 7f1bc1c78640 1 -- 172.21.15.153:0/959481134 wait complete. 2024-09-15T03:42:02.901 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:42:02.901 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:42:02.901 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:42:02.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:42:02.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.994+0000 7f02fc63f640 10 monclient: get_monmap_and_config 2024-09-15T03:42:02.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.994+0000 7f02fc63f640 10 monclient: build_initial_monmap 2024-09-15T03:42:02.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.994+0000 7f02fc63f640 10 monclient: monmap: 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:02.995020+0000 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:02.995020+0000 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:02.996 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:02.997 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:02.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.994+0000 7f02fc63f640 1 Processor -- start 2024-09-15T03:42:02.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.994+0000 7f02fc63f640 1 -- start start 2024-09-15T03:42:02.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.994+0000 7f02fc63f640 10 monclient: init 2024-09-15T03:42:03.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:03.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T03:42:03.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): picked mon.noname-b con 0x7f02f41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02f409ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f02f4136c20 0x7f02f4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): picked mon.noname-c con 0x7f02f4136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f4131cc0 con 0x7f02f409ea50 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f409f530 con 0x7f02f4136c20 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f409fbf0 con 0x7f02f41318f0 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fc63f640 10 monclient(hunting): authenticate will time out at 1520.634528s 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02f9bda640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f02f409ea50 0x7f02f409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50922/0 (socket says 172.21.15.153:50922) 2024-09-15T03:42:03.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46114/0 (socket says 172.21.15.153:46114) 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02f9bda640 1 -- 172.21.15.153:0/246103212 learned_addr learned my addr 172.21.15.153:0/246103212 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): get_auth_request con 0x7f02f41318f0 auth_method 0 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient(hunting): handle_auth_done global_id 124649 payload 274 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 -- 172.21.15.153:0/246103212 >> v2:172.21.15.181:3300/0 conn(0x7f02f4136c20 msgr2=0x7f02f4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 --2- 172.21.15.153:0/246103212 >> v2:172.21.15.181:3300/0 conn(0x7f02f4136c20 0x7f02f4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 -- 172.21.15.153:0/246103212 >> v1:172.21.15.153:6789/0 conn(0x7f02f409ea50 legacy=0x7f02f409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:03.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient: _finish_hunting 0 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 monclient: found mon.noname-b 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.995+0000 7f02fa3db640 1 -- 172.21.15.153:0/246103212 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f409fe30 con 0x7f02f41318f0 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02fa3db640 1 -- 172.21.15.153:0/246103212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 msgr2=0x7f02f4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02fa3db640 1 --2- 172.21.15.153:0/246103212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4131f30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009e60 tx=0x7f02ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3428100b13cafcfa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 1 -- 172.21.15.153:0/246103212 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec007840 con 0x7f02f41318f0 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: dump: 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: _finish_auth 0 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 1 -- 172.21.15.153:0/246103212 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec02d050 con 0x7f02f41318f0 2024-09-15T03:42:03.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02fc63f640 5 monclient: authenticate success, global_id 124649 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02fc63f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02fc63f640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 1 -- 172.21.15.153:0/246103212 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec007c10 con 0x7f02f41318f0 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02fc63f640 4 set_mon_vals no callback set 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.998+0000 7f02f93d9640 10 monclient: dump: 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.008 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 10 monclient: shutdown 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 -- 172.21.15.153:0/246103212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 msgr2=0x7f02f4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 --2- 172.21.15.153:0/246103212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4131f30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009e60 tx=0x7f02ec009780 comp rx=0 tx=0).stop 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 -- 172.21.15.153:0/246103212 shutdown_connections 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 --2- 172.21.15.153:0/246103212 >> v2:172.21.15.181:3300/0 conn(0x7f02f4136c20 0x7f02f4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 --2- 172.21.15.153:0/246103212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4131f30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 -- 172.21.15.153:0/246103212 >> 172.21.15.153:0/246103212 conn(0x7f02f4067110 msgr2=0x7f02f4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 -- 172.21.15.153:0/246103212 shutdown_connections 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:02.999+0000 7f02fc63f640 1 -- 172.21.15.153:0/246103212 wait complete. 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient: build_initial_monmap 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient: monmap: 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:03.010 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 Processor -- start 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 -- start start 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient: init 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f409ea50 0x7f02f4141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): picked mon.noname-b con 0x7f02f409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): picked mon.noname-c con 0x7f02f41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4136c20 0x7f02f4099840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): picked mon.noname-a con 0x7f02f4136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:03.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f4141fe0 con 0x7f02f4136c20 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f409e0e0 con 0x7f02f409ea50 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f411ce90 con 0x7f02f41318f0 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fc63f640 10 monclient(hunting): authenticate will time out at 1520.639821s 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46126/0 (socket says 172.21.15.153:46126) 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fabdc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4136c20 0x7f02f4099840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 1 -- 172.21.15.153:0/2187139001 learned_addr learned my addr 172.21.15.153:0/2187139001 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 10 monclient(hunting): get_auth_request con 0x7f02f41318f0 auth_method 0 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02f9bda640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fabdc640 10 monclient(hunting): get_auth_request con 0x7f02f4136c20 auth_method 0 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fabdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fabdc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.000+0000 7f02fabdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fabdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fabdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fabdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fa3db640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f409ea50 0x7f02f4141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fa3db640 10 monclient(hunting): get_auth_request con 0x7f02f409ea50 auth_method 0 2024-09-15T03:42:03.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fa3db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fa3db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02fa3db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 10 monclient(hunting): handle_auth_done global_id 124652 payload 824 2024-09-15T03:42:03.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f409ea50 msgr2=0x7f02f4141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f409ea50 0x7f02f4141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4136c20 msgr2=0x7f02f4099840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4136c20 0x7f02f4099840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 10 monclient: _finish_hunting 0 2024-09-15T03:42:03.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 1 monclient: found mon.noname-c 2024-09-15T03:42:03.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:03.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:42:03.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.001+0000 7f02f9bda640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec009a20 con 0x7f02f41318f0 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f9bda640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 msgr2=0x7f02f4130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f9bda640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4130aa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02e800c5e0 tx=0x7f02e8004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8005cb0 con 0x7f02f41318f0 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: dump: 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.022 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: _finish_auth 0 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e8005e30 con 0x7f02f41318f0 2024-09-15T03:42:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 5 monclient: authenticate success, global_id 124652 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 10 monclient: _renew_subs 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f411d170 con 0x7f02f41318f0 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8007340 con 0x7f02f41318f0 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f27fc640 10 monclient: dump: 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02f8bd8640 4 set_mon_vals no callback set 2024-09-15T03:42:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 10 monclient: _renew_subs 2024-09-15T03:42:03.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:42:03.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.003+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f411bcc0 con 0x7f02f41318f0 2024-09-15T03:42:03.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fc63f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:03.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fc63f640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 0x7f02f4099d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f4065e00 con 0x7f02f4141640 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f02e801b400 con 0x7f02f41318f0 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 0x7f02f4099d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient: get_auth_request con 0x7f02f4141640 auth_method 0 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02f27fc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02e0038740 0x7f02e003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f02e804c180 con 0x7f02f41318f0 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 10 monclient(hunting): handle_auth_done global_id 174135 payload 824 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.004+0000 7f02fa3db640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02f404bf60 con 0x7f02f4141640 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.005+0000 7f02fabdc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02e0038740 0x7f02e003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.005+0000 7f02fabdc640 10 monclient: get_auth_request con 0x7f02e0038740 auth_method 0 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02fa3db640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 msgr2=0x7f02f4099d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02fa3db640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 0x7f02f4099d80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02f4068160 tx=0x7f02ec037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec031b90 con 0x7f02f4141640 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 10 monclient: dump: 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.029 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02fabdc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02e0038740 0x7f02e003abf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f02f40a0240 tx=0x7f02e4006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f02ec031cf0 con 0x7f02f4141640 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 msgr2=0x7f02f4099d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 0x7f02f4099d80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02f4068160 tx=0x7f02ec037000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 msgr2=0x7f02f4099d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.007+0000 7f02f27fc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4141640 0x7f02f4099d80 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02f4068160 tx=0x7f02ec037000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fc63f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fc63f640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 0x7f02f409a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fc63f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f411d300 con 0x7f02f4130fe0 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 0x7f02f409a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient: get_auth_request con 0x7f02f4130fe0 auth_method 0 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 10 monclient(hunting): handle_auth_done global_id 174138 payload 824 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.013+0000 7f02fa3db640 1 -- 172.21.15.153:0/2187139001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02f40a4cc0 con 0x7f02f4130fe0 2024-09-15T03:42:03.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.015+0000 7f02fa3db640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 msgr2=0x7f02f409a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.015+0000 7f02fa3db640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 0x7f02f409a2c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0097c0 tx=0x7f02ec03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.015+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec045640 con 0x7f02f4130fe0 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.015+0000 7f02f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.015+0000 7f02f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.015+0000 7f02f27fc640 10 monclient: dump: 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.032 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f02ec03b420 con 0x7f02f4130fe0 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 msgr2=0x7f02f409a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 0x7f02f409a2c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0097c0 tx=0x7f02ec03f040 comp rx=0 tx=0).stop 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 msgr2=0x7f02f409a2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:03.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02f27fc640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f4130fe0 0x7f02f409a2c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 138, 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:03.034 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.035 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.036 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:03.037 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.038 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:03.039 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.043 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.045 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.046 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.047 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:03.047 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:03.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02e0038740 msgr2=0x7f02e003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f02e0038740 0x7f02e003abf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f02f40a0240 tx=0x7f02e4006f90 comp rx=0 tx=0).stop 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 10 monclient: shutdown 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 msgr2=0x7f02f4130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 --2- 172.21.15.153:0/2187139001 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02f41318f0 0x7f02f4130aa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02e800c5e0 tx=0x7f02e8004960 comp rx=0 tx=0).stop 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fabdc640 1 -- 172.21.15.153:0/2187139001 reap_dead start 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fabdc640 1 -- 172.21.15.153:0/2187139001 reap_dead start 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 shutdown_connections 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 >> 172.21.15.153:0/2187139001 conn(0x7f02f4067110 msgr2=0x7f02f40a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 shutdown_connections 2024-09-15T03:42:03.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.016+0000 7f02fc63f640 1 -- 172.21.15.153:0/2187139001 wait complete. 2024-09-15T03:42:03.050 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:42:03.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:42:03.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:42:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.144+0000 7fb21875d640 10 monclient: get_monmap_and_config 2024-09-15T03:42:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.144+0000 7fb21875d640 10 monclient: build_initial_monmap 2024-09-15T03:42:03.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.144+0000 7fb21875d640 10 monclient: monmap: 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:03.144903+0000 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:03.144903+0000 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:03.146 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.144+0000 7fb21875d640 1 Processor -- start 2024-09-15T03:42:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.144+0000 7fb21875d640 1 -- start start 2024-09-15T03:42:03.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.144+0000 7fb21875d640 10 monclient: init 2024-09-15T03:42:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T03:42:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2101318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb21009ea50 0x7fb21009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient(hunting): picked mon.noname-a con 0x7fb21009ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb210136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb210131cc0 con 0x7fb210136c20 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21009fbf0 con 0x7fb2101318f0 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21009fe30 con 0x7fb21009ea50 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb21875d640 10 monclient(hunting): authenticate will time out at 1520.784398s 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb21009ea50 0x7fb21009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb216cfa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb210136c20 0x7fb210139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50934/0 (socket says 172.21.15.153:50934) 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb216cfa640 1 -- 172.21.15.153:0/1144021049 learned_addr learned my addr 172.21.15.153:0/1144021049 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:42:03.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): get_auth_request con 0x7fb21009ea50 auth_method 0 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient(hunting): handle_auth_done global_id 174141 payload 274 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 1 -- 172.21.15.153:0/1144021049 >> v1:172.21.15.181:6789/0 conn(0x7fb2101318f0 legacy=0x7fb210131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 1 -- 172.21.15.153:0/1144021049 >> v1:172.21.15.153:6789/0 conn(0x7fb210136c20 legacy=0x7fb210139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient: _finish_hunting 0 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 1 monclient: found mon.noname-a 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.145+0000 7fb215cf8640 1 -- 172.21.15.153:0/1144021049 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100a1210 con 0x7fb21009ea50 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb215cf8640 1 -- 172.21.15.153:0/1144021049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb21009ea50 msgr2=0x7fb21009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb215cf8640 1 --2- 172.21.15.153:0/1144021049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb21009ea50 0x7fb21009ee20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb204009850 tx=0x7fb2040090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1720d54c7ca5b0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 1 -- 172.21.15.153:0/1144021049 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204032440 con 0x7fb21009ea50 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: dump: 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: _finish_auth 0 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 1 -- 172.21.15.153:0/1144021049 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2040325a0 con 0x7fb21009ea50 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb21875d640 5 monclient: authenticate success, global_id 174141 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb21875d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb21875d640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 1 -- 172.21.15.153:0/1144021049 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204032940 con 0x7fb21009ea50 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb21875d640 4 set_mon_vals no callback set 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.148+0000 7fb2154f7640 10 monclient: dump: 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.158 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 10 monclient: shutdown 2024-09-15T03:42:03.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 -- 172.21.15.153:0/1144021049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb21009ea50 msgr2=0x7fb21009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 --2- 172.21.15.153:0/1144021049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb21009ea50 0x7fb21009ee20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb204009850 tx=0x7fb2040090f0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 -- 172.21.15.153:0/1144021049 shutdown_connections 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 --2- 172.21.15.153:0/1144021049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb21009ea50 0x7fb21009ee20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 -- 172.21.15.153:0/1144021049 >> 172.21.15.153:0/1144021049 conn(0x7fb210067110 msgr2=0x7fb210067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 -- 172.21.15.153:0/1144021049 shutdown_connections 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 1 -- 172.21.15.153:0/1144021049 wait complete. 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 10 monclient: build_initial_monmap 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.149+0000 7fb21875d640 10 monclient: monmap: 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 Processor -- start 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 -- start start 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient: init 2024-09-15T03:42:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:03.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:42:03.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb21009ea50 0x7fb210130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): picked mon.noname-b con 0x7fb21009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:03.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2101318f0 0x7fb210099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2101318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb210136c20 0x7fb210099e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): picked mon.noname-a con 0x7fb210136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21009b8e0 con 0x7fb210136c20 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21009e000 con 0x7fb21009ea50 2024-09-15T03:42:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21011bba0 con 0x7fb2101318f0 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb21875d640 10 monclient(hunting): authenticate will time out at 1520.789811s 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2101318f0 0x7fb210099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb216cfa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb210136c20 0x7fb210099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2101318f0 0x7fb210099920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46138/0 (socket says 172.21.15.153:46138) 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 1 -- 172.21.15.153:0/4280933633 learned_addr learned my addr 172.21.15.153:0/4280933633 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 10 monclient(hunting): get_auth_request con 0x7fb2101318f0 auth_method 0 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb215cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb216cfa640 10 monclient(hunting): get_auth_request con 0x7fb210136c20 auth_method 0 2024-09-15T03:42:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb216cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb216cfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.150+0000 7fb216cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb215cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb215cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb215cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb2164f9640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb21009ea50 0x7fb210130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb2164f9640 10 monclient(hunting): get_auth_request con 0x7fb21009ea50 auth_method 0 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb2164f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb2164f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb2164f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 10 monclient(hunting): handle_auth_done global_id 174144 payload 824 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2101318f0 msgr2=0x7fb210099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2101318f0 0x7fb210099920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb21009ea50 msgr2=0x7fb210130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb21009ea50 0x7fb210130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 10 monclient: _finish_hunting 0 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 1 monclient: found mon.noname-a 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb216cfa640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204009390 con 0x7fb210136c20 2024-09-15T03:42:03.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.151+0000 7fb215cf8640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2101318f0 0x7fb210099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb216cfa640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb210136c20 msgr2=0x7fb210099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb216cfa640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb210136c20 0x7fb210099e60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb20000cb60 tx=0x7fb2000052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000a850 con 0x7fb210136c20 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: dump: 2024-09-15T03:42:03.171 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20000a9d0 con 0x7fb210136c20 2024-09-15T03:42:03.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 5 monclient: authenticate success, global_id 174144 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 10 monclient: _renew_subs 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21009bf10 con 0x7fb210136c20 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000ad90 con 0x7fb210136c20 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb20e7fc640 10 monclient: dump: 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.173 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb214cf6640 4 set_mon_vals no callback set 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 10 monclient: _renew_subs 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.153+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21009b3a0 con 0x7fb210136c20 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb200007300 con 0x7fb210136c20 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb21875d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb21875d640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 0x7fb2100298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb21875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb210065e00 con 0x7fb210130ec0 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb20e7fc640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1fc038740 0x7fb1fc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 0x7fb2100298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient: get_auth_request con 0x7fb210130ec0 auth_method 0 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb20004c680 con 0x7fb210136c20 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 10 monclient(hunting): handle_auth_done global_id 124658 payload 824 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb2164f9640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb21011cba0 con 0x7fb210130ec0 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.154+0000 7fb215cf8640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1fc038740 0x7fb1fc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.155+0000 7fb215cf8640 10 monclient: get_auth_request con 0x7fb1fc038740 auth_method 0 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb2164f9640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 msgr2=0x7fb2100298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb2164f9640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 0x7fb2100298c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e510 tx=0x7fb208009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208007450 con 0x7fb210130ec0 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 10 monclient: dump: 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.176 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb2080075b0 con 0x7fb210130ec0 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 msgr2=0x7fb2100298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb215cf8640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1fc038740 0x7fb1fc03abf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb204009130 tx=0x7fb204032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 0x7fb2100298c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e510 tx=0x7fb208009040 comp rx=0 tx=0).stop 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 msgr2=0x7fb2100298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.157+0000 7fb20e7fc640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210130ec0 0x7fb2100298c0 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e510 tx=0x7fb208009040 comp rx=0 tx=0).stop 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb21875d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:03.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb21875d640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 0x7fb21002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb21875d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb210131290 con 0x7fb210029e00 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 0x7fb21002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient: get_auth_request con 0x7fb210029e00 auth_method 0 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 10 monclient(hunting): handle_auth_done global_id 124661 payload 824 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.163+0000 7fb2164f9640 1 -- 172.21.15.153:0/4280933633 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb21004bf60 con 0x7fb210029e00 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.165+0000 7fb2164f9640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 msgr2=0x7fb21002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.165+0000 7fb2164f9640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 0x7fb21002a200 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e510 tx=0x7fb20801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.165+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208013c40 con 0x7fb210029e00 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.165+0000 7fb20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.165+0000 7fb20e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.165+0000 7fb20e7fc640 10 monclient: dump: 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.179 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fb208013da0 con 0x7fb210029e00 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 msgr2=0x7fb21002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 0x7fb21002a200 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e510 tx=0x7fb20801b000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 msgr2=0x7fb21002a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb20e7fc640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb210029e00 0x7fb21002a200 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e510 tx=0x7fb20801b000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:03.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 138, 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.181 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.182 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:03.183 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.184 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:03.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:03.186 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.186 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.186 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.187 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.189 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:03.190 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:03.190 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.190 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.192 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:03.194 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1fc038740 msgr2=0x7fb1fc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1fc038740 0x7fb1fc03abf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb204009130 tx=0x7fb204032000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 10 monclient: shutdown 2024-09-15T03:42:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb210136c20 msgr2=0x7fb210099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 --2- 172.21.15.153:0/4280933633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb210136c20 0x7fb210099e60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb20000cb60 tx=0x7fb2000052b0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb216cfa640 1 -- 172.21.15.153:0/4280933633 reap_dead start 2024-09-15T03:42:03.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb216cfa640 1 -- 172.21.15.153:0/4280933633 reap_dead start 2024-09-15T03:42:03.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 shutdown_connections 2024-09-15T03:42:03.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 >> 172.21.15.153:0/4280933633 conn(0x7fb210067110 msgr2=0x7fb2100a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:03.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 shutdown_connections 2024-09-15T03:42:03.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.166+0000 7fb21875d640 1 -- 172.21.15.153:0/4280933633 wait complete. 2024-09-15T03:42:03.197 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:42:03.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:42:03.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:42:03.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.290+0000 7fad8c0f8640 10 monclient: get_monmap_and_config 2024-09-15T03:42:03.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.290+0000 7fad8c0f8640 10 monclient: build_initial_monmap 2024-09-15T03:42:03.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.290+0000 7fad8c0f8640 10 monclient: monmap: 2024-09-15T03:42:03.291 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:03.291389+0000 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:03.291389+0000 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:03.292 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 1 Processor -- start 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 1 -- start start 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient: init 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad840974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad8409a8a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fad84134080 0x7fad84136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient(hunting): picked mon.noname-a con 0x7fad84134080 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8412b210 con 0x7fad840974f0 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad840978c0 con 0x7fad8409a8a0 2024-09-15T03:42:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8409ca20 con 0x7fad84134080 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.291+0000 7fad8c0f8640 10 monclient(hunting): authenticate will time out at 1520.930994s 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad89e94640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fad840974f0 0x7fad8412ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50944/0 (socket says 172.21.15.153:50944) 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad89e94640 1 -- 172.21.15.153:0/68190523 learned_addr learned my addr 172.21.15.153:0/68190523 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 1 --2- 172.21.15.153:0/68190523 >> v2:172.21.15.153:3300/0 conn(0x7fad84134080 0x7fad84136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): get_auth_request con 0x7fad84134080 auth_method 0 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient(hunting): handle_auth_done global_id 174147 payload 274 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 1 -- 172.21.15.153:0/68190523 >> v1:172.21.15.181:6789/0 conn(0x7fad8409a8a0 legacy=0x7fad8409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 1 -- 172.21.15.153:0/68190523 >> v1:172.21.15.153:6789/0 conn(0x7fad840974f0 legacy=0x7fad8412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient: _finish_hunting 0 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 1 monclient: found mon.noname-a 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.292+0000 7fad8a695640 1 -- 172.21.15.153:0/68190523 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84098ad0 con 0x7fad84134080 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8a695640 1 -- 172.21.15.153:0/68190523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad84134080 msgr2=0x7fad84136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8a695640 1 --2- 172.21.15.153:0/68190523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad84134080 0x7fad84136460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009e60 tx=0x7fad7c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688374f5fadb648f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 1 -- 172.21.15.153:0/68190523 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c007840 con 0x7fad84134080 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: dump: 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: _finish_auth 0 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 1 -- 172.21.15.153:0/68190523 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c02e050 con 0x7fad84134080 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 5 monclient: authenticate success, global_id 174147 2024-09-15T03:42:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 1 -- 172.21.15.153:0/68190523 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c007bd0 con 0x7fad84134080 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 4 set_mon_vals no callback set 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad88e92640 10 monclient: dump: 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 10 monclient: shutdown 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.295+0000 7fad8c0f8640 1 -- 172.21.15.153:0/68190523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad84134080 msgr2=0x7fad84136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 --2- 172.21.15.153:0/68190523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad84134080 0x7fad84136460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009e60 tx=0x7fad7c009780 comp rx=0 tx=0).stop 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 -- 172.21.15.153:0/68190523 shutdown_connections 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 --2- 172.21.15.153:0/68190523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad84134080 0x7fad84136460 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 -- 172.21.15.153:0/68190523 >> 172.21.15.153:0/68190523 conn(0x7fad840672b0 msgr2=0x7fad840676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 -- 172.21.15.153:0/68190523 shutdown_connections 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 -- 172.21.15.153:0/68190523 wait complete. 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 10 monclient: build_initial_monmap 2024-09-15T03:42:03.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 10 monclient: monmap: 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 Processor -- start 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 1 -- start start 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.296+0000 7fad8c0f8640 10 monclient: init 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:03.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 0x7fad841281b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): picked mon.noname-a con 0x7fad840974f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad8409a8a0 0x7fad8412fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): picked mon.noname-c con 0x7fad8409a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad84134080 0x7fad84130280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): picked mon.noname-b con 0x7fad84134080 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84131690 con 0x7fad840974f0 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84098300 con 0x7fad84134080 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8412a370 con 0x7fad8409a8a0 2024-09-15T03:42:03.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8c0f8640 10 monclient(hunting): authenticate will time out at 1520.936379s 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 0x7fad841281b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 0x7fad841281b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60042/0 (socket says 172.21.15.153:60042) 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 -- 172.21.15.153:0/3271880989 learned_addr learned my addr 172.21.15.153:0/3271880989 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad8409a8a0 0x7fad8412fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): get_auth_request con 0x7fad840974f0 auth_method 0 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): get_auth_request con 0x7fad8409a8a0 auth_method 0 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8a695640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad84134080 0x7fad84130280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient(hunting): handle_auth_done global_id 174150 payload 824 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad8409a8a0 msgr2=0x7fad8412fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad8409a8a0 0x7fad8412fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad84134080 msgr2=0x7fad84130280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad84134080 0x7fad84130280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient: _finish_hunting 0 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 monclient: found mon.noname-a 2024-09-15T03:42:03.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89e94640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c009a20 con 0x7fad840974f0 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8a695640 10 monclient: get_auth_request con 0x7fad84134080 auth_method 0 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad8a695640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad84134080 0x7fad84130280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.297+0000 7fad89693640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad8409a8a0 0x7fad8412fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad89e94640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 msgr2=0x7fad841281b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad89e94640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 0x7fad841281b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad8000c5e0 tx=0x7fad80004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80005cb0 con 0x7fad840974f0 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: dump: 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: _finish_auth 0 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad80005e30 con 0x7fad840974f0 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 5 monclient: authenticate success, global_id 174150 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80007340 con 0x7fad840974f0 2024-09-15T03:42:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 10 monclient: _renew_subs 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad84130ec0 con 0x7fad840974f0 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad7bfff640 4 set_mon_vals no callback set 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad79ffb640 10 monclient: dump: 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 10 monclient: _renew_subs 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.300+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8406f450 con 0x7fad840974f0 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad8001b400 con 0x7fad840974f0 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad8c0f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad8c0f8640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 0x7fad8409ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad8c0f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84065f20 con 0x7fad841307c0 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad79ffb640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad68038740 0x7fad6803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fad8004c0c0 con 0x7fad840974f0 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 0x7fad8409ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient: get_auth_request con 0x7fad841307c0 auth_method 0 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad8a695640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad68038740 0x7fad6803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad8a695640 10 monclient: get_auth_request con 0x7fad68038740 auth_method 0 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 10 monclient(hunting): handle_auth_done global_id 144370 payload 824 2024-09-15T03:42:03.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.301+0000 7fad89693640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad84130b90 con 0x7fad841307c0 2024-09-15T03:42:03.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad89693640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 msgr2=0x7fad8409ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad89693640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 0x7fad8409ea80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fad7400fa50 tx=0x7fad74007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad8a695640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad68038740 0x7fad6803abf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fad7c035000 tx=0x7fad7c038000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7400dda0 con 0x7fad841307c0 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 10 monclient: dump: 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.318 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fad74014470 con 0x7fad841307c0 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 msgr2=0x7fad8409ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 0x7fad8409ea80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fad7400fa50 tx=0x7fad74007e10 comp rx=0 tx=0).stop 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 msgr2=0x7fad8409ea80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.304+0000 7fad79ffb640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad841307c0 0x7fad8409ea80 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fad7400fa50 tx=0x7fad74007e10 comp rx=0 tx=0).stop 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad8c0f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad8c0f8640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 0x7fad8409f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad8c0f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:03.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8404bf60 con 0x7fad8409efc0 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 0x7fad8409f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 10 monclient: get_auth_request con 0x7fad8409efc0 auth_method 0 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.310+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.311+0000 7fad89693640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.311+0000 7fad89693640 10 monclient(hunting): handle_auth_done global_id 144373 payload 824 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.311+0000 7fad89693640 1 -- 172.21.15.153:0/3271880989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fad84132b20 con 0x7fad8409efc0 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.313+0000 7fad89693640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 msgr2=0x7fad8409f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.313+0000 7fad89693640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 0x7fad8409f3c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fad74019000 tx=0x7fad7401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.313+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7401ec50 con 0x7fad8409efc0 2024-09-15T03:42:03.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.313+0000 7fad79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.313+0000 7fad79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.313+0000 7fad79ffb640 10 monclient: dump: 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fad7401edb0 con 0x7fad8409efc0 2024-09-15T03:42:03.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 msgr2=0x7fad8409f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 0x7fad8409f3c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fad74019000 tx=0x7fad7401a000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 msgr2=0x7fad8409f3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad79ffb640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad8409efc0 0x7fad8409f3c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:42:03.322 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 138, 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.323 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.324 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:03.325 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:03.326 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.328 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.329 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:03.331 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:03.332 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.333 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:03.334 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:03.335 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad68038740 msgr2=0x7fad6803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad68038740 0x7fad6803abf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fad7c035000 tx=0x7fad7c038000 comp rx=0 tx=0).stop 2024-09-15T03:42:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 10 monclient: shutdown 2024-09-15T03:42:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 msgr2=0x7fad841281b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 --2- 172.21.15.153:0/3271880989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad840974f0 0x7fad841281b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad8000c5e0 tx=0x7fad80004960 comp rx=0 tx=0).stop 2024-09-15T03:42:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8a695640 1 -- 172.21.15.153:0/3271880989 reap_dead start 2024-09-15T03:42:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8a695640 1 -- 172.21.15.153:0/3271880989 reap_dead start 2024-09-15T03:42:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 shutdown_connections 2024-09-15T03:42:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 >> 172.21.15.153:0/3271880989 conn(0x7fad840672b0 msgr2=0x7fad84124750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 shutdown_connections 2024-09-15T03:42:03.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:03.314+0000 7fad8c0f8640 1 -- 172.21.15.153:0/3271880989 wait complete. 2024-09-15T03:42:03.337 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:42:03.337 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:42:03.338 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T03:42:03.338 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:42:03.338 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:42:03.338 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:42:03.338 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:42:03.353 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:42:03.353 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:42:03.353 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:42:03.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:42:18.354 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:42:18.355 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:42:18.355 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:42:18.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:42:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.458+0000 7f004e11d640 10 monclient: get_monmap_and_config 2024-09-15T03:42:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.458+0000 7f004e11d640 10 monclient: build_initial_monmap 2024-09-15T03:42:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.458+0000 7f004e11d640 10 monclient: monmap: 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:18.459305+0000 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:18.459305+0000 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:18.461 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 1 Processor -- start 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 1 -- start start 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient: init 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00480981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 0x7f0048136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient(hunting): picked mon.noname-b con 0x7f0048131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0048132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004809ca60 con 0x7f00480981c0 2024-09-15T03:42:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004809cca0 con 0x7f0048132170 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0048131bd0 con 0x7f0048131da0 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.459+0000 7f004e11d640 10 monclient(hunting): authenticate will time out at 1536.098988s 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.460+0000 7f0046ffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 msgr2=0x7f0048136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.460+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 0x7f0048136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.460+0000 7f00477fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f00480981c0 0x7f0048098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34664/0 (socket says 172.21.15.153:34664) 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.460+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 learned_addr learned my addr 172.21.15.153:0/3266487106 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.462+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v1:172.21.15.153:6789/0 conn(0x7f00480981c0 legacy=0x7f0048098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:18.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.462+0000 7f0047fff640 1 -- 172.21.15.153:0/3266487106 >> v1:172.21.15.181:6789/0 conn(0x7f0048132170 legacy=0x7f00481390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:18.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.661+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 msgr2=0x7f0048136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:18.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:18.661+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 0x7f0048136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:42:19.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:19.062+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 msgr2=0x7f0048136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:19.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:19.062+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 0x7f0048136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:42:19.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:19.863+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 msgr2=0x7f0048136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:19.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:19.863+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 0x7f0048136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): tick 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 monclient(hunting): continuing hunt 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 msgr2=0x7f0048136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0048131da0 0x7f0048136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 >> v1:172.21.15.181:6789/0 conn(0x7f0048132170 legacy=0x7f00481390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 >> v1:172.21.15.153:6789/0 conn(0x7f00480981c0 legacy=0x7f0048098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient: _add_conns ranks=[3,4] 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): picked mon.noname-a con 0x7f00340037d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:21.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0034006330 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0048131bd0 con 0x7f00340037d0 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0034008e80 con 0x7f0034006330 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00457fa640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f00340037d0 auth_method 0 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:21.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f0034006330 auth_method 0 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:21.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.460+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:21.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:21.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f00340037d0 auth_method 0 2024-09-15T03:42:21.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f0034006330 auth_method 0 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:42:21.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:21.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:21.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:21.661+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:42:22.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:22.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f00340037d0 auth_method 0 2024-09-15T03:42:22.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:22.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:22.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:22.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f0034006330 auth_method 0 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:42:22.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:22.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:42:22.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:22.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.062+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f00340037d0 auth_method 0 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:22.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f0034006330 auth_method 0 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:42:22.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:22.863+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): tick 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 monclient(hunting): continuing hunt 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 msgr2=0x7f0034005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0034006330 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0047fff640 1 -- 172.21.15.153:0/3266487106 reap_dead start 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): picked mon.noname-a con 0x7f00340037d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0034009290 addr v1:172.21.15.153:6789/0 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004809e010 con 0x7f0034009290 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0048131bd0 con 0x7f00340037d0 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 1 -- 172.21.15.153:0/3266487106 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0034008e80 con 0x7f0034006330 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00457fa640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.153:3300/0 conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f00340037d0 auth_method 0 2024-09-15T03:42:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00477fe640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f0046ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f0034006330 auth_method 0 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.460+0000 7f00477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 10 monclient(hunting): handle_auth_done global_id 174153 payload 274 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 msgr2=0x7f00340087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> v1:172.21.15.153:6789/0 conn(0x7f0034009290 legacy=0x7f003400b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 1 monclient: found mon.noname-a 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.461+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003400dfc0 con 0x7f00340037d0 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f0046ffd640 1 -- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00340037d0 msgr2=0x7f0034005c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f0046ffd640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00340037d0 0x7f0034005c40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f004806f850 tx=0x7f003c0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cb0f53938a83b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 1 -- 172.21.15.153:0/3266487106 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c02c950 con 0x7f00340037d0 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: dump: 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.473 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: _finish_auth 0 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 1 -- 172.21.15.153:0/3266487106 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c030070 con 0x7f00340037d0 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f004e11d640 5 monclient: authenticate success, global_id 174153 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f004e11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f004e11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 1 -- 172.21.15.153:0/3266487106 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c02cd20 con 0x7f00340037d0 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f004e11d640 4 set_mon_vals no callback set 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.464+0000 7f00467fc640 10 monclient: dump: 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.475 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 10 monclient: shutdown 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 -- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00340037d0 msgr2=0x7f0034005c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00340037d0 0x7f0034005c40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f004806f850 tx=0x7f003c0079d0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 -- 172.21.15.153:0/3266487106 shutdown_connections 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 --2- 172.21.15.153:0/3266487106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00340037d0 0x7f0034005c40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 --2- 172.21.15.153:0/3266487106 >> v2:172.21.15.181:3300/0 conn(0x7f0034006330 0x7f00340087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 -- 172.21.15.153:0/3266487106 >> 172.21.15.153:0/3266487106 conn(0x7f00480a1e30 msgr2=0x7f00480a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 -- 172.21.15.153:0/3266487106 shutdown_connections 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 1 -- 172.21.15.153:0/3266487106 wait complete. 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 10 monclient: build_initial_monmap 2024-09-15T03:42:24.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.465+0000 7f004e11d640 10 monclient: monmap: 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.466+0000 7f004e11d640 1 Processor -- start 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.466+0000 7f004e11d640 1 -- start start 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.466+0000 7f004e11d640 10 monclient: init 2024-09-15T03:42:24.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00340037d0 0x7f0048141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): picked mon.noname-b con 0x7f00340037d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 0x7f0048130920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): picked mon.noname-a con 0x7f0034006330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0034009290 0x7f004811a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): picked mon.noname-c con 0x7f0034009290 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00480a0560 con 0x7f0034006330 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004811bf80 con 0x7f00340037d0 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004811c490 con 0x7f0034009290 2024-09-15T03:42:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f004e11d640 10 monclient(hunting): authenticate will time out at 1542.106237s 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0047fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0034009290 msgr2=0x7f004811a3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0047fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0034009290 0x7f004811a3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 0x7f0048130920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 0x7f0048130920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33866/0 (socket says 172.21.15.153:33866) 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 -- 172.21.15.153:0/2326066237 learned_addr learned my addr 172.21.15.153:0/2326066237 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f0034006330 auth_method 0 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00340037d0 0x7f0048141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f00340037d0 auth_method 0 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient(hunting): handle_auth_done global_id 174156 payload 824 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 -- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0034009290 msgr2=0x7f004811a3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0034009290 0x7f004811a3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 -- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00340037d0 msgr2=0x7f0048141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00340037d0 0x7f0048141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 monclient: found mon.noname-a 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.467+0000 7f0046ffd640 1 -- 172.21.15.153:0/2326066237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c007cb0 con 0x7f0034006330 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f0046ffd640 1 -- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 msgr2=0x7f0048130920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f0046ffd640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 0x7f0048130920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f003c02c390 tx=0x7f003c038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c036a30 con 0x7f0034006330 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: dump: 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c036b90 con 0x7f0034006330 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c042470 con 0x7f0034006330 2024-09-15T03:42:24.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 5 monclient: authenticate success, global_id 174156 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f002f7fe640 10 monclient: dump: 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 10 monclient: _renew_subs 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0048098cb0 con 0x7f0034006330 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f0045ffb640 4 set_mon_vals no callback set 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 10 monclient: _renew_subs 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.470+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004811da40 con 0x7f0034006330 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f003c00d040 con 0x7f0034006330 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f004e11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f002f7fe640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0034039c70 0x7f003403c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f004e11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f004e11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004804bf60 con 0x7f0034006330 2024-09-15T03:42:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f003c06dbf0 con 0x7f0034006330 2024-09-15T03:42:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.471+0000 7f00477fe640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0034039c70 0x7f003403c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.472+0000 7f00477fe640 10 monclient: get_auth_request con 0x7f0034039c70 auth_method 0 2024-09-15T03:42:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.474+0000 7f00477fe640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0034039c70 0x7f003403c120 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0038000c00 tx=0x7f0038009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.476+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f003c008040 con 0x7f0034006330 2024-09-15T03:42:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.476+0000 7f002f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.476+0000 7f002f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:24.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.693+0000 7f004e11d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:42:24.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.693+0000 7f004e11d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:42:24.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.693+0000 7f004e11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.693+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f004811d6b0 con 0x7f0034006330 2024-09-15T03:42:24.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.694+0000 7f002f7fe640 1 -- 172.21.15.153:0/2326066237 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f003c03e320 con 0x7f0034006330 2024-09-15T03:42:24.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.694+0000 7f002f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:42:24.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.694+0000 7f002f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:24.695 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":140,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0034039c70 msgr2=0x7f003403c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0034039c70 0x7f003403c120 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0038000c00 tx=0x7f0038009040 comp rx=0 tx=0).stop 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 10 monclient: shutdown 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 msgr2=0x7f0048130920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 0x7f0048130920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f003c02c390 tx=0x7f003c038000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 shutdown_connections 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0034039c70 0x7f003403c120 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0034009290 0x7f004811a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0034006330 0x7f0048130920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 --2- 172.21.15.153:0/2326066237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00340037d0 0x7f0048141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 >> 172.21.15.153:0/2326066237 conn(0x7f00480a1e30 msgr2=0x7f0048096e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:24.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 shutdown_connections 2024-09-15T03:42:24.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.696+0000 7f004e11d640 1 -- 172.21.15.153:0/2326066237 wait complete. 2024-09-15T03:42:24.708 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:42:24.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:42:24.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.802+0000 7f901b99d640 10 monclient: get_monmap_and_config 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.802+0000 7f901b99d640 10 monclient: build_initial_monmap 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.802+0000 7f901b99d640 10 monclient: monmap: 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:24.802897+0000 2024-09-15T03:42:24.803 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:24.802897+0000 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:24.804 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.802+0000 7f901b99d640 1 Processor -- start 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.802+0000 7f901b99d640 1 -- start start 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.802+0000 7f901b99d640 10 monclient: init 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9014131ab0 0x7f9014131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f9014131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f901409e910 0x7f901409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f901409e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:24.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409f250 0x7f9014068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): picked mon.noname-b con 0x7f901409f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901409faf0 con 0x7f901409e910 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901409fd30 con 0x7f9014131ab0 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140a1110 con 0x7f901409f250 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f901b99d640 10 monclient(hunting): authenticate will time out at 1542.442478s 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019f3a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409f250 msgr2=0x7f9014068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019f3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409f250 0x7f9014068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f901409e910 0x7f901409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f901409e910 0x7f901409ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33880/0 (socket says 172.21.15.153:33880) 2024-09-15T03:42:24.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 -- 172.21.15.153:0/592318143 learned_addr learned my addr 172.21.15.153:0/592318143 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): get_auth_request con 0x7f901409e910 auth_method 0 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019739640 1 --2- 172.21.15.153:0/592318143 >> v2:172.21.15.181:3300/0 conn(0x7f9014131ab0 0x7f9014131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019739640 10 monclient(hunting): get_auth_request con 0x7f9014131ab0 auth_method 0 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9019739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient(hunting): handle_auth_done global_id 174159 payload 274 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 -- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409f250 msgr2=0x7f9014068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 --2- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409f250 0x7f9014068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 -- 172.21.15.153:0/592318143 >> v2:172.21.15.181:3300/0 conn(0x7f9014131ab0 msgr2=0x7f9014131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 --2- 172.21.15.153:0/592318143 >> v2:172.21.15.181:3300/0 conn(0x7f9014131ab0 0x7f9014131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient: _finish_hunting 0 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 monclient: found mon.noname-a 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.803+0000 7f9018f38640 1 -- 172.21.15.153:0/592318143 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140a13f0 con 0x7f901409e910 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.806+0000 7f9018f38640 1 -- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901409e910 msgr2=0x7f901409ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.806+0000 7f9018f38640 1 --2- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901409e910 0x7f901409ed10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9014076760 tx=0x7f90040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4c142d9c54d1c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 1 -- 172.21.15.153:0/592318143 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004032440 con 0x7f901409e910 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: dump: 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.813 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: _finish_auth 0 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 5 monclient: authenticate success, global_id 174159 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 1 -- 172.21.15.153:0/592318143 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90040325a0 con 0x7f901409e910 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 1 -- 172.21.15.153:0/592318143 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004032940 con 0x7f901409e910 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 4 set_mon_vals no callback set 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f900bfff640 10 monclient: dump: 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 10 monclient: shutdown 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:24.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 -- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901409e910 msgr2=0x7f901409ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 --2- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901409e910 0x7f901409ed10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9014076760 tx=0x7f90040091d0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 -- 172.21.15.153:0/592318143 shutdown_connections 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 --2- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409f250 0x7f9014068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 --2- 172.21.15.153:0/592318143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901409e910 0x7f901409ed10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 --2- 172.21.15.153:0/592318143 >> v2:172.21.15.181:3300/0 conn(0x7f9014131ab0 0x7f9014131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 -- 172.21.15.153:0/592318143 >> 172.21.15.153:0/592318143 conn(0x7f9014067d30 msgr2=0x7f90140a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 -- 172.21.15.153:0/592318143 shutdown_connections 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.807+0000 7f901b99d640 1 -- 172.21.15.153:0/592318143 wait complete. 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.808+0000 7f901b99d640 10 monclient: build_initial_monmap 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.808+0000 7f901b99d640 10 monclient: monmap: 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:24.816 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.808+0000 7f901b99d640 1 Processor -- start 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.808+0000 7f901b99d640 1 -- start start 2024-09-15T03:42:24.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.808+0000 7f901b99d640 10 monclient: init 2024-09-15T03:42:24.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:24.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:42:24.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409e910 0x7f901411dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f901409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901409f250 0x7f901411e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): picked mon.noname-b con 0x7f901409f250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 0x7f901411e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019739640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409e910 msgr2=0x7f901411dea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f9014131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901411f990 con 0x7f9014131ab0 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901409bde0 con 0x7f901409f250 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140a43f0 con 0x7f901409e910 2024-09-15T03:42:24.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019739640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409e910 0x7f901411dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f901b99d640 10 monclient(hunting): authenticate will time out at 1542.448262s 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 0x7f901411e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 0x7f901411e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33886/0 (socket says 172.21.15.153:33886) 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 -- 172.21.15.153:0/1085450241 learned_addr learned my addr 172.21.15.153:0/1085450241 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): get_auth_request con 0x7f9014131ab0 auth_method 0 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient(hunting): handle_auth_done global_id 174162 payload 824 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9018f38640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901409f250 0x7f901411e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409e910 msgr2=0x7f901411dea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f901409e910 0x7f901411dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901409f250 msgr2=0x7f901411e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901409f250 0x7f901411e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient: _finish_hunting 0 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 monclient: found mon.noname-a 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.809+0000 7f9019f3a640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004009450 con 0x7f9014131ab0 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f9019f3a640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 msgr2=0x7f901411e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f9019f3a640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 0x7f901411e920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f900c00bf80 tx=0x7f900c009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c016430 con 0x7f9014131ab0 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: dump: 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: _finish_auth 0 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c0165b0 con 0x7f9014131ab0 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c016760 con 0x7f9014131ab0 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f90097fa640 10 monclient: dump: 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.828 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f901b99d640 5 monclient: authenticate success, global_id 174162 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f901b99d640 10 monclient: _renew_subs 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f900b7fe640 4 set_mon_vals no callback set 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f901b99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.811+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90141205e0 con 0x7f9014131ab0 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 10 monclient: _renew_subs 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9014131450 con 0x7f9014131ab0 2024-09-15T03:42:24.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 0x7f901411ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141316d0 con 0x7f901402aa40 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 0x7f901411ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f900c012020 con 0x7f9014131ab0 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient: get_auth_request con 0x7f901402aa40 auth_method 0 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.812+0000 7f9019739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.813+0000 7f90097fa640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ff80386f0 0x7f8ff803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.813+0000 7f9019739640 10 monclient(hunting): handle_auth_done global_id 174165 payload 824 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.813+0000 7f9019739640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f901402ae10 con 0x7f901402aa40 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.813+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f900c04c830 con 0x7f9014131ab0 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.813+0000 7f9018f38640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ff80386f0 0x7f8ff803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.813+0000 7f9018f38640 10 monclient: get_auth_request con 0x7f8ff80386f0 auth_method 0 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f9019739640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 msgr2=0x7f901411ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f9019739640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 0x7f901411ee60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9010006f90 tx=0x7f9010008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010012800 con 0x7f901402aa40 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 10 monclient: dump: 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9010012960 con 0x7f901402aa40 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 msgr2=0x7f901411ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 0x7f901411ee60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9010006f90 tx=0x7f9010008040 comp rx=0 tx=0).stop 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 msgr2=0x7f901411ee60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f90097fa640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901402aa40 0x7f901411ee60 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9010006f90 tx=0x7f9010008040 comp rx=0 tx=0).stop 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.815+0000 7f9018f38640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ff80386f0 0x7f8ff803aba0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f901409bdb0 tx=0x7f9004032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f901b99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f901b99d640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 0x7f901405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f901b99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141215d0 con 0x7f901405e310 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 0x7f901405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient: get_auth_request con 0x7f901405e310 auth_method 0 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 10 monclient(hunting): handle_auth_done global_id 174168 payload 824 2024-09-15T03:42:24.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.821+0000 7f9019739640 1 -- 172.21.15.153:0/1085450241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f901411f3a0 con 0x7f901405e310 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f9019739640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 msgr2=0x7f901405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f9019739640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 0x7f901405e710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9010018000 tx=0x7f901001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010014640 con 0x7f901405e310 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 10 monclient: dump: 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.835 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f90100147a0 con 0x7f901405e310 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 msgr2=0x7f901405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 0x7f901405e710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9010018000 tx=0x7f901001b000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 msgr2=0x7f901405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:24.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f90097fa640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f901405e310 0x7f901405e710 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9010018000 tx=0x7f901001b000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 140, 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.837 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:24.838 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:24.840 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:24.841 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:24.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.844 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.845 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.846 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:24.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.848 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.848 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.848 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:24.848 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.848 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:42:24.849 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.850 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.850 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:24.850 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ff80386f0 msgr2=0x7f8ff803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ff80386f0 0x7f8ff803aba0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f901409bdb0 tx=0x7f9004032000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 10 monclient: shutdown 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 msgr2=0x7f901411e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 0x7f901411e920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f900c00bf80 tx=0x7f900c009730 comp rx=0 tx=0).stop 2024-09-15T03:42:24.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f9019f3a640 1 -- 172.21.15.153:0/1085450241 reap_dead start 2024-09-15T03:42:24.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 shutdown_connections 2024-09-15T03:42:24.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 --2- 172.21.15.153:0/1085450241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9014131ab0 0x7f901411e920 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 >> 172.21.15.153:0/1085450241 conn(0x7f9014067d30 msgr2=0x7f901409b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:24.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 shutdown_connections 2024-09-15T03:42:24.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.824+0000 7f901b99d640 1 -- 172.21.15.153:0/1085450241 wait complete. 2024-09-15T03:42:24.852 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:42:24.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:42:24.852 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.945+0000 7f1227f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.945+0000 7f1227f5d640 10 monclient: build_initial_monmap 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.945+0000 7f1227f5d640 10 monclient: monmap: 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:42:24.945926+0000 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:42:24.945926+0000 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:42:24.946 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:42:24.947 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:42:24.947 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:42:24.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.945+0000 7f1227f5d640 1 Processor -- start 2024-09-15T03:42:24.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.945+0000 7f1227f5d640 1 -- start start 2024-09-15T03:42:24.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.945+0000 7f1227f5d640 10 monclient: init 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12200981c0 0x7f1220098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f12200981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1220131da0 0x7f1220067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1220131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1220132170 0x7f1220067650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1220132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1220131bd0 con 0x7f1220132170 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1220098ad0 con 0x7f1220131da0 2024-09-15T03:42:24.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122009ca60 con 0x7f12200981c0 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1227f5d640 10 monclient(hunting): authenticate will time out at 1542.585501s 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1225cf9640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12200981c0 msgr2=0x7f1220098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f1225cf9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12200981c0 0x7f1220098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1220132170 0x7f1220067650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1220132170 0x7f1220067650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33920/0 (socket says 172.21.15.153:33920) 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 -- 172.21.15.153:0/4217523707 learned_addr learned my addr 172.21.15.153:0/4217523707 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): get_auth_request con 0x7f1220132170 auth_method 0 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12254f8640 1 --2- 172.21.15.153:0/4217523707 >> v2:172.21.15.181:3300/0 conn(0x7f1220131da0 0x7f1220067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient(hunting): handle_auth_done global_id 174171 payload 274 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 -- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12200981c0 msgr2=0x7f1220098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 --2- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12200981c0 0x7f1220098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 -- 172.21.15.153:0/4217523707 >> v2:172.21.15.181:3300/0 conn(0x7f1220131da0 msgr2=0x7f1220067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 --2- 172.21.15.153:0/4217523707 >> v2:172.21.15.181:3300/0 conn(0x7f1220131da0 0x7f1220067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient: _finish_hunting 0 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 monclient: found mon.noname-a 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12264fa640 1 -- 172.21.15.153:0/4217523707 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122009cca0 con 0x7f1220132170 2024-09-15T03:42:24.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12254f8640 10 monclient: get_auth_request con 0x7f1220131da0 auth_method 0 2024-09-15T03:42:24.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.946+0000 7f12254f8640 1 --2- 172.21.15.153:0/4217523707 >> v2:172.21.15.181:3300/0 conn(0x7f1220131da0 0x7f1220067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f12264fa640 1 -- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 msgr2=0x7f1220067650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f12264fa640 1 --2- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220067650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f122006f880 tx=0x7f1210009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b3ddcdc87eebc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 1 -- 172.21.15.153:0/4217523707 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210007610 con 0x7f1220132170 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: dump: 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: _finish_auth 0 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 1 -- 172.21.15.153:0/4217523707 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121002d050 con 0x7f1220132170 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1227f5d640 5 monclient: authenticate success, global_id 174171 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1227f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:24.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1227f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 1 -- 172.21.15.153:0/4217523707 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12100079a0 con 0x7f1220132170 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1227f5d640 4 set_mon_vals no callback set 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.949+0000 7f1224cf7640 10 monclient: dump: 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.962 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 10 monclient: shutdown 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 -- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 msgr2=0x7f1220067650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 --2- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220067650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f122006f880 tx=0x7f1210009840 comp rx=0 tx=0).stop 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 -- 172.21.15.153:0/4217523707 shutdown_connections 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 --2- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220067650 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 --2- 172.21.15.153:0/4217523707 >> v2:172.21.15.181:3300/0 conn(0x7f1220131da0 0x7f1220067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 --2- 172.21.15.153:0/4217523707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12200981c0 0x7f1220098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 -- 172.21.15.153:0/4217523707 >> 172.21.15.153:0/4217523707 conn(0x7f12200a1e30 msgr2=0x7f12200a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 -- 172.21.15.153:0/4217523707 shutdown_connections 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 1 -- 172.21.15.153:0/4217523707 wait complete. 2024-09-15T03:42:24.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 10 monclient: build_initial_monmap 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.950+0000 7f1227f5d640 10 monclient: monmap: 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 Processor -- start 2024-09-15T03:42:24.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 -- start start 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient: init 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200981c0 0x7f1220140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f12200981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1220131da0 0x7f1220130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1220131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1220132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1220141e50 con 0x7f1220132170 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1220098cb0 con 0x7f12200981c0 2024-09-15T03:42:24.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12200a0cf0 con 0x7f1220131da0 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f1227f5d640 10 monclient(hunting): authenticate will time out at 1542.590807s 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12254f8640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1220131da0 msgr2=0x7f1220130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12254f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1220131da0 0x7f1220130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220131090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33922/0 (socket says 172.21.15.153:33922) 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 learned_addr learned my addr 172.21.15.153:0/2488603009 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 10 monclient(hunting): get_auth_request con 0x7f1220132170 auth_method 0 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.951+0000 7f12264fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f1225cf9640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200981c0 0x7f1220140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f1225cf9640 10 monclient(hunting): get_auth_request con 0x7f12200981c0 auth_method 0 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f1225cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f1225cf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f1225cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 10 monclient(hunting): handle_auth_done global_id 174174 payload 824 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1220131da0 msgr2=0x7f1220130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1220131da0 0x7f1220130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200981c0 msgr2=0x7f1220140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200981c0 0x7f1220140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 10 monclient: _finish_hunting 0 2024-09-15T03:42:24.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 1 monclient: found mon.noname-a 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.952+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210009ae0 con 0x7f1220132170 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 msgr2=0x7f1220131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f12264fa640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220131090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1210009ab0 tx=0x7f1210036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210032ce0 con 0x7f1220132170 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: dump: 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.968 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: _finish_auth 0 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 5 monclient: authenticate success, global_id 174174 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 10 monclient: _renew_subs 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122011df50 con 0x7f1220132170 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1210032e40 con 0x7f1220132170 2024-09-15T03:42:24.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121003b620 con 0x7f1220132170 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121dffb640 10 monclient: dump: 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.970 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f121ffff640 4 set_mon_vals no callback set 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 10 monclient: _renew_subs 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.954+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122011cd80 con 0x7f1220132170 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1210003020 con 0x7f1220132170 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f121dffb640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f120c0386f0 0x7f120c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f1227f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f1227f5d640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 0x7f122011ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1220065f90 con 0x7f1220141460 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.955+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f121006d280 con 0x7f1220132170 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f1225cf9640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f120c0386f0 0x7f120c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 0x7f122011ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f1225cf9640 10 monclient: get_auth_request con 0x7f120c0386f0 auth_method 0 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient: get_auth_request con 0x7f1220141460 auth_method 0 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 10 monclient(hunting): handle_auth_done global_id 144388 payload 824 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.956+0000 7f12254f8640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f122011e0e0 con 0x7f1220141460 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f1225cf9640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f120c0386f0 0x7f120c03aba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1218003940 tx=0x7f1218006ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f12254f8640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 msgr2=0x7f122011ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f12254f8640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 0x7f122011ad90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f121400c650 tx=0x7f1214008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214011810 con 0x7f1220141460 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f121dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f121dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.958+0000 7f121dffb640 10 monclient: dump: 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.973 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1214011970 con 0x7f1220141460 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 msgr2=0x7f122011ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 0x7f122011ad90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f121400c650 tx=0x7f1214008040 comp rx=0 tx=0).stop 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 msgr2=0x7f122011ad90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:24.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.959+0000 7f121dffb640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1220141460 0x7f122011ad90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.964+0000 7f1227f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.964+0000 7f1227f5d640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 0x7f1220029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.964+0000 7f1227f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.964+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122004bf60 con 0x7f12200298c0 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.964+0000 7f12254f8640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 0x7f1220029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient: get_auth_request con 0x7f12200298c0 auth_method 0 2024-09-15T03:42:24.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 10 monclient(hunting): handle_auth_done global_id 144391 payload 824 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.965+0000 7f12254f8640 1 -- 172.21.15.153:0/2488603009 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1220098ee0 con 0x7f12200298c0 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.967+0000 7f12254f8640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 msgr2=0x7f1220029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.967+0000 7f12254f8640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 0x7f1220029cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1214017000 tx=0x7f121401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.967+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121401f070 con 0x7f12200298c0 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.967+0000 7f121dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.967+0000 7f121dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:42:24.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.967+0000 7f121dffb640 10 monclient: dump: 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f1214011600 con 0x7f12200298c0 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:42:24.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 msgr2=0x7f1220029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 0x7f1220029cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1214017000 tx=0x7f121401a000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 msgr2=0x7f1220029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f121dffb640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12200298c0 0x7f1220029cc0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1214017000 tx=0x7f121401a000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 140, 2024-09-15T03:42:24.979 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:24.980 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:42:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:42:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.984 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:24.985 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:24.986 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:24.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.986 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.986 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:42:24.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.988 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:42:24.989 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:42:24.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:42:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:42:24.992 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f120c0386f0 msgr2=0x7f120c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f120c0386f0 0x7f120c03aba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1218003940 tx=0x7f1218006ef0 comp rx=0 tx=0).stop 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 10 monclient: shutdown 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 msgr2=0x7f1220131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 --2- 172.21.15.153:0/2488603009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1220132170 0x7f1220131090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1210009ab0 tx=0x7f1210036000 comp rx=0 tx=0).stop 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 reap_dead start 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f12264fa640 1 -- 172.21.15.153:0/2488603009 reap_dead start 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 shutdown_connections 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 >> 172.21.15.153:0/2488603009 conn(0x7f12200a1e30 msgr2=0x7f12200977a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 shutdown_connections 2024-09-15T03:42:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:42:24.968+0000 7f1227f5d640 1 -- 172.21.15.153:0/2488603009 wait complete. 2024-09-15T03:42:24.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:42:24.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:42:44.996 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:42:44.996 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:42:44.996 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:42:44.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:42:45.000 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:42:45.000 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:42:45.000 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:42:45.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:43:00.001 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:43:00.002 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:43:00.002 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:43:00.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.105+0000 7f19f049f640 10 monclient: get_monmap_and_config 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.105+0000 7f19f049f640 10 monclient: build_initial_monmap 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.105+0000 7f19f049f640 10 monclient: monmap: 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:00.106422+0000 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:00.106422+0000 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:00.107 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 1 Processor -- start 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 1 -- start start 2024-09-15T03:43:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient: init 2024-09-15T03:43:00.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:00.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:43:00.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19e809a510 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:00.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:00.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient(hunting): picked mon.noname-b con 0x7f19e80973b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19e8097780 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e806ee10 con 0x7f19e809a510 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e809cb00 con 0x7f19e8097780 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.106+0000 7f19f049f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19f049f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e809a8e0 con 0x7f19e80973b0 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19f049f640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19f049f640 10 monclient(hunting): authenticate will time out at 1577.746013s 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19ee23b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f19e809a510 0x7f19e8096ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:57392/0 (socket says 172.21.15.153:57392) 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 learned_addr learned my addr 172.21.15.153:0/1376502310 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19e80973b0 auth_method 0 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.107+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.109+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v1:172.21.15.153:6789/0 conn(0x7f19e809a510 legacy=0x7f19e8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.109+0000 7f19eea3c640 1 -- 172.21.15.153:0/1376502310 >> v1:172.21.15.181:6789/0 conn(0x7f19e8097780 legacy=0x7f19e8136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19e80973b0 auth_method 0 2024-09-15T03:43:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:00.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.308+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19e80973b0 auth_method 0 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:00.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:00.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:00.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:00.709+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:43:01.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19e80973b0 auth_method 0 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:01.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:01.510+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:43:03.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): tick 2024-09-15T03:43:03.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 monclient(hunting): continuing hunt 2024-09-15T03:43:03.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:43:03.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 msgr2=0x7f19e81349a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:03.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19e80973b0 0x7f19e81349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 -- 172.21.15.153:0/1376502310 >> v1:172.21.15.181:6789/0 conn(0x7f19e8097780 legacy=0x7f19e8136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 -- 172.21.15.153:0/1376502310 >> v1:172.21.15.153:6789/0 conn(0x7f19e809a510 legacy=0x7f19e8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient: _add_conns ranks=[4,3] 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f19cc0037d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f19cc006270 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 -- 172.21.15.153:0/1376502310 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e809a8e0 con 0x7f19cc006270 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 1 -- 172.21.15.153:0/1376502310 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc008de0 con 0x7f19cc0037d0 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19dffff640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:03.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19cc006270 auth_method 0 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 msgr2=0x7f19cc008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 msgr2=0x7f19cc008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 10 monclient(hunting): get_auth_request con 0x7f19cc0037d0 auth_method 0 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:03.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:03.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:03.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:03.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:03.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.107+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19cc006270 auth_method 0 2024-09-15T03:43:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 msgr2=0x7f19cc008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 msgr2=0x7f19cc008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 10 monclient(hunting): get_auth_request con 0x7f19cc0037d0 auth_method 0 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:03.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.308+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:43:03.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19cc006270 auth_method 0 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 msgr2=0x7f19cc008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 msgr2=0x7f19cc008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 10 monclient(hunting): get_auth_request con 0x7f19cc0037d0 auth_method 0 2024-09-15T03:43:03.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:03.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:03.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:03.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:03.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:03.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:03.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:03.709+0000 7f19ee23b640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:43:04.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.153:3300/0 conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:04.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19cc006270 auth_method 0 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient(hunting): handle_auth_done global_id 174177 payload 274 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 msgr2=0x7f19cc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> v2:172.21.15.181:3300/0 conn(0x7f19cc0037d0 0x7f19cc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 1 monclient: found mon.noname-a 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:04.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:04.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.510+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e809d590 con 0x7f19cc006270 2024-09-15T03:43:04.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.513+0000 7f19eda3a640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19cc006270 msgr2=0x7f19cc008720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:04.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.513+0000 7f19eda3a640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19cc006270 0x7f19cc008720 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f19cc003740 tx=0x7f19d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563789bed78bed3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 1 -- 172.21.15.153:0/1376502310 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d802ca90 con 0x7f19cc006270 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: dump: 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.515 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: _finish_auth 0 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 1 -- 172.21.15.153:0/1376502310 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8030070 con 0x7f19cc006270 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19f049f640 5 monclient: authenticate success, global_id 174177 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19f049f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19f049f640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 1 -- 172.21.15.153:0/1376502310 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d802ce60 con 0x7f19cc006270 2024-09-15T03:43:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19f049f640 4 set_mon_vals no callback set 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.514+0000 7f19ed239640 10 monclient: dump: 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.517 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.518 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:04.518 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:04.518 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:04.518 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:04.518 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:04.518 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 10 monclient: shutdown 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 -- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19cc006270 msgr2=0x7f19cc008720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19cc006270 0x7f19cc008720 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f19cc003740 tx=0x7f19d8008040 comp rx=0 tx=0).stop 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19eea3c640 1 -- 172.21.15.153:0/1376502310 reap_dead start 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 -- 172.21.15.153:0/1376502310 shutdown_connections 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 --2- 172.21.15.153:0/1376502310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19cc006270 0x7f19cc008720 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 -- 172.21.15.153:0/1376502310 >> 172.21.15.153:0/1376502310 conn(0x7f19e80672b0 msgr2=0x7f19e80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:04.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 -- 172.21.15.153:0/1376502310 shutdown_connections 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 1 -- 172.21.15.153:0/1376502310 wait complete. 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 10 monclient: build_initial_monmap 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.515+0000 7f19f049f640 10 monclient: monmap: 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:04.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 Processor -- start 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 -- start start 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient: init 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): picked mon.noname-c con 0x7f19cc006270 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 0x7f19e812f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): picked mon.noname-a con 0x7f19e811ba00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 0x7f19e8119330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): picked mon.noname-b con 0x7f19e812fd10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:04.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e809c370 con 0x7f19e811ba00 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e806d9f0 con 0x7f19e812fd10 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e811cbd0 con 0x7f19cc006270 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19f049f640 10 monclient(hunting): authenticate will time out at 1582.155862s 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19ee23b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.516+0000 7f19eda3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 0x7f19e812f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:53266/0 (socket says 172.21.15.153:53266) 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 1 -- 172.21.15.153:0/2566810363 learned_addr learned my addr 172.21.15.153:0/2566810363 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): get_auth_request con 0x7f19e811ba00 auth_method 0 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:04.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 10 monclient(hunting): get_auth_request con 0x7f19cc006270 auth_method 0 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 msgr2=0x7f19e813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 msgr2=0x7f19e813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:43:04.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19ee23b640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 0x7f19e8119330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 10 monclient(hunting): get_auth_request con 0x7f19e812fd10 auth_method 0 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient(hunting): handle_auth_done global_id 174180 payload 824 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 msgr2=0x7f19e8119330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 msgr2=0x7f19e8119330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 msgr2=0x7f19e813ee50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:04.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 0x7f19e8119330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 msgr2=0x7f19e8119330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eea3c640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 0x7f19e8119330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 0x7f19e8119330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 1 monclient: found mon.noname-a 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.517+0000 7f19eda3a640 1 -- 172.21.15.153:0/2566810363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d800e040 con 0x7f19e811ba00 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19eda3a640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 msgr2=0x7f19e812f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19eda3a640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 0x7f19e812f7d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19d8034000 tx=0x7f19d8038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8041400 con 0x7f19e811ba00 2024-09-15T03:43:04.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: dump: 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:04.536 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8041560 con 0x7f19e811ba00 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8041920 con 0x7f19e811ba00 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19f049f640 5 monclient: authenticate success, global_id 174180 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19f049f640 10 monclient: _renew_subs 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19f049f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e811c8f0 con 0x7f19e811ba00 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:04.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19eca38640 4 set_mon_vals no callback set 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.519+0000 7f19de7fc640 10 monclient: dump: 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.520+0000 7f19f049f640 10 monclient: _renew_subs 2024-09-15T03:43:04.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.520+0000 7f19f049f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.520+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e813f390 con 0x7f19e811ba00 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f19d8041d10 con 0x7f19e811ba00 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19f049f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19f049f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19f049f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e811cd90 con 0x7f19e811ba00 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19de7fc640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19cc039490 0x7f19cc03b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f19d806ede0 con 0x7f19e811ba00 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.521+0000 7f19ee23b640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19cc039490 0x7f19cc03b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.522+0000 7f19ee23b640 10 monclient: get_auth_request con 0x7f19cc039490 auth_method 0 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.524+0000 7f19ee23b640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19cc039490 0x7f19cc03b940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f19e4007fd0 tx=0x7f19e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.526+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f19d803f9b0 con 0x7f19e811ba00 2024-09-15T03:43:04.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.526+0000 7f19de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:04.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.526+0000 7f19de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19f049f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19f049f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19f049f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f19e811ae90 con 0x7f19e811ba00 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19de7fc640 1 -- 172.21.15.153:0/2566810363 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f19d804a3c0 con 0x7f19e811ba00 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:04.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.748+0000 7f19de7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:04.751 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:04.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19cc039490 msgr2=0x7f19cc03b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19cc039490 0x7f19cc03b940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f19e4007fd0 tx=0x7f19e4009040 comp rx=0 tx=0).stop 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 10 monclient: shutdown 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 msgr2=0x7f19e812f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 0x7f19e812f7d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19d8034000 tx=0x7f19d8038000 comp rx=0 tx=0).stop 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 shutdown_connections 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19cc039490 0x7f19cc03b940 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19e812fd10 0x7f19e8119330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e811ba00 0x7f19e812f7d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 --2- 172.21.15.153:0/2566810363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f19cc006270 0x7f19e813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 >> 172.21.15.153:0/2566810363 conn(0x7f19e80672b0 msgr2=0x7f19e806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 shutdown_connections 2024-09-15T03:43:04.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:04.750+0000 7f19f049f640 1 -- 172.21.15.153:0/2566810363 wait complete. 2024-09-15T03:43:07.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.866+0000 7f54d919d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.866+0000 7f54d919d640 10 monclient: build_initial_monmap 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.866+0000 7f54d919d640 10 monclient: monmap: 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:07.866886+0000 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:07.866886+0000 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:07.867 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:07.868 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:07.868 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:07.868 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.866+0000 7f54d919d640 1 Processor -- start 2024-09-15T03:43:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.866+0000 7f54d919d640 1 -- start start 2024-09-15T03:43:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.866+0000 7f54d919d640 10 monclient: init 2024-09-15T03:43:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:43:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54d41318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54d409ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f54d4137c30 0x7f54d413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient(hunting): picked mon.noname-c con 0x7f54d4137c30 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4131cc0 con 0x7f54d41318f0 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d409f530 con 0x7f54d409ea50 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d409fbf0 con 0x7f54d4137c30 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d919d640 10 monclient(hunting): authenticate will time out at 1585.506367s 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d2d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f54d41318f0 0x7f54d4131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44762/0 (socket says 172.21.15.153:44762) 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d2d9d640 1 -- 172.21.15.153:0/94434788 learned_addr learned my addr 172.21.15.153:0/94434788 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 1 --2- 172.21.15.153:0/94434788 >> v2:172.21.15.181:3300/0 conn(0x7f54d4137c30 0x7f54d413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): get_auth_request con 0x7f54d4137c30 auth_method 0 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 10 monclient(hunting): handle_auth_done global_id 144394 payload 274 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 1 -- 172.21.15.153:0/94434788 >> v1:172.21.15.181:6789/0 conn(0x7f54d409ea50 legacy=0x7f54d409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.867+0000 7f54d359e640 1 -- 172.21.15.153:0/94434788 >> v1:172.21.15.153:6789/0 conn(0x7f54d41318f0 legacy=0x7f54d4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.868+0000 7f54d359e640 10 monclient: _finish_hunting 0 2024-09-15T03:43:07.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.868+0000 7f54d359e640 1 monclient: found mon.noname-c 2024-09-15T03:43:07.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.868+0000 7f54d359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:07.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.868+0000 7f54d359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:43:07.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.868+0000 7f54d359e640 1 -- 172.21.15.153:0/94434788 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d409fe30 con 0x7f54d4137c30 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d359e640 1 -- 172.21.15.153:0/94434788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 msgr2=0x7f54d413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d359e640 1 --2- 172.21.15.153:0/94434788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d413a010 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009850 tx=0x7f54c40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=521d88c0d985a59e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 1 -- 172.21.15.153:0/94434788 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4032440 con 0x7f54d4137c30 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: dump: 2024-09-15T03:43:07.878 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 5 monclient: authenticate success, global_id 144394 2024-09-15T03:43:07.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 1 -- 172.21.15.153:0/94434788 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c40325a0 con 0x7f54d4137c30 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 1 -- 172.21.15.153:0/94434788 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4032940 con 0x7f54d4137c30 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 4 set_mon_vals no callback set 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d1d9b640 10 monclient: dump: 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.880 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 10 monclient: shutdown 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 -- 172.21.15.153:0/94434788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 msgr2=0x7f54d413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 --2- 172.21.15.153:0/94434788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d413a010 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009850 tx=0x7f54c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 -- 172.21.15.153:0/94434788 shutdown_connections 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 --2- 172.21.15.153:0/94434788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d413a010 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:07.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 -- 172.21.15.153:0/94434788 >> 172.21.15.153:0/94434788 conn(0x7f54d4067110 msgr2=0x7f54d4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 -- 172.21.15.153:0/94434788 shutdown_connections 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.871+0000 7f54d919d640 1 -- 172.21.15.153:0/94434788 wait complete. 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.872+0000 7f54d919d640 10 monclient: build_initial_monmap 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.872+0000 7f54d919d640 10 monclient: monmap: 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:07.882 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.872+0000 7f54d919d640 1 Processor -- start 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.872+0000 7f54d919d640 1 -- start start 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.872+0000 7f54d919d640 10 monclient: init 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 0x7f54d409b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): picked mon.noname-a con 0x7f54d409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 0x7f54d4132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): picked mon.noname-c con 0x7f54d41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d4132c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): picked mon.noname-b con 0x7f54d4137c30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:07.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d409bf10 con 0x7f54d409ea50 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4133f70 con 0x7f54d4137c30 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d409d4f0 con 0x7f54d41318f0 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d919d640 10 monclient(hunting): authenticate will time out at 1585.512444s 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 0x7f54d409b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 0x7f54d409b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44656/0 (socket says 172.21.15.153:44656) 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 1 -- 172.21.15.153:0/3463272130 learned_addr learned my addr 172.21.15.153:0/3463272130 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): get_auth_request con 0x7f54d409ea50 auth_method 0 2024-09-15T03:43:07.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 0x7f54d4132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): get_auth_request con 0x7f54d41318f0 auth_method 0 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:07.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d359e640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d4132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d359e640 10 monclient(hunting): get_auth_request con 0x7f54d4137c30 auth_method 0 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient(hunting): handle_auth_done global_id 184100 payload 824 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 -- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 msgr2=0x7f54d4132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d4132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 -- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 msgr2=0x7f54d409b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 0x7f54d409b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient: _finish_hunting 0 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 monclient: found mon.noname-c 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:07.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:07.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d259c640 1 -- 172.21.15.153:0/3463272130 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c4009390 con 0x7f54d41318f0 2024-09-15T03:43:07.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.873+0000 7f54d2d9d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 0x7f54d409b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:07.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.876+0000 7f54d259c640 1 -- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 msgr2=0x7f54d4132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:07.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.876+0000 7f54d259c640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 0x7f54d4132700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00c510 tx=0x7f54bc007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:07.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0117a0 con 0x7f54d41318f0 2024-09-15T03:43:07.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:07.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:07.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: dump: 2024-09-15T03:43:07.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:07.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc011920 con 0x7f54d41318f0 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 5 monclient: authenticate success, global_id 184100 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc011ce0 con 0x7f54d41318f0 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 10 monclient: _renew_subs 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d4133cc0 con 0x7f54d41318f0 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54caffd640 10 monclient: dump: 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:07.891 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d159a640 4 set_mon_vals no callback set 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 10 monclient: _renew_subs 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.878+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d4136730 con 0x7f54d41318f0 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f54bc012630 con 0x7f54d41318f0 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54d919d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:07.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54d919d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54d919d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d404bf60 con 0x7f54d41318f0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54caffd640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54b40387c0 0x7f54b403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f54bc04c0d0 con 0x7f54d41318f0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54d2d9d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54b40387c0 0x7f54b403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.879+0000 7f54d2d9d640 10 monclient: get_auth_request con 0x7f54b40387c0 auth_method 0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.882+0000 7f54d2d9d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54b40387c0 0x7f54b403ac70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00d7e0 tx=0x7f54cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.883+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f54bc0128e0 con 0x7f54d41318f0 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.884+0000 7f54caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:07.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:07.884+0000 7f54caffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:08.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54d919d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:08.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54d919d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:08.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54d919d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f54d41352e0 con 0x7f54d41318f0 2024-09-15T03:43:08.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54caffd640 1 -- 172.21.15.153:0/3463272130 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f54bc0102e0 con 0x7f54d41318f0 2024-09-15T03:43:08.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:08.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.097+0000 7f54caffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:08.099 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54b40387c0 msgr2=0x7f54b403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54b40387c0 0x7f54b403ac70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00d7e0 tx=0x7f54cc009040 comp rx=0 tx=0).stop 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 10 monclient: shutdown 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 msgr2=0x7f54d4132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 0x7f54d4132700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00c510 tx=0x7f54bc007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 shutdown_connections 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f54b40387c0 0x7f54b403ac70 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54d4137c30 0x7f54d4132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54d41318f0 0x7f54d4132700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 --2- 172.21.15.153:0/3463272130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d409ea50 0x7f54d409b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.099+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 >> 172.21.15.153:0/3463272130 conn(0x7f54d4067110 msgr2=0x7f54d40a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.100+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 shutdown_connections 2024-09-15T03:43:08.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.100+0000 7f54d919d640 1 -- 172.21.15.153:0/3463272130 wait complete. 2024-09-15T03:43:08.113 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:43:08.113 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:43:08.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:43:08.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.207+0000 7fcd55a3f640 10 monclient: get_monmap_and_config 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.207+0000 7fcd55a3f640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.207+0000 7fcd55a3f640 10 monclient: monmap: 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:08.208696+0000 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:08.208696+0000 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:08.210 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.208+0000 7fcd55a3f640 1 Processor -- start 2024-09-15T03:43:08.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.208+0000 7fcd55a3f640 1 -- start start 2024-09-15T03:43:08.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.208+0000 7fcd55a3f640 10 monclient: init 2024-09-15T03:43:08.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 0x7fcd5009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): picked mon.noname-b con 0x7fcd50096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd50097030 0x7fcd50097400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): picked mon.noname-a con 0x7fcd50097030 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd5012e5f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5009d2e0 con 0x7fcd5012e5f0 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5009ac00 con 0x7fcd50097030 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5009d6b0 con 0x7fcd50096c60 2024-09-15T03:43:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd55a3f640 10 monclient(hunting): authenticate will time out at 1585.848136s 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 0x7fcd5009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd50097030 0x7fcd50097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd50097030 0x7fcd50097400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44672/0 (socket says 172.21.15.153:44672) 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 0x7fcd5009a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54956/0 (socket says 172.21.15.153:54956) 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/2639069764 learned_addr learned my addr 172.21.15.153:0/2639069764 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): get_auth_request con 0x7fcd50097030 auth_method 0 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): get_auth_request con 0x7fcd50096c60 auth_method 0 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_done global_id 174186 payload 274 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 msgr2=0x7fcd5009a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 --2- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 0x7fcd5009a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/2639069764 >> v1:172.21.15.181:6789/0 conn(0x7fcd5012e5f0 legacy=0x7fcd5012ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 monclient: found mon.noname-a 2024-09-15T03:43:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/2639069764 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd500992f0 con 0x7fcd50097030 2024-09-15T03:43:08.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.209+0000 7fcd4effd640 1 --2- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 0x7fcd5009a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd50097030 msgr2=0x7fcd50097400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4e7fc640 1 --2- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd50097030 0x7fcd50097400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400bbf0 tx=0x7fcd4400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1cba680db158bfa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 1 -- 172.21.15.153:0/2639069764 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44032530 con 0x7fcd50097030 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: dump: 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.225 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 1 -- 172.21.15.153:0/2639069764 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44032690 con 0x7fcd50097030 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd55a3f640 5 monclient: authenticate success, global_id 174186 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd55a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:08.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd55a3f640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 1 -- 172.21.15.153:0/2639069764 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44032a30 con 0x7fcd50097030 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd55a3f640 4 set_mon_vals no callback set 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.212+0000 7fcd4dffb640 10 monclient: dump: 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.227 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 10 monclient: shutdown 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 -- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd50097030 msgr2=0x7fcd50097400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd50097030 0x7fcd50097400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400bbf0 tx=0x7fcd4400b460 comp rx=0 tx=0).stop 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 -- 172.21.15.153:0/2639069764 shutdown_connections 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd50097030 0x7fcd50097400 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/2639069764 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50096c60 0x7fcd5009a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 -- 172.21.15.153:0/2639069764 >> 172.21.15.153:0/2639069764 conn(0x7fcd500672b0 msgr2=0x7fcd500676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 -- 172.21.15.153:0/2639069764 shutdown_connections 2024-09-15T03:43:08.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 1 -- 172.21.15.153:0/2639069764 wait complete. 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.213+0000 7fcd55a3f640 10 monclient: monmap: 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 Processor -- start 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 -- start start 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient: init 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd50096c60 0x7fcd501411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): picked mon.noname-b con 0x7fcd50096c60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 0x7fcd5012da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): picked mon.noname-c con 0x7fcd50097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5012e5f0 0x7fcd5012dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): picked mon.noname-a con 0x7fcd5012e5f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5011eed0 con 0x7fcd5012e5f0 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50142080 con 0x7fcd50096c60 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5011d0e0 con 0x7fcd50097030 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd55a3f640 10 monclient(hunting): authenticate will time out at 1585.853817s 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 0x7fcd5012da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5012e5f0 0x7fcd5012dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 0x7fcd5012da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54970/0 (socket says 172.21.15.153:54970) 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/499730509 learned_addr learned my addr 172.21.15.153:0/499730509 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 10 monclient(hunting): get_auth_request con 0x7fcd50097030 auth_method 0 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4f7fe640 10 monclient(hunting): get_auth_request con 0x7fcd5012e5f0 auth_method 0 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.214+0000 7fcd4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4effd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd50096c60 0x7fcd501411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4effd640 10 monclient(hunting): get_auth_request con 0x7fcd50096c60 auth_method 0 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 10 monclient(hunting): handle_auth_done global_id 184106 payload 824 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd50096c60 msgr2=0x7fcd501411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd50096c60 0x7fcd501411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5012e5f0 msgr2=0x7fcd5012dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5012e5f0 0x7fcd5012dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 1 monclient: found mon.noname-c 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd40009a20 con 0x7fcd50097030 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.215+0000 7fcd4f7fe640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5012e5f0 0x7fcd5012dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd4e7fc640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 msgr2=0x7fcd5012da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd4e7fc640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 0x7fcd5012da90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400b2c0 tx=0x7fcd4403a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd440349c0 con 0x7fcd50097030 2024-09-15T03:43:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: dump: 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 5 monclient: authenticate success, global_id 184106 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44034b40 con 0x7fcd50097030 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 10 monclient: _renew_subs 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4400b700 con 0x7fcd50097030 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4403d430 con 0x7fcd50097030 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd36ffd640 10 monclient: dump: 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.236 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 10 monclient: _renew_subs 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd4d7fa640 4 set_mon_vals no callback set 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.217+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5011ebf0 con 0x7fcd50097030 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd55a3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcd44032070 con 0x7fcd50097030 2024-09-15T03:43:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 0x7fcd5011bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50065f20 con 0x7fcd5011b890 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 0x7fcd5011bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient: get_auth_request con 0x7fcd5011b890 auth_method 0 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd36ffd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd3c038740 0x7fcd3c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcd4406d5e0 con 0x7fcd50097030 2024-09-15T03:43:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 10 monclient(hunting): handle_auth_done global_id 174192 payload 824 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.218+0000 7fcd4effd640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd5004bf60 con 0x7fcd5011b890 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.219+0000 7fcd4f7fe640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd3c038740 0x7fcd3c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.219+0000 7fcd4f7fe640 10 monclient: get_auth_request con 0x7fcd3c038740 auth_method 0 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd4effd640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 msgr2=0x7fcd5011bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd4effd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 0x7fcd5011bc60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcd40011430 tx=0x7fcd4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000e030 con 0x7fcd5011b890 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 10 monclient: dump: 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcd40016970 con 0x7fcd5011b890 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 msgr2=0x7fcd5011bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 0x7fcd5011bc60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcd40011430 tx=0x7fcd4000b040 comp rx=0 tx=0).stop 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 msgr2=0x7fcd5011bc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd36ffd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011b890 0x7fcd5011bc60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.221+0000 7fcd4f7fe640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd3c038740 0x7fcd3c03abf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcd5011d6c0 tx=0x7fcd48007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd55a3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:08.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 0x7fcd5006b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd55a3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5011f110 con 0x7fcd5011c1a0 2024-09-15T03:43:08.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 0x7fcd5006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient: get_auth_request con 0x7fcd5011c1a0 auth_method 0 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 10 monclient(hunting): handle_auth_done global_id 174195 payload 824 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.227+0000 7fcd4effd640 1 -- 172.21.15.153:0/499730509 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcd500983b0 con 0x7fcd5011c1a0 2024-09-15T03:43:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.229+0000 7fcd4effd640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 msgr2=0x7fcd5006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.229+0000 7fcd4effd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 0x7fcd5006b840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd4001a000 tx=0x7fcd4001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.229+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40015a70 con 0x7fcd5011c1a0 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.229+0000 7fcd36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.229+0000 7fcd36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.229+0000 7fcd36ffd640 10 monclient: dump: 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fcd40016760 con 0x7fcd5011c1a0 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 msgr2=0x7fcd5006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 0x7fcd5006b840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd4001a000 tx=0x7fcd4001b000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 msgr2=0x7fcd5006b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:08.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd36ffd640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd5011c1a0 0x7fcd5006b840 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd4001a000 tx=0x7fcd4001b000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 146, 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:43:08.246 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.247 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.250 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:08.253 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:08.254 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.256 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:08.258 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:08.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd3c038740 msgr2=0x7fcd3c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcd3c038740 0x7fcd3c03abf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcd5011d6c0 tx=0x7fcd48007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 10 monclient: shutdown 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 msgr2=0x7fcd5012da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 --2- 172.21.15.153:0/499730509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd50097030 0x7fcd5012da90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400b2c0 tx=0x7fcd4403a000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd4f7fe640 1 -- 172.21.15.153:0/499730509 reap_dead start 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd4f7fe640 1 -- 172.21.15.153:0/499730509 reap_dead start 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 shutdown_connections 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 >> 172.21.15.153:0/499730509 conn(0x7fcd500672b0 msgr2=0x7fcd5012f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 shutdown_connections 2024-09-15T03:43:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.230+0000 7fcd55a3f640 1 -- 172.21.15.153:0/499730509 wait complete. 2024-09-15T03:43:08.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:08.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:43:08.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.354+0000 7f87a6834640 10 monclient: get_monmap_and_config 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.354+0000 7f87a6834640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.354+0000 7f87a6834640 10 monclient: monmap: 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:08.355010+0000 2024-09-15T03:43:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:08.355010+0000 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:08.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.354+0000 7f87a6834640 1 Processor -- start 2024-09-15T03:43:08.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 1 -- start start 2024-09-15T03:43:08.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient: init 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87a0131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): picked mon.noname-b con 0x7f87a009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f87a009f220 0x7f87a0138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): picked mon.noname-a con 0x7f87a009f220 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a009faf0 con 0x7f87a0131ab0 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a0132590 con 0x7f87a009f220 2024-09-15T03:43:08.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a009fd30 con 0x7f87a009e910 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a6834640 10 monclient(hunting): authenticate will time out at 1585.994571s 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f87a009f220 0x7f87a0138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f87a009f220 0x7f87a0138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44710/0 (socket says 172.21.15.153:44710) 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): get_auth_request con 0x7f87a009e910 auth_method 0 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 1 -- 172.21.15.153:0/744132499 learned_addr learned my addr 172.21.15.153:0/744132499 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): get_auth_request con 0x7f87a009f220 auth_method 0 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 10 monclient(hunting): handle_auth_done global_id 174198 payload 274 2024-09-15T03:43:08.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 1 -- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 msgr2=0x7f87a009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.355+0000 7f87a4dd1640 1 --2- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f87a4dd1640 1 -- 172.21.15.153:0/744132499 >> v1:172.21.15.181:6789/0 conn(0x7f87a0131ab0 legacy=0x7f87a0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f87a4dd1640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f87a4dd1640 1 monclient: found mon.noname-a 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f87a4dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f87a4dd1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f87a4dd1640 1 -- 172.21.15.153:0/744132499 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a00a1110 con 0x7f87a009f220 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.356+0000 7f879f7fe640 1 --2- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f87a4dd1640 1 -- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 msgr2=0x7f87a0138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f87a4dd1640 1 --2- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0138bc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8798009f20 tx=0x7f8798009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e43aee8821bfa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 1 -- 172.21.15.153:0/744132499 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798007610 con 0x7f87a009f220 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 10 monclient: dump: 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.365 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 5 monclient: authenticate success, global_id 174198 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.358+0000 7f879effd640 1 -- 172.21.15.153:0/744132499 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879802d050 con 0x7f87a009f220 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f879effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f879effd640 1 -- 172.21.15.153:0/744132499 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980079a0 con 0x7f87a009f220 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f879effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f879effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f879effd640 10 monclient: dump: 2024-09-15T03:43:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:08.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 4 set_mon_vals no callback set 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 10 monclient: shutdown 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 -- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 msgr2=0x7f87a0138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 --2- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0138bc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8798009f20 tx=0x7f8798009840 comp rx=0 tx=0).stop 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 -- 172.21.15.153:0/744132499 shutdown_connections 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 --2- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0138bc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 --2- 172.21.15.153:0/744132499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 -- 172.21.15.153:0/744132499 >> 172.21.15.153:0/744132499 conn(0x7f87a0067d30 msgr2=0x7f87a00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 -- 172.21.15.153:0/744132499 shutdown_connections 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.359+0000 7f87a6834640 1 -- 172.21.15.153:0/744132499 wait complete. 2024-09-15T03:43:08.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.360+0000 7f87a6834640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.360+0000 7f87a6834640 10 monclient: monmap: 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:08.369 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.360+0000 7f87a6834640 1 Processor -- start 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.360+0000 7f87a6834640 1 -- start start 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.360+0000 7f87a6834640 10 monclient: init 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): picked mon.noname-c con 0x7f87a009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): picked mon.noname-a con 0x7f87a009f220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87a0131ab0 0x7f87a0096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): picked mon.noname-b con 0x7f87a0131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a009a200 con 0x7f87a009f220 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0098650 con 0x7f87a0131ab0 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a009b8e0 con 0x7f87a009e910 2024-09-15T03:43:08.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a6834640 10 monclient(hunting): authenticate will time out at 1586.000168s 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54982/0 (socket says 172.21.15.153:54982) 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 -- 172.21.15.153:0/229001259 learned_addr learned my addr 172.21.15.153:0/229001259 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): get_auth_request con 0x7f87a009e910 auth_method 0 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): get_auth_request con 0x7f87a009f220 auth_method 0 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a4dd1640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87a0131ab0 0x7f87a0096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a4dd1640 10 monclient(hunting): get_auth_request con 0x7f87a0131ab0 auth_method 0 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a4dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a4dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a4dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient(hunting): handle_auth_done global_id 184112 payload 824 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87a0131ab0 msgr2=0x7f87a0096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87a0131ab0 0x7f87a0096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 msgr2=0x7f87a0096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 monclient: found mon.noname-c 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879ffff640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788009390 con 0x7f87a009e910 2024-09-15T03:43:08.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f87a4dd1640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87a0131ab0 0x7f87a0096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.361+0000 7f879f7fe640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87a009f220 0x7f87a0096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.363+0000 7f879ffff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 msgr2=0x7f87a009a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.363+0000 7f879ffff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009a620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f879000c610 tx=0x7f8790007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.363+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790012840 con 0x7f87a009e910 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.363+0000 7f8797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: dump: 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 5 monclient: authenticate success, global_id 184112 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87900129c0 con 0x7f87a009e910 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 10 monclient: _renew_subs 2024-09-15T03:43:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8798009ae0 con 0x7f87a009e910 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790012d80 con 0x7f87a009e910 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 10 monclient: dump: 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.384 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.384 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.384 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.384 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.384 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 10 monclient: _renew_subs 2024-09-15T03:43:08.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f879e7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:08.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a0099b00 con 0x7f87a009e910 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8790011590 con 0x7f87a009e910 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 0x7f87a00974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0063740 con 0x7f87a01312a0 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.364+0000 7f879f7fe640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 0x7f87a00974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient: get_auth_request con 0x7f87a01312a0 auth_method 0 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f8797fff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8780038740 0x7f878003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f879004c7a0 con 0x7f87a009e910 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 10 monclient(hunting): handle_auth_done global_id 184115 payload 824 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f879f7fe640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a00996d0 con 0x7f87a01312a0 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f87a4dd1640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8780038740 0x7f878003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.365+0000 7f87a4dd1640 10 monclient: get_auth_request con 0x7f8780038740 auth_method 0 2024-09-15T03:43:08.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f879f7fe640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 msgr2=0x7f87a00974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f879f7fe640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 0x7f87a00974e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8788011ef0 tx=0x7f87880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788014940 con 0x7f87a01312a0 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f8797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f8797fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f8797fff640 10 monclient: dump: 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.367+0000 7f87a4dd1640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8780038740 0x7f878003abf0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f87a0076760 tx=0x7f8798036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8788014aa0 con 0x7f87a01312a0 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 msgr2=0x7f87a00974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 0x7f87a00974e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8788011ef0 tx=0x7f87880090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 msgr2=0x7f87a00974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.368+0000 7f8797fff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a01312a0 0x7f87a00974e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8788011ef0 tx=0x7f87880090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f87a6834640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f87a6834640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 0x7f87a009d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f87a6834640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a004bf60 con 0x7f87a009d0e0 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 0x7f87a009d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient: get_auth_request con 0x7f87a009d0e0 auth_method 0 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.373+0000 7f879f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.374+0000 7f879f7fe640 10 monclient(hunting): handle_auth_done global_id 184118 payload 824 2024-09-15T03:43:08.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.374+0000 7f879f7fe640 1 -- 172.21.15.153:0/229001259 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f87a0066280 con 0x7f87a009d0e0 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f879f7fe640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 msgr2=0x7f87a009d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f879f7fe640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 0x7f87a009d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f878801b000 tx=0x7f878801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878801c800 con 0x7f87a009d0e0 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 10 monclient: dump: 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.392 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f878801c960 con 0x7f87a009d0e0 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 msgr2=0x7f87a009d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 0x7f87a009d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f878801b000 tx=0x7f878801c000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 msgr2=0x7f87a009d4e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.376+0000 7f8797fff640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009d0e0 0x7f87a009d4e0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f878801b000 tx=0x7f878801c000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:08.394 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 146, 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.395 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.396 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:08.397 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:08.398 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.399 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.402 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.403 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.404 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.405 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:08.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.407 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.408 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:08.408 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:08.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8780038740 msgr2=0x7f878003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8780038740 0x7f878003abf0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f87a0076760 tx=0x7f8798036000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 10 monclient: shutdown 2024-09-15T03:43:08.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 msgr2=0x7f87a009a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 --2- 172.21.15.153:0/229001259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87a009e910 0x7f87a009a620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f879000c610 tx=0x7f8790007ed0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a4dd1640 1 -- 172.21.15.153:0/229001259 reap_dead start 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a4dd1640 1 -- 172.21.15.153:0/229001259 reap_dead start 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 shutdown_connections 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 >> 172.21.15.153:0/229001259 conn(0x7f87a0067d30 msgr2=0x7f87a0068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 shutdown_connections 2024-09-15T03:43:08.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.377+0000 7f87a6834640 1 -- 172.21.15.153:0/229001259 wait complete. 2024-09-15T03:43:08.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:08.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:43:08.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:43:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.503+0000 7f81d6300640 10 monclient: get_monmap_and_config 2024-09-15T03:43:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.503+0000 7f81d6300640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.503+0000 7f81d6300640 10 monclient: monmap: 2024-09-15T03:43:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:08.504237+0000 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:08.504237+0000 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:08.505 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 Processor -- start 2024-09-15T03:43:08.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 -- start start 2024-09-15T03:43:08.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient: init 2024-09-15T03:43:08.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 0x7f81d0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): picked mon.noname-a con 0x7f81d01318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 0x7f81d009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): picked mon.noname-b con 0x7f81d009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81d0068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d0131cc0 con 0x7f81d0068d90 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d009fbf0 con 0x7f81d01318f0 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d009fe30 con 0x7f81d009ea50 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81d6300640 10 monclient(hunting): authenticate will time out at 1586.143801s 2024-09-15T03:43:08.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 0x7f81d0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 0x7f81d009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 0x7f81d0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44734/0 (socket says 172.21.15.153:44734) 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 1 -- 172.21.15.153:0/546343777 learned_addr learned my addr 172.21.15.153:0/546343777 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d01318f0 auth_method 0 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request con 0x7f81d009ea50 auth_method 0 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.504+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_done global_id 184121 payload 274 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 1 -- 172.21.15.153:0/546343777 >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 msgr2=0x7f81d0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/546343777 >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 0x7f81d0131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 1 -- 172.21.15.153:0/546343777 >> v1:172.21.15.181:6789/0 conn(0x7f81d0068d90 legacy=0x7f81d00691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 1 monclient: found mon.noname-b 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cf7fe640 1 -- 172.21.15.153:0/546343777 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00a1210 con 0x7f81d009ea50 2024-09-15T03:43:08.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.505+0000 7f81cffff640 1 --2- 172.21.15.153:0/546343777 >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 0x7f81d0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.507+0000 7f81cf7fe640 1 -- 172.21.15.153:0/546343777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 msgr2=0x7f81d009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.507+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/546343777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 0x7f81d009ee20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009850 tx=0x7f81c40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e2fb1cfc1ebf2f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 1 -- 172.21.15.153:0/546343777 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c402adf0 con 0x7f81d009ea50 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: dump: 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 5 monclient: authenticate success, global_id 184121 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 1 -- 172.21.15.153:0/546343777 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4032440 con 0x7f81d009ea50 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 4 set_mon_vals no callback set 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 1 -- 172.21.15.153:0/546343777 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c40327e0 con 0x7f81d009ea50 2024-09-15T03:43:08.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81ceffd640 10 monclient: dump: 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 10 monclient: shutdown 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 -- 172.21.15.153:0/546343777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 msgr2=0x7f81d009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 --2- 172.21.15.153:0/546343777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 0x7f81d009ee20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009850 tx=0x7f81c40090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 -- 172.21.15.153:0/546343777 shutdown_connections 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 --2- 172.21.15.153:0/546343777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d009ea50 0x7f81d009ee20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 --2- 172.21.15.153:0/546343777 >> v2:172.21.15.153:3300/0 conn(0x7f81d01318f0 0x7f81d0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 -- 172.21.15.153:0/546343777 >> 172.21.15.153:0/546343777 conn(0x7f81d0067110 msgr2=0x7f81d0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 -- 172.21.15.153:0/546343777 shutdown_connections 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.508+0000 7f81d6300640 1 -- 172.21.15.153:0/546343777 wait complete. 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.509+0000 7f81d6300640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.509+0000 7f81d6300640 10 monclient: monmap: 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:08.520 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.509+0000 7f81d6300640 1 Processor -- start 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.509+0000 7f81d6300640 1 -- start start 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.509+0000 7f81d6300640 10 monclient: init 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 0x7f81d009a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): picked mon.noname-c con 0x7f81d0068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d009ea50 0x7f81d0130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): picked mon.noname-b con 0x7f81d009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d01318f0 0x7f81d011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): picked mon.noname-a con 0x7f81d01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d009bf10 con 0x7f81d01318f0 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d009e790 con 0x7f81d009ea50 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d011c950 con 0x7f81d0068d90 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d6300640 10 monclient(hunting): authenticate will time out at 1586.149257s 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 0x7f81d009a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 0x7f81d009a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55026/0 (socket says 172.21.15.153:55026) 2024-09-15T03:43:08.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 -- 172.21.15.153:0/248473425 learned_addr learned my addr 172.21.15.153:0/248473425 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d0068d90 auth_method 0 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d01318f0 0x7f81d011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): get_auth_request con 0x7f81d01318f0 auth_method 0 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d009ea50 0x7f81d0130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request con 0x7f81d009ea50 auth_method 0 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient(hunting): handle_auth_done global_id 184124 payload 824 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d009ea50 msgr2=0x7f81d0130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d009ea50 0x7f81d0130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d01318f0 msgr2=0x7f81d011b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d01318f0 0x7f81d011b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 monclient: found mon.noname-c 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81cffff640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c0009450 con 0x7f81d0068d90 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.510+0000 7f81d489d640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d01318f0 0x7f81d011b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.512+0000 7f81cffff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 msgr2=0x7f81d009a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.512+0000 7f81cffff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 0x7f81d009a9a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81c000ec80 tx=0x7f81c00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.512+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0009730 con 0x7f81d0068d90 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.512+0000 7f81b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: dump: 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.525 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 5 monclient: authenticate success, global_id 184124 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c000eec0 con 0x7f81d0068d90 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 10 monclient: _renew_subs 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c4009390 con 0x7f81d0068d90 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0013470 con 0x7f81d0068d90 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 10 monclient: dump: 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 10 monclient: _renew_subs 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d01413a0 con 0x7f81d0068d90 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81ce7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f81c0013820 con 0x7f81d0068d90 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 0x7f81d011bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.513+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d0065e00 con 0x7f81d0131150 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81b7fff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81bc038740 0x7f81bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f81c004ca30 con 0x7f81d0068d90 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 0x7f81d011bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient: get_auth_request con 0x7f81d0131150 auth_method 0 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81d489d640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81bc038740 0x7f81bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81d489d640 10 monclient: get_auth_request con 0x7f81bc038740 auth_method 0 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_done global_id 144409 payload 824 2024-09-15T03:43:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.514+0000 7f81cf7fe640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81d011de90 con 0x7f81d0131150 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.516+0000 7f81d489d640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81bc038740 0x7f81bc03abf0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f81c8007fd0 tx=0x7f81c80025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.516+0000 7f81cf7fe640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 msgr2=0x7f81d011bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.516+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 0x7f81d011bc60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81c4034000 tx=0x7f81c4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c403ca40 con 0x7f81d0131150 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 10 monclient: dump: 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f81c403cba0 con 0x7f81d0131150 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 msgr2=0x7f81d011bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 0x7f81d011bc60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81c4034000 tx=0x7f81c4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 msgr2=0x7f81d011bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.517+0000 7f81b7fff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d0131150 0x7f81d011bc60 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81c4034000 tx=0x7f81c4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81d6300640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81d6300640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 0x7f81d0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81d6300640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d0131520 con 0x7f81d011c1a0 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 0x7f81d0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient: get_auth_request con 0x7f81d011c1a0 auth_method 0 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_done global_id 144412 payload 824 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.523+0000 7f81cf7fe640 1 -- 172.21.15.153:0/248473425 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81d011c5a0 con 0x7f81d011c1a0 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81cf7fe640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 msgr2=0x7f81d0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81cf7fe640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 0x7f81d0096a60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81c4042000 tx=0x7f81c403c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c40479a0 con 0x7f81d011c1a0 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 10 monclient: dump: 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f81c4047b00 con 0x7f81d011c1a0 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 msgr2=0x7f81d0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 0x7f81d0096a60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81c4042000 tx=0x7f81c403c000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 msgr2=0x7f81d0096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.526+0000 7f81b7fff640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81d011c1a0 0x7f81d0096a60 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81c4042000 tx=0x7f81c403c000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 146, 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.539 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:08.541 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.542 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.543 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:08.546 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:08.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.549 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:08.550 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81bc038740 msgr2=0x7f81bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81bc038740 0x7f81bc03abf0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f81c8007fd0 tx=0x7f81c80025b0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 10 monclient: shutdown 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 msgr2=0x7f81d009a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 --2- 172.21.15.153:0/248473425 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d0068d90 0x7f81d009a9a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81c000ec80 tx=0x7f81c00091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d489d640 1 -- 172.21.15.153:0/248473425 reap_dead start 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d489d640 1 -- 172.21.15.153:0/248473425 reap_dead start 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 shutdown_connections 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 >> 172.21.15.153:0/248473425 conn(0x7f81d0067110 msgr2=0x7f81d00a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 shutdown_connections 2024-09-15T03:43:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.527+0000 7f81d6300640 1 -- 172.21.15.153:0/248473425 wait complete. 2024-09-15T03:43:08.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:08.553 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:43:08.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:43:08.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.645+0000 7f3a8d8c9640 10 monclient: get_monmap_and_config 2024-09-15T03:43:08.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.645+0000 7f3a8d8c9640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.645+0000 7f3a8d8c9640 10 monclient: monmap: 2024-09-15T03:43:08.646 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.646 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:08.646385+0000 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:08.646385+0000 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:08.647 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 Processor -- start 2024-09-15T03:43:08.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 -- start start 2024-09-15T03:43:08.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient: init 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3a88131ab0 0x7f3a88131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3a88131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a8809e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3a88136bb0 0x7f3a88138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3a88136bb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809faf0 con 0x7f3a8809e910 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8809fd30 con 0x7f3a88131ab0 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a880a1110 con 0x7f3a88136bb0 2024-09-15T03:43:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.646+0000 7f3a8d8c9640 10 monclient(hunting): authenticate will time out at 1586.285870s 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3a88131ab0 0x7f3a88131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3a88131ab0 0x7f3a88131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44754/0 (socket says 172.21.15.153:44754) 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1377521957 learned_addr learned my addr 172.21.15.153:0/1377521957 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): get_auth_request con 0x7f3a88131ab0 auth_method 0 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 1 --2- 172.21.15.153:0/1377521957 >> v2:172.21.15.181:3300/0 conn(0x7f3a88136bb0 0x7f3a88138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): get_auth_request con 0x7f3a88136bb0 auth_method 0 2024-09-15T03:43:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_done global_id 174210 payload 274 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1377521957 >> v2:172.21.15.181:3300/0 conn(0x7f3a88136bb0 msgr2=0x7f3a88138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 --2- 172.21.15.153:0/1377521957 >> v2:172.21.15.181:3300/0 conn(0x7f3a88136bb0 0x7f3a88138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1377521957 >> v1:172.21.15.181:6789/0 conn(0x7f3a8809e910 legacy=0x7f3a8809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 monclient: found mon.noname-a 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.647+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1377521957 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a880a13f0 con 0x7f3a88131ab0 2024-09-15T03:43:08.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1377521957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a88131ab0 msgr2=0x7f3a88131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a86ffd640 1 --2- 172.21.15.153:0/1377521957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a88131ab0 0x7f3a88131e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009910 tx=0x7f3a780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218d45acfd7f576 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 1 -- 172.21.15.153:0/1377521957 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78032440 con 0x7f3a88131ab0 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: dump: 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.659 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 1 -- 172.21.15.153:0/1377521957 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a780325a0 con 0x7f3a88131ab0 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 5 monclient: authenticate success, global_id 174210 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 1 -- 172.21.15.153:0/1377521957 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78032940 con 0x7f3a88131ab0 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 4 set_mon_vals no callback set 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:08.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a867fc640 10 monclient: dump: 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 10 monclient: shutdown 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1377521957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a88131ab0 msgr2=0x7f3a88131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1377521957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a88131ab0 0x7f3a88131e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009910 tx=0x7f3a780091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1377521957 shutdown_connections 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1377521957 >> v2:172.21.15.181:3300/0 conn(0x7f3a88136bb0 0x7f3a88138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1377521957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a88131ab0 0x7f3a88131e80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1377521957 >> 172.21.15.153:0/1377521957 conn(0x7f3a88067d30 msgr2=0x7f3a880a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1377521957 shutdown_connections 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.650+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1377521957 wait complete. 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.651+0000 7f3a8d8c9640 10 monclient: build_initial_monmap 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.651+0000 7f3a8d8c9640 10 monclient: monmap: 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:08.662 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.651+0000 7f3a8d8c9640 1 Processor -- start 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.651+0000 7f3a8d8c9640 1 -- start start 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.651+0000 7f3a8d8c9640 10 monclient: init 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 0x7f3a8812fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3a8809e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a88131ab0 0x7f3a88130320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f3a88131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a88136bb0 0x7f3a88097520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3a88136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a880a2240 con 0x7f3a8809e910 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8809ac40 con 0x7f3a88131ab0 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8809af20 con 0x7f3a88136bb0 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a8d8c9640 10 monclient(hunting): authenticate will time out at 1586.291148s 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 0x7f3a8812fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 0x7f3a8812fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44758/0 (socket says 172.21.15.153:44758) 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1896017056 learned_addr learned my addr 172.21.15.153:0/1896017056 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): get_auth_request con 0x7f3a8809e910 auth_method 0 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a88136bb0 0x7f3a88097520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): get_auth_request con 0x7f3a88136bb0 auth_method 0 2024-09-15T03:43:08.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a88131ab0 0x7f3a88130320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): get_auth_request con 0x7f3a88131ab0 auth_method 0 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient(hunting): handle_auth_done global_id 174213 payload 824 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a88136bb0 msgr2=0x7f3a88097520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a88136bb0 0x7f3a88097520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a88131ab0 msgr2=0x7f3a88130320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a88131ab0 0x7f3a88130320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 monclient: found mon.noname-a 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.652+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1896017056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a78009450 con 0x7f3a8809e910 2024-09-15T03:43:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a86ffd640 1 -- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 msgr2=0x7f3a8812fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a86ffd640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 0x7f3a8812fde0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a78035040 tx=0x7f3a78032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78031a10 con 0x7f3a8809e910 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: dump: 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: _finish_auth 0 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a78031b90 con 0x7f3a8809e910 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a8d8c9640 5 monclient: authenticate success, global_id 174213 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a8d8c9640 10 monclient: _renew_subs 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a8d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a88099180 con 0x7f3a8809e910 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7803d440 con 0x7f3a8809e910 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a7effd640 10 monclient: dump: 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a85ffb640 4 set_mon_vals no callback set 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.654+0000 7f3a8d8c9640 10 monclient: _renew_subs 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a8d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a88099d50 con 0x7f3a8809e910 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3a78039040 con 0x7f3a8809e910 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a8d8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a8d8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a8d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8804bf60 con 0x7f3a8809e910 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a7effd640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.655+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3a7806c0e0 con 0x7f3a8809e910 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.656+0000 7f3a7ffff640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:08.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.656+0000 7f3a7ffff640 10 monclient: get_auth_request con 0x7f3a68038740 auth_method 0 2024-09-15T03:43:08.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.658+0000 7f3a7ffff640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3a8809df00 tx=0x7f3a70007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:08.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.660+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3a78044af0 con 0x7f3a8809e910 2024-09-15T03:43:08.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.660+0000 7f3a7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:08.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.660+0000 7f3a7effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:08.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.874+0000 7f3a8d8c9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:43:08.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.874+0000 7f3a8d8c9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:43:08.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.874+0000 7f3a8d8c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:08.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.874+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3a880a26a0 con 0x7f3a8809e910 2024-09-15T03:43:08.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f3a7effd640 1 -- 172.21.15.153:0/1896017056 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3a78044310 con 0x7f3a8809e910 2024-09-15T03:43:08.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f3a7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:43:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f3a7effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:08.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=8,logm=50} crc {auth=3806993889,config=3140514009,health=837413035,kv=3235807718,logm=42705250}) 2024-09-15T03:43:08.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=1683593157,config=3140514009,health=2867440668,kv=3235807718,logm=3512690351}) 2024-09-15T03:43:08.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=10,kv=8,logm=50} crc {auth=3806993889,config=3140514009,health=837413035,kv=3235807718,logm=42705250}) 2024-09-15T03:43:08.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.875+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=7,logm=51} crc {auth=3447334624,config=3140514009,health=2132524724,kv=1577269308,logm=1096854174}) 2024-09-15T03:43:08.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.876+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.876+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2059033778}) 2024-09-15T03:43:08.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.876+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255273458}) 2024-09-15T03:43:08.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.876+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.877 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.876+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2059033778}) 2024-09-15T03:43:08.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.876+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036275770}) 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 msgr2=0x7f3a6803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3a8809df00 tx=0x7f3a70007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 10 monclient: shutdown 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 msgr2=0x7f3a8812fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 0x7f3a8812fde0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a78035040 tx=0x7f3a78032000 comp rx=0 tx=0).stop 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 shutdown_connections 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a88136bb0 0x7f3a88097520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a88131ab0 0x7f3a88130320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 --2- 172.21.15.153:0/1896017056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a8809e910 0x7f3a8812fde0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 >> 172.21.15.153:0/1896017056 conn(0x7f3a88067d30 msgr2=0x7f3a8809d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 shutdown_connections 2024-09-15T03:43:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:08.877+0000 7f3a8d8c9640 1 -- 172.21.15.153:0/1896017056 wait complete. 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.878+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.878+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519061137}) 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.878+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1575611871}) 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.878+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.878+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519061137}) 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.878+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465799019}) 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.879+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.882 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.879+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203777386}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.879+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1280920537}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.879+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.879+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203777386}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.879+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040340836}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.880+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.880+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181178720}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.880+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1150914710}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.880+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.880+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181178720}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.880+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1856279424}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.881+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.881+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044114232}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.881+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1063117180}) 2024-09-15T03:43:08.883 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.881+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.881+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044114232}) 2024-09-15T03:43:08.884 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.881+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320638189}) 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.882+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.882+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953989888}) 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.882+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68737053}) 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.882+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.882+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953989888}) 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.882+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985048613}) 2024-09-15T03:43:08.886 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.883+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.883+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469651455}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.883+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2879224461}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.883+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.883+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469651455}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.883+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156430199}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.884+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.884+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068010105}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.884+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661257418}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.884+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.884+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068010105}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.884+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1345295102}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.885+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.885+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782224976}) 2024-09-15T03:43:08.887 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.885+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280852638}) 2024-09-15T03:43:08.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.885+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.885+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782224976}) 2024-09-15T03:43:08.888 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.885+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109404698}) 2024-09-15T03:43:08.890 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.886+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.886+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405017959}) 2024-09-15T03:43:08.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.886+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002094940}) 2024-09-15T03:43:08.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.886+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.886+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405017959}) 2024-09-15T03:43:08.891 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.886+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912782987}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.887+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.887+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102684938}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.887+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590227990}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.887+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.887+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102684938}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.887+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493425803}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479857224}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1760062867}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479857224}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3346370782}) 2024-09-15T03:43:08.892 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.888+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231045136}) 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975730335}) 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231045136}) 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750635674}) 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555440702}) 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375855047}) 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.893 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555440702}) 2024-09-15T03:43:08.894 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.889+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267958130}) 2024-09-15T03:43:08.895 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.890+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.895 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.890+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421357485}) 2024-09-15T03:43:08.895 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.890+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700372347}) 2024-09-15T03:43:08.895 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.890+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.895 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.890+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421357485}) 2024-09-15T03:43:08.895 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.890+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328633275}) 2024-09-15T03:43:08.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.891+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.891+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583467987}) 2024-09-15T03:43:08.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.891+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=530143150}) 2024-09-15T03:43:08.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.891+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.891+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583467987}) 2024-09-15T03:43:08.897 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.891+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569638779}) 2024-09-15T03:43:08.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.892+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.892+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769705021}) 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.892+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=284572496}) 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.892+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.892+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769705021}) 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.892+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993706449}) 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880143125}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481309836}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880143125}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750519543}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456193349}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812361349}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456193349}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.893+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201481858}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.894+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.894+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065250781}) 2024-09-15T03:43:08.911 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.894+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057315835}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.894+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.894+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065250781}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.894+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=631081129}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.895+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.895+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663712512}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.895+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792459746}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.895+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.895+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663712512}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.895+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408039551}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4088246796}) 2024-09-15T03:43:08.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092395126}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4088246796}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2760447055}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2098458220}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4088843670}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2098458220}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.896+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688905435}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.897+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.897+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3142469226}) 2024-09-15T03:43:08.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.897+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244817513}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.897+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.897+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3142469226}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.897+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574153040}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1947306140}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187326823}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1947306140}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853777603}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4463209}) 2024-09-15T03:43:08.914 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525366435}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4463209}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.898+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040406326}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.899+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.899+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636845084}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.899+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3210096299}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.899+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.899+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636845084}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.899+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2030294090}) 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226885843}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522931435}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226885843}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020625794}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610185687}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.900+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244646725}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610185687}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=78997935}) 2024-09-15T03:43:08.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163568355}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545821997}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163568355}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.901+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471773206}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.902+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.902+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044724187}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.902+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386793084}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.902+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.902+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044724187}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.902+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515313745}) 2024-09-15T03:43:08.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.903+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.903+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664637045}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.903+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3095238680}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.903+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.903+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664637045}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.903+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428373810}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421806485}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389754693}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421806485}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838905815}) 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165780916}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678824028}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165780916}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.904+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292665122}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.905+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.905+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259739214}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.905+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2214643192}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.905+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.905+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259739214}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.905+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511768016}) 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956217042}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924475858}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956217042}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233669259}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791121686}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2598818684}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791121686}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.906+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2408329255}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.907+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.907+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274162575}) 2024-09-15T03:43:08.920 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.907+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827518685}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.907+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.907+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274162575}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.907+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026411251}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126330670}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3609601121}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126330670}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2779705655}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609188341}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501253470}) 2024-09-15T03:43:08.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609188341}) 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.908+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710615899}) 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.909+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.909+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076645850}) 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.909+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1443280824}) 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.909+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.909+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076645850}) 2024-09-15T03:43:08.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.909+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3693610129}) 2024-09-15T03:43:08.930 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416820337}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734717619}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416820337}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111202530}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921506604}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2350188635}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921506604}) 2024-09-15T03:43:08.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.910+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3528455643}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.911+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.911+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574037703}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.911+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1462686642}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.911+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.911+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574037703}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.911+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24503127}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886521938}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256576401}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886521938}) 2024-09-15T03:43:08.932 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2739139964}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=783448887}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3166079794}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=783448887}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.912+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217679566}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.913+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.913+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868726723}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.913+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3295958472}) 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.913+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.913+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868726723}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.913+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943817005}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158405548}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386706931}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158405548}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747676087}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301100321}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147369014}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301100321}) 2024-09-15T03:43:08.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.914+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980726731}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4685189}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1764030829}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4685189}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778552504}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.915+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826869596}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596157368}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826869596}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193735213}) 2024-09-15T03:43:08.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605357238}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3517796194}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605357238}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.916+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2540969334}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946408436}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234401615}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946408436}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052281144}) 2024-09-15T03:43:08.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650683748}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718878678}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650683748}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.917+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206787713}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.918+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.918+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756669092}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.918+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2908610155}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.918+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.918+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756669092}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.918+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573247627}) 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613410759}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712657439}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613410759}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428877610}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642202693}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2013359770}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642202693}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.919+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714421865}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935224526}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670734025}) 2024-09-15T03:43:08.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935224526}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606525099}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.920+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973347626}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2163629701}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973347626}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995791954}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654723775}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779678042}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654723775}) 2024-09-15T03:43:08.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.921+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017449915}) 2024-09-15T03:43:08.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.922+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.922+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431961661}) 2024-09-15T03:43:08.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.922+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810985990}) 2024-09-15T03:43:08.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.922+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:08.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.922+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431961661}) 2024-09-15T03:43:08.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:08.922+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428507926}) 2024-09-15T03:43:09.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.007+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.007+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668984012}) 2024-09-15T03:43:09.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.007+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=23931110}) 2024-09-15T03:43:09.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.007+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.007+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668984012}) 2024-09-15T03:43:09.010 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.007+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2300636899}) 2024-09-15T03:43:09.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207231954}) 2024-09-15T03:43:09.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1463022347}) 2024-09-15T03:43:09.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207231954}) 2024-09-15T03:43:09.110 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.107+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787141881}) 2024-09-15T03:43:09.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.119+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.119+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309080032}) 2024-09-15T03:43:09.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.119+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2206456654}) 2024-09-15T03:43:09.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.119+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.119+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309080032}) 2024-09-15T03:43:09.120 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.119+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451705982}) 2024-09-15T03:43:09.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.278+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.278+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089313231}) 2024-09-15T03:43:09.280 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.278+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2742972433}) 2024-09-15T03:43:09.280 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.278+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.280 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.278+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089313231}) 2024-09-15T03:43:09.280 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.278+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022587889}) 2024-09-15T03:43:09.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.281+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.281+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613720446}) 2024-09-15T03:43:09.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.281+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317663399}) 2024-09-15T03:43:09.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.281+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.281+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613720446}) 2024-09-15T03:43:09.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.281+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1998699871}) 2024-09-15T03:43:09.899 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:43:09.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:09.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052857776}) 2024-09-15T03:43:09.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2714043771}) 2024-09-15T03:43:09.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:09.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052857776}) 2024-09-15T03:43:09.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:09.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181168761}) 2024-09-15T03:43:09.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.997+0000 7fd1aa59d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:09.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.997+0000 7fd1aa59d640 10 monclient: build_initial_monmap 2024-09-15T03:43:09.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.997+0000 7fd1aa59d640 10 monclient: monmap: 2024-09-15T03:43:09.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:09.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:09.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:09.998355+0000 2024-09-15T03:43:09.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:09.998355+0000 2024-09-15T03:43:10.000 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.000 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.000 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:10.000 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:10.000 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:10.001 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:10.001 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:10.001 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 1 Processor -- start 2024-09-15T03:43:10.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 1 -- start start 2024-09-15T03:43:10.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 10 monclient: init 2024-09-15T03:43:10.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:43:10.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1a40975c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:10.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1a412ea30 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:10.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 0x7fd1a413a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.998+0000 7fd1aa59d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a412ee50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1aa59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a406e9e0 con 0x7fd1a412ea30 2024-09-15T03:43:10.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1aa59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a409d1c0 con 0x7fd1a40975c0 2024-09-15T03:43:10.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1aa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1aa59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a4063950 con 0x7fd1a412ee50 2024-09-15T03:43:10.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1aa59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1aa59d640 10 monclient(hunting): authenticate will time out at 1587.638055s 2024-09-15T03:43:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd1a412ea30 0x7fd1a4136c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44768/0 (socket says 172.21.15.153:44768) 2024-09-15T03:43:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 0x7fd1a413a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a37fe640 1 -- 172.21.15.153:0/2659256592 learned_addr learned my addr 172.21.15.153:0/2659256592 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): get_auth_request con 0x7fd1a412ee50 auth_method 0 2024-09-15T03:43:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient(hunting): handle_auth_done global_id 184130 payload 274 2024-09-15T03:43:10.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 1 -- 172.21.15.153:0/2659256592 >> v1:172.21.15.181:6789/0 conn(0x7fd1a40975c0 legacy=0x7fd1a4097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 1 -- 172.21.15.153:0/2659256592 >> v1:172.21.15.153:6789/0 conn(0x7fd1a412ea30 legacy=0x7fd1a4136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 1 monclient: found mon.noname-b 2024-09-15T03:43:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:09.999+0000 7fd1a8b3a640 1 -- 172.21.15.153:0/2659256592 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a409d4a0 con 0x7fd1a412ee50 2024-09-15T03:43:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a8b3a640 1 -- 172.21.15.153:0/2659256592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 msgr2=0x7fd1a413a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a8b3a640 1 --2- 172.21.15.153:0/2659256592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 0x7fd1a413a3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4064510 tx=0x7fd19c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=943927a117cf3b1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 1 -- 172.21.15.153:0/2659256592 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c007840 con 0x7fd1a412ee50 2024-09-15T03:43:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: dump: 2024-09-15T03:43:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.027 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.027 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.027 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.028 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.028 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.028 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 1 -- 172.21.15.153:0/2659256592 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c02d050 con 0x7fd1a412ee50 2024-09-15T03:43:10.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 5 monclient: authenticate success, global_id 184130 2024-09-15T03:43:10.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:10.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 1 -- 172.21.15.153:0/2659256592 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c007bd0 con 0x7fd1a412ee50 2024-09-15T03:43:10.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:10.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 4 set_mon_vals no callback set 2024-09-15T03:43:10.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1a2ffd640 10 monclient: dump: 2024-09-15T03:43:10.031 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.033 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 10 monclient: shutdown 2024-09-15T03:43:10.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2659256592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 msgr2=0x7fd1a413a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2659256592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 0x7fd1a413a3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4064510 tx=0x7fd19c009780 comp rx=0 tx=0).stop 2024-09-15T03:43:10.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2659256592 shutdown_connections 2024-09-15T03:43:10.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2659256592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ee50 0x7fd1a413a3b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2659256592 >> 172.21.15.153:0/2659256592 conn(0x7fd1a4067b50 msgr2=0x7fd1a40a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2659256592 shutdown_connections 2024-09-15T03:43:10.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.002+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2659256592 wait complete. 2024-09-15T03:43:10.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.003+0000 7fd1aa59d640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.003+0000 7fd1aa59d640 10 monclient: monmap: 2024-09-15T03:43:10.035 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.036 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:10.036 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:10.036 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:10.036 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.036 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.037 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:10.037 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:10.037 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:10.037 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.003+0000 7fd1aa59d640 1 Processor -- start 2024-09-15T03:43:10.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.003+0000 7fd1aa59d640 1 -- start start 2024-09-15T03:43:10.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.003+0000 7fd1aa59d640 10 monclient: init 2024-09-15T03:43:10.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:43:10.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 0x7fd1a413a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a40975c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:10.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 0x7fd1a4128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a412ea30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a412ee50 0x7fd1a41291d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a412ee50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:10.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a412f9c0 con 0x7fd1a40975c0 2024-09-15T03:43:10.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a412b200 con 0x7fd1a412ee50 2024-09-15T03:43:10.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a409ce40 con 0x7fd1a412ea30 2024-09-15T03:43:10.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1aa59d640 10 monclient(hunting): authenticate will time out at 1587.643427s 2024-09-15T03:43:10.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 0x7fd1a4128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 0x7fd1a413a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 0x7fd1a413a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44772/0 (socket says 172.21.15.153:44772) 2024-09-15T03:43:10.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 0x7fd1a4128c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55070/0 (socket says 172.21.15.153:55070) 2024-09-15T03:43:10.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 -- 172.21.15.153:0/2921800452 learned_addr learned my addr 172.21.15.153:0/2921800452 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): get_auth_request con 0x7fd1a40975c0 auth_method 0 2024-09-15T03:43:10.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): get_auth_request con 0x7fd1a412ea30 auth_method 0 2024-09-15T03:43:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a8b3a640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a412ee50 0x7fd1a41291d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a8b3a640 10 monclient(hunting): get_auth_request con 0x7fd1a412ee50 auth_method 0 2024-09-15T03:43:10.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a8b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a8b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a8b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient(hunting): handle_auth_done global_id 174216 payload 824 2024-09-15T03:43:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 -- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 msgr2=0x7fd1a4128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 0x7fd1a4128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 -- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a412ee50 msgr2=0x7fd1a41291d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a412ee50 0x7fd1a41291d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 monclient: found mon.noname-a 2024-09-15T03:43:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a3fff640 1 -- 172.21.15.153:0/2921800452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c009a20 con 0x7fd1a40975c0 2024-09-15T03:43:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.004+0000 7fd1a37fe640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 0x7fd1a4128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1a3fff640 1 -- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 msgr2=0x7fd1a413a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1a3fff640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 0x7fd1a413a1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd19400c610 tx=0x7fd194007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194012840 con 0x7fd1a40975c0 2024-09-15T03:43:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: dump: 2024-09-15T03:43:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1940129c0 con 0x7fd1a40975c0 2024-09-15T03:43:10.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 5 monclient: authenticate success, global_id 174216 2024-09-15T03:43:10.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 10 monclient: _renew_subs 2024-09-15T03:43:10.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194012d80 con 0x7fd1a40975c0 2024-09-15T03:43:10.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a412a690 con 0x7fd1a40975c0 2024-09-15T03:43:10.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd18bfff640 10 monclient: dump: 2024-09-15T03:43:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.054 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1a27fc640 4 set_mon_vals no callback set 2024-09-15T03:43:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 10 monclient: _renew_subs 2024-09-15T03:43:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.007+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a412c2b0 con 0x7fd1a40975c0 2024-09-15T03:43:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd194011590 con 0x7fd1a40975c0 2024-09-15T03:43:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd1aa59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd1aa59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd1aa59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a404bf60 con 0x7fd1a40975c0 2024-09-15T03:43:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd18bfff640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd190038740 0x7fd19003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd19404c770 con 0x7fd1a40975c0 2024-09-15T03:43:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd1a37fe640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd190038740 0x7fd19003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.008+0000 7fd1a37fe640 10 monclient: get_auth_request con 0x7fd190038740 auth_method 0 2024-09-15T03:43:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.011+0000 7fd1a37fe640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd190038740 0x7fd19003abf0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4095270 tx=0x7fd198007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.012+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd194010ac0 con 0x7fd1a40975c0 2024-09-15T03:43:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.012+0000 7fd18bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.012+0000 7fd18bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:10.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.072+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.072+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795054718}) 2024-09-15T03:43:10.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.072+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187080059}) 2024-09-15T03:43:10.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.072+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.072+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795054718}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.072+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850681690}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.073+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.073+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153715046}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.073+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2145133490}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.073+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.073+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153715046}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.073+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652131101}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.074+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.074+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532833554}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.074+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499588483}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.074+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.074+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532833554}) 2024-09-15T03:43:10.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.074+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334360665}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.075+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.075+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3470527763,mdsmap=1531151339,mgr=2572398688,mgr_command_descs=3545737492,mgr_metadata=2280977773,mgrstat=1076642982}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.075+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=145601552,mdsmap=1531151339,mgr=1088511988,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2391386471}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.075+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.075+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3470527763,mdsmap=1531151339,mgr=2572398688,mgr_command_descs=3545737492,mgr_metadata=2280977773,mgrstat=1076642982}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.075+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=468394883,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3804242477,mon_config_key=292861232}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.076+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.076+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=83} crc {mgrstat=1226747557,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=3646047294}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.076+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=1,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=83} crc {mgrstat=1226747557,mon_config_key=3725707551,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3454460932}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.076+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.076+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=83} crc {mgrstat=1226747557,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=3646047294}) 2024-09-15T03:43:10.079 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.076+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=83} crc {mgrstat=1226747557,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=4055719247}) 2024-09-15T03:43:10.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.082+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:10.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.082+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=3857030668}) 2024-09-15T03:43:10.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:10.082+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=2612194034}) 2024-09-15T03:43:10.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.233+0000 7fd1aa59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:10.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.233+0000 7fd1aa59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:10.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.233+0000 7fd1aa59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.233+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd1a412fbc0 con 0x7fd1a40975c0 2024-09-15T03:43:10.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.234+0000 7fd18bfff640 1 -- 172.21.15.153:0/2921800452 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fd194016020 con 0x7fd1a40975c0 2024-09-15T03:43:10.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.234+0000 7fd18bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:10.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.234+0000 7fd18bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:10.236 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:10.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd190038740 msgr2=0x7fd19003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd190038740 0x7fd19003abf0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4095270 tx=0x7fd198007f00 comp rx=0 tx=0).stop 2024-09-15T03:43:10.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 10 monclient: shutdown 2024-09-15T03:43:10.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 msgr2=0x7fd1a413a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 0x7fd1a413a1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd19400c610 tx=0x7fd194007ed0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 shutdown_connections 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd190038740 0x7fd19003abf0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1a412ee50 0x7fd1a41291d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1a412ea30 0x7fd1a4128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 --2- 172.21.15.153:0/2921800452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1a40975c0 0x7fd1a413a1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 >> 172.21.15.153:0/2921800452 conn(0x7fd1a4067b50 msgr2=0x7fd1a40955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 shutdown_connections 2024-09-15T03:43:10.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.236+0000 7fd1aa59d640 1 -- 172.21.15.153:0/2921800452 wait complete. 2024-09-15T03:43:10.248 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:43:10.248 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:43:10.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:43:10.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.381+0000 7f84ede17640 10 monclient: get_monmap_and_config 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.381+0000 7f84ede17640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.381+0000 7f84ede17640 10 monclient: monmap: 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:10.381869+0000 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:10.381869+0000 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.382 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:10.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:10.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:10.383 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:10.383 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:10.383 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.381+0000 7f84ede17640 1 Processor -- start 2024-09-15T03:43:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 1 -- start start 2024-09-15T03:43:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient: init 2024-09-15T03:43:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84e81318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): picked mon.noname-b con 0x7f84e809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f84e8136c20 0x7f84e8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): picked mon.noname-c con 0x7f84e8136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e8132640 con 0x7f84e81318f0 2024-09-15T03:43:10.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e8131d00 con 0x7f84e8136c20 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e809fbf0 con 0x7f84e809ea50 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84ede17640 10 monclient(hunting): authenticate will time out at 1588.021558s 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55072/0 (socket says 172.21.15.153:55072) 2024-09-15T03:43:10.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 -- 172.21.15.153:0/957042436 learned_addr learned my addr 172.21.15.153:0/957042436 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): get_auth_request con 0x7f84e809ea50 auth_method 0 2024-09-15T03:43:10.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient(hunting): handle_auth_done global_id 184136 payload 274 2024-09-15T03:43:10.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 -- 172.21.15.153:0/957042436 >> v2:172.21.15.181:3300/0 conn(0x7f84e8136c20 msgr2=0x7f84e8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 --2- 172.21.15.153:0/957042436 >> v2:172.21.15.181:3300/0 conn(0x7f84e8136c20 0x7f84e8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 -- 172.21.15.153:0/957042436 >> v1:172.21.15.181:6789/0 conn(0x7f84e81318f0 legacy=0x7f84e8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 monclient: found mon.noname-b 2024-09-15T03:43:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.382+0000 7f84dffff640 1 -- 172.21.15.153:0/957042436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e809fe30 con 0x7f84e809ea50 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84dffff640 1 -- 172.21.15.153:0/957042436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 msgr2=0x7f84e809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84dffff640 1 --2- 172.21.15.153:0/957042436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e809ee20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009910 tx=0x7f84d00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6207f53fe8ce0fa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 1 -- 172.21.15.153:0/957042436 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0032440 con 0x7f84e809ea50 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: dump: 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.406 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 1 -- 172.21.15.153:0/957042436 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d00325a0 con 0x7f84e809ea50 2024-09-15T03:43:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84ede17640 5 monclient: authenticate success, global_id 184136 2024-09-15T03:43:10.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84ede17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84ede17640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 1 -- 172.21.15.153:0/957042436 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0032940 con 0x7f84e809ea50 2024-09-15T03:43:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84ede17640 4 set_mon_vals no callback set 2024-09-15T03:43:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.385+0000 7f84e6ffd640 10 monclient: dump: 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 10 monclient: shutdown 2024-09-15T03:43:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 -- 172.21.15.153:0/957042436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 msgr2=0x7f84e809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 --2- 172.21.15.153:0/957042436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e809ee20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009910 tx=0x7f84d00091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 -- 172.21.15.153:0/957042436 shutdown_connections 2024-09-15T03:43:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 --2- 172.21.15.153:0/957042436 >> v2:172.21.15.181:3300/0 conn(0x7f84e8136c20 0x7f84e8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 --2- 172.21.15.153:0/957042436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e809ee20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 -- 172.21.15.153:0/957042436 >> 172.21.15.153:0/957042436 conn(0x7f84e8067110 msgr2=0x7f84e8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 -- 172.21.15.153:0/957042436 shutdown_connections 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 1 -- 172.21.15.153:0/957042436 wait complete. 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.386+0000 7f84ede17640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient: monmap: 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.413 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:10.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 Processor -- start 2024-09-15T03:43:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 -- start start 2024-09-15T03:43:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient: init 2024-09-15T03:43:10.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e8097530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): picked mon.noname-c con 0x7f84e809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84e81318f0 0x7f84e8097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): picked mon.noname-b con 0x7f84e81318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:10.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8136c20 0x7f84e8097fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): picked mon.noname-a con 0x7f84e8136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e809bf10 con 0x7f84e8136c20 2024-09-15T03:43:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e809daa0 con 0x7f84e81318f0 2024-09-15T03:43:10.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e809ac30 con 0x7f84e809ea50 2024-09-15T03:43:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84ede17640 10 monclient(hunting): authenticate will time out at 1588.026793s 2024-09-15T03:43:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e8097530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e8097530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55086/0 (socket says 172.21.15.153:55086) 2024-09-15T03:43:10.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 1 -- 172.21.15.153:0/923574435 learned_addr learned my addr 172.21.15.153:0/923574435 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): get_auth_request con 0x7f84e809ea50 auth_method 0 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e7fff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8136c20 0x7f84e8097fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e7fff640 10 monclient(hunting): get_auth_request con 0x7f84e8136c20 auth_method 0 2024-09-15T03:43:10.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.387+0000 7f84e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84dffff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84e81318f0 0x7f84e8097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84dffff640 10 monclient(hunting): get_auth_request con 0x7f84e81318f0 auth_method 0 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 10 monclient(hunting): handle_auth_done global_id 184139 payload 824 2024-09-15T03:43:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84e81318f0 msgr2=0x7f84e8097a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84e81318f0 0x7f84e8097a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8136c20 msgr2=0x7f84e8097fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8136c20 0x7f84e8097fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 1 monclient: found mon.noname-c 2024-09-15T03:43:10.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84e77fe640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0009450 con 0x7f84e809ea50 2024-09-15T03:43:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.388+0000 7f84dffff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84e81318f0 0x7f84e8097a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:43:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84e77fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 msgr2=0x7f84e8097530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84e77fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e8097530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84d800b610 tx=0x7f84d80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800f7a0 con 0x7f84e809ea50 2024-09-15T03:43:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: dump: 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.426 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d800f920 con 0x7f84e809ea50 2024-09-15T03:43:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 5 monclient: authenticate success, global_id 184139 2024-09-15T03:43:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 10 monclient: _renew_subs 2024-09-15T03:43:10.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e8098e00 con 0x7f84e809ea50 2024-09-15T03:43:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800fce0 con 0x7f84e809ea50 2024-09-15T03:43:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84df7fe640 10 monclient: dump: 2024-09-15T03:43:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.430 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 10 monclient: _renew_subs 2024-09-15T03:43:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84e67fc640 4 set_mon_vals no callback set 2024-09-15T03:43:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.390+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e80984f0 con 0x7f84e809ea50 2024-09-15T03:43:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f84d8010630 con 0x7f84e809ea50 2024-09-15T03:43:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84ede17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84ede17640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 0x7f84e802aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8065e00 con 0x7f84e809b580 2024-09-15T03:43:10.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84df7fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84c8038770 0x7f84c803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f84d804c250 con 0x7f84e809ea50 2024-09-15T03:43:10.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 0x7f84e802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient: get_auth_request con 0x7f84e809b580 auth_method 0 2024-09-15T03:43:10.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84e7fff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84c8038770 0x7f84c803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84e7fff640 10 monclient: get_auth_request con 0x7f84c8038770 auth_method 0 2024-09-15T03:43:10.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 10 monclient(hunting): handle_auth_done global_id 174222 payload 824 2024-09-15T03:43:10.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.391+0000 7f84dffff640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84e809aa30 con 0x7f84e809b580 2024-09-15T03:43:10.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84dffff640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 msgr2=0x7f84e802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84dffff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 0x7f84e802aa40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84d0034000 tx=0x7f84d0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d003d980 con 0x7f84e809b580 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 10 monclient: dump: 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.436 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.437 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84e7fff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84c8038770 0x7f84c803ac20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f84e80a0240 tx=0x7f84e0009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f84d003dae0 con 0x7f84e809b580 2024-09-15T03:43:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 msgr2=0x7f84e802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 0x7f84e802aa40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84d0034000 tx=0x7f84d0032000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 msgr2=0x7f84e802aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.394+0000 7f84df7fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e809b580 0x7f84e802aa40 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84d0034000 tx=0x7f84d0032000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84ede17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84ede17640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 0x7f84e805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84ede17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e809ada0 con 0x7f84e805e310 2024-09-15T03:43:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 0x7f84e805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient: get_auth_request con 0x7f84e805e310 auth_method 0 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 10 monclient(hunting): handle_auth_done global_id 174225 payload 824 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.400+0000 7f84dffff640 1 -- 172.21.15.153:0/923574435 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f84e809b060 con 0x7f84e805e310 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.402+0000 7f84dffff640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 msgr2=0x7f84e805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.402+0000 7f84dffff640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 0x7f84e805e710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009210 tx=0x7f84d003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.402+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d003cab0 con 0x7f84e805e310 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.402+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.402+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.402+0000 7f84df7fe640 10 monclient: dump: 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f84d003cc10 con 0x7f84e805e310 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 msgr2=0x7f84e805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 0x7f84e805e710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009210 tx=0x7f84d003d000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 msgr2=0x7f84e805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84df7fe640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e805e310 0x7f84e805e710 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009210 tx=0x7f84d003d000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.445 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 146, 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.458 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:10.459 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:10.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84c8038770 msgr2=0x7f84c803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84c8038770 0x7f84c803ac20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f84e80a0240 tx=0x7f84e0009780 comp rx=0 tx=0).stop 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 10 monclient: shutdown 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 msgr2=0x7f84e8097530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 --2- 172.21.15.153:0/923574435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84e809ea50 0x7f84e8097530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84d800b610 tx=0x7f84d80090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84e7fff640 1 -- 172.21.15.153:0/923574435 reap_dead start 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84e7fff640 1 -- 172.21.15.153:0/923574435 reap_dead start 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 shutdown_connections 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 >> 172.21.15.153:0/923574435 conn(0x7f84e8067110 msgr2=0x7f84e80a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 shutdown_connections 2024-09-15T03:43:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.403+0000 7f84ede17640 1 -- 172.21.15.153:0/923574435 wait complete. 2024-09-15T03:43:10.462 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:10.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:43:10.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:43:10.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.556+0000 7f09c5a80640 10 monclient: get_monmap_and_config 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.556+0000 7f09c5a80640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.556+0000 7f09c5a80640 10 monclient: monmap: 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:10.556825+0000 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:10.556825+0000 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:10.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:10.558 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:10.558 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.556+0000 7f09c5a80640 1 Processor -- start 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.556+0000 7f09c5a80640 1 -- start start 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.556+0000 7f09c5a80640 10 monclient: init 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09c0131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:10.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 0x7f09c009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): picked mon.noname-b con 0x7f09c009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f09c009f220 0x7f09c0138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): picked mon.noname-c con 0x7f09c009f220 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c009faf0 con 0x7f09c0131ab0 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c0132590 con 0x7f09c009f220 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c009fd30 con 0x7f09c009e910 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09c5a80640 10 monclient(hunting): authenticate will time out at 1588.196381s 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 0x7f09c009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 0x7f09c009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55096/0 (socket says 172.21.15.153:55096) 2024-09-15T03:43:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 -- 172.21.15.153:0/3713714018 learned_addr learned my addr 172.21.15.153:0/3713714018 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request con 0x7f09c009e910 auth_method 0 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_done global_id 184142 payload 274 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 -- 172.21.15.153:0/3713714018 >> v2:172.21.15.181:3300/0 conn(0x7f09c009f220 msgr2=0x7f09c0138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09bffff640 1 --2- 172.21.15.153:0/3713714018 >> v2:172.21.15.181:3300/0 conn(0x7f09c009f220 0x7f09c0138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 --2- 172.21.15.153:0/3713714018 >> v2:172.21.15.181:3300/0 conn(0x7f09c009f220 0x7f09c0138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 -- 172.21.15.153:0/3713714018 >> v1:172.21.15.181:6789/0 conn(0x7f09c0131ab0 legacy=0x7f09c0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 monclient: found mon.noname-b 2024-09-15T03:43:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09b6ffd640 1 -- 172.21.15.153:0/3713714018 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00a1110 con 0x7f09c009e910 2024-09-15T03:43:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09bffff640 10 monclient: get_auth_request con 0x7f09c009f220 auth_method 0 2024-09-15T03:43:10.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.557+0000 7f09bffff640 1 --2- 172.21.15.153:0/3713714018 >> v2:172.21.15.181:3300/0 conn(0x7f09c009f220 0x7f09c0138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09b6ffd640 1 -- 172.21.15.153:0/3713714018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 msgr2=0x7f09c009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09b6ffd640 1 --2- 172.21.15.153:0/3713714018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 0x7f09c009ece0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009910 tx=0x7f09a80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba07d8cfca4c5ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 1 -- 172.21.15.153:0/3713714018 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802ae00 con 0x7f09c009e910 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: dump: 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 1 -- 172.21.15.153:0/3713714018 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a802f4b0 con 0x7f09c009e910 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09c5a80640 5 monclient: authenticate success, global_id 184142 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09c5a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 1 -- 172.21.15.153:0/3713714018 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802f850 con 0x7f09c009e910 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09c5a80640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09c5a80640 4 set_mon_vals no callback set 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.560+0000 7f09beffd640 10 monclient: dump: 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.571 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 10 monclient: shutdown 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 -- 172.21.15.153:0/3713714018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 msgr2=0x7f09c009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 --2- 172.21.15.153:0/3713714018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 0x7f09c009ece0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009910 tx=0x7f09a80091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 -- 172.21.15.153:0/3713714018 shutdown_connections 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 --2- 172.21.15.153:0/3713714018 >> v2:172.21.15.181:3300/0 conn(0x7f09c009f220 0x7f09c0138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 --2- 172.21.15.153:0/3713714018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009e910 0x7f09c009ece0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 -- 172.21.15.153:0/3713714018 >> 172.21.15.153:0/3713714018 conn(0x7f09c0067d30 msgr2=0x7f09c00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 -- 172.21.15.153:0/3713714018 shutdown_connections 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 1 -- 172.21.15.153:0/3713714018 wait complete. 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.561+0000 7f09c5a80640 10 monclient: monmap: 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 Processor -- start 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 -- start start 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient: init 2024-09-15T03:43:10.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c009e910 0x7f09c01381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): picked mon.noname-a con 0x7f09c009e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 0x7f09c0133f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): picked mon.noname-c con 0x7f09c009f220 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c0131ab0 0x7f09c01344d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): picked mon.noname-b con 0x7f09c0131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0138730 con 0x7f09c009e910 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0135f80 con 0x7f09c0131ab0 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0137810 con 0x7f09c009f220 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.562+0000 7f09c5a80640 10 monclient(hunting): authenticate will time out at 1588.201952s 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 0x7f09c0133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c009e910 0x7f09c01381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 0x7f09c0133f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55104/0 (socket says 172.21.15.153:55104) 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 -- 172.21.15.153:0/1668641175 learned_addr learned my addr 172.21.15.153:0/1668641175 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request con 0x7f09c009f220 auth_method 0 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request con 0x7f09c009e910 auth_method 0 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bffff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c0131ab0 0x7f09c01344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient(hunting): handle_auth_done global_id 184145 payload 824 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c0131ab0 msgr2=0x7f09c01344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c0131ab0 0x7f09c01344d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c009e910 msgr2=0x7f09c01381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c009e910 0x7f09c01381f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 monclient: found mon.noname-c 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09b6ffd640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009450 con 0x7f09c009f220 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bffff640 10 monclient: get_auth_request con 0x7f09c0131ab0 auth_method 0 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bffff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c0131ab0 0x7f09c01344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.563+0000 7f09bf7fe640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c009e910 0x7f09c01381f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b6ffd640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 msgr2=0x7f09c0133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b6ffd640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 0x7f09c0133f90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09a8035040 tx=0x7f09a8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a80329b0 con 0x7f09c009f220 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:10.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: dump: 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8032b30 con 0x7f09c009f220 2024-09-15T03:43:10.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 5 monclient: authenticate success, global_id 184145 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 10 monclient: _renew_subs 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a803d440 con 0x7f09c009f220 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c009bde0 con 0x7f09c009f220 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09b7fff640 10 monclient: dump: 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09be7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 10 monclient: _renew_subs 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.565+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c0137530 con 0x7f09c009f220 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f09a8039040 con 0x7f09c009f220 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09c5a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09c5a80640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 0x7f09c0134a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0063740 con 0x7f09c002aa40 2024-09-15T03:43:10.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 0x7f09c0134a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient: get_auth_request con 0x7f09c002aa40 auth_method 0 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09b7fff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09a0038740 0x7f09a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f09a806c140 con 0x7f09c009f220 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.566+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_done global_id 184148 payload 824 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.567+0000 7f09bf7fe640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09c002ae10 con 0x7f09c002aa40 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.567+0000 7f09bffff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09a0038740 0x7f09a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.567+0000 7f09bffff640 10 monclient: get_auth_request con 0x7f09a0038740 auth_method 0 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09bf7fe640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 msgr2=0x7f09c0134a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09bf7fe640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 0x7f09c0134a10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f09b000ea80 tx=0x7f09b0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b000cdf0 con 0x7f09c002aa40 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 10 monclient: dump: 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f09b00123f0 con 0x7f09c002aa40 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 msgr2=0x7f09c0134a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 0x7f09c0134a10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f09b000ea80 tx=0x7f09b0009060 comp rx=0 tx=0).stop 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 msgr2=0x7f09c0134a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09b7fff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c002aa40 0x7f09c0134a10 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f09b000ea80 tx=0x7f09b0009060 comp rx=0 tx=0).stop 2024-09-15T03:43:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.569+0000 7f09bffff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09a0038740 0x7f09a003abf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f09b800d640 tx=0x7f09b8007650 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09c5a80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09c5a80640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 0x7f09c005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09c5a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c004bf60 con 0x7f09c005e310 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 0x7f09c005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient: get_auth_request con 0x7f09c005e310 auth_method 0 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_done global_id 184151 payload 824 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.575+0000 7f09bf7fe640 1 -- 172.21.15.153:0/1668641175 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09c009a140 con 0x7f09c005e310 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.577+0000 7f09bf7fe640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 msgr2=0x7f09c005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.577+0000 7f09bf7fe640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 0x7f09c005e710 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f09b0017000 tx=0x7f09b0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0005dc0 con 0x7f09c005e310 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 10 monclient: dump: 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.587 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f09b001c830 con 0x7f09c005e310 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 msgr2=0x7f09c005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 0x7f09c005e710 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f09b0017000 tx=0x7f09b0015000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 msgr2=0x7f09c005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09b7fff640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c005e310 0x7f09c005e710 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.588 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 146, 2024-09-15T03:43:10.589 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:10.590 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.591 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.592 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:10.593 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:10.594 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.595 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:10.596 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.597 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.597 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.597 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:10.598 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.598 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:10.599 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:10.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:10.602 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.602 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.603 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.603 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.604 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:10.606 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:10.607 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:10.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09a0038740 msgr2=0x7f09a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09a0038740 0x7f09a003abf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f09b800d640 tx=0x7f09b8007650 comp rx=0 tx=0).stop 2024-09-15T03:43:10.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 10 monclient: shutdown 2024-09-15T03:43:10.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 msgr2=0x7f09c0133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 --2- 172.21.15.153:0/1668641175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09c009f220 0x7f09c0133f90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09a8035040 tx=0x7f09a8032000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09bffff640 1 -- 172.21.15.153:0/1668641175 reap_dead start 2024-09-15T03:43:10.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09bffff640 1 -- 172.21.15.153:0/1668641175 reap_dead start 2024-09-15T03:43:10.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 shutdown_connections 2024-09-15T03:43:10.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 >> 172.21.15.153:0/1668641175 conn(0x7f09c0067d30 msgr2=0x7f09c009e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 shutdown_connections 2024-09-15T03:43:10.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.578+0000 7f09c5a80640 1 -- 172.21.15.153:0/1668641175 wait complete. 2024-09-15T03:43:10.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:10.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:43:10.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:43:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.703+0000 7f7362354640 10 monclient: get_monmap_and_config 2024-09-15T03:43:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.703+0000 7f7362354640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.703+0000 7f7362354640 10 monclient: monmap: 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:10.704412+0000 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:10.704412+0000 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:10.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:10.706 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:10.706 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:10.706 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:10.706 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 1 Processor -- start 2024-09-15T03:43:10.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 1 -- start start 2024-09-15T03:43:10.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient: init 2024-09-15T03:43:10.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f735c096c60 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient(hunting): picked mon.noname-b con 0x7f735c097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f735c133b40 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c09d2e0 con 0x7f735c133b40 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c09d6b0 con 0x7f735c096c60 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0992f0 con 0x7f735c097030 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.704+0000 7f7362354640 10 monclient(hunting): authenticate will time out at 1588.343945s 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f73608f1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f735c133b40 0x7f735c135f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44774/0 (socket says 172.21.15.153:44774) 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f73608f1640 1 -- 172.21.15.153:0/1638047574 learned_addr learned my addr 172.21.15.153:0/1638047574 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): get_auth_request con 0x7f735c097030 auth_method 0 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient(hunting): handle_auth_done global_id 184154 payload 274 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 1 -- 172.21.15.153:0/1638047574 >> v1:172.21.15.181:6789/0 conn(0x7f735c096c60 legacy=0x7f735c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 1 -- 172.21.15.153:0/1638047574 >> v1:172.21.15.153:6789/0 conn(0x7f735c133b40 legacy=0x7f735c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:43:10.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.705+0000 7f735b7fe640 1 -- 172.21.15.153:0/1638047574 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0995d0 con 0x7f735c097030 2024-09-15T03:43:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735b7fe640 1 -- 172.21.15.153:0/1638047574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 msgr2=0x7f735c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735b7fe640 1 --2- 172.21.15.153:0/1638047574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c097430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7350009880 tx=0x7f73500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b61e1a3e0c272bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 1 -- 172.21.15.153:0/1638047574 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350032440 con 0x7f735c097030 2024-09-15T03:43:10.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:10.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: dump: 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.732 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 1 -- 172.21.15.153:0/1638047574 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73500325a0 con 0x7f735c097030 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 5 monclient: authenticate success, global_id 184154 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 1 -- 172.21.15.153:0/1638047574 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350032940 con 0x7f735c097030 2024-09-15T03:43:10.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 4 set_mon_vals no callback set 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f735affd640 10 monclient: dump: 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.734 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 10 monclient: shutdown 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 -- 172.21.15.153:0/1638047574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 msgr2=0x7f735c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 --2- 172.21.15.153:0/1638047574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c097430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7350009880 tx=0x7f73500090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 -- 172.21.15.153:0/1638047574 shutdown_connections 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 --2- 172.21.15.153:0/1638047574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c097430 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 -- 172.21.15.153:0/1638047574 >> 172.21.15.153:0/1638047574 conn(0x7f735c0672b0 msgr2=0x7f735c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 -- 172.21.15.153:0/1638047574 shutdown_connections 2024-09-15T03:43:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.708+0000 7f7362354640 1 -- 172.21.15.153:0/1638047574 wait complete. 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.709+0000 7f7362354640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.709+0000 7f7362354640 10 monclient: monmap: 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:10.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.709+0000 7f7362354640 1 Processor -- start 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.709+0000 7f7362354640 1 -- start start 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.709+0000 7f7362354640 10 monclient: init 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c096c60 0x7f735c141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): picked mon.noname-b con 0x7f735c096c60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): picked mon.noname-c con 0x7f735c097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c133b40 0x7f735c06c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): picked mon.noname-a con 0x7f735c133b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c12ed90 con 0x7f735c133b40 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c09c3b0 con 0x7f735c096c60 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06e960 con 0x7f735c097030 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f7362354640 10 monclient(hunting): authenticate will time out at 1588.349231s 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55140/0 (socket says 172.21.15.153:55140) 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 -- 172.21.15.153:0/1409278772 learned_addr learned my addr 172.21.15.153:0/1409278772 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): get_auth_request con 0x7f735c097030 auth_method 0 2024-09-15T03:43:10.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c133b40 0x7f735c06c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): get_auth_request con 0x7f735c133b40 auth_method 0 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c096c60 0x7f735c141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f73608f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): get_auth_request con 0x7f735c096c60 auth_method 0 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient(hunting): handle_auth_done global_id 184157 payload 824 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c096c60 msgr2=0x7f735c141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c096c60 0x7f735c141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c133b40 msgr2=0x7f735c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c133b40 0x7f735c06c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.710+0000 7f735b7fe640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7350009390 con 0x7f735c097030 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.712+0000 7f735b7fe640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 msgr2=0x7f735c141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.712+0000 7f735b7fe640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c141e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f73500097d0 tx=0x7f7350032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.712+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350031990 con 0x7f735c097030 2024-09-15T03:43:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: dump: 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.742 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350031af0 con 0x7f735c097030 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 5 monclient: authenticate success, global_id 184157 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 10 monclient: _renew_subs 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c06e680 con 0x7f735c097030 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350044410 con 0x7f735c097030 2024-09-15T03:43:10.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 10 monclient: dump: 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.744 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 10 monclient: _renew_subs 2024-09-15T03:43:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f735a7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c06d790 con 0x7f735c097030 2024-09-15T03:43:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7350039040 con 0x7f735c097030 2024-09-15T03:43:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 0x7f735c06ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.713+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06ede0 con 0x7f735c1423d0 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f7343fff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7348038770 0x7f734803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f735006c920 con 0x7f735c097030 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 0x7f735c06ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient: get_auth_request con 0x7f735c1423d0 auth_method 0 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f73608f1640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7348038770 0x7f734803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f73608f1640 10 monclient: get_auth_request con 0x7f7348038770 auth_method 0 2024-09-15T03:43:10.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 10 monclient(hunting): handle_auth_done global_id 144430 payload 824 2024-09-15T03:43:10.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.714+0000 7f735bfff640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f735c06f340 con 0x7f735c1423d0 2024-09-15T03:43:10.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f73608f1640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7348038770 0x7f734803ac20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7354006d70 tx=0x7f73540052b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f735bfff640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 msgr2=0x7f735c06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f735bfff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 0x7f735c06ce40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f734c00f780 tx=0x7f734c007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c00d8b0 con 0x7f735c1423d0 2024-09-15T03:43:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f7343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f7343fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.716+0000 7f7343fff640 10 monclient: dump: 2024-09-15T03:43:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.750 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f734c00da10 con 0x7f735c1423d0 2024-09-15T03:43:10.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:10.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 msgr2=0x7f735c06ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 0x7f735c06ce40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f734c00f780 tx=0x7f734c007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:10.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 msgr2=0x7f735c06ce40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:10.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.717+0000 7f7343fff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c1423d0 0x7f735c06ce40 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f734c00f780 tx=0x7f734c007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:10.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.722+0000 7f7362354640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:10.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.722+0000 7f7362354640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 0x7f735c029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.722+0000 7f7362354640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.722+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06d380 con 0x7f735c0298c0 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 0x7f735c029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient: get_auth_request con 0x7f735c0298c0 auth_method 0 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 10 monclient(hunting): handle_auth_done global_id 144433 payload 824 2024-09-15T03:43:10.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.723+0000 7f735bfff640 1 -- 172.21.15.153:0/1409278772 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f735c06eaa0 con 0x7f735c0298c0 2024-09-15T03:43:10.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.725+0000 7f735bfff640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 msgr2=0x7f735c029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.725+0000 7f735bfff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 0x7f735c029cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f734c003420 tx=0x7f734c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c007240 con 0x7f735c0298c0 2024-09-15T03:43:10.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:10.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 10 monclient: dump: 2024-09-15T03:43:10.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.756 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.756 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.756 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.756 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.756 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.756 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.757 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.757 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f734c0073a0 con 0x7f735c0298c0 2024-09-15T03:43:10.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:10.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:10.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 msgr2=0x7f735c029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 0x7f735c029cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f734c003420 tx=0x7f734c019000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 msgr2=0x7f735c029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:10.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7343fff640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f735c0298c0 0x7f735c029cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.759 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:10.759 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:10.759 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:10.759 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:10.759 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 146, 2024-09-15T03:43:10.760 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:10.760 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:10.760 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:10.760 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:10.760 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.760 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:43:10.761 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:10.761 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:10.761 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:10.761 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.761 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.762 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.762 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.762 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.762 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.762 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.763 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:10.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:10.763 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.763 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.764 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.764 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.764 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.764 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.764 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.764 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.765 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.765 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.765 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:10.765 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:10.766 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:10.766 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:10.766 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:10.766 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:10.766 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:10.766 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:10.767 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:10.767 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:10.767 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:10.767 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:10.767 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:10.768 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:10.768 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:10.768 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:10.768 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:10.768 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:10.768 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:10.769 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:10.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:10.769 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:10.769 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:10.769 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:10.770 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:10.772 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:10.774 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:10.775 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.776 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.777 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:10.778 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7348038770 msgr2=0x7f734803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7348038770 0x7f734803ac20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7354006d70 tx=0x7f73540052b0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 10 monclient: shutdown 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 msgr2=0x7f735c141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c141e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f73500097d0 tx=0x7f7350032000 comp rx=0 tx=0).stop 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f73608f1640 1 -- 172.21.15.153:0/1409278772 reap_dead start 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 shutdown_connections 2024-09-15T03:43:10.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 --2- 172.21.15.153:0/1409278772 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c097030 0x7f735c141e90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 >> 172.21.15.153:0/1409278772 conn(0x7f735c0672b0 msgr2=0x7f735c136350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 shutdown_connections 2024-09-15T03:43:10.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.726+0000 7f7362354640 1 -- 172.21.15.153:0/1409278772 wait complete. 2024-09-15T03:43:10.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:10.781 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:43:10.781 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:43:10.781 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:43:10.781 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:43:10.782 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:43:10.782 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:10.792 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:43:10.792 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:43:10.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:10.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.924+0000 7f81c16c9640 10 monclient: get_monmap_and_config 2024-09-15T03:43:10.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.924+0000 7f81c16c9640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.924+0000 7f81c16c9640 10 monclient: monmap: 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:10.925362+0000 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:10.925362+0000 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:10.928 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 1 Processor -- start 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 1 -- start start 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient: init 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81bc131ab0 0x7f81bc131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient(hunting): picked mon.noname-a con 0x7f81bc131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81bc09e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81bc136bb0 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81bc09faf0 con 0x7f81bc136bb0 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81bc09fd30 con 0x7f81bc09e910 2024-09-15T03:43:10.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc0a1110 con 0x7f81bc131ab0 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81c16c9640 10 monclient(hunting): authenticate will time out at 1588.564705s 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81bc131ab0 0x7f81bc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f81bc131ab0 0x7f81bc131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44846/0 (socket says 172.21.15.153:44846) 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 1 -- 172.21.15.153:0/130373176 learned_addr learned my addr 172.21.15.153:0/130373176 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): get_auth_request con 0x7f81bc131ab0 auth_method 0 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81bb7fe640 1 --1- 172.21.15.153:0/130373176 >> v1:172.21.15.153:6789/0 conn(0x7f81bc136bb0 0x7f81bc138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44776/0 (socket says 172.21.15.153:44776) 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.925+0000 7f81baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 10 monclient(hunting): handle_auth_done global_id 174234 payload 274 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 1 -- 172.21.15.153:0/130373176 >> v1:172.21.15.181:6789/0 conn(0x7f81bc09e910 legacy=0x7f81bc09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 1 -- 172.21.15.153:0/130373176 >> v1:172.21.15.153:6789/0 conn(0x7f81bc136bb0 legacy=0x7f81bc138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 1 monclient: found mon.noname-a 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.926+0000 7f81baffd640 1 -- 172.21.15.153:0/130373176 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0a13f0 con 0x7f81bc131ab0 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.928+0000 7f81baffd640 1 -- 172.21.15.153:0/130373176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc131ab0 msgr2=0x7f81bc131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.928+0000 7f81baffd640 1 --2- 172.21.15.153:0/130373176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc131ab0 0x7f81bc131e80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009940 tx=0x7f81ac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a283a038551e22d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 1 -- 172.21.15.153:0/130373176 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac02ae00 con 0x7f81bc131ab0 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: dump: 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.937 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 1 -- 172.21.15.153:0/130373176 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac02f4b0 con 0x7f81bc131ab0 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 5 monclient: authenticate success, global_id 174234 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 1 -- 172.21.15.153:0/130373176 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac02f850 con 0x7f81bc131ab0 2024-09-15T03:43:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 4 set_mon_vals no callback set 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81b9ffb640 10 monclient: dump: 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.939 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 10 monclient: shutdown 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 -- 172.21.15.153:0/130373176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc131ab0 msgr2=0x7f81bc131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 --2- 172.21.15.153:0/130373176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc131ab0 0x7f81bc131e80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009940 tx=0x7f81ac0091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 -- 172.21.15.153:0/130373176 shutdown_connections 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 --2- 172.21.15.153:0/130373176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc131ab0 0x7f81bc131e80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 -- 172.21.15.153:0/130373176 >> 172.21.15.153:0/130373176 conn(0x7f81bc067d30 msgr2=0x7f81bc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 -- 172.21.15.153:0/130373176 shutdown_connections 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.929+0000 7f81c16c9640 1 -- 172.21.15.153:0/130373176 wait complete. 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.930+0000 7f81c16c9640 10 monclient: build_initial_monmap 2024-09-15T03:43:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.930+0000 7f81c16c9640 10 monclient: monmap: 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.930+0000 7f81c16c9640 1 Processor -- start 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.930+0000 7f81c16c9640 1 -- start start 2024-09-15T03:43:10.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.930+0000 7f81c16c9640 10 monclient: init 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09e910 0x7f81bc1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): picked mon.noname-c con 0x7f81bc09e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc131ab0 0x7f81bc132be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): picked mon.noname-b con 0x7f81bc131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 0x7f81bc133120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): picked mon.noname-a con 0x7f81bc136bb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81baffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09e910 msgr2=0x7f81bc1326a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc1313a0 con 0x7f81bc136bb0 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc09bde0 con 0x7f81bc131ab0 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc13a3a0 con 0x7f81bc09e910 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81baffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09e910 0x7f81bc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81c16c9640 10 monclient(hunting): authenticate will time out at 1588.570238s 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 0x7f81bc133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 0x7f81bc133120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44856/0 (socket says 172.21.15.153:44856) 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 -- 172.21.15.153:0/2412564401 learned_addr learned my addr 172.21.15.153:0/2412564401 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): get_auth_request con 0x7f81bc136bb0 auth_method 0 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81ba7fc640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc131ab0 0x7f81bc132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81ba7fc640 10 monclient(hunting): get_auth_request con 0x7f81bc131ab0 auth_method 0 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient(hunting): handle_auth_done global_id 174237 payload 824 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 -- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09e910 msgr2=0x7f81bc1326a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09e910 0x7f81bc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 -- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc131ab0 msgr2=0x7f81bc132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc131ab0 0x7f81bc132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:10.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 monclient: found mon.noname-a 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.931+0000 7f81bb7fe640 1 -- 172.21.15.153:0/2412564401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac009450 con 0x7f81bc136bb0 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81bb7fe640 1 -- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 msgr2=0x7f81bc133120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81bb7fe640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 0x7f81bc133120 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81b4007fd0 tx=0x7f81b4004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b400ce40 con 0x7f81bc136bb0 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: dump: 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b400a640 con 0x7f81bc136bb0 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81c16c9640 5 monclient: authenticate success, global_id 174237 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81c16c9640 10 monclient: _renew_subs 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81c16c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc130820 con 0x7f81bc136bb0 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b400aa00 con 0x7f81bc136bb0 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81a2ffd640 10 monclient: dump: 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:10.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 10 monclient: _renew_subs 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.933+0000 7f81b97fa640 4 set_mon_vals no callback set 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc134b90 con 0x7f81bc136bb0 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f81b40070f0 con 0x7f81bc136bb0 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.934+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc04bf60 con 0x7f81bc136bb0 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.935+0000 7f81a2ffd640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81a80386f0 0x7f81a803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.935+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f81b404c410 con 0x7f81bc136bb0 2024-09-15T03:43:10.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.935+0000 7f81baffd640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81a80386f0 0x7f81a803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:10.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.935+0000 7f81baffd640 10 monclient: get_auth_request con 0x7f81a80386f0 auth_method 0 2024-09-15T03:43:10.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.938+0000 7f81baffd640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81a80386f0 0x7f81a803aba0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009910 tx=0x7f81ac032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:10.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.939+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f81b40073a0 con 0x7f81bc136bb0 2024-09-15T03:43:10.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.939+0000 7f81a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:10.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:10.939+0000 7f81a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81c16c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81c16c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81c16c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f81bc136160 con 0x7f81bc136bb0 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81a2ffd640 1 -- 172.21.15.153:0/2412564401 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f81b4010300 con 0x7f81bc136bb0 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:11.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.150+0000 7f81a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:11.152 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81a80386f0 msgr2=0x7f81a803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81a80386f0 0x7f81a803aba0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009910 tx=0x7f81ac032000 comp rx=0 tx=0).stop 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 10 monclient: shutdown 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 msgr2=0x7f81bc133120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 0x7f81bc133120 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81b4007fd0 tx=0x7f81b4004ef0 comp rx=0 tx=0).stop 2024-09-15T03:43:11.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 shutdown_connections 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f81a80386f0 0x7f81a803aba0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81bc136bb0 0x7f81bc133120 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81bc131ab0 0x7f81bc132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 --2- 172.21.15.153:0/2412564401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81bc09e910 0x7f81bc1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 >> 172.21.15.153:0/2412564401 conn(0x7f81bc067d30 msgr2=0x7f81bc12bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 shutdown_connections 2024-09-15T03:43:11.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:11.152+0000 7f81c16c9640 1 -- 172.21.15.153:0/2412564401 wait complete. 2024-09-15T03:43:14.165 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.269+0000 7f841c59d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:14.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.269+0000 7f841c59d640 10 monclient: build_initial_monmap 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.269+0000 7f841c59d640 10 monclient: monmap: 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:14.270519+0000 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:14.270519+0000 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:14.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:14.272 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:14.272 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.270+0000 7f841c59d640 1 Processor -- start 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.270+0000 7f841c59d640 1 -- start start 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.270+0000 7f841c59d640 10 monclient: init 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f841406eb80 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:14.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f841409c6d0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f841409cac0 0x7f841413a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient(hunting): picked mon.noname-c con 0x7f841409cac0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8414068a20 con 0x7f841406eb80 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841406ef50 con 0x7f841409c6d0 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8414099ef0 con 0x7f841409cac0 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841c59d640 10 monclient(hunting): authenticate will time out at 1591.910116s 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841a339640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f841406eb80 0x7f841406f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44778/0 (socket says 172.21.15.153:44778) 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841a339640 1 -- 172.21.15.153:0/1768998684 learned_addr learned my addr 172.21.15.153:0/1768998684 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 1 --2- 172.21.15.153:0/1768998684 >> v2:172.21.15.181:3300/0 conn(0x7f841409cac0 0x7f841413a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): get_auth_request con 0x7f841409cac0 auth_method 0 2024-09-15T03:43:14.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient(hunting): handle_auth_done global_id 144439 payload 274 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 1 -- 172.21.15.153:0/1768998684 >> v1:172.21.15.181:6789/0 conn(0x7f841409c6d0 legacy=0x7f8414136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 1 -- 172.21.15.153:0/1768998684 >> v1:172.21.15.153:6789/0 conn(0x7f841406eb80 legacy=0x7f841406f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 1 monclient: found mon.noname-c 2024-09-15T03:43:14.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:14.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:43:14.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.271+0000 7f841ab3a640 1 -- 172.21.15.153:0/1768998684 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8414068c20 con 0x7f841409cac0 2024-09-15T03:43:14.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.274+0000 7f841ab3a640 1 -- 172.21.15.153:0/1768998684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409cac0 msgr2=0x7f841413a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:14.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.274+0000 7f841ab3a640 1 --2- 172.21.15.153:0/1768998684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409cac0 0x7f841413a1e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8404009e60 tx=0x7f8404009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ba94601bdc1b104 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:14.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 1 -- 172.21.15.153:0/1768998684 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404007630 con 0x7f841409cac0 2024-09-15T03:43:14.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:14.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: dump: 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: _finish_auth 0 2024-09-15T03:43:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 1 -- 172.21.15.153:0/1768998684 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840402d050 con 0x7f841409cac0 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 1 -- 172.21.15.153:0/1768998684 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040079c0 con 0x7f841409cac0 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 5 monclient: authenticate success, global_id 144439 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 4 set_mon_vals no callback set 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f8419337640 10 monclient: dump: 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:14.284 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 10 monclient: shutdown 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 -- 172.21.15.153:0/1768998684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409cac0 msgr2=0x7f841413a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 --2- 172.21.15.153:0/1768998684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409cac0 0x7f841413a1e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8404009e60 tx=0x7f8404009780 comp rx=0 tx=0).stop 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 -- 172.21.15.153:0/1768998684 shutdown_connections 2024-09-15T03:43:14.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 --2- 172.21.15.153:0/1768998684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409cac0 0x7f841413a1e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 -- 172.21.15.153:0/1768998684 >> 172.21.15.153:0/1768998684 conn(0x7f8414067cd0 msgr2=0x7f84140a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 -- 172.21.15.153:0/1768998684 shutdown_connections 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.275+0000 7f841c59d640 1 -- 172.21.15.153:0/1768998684 wait complete. 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.276+0000 7f841c59d640 10 monclient: build_initial_monmap 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.276+0000 7f841c59d640 10 monclient: monmap: 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:14.286 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.276+0000 7f841c59d640 1 Processor -- start 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.276+0000 7f841c59d640 1 -- start start 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.276+0000 7f841c59d640 10 monclient: init 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f841406eb80 0x7f841411ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): picked mon.noname-c con 0x7f841406eb80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409c6d0 0x7f841411efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): picked mon.noname-b con 0x7f841409c6d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 0x7f841411f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): picked mon.noname-a con 0x7f841409cac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841a339640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f841406eb80 msgr2=0x7f841411ea70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841409cf40 con 0x7f841409cac0 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841409c4a0 con 0x7f841409c6d0 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8414120ad0 con 0x7f841406eb80 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841a339640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f841406eb80 0x7f841411ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841c59d640 10 monclient(hunting): authenticate will time out at 1591.916239s 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 0x7f841411f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 0x7f841411f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44868/0 (socket says 172.21.15.153:44868) 2024-09-15T03:43:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 -- 172.21.15.153:0/113339556 learned_addr learned my addr 172.21.15.153:0/113339556 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): get_auth_request con 0x7f841409cac0 auth_method 0 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409c6d0 0x7f841411efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): get_auth_request con 0x7f841409c6d0 auth_method 0 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:14.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f8419b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient(hunting): handle_auth_done global_id 174240 payload 824 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 -- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f841406eb80 msgr2=0x7f841411ea70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f841406eb80 0x7f841411ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 -- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409c6d0 msgr2=0x7f841411efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409c6d0 0x7f841411efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 monclient: found mon.noname-a 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.277+0000 7f841ab3a640 1 -- 172.21.15.153:0/113339556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8404009a20 con 0x7f841409cac0 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.279+0000 7f841ab3a640 1 -- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 msgr2=0x7f841411f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.279+0000 7f841ab3a640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 0x7f841411f4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84040099f0 tx=0x7f8404037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404007e70 con 0x7f841409cac0 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: dump: 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8404040070 con 0x7f841409cac0 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 5 monclient: authenticate success, global_id 174240 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 10 monclient: _renew_subs 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8414122080 con 0x7f841409cac0 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404031d10 con 0x7f841409cac0 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f84127fc640 10 monclient: dump: 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 10 monclient: _renew_subs 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f8418b36640 4 set_mon_vals no callback set 2024-09-15T03:43:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.280+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84141205c0 con 0x7f841409cac0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8404042030 con 0x7f841409cac0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f841c59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f841c59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f841c59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841404bf60 con 0x7f841409cac0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f84127fc640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84000386f0 0x7f840003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f840406dea0 con 0x7f841409cac0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f841a339640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84000386f0 0x7f840003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:14.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.281+0000 7f841a339640 10 monclient: get_auth_request con 0x7f84000386f0 auth_method 0 2024-09-15T03:43:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.284+0000 7f841a339640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84000386f0 0x7f840003aba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f840c0047d0 tx=0x7f840c007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.285+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8404030b80 con 0x7f841409cac0 2024-09-15T03:43:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.285+0000 7f84127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.285+0000 7f84127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.499+0000 7f841c59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.499+0000 7f841c59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.499+0000 7f841c59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.499+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f841404bf60 con 0x7f841409cac0 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.500+0000 7f84127fc640 1 -- 172.21.15.153:0/113339556 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f84040303a0 con 0x7f841409cac0 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.500+0000 7f84127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.500+0000 7f84127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:14.501 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84000386f0 msgr2=0x7f840003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84000386f0 0x7f840003aba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f840c0047d0 tx=0x7f840c007b80 comp rx=0 tx=0).stop 2024-09-15T03:43:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 10 monclient: shutdown 2024-09-15T03:43:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 msgr2=0x7f841411f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 0x7f841411f4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84040099f0 tx=0x7f8404037000 comp rx=0 tx=0).stop 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 shutdown_connections 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f84000386f0 0x7f840003aba0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f841409cac0 0x7f841411f4f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f841409c6d0 0x7f841411efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 --2- 172.21.15.153:0/113339556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f841406eb80 0x7f841411ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 >> 172.21.15.153:0/113339556 conn(0x7f8414067cd0 msgr2=0x7f841406d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 shutdown_connections 2024-09-15T03:43:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:14.502+0000 7f841c59d640 1 -- 172.21.15.153:0/113339556 wait complete. 2024-09-15T03:43:17.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:17.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.619+0000 7f74be4e8640 10 monclient: get_monmap_and_config 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.619+0000 7f74be4e8640 10 monclient: build_initial_monmap 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.620+0000 7f74be4e8640 10 monclient: monmap: 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:17.620738+0000 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:17.620738+0000 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:17.621 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:17.622 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:17.622 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.620+0000 7f74be4e8640 1 Processor -- start 2024-09-15T03:43:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.620+0000 7f74be4e8640 1 -- start start 2024-09-15T03:43:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.620+0000 7f74be4e8640 10 monclient: init 2024-09-15T03:43:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74b80975c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f74b812ea30 0x7f74b8136c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): picked mon.noname-a con 0x7f74b812ea30 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f74b812ee50 0x7f74b81391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): picked mon.noname-c con 0x7f74b812ee50 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b806e9e0 con 0x7f74b80975c0 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b8063950 con 0x7f74b812ea30 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b809d1c0 con 0x7f74b812ee50 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74be4e8640 10 monclient(hunting): authenticate will time out at 1595.260365s 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f74b812ea30 0x7f74b8136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f74b812ea30 0x7f74b8136c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40418/0 (socket says 172.21.15.153:40418) 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 -- 172.21.15.153:0/3773533061 learned_addr learned my addr 172.21.15.153:0/3773533061 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): get_auth_request con 0x7f74b812ea30 auth_method 0 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74bca85640 1 --2- 172.21.15.153:0/3773533061 >> v2:172.21.15.181:3300/0 conn(0x7f74b812ee50 0x7f74b81391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74bca85640 10 monclient(hunting): get_auth_request con 0x7f74b812ee50 auth_method 0 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74bca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74bca85640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74bca85640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient(hunting): handle_auth_done global_id 174243 payload 274 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 -- 172.21.15.153:0/3773533061 >> v2:172.21.15.181:3300/0 conn(0x7f74b812ee50 msgr2=0x7f74b81391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 --2- 172.21.15.153:0/3773533061 >> v2:172.21.15.181:3300/0 conn(0x7f74b812ee50 0x7f74b81391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 -- 172.21.15.153:0/3773533061 >> v1:172.21.15.181:6789/0 conn(0x7f74b80975c0 legacy=0x7f74b8097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 monclient: found mon.noname-a 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:17.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.621+0000 7f74b77fe640 1 -- 172.21.15.153:0/3773533061 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b809d4a0 con 0x7f74b812ea30 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b77fe640 1 -- 172.21.15.153:0/3773533061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 msgr2=0x7f74b8136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b77fe640 1 --2- 172.21.15.153:0/3773533061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8136c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f74b806f410 tx=0x7f74a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f27a57b4b5e5e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 1 -- 172.21.15.153:0/3773533061 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0032440 con 0x7f74b812ea30 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: dump: 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 1 -- 172.21.15.153:0/3773533061 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a00325a0 con 0x7f74b812ea30 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74be4e8640 5 monclient: authenticate success, global_id 174243 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74be4e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74be4e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 1 -- 172.21.15.153:0/3773533061 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0032940 con 0x7f74b812ea30 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74be4e8640 4 set_mon_vals no callback set 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.624+0000 7f74b6ffd640 10 monclient: dump: 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 10 monclient: shutdown 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 -- 172.21.15.153:0/3773533061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 msgr2=0x7f74b8136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 --2- 172.21.15.153:0/3773533061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8136c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f74b806f410 tx=0x7f74a00091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 -- 172.21.15.153:0/3773533061 shutdown_connections 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 --2- 172.21.15.153:0/3773533061 >> v2:172.21.15.181:3300/0 conn(0x7f74b812ee50 0x7f74b81391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 --2- 172.21.15.153:0/3773533061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8136c80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 -- 172.21.15.153:0/3773533061 >> 172.21.15.153:0/3773533061 conn(0x7f74b8067b50 msgr2=0x7f74b80a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 -- 172.21.15.153:0/3773533061 shutdown_connections 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 1 -- 172.21.15.153:0/3773533061 wait complete. 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 10 monclient: build_initial_monmap 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.625+0000 7f74be4e8640 10 monclient: monmap: 2024-09-15T03:43:17.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 Processor -- start 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 -- start start 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient: init 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:17.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74b80975c0 0x7f74b8128e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): picked mon.noname-c con 0x7f74b80975c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8129380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): picked mon.noname-a con 0x7f74b812ea30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b812ee50 0x7f74b81298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): picked mon.noname-b con 0x7f74b812ee50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b812c770 con 0x7f74b812ea30 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b812fb00 con 0x7f74b812ee50 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b81303a0 con 0x7f74b80975c0 2024-09-15T03:43:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74b7fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74b80975c0 msgr2=0x7f74b8128e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74be4e8640 10 monclient(hunting): authenticate will time out at 1595.265956s 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.626+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74b80975c0 0x7f74b8128e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8129380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8129380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40434/0 (socket says 172.21.15.153:40434) 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 -- 172.21.15.153:0/2273876869 learned_addr learned my addr 172.21.15.153:0/2273876869 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): get_auth_request con 0x7f74b812ea30 auth_method 0 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b812ee50 0x7f74b81298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:17.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): get_auth_request con 0x7f74b812ee50 auth_method 0 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74bca85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient(hunting): handle_auth_done global_id 174246 payload 824 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 -- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74b80975c0 msgr2=0x7f74b8128e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74b80975c0 0x7f74b8128e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 -- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b812ee50 msgr2=0x7f74b81298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b812ee50 0x7f74b81298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 monclient: found mon.noname-a 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.627+0000 7f74b77fe640 1 -- 172.21.15.153:0/2273876869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0009450 con 0x7f74b812ea30 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74b77fe640 1 -- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 msgr2=0x7f74b8129380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74b77fe640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8129380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74a0035040 tx=0x7f74a0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0031a40 con 0x7f74b812ea30 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: dump: 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: _finish_auth 0 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:17.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0031be0 con 0x7f74b812ea30 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a003d440 con 0x7f74b812ea30 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74affff640 10 monclient: dump: 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 5 monclient: authenticate success, global_id 174246 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74b67fc640 4 set_mon_vals no callback set 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 10 monclient: _renew_subs 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b812f820 con 0x7f74b812ea30 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 10 monclient: _renew_subs 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.629+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b812a760 con 0x7f74b812ea30 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f74a0039040 con 0x7f74b812ea30 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74be4e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74be4e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74be4e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b812f660 con 0x7f74b812ea30 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74affff640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74980386f0 0x7f749803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.630+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f74a006c5c0 con 0x7f74b812ea30 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.631+0000 7f74b7fff640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74980386f0 0x7f749803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.631+0000 7f74b7fff640 10 monclient: get_auth_request con 0x7f74980386f0 auth_method 0 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.633+0000 7f74b7fff640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74980386f0 0x7f749803aba0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f74a8007fd0 tx=0x7f74a8007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.635+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f74a0030b20 con 0x7f74b812ea30 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.635+0000 7f74affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.635+0000 7f74affff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:17.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.847+0000 7f74be4e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:17.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.847+0000 7f74be4e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:17.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.847+0000 7f74be4e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:17.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.847+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f74b804bf60 con 0x7f74b812ea30 2024-09-15T03:43:17.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.848+0000 7f74affff640 1 -- 172.21.15.153:0/2273876869 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f74a0037080 con 0x7f74b812ea30 2024-09-15T03:43:17.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.848+0000 7f74affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:17.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.848+0000 7f74affff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:17.850 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:17.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74980386f0 msgr2=0x7f749803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:17.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74980386f0 0x7f749803aba0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f74a8007fd0 tx=0x7f74a8007a70 comp rx=0 tx=0).stop 2024-09-15T03:43:17.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 10 monclient: shutdown 2024-09-15T03:43:17.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 msgr2=0x7f74b8129380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8129380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74a0035040 tx=0x7f74a0032000 comp rx=0 tx=0).stop 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 shutdown_connections 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f74980386f0 0x7f749803aba0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b812ee50 0x7f74b81298c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74b812ea30 0x7f74b8129380 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 --2- 172.21.15.153:0/2273876869 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74b80975c0 0x7f74b8128e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 >> 172.21.15.153:0/2273876869 conn(0x7f74b8067b50 msgr2=0x7f74b80955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 shutdown_connections 2024-09-15T03:43:17.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:17.850+0000 7f74be4e8640 1 -- 172.21.15.153:0/2273876869 wait complete. 2024-09-15T03:43:19.994 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=110 2024-09-15T03:43:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 110 -ne 110 ]] 2024-09-15T03:43:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:+ export CEPH_ARGS= 2024-09-15T03:43:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:+ CEPH_ARGS= 2024-09-15T03:43:19.995 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph auth del client.foo 2024-09-15T03:43:20.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.067+0000 7feeb059d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:20.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.067+0000 7feeb059d640 10 monclient: build_initial_monmap 2024-09-15T03:43:20.069 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.067+0000 7feeb059d640 10 monclient: monmap: 2024-09-15T03:43:20.069 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:43:20.069483+0000 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:43:20.069483+0000 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:20.070 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:20.071 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 Processor -- start 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 -- start start 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient: init 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feea806e9d0 0x7feea809bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): picked mon.noname-a con 0x7feea806e9d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7feea806eda0 0x7feea812f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.073 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): picked mon.noname-c con 0x7feea806eda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806f170 0x7feea812fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): picked mon.noname-b con 0x7feea806f170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea809cd60 con 0x7feea806e9d0 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea809ca80 con 0x7feea806eda0 2024-09-15T03:43:20.074 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea809d040 con 0x7feea806f170 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.068+0000 7feeb059d640 10 monclient(hunting): authenticate will time out at 1604.550974s 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7feea806eda0 0x7feea812f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeaeb3a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806f170 msgr2=0x7feea812fd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeaeb3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806f170 0x7feea812fd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:20.075 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7feea806eda0 0x7feea812f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55394/0 (socket says 172.21.15.181:55394) 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 -- 172.21.15.181:0/955212779 learned_addr learned my addr 172.21.15.181:0/955212779 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): get_auth_request con 0x7feea806eda0 auth_method 0 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feead337640 1 -- 172.21.15.181:0/955212779 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient(hunting): handle_auth_done global_id 144451 payload 274 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeae339640 1 --2- 172.21.15.181:0/955212779 >> v2:172.21.15.153:3300/0 conn(0x7feea806e9d0 0x7feea809bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 -- 172.21.15.181:0/955212779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806f170 msgr2=0x7feea812fd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 --2- 172.21.15.181:0/955212779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806f170 0x7feea812fd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 -- 172.21.15.181:0/955212779 >> v2:172.21.15.153:3300/0 conn(0x7feea806e9d0 msgr2=0x7feea809bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 --2- 172.21.15.181:0/955212779 >> v2:172.21.15.153:3300/0 conn(0x7feea806e9d0 0x7feea809bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient: _finish_hunting 0 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 monclient: found mon.noname-c 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeadb38640 1 -- 172.21.15.181:0/955212779 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea8097e80 con 0x7feea806eda0 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeae339640 10 monclient: get_auth_request con 0x7feea806e9d0 auth_method 0 2024-09-15T03:43:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.069+0000 7feeae339640 1 --2- 172.21.15.181:0/955212779 >> v2:172.21.15.153:3300/0 conn(0x7feea806e9d0 0x7feea809bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:20.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeadb38640 1 -- 172.21.15.181:0/955212779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806eda0 msgr2=0x7feea812f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeadb38640 1 --2- 172.21.15.181:0/955212779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806eda0 0x7feea812f7f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7feea809c9b0 tx=0x7feea40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e7296e350797aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 1 -- 172.21.15.181:0/955212779 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4032440 con 0x7feea806eda0 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: dump: 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: _finish_auth 0 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 5 monclient: authenticate success, global_id 144451 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 1 -- 172.21.15.181:0/955212779 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea40325a0 con 0x7feea806eda0 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 4 set_mon_vals no callback set 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 1 -- 172.21.15.181:0/955212779 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4032940 con 0x7feea806eda0 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feead337640 10 monclient: dump: 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 10 monclient: shutdown 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 1 -- 172.21.15.181:0/955212779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806eda0 msgr2=0x7feea812f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 1 --2- 172.21.15.181:0/955212779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806eda0 0x7feea812f7f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7feea809c9b0 tx=0x7feea40091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.072+0000 7feeb059d640 1 -- 172.21.15.181:0/955212779 shutdown_connections 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 --2- 172.21.15.181:0/955212779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806f170 0x7feea812fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 --2- 172.21.15.181:0/955212779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806eda0 0x7feea812f7f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 --2- 172.21.15.181:0/955212779 >> v2:172.21.15.153:3300/0 conn(0x7feea806e9d0 0x7feea809bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 -- 172.21.15.181:0/955212779 >> 172.21.15.181:0/955212779 conn(0x7feea80a18b0 msgr2=0x7feea809d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 -- 172.21.15.181:0/955212779 shutdown_connections 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 -- 172.21.15.181:0/955212779 wait complete. 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 10 monclient: build_initial_monmap 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 10 monclient: monmap: 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:20.088 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 Processor -- start 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 1 -- start start 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.073+0000 7feeb059d640 10 monclient: init 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 0x7feea81410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): picked mon.noname-b con 0x7feea806e9d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806eda0 0x7feea812fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): picked mon.noname-c con 0x7feea806eda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feea806f170 0x7feea8130040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): picked mon.noname-a con 0x7feea806f170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea8141fc0 con 0x7feea806f170 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea806db60 con 0x7feea806e9d0 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea8099fd0 con 0x7feea806eda0 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeb059d640 10 monclient(hunting): authenticate will time out at 1604.556369s 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 0x7feea81410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 0x7feea81410a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55396/0 (socket says 172.21.15.181:55396) 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeadb38640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806eda0 msgr2=0x7feea812fb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:20.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 -- 172.21.15.181:0/3043247123 learned_addr learned my addr 172.21.15.181:0/3043247123 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): get_auth_request con 0x7feea806e9d0 auth_method 0 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeadb38640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806eda0 0x7feea812fb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeaeb3a640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feea806f170 0x7feea8130040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeaeb3a640 10 monclient(hunting): get_auth_request con 0x7feea806f170 auth_method 0 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeaeb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeaeb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:20.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeaeb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient(hunting): handle_auth_done global_id 144454 payload 824 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 -- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806eda0 msgr2=0x7feea812fb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806eda0 0x7feea812fb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 -- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feea806f170 msgr2=0x7feea8130040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feea806f170 0x7feea8130040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient: _finish_hunting 0 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 monclient: found mon.noname-b 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.074+0000 7feeae339640 1 -- 172.21.15.181:0/3043247123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4009450 con 0x7feea806e9d0 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.076+0000 7feeae339640 1 -- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 msgr2=0x7feea81410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.076+0000 7feeae339640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 0x7feea81410a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00c880 tx=0x7fee9c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:20.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c00ce10 con 0x7feea806e9d0 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: dump: 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:20.093 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 5 monclient: authenticate success, global_id 144454 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9c007a40 con 0x7feea806e9d0 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 10 monclient: _renew_subs 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea811ef20 con 0x7feea806e9d0 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c007e00 con 0x7feea806e9d0 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7fee9a7fc640 10 monclient: dump: 2024-09-15T03:43:20.094 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 10 monclient: _renew_subs 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:20.095 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea811dd50 con 0x7feea806e9d0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.077+0000 7feeacb36640 4 set_mon_vals no callback set 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fee9c011600 con 0x7feea806e9d0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7feeb059d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7feeb059d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7feeb059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea8066600 con 0x7feea806e9d0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7fee9a7fc640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee8c038720 0x7fee8c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fee9c04c730 con 0x7feea806e9d0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7feeadb38640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee8c038720 0x7fee8c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.078+0000 7feeadb38640 10 monclient: get_auth_request con 0x7fee8c038720 auth_method 0 2024-09-15T03:43:20.096 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.080+0000 7feeadb38640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee8c038720 0x7fee8c03abd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7feea806fee0 tx=0x7feea4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.083+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fee9c021ae0 con 0x7feea806e9d0 2024-09-15T03:43:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.083+0000 7fee9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:20.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.083+0000 7fee9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:20.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.298+0000 7feeb059d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T03:43:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.298+0000 7feeb059d640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T03:43:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.298+0000 7feeb059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:20.298+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7feea8067700 con 0x7feea806e9d0 2024-09-15T03:43:20.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:20.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.966+0000 7fc64b19d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.966+0000 7fc64b19d640 10 monclient: build_initial_monmap 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.966+0000 7fc64b19d640 10 monclient: monmap: 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:20.966847+0000 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:20.966847+0000 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:20.967 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:20.968 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:20.968 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:20.968 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:20.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.966+0000 7fc64b19d640 1 Processor -- start 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.966+0000 7fc64b19d640 1 -- start start 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.966+0000 7fc64b19d640 10 monclient: init 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc644131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc64409e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 0x7fc644139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient(hunting): picked mon.noname-b con 0x7fc644137bc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64409faf0 con 0x7fc644131ab0 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64409fd30 con 0x7fc64409e910 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:20.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc644132590 con 0x7fc644137bc0 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64b19d640 10 monclient(hunting): authenticate will time out at 1598.606469s 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64973a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 msgr2=0x7fc644139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc64973a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 0x7fc644139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc648f39640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc644131ab0 0x7fc644131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39732/0 (socket says 172.21.15.153:39732) 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.967+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 learned_addr learned my addr 172.21.15.153:0/1171525504 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.969+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v1:172.21.15.153:6789/0 conn(0x7fc644131ab0 legacy=0x7fc644131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:20.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:20.970+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v1:172.21.15.181:6789/0 conn(0x7fc64409e910 legacy=0x7fc64409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:21.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:21.168+0000 7fc64973a640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 msgr2=0x7fc644139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:21.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:21.168+0000 7fc64973a640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 0x7fc644139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:43:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:21.569+0000 7fc64973a640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 msgr2=0x7fc644139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:21.569+0000 7fc64973a640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 0x7fc644139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:43:22.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:22.370+0000 7fc64973a640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 msgr2=0x7fc644139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:22.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:22.370+0000 7fc64973a640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 0x7fc644139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:43:23.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:23.075+0000 7fee997fa640 10 monclient: tick 2024-09-15T03:43:23.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:23.075+0000 7fee997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:23.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:23.075+0000 7fee997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:23.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): tick 2024-09-15T03:43:23.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 msgr2=0x7fc644139fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc644137bc0 0x7fc644139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 >> v1:172.21.15.181:6789/0 conn(0x7fc64409e910 legacy=0x7fc64409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 >> v1:172.21.15.153:6789/0 conn(0x7fc644131ab0 legacy=0x7fc644131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient: _add_conns ranks=[3,4] 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc6300037d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc630006290 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc644132590 con 0x7fc6300037d0 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc630008e00 con 0x7fc630006290 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc6427fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc648f39640 10 monclient(hunting): get_auth_request con 0x7fc6300037d0 auth_method 0 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc648f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc648f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc648f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.967+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:23.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 10 monclient(hunting): get_auth_request con 0x7fc630006290 auth_method 0 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:23.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:23.968+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:24.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:24.075+0000 7fee997fa640 10 monclient: tick 2024-09-15T03:43:24.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:24.075+0000 7fee997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:24.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 10 monclient(hunting): get_auth_request con 0x7fc6300037d0 auth_method 0 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 10 monclient(hunting): get_auth_request con 0x7fc630006290 auth_method 0 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:24.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:43:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:24.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.169+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 10 monclient(hunting): get_auth_request con 0x7fc630006290 auth_method 0 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 10 monclient(hunting): get_auth_request con 0x7fc6300037d0 auth_method 0 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:24.570+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:43:25.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.074+0000 7fee997fa640 10 monclient: tick 2024-09-15T03:43:25.077 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.074+0000 7fee997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 10 monclient(hunting): get_auth_request con 0x7fc6300037d0 auth_method 0 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 10 monclient(hunting): get_auth_request con 0x7fc630006290 auth_method 0 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:25.371+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:43:25.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.663+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee9c021300 con 0x7feea806e9d0 2024-09-15T03:43:25.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.663+0000 7fee9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T03:43:25.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.663+0000 7fee9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:25.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.818+0000 7feeb059d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T03:43:25.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.818+0000 7feeb059d640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T03:43:25.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.818+0000 7feeb059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:25.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.818+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7feea8141640 con 0x7feea806e9d0 2024-09-15T03:43:25.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.819+0000 7fee9a7fc640 1 -- 172.21.15.181:0/3043247123 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee9c0214e0 con 0x7feea806e9d0 2024-09-15T03:43:25.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.819+0000 7fee9a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-15T03:43:25.821 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.819+0000 7fee9a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee8c038720 msgr2=0x7fee8c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee8c038720 0x7fee8c03abd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7feea806fee0 tx=0x7feea4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 10 monclient: shutdown 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 msgr2=0x7feea81410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 0x7feea81410a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00c880 tx=0x7fee9c008040 comp rx=0 tx=0).stop 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 shutdown_connections 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fee8c038720 0x7fee8c03abd0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feea806f170 0x7feea8130040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feea806eda0 0x7feea812fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.821+0000 7feeb059d640 1 --2- 172.21.15.181:0/3043247123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feea806e9d0 0x7feea81410a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:25.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.822+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 >> 172.21.15.181:0/3043247123 conn(0x7feea80a18b0 msgr2=0x7feea809fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:25.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.822+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 shutdown_connections 2024-09-15T03:43:25.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:43:25.822+0000 7feeb059d640 1 -- 172.21.15.181:0/3043247123 wait complete. 2024-09-15T03:43:25.832 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-15T03:43:25.832 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-15T03:43:25.832 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:43:25.833 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T03:43:25.833 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T03:43:25.833 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-15T03:43:25.904 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=0 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T03:43:25.905 INFO:tasks.workunit.client.0.smithi181.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T03:43:25.906 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=0 2024-09-15T03:43:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T03:43:26.377 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T03:43:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T03:43:26.830 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=0 2024-09-15T03:43:26.830 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 0 -ne 0 ]] 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-15T03:43:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-15T03:43:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): tick 2024-09-15T03:43:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:43:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 msgr2=0x7fc630008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc630006290 0x7fc630008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 msgr2=0x7fc630005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 1 -- 172.21.15.153:0/1171525504 reap_dead start 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc6300037d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc6300091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc6300066b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc6300096f0 0x7fc63000bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc6300096f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc644132590 con 0x7fc6300037d0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc630008e00 con 0x7fc6300096f0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 1 -- 172.21.15.153:0/1171525504 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63000c160 con 0x7fc6300066b0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc6427fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc643fff640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 msgr2=0x7fc6300091b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc643fff640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc6300091b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.153:3300/0 conn(0x7fc6300037d0 0x7fc630005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): get_auth_request con 0x7fc6300037d0 auth_method 0 2024-09-15T03:43:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc6300096f0 0x7fc63000bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): get_auth_request con 0x7fc6300096f0 auth_method 0 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc64973a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient(hunting): handle_auth_done global_id 174267 payload 274 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 msgr2=0x7fc6300091b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc6300091b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc6300096f0 msgr2=0x7fc63000bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc6300096f0 0x7fc63000bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient: _finish_hunting 0 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 monclient: found mon.noname-a 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:26.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.968+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6440a1110 con 0x7fc6300037d0 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc648f39640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 msgr2=0x7fc630005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc648f39640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc630005be0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc6300035d0 tx=0x7fc634009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1669d4aaa2949c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 1 -- 172.21.15.153:0/1171525504 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63402b760 con 0x7fc6300037d0 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: dump: 2024-09-15T03:43:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 1 -- 172.21.15.153:0/1171525504 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63402f070 con 0x7fc6300037d0 2024-09-15T03:43:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc64b19d640 5 monclient: authenticate success, global_id 174267 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc64b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc64b19d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 1 -- 172.21.15.153:0/1171525504 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63402bb50 con 0x7fc6300037d0 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc64b19d640 4 set_mon_vals no callback set 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.971+0000 7fc6437fe640 10 monclient: dump: 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.982 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 10 monclient: shutdown 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 -- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 msgr2=0x7fc630005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc630005be0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc6300035d0 tx=0x7fc634009040 comp rx=0 tx=0).stop 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 -- 172.21.15.153:0/1171525504 shutdown_connections 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 --2- 172.21.15.153:0/1171525504 >> v2:172.21.15.181:3300/0 conn(0x7fc6300096f0 0x7fc63000bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc6300091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 --2- 172.21.15.153:0/1171525504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc630005be0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 -- 172.21.15.153:0/1171525504 >> 172.21.15.153:0/1171525504 conn(0x7fc644067d30 msgr2=0x7fc6440a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 -- 172.21.15.153:0/1171525504 shutdown_connections 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.972+0000 7fc64b19d640 1 -- 172.21.15.153:0/1171525504 wait complete. 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.973+0000 7fc64b19d640 10 monclient: build_initial_monmap 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.973+0000 7fc64b19d640 10 monclient: monmap: 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:26.984 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.973+0000 7fc64b19d640 1 Processor -- start 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 -- start start 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient: init 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc644130ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): picked mon.noname-a con 0x7fc6300037d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc644141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): picked mon.noname-c con 0x7fc6300066b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 0x7fc644141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): picked mon.noname-b con 0x7fc6300096f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6441311e0 con 0x7fc6300037d0 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64409a9e0 con 0x7fc6300096f0 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64409df20 con 0x7fc6300066b0 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc64b19d640 10 monclient(hunting): authenticate will time out at 1604.613876s 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc643fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 msgr2=0x7fc644141330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc643fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc644141330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.974+0000 7fc648f39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc644130ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc644130ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53180/0 (socket says 172.21.15.153:53180) 2024-09-15T03:43:26.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 1 -- 172.21.15.153:0/3003058408 learned_addr learned my addr 172.21.15.153:0/3003058408 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): get_auth_request con 0x7fc6300037d0 auth_method 0 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 0x7fc644141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): get_auth_request con 0x7fc6300096f0 auth_method 0 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient(hunting): handle_auth_done global_id 144478 payload 824 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 -- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 msgr2=0x7fc644141330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc644141330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 -- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 msgr2=0x7fc644130ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc644130ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 monclient: found mon.noname-b 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc64973a640 1 -- 172.21.15.153:0/3003058408 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63400f040 con 0x7fc6300096f0 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.975+0000 7fc648f39640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc644130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.977+0000 7fc64973a640 1 -- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 msgr2=0x7fc644141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.977+0000 7fc64973a640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 0x7fc644141870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc644066000 tx=0x7fc63c007750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63c0116a0 con 0x7fc6300096f0 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: dump: 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: _finish_auth 0 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63c004e90 con 0x7fc6300096f0 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 5 monclient: authenticate success, global_id 144478 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63c011a50 con 0x7fc6300096f0 2024-09-15T03:43:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc640ff9640 10 monclient: dump: 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 10 monclient: _renew_subs 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64409b070 con 0x7fc6300096f0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc642ffd640 4 set_mon_vals no callback set 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 10 monclient: _renew_subs 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.978+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64411be30 con 0x7fc6300096f0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc63c012460 con 0x7fc6300096f0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc640ff9640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc630039ec0 0x7fc63003c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc64b19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc63c04c1a0 con 0x7fc6300096f0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc64b19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc64b19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64409a500 con 0x7fc6300096f0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc648f39640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc630039ec0 0x7fc63003c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.979+0000 7fc648f39640 10 monclient: get_auth_request con 0x7fc630039ec0 auth_method 0 2024-09-15T03:43:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.982+0000 7fc648f39640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc630039ec0 0x7fc63003c370 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc634007d50 tx=0x7fc634037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.984+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc63c011e00 con 0x7fc6300096f0 2024-09-15T03:43:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.984+0000 7fc640ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:26.984+0000 7fc640ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:27.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc64b19d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc64b19d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc64b19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc64404bf60 con 0x7fc6300096f0 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc640ff9640 1 -- 172.21.15.153:0/3003058408 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fc63c010e40 con 0x7fc6300096f0 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc640ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.201+0000 7fc640ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:27.205 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":150,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:27.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc630039ec0 msgr2=0x7fc63003c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc630039ec0 0x7fc63003c370 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc634007d50 tx=0x7fc634037000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 10 monclient: shutdown 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 msgr2=0x7fc644141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 0x7fc644141870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc644066000 tx=0x7fc63c007750 comp rx=0 tx=0).stop 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 shutdown_connections 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc630039ec0 0x7fc63003c370 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6300096f0 0x7fc644141870 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6300066b0 0x7fc644141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 --2- 172.21.15.153:0/3003058408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6300037d0 0x7fc644130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 >> 172.21.15.153:0/3003058408 conn(0x7fc644067d30 msgr2=0x7fc6440a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:27.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 shutdown_connections 2024-09-15T03:43:27.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.203+0000 7fc64b19d640 1 -- 172.21.15.153:0/3003058408 wait complete. 2024-09-15T03:43:27.215 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:43:27.215 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:43:27.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:43:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.308+0000 7f94a651d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.308+0000 7f94a651d640 10 monclient: build_initial_monmap 2024-09-15T03:43:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.308+0000 7f94a651d640 10 monclient: monmap: 2024-09-15T03:43:27.309 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:27.309369+0000 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:27.309369+0000 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:27.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 1 Processor -- start 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 1 -- start start 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient: init 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 0x7f94a0098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient(hunting): picked mon.noname-b con 0x7f94a00981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:27.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94a0131da0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94a0132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a0098ad0 con 0x7f94a0132170 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a0067f30 con 0x7f94a0131da0 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0131bd0 con 0x7f94a00981c0 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f94a651d640 10 monclient(hunting): authenticate will time out at 1604.948925s 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f949ffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 msgr2=0x7f94a0098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.309+0000 7f949ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 0x7f94a0098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.310+0000 7f94a4aba640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f94a0132170 0x7f94a0067820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:51258/0 (socket says 172.21.15.153:51258) 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.310+0000 7f94a4aba640 1 -- 172.21.15.153:0/5483330 learned_addr learned my addr 172.21.15.153:0/5483330 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f94a4aba640 1 -- 172.21.15.153:0/5483330 >> v1:172.21.15.153:6789/0 conn(0x7f94a0132170 legacy=0x7f94a0067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90434405 0 0) 0x7f94a0098ad0 con 0x7f94a0132170 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): my global_id is 174273 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94880034e0 con 0x7f94a0132170 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 1 -- 172.21.15.153:0/5483330 >> v1:172.21.15.181:6789/0 conn(0x7f94a0131da0 legacy=0x7f94a0067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2082754118 0 0) 0x7f94880034e0 con 0x7f94a0132170 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 msgr2=0x7f94a0098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 0x7f94a0098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 >> v1:172.21.15.181:6789/0 conn(0x7f94a0131da0 legacy=0x7f94a0067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a009ca60 con 0x7f94a0132170 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: _finish_auth 0 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f9498001aa0 con 0x7f94a0132170 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: dump: 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.327 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 >> v1:172.21.15.153:6789/0 conn(0x7f94a0132170 legacy=0x7f94a0067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 0x7f9488003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): picked mon.c con 0x7f94a00981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94880041b0 0x7f9488004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): picked mon.a con 0x7f94880041b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9488004e70 0x7f948800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): picked mon.b con 0x7f9488004e70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0131bd0 con 0x7f94880041b0 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948800bf10 con 0x7f9488004e70 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948800c050 con 0x7f94a00981c0 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949ffff640 1 -- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 msgr2=0x7f9488003aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:43:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949ffff640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 0x7f9488003aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94880041b0 0x7f9488004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): get_auth_request con 0x7f94880041b0 auth_method 0 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f94a4aba640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9488004e70 0x7f948800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f949f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f94a4aba640 10 monclient(hunting): get_auth_request con 0x7f9488004e70 auth_method 0 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f94a4aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f94a4aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.312+0000 7f94a4aba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f94a4aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f94a4aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f94a4aba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:43:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 10 monclient(hunting): handle_auth_done global_id 174273 payload 293 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 1 -- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 msgr2=0x7f9488003aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94a00981c0 0x7f9488003aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 1 -- 172.21.15.153:0/5483330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9488004e70 msgr2=0x7f948800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9488004e70 0x7f948800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 1 monclient: found mon.a 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 1 -- 172.21.15.153:0/5483330 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94880034e0 con 0x7f94880041b0 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f949f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f94a651d640 5 monclient: authenticate success, global_id 174273 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.313+0000 7f94a651d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.315+0000 7f949f7fe640 1 -- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94880041b0 msgr2=0x7f9488004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.315+0000 7f949f7fe640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94880041b0 0x7f9488004780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9494001b30 tx=0x7f9494007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2141424d81cb7850 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949402ac70 con 0x7f94880041b0 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: dump: 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949402add0 con 0x7f94880041b0 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 1 -- 172.21.15.153:0/5483330 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494007340 con 0x7f94880041b0 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 4 set_mon_vals no callback set 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f949effd640 10 monclient: dump: 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 10 monclient: shutdown 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 1 -- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94880041b0 msgr2=0x7f9488004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 1 --2- 172.21.15.153:0/5483330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94880041b0 0x7f9488004780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9494001b30 tx=0x7f9494007c40 comp rx=0 tx=0).stop 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a4aba640 1 -- 172.21.15.153:0/5483330 reap_dead start 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 1 -- 172.21.15.153:0/5483330 shutdown_connections 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 1 -- 172.21.15.153:0/5483330 >> 172.21.15.153:0/5483330 conn(0x7f94a00a1e30 msgr2=0x7f94a00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 1 -- 172.21.15.153:0/5483330 shutdown_connections 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.316+0000 7f94a651d640 1 -- 172.21.15.153:0/5483330 wait complete. 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient: build_initial_monmap 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient: monmap: 2024-09-15T03:43:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 1 Processor -- start 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 1 -- start start 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient: init 2024-09-15T03:43:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94880041b0 0x7f94a011de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient(hunting): picked mon.noname-c con 0x7f94880041b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 0x7f94a011e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.317+0000 7f94a651d640 10 monclient(hunting): picked mon.noname-a con 0x7f94a011e340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94a011ec50 0x7f94a0147760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 10 monclient(hunting): picked mon.noname-b con 0x7f94a011ec50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0098cb0 con 0x7f94a011e340 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a00a09c0 con 0x7f94a011ec50 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a01204c0 con 0x7f94880041b0 2024-09-15T03:43:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a651d640 10 monclient(hunting): authenticate will time out at 1604.957028s 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949ffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94880041b0 msgr2=0x7f94a011de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94880041b0 0x7f94a011de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 0x7f94a011e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 0x7f94a011e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53196/0 (socket says 172.21.15.153:53196) 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 -- 172.21.15.153:0/3764275087 learned_addr learned my addr 172.21.15.153:0/3764275087 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): get_auth_request con 0x7f94a011e340 auth_method 0 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94a011ec50 0x7f94a0147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): get_auth_request con 0x7f94a011ec50 auth_method 0 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f94a4aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient(hunting): handle_auth_done global_id 174276 payload 824 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94880041b0 msgr2=0x7f94a011de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94880041b0 0x7f94a011de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94a011ec50 msgr2=0x7f94a0147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94a011ec50 0x7f94a0147760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.318+0000 7f949f7fe640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0121380 con 0x7f94a011e340 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f949f7fe640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 msgr2=0x7f94a011e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f949f7fe640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 0x7f94a011e710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9494001b30 tx=0x7f9494030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494037070 con 0x7f94a011e340 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: dump: 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9494005df0 con 0x7f94a011e340 2024-09-15T03:43:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 5 monclient: authenticate success, global_id 174276 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 10 monclient: _renew_subs 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a011f850 con 0x7f94a011e340 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940336a0 con 0x7f94a011e340 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f9487fff640 10 monclient: dump: 2024-09-15T03:43:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 10 monclient: _renew_subs 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f949e7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.320+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a011f0c0 con 0x7f94a011e340 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f94a651d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f94a651d640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 0x7f94a005e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f94a651d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0065f90 con 0x7f94a002aa40 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9494030070 con 0x7f94a011e340 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 0x7f94a005e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient: get_auth_request con 0x7f94a002aa40 auth_method 0 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.321+0000 7f9487fff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f948c0386f0 0x7f948c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.322+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f949403a040 con 0x7f94a011e340 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.322+0000 7f949ffff640 10 monclient(hunting): handle_auth_done global_id 174279 payload 824 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.322+0000 7f949ffff640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94a002ae10 con 0x7f94a002aa40 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.322+0000 7f94a4aba640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f948c0386f0 0x7f948c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.322+0000 7f94a4aba640 10 monclient: get_auth_request con 0x7f948c0386f0 auth_method 0 2024-09-15T03:43:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f949ffff640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 msgr2=0x7f94a005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f949ffff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 0x7f94a005e310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f949000b690 tx=0x7f94900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490006150 con 0x7f94a002aa40 2024-09-15T03:43:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 10 monclient: dump: 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f94a4aba640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f948c0386f0 0x7f948c03aba0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f94a009d0b0 tx=0x7f9498006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f94900062b0 con 0x7f94a002aa40 2024-09-15T03:43:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 msgr2=0x7f94a005e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 0x7f94a005e310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f949000b690 tx=0x7f94900091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 msgr2=0x7f94a005e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.324+0000 7f9487fff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a002aa40 0x7f94a005e310 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f949000b690 tx=0x7f94900091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f94a651d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f94a651d640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 0x7f94a014bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f94a651d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a011f2b0 con 0x7f94a005e850 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 0x7f94a014bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient: get_auth_request con 0x7f94a005e850 auth_method 0 2024-09-15T03:43:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 10 monclient(hunting): handle_auth_done global_id 174282 payload 824 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.330+0000 7f949ffff640 1 -- 172.21.15.153:0/3764275087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f94a004bf60 con 0x7f94a005e850 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.332+0000 7f949ffff640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 msgr2=0x7f94a014bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.332+0000 7f949ffff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 0x7f94a014bcc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9490013000 tx=0x7f9490011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000fd30 con 0x7f94a005e850 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 10 monclient: dump: 2024-09-15T03:43:27.352 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f949000fe90 con 0x7f94a005e850 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 msgr2=0x7f94a014bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 0x7f94a014bcc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9490013000 tx=0x7f9490011000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 msgr2=0x7f94a014bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f9487fff640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a005e850 0x7f94a014bcc0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9490013000 tx=0x7f9490011000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 150, 2024-09-15T03:43:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.365 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.366 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:27.367 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f948c0386f0 msgr2=0x7f948c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f948c0386f0 0x7f948c03aba0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f94a009d0b0 tx=0x7f9498006040 comp rx=0 tx=0).stop 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 10 monclient: shutdown 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 msgr2=0x7f94a011e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 --2- 172.21.15.153:0/3764275087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94a011e340 0x7f94a011e710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9494001b30 tx=0x7f9494030000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a4aba640 1 -- 172.21.15.153:0/3764275087 reap_dead start 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a4aba640 1 -- 172.21.15.153:0/3764275087 reap_dead start 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 shutdown_connections 2024-09-15T03:43:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 >> 172.21.15.153:0/3764275087 conn(0x7f94a00a1e30 msgr2=0x7f94a009b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:27.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 shutdown_connections 2024-09-15T03:43:27.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.333+0000 7f94a651d640 1 -- 172.21.15.153:0/3764275087 wait complete. 2024-09-15T03:43:27.370 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:43:27.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:43:27.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:43:27.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.463+0000 7f653699d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.463+0000 7f653699d640 10 monclient: build_initial_monmap 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.463+0000 7f653699d640 10 monclient: monmap: 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:27.463767+0000 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:27.463767+0000 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:27.465 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:27.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.463+0000 7f653699d640 1 Processor -- start 2024-09-15T03:43:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.463+0000 7f653699d640 1 -- start start 2024-09-15T03:43:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.463+0000 7f653699d640 10 monclient: init 2024-09-15T03:43:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6530131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f653009fe10 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f653009f140 0x7f6530139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient(hunting): picked mon.noname-a con 0x7f653009f140 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65300a14d0 con 0x7f6530131980 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65301325b0 con 0x7f653009fe10 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653009f560 con 0x7f653009f140 2024-09-15T03:43:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f653699d640 10 monclient(hunting): authenticate will time out at 1605.103419s 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f652ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6530131980 0x7f6530131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:51272/0 (socket says 172.21.15.153:51272) 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f652ffff640 1 -- 172.21.15.153:0/3398929108 learned_addr learned my addr 172.21.15.153:0/3398929108 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 1 --2- 172.21.15.153:0/3398929108 >> v2:172.21.15.153:3300/0 conn(0x7f653009f140 0x7f6530139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): get_auth_request con 0x7f653009f140 auth_method 0 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient(hunting): handle_auth_done global_id 174285 payload 274 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 1 -- 172.21.15.153:0/3398929108 >> v1:172.21.15.181:6789/0 conn(0x7f653009fe10 legacy=0x7f653009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 1 -- 172.21.15.153:0/3398929108 >> v1:172.21.15.153:6789/0 conn(0x7f6530131980 legacy=0x7f6530131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 1 monclient: found mon.noname-a 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.464+0000 7f6534f3a640 1 -- 172.21.15.153:0/3398929108 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65300a11f0 con 0x7f653009f140 2024-09-15T03:43:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f6534f3a640 1 -- 172.21.15.153:0/3398929108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 msgr2=0x7f6530139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f6534f3a640 1 --2- 172.21.15.153:0/3398929108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530139cd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6528009e60 tx=0x7f6528009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78237462defd74e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 1 -- 172.21.15.153:0/3398929108 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6528007630 con 0x7f653009f140 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: dump: 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: _finish_auth 0 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 1 -- 172.21.15.153:0/3398929108 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652802d050 con 0x7f653009f140 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f653699d640 5 monclient: authenticate success, global_id 174285 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f653699d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f653699d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f653699d640 4 set_mon_vals no callback set 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 1 -- 172.21.15.153:0/3398929108 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65280079c0 con 0x7f653009f140 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.467+0000 7f652effd640 10 monclient: dump: 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 10 monclient: shutdown 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 -- 172.21.15.153:0/3398929108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 msgr2=0x7f6530139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 --2- 172.21.15.153:0/3398929108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530139cd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6528009e60 tx=0x7f6528009780 comp rx=0 tx=0).stop 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 -- 172.21.15.153:0/3398929108 shutdown_connections 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 --2- 172.21.15.153:0/3398929108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530139cd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 -- 172.21.15.153:0/3398929108 >> 172.21.15.153:0/3398929108 conn(0x7f65300a5050 msgr2=0x7f65300a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 -- 172.21.15.153:0/3398929108 shutdown_connections 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.468+0000 7f653699d640 1 -- 172.21.15.153:0/3398929108 wait complete. 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient: build_initial_monmap 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient: monmap: 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 Processor -- start 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 -- start start 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient: init 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530138550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): picked mon.noname-a con 0x7f653009f140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f653009fe10 0x7f6530134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): picked mon.noname-c con 0x7f653009fe10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6530131980 0x7f6530134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): picked mon.noname-b con 0x7f6530131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6530069eb0 con 0x7f653009f140 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65300a41d0 con 0x7f6530131980 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6530137890 con 0x7f653009fe10 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f653699d640 10 monclient(hunting): authenticate will time out at 1605.108956s 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.469+0000 7f652f7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f653009fe10 msgr2=0x7f6530134220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f653009fe10 0x7f6530134220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53234/0 (socket says 172.21.15.153:53234) 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 -- 172.21.15.153:0/3080215374 learned_addr learned my addr 172.21.15.153:0/3080215374 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): get_auth_request con 0x7f653009f140 auth_method 0 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f6534f3a640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6530131980 0x7f6530134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f6534f3a640 10 monclient(hunting): get_auth_request con 0x7f6530131980 auth_method 0 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f6534f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f6534f3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f6534f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient(hunting): handle_auth_done global_id 174288 payload 824 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f653009fe10 msgr2=0x7f6530134220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f653009fe10 0x7f6530134220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6530131980 msgr2=0x7f6530134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6530131980 0x7f6530134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 monclient: found mon.noname-a 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.470+0000 7f652ffff640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6528009a20 con 0x7f653009f140 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f652ffff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 msgr2=0x7f6530138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f652ffff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530138550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f652000c5e0 tx=0x7f6520004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6520005cb0 con 0x7f653009f140 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: dump: 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.487 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 5 monclient: authenticate success, global_id 174288 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6520005e30 con 0x7f653009f140 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 10 monclient: _renew_subs 2024-09-15T03:43:27.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6530135d60 con 0x7f653009f140 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6520007320 con 0x7f653009f140 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f6517fff640 10 monclient: dump: 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f652e7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 10 monclient: _renew_subs 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.472+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6530138a90 con 0x7f653009f140 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f653699d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f653699d640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 0x7f6530134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f653699d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6530065e00 con 0x7f653002aa40 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f652001b3b0 con 0x7f653009f140 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 0x7f6530134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient: get_auth_request con 0x7f653002aa40 auth_method 0 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f6517fff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f651c0386f0 0x7f651c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 10 monclient(hunting): handle_auth_done global_id 144490 payload 824 2024-09-15T03:43:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f652004bab0 con 0x7f653009f140 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.473+0000 7f652f7fe640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6530137a00 con 0x7f653002aa40 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.474+0000 7f6534f3a640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f651c0386f0 0x7f651c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.474+0000 7f6534f3a640 10 monclient: get_auth_request con 0x7f651c0386f0 auth_method 0 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f652f7fe640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 msgr2=0x7f6530134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f652f7fe640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 0x7f6530134ca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6524002060 tx=0x7f6524007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65240075d0 con 0x7f653002aa40 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 10 monclient: dump: 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6534f3a640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f651c0386f0 0x7f651c03aba0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6528034000 tx=0x7f6528037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6524005780 con 0x7f653002aa40 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 msgr2=0x7f6530134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 0x7f6530134ca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6524002060 tx=0x7f6524007df0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 msgr2=0x7f6530134ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.476+0000 7f6517fff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653002aa40 0x7f6530134ca0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f653699d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f653699d640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 0x7f653005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f653699d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653002ae10 con 0x7f653005e310 2024-09-15T03:43:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 0x7f653005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient: get_auth_request con 0x7f653005e310 auth_method 0 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 10 monclient(hunting): handle_auth_done global_id 144493 payload 824 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.482+0000 7f652f7fe640 1 -- 172.21.15.153:0/3080215374 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6530138d10 con 0x7f653005e310 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.484+0000 7f652f7fe640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 msgr2=0x7f653005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.484+0000 7f652f7fe640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 0x7f653005e710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6524002060 tx=0x7f6524015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524020900 con 0x7f653005e310 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 10 monclient: dump: 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f65240073c0 con 0x7f653005e310 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 msgr2=0x7f653005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 0x7f653005e710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6524002060 tx=0x7f6524015000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 msgr2=0x7f653005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6517fff640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f653005e310 0x7f653005e710 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6524002060 tx=0x7f6524015000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 150, 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.502 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:27.503 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:27.504 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:27.505 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:27.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.507 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:27.517 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:27.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f651c0386f0 msgr2=0x7f651c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f651c0386f0 0x7f651c03aba0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6528034000 tx=0x7f6528037000 comp rx=0 tx=0).stop 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 10 monclient: shutdown 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 msgr2=0x7f6530138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 1 --2- 172.21.15.153:0/3080215374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f653009f140 0x7f6530138550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f652000c5e0 tx=0x7f6520004960 comp rx=0 tx=0).stop 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6534f3a640 1 -- 172.21.15.153:0/3080215374 reap_dead start 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f6534f3a640 1 -- 172.21.15.153:0/3080215374 reap_dead start 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 shutdown_connections 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.485+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 >> 172.21.15.153:0/3080215374 conn(0x7f65300a5050 msgr2=0x7f653009d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.486+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 shutdown_connections 2024-09-15T03:43:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:27.486+0000 7f653699d640 1 -- 172.21.15.153:0/3080215374 wait complete. 2024-09-15T03:43:27.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:43:27.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:43:27.744 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:43:27.745 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:43:27.745 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:27.745 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-15T03:43:27.745 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-15T03:43:27.745 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:43:27.745 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-15T03:43:27.746 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-15T03:43:27.746 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:43:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-15T03:43:28.042 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-15T03:43:28.042 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-15T03:43:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:43:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:43:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-15T03:43:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-15T03:43:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:43:28.341 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-15T03:43:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-15T03:43:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-15T03:43:28.580 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:43:28.580 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=110 2024-09-15T03:43:28.581 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-15T03:43:28.582 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-15T03:43:28.582 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-15T03:43:47.521 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:43:47.521 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:43:47.521 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:43:47.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:43:47.525 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:43:47.525 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:43:47.525 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:47.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.628+0000 7fb505617640 10 monclient: get_monmap_and_config 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.628+0000 7fb505617640 10 monclient: build_initial_monmap 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.628+0000 7fb505617640 10 monclient: monmap: 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:47.629630+0000 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:47.629630+0000 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:47.631 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:47.631 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:47.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.629+0000 7fb505617640 1 Processor -- start 2024-09-15T03:43:47.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.629+0000 7fb505617640 1 -- start start 2024-09-15T03:43:47.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.629+0000 7fb505617640 10 monclient: init 2024-09-15T03:43:47.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.629+0000 7fb505617640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.629+0000 7fb505617640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.629+0000 7fb505617640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): picked mon.noname-b con 0x7fb5001318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb50009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb500136c20 0x7fb500139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): picked mon.noname-a con 0x7fb500136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb500131cc0 con 0x7fb50009ea50 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50009f530 con 0x7fb500136c20 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50009fbf0 con 0x7fb5001318f0 2024-09-15T03:43:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb505617640 10 monclient(hunting): authenticate will time out at 1625.269115s 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb500136c20 0x7fb500139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58664/0 (socket says 172.21.15.153:58664) 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 -- 172.21.15.153:0/743205220 learned_addr learned my addr 172.21.15.153:0/743205220 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb500136c20 auth_method 0 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 10 monclient(hunting): get_auth_request con 0x7fb5001318f0 auth_method 0 2024-09-15T03:43:47.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:47.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:47.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 -- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 msgr2=0x7fb500131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 -- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 msgr2=0x7fb500131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:47.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4feffd640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_done global_id 174309 payload 274 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 msgr2=0x7fb500131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/743205220 >> v1:172.21.15.181:6789/0 conn(0x7fb50009ea50 legacy=0x7fb50009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 1 monclient: found mon.noname-a 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:47.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:47.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.630+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/743205220 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50009fe30 con 0x7fb500136c20 2024-09-15T03:43:47.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 msgr2=0x7fb500139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:47.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4ff7fe640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500139000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009e60 tx=0x7fb4f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c245247209a636b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:47.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 1 -- 172.21.15.153:0/743205220 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8007760 con 0x7fb500136c20 2024-09-15T03:43:47.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: dump: 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:47.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: _finish_auth 0 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 1 -- 172.21.15.153:0/743205220 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f802d050 con 0x7fb500136c20 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb505617640 5 monclient: authenticate success, global_id 174309 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb505617640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 1 -- 172.21.15.153:0/743205220 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8007af0 con 0x7fb500136c20 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb505617640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb505617640 4 set_mon_vals no callback set 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:47.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.633+0000 7fb4fdffb640 10 monclient: dump: 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 10 monclient: shutdown 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 -- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 msgr2=0x7fb500139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500139000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009e60 tx=0x7fb4f8009780 comp rx=0 tx=0).stop 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 -- 172.21.15.153:0/743205220 shutdown_connections 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500139000 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 --2- 172.21.15.153:0/743205220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 -- 172.21.15.153:0/743205220 >> 172.21.15.153:0/743205220 conn(0x7fb500067110 msgr2=0x7fb500067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 -- 172.21.15.153:0/743205220 shutdown_connections 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 1 -- 172.21.15.153:0/743205220 wait complete. 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 10 monclient: build_initial_monmap 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.634+0000 7fb505617640 10 monclient: monmap: 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 Processor -- start 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 -- start start 2024-09-15T03:43:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient: init 2024-09-15T03:43:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb50009ea50 0x7fb500141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): picked mon.noname-b con 0x7fb50009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): picked mon.noname-c con 0x7fb5001318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500099610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): picked mon.noname-a con 0x7fb500136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50011c7a0 con 0x7fb500136c20 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50009dcb0 con 0x7fb50009ea50 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50009bf10 con 0x7fb5001318f0 2024-09-15T03:43:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb505617640 10 monclient(hunting): authenticate will time out at 1625.274725s 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500099610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500099610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39588/0 (socket says 172.21.15.153:39588) 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 10 monclient(hunting): get_auth_request con 0x7fb5001318f0 auth_method 0 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/866449542 learned_addr learned my addr 172.21.15.153:0/866449542 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb500136c20 auth_method 0 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:47.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4feffd640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb50009ea50 0x7fb500141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 msgr2=0x7fb500141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 msgr2=0x7fb500141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4fe7fc640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500141770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4feffd640 10 monclient(hunting): get_auth_request con 0x7fb50009ea50 auth_method 0 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4feffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:47.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.635+0000 7fb4feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4fe7fc640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500141770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 10 monclient(hunting): handle_auth_done global_id 174312 payload 824 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 msgr2=0x7fb500141770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500141770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb50009ea50 msgr2=0x7fb500141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb50009ea50 0x7fb500141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:47.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 1 monclient: found mon.noname-a 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.636+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/866449542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f8009a20 con 0x7fb500136c20 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4ff7fe640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 msgr2=0x7fb500099610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4ff7fe640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500099610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009e60 tx=0x7fb4f8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8031c10 con 0x7fb500136c20 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: dump: 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f8040070 con 0x7fb500136c20 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 5 monclient: authenticate success, global_id 174312 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 10 monclient: _renew_subs 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50009b8d0 con 0x7fb500136c20 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8045430 con 0x7fb500136c20 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4e6ffd640 10 monclient: dump: 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 10 monclient: _renew_subs 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50009b160 con 0x7fb500136c20 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.638+0000 7fb4fd7fa640 4 set_mon_vals no callback set 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb4f8042030 con 0x7fb500136c20 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb505617640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb505617640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb505617640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50004bf60 con 0x7fb500136c20 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb4e6ffd640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4ec038740 0x7fb4ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.639+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb4f806e030 con 0x7fb500136c20 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.640+0000 7fb4feffd640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4ec038740 0x7fb4ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.640+0000 7fb4feffd640 10 monclient: get_auth_request con 0x7fb4ec038740 auth_method 0 2024-09-15T03:43:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.642+0000 7fb4feffd640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4ec038740 0x7fb4ec03abf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006a70 tx=0x7fb4f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.644+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb4f809fe40 con 0x7fb500136c20 2024-09-15T03:43:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.644+0000 7fb4e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.644+0000 7fb4e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:47.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.858+0000 7fb505617640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.858+0000 7fb505617640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.858+0000 7fb505617640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.858+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb5000638d0 con 0x7fb500136c20 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.859+0000 7fb4e6ffd640 1 -- 172.21.15.153:0/866449542 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fb4f803ab80 con 0x7fb500136c20 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.859+0000 7fb4e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.859+0000 7fb4e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:47.860 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":150,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4ec038740 msgr2=0x7fb4ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4ec038740 0x7fb4ec03abf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006a70 tx=0x7fb4f0008040 comp rx=0 tx=0).stop 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 10 monclient: shutdown 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 msgr2=0x7fb500099610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500099610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009e60 tx=0x7fb4f8037000 comp rx=0 tx=0).stop 2024-09-15T03:43:47.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 shutdown_connections 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb4ec038740 0x7fb4ec03abf0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb500136c20 0x7fb500099610 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5001318f0 0x7fb500141770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 --2- 172.21.15.153:0/866449542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb50009ea50 0x7fb500141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 >> 172.21.15.153:0/866449542 conn(0x7fb500067110 msgr2=0x7fb5000a3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 shutdown_connections 2024-09-15T03:43:47.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:47.861+0000 7fb505617640 1 -- 172.21.15.153:0/866449542 wait complete. 2024-09-15T03:43:50.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.975+0000 7fe8a8f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.975+0000 7fe8a8f1d640 10 monclient: build_initial_monmap 2024-09-15T03:43:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.975+0000 7fe8a8f1d640 10 monclient: monmap: 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:50.976231+0000 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:50.976231+0000 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:50.978 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 Processor -- start 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 -- start start 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient: init 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8a4096c60 0x7fe8a409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8a4096c60 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8a4097030 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe8a4133b40 0x7fe8a4135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8a4133b40 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a40696a0 con 0x7fe8a4097030 2024-09-15T03:43:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a409ac40 con 0x7fe8a4096c60 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a409d6b0 con 0x7fe8a4133b40 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a8f1d640 10 monclient(hunting): authenticate will time out at 1628.615817s 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8a4096c60 0x7fe8a409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe89bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe8a4097030 0x7fe8a4097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39096/0 (socket says 172.21.15.153:39096) 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe89bfff640 1 -- 172.21.15.153:0/951966249 learned_addr learned my addr 172.21.15.153:0/951966249 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8a4096c60 0x7fe8a409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39644/0 (socket says 172.21.15.153:39644) 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.976+0000 7fe8a259c640 10 monclient(hunting): get_auth_request con 0x7fe8a4096c60 auth_method 0 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a2d9d640 1 --2- 172.21.15.153:0/951966249 >> v2:172.21.15.181:3300/0 conn(0x7fe8a4133b40 0x7fe8a4135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a2d9d640 10 monclient(hunting): get_auth_request con 0x7fe8a4133b40 auth_method 0 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient(hunting): handle_auth_done global_id 174315 payload 274 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 1 -- 172.21.15.153:0/951966249 >> v2:172.21.15.181:3300/0 conn(0x7fe8a4133b40 msgr2=0x7fe8a4135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 1 --2- 172.21.15.153:0/951966249 >> v2:172.21.15.181:3300/0 conn(0x7fe8a4133b40 0x7fe8a4135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 1 -- 172.21.15.153:0/951966249 >> v1:172.21.15.153:6789/0 conn(0x7fe8a4097030 legacy=0x7fe8a4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient: _finish_hunting 0 2024-09-15T03:43:50.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 1 monclient: found mon.noname-a 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.977+0000 7fe8a259c640 1 -- 172.21.15.153:0/951966249 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a409d2e0 con 0x7fe8a4096c60 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a259c640 1 -- 172.21.15.153:0/951966249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 msgr2=0x7fe8a409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a259c640 1 --2- 172.21.15.153:0/951966249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a409a6c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4097b70 tx=0x7fe8940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be620adf3403338 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 1 -- 172.21.15.153:0/951966249 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89402ae00 con 0x7fe8a4096c60 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: dump: 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 1 -- 172.21.15.153:0/951966249 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894032440 con 0x7fe8a4096c60 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 5 monclient: authenticate success, global_id 174315 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 4 set_mon_vals no callback set 2024-09-15T03:43:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 1 -- 172.21.15.153:0/951966249 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940327e0 con 0x7fe8a4096c60 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a1d9b640 10 monclient: dump: 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 10 monclient: shutdown 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/951966249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 msgr2=0x7fe8a409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/951966249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a409a6c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4097b70 tx=0x7fe8940090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/951966249 shutdown_connections 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/951966249 >> v2:172.21.15.181:3300/0 conn(0x7fe8a4133b40 0x7fe8a4135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/951966249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a409a6c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/951966249 >> 172.21.15.153:0/951966249 conn(0x7fe8a40672b0 msgr2=0x7fe8a40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/951966249 shutdown_connections 2024-09-15T03:43:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.980+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/951966249 wait complete. 2024-09-15T03:43:50.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.981+0000 7fe8a8f1d640 10 monclient: build_initial_monmap 2024-09-15T03:43:50.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.981+0000 7fe8a8f1d640 10 monclient: monmap: 2024-09-15T03:43:50.993 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:50.993 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:50.994 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:50.994 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:50.994 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:50.994 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:50.994 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:50.994 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.981+0000 7fe8a8f1d640 1 Processor -- start 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.981+0000 7fe8a8f1d640 1 -- start start 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.981+0000 7fe8a8f1d640 10 monclient: init 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a406bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8a4096c60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:50.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 0x7fe8a406c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:50.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8a4097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:50.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a4133b40 0x7fe8a406ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:50.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8a4133b40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:50.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:50.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a406e6e0 con 0x7fe8a4096c60 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a406f740 con 0x7fe8a4133b40 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4117870 con 0x7fe8a4097030 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a8f1d640 10 monclient(hunting): authenticate will time out at 1628.621184s 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a406bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a406bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39656/0 (socket says 172.21.15.153:39656) 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 -- 172.21.15.153:0/1124896452 learned_addr learned my addr 172.21.15.153:0/1124896452 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:50.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): get_auth_request con 0x7fe8a4096c60 auth_method 0 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 0x7fe8a406c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 10 monclient(hunting): get_auth_request con 0x7fe8a4097030 auth_method 0 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:50.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a4133b40 0x7fe8a406ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 msgr2=0x7fe8a406c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 msgr2=0x7fe8a406c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): get_auth_request con 0x7fe8a4133b40 auth_method 0 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 0x7fe8a406c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe89bfff640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 0x7fe8a406c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:50.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient(hunting): handle_auth_done global_id 174318 payload 824 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 msgr2=0x7fe8a406c520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 0x7fe8a406c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a4133b40 msgr2=0x7fe8a406ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a4133b40 0x7fe8a406ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient: _finish_hunting 0 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 monclient: found mon.noname-a 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:51.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.982+0000 7fe8a259c640 1 -- 172.21.15.153:0/1124896452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894009390 con 0x7fe8a4096c60 2024-09-15T03:43:51.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.984+0000 7fe8a259c640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 msgr2=0x7fe8a406bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:51.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.984+0000 7fe8a259c640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a406bfe0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe894035040 tx=0x7fe894032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.984+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894031990 con 0x7fe8a4096c60 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.984+0000 7fe89a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.984+0000 7fe89a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.984+0000 7fe89a7fc640 10 monclient: dump: 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:51.003 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894031af0 con 0x7fe8a4096c60 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 5 monclient: authenticate success, global_id 174318 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 monclient: _renew_subs 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a4135a20 con 0x7fe8a4096c60 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894031e70 con 0x7fe8a4096c60 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 10 monclient: dump: 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:51.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a159a640 4 set_mon_vals no callback set 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 monclient: _renew_subs 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a412f2c0 con 0x7fe8a4096c60 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe894039040 con 0x7fe8a4096c60 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.985+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4117620 con 0x7fe8a4096c60 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.986+0000 7fe89a7fc640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe884038740 0x7fe88403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.986+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe89406c950 con 0x7fe8a4096c60 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.986+0000 7fe89bfff640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe884038740 0x7fe88403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.986+0000 7fe89bfff640 10 monclient: get_auth_request con 0x7fe884038740 auth_method 0 2024-09-15T03:43:51.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.988+0000 7fe89bfff640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe884038740 0x7fe88403abf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe8a406d890 tx=0x7fe88c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.990+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe89403b610 con 0x7fe8a4096c60 2024-09-15T03:43:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.990+0000 7fe89a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:50.990+0000 7fe89a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:51.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe8a8f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:51.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe8a8f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:51.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe8a8f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:51.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe8a404bf60 con 0x7fe8a4096c60 2024-09-15T03:43:51.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe89a7fc640 1 -- 172.21.15.153:0/1124896452 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fe89403f080 con 0x7fe8a4096c60 2024-09-15T03:43:51.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe89a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:51.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.205+0000 7fe89a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:51.207 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":150,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:51.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe884038740 msgr2=0x7fe88403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:51.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe884038740 0x7fe88403abf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe8a406d890 tx=0x7fe88c009040 comp rx=0 tx=0).stop 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 10 monclient: shutdown 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 msgr2=0x7fe8a406bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a406bfe0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe894035040 tx=0x7fe894032000 comp rx=0 tx=0).stop 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 shutdown_connections 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe884038740 0x7fe88403abf0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a4133b40 0x7fe8a406ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8a4097030 0x7fe8a406c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 --2- 172.21.15.153:0/1124896452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a4096c60 0x7fe8a406bfe0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 >> 172.21.15.153:0/1124896452 conn(0x7fe8a40672b0 msgr2=0x7fe8a4134130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 shutdown_connections 2024-09-15T03:43:51.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:51.207+0000 7fe8a8f1d640 1 -- 172.21.15.153:0/1124896452 wait complete. 2024-09-15T03:43:54.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.323+0000 7f3b4455d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.323+0000 7f3b4455d640 10 monclient: build_initial_monmap 2024-09-15T03:43:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.323+0000 7f3b4455d640 10 monclient: monmap: 2024-09-15T03:43:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:54.324391+0000 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:54.324391+0000 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:54.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 Processor -- start 2024-09-15T03:43:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 -- start start 2024-09-15T03:43:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient: init 2024-09-15T03:43:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3b3c0975f0 0x7f3b3c095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3c0975f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b3c0979c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3b3c097dc0 0x7f3b3c09c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3c097dc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c098200 con 0x7f3b3c0979c0 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c09d1b0 con 0x7f3b3c097dc0 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c09d420 con 0x7f3b3c0975f0 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.324+0000 7f3b4455d640 10 monclient(hunting): authenticate will time out at 1631.963974s 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3b3c097dc0 0x7f3b3c09c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3b3c097dc0 0x7f3b3c09c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39664/0 (socket says 172.21.15.153:39664) 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 -- 172.21.15.153:0/2053903024 learned_addr learned my addr 172.21.15.153:0/2053903024 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): get_auth_request con 0x7f3b3c097dc0 auth_method 0 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b41af8640 1 --1- 172.21.15.153:0/2053903024 >> v1:172.21.15.153:6789/0 conn(0x7f3b3c0979c0 0x7f3b3c09bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39108/0 (socket says 172.21.15.153:39108) 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b422f9640 1 --2- 172.21.15.153:0/2053903024 >> v2:172.21.15.181:3300/0 conn(0x7f3b3c0975f0 0x7f3b3c095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b422f9640 10 monclient(hunting): get_auth_request con 0x7f3b3c0975f0 auth_method 0 2024-09-15T03:43:54.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b422f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b422f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b422f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient(hunting): handle_auth_done global_id 174321 payload 274 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 -- 172.21.15.153:0/2053903024 >> v2:172.21.15.181:3300/0 conn(0x7f3b3c0975f0 msgr2=0x7f3b3c095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 --2- 172.21.15.153:0/2053903024 >> v2:172.21.15.181:3300/0 conn(0x7f3b3c0975f0 0x7f3b3c095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 -- 172.21.15.153:0/2053903024 >> v1:172.21.15.153:6789/0 conn(0x7f3b3c0979c0 legacy=0x7f3b3c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 monclient: found mon.noname-a 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.325+0000 7f3b42afa640 1 -- 172.21.15.153:0/2053903024 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c09d9e0 con 0x7f3b3c097dc0 2024-09-15T03:43:54.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b42afa640 1 -- 172.21.15.153:0/2053903024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c097dc0 msgr2=0x7f3b3c09c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b42afa640 1 --2- 172.21.15.153:0/2053903024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c09c660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0713a0 tx=0x7f3b34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38dd3e1fa6eaab2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 1 -- 172.21.15.153:0/2053903024 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34007610 con 0x7f3b3c097dc0 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: dump: 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.340 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 1 -- 172.21.15.153:0/2053903024 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3402d050 con 0x7f3b3c097dc0 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 5 monclient: authenticate success, global_id 174321 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 4 set_mon_vals no callback set 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 1 -- 172.21.15.153:0/2053903024 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b340079a0 con 0x7f3b3c097dc0 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b412f7640 10 monclient: dump: 2024-09-15T03:43:54.341 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 10 monclient: shutdown 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 1 -- 172.21.15.153:0/2053903024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c097dc0 msgr2=0x7f3b3c09c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.328+0000 7f3b4455d640 1 --2- 172.21.15.153:0/2053903024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c09c660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0713a0 tx=0x7f3b34009840 comp rx=0 tx=0).stop 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 -- 172.21.15.153:0/2053903024 shutdown_connections 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 --2- 172.21.15.153:0/2053903024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c09c660 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 --2- 172.21.15.153:0/2053903024 >> v2:172.21.15.181:3300/0 conn(0x7f3b3c0975f0 0x7f3b3c095ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 -- 172.21.15.153:0/2053903024 >> 172.21.15.153:0/2053903024 conn(0x7f3b3c0678a0 msgr2=0x7f3b3c0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 -- 172.21.15.153:0/2053903024 shutdown_connections 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 -- 172.21.15.153:0/2053903024 wait complete. 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 10 monclient: build_initial_monmap 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 10 monclient: monmap: 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:54.343 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.329+0000 7f3b4455d640 1 Processor -- start 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 -- start start 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient: init 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b3c0975f0 0x7f3b3c140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3c0975f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 0x7f3b3c131a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3c0979c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b3c097dc0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c132740 con 0x7f3b3c0979c0 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0983c0 con 0x7f3b3c097dc0 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c09a510 con 0x7f3b3c0975f0 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b4455d640 10 monclient(hunting): authenticate will time out at 1631.969531s 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b3c0975f0 0x7f3b3c140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 0x7f3b3c131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b3c0975f0 0x7f3b3c140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58700/0 (socket says 172.21.15.153:58700) 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): get_auth_request con 0x7f3b3c0979c0 auth_method 0 2024-09-15T03:43:54.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 1 -- 172.21.15.153:0/881925547 learned_addr learned my addr 172.21.15.153:0/881925547 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): get_auth_request con 0x7f3b3c0975f0 auth_method 0 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b422f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): get_auth_request con 0x7f3b3c097dc0 auth_method 0 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b42afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient(hunting): handle_auth_done global_id 174324 payload 824 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 -- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b3c0975f0 msgr2=0x7f3b3c140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b3c0975f0 0x7f3b3c140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 -- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b3c097dc0 msgr2=0x7f3b3c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c098a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 monclient: found mon.noname-a 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.330+0000 7f3b41af8640 1 -- 172.21.15.153:0/881925547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34009ae0 con 0x7f3b3c0979c0 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b41af8640 1 -- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 msgr2=0x7f3b3c131a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b41af8640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 0x7f3b3c131a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00c490 tx=0x7f3b2c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c0074f0 con 0x7f3b3c0979c0 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: dump: 2024-09-15T03:43:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c007670 con 0x7f3b3c0979c0 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 5 monclient: authenticate success, global_id 174324 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 10 monclient: _renew_subs 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c141c30 con 0x7f3b3c0979c0 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c005a20 con 0x7f3b3c0979c0 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b327fc640 10 monclient: dump: 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.354 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 10 monclient: _renew_subs 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b40af6640 4 set_mon_vals no callback set 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.333+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c12df70 con 0x7f3b3c0979c0 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3b2c01b3b0 con 0x7f3b3c0979c0 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b4455d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b4455d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b4455d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c04bf60 con 0x7f3b3c0979c0 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b327fc640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b200387c0 0x7f3b2003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3b2c04bac0 con 0x7f3b3c0979c0 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b422f9640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b200387c0 0x7f3b2003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.334+0000 7f3b422f9640 10 monclient: get_auth_request con 0x7f3b200387c0 auth_method 0 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.337+0000 7f3b422f9640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b200387c0 0x7f3b2003ac70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3b38007c60 tx=0x7f3b38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.338+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3b2c01be60 con 0x7f3b3c0979c0 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.338+0000 7f3b327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.338+0000 7f3b327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.554+0000 7f3b4455d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.554+0000 7f3b4455d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.554+0000 7f3b4455d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.554+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3b3c0008d0 con 0x7f3b3c0979c0 2024-09-15T03:43:54.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.555+0000 7f3b327fc640 1 -- 172.21.15.153:0/881925547 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f3b2c017080 con 0x7f3b3c0979c0 2024-09-15T03:43:54.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.555+0000 7f3b327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:54.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.555+0000 7f3b327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:54.556 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":152,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:54.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b200387c0 msgr2=0x7f3b2003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b200387c0 0x7f3b2003ac70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3b38007c60 tx=0x7f3b38008040 comp rx=0 tx=0).stop 2024-09-15T03:43:54.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 10 monclient: shutdown 2024-09-15T03:43:54.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:54.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 msgr2=0x7f3b3c131a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 0x7f3b3c131a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00c490 tx=0x7f3b2c007e40 comp rx=0 tx=0).stop 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 shutdown_connections 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3b200387c0 0x7f3b2003ac70 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b3c097dc0 0x7f3b3c098a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b3c0979c0 0x7f3b3c131a80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 --2- 172.21.15.153:0/881925547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b3c0975f0 0x7f3b3c140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 >> 172.21.15.153:0/881925547 conn(0x7f3b3c0678a0 msgr2=0x7f3b3c132900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 shutdown_connections 2024-09-15T03:43:54.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.557+0000 7f3b4455d640 1 -- 172.21.15.153:0/881925547 wait complete. 2024-09-15T03:43:54.568 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:43:54.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:43:54.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:43:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.661+0000 7fb7f239d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.661+0000 7fb7f239d640 10 monclient: build_initial_monmap 2024-09-15T03:43:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.661+0000 7fb7f239d640 10 monclient: monmap: 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:54.662474+0000 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:54.662474+0000 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:54.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 Processor -- start 2024-09-15T03:43:54.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 -- start start 2024-09-15T03:43:54.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient: init 2024-09-15T03:43:54.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7ec096e10 0x7fb7ec09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7ec096e10 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7ec09aaa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb7ec0978f0 0x7fb7ec135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ec0978f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec09d4a0 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec099620 con 0x7fb7ec096e10 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec099340 con 0x7fb7ec0978f0 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.662+0000 7fb7f239d640 10 monclient(hunting): authenticate will time out at 1632.301998s 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7ec096e10 0x7fb7ec09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7ec096e10 0x7fb7ec09a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39686/0 (socket says 172.21.15.153:39686) 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 -- 172.21.15.153:0/4234613499 learned_addr learned my addr 172.21.15.153:0/4234613499 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request con 0x7fb7ec096e10 auth_method 0 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_done global_id 174327 payload 274 2024-09-15T03:43:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7f093a640 1 --2- 172.21.15.153:0/4234613499 >> v2:172.21.15.181:3300/0 conn(0x7fb7ec0978f0 0x7fb7ec135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 -- 172.21.15.153:0/4234613499 >> v2:172.21.15.181:3300/0 conn(0x7fb7ec0978f0 msgr2=0x7fb7ec135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/4234613499 >> v2:172.21.15.181:3300/0 conn(0x7fb7ec0978f0 0x7fb7ec135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 -- 172.21.15.153:0/4234613499 >> v1:172.21.15.181:6789/0 conn(0x7fb7ec09aaa0 legacy=0x7fb7ec0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 monclient: found mon.noname-a 2024-09-15T03:43:54.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.663+0000 7fb7ebfff640 1 -- 172.21.15.153:0/4234613499 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec099900 con 0x7fb7ec096e10 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7ebfff640 1 -- 172.21.15.153:0/4234613499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 msgr2=0x7fb7ec09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/4234613499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec09a560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec09d0e0 tx=0x7fb7d40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63e4865ed0fd733 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 1 -- 172.21.15.153:0/4234613499 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4032440 con 0x7fb7ec096e10 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: dump: 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.681 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 1 -- 172.21.15.153:0/4234613499 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d40325a0 con 0x7fb7ec096e10 2024-09-15T03:43:54.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 5 monclient: authenticate success, global_id 174327 2024-09-15T03:43:54.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:54.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:54.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 1 -- 172.21.15.153:0/4234613499 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4032940 con 0x7fb7ec096e10 2024-09-15T03:43:54.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 4 set_mon_vals no callback set 2024-09-15T03:43:54.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:54.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7eaffd640 10 monclient: dump: 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 10 monclient: shutdown 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 -- 172.21.15.153:0/4234613499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 msgr2=0x7fb7ec09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 --2- 172.21.15.153:0/4234613499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec09a560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec09d0e0 tx=0x7fb7d40091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 -- 172.21.15.153:0/4234613499 shutdown_connections 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 --2- 172.21.15.153:0/4234613499 >> v2:172.21.15.181:3300/0 conn(0x7fb7ec0978f0 0x7fb7ec135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 --2- 172.21.15.153:0/4234613499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec09a560 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 -- 172.21.15.153:0/4234613499 >> 172.21.15.153:0/4234613499 conn(0x7fb7ec0672b0 msgr2=0x7fb7ec0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 -- 172.21.15.153:0/4234613499 shutdown_connections 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.666+0000 7fb7f239d640 1 -- 172.21.15.153:0/4234613499 wait complete. 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.667+0000 7fb7f239d640 10 monclient: build_initial_monmap 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.667+0000 7fb7f239d640 10 monclient: monmap: 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:54.687 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.667+0000 7fb7f239d640 1 Processor -- start 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.667+0000 7fb7f239d640 1 -- start start 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.667+0000 7fb7f239d640 10 monclient: init 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:43:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec1346a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7ec096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7ec134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7ec0978f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 0x7fb7ec06ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ec09aaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec121e20 con 0x7fb7ec096e10 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec122090 con 0x7fb7ec0978f0 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0696a0 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f239d640 10 monclient(hunting): authenticate will time out at 1632.307349s 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec1346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 0x7fb7ec06ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec1346a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39698/0 (socket says 172.21.15.153:39698) 2024-09-15T03:43:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 1 -- 172.21.15.153:0/3964756922 learned_addr learned my addr 172.21.15.153:0/3964756922 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request con 0x7fb7ec096e10 auth_method 0 2024-09-15T03:43:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): get_auth_request con 0x7fb7ec09aaa0 auth_method 0 2024-09-15T03:43:54.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7ec134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): get_auth_request con 0x7fb7ec0978f0 auth_method 0 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient(hunting): handle_auth_done global_id 194102 payload 824 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 msgr2=0x7fb7ec134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7ec134be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 msgr2=0x7fb7ec1346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec1346a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 monclient: found mon.noname-c 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4009450 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.668+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7ec1346a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 msgr2=0x7fb7ec06ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f093a640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 0x7fb7ec06ae60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400ce50 tx=0x7fb7e40077b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400e610 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: dump: 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.695 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 5 monclient: authenticate success, global_id 194102 2024-09-15T03:43:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e400e770 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 10 monclient: _renew_subs 2024-09-15T03:43:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec142130 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400eb30 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7e3fff640 10 monclient: dump: 2024-09-15T03:43:54.698 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 10 monclient: _renew_subs 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7ea7fc640 4 set_mon_vals no callback set 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec06ca20 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 msgr2=0x7fb7ec06ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 msgr2=0x7fb7ec06ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f093a640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 0x7fb7ec06ae60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400ce50 tx=0x7fb7e40077b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f093a640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 0x7fb7ec06ae60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400ce50 tx=0x7fb7e40077b0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.671+0000 7fb7f239d640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 0x7fb7ec0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec065f20 con 0x7fb7ec06bb60 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 msgr2=0x7fb7ec06ae60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec09aaa0 0x7fb7ec06ae60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:43:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7cc0057a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): picked mon.b con 0x7fb7ec0978f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7cc0060a0 0x7fb7cc008520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): picked mon.c con 0x7fb7cc0060a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7cc00b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): picked mon.a con 0x7fb7ec096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00b980 con 0x7fb7ec096e10 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00bac0 con 0x7fb7ec0978f0 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00bc00 con 0x7fb7cc0060a0 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (unknown 0 0 0) 0x7fb7e4009030 con 0x7fb7ec09aaa0 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 0x7fb7ec0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request con 0x7fb7ec06bb60 auth_method 0 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7cc0060a0 0x7fb7cc008520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7cc00b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7cc0057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): get_auth_request con 0x7fb7ec0978f0 auth_method 0 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:54.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request con 0x7fb7ec096e10 auth_method 0 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): get_auth_request con 0x7fb7cc0060a0 auth_method 0 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7f093a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient(hunting): handle_auth_done global_id 194102 payload 293 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7cc0413f0 0x7fb7cc0438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7cc0060a0 msgr2=0x7fb7cc008520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7cc0060a0 0x7fb7cc008520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 msgr2=0x7fb7cc00b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec096e10 0x7fb7cc00b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 monclient: found mon.b 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fb7cc00bef0 con 0x7fb7ec0978f0 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_done global_id 174333 payload 824 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7ebfff640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7ec1221d0 con 0x7fb7ec06bb60 2024-09-15T03:43:54.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7cc0413f0 0x7fb7cc0438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.672+0000 7fb7eb7fe640 10 monclient: get_auth_request con 0x7fb7cc0413f0 auth_method 0 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.674+0000 7fb7ebfff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 msgr2=0x7fb7ec0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.674+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 0x7fb7ec0298c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009420 tx=0x7fb7d4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.674+0000 7fb7eb7fe640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 msgr2=0x7fb7cc0057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.674+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7cc0057a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc003990 tx=0x7fb7dc007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d403c7d0 con 0x7fb7ec06bb60 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: dump: 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.710 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00f920 con 0x7fb7ec0978f0 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: dump: 2024-09-15T03:43:54.711 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc013a60 con 0x7fb7ec0978f0 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e9ffb640 4 set_mon_vals no callback set 2024-09-15T03:43:54.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb7d4045040 con 0x7fb7ec06bb60 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 msgr2=0x7fb7ec0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 0x7fb7ec0298c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009420 tx=0x7fb7d4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 msgr2=0x7fb7ec0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bb60 0x7fb7ec0298c0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009420 tx=0x7fb7d4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.675+0000 7fb7eb7fe640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7cc0413f0 0x7fb7cc0438a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc003f10 tx=0x7fb7dc014000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.676+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb7dc013bd0 con 0x7fb7ec0978f0 2024-09-15T03:43:54.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.676+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.1 v2:172.21.15.181:3300/0 4 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb7dc04e170 con 0x7fb7ec0978f0 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7f239d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7f239d640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 0x7fb7ec06ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7f239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec09a290 con 0x7fb7ec06bf30 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 0x7fb7ec06ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient: get_auth_request con 0x7fb7ec06bf30 auth_method 0 2024-09-15T03:43:54.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 10 monclient(hunting): handle_auth_done global_id 174336 payload 824 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.681+0000 7fb7ebfff640 1 -- 172.21.15.153:0/3964756922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb7ec04bf60 con 0x7fb7ec06bf30 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7ebfff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 msgr2=0x7fb7ec06ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7ebfff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 0x7fb7ec06ae60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003080 tx=0x7fb7d4041050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d403dc20 con 0x7fb7ec06bf30 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 10 monclient: dump: 2024-09-15T03:43:54.716 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.717 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fb7d403f110 con 0x7fb7ec06bf30 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 msgr2=0x7fb7ec06ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 0x7fb7ec06ae60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003080 tx=0x7fb7d4041050 comp rx=0 tx=0).stop 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 msgr2=0x7fb7ec06ae60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.683+0000 7fb7e3fff640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec06bf30 0x7fb7ec06ae60 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003080 tx=0x7fb7d4041050 comp rx=0 tx=0).stop 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:54.718 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 152, 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:54.719 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.724 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.725 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:54.726 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.727 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:54.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.729 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.730 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:54.731 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:54.732 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7cc0413f0 msgr2=0x7fb7cc0438a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7cc0413f0 0x7fb7cc0438a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc003f10 tx=0x7fb7dc014000 comp rx=0 tx=0).stop 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 10 monclient: shutdown 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 msgr2=0x7fb7cc0057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7cc0057a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc003990 tx=0x7fb7dc007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f093a640 1 -- 172.21.15.153:0/3964756922 reap_dead start 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 shutdown_connections 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 --2- 172.21.15.153:0/3964756922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0978f0 0x7fb7cc0057a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 >> 172.21.15.153:0/3964756922 conn(0x7fb7ec0672b0 msgr2=0x7fb7ec09b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:54.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 shutdown_connections 2024-09-15T03:43:54.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.684+0000 7fb7f239d640 1 -- 172.21.15.153:0/3964756922 wait complete. 2024-09-15T03:43:54.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:54.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:43:54.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:43:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.828+0000 7f0d29b1d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.828+0000 7f0d29b1d640 10 monclient: build_initial_monmap 2024-09-15T03:43:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.828+0000 7f0d29b1d640 10 monclient: monmap: 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:54.829515+0000 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:54.829515+0000 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:54.830 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.829+0000 7f0d29b1d640 1 Processor -- start 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.829+0000 7f0d29b1d640 1 -- start start 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.829+0000 7f0d29b1d640 10 monclient: init 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d24067110 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d2409b010 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d2409b400 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24130540 con 0x7f0d2409b010 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24099800 con 0x7f0d2409b400 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24067ed0 con 0x7f0d24067110 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d29b1d640 10 monclient(hunting): authenticate will time out at 1632.469152s 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d22ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0d2409b010 0x7f0d2409c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39116/0 (socket says 172.21.15.153:39116) 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d22ffd640 1 -- 172.21.15.153:0/3975902437 learned_addr learned my addr 172.21.15.153:0/3975902437 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): get_auth_request con 0x7f0d24067110 auth_method 0 2024-09-15T03:43:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient(hunting): handle_auth_done global_id 194105 payload 274 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 1 -- 172.21.15.153:0/3975902437 >> v1:172.21.15.181:6789/0 conn(0x7f0d2409b400 legacy=0x7f0d2409cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 1 -- 172.21.15.153:0/3975902437 >> v1:172.21.15.153:6789/0 conn(0x7f0d2409b010 legacy=0x7f0d2409c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 1 monclient: found mon.noname-b 2024-09-15T03:43:54.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.830+0000 7f0d237fe640 1 -- 172.21.15.153:0/3975902437 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240674e0 con 0x7f0d24067110 2024-09-15T03:43:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d237fe640 1 -- 172.21.15.153:0/3975902437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 msgr2=0x7f0d2412f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d237fe640 1 --2- 172.21.15.153:0/3975902437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412f670 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009e90 tx=0x7f0d14009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aaa8b989abc12a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 1 -- 172.21.15.153:0/3975902437 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14007840 con 0x7f0d24067110 2024-09-15T03:43:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: dump: 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 1 -- 172.21.15.153:0/3975902437 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1402d050 con 0x7f0d24067110 2024-09-15T03:43:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d29b1d640 5 monclient: authenticate success, global_id 194105 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d29b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d29b1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d29b1d640 4 set_mon_vals no callback set 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 1 -- 172.21.15.153:0/3975902437 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14007bd0 con 0x7f0d24067110 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.833+0000 7f0d227fc640 10 monclient: dump: 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 10 monclient: shutdown 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 -- 172.21.15.153:0/3975902437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 msgr2=0x7f0d2412f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 --2- 172.21.15.153:0/3975902437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412f670 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009e90 tx=0x7f0d14009780 comp rx=0 tx=0).stop 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 -- 172.21.15.153:0/3975902437 shutdown_connections 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 --2- 172.21.15.153:0/3975902437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412f670 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 -- 172.21.15.153:0/3975902437 >> 172.21.15.153:0/3975902437 conn(0x7f0d240a1e30 msgr2=0x7f0d240a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 -- 172.21.15.153:0/3975902437 shutdown_connections 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 1 -- 172.21.15.153:0/3975902437 wait complete. 2024-09-15T03:43:54.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 10 monclient: build_initial_monmap 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.834+0000 7f0d29b1d640 10 monclient: monmap: 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 Processor -- start 2024-09-15T03:43:54.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 -- start start 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient: init 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d24067110 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2409b010 0x7f0d2406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d2409b010 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d2409b400 0x7f0d2406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d2409b400 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2406f210 con 0x7f0d2409b400 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24132800 con 0x7f0d2409b010 2024-09-15T03:43:54.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2413fbe0 con 0x7f0d24067110 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d29b1d640 10 monclient(hunting): authenticate will time out at 1632.474810s 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d23fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d2409b400 0x7f0d2406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:58746/0 (socket says 172.21.15.153:58746) 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 1 -- 172.21.15.153:0/1269856411 learned_addr learned my addr 172.21.15.153:0/1269856411 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 10 monclient(hunting): get_auth_request con 0x7f0d24067110 auth_method 0 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d23fff640 10 monclient(hunting): get_auth_request con 0x7f0d2409b400 auth_method 0 2024-09-15T03:43:54.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d23fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.835+0000 7f0d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2409b010 0x7f0d2406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 10 monclient(hunting): handle_auth_done global_id 194108 payload 824 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2409b010 msgr2=0x7f0d2406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2409b010 0x7f0d2406d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d2409b400 msgr2=0x7f0d2406db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d2409b400 0x7f0d2406db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 1 monclient: found mon.noname-c 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d237fe640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d14009a20 con 0x7f0d24067110 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d22ffd640 10 monclient: get_auth_request con 0x7f0d2409b010 auth_method 0 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d23fff640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d2409b400 0x7f0d2406db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.836+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2409b010 0x7f0d2406d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d237fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 msgr2=0x7f0d2412c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d237fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412c700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009de0 tx=0x7f0d14037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14031a20 con 0x7f0d24067110 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: dump: 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d14040070 con 0x7f0d24067110 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 5 monclient: authenticate success, global_id 194108 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 10 monclient: _renew_subs 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d24119820 con 0x7f0d24067110 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14031d90 con 0x7f0d24067110 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d0b7fe640 10 monclient: dump: 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 10 monclient: _renew_subs 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d24118930 con 0x7f0d24067110 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.838+0000 7f0d21ffb640 4 set_mon_vals no callback set 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0d14042030 con 0x7f0d24067110 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d29b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d29b1d640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 0x7f0d2406e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d29b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2406f770 con 0x7f0d24134100 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 0x7f0d2406e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient: get_auth_request con 0x7f0d24134100 auth_method 0 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d0b7fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0d10038790 0x7f0d1003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0d1406dcb0 con 0x7f0d24067110 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_done global_id 194111 payload 824 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d22ffd640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d2404bf60 con 0x7f0d24134100 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.839+0000 7f0d23fff640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0d10038790 0x7f0d1003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.840+0000 7f0d23fff640 10 monclient: get_auth_request con 0x7f0d10038790 auth_method 0 2024-09-15T03:43:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.841+0000 7f0d22ffd640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 msgr2=0x7f0d2406e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.841+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 0x7f0d2406e070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800d500 tx=0x7f0d18009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18005cb0 con 0x7f0d24134100 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 10 monclient: dump: 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0d18005e10 con 0x7f0d24134100 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 msgr2=0x7f0d2406e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 0x7f0d2406e070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800d500 tx=0x7f0d18009040 comp rx=0 tx=0).stop 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 msgr2=0x7f0d2406e070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d0b7fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24134100 0x7f0d2406e070 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800d500 tx=0x7f0d18009040 comp rx=0 tx=0).stop 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.842+0000 7f0d23fff640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0d10038790 0x7f0d1003ac40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c007fd0 tx=0x7f0d1c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d29b1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d29b1d640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d29b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24065f90 con 0x7f0d2412cc40 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient: get_auth_request con 0x7f0d2412cc40 auth_method 0 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 msgr2=0x7f0d2406e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 msgr2=0x7f0d2406e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:43:54.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:54.848+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:43:55.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient: get_auth_request con 0x7f0d2412cc40 auth_method 0 2024-09-15T03:43:55.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:43:55.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 10 monclient(hunting): handle_auth_done global_id 194117 payload 824 2024-09-15T03:43:55.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.049+0000 7f0d22ffd640 1 -- 172.21.15.153:0/1269856411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0d24119e70 con 0x7f0d2412cc40 2024-09-15T03:43:55.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.051+0000 7f0d22ffd640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 msgr2=0x7f0d2406e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.051+0000 7f0d22ffd640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800d500 tx=0x7f0d18019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.051+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18025070 con 0x7f0d2412cc40 2024-09-15T03:43:55.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.051+0000 7f0d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.051+0000 7f0d0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:55.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.051+0000 7f0d0b7fe640 10 monclient: dump: 2024-09-15T03:43:55.056 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.057 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.058 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.058 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.058 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f0d18012d10 con 0x7f0d2412cc40 2024-09-15T03:43:55.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:55.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:55.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 msgr2=0x7f0d2406e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800d500 tx=0x7f0d18019000 comp rx=0 tx=0).stop 2024-09-15T03:43:55.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 msgr2=0x7f0d2406e5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:55.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d0b7fe640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d2412cc40 0x7f0d2406e5b0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800d500 tx=0x7f0d18019000 comp rx=0 tx=0).stop 2024-09-15T03:43:55.060 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:55.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:55.060 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:55.060 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:55.060 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 152, 2024-09-15T03:43:55.061 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:55.061 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:55.061 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:55.061 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:55.061 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:55.062 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:55.063 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:55.064 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.064 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:55.064 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:55.064 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:55.064 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:55.065 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:55.066 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.066 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:55.066 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:55.066 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:55.066 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:55.067 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:55.067 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:55.067 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:55.067 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:55.067 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:55.068 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:55.068 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:55.068 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:55.068 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:55.068 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:55.068 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:55.069 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:55.069 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:55.069 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:55.069 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:55.069 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:55.069 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:55.070 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:55.070 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:55.071 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:55.071 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:55.071 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:55.071 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.071 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:55.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.072 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:55.072 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.072 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:55.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:55.072 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:55.072 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.074 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:55.076 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.077 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.078 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:55.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:55.081 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0d10038790 msgr2=0x7f0d1003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0d10038790 0x7f0d1003ac40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c007fd0 tx=0x7f0d1c007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 10 monclient: shutdown 2024-09-15T03:43:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 msgr2=0x7f0d2412c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 --2- 172.21.15.153:0/1269856411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d24067110 0x7f0d2412c700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009de0 tx=0x7f0d14037000 comp rx=0 tx=0).stop 2024-09-15T03:43:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d23fff640 1 -- 172.21.15.153:0/1269856411 reap_dead start 2024-09-15T03:43:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d23fff640 1 -- 172.21.15.153:0/1269856411 reap_dead start 2024-09-15T03:43:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 shutdown_connections 2024-09-15T03:43:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 >> 172.21.15.153:0/1269856411 conn(0x7f0d240a1e30 msgr2=0x7f0d2409e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 shutdown_connections 2024-09-15T03:43:55.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.052+0000 7f0d29b1d640 1 -- 172.21.15.153:0/1269856411 wait complete. 2024-09-15T03:43:55.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:55.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:43:55.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:43:55.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.178+0000 7fc6eb4dd640 10 monclient: get_monmap_and_config 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.178+0000 7fc6eb4dd640 10 monclient: build_initial_monmap 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.178+0000 7fc6eb4dd640 10 monclient: monmap: 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:55.179262+0000 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:55.179262+0000 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:55.180 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:55.181 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 Processor -- start 2024-09-15T03:43:55.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 -- start start 2024-09-15T03:43:55.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient: init 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6e41318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 0x7fc6e409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc6e4136c20 0x7fc6e4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e4136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e4132640 con 0x7fc6e41318f0 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e4131d00 con 0x7fc6e4136c20 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e409fbf0 con 0x7fc6e409ea50 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6eb4dd640 10 monclient(hunting): authenticate will time out at 1632.818767s 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e8a78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 0x7fc6e409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9279640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc6e41318f0 0x7fc6e4131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:59916/0 (socket says 172.21.15.153:59916) 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9279640 1 -- 172.21.15.153:0/1647408233 learned_addr learned my addr 172.21.15.153:0/1647408233 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e8a78640 10 monclient(hunting): get_auth_request con 0x7fc6e409ea50 auth_method 0 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e8a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e8a78640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e8a78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9a7a640 1 --2- 172.21.15.153:0/1647408233 >> v2:172.21.15.153:3300/0 conn(0x7fc6e4136c20 0x7fc6e4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9a7a640 10 monclient(hunting): get_auth_request con 0x7fc6e4136c20 auth_method 0 2024-09-15T03:43:55.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9a7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9a7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e9a7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.179+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e9a7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e9a7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e9a7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_done global_id 194120 payload 274 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 1 -- 172.21.15.153:0/1647408233 >> v2:172.21.15.153:3300/0 conn(0x7fc6e4136c20 msgr2=0x7fc6e4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 1 --2- 172.21.15.153:0/1647408233 >> v2:172.21.15.153:3300/0 conn(0x7fc6e4136c20 0x7fc6e4139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 1 -- 172.21.15.153:0/1647408233 >> v1:172.21.15.153:6789/0 conn(0x7fc6e41318f0 legacy=0x7fc6e4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 10 monclient: _finish_hunting 0 2024-09-15T03:43:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 1 monclient: found mon.noname-b 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e8a78640 1 -- 172.21.15.153:0/1647408233 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e409fe30 con 0x7fc6e409ea50 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.180+0000 7fc6e9a7a640 1 --2- 172.21.15.153:0/1647408233 >> v2:172.21.15.153:3300/0 conn(0x7fc6e4136c20 0x7fc6e4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.182+0000 7fc6e8a78640 1 -- 172.21.15.153:0/1647408233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 msgr2=0x7fc6e409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.182+0000 7fc6e8a78640 1 --2- 172.21.15.153:0/1647408233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 0x7fc6e409ee20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009850 tx=0x7fc6d40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad479de0bc804353 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 1 -- 172.21.15.153:0/1647408233 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d402adf0 con 0x7fc6e409ea50 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: dump: 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.188 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: _finish_auth 0 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 1 -- 172.21.15.153:0/1647408233 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4032440 con 0x7fc6e409ea50 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 5 monclient: authenticate success, global_id 194120 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:55.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 1 -- 172.21.15.153:0/1647408233 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d40327e0 con 0x7fc6e409ea50 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 4 set_mon_vals no callback set 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6dbfff640 10 monclient: dump: 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.190 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 10 monclient: shutdown 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/1647408233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 msgr2=0x7fc6e409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/1647408233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 0x7fc6e409ee20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009850 tx=0x7fc6d40090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/1647408233 shutdown_connections 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/1647408233 >> v2:172.21.15.153:3300/0 conn(0x7fc6e4136c20 0x7fc6e4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/1647408233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e409ea50 0x7fc6e409ee20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/1647408233 >> 172.21.15.153:0/1647408233 conn(0x7fc6e4067110 msgr2=0x7fc6e4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/1647408233 shutdown_connections 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.183+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/1647408233 wait complete. 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.184+0000 7fc6eb4dd640 10 monclient: build_initial_monmap 2024-09-15T03:43:55.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.184+0000 7fc6eb4dd640 10 monclient: monmap: 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.184+0000 7fc6eb4dd640 1 Processor -- start 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.184+0000 7fc6eb4dd640 1 -- start start 2024-09-15T03:43:55.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.184+0000 7fc6eb4dd640 10 monclient: init 2024-09-15T03:43:55.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409ea50 0x7fc6e4130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 0x7fc6e4099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e41318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e4136c20 0x7fc6e4099d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e4136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e411ca90 con 0x7fc6e4136c20 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e409b8e0 con 0x7fc6e409ea50 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e409bf10 con 0x7fc6e41318f0 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6eb4dd640 10 monclient(hunting): authenticate will time out at 1632.824178s 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 0x7fc6e4099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e4136c20 0x7fc6e4099d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 0x7fc6e4099810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34956/0 (socket says 172.21.15.153:34956) 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 -- 172.21.15.153:0/2896172769 learned_addr learned my addr 172.21.15.153:0/2896172769 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): get_auth_request con 0x7fc6e41318f0 auth_method 0 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): get_auth_request con 0x7fc6e4136c20 auth_method 0 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409ea50 0x7fc6e4130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): get_auth_request con 0x7fc6e409ea50 auth_method 0 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient(hunting): handle_auth_done global_id 194123 payload 824 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409ea50 msgr2=0x7fc6e4130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409ea50 0x7fc6e4130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e4136c20 msgr2=0x7fc6e4099d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e4136c20 0x7fc6e4099d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient: _finish_hunting 0 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 monclient: found mon.noname-c 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e8a78640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc009a20 con 0x7fc6e41318f0 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.185+0000 7fc6e9a7a640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e4136c20 0x7fc6e4099d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:55.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6e8a78640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 msgr2=0x7fc6e4099810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6e8a78640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 0x7fc6e4099810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009850 tx=0x7fc6d4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4031a40 con 0x7fc6e41318f0 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 10 monclient: dump: 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.202 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 10 monclient: _finish_auth 0 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4031ba0 con 0x7fc6e41318f0 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6eb4dd640 5 monclient: authenticate success, global_id 194123 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.187+0000 7fc6d97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d403d440 con 0x7fc6e41318f0 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 10 monclient: _renew_subs 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d4009390 con 0x7fc6e41318f0 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:55.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6db7fe640 4 set_mon_vals no callback set 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6d97fa640 10 monclient: dump: 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 10 monclient: _renew_subs 2024-09-15T03:43:55.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e4141eb0 con 0x7fc6e41318f0 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 0x7fc6e40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4065e00 con 0x7fc6e409aa50 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.188+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc6d4039040 con 0x7fc6e41318f0 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 0x7fc6e40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient: get_auth_request con 0x7fc6e409aa50 auth_method 0 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6d97fa640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6c8038740 0x7fc6c803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc6d406c0b0 con 0x7fc6e41318f0 2024-09-15T03:43:55.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9a7a640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6c8038740 0x7fc6c803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 10 monclient(hunting): handle_auth_done global_id 144535 payload 824 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9279640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6e404bf60 con 0x7fc6e409aa50 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.189+0000 7fc6e9a7a640 10 monclient: get_auth_request con 0x7fc6c8038740 auth_method 0 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6e9279640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 msgr2=0x7fc6e40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6e9279640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 0x7fc6e40298c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e710 tx=0x7fc6e0007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0004b50 con 0x7fc6e409aa50 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6e9a7a640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6c8038740 0x7fc6c803abf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40a0a20 tx=0x7fc6dc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.191+0000 7fc6d97fa640 10 monclient: dump: 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc6e0004cb0 con 0x7fc6e409aa50 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:55.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 msgr2=0x7fc6e40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 0x7fc6e40298c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e710 tx=0x7fc6e0007f00 comp rx=0 tx=0).stop 2024-09-15T03:43:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 msgr2=0x7fc6e40298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:55.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.192+0000 7fc6d97fa640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e409aa50 0x7fc6e40298c0 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e710 tx=0x7fc6e0007f00 comp rx=0 tx=0).stop 2024-09-15T03:43:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.197+0000 7fc6eb4dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.197+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 0x7fc6e4029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.197+0000 7fc6eb4dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.197+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e411ce20 con 0x7fc6e4130ec0 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 0x7fc6e4029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient: get_auth_request con 0x7fc6e4130ec0 auth_method 0 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 10 monclient(hunting): handle_auth_done global_id 144538 payload 824 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.198+0000 7fc6e9279640 1 -- 172.21.15.153:0/2896172769 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc6e411bba0 con 0x7fc6e4130ec0 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6e9279640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 msgr2=0x7fc6e4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6e9279640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 0x7fc6e4029e00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e710 tx=0x7fc6e0019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0015d30 con 0x7fc6e4130ec0 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 10 monclient: dump: 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.211 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fc6e0015e90 con 0x7fc6e4130ec0 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 msgr2=0x7fc6e4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 0x7fc6e4029e00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e710 tx=0x7fc6e0019000 comp rx=0 tx=0).stop 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 msgr2=0x7fc6e4029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6d97fa640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6e4130ec0 0x7fc6e4029e00 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e710 tx=0x7fc6e0019000 comp rx=0 tx=0).stop 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:55.212 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 152, 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:55.213 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:55.214 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:55.215 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:55.216 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.217 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.218 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.219 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:55.220 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.222 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:55.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:55.225 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:55.225 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:55.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6c8038740 msgr2=0x7fc6c803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc6c8038740 0x7fc6c803abf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40a0a20 tx=0x7fc6dc00b040 comp rx=0 tx=0).stop 2024-09-15T03:43:55.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6eb4dd640 10 monclient: shutdown 2024-09-15T03:43:55.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6eb4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:55.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 msgr2=0x7fc6e4099810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6eb4dd640 1 --2- 172.21.15.153:0/2896172769 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e41318f0 0x7fc6e4099810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009850 tx=0x7fc6d4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:55.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.201+0000 7fc6e9a7a640 1 -- 172.21.15.153:0/2896172769 reap_dead start 2024-09-15T03:43:55.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.202+0000 7fc6e9a7a640 1 -- 172.21.15.153:0/2896172769 reap_dead start 2024-09-15T03:43:55.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.202+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 shutdown_connections 2024-09-15T03:43:55.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.202+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 >> 172.21.15.153:0/2896172769 conn(0x7fc6e4067110 msgr2=0x7fc6e40a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:55.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.202+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 shutdown_connections 2024-09-15T03:43:55.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.202+0000 7fc6eb4dd640 1 -- 172.21.15.153:0/2896172769 wait complete. 2024-09-15T03:43:55.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:55.227 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:43:55.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.320+0000 7f77d8880640 10 monclient: get_monmap_and_config 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.320+0000 7f77d8880640 10 monclient: build_initial_monmap 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.320+0000 7f77d8880640 10 monclient: monmap: 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:55.320920+0000 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:55.320920+0000 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:55.322 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.320+0000 7f77d8880640 1 Processor -- start 2024-09-15T03:43:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.320+0000 7f77d8880640 1 -- start start 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.320+0000 7f77d8880640 10 monclient: init 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f77d4131ab0 0x7f77d4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): picked mon.noname-c con 0x7f77d4131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77d409e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 0x7f77d4138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): picked mon.noname-b con 0x7f77d4136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d409faf0 con 0x7f77d409e910 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d409fd30 con 0x7f77d4131ab0 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40a1110 con 0x7f77d4136bb0 2024-09-15T03:43:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d8880640 10 monclient(hunting): authenticate will time out at 1632.960433s 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 0x7f77d4138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 0x7f77d4138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34968/0 (socket says 172.21.15.153:34968) 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 -- 172.21.15.153:0/3833803167 learned_addr learned my addr 172.21.15.153:0/3833803167 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): get_auth_request con 0x7f77d4136bb0 auth_method 0 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d259c640 1 --2- 172.21.15.153:0/3833803167 >> v2:172.21.15.181:3300/0 conn(0x7f77d4131ab0 0x7f77d4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d259c640 10 monclient(hunting): get_auth_request con 0x7f77d4131ab0 auth_method 0 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient(hunting): handle_auth_done global_id 194126 payload 274 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 -- 172.21.15.153:0/3833803167 >> v2:172.21.15.181:3300/0 conn(0x7f77d4131ab0 msgr2=0x7f77d4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 --2- 172.21.15.153:0/3833803167 >> v2:172.21.15.181:3300/0 conn(0x7f77d4131ab0 0x7f77d4131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 -- 172.21.15.153:0/3833803167 >> v1:172.21.15.181:6789/0 conn(0x7f77d409e910 legacy=0x7f77d409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 monclient: found mon.noname-b 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.321+0000 7f77d2d9d640 1 -- 172.21.15.153:0/3833803167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40a13f0 con 0x7f77d4136bb0 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d2d9d640 1 -- 172.21.15.153:0/3833803167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 msgr2=0x7f77d4138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d2d9d640 1 --2- 172.21.15.153:0/3833803167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 0x7f77d4138f90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009910 tx=0x7f77c40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dadaeebccb772a9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 1 -- 172.21.15.153:0/3833803167 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4032440 con 0x7f77d4136bb0 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: dump: 2024-09-15T03:43:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.335 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: _finish_auth 0 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 1 -- 172.21.15.153:0/3833803167 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c40325a0 con 0x7f77d4136bb0 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d8880640 5 monclient: authenticate success, global_id 194126 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d8880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d8880640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 1 -- 172.21.15.153:0/3833803167 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4032980 con 0x7f77d4136bb0 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d8880640 4 set_mon_vals no callback set 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.324+0000 7f77d159a640 10 monclient: dump: 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 10 monclient: shutdown 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 -- 172.21.15.153:0/3833803167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 msgr2=0x7f77d4138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 --2- 172.21.15.153:0/3833803167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 0x7f77d4138f90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009910 tx=0x7f77c40091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 -- 172.21.15.153:0/3833803167 shutdown_connections 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 --2- 172.21.15.153:0/3833803167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d4136bb0 0x7f77d4138f90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 --2- 172.21.15.153:0/3833803167 >> v2:172.21.15.181:3300/0 conn(0x7f77d4131ab0 0x7f77d4131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 -- 172.21.15.153:0/3833803167 >> 172.21.15.153:0/3833803167 conn(0x7f77d4067d30 msgr2=0x7f77d40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 -- 172.21.15.153:0/3833803167 shutdown_connections 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 1 -- 172.21.15.153:0/3833803167 wait complete. 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 10 monclient: build_initial_monmap 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.325+0000 7f77d8880640 10 monclient: monmap: 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 Processor -- start 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 -- start start 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient: init 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 0x7f77d4135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): picked mon.noname-c con 0x7f77d409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4131ab0 0x7f77d412f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): picked mon.noname-a con 0x7f77d4131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77d4136bb0 0x7f77d412f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): picked mon.noname-b con 0x7f77d4136bb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d409da90 con 0x7f77d4131ab0 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4133eb0 con 0x7f77d4136bb0 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41331f0 con 0x7f77d409e910 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d8880640 10 monclient(hunting): authenticate will time out at 1632.965643s 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 0x7f77d4135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 0x7f77d4135350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34984/0 (socket says 172.21.15.153:34984) 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 1 -- 172.21.15.153:0/2600500671 learned_addr learned my addr 172.21.15.153:0/2600500671 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): get_auth_request con 0x7f77d409e910 auth_method 0 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4131ab0 0x7f77d412f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): get_auth_request con 0x7f77d4131ab0 auth_method 0 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d2d9d640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77d4136bb0 0x7f77d412f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.326+0000 7f77d259c640 10 monclient(hunting): handle_auth_done global_id 194129 payload 824 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 1 -- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77d4136bb0 msgr2=0x7f77d412f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77d4136bb0 0x7f77d412f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 1 -- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4131ab0 msgr2=0x7f77d412f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4131ab0 0x7f77d412f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 10 monclient: _finish_hunting 0 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 1 monclient: found mon.noname-c 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d259c640 1 -- 172.21.15.153:0/2600500671 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4009450 con 0x7f77d409e910 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d2d9d640 10 monclient: get_auth_request con 0x7f77d4136bb0 auth_method 0 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d2d9d640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77d4136bb0 0x7f77d412f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.327+0000 7f77d1d9b640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4131ab0 0x7f77d412f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d259c640 1 -- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 msgr2=0x7f77d4135350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d259c640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 0x7f77d4135350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00e190 tx=0x7f77cc009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc013590 con 0x7f77d409e910 2024-09-15T03:43:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: dump: 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 5 monclient: authenticate success, global_id 194129 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77cc013710 con 0x7f77d409e910 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc013ad0 con 0x7f77d409e910 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 10 monclient: _renew_subs 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d4130cf0 con 0x7f77d409e910 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d0d99640 4 set_mon_vals no callback set 2024-09-15T03:43:55.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77ca7fc640 10 monclient: dump: 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:55.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 10 monclient: _renew_subs 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.329+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d4130900 con 0x7f77d409e910 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f77cc01e460 con 0x7f77d409e910 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77ca7fc640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f77b4038790 0x7f77b403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77d8880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77d8880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77d8880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d4130af0 con 0x7f77d409e910 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f77cc04ce90 con 0x7f77d409e910 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77d1d9b640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f77b4038790 0x7f77b403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.330+0000 7f77d1d9b640 10 monclient: get_auth_request con 0x7f77b4038790 auth_method 0 2024-09-15T03:43:55.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.333+0000 7f77d1d9b640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f77b4038790 0x7f77b403ac40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f77bc007fd0 tx=0x7f77bc007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.334+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f77cc0128c0 con 0x7f77d409e910 2024-09-15T03:43:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.335+0000 7f77ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.335+0000 7f77ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.550+0000 7f77d8880640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:43:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.550+0000 7f77d8880640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:43:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.550+0000 7f77d8880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.550+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f77d404bf60 con 0x7f77d409e910 2024-09-15T03:43:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f77ca7fc640 1 -- 172.21.15.153:0/2600500671 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f77cc0122e0 con 0x7f77d409e910 2024-09-15T03:43:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f77ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:43:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f77ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:55.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=3381232033,config=3140514009,health=2002173999,kv=3235807718,logm=2544071823}) 2024-09-15T03:43:55.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=7,logm=52} crc {auth=785265808,config=3140514009,health=993608205,kv=1577269308,logm=1200679250}) 2024-09-15T03:43:55.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=3381232033,config=3140514009,health=2002173999,kv=3235807718,logm=2544071823}) 2024-09-15T03:43:55.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.551+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=785265808,config=3140514009,health=1538980112,kv=3235807718,logm=1848354946}) 2024-09-15T03:43:55.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200337283}) 2024-09-15T03:43:55.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=924754210}) 2024-09-15T03:43:55.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200337283}) 2024-09-15T03:43:55.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.552+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2920790724}) 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f77b4038790 msgr2=0x7f77b403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f77b4038790 0x7f77b403ac40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f77bc007fd0 tx=0x7f77bc007e10 comp rx=0 tx=0).stop 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 10 monclient: shutdown 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 msgr2=0x7f77d4135350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 0x7f77d4135350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00e190 tx=0x7f77cc009900 comp rx=0 tx=0).stop 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 shutdown_connections 2024-09-15T03:43:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f77b4038790 0x7f77b403ac40 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77d4136bb0 0x7f77d412f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4131ab0 0x7f77d412f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 --2- 172.21.15.153:0/2600500671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d409e910 0x7f77d4135350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:55.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 >> 172.21.15.153:0/2600500671 conn(0x7f77d4067d30 msgr2=0x7f77d412cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:55.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 shutdown_connections 2024-09-15T03:43:55.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:55.553+0000 7f77d8880640 1 -- 172.21.15.153:0/2600500671 wait complete. 2024-09-15T03:43:55.558 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802365430}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765463969}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802365430}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.554+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722827475}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053836208}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=115846112}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053836208}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.555+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2029446918}) 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.559 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098925724}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670509947}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098925724}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.556+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294714653}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750258612}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603044679}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750258612}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653362944}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573027894}) 2024-09-15T03:43:55.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=856925662}) 2024-09-15T03:43:55.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573027894}) 2024-09-15T03:43:55.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.557+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456865650}) 2024-09-15T03:43:55.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.558+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.558+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034370595}) 2024-09-15T03:43:55.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.558+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4267266720}) 2024-09-15T03:43:55.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.558+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.558+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034370595}) 2024-09-15T03:43:55.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.558+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561436337}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.559+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.559+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792953929}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.559+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456964678}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.559+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.559+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792953929}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.559+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961868215}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078774357}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190579212}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078774357}) 2024-09-15T03:43:55.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226054826}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.560+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996121165}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128536911}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996121165}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2089447670}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759634494}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2636486420}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759634494}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.561+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46034594}) 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.562+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.562+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179201069}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.562+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2824510386}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.562+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.562+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179201069}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.562+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=558054728}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193034676}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4036942904}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193034676}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4116932301}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695247756}) 2024-09-15T03:43:55.567 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476625153}) 2024-09-15T03:43:55.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695247756}) 2024-09-15T03:43:55.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.563+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140322272}) 2024-09-15T03:43:55.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.564+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.564+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039966607}) 2024-09-15T03:43:55.568 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.564+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234419297}) 2024-09-15T03:43:55.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.564+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.564+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039966607}) 2024-09-15T03:43:55.569 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.564+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393193440}) 2024-09-15T03:43:55.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:43:55.576 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.565+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.565+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306554212}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.565+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007782692}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.565+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.565+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306554212}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.565+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556638106}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339522254}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2904839870}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339522254}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=547608412}) 2024-09-15T03:43:55.577 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157108943}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115647886}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157108943}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.566+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122167889}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.567+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.567+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2578142808}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.567+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369843155}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.567+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.567+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2578142808}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.567+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445407509}) 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.578 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374857314}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854584522}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374857314}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367027175}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968248119}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001684809}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968248119}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.568+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747451450}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.569+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.569+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831057758}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.569+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4286905056}) 2024-09-15T03:43:55.579 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.569+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.569+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831057758}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.569+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1791968637}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675315219}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725605292}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675315219}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903318922}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147575813}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044879115}) 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.580 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147575813}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.570+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213835321}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780078495}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1284248822}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780078495}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343169831}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694971536}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921895558}) 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.581 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.571+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694971536}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081748482}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369494427}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258839372}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369494427}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.572+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941994153}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.573+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.573+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500492263}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.573+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1154001807}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.573+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.573+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500492263}) 2024-09-15T03:43:55.582 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.573+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3852165947}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812542779}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1662505346}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812542779}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652613733}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694654358}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4120102918}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694654358}) 2024-09-15T03:43:55.583 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.574+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514578937}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.575+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.575+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844549872}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.575+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248448719}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.575+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.575+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844549872}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.575+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090245976}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3805945653}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983960350}) 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.584 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3805945653}) 2024-09-15T03:43:55.585 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240966541}) 2024-09-15T03:43:55.586 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.586 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974839488}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3926425396}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974839488}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.576+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=168115363}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334638272}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943326629}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334638272}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3262156542}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60936992}) 2024-09-15T03:43:55.587 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345974642}) 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60936992}) 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.577+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483575828}) 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570853297}) 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529015146}) 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570853297}) 2024-09-15T03:43:55.588 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.578+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107930501}) 2024-09-15T03:43:55.823 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.822+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.823 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.822+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726266432}) 2024-09-15T03:43:55.823 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.822+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059951718}) 2024-09-15T03:43:55.823 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.822+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.824 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.822+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726266432}) 2024-09-15T03:43:55.824 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.822+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763784032}) 2024-09-15T03:43:55.824 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.823+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.824 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.823+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443859883}) 2024-09-15T03:43:55.824 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.823+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=184646396}) 2024-09-15T03:43:55.825 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.823+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.825 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.823+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443859883}) 2024-09-15T03:43:55.825 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.823+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1647334932}) 2024-09-15T03:43:55.826 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.824+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.826 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.824+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998410761}) 2024-09-15T03:43:55.826 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.824+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2150574442}) 2024-09-15T03:43:55.826 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.824+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.826 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.824+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998410761}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.824+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3569259043}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32850533}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3903961760}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32850533}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380308468}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437582396}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023291591}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437582396}) 2024-09-15T03:43:55.827 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.825+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046180703}) 2024-09-15T03:43:55.829 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.826+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.826+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880532533}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.826+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198121657}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.826+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.826+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880532533}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.826+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4170018305}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744205313}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110442795}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744205313}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3760886258}) 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.830 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367637504}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641342298}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367637504}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.827+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1882188514}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.828+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.828+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243555358}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.828+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1982757134}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.828+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.828+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243555358}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.828+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457335797}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716061930}) 2024-09-15T03:43:55.831 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061007955}) 2024-09-15T03:43:55.832 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.832 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716061930}) 2024-09-15T03:43:55.832 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2806008934}) 2024-09-15T03:43:55.835 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.835 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4126088739}) 2024-09-15T03:43:55.835 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590218151}) 2024-09-15T03:43:55.835 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.835 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4126088739}) 2024-09-15T03:43:55.835 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.829+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1317961730}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.830+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.830+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244346802}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.830+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168285614}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.830+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.830+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244346802}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.830+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147063987}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609491030}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3063635660}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609491030}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264870556}) 2024-09-15T03:43:55.836 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518274013}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3460371376}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518274013}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.831+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2685257446}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.832+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.832+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662838207}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.832+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245639472}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.832+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.832+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662838207}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.832+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672352033}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053892561}) 2024-09-15T03:43:55.837 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=324228865}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053892561}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3711045368}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621416348}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193988771}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621416348}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.833+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396242726}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.834+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.834+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748750285}) 2024-09-15T03:43:55.838 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.834+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503929935}) 2024-09-15T03:43:55.839 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.834+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.839 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.834+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748750285}) 2024-09-15T03:43:55.839 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.834+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018309787}) 2024-09-15T03:43:55.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650327729}) 2024-09-15T03:43:55.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3589307437}) 2024-09-15T03:43:55.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.843 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650327729}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095749568}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3817306266}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893352576}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3817306266}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.835+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992654259}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.836+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.836+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796712399}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.836+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2782572289}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.836+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.836+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796712399}) 2024-09-15T03:43:55.844 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.836+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126144421}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.837+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.837+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157325019}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.837+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=149999717}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.837+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.837+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157325019}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.837+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682144439}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467417726}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484758154}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467417726}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644568235}) 2024-09-15T03:43:55.845 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696148775}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013216977}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696148775}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.838+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31333289}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.839+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.839+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370623523}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.839+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927740208}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.839+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.839+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370623523}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.839+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149108924}) 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.840+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.846 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.840+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113170624}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.840+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300857695}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.840+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.840+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113170624}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.840+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2871055003}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890908523}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403335994}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890908523}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813137106}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596362713}) 2024-09-15T03:43:55.847 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1834867235}) 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596362713}) 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.841+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=313490284}) 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.842+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.842+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839303836}) 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.842+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978617028}) 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.842+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.842+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839303836}) 2024-09-15T03:43:55.848 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.842+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3580259972}) 2024-09-15T03:43:55.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.843+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.843+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291053449}) 2024-09-15T03:43:55.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.843+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2438597376}) 2024-09-15T03:43:55.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.843+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.843+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291053449}) 2024-09-15T03:43:55.852 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.843+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979132123}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103636627}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3163470738}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103636627}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697211364}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521403884}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3361097943}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521403884}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.844+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1276032692}) 2024-09-15T03:43:55.853 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.845+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.845+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895310388}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.845+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3297895135}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.845+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.845+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895310388}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.845+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463790586}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.846+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.846+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101809000}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.846+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024020038}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.846+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.846+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101809000}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.846+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269124984}) 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.854 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885528975}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2537370005}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885528975}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3819716449}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267189883}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1796981661}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267189883}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.847+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3997142991}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.848+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.848+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=554634624,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3830687212,mon_config_key=3725707551}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.848+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5} crc {logm=4253989489,mdsmap=2826658659,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=564259083,mon_config_key=4117641549}) 2024-09-15T03:43:55.855 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.848+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.848+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=554634624,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3830687212,mon_config_key=3725707551}) 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.848+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=4158530222,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3971553023,mon_config_key=753185158}) 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.850+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.850+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=1685524504,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=2536890333}) 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.850+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {mon_config_key=1685524504,monmap=244636553,osd_metadata=540015074,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3105462808}) 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.850+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.850+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=1685524504,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=2536890333}) 2024-09-15T03:43:55.856 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.850+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=1685524504,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1169988766}) 2024-09-15T03:43:55.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.856+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.856+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=3732344400}) 2024-09-15T03:43:55.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.856+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=18} crc {osdmap=2998009676}) 2024-09-15T03:43:55.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.856+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:43:55.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.856+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=3732344400}) 2024-09-15T03:43:55.857 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:43:55.856+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=18} crc {osdmap=2998009676}) 2024-09-15T03:43:56.571 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:43:56.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:43:56.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.669+0000 7f6c8595d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:56.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.669+0000 7f6c8595d640 10 monclient: build_initial_monmap 2024-09-15T03:43:56.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.669+0000 7f6c8595d640 10 monclient: monmap: 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:56.670224+0000 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:56.670224+0000 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:56.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 Processor -- start 2024-09-15T03:43:56.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 -- start start 2024-09-15T03:43:56.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient: init 2024-09-15T03:43:56.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:56.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T03:43:56.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c8009c860 addr v1:172.21.15.153:6789/0 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 0x7f6c8009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): picked mon.noname-a con 0x7f6c8009e840 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c80139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): picked mon.noname-b con 0x7f6c80136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8009ec40 con 0x7f6c8009c860 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8009a9d0 con 0x7f6c8009e840 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8006a320 con 0x7f6c80136cb0 2024-09-15T03:43:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c8595d640 10 monclient(hunting): authenticate will time out at 1634.309581s 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c80139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 0x7f6c8009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c80139090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34994/0 (socket says 172.21.15.153:34994) 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 0x7f6c8009aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42014/0 (socket says 172.21.15.153:42014) 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 1 -- 172.21.15.153:0/29706730 learned_addr learned my addr 172.21.15.153:0/29706730 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:56.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6c8009c860 0x7f6c8009cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:59924/0 (socket says 172.21.15.153:59924) 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6c80136cb0 auth_method 0 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): get_auth_request con 0x7f6c8009e840 auth_method 0 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_done global_id 194132 payload 274 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.670+0000 7f6c7f7fe640 1 -- 172.21.15.153:0/29706730 >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 msgr2=0x7f6c8009aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 1 --2- 172.21.15.153:0/29706730 >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 0x7f6c8009aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 1 -- 172.21.15.153:0/29706730 >> v1:172.21.15.153:6789/0 conn(0x7f6c8009c860 legacy=0x7f6c8009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 1 monclient: found mon.noname-b 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7f7fe640 1 -- 172.21.15.153:0/29706730 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80068a20 con 0x7f6c80136cb0 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.671+0000 7f6c7e7fc640 1 --2- 172.21.15.153:0/29706730 >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 0x7f6c8009aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7f7fe640 1 -- 172.21.15.153:0/29706730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 msgr2=0x7f6c80139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7f7fe640 1 --2- 172.21.15.153:0/29706730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c80139090 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009910 tx=0x7f6c700091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf6ca7298ef5f3c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 1 -- 172.21.15.153:0/29706730 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7002aea0 con 0x7f6c80136cb0 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: dump: 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:56.679 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: _finish_auth 0 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 1 -- 172.21.15.153:0/29706730 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c70032440 con 0x7f6c80136cb0 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c8595d640 5 monclient: authenticate success, global_id 194132 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c8595d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c8595d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 1 -- 172.21.15.153:0/29706730 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c700327e0 con 0x7f6c80136cb0 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c8595d640 4 set_mon_vals no callback set 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.673+0000 7f6c7dffb640 10 monclient: dump: 2024-09-15T03:43:56.680 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:56.681 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 10 monclient: shutdown 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 -- 172.21.15.153:0/29706730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 msgr2=0x7f6c80139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 --2- 172.21.15.153:0/29706730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c80139090 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009910 tx=0x7f6c700091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 -- 172.21.15.153:0/29706730 shutdown_connections 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 --2- 172.21.15.153:0/29706730 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c80139090 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 --2- 172.21.15.153:0/29706730 >> v2:172.21.15.153:3300/0 conn(0x7f6c8009e840 0x7f6c8009aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 -- 172.21.15.153:0/29706730 >> 172.21.15.153:0/29706730 conn(0x7f6c800a2e40 msgr2=0x7f6c800a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 -- 172.21.15.153:0/29706730 shutdown_connections 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.674+0000 7f6c8595d640 1 -- 172.21.15.153:0/29706730 wait complete. 2024-09-15T03:43:56.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient: build_initial_monmap 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient: monmap: 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:56.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 Processor -- start 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 -- start start 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient: init 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 0x7f6c80098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): picked mon.noname-a con 0x7f6c8009c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8009e840 0x7f6c80133070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): picked mon.noname-b con 0x7f6c8009e840 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c801335b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): picked mon.noname-c con 0x7f6c80136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8009e360 con 0x7f6c8009c860 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:56.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80135b20 con 0x7f6c8009e840 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80135560 con 0x7f6c80136cb0 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.675+0000 7f6c8595d640 10 monclient(hunting): authenticate will time out at 1634.315000s 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 0x7f6c80098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 0x7f6c80098b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42016/0 (socket says 172.21.15.153:42016) 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 -- 172.21.15.153:0/3778237480 learned_addr learned my addr 172.21.15.153:0/3778237480 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): get_auth_request con 0x7f6c8009c860 auth_method 0 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c801335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6c80136cb0 auth_method 0 2024-09-15T03:43:56.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient(hunting): handle_auth_done global_id 174354 payload 824 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7e7fc640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8009e840 0x7f6c80133070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 -- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 msgr2=0x7f6c801335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c801335b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 -- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8009e840 msgr2=0x7f6c80133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8009e840 0x7f6c80133070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient: _finish_hunting 0 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 monclient: found mon.noname-a 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7effd640 1 -- 172.21.15.153:0/3778237480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c68009390 con 0x7f6c8009c860 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7e7fc640 10 monclient: get_auth_request con 0x7f6c8009e840 auth_method 0 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.676+0000 7f6c7e7fc640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8009e840 0x7f6c80133070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c7effd640 1 -- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 msgr2=0x7f6c80098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c7effd640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 0x7f6c80098b00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800c930 tx=0x7f6c78004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800a5e0 con 0x7f6c8009c860 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 10 monclient: dump: 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 10 monclient: _finish_auth 0 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:56.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7800a740 con 0x7f6c8009c860 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c8595d640 5 monclient: authenticate success, global_id 174354 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c8595d640 10 monclient: _renew_subs 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c8595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.678+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c70009450 con 0x7f6c8009c860 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c76ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800ab00 con 0x7f6c8009c860 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:56.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c76ffd640 10 monclient: dump: 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c7d7fa640 4 set_mon_vals no callback set 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 10 monclient: _renew_subs 2024-09-15T03:43:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c80134360 con 0x7f6c8009c860 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6c780070f0 con 0x7f6c8009c860 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.679+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c80135d60 con 0x7f6c8009c860 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.680+0000 7f6c76ffd640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c60038740 0x7f6c6003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.680+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6c7804c410 con 0x7f6c8009c860 2024-09-15T03:43:56.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.680+0000 7f6c7e7fc640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c60038740 0x7f6c6003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:56.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.680+0000 7f6c7e7fc640 10 monclient: get_auth_request con 0x7f6c60038740 auth_method 0 2024-09-15T03:43:56.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.683+0000 7f6c7e7fc640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c60038740 0x7f6c6003abf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009360 tx=0x7f6c68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:56.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.684+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6c80135d60 con 0x7f6c8009c860 2024-09-15T03:43:56.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.684+0000 7f6c76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:56.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.684+0000 7f6c76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c8595d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:43:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c8595d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c8595d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6c801357a0 con 0x7f6c8009c860 2024-09-15T03:43:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c76ffd640 1 -- 172.21.15.153:0/3778237480 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f6c801357a0 con 0x7f6c8009c860 2024-09-15T03:43:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:43:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.896+0000 7f6c76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:56.898 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":152,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:43:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c60038740 msgr2=0x7f6c6003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c60038740 0x7f6c6003abf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009360 tx=0x7f6c68008040 comp rx=0 tx=0).stop 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 10 monclient: shutdown 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 msgr2=0x7f6c80098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 0x7f6c80098b00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800c930 tx=0x7f6c78004f70 comp rx=0 tx=0).stop 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 shutdown_connections 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f6c60038740 0x7f6c6003abf0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c80136cb0 0x7f6c801335b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8009e840 0x7f6c80133070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 --2- 172.21.15.153:0/3778237480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c8009c860 0x7f6c80098b00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 >> 172.21.15.153:0/3778237480 conn(0x7f6c800a2e40 msgr2=0x7f6c800a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 shutdown_connections 2024-09-15T03:43:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:56.898+0000 7f6c8595d640 1 -- 172.21.15.153:0/3778237480 wait complete. 2024-09-15T03:43:56.909 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:43:56.910 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:43:56.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:43:56.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:43:57.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.003+0000 7f16aa11d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:57.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.003+0000 7f16aa11d640 10 monclient: build_initial_monmap 2024-09-15T03:43:57.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.003+0000 7f16aa11d640 10 monclient: monmap: 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:57.004418+0000 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:57.004418+0000 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:57.005 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:57.006 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 Processor -- start 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 -- start start 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient: init 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 0x7f16a409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): picked mon.noname-b con 0x7f16a409c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16a409aba0 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 0x7f16a409df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): picked mon.noname-a con 0x7f16a409af70 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a406a320 con 0x7f16a409aba0 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a409a9d0 con 0x7f16a409af70 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4068a20 con 0x7f16a409c860 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.004+0000 7f16aa11d640 10 monclient(hunting): authenticate will time out at 1634.643939s 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 0x7f16a409cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 0x7f16a409df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 0x7f16a409cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35020/0 (socket says 172.21.15.153:35020) 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 0x7f16a409df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42026/0 (socket says 172.21.15.153:42026) 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 -- 172.21.15.153:0/3975108418 learned_addr learned my addr 172.21.15.153:0/3975108418 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): get_auth_request con 0x7f16a409c860 auth_method 0 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): get_auth_request con 0x7f16a409af70 auth_method 0 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient(hunting): handle_auth_done global_id 194138 payload 274 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 -- 172.21.15.153:0/3975108418 >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 msgr2=0x7f16a409df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 --2- 172.21.15.153:0/3975108418 >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 0x7f16a409df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 -- 172.21.15.153:0/3975108418 >> v1:172.21.15.181:6789/0 conn(0x7f16a409aba0 legacy=0x7f16a409d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:57.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 monclient: found mon.noname-b 2024-09-15T03:43:57.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:57.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:57.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a37fe640 1 -- 172.21.15.153:0/3975108418 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4068e60 con 0x7f16a409c860 2024-09-15T03:43:57.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.005+0000 7f16a3fff640 1 --2- 172.21.15.153:0/3975108418 >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 0x7f16a409df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:57.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a37fe640 1 -- 172.21.15.153:0/3975108418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 msgr2=0x7f16a409cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a37fe640 1 --2- 172.21.15.153:0/3975108418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 0x7f16a409cc30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1694009910 tx=0x7f16940091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98586498c509c70a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 1 -- 172.21.15.153:0/3975108418 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694032440 con 0x7f16a409c860 2024-09-15T03:43:57.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: dump: 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:57.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 1 -- 172.21.15.153:0/3975108418 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16940325a0 con 0x7f16a409c860 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 5 monclient: authenticate success, global_id 194138 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 4 set_mon_vals no callback set 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 1 -- 172.21.15.153:0/3975108418 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694032940 con 0x7f16a409c860 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16a27fc640 10 monclient: dump: 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.021 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 10 monclient: shutdown 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 -- 172.21.15.153:0/3975108418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 msgr2=0x7f16a409cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3975108418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 0x7f16a409cc30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1694009910 tx=0x7f16940091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 -- 172.21.15.153:0/3975108418 shutdown_connections 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3975108418 >> v2:172.21.15.153:3300/0 conn(0x7f16a409af70 0x7f16a409df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3975108418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409c860 0x7f16a409cc30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 -- 172.21.15.153:0/3975108418 >> 172.21.15.153:0/3975108418 conn(0x7f16a40a2e40 msgr2=0x7f16a40a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 -- 172.21.15.153:0/3975108418 shutdown_connections 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.008+0000 7f16aa11d640 1 -- 172.21.15.153:0/3975108418 wait complete. 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.009+0000 7f16aa11d640 10 monclient: build_initial_monmap 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.009+0000 7f16aa11d640 10 monclient: monmap: 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.023 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.009+0000 7f16aa11d640 1 Processor -- start 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.009+0000 7f16aa11d640 1 -- start start 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.009+0000 7f16aa11d640 10 monclient: init 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 0x7f16a4131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): picked mon.noname-c con 0x7f16a409aba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16a409af70 0x7f16a406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): picked mon.noname-b con 0x7f16a409af70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a409c860 0x7f16a406dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): picked mon.noname-a con 0x7f16a409c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4134ac0 con 0x7f16a409c860 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a411ecd0 con 0x7f16a409af70 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a411efb0 con 0x7f16a409aba0 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16aa11d640 10 monclient(hunting): authenticate will time out at 1634.649282s 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 0x7f16a4131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a409c860 0x7f16a406dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 0x7f16a4131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35032/0 (socket says 172.21.15.153:35032) 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 -- 172.21.15.153:0/3752604427 learned_addr learned my addr 172.21.15.153:0/3752604427 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:57.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): get_auth_request con 0x7f16a409aba0 auth_method 0 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): get_auth_request con 0x7f16a409c860 auth_method 0 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a2ffd640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16a409af70 0x7f16a406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient(hunting): handle_auth_done global_id 194141 payload 824 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16a409af70 msgr2=0x7f16a406d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16a409af70 0x7f16a406d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a409c860 msgr2=0x7f16a406dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a409c860 0x7f16a406dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 monclient: found mon.noname-c 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a37fe640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c009a20 con 0x7f16a409aba0 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a3fff640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a409c860 0x7f16a406dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a2ffd640 10 monclient: get_auth_request con 0x7f16a409af70 auth_method 0 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.010+0000 7f16a2ffd640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16a409af70 0x7f16a406d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.012+0000 7f16a37fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 msgr2=0x7f16a4131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.012+0000 7f16a37fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 0x7f16a4131b90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1694009910 tx=0x7f1694032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694031a40 con 0x7f16a409aba0 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: dump: 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.028 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 5 monclient: authenticate success, global_id 194141 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694031bc0 con 0x7f16a409aba0 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 10 monclient: _renew_subs 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1694009450 con 0x7f16a409aba0 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169403d440 con 0x7f16a409aba0 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f169b7fe640 10 monclient: dump: 2024-09-15T03:43:57.029 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16a1ffb640 4 set_mon_vals no callback set 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 10 monclient: _renew_subs 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:43:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a406f1d0 con 0x7f16a409aba0 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 0x7f16a406e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.013+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4065e00 con 0x7f16a41320d0 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1694039040 con 0x7f16a409aba0 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 0x7f16a406e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient: get_auth_request con 0x7f16a41320d0 auth_method 0 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_done global_id 174363 payload 824 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f169b7fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1684038740 0x7f168403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a2ffd640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a404bf60 con 0x7f16a41320d0 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f169406c2d0 con 0x7f16a409aba0 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a3fff640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1684038740 0x7f168403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.014+0000 7f16a3fff640 10 monclient: get_auth_request con 0x7f1684038740 auth_method 0 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f16a2ffd640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 msgr2=0x7f16a406e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f16a2ffd640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 0x7f16a406e1e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f16a406f860 tx=0x7f168c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c012810 con 0x7f16a41320d0 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 10 monclient: dump: 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f168c012970 con 0x7f16a41320d0 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 msgr2=0x7f16a406e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 0x7f16a406e1e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f16a406f860 tx=0x7f168c009040 comp rx=0 tx=0).stop 2024-09-15T03:43:57.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 msgr2=0x7f16a406e1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:57.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.016+0000 7f169b7fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a41320d0 0x7f16a406e1e0 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f16a406f860 tx=0x7f168c009040 comp rx=0 tx=0).stop 2024-09-15T03:43:57.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.017+0000 7f16a3fff640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1684038740 0x7f168403abf0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f169c00eb00 tx=0x7f169c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16aa11d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 0x7f16a4097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16aa11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a406eb20 con 0x7f16a406e720 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 0x7f16a4097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient: get_auth_request con 0x7f16a406e720 auth_method 0 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.022+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.023+0000 7f16a2ffd640 10 monclient(hunting): handle_auth_done global_id 174366 payload 824 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.023+0000 7f16a2ffd640 1 -- 172.21.15.153:0/3752604427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f16a41363c0 con 0x7f16a406e720 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16a2ffd640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 msgr2=0x7f16a4097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16a2ffd640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 0x7f16a4097a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f16a406f860 tx=0x7f168c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c021070 con 0x7f16a406e720 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 10 monclient: dump: 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f168c01b450 con 0x7f16a406e720 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 msgr2=0x7f16a4097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 0x7f16a4097a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f16a406f860 tx=0x7f168c01b000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 msgr2=0x7f16a4097a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f169b7fe640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16a406e720 0x7f16a4097a70 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f16a406f860 tx=0x7f168c01b000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 152, 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:57.039 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.041 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:57.042 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.044 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:57.045 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.047 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.049 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.050 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:43:57.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.051 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.051 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:57.051 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1684038740 msgr2=0x7f168403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1684038740 0x7f168403abf0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f169c00eb00 tx=0x7f169c00b040 comp rx=0 tx=0).stop 2024-09-15T03:43:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 10 monclient: shutdown 2024-09-15T03:43:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 msgr2=0x7f16a4131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 --2- 172.21.15.153:0/3752604427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16a409aba0 0x7f16a4131b90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1694009910 tx=0x7f1694032000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16a3fff640 1 -- 172.21.15.153:0/3752604427 reap_dead start 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16a3fff640 1 -- 172.21.15.153:0/3752604427 reap_dead start 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 shutdown_connections 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 >> 172.21.15.153:0/3752604427 conn(0x7f16a40a2e40 msgr2=0x7f16a4134cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 shutdown_connections 2024-09-15T03:43:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.025+0000 7f16aa11d640 1 -- 172.21.15.153:0/3752604427 wait complete. 2024-09-15T03:43:57.053 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:57.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:43:57.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:43:57.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.145+0000 7f17df86f640 10 monclient: get_monmap_and_config 2024-09-15T03:43:57.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.145+0000 7f17df86f640 10 monclient: build_initial_monmap 2024-09-15T03:43:57.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.145+0000 7f17df86f640 10 monclient: monmap: 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:57.146520+0000 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:57.146520+0000 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:57.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 1 Processor -- start 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 1 -- start start 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 10 monclient: init 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17d8131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f17d809fda0 0x7f17d809e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 10 monclient(hunting): picked mon.noname-a con 0x7f17d809fda0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f17d809ef00 0x7f17d8138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 10 monclient(hunting): picked mon.noname-c con 0x7f17d809ef00 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:57.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.146+0000 7f17df86f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d80a1180 con 0x7f17d8131980 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17df86f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d80a1460 con 0x7f17d809fda0 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17df86f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d8132460 con 0x7f17d809ef00 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17df86f640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17df86f640 10 monclient(hunting): authenticate will time out at 1634.786027s 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f17d809fda0 0x7f17d809e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f17d809fda0 0x7f17d809e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42046/0 (socket says 172.21.15.153:42046) 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 -- 172.21.15.153:0/2916610591 learned_addr learned my addr 172.21.15.153:0/2916610591 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): get_auth_request con 0x7f17d809fda0 auth_method 0 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 1 --2- 172.21.15.153:0/2916610591 >> v2:172.21.15.181:3300/0 conn(0x7f17d809ef00 0x7f17d8138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): get_auth_request con 0x7f17d809ef00 auth_method 0 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dde0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient(hunting): handle_auth_done global_id 174369 payload 274 2024-09-15T03:43:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 -- 172.21.15.153:0/2916610591 >> v2:172.21.15.181:3300/0 conn(0x7f17d809ef00 msgr2=0x7f17d8138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 --2- 172.21.15.153:0/2916610591 >> v2:172.21.15.181:3300/0 conn(0x7f17d809ef00 0x7f17d8138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 -- 172.21.15.153:0/2916610591 >> v1:172.21.15.181:6789/0 conn(0x7f17d8131980 legacy=0x7f17d8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient: _finish_hunting 0 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 monclient: found mon.noname-a 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.147+0000 7f17dce0a640 1 -- 172.21.15.153:0/2916610591 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d81325d0 con 0x7f17d809fda0 2024-09-15T03:43:57.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17dce0a640 1 -- 172.21.15.153:0/2916610591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d809fda0 msgr2=0x7f17d809e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17dce0a640 1 --2- 172.21.15.153:0/2916610591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d809fda0 0x7f17d809e9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f17d8068aa0 tx=0x7f17c80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56e81217e083f7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 1 -- 172.21.15.153:0/2916610591 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8032440 con 0x7f17d809fda0 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: dump: 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.163 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: _finish_auth 0 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 5 monclient: authenticate success, global_id 174369 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 1 -- 172.21.15.153:0/2916610591 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c80325a0 con 0x7f17d809fda0 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 1 -- 172.21.15.153:0/2916610591 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8032980 con 0x7f17d809fda0 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 4 set_mon_vals no callback set 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17cffff640 10 monclient: dump: 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 10 monclient: shutdown 2024-09-15T03:43:57.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 1 -- 172.21.15.153:0/2916610591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d809fda0 msgr2=0x7f17d809e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 1 --2- 172.21.15.153:0/2916610591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d809fda0 0x7f17d809e9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f17d8068aa0 tx=0x7f17c80091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 1 -- 172.21.15.153:0/2916610591 shutdown_connections 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 1 --2- 172.21.15.153:0/2916610591 >> v2:172.21.15.181:3300/0 conn(0x7f17d809ef00 0x7f17d8138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 1 --2- 172.21.15.153:0/2916610591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d809fda0 0x7f17d809e9c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.150+0000 7f17df86f640 1 -- 172.21.15.153:0/2916610591 >> 172.21.15.153:0/2916610591 conn(0x7f17d8067bd0 msgr2=0x7f17d80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 1 -- 172.21.15.153:0/2916610591 shutdown_connections 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 1 -- 172.21.15.153:0/2916610591 wait complete. 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 10 monclient: build_initial_monmap 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 10 monclient: monmap: 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:57.166 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 1 Processor -- start 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 1 -- start start 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.151+0000 7f17df86f640 10 monclient: init 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d809ef00 0x7f17d8133f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): picked mon.noname-c con 0x7f17d809ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d809fda0 0x7f17d81344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): picked mon.noname-b con 0x7f17d809fda0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d8131980 0x7f17d81349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): picked mon.noname-a con 0x7f17d8131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8131560 con 0x7f17d8131980 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8135fd0 con 0x7f17d809fda0 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d81375d0 con 0x7f17d809ef00 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17df86f640 10 monclient(hunting): authenticate will time out at 1634.791517s 2024-09-15T03:43:57.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d809ef00 0x7f17d8133f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d8131980 0x7f17d81349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d809ef00 0x7f17d8133f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35040/0 (socket says 172.21.15.153:35040) 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 1 -- 172.21.15.153:0/3761375093 learned_addr learned my addr 172.21.15.153:0/3761375093 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): get_auth_request con 0x7f17d809ef00 auth_method 0 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): get_auth_request con 0x7f17d8131980 auth_method 0 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d809fda0 0x7f17d81344a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): get_auth_request con 0x7f17d809fda0 auth_method 0 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dce0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient(hunting): handle_auth_done global_id 174372 payload 824 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d809ef00 msgr2=0x7f17d8133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d809ef00 0x7f17d8133f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d809fda0 msgr2=0x7f17d81344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d809fda0 0x7f17d81344a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient: _finish_hunting 0 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 monclient: found mon.noname-a 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dde0c640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8009450 con 0x7f17d8131980 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.152+0000 7f17dd60b640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d809ef00 0x7f17d8133f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17dde0c640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d8131980 msgr2=0x7f17d81349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17dde0c640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d8131980 0x7f17d81349e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009e50 tx=0x7f17d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0014480 con 0x7f17d8131980 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: dump: 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d000f040 con 0x7f17d8131980 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 5 monclient: authenticate success, global_id 174372 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 10 monclient: _renew_subs 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d809bf10 con 0x7f17d8131980 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0014830 con 0x7f17d8131980 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cd7fa640 10 monclient: dump: 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17cf7fe640 4 set_mon_vals no callback set 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 10 monclient: _renew_subs 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.155+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d8135b20 con 0x7f17d8131980 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17df86f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:57.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17df86f640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 0x7f17d8098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8065e00 con 0x7f17d8134f20 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f17d0014ba0 con 0x7f17d8131980 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 0x7f17d8098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient: get_auth_request con 0x7f17d8134f20 auth_method 0 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17cd7fa640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17bc038740 0x7f17bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f17d004d600 con 0x7f17d8131980 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 10 monclient(hunting): handle_auth_done global_id 194147 payload 824 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dd60b640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17d8137940 con 0x7f17d8134f20 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dce0a640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17bc038740 0x7f17bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.156+0000 7f17dce0a640 10 monclient: get_auth_request con 0x7f17bc038740 auth_method 0 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.158+0000 7f17dd60b640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 msgr2=0x7f17d8098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.158+0000 7f17dd60b640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 0x7f17d8098a70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17d400fbb0 tx=0x7f17d4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.158+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d400de70 con 0x7f17d8134f20 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.158+0000 7f17cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.158+0000 7f17cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.158+0000 7f17cd7fa640 10 monclient: dump: 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.176 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f17d4014470 con 0x7f17d8134f20 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 msgr2=0x7f17d8098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 0x7f17d8098a70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17d400fbb0 tx=0x7f17d4007ed0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 msgr2=0x7f17d8098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17cd7fa640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8134f20 0x7f17d8098a70 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17d400fbb0 tx=0x7f17d4007ed0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.159+0000 7f17dce0a640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17bc038740 0x7f17bc03abf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f17c8034000 tx=0x7f17c8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17df86f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17df86f640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 0x7f17d80993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17df86f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d804bf60 con 0x7f17d8098fb0 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17dd60b640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 0x7f17d80993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17dd60b640 10 monclient: get_auth_request con 0x7f17d8098fb0 auth_method 0 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17dd60b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17dd60b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.164+0000 7f17dd60b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.165+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.165+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.165+0000 7f17dd60b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.165+0000 7f17dd60b640 10 monclient(hunting): handle_auth_done global_id 194150 payload 824 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.165+0000 7f17dd60b640 1 -- 172.21.15.153:0/3761375093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f17d81352f0 con 0x7f17d8098fb0 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17dd60b640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 msgr2=0x7f17d80993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17dd60b640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 0x7f17d80993b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f17d400fbb0 tx=0x7f17d401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d401ed50 con 0x7f17d8098fb0 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 10 monclient: dump: 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f17d4007020 con 0x7f17d8098fb0 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 msgr2=0x7f17d80993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 0x7f17d80993b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f17d400fbb0 tx=0x7f17d401a000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 msgr2=0x7f17d80993b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.167+0000 7f17cd7fa640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17d8098fb0 0x7f17d80993b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 152, 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:57.183 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.184 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.185 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:57.186 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.187 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:57.189 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:57.193 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:57.194 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.195 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.195 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:57.195 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:57.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17bc038740 msgr2=0x7f17bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f17bc038740 0x7f17bc03abf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f17c8034000 tx=0x7f17c8032000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 10 monclient: shutdown 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d8131980 msgr2=0x7f17d81349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 --2- 172.21.15.153:0/3761375093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17d8131980 0x7f17d81349e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009e50 tx=0x7f17d000b040 comp rx=0 tx=0).stop 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17dde0c640 1 -- 172.21.15.153:0/3761375093 reap_dead start 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17dde0c640 1 -- 172.21.15.153:0/3761375093 reap_dead start 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 shutdown_connections 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 >> 172.21.15.153:0/3761375093 conn(0x7f17d8067bd0 msgr2=0x7f17d80a48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 shutdown_connections 2024-09-15T03:43:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.168+0000 7f17df86f640 1 -- 172.21.15.153:0/3761375093 wait complete. 2024-09-15T03:43:57.197 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:57.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:43:57.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:43:57.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.291+0000 7fbac159d640 10 monclient: get_monmap_and_config 2024-09-15T03:43:57.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.291+0000 7fbac159d640 10 monclient: build_initial_monmap 2024-09-15T03:43:57.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.291+0000 7fbac159d640 10 monclient: monmap: 2024-09-15T03:43:57.292 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:43:57.291876+0000 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:43:57.291876+0000 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.291+0000 7fbac159d640 1 Processor -- start 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.291+0000 7fbac159d640 1 -- start start 2024-09-15T03:43:57.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.291+0000 7fbac159d640 10 monclient: init 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbabc1318f0 0x7fbabc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): picked mon.noname-a con 0x7fbabc1318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 0x7fbabc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): picked mon.noname-b con 0x7fbabc09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbabc068d90 0x7fbabc0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): picked mon.noname-c con 0x7fbabc068d90 addr v2:172.21.15.181:3300/0 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbabc131cc0 con 0x7fbabc1318f0 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbabc09fbf0 con 0x7fbabc068d90 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc09fe30 con 0x7fbabc09ea50 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbac159d640 10 monclient(hunting): authenticate will time out at 1634.931254s 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 0x7fbabc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbabc1318f0 0x7fbabc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 0x7fbabc09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35058/0 (socket says 172.21.15.153:35058) 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 -- 172.21.15.153:0/2985481209 learned_addr learned my addr 172.21.15.153:0/2985481209 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): get_auth_request con 0x7fbabc09ea50 auth_method 0 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): get_auth_request con 0x7fbabc1318f0 auth_method 0 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabb7fe640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.181:3300/0 conn(0x7fbabc068d90 0x7fbabc0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient(hunting): handle_auth_done global_id 194153 payload 274 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 -- 172.21.15.153:0/2985481209 >> v2:172.21.15.181:3300/0 conn(0x7fbabc068d90 msgr2=0x7fbabc0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.181:3300/0 conn(0x7fbabc068d90 0x7fbabc0691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 -- 172.21.15.153:0/2985481209 >> v2:172.21.15.153:3300/0 conn(0x7fbabc1318f0 msgr2=0x7fbabc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.153:3300/0 conn(0x7fbabc1318f0 0x7fbabc131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 monclient: found mon.noname-b 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbab27fc640 1 -- 172.21.15.153:0/2985481209 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc0a1210 con 0x7fbabc09ea50 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabb7fe640 10 monclient: get_auth_request con 0x7fbabc068d90 auth_method 0 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabaffd640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.153:3300/0 conn(0x7fbabc1318f0 0x7fbabc131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.292+0000 7fbabb7fe640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.181:3300/0 conn(0x7fbabc068d90 0x7fbabc0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:43:57.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbab27fc640 1 -- 172.21.15.153:0/2985481209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 msgr2=0x7fbabc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbab27fc640 1 --2- 172.21.15.153:0/2985481209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 0x7fbabc09ee20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbabc064050 tx=0x7fbaa40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec95c45606c687ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 1 -- 172.21.15.153:0/2985481209 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4032440 con 0x7fbabc09ea50 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: dump: 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.301 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: _finish_auth 0 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 1 -- 172.21.15.153:0/2985481209 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa40325a0 con 0x7fbabc09ea50 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbac159d640 5 monclient: authenticate success, global_id 194153 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbac159d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbac159d640 10 monclient: get_monmap_and_config success 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 1 -- 172.21.15.153:0/2985481209 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4032940 con 0x7fbabc09ea50 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbac159d640 4 set_mon_vals no callback set 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.295+0000 7fbaba7fc640 10 monclient: dump: 2024-09-15T03:43:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 10 monclient: shutdown 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 -- 172.21.15.153:0/2985481209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 msgr2=0x7fbabc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 --2- 172.21.15.153:0/2985481209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 0x7fbabc09ee20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbabc064050 tx=0x7fbaa40091d0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 -- 172.21.15.153:0/2985481209 shutdown_connections 2024-09-15T03:43:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.181:3300/0 conn(0x7fbabc068d90 0x7fbabc0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 --2- 172.21.15.153:0/2985481209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc09ea50 0x7fbabc09ee20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 --2- 172.21.15.153:0/2985481209 >> v2:172.21.15.153:3300/0 conn(0x7fbabc1318f0 0x7fbabc131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 -- 172.21.15.153:0/2985481209 >> 172.21.15.153:0/2985481209 conn(0x7fbabc067110 msgr2=0x7fbabc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 -- 172.21.15.153:0/2985481209 shutdown_connections 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 1 -- 172.21.15.153:0/2985481209 wait complete. 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 10 monclient: build_initial_monmap 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.296+0000 7fbac159d640 10 monclient: monmap: 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.304 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:43:57.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:43:57.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:43:57.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 Processor -- start 2024-09-15T03:43:57.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 -- start start 2024-09-15T03:43:57.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient: init 2024-09-15T03:43:57.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient: _reopen_session rank -1 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc068d90 0x7fbabc09ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): picked mon.noname-b con 0x7fbabc068d90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc09ea50 0x7fbabc130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): picked mon.noname-a con 0x7fbabc09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc1318f0 0x7fbabc140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): picked mon.noname-c con 0x7fbabc1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc09bf80 con 0x7fbabc09ea50 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc11e0e0 con 0x7fbabc068d90 2024-09-15T03:43:57.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0a4390 con 0x7fbabc1318f0 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): _renew_subs 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbac159d640 10 monclient(hunting): authenticate will time out at 1634.936689s 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc1318f0 0x7fbabc140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc09ea50 0x7fbabc130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc1318f0 0x7fbabc140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35062/0 (socket says 172.21.15.153:35062) 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 1 -- 172.21.15.153:0/2791794031 learned_addr learned my addr 172.21.15.153:0/2791794031 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): get_auth_request con 0x7fbabc09ea50 auth_method 0 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): get_auth_request con 0x7fbabc1318f0 auth_method 0 2024-09-15T03:43:57.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc068d90 0x7fbabc09ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): get_auth_request con 0x7fbabc068d90 auth_method 0 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbab27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.297+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 10 monclient(hunting): handle_auth_done global_id 174378 payload 824 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc1318f0 msgr2=0x7fbabc140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc1318f0 0x7fbabc140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc068d90 msgr2=0x7fbabc09ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc068d90 0x7fbabc09ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 1 monclient: found mon.noname-a 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbab27fc640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac009390 con 0x7fbabc09ea50 2024-09-15T03:43:57.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.298+0000 7fbabb7fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbabc1318f0 0x7fbabc140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab27fc640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc09ea50 msgr2=0x7fbabc130ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab27fc640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc09ea50 0x7fbabc130ce0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4031000 tx=0x7fbaa4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4031a90 con 0x7fbabc09ea50 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: dump: 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: _finish_auth 0 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 5 monclient: authenticate success, global_id 174378 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 10 monclient: _renew_subs 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa4031bf0 con 0x7fbabc09ea50 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa4009450 con 0x7fbabc09ea50 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa403d440 con 0x7fbabc09ea50 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab37fe640 10 monclient: dump: 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbab9ffb640 4 set_mon_vals no callback set 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 10 monclient: _renew_subs 2024-09-15T03:43:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:43:57.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.300+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc11de00 con 0x7fbabc09ea50 2024-09-15T03:43:57.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbaa4039040 con 0x7fbabc09ea50 2024-09-15T03:43:57.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbac159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:57.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbac159d640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 0x7fbabc1413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc065e00 con 0x7fbabc131220 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbab37fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba9c038740 0x7fba9c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbaa406c120 con 0x7fbabc09ea50 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbabaffd640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 0x7fbabc1413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbabaffd640 10 monclient: get_auth_request con 0x7fbabc131220 auth_method 0 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbabaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.301+0000 7fbabb7fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba9c038740 0x7fba9c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.302+0000 7fbabb7fe640 10 monclient: get_auth_request con 0x7fba9c038740 auth_method 0 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.302+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.302+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.302+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.302+0000 7fbabaffd640 10 monclient(hunting): handle_auth_done global_id 144553 payload 824 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.302+0000 7fbabaffd640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbabc11c850 con 0x7fbabc131220 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbabb7fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba9c038740 0x7fba9c03abf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbabc11cc30 tx=0x7fbab4007860 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbabaffd640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 msgr2=0x7fbabc1413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbabaffd640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 0x7fbabc1413e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbaac011bf0 tx=0x7fbaac0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00fe90 con 0x7fbabc131220 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbab37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.304+0000 7fbab37fe640 10 monclient: dump: 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbaac014650 con 0x7fbabc131220 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:43:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:43:57.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 msgr2=0x7fbabc1413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 0x7fbabc1413e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbaac011bf0 tx=0x7fbaac0090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 msgr2=0x7fbabc1413e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:57.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.305+0000 7fbab37fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc131220 0x7fbabc1413e0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbaac011bf0 tx=0x7fbaac0090f0 comp rx=0 tx=0).stop 2024-09-15T03:43:57.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.310+0000 7fbac159d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.310+0000 7fbac159d640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 0x7fbabc096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.310+0000 7fbac159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.310+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc1315f0 con 0x7fbabc141920 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 0x7fbabc096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient: get_auth_request con 0x7fbabc141920 auth_method 0 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 10 monclient(hunting): handle_auth_done global_id 144556 payload 824 2024-09-15T03:43:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.311+0000 7fbabaffd640 1 -- 172.21.15.153:0/2791794031 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbabc141d20 con 0x7fbabc141920 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.313+0000 7fbabaffd640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 msgr2=0x7fbabc096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.313+0000 7fbabaffd640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 0x7fbabc096a60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbaac011bf0 tx=0x7fbaac01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.313+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac025040 con 0x7fbabc141920 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.313+0000 7fbab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.313+0000 7fbab37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.313+0000 7fbab37fe640 10 monclient: dump: 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:43:57.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fbaac01c3e0 con 0x7fbabc141920 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 msgr2=0x7fbabc096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 0x7fbabc096a60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbaac011bf0 tx=0x7fbaac01d000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 msgr2=0x7fbabc096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbab37fe640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbabc141920 0x7fbabc096a60 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbaac011bf0 tx=0x7fbaac01d000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.326 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 152, 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:43:57.327 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.328 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:43:57.329 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:43:57.330 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:43:57.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:43:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:43:57.334 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:43:57.335 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:43:57.336 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:57.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.338 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:43:57.339 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba9c038740 msgr2=0x7fba9c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba9c038740 0x7fba9c03abf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbabc11cc30 tx=0x7fbab4007860 comp rx=0 tx=0).stop 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 10 monclient: shutdown 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc09ea50 msgr2=0x7fbabc130ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 --2- 172.21.15.153:0/2791794031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbabc09ea50 0x7fbabc130ce0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4031000 tx=0x7fbaa4032000 comp rx=0 tx=0).stop 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbabb7fe640 1 -- 172.21.15.153:0/2791794031 reap_dead start 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbabb7fe640 1 -- 172.21.15.153:0/2791794031 reap_dead start 2024-09-15T03:43:57.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 shutdown_connections 2024-09-15T03:43:57.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 >> 172.21.15.153:0/2791794031 conn(0x7fbabc067110 msgr2=0x7fbabc0a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:43:57.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 shutdown_connections 2024-09-15T03:43:57.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:43:57.314+0000 7fbac159d640 1 -- 172.21.15.153:0/2791794031 wait complete. 2024-09-15T03:43:57.342 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:43:57.342 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:43:57.342 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-15T03:43:57.342 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:43:57.342 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:43:57.343 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:43:57.343 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:57.363 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:43:57.363 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T03:43:57.364 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T03:43:57.364 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:44:12.365 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T03:44:12.366 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T03:44:12.366 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:44:12.366 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:44:12.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.466+0000 7ff8550c0640 10 monclient: get_monmap_and_config 2024-09-15T03:44:12.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.466+0000 7ff8550c0640 10 monclient: build_initial_monmap 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.466+0000 7ff8550c0640 10 monclient: monmap: 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:44:12.467637+0000 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:44:12.467637+0000 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:44:12.468 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:44:12.469 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:44:12.469 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:12.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 1 Processor -- start 2024-09-15T03:44:12.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 1 -- start start 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 10 monclient: init 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 0x7ff850098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff8500981c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:12.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff850131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:12.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.467+0000 7ff8550c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff850132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:44:12.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff850098ad0 con 0x7ff850132170 2024-09-15T03:44:12.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:12.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff850131bd0 con 0x7ff8500981c0 2024-09-15T03:44:12.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:12.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85009ca60 con 0x7ff850131da0 2024-09-15T03:44:12.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:12.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff8550c0640 10 monclient(hunting): authenticate will time out at 1650.107032s 2024-09-15T03:44:12.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84f59e640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7ff850132170 legacy=0x7ff850067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:44:12.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84ed9d640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 msgr2=0x7ff850098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:12.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84ed9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 0x7ff850098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:12.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35384/0 (socket says 172.21.15.153:35384) 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 learned_addr learned my addr 172.21.15.153:0/2407679400 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 10 monclient(hunting): get_auth_request con 0x7ff850131da0 auth_method 0 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:12.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.468+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:12.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:12.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 10 monclient(hunting): get_auth_request con 0x7ff850131da0 auth_method 0 2024-09-15T03:44:12.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:12.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:12.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 msgr2=0x7ff850098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84f59e640 1 -- 172.21.15.153:0/2407679400 >> v1:172.21.15.181:6789/0 conn(0x7ff850132170 legacy=0x7ff850067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:44:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 0x7ff850098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:44:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:12.669+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 10 monclient(hunting): get_auth_request con 0x7ff850131da0 auth_method 0 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 msgr2=0x7ff850098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84f59e640 1 -- 172.21.15.153:0/2407679400 >> v1:172.21.15.181:6789/0 conn(0x7ff850132170 legacy=0x7ff850067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:44:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 0x7ff850098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:44:13.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:44:13.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:13.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:13.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.070+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:44:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 msgr2=0x7ff850098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84f59e640 1 -- 172.21.15.153:0/2407679400 >> v1:172.21.15.181:6789/0 conn(0x7ff850132170 legacy=0x7ff850067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:44:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 0x7ff850098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 10 monclient(hunting): get_auth_request con 0x7ff850131da0 auth_method 0 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:13.871+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:44:15.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient(hunting): tick 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 monclient(hunting): continuing hunt 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 msgr2=0x7ff850067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff850131da0 0x7ff850067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 msgr2=0x7ff850098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.181:3300/0 conn(0x7ff8500981c0 0x7ff850098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 -- 172.21.15.153:0/2407679400 >> v1:172.21.15.181:6789/0 conn(0x7ff850132170 legacy=0x7ff850067650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient: _add_conns ranks=[3,0] 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient(hunting): picked mon.noname-a con 0x7ff83c003680 addr v2:172.21.15.153:3300/0 2024-09-15T03:44:15.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff83c0061c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 -- 172.21.15.153:0/2407679400 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85009cca0 con 0x7ff83c0061c0 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 1 -- 172.21.15.153:0/2407679400 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff850131bd0 con 0x7ff83c003680 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84cd99640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request con 0x7ff83c003680 auth_method 0 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:15.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.468+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:15.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.470+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> v1:172.21.15.153:6789/0 conn(0x7ff83c0061c0 legacy=0x7ff83c008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request con 0x7ff83c003680 auth_method 0 2024-09-15T03:44:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:15.669+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:44:16.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:16.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request con 0x7ff83c003680 auth_method 0 2024-09-15T03:44:16.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:16.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:16.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:16.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:16.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:16.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:16.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.070+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:44:16.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:16.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request con 0x7ff83c003680 auth_method 0 2024-09-15T03:44:16.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:16.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:16.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:16.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:16.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:16.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:16.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:16.871+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988297817 0 0) 0x7ff85009cca0 con 0x7ff83c0061c0 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient(hunting): my global_id is 174381 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8380034e0 con 0x7ff83c0061c0 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286050167 0 0) 0x7ff8380034e0 con 0x7ff83c0061c0 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 msgr2=0x7ff83c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 --2- 172.21.15.153:0/2407679400 >> v2:172.21.15.153:3300/0 conn(0x7ff83c003680 0x7ff83c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:44:17.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850098ad0 con 0x7ff83c0061c0 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7ff844007540 con 0x7ff83c0061c0 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff8550c0640 5 monclient: authenticate success, global_id 174381 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff8550c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: dump: 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:44:17.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 >> v1:172.21.15.153:6789/0 conn(0x7ff83c0061c0 legacy=0x7ff83c008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84f59e640 1 -- 172.21.15.153:0/2407679400 reap_dead start 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.388+0000 7ff84dd9b640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380039d0 0x7ff838003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): picked mon.c con 0x7ff8380039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 0x7ff838004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): picked mon.a con 0x7ff838004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8380051a0 0x7ff83800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): picked mon.b con 0x7ff8380051a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850131bd0 con 0x7ff838004780 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83800c170 con 0x7ff8380051a0 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83800c330 con 0x7ff8380039d0 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff8550c0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84f59e640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8380051a0 msgr2=0x7ff83800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84f59e640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8380051a0 0x7ff83800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 0x7ff838004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): get_auth_request con 0x7ff838004780 auth_method 0 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380039d0 0x7ff838003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request con 0x7ff8380039d0 auth_method 0 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient(hunting): handle_auth_done global_id 174381 payload 293 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380039d0 msgr2=0x7ff838003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380039d0 0x7ff838003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8380051a0 msgr2=0x7ff83800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8380051a0 0x7ff83800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 monclient: found mon.a 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8380034e0 con 0x7ff838004780 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.389+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380039d0 0x7ff838003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:44:17.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84e59c640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 msgr2=0x7ff838004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84e59c640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 0x7ff838004bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff844009650 tx=0x7ff844009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdb3953a5b70b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400b0d0 con 0x7ff838004780 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: dump: 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff8550c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84402e050 con 0x7ff838004780 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff8550c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 1 -- 172.21.15.153:0/2407679400 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400b4a0 con 0x7ff838004780 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff8550c0640 4 set_mon_vals no callback set 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.392+0000 7ff84dd9b640 10 monclient: dump: 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.405 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 10 monclient: shutdown 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 -- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 msgr2=0x7ff838004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 0x7ff838004bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff844009650 tx=0x7ff844009040 comp rx=0 tx=0).stop 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 -- 172.21.15.153:0/2407679400 shutdown_connections 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8380051a0 0x7ff83800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff838004780 0x7ff838004bf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 --2- 172.21.15.153:0/2407679400 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380039d0 0x7ff838003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 -- 172.21.15.153:0/2407679400 >> 172.21.15.153:0/2407679400 conn(0x7ff8500a1e30 msgr2=0x7ff8500a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 -- 172.21.15.153:0/2407679400 shutdown_connections 2024-09-15T03:44:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.393+0000 7ff8550c0640 1 -- 172.21.15.153:0/2407679400 wait complete. 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.394+0000 7ff8550c0640 10 monclient: build_initial_monmap 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.394+0000 7ff8550c0640 10 monclient: monmap: 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:44:17.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.394+0000 7ff8550c0640 1 Processor -- start 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 -- start start 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient: init 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 0x7ff850140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff8380039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff838004780 0x7ff850130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff838004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 0x7ff8501311f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff8380051a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850098cb0 con 0x7ff8380039d0 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85009b7b0 con 0x7ff838004780 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85011c310 con 0x7ff8380051a0 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.395+0000 7ff8550c0640 10 monclient(hunting): authenticate will time out at 1655.034906s 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 0x7ff8501311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84e59c640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff838004780 msgr2=0x7ff850130cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 0x7ff850140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff838004780 0x7ff850130cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 0x7ff850140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39126/0 (socket says 172.21.15.153:39126) 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): get_auth_request con 0x7ff8380051a0 auth_method 0 2024-09-15T03:44:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 1 -- 172.21.15.153:0/956198132 learned_addr learned my addr 172.21.15.153:0/956198132 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request con 0x7ff8380039d0 auth_method 0 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient(hunting): handle_auth_done global_id 194159 payload 824 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 -- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff838004780 msgr2=0x7ff850130cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff838004780 0x7ff850130cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 -- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 msgr2=0x7ff850140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 0x7ff850140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 monclient: found mon.noname-c 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84f59e640 1 -- 172.21.15.153:0/956198132 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85011ddd0 con 0x7ff8380051a0 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.396+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 0x7ff850140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.398+0000 7ff84f59e640 1 -- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 msgr2=0x7ff8501311f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.398+0000 7ff84f59e640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 0x7ff8501311f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff85006f820 tx=0x7ff840006e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840003d40 con 0x7ff8380051a0 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: dump: 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840006170 con 0x7ff8380051a0 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840006530 con 0x7ff8380051a0 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff836ffd640 10 monclient: dump: 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 5 monclient: authenticate success, global_id 194159 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 10 monclient: _renew_subs 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85011e0b0 con 0x7ff8380051a0 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff84d59a640 4 set_mon_vals no callback set 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 10 monclient: _renew_subs 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.399+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8501414a0 con 0x7ff8380051a0 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff840009750 con 0x7ff8380051a0 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff8550c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff8550c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff8550c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:44:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85004bf60 con 0x7ff8380051a0 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff836ffd640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff83c038e30 0x7ff83c03b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff84004b160 con 0x7ff8380051a0 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff83c038e30 0x7ff83c03b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.400+0000 7ff84ed9d640 10 monclient: get_auth_request con 0x7ff83c038e30 auth_method 0 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.403+0000 7ff84ed9d640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff83c038e30 0x7ff83c03b2e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff8480039b0 tx=0x7ff84800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.405+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff84000e260 con 0x7ff8380051a0 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.405+0000 7ff836ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.405+0000 7ff836ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.621+0000 7ff8550c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.621+0000 7ff8550c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.621+0000 7ff8550c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.621+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff850098ee0 con 0x7ff8380051a0 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.622+0000 7ff836ffd640 1 -- 172.21.15.153:0/956198132 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7ff84000e260 con 0x7ff8380051a0 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.622+0000 7ff836ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.622+0000 7ff836ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:44:17.623 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":156,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff83c038e30 msgr2=0x7ff83c03b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff83c038e30 0x7ff83c03b2e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff8480039b0 tx=0x7ff84800b040 comp rx=0 tx=0).stop 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 10 monclient: shutdown 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 msgr2=0x7ff8501311f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 0x7ff8501311f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff85006f820 tx=0x7ff840006e20 comp rx=0 tx=0).stop 2024-09-15T03:44:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 shutdown_connections 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff83c038e30 0x7ff83c03b2e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8380051a0 0x7ff8501311f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff838004780 0x7ff850130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 --2- 172.21.15.153:0/956198132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8380039d0 0x7ff850140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 >> 172.21.15.153:0/956198132 conn(0x7ff8500a1e30 msgr2=0x7ff850096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 shutdown_connections 2024-09-15T03:44:17.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.624+0000 7ff8550c0640 1 -- 172.21.15.153:0/956198132 wait complete. 2024-09-15T03:44:17.637 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:44:17.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:44:17.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.730+0000 7f269b700640 10 monclient: get_monmap_and_config 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.730+0000 7f269b700640 10 monclient: build_initial_monmap 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.730+0000 7f269b700640 10 monclient: monmap: 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:44:17.730910+0000 2024-09-15T03:44:17.731 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:44:17.730910+0000 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.730+0000 7f269b700640 1 Processor -- start 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.730+0000 7f269b700640 1 -- start start 2024-09-15T03:44:17.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.730+0000 7f269b700640 10 monclient: init 2024-09-15T03:44:17.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:17.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:44:17.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f26940981c0 0x7f2694098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): picked mon.noname-a con 0x7f26940981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:44:17.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2694131da0 0x7f2694067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): picked mon.noname-c con 0x7f2694131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2694132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2694098ad0 con 0x7f2694132170 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2694131bd0 con 0x7f26940981c0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269409ca60 con 0x7f2694131da0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269b700640 10 monclient(hunting): authenticate will time out at 1655.370476s 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f26940981c0 0x7f2694098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f2699c9d640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f2694132170 legacy=0x7f2694067650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f2698c9b640 1 -- >> v2:172.21.15.181:3300/0 conn(0x7f2694131da0 msgr2=0x7f2694067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f2698c9b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2694131da0 0x7f2694067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:17.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f26940981c0 0x7f2694098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39140/0 (socket says 172.21.15.153:39140) 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 -- 172.21.15.153:0/3281223796 learned_addr learned my addr 172.21.15.153:0/3281223796 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): get_auth_request con 0x7f26940981c0 auth_method 0 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient(hunting): handle_auth_done global_id 174387 payload 274 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 -- 172.21.15.153:0/3281223796 >> v2:172.21.15.181:3300/0 conn(0x7f2694131da0 msgr2=0x7f2694067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 --2- 172.21.15.153:0/3281223796 >> v2:172.21.15.181:3300/0 conn(0x7f2694131da0 0x7f2694067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 -- 172.21.15.153:0/3281223796 >> v1:172.21.15.181:6789/0 conn(0x7f2694132170 legacy=0x7f2694067650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:17.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 monclient: found mon.noname-a 2024-09-15T03:44:17.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.731+0000 7f269949c640 1 -- 172.21.15.153:0/3281223796 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269409cca0 con 0x7f26940981c0 2024-09-15T03:44:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f269949c640 1 -- 172.21.15.153:0/3281223796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 msgr2=0x7f2694098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f269949c640 1 --2- 172.21.15.153:0/3281223796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694098590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2694132870 tx=0x7f26900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b5f68acbc323ad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 1 -- 172.21.15.153:0/3281223796 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690032440 con 0x7f26940981c0 2024-09-15T03:44:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:44:17.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: dump: 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 1 -- 172.21.15.153:0/3281223796 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26900325a0 con 0x7f26940981c0 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f269b700640 5 monclient: authenticate success, global_id 174387 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f269b700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f269b700640 10 monclient: get_monmap_and_config success 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f269b700640 4 set_mon_vals no callback set 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 1 -- 172.21.15.153:0/3281223796 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690032940 con 0x7f26940981c0 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.734+0000 7f268bfff640 10 monclient: dump: 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.746 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 10 monclient: shutdown 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 -- 172.21.15.153:0/3281223796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 msgr2=0x7f2694098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 --2- 172.21.15.153:0/3281223796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694098590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2694132870 tx=0x7f26900091d0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 -- 172.21.15.153:0/3281223796 shutdown_connections 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 --2- 172.21.15.153:0/3281223796 >> v2:172.21.15.181:3300/0 conn(0x7f2694131da0 0x7f2694067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 --2- 172.21.15.153:0/3281223796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694098590 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 -- 172.21.15.153:0/3281223796 >> 172.21.15.153:0/3281223796 conn(0x7f26940a1e30 msgr2=0x7f26940a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 -- 172.21.15.153:0/3281223796 shutdown_connections 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 1 -- 172.21.15.153:0/3281223796 wait complete. 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 10 monclient: build_initial_monmap 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.735+0000 7f269b700640 10 monclient: monmap: 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:44:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 Processor -- start 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 -- start start 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient: init 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient(hunting): picked mon.noname-a con 0x7f26940981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2694131da0 0x7f2694141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient(hunting): picked mon.noname-c con 0x7f2694131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2694132170 0x7f26941421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient(hunting): picked mon.noname-b con 0x7f2694132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:17.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694121b00 con 0x7f26940981c0 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694098cb0 con 0x7f2694132170 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.736+0000 7f269b700640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26940a0be0 con 0x7f2694131da0 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269b700640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269b700640 10 monclient(hunting): authenticate will time out at 1655.376012s 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39150/0 (socket says 172.21.15.153:39150) 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 -- 172.21.15.153:0/2226850447 learned_addr learned my addr 172.21.15.153:0/2226850447 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): get_auth_request con 0x7f26940981c0 auth_method 0 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2694131da0 0x7f2694141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): get_auth_request con 0x7f2694131da0 auth_method 0 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2699c9d640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2694132170 msgr2=0x7f26941421b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2699c9d640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2694132170 0x7f26941421b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient(hunting): handle_auth_done global_id 174390 payload 824 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2694131da0 msgr2=0x7f2694141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2694131da0 0x7f2694141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2694132170 msgr2=0x7f26941421b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2694132170 0x7f26941421b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 monclient: found mon.noname-a 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f269949c640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2690009450 con 0x7f26940981c0 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.737+0000 7f2698c9b640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2694131da0 0x7f2694141c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:44:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269949c640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 msgr2=0x7f2694141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269949c640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694141730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2690009210 tx=0x7f2690032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269003c960 con 0x7f26940981c0 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: dump: 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.753 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269003cac0 con 0x7f26940981c0 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 5 monclient: authenticate success, global_id 174390 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 10 monclient: _renew_subs 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269003ce40 con 0x7f26940981c0 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269411f980 con 0x7f26940981c0 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f268b7fe640 4 set_mon_vals no callback set 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f26897fa640 10 monclient: dump: 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 10 monclient: _renew_subs 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.739+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26941202d0 con 0x7f26940981c0 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f269b700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f269b700640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 0x7f269411a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694065f90 con 0x7f269411a4a0 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2690039040 con 0x7f26940981c0 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 0x7f269411a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient: get_auth_request con 0x7f269411a4a0 auth_method 0 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f26897fa640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26780386f0 0x7f267803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f269006cce0 con 0x7f26940981c0 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 10 monclient(hunting): handle_auth_done global_id 174393 payload 824 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.740+0000 7f2698c9b640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2694121620 con 0x7f269411a4a0 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.741+0000 7f2699c9d640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26780386f0 0x7f267803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.741+0000 7f2699c9d640 10 monclient: get_auth_request con 0x7f26780386f0 auth_method 0 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f2698c9b640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 msgr2=0x7f269411a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f2698c9b640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 0x7f269411a870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2694131670 tx=0x7f2684006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268400ed20 con 0x7f269411a4a0 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 10 monclient: dump: 2024-09-15T03:44:17.757 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f268400ee80 con 0x7f269411a4a0 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:44:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 msgr2=0x7f269411a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 0x7f269411a870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2694131670 tx=0x7f2684006f90 comp rx=0 tx=0).stop 2024-09-15T03:44:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 msgr2=0x7f269411a870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:44:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f26897fa640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411a4a0 0x7f269411a870 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2694131670 tx=0x7f2684006f90 comp rx=0 tx=0).stop 2024-09-15T03:44:17.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.743+0000 7f2699c9d640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26780386f0 0x7f267803aba0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f269411fdb0 tx=0x7f268c009310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f269b700640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f269b700640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 0x7f269411de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f269b700640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26941218b0 con 0x7f269411adb0 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 0x7f269411de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient: get_auth_request con 0x7f269411adb0 auth_method 0 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 10 monclient(hunting): handle_auth_done global_id 174396 payload 824 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.749+0000 7f2698c9b640 1 -- 172.21.15.153:0/2226850447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f269404bf60 con 0x7f269411adb0 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.751+0000 7f2698c9b640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 msgr2=0x7f269411de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.751+0000 7f2698c9b640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 0x7f269411de40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2684018000 tx=0x7f2684019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.751+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684022070 con 0x7f269411adb0 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.751+0000 7f26897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.751+0000 7f26897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.751+0000 7f26897fa640 10 monclient: dump: 2024-09-15T03:44:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7f268401b870 con 0x7f269411adb0 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 msgr2=0x7f269411de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 0x7f269411de40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2684018000 tx=0x7f2684019000 comp rx=0 tx=0).stop 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 msgr2=0x7f269411de40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f26897fa640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f269411adb0 0x7f269411de40 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2684018000 tx=0x7f2684019000 comp rx=0 tx=0).stop 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 156, 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.764 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:44:17.765 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:44:17.767 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.768 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:44:17.771 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:44:17.772 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.774 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.775 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:44:17.776 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26780386f0 msgr2=0x7f267803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26780386f0 0x7f267803aba0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f269411fdb0 tx=0x7f268c009310 comp rx=0 tx=0).stop 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 10 monclient: shutdown 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 msgr2=0x7f2694141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 --2- 172.21.15.153:0/2226850447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26940981c0 0x7f2694141730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2690009210 tx=0x7f2690032000 comp rx=0 tx=0).stop 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f2699c9d640 1 -- 172.21.15.153:0/2226850447 reap_dead start 2024-09-15T03:44:17.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f2699c9d640 1 -- 172.21.15.153:0/2226850447 reap_dead start 2024-09-15T03:44:17.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 shutdown_connections 2024-09-15T03:44:17.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 >> 172.21.15.153:0/2226850447 conn(0x7f26940a1e30 msgr2=0x7f2694096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:17.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 shutdown_connections 2024-09-15T03:44:17.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.752+0000 7f269b700640 1 -- 172.21.15.153:0/2226850447 wait complete. 2024-09-15T03:44:17.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:44:17.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:44:17.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:44:17.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.871+0000 7f957c299640 10 monclient: get_monmap_and_config 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.871+0000 7f957c299640 10 monclient: build_initial_monmap 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.871+0000 7f957c299640 10 monclient: monmap: 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:44:17.872711+0000 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:44:17.872711+0000 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:44:17.873 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:44:17.874 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:44:17.874 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.872+0000 7f957c299640 1 Processor -- start 2024-09-15T03:44:17.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.872+0000 7f957c299640 1 -- start start 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.872+0000 7f957c299640 10 monclient: init 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f957409caa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f957409a540 0x7f957409ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient(hunting): picked mon.noname-a con 0x7f957409a540 addr v2:172.21.15.153:3300/0 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95741346c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957409abd0 con 0x7f957409caa0 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957409b920 con 0x7f95741346c0 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957409ed40 con 0x7f957409a540 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:17.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957c299640 10 monclient(hunting): authenticate will time out at 1655.512264s 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f957409a540 0x7f957409ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957a836640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f95741346c0 legacy=0x7f9574136aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f957409a540 0x7f957409ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39180/0 (socket says 172.21.15.153:39180) 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 1 -- 172.21.15.153:0/535261330 learned_addr learned my addr 172.21.15.153:0/535261330 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 10 monclient(hunting): get_auth_request con 0x7f957409a540 auth_method 0 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f957a035640 1 --1- 172.21.15.153:0/535261330 >> v1:172.21.15.153:6789/0 conn(0x7f957409caa0 0x7f957406ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52746/0 (socket says 172.21.15.153:52746) 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.873+0000 7f9579834640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f957a035640 1 -- 172.21.15.153:0/535261330 >> v1:172.21.15.153:6789/0 conn(0x7f957409caa0 legacy=0x7f957406ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 10 monclient(hunting): handle_auth_done global_id 174399 payload 274 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 1 -- 172.21.15.153:0/535261330 >> v1:172.21.15.181:6789/0 conn(0x7f95741346c0 legacy=0x7f9574136aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 1 -- 172.21.15.153:0/535261330 >> v1:172.21.15.153:6789/0 conn(0x7f957409caa0 legacy=0x7f957406ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.875+0000 7f9579834640 1 monclient: found mon.noname-a 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.876+0000 7f9579834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.876+0000 7f9579834640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.876+0000 7f9579834640 1 -- 172.21.15.153:0/535261330 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957409c620 con 0x7f957409a540 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579834640 1 -- 172.21.15.153:0/535261330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f957409a540 msgr2=0x7f957409ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579834640 1 --2- 172.21.15.153:0/535261330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f957409a540 0x7f957409ce70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9564009990 tx=0x7f95640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4137777a3c2f4e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 1 -- 172.21.15.153:0/535261330 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956402f4b0 con 0x7f957409a540 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 10 monclient: dump: 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.886 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f957c299640 5 monclient: authenticate success, global_id 174399 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 1 -- 172.21.15.153:0/535261330 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956402f610 con 0x7f957409a540 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f957c299640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:17.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f957c299640 10 monclient: get_monmap_and_config success 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f957c299640 4 set_mon_vals no callback set 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 1 -- 172.21.15.153:0/535261330 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956402f9b0 con 0x7f957409a540 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.878+0000 7f9579033640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f9579033640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f9579033640 10 monclient: dump: 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.888 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 10 monclient: shutdown 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 -- 172.21.15.153:0/535261330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f957409a540 msgr2=0x7f957409ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 --2- 172.21.15.153:0/535261330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f957409a540 0x7f957409ce70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9564009990 tx=0x7f95640091d0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 -- 172.21.15.153:0/535261330 shutdown_connections 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 --2- 172.21.15.153:0/535261330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f957409a540 0x7f957409ce70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 -- 172.21.15.153:0/535261330 >> 172.21.15.153:0/535261330 conn(0x7f9574067cd0 msgr2=0x7f95740a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 -- 172.21.15.153:0/535261330 shutdown_connections 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.879+0000 7f957c299640 1 -- 172.21.15.153:0/535261330 wait complete. 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient: build_initial_monmap 2024-09-15T03:44:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient: monmap: 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 Processor -- start 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 -- start start 2024-09-15T03:44:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient: init 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f957409a540 0x7f9574119530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): picked mon.noname-b con 0x7f957409a540 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957409caa0 0x7f9574119a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): picked mon.noname-c con 0x7f957409caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 0x7f9574119fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): picked mon.noname-a con 0x7f95741346c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9574133930 con 0x7f95741346c0 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957411ad40 con 0x7f957409a540 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9574099d50 con 0x7f957409caa0 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957c299640 10 monclient(hunting): authenticate will time out at 1655.519944s 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.880+0000 7f957a035640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f957409a540 msgr2=0x7f9574119530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a035640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f957409a540 0x7f9574119530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 0x7f9574119fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957409caa0 0x7f9574119a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 0x7f9574119fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39182/0 (socket says 172.21.15.153:39182) 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 learned_addr learned my addr 172.21.15.153:0/4155736052 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): get_auth_request con 0x7f95741346c0 auth_method 0 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): get_auth_request con 0x7f957409caa0 auth_method 0 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f9579834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient(hunting): handle_auth_done global_id 174405 payload 824 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957409caa0 msgr2=0x7f9574119a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957409caa0 0x7f9574119a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f957409a540 msgr2=0x7f9574119530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:44:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f957409a540 0x7f9574119530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient: _finish_hunting 0 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 monclient: found mon.noname-a 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.881+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564009450 con 0x7f95741346c0 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 msgr2=0x7f9574119fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957a836640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 0x7f9574119fb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f956c00b4c0 tx=0x7f956c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c003d40 con 0x7f95741346c0 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: dump: 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 5 monclient: authenticate success, global_id 174405 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c006150 con 0x7f95741346c0 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 10 monclient: _renew_subs 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9574137f20 con 0x7f95741346c0 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c006510 con 0x7f95741346c0 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f956a7fc640 10 monclient: dump: 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f9578832640 4 set_mon_vals no callback set 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 10 monclient: _renew_subs 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.883+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957411c330 con 0x7f95741346c0 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f956c01c440 con 0x7f95741346c0 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957c299640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957c299640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 0x7f957406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957c299640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9574065e00 con 0x7f9574099660 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 0x7f957406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient: get_auth_request con 0x7f9574099660 auth_method 0 2024-09-15T03:44:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f956a7fc640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95580386f0 0x7f955803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 199..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f956c04ba10 con 0x7f95741346c0 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 10 monclient(hunting): handle_auth_done global_id 194168 payload 824 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.884+0000 7f957a035640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95741344b0 con 0x7f9574099660 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.885+0000 7f9579834640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95580386f0 0x7f955803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.885+0000 7f9579834640 10 monclient: get_auth_request con 0x7f95580386f0 auth_method 0 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f957a035640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 msgr2=0x7f957406bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f957a035640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 0x7f957406bfb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f957000c7e0 tx=0x7f95700018a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957000ceb0 con 0x7f9574099660 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 10 monclient: dump: 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f957000a6e0 con 0x7f9574099660 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 msgr2=0x7f957406bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 0x7f957406bfb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f957000c7e0 tx=0x7f95700018a0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 msgr2=0x7f957406bfb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f956a7fc640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9574099660 0x7f957406bfb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.887+0000 7f9579834640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95580386f0 0x7f955803aba0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f957409f920 tx=0x7f9564032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957c299640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957c299640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 0x7f957406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957c299640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957404bf60 con 0x7f957406c4f0 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 0x7f957406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient: get_auth_request con 0x7f957406c4f0 auth_method 0 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 10 monclient(hunting): handle_auth_done global_id 194171 payload 824 2024-09-15T03:44:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.893+0000 7f957a035640 1 -- 172.21.15.153:0/4155736052 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95740638d0 con 0x7f957406c4f0 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.895+0000 7f957a035640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 msgr2=0x7f957406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.895+0000 7f957a035640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 0x7f957406c8f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9570015000 tx=0x7f9570016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.895+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570021920 con 0x7f957406c4f0 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.895+0000 7f956a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.895+0000 7f956a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.895+0000 7f956a7fc640 10 monclient: dump: 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f9570021a80 con 0x7f957406c4f0 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 msgr2=0x7f957406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 0x7f957406c8f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9570015000 tx=0x7f9570016000 comp rx=0 tx=0).stop 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 msgr2=0x7f957406c8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f956a7fc640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f957406c4f0 0x7f957406c8f0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9570015000 tx=0x7f9570016000 comp rx=0 tx=0).stop 2024-09-15T03:44:17.905 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 156, 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:44:17.906 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:44:17.907 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:44:17.908 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:44:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:44:17.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.911 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:44:17.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:44:17.913 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:44:17.915 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.916 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:44:17.917 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95580386f0 msgr2=0x7f955803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95580386f0 0x7f955803aba0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f957409f920 tx=0x7f9564032000 comp rx=0 tx=0).stop 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 10 monclient: shutdown 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 msgr2=0x7f9574119fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 --2- 172.21.15.153:0/4155736052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95741346c0 0x7f9574119fb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f956c00b4c0 tx=0x7f956c009060 comp rx=0 tx=0).stop 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 reap_dead start 2024-09-15T03:44:17.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957a836640 1 -- 172.21.15.153:0/4155736052 reap_dead start 2024-09-15T03:44:17.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 shutdown_connections 2024-09-15T03:44:17.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 >> 172.21.15.153:0/4155736052 conn(0x7f9574067cd0 msgr2=0x7f957406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:17.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 shutdown_connections 2024-09-15T03:44:17.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:17.896+0000 7f957c299640 1 -- 172.21.15.153:0/4155736052 wait complete. 2024-09-15T03:44:17.920 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:44:17.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:44:37.921 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:44:37.922 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:44:37.922 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:44:37.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:44:37.926 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:44:37.926 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T03:44:37.926 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T03:44:37.926 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:44:52.927 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T03:44:52.928 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T03:44:52.928 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:44:52.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.030+0000 7fa00591d640 10 monclient: get_monmap_and_config 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.030+0000 7fa00591d640 10 monclient: build_initial_monmap 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.030+0000 7fa00591d640 10 monclient: monmap: 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:44:53.030882+0000 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:44:53.030882+0000 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:53.031 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:53.032 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:44:53.032 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:44:53.032 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:44:53.032 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:44:53.032 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:44:53.032 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:53.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.030+0000 7fa00591d640 1 Processor -- start 2024-09-15T03:44:53.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.030+0000 7fa00591d640 1 -- start start 2024-09-15T03:44:53.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.030+0000 7fa00591d640 10 monclient: init 2024-09-15T03:44:53.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0001318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): picked mon.noname-b con 0x7fa00009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): picked mon.noname-c con 0x7fa000068d90 addr v2:172.21.15.181:3300/0 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa000131cc0 con 0x7fa0001318f0 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00009fbf0 con 0x7fa000068d90 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00009fe30 con 0x7fa00009ea50 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7fa00591d640 10 monclient(hunting): authenticate will time out at 1690.670377s 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36124/0 (socket says 172.21.15.153:36124) 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35934/0 (socket says 172.21.15.153:35934) 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 learned_addr learned my addr 172.21.15.153:0/751563478 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7fa00009ea50 auth_method 0 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7fa0001318f0 auth_method 0 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7fa000068d90 auth_method 0 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:44:53.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.031+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7fa00009ea50 auth_method 0 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7fa0001318f0 auth_method 0 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7fa000068d90 auth_method 0 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:44:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:44:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:44:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.232+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7fa00009ea50 auth_method 0 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7fa0001318f0 auth_method 0 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:53.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7fa000068d90 auth_method 0 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:53.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:44:53.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:53.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:53.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:53.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:53.633+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:44:54.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:54.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:54.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7fa0001318f0 auth_method 0 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7fa00009ea50 auth_method 0 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:54.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.434+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7fa000068d90 auth_method 0 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:54.435+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:44:56.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 10 monclient(hunting): tick 2024-09-15T03:44:56.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 monclient(hunting): continuing hunt 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 msgr2=0x7fa00009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa00009ea50 0x7fa00009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 msgr2=0x7fa0000691f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.181:3300/0 conn(0x7fa000068d90 0x7fa0000691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 -- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 msgr2=0x7fa000131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:44:56.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 --2- 172.21.15.153:0/751563478 >> v2:172.21.15.153:3300/0 conn(0x7fa0001318f0 0x7fa000131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 10 monclient: _add_conns ranks=[1,0] 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fe8003650 addr v1:172.21.15.181:6789/0 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9fe8007350 addr v1:172.21.15.153:6789/0 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 -- 172.21.15.153:0/751563478 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000a1210 con 0x7f9fe8007350 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 1 -- 172.21.15.153:0/751563478 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe800c0c0 con 0x7f9fe8003650 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.031+0000 7f9ffcff9640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:56.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.033+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/751563478 >> v1:172.21.15.153:6789/0 conn(0x7f9fe8007350 legacy=0x7f9fe8009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:56.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:56.034+0000 7f9ffeffd640 1 -- 172.21.15.153:0/751563478 >> v1:172.21.15.181:6789/0 conn(0x7f9fe8003650 legacy=0x7f9fe8005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038016509 0 0) 0x7fa0000a1210 con 0x7f9fe8007350 2024-09-15T03:44:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient(hunting): my global_id is 174408 2024-09-15T03:44:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec0034e0 con 0x7f9fe8007350 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3521032335 0 0) 0x7f9fec0034e0 con 0x7f9fe8007350 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 >> v1:172.21.15.181:6789/0 conn(0x7f9fe8003650 legacy=0x7f9fe8005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient: _finish_hunting 0 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe800d320 con 0x7f9fe8007350 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient: _finish_auth 0 2024-09-15T03:44:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f9ff400c680 con 0x7f9fe8007350 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.937+0000 7f9ffdffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient: dump: 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 >> v1:172.21.15.153:6789/0 conn(0x7f9fe8007350 legacy=0x7f9fe8009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 reap_dead start 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7f9fec003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): picked mon.c con 0x7f9fec003b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec004780 0x7f9fec004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): picked mon.b con 0x7f9fec004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 0x7f9fec00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): picked mon.a con 0x7f9fec0099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec00c3f0 con 0x7f9fec0099c0 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec00c560 con 0x7f9fec004780 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec00c720 con 0x7f9fec003b20 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffdffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 0x7f9fec00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7f9fec0099c0 auth_method 0 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec004780 0x7f9fec004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7f9fec004780 auth_method 0 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_done global_id 174408 payload 293 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 msgr2=0x7f9fec003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7f9fec003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec004780 msgr2=0x7f9fec004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec004780 0x7f9fec004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 monclient: found mon.a 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0034e0 con 0x7f9fec0099c0 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 10 monclient: _finish_auth 0 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7f9fff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7fa00591d640 5 monclient: authenticate success, global_id 174408 2024-09-15T03:44:57.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.938+0000 7fa00591d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9fff7fe640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 msgr2=0x7f9fec00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 0x7f9fec00bda0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa000068d40 tx=0x7f9ff8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfe76cad85c0d53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff802ad00 con 0x7f9fec0099c0 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: dump: 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:57.956 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff802e070 con 0x7f9fec0099c0 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7fa00591d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7fa00591d640 10 monclient: get_monmap_and_config success 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 1 -- 172.21.15.153:0/751563478 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8034650 con 0x7f9fec0099c0 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7fa00591d640 4 set_mon_vals no callback set 2024-09-15T03:44:57.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.941+0000 7f9ffdffb640 10 monclient: dump: 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 10 monclient: shutdown 2024-09-15T03:44:57.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 -- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 msgr2=0x7f9fec00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 0x7f9fec00bda0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa000068d40 tx=0x7f9ff8008040 comp rx=0 tx=0).stop 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 -- 172.21.15.153:0/751563478 shutdown_connections 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec0099c0 0x7f9fec00bda0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec004780 0x7f9fec004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 --2- 172.21.15.153:0/751563478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7f9fec003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 -- 172.21.15.153:0/751563478 >> 172.21.15.153:0/751563478 conn(0x7fa000067110 msgr2=0x7fa000067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 -- 172.21.15.153:0/751563478 shutdown_connections 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.942+0000 7fa00591d640 1 -- 172.21.15.153:0/751563478 wait complete. 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.943+0000 7fa00591d640 10 monclient: build_initial_monmap 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.943+0000 7fa00591d640 10 monclient: monmap: 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:44:57.959 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 Processor -- start 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 -- start start 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient: init 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:44:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7fa0001404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec003b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7fa000130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7fa00011d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec0099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa000120f20 con 0x7f9fec004780 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0001413d0 con 0x7f9fec0099c0 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00009dc00 con 0x7f9fec003b20 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7fa00591d640 10 monclient(hunting): authenticate will time out at 1695.583735s 2024-09-15T03:44:57.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7fa000130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7fa0001404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7fa000130af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60906/0 (socket says 172.21.15.153:60906) 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 learned_addr learned my addr 172.21.15.153:0/833559834 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7f9fec004780 auth_method 0 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7f9fec003b20 auth_method 0 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.944+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7fa00011d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7f9fec0099c0 auth_method 0 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7fa00011d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7fa00011d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7fa00011d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7fa00011d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_done global_id 194174 payload 824 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7fa00011d700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:44:57.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7fa00011d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 msgr2=0x7fa000130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7fa000130af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 1 monclient: found mon.noname-c 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffeffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000120c40 con 0x7f9fec003b20 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.945+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7fa000130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.947+0000 7f9ffeffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 msgr2=0x7fa0001404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.947+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7fa0001404f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0039d0 tx=0x7f9ff000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00098c0 con 0x7f9fec003b20 2024-09-15T03:44:57.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: dump: 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:44:57.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff0009a20 con 0x7f9fec003b20 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0009de0 con 0x7f9fec003b20 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 5 monclient: authenticate success, global_id 194174 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 10 monclient: _renew_subs 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa000140a30 con 0x7f9fec003b20 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9fe6ffd640 10 monclient: dump: 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.966 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9ffd7fa640 4 set_mon_vals no callback set 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 10 monclient: _renew_subs 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00011fe80 con 0x7f9fec003b20 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9ffeffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 msgr2=0x7fa0001404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9ffeffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 msgr2=0x7fa0001404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:44:57.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.949+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7fa0001404f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0039d0 tx=0x7f9ff000b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7fa0001404f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0039d0 tx=0x7f9ff000b040 comp rx=0 tx=0).stop 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 msgr2=0x7fa0001404f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fec003b20 0x7fa0001404f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): picked mon.a con 0x7f9fec004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7f9fe800b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): picked mon.b con 0x7f9fec0099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe800c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): picked mon.c con 0x7f9fe800baa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:57.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe800cb60 con 0x7f9fec004780 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe800cca0 con 0x7f9fec0099c0 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe800ce10 con 0x7f9fe800baa0 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fe6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe800c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7f9fe800b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request con 0x7f9fe800baa0 auth_method 0 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7f9fec0099c0 auth_method 0 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7f9fec004780 auth_method 0 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9fff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_done global_id 194174 payload 293 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 msgr2=0x7f9fe800c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe800c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 msgr2=0x7f9fe8009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 monclient: found mon.b 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:44:57.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f9fe800d0a0 con 0x7f9fec0099c0 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 10 monclient: _finish_auth 0 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.950+0000 7f9ffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.951+0000 7fa00591d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.951+0000 7fa00591d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.951+0000 7fa00591d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.951+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00009bf80 con 0x7f9fec0099c0 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7f9fe800b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7f9fe800b2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0000a2b40 tx=0x7f9ff4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4017e40 con 0x7f9fec0099c0 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9fe6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9fe6ffd640 10 monclient: dump: 2024-09-15T03:44:57.972 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff400c040 con 0x7f9fec0099c0 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.953+0000 7f9fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.954+0000 7f9ffcff9640 4 set_mon_vals no callback set 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.954+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ff4029020 con 0x7f9fec0099c0 2024-09-15T03:44:57.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.954+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9fe8041d40 0x7f9fe80441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:57.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.955+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.1 v2:172.21.15.181:3300/0 4 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9ff404bf60 con 0x7f9fec0099c0 2024-09-15T03:44:57.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.955+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9fe8041d40 0x7f9fe80441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:57.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.955+0000 7f9ffeffd640 10 monclient: get_auth_request con 0x7f9fe8041d40 auth_method 0 2024-09-15T03:44:57.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.957+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9fe8041d40 0x7f9fe80441f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0002230 tx=0x7f9ff001f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:57.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.959+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.1 v2:172.21.15.181:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ff4014530 con 0x7f9fec0099c0 2024-09-15T03:44:57.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.959+0000 7f9fe6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:44:57.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:57.959+0000 7f9fe6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:44:58.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7f9fe800b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:44:58.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7f9fe800b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:44:58.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7f9fe800b2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0000a2b40 tx=0x7f9ff4008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7f9fe800b2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0000a2b40 tx=0x7f9ff4008040 comp rx=0 tx=0).stop 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 msgr2=0x7f9fe800b2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fec0099c0 0x7f9fe800b2f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient(hunting): picked mon.a con 0x7f9fec004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 0x7f9fe804ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient(hunting): picked mon.b con 0x7f9fe804aa00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe804d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:44:58.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient(hunting): picked mon.c con 0x7f9fe800baa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe804dda0 con 0x7f9fec004780 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe804dee0 con 0x7f9fe804aa00 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.015+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe804e020 con 0x7f9fe800baa0 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9fe6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request con 0x7f9fec004780 auth_method 0 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:58.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 0x7f9fe804ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9fff7fe640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 msgr2=0x7f9fe804ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9fff7fe640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 msgr2=0x7f9fe804ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 0x7f9fe804ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe804d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9fff7fe640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 0x7f9fe804ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request con 0x7f9fe800baa0 auth_method 0 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient(hunting): handle_auth_done global_id 194174 payload 293 2024-09-15T03:44:58.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 msgr2=0x7f9fe804d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe804d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 msgr2=0x7f9fe804ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 0x7f9fe804ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 monclient: found mon.a 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f9fe804e1f0 con 0x7f9fec004780 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 10 monclient: _finish_auth 0 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:44:58.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.016+0000 7f9ffeffd640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe804d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:44:58.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.018+0000 7f9ffe7fc640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 msgr2=0x7f9fe8009300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:44:58.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.018+0000 7f9ffe7fc640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40148e0 tx=0x7f9ff404b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:44:58.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40496b0 con 0x7f9fec004780 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 10 monclient: dump: 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:44:58.024 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:44:58.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:44:58.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff4049930 con 0x7f9fec004780 2024-09-15T03:44:58.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:44:58.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9ffcff9640 4 set_mon_vals no callback set 2024-09-15T03:44:58.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.019+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ff4019960 con 0x7f9fec004780 2024-09-15T03:44:58.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7fa00591d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:44:58.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7fa00591d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:44:58.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7fa00591d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:44:58.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa00009bf80 con 0x7f9fec004780 2024-09-15T03:44:58.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7f9fe6ffd640 1 -- 172.21.15.153:0/833559834 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa00009bf80 con 0x7f9fec004780 2024-09-15T03:44:58.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7f9fe6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:44:58.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.179+0000 7f9fe6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:44:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:44:58.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9fe8041d40 msgr2=0x7f9fe80441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9fe8041d40 0x7f9fe80441f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0002230 tx=0x7f9ff001f000 comp rx=0 tx=0).stop 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 10 monclient: shutdown 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 msgr2=0x7f9fe8009300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40148e0 tx=0x7f9ff404b040 comp rx=0 tx=0).stop 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 shutdown_connections 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fe800baa0 0x7f9fe804d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe804aa00 0x7f9fe804ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fec004780 0x7f9fe8009300 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 --2- 172.21.15.153:0/833559834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9fe8041d40 0x7f9fe80441f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 >> 172.21.15.153:0/833559834 conn(0x7fa000067110 msgr2=0x7fa0000a3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:44:58.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 shutdown_connections 2024-09-15T03:44:58.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:44:58.181+0000 7fa00591d640 1 -- 172.21.15.153:0/833559834 wait complete. 2024-09-15T03:45:01.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.298+0000 7f9743d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.298+0000 7f9743d9d640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.298+0000 7f9743d9d640 10 monclient: monmap: 2024-09-15T03:45:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.299 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:01.299356+0000 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:01.299356+0000 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:01.300 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 1 Processor -- start 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 1 -- start start 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 10 monclient: init 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f973c09a510 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f973c097370 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f973c097770 0x7f973c1380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.299+0000 7f9743d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f973c097770 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9743d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c09cb00 con 0x7f973c097370 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9743d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c09a8e0 con 0x7f973c09a510 2024-09-15T03:45:01.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9743d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9743d9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c06ed80 con 0x7f973c097770 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9743d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9743d9d640 10 monclient(hunting): authenticate will time out at 1698.939068s 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f973c097770 0x7f973c1380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f973c097770 0x7f973c1380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60940/0 (socket says 172.21.15.153:60940) 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f9741338640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f973c097370 0x7f973c1349a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44830/0 (socket says 172.21.15.153:44830) 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 -- 172.21.15.153:0/638760517 learned_addr learned my addr 172.21.15.153:0/638760517 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): get_auth_request con 0x7f973c097770 auth_method 0 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient(hunting): handle_auth_done global_id 174414 payload 274 2024-09-15T03:45:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 -- 172.21.15.153:0/638760517 >> v1:172.21.15.181:6789/0 conn(0x7f973c09a510 legacy=0x7f973c096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 -- 172.21.15.153:0/638760517 >> v1:172.21.15.153:6789/0 conn(0x7f973c097370 legacy=0x7f973c1349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 monclient: found mon.noname-a 2024-09-15T03:45:01.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.300+0000 7f974233a640 1 -- 172.21.15.153:0/638760517 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c09c820 con 0x7f973c097770 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f974233a640 1 -- 172.21.15.153:0/638760517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c097770 msgr2=0x7f973c1380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f974233a640 1 --2- 172.21.15.153:0/638760517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c097770 0x7f973c1380d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9734009f50 tx=0x7f9734009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47153bc369b9a086 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 1 -- 172.21.15.153:0/638760517 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734007610 con 0x7f973c097770 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: dump: 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 1 -- 172.21.15.153:0/638760517 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973402d050 con 0x7f973c097770 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 5 monclient: authenticate success, global_id 174414 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 1 -- 172.21.15.153:0/638760517 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340079a0 con 0x7f973c097770 2024-09-15T03:45:01.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 4 set_mon_vals no callback set 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9740b37640 10 monclient: dump: 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 10 monclient: shutdown 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 1 -- 172.21.15.153:0/638760517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c097770 msgr2=0x7f973c1380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.303+0000 7f9743d9d640 1 --2- 172.21.15.153:0/638760517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c097770 0x7f973c1380d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9734009f50 tx=0x7f9734009840 comp rx=0 tx=0).stop 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 -- 172.21.15.153:0/638760517 shutdown_connections 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 --2- 172.21.15.153:0/638760517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c097770 0x7f973c1380d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 -- 172.21.15.153:0/638760517 >> 172.21.15.153:0/638760517 conn(0x7f973c0672b0 msgr2=0x7f973c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 -- 172.21.15.153:0/638760517 shutdown_connections 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 -- 172.21.15.153:0/638760517 wait complete. 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 10 monclient: monmap: 2024-09-15T03:45:01.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 Processor -- start 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 1 -- start start 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.304+0000 7f9743d9d640 10 monclient: init 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f973c097370 0x7f973c140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f973c097370 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f973c097770 0x7f973c12fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f973c097770 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 0x7f973c130110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f973c09a510 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c141c50 con 0x7f973c09a510 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c120000 con 0x7f973c097770 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c11ee30 con 0x7f973c097370 2024-09-15T03:45:01.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9743d9d640 10 monclient(hunting): authenticate will time out at 1698.944400s 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 0x7f973c130110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f973c097370 0x7f973c140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 0x7f973c130110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60956/0 (socket says 172.21.15.153:60956) 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 -- 172.21.15.153:0/199638773 learned_addr learned my addr 172.21.15.153:0/199638773 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): get_auth_request con 0x7f973c09a510 auth_method 0 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): get_auth_request con 0x7f973c097370 auth_method 0 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f9741338640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f973c097770 0x7f973c12fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient(hunting): handle_auth_done global_id 174417 payload 824 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 -- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f973c097370 msgr2=0x7f973c140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f973c097370 0x7f973c140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 -- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f973c097770 msgr2=0x7f973c12fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f973c097770 0x7f973c12fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 monclient: found mon.noname-a 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.305+0000 7f974233a640 1 -- 172.21.15.153:0/199638773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734009ae0 con 0x7f973c09a510 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f974233a640 1 -- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 msgr2=0x7f973c130110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f974233a640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 0x7f973c130110 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9734009f50 tx=0x7f9734036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734031cb0 con 0x7f973c09a510 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: dump: 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9734031e10 con 0x7f973c09a510 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 5 monclient: authenticate success, global_id 174417 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 10 monclient: _renew_subs 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c11fd20 con 0x7f973c09a510 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973403c650 con 0x7f973c09a510 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9731ffb640 10 monclient: dump: 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9733fff640 4 set_mon_vals no callback set 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 10 monclient: _renew_subs 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.308+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c1412c0 con 0x7f973c09a510 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f973403ca00 con 0x7f973c09a510 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9743d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9743d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9743d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c09c3e0 con 0x7f973c09a510 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9731ffb640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9720038740 0x7f972003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f973406d9b0 con 0x7f973c09a510 2024-09-15T03:45:01.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9741b39640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9720038740 0x7f972003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.309+0000 7f9741b39640 10 monclient: get_auth_request con 0x7f9720038740 auth_method 0 2024-09-15T03:45:01.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.312+0000 7f9741b39640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9720038740 0x7f972003abf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9738007fd0 tx=0x7f9738007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.314+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f973c09c3e0 con 0x7f973c09a510 2024-09-15T03:45:01.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.314+0000 7f9731ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.314+0000 7f9731ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9743d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9743d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9743d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f973c09c3e0 con 0x7f973c09a510 2024-09-15T03:45:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9731ffb640 1 -- 172.21.15.153:0/199638773 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f973c09c3e0 con 0x7f973c09a510 2024-09-15T03:45:01.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9731ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.525+0000 7f9731ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:01.527 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9720038740 msgr2=0x7f972003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9720038740 0x7f972003abf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9738007fd0 tx=0x7f9738007e10 comp rx=0 tx=0).stop 2024-09-15T03:45:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 10 monclient: shutdown 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 msgr2=0x7f973c130110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 0x7f973c130110 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9734009f50 tx=0x7f9734036000 comp rx=0 tx=0).stop 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 shutdown_connections 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9720038740 0x7f972003abf0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f973c09a510 0x7f973c130110 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f973c097770 0x7f973c12fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 --2- 172.21.15.153:0/199638773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f973c097370 0x7f973c140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 >> 172.21.15.153:0/199638773 conn(0x7f973c0672b0 msgr2=0x7f973c06ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 shutdown_connections 2024-09-15T03:45:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.527+0000 7f9743d9d640 1 -- 172.21.15.153:0/199638773 wait complete. 2024-09-15T03:45:01.539 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:45:01.539 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:45:01.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:45:01.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.631+0000 7ff18739d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.631+0000 7ff18739d640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.631+0000 7ff18739d640 10 monclient: monmap: 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:01.632428+0000 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:01.632428+0000 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.633 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 Processor -- start 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 -- start start 2024-09-15T03:45:01.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient: init 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff18009c860 0x7ff18009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): picked mon.noname-a con 0x7ff18009c860 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff18009aba0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 0x7ff18009df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): picked mon.noname-b con 0x7ff18009afa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18006a320 con 0x7ff18009aba0 2024-09-15T03:45:01.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18009a9d0 con 0x7ff18009c860 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff180068a20 con 0x7ff18009afa0 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.632+0000 7ff18739d640 10 monclient(hunting): authenticate will time out at 1699.271945s 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff18009c860 0x7ff18009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 0x7ff18009df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 0x7ff18009df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36170/0 (socket says 172.21.15.153:36170) 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff18009c860 0x7ff18009cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60964/0 (socket says 172.21.15.153:60964) 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 1 -- 172.21.15.153:0/217953292 learned_addr learned my addr 172.21.15.153:0/217953292 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): get_auth_request con 0x7ff18009afa0 auth_method 0 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): get_auth_request con 0x7ff18009c860 auth_method 0 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient(hunting): handle_auth_done global_id 174420 payload 274 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 -- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 msgr2=0x7ff18009df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 --2- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 0x7ff18009df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 -- 172.21.15.153:0/217953292 >> v1:172.21.15.181:6789/0 conn(0x7ff18009aba0 legacy=0x7ff18009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 monclient: found mon.noname-a 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff185139640 1 -- 172.21.15.153:0/217953292 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180068e60 con 0x7ff18009c860 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.633+0000 7ff18593a640 1 --2- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 0x7ff18009df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff185139640 1 -- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009c860 msgr2=0x7ff18009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff185139640 1 --2- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009c860 0x7ff18009cc30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009850 tx=0x7ff17c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e4325f2fbe79b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 1 -- 172.21.15.153:0/217953292 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c02ae80 con 0x7ff18009c860 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: dump: 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.645 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 1 -- 172.21.15.153:0/217953292 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c032440 con 0x7ff18009c860 2024-09-15T03:45:01.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 5 monclient: authenticate success, global_id 174420 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 4 set_mon_vals no callback set 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 1 -- 172.21.15.153:0/217953292 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0327c0 con 0x7ff18009c860 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff177fff640 10 monclient: dump: 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 10 monclient: shutdown 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 -- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009c860 msgr2=0x7ff18009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 --2- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009c860 0x7ff18009cc30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009850 tx=0x7ff17c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 -- 172.21.15.153:0/217953292 shutdown_connections 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 --2- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009afa0 0x7ff18009df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 --2- 172.21.15.153:0/217953292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009c860 0x7ff18009cc30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 -- 172.21.15.153:0/217953292 >> 172.21.15.153:0/217953292 conn(0x7ff1800a2e40 msgr2=0x7ff1800a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 -- 172.21.15.153:0/217953292 shutdown_connections 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.636+0000 7ff18739d640 1 -- 172.21.15.153:0/217953292 wait complete. 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.637+0000 7ff18739d640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.637+0000 7ff18739d640 10 monclient: monmap: 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.637+0000 7ff18739d640 1 Processor -- start 2024-09-15T03:45:01.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.637+0000 7ff18739d640 1 -- start start 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.637+0000 7ff18739d640 10 monclient: init 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 0x7ff1801411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): picked mon.noname-b con 0x7ff18009aba0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 0x7ff180133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): picked mon.noname-a con 0x7ff18009afa0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009c860 0x7ff18011d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): picked mon.noname-c con 0x7ff18009c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180121430 con 0x7ff18009afa0 2024-09-15T03:45:01.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180099cc0 con 0x7ff18009aba0 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff180134b90 con 0x7ff18009c860 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18739d640 10 monclient(hunting): authenticate will time out at 1699.277303s 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 0x7ff180133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009c860 0x7ff18011d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 0x7ff180133cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60972/0 (socket says 172.21.15.153:60972) 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 -- 172.21.15.153:0/1389640005 learned_addr learned my addr 172.21.15.153:0/1389640005 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): get_auth_request con 0x7ff18009afa0 auth_method 0 2024-09-15T03:45:01.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): get_auth_request con 0x7ff18009c860 auth_method 0 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 0x7ff1801411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 10 monclient(hunting): get_auth_request con 0x7ff18009aba0 auth_method 0 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient(hunting): handle_auth_done global_id 174423 payload 824 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 msgr2=0x7ff1801411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 msgr2=0x7ff1801411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 0x7ff1801411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009c860 msgr2=0x7ff18011d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009c860 0x7ff18011d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 0x7ff1801411e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 msgr2=0x7ff1801411e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18009aba0 0x7ff1801411e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 monclient: found mon.noname-a 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff184938640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff178009ae0 con 0x7ff18009afa0 2024-09-15T03:45:01.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.638+0000 7ff18593a640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff18009c860 0x7ff18011d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.640+0000 7ff184938640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 msgr2=0x7ff180133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.640+0000 7ff184938640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 0x7ff180133cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff17000c610 tx=0x7ff170007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170012840 con 0x7ff18009afa0 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: dump: 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1700129e0 con 0x7ff18009afa0 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 5 monclient: authenticate success, global_id 174423 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 10 monclient: _renew_subs 2024-09-15T03:45:01.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17c009390 con 0x7ff18009afa0 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170012da0 con 0x7ff18009afa0 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1757fa640 10 monclient: dump: 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff1777fe640 4 set_mon_vals no callback set 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 10 monclient: _renew_subs 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.641+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18011f7e0 con 0x7ff18009afa0 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff170011590 con 0x7ff18009afa0 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff18739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff18739d640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 0x7ff18011ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180065e00 con 0x7ff180134200 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 0x7ff18011ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient: get_auth_request con 0x7ff180134200 auth_method 0 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff1757fa640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff164038740 0x7ff16403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff17004c730 con 0x7ff18009afa0 2024-09-15T03:45:01.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 10 monclient(hunting): handle_auth_done global_id 174426 payload 824 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff185139640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff18004bf60 con 0x7ff180134200 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff18593a640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff164038740 0x7ff16403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.642+0000 7ff18593a640 10 monclient: get_auth_request con 0x7ff164038740 auth_method 0 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.644+0000 7ff185139640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 msgr2=0x7ff18011ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.644+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 0x7ff18011ddf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff17c034000 tx=0x7ff17c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c031c50 con 0x7ff180134200 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 10 monclient: dump: 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.666 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff18593a640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff164038740 0x7ff16403abf0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff1780072f0 tx=0x7ff17800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff17c031db0 con 0x7ff180134200 2024-09-15T03:45:01.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 msgr2=0x7ff18011ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 0x7ff18011ddf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff17c034000 tx=0x7ff17c032000 comp rx=0 tx=0).stop 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 msgr2=0x7ff18011ddf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.645+0000 7ff1757fa640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff180134200 0x7ff18011ddf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff18739d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff18739d640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 0x7ff18006b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff18739d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180142100 con 0x7ff18011e330 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 0x7ff18006b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient: get_auth_request con 0x7ff18011e330 auth_method 0 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 10 monclient(hunting): handle_auth_done global_id 174429 payload 824 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.651+0000 7ff185139640 1 -- 172.21.15.153:0/1389640005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff18011e730 con 0x7ff18011e330 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff185139640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 msgr2=0x7ff18006b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff185139640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 0x7ff18006b8b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff180120090 tx=0x7ff17c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c04f410 con 0x7ff18011e330 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 10 monclient: dump: 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7ff17c031a40 con 0x7ff18011e330 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 msgr2=0x7ff18006b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 0x7ff18006b8b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff180120090 tx=0x7ff17c03c000 comp rx=0 tx=0).stop 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 msgr2=0x7ff18006b8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.653+0000 7ff1757fa640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18011e330 0x7ff18006b8b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.671 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 166, 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:45:01.672 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:01.673 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:01.675 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:01.676 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:01.677 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.678 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.684 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:45:01.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:01.686 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:01.687 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:01.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff164038740 msgr2=0x7ff16403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff164038740 0x7ff16403abf0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff1780072f0 tx=0x7ff17800b040 comp rx=0 tx=0).stop 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 10 monclient: shutdown 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 msgr2=0x7ff180133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 0x7ff180133cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff17000c610 tx=0x7ff170007ed0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18593a640 1 -- 172.21.15.153:0/1389640005 reap_dead start 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 shutdown_connections 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 --2- 172.21.15.153:0/1389640005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18009afa0 0x7ff180133cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 >> 172.21.15.153:0/1389640005 conn(0x7ff1800a2e40 msgr2=0x7ff180098c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 shutdown_connections 2024-09-15T03:45:01.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.654+0000 7ff18739d640 1 -- 172.21.15.153:0/1389640005 wait complete. 2024-09-15T03:45:01.690 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:01.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:45:01.690 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:45:01.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.782+0000 7f3d6306f640 10 monclient: get_monmap_and_config 2024-09-15T03:45:01.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.782+0000 7f3d6306f640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.782+0000 7f3d6306f640 10 monclient: monmap: 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:01.783554+0000 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:01.783554+0000 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:01.784 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:01.785 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 1 Processor -- start 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 1 -- start start 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient: init 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c0974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 0x7f3d5c09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c09a8a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d5c12dd40 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c0978c0 con 0x7f3d5c12dd40 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.783+0000 7f3d6306f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5c09ca20 con 0x7f3d5c09a8a0 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d6306f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c09b1f0 con 0x7f3d5c0974f0 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d6306f640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d6306f640 10 monclient(hunting): authenticate will time out at 1699.423016s 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36188/0 (socket says 172.21.15.153:36188) 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 -- 172.21.15.153:0/1807057070 learned_addr learned my addr 172.21.15.153:0/1807057070 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): get_auth_request con 0x7f3d5c0974f0 auth_method 0 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 0x7f3d5c09acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request con 0x7f3d5c09a8a0 auth_method 0 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 1 -- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 msgr2=0x7f3d5c09acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 1 -- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 msgr2=0x7f3d5c09acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_done global_id 194186 payload 274 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 0x7f3d5c09acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 -- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 msgr2=0x7f3d5c09acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 0x7f3d5c09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 --2- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 0x7f3d5c09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 -- 172.21.15.153:0/1807057070 >> v1:172.21.15.181:6789/0 conn(0x7f3d5c12dd40 legacy=0x7f3d5c12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 monclient: found mon.noname-b 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.784+0000 7f3d60e0b640 1 -- 172.21.15.153:0/1807057070 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c098ad0 con 0x7f3d5c0974f0 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d60e0b640 1 -- 172.21.15.153:0/1807057070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 msgr2=0x7f3d5c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d60e0b640 1 --2- 172.21.15.153:0/1807057070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c12ab00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000bbb0 tx=0x7f3d5000b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a8fee1ab4a67cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 1 -- 172.21.15.153:0/1807057070 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50032690 con 0x7f3d5c0974f0 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:01.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: dump: 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 5 monclient: authenticate success, global_id 194186 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 1 -- 172.21.15.153:0/1807057070 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5002f030 con 0x7f3d5c0974f0 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 1 -- 172.21.15.153:0/1807057070 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50032a60 con 0x7f3d5c0974f0 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 4 set_mon_vals no callback set 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:01.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d5b7fe640 10 monclient: dump: 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 10 monclient: shutdown 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 1 -- 172.21.15.153:0/1807057070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 msgr2=0x7f3d5c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 1 --2- 172.21.15.153:0/1807057070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c12ab00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000bbb0 tx=0x7f3d5000b4b0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 1 -- 172.21.15.153:0/1807057070 shutdown_connections 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 1 --2- 172.21.15.153:0/1807057070 >> v2:172.21.15.181:3300/0 conn(0x7f3d5c09a8a0 0x7f3d5c09acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 1 --2- 172.21.15.153:0/1807057070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c12ab00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.787+0000 7f3d6306f640 1 -- 172.21.15.153:0/1807057070 >> 172.21.15.153:0/1807057070 conn(0x7f3d5c0672b0 msgr2=0x7f3d5c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 1 -- 172.21.15.153:0/1807057070 shutdown_connections 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 1 -- 172.21.15.153:0/1807057070 wait complete. 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 10 monclient: monmap: 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 1 Processor -- start 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 1 -- start start 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.788+0000 7f3d6306f640 10 monclient: init 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c13d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c0974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 0x7f3d5c13d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c09a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c12dd40 0x7f3d5c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c12dd40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c098300 con 0x7f3d5c12dd40 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c1199b0 con 0x7f3d5c09a8a0 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c129ee0 con 0x7f3d5c0974f0 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6306f640 10 monclient(hunting): authenticate will time out at 1699.428404s 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c13d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c12dd40 0x7f3d5c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c13d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36204/0 (socket says 172.21.15.153:36204) 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 -- 172.21.15.153:0/3869153108 learned_addr learned my addr 172.21.15.153:0/3869153108 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): get_auth_request con 0x7f3d5c0974f0 auth_method 0 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): get_auth_request con 0x7f3d5c12dd40 auth_method 0 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 0x7f3d5c13d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request con 0x7f3d5c09a8a0 auth_method 0 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 msgr2=0x7f3d5c13d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:45:01.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 msgr2=0x7f3d5c13d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 0x7f3d5c13d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 0x7f3d5c13d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient(hunting): handle_auth_done global_id 194189 payload 824 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 msgr2=0x7f3d5c13d9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c09a8a0 0x7f3d5c13d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c12dd40 msgr2=0x7f3d5c06d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c12dd40 0x7f3d5c06d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 monclient: found mon.noname-c 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d60e0b640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5000b6f0 con 0x7f3d5c0974f0 2024-09-15T03:45:01.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.789+0000 7f3d6160c640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c12dd40 0x7f3d5c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:01.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d60e0b640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 msgr2=0x7f3d5c13d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d60e0b640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c13d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000b4f0 tx=0x7f3d5003a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d500359b0 con 0x7f3d5c0974f0 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: dump: 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.813 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d50035b30 con 0x7f3d5c0974f0 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 5 monclient: authenticate success, global_id 194189 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 10 monclient: _renew_subs 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:01.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c06efd0 con 0x7f3d5c0974f0 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5003e400 con 0x7f3d5c0974f0 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d58ff9640 10 monclient: dump: 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d5affd640 4 set_mon_vals no callback set 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 10 monclient: _renew_subs 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.792+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c117680 con 0x7f3d5c0974f0 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d6306f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d6306f640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 0x7f3d5c06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c065f20 con 0x7f3d5c06da20 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3d50046030 con 0x7f3d5c0974f0 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 0x7f3d5c06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient: get_auth_request con 0x7f3d5c06da20 auth_method 0 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d58ff9640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038790 0x7f3d4003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3d50044070 con 0x7f3d5c0974f0 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_done global_id 194192 payload 824 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d5bfff640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d5c04bf60 con 0x7f3d5c06da20 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d6160c640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038790 0x7f3d4003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.793+0000 7f3d6160c640 10 monclient: get_auth_request con 0x7f3d40038790 auth_method 0 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.795+0000 7f3d5bfff640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 msgr2=0x7f3d5c06ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.795+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 0x7f3d5c06ddf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.795+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48014880 con 0x7f3d5c06da20 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.795+0000 7f3d58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.795+0000 7f3d58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.795+0000 7f3d58ff9640 10 monclient: dump: 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.819 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3d480149e0 con 0x7f3d5c06da20 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 msgr2=0x7f3d5c06ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 0x7f3d5c06ddf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48008040 comp rx=0 tx=0).stop 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 msgr2=0x7f3d5c06ddf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:01.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d58ff9640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06da20 0x7f3d5c06ddf0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48008040 comp rx=0 tx=0).stop 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.796+0000 7f3d6160c640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038790 0x7f3d4003ac40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3d54007fd0 tx=0x7f3d540025b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d6306f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d6306f640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 0x7f3d5c0696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d6306f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c097aa0 con 0x7f3d5c06e330 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 0x7f3d5c0696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 10 monclient: get_auth_request con 0x7f3d5c06e330 auth_method 0 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.801+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.802+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.802+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_done global_id 194195 payload 824 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.802+0000 7f3d5bfff640 1 -- 172.21.15.153:0/3869153108 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d5c12a5e0 con 0x7f3d5c06e330 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d5bfff640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 msgr2=0x7f3d5c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d5bfff640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 0x7f3d5c0696a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d4801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48021840 con 0x7f3d5c06e330 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:01.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 10 monclient: dump: 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f3d48026070 con 0x7f3d5c06e330 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 msgr2=0x7f3d5c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 0x7f3d5c0696a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d4801d000 comp rx=0 tx=0).stop 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 msgr2=0x7f3d5c0696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d58ff9640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c06e330 0x7f3d5c0696a0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d4801d000 comp rx=0 tx=0).stop 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:01.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 166, 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:01.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:45:01.826 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.827 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.828 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:01.829 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.830 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.831 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.832 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:01.834 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:01.835 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:01.836 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:01.837 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.804+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038790 msgr2=0x7f3d4003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3d40038790 0x7f3d4003ac40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3d54007fd0 tx=0x7f3d540025b0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 10 monclient: shutdown 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 msgr2=0x7f3d5c13d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 --2- 172.21.15.153:0/3869153108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d5c0974f0 0x7f3d5c13d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000b4f0 tx=0x7f3d5003a000 comp rx=0 tx=0).stop 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6160c640 1 -- 172.21.15.153:0/3869153108 reap_dead start 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6160c640 1 -- 172.21.15.153:0/3869153108 reap_dead start 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 shutdown_connections 2024-09-15T03:45:01.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 >> 172.21.15.153:0/3869153108 conn(0x7f3d5c0672b0 msgr2=0x7f3d5c12d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 shutdown_connections 2024-09-15T03:45:01.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.805+0000 7f3d6306f640 1 -- 172.21.15.153:0/3869153108 wait complete. 2024-09-15T03:45:01.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:01.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:45:01.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:45:01.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.933+0000 7fad5144f640 10 monclient: get_monmap_and_config 2024-09-15T03:45:01.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.933+0000 7fad5144f640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.933+0000 7fad5144f640 10 monclient: monmap: 2024-09-15T03:45:01.934 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.934 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:01.934104+0000 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:01.934104+0000 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:01.935 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 Processor -- start 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 -- start start 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient: init 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad4c1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fad4c09ea50 0x7fad4c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): picked mon.noname-c con 0x7fad4c09ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): picked mon.noname-b con 0x7fad4c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c132640 con 0x7fad4c1318f0 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c131cc0 con 0x7fad4c09ea50 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c09fbf0 con 0x7fad4c136c20 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad5144f640 10 monclient(hunting): authenticate will time out at 1699.573744s 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fad4c1318f0 0x7fad4c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44846/0 (socket says 172.21.15.153:44846) 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4affd640 1 -- 172.21.15.153:0/1116267673 learned_addr learned my addr 172.21.15.153:0/1116267673 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request con 0x7fad4c136c20 auth_method 0 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.934+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_done global_id 194198 payload 274 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad43fff640 1 --2- 172.21.15.153:0/1116267673 >> v2:172.21.15.181:3300/0 conn(0x7fad4c09ea50 0x7fad4c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 1 -- 172.21.15.153:0/1116267673 >> v2:172.21.15.181:3300/0 conn(0x7fad4c09ea50 msgr2=0x7fad4c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 1 --2- 172.21.15.153:0/1116267673 >> v2:172.21.15.181:3300/0 conn(0x7fad4c09ea50 0x7fad4c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 1 -- 172.21.15.153:0/1116267673 >> v1:172.21.15.153:6789/0 conn(0x7fad4c1318f0 legacy=0x7fad4c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad4b7fe640 1 -- 172.21.15.153:0/1116267673 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c09fe30 con 0x7fad4c136c20 2024-09-15T03:45:01.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad43fff640 10 monclient: get_auth_request con 0x7fad4c09ea50 auth_method 0 2024-09-15T03:45:01.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.935+0000 7fad43fff640 1 --2- 172.21.15.153:0/1116267673 >> v2:172.21.15.181:3300/0 conn(0x7fad4c09ea50 0x7fad4c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.937+0000 7fad4b7fe640 1 -- 172.21.15.153:0/1116267673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 msgr2=0x7fad4c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.937+0000 7fad4b7fe640 1 --2- 172.21.15.153:0/1116267673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c139000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fad44009e60 tx=0x7fad44009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dca09c9bf6948be4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 1 -- 172.21.15.153:0/1116267673 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44007840 con 0x7fad4c136c20 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: dump: 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.947 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 1 -- 172.21.15.153:0/1116267673 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4402d050 con 0x7fad4c136c20 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 5 monclient: authenticate success, global_id 194198 2024-09-15T03:45:01.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:01.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:01.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 1 -- 172.21.15.153:0/1116267673 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44007bd0 con 0x7fad4c136c20 2024-09-15T03:45:01.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 4 set_mon_vals no callback set 2024-09-15T03:45:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad4a7fc640 10 monclient: dump: 2024-09-15T03:45:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 10 monclient: shutdown 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 -- 172.21.15.153:0/1116267673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 msgr2=0x7fad4c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 --2- 172.21.15.153:0/1116267673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c139000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fad44009e60 tx=0x7fad44009780 comp rx=0 tx=0).stop 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 -- 172.21.15.153:0/1116267673 shutdown_connections 2024-09-15T03:45:01.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 --2- 172.21.15.153:0/1116267673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c139000 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 --2- 172.21.15.153:0/1116267673 >> v2:172.21.15.181:3300/0 conn(0x7fad4c09ea50 0x7fad4c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 -- 172.21.15.153:0/1116267673 >> 172.21.15.153:0/1116267673 conn(0x7fad4c067110 msgr2=0x7fad4c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 -- 172.21.15.153:0/1116267673 shutdown_connections 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.938+0000 7fad5144f640 1 -- 172.21.15.153:0/1116267673 wait complete. 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.939+0000 7fad5144f640 10 monclient: build_initial_monmap 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.939+0000 7fad5144f640 10 monclient: monmap: 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:01.954 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.939+0000 7fad5144f640 1 Processor -- start 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.939+0000 7fad5144f640 1 -- start start 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.939+0000 7fad5144f640 10 monclient: init 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 0x7fad4c099680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): picked mon.noname-a con 0x7fad4c09ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 0x7fad4c099bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): picked mon.noname-b con 0x7fad4c1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c09a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): picked mon.noname-c con 0x7fad4c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c11cd80 con 0x7fad4c09ea50 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c09afa0 con 0x7fad4c1318f0 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c11c9a0 con 0x7fad4c136c20 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad5144f640 10 monclient(hunting): authenticate will time out at 1699.579148s 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 0x7fad4c099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c09a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 0x7fad4c099680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32772/0 (socket says 172.21.15.153:32772) 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 -- 172.21.15.153:0/2580962566 learned_addr learned my addr 172.21.15.153:0/2580962566 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): get_auth_request con 0x7fad4c09ea50 auth_method 0 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request con 0x7fad4c136c20 auth_method 0 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 0x7fad4c099bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 10 monclient(hunting): get_auth_request con 0x7fad4c1318f0 auth_method 0 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 msgr2=0x7fad4c099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 msgr2=0x7fad4c099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 0x7fad4c099bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 0x7fad4c099bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:01.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient(hunting): handle_auth_done global_id 174435 payload 824 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 msgr2=0x7fad4c09a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c09a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 msgr2=0x7fad4c099bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c1318f0 0x7fad4c099bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 monclient: found mon.noname-a 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4affd640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44009a20 con 0x7fad4c09ea50 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.940+0000 7fad4b7fe640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad4c136c20 0x7fad4c09a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad4affd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 msgr2=0x7fad4c099680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:01.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad4affd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 0x7fad4c099680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00c610 tx=0x7fad3c004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c005cb0 con 0x7fad4c09ea50 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: dump: 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.960 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad5144f640 5 monclient: authenticate success, global_id 174435 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c005e30 con 0x7fad4c09ea50 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad5144f640 10 monclient: _renew_subs 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad5144f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c11c6c0 con 0x7fad4c09ea50 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:01.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c007340 con 0x7fad4c09ea50 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad42ffd640 10 monclient: dump: 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:01.962 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.942+0000 7fad49ffb640 4 set_mon_vals no callback set 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 10 monclient: _renew_subs 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c09b830 con 0x7fad4c09ea50 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fad3c01b400 con 0x7fad4c09ea50 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.943+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c065e00 con 0x7fad4c09a640 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad42ffd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fad3c04c190 con 0x7fad4c09ea50 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 10 monclient: get_auth_request con 0x7fad4c09a640 auth_method 0 2024-09-15T03:45:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad4b7fe640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad4b7fe640 10 monclient: get_auth_request con 0x7fad2c038740 auth_method 0 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.944+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:01.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:01.946+0000 7fad4b7fe640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fad4c076600 tx=0x7fad44037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 10 monclient: get_auth_request con 0x7fad4c09a640 auth_method 0 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:02.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.145+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:02.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:02.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 10 monclient: get_auth_request con 0x7fad4c09a640 auth_method 0 2024-09-15T03:45:02.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:02.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:02.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:02.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:45:02.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:02.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:02.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:02.546+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.347+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.347+0000 7fad43fff640 10 monclient: get_auth_request con 0x7fad4c09a640 auth_method 0 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.347+0000 7fad43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.347+0000 7fad43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.347+0000 7fad43fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.348+0000 7fad43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.348+0000 7fad43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.348+0000 7fad43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.348+0000 7fad43fff640 10 monclient(hunting): handle_auth_done global_id 204100 payload 824 2024-09-15T03:45:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.348+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad4c11b900 con 0x7fad4c09a640 2024-09-15T03:45:03.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.350+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:03.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.350+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0683f0 tx=0x7fad34008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400abd0 con 0x7fad4c09a640 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 10 monclient: dump: 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fad3400c040 con 0x7fad4c09a640 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0683f0 tx=0x7fad34008040 comp rx=0 tx=0).stop 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 msgr2=0x7fad4c02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:03.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.351+0000 7fad42ffd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c09a640 0x7fad4c02aa40 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0683f0 tx=0x7fad34008040 comp rx=0 tx=0).stop 2024-09-15T03:45:03.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.357+0000 7fad5144f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:03.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.357+0000 7fad5144f640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 0x7fad4c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:03.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.357+0000 7fad5144f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:03.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.357+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c09ba50 con 0x7fad4c05e310 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 0x7fad4c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient: get_auth_request con 0x7fad4c05e310 auth_method 0 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:03.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 10 monclient(hunting): handle_auth_done global_id 204103 payload 824 2024-09-15T03:45:03.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.358+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fad4c04bf60 con 0x7fad4c05e310 2024-09-15T03:45:03.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.360+0000 7fad43fff640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 msgr2=0x7fad4c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:03.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.360+0000 7fad43fff640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 0x7fad4c05e710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad3401e000 tx=0x7fad34021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34027070 con 0x7fad4c05e310 2024-09-15T03:45:03.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 10 monclient: dump: 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:03.364 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fad340219e0 con 0x7fad4c05e310 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 msgr2=0x7fad4c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 0x7fad4c05e710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad3401e000 tx=0x7fad34021000 comp rx=0 tx=0).stop 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 msgr2=0x7fad4c05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:03.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad42ffd640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c05e310 0x7fad4c05e710 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad3401e000 tx=0x7fad34021000 comp rx=0 tx=0).stop 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 166, 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:03.366 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:03.367 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:03.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:03.369 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:03.370 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:03.370 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:03.370 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:03.371 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:03.371 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:03.371 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:03.371 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:03.373 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:03.373 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:03.373 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:03.373 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:03.373 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:03.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.375 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:03.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:03.375 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:03.376 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:03.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.376 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:03.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:03.377 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:03.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:03.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:03.378 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:03.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:03.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.378 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:03.378 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:03.379 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:03.379 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:03.379 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:03.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.380 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:03.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:03.380 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:03.381 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:03.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.381 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:03.382 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:03.382 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:03.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.382 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:03.383 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:03.383 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:03.383 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.383 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:03.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.383 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:03.384 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:03.384 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:03.384 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:03.384 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:03.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.385 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:03.385 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:03.385 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:03.385 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:03.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.385 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:03.386 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:03.386 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:03.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.387 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:03.387 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:03.387 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:03.387 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.387 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:03.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.388 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:03.388 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:03.388 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:03.388 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:03.389 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:03.389 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.389 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:03.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.389 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:03.390 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:03.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.390 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:03.390 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:03.390 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:03.390 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.391 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:03.391 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:03.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.391 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:03.392 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:03.392 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:03.392 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:45:03.393 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:03.394 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:03.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:03.394 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:03.394 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:03.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 msgr2=0x7fad2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fad2c038740 0x7fad2c03abf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fad4c076600 tx=0x7fad44037000 comp rx=0 tx=0).stop 2024-09-15T03:45:03.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 10 monclient: shutdown 2024-09-15T03:45:03.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:03.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 msgr2=0x7fad4c099680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 --2- 172.21.15.153:0/2580962566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad4c09ea50 0x7fad4c099680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00c610 tx=0x7fad3c004960 comp rx=0 tx=0).stop 2024-09-15T03:45:03.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad4b7fe640 1 -- 172.21.15.153:0/2580962566 reap_dead start 2024-09-15T03:45:03.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad4b7fe640 1 -- 172.21.15.153:0/2580962566 reap_dead start 2024-09-15T03:45:03.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 shutdown_connections 2024-09-15T03:45:03.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 >> 172.21.15.153:0/2580962566 conn(0x7fad4c067110 msgr2=0x7fad4c0a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:03.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 shutdown_connections 2024-09-15T03:45:03.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.361+0000 7fad5144f640 1 -- 172.21.15.153:0/2580962566 wait complete. 2024-09-15T03:45:03.398 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:03.399 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:45:03.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:45:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.500+0000 7f175831d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.500+0000 7f175831d640 10 monclient: build_initial_monmap 2024-09-15T03:45:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.500+0000 7f175831d640 10 monclient: monmap: 2024-09-15T03:45:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:03.501564+0000 2024-09-15T03:45:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:03.501564+0000 2024-09-15T03:45:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.501+0000 7f175831d640 1 Processor -- start 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.501+0000 7f175831d640 1 -- start start 2024-09-15T03:45:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.501+0000 7f175831d640 10 monclient: init 2024-09-15T03:45:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:45:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient(hunting): picked mon.noname-b con 0x7f17500981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1750131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1750132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1750098ad0 con 0x7f1750131da0 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1750067f30 con 0x7f1750132170 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1750131bd0 con 0x7f17500981c0 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f175831d640 10 monclient(hunting): authenticate will time out at 1701.141198s 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17558b8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1750131da0 0x7f1750067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44854/0 (socket says 172.21.15.153:44854) 2024-09-15T03:45:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17558b8640 1 -- 172.21.15.153:0/316790437 learned_addr learned my addr 172.21.15.153:0/316790437 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): get_auth_request con 0x7f17500981c0 auth_method 0 2024-09-15T03:45:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient(hunting): handle_auth_done global_id 194204 payload 274 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 1 -- 172.21.15.153:0/316790437 >> v1:172.21.15.181:6789/0 conn(0x7f1750132170 legacy=0x7f1750067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 1 -- 172.21.15.153:0/316790437 >> v1:172.21.15.153:6789/0 conn(0x7f1750131da0 legacy=0x7f1750067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient: _finish_hunting 0 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 1 monclient: found mon.noname-b 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.502+0000 7f17560b9640 1 -- 172.21.15.153:0/316790437 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175009ca60 con 0x7f17500981c0 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17560b9640 1 -- 172.21.15.153:0/316790437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 msgr2=0x7f1750098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17560b9640 1 --2- 172.21.15.153:0/316790437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750098590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f174c009880 tx=0x7f174c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76845d187de83465 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 1 -- 172.21.15.153:0/316790437 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c032440 con 0x7f17500981c0 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: dump: 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: _finish_auth 0 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 1 -- 172.21.15.153:0/316790437 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c0325a0 con 0x7f17500981c0 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 1 -- 172.21.15.153:0/316790437 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c032940 con 0x7f17500981c0 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f175831d640 5 monclient: authenticate success, global_id 194204 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f175831d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f175831d640 4 set_mon_vals no callback set 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.505+0000 7f17550b7640 10 monclient: dump: 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.519 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 10 monclient: shutdown 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 -- 172.21.15.153:0/316790437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 msgr2=0x7f1750098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 --2- 172.21.15.153:0/316790437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750098590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f174c009880 tx=0x7f174c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 -- 172.21.15.153:0/316790437 shutdown_connections 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 --2- 172.21.15.153:0/316790437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750098590 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 -- 172.21.15.153:0/316790437 >> 172.21.15.153:0/316790437 conn(0x7f17500a1e30 msgr2=0x7f17500a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 -- 172.21.15.153:0/316790437 shutdown_connections 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 1 -- 172.21.15.153:0/316790437 wait complete. 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 10 monclient: build_initial_monmap 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.506+0000 7f175831d640 10 monclient: monmap: 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 Processor -- start 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 -- start start 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient: init 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): picked mon.noname-c con 0x7f17500981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 0x7f1750141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:03.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): picked mon.noname-a con 0x7f1750131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1750132170 0x7f17501421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): picked mon.noname-b con 0x7f1750132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1750121a80 con 0x7f1750131da0 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1750098cb0 con 0x7f1750132170 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500a0bf0 con 0x7f17500981c0 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f175831d640 10 monclient(hunting): authenticate will time out at 1701.146516s 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 0x7f1750141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36248/0 (socket says 172.21.15.153:36248) 2024-09-15T03:45:03.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 1 -- 172.21.15.153:0/3520189274 learned_addr learned my addr 172.21.15.153:0/3520189274 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:03.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): get_auth_request con 0x7f1750131da0 auth_method 0 2024-09-15T03:45:03.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:03.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:03.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:03.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): get_auth_request con 0x7f17500981c0 auth_method 0 2024-09-15T03:45:03.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:03.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:03.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:03.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:03.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:03.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:03.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:03.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:03.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:03.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17568ba640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1750132170 0x7f17501421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17568ba640 10 monclient(hunting): get_auth_request con 0x7f1750132170 auth_method 0 2024-09-15T03:45:03.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17568ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:03.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17568ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:03.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17568ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient(hunting): handle_auth_done global_id 174438 payload 824 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 -- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 msgr2=0x7f1750141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 -- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1750132170 msgr2=0x7f17501421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1750132170 0x7f17501421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient: _finish_hunting 0 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 monclient: found mon.noname-a 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17558b8640 1 -- 172.21.15.153:0/3520189274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c009390 con 0x7f1750131da0 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17560b9640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.507+0000 7f17568ba640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1750132170 0x7f17501421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:03.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17558b8640 1 -- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 msgr2=0x7f1750141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17558b8640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 0x7f1750141c70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f174000c4c0 tx=0x7f1740007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740005cb0 con 0x7f1750131da0 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: dump: 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:03.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 5 monclient: authenticate success, global_id 174438 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1740005e30 con 0x7f1750131da0 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 10 monclient: _renew_subs 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1750120540 con 0x7f1750131da0 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740007320 con 0x7f1750131da0 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:03.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17467fc640 10 monclient: dump: 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 10 monclient: _renew_subs 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175011f970 con 0x7f1750131da0 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.510+0000 7f17548b6640 4 set_mon_vals no callback set 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f174001b3b0 con 0x7f1750131da0 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f17467fc640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1734038740 0x7f173403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f175831d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f175831d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f175831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1750120350 con 0x7f1750131da0 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f174004ba90 con 0x7f1750131da0 2024-09-15T03:45:03.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f17560b9640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1734038740 0x7f173403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:03.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.511+0000 7f17560b9640 10 monclient: get_auth_request con 0x7f1734038740 auth_method 0 2024-09-15T03:45:03.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.514+0000 7f17560b9640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1734038740 0x7f173403abf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f174c035040 tx=0x7f174c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:03.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.515+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1740017080 con 0x7f1750131da0 2024-09-15T03:45:03.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.515+0000 7f17467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:03.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.515+0000 7f17467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:03.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.729+0000 7f175831d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:45:03.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.729+0000 7f175831d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:45:03.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.729+0000 7f175831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:03.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.729+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f17501226c0 con 0x7f1750131da0 2024-09-15T03:45:03.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.730+0000 7f17467fc640 1 -- 172.21.15.153:0/3520189274 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f17400076d0 con 0x7f1750131da0 2024-09-15T03:45:03.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.730+0000 7f17467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:45:03.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.730+0000 7f17467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:03.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.730+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.730+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=1,health=12,kv=8,logm=51} crc {auth=3482284107,config=1228700967,health=1287409668,kv=3235807718,logm=577408539}) 2024-09-15T03:45:03.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.730+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=7,logm=50} crc {auth=4282824459,config=3140514009,health=1287409668,kv=2145712985,logm=2242590633}) 2024-09-15T03:45:03.734 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.731+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.734 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.731+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=1,health=12,kv=8,logm=51} crc {auth=3482284107,config=1228700967,health=1287409668,kv=3235807718,logm=577408539}) 2024-09-15T03:45:03.734 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.731+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=8,logm=51} crc {auth=462246229,config=3140514009,health=89257251,kv=3235807718,logm=2006733512}) 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.732+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.732+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425238372}) 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.732+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2868522322}) 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.732+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.732+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425238372}) 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.732+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835463178}) 2024-09-15T03:45:03.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.736 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889757598}) 2024-09-15T03:45:03.736 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707409791}) 2024-09-15T03:45:03.736 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.736 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889757598}) 2024-09-15T03:45:03.736 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852764669}) 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1734038740 msgr2=0x7f173403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1734038740 0x7f173403abf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f174c035040 tx=0x7f174c032000 comp rx=0 tx=0).stop 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 10 monclient: shutdown 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 msgr2=0x7f1750141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 0x7f1750141c70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f174000c4c0 tx=0x7f1740007f00 comp rx=0 tx=0).stop 2024-09-15T03:45:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 shutdown_connections 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1734038740 0x7f173403abf0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1750132170 0x7f17501421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1750131da0 0x7f1750141c70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 --2- 172.21.15.153:0/3520189274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17500981c0 0x7f1750141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 >> 172.21.15.153:0/3520189274 conn(0x7f17500a1e30 msgr2=0x7f1750096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 shutdown_connections 2024-09-15T03:45:03.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:03.733+0000 7f175831d640 1 -- 172.21.15.153:0/3520189274 wait complete. 2024-09-15T03:45:03.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.734+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.734+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957842133}) 2024-09-15T03:45:03.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.734+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1556030020}) 2024-09-15T03:45:03.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.734+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.734+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957842133}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.734+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635021464}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.735+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.735+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344517861}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.735+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854344146}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.735+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.735+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344517861}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.735+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553737191}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.736+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.736+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870481203}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.736+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=165108615}) 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.736+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.741 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.736+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870481203}) 2024-09-15T03:45:03.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.736+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105397392}) 2024-09-15T03:45:03.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998080678}) 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848533848}) 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998080678}) 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186697364}) 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989176263}) 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763720112}) 2024-09-15T03:45:03.743 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989176263}) 2024-09-15T03:45:03.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.737+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837889348}) 2024-09-15T03:45:03.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:45:03.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373235870}) 2024-09-15T03:45:03.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307133663}) 2024-09-15T03:45:03.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373235870}) 2024-09-15T03:45:03.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133564537}) 2024-09-15T03:45:03.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.935+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.935+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2321538749}) 2024-09-15T03:45:03.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.935+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2349124897}) 2024-09-15T03:45:03.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.935+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:03.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.935+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2321538749}) 2024-09-15T03:45:03.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:03.935+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766589173}) 2024-09-15T03:45:04.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.318+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.318+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725393930}) 2024-09-15T03:45:04.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.318+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1506956775}) 2024-09-15T03:45:04.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.318+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.318+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725393930}) 2024-09-15T03:45:04.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.318+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43953186}) 2024-09-15T03:45:04.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.321+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.321+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886064063}) 2024-09-15T03:45:04.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.321+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2658614986}) 2024-09-15T03:45:04.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.321+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.321+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886064063}) 2024-09-15T03:45:04.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.321+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047771135}) 2024-09-15T03:45:04.327 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.327 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119734563}) 2024-09-15T03:45:04.327 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2308241400}) 2024-09-15T03:45:04.327 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119734563}) 2024-09-15T03:45:04.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449787346}) 2024-09-15T03:45:04.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.692+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.692+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731644591}) 2024-09-15T03:45:04.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.692+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537336931}) 2024-09-15T03:45:04.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.692+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.692+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731644591}) 2024-09-15T03:45:04.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.692+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1930228707}) 2024-09-15T03:45:04.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.693+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.693+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807321078}) 2024-09-15T03:45:04.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.693+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484310630}) 2024-09-15T03:45:04.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.693+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.693+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=807321078}) 2024-09-15T03:45:04.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.693+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1616442309}) 2024-09-15T03:45:04.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.694+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.694+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920006954}) 2024-09-15T03:45:04.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.694+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030139873}) 2024-09-15T03:45:04.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.694+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.694+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920006954}) 2024-09-15T03:45:04.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.694+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334373956}) 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.695+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.695+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087702}) 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.695+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2667813377}) 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.695+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.695+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087702}) 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.695+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683118490}) 2024-09-15T03:45:04.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.696+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.696+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155241326}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.696+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=544448365}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.696+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.696+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155241326}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.696+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1340236550}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202918630}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325191971}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202918630}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527548654}) 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761855785}) 2024-09-15T03:45:04.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1897537492}) 2024-09-15T03:45:04.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761855785}) 2024-09-15T03:45:04.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.697+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978482609}) 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.698+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.698+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305807517}) 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.698+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2518499839}) 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.698+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.698+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305807517}) 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.698+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721797319}) 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.699+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.699+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=962247408}) 2024-09-15T03:45:04.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.699+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344324943}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.699+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.699+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=962247408}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.699+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717833237}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229243611}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235220893}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229243611}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274435221}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163960841}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771083887}) 2024-09-15T03:45:04.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163960841}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.700+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615262669}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.701+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.701+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3752676726}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.701+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=334388192}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.701+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.701+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3752676726}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.701+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815232322}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.702+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.702+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901730283}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.702+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1236487934}) 2024-09-15T03:45:04.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.702+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.702+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901730283}) 2024-09-15T03:45:04.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.702+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1816692356}) 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.703+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.703+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995544519}) 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.703+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2846156247}) 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.703+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.703+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995544519}) 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.703+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2512527334}) 2024-09-15T03:45:04.710 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277192568}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797342789}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277192568}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192692344}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881508814}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2446030274}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.704+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881508814}) 2024-09-15T03:45:04.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3608971016}) 2024-09-15T03:45:04.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457203775}) 2024-09-15T03:45:04.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139292026}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457203775}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.705+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824143374}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.706+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.706+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001070289}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.706+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=530131418}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.706+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.706+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001070289}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.706+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2155759855}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.707+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.707+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826906946}) 2024-09-15T03:45:04.713 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.707+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4289447126}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.707+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.707+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826906946}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.707+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824698771}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001173492}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628906867}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001173492}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646778948}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.708+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297130906}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1218323731}) 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297130906}) 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512869730}) 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033988993}) 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4001488532}) 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033988993}) 2024-09-15T03:45:04.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.709+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=615169569}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.710+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.710+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256976613}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.710+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1704453426}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.710+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.710+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256976613}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.710+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188038208}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.711+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.711+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276039150}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.711+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585162665}) 2024-09-15T03:45:04.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.711+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.711+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276039150}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.711+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528946724}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390890309}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3587105243}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390890309}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210843719}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403947478}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1918332197}) 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.712+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403947478}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3088754584}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808242706}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993506923}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808242706}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.713+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1974269270}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.714+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.714+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828855795}) 2024-09-15T03:45:04.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.714+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2371636629}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.714+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.714+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828855795}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.714+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516403936}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.715+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.715+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344343587}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.715+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516697312}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.715+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.715+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344343587}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.715+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984068479}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.716+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.716+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279831275}) 2024-09-15T03:45:04.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.716+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256839850}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.716+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.716+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279831275}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.716+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=445186918}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149441476}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823658341}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149441476}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3459723085}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291471113}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536906898}) 2024-09-15T03:45:04.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291471113}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.717+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837416226}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.718+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.718+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337299532}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.718+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613766689}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.718+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.718+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337299532}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.718+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3432836029}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.719+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.719+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152052092}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.719+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804415494}) 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.719+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.727 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.719+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152052092}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.719+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3760511063}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.720+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.720+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405083316}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.720+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3317451559}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.720+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.720+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405083316}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.720+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522935220}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.721+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.721+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392492310}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.721+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669733354}) 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.721+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.721+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392492310}) 2024-09-15T03:45:04.729 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.721+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2820977488}) 2024-09-15T03:45:04.747 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:45:04.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.846+0000 7f365d55d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.846+0000 7f365d55d640 10 monclient: build_initial_monmap 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.846+0000 7f365d55d640 10 monclient: monmap: 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:04.847192+0000 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:04.847192+0000 2024-09-15T03:45:04.847 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:04.848 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 1 Processor -- start 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 1 -- start start 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient: init 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3658131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f365809e910 0x7f365809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient(hunting): picked mon.noname-c con 0x7f365809e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:04.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365809f250 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365809faf0 con 0x7f365809f250 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365809fd30 con 0x7f3658131ab0 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3658132590 con 0x7f365809e910 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f365d55d640 10 monclient(hunting): authenticate will time out at 1702.486586s 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36577fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f365809f250 0x7f3658138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44870/0 (socket says 172.21.15.153:44870) 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36577fe640 1 -- 172.21.15.153:0/3460857745 learned_addr learned my addr 172.21.15.153:0/3460857745 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 1 --2- 172.21.15.153:0/3460857745 >> v2:172.21.15.181:3300/0 conn(0x7f365809e910 0x7f365809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): get_auth_request con 0x7f365809e910 auth_method 0 2024-09-15T03:45:04.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:04.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:04.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:04.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:04.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:04.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.847+0000 7f36567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:04.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.849+0000 7f36577fe640 1 -- 172.21.15.153:0/3460857745 >> v1:172.21.15.153:6789/0 conn(0x7f365809f250 legacy=0x7f3658138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f3655ffb640 1 -- 172.21.15.153:0/3460857745 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92220817 0 0) 0x7f365809faf0 con 0x7f365809f250 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f3655ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f3655ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f3655ffb640 10 monclient(hunting): my global_id is 174441 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f3656ffd640 1 -- 172.21.15.153:0/3460857745 >> v1:172.21.15.181:6789/0 conn(0x7f3658131ab0 legacy=0x7f3658131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f3655ffb640 1 -- 172.21.15.153:0/3460857745 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36440034e0 con 0x7f365809f250 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 10 monclient(hunting): handle_auth_done global_id 204109 payload 274 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 1 -- 172.21.15.153:0/3460857745 >> v1:172.21.15.181:6789/0 conn(0x7f3658131ab0 legacy=0x7f3658131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 1 -- 172.21.15.153:0/3460857745 >> v1:172.21.15.153:6789/0 conn(0x7f365809f250 legacy=0x7f3658138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:04.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 10 monclient: _finish_hunting 0 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 1 monclient: found mon.noname-c 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.850+0000 7f36567fc640 1 -- 172.21.15.153:0/3460857745 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36580a1110 con 0x7f365809e910 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f36567fc640 1 -- 172.21.15.153:0/3460857745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f365809e910 msgr2=0x7f365809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f36567fc640 1 --2- 172.21.15.153:0/3460857745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f365809e910 0x7f365809ed10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3640009950 tx=0x7f36400090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab8205231c7211 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 1 -- 172.21.15.153:0/3460857745 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364002ae10 con 0x7f365809e910 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: dump: 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: _finish_auth 0 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 5 monclient: authenticate success, global_id 204109 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 1 -- 172.21.15.153:0/3460857745 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364002f4b0 con 0x7f365809e910 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:04.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 1 -- 172.21.15.153:0/3460857745 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364002f830 con 0x7f365809e910 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 4 set_mon_vals no callback set 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f3655ffb640 10 monclient: dump: 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:04.858 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 10 monclient: shutdown 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 1 -- 172.21.15.153:0/3460857745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f365809e910 msgr2=0x7f365809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 1 --2- 172.21.15.153:0/3460857745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f365809e910 0x7f365809ed10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3640009950 tx=0x7f36400090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 1 -- 172.21.15.153:0/3460857745 shutdown_connections 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 1 --2- 172.21.15.153:0/3460857745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f365809e910 0x7f365809ed10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 1 -- 172.21.15.153:0/3460857745 >> 172.21.15.153:0/3460857745 conn(0x7f3658067d30 msgr2=0x7f36580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.853+0000 7f365d55d640 1 -- 172.21.15.153:0/3460857745 shutdown_connections 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.854+0000 7f365d55d640 1 -- 172.21.15.153:0/3460857745 wait complete. 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.854+0000 7f365d55d640 10 monclient: build_initial_monmap 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.854+0000 7f365d55d640 10 monclient: monmap: 2024-09-15T03:45:04.859 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:04.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.854+0000 7f365d55d640 1 Processor -- start 2024-09-15T03:45:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 -- start start 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient: init 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 0x7f3658134210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): picked mon.noname-a con 0x7f365809e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f365809f250 0x7f3658134750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): picked mon.noname-c con 0x7f365809f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3658131ab0 0x7f3658134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): picked mon.noname-b con 0x7f3658131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365809bde0 con 0x7f365809e910 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365809dc30 con 0x7f3658131ab0 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36581379b0 con 0x7f365809f250 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f365d55d640 10 monclient(hunting): authenticate will time out at 1702.494426s 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 0x7f3658134210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f365809f250 0x7f3658134750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 0x7f3658134210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32778/0 (socket says 172.21.15.153:32778) 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 -- 172.21.15.153:0/2464927504 learned_addr learned my addr 172.21.15.153:0/2464927504 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): get_auth_request con 0x7f365809f250 auth_method 0 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:04.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): get_auth_request con 0x7f365809e910 auth_method 0 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36577fe640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3658131ab0 0x7f3658134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:04.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient(hunting): handle_auth_done global_id 174444 payload 824 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 -- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f365809f250 msgr2=0x7f3658134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f365809f250 0x7f3658134750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 -- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3658131ab0 msgr2=0x7f3658134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3658131ab0 0x7f3658134c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 monclient: found mon.noname-a 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f3656ffd640 1 -- 172.21.15.153:0/2464927504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3640009390 con 0x7f365809e910 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.855+0000 7f36567fc640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f365809f250 0x7f3658134750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f3656ffd640 1 -- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 msgr2=0x7f3658134210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f3656ffd640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 0x7f3658134210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f365000c910 tx=0x7f36500018f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365000a640 con 0x7f365809e910 2024-09-15T03:45:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: dump: 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.870 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:04.871 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:04.871 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:04.871 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:04.871 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: _finish_auth 0 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 5 monclient: authenticate success, global_id 174444 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 10 monclient: _renew_subs 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365000a7c0 con 0x7f365809e910 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3658137420 con 0x7f365809e910 2024-09-15T03:45:04.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365000ab80 con 0x7f365809e910 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f364effd640 10 monclient: dump: 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:04.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f36557fa640 4 set_mon_vals no callback set 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 10 monclient: _renew_subs 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.858+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3658136770 con 0x7f365809e910 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f36500070f0 con 0x7f365809e910 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f365d55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f365d55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f365d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365804bf60 con 0x7f365809e910 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f364effd640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3638038740 0x7f363803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f365004c310 con 0x7f365809e910 2024-09-15T03:45:04.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f36567fc640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3638038740 0x7f363803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:04.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.859+0000 7f36567fc640 10 monclient: get_auth_request con 0x7f3638038740 auth_method 0 2024-09-15T03:45:04.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.862+0000 7f36567fc640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3638038740 0x7f363803abf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f365806a250 tx=0x7f3640032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:04.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.864+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f36500073a0 con 0x7f365809e910 2024-09-15T03:45:04.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.864+0000 7f364effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:04.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:04.864+0000 7f364effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:04.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687592800}) 2024-09-15T03:45:04.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473942154}) 2024-09-15T03:45:04.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687592800}) 2024-09-15T03:45:04.933 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.932+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1776856901}) 2024-09-15T03:45:04.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.934+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.934+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982757520}) 2024-09-15T03:45:04.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.934+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=629583887}) 2024-09-15T03:45:04.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.934+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:04.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.934+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982757520}) 2024-09-15T03:45:04.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:04.934+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1786476172}) 2024-09-15T03:45:05.047 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.047 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1172310222}) 2024-09-15T03:45:05.047 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2173668336}) 2024-09-15T03:45:05.047 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.047 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1172310222}) 2024-09-15T03:45:05.047 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366192610}) 2024-09-15T03:45:05.049 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.049 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640934000}) 2024-09-15T03:45:05.049 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220383390}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.046+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640934000}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2337627736}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515810794}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482378330}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515810794}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.047+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981645711}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39344518}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73324152}) 2024-09-15T03:45:05.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39344518}) 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179166951}) 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.048+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803494459}) 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755732811}) 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803494459}) 2024-09-15T03:45:05.051 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886568809}) 2024-09-15T03:45:05.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900694794}) 2024-09-15T03:45:05.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997373381}) 2024-09-15T03:45:05.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900694794}) 2024-09-15T03:45:05.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.049+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927619556}) 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983189425}) 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4207140202}) 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983189425}) 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1628715072}) 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511708938}) 2024-09-15T03:45:05.056 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3662746591}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.050+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511708938}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457290668}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044478985}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1711585377}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044478985}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.051+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4053748094}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994523076}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3063700180}) 2024-09-15T03:45:05.057 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994523076}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652081258}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445226121}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679123835}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445226121}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.052+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1257038436}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.053+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.053+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175054403}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.053+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=685062800}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.053+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.053+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175054403}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.053+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939117487}) 2024-09-15T03:45:05.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.054+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.059 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.054+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3904763641}) 2024-09-15T03:45:05.059 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.054+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591300917}) 2024-09-15T03:45:05.059 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.054+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.059 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.054+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3904763641}) 2024-09-15T03:45:05.059 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.054+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547354778}) 2024-09-15T03:45:05.064 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.055+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.064 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.055+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852038627}) 2024-09-15T03:45:05.064 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.055+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413555647}) 2024-09-15T03:45:05.064 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.055+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.055+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852038627}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.055+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026012693}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.056+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.056+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829723059}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.056+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=725427697}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.056+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.056+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829723059}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.056+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346818917}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49239516}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532639382}) 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.065 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49239516}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3929635257}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929048950}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808679326}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929048950}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.057+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042463475}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.058+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.058+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337378535}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.058+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068322516}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.058+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.058+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337378535}) 2024-09-15T03:45:05.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.058+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723769203}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368381376}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3706218099}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368381376}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813160590}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.059+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462075645}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459700923}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462075645}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4294699792}) 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391627935}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512915809}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391627935}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.060+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3273704506}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.061+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.061+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1618572857}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.061+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2609211730}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.061+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.061+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1618572857}) 2024-09-15T03:45:05.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.061+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553154962}) 2024-09-15T03:45:05.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.081+0000 7f365d55d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:05.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.081+0000 7f365d55d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:05.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.081+0000 7f365d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:05.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.081+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f365809d6e0 con 0x7f365809e910 2024-09-15T03:45:05.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.082+0000 7f364effd640 1 -- 172.21.15.153:0/2464927504 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f3650010770 con 0x7f365809e910 2024-09-15T03:45:05.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.082+0000 7f364effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:05.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.082+0000 7f364effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:05.083 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:05.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3638038740 msgr2=0x7f363803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3638038740 0x7f363803abf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f365806a250 tx=0x7f3640032000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 10 monclient: shutdown 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 msgr2=0x7f3658134210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 0x7f3658134210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f365000c910 tx=0x7f36500018f0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 shutdown_connections 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3638038740 0x7f363803abf0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3658131ab0 0x7f3658134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f365809f250 0x7f3658134750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 --2- 172.21.15.153:0/2464927504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f365809e910 0x7f3658134210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 >> 172.21.15.153:0/2464927504 conn(0x7f3658067d30 msgr2=0x7f36580a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 shutdown_connections 2024-09-15T03:45:05.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.084+0000 7f365d55d640 1 -- 172.21.15.153:0/2464927504 wait complete. 2024-09-15T03:45:05.095 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:45:05.095 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:45:05.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:45:05.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:45:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.229+0000 7f984b15d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.229+0000 7f984b15d640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.229+0000 7f984b15d640 10 monclient: monmap: 2024-09-15T03:45:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:05.230500+0000 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:05.230500+0000 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:05.231 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.230+0000 7f984b15d640 1 Processor -- start 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.230+0000 7f984b15d640 1 -- start start 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.230+0000 7f984b15d640 10 monclient: init 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.230+0000 7f984b15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.230+0000 7f984b15d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.230+0000 7f984b15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 0x7f984409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): picked mon.noname-b con 0x7f98440a1ef0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f984409abd0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9844136d00 0x7f98441390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): picked mon.noname-c con 0x7f9844136d00 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:05.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984406f010 con 0x7f984409abd0 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984409d9e0 con 0x7f9844136d00 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984409dcc0 con 0x7f98440a1ef0 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f984b15d640 10 monclient(hunting): authenticate will time out at 1702.870087s 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 0x7f984409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 0x7f984409c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36052/0 (socket says 172.21.15.153:36052) 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 -- 172.21.15.153:0/4240725580 learned_addr learned my addr 172.21.15.153:0/4240725580 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): get_auth_request con 0x7f98440a1ef0 auth_method 0 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f98496fa640 1 --2- 172.21.15.153:0/4240725580 >> v2:172.21.15.181:3300/0 conn(0x7f9844136d00 0x7f98441390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f98496fa640 10 monclient(hunting): get_auth_request con 0x7f9844136d00 auth_method 0 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f98496fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f98496fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f98496fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient(hunting): handle_auth_done global_id 194213 payload 274 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 -- 172.21.15.153:0/4240725580 >> v2:172.21.15.181:3300/0 conn(0x7f9844136d00 msgr2=0x7f98441390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 --2- 172.21.15.153:0/4240725580 >> v2:172.21.15.181:3300/0 conn(0x7f9844136d00 0x7f98441390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 -- 172.21.15.153:0/4240725580 >> v1:172.21.15.181:6789/0 conn(0x7f984409abd0 legacy=0x7f984409afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:05.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 monclient: found mon.noname-b 2024-09-15T03:45:05.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.231+0000 7f9848ef9640 1 -- 172.21.15.153:0/4240725580 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98440a22c0 con 0x7f98440a1ef0 2024-09-15T03:45:05.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f9848ef9640 1 -- 172.21.15.153:0/4240725580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 msgr2=0x7f984409c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f9848ef9640 1 --2- 172.21.15.153:0/4240725580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 0x7f984409c6a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9840009910 tx=0x7f98400091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d00e1e6bdda9751f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 1 -- 172.21.15.153:0/4240725580 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840032440 con 0x7f98440a1ef0 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: dump: 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.243 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 1 -- 172.21.15.153:0/4240725580 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98400325a0 con 0x7f98440a1ef0 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 5 monclient: authenticate success, global_id 194213 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 1 -- 172.21.15.153:0/4240725580 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840032940 con 0x7f98440a1ef0 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 4 set_mon_vals no callback set 2024-09-15T03:45:05.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f983b7fe640 10 monclient: dump: 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 10 monclient: shutdown 2024-09-15T03:45:05.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 1 -- 172.21.15.153:0/4240725580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 msgr2=0x7f984409c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 1 --2- 172.21.15.153:0/4240725580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 0x7f984409c6a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9840009910 tx=0x7f98400091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 1 -- 172.21.15.153:0/4240725580 shutdown_connections 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 1 --2- 172.21.15.153:0/4240725580 >> v2:172.21.15.181:3300/0 conn(0x7f9844136d00 0x7f98441390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 1 --2- 172.21.15.153:0/4240725580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98440a1ef0 0x7f984409c6a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.234+0000 7f984b15d640 1 -- 172.21.15.153:0/4240725580 >> 172.21.15.153:0/4240725580 conn(0x7f98440678a0 msgr2=0x7f98440a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.235+0000 7f984b15d640 1 -- 172.21.15.153:0/4240725580 shutdown_connections 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.235+0000 7f984b15d640 1 -- 172.21.15.153:0/4240725580 wait complete. 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.235+0000 7f984b15d640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.235+0000 7f984b15d640 10 monclient: monmap: 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:05.246 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.235+0000 7f984b15d640 1 Processor -- start 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 -- start start 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient: init 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f984409abd0 0x7f98440942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): picked mon.noname-b con 0x7f984409abd0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98440a1ef0 0x7f9844094820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): picked mon.noname-a con 0x7f98440a1ef0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 0x7f98441346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): picked mon.noname-c con 0x7f9844136d00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844128090 con 0x7f98440a1ef0 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844127db0 con 0x7f984409abd0 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98441266e0 con 0x7f9844136d00 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f984b15d640 10 monclient(hunting): authenticate will time out at 1702.875538s 2024-09-15T03:45:05.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 0x7f98441346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 0x7f98441346c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36064/0 (socket says 172.21.15.153:36064) 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98440a1ef0 0x7f9844094820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 learned_addr learned my addr 172.21.15.153:0/368887398 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): get_auth_request con 0x7f9844136d00 auth_method 0 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): get_auth_request con 0x7f98440a1ef0 auth_method 0 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f983bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f9848ef9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f984409abd0 0x7f98440942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient(hunting): handle_auth_done global_id 194216 payload 824 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f984409abd0 msgr2=0x7f98440942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f984409abd0 0x7f98440942e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98440a1ef0 msgr2=0x7f9844094820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98440a1ef0 0x7f9844094820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 monclient: found mon.noname-c 2024-09-15T03:45:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840009450 con 0x7f9844136d00 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f9848ef9640 10 monclient: get_auth_request con 0x7f984409abd0 auth_method 0 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.236+0000 7f9848ef9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f984409abd0 0x7f98440942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 msgr2=0x7f98441346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f98496fa640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 0x7f98441346c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f983c009fd0 tx=0x7f983c009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c00ee90 con 0x7f9844136d00 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: dump: 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c013590 con 0x7f9844136d00 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 5 monclient: authenticate success, global_id 194216 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 10 monclient: _renew_subs 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9844127b30 con 0x7f9844136d00 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c013950 con 0x7f9844136d00 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f9838ff9640 10 monclient: dump: 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f983affd640 4 set_mon_vals no callback set 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 10 monclient: _renew_subs 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.239+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9844126d00 con 0x7f9844136d00 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f983c013d00 con 0x7f9844136d00 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f984b15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f984b15d640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 0x7f9844134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844065e00 con 0x7f9844094d60 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 0x7f9844134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient: get_auth_request con 0x7f9844094d60 auth_method 0 2024-09-15T03:45:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9838ff9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9830038740 0x7f983003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f983c04cda0 con 0x7f9844136d00 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 10 monclient(hunting): handle_auth_done global_id 174450 payload 824 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f9848ef9640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98441282d0 con 0x7f9844094d60 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f983bfff640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9830038740 0x7f983003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.240+0000 7f983bfff640 10 monclient: get_auth_request con 0x7f9830038740 auth_method 0 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.242+0000 7f9848ef9640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 msgr2=0x7f9844134c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.242+0000 7f9848ef9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 0x7f9844134c00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9840034000 tx=0x7f9840032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.242+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840031a30 con 0x7f9844094d60 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.242+0000 7f9838ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 10 monclient: dump: 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9840031b90 con 0x7f9844094d60 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 msgr2=0x7f9844134c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 0x7f9844134c00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9840034000 tx=0x7f9840032000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 msgr2=0x7f9844134c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f9838ff9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844094d60 0x7f9844134c00 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9840034000 tx=0x7f9840032000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.243+0000 7f983bfff640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9830038740 0x7f983003abf0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9834009a60 tx=0x7f9834006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.248+0000 7f984b15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.248+0000 7f984b15d640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 0x7f9844135510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.248+0000 7f984b15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.248+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844135a50 con 0x7f9844135140 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 0x7f9844135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient: get_auth_request con 0x7f9844135140 auth_method 0 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 10 monclient(hunting): handle_auth_done global_id 174453 payload 824 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.249+0000 7f9848ef9640 1 -- 172.21.15.153:0/368887398 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f984404bf60 con 0x7f9844135140 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9848ef9640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 msgr2=0x7f9844135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9848ef9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 0x7f9844135510 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9840006c70 tx=0x7f984003b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984003b610 con 0x7f9844135140 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 10 monclient: dump: 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7f9840031820 con 0x7f9844135140 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 msgr2=0x7f9844135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 0x7f9844135510 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9840006c70 tx=0x7f984003b000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 msgr2=0x7f9844135510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.251+0000 7f9838ff9640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9844135140 0x7f9844135510 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9840006c70 tx=0x7f984003b000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 166, 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:05.263 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.264 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.267 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:05.270 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.271 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.272 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.273 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.274 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:05.275 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.276 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.276 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:05.276 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:05.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9830038740 msgr2=0x7f983003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9830038740 0x7f983003abf0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9834009a60 tx=0x7f9834006f50 comp rx=0 tx=0).stop 2024-09-15T03:45:05.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 10 monclient: shutdown 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 msgr2=0x7f98441346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 --2- 172.21.15.153:0/368887398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9844136d00 0x7f98441346c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f983c009fd0 tx=0x7f983c009900 comp rx=0 tx=0).stop 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 reap_dead start 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f98496fa640 1 -- 172.21.15.153:0/368887398 reap_dead start 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 shutdown_connections 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 >> 172.21.15.153:0/368887398 conn(0x7f98440678a0 msgr2=0x7f98440a02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 shutdown_connections 2024-09-15T03:45:05.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.252+0000 7f984b15d640 1 -- 172.21.15.153:0/368887398 wait complete. 2024-09-15T03:45:05.279 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:05.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:45:05.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:45:05.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835250502}) 2024-09-15T03:45:05.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992805700}) 2024-09-15T03:45:05.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835250502}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932094707}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.271+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811596702}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293529842}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811596702}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264818034}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938304760}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351679655}) 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.282 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938304760}) 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.272+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1000773220}) 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.273+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.273+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996832031}) 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.273+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2200686459}) 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.273+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.273+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996832031}) 2024-09-15T03:45:05.283 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.273+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3190734166}) 2024-09-15T03:45:05.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=433484250}) 2024-09-15T03:45:05.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=941659635}) 2024-09-15T03:45:05.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=433484250}) 2024-09-15T03:45:05.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.326+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184525531}) 2024-09-15T03:45:05.338 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.337+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.337+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977587550}) 2024-09-15T03:45:05.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.337+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174048164}) 2024-09-15T03:45:05.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.337+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:05.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.338+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977587550}) 2024-09-15T03:45:05.339 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:05.338+0000 7f8586bf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723808315}) 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.380+0000 7f446ef1d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.380+0000 7f446ef1d640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.380+0000 7f446ef1d640 10 monclient: monmap: 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:05.380907+0000 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:05.380907+0000 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:05.382 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:05.382 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:05.382 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.380+0000 7f446ef1d640 1 Processor -- start 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.380+0000 7f446ef1d640 1 -- start start 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.380+0000 7f446ef1d640 10 monclient: init 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44681318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f446809ea50 0x7f446809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): picked mon.noname-a con 0x7f446809ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4468136c20 0x7f4468139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): picked mon.noname-c con 0x7f4468136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:05.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4468131cc0 con 0x7f44681318f0 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446809fbf0 con 0x7f446809ea50 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446809fe30 con 0x7f4468136c20 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446ef1d640 10 monclient(hunting): authenticate will time out at 1703.020491s 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f446809ea50 0x7f446809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f446809ea50 0x7f446809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47600/0 (socket says 172.21.15.153:47600) 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 -- 172.21.15.153:0/2947810817 learned_addr learned my addr 172.21.15.153:0/2947810817 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): get_auth_request con 0x7f446809ea50 auth_method 0 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446d4ba640 1 --2- 172.21.15.153:0/2947810817 >> v2:172.21.15.181:3300/0 conn(0x7f4468136c20 0x7f4468139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446d4ba640 10 monclient(hunting): get_auth_request con 0x7f4468136c20 auth_method 0 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446d4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446d4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f446d4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient(hunting): handle_auth_done global_id 174456 payload 274 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 -- 172.21.15.153:0/2947810817 >> v2:172.21.15.181:3300/0 conn(0x7f4468136c20 msgr2=0x7f4468139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 --2- 172.21.15.153:0/2947810817 >> v2:172.21.15.181:3300/0 conn(0x7f4468136c20 0x7f4468139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 -- 172.21.15.153:0/2947810817 >> v1:172.21.15.181:6789/0 conn(0x7f44681318f0 legacy=0x7f4468131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 monclient: found mon.noname-a 2024-09-15T03:45:05.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:05.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.381+0000 7f4467fff640 1 -- 172.21.15.153:0/2947810817 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44680a1210 con 0x7f446809ea50 2024-09-15T03:45:05.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f4467fff640 1 -- 172.21.15.153:0/2947810817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446809ea50 msgr2=0x7f446809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f4467fff640 1 --2- 172.21.15.153:0/2947810817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446809ea50 0x7f446809ee20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f446809c9e0 tx=0x7f445c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737dff9c5c3ab773 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 1 -- 172.21.15.153:0/2947810817 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c032440 con 0x7f446809ea50 2024-09-15T03:45:05.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:05.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: dump: 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 1 -- 172.21.15.153:0/2947810817 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c0325a0 con 0x7f446809ea50 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f446ef1d640 5 monclient: authenticate success, global_id 174456 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f446ef1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 1 -- 172.21.15.153:0/2947810817 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c032940 con 0x7f446809ea50 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f44677fe640 10 monclient: dump: 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.397 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f446ef1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.384+0000 7f446ef1d640 4 set_mon_vals no callback set 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 10 monclient: shutdown 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 -- 172.21.15.153:0/2947810817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446809ea50 msgr2=0x7f446809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 --2- 172.21.15.153:0/2947810817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446809ea50 0x7f446809ee20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f446809c9e0 tx=0x7f445c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 -- 172.21.15.153:0/2947810817 shutdown_connections 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 --2- 172.21.15.153:0/2947810817 >> v2:172.21.15.181:3300/0 conn(0x7f4468136c20 0x7f4468139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 --2- 172.21.15.153:0/2947810817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446809ea50 0x7f446809ee20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 -- 172.21.15.153:0/2947810817 >> 172.21.15.153:0/2947810817 conn(0x7f4468067110 msgr2=0x7f4468067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 -- 172.21.15.153:0/2947810817 shutdown_connections 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 1 -- 172.21.15.153:0/2947810817 wait complete. 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.385+0000 7f446ef1d640 10 monclient: monmap: 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:05.399 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 Processor -- start 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 -- start start 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient: init 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f446809ea50 0x7f44681309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): picked mon.noname-b con 0x7f446809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:05.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44681318f0 0x7f4468130f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): picked mon.noname-c con 0x7f44681318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 0x7f4468099930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): picked mon.noname-a con 0x7f4468136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446809dfa0 con 0x7f4468136c20 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446809b310 con 0x7f446809ea50 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446811cae0 con 0x7f44681318f0 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446ef1d640 10 monclient(hunting): authenticate will time out at 1703.025887s 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f446d4ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 0x7f4468099930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.386+0000 7f4467fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44681318f0 0x7f4468130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 0x7f4468099930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47612/0 (socket says 172.21.15.153:47612) 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44681318f0 0x7f4468130f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36074/0 (socket says 172.21.15.153:36074) 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 -- 172.21.15.153:0/1798797994 learned_addr learned my addr 172.21.15.153:0/1798797994 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): get_auth_request con 0x7f4468136c20 auth_method 0 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): get_auth_request con 0x7f44681318f0 auth_method 0 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f446809ea50 0x7f44681309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): get_auth_request con 0x7f446809ea50 auth_method 0 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient(hunting): handle_auth_done global_id 174459 payload 824 2024-09-15T03:45:05.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44681318f0 msgr2=0x7f4468130f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44681318f0 0x7f4468130f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f446809ea50 msgr2=0x7f44681309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f446809ea50 0x7f44681309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 monclient: found mon.noname-a 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f446d4ba640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c009450 con 0x7f4468136c20 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.387+0000 7f4467fff640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44681318f0 0x7f4468130f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446d4ba640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 msgr2=0x7f4468099930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446d4ba640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 0x7f4468099930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4460009fa0 tx=0x7f4460009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446000ee60 con 0x7f4468136c20 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: dump: 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460013590 con 0x7f4468136c20 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 5 monclient: authenticate success, global_id 174459 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 10 monclient: _renew_subs 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460013950 con 0x7f4468136c20 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4468141e40 con 0x7f4468136c20 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4464ff9640 10 monclient: dump: 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.406 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f4466ffd640 4 set_mon_vals no callback set 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 10 monclient: _renew_subs 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.389+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4468137500 con 0x7f4468136c20 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4460013cc0 con 0x7f4468136c20 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ef1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ef1d640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 0x7f446809a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4468065e00 con 0x7f446811ce30 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 0x7f446809a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient: get_auth_request con 0x7f446811ce30 auth_method 0 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f4464ff9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4454038740 0x7f445403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f446004cdc0 con 0x7f4468136c20 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.390+0000 7f446ccb9640 10 monclient(hunting): handle_auth_done global_id 194222 payload 824 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.391+0000 7f446ccb9640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f446804bf60 con 0x7f446811ce30 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.391+0000 7f4467fff640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4454038740 0x7f445403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.391+0000 7f4467fff640 10 monclient: get_auth_request con 0x7f4454038740 auth_method 0 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f446ccb9640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 msgr2=0x7f446809a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f446ccb9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 0x7f446809a630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f445800f780 tx=0x7f4458007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445800d8b0 con 0x7f446811ce30 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 10 monclient: dump: 2024-09-15T03:45:05.409 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f445800da10 con 0x7f446811ce30 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:05.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 msgr2=0x7f446809a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 0x7f446809a630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f445800f780 tx=0x7f4458007e10 comp rx=0 tx=0).stop 2024-09-15T03:45:05.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 msgr2=0x7f446809a630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:05.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4464ff9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f446811ce30 0x7f446809a630 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f445800f780 tx=0x7f4458007e10 comp rx=0 tx=0).stop 2024-09-15T03:45:05.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.393+0000 7f4467fff640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4454038740 0x7f445403abf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4468076600 tx=0x7f445c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ef1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ef1d640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 0x7f4468029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ef1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4468135e70 con 0x7f44680298c0 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 0x7f4468029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient: get_auth_request con 0x7f44680298c0 auth_method 0 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 10 monclient(hunting): handle_auth_done global_id 194225 payload 824 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.399+0000 7f446ccb9640 1 -- 172.21.15.153:0/1798797994 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f44680638d0 con 0x7f44680298c0 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.401+0000 7f446ccb9640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 msgr2=0x7f4468029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.401+0000 7f446ccb9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 0x7f4468029cc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f445800f780 tx=0x7f4458019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.401+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458014e20 con 0x7f44680298c0 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.401+0000 7f4464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.401+0000 7f4464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.401+0000 7f4464ff9640 10 monclient: dump: 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f445801d610 con 0x7f44680298c0 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 msgr2=0x7f4468029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 0x7f4468029cc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f445800f780 tx=0x7f4458019000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 msgr2=0x7f4468029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:05.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f4464ff9640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44680298c0 0x7f4468029cc0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f445800f780 tx=0x7f4458019000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 166, 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:05.416 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.417 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.418 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:05.419 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.420 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.421 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:05.422 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:05.423 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.424 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.425 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.426 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:45:05.427 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.428 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.428 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:05.428 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:05.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4454038740 msgr2=0x7f445403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4454038740 0x7f445403abf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4468076600 tx=0x7f445c032000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 10 monclient: shutdown 2024-09-15T03:45:05.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 msgr2=0x7f4468099930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446d4ba640 1 -- 172.21.15.153:0/1798797994 reap_dead start 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 0x7f4468099930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4460009fa0 tx=0x7f4460009900 comp rx=0 tx=0).stop 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 shutdown_connections 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 --2- 172.21.15.153:0/1798797994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4468136c20 0x7f4468099930 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 >> 172.21.15.153:0/1798797994 conn(0x7f4468067110 msgr2=0x7f44680a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 shutdown_connections 2024-09-15T03:45:05.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.402+0000 7f446ef1d640 1 -- 172.21.15.153:0/1798797994 wait complete. 2024-09-15T03:45:05.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:05.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:45:05.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.523+0000 7f3674c2b640 10 monclient: get_monmap_and_config 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.523+0000 7f3674c2b640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.523+0000 7f3674c2b640 10 monclient: monmap: 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:05.523914+0000 2024-09-15T03:45:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:05.523914+0000 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.523+0000 7f3674c2b640 1 Processor -- start 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.523+0000 7f3674c2b640 1 -- start start 2024-09-15T03:45:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.523+0000 7f3674c2b640 10 monclient: init 2024-09-15T03:45:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3670131ab0 0x7f3670131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f3670131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 0x7f367009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f367009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f367009f220 0x7f3670068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): picked mon.noname-c con 0x7f367009f220 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367009faf0 con 0x7f3670131ab0 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367009fd30 con 0x7f367009f220 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36700a1110 con 0x7f367009e910 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f3674c2b640 10 monclient(hunting): authenticate will time out at 1703.163541s 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 0x7f367009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3670131ab0 0x7f3670131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 0x7f367009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36084/0 (socket says 172.21.15.153:36084) 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 -- 172.21.15.153:0/706193152 learned_addr learned my addr 172.21.15.153:0/706193152 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): get_auth_request con 0x7f367009e910 auth_method 0 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): get_auth_request con 0x7f3670131ab0 auth_method 0 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366ed9d640 1 --2- 172.21.15.153:0/706193152 >> v2:172.21.15.181:3300/0 conn(0x7f367009f220 0x7f3670068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366ed9d640 10 monclient(hunting): get_auth_request con 0x7f367009f220 auth_method 0 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient(hunting): handle_auth_done global_id 194228 payload 274 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 -- 172.21.15.153:0/706193152 >> v2:172.21.15.181:3300/0 conn(0x7f367009f220 msgr2=0x7f3670068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 --2- 172.21.15.153:0/706193152 >> v2:172.21.15.181:3300/0 conn(0x7f367009f220 0x7f3670068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 -- 172.21.15.153:0/706193152 >> v2:172.21.15.153:3300/0 conn(0x7f3670131ab0 msgr2=0x7f3670131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 --2- 172.21.15.153:0/706193152 >> v2:172.21.15.153:3300/0 conn(0x7f3670131ab0 0x7f3670131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 monclient: found mon.noname-b 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366dd9b640 1 -- 172.21.15.153:0/706193152 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36700a13f0 con 0x7f367009e910 2024-09-15T03:45:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.524+0000 7f366e59c640 1 --2- 172.21.15.153:0/706193152 >> v2:172.21.15.153:3300/0 conn(0x7f3670131ab0 0x7f3670131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:05.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366dd9b640 1 -- 172.21.15.153:0/706193152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 msgr2=0x7f367009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366dd9b640 1 --2- 172.21.15.153:0/706193152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 0x7f367009ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f367009bd80 tx=0x7f36580091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825e2ff361c58682 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 1 -- 172.21.15.153:0/706193152 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658032440 con 0x7f367009e910 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 10 monclient: dump: 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.540 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f3674c2b640 5 monclient: authenticate success, global_id 194228 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.527+0000 7f366d59a640 1 -- 172.21.15.153:0/706193152 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36580325a0 con 0x7f367009e910 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f366d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f366d59a640 1 -- 172.21.15.153:0/706193152 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658032940 con 0x7f367009e910 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:05.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f366d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 4 set_mon_vals no callback set 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f366d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f366d59a640 10 monclient: dump: 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 10 monclient: shutdown 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 -- 172.21.15.153:0/706193152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 msgr2=0x7f367009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 --2- 172.21.15.153:0/706193152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 0x7f367009ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f367009bd80 tx=0x7f36580091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 -- 172.21.15.153:0/706193152 shutdown_connections 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 --2- 172.21.15.153:0/706193152 >> v2:172.21.15.181:3300/0 conn(0x7f367009f220 0x7f3670068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 --2- 172.21.15.153:0/706193152 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f367009e910 0x7f367009ece0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 --2- 172.21.15.153:0/706193152 >> v2:172.21.15.153:3300/0 conn(0x7f3670131ab0 0x7f3670131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 -- 172.21.15.153:0/706193152 >> 172.21.15.153:0/706193152 conn(0x7f3670067d30 msgr2=0x7f36700a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 -- 172.21.15.153:0/706193152 shutdown_connections 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.528+0000 7f3674c2b640 1 -- 172.21.15.153:0/706193152 wait complete. 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient: monmap: 2024-09-15T03:45:05.543 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 1 Processor -- start 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 1 -- start start 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient: init 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f367009e910 0x7f367009a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f367009e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367009f220 0x7f3670130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f367009f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3670131ab0 0x7f3670140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient(hunting): picked mon.noname-c con 0x7f3670131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.529+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367009bde0 con 0x7f367009e910 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367009dd90 con 0x7f367009f220 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367011def0 con 0x7f3670131ab0 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f3674c2b640 10 monclient(hunting): authenticate will time out at 1703.169021s 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f367009e910 0x7f367009a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f367009e910 0x7f367009a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47624/0 (socket says 172.21.15.153:47624) 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 1 -- 172.21.15.153:0/700115934 learned_addr learned my addr 172.21.15.153:0/700115934 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): get_auth_request con 0x7f367009e910 auth_method 0 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3670131ab0 0x7f3670140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): get_auth_request con 0x7f3670131ab0 auth_method 0 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366dd9b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367009f220 0x7f3670130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366dd9b640 10 monclient(hunting): get_auth_request con 0x7f367009f220 auth_method 0 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient(hunting): handle_auth_done global_id 194231 payload 824 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367009f220 msgr2=0x7f3670130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367009f220 0x7f3670130af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f367009e910 msgr2=0x7f367009a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f367009e910 0x7f367009a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366ed9d640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660009390 con 0x7f3670131ab0 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366dd9b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367009f220 0x7f3670130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.530+0000 7f366e59c640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f367009e910 0x7f367009a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f366ed9d640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3670131ab0 msgr2=0x7f3670140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f366ed9d640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3670131ab0 0x7f3670140e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3668007f00 tx=0x7f3668009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668014420 con 0x7f3670131ab0 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: dump: 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 5 monclient: authenticate success, global_id 194231 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668014580 con 0x7f3670131ab0 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 10 monclient: _renew_subs 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3658009450 con 0x7f3670131ab0 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668014940 con 0x7f3670131ab0 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f36667fc640 10 monclient: dump: 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f366cd99640 4 set_mon_vals no callback set 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 10 monclient: _renew_subs 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.532+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367011ccb0 con 0x7f3670131ab0 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f3674c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f3674c2b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 0x7f3670141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367011ddb0 con 0x7f367011bb50 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3668003020 con 0x7f3670131ab0 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f366e59c640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 0x7f3670141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f36667fc640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3650038740 0x7f365003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f366e59c640 10 monclient: get_auth_request con 0x7f367011bb50 auth_method 0 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f366e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f366e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f366e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.533+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f366804caf0 con 0x7f3670131ab0 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366dd9b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3650038740 0x7f365003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366dd9b640 10 monclient: get_auth_request con 0x7f3650038740 auth_method 0 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366e59c640 10 monclient(hunting): handle_auth_done global_id 204127 payload 824 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.534+0000 7f366e59c640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f367011e110 con 0x7f367011bb50 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f366e59c640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 msgr2=0x7f3670141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f366e59c640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 0x7f3670141370 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f366000f4a0 tx=0x7f36600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f366dd9b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3650038740 0x7f365003abf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3670076760 tx=0x7f3658032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660013a40 con 0x7f367011bb50 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 10 monclient: dump: 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3660013ba0 con 0x7f367011bb50 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 msgr2=0x7f3670141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 0x7f3670141370 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f366000f4a0 tx=0x7f36600090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 msgr2=0x7f3670141370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.536+0000 7f36667fc640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f367011bb50 0x7f3670141370 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f3674c2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f3674c2b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 0x7f36701418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f3674c2b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670063740 con 0x7f3670131030 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f366e59c640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 0x7f36701418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f366e59c640 10 monclient: get_auth_request con 0x7f3670131030 auth_method 0 2024-09-15T03:45:05.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f366e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f366e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.542+0000 7f366e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.543+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.543+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.543+0000 7f366e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.543+0000 7f366e59c640 10 monclient(hunting): handle_auth_done global_id 204130 payload 824 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.543+0000 7f366e59c640 1 -- 172.21.15.153:0/700115934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f367004bf60 con 0x7f3670131030 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f366e59c640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 msgr2=0x7f36701418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f366e59c640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 0x7f36701418b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3660000c00 tx=0x7f366001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660024040 con 0x7f3670131030 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 10 monclient: dump: 2024-09-15T03:45:05.557 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f3660013830 con 0x7f3670131030 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 msgr2=0x7f36701418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 0x7f36701418b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3660000c00 tx=0x7f366001c000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 msgr2=0x7f36701418b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 166, 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.567 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:05.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:05.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:05.572 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:05.573 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.545+0000 7f36667fc640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3670131030 0x7f36701418b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3650038740 msgr2=0x7f365003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3650038740 0x7f365003abf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3670076760 tx=0x7f3658032000 comp rx=0 tx=0).stop 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 10 monclient: shutdown 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3670131ab0 msgr2=0x7f3670140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 --2- 172.21.15.153:0/700115934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3670131ab0 0x7f3670140e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3668007f00 tx=0x7f3668009850 comp rx=0 tx=0).stop 2024-09-15T03:45:05.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f366ed9d640 1 -- 172.21.15.153:0/700115934 reap_dead start 2024-09-15T03:45:05.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f366ed9d640 1 -- 172.21.15.153:0/700115934 reap_dead start 2024-09-15T03:45:05.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 shutdown_connections 2024-09-15T03:45:05.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 >> 172.21.15.153:0/700115934 conn(0x7f3670067d30 msgr2=0x7f36700a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 shutdown_connections 2024-09-15T03:45:05.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.546+0000 7f3674c2b640 1 -- 172.21.15.153:0/700115934 wait complete. 2024-09-15T03:45:05.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:05.575 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:45:05.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:45:05.576 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:45:05.576 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:45:05.576 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:45:05.576 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:45:05.590 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:45:05.590 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:45:05.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:05.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.721+0000 7f9674f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.721+0000 7f9674f1d640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.721+0000 7f9674f1d640 10 monclient: monmap: 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:05.722469+0000 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:05.722469+0000 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:05.723 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 Processor -- start 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 -- start start 2024-09-15T03:45:05.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient: init 2024-09-15T03:45:05.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f96700975f0 0x7f9670095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f96700975f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96700979c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9670097dc0 0x7f967009c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9670097dc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9670098200 con 0x7f96700979c0 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967009d1b0 con 0x7f9670097dc0 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967009d420 con 0x7f96700975f0 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f9674f1d640 10 monclient(hunting): authenticate will time out at 1703.361982s 2024-09-15T03:45:05.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.722+0000 7f966ed9d640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9670097dc0 msgr2=0x7f967009c660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966dd9b640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f96700979c0 legacy=0x7f967009bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966ed9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9670097dc0 0x7f967009c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f96700975f0 0x7f9670095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f96700975f0 0x7f9670095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:55916/0 (socket says 172.21.15.153:55916) 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 -- 172.21.15.153:0/2988816275 learned_addr learned my addr 172.21.15.153:0/2988816275 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): get_auth_request con 0x7f96700975f0 auth_method 0 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient(hunting): handle_auth_done global_id 204133 payload 274 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 -- 172.21.15.153:0/2988816275 >> v2:172.21.15.153:3300/0 conn(0x7f9670097dc0 msgr2=0x7f967009c660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 --2- 172.21.15.153:0/2988816275 >> v2:172.21.15.153:3300/0 conn(0x7f9670097dc0 0x7f967009c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 -- 172.21.15.153:0/2988816275 >> v1:172.21.15.153:6789/0 conn(0x7f96700979c0 legacy=0x7f967009bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 monclient: found mon.noname-c 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:45:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.723+0000 7f966e59c640 1 -- 172.21.15.153:0/2988816275 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967009d9e0 con 0x7f96700975f0 2024-09-15T03:45:05.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966e59c640 1 -- 172.21.15.153:0/2988816275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 msgr2=0x7f9670095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966e59c640 1 --2- 172.21.15.153:0/2988816275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670095ad0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96700713a0 tx=0x7f96600091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31037f5d75305f1a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 1 -- 172.21.15.153:0/2988816275 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660032440 con 0x7f96700975f0 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: dump: 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 1 -- 172.21.15.153:0/2988816275 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96600325a0 con 0x7f96700975f0 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f9674f1d640 5 monclient: authenticate success, global_id 204133 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f9674f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f9674f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 1 -- 172.21.15.153:0/2988816275 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660032940 con 0x7f96700975f0 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f9674f1d640 4 set_mon_vals no callback set 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.726+0000 7f966d59a640 10 monclient: dump: 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 10 monclient: shutdown 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:05.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 -- 172.21.15.153:0/2988816275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 msgr2=0x7f9670095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 --2- 172.21.15.153:0/2988816275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670095ad0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96700713a0 tx=0x7f96600091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 -- 172.21.15.153:0/2988816275 shutdown_connections 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 --2- 172.21.15.153:0/2988816275 >> v2:172.21.15.153:3300/0 conn(0x7f9670097dc0 0x7f967009c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 --2- 172.21.15.153:0/2988816275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670095ad0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 -- 172.21.15.153:0/2988816275 >> 172.21.15.153:0/2988816275 conn(0x7f96700678a0 msgr2=0x7f96700a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 -- 172.21.15.153:0/2988816275 shutdown_connections 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 1 -- 172.21.15.153:0/2988816275 wait complete. 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 10 monclient: build_initial_monmap 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.727+0000 7f9674f1d640 10 monclient: monmap: 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:05.739 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 Processor -- start 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 -- start start 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient: init 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f96700975f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:05.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f96700979c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670097dc0 0x7f9670130120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9670097dc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967012c3b0 con 0x7f9670097dc0 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9670142040 con 0x7f96700975f0 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967012c0d0 con 0x7f96700979c0 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f9674f1d640 10 monclient(hunting): authenticate will time out at 1703.367700s 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966ed9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670097dc0 msgr2=0x7f9670130120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:05.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670097dc0 0x7f9670130120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36132/0 (socket says 172.21.15.153:36132) 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 1 -- 172.21.15.153:0/785932927 learned_addr learned my addr 172.21.15.153:0/785932927 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): get_auth_request con 0x7f96700979c0 auth_method 0 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966e59c640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966e59c640 10 monclient(hunting): get_auth_request con 0x7f96700975f0 auth_method 0 2024-09-15T03:45:05.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.728+0000 7f966e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 10 monclient(hunting): handle_auth_done global_id 194234 payload 824 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 1 -- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 msgr2=0x7f9670141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 1 -- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670097dc0 msgr2=0x7f9670130120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670097dc0 0x7f9670130120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 1 monclient: found mon.noname-c 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:05.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.729+0000 7f966dd9b640 1 -- 172.21.15.153:0/785932927 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660009450 con 0x7f96700979c0 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f966dd9b640 1 -- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 msgr2=0x7f967012fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f966dd9b640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f966400b5c0 tx=0x7f9664009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664003cc0 con 0x7f96700979c0 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: dump: 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.744 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664003e40 con 0x7f96700979c0 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 5 monclient: authenticate success, global_id 194234 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664005340 con 0x7f96700979c0 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f965e7fc640 10 monclient: dump: 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:05.745 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 10 monclient: _renew_subs 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f966cd99640 4 set_mon_vals no callback set 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967009a340 con 0x7f96700979c0 2024-09-15T03:45:05.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 10 monclient: _renew_subs 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.731+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967009a9e0 con 0x7f96700979c0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f966401a460 con 0x7f96700979c0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f9674f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f9674f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f9674f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967004bf60 con 0x7f96700979c0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f965e7fc640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f96500386f0 0x7f965003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f966404c180 con 0x7f96700979c0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f966e59c640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f96500386f0 0x7f965003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.732+0000 7f966e59c640 10 monclient: get_auth_request con 0x7f96500386f0 auth_method 0 2024-09-15T03:45:05.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.735+0000 7f966e59c640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f96500386f0 0x7f965003aba0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9660031000 tx=0x7f9660032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:05.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:05.737+0000 7f966dd9b640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f966400b5c0 tx=0x7f9664009120 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f966401a710 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:45:06.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.565+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f966401a710 con 0x7f96700979c0 2024-09-15T03:45:06.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.565+0000 7f965e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:06.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.565+0000 7f965e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:06.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f9674f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:06.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f9674f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:06.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f9674f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:06.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f967012be50 con 0x7f96700979c0 2024-09-15T03:45:06.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f965e7fc640 1 -- 172.21.15.153:0/785932927 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f9664017310 con 0x7f96700979c0 2024-09-15T03:45:06.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f965e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:06.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.784+0000 7f965e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:06.786 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:06.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f96500386f0 msgr2=0x7f965003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:06.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f96500386f0 0x7f965003aba0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9660031000 tx=0x7f9660032000 comp rx=0 tx=0).stop 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 10 monclient: shutdown 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 msgr2=0x7f967012fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f966400b5c0 tx=0x7f9664009120 comp rx=0 tx=0).stop 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 shutdown_connections 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f96500386f0 0x7f965003aba0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670097dc0 0x7f9670130120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96700979c0 0x7f967012fbe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 --2- 172.21.15.153:0/785932927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96700975f0 0x7f9670141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:06.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 >> 172.21.15.153:0/785932927 conn(0x7f96700678a0 msgr2=0x7f96700961d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:06.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 shutdown_connections 2024-09-15T03:45:06.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:06.786+0000 7f9674f1d640 1 -- 172.21.15.153:0/785932927 wait complete. 2024-09-15T03:45:09.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.903+0000 7fc818f9d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.903+0000 7fc818f9d640 10 monclient: build_initial_monmap 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.903+0000 7fc818f9d640 10 monclient: monmap: 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:09.904006+0000 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:09.904006+0000 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:09.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:09.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:09.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:09.905 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:09.905 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:09.905 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:09.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.903+0000 7fc818f9d640 1 Processor -- start 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.903+0000 7fc818f9d640 1 -- start start 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.903+0000 7fc818f9d640 10 monclient: init 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc8141318f0 0x7fc814131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8141318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc81409ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 0x7fc814139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc814136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc814131cc0 con 0x7fc81409ea50 2024-09-15T03:45:09.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81409f530 con 0x7fc8141318f0 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81409fbf0 con 0x7fc814136c20 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc818f9d640 10 monclient(hunting): authenticate will time out at 1707.543678s 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 0x7fc814139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 0x7fc814139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36140/0 (socket says 172.21.15.153:36140) 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 1 -- 172.21.15.153:0/980054352 learned_addr learned my addr 172.21.15.153:0/980054352 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): get_auth_request con 0x7fc814136c20 auth_method 0 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:09.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.904+0000 7fc812d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc81259c640 1 --2- 172.21.15.153:0/980054352 >> v2:172.21.15.181:3300/0 conn(0x7fc8141318f0 0x7fc814131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc81259c640 10 monclient(hunting): get_auth_request con 0x7fc8141318f0 auth_method 0 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc81259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc81259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc81259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 10 monclient(hunting): handle_auth_done global_id 194237 payload 274 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 1 -- 172.21.15.153:0/980054352 >> v2:172.21.15.181:3300/0 conn(0x7fc8141318f0 msgr2=0x7fc814131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 1 --2- 172.21.15.153:0/980054352 >> v2:172.21.15.181:3300/0 conn(0x7fc8141318f0 0x7fc814131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 1 -- 172.21.15.153:0/980054352 >> v1:172.21.15.181:6789/0 conn(0x7fc81409ea50 legacy=0x7fc81409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 1 monclient: found mon.noname-b 2024-09-15T03:45:09.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:09.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:09.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.905+0000 7fc812d9d640 1 -- 172.21.15.153:0/980054352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81409fe30 con 0x7fc814136c20 2024-09-15T03:45:09.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.907+0000 7fc812d9d640 1 -- 172.21.15.153:0/980054352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 msgr2=0x7fc814139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:09.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.907+0000 7fc812d9d640 1 --2- 172.21.15.153:0/980054352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 0x7fc814139000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009f20 tx=0x7fc80c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45d0c4f6832460f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:09.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 1 -- 172.21.15.153:0/980054352 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c007610 con 0x7fc814136c20 2024-09-15T03:45:09.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: dump: 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:09.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: _finish_auth 0 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 1 -- 172.21.15.153:0/980054352 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c02d050 con 0x7fc814136c20 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 5 monclient: authenticate success, global_id 194237 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 1 -- 172.21.15.153:0/980054352 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c0079a0 con 0x7fc814136c20 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 4 set_mon_vals no callback set 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc81159a640 10 monclient: dump: 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:09.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 10 monclient: shutdown 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 -- 172.21.15.153:0/980054352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 msgr2=0x7fc814139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 --2- 172.21.15.153:0/980054352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 0x7fc814139000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009f20 tx=0x7fc80c009840 comp rx=0 tx=0).stop 2024-09-15T03:45:09.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 -- 172.21.15.153:0/980054352 shutdown_connections 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 --2- 172.21.15.153:0/980054352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc814136c20 0x7fc814139000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 --2- 172.21.15.153:0/980054352 >> v2:172.21.15.181:3300/0 conn(0x7fc8141318f0 0x7fc814131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 -- 172.21.15.153:0/980054352 >> 172.21.15.153:0/980054352 conn(0x7fc814067110 msgr2=0x7fc814067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 -- 172.21.15.153:0/980054352 shutdown_connections 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.908+0000 7fc818f9d640 1 -- 172.21.15.153:0/980054352 wait complete. 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.909+0000 7fc818f9d640 10 monclient: build_initial_monmap 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.909+0000 7fc818f9d640 10 monclient: monmap: 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:09.922 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.909+0000 7fc818f9d640 1 Processor -- start 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.909+0000 7fc818f9d640 1 -- start start 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.909+0000 7fc818f9d640 10 monclient: init 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 0x7fc81411daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc81409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8141318f0 0x7fc814130760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8141318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:09.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc814136c20 0x7fc814099590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc814136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81409dd10 con 0x7fc814136c20 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81409bf10 con 0x7fc8141318f0 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81409b830 con 0x7fc81409ea50 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc818f9d640 10 monclient(hunting): authenticate will time out at 1707.549104s 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc812d9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc814136c20 msgr2=0x7fc814099590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc812d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc814136c20 0x7fc814099590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 0x7fc81411daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:09.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 0x7fc81411daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36144/0 (socket says 172.21.15.153:36144) 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 -- 172.21.15.153:0/1789133045 learned_addr learned my addr 172.21.15.153:0/1789133045 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): get_auth_request con 0x7fc81409ea50 auth_method 0 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc811d9b640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8141318f0 0x7fc814130760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient(hunting): handle_auth_done global_id 194240 payload 824 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 -- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8141318f0 msgr2=0x7fc814130760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8141318f0 0x7fc814130760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 -- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc814136c20 msgr2=0x7fc814099590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc814136c20 0x7fc814099590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 monclient: found mon.noname-c 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc81259c640 1 -- 172.21.15.153:0/1789133045 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c009ae0 con 0x7fc81409ea50 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc811d9b640 10 monclient: get_auth_request con 0x7fc8141318f0 auth_method 0 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.910+0000 7fc811d9b640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8141318f0 0x7fc814130760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.912+0000 7fc81259c640 1 -- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 msgr2=0x7fc81411daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.912+0000 7fc81259c640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 0x7fc81411daa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00d8c0 tx=0x7fc7fc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00de50 con 0x7fc81409ea50 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:09.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: dump: 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:09.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 5 monclient: authenticate success, global_id 194240 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc00bae0 con 0x7fc81409ea50 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 10 monclient: _renew_subs 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81409b070 con 0x7fc81409ea50 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00be80 con 0x7fc81409ea50 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc80a7fc640 10 monclient: dump: 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc810d99640 4 set_mon_vals no callback set 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 10 monclient: _renew_subs 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.913+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc814130ca0 con 0x7fc81409ea50 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc818f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc818f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc818f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81404bf60 con 0x7fc81409ea50 2024-09-15T03:45:09.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc7fc014640 con 0x7fc81409ea50 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc80a7fc640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7f40386f0 0x7fc7f403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc7fc04cc80 con 0x7fc81409ea50 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc811d9b640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7f40386f0 0x7fc7f403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.914+0000 7fc811d9b640 10 monclient: get_auth_request con 0x7fc7f40386f0 auth_method 0 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.917+0000 7fc811d9b640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7f40386f0 0x7fc7f403aba0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc8140a0900 tx=0x7fc804009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.918+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc7fc01e460 con 0x7fc81409ea50 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.918+0000 7fc80a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:09.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:09.918+0000 7fc80a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc818f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc818f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc818f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc81411c9b0 con 0x7fc81409ea50 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc80a7fc640 1 -- 172.21.15.153:0/1789133045 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fc7fc01dbb0 con 0x7fc81409ea50 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc80a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:10.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.130+0000 7fc80a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:10.132 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7f40386f0 msgr2=0x7fc7f403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7f40386f0 0x7fc7f403aba0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc8140a0900 tx=0x7fc804009040 comp rx=0 tx=0).stop 2024-09-15T03:45:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 10 monclient: shutdown 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 msgr2=0x7fc81411daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 0x7fc81411daa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00d8c0 tx=0x7fc7fc008040 comp rx=0 tx=0).stop 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 shutdown_connections 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc7f40386f0 0x7fc7f403aba0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc814136c20 0x7fc814099590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8141318f0 0x7fc814130760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 --2- 172.21.15.153:0/1789133045 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc81409ea50 0x7fc81411daa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 >> 172.21.15.153:0/1789133045 conn(0x7fc814067110 msgr2=0x7fc8140a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 shutdown_connections 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:10.132+0000 7fc818f9d640 1 -- 172.21.15.153:0/1789133045 wait complete. 2024-09-15T03:45:13.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:13.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.248+0000 7fcc3c0e8640 10 monclient: get_monmap_and_config 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.248+0000 7fcc3c0e8640 10 monclient: build_initial_monmap 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.248+0000 7fcc3c0e8640 10 monclient: monmap: 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:13.248818+0000 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:13.248818+0000 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:13.250 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:13.250 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:13.250 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:13.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.248+0000 7fcc3c0e8640 1 Processor -- start 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.248+0000 7fcc3c0e8640 1 -- start start 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.248+0000 7fcc3c0e8640 10 monclient: init 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc340981c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc34131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 0x7fcc3413a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient(hunting): picked mon.noname-b con 0x7fcc34132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3409ca60 con 0x7fcc34131da0 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3409cca0 con 0x7fcc340981c0 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:13.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc34131bd0 con 0x7fcc34132170 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3c0e8640 10 monclient(hunting): authenticate will time out at 1710.888422s 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc39683640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fcc34131da0 legacy=0x7fcc34136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 0x7fcc3413a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 0x7fcc3413a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36158/0 (socket says 172.21.15.153:36158) 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 -- 172.21.15.153:0/3324148575 learned_addr learned my addr 172.21.15.153:0/3324148575 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): get_auth_request con 0x7fcc34132170 auth_method 0 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:13.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient(hunting): handle_auth_done global_id 194243 payload 274 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 -- 172.21.15.153:0/3324148575 >> v1:172.21.15.181:6789/0 conn(0x7fcc340981c0 legacy=0x7fcc34098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 -- 172.21.15.153:0/3324148575 >> v1:172.21.15.153:6789/0 conn(0x7fcc34131da0 legacy=0x7fcc34136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient: _finish_hunting 0 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 monclient: found mon.noname-b 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:13.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.249+0000 7fcc3a685640 1 -- 172.21.15.153:0/3324148575 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3409e010 con 0x7fcc34132170 2024-09-15T03:45:13.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc3a685640 1 -- 172.21.15.153:0/3324148575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 msgr2=0x7fcc3413a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:13.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc3a685640 1 --2- 172.21.15.153:0/3324148575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 0x7fcc3413a2a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcc34066560 tx=0x7fcc240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78bff339a0f4c0df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:13.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 1 -- 172.21.15.153:0/3324148575 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24032440 con 0x7fcc34132170 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: dump: 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:13.259 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: _finish_auth 0 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 1 -- 172.21.15.153:0/3324148575 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc240325a0 con 0x7fcc34132170 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc3c0e8640 5 monclient: authenticate success, global_id 194243 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc3c0e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc3c0e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc3c0e8640 4 set_mon_vals no callback set 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 1 -- 172.21.15.153:0/3324148575 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24032940 con 0x7fcc34132170 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:13.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.252+0000 7fcc38e82640 10 monclient: dump: 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 10 monclient: shutdown 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/3324148575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 msgr2=0x7fcc3413a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/3324148575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 0x7fcc3413a2a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcc34066560 tx=0x7fcc240091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/3324148575 shutdown_connections 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/3324148575 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34132170 0x7fcc3413a2a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/3324148575 >> 172.21.15.153:0/3324148575 conn(0x7fcc340a1e30 msgr2=0x7fcc340a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/3324148575 shutdown_connections 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/3324148575 wait complete. 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 10 monclient: build_initial_monmap 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.253+0000 7fcc3c0e8640 10 monclient: monmap: 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:13.263 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 Processor -- start 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 -- start start 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient: init 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc340981c0 0x7fcc34137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): picked mon.noname-b con 0x7fcc340981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 0x7fcc341377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): picked mon.noname-c con 0x7fcc34131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc34132170 0x7fcc34137cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:13.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): picked mon.noname-a con 0x7fcc34132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc34119430 con 0x7fcc34132170 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc34119a80 con 0x7fcc340981c0 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3411a0f0 con 0x7fcc34131da0 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3c0e8640 10 monclient(hunting): authenticate will time out at 1710.893772s 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3a685640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc34132170 msgr2=0x7fcc34137cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc3a685640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc34132170 0x7fcc34137cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:13.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 0x7fcc341377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 0x7fcc341377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36166/0 (socket says 172.21.15.153:36166) 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 1 -- 172.21.15.153:0/1129200451 learned_addr learned my addr 172.21.15.153:0/1129200451 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): get_auth_request con 0x7fcc34131da0 auth_method 0 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.254+0000 7fcc39683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 10 monclient(hunting): handle_auth_done global_id 194246 payload 824 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39e84640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc340981c0 0x7fcc34137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 1 -- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc340981c0 msgr2=0x7fcc34137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:13.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc340981c0 0x7fcc34137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 1 -- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc34132170 msgr2=0x7fcc34137cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc34132170 0x7fcc34137cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 10 monclient: _finish_hunting 0 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 1 monclient: found mon.noname-c 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.255+0000 7fcc39683640 1 -- 172.21.15.153:0/1129200451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24009450 con 0x7fcc34131da0 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc39683640 1 -- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 msgr2=0x7fcc341377b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc39683640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 0x7fcc341377b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800bbb0 tx=0x7fcc280097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800cc80 con 0x7fcc34131da0 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:13.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: dump: 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: _finish_auth 0 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 5 monclient: authenticate success, global_id 194246 2024-09-15T03:45:13.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800cde0 con 0x7fcc34131da0 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 10 monclient: _renew_subs 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc34098cb0 con 0x7fcc34131da0 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28016430 con 0x7fcc34131da0 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc31ffb640 10 monclient: dump: 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:13.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 10 monclient: _renew_subs 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc341309a0 con 0x7fcc34131da0 2024-09-15T03:45:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.257+0000 7fcc33fff640 4 set_mon_vals no callback set 2024-09-15T03:45:13.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcc28012020 con 0x7fcc34131da0 2024-09-15T03:45:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc3c0e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc3c0e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc3c0e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc31ffb640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc20038720 0x7fcc2003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3404bf60 con 0x7fcc34131da0 2024-09-15T03:45:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.258+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcc2804c890 con 0x7fcc34131da0 2024-09-15T03:45:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.259+0000 7fcc39e84640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc20038720 0x7fcc2003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.259+0000 7fcc39e84640 10 monclient: get_auth_request con 0x7fcc20038720 auth_method 0 2024-09-15T03:45:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.261+0000 7fcc39e84640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc20038720 0x7fcc2003abd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c002750 tx=0x7fcc2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.263+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcc28006150 con 0x7fcc34131da0 2024-09-15T03:45:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.263+0000 7fcc31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:13.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.263+0000 7fcc31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc3c0e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc3c0e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc3c0e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcc34098ee0 con 0x7fcc34131da0 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc31ffb640 1 -- 172.21.15.153:0/1129200451 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fcc28014340 con 0x7fcc34131da0 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:13.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.479+0000 7fcc31ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:13.481 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:13.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc20038720 msgr2=0x7fcc2003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc20038720 0x7fcc2003abd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c002750 tx=0x7fcc2c009040 comp rx=0 tx=0).stop 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 10 monclient: shutdown 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 msgr2=0x7fcc341377b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 0x7fcc341377b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800bbb0 tx=0x7fcc280097f0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 shutdown_connections 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc20038720 0x7fcc2003abd0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc34132170 0x7fcc34137cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc34131da0 0x7fcc341377b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 --2- 172.21.15.153:0/1129200451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc340981c0 0x7fcc34137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 >> 172.21.15.153:0/1129200451 conn(0x7fcc340a1e30 msgr2=0x7fcc3413a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 shutdown_connections 2024-09-15T03:45:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:13.481+0000 7fcc3c0e8640 1 -- 172.21.15.153:0/1129200451 wait complete. 2024-09-15T03:45:16.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.598+0000 7fa9c255d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.598+0000 7fa9c255d640 10 monclient: build_initial_monmap 2024-09-15T03:45:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.598+0000 7fa9c255d640 10 monclient: monmap: 2024-09-15T03:45:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:16.599 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:16.599252+0000 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:16.599252+0000 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:16.600 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 1 Processor -- start 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 1 -- start start 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient: init 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 0x7fa9bc098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9bc0981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9bc131da0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:16.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9bc132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc098ad0 con 0x7fa9bc132170 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc067f30 con 0x7fa9bc131da0 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9bc131bd0 con 0x7fa9bc0981c0 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c255d640 10 monclient(hunting): authenticate will time out at 1714.238933s 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9c0afa640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa9bc132170 legacy=0x7fa9bc067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9bbfff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 msgr2=0x7fa9bc098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.599+0000 7fa9bbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 0x7fa9bc098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:16.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.600+0000 7fa9bb7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa9bc131da0 0x7fa9bc067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:56160/0 (socket says 172.21.15.153:56160) 2024-09-15T03:45:16.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.600+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 learned_addr learned my addr 172.21.15.153:0/1533139471 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:16.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.602+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> v1:172.21.15.181:6789/0 conn(0x7fa9bc131da0 legacy=0x7fa9bc067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.800+0000 7fa9c0afa640 1 -- 172.21.15.153:0/1533139471 >> v1:172.21.15.153:6789/0 conn(0x7fa9bc132170 legacy=0x7fa9bc067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.800+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 msgr2=0x7fa9bc098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:16.800+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 0x7fa9bc098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:17.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:17.201+0000 7fa9c0afa640 1 -- 172.21.15.153:0/1533139471 >> v1:172.21.15.153:6789/0 conn(0x7fa9bc132170 legacy=0x7fa9bc067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:17.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:17.201+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 msgr2=0x7fa9bc098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:17.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:17.201+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 0x7fa9bc098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:18.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:18.002+0000 7fa9c0afa640 1 -- 172.21.15.153:0/1533139471 >> v1:172.21.15.153:6789/0 conn(0x7fa9bc132170 legacy=0x7fa9bc067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:18.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:18.002+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 msgr2=0x7fa9bc098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:18.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:18.002+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 0x7fa9bc098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): tick 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 monclient(hunting): continuing hunt 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 msgr2=0x7fa9bc098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.153:3300/0 conn(0x7fa9bc0981c0 0x7fa9bc098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 -- 172.21.15.153:0/1533139471 >> v1:172.21.15.181:6789/0 conn(0x7fa9bc131da0 legacy=0x7fa9bc067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 -- 172.21.15.153:0/1533139471 >> v1:172.21.15.153:6789/0 conn(0x7fa9bc132170 legacy=0x7fa9bc067820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient: _add_conns ranks=[4,2] 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fa9a8003700 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fa9a8006200 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 -- 172.21.15.153:0/1533139471 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9bc131bd0 con 0x7fa9a8003700 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 1 -- 172.21.15.153:0/1533139471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8008de0 con 0x7fa9a8006200 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9b9ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request con 0x7fa9a8006200 auth_method 0 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:19.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request con 0x7fa9a8003700 auth_method 0 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.600+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:19.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:19.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request con 0x7fa9a8006200 auth_method 0 2024-09-15T03:45:19.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request con 0x7fa9a8003700 auth_method 0 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:19.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:19.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:19.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:19.801+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:20.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request con 0x7fa9a8006200 auth_method 0 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request con 0x7fa9a8003700 auth_method 0 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:20.202+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request con 0x7fa9a8006200 auth_method 0 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request con 0x7fa9a8003700 auth_method 0 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_done global_id 194249 payload 274 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 msgr2=0x7fa9a8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.002+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> v2:172.21.15.181:3300/0 conn(0x7fa9a8003700 0x7fa9a8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.003+0000 7fa9bb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.003+0000 7fa9bb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.003+0000 7fa9bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.003+0000 7fa9bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.003+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc09ca60 con 0x7fa9a8006200 2024-09-15T03:45:21.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.005+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.005+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0007c10 tx=0x7fa9b0009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2717eadc8a4be97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 1 -- 172.21.15.153:0/1533139471 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b00313f0 con 0x7fa9a8006200 2024-09-15T03:45:21.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:21.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: dump: 2024-09-15T03:45:21.009 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.009 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 1 -- 172.21.15.153:0/1533139471 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b0031550 con 0x7fa9a8006200 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 1 -- 172.21.15.153:0/1533139471 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b00318f0 con 0x7fa9a8006200 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9c255d640 5 monclient: authenticate success, global_id 194249 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9c255d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9c255d640 4 set_mon_vals no callback set 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.006+0000 7fa9baffd640 10 monclient: dump: 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 10 monclient: shutdown 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 1 -- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 msgr2=0x7fa9a80086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 1 --2- 172.21.15.153:0/1533139471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9a8006200 0x7fa9a80086b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0007c10 tx=0x7fa9b0009040 comp rx=0 tx=0).stop 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c0afa640 1 -- 172.21.15.153:0/1533139471 reap_dead start 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 1 -- 172.21.15.153:0/1533139471 shutdown_connections 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 1 -- 172.21.15.153:0/1533139471 >> 172.21.15.153:0/1533139471 conn(0x7fa9bc0a1e30 msgr2=0x7fa9bc0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 1 -- 172.21.15.153:0/1533139471 shutdown_connections 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 1 -- 172.21.15.153:0/1533139471 wait complete. 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 10 monclient: build_initial_monmap 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.007+0000 7fa9c255d640 10 monclient: monmap: 2024-09-15T03:45:21.013 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:21.014 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.008+0000 7fa9c255d640 1 Processor -- start 2024-09-15T03:45:21.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.008+0000 7fa9c255d640 1 -- start start 2024-09-15T03:45:21.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.008+0000 7fa9c255d640 10 monclient: init 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9bc11ded0 0x7fa9bc11e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9bc11ded0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 0x7fa9bc11ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9bc11e7e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:21.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9bc145730 0x7fa9bc147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9bc145730 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc1308e0 con 0x7fa9bc145730 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc11f0f0 con 0x7fa9bc11ded0 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc098cb0 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c255d640 10 monclient(hunting): authenticate will time out at 1718.649143s 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c0afa640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9bc145730 msgr2=0x7fa9bc147b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9c0afa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9bc145730 0x7fa9bc147b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 0x7fa9bc11ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9bc11ded0 0x7fa9bc11e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 0x7fa9bc11ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50918/0 (socket says 172.21.15.153:50918) 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9bc11ded0 0x7fa9bc11e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:52160/0 (socket says 172.21.15.153:52160) 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/3480637138 learned_addr learned my addr 172.21.15.153:0/3480637138 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request con 0x7fa9bc11e7e0 auth_method 0 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request con 0x7fa9bc11ded0 auth_method 0 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient(hunting): handle_auth_done global_id 194252 payload 824 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9bc11ded0 msgr2=0x7fa9bc11e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9bc11ded0 0x7fa9bc11e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9bc145730 msgr2=0x7fa9bc147b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9bc145730 0x7fa9bc147b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 monclient: found mon.noname-c 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.009+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/3480637138 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b000f040 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9bb7fe640 1 -- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 msgr2=0x7fa9bc11ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9bb7fe640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 0x7fa9bc11ebb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9b002aeb0 tx=0x7fa9b0039000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b00313f0 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: dump: 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b00316c0 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0031c60 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9a3fff640 10 monclient: dump: 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.029 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 5 monclient: authenticate success, global_id 194252 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 10 monclient: _renew_subs 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9bc121590 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9ba7fc640 4 set_mon_vals no callback set 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 10 monclient: _renew_subs 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.012+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9bc11f7f0 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa9b0043070 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9c255d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9c255d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9c255d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9bc121720 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9a3fff640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa9a8039550 0x7fa9a803ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.013+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa9b006d110 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.014+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa9a8039550 0x7fa9a803ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.014+0000 7fa9bbfff640 10 monclient: get_auth_request con 0x7fa9a8039550 auth_method 0 2024-09-15T03:45:21.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.016+0000 7fa9bbfff640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa9a8039550 0x7fa9a803ba00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac000ea0 tx=0x7fa9ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.018+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa9b0043d50 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.018+0000 7fa9a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.018+0000 7fa9a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9c255d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9c255d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9c255d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa9bc04bf60 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9a3fff640 1 -- 172.21.15.153:0/3480637138 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa9b0035020 con 0x7fa9bc11e7e0 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:21.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.237+0000 7fa9a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:21.240 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":170,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa9a8039550 msgr2=0x7fa9a803ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa9a8039550 0x7fa9a803ba00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac000ea0 tx=0x7fa9ac008040 comp rx=0 tx=0).stop 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 10 monclient: shutdown 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 msgr2=0x7fa9bc11ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 0x7fa9bc11ebb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9b002aeb0 tx=0x7fa9b0039000 comp rx=0 tx=0).stop 2024-09-15T03:45:21.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 shutdown_connections 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa9a8039550 0x7fa9a803ba00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9bc145730 0x7fa9bc147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9bc11e7e0 0x7fa9bc11ebb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 --2- 172.21.15.153:0/3480637138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9bc11ded0 0x7fa9bc11e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 >> 172.21.15.153:0/3480637138 conn(0x7fa9bc0a1e30 msgr2=0x7fa9bc096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 shutdown_connections 2024-09-15T03:45:21.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.239+0000 7fa9c255d640 1 -- 172.21.15.153:0/3480637138 wait complete. 2024-09-15T03:45:21.251 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:45:21.252 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:45:21.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:45:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.345+0000 7f90c911d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.345+0000 7f90c911d640 10 monclient: build_initial_monmap 2024-09-15T03:45:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.345+0000 7f90c911d640 10 monclient: monmap: 2024-09-15T03:45:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:21.347206+0000 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:21.347206+0000 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 1 Processor -- start 2024-09-15T03:45:21.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 1 -- start start 2024-09-15T03:45:21.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient: init 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient(hunting): picked mon.noname-c con 0x7f90c4096c60 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90c4097030 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90c4133b40 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c40696a0 con 0x7f90c4133b40 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c409d2e0 con 0x7f90c4097030 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c409ac60 con 0x7f90c4096c60 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c911d640 10 monclient(hunting): authenticate will time out at 1718.986550s 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c359e640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f90c4133b40 legacy=0x7f90c4135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:52172/0 (socket says 172.21.15.153:52172) 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 learned_addr learned my addr 172.21.15.153:0/910239465 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request con 0x7f90c4096c60 auth_method 0 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c259c640 1 --1- 172.21.15.153:0/910239465 >> v1:172.21.15.181:6789/0 conn(0x7f90c4097030 0x7f90c4097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:56174/0 (socket says 172.21.15.153:56174) 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.346+0000 7f90c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_done global_id 204148 payload 274 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 >> v1:172.21.15.181:6789/0 conn(0x7f90c4097030 legacy=0x7f90c4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 >> v1:172.21.15.153:6789/0 conn(0x7f90c4133b40 legacy=0x7f90c4135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 1 monclient: found mon.noname-c 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.347+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c409d6b0 con 0x7f90c4096c60 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.350+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 msgr2=0x7f90c409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.350+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 msgr2=0x7f90c409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.350+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f90b400bbc0 tx=0x7f90b400b460 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:21.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.350+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient: get_auth_request con 0x7f90c4096c60 auth_method 0 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 0 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient: ms_handle_reset current mon v2:172.21.15.181:3300/0 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 1 -- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 msgr2=0x7f90c409a6c0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.181:3300/0 conn(0x7f90c4096c60 0x7f90c409a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient: _add_conns ranks=[2,3] 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 1 --2- 172.21.15.153:0/910239465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90b00034c0 0x7f90b0005900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient(hunting): picked mon.noname-b con 0x7f90b00034c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.153:3300/0 conn(0x7f90b0006220 0x7f90b00086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient(hunting): picked mon.noname-a con 0x7f90b0006220 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 1 -- 172.21.15.153:0/910239465 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b0008fa0 con 0x7f90b0006220 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 1 -- 172.21.15.153:0/910239465 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0009110 con 0x7f90b00034c0 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c1d9b640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.181:3300/0 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c259c640 1 -- 172.21.15.153:0/910239465 >> v2:172.21.15.153:3300/0 conn(0x7f90b0006220 msgr2=0x7f90b00086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c259c640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.153:3300/0 conn(0x7f90b0006220 0x7f90b00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90b00034c0 0x7f90b0005900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request con 0x7f90b00034c0 auth_method 0 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_done global_id 204148 payload 293 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 >> v2:172.21.15.153:3300/0 conn(0x7f90b0006220 msgr2=0x7f90b00086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> v2:172.21.15.153:3300/0 conn(0x7f90b0006220 0x7f90b00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 monclient: found mon.noname-b 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.551+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0009390 con 0x7f90b00034c0 2024-09-15T03:45:21.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c2d9d640 1 -- 172.21.15.153:0/910239465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90b00034c0 msgr2=0x7f90b0005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/910239465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90b00034c0 0x7f90b0005900 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90c4068d80 tx=0x7f90c409d840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffd73db018ccff16 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 1 -- 172.21.15.153:0/910239465 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4032760 con 0x7f90b00034c0 2024-09-15T03:45:21.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: dump: 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.564 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 5 monclient: authenticate success, global_id 204148 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 1 -- 172.21.15.153:0/910239465 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b403a040 con 0x7f90b00034c0 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 4 set_mon_vals no callback set 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 1 -- 172.21.15.153:0/910239465 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4034450 con 0x7f90b00034c0 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c1d9b640 10 monclient: dump: 2024-09-15T03:45:21.565 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 10 monclient: shutdown 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 1 -- 172.21.15.153:0/910239465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90b00034c0 msgr2=0x7f90b0005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 1 --2- 172.21.15.153:0/910239465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90b00034c0 0x7f90b0005900 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90c4068d80 tx=0x7f90c409d840 comp rx=0 tx=0).stop 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c359e640 1 -- 172.21.15.153:0/910239465 reap_dead start 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 1 -- 172.21.15.153:0/910239465 shutdown_connections 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 1 -- 172.21.15.153:0/910239465 >> 172.21.15.153:0/910239465 conn(0x7f90c40672b0 msgr2=0x7f90c40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 1 -- 172.21.15.153:0/910239465 shutdown_connections 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.554+0000 7f90c911d640 1 -- 172.21.15.153:0/910239465 wait complete. 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.555+0000 7f90c911d640 10 monclient: build_initial_monmap 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.555+0000 7f90c911d640 10 monclient: monmap: 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:21.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.555+0000 7f90c911d640 1 Processor -- start 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.555+0000 7f90c911d640 1 -- start start 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.555+0000 7f90c911d640 10 monclient: init 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b00034c0 0x7f90c41413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): picked mon.noname-a con 0x7f90b00034c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:21.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 0x7f90c412eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): picked mon.noname-c con 0x7f90c4141910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90c412f090 0x7f90c411a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): picked mon.noname-b con 0x7f90c412f090 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c412f4a0 con 0x7f90b00034c0 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c409add0 con 0x7f90c412f090 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c411e000 con 0x7f90c4141910 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c911d640 10 monclient(hunting): authenticate will time out at 1719.196431s 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c2d9d640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b00034c0 msgr2=0x7f90c41413d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b00034c0 0x7f90c41413d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 0x7f90c412eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 0x7f90c412eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50940/0 (socket says 172.21.15.153:50940) 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 -- 172.21.15.153:0/913779570 learned_addr learned my addr 172.21.15.153:0/913779570 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): get_auth_request con 0x7f90c4141910 auth_method 0 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c359e640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90c412f090 0x7f90c411a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c359e640 10 monclient(hunting): get_auth_request con 0x7f90c412f090 auth_method 0 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient(hunting): handle_auth_done global_id 194255 payload 824 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90c412f090 msgr2=0x7f90c411a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90c412f090 0x7f90c411a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b00034c0 msgr2=0x7f90c41413d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b00034c0 0x7f90c41413d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 monclient: found mon.noname-c 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.556+0000 7f90c259c640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c41336c0 con 0x7f90c4141910 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c259c640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 msgr2=0x7f90c412eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c259c640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 0x7f90c412eb50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c640 tx=0x7f90ac007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac007510 con 0x7f90c4141910 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: dump: 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 5 monclient: authenticate success, global_id 194255 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac0057d0 con 0x7f90c4141910 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac007690 con 0x7f90c4141910 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 10 monclient: _renew_subs 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c4142230 con 0x7f90c4141910 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c159a640 4 set_mon_vals no callback set 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90baffd640 10 monclient: dump: 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 10 monclient: _renew_subs 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.559+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c411cd30 con 0x7f90c4141910 2024-09-15T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c911d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c911d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 0x7f90c4029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c411de30 con 0x7f90c40298c0 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f90ac01b400 con 0x7f90c4141910 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 0x7f90c4029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient: get_auth_request con 0x7f90c40298c0 auth_method 0 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90baffd640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90a40386f0 0x7f90a403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_done global_id 194258 payload 824 2024-09-15T03:45:21.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c2d9d640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90c404bf60 con 0x7f90c40298c0 2024-09-15T03:45:21.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f90ac04c1b0 con 0x7f90c4141910 2024-09-15T03:45:21.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c359e640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90a40386f0 0x7f90a403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.560+0000 7f90c359e640 10 monclient: get_auth_request con 0x7f90a40386f0 auth_method 0 2024-09-15T03:45:21.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.562+0000 7f90c2d9d640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 msgr2=0x7f90c4029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.562+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 0x7f90c4029c90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90b00092a0 tx=0x7f90b403b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.562+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4049460 con 0x7f90c40298c0 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.562+0000 7f90baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.562+0000 7f90baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.562+0000 7f90baffd640 10 monclient: dump: 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f90b4047450 con 0x7f90c40298c0 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 msgr2=0x7f90c4029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 0x7f90c4029c90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90b00092a0 tx=0x7f90b403b000 comp rx=0 tx=0).stop 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 msgr2=0x7f90c4029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90baffd640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c40298c0 0x7f90c4029c90 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90b00092a0 tx=0x7f90b403b000 comp rx=0 tx=0).stop 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.563+0000 7f90c359e640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90a40386f0 0x7f90a403aba0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f90c4098740 tx=0x7f90bc0094c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.568+0000 7f90c911d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.568+0000 7f90c911d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 0x7f90c402a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.568+0000 7f90c911d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.568+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c4065f20 con 0x7f90c402a1d0 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 0x7f90c402a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient: get_auth_request con 0x7f90c402a1d0 auth_method 0 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 10 monclient(hunting): handle_auth_done global_id 194261 payload 824 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.569+0000 7f90c2d9d640 1 -- 172.21.15.153:0/913779570 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f90c40983b0 con 0x7f90c402a1d0 2024-09-15T03:45:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90c2d9d640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 msgr2=0x7f90c402a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90c2d9d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 0x7f90c402a5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f90b400bb10 tx=0x7f90b4008050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4055410 con 0x7f90c402a1d0 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90baffd640 10 monclient: dump: 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.582 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.571+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f90b403e3f0 con 0x7f90c402a1d0 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90baffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 msgr2=0x7f90c402a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90baffd640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 0x7f90c402a5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f90b400bb10 tx=0x7f90b4008050 comp rx=0 tx=0).stop 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90baffd640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 msgr2=0x7f90c402a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90baffd640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c402a1d0 0x7f90c402a5d0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f90b400bb10 tx=0x7f90b4008050 comp rx=0 tx=0).stop 2024-09-15T03:45:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 170, 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:21.584 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:21.585 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:21.586 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:21.587 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.588 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:21.589 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:21.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:21.591 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:21.593 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.594 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:21.595 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90a40386f0 msgr2=0x7f90a403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90a40386f0 0x7f90a403aba0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f90c4098740 tx=0x7f90bc0094c0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 10 monclient: shutdown 2024-09-15T03:45:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:21.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 msgr2=0x7f90c412eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 --2- 172.21.15.153:0/913779570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90c4141910 0x7f90c412eb50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c640 tx=0x7f90ac007ed0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c359e640 1 -- 172.21.15.153:0/913779570 reap_dead start 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c359e640 1 -- 172.21.15.153:0/913779570 reap_dead start 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 shutdown_connections 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 >> 172.21.15.153:0/913779570 conn(0x7f90c40672b0 msgr2=0x7f90c409f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 shutdown_connections 2024-09-15T03:45:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.572+0000 7f90c911d640 1 -- 172.21.15.153:0/913779570 wait complete. 2024-09-15T03:45:21.598 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:45:21.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:45:21.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.691+0000 7fa82179d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.691+0000 7fa82179d640 10 monclient: build_initial_monmap 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.691+0000 7fa82179d640 10 monclient: monmap: 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:21.693221+0000 2024-09-15T03:45:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:21.693221+0000 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:21.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 1 Processor -- start 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 1 -- start start 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient: init 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa81c131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa81c09e910 0x7fa81c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient(hunting): picked mon.noname-a con 0x7fa81c09e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa81c09f250 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c09faf0 con 0x7fa81c09f250 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c09fd30 con 0x7fa81c131ab0 2024-09-15T03:45:21.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c132590 con 0x7fa81c09e910 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa82179d640 10 monclient(hunting): authenticate will time out at 1719.332605s 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa81b7fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fa81c09f250 legacy=0x7fa81c138bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa81a7fc640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fa81c09e910 msgr2=0x7fa81c09ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa81a7fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa81c09e910 0x7fa81c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa81affd640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fa81c131ab0 0x7fa81c131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:56186/0 (socket says 172.21.15.153:56186) 2024-09-15T03:45:21.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.692+0000 7fa81affd640 1 -- 172.21.15.153:0/449906063 learned_addr learned my addr 172.21.15.153:0/449906063 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 1 -- 172.21.15.153:0/449906063 >> v1:172.21.15.181:6789/0 conn(0x7fa81c131ab0 legacy=0x7fa81c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073928424 0 0) 0x7fa81c09fd30 con 0x7fa81c131ab0 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): my global_id is 204154 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8040034e0 con 0x7fa81c131ab0 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671594979 0 0) 0x7fa8040034e0 con 0x7fa81c131ab0 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 >> v2:172.21.15.153:3300/0 conn(0x7fa81c09e910 msgr2=0x7fa81c09ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 --2- 172.21.15.153:0/449906063 >> v2:172.21.15.153:3300/0 conn(0x7fa81c09e910 0x7fa81c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 >> v1:172.21.15.153:6789/0 conn(0x7fa81c09f250 legacy=0x7fa81c138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0a1110 con 0x7fa81c131ab0 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fa8140017e0 con 0x7fa81c131ab0 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:45:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: dump: 2024-09-15T03:45:21.704 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.704 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.704 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 >> v1:172.21.15.181:6789/0 conn(0x7fa81c131ab0 legacy=0x7fa81c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8040039d0 0x7fa804003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): picked mon.b con 0x7fa8040039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 0x7fa804004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): picked mon.c con 0x7fa804004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa804005330 0x7fa80400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): picked mon.a con 0x7fa804005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c132590 con 0x7fa804005330 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80400c290 con 0x7fa8040039d0 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80400c450 con 0x7fa804004780 2024-09-15T03:45:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa819ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81b7fe640 1 -- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa804005330 msgr2=0x7fa80400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81b7fe640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa804005330 0x7fa80400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 0x7fa804004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8040039d0 0x7fa804003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): get_auth_request con 0x7fa804004780 auth_method 0 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): get_auth_request con 0x7fa8040039d0 auth_method 0 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.695+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_done global_id 204154 payload 293 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 1 -- 172.21.15.153:0/449906063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8040039d0 msgr2=0x7fa804003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8040039d0 0x7fa804003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 1 -- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa804005330 msgr2=0x7fa80400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:21.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa804005330 0x7fa80400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 1 monclient: found mon.c 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81b7fe640 1 -- 172.21.15.153:0/449906063 reap_dead start 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 1 -- 172.21.15.153:0/449906063 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8040034e0 con 0x7fa804004780 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa81a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa82179d640 5 monclient: authenticate success, global_id 204154 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.696+0000 7fa82179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.698+0000 7fa81a7fc640 1 -- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 msgr2=0x7fa804004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.698+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 0x7fa804004bf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa810009f90 tx=0x7fa8100095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=986592d0078196e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8100050d0 con 0x7fa804004780 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: dump: 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.710 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81002d070 con 0x7fa804004780 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 1 -- 172.21.15.153:0/449906063 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8100054c0 con 0x7fa804004780 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 4 set_mon_vals no callback set 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa819ffb640 10 monclient: dump: 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.711 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 10 monclient: shutdown 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 -- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 msgr2=0x7fa804004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 0x7fa804004bf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa810009f90 tx=0x7fa8100095d0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 -- 172.21.15.153:0/449906063 shutdown_connections 2024-09-15T03:45:21.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 --2- 172.21.15.153:0/449906063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa804004780 0x7fa804004bf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 -- 172.21.15.153:0/449906063 >> 172.21.15.153:0/449906063 conn(0x7fa81c067d30 msgr2=0x7fa81c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 -- 172.21.15.153:0/449906063 shutdown_connections 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.699+0000 7fa82179d640 1 -- 172.21.15.153:0/449906063 wait complete. 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.700+0000 7fa82179d640 10 monclient: build_initial_monmap 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.700+0000 7fa82179d640 10 monclient: monmap: 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:21.713 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:21.714 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:21.714 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.700+0000 7fa82179d640 1 Processor -- start 2024-09-15T03:45:21.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.700+0000 7fa82179d640 1 -- start start 2024-09-15T03:45:21.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.700+0000 7fa82179d640 10 monclient: init 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa804004780 0x7fa81c140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): picked mon.noname-b con 0x7fa804004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 0x7fa81c1300b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): picked mon.noname-c con 0x7fa81c12fc60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa81c099b00 0x7fa81c099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): picked mon.noname-a con 0x7fa81c099b00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c11b7d0 con 0x7fa81c099b00 2024-09-15T03:45:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c09bde0 con 0x7fa804004780 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c130ac0 con 0x7fa81c12fc60 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa82179d640 10 monclient(hunting): authenticate will time out at 1719.341246s 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81b7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa81c099b00 msgr2=0x7fa81c099fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa81c099b00 0x7fa81c099fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 0x7fa81c1300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 0x7fa81c1300b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50966/0 (socket says 172.21.15.153:50966) 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 -- 172.21.15.153:0/4059100616 learned_addr learned my addr 172.21.15.153:0/4059100616 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): get_auth_request con 0x7fa81c12fc60 auth_method 0 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa804004780 0x7fa81c140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81affd640 10 monclient(hunting): get_auth_request con 0x7fa804004780 auth_method 0 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient(hunting): handle_auth_done global_id 194264 payload 824 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa804004780 msgr2=0x7fa81c140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa804004780 0x7fa81c140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa81c099b00 msgr2=0x7fa81c099fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:21.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa81c099b00 0x7fa81c099fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 monclient: found mon.noname-c 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81a7fc640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c11be20 con 0x7fa81c12fc60 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.701+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa804004780 0x7fa81c140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa81a7fc640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 msgr2=0x7fa81c1300b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa81a7fc640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 0x7fa81c1300b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa8100099f0 tx=0x7fa810035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810032640 con 0x7fa81c12fc60 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 10 monclient: dump: 2024-09-15T03:45:21.724 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa82179d640 5 monclient: authenticate success, global_id 194264 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8100327c0 con 0x7fa81c12fc60 2024-09-15T03:45:21.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa82179d640 10 monclient: _renew_subs 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810032b80 con 0x7fa81c12fc60 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa82179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c11cd80 con 0x7fa81c12fc60 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa802ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa802ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa802ffd640 10 monclient: dump: 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.703+0000 7fa8197fa640 4 set_mon_vals no callback set 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 10 monclient: _renew_subs 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c141be0 con 0x7fa81c12fc60 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa810039070 con 0x7fa81c12fc60 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 0x7fa81c09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c063740 con 0x7fa81c09a4e0 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.704+0000 7fa802ffd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8080386f0 0x7fa80803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa81006da00 con 0x7fa81c12fc60 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 0x7fa81c09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient: get_auth_request con 0x7fa81c09a4e0 auth_method 0 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81b7fe640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8080386f0 0x7fa80803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81b7fe640 10 monclient: get_auth_request con 0x7fa8080386f0 auth_method 0 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 10 monclient(hunting): handle_auth_done global_id 204160 payload 824 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.705+0000 7fa81affd640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa81c04bf60 con 0x7fa81c09a4e0 2024-09-15T03:45:21.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa81affd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 msgr2=0x7fa81c09cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 0x7fa81c09cfd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa814001490 tx=0x7fa814009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa81b7fe640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8080386f0 0x7fa80803aba0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa81c09f620 tx=0x7fa80c006c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814011d30 con 0x7fa81c09a4e0 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa802ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa802ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.707+0000 7fa802ffd640 10 monclient: dump: 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa814011e90 con 0x7fa81c09a4e0 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 msgr2=0x7fa81c09cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 0x7fa81c09cfd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa814001490 tx=0x7fa814009040 comp rx=0 tx=0).stop 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 msgr2=0x7fa81c09cfd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.708+0000 7fa802ffd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09a4e0 0x7fa81c09cfd0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa814001490 tx=0x7fa814009040 comp rx=0 tx=0).stop 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.713+0000 7fa82179d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.713+0000 7fa82179d640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:21.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa82179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c11cfc0 con 0x7fa81c09d510 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 10 monclient: get_auth_request con 0x7fa81c09d510 auth_method 0 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 msgr2=0x7fa81c09d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 msgr2=0x7fa81c09d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:21.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.714+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient: get_auth_request con 0x7fa81c09d510 auth_method 0 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:21.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:21.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 10 monclient(hunting): handle_auth_done global_id 204166 payload 824 2024-09-15T03:45:21.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.915+0000 7fa81affd640 1 -- 172.21.15.153:0/4059100616 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa81c09a8b0 con 0x7fa81c09d510 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa81affd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 msgr2=0x7fa81c09d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa81affd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa814001490 tx=0x7fa814015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81401e8f0 con 0x7fa81c09d510 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 10 monclient: dump: 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:21.920 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7fa81401ea50 con 0x7fa81c09d510 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 msgr2=0x7fa81c09d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa814001490 tx=0x7fa814015000 comp rx=0 tx=0).stop 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 msgr2=0x7fa81c09d910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:21.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa802ffd640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c09d510 0x7fa81c09d910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 170, 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:21.922 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:21.923 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.924 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:21.925 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:21.926 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:21.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:21.928 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:21.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:21.931 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:21.932 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:21.933 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:21.934 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:21.935 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8080386f0 msgr2=0x7fa80803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa8080386f0 0x7fa80803aba0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa81c09f620 tx=0x7fa80c006c10 comp rx=0 tx=0).stop 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 10 monclient: shutdown 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 msgr2=0x7fa81c1300b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 --2- 172.21.15.153:0/4059100616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa81c12fc60 0x7fa81c1300b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa8100099f0 tx=0x7fa810035000 comp rx=0 tx=0).stop 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa81b7fe640 1 -- 172.21.15.153:0/4059100616 reap_dead start 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa81b7fe640 1 -- 172.21.15.153:0/4059100616 reap_dead start 2024-09-15T03:45:21.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 shutdown_connections 2024-09-15T03:45:21.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 >> 172.21.15.153:0/4059100616 conn(0x7fa81c067d30 msgr2=0x7fa81c0a2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:21.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 shutdown_connections 2024-09-15T03:45:21.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:21.918+0000 7fa82179d640 1 -- 172.21.15.153:0/4059100616 wait complete. 2024-09-15T03:45:21.937 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:45:21.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:45:41.938 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:45:41.938 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:45:41.938 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:45:41.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:45:41.942 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:45:41.942 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:45:41.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.046+0000 7fa0ed11d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.046+0000 7fa0ed11d640 10 monclient: build_initial_monmap 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.046+0000 7fa0ed11d640 10 monclient: monmap: 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:42.046941+0000 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:42.046941+0000 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:42.047 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:42.048 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:42.048 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:42.048 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:42.048 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:42.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.046+0000 7fa0ed11d640 1 Processor -- start 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.046+0000 7fa0ed11d640 1 -- start start 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.046+0000 7fa0ed11d640 10 monclient: init 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0e8131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 0x7fa0e809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0e809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0e809f220 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e809faf0 con 0x7fa0e8131ab0 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:42.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e8132590 con 0x7fa0e809f220 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e809fd30 con 0x7fa0e809e910 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0ed11d640 10 monclient(hunting): authenticate will time out at 1739.686333s 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 0x7fa0e809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39562/0 (socket says 172.21.15.153:39562) 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 0x7fa0e809ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34860/0 (socket says 172.21.15.153:34860) 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 -- 172.21.15.153:0/4230965429 learned_addr learned my addr 172.21.15.153:0/4230965429 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 10 monclient(hunting): get_auth_request con 0x7fa0e809f220 auth_method 0 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): get_auth_request con 0x7fa0e809e910 auth_method 0 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 -- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 msgr2=0x7fa0e8138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 -- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 msgr2=0x7fa0e8138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 --2- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e759e640 1 --2- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:42.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient(hunting): handle_auth_done global_id 194267 payload 274 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 -- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 msgr2=0x7fa0e8138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 --2- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 -- 172.21.15.153:0/4230965429 >> v1:172.21.15.181:6789/0 conn(0x7fa0e8131ab0 legacy=0x7fa0e8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 monclient: found mon.noname-b 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:42.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.047+0000 7fa0e659c640 1 -- 172.21.15.153:0/4230965429 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e80a1110 con 0x7fa0e809e910 2024-09-15T03:45:42.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e659c640 1 -- 172.21.15.153:0/4230965429 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 msgr2=0x7fa0e809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:42.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e659c640 1 --2- 172.21.15.153:0/4230965429 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 0x7fa0e809ece0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009850 tx=0x7fa0dc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c97dcc1b9dd2719e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:42.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 1 -- 172.21.15.153:0/4230965429 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc02ae80 con 0x7fa0e809e910 2024-09-15T03:45:42.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:42.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:42.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: dump: 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: _finish_auth 0 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 1 -- 172.21.15.153:0/4230965429 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc032440 con 0x7fa0e809e910 2024-09-15T03:45:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0ed11d640 5 monclient: authenticate success, global_id 194267 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0ed11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0ed11d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 1 -- 172.21.15.153:0/4230965429 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0327c0 con 0x7fa0e809e910 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0ed11d640 4 set_mon_vals no callback set 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.050+0000 7fa0e5d9b640 10 monclient: dump: 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.062 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 10 monclient: shutdown 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 -- 172.21.15.153:0/4230965429 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 msgr2=0x7fa0e809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/4230965429 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 0x7fa0e809ece0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009850 tx=0x7fa0dc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 -- 172.21.15.153:0/4230965429 shutdown_connections 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/4230965429 >> v2:172.21.15.153:3300/0 conn(0x7fa0e809f220 0x7fa0e8138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/4230965429 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809e910 0x7fa0e809ece0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 -- 172.21.15.153:0/4230965429 >> 172.21.15.153:0/4230965429 conn(0x7fa0e8067d30 msgr2=0x7fa0e80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 -- 172.21.15.153:0/4230965429 shutdown_connections 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 1 -- 172.21.15.153:0/4230965429 wait complete. 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 10 monclient: build_initial_monmap 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.051+0000 7fa0ed11d640 10 monclient: monmap: 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:42.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 Processor -- start 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 -- start start 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient: init 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0e809e910 0x7fa0e8133ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0e809e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 0x7fa0e8134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0e809f220 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 0x7fa0e8134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0e8131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8136550 con 0x7fa0e8131ab0 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8135f00 con 0x7fa0e809e910 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e80a3cd0 con 0x7fa0e809f220 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0ed11d640 10 monclient(hunting): authenticate will time out at 1739.691690s 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 0x7fa0e8134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 0x7fa0e8134410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34870/0 (socket says 172.21.15.153:34870) 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 1 -- 172.21.15.153:0/2401733706 learned_addr learned my addr 172.21.15.153:0/2401733706 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): get_auth_request con 0x7fa0e809f220 auth_method 0 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 0x7fa0e8134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 10 monclient(hunting): get_auth_request con 0x7fa0e8131ab0 auth_method 0 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e6d9d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0e809e910 0x7fa0e8133ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e6d9d640 10 monclient(hunting): get_auth_request con 0x7fa0e809e910 auth_method 0 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 msgr2=0x7fa0e8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:42.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 msgr2=0x7fa0e8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 0x7fa0e8134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.052+0000 7fa0e759e640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 0x7fa0e8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 10 monclient(hunting): handle_auth_done global_id 194270 payload 824 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0e809e910 msgr2=0x7fa0e8133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0e809e910 0x7fa0e8133ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 msgr2=0x7fa0e8134950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 0x7fa0e8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 1 monclient: found mon.noname-c 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e659c640 1 -- 172.21.15.153:0/2401733706 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc009390 con 0x7fa0e809f220 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.053+0000 7fa0e6d9d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0e809e910 0x7fa0e8133ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0e659c640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 msgr2=0x7fa0e8134410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0e659c640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 0x7fa0e8134410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009850 tx=0x7fa0dc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc031d40 con 0x7fa0e809f220 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: dump: 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc03d440 con 0x7fa0e809f220 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ed11d640 5 monclient: authenticate success, global_id 194270 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ed11d640 10 monclient: _renew_subs 2024-09-15T03:45:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc031ea0 con 0x7fa0e809f220 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ed11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e809bde0 con 0x7fa0e809f220 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0ceffd640 10 monclient: dump: 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.055+0000 7fa0e559a640 4 set_mon_vals no callback set 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 10 monclient: _renew_subs 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e8137560 con 0x7fa0e809f220 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa0dc039040 con 0x7fa0e809f220 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.056+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e8137820 con 0x7fa0e809f220 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.057+0000 7fa0ceffd640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0d4038740 0x7fa0d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.057+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa0dc06c340 con 0x7fa0e809f220 2024-09-15T03:45:42.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.057+0000 7fa0e6d9d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0d4038740 0x7fa0d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.057+0000 7fa0e6d9d640 10 monclient: get_auth_request con 0x7fa0d4038740 auth_method 0 2024-09-15T03:45:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.059+0000 7fa0e6d9d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0d4038740 0x7fa0d403abf0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8002860 tx=0x7fa0d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.061+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa0dc044ac0 con 0x7fa0e809f220 2024-09-15T03:45:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.061+0000 7fa0ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.061+0000 7fa0ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:42.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.277+0000 7fa0ed11d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:42.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.277+0000 7fa0ed11d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:42.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.277+0000 7fa0ed11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:42.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.277+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa0e804bf60 con 0x7fa0e809f220 2024-09-15T03:45:42.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.278+0000 7fa0ceffd640 1 -- 172.21.15.153:0/2401733706 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fa0dc042610 con 0x7fa0e809f220 2024-09-15T03:45:42.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.278+0000 7fa0ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:42.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.278+0000 7fa0ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:42.279 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":170,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0d4038740 msgr2=0x7fa0d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0d4038740 0x7fa0d403abf0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8002860 tx=0x7fa0d8009040 comp rx=0 tx=0).stop 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 10 monclient: shutdown 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 msgr2=0x7fa0e8134410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 0x7fa0e8134410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009850 tx=0x7fa0dc032000 comp rx=0 tx=0).stop 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 shutdown_connections 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa0d4038740 0x7fa0d403abf0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0e8131ab0 0x7fa0e8134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0e809f220 0x7fa0e8134410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 --2- 172.21.15.153:0/2401733706 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0e809e910 0x7fa0e8133ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:42.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 >> 172.21.15.153:0/2401733706 conn(0x7fa0e8067d30 msgr2=0x7fa0e809d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:42.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 shutdown_connections 2024-09-15T03:45:42.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:42.280+0000 7fa0ed11d640 1 -- 172.21.15.153:0/2401733706 wait complete. 2024-09-15T03:45:45.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.396+0000 7f22fe51d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.396+0000 7f22fe51d640 10 monclient: build_initial_monmap 2024-09-15T03:45:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.396+0000 7f22fe51d640 10 monclient: monmap: 2024-09-15T03:45:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:45.397516+0000 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:45.397516+0000 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:45.398 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.397+0000 7f22fe51d640 1 Processor -- start 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.397+0000 7f22fe51d640 1 -- start start 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.397+0000 7f22fe51d640 10 monclient: init 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22f80981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 0x7f22f8136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient(hunting): picked mon.noname-a con 0x7f22f8131da0 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22f8132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f809ca60 con 0x7f22f80981c0 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f809cca0 con 0x7f22f8132170 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f8131bd0 con 0x7f22f8131da0 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22fe51d640 10 monclient(hunting): authenticate will time out at 1743.037351s 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 0x7f22f8136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22f7fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f22f80981c0 0x7f22f8098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60296/0 (socket says 172.21.15.153:60296) 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22f7fff640 1 -- 172.21.15.153:0/2018000221 learned_addr learned my addr 172.21.15.153:0/2018000221 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 10 monclient(hunting): get_auth_request con 0x7f22f8131da0 auth_method 0 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 1 -- 172.21.15.153:0/2018000221 >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 msgr2=0x7f22f8136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 1 -- 172.21.15.153:0/2018000221 >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 msgr2=0x7f22f8136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 1 --2- 172.21.15.153:0/2018000221 >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 0x7f22f8136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.398+0000 7f22effff640 1 --2- 172.21.15.153:0/2018000221 >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 0x7f22f8136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:45.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.400+0000 7f22f7fff640 1 -- 172.21.15.153:0/2018000221 >> v1:172.21.15.153:6789/0 conn(0x7f22f80981c0 legacy=0x7f22f8098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.400+0000 7f22fcaba640 1 -- 172.21.15.153:0/2018000221 >> v1:172.21.15.181:6789/0 conn(0x7f22f8132170 legacy=0x7f22f81390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.400+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389598928 0 0) 0x7f22f809cca0 con 0x7f22f8132170 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): my global_id is 204172 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e40034e0 con 0x7f22f8132170 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876051048 0 0) 0x7f22e40034e0 con 0x7f22f8132170 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 msgr2=0x7f22f8136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 --2- 172.21.15.153:0/2018000221 >> v2:172.21.15.153:3300/0 conn(0x7f22f8131da0 0x7f22f8136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 >> v1:172.21.15.153:6789/0 conn(0x7f22f80981c0 legacy=0x7f22f8098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:45.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f809e010 con 0x7f22f8132170 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f22f0001bb0 con 0x7f22f8132170 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fe51d640 5 monclient: authenticate success, global_id 204172 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fe51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: dump: 2024-09-15T03:45:45.413 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 >> v1:172.21.15.181:6789/0 conn(0x7f22f8132170 legacy=0x7f22f81390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22e4003a50 0x7f22e4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): picked mon.c con 0x7f22e4003a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 0x7f22e4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): picked mon.a con 0x7f22e4004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e40052e0 0x7f22e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): picked mon.b con 0x7f22e40052e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f8131bd0 con 0x7f22e4004780 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e400c2b0 con 0x7f22e40052e0 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e400c470 con 0x7f22e4003a50 2024-09-15T03:45:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fe51d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 0x7f22e4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22e4003a50 0x7f22e4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): get_auth_request con 0x7f22e4004780 auth_method 0 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): get_auth_request con 0x7f22e4003a50 auth_method 0 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:45.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e40052e0 0x7f22e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): get_auth_request con 0x7f22e40052e0 auth_method 0 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient(hunting): handle_auth_done global_id 204172 payload 293 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 -- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22e4003a50 msgr2=0x7f22e4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22e4003a50 0x7f22e4003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 -- 172.21.15.153:0/2018000221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e40052e0 msgr2=0x7f22e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e40052e0 0x7f22e400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient: _finish_hunting 0 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 monclient: found mon.a 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 1 -- 172.21.15.153:0/2018000221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e40034e0 con 0x7f22e4004780 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22fcaba640 1 -- 172.21.15.153:0/2018000221 reap_dead start 2024-09-15T03:45:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 10 monclient: _finish_auth 0 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.401+0000 7f22f7fff640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22e4003a50 0x7f22e4003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22effff640 1 -- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 msgr2=0x7f22e4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22effff640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 0x7f22e4004bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006fd0 tx=0x7f22e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b4bf64bc316d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e002add0 con 0x7f22e4004780 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: dump: 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.419 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22fe51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e002e070 con 0x7f22e4004780 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 1 -- 172.21.15.153:0/2018000221 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0034660 con 0x7f22e4004780 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22fe51d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22fe51d640 4 set_mon_vals no callback set 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:45.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.404+0000 7f22f77fe640 10 monclient: dump: 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 10 monclient: shutdown 2024-09-15T03:45:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 -- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 msgr2=0x7f22e4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 0x7f22e4004bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006fd0 tx=0x7f22e0008040 comp rx=0 tx=0).stop 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 -- 172.21.15.153:0/2018000221 shutdown_connections 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 --2- 172.21.15.153:0/2018000221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e4004780 0x7f22e4004bf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 -- 172.21.15.153:0/2018000221 >> 172.21.15.153:0/2018000221 conn(0x7f22f80a1e30 msgr2=0x7f22f80a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 -- 172.21.15.153:0/2018000221 shutdown_connections 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 1 -- 172.21.15.153:0/2018000221 wait complete. 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 10 monclient: build_initial_monmap 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.405+0000 7f22fe51d640 10 monclient: monmap: 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:45.422 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 Processor -- start 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 -- start start 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient: init 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e4004780 0x7f22f811ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): picked mon.noname-b con 0x7f22e4004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:45.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 0x7f22f811e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): picked mon.noname-c con 0x7f22f811e300 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): picked mon.noname-a con 0x7f22f811ec10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f81218f0 con 0x7f22f811ec10 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f811f080 con 0x7f22e4004780 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f8120170 con 0x7f22f811e300 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fe51d640 10 monclient(hunting): authenticate will time out at 1743.045775s 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 0x7f22f811e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60906/0 (socket says 172.21.15.153:60906) 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 -- 172.21.15.153:0/907844288 learned_addr learned my addr 172.21.15.153:0/907844288 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 10 monclient(hunting): get_auth_request con 0x7f22f811ec10 auth_method 0 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 10 monclient(hunting): get_auth_request con 0x7f22f811e300 auth_method 0 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 msgr2=0x7f22f8147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 msgr2=0x7f22f8147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.406+0000 7f22fcaba640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e4004780 0x7f22f811ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): get_auth_request con 0x7f22e4004780 auth_method 0 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 10 monclient(hunting): handle_auth_done global_id 194273 payload 824 2024-09-15T03:45:45.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e4004780 msgr2=0x7f22f811ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e4004780 0x7f22f811ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 msgr2=0x7f22f8147740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 10 monclient: _finish_hunting 0 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 1 monclient: found mon.noname-c 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.407+0000 7f22effff640 1 -- 172.21.15.153:0/907844288 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80a06d0 con 0x7f22f811e300 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22effff640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 msgr2=0x7f22f811e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22effff640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 0x7f22f811e6d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22e0032000 tx=0x7f22e0035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0043070 con 0x7f22f811e300 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:45.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: dump: 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:45.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e0033760 con 0x7f22f811e300 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 5 monclient: authenticate success, global_id 194273 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 10 monclient: _renew_subs 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f8098cb0 con 0x7f22f811e300 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0033b20 con 0x7f22f811e300 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f4ff9640 10 monclient: dump: 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:45.429 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22f6ffd640 4 set_mon_vals no callback set 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 10 monclient: _renew_subs 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.409+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f8121230 con 0x7f22f811e300 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f22e003f440 con 0x7f22f811e300 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22f4ff9640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22d8038740 0x7f22d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f22e006cef0 con 0x7f22f811e300 2024-09-15T03:45:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22fe51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22fe51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22fe51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.410+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f811f830 con 0x7f22f811e300 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.411+0000 7f22f7fff640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22d8038740 0x7f22d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.411+0000 7f22f7fff640 10 monclient: get_auth_request con 0x7f22d8038740 auth_method 0 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.413+0000 7f22f7fff640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22d8038740 0x7f22d803abf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f22f804cdd0 tx=0x7f22e8007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:45.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.415+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f22f811f830 con 0x7f22f811e300 2024-09-15T03:45:45.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.415+0000 7f22f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:45.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.415+0000 7f22f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:45.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.627+0000 7f22fe51d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.627+0000 7f22fe51d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.627+0000 7f22fe51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.627+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f22f811f830 con 0x7f22f811e300 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.628+0000 7f22f4ff9640 1 -- 172.21.15.153:0/907844288 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f22f811f830 con 0x7f22f811e300 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.628+0000 7f22f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.628+0000 7f22f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:45.629 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":170,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22d8038740 msgr2=0x7f22d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:45.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22d8038740 0x7f22d803abf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f22f804cdd0 tx=0x7f22e8007d50 comp rx=0 tx=0).stop 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 10 monclient: shutdown 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 msgr2=0x7f22f811e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 0x7f22f811e6d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22e0032000 tx=0x7f22e0035000 comp rx=0 tx=0).stop 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 shutdown_connections 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22d8038740 0x7f22d803abf0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22f811ec10 0x7f22f8147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22f811e300 0x7f22f811e6d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 --2- 172.21.15.153:0/907844288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22e4004780 0x7f22f811ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 >> 172.21.15.153:0/907844288 conn(0x7f22f80a1e30 msgr2=0x7f22f809b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 shutdown_connections 2024-09-15T03:45:45.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:45.630+0000 7f22fe51d640 1 -- 172.21.15.153:0/907844288 wait complete. 2024-09-15T03:45:48.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:48.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.747+0000 7f8af055d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.747+0000 7f8af055d640 10 monclient: build_initial_monmap 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.747+0000 7f8af055d640 10 monclient: monmap: 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:48.747886+0000 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:48.747886+0000 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:48.748 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:48.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:48.749 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:48.749 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:48.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:48.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.747+0000 7f8af055d640 1 Processor -- start 2024-09-15T03:45:48.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.747+0000 7f8af055d640 1 -- start start 2024-09-15T03:45:48.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.747+0000 7f8af055d640 10 monclient: init 2024-09-15T03:45:48.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:48.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae8131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae809e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ae809f220 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae809faf0 con 0x7f8ae809f220 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae809fd30 con 0x7f8ae809e910 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80a1110 con 0x7f8ae8131ab0 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8af055d640 10 monclient(hunting): authenticate will time out at 1746.387591s 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56040/0 (socket says 172.21.15.153:56040) 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60918/0 (socket says 172.21.15.153:60918) 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 learned_addr learned my addr 172.21.15.153:0/2111487376 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request con 0x7f8ae8131ab0 auth_method 0 2024-09-15T03:45:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ae809e910 auth_method 0 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.748+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.751+0000 7f8aeeafa640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.181:6789/0 conn(0x7f8ae809f220 legacy=0x7f8ae8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:48.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request con 0x7f8ae8131ab0 auth_method 0 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ae809e910 auth_method 0 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:48.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:48.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:48.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:48.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:48.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:48.949+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:49.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:49.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request con 0x7f8ae8131ab0 auth_method 0 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ae809e910 auth_method 0 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:49.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:49.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:49.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:49.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:49.350+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:50.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:50.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:50.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request con 0x7f8ae8131ab0 auth_method 0 2024-09-15T03:45:50.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ae809e910 auth_method 0 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:50.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:50.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:50.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:50.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:50.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:45:50.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:50.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:50.151+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient(hunting): tick 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae8131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 msgr2=0x7f8ae809ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.153:3300/0 conn(0x7f8ae809e910 0x7f8ae809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.181:6789/0 conn(0x7f8ae809f220 legacy=0x7f8ae8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient: _add_conns ranks=[0,4] 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ad4003700 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8ad40073a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae80a13f0 con 0x7f8ad4003700 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae809fd30 con 0x7f8ad40073a0 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.748+0000 7f8ae7fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ad40073a0 auth_method 0 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:51.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:51.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:51.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:51.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:51.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.749+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:51.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.751+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.153:6789/0 conn(0x7f8ad4003700 legacy=0x7f8ad4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ad40073a0 auth_method 0 2024-09-15T03:45:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:51.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:45:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:51.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:51.950+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:45:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.037+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.153:6789/0 conn(0x7f8ad4003700 legacy=0x7f8ad4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T03:45:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.037+0000 7f8aee2f9640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.153:6789/0 conn(0x7f8ad4003700 legacy=0x7f8ad4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:45:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.037+0000 7f8aee2f9640 1 --1- 172.21.15.153:0/2111487376 >> v1:172.21.15.153:6789/0 conn(0x7f8ad4003700 0x7f8ad4005b70 :-1 s=OPENED pgs=26 cs=1 l=1).handle_message read tag failed 2024-09-15T03:45:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.037+0000 7f8aee2f9640 1 --1- 172.21.15.153:0/2111487376 >> v1:172.21.15.153:6789/0 conn(0x7f8ad4003700 0x7f8ad4005b70 :-1 s=OPENED pgs=26 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:45:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.037+0000 7f8aed2f7640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6789/0 2024-09-15T03:45:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ad40073a0 auth_method 0 2024-09-15T03:45:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:52.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:52.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:52.351+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:45:53.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ad40073a0 auth_method 0 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.152+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:45:53.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:53.153+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:45:54.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.748+0000 7f8ae7fff640 10 monclient(hunting): tick 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.748+0000 7f8ae7fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.748+0000 7f8ae7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.748+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad4009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad4009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.153:6789/0 conn(0x7f8ad4003700 legacy=0x7f8ad4005b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 -- 172.21.15.153:0/2111487376 reap_dead start 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ae809f220 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:54.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad40097d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8ad40073a0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ad400a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae8131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad400af60 con 0x7f8ae809f220 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae809fd30 con 0x7f8ad40073a0 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 1 -- 172.21.15.153:0/2111487376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80a1110 con 0x7f8ae8131ab0 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8ae7fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:54.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ad400a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad40097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): get_auth_request con 0x7f8ae8131ab0 auth_method 0 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ad40073a0 auth_method 0 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:54.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_done global_id 194276 payload 274 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 -- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 msgr2=0x7f8ad40097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad40097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 -- 172.21.15.153:0/2111487376 >> v1:172.21.15.181:6789/0 conn(0x7f8ae809f220 legacy=0x7f8ad4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient: _finish_hunting 0 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 monclient: found mon.noname-b 2024-09-15T03:45:54.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:54.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:54.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.749+0000 7f8aeeafa640 1 -- 172.21.15.153:0/2111487376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad400acd0 con 0x7f8ae8131ab0 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.752+0000 7f8aeeafa640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ad400a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.752+0000 7f8aeeafa640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ad400a6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4003ad0 tx=0x7f8ad8007c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4648f70da934bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 1 -- 172.21.15.153:0/2111487376 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80070f0 con 0x7f8ae8131ab0 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: dump: 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:54.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: _finish_auth 0 2024-09-15T03:45:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 1 -- 172.21.15.153:0/2111487376 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad802c070 con 0x7f8ae8131ab0 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 1 -- 172.21.15.153:0/2111487376 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80074a0 con 0x7f8ae8131ab0 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 5 monclient: authenticate success, global_id 194276 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 4 set_mon_vals no callback set 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8aed2f7640 10 monclient: dump: 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 10 monclient: shutdown 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 1 -- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ad400a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ad400a6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4003ad0 tx=0x7f8ad8007c60 comp rx=0 tx=0).stop 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 1 -- 172.21.15.153:0/2111487376 shutdown_connections 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.753+0000 7f8af055d640 1 --2- 172.21.15.153:0/2111487376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ad400a6b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.754+0000 7f8af055d640 1 --2- 172.21.15.153:0/2111487376 >> v2:172.21.15.181:3300/0 conn(0x7f8ad40073a0 0x7f8ad40097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.754+0000 7f8af055d640 1 -- 172.21.15.153:0/2111487376 >> 172.21.15.153:0/2111487376 conn(0x7f8ae8067d30 msgr2=0x7f8ae80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.754+0000 7f8af055d640 1 -- 172.21.15.153:0/2111487376 shutdown_connections 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.754+0000 7f8af055d640 1 -- 172.21.15.153:0/2111487376 wait complete. 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.754+0000 7f8af055d640 10 monclient: build_initial_monmap 2024-09-15T03:45:54.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.754+0000 7f8af055d640 10 monclient: monmap: 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 Processor -- start 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 -- start start 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient: init 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ad40073a0 0x7f8ae8141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ad40073a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 0x7f8ae8141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae809f220 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae811e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ae8131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80a46a0 con 0x7f8ae809f220 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8122ad0 con 0x7f8ad40073a0 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8121e00 con 0x7f8ae8131ab0 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8af055d640 10 monclient(hunting): authenticate will time out at 1752.394629s 2024-09-15T03:45:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae811e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 0x7f8ae8141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae811e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56108/0 (socket says 172.21.15.153:56108) 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 0x7f8ae8141870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60968/0 (socket says 172.21.15.153:60968) 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 1 -- 172.21.15.153:0/1067668847 learned_addr learned my addr 172.21.15.153:0/1067668847 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): get_auth_request con 0x7f8ae8131ab0 auth_method 0 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request con 0x7f8ae809f220 auth_method 0 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aeeafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.755+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ad40073a0 0x7f8ae8141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request con 0x7f8ad40073a0 auth_method 0 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aee2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aee2f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aee2f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 10 monclient(hunting): handle_auth_done global_id 214098 payload 824 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 1 -- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 msgr2=0x7f8ae811e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae811e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 1 -- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ad40073a0 msgr2=0x7f8ae8141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ad40073a0 0x7f8ae8141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 10 monclient: _finish_hunting 0 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 1 monclient: found mon.noname-a 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aedaf8640 1 -- 172.21.15.153:0/1067668847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae809faf0 con 0x7f8ae809f220 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ad40073a0 0x7f8ae8141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.756+0000 7f8aeeafa640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae811e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8aedaf8640 1 -- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 msgr2=0x7f8ae8141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8aedaf8640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 0x7f8ae8141870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8adc018b50 tx=0x7f8adc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc027070 con 0x7f8ae809f220 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: dump: 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc01f040 con 0x7f8ae809f220 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 5 monclient: authenticate success, global_id 214098 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 10 monclient: _renew_subs 2024-09-15T03:45:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae8120950 con 0x7f8ae809f220 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc01e050 con 0x7f8ae809f220 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae67fc640 10 monclient: dump: 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 10 monclient: _renew_subs 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae811fd80 con 0x7f8ae809f220 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.758+0000 7f8ae7fff640 4 set_mon_vals no callback set 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.759+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8adc023900 con 0x7f8ae809f220 2024-09-15T03:45:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8af055d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8af055d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8af055d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8ae67fc640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ad40395f0 0x7f8ad403baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae81219a0 con 0x7f8ae809f220 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f8adc04da30 con 0x7f8ae809f220 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ad40395f0 0x7f8ad403baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:54.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.760+0000 7f8aee2f9640 10 monclient: get_auth_request con 0x7f8ad40395f0 auth_method 0 2024-09-15T03:45:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.763+0000 7f8aee2f9640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ad40395f0 0x7f8ad403baa0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0013380 tx=0x7f8ae0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.764+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8adc02c410 con 0x7f8ae809f220 2024-09-15T03:45:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.765+0000 7f8ae67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.765+0000 7f8ae67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:54.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.986+0000 7f8af055d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:54.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.986+0000 7f8af055d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:54.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.986+0000 7f8af055d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:54.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.986+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ae804bf60 con 0x7f8ae809f220 2024-09-15T03:45:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.987+0000 7f8ae67fc640 1 -- 172.21.15.153:0/1067668847 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f8adc0250d0 con 0x7f8ae809f220 2024-09-15T03:45:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.987+0000 7f8ae67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.987+0000 7f8ae67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:54.988 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ad40395f0 msgr2=0x7f8ad403baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ad40395f0 0x7f8ad403baa0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0013380 tx=0x7f8ae0009040 comp rx=0 tx=0).stop 2024-09-15T03:45:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 10 monclient: shutdown 2024-09-15T03:45:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 msgr2=0x7f8ae8141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 0x7f8ae8141870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8adc018b50 tx=0x7f8adc008040 comp rx=0 tx=0).stop 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 shutdown_connections 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ad40395f0 0x7f8ad403baa0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ae8131ab0 0x7f8ae811e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ae809f220 0x7f8ae8141870 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 --2- 172.21.15.153:0/1067668847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ad40073a0 0x7f8ae8141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 >> 172.21.15.153:0/1067668847 conn(0x7f8ae8067d30 msgr2=0x7f8ae809c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 shutdown_connections 2024-09-15T03:45:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:54.989+0000 7f8af055d640 1 -- 172.21.15.153:0/1067668847 wait complete. 2024-09-15T03:45:55.000 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:45:55.001 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:45:55.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:45:55.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.094+0000 7fccd219d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:55.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.094+0000 7fccd219d640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.094+0000 7fccd219d640 10 monclient: monmap: 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:55.095561+0000 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:55.095561+0000 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:55.096 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:55.097 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.095+0000 7fccd219d640 1 Processor -- start 2024-09-15T03:45:55.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.095+0000 7fccd219d640 1 -- start start 2024-09-15T03:45:55.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.095+0000 7fccd219d640 10 monclient: init 2024-09-15T03:45:55.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.095+0000 7fccd219d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.095+0000 7fccd219d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:45:55.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 0x7fcccc09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): picked mon.noname-b con 0x7fcccc096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fcccc09aaa0 0x7fcccc0971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): picked mon.noname-c con 0x7fcccc09aaa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcccc097720 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc099620 con 0x7fcccc097720 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc099340 con 0x7fcccc09aaa0 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc099900 con 0x7fcccc096e10 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fccd219d640 10 monclient(hunting): authenticate will time out at 1752.735090s 2024-09-15T03:45:55.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 0x7fcccc09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 0x7fcccc09a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:56116/0 (socket says 172.21.15.153:56116) 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 -- 172.21.15.153:0/4223632618 learned_addr learned my addr 172.21.15.153:0/4223632618 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): get_auth_request con 0x7fcccc096e10 auth_method 0 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccaffd640 1 --2- 172.21.15.153:0/4223632618 >> v2:172.21.15.181:3300/0 conn(0x7fcccc09aaa0 0x7fcccc0971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_done global_id 194282 payload 274 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 -- 172.21.15.153:0/4223632618 >> v2:172.21.15.181:3300/0 conn(0x7fcccc09aaa0 msgr2=0x7fcccc0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 --2- 172.21.15.153:0/4223632618 >> v2:172.21.15.181:3300/0 conn(0x7fcccc09aaa0 0x7fcccc0971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 -- 172.21.15.153:0/4223632618 >> v1:172.21.15.181:6789/0 conn(0x7fcccc097720 legacy=0x7fcccc12e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:45:55.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.096+0000 7fcccb7fe640 1 -- 172.21.15.153:0/4223632618 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc097b10 con 0x7fcccc096e10 2024-09-15T03:45:55.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fcccb7fe640 1 -- 172.21.15.153:0/4223632618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 msgr2=0x7fcccc09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fcccb7fe640 1 --2- 172.21.15.153:0/4223632618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 0x7fcccc09a560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009910 tx=0x7fccbc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4dee9ecba265a2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 1 -- 172.21.15.153:0/4223632618 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc032440 con 0x7fcccc096e10 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: dump: 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.110 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 1 -- 172.21.15.153:0/4223632618 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc0325a0 con 0x7fcccc096e10 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 5 monclient: authenticate success, global_id 194282 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 1 -- 172.21.15.153:0/4223632618 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc032980 con 0x7fcccc096e10 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccca7fc640 10 monclient: dump: 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 4 set_mon_vals no callback set 2024-09-15T03:45:55.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 10 monclient: shutdown 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 1 -- 172.21.15.153:0/4223632618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 msgr2=0x7fcccc09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 1 --2- 172.21.15.153:0/4223632618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 0x7fcccc09a560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009910 tx=0x7fccbc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.099+0000 7fccd219d640 1 -- 172.21.15.153:0/4223632618 shutdown_connections 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 1 --2- 172.21.15.153:0/4223632618 >> v2:172.21.15.181:3300/0 conn(0x7fcccc09aaa0 0x7fcccc0971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 1 --2- 172.21.15.153:0/4223632618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc096e10 0x7fcccc09a560 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 1 -- 172.21.15.153:0/4223632618 >> 172.21.15.153:0/4223632618 conn(0x7fcccc0672b0 msgr2=0x7fcccc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 1 -- 172.21.15.153:0/4223632618 shutdown_connections 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 1 -- 172.21.15.153:0/4223632618 wait complete. 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 10 monclient: monmap: 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:55.113 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.100+0000 7fccd219d640 1 Processor -- start 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 -- start start 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient: init 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:45:55.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc096e10 0x7fcccc13f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): picked mon.noname-a con 0x7fcccc096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc097720 0x7fcccc12e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): picked mon.noname-b con 0x7fcccc097720 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 0x7fcccc128be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): picked mon.noname-c con 0x7fcccc09aaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc06e790 con 0x7fcccc096e10 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc09b050 con 0x7fcccc097720 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc06ed60 con 0x7fcccc09aaa0 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fccd219d640 10 monclient(hunting): authenticate will time out at 1752.740517s 2024-09-15T03:45:55.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc096e10 0x7fcccc13f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 0x7fcccc128be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc096e10 0x7fcccc13f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60978/0 (socket says 172.21.15.153:60978) 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 1 -- 172.21.15.153:0/289739374 learned_addr learned my addr 172.21.15.153:0/289739374 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): get_auth_request con 0x7fcccc09aaa0 auth_method 0 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): get_auth_request con 0x7fcccc096e10 auth_method 0 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccaffd640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc097720 0x7fcccc12e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccaffd640 10 monclient(hunting): get_auth_request con 0x7fcccc097720 auth_method 0 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient(hunting): handle_auth_done global_id 194285 payload 824 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc097720 msgr2=0x7fcccc12e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc097720 0x7fcccc12e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc096e10 msgr2=0x7fcccc13f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc096e10 0x7fcccc13f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 monclient: found mon.noname-c 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccbfff640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc009450 con 0x7fcccc09aaa0 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.101+0000 7fcccaffd640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc097720 0x7fcccc12e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fcccbfff640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 msgr2=0x7fcccc128be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fcccbfff640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 0x7fcccc128be0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c810 tx=0x7fccc40025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400a850 con 0x7fcccc09aaa0 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: dump: 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc400a9b0 con 0x7fcccc09aaa0 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 5 monclient: authenticate success, global_id 194285 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400ad70 con 0x7fcccc09aaa0 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 10 monclient: _renew_subs 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc13fde0 con 0x7fcccc09aaa0 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccb37fe640 10 monclient: dump: 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccc9ffb640 4 set_mon_vals no callback set 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 10 monclient: _renew_subs 2024-09-15T03:45:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.104+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc06ddb0 con 0x7fcccc09aaa0 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fccc4012630 con 0x7fcccc09aaa0 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccd219d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccd219d640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 0x7fcccc129120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc12b3f0 con 0x7fcccc12e570 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccb37fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fccb8038740 0x7fccb803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 0x7fcccc129120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fccc404c840 con 0x7fcccc09aaa0 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient: get_auth_request con 0x7fcccc12e570 auth_method 0 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccaffd640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fccb8038740 0x7fccb803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_done global_id 214104 payload 824 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccb7fe640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcccc12b8a0 con 0x7fcccc12e570 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.105+0000 7fcccaffd640 10 monclient: get_auth_request con 0x7fccb8038740 auth_method 0 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.107+0000 7fcccb7fe640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 msgr2=0x7fcccc129120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.107+0000 7fcccb7fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 0x7fcccc129120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009420 tx=0x7fccbc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.107+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc031a70 con 0x7fcccc12e570 2024-09-15T03:45:55.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.107+0000 7fccb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.107+0000 7fccb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.107+0000 7fccb37fe640 10 monclient: dump: 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.124 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fcccaffd640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fccb8038740 0x7fccb803abf0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fccc000aaa0 tx=0x7fccc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fccbc031bd0 con 0x7fcccc12e570 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 msgr2=0x7fcccc129120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 0x7fcccc129120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009420 tx=0x7fccbc032000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 msgr2=0x7fcccc129120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:55.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.108+0000 7fccb37fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12e570 0x7fcccc129120 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009420 tx=0x7fccbc032000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.113+0000 7fccd219d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:55.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.113+0000 7fccd219d640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 0x7fcccc129660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.113+0000 7fccd219d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.113+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc065f20 con 0x7fcccc12a360 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 0x7fcccc129660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient: get_auth_request con 0x7fcccc12a360 auth_method 0 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 10 monclient(hunting): handle_auth_done global_id 214107 payload 824 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.114+0000 7fcccb7fe640 1 -- 172.21.15.153:0/289739374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcccc04bf60 con 0x7fcccc12a360 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fcccb7fe640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 msgr2=0x7fcccc129660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fcccb7fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 0x7fcccc129660 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcccc09b230 tx=0x7fccbc03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc03c860 con 0x7fcccc12a360 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 10 monclient: dump: 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.129 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fccbc03c9c0 con 0x7fcccc12a360 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 msgr2=0x7fcccc129660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 0x7fcccc129660 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcccc09b230 tx=0x7fccbc03d000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 msgr2=0x7fcccc129660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.116+0000 7fccb37fe640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcccc12a360 0x7fcccc129660 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcccc09b230 tx=0x7fccbc03d000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.130 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 178, 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:55.131 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:55.133 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:55.134 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.135 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:55.136 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.138 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.140 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:55.143 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.143 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.143 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:55.143 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:55.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fccb8038740 msgr2=0x7fccb803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fccb8038740 0x7fccb803abf0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fccc000aaa0 tx=0x7fccc0008040 comp rx=0 tx=0).stop 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 10 monclient: shutdown 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 msgr2=0x7fcccc128be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fcccbfff640 1 -- 172.21.15.153:0/289739374 reap_dead start 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 0x7fcccc128be0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c810 tx=0x7fccc40025b0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 shutdown_connections 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 --2- 172.21.15.153:0/289739374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcccc09aaa0 0x7fcccc128be0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 >> 172.21.15.153:0/289739374 conn(0x7fcccc0672b0 msgr2=0x7fcccc09d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 shutdown_connections 2024-09-15T03:45:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.117+0000 7fccd219d640 1 -- 172.21.15.153:0/289739374 wait complete. 2024-09-15T03:45:55.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:55.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:45:55.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:45:55.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.239+0000 7fcb650c0640 10 monclient: get_monmap_and_config 2024-09-15T03:45:55.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.239+0000 7fcb650c0640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.239+0000 7fcb650c0640 10 monclient: monmap: 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:55.239905+0000 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:55.239905+0000 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.239+0000 7fcb650c0640 1 Processor -- start 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.239+0000 7fcb650c0640 1 -- start start 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.239+0000 7fcb650c0640 10 monclient: init 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb60131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb6009e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 0x7fcb60139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient(hunting): picked mon.noname-b con 0x7fcb60137bc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6009faf0 con 0x7fcb6009e910 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6009fd30 con 0x7fcb60131ab0 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60132590 con 0x7fcb60137bc0 2024-09-15T03:45:55.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb650c0640 10 monclient(hunting): authenticate will time out at 1752.879383s 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 0x7fcb60139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcb6009e910 0x7fcb6009ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50598/0 (socket says 172.21.15.153:50598) 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5e59c640 1 -- 172.21.15.153:0/3740721227 learned_addr learned my addr 172.21.15.153:0/3740721227 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): get_auth_request con 0x7fcb60137bc0 auth_method 0 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_done global_id 194288 payload 274 2024-09-15T03:45:55.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 1 -- 172.21.15.153:0/3740721227 >> v1:172.21.15.181:6789/0 conn(0x7fcb60131ab0 legacy=0x7fcb60131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 1 -- 172.21.15.153:0/3740721227 >> v1:172.21.15.153:6789/0 conn(0x7fcb6009e910 legacy=0x7fcb6009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 1 monclient: found mon.noname-b 2024-09-15T03:45:55.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.240+0000 7fcb5f59e640 1 -- 172.21.15.153:0/3740721227 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb600a1110 con 0x7fcb60137bc0 2024-09-15T03:45:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.243+0000 7fcb5f59e640 1 -- 172.21.15.153:0/3740721227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 msgr2=0x7fcb60139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.243+0000 7fcb5f59e640 1 --2- 172.21.15.153:0/3740721227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 0x7fcb60139fa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009e90 tx=0x7fcb58009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f5253100e4f912e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.243+0000 7fcb5dd9b640 1 -- 172.21.15.153:0/3740721227 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007840 con 0x7fcb60137bc0 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.243+0000 7fcb5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: dump: 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 1 -- 172.21.15.153:0/3740721227 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5802d050 con 0x7fcb60137bc0 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 1 -- 172.21.15.153:0/3740721227 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007bd0 con 0x7fcb60137bc0 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 5 monclient: authenticate success, global_id 194288 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 4 set_mon_vals no callback set 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb5dd9b640 10 monclient: dump: 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 10 monclient: shutdown 2024-09-15T03:45:55.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 -- 172.21.15.153:0/3740721227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 msgr2=0x7fcb60139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 --2- 172.21.15.153:0/3740721227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 0x7fcb60139fa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009e90 tx=0x7fcb58009780 comp rx=0 tx=0).stop 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 -- 172.21.15.153:0/3740721227 shutdown_connections 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 --2- 172.21.15.153:0/3740721227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb60137bc0 0x7fcb60139fa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 -- 172.21.15.153:0/3740721227 >> 172.21.15.153:0/3740721227 conn(0x7fcb60067d30 msgr2=0x7fcb600a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 -- 172.21.15.153:0/3740721227 shutdown_connections 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.244+0000 7fcb650c0640 1 -- 172.21.15.153:0/3740721227 wait complete. 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 10 monclient: monmap: 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 1 Processor -- start 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 1 -- start start 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 10 monclient: init 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.245+0000 7fcb650c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 0x7fcb60137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb60131ab0 0x7fcb6012e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): picked mon.noname-b con 0x7fcb60131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60137bc0 0x7fcb6012e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): picked mon.noname-a con 0x7fcb60137bc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60132770 con 0x7fcb60137bc0 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6012f920 con 0x7fcb60131ab0 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb600a54d0 con 0x7fcb6009e910 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb650c0640 10 monclient(hunting): authenticate will time out at 1752.885080s 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 0x7fcb60137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60137bc0 0x7fcb6012e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 0x7fcb60137090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37636/0 (socket says 172.21.15.153:37636) 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 -- 172.21.15.153:0/1888604228 learned_addr learned my addr 172.21.15.153:0/1888604228 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): get_auth_request con 0x7fcb6009e910 auth_method 0 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): get_auth_request con 0x7fcb60137bc0 auth_method 0 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb60131ab0 0x7fcb6012e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): get_auth_request con 0x7fcb60131ab0 auth_method 0 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient(hunting): handle_auth_done global_id 194291 payload 824 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb60131ab0 msgr2=0x7fcb6012e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb60131ab0 0x7fcb6012e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60137bc0 msgr2=0x7fcb6012e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60137bc0 0x7fcb6012e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.246+0000 7fcb5ed9d640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb58009a20 con 0x7fcb6009e910 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb5ed9d640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 msgr2=0x7fcb60137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb5ed9d640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 0x7fcb60137090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000c5e0 tx=0x7fcb50007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50012840 con 0x7fcb6009e910 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:55.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: dump: 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 5 monclient: authenticate success, global_id 194291 2024-09-15T03:45:55.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb500129a0 con 0x7fcb6009e910 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 10 monclient: _renew_subs 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb60131300 con 0x7fcb6009e910 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50012da0 con 0x7fcb6009e910 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb46ffd640 10 monclient: dump: 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 10 monclient: _renew_subs 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6012fc80 con 0x7fcb6009e910 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.248+0000 7fcb5d59a640 4 set_mon_vals no callback set 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb650c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb650c0640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 0x7fcb6009d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb650c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60063740 con 0x7fcb6012ead0 2024-09-15T03:45:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcb50011590 con 0x7fcb6009e910 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 0x7fcb6009d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient: get_auth_request con 0x7fcb6012ead0 auth_method 0 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb46ffd640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.249+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcb5004c6b0 con 0x7fcb6009e910 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.250+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_done global_id 194294 payload 824 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.250+0000 7fcb5e59c640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb60132910 con 0x7fcb6012ead0 2024-09-15T03:45:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.250+0000 7fcb5f59e640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.250+0000 7fcb5f59e640 10 monclient: get_auth_request con 0x7fcb4c038740 auth_method 0 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb5e59c640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 msgr2=0x7fcb6009d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb5e59c640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 0x7fcb6009d560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e710 tx=0x7fcb54007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54004b50 con 0x7fcb6012ead0 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 10 monclient: dump: 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcb54004cb0 con 0x7fcb6012ead0 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 msgr2=0x7fcb6009d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 0x7fcb6009d560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e710 tx=0x7fcb54007f00 comp rx=0 tx=0).stop 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 msgr2=0x7fcb6009d560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb46ffd640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6012ead0 0x7fcb6009d560 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e710 tx=0x7fcb54007f00 comp rx=0 tx=0).stop 2024-09-15T03:45:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.252+0000 7fcb5f59e640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcb58034000 tx=0x7fcb58037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb650c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:55.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb650c0640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 0x7fcb6009dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb650c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6012eea0 con 0x7fcb6009daa0 2024-09-15T03:45:55.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 0x7fcb6009dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient: get_auth_request con 0x7fcb6009daa0 auth_method 0 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 10 monclient(hunting): handle_auth_done global_id 194297 payload 824 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.258+0000 7fcb5e59c640 1 -- 172.21.15.153:0/1888604228 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcb6004bf60 con 0x7fcb6009daa0 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.260+0000 7fcb5e59c640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 msgr2=0x7fcb6009dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb5e59c640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 0x7fcb6009dea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e710 tx=0x7fcb54019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54015d30 con 0x7fcb6009daa0 2024-09-15T03:45:55.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 10 monclient: dump: 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fcb54015e90 con 0x7fcb6009daa0 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 msgr2=0x7fcb6009dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 0x7fcb6009dea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e710 tx=0x7fcb54019000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 msgr2=0x7fcb6009dea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb46ffd640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009daa0 0x7fcb6009dea0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e710 tx=0x7fcb54019000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:55.273 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 178, 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.274 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.275 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:55.276 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:55.277 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.278 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:55.279 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:55.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.281 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.283 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.284 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.285 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:55.286 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.286 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.286 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:55.286 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:55.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 msgr2=0x7fcb4c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcb4c038740 0x7fcb4c03abf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcb58034000 tx=0x7fcb58037000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 10 monclient: shutdown 2024-09-15T03:45:55.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 msgr2=0x7fcb60137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 --2- 172.21.15.153:0/1888604228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6009e910 0x7fcb60137090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000c5e0 tx=0x7fcb50007ed0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb5f59e640 1 -- 172.21.15.153:0/1888604228 reap_dead start 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb5f59e640 1 -- 172.21.15.153:0/1888604228 reap_dead start 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 shutdown_connections 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 >> 172.21.15.153:0/1888604228 conn(0x7fcb60067d30 msgr2=0x7fcb601390e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 shutdown_connections 2024-09-15T03:45:55.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.261+0000 7fcb650c0640 1 -- 172.21.15.153:0/1888604228 wait complete. 2024-09-15T03:45:55.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:55.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:45:55.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:45:55.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.381+0000 7fd6b6b5d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:55.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.381+0000 7fd6b6b5d640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.381+0000 7fd6b6b5d640 10 monclient: monmap: 2024-09-15T03:45:55.382 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.382 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:55.382362+0000 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:55.382362+0000 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:55.383 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 Processor -- start 2024-09-15T03:45:55.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 -- start start 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient: init 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6b00981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 0x7fd6b0136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b0131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd6b0132170 0x7fd6b01390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b0132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b009ca60 con 0x7fd6b00981c0 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b0131bd0 con 0x7fd6b0132170 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b009cca0 con 0x7fd6b0131da0 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b6b5d640 10 monclient(hunting): authenticate will time out at 1753.021854s 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6a7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 0x7fd6b0136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b48f9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd6b00981c0 0x7fd6b0098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50606/0 (socket says 172.21.15.153:50606) 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.382+0000 7fd6b48f9640 1 -- 172.21.15.153:0/1521889258 learned_addr learned my addr 172.21.15.153:0/1521889258 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): get_auth_request con 0x7fd6b0131da0 auth_method 0 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_done global_id 194300 payload 274 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6b50fa640 1 --2- 172.21.15.153:0/1521889258 >> v2:172.21.15.181:3300/0 conn(0x7fd6b0132170 0x7fd6b01390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 1 -- 172.21.15.153:0/1521889258 >> v2:172.21.15.181:3300/0 conn(0x7fd6b0132170 msgr2=0x7fd6b01390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 1 --2- 172.21.15.153:0/1521889258 >> v2:172.21.15.181:3300/0 conn(0x7fd6b0132170 0x7fd6b01390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 1 -- 172.21.15.153:0/1521889258 >> v1:172.21.15.153:6789/0 conn(0x7fd6b00981c0 legacy=0x7fd6b0098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 1 monclient: found mon.noname-b 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.383+0000 7fd6a7fff640 1 -- 172.21.15.153:0/1521889258 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b009e010 con 0x7fd6b0131da0 2024-09-15T03:45:55.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a7fff640 1 -- 172.21.15.153:0/1521889258 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 msgr2=0x7fd6b0136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a7fff640 1 --2- 172.21.15.153:0/1521889258 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 0x7fd6b0136b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009850 tx=0x7fd6a00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c450053bb1edbff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 1 -- 172.21.15.153:0/1521889258 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0032440 con 0x7fd6b0131da0 2024-09-15T03:45:55.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:55.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: dump: 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 1 -- 172.21.15.153:0/1521889258 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a00325a0 con 0x7fd6b0131da0 2024-09-15T03:45:55.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 5 monclient: authenticate success, global_id 194300 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 1 -- 172.21.15.153:0/1521889258 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0032940 con 0x7fd6b0131da0 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 4 set_mon_vals no callback set 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6a77fe640 10 monclient: dump: 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.398 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 10 monclient: shutdown 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/1521889258 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 msgr2=0x7fd6b0136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/1521889258 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 0x7fd6b0136b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009850 tx=0x7fd6a00090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/1521889258 shutdown_connections 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/1521889258 >> v2:172.21.15.181:3300/0 conn(0x7fd6b0132170 0x7fd6b01390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/1521889258 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0131da0 0x7fd6b0136b70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/1521889258 >> 172.21.15.153:0/1521889258 conn(0x7fd6b00a1e30 msgr2=0x7fd6b00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/1521889258 shutdown_connections 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.386+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/1521889258 wait complete. 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.387+0000 7fd6b6b5d640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.387+0000 7fd6b6b5d640 10 monclient: monmap: 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:55.400 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.387+0000 7fd6b6b5d640 1 Processor -- start 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.387+0000 7fd6b6b5d640 1 -- start start 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.387+0000 7fd6b6b5d640 10 monclient: init 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:45:55.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b00981c0 0x7fd6b0137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b00981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b0131da0 0x7fd6b01384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b0131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 0x7fd6b0138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b0132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0130b50 con 0x7fd6b0131da0 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0118f70 con 0x7fd6b00981c0 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00a09e0 con 0x7fd6b0132170 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b6b5d640 10 monclient(hunting): authenticate will time out at 1753.027220s 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b0131da0 0x7fd6b01384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 0x7fd6b0138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b0131da0 0x7fd6b01384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47048/0 (socket says 172.21.15.153:47048) 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 0x7fd6b0138a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37688/0 (socket says 172.21.15.153:37688) 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 1 -- 172.21.15.153:0/3441596209 learned_addr learned my addr 172.21.15.153:0/3441596209 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): get_auth_request con 0x7fd6b0131da0 auth_method 0 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): get_auth_request con 0x7fd6b0132170 auth_method 0 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b48f9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b00981c0 0x7fd6b0137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b48f9640 10 monclient(hunting): get_auth_request con 0x7fd6b00981c0 auth_method 0 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b48f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b48f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b48f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient(hunting): handle_auth_done global_id 194303 payload 824 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b00981c0 msgr2=0x7fd6b0137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b00981c0 0x7fd6b0137f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b0131da0 msgr2=0x7fd6b01384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b0131da0 0x7fd6b01384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 monclient: found mon.noname-c 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6b50fa640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0009390 con 0x7fd6b0132170 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.388+0000 7fd6a7fff640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b0131da0 0x7fd6b01384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.390+0000 7fd6b50fa640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 msgr2=0x7fd6b0138a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.390+0000 7fd6b50fa640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 0x7fd6b0138a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800c650 tx=0x7fd6a8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.390+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8011810 con 0x7fd6b0132170 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: dump: 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a8011970 con 0x7fd6b0132170 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 5 monclient: authenticate success, global_id 194303 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 10 monclient: _renew_subs 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b0098cb0 con 0x7fd6b0132170 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8011d30 con 0x7fd6b0132170 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 10 monclient: dump: 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 10 monclient: _renew_subs 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b01189f0 con 0x7fd6b0132170 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a6ffd640 4 set_mon_vals no callback set 2024-09-15T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 0x7fd6b01345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0065f90 con 0x7fd6b0138f40 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.391+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd6a80195c0 con 0x7fd6b0132170 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 0x7fd6b01345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6a4ff9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd69c0387c0 0x7fd69c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient: get_auth_request con 0x7fd6b0138f40 auth_method 0 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd6a804c550 con 0x7fd6b0132170 2024-09-15T03:45:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6a7fff640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd69c0387c0 0x7fd69c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6a7fff640 10 monclient: get_auth_request con 0x7fd69c0387c0 auth_method 0 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_done global_id 204193 payload 824 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.392+0000 7fd6b48f9640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6b004bf60 con 0x7fd6b0138f40 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6b48f9640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 msgr2=0x7fd6b01345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6b48f9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 0x7fd6b01345e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00ec30 tx=0x7fd6ac009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00ee50 con 0x7fd6b0138f40 2024-09-15T03:45:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6a4ff9640 10 monclient: dump: 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.394+0000 7fd6a7fff640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd69c0387c0 0x7fd69c03ac70 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd6b01185f0 tx=0x7fd6a0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd6ac00cc60 con 0x7fd6b0138f40 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 msgr2=0x7fd6b01345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 0x7fd6b01345e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00ec30 tx=0x7fd6ac009040 comp rx=0 tx=0).stop 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 msgr2=0x7fd6b01345e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.395+0000 7fd6a4ff9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0138f40 0x7fd6b01345e0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00ec30 tx=0x7fd6ac009040 comp rx=0 tx=0).stop 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.400+0000 7fd6b6b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.400+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 0x7fd6b0134f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.400+0000 7fd6b6b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.400+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0098ee0 con 0x7fd6b0134b20 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 0x7fd6b0134f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient: get_auth_request con 0x7fd6b0134b20 auth_method 0 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 10 monclient(hunting): handle_auth_done global_id 204196 payload 824 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.401+0000 7fd6b48f9640 1 -- 172.21.15.153:0/3441596209 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6b0139310 con 0x7fd6b0134b20 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.403+0000 7fd6b48f9640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 msgr2=0x7fd6b0134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.403+0000 7fd6b48f9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 0x7fd6b0134f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00ec30 tx=0x7fd6ac01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.403+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac013e00 con 0x7fd6b0134b20 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.403+0000 7fd6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.403+0000 7fd6a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.403+0000 7fd6a4ff9640 10 monclient: dump: 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fd6ac01dc50 con 0x7fd6b0134b20 2024-09-15T03:45:55.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 msgr2=0x7fd6b0134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 0x7fd6b0134f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00ec30 tx=0x7fd6ac01b000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 msgr2=0x7fd6b0134f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6a4ff9640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6b0134b20 0x7fd6b0134f20 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00ec30 tx=0x7fd6ac01b000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:55.417 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 178, 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:55.418 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.419 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:55.421 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:55.422 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:55.423 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:55.424 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.425 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.426 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.427 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:55.428 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:55.429 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.430 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:55.431 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd69c0387c0 msgr2=0x7fd69c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd69c0387c0 0x7fd69c03ac70 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd6b01185f0 tx=0x7fd6a0032000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 10 monclient: shutdown 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 msgr2=0x7fd6b0138a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 --2- 172.21.15.153:0/3441596209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b0132170 0x7fd6b0138a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800c650 tx=0x7fd6a8008040 comp rx=0 tx=0).stop 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b50fa640 1 -- 172.21.15.153:0/3441596209 reap_dead start 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b50fa640 1 -- 172.21.15.153:0/3441596209 reap_dead start 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 shutdown_connections 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 >> 172.21.15.153:0/3441596209 conn(0x7fd6b00a1e30 msgr2=0x7fd6b0096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 shutdown_connections 2024-09-15T03:45:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.404+0000 7fd6b6b5d640 1 -- 172.21.15.153:0/3441596209 wait complete. 2024-09-15T03:45:55.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:55.434 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:45:55.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:45:55.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.526+0000 7fa7aa0ef640 10 monclient: get_monmap_and_config 2024-09-15T03:45:55.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.526+0000 7fa7aa0ef640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.526+0000 7fa7aa0ef640 10 monclient: monmap: 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:55.527396+0000 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:55.527396+0000 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:55.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:55.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:55.529 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:55.529 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:55.529 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 1 Processor -- start 2024-09-15T03:45:55.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 1 -- start start 2024-09-15T03:45:55.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient: init 2024-09-15T03:45:55.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7a40974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 0x7fa7a409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a409a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a4133070 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a412b210 con 0x7fa7a40974f0 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a40978c0 con 0x7fa7a4133070 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a409ca20 con 0x7fa7a409a8a0 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.527+0000 7fa7aa0ef640 10 monclient(hunting): authenticate will time out at 1753.166942s 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa7a40974f0 0x7fa7a412ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50618/0 (socket says 172.21.15.153:50618) 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a37fe640 1 -- 172.21.15.153:0/1366742796 learned_addr learned my addr 172.21.15.153:0/1366742796 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 0x7fa7a409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): get_auth_request con 0x7fa7a409a8a0 auth_method 0 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_done global_id 194306 payload 274 2024-09-15T03:45:55.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 1 -- 172.21.15.153:0/1366742796 >> v1:172.21.15.181:6789/0 conn(0x7fa7a4133070 legacy=0x7fa7a4135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 1 -- 172.21.15.153:0/1366742796 >> v1:172.21.15.153:6789/0 conn(0x7fa7a40974f0 legacy=0x7fa7a412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 1 monclient: found mon.noname-b 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.528+0000 7fa7a2ffd640 1 -- 172.21.15.153:0/1366742796 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a409b1f0 con 0x7fa7a409a8a0 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a2ffd640 1 -- 172.21.15.153:0/1366742796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 msgr2=0x7fa7a409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a2ffd640 1 --2- 172.21.15.153:0/1366742796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 0x7fa7a409acb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa798009880 tx=0x7fa7980090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe87560cd4552ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 1 -- 172.21.15.153:0/1366742796 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798032440 con 0x7fa7a409a8a0 2024-09-15T03:45:55.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: dump: 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 1 -- 172.21.15.153:0/1366742796 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7980325a0 con 0x7fa7a409a8a0 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 5 monclient: authenticate success, global_id 194306 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 1 -- 172.21.15.153:0/1366742796 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798032940 con 0x7fa7a409a8a0 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 4 set_mon_vals no callback set 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7a27fc640 10 monclient: dump: 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 10 monclient: shutdown 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/1366742796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 msgr2=0x7fa7a409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/1366742796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 0x7fa7a409acb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa798009880 tx=0x7fa7980090f0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/1366742796 shutdown_connections 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/1366742796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a409a8a0 0x7fa7a409acb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.531+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/1366742796 >> 172.21.15.153:0/1366742796 conn(0x7fa7a40672b0 msgr2=0x7fa7a40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.532+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/1366742796 shutdown_connections 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.532+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/1366742796 wait complete. 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.532+0000 7fa7aa0ef640 10 monclient: build_initial_monmap 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.532+0000 7fa7aa0ef640 10 monclient: monmap: 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:55.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.532+0000 7fa7aa0ef640 1 Processor -- start 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 -- start start 2024-09-15T03:45:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient: init 2024-09-15T03:45:55.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:55.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:45:55.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 0x7fa7a413de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a40974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a409a8a0 0x7fa7a4132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a409a8a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a4133070 0x7fa7a406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a4133070 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a406eb30 con 0x7fa7a409a8a0 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4098300 con 0x7fa7a4133070 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4099fe0 con 0x7fa7a40974f0 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7aa0ef640 10 monclient(hunting): authenticate will time out at 1753.172516s 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 0x7fa7a413de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a409a8a0 0x7fa7a4132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 0x7fa7a413de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37700/0 (socket says 172.21.15.153:37700) 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a409a8a0 0x7fa7a4132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47050/0 (socket says 172.21.15.153:47050) 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 -- 172.21.15.153:0/2519572451 learned_addr learned my addr 172.21.15.153:0/2519572451 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): get_auth_request con 0x7fa7a40974f0 auth_method 0 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): get_auth_request con 0x7fa7a409a8a0 auth_method 0 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a3fff640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a4133070 0x7fa7a406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a3fff640 10 monclient(hunting): get_auth_request con 0x7fa7a4133070 auth_method 0 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:55.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient(hunting): handle_auth_done global_id 194309 payload 824 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 -- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a4133070 msgr2=0x7fa7a406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a4133070 0x7fa7a406ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 -- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a409a8a0 msgr2=0x7fa7a4132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a409a8a0 0x7fa7a4132010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 monclient: found mon.noname-c 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.533+0000 7fa7a37fe640 1 -- 172.21.15.153:0/2519572451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798009390 con 0x7fa7a40974f0 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7a37fe640 1 -- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 msgr2=0x7fa7a413de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7a37fe640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 0x7fa7a413de50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa79400ef40 tx=0x7fa794005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79400ce00 con 0x7fa7a40974f0 2024-09-15T03:45:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: dump: 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.550 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 5 monclient: authenticate success, global_id 194309 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7940070f0 con 0x7fa7a40974f0 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7940074b0 con 0x7fa7a40974f0 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 10 monclient: _renew_subs 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a413eca0 con 0x7fa7a40974f0 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa78b7fe640 10 monclient: dump: 2024-09-15T03:45:55.551 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7a1ffb640 4 set_mon_vals no callback set 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 10 monclient: _renew_subs 2024-09-15T03:45:55.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.536+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a412c200 con 0x7fa7a40974f0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa7aa0ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa7aa0ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa7aa0ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa794007920 con 0x7fa7a40974f0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a404bf60 con 0x7fa7a40974f0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa78b7fe640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa790038740 0x7fa79003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa79404f250 con 0x7fa7a40974f0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa7a2ffd640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa790038740 0x7fa79003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.537+0000 7fa7a2ffd640 10 monclient: get_auth_request con 0x7fa790038740 auth_method 0 2024-09-15T03:45:55.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.540+0000 7fa7a2ffd640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa790038740 0x7fa79003abf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa798034000 tx=0x7fa798032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:55.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.541+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa794018360 con 0x7fa7a40974f0 2024-09-15T03:45:55.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.541+0000 7fa78b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:55.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.541+0000 7fa78b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:55.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.758+0000 7fa7aa0ef640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:45:55.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.758+0000 7fa7aa0ef640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:45:55.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.758+0000 7fa7aa0ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:55.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.758+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa7a406e610 con 0x7fa7a40974f0 2024-09-15T03:45:55.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.759+0000 7fa78b7fe640 1 -- 172.21.15.153:0/2519572451 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fa79401f020 con 0x7fa7a40974f0 2024-09-15T03:45:55.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.759+0000 7fa78b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:45:55.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.759+0000 7fa78b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:55.761 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:55.760+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:55.761 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:55.760+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=1,health=12,kv=8,logm=50} crc {auth=3655577910,config=1228700967,health=1569412615,kv=3235807718,logm=3554470024}) 2024-09-15T03:45:55.761 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:55.760+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=1231803550,config=3140514009,health=3441423671,kv=3235807718,logm=499942927}) 2024-09-15T03:45:55.761 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:55.760+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:55.761 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:55.760+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=1,health=12,kv=8,logm=50} crc {auth=3655577910,config=1228700967,health=1569412615,kv=3235807718,logm=3554470024}) 2024-09-15T03:45:55.761 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:55.760+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=7,logm=51} crc {auth=3124609362,config=3140514009,health=652602147,kv=1577269308,logm=4172953653}) 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa790038740 msgr2=0x7fa79003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa790038740 0x7fa79003abf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa798034000 tx=0x7fa798032000 comp rx=0 tx=0).stop 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 10 monclient: shutdown 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 msgr2=0x7fa7a413de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 0x7fa7a413de50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa79400ef40 tx=0x7fa794005480 comp rx=0 tx=0).stop 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 shutdown_connections 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa790038740 0x7fa79003abf0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.761+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a4133070 0x7fa7a406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.762+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a409a8a0 0x7fa7a4132010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.762+0000 7fa7aa0ef640 1 --2- 172.21.15.153:0/2519572451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa7a40974f0 0x7fa7a413de50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.762+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 >> 172.21.15.153:0/2519572451 conn(0x7fa7a40672b0 msgr2=0x7fa7a409d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:55.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.762+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 shutdown_connections 2024-09-15T03:45:55.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:55.762+0000 7fa7aa0ef640 1 -- 172.21.15.153:0/2519572451 wait complete. 2024-09-15T03:45:55.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:45:56.266 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.266 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302193489}) 2024-09-15T03:45:56.267 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882147430}) 2024-09-15T03:45:56.267 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.267 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302193489}) 2024-09-15T03:45:56.267 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733308842}) 2024-09-15T03:45:56.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1262527799}) 2024-09-15T03:45:56.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816187028}) 2024-09-15T03:45:56.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1262527799}) 2024-09-15T03:45:56.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939654013}) 2024-09-15T03:45:56.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226336765}) 2024-09-15T03:45:56.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1164612915}) 2024-09-15T03:45:56.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226336765}) 2024-09-15T03:45:56.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3917053520}) 2024-09-15T03:45:56.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892191727}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750809189}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892191727}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096633852}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733793815}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898878033}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733793815}) 2024-09-15T03:45:56.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403192733}) 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114281478}) 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793405627}) 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114281478}) 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2758159381}) 2024-09-15T03:45:56.535 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213970645}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423915653}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213970645}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912796407}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993684172}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334467748}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993684172}) 2024-09-15T03:45:56.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4121832401}) 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935476108}) 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1978048802}) 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935476108}) 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3812640137}) 2024-09-15T03:45:56.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382811768}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253228802}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382811768}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1351667434}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028564239}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178697952}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028564239}) 2024-09-15T03:45:56.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235249645}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796085824}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322473445}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796085824}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521377295}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2500803729}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690923868}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2500803729}) 2024-09-15T03:45:56.541 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2263615316}) 2024-09-15T03:45:56.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841188650}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101889268}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841188650}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048526047}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680853390}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940999688}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680853390}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027937586}) 2024-09-15T03:45:56.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041870198}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4007273875}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041870198}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=283017225}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544448365}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1042127949}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544448365}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128279628}) 2024-09-15T03:45:56.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277868703}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839578130}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277868703}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658864177}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487590128}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3884455337}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487590128}) 2024-09-15T03:45:56.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=806751494}) 2024-09-15T03:45:56.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3661529270}) 2024-09-15T03:45:56.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3039567086}) 2024-09-15T03:45:56.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3661529270}) 2024-09-15T03:45:56.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750932976}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559092754}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872520301}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559092754}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334651264}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974977811}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1937869794}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974977811}) 2024-09-15T03:45:56.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080985407}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711391809}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640344427}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711391809}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2693356295}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710396831}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=681578530}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710396831}) 2024-09-15T03:45:56.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476351921}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083562489}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2666468663}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083562489}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662815140}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168657738}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792507938}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168657738}) 2024-09-15T03:45:56.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=689899209}) 2024-09-15T03:45:56.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998382236}) 2024-09-15T03:45:56.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940817781}) 2024-09-15T03:45:56.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998382236}) 2024-09-15T03:45:56.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038558879}) 2024-09-15T03:45:56.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.551+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.551+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434008678}) 2024-09-15T03:45:56.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.551+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336102134}) 2024-09-15T03:45:56.560 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.551+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.551+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434008678}) 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.551+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280273985}) 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.552+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.552+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664598606}) 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.552+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934450686}) 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.552+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.552+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664598606}) 2024-09-15T03:45:56.561 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.552+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462088942}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.553+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.553+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686108161}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.553+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1343071202}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.553+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.553+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686108161}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.553+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3254492111}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734098802}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3471453743}) 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.562 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734098802}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618785632}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150024058}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3913439332}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150024058}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610584247}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.555+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.555+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352777512}) 2024-09-15T03:45:56.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.555+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2625363043}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.555+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.555+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352777512}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.555+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24351137}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.556+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.556+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520770584}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.556+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231718029}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.556+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.556+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520770584}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.556+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1740839675}) 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.557+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.557+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063340247}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.557+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376807895}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.557+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.557+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063340247}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.557+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1400525474}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.558+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.558+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732801724}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.558+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1110226013}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.558+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.558+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732801724}) 2024-09-15T03:45:56.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.558+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229044618}) 2024-09-15T03:45:56.589 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.589+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.590 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.589+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980178441}) 2024-09-15T03:45:56.590 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.589+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997905202}) 2024-09-15T03:45:56.590 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.589+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.590 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.589+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980178441}) 2024-09-15T03:45:56.590 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.589+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2420733394}) 2024-09-15T03:45:56.591 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.590+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.591 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.590+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106309263}) 2024-09-15T03:45:56.591 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.590+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2180056939}) 2024-09-15T03:45:56.591 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.590+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.591 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.590+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106309263}) 2024-09-15T03:45:56.591 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.590+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224367689}) 2024-09-15T03:45:56.593 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.593 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830695654}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543776979}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830695654}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684785915}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612236821}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.591+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723153590}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612236821}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571615112}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37850333}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1696632873}) 2024-09-15T03:45:56.594 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.595 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37850333}) 2024-09-15T03:45:56.595 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.592+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601803426}) 2024-09-15T03:45:56.595 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.593+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.596 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.593+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265639708}) 2024-09-15T03:45:56.596 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.593+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179311821}) 2024-09-15T03:45:56.596 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.593+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.596 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.593+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265639708}) 2024-09-15T03:45:56.596 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.593+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830141404}) 2024-09-15T03:45:56.602 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.601+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.602 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.601+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323327825}) 2024-09-15T03:45:56.602 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.601+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=217369360}) 2024-09-15T03:45:56.602 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.601+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.602 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.601+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323327825}) 2024-09-15T03:45:56.602 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.601+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278965371}) 2024-09-15T03:45:56.603 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.602+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.603 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.602+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731417865}) 2024-09-15T03:45:56.603 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.602+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97025895}) 2024-09-15T03:45:56.603 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.602+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.603 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.602+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731417865}) 2024-09-15T03:45:56.603 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.602+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414937704}) 2024-09-15T03:45:56.605 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.603+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.605 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.603+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661007604}) 2024-09-15T03:45:56.605 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.603+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462832214}) 2024-09-15T03:45:56.605 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.603+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.605 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.603+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661007604}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.603+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064598850}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497876749}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=323769605}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497876749}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275620045}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498046625}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319159458}) 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.606 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498046625}) 2024-09-15T03:45:56.607 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.604+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158115690}) 2024-09-15T03:45:56.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.605+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.605+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273205541}) 2024-09-15T03:45:56.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.605+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107507690}) 2024-09-15T03:45:56.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.605+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.609 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.605+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273205541}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.605+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448392297}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658645774}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4092505417}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658645774}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067179721}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242291576}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2459955648}) 2024-09-15T03:45:56.610 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.606+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242291576}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479381085}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2280813574}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1570461980}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2280813574}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.607+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3358624001}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.608+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.608+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971730780}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.608+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3027551922}) 2024-09-15T03:45:56.611 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.608+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.608+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971730780}) 2024-09-15T03:45:56.612 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.608+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2031330218}) 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.609+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.609+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676439360}) 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.609+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1744290432}) 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.609+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.609+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676439360}) 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.609+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192241365}) 2024-09-15T03:45:56.617 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265105890}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933029143}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265105890}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210783180}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574581069}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=152629161}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574581069}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.610+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446400806}) 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.611+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.618 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.611+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569303274}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.611+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3142335883}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.611+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.611+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569303274}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.611+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094781623}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.612+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.612+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638439590}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.612+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936954606}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.612+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.612+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638439590}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.612+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516695205}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499246551}) 2024-09-15T03:45:56.619 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489815396}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499246551}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536011924}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498147066}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007013044}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.613+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498147066}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2413689205}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464684789}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1110120643}) 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.620 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464684789}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.614+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365991553}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.615+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.615+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1178516797}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.615+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616470848}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.615+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.615+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1178516797}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.615+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348759407}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.616+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.616+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421569262}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.616+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689659265}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.616+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.616+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421569262}) 2024-09-15T03:45:56.621 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.616+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4197113962}) 2024-09-15T03:45:56.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.700+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.700+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170377832}) 2024-09-15T03:45:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.700+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385363082}) 2024-09-15T03:45:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.700+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.700+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170377832}) 2024-09-15T03:45:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.700+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904205969}) 2024-09-15T03:45:56.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.704+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.704+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909669376}) 2024-09-15T03:45:56.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.704+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297814688}) 2024-09-15T03:45:56.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.704+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.704+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909669376}) 2024-09-15T03:45:56.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.704+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444228706}) 2024-09-15T03:45:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.715+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.715+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296761352}) 2024-09-15T03:45:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.715+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321505396}) 2024-09-15T03:45:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.715+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.715+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296761352}) 2024-09-15T03:45:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:56.715+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174864344}) 2024-09-15T03:45:56.774 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:45:56.774 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:56.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.874+0000 7f742d8dd640 10 monclient: get_monmap_and_config 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.874+0000 7f742d8dd640 10 monclient: build_initial_monmap 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.874+0000 7f742d8dd640 10 monclient: monmap: 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:56.874996+0000 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:56.874996+0000 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:56.879 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.874+0000 7f742d8dd640 1 Processor -- start 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.874+0000 7f742d8dd640 1 -- start start 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.874+0000 7f742d8dd640 10 monclient: init 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 0x7f7428131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f74281318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f742809ea50 0x7f742809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f742809ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:56.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7428068d90 0x7f74280691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f7428068d90 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7428131cc0 con 0x7f7428068d90 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742809fbf0 con 0x7f742809ea50 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742809fe30 con 0x7f74281318f0 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f742d8dd640 10 monclient(hunting): authenticate will time out at 1754.514410s 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 0x7f7428131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7428068d90 0x7f74280691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 0x7f7428131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37702/0 (socket says 172.21.15.153:37702) 2024-09-15T03:45:56.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 -- 172.21.15.153:0/246639279 learned_addr learned my addr 172.21.15.153:0/246639279 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): get_auth_request con 0x7f74281318f0 auth_method 0 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): get_auth_request con 0x7f7428068d90 auth_method 0 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74267fc640 1 --2- 172.21.15.153:0/246639279 >> v2:172.21.15.181:3300/0 conn(0x7f742809ea50 0x7f742809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74267fc640 10 monclient(hunting): get_auth_request con 0x7f742809ea50 auth_method 0 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient(hunting): handle_auth_done global_id 194312 payload 274 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 -- 172.21.15.153:0/246639279 >> v2:172.21.15.181:3300/0 conn(0x7f742809ea50 msgr2=0x7f742809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 --2- 172.21.15.153:0/246639279 >> v2:172.21.15.181:3300/0 conn(0x7f742809ea50 0x7f742809ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 -- 172.21.15.153:0/246639279 >> v2:172.21.15.153:3300/0 conn(0x7f7428068d90 msgr2=0x7f74280691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 --2- 172.21.15.153:0/246639279 >> v2:172.21.15.153:3300/0 conn(0x7f7428068d90 0x7f74280691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 monclient: found mon.noname-b 2024-09-15T03:45:56.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:56.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:45:56.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f7426ffd640 1 -- 172.21.15.153:0/246639279 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280a1210 con 0x7f74281318f0 2024-09-15T03:45:56.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.875+0000 7f74277fe640 1 --2- 172.21.15.153:0/246639279 >> v2:172.21.15.153:3300/0 conn(0x7f7428068d90 0x7f74280691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:56.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7426ffd640 1 -- 172.21.15.153:0/246639279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 msgr2=0x7f7428131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:56.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7426ffd640 1 --2- 172.21.15.153:0/246639279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 0x7f7428131f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7428068160 tx=0x7f7418009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b08991436d32167 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:56.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 1 -- 172.21.15.153:0/246639279 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418007610 con 0x7f74281318f0 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: dump: 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:56.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: _finish_auth 0 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 1 -- 172.21.15.153:0/246639279 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741802d050 con 0x7f74281318f0 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 1 -- 172.21.15.153:0/246639279 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74180079a0 con 0x7f74281318f0 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f7425ffb640 10 monclient: dump: 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.891 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f742d8dd640 5 monclient: authenticate success, global_id 194312 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f742d8dd640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.878+0000 7f742d8dd640 4 set_mon_vals no callback set 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 10 monclient: shutdown 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 -- 172.21.15.153:0/246639279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 msgr2=0x7f7428131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:56.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 --2- 172.21.15.153:0/246639279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 0x7f7428131f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7428068160 tx=0x7f7418009840 comp rx=0 tx=0).stop 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 -- 172.21.15.153:0/246639279 shutdown_connections 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 --2- 172.21.15.153:0/246639279 >> v2:172.21.15.153:3300/0 conn(0x7f7428068d90 0x7f74280691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 --2- 172.21.15.153:0/246639279 >> v2:172.21.15.181:3300/0 conn(0x7f742809ea50 0x7f742809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 --2- 172.21.15.153:0/246639279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74281318f0 0x7f7428131f30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 -- 172.21.15.153:0/246639279 >> 172.21.15.153:0/246639279 conn(0x7f7428067110 msgr2=0x7f7428067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 -- 172.21.15.153:0/246639279 shutdown_connections 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.879+0000 7f742d8dd640 1 -- 172.21.15.153:0/246639279 wait complete. 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient: build_initial_monmap 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient: monmap: 2024-09-15T03:45:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 Processor -- start 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 -- start start 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient: init 2024-09-15T03:45:56.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 0x7f742809adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f7428068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 0x7f742806ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f742809ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74281318f0 0x7f742806b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f74281318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742806ea90 con 0x7f742809ea50 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742806d610 con 0x7f74281318f0 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742809bf10 con 0x7f7428068d90 2024-09-15T03:45:56.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.880+0000 7f742d8dd640 10 monclient(hunting): authenticate will time out at 1754.519969s 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 0x7f742806ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 0x7f742809adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 0x7f742809adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37706/0 (socket says 172.21.15.153:37706) 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 0x7f742806ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47082/0 (socket says 172.21.15.153:47082) 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 -- 172.21.15.153:0/4230262253 learned_addr learned my addr 172.21.15.153:0/4230262253 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): get_auth_request con 0x7f7428068d90 auth_method 0 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): get_auth_request con 0x7f742809ea50 auth_method 0 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:56.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74277fe640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74281318f0 0x7f742806b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74277fe640 10 monclient(hunting): get_auth_request con 0x7f74281318f0 auth_method 0 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:56.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient(hunting): handle_auth_done global_id 194315 payload 824 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 -- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74281318f0 msgr2=0x7f742806b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74281318f0 0x7f742806b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 -- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 msgr2=0x7f742806ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 0x7f742806ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 monclient: found mon.noname-c 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f7426ffd640 1 -- 172.21.15.153:0/4230262253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420009390 con 0x7f7428068d90 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74277fe640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74281318f0 0x7f742806b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.881+0000 7f74267fc640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 0x7f742806ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f7426ffd640 1 -- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 msgr2=0x7f742809adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:56.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f7426ffd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 0x7f742809adc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f741802abb0 tx=0x7f7418036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741803c840 con 0x7f7428068d90 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: dump: 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:56.899 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: _finish_auth 0 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741803c9a0 con 0x7f7428068d90 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 5 monclient: authenticate success, global_id 194315 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 10 monclient: _renew_subs 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741803cd60 con 0x7f7428068d90 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7418009ae0 con 0x7f7428068d90 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f740effd640 10 monclient: dump: 2024-09-15T03:45:56.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f74257fa640 4 set_mon_vals no callback set 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 10 monclient: _renew_subs 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:56.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.883+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742806d330 con 0x7f7428068d90 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.884+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7418031610 con 0x7f7428068d90 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.884+0000 7f742d8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.884+0000 7f742d8dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.884+0000 7f742d8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.884+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742804bf60 con 0x7f7428068d90 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.885+0000 7f740effd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7414038740 0x7f741403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.885+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f741806cd40 con 0x7f7428068d90 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.885+0000 7f74267fc640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7414038740 0x7f741403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.885+0000 7f74267fc640 10 monclient: get_auth_request con 0x7f7414038740 auth_method 0 2024-09-15T03:45:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.888+0000 7f74267fc640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7414038740 0x7f741403abf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f741c007fa0 tx=0x7f741c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:56.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.889+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f74180318c0 con 0x7f7428068d90 2024-09-15T03:45:56.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.889+0000 7f740effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:56.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:56.889+0000 7f740effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:57.038 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.037+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.039 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.037+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319977047}) 2024-09-15T03:45:57.039 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.037+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2425174423}) 2024-09-15T03:45:57.039 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.037+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.039 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.037+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319977047}) 2024-09-15T03:45:57.039 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.037+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=188698320}) 2024-09-15T03:45:57.042 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.041+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.042 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.041+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429430099}) 2024-09-15T03:45:57.042 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.041+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53260869}) 2024-09-15T03:45:57.042 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.041+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.042 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.041+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429430099}) 2024-09-15T03:45:57.042 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.041+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4014549778}) 2024-09-15T03:45:57.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.072+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.072+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562095074}) 2024-09-15T03:45:57.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.072+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1474894671}) 2024-09-15T03:45:57.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.072+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.072+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562095074}) 2024-09-15T03:45:57.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.072+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3452489886}) 2024-09-15T03:45:57.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860870882}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2977101480}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860870882}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697278880}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069937454}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649145334}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069937454}) 2024-09-15T03:45:57.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.073+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204565579}) 2024-09-15T03:45:57.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.074+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.074+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385254541}) 2024-09-15T03:45:57.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.074+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269949445}) 2024-09-15T03:45:57.077 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.074+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.074+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385254541}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.074+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706178401}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.075+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.075+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1348417831}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.075+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1201945572}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.075+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.075+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1348417831}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.075+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3057904020}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.076+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.076+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707335002}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.076+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2364458349}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.076+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.076+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707335002}) 2024-09-15T03:45:57.078 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.076+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3346964370}) 2024-09-15T03:45:57.081 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.077+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.077+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967666043}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.077+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314389674}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.077+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.077+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967666043}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.077+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609035369}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.078+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.078+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649822699}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.078+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=608375720}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.078+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.078+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649822699}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.078+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266607415}) 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.079+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.082 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.079+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253365332}) 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.079+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741663561}) 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.079+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.079+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253365332}) 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.079+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154785702}) 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.080+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.080+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088684389}) 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.080+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572439346}) 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.080+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.083 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.080+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088684389}) 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.080+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252237153}) 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455098203}) 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768821131}) 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455098203}) 2024-09-15T03:45:57.084 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040832165}) 2024-09-15T03:45:57.090 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.090 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617289687}) 2024-09-15T03:45:57.090 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1632181992}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617289687}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.081+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673756269}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.082+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.082+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648856226}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.082+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622300575}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.082+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.082+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648856226}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.082+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940923726}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.083+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.083+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829321362}) 2024-09-15T03:45:57.091 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.083+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452166428}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.083+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.083+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829321362}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.083+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=78060062}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.084+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.084+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118511763}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.084+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109768895}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.084+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.084+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118511763}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.084+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129784131}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.085+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.085+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=634662784}) 2024-09-15T03:45:57.092 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.085+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2959035312}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.085+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.085+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=634662784}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.085+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564002143}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.086+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.086+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018990150}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.086+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1106221478}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.086+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.086+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018990150}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.086+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187297316}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.087+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.087+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359001115}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.087+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3152408998}) 2024-09-15T03:45:57.093 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.087+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.087+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359001115}) 2024-09-15T03:45:57.094 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.087+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=527795611}) 2024-09-15T03:45:57.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.116+0000 7f742d8dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:57.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.116+0000 7f742d8dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:57.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.116+0000 7f742d8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:45:57.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.116+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f742809bbd0 con 0x7f7428068d90 2024-09-15T03:45:57.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f740effd640 1 -- 172.21.15.153:0/4230262253 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f7418040020 con 0x7f7428068d90 2024-09-15T03:45:57.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f740effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:57.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f740effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:57.118 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:57.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750951414}) 2024-09-15T03:45:57.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986126929}) 2024-09-15T03:45:57.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750951414}) 2024-09-15T03:45:57.119 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.117+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903203095}) 2024-09-15T03:45:57.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7414038740 msgr2=0x7f741403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7414038740 0x7f741403abf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f741c007fa0 tx=0x7f741c008040 comp rx=0 tx=0).stop 2024-09-15T03:45:57.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 10 monclient: shutdown 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 msgr2=0x7f742809adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 0x7f742809adc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f741802abb0 tx=0x7f7418036000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 shutdown_connections 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7414038740 0x7f741403abf0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74281318f0 0x7f742806b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742809ea50 0x7f742806ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 --2- 172.21.15.153:0/4230262253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7428068d90 0x7f742809adc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 >> 172.21.15.153:0/4230262253 conn(0x7f7428067110 msgr2=0x7f742809d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 shutdown_connections 2024-09-15T03:45:57.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.119+0000 7f742d8dd640 1 -- 172.21.15.153:0/4230262253 wait complete. 2024-09-15T03:45:57.131 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:45:57.131 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:45:57.131 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:45:57.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:45:57.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.224+0000 7f8988c20640 10 monclient: get_monmap_and_config 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.224+0000 7f8988c20640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.224+0000 7f8988c20640 10 monclient: monmap: 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:57.225673+0000 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:57.225673+0000 2024-09-15T03:45:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:57.227 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.225+0000 7f8988c20640 1 Processor -- start 2024-09-15T03:45:57.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.225+0000 7f8988c20640 1 -- start start 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.225+0000 7f8988c20640 10 monclient: init 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8984131980 0x7f8984131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): picked mon.noname-a con 0x7f8984131980 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f898409fda0 0x7f8984132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): picked mon.noname-c con 0x7f898409fda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f898409e9c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898409f510 con 0x7f898409e9c0 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840a1180 con 0x7f8984131980 2024-09-15T03:45:57.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840a1460 con 0x7f898409fda0 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8988c20640 10 monclient(hunting): authenticate will time out at 1754.865263s 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8984131980 0x7f8984131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8984131980 0x7f8984131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47094/0 (socket says 172.21.15.153:47094) 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 -- 172.21.15.153:0/3929369604 learned_addr learned my addr 172.21.15.153:0/3929369604 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): get_auth_request con 0x7f8984131980 auth_method 0 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f8981d9b640 1 --2- 172.21.15.153:0/3929369604 >> v2:172.21.15.181:3300/0 conn(0x7f898409fda0 0x7f8984132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient(hunting): handle_auth_done global_id 214125 payload 274 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 -- 172.21.15.153:0/3929369604 >> v2:172.21.15.181:3300/0 conn(0x7f898409fda0 msgr2=0x7f8984132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 --2- 172.21.15.153:0/3929369604 >> v2:172.21.15.181:3300/0 conn(0x7f898409fda0 0x7f8984132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 -- 172.21.15.153:0/3929369604 >> v1:172.21.15.181:6789/0 conn(0x7f898409e9c0 legacy=0x7f898409ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.226+0000 7f898259c640 1 -- 172.21.15.153:0/3929369604 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984139cd0 con 0x7f8984131980 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898259c640 1 -- 172.21.15.153:0/3929369604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984131980 msgr2=0x7f8984131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898259c640 1 --2- 172.21.15.153:0/3929369604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984131980 0x7f8984131d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8984064050 tx=0x7f89740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bc6c5131e77caf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 1 -- 172.21.15.153:0/3929369604 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974032440 con 0x7f8984131980 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: dump: 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.242 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 1 -- 172.21.15.153:0/3929369604 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89740325a0 con 0x7f8984131980 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f8988c20640 5 monclient: authenticate success, global_id 214125 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f8988c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f8988c20640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 1 -- 172.21.15.153:0/3929369604 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974032940 con 0x7f8984131980 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f8988c20640 4 set_mon_vals no callback set 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.229+0000 7f898159a640 10 monclient: dump: 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 10 monclient: shutdown 2024-09-15T03:45:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 -- 172.21.15.153:0/3929369604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984131980 msgr2=0x7f8984131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 --2- 172.21.15.153:0/3929369604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984131980 0x7f8984131d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8984064050 tx=0x7f89740091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 -- 172.21.15.153:0/3929369604 shutdown_connections 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 --2- 172.21.15.153:0/3929369604 >> v2:172.21.15.181:3300/0 conn(0x7f898409fda0 0x7f8984132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 --2- 172.21.15.153:0/3929369604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984131980 0x7f8984131d50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 -- 172.21.15.153:0/3929369604 >> 172.21.15.153:0/3929369604 conn(0x7f8984067bd0 msgr2=0x7f89840a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 -- 172.21.15.153:0/3929369604 shutdown_connections 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 1 -- 172.21.15.153:0/3929369604 wait complete. 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.230+0000 7f8988c20640 10 monclient: monmap: 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 Processor -- start 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 -- start start 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient: init 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:45:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 0x7f8984138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): picked mon.noname-a con 0x7f898409e9c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f898409fda0 0x7f8984140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): picked mon.noname-c con 0x7f898409fda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8984131980 0x7f8984141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): picked mon.noname-b con 0x7f8984131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898409bf10 con 0x7f898409e9c0 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898409e450 con 0x7f8984131980 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8984137470 con 0x7f898409fda0 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8988c20640 10 monclient(hunting): authenticate will time out at 1754.870809s 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 0x7f8984138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 0x7f8984138180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47102/0 (socket says 172.21.15.153:47102) 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 1 -- 172.21.15.153:0/2860640596 learned_addr learned my addr 172.21.15.153:0/2860640596 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): get_auth_request con 0x7f898409e9c0 auth_method 0 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8981d9b640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f898409fda0 0x7f8984140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8981d9b640 10 monclient(hunting): get_auth_request con 0x7f898409fda0 auth_method 0 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f8981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.231+0000 7f898259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8982d9d640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8984131980 0x7f8984141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8982d9d640 10 monclient(hunting): get_auth_request con 0x7f8984131980 auth_method 0 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8982d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8982d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f8982d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 10 monclient(hunting): handle_auth_done global_id 214128 payload 824 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f898409fda0 msgr2=0x7f8984140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f898409fda0 0x7f8984140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8984131980 msgr2=0x7f8984141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8984131980 0x7f8984141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.232+0000 7f898259c640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8974009450 con 0x7f898409e9c0 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f898259c640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 msgr2=0x7f8984138180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f898259c640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 0x7f8984138180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8974009940 tx=0x7f8974032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974031a90 con 0x7f898409e9c0 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: dump: 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8974031c10 con 0x7f898409e9c0 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 5 monclient: authenticate success, global_id 214128 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 10 monclient: _renew_subs 2024-09-15T03:45:57.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89841386c0 con 0x7f898409e9c0 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897403b400 con 0x7f898409e9c0 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f89727fc640 10 monclient: dump: 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.253 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 10 monclient: _renew_subs 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8980d99640 4 set_mon_vals no callback set 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.234+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89841362a0 con 0x7f898409e9c0 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8988c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8988c20640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 0x7f8984130800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8974039040 con 0x7f898409e9c0 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8984065e00 con 0x7f89841419c0 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 0x7f8984130800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient: get_auth_request con 0x7f89841419c0 auth_method 0 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f89727fc640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8964038740 0x7f896403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f897406c540 con 0x7f898409e9c0 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 10 monclient(hunting): handle_auth_done global_id 214131 payload 824 2024-09-15T03:45:57.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.235+0000 7f8981d9b640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8984137240 con 0x7f89841419c0 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.236+0000 7f8982d9d640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8964038740 0x7f896403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.236+0000 7f8982d9d640 10 monclient: get_auth_request con 0x7f8964038740 auth_method 0 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f8981d9b640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 msgr2=0x7f8984130800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f8981d9b640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 0x7f8984130800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f897800afd0 tx=0x7f8978008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897800db30 con 0x7f89841419c0 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 10 monclient: dump: 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f897800c040 con 0x7f89841419c0 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 msgr2=0x7f8984130800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 0x7f8984130800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f897800afd0 tx=0x7f8978008040 comp rx=0 tx=0).stop 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 msgr2=0x7f8984130800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f89727fc640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89841419c0 0x7f8984130800 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f897800afd0 tx=0x7f8978008040 comp rx=0 tx=0).stop 2024-09-15T03:45:57.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.238+0000 7f8982d9d640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8964038740 0x7f896403abf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8984136030 tx=0x7f897c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8988c20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8988c20640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 0x7f8984131110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8988c20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898409f510 con 0x7f8984130d40 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 0x7f8984131110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient: get_auth_request con 0x7f8984130d40 auth_method 0 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 10 monclient(hunting): handle_auth_done global_id 214134 payload 824 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.244+0000 7f8981d9b640 1 -- 172.21.15.153:0/2860640596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f89841376b0 con 0x7f8984130d40 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.246+0000 7f8981d9b640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 msgr2=0x7f8984131110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.246+0000 7f8981d9b640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 0x7f8984131110 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8978000c00 tx=0x7f897801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.246+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897801b900 con 0x7f8984130d40 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.246+0000 7f89727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.246+0000 7f89727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.246+0000 7f89727fc640 10 monclient: dump: 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.261 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f897801ba60 con 0x7f8984130d40 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 msgr2=0x7f8984131110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 0x7f8984131110 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8978000c00 tx=0x7f897801c000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 msgr2=0x7f8984131110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f89727fc640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8984130d40 0x7f8984131110 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8978000c00 tx=0x7f897801c000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:57.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 178, 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:57.263 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.264 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.265 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:57.266 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.268 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.269 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:57.271 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.273 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:57.274 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8964038740 msgr2=0x7f896403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8964038740 0x7f896403abf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8984136030 tx=0x7f897c009040 comp rx=0 tx=0).stop 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 10 monclient: shutdown 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 msgr2=0x7f8984138180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 --2- 172.21.15.153:0/2860640596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f898409e9c0 0x7f8984138180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8974009940 tx=0x7f8974032000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8982d9d640 1 -- 172.21.15.153:0/2860640596 reap_dead start 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8982d9d640 1 -- 172.21.15.153:0/2860640596 reap_dead start 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 shutdown_connections 2024-09-15T03:45:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 >> 172.21.15.153:0/2860640596 conn(0x7f8984067bd0 msgr2=0x7f89840a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 shutdown_connections 2024-09-15T03:45:57.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.247+0000 7f8988c20640 1 -- 172.21.15.153:0/2860640596 wait complete. 2024-09-15T03:45:57.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:57.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:45:57.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:45:57.278 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.265+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.266+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=479043782,mdsmap=1531151339,mgr=2291727002,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2568943349,mon_config_key=4117641549,monmap=244636553,osd_metadata=2249018645}) 2024-09-15T03:45:57.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.266+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=3} crc {logm=3195889777,mdsmap=1531151339,mgr=4081032428,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2318863453,mon_config_key=1321035168,monmap=244636553,osd_metadata=2704803675}) 2024-09-15T03:45:57.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.266+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.266+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=479043782,mdsmap=1531151339,mgr=2291727002,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2568943349,mon_config_key=4117641549,monmap=244636553,osd_metadata=2249018645}) 2024-09-15T03:45:57.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.266+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=67,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=3} crc {logm=3247330846,mdsmap=3585761405,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2568943349,mon_config_key=4117641549,monmap=244636553,osd_metadata=2570910634}) 2024-09-15T03:45:57.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.371+0000 7ff6e8b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.371+0000 7ff6e8b9d640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.371+0000 7ff6e8b9d640 10 monclient: monmap: 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:57.372707+0000 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:57.372707+0000 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.373 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:57.374 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:57.374 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.372+0000 7ff6e8b9d640 1 Processor -- start 2024-09-15T03:45:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.372+0000 7ff6e8b9d640 1 -- start start 2024-09-15T03:45:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.372+0000 7ff6e8b9d640 10 monclient: init 2024-09-15T03:45:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff6e4131980 0x7ff6e4131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e4131980 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6e409fda0 0x7ff6e4132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e409fda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6e409e9c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e409f510 con 0x7ff6e409e9c0 2024-09-15T03:45:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40a1180 con 0x7ff6e4131980 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40a1460 con 0x7ff6e409fda0 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e8b9d640 10 monclient(hunting): authenticate will time out at 1755.012147s 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff6e4131980 0x7ff6e4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff6e4131980 0x7ff6e4131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47146/0 (socket says 172.21.15.153:47146) 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 -- 172.21.15.153:0/3480260605 learned_addr learned my addr 172.21.15.153:0/3480260605 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e2d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff6e409e9c0 0x7ff6e409ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50626/0 (socket says 172.21.15.153:50626) 2024-09-15T03:45:57.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): get_auth_request con 0x7ff6e4131980 auth_method 0 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient(hunting): handle_auth_done global_id 214137 payload 274 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 -- 172.21.15.153:0/3480260605 >> v2:172.21.15.181:3300/0 conn(0x7ff6e409fda0 msgr2=0x7ff6e4132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 --2- 172.21.15.153:0/3480260605 >> v2:172.21.15.181:3300/0 conn(0x7ff6e409fda0 0x7ff6e4132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 -- 172.21.15.153:0/3480260605 >> v1:172.21.15.153:6789/0 conn(0x7ff6e409e9c0 legacy=0x7ff6e409ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.373+0000 7ff6e259c640 1 -- 172.21.15.153:0/3480260605 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4139cd0 con 0x7ff6e4131980 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e259c640 1 -- 172.21.15.153:0/3480260605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e4131980 msgr2=0x7ff6e4131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e259c640 1 --2- 172.21.15.153:0/3480260605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e4131980 0x7ff6e4131d50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4064050 tx=0x7ff6d40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0435eaaccf3345 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 1 -- 172.21.15.153:0/3480260605 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4032440 con 0x7ff6e4131980 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: dump: 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e8b9d640 5 monclient: authenticate success, global_id 214137 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e8b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 1 -- 172.21.15.153:0/3480260605 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d40325a0 con 0x7ff6e4131980 2024-09-15T03:45:57.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 1 -- 172.21.15.153:0/3480260605 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4032940 con 0x7ff6e4131980 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e8b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e8b9d640 4 set_mon_vals no callback set 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.376+0000 7ff6e1d9b640 10 monclient: dump: 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 10 monclient: shutdown 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/3480260605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e4131980 msgr2=0x7ff6e4131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/3480260605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e4131980 0x7ff6e4131d50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4064050 tx=0x7ff6d40091d0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/3480260605 shutdown_connections 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/3480260605 >> v2:172.21.15.181:3300/0 conn(0x7ff6e409fda0 0x7ff6e4132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/3480260605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e4131980 0x7ff6e4131d50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/3480260605 >> 172.21.15.153:0/3480260605 conn(0x7ff6e4067bd0 msgr2=0x7ff6e40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/3480260605 shutdown_connections 2024-09-15T03:45:57.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/3480260605 wait complete. 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.377+0000 7ff6e8b9d640 10 monclient: monmap: 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 Processor -- start 2024-09-15T03:45:57.389 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.390 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {osd_metadata=205594370,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3129237019}) 2024-09-15T03:45:57.390 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {osd_metadata=205594370,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=608541313}) 2024-09-15T03:45:57.390 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:45:57.390 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {osd_metadata=205594370,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3129237019}) 2024-09-15T03:45:57.390 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:45:57.378+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=55} crc {osd_metadata=205594370,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=1617945301}) 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 -- start start 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient: init 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 0x7ff6e41410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e409e9c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e409fda0 0x7ff6e41415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e409fda0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:57.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4131980 0x7ff6e4141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e4131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e409bf80 con 0x7ff6e409e9c0 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e409e440 con 0x7ff6e409fda0 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4135560 con 0x7ff6e4131980 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e8b9d640 10 monclient(hunting): authenticate will time out at 1755.017717s 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 0x7ff6e41410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 0x7ff6e41410b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47152/0 (socket says 172.21.15.153:47152) 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 1 -- 172.21.15.153:0/2166265818 learned_addr learned my addr 172.21.15.153:0/2166265818 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): get_auth_request con 0x7ff6e409e9c0 auth_method 0 2024-09-15T03:45:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4131980 0x7ff6e4141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): get_auth_request con 0x7ff6e4131980 auth_method 0 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.378+0000 7ff6e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 10 monclient(hunting): handle_auth_done global_id 214140 payload 824 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4131980 msgr2=0x7ff6e4141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6d9d9b640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e409fda0 0x7ff6e41415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4131980 0x7ff6e4141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e409fda0 msgr2=0x7ff6e41415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e409fda0 0x7ff6e41415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.379+0000 7ff6e259c640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009450 con 0x7ff6e409e9c0 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e259c640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 msgr2=0x7ff6e41410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e259c640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 0x7ff6e41410b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009210 tx=0x7ff6d4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4031990 con 0x7ff6e409e9c0 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: dump: 2024-09-15T03:45:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d4031af0 con 0x7ff6e409e9c0 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 5 monclient: authenticate success, global_id 214140 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 10 monclient: _renew_subs 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e41364c0 con 0x7ff6e409e9c0 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d403b400 con 0x7ff6e409e9c0 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6daffd640 10 monclient: dump: 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 10 monclient: _renew_subs 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e41374b0 con 0x7ff6e409e9c0 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.381+0000 7ff6e159a640 4 set_mon_vals no callback set 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6e8b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 0x7ff6e4130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4136940 con 0x7ff6e4132ee0 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 0x7ff6e4130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient: get_auth_request con 0x7ff6e4132ee0 auth_method 0 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff6d4039040 con 0x7ff6e409e9c0 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6daffd640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff6c4038740 0x7ff6c403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_done global_id 194324 payload 824 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff6d406c790 con 0x7ff6e409e9c0 2024-09-15T03:45:57.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.382+0000 7ff6d9d9b640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6e4136b80 con 0x7ff6e4132ee0 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.383+0000 7ff6e2d9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff6c4038740 0x7ff6c403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.383+0000 7ff6e2d9d640 10 monclient: get_auth_request con 0x7ff6c4038740 auth_method 0 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.384+0000 7ff6d9d9b640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 msgr2=0x7ff6e4130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.384+0000 7ff6d9d9b640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 0x7ff6e4130fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff6e406a530 tx=0x7ff6cc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc010790 con 0x7ff6e4132ee0 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 10 monclient: dump: 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.409 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff6cc0108f0 con 0x7ff6e4132ee0 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 msgr2=0x7ff6e4130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 0x7ff6e4130fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff6e406a530 tx=0x7ff6cc008040 comp rx=0 tx=0).stop 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 msgr2=0x7ff6e4130fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6daffd640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e4132ee0 0x7ff6e4130fc0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff6e406a530 tx=0x7ff6cc008040 comp rx=0 tx=0).stop 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.385+0000 7ff6e2d9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff6c4038740 0x7ff6c403abf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0079b0 tx=0x7ff6dc004e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6e8b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 0x7ff6e4029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6e8b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4065e00 con 0x7ff6e40298c0 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 0x7ff6e4029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient: get_auth_request con 0x7ff6e40298c0 auth_method 0 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 10 monclient(hunting): handle_auth_done global_id 194327 payload 824 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.391+0000 7ff6d9d9b640 1 -- 172.21.15.153:0/2166265818 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6e409f510 con 0x7ff6e40298c0 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.393+0000 7ff6d9d9b640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 msgr2=0x7ff6e4029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.393+0000 7ff6d9d9b640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 0x7ff6e4029cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc015000 tx=0x7ff6cc018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.393+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc01a400 con 0x7ff6e40298c0 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.393+0000 7ff6daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.393+0000 7ff6daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.393+0000 7ff6daffd640 10 monclient: dump: 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7ff6cc01a560 con 0x7ff6e40298c0 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 msgr2=0x7ff6e4029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 0x7ff6e4029cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc015000 tx=0x7ff6cc018000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 msgr2=0x7ff6e4029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6daffd640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e40298c0 0x7ff6e4029cc0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc015000 tx=0x7ff6cc018000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 178, 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.414 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.416 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:57.417 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.418 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:57.419 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.420 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.421 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:57.422 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.425 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.427 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:57.427 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff6c4038740 msgr2=0x7ff6c403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff6c4038740 0x7ff6c403abf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0079b0 tx=0x7ff6dc004e20 comp rx=0 tx=0).stop 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 10 monclient: shutdown 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 msgr2=0x7ff6e41410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 --2- 172.21.15.153:0/2166265818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e409e9c0 0x7ff6e41410b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009210 tx=0x7ff6d4032000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e2d9d640 1 -- 172.21.15.153:0/2166265818 reap_dead start 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e2d9d640 1 -- 172.21.15.153:0/2166265818 reap_dead start 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 shutdown_connections 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 >> 172.21.15.153:0/2166265818 conn(0x7ff6e4067bd0 msgr2=0x7ff6e4137fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 shutdown_connections 2024-09-15T03:45:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.394+0000 7ff6e8b9d640 1 -- 172.21.15.153:0/2166265818 wait complete. 2024-09-15T03:45:57.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:57.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:45:57.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:45:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.523+0000 7f185039d640 10 monclient: get_monmap_and_config 2024-09-15T03:45:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.523+0000 7f185039d640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.523+0000 7f185039d640 10 monclient: monmap: 2024-09-15T03:45:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:57.524150+0000 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:57.524150+0000 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:57.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 Processor -- start 2024-09-15T03:45:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 -- start start 2024-09-15T03:45:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient: init 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1848096c60 addr v1:172.21.15.153:6789/0 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1848097030 0x7f1848097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): picked mon.noname-c con 0x7f1848097030 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1848133b40 0x7f1848135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): picked mon.noname-a con 0x7f1848133b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184809d2e0 con 0x7f1848096c60 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184809d6b0 con 0x7f1848133b40 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18480992f0 con 0x7f1848097030 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f185039d640 10 monclient(hunting): authenticate will time out at 1755.163792s 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1848133b40 0x7f1848135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e139640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1848096c60 0x7f184809a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50636/0 (socket says 172.21.15.153:50636) 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e139640 1 -- 172.21.15.153:0/631624491 learned_addr learned my addr 172.21.15.153:0/631624491 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 10 monclient(hunting): get_auth_request con 0x7f1848133b40 auth_method 0 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.524+0000 7f184e93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 10 monclient(hunting): handle_auth_done global_id 214143 payload 274 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 1 -- 172.21.15.153:0/631624491 >> v2:172.21.15.181:3300/0 conn(0x7f1848097030 msgr2=0x7f1848097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 1 --2- 172.21.15.153:0/631624491 >> v2:172.21.15.181:3300/0 conn(0x7f1848097030 0x7f1848097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 1 -- 172.21.15.153:0/631624491 >> v1:172.21.15.153:6789/0 conn(0x7f1848096c60 legacy=0x7f184809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.525+0000 7f184e93a640 1 -- 172.21.15.153:0/631624491 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18480995d0 con 0x7f1848133b40 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.527+0000 7f184e93a640 1 -- 172.21.15.153:0/631624491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848133b40 msgr2=0x7f1848135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.527+0000 7f184e93a640 1 --2- 172.21.15.153:0/631624491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848133b40 0x7f1848135f20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f184809ecd0 tx=0x7f1840009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42849ee317a56172 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 1 -- 172.21.15.153:0/631624491 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840007840 con 0x7f1848133b40 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: dump: 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 1 -- 172.21.15.153:0/631624491 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184002d050 con 0x7f1848133b40 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 5 monclient: authenticate success, global_id 214143 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 1 -- 172.21.15.153:0/631624491 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840007bd0 con 0x7f1848133b40 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:57.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 4 set_mon_vals no callback set 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f184d137640 10 monclient: dump: 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 10 monclient: shutdown 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 -- 172.21.15.153:0/631624491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848133b40 msgr2=0x7f1848135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 --2- 172.21.15.153:0/631624491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848133b40 0x7f1848135f20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f184809ecd0 tx=0x7f1840009780 comp rx=0 tx=0).stop 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 -- 172.21.15.153:0/631624491 shutdown_connections 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 --2- 172.21.15.153:0/631624491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848133b40 0x7f1848135f20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 --2- 172.21.15.153:0/631624491 >> v2:172.21.15.181:3300/0 conn(0x7f1848097030 0x7f1848097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 -- 172.21.15.153:0/631624491 >> 172.21.15.153:0/631624491 conn(0x7f18480672b0 msgr2=0x7f18480676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 -- 172.21.15.153:0/631624491 shutdown_connections 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.528+0000 7f185039d640 1 -- 172.21.15.153:0/631624491 wait complete. 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 10 monclient: monmap: 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.541 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 1 Processor -- start 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 1 -- start start 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 10 monclient: init 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.529+0000 7f185039d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1848096c60 0x7f1848130210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): picked mon.noname-b con 0x7f1848096c60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 0x7f1848130750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): picked mon.noname-a con 0x7f1848097030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1848133b40 0x7f1848130c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): picked mon.noname-c con 0x7f1848133b40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848135720 con 0x7f1848097030 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184806f5e0 con 0x7f1848096c60 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18480698a0 con 0x7f1848133b40 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f185039d640 10 monclient(hunting): authenticate will time out at 1755.169075s 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 0x7f1848130750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1848133b40 0x7f1848130c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 0x7f1848130750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47168/0 (socket says 172.21.15.153:47168) 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 -- 172.21.15.153:0/3209837225 learned_addr learned my addr 172.21.15.153:0/3209837225 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): get_auth_request con 0x7f1848097030 auth_method 0 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): get_auth_request con 0x7f1848133b40 auth_method 0 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1848096c60 0x7f1848130210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): get_auth_request con 0x7f1848096c60 auth_method 0 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient(hunting): handle_auth_done global_id 214146 payload 824 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1848133b40 msgr2=0x7f1848130c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1848133b40 0x7f1848130c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1848096c60 msgr2=0x7f1848130210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1848096c60 0x7f1848130210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184d938640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1840009a20 con 0x7f1848097030 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.530+0000 7f184e93a640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1848133b40 0x7f1848130c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f184d938640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 msgr2=0x7f1848130750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f184d938640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 0x7f1848130750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f184400db70 tx=0x7f184400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184400ee40 con 0x7f1848097030 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: dump: 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.547 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f185039d640 5 monclient: authenticate success, global_id 214146 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1844017470 con 0x7f1848097030 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844017830 con 0x7f1848097030 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f185039d640 10 monclient: _renew_subs 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f185039d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1848069b80 con 0x7f1848097030 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f184c936640 4 set_mon_vals no callback set 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.532+0000 7f183e7fc640 10 monclient: dump: 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.549 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 10 monclient: _renew_subs 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848132970 con 0x7f1848097030 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 0x7f184809b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848065f20 con 0x7f18481311d0 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.533+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1844014070 con 0x7f1848097030 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f183e7fc640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f182c038790 0x7f182c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 0x7f184809b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f184404e4c0 con 0x7f1848097030 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient: get_auth_request con 0x7f18481311d0 auth_method 0 2024-09-15T03:45:57.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e93a640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f182c038790 0x7f182c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e93a640 10 monclient: get_auth_request con 0x7f182c038790 auth_method 0 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 10 monclient(hunting): handle_auth_done global_id 204214 payload 824 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.534+0000 7f184e139640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1848131e30 con 0x7f18481311d0 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.536+0000 7f184e139640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 msgr2=0x7f184809b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.536+0000 7f184e139640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 0x7f184809b810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f183800e710 tx=0x7f1838007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.536+0000 7f184e93a640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f182c038790 0x7f182c03ac40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f18480637c0 tx=0x7f1840037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838004b50 con 0x7f18481311d0 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 10 monclient: dump: 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1838004cb0 con 0x7f18481311d0 2024-09-15T03:45:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 msgr2=0x7f184809b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 0x7f184809b810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f183800e710 tx=0x7f1838007f00 comp rx=0 tx=0).stop 2024-09-15T03:45:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 msgr2=0x7f184809b810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:57.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.537+0000 7f183e7fc640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18481311d0 0x7f184809b810 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f183800e710 tx=0x7f1838007f00 comp rx=0 tx=0).stop 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f185039d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f185039d640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 0x7f184809c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f185039d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848069d10 con 0x7f184809bd50 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 0x7f184809c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient: get_auth_request con 0x7f184809bd50 auth_method 0 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 10 monclient(hunting): handle_auth_done global_id 204217 payload 824 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.543+0000 7f184e139640 1 -- 172.21.15.153:0/3209837225 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f184804bf60 con 0x7f184809bd50 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f184e139640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 msgr2=0x7f184809c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f184e139640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 0x7f184809c150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f183800e710 tx=0x7f1838019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838015d30 con 0x7f184809bd50 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 10 monclient: dump: 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.556 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f1838015e90 con 0x7f184809bd50 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 msgr2=0x7f184809c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 0x7f184809c150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f183800e710 tx=0x7f1838019000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 msgr2=0x7f184809c150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:45:57.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f183e7fc640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f184809bd50 0x7f184809c150 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f183800e710 tx=0x7f1838019000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 178, 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:45:57.558 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.560 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:45:57.561 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:45:57.562 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:45:57.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.568 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.569 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:45:57.570 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f182c038790 msgr2=0x7f182c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f182c038790 0x7f182c03ac40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f18480637c0 tx=0x7f1840037000 comp rx=0 tx=0).stop 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 10 monclient: shutdown 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 msgr2=0x7f1848130750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 1 --2- 172.21.15.153:0/3209837225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1848097030 0x7f1848130750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f184400db70 tx=0x7f184400b460 comp rx=0 tx=0).stop 2024-09-15T03:45:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f184e93a640 1 -- 172.21.15.153:0/3209837225 reap_dead start 2024-09-15T03:45:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f184e93a640 1 -- 172.21.15.153:0/3209837225 reap_dead start 2024-09-15T03:45:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 shutdown_connections 2024-09-15T03:45:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.546+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 >> 172.21.15.153:0/3209837225 conn(0x7f18480672b0 msgr2=0x7f1848135be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.547+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 shutdown_connections 2024-09-15T03:45:57.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.547+0000 7f185039d640 1 -- 172.21.15.153:0/3209837225 wait complete. 2024-09-15T03:45:57.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:45:57.574 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:45:57.574 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:45:57.574 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:45:57.574 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:45:57.574 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:45:57.574 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:45:57.586 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:45:57.586 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:45:57.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.719+0000 7fb7cda17640 10 monclient: get_monmap_and_config 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.719+0000 7fb7cda17640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.719+0000 7fb7cda17640 10 monclient: monmap: 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:45:57.720040+0000 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:45:57.720040+0000 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:45:57.721 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:45:57.721 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:45:57.721 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:45:57.721 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:45:57.721 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.719+0000 7fb7cda17640 1 Processor -- start 2024-09-15T03:45:57.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.719+0000 7fb7cda17640 1 -- start start 2024-09-15T03:45:57.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.719+0000 7fb7cda17640 10 monclient: init 2024-09-15T03:45:57.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7c809caa0 0x7fb7c809ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c809caa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb7c809a540 0x7fb7c806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c809a540 addr v2:172.21.15.181:3300/0 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c806f490 0x7fb7c809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c806f490 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c809ed10 con 0x7fb7c809caa0 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c809d3b0 con 0x7fb7c809a540 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c809abd0 con 0x7fb7c806f490 2024-09-15T03:45:57.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7cda17640 10 monclient(hunting): authenticate will time out at 1755.359529s 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c77fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c806f490 msgr2=0x7fb7c809d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c806f490 0x7fb7c809d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7c809caa0 0x7fb7c809ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb7c809caa0 0x7fb7c809ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47204/0 (socket says 172.21.15.153:47204) 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/578168312 learned_addr learned my addr 172.21.15.153:0/578168312 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): get_auth_request con 0x7fb7c809caa0 auth_method 0 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_done global_id 214149 payload 274 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c806f490 msgr2=0x7fb7c809d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c806f490 0x7fb7c809d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/578168312 >> v2:172.21.15.181:3300/0 conn(0x7fb7c809a540 msgr2=0x7fb7c806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/578168312 >> v2:172.21.15.181:3300/0 conn(0x7fb7c809a540 0x7fb7c806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.720+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/578168312 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c809b920 con 0x7fb7c809caa0 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.723+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c809caa0 msgr2=0x7fb7c809ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.723+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c809caa0 0x7fb7c809ce70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb7c809f550 tx=0x7fb7b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3a30b665c13ee6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.723+0000 7fb7c5ffb640 1 -- 172.21.15.153:0/578168312 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8007610 con 0x7fb7c809caa0 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.723+0000 7fb7c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.723+0000 7fb7c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.723+0000 7fb7c5ffb640 10 monclient: dump: 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 1 -- 172.21.15.153:0/578168312 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b802d050 con 0x7fb7c809caa0 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 5 monclient: authenticate success, global_id 214149 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 10 monclient: get_monmap_and_config success 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 4 set_mon_vals no callback set 2024-09-15T03:45:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 1 -- 172.21.15.153:0/578168312 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80079e0 con 0x7fb7c809caa0 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7c5ffb640 10 monclient: dump: 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 10 monclient: shutdown 2024-09-15T03:45:57.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 -- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c809caa0 msgr2=0x7fb7c809ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 --2- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c809caa0 0x7fb7c809ce70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb7c809f550 tx=0x7fb7b8009840 comp rx=0 tx=0).stop 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 -- 172.21.15.153:0/578168312 shutdown_connections 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 --2- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c806f490 0x7fb7c809d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 --2- 172.21.15.153:0/578168312 >> v2:172.21.15.181:3300/0 conn(0x7fb7c809a540 0x7fb7c806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 --2- 172.21.15.153:0/578168312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c809caa0 0x7fb7c809ce70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 -- 172.21.15.153:0/578168312 >> 172.21.15.153:0/578168312 conn(0x7fb7c8067cd0 msgr2=0x7fb7c80a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 -- 172.21.15.153:0/578168312 shutdown_connections 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.724+0000 7fb7cda17640 1 -- 172.21.15.153:0/578168312 wait complete. 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.725+0000 7fb7cda17640 10 monclient: build_initial_monmap 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.725+0000 7fb7cda17640 10 monclient: monmap: 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:45:57.735 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.725+0000 7fb7cda17640 1 Processor -- start 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.725+0000 7fb7cda17640 1 -- start start 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.725+0000 7fb7cda17640 10 monclient: init 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.725+0000 7fb7cda17640 10 monclient: _reopen_session rank -1 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:45:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c806f490 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7c809a540 0x7fb7c8131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c809a540 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c809caa0 0x7fb7c8068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c809caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c806a490 con 0x7fb7c806f490 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c806dcd0 con 0x7fb7c809a540 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8141ed0 con 0x7fb7c809caa0 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): _renew_subs 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c77fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c809caa0 msgr2=0x7fb7c8068900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:45:57.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7cda17640 10 monclient(hunting): authenticate will time out at 1755.365129s 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c809caa0 0x7fb7c8068900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47206/0 (socket says 172.21.15.153:47206) 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/2395872933 learned_addr learned my addr 172.21.15.153:0/2395872933 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): get_auth_request con 0x7fb7c806f490 auth_method 0 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c67fc640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7c809a540 0x7fb7c8131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c67fc640 10 monclient(hunting): get_auth_request con 0x7fb7c809a540 auth_method 0 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient(hunting): handle_auth_done global_id 214152 payload 824 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c809caa0 msgr2=0x7fb7c8068900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c809caa0 0x7fb7c8068900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7c809a540 msgr2=0x7fb7c8131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7c809a540 0x7fb7c8131d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/2395872933 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8009ae0 con 0x7fb7c806f490 2024-09-15T03:45:57.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.726+0000 7fb7c67fc640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7c809a540 0x7fb7c8131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7c6ffd640 1 -- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 msgr2=0x7fb7c8140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009ea0 tx=0x7fb7b8036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8031a70 con 0x7fb7c806f490 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: dump: 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: _finish_auth 0 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7cda17640 5 monclient: authenticate success, global_id 214152 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b8031bd0 con 0x7fb7c806f490 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b803c420 con 0x7fb7c806f490 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7aeffd640 10 monclient: dump: 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:45:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7cda17640 10 monclient: _renew_subs 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7cda17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c81211b0 con 0x7fb7c806f490 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.728+0000 7fb7c57fa640 4 set_mon_vals no callback set 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.729+0000 7fb7cda17640 10 monclient: _renew_subs 2024-09-15T03:45:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.729+0000 7fb7cda17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.729+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c811ff70 con 0x7fb7c806f490 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.729+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb7b803c7d0 con 0x7fb7c806f490 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7cda17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7cda17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7cda17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c804bf60 con 0x7fb7c806f490 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7aeffd640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7b40386f0 0x7fb7b403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb7b806d760 con 0x7fb7c806f490 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7c67fc640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7b40386f0 0x7fb7b403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:45:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.730+0000 7fb7c67fc640 10 monclient: get_auth_request con 0x7fb7b40386f0 auth_method 0 2024-09-15T03:45:57.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.733+0000 7fb7c67fc640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7b40386f0 0x7fb7b403aba0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb7c809e700 tx=0x7fb7bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:45:57.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:57.734+0000 7fb7c6ffd640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009ea0 tx=0x7fb7b8036000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fb7b803ce00 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:45:58.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.563+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb7b803ce00 con 0x7fb7c806f490 2024-09-15T03:45:58.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.563+0000 7fb7aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:45:58.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.563+0000 7fb7aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:45:58.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7cda17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:45:58.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7cda17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:58.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7cda17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:45:58.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb7c80638d0 con 0x7fb7c806f490 2024-09-15T03:45:58.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7aeffd640 1 -- 172.21.15.153:0/2395872933 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb7b804a080 con 0x7fb7c806f490 2024-09-15T03:45:58.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:45:58.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.785+0000 7fb7aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:45:58.787 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.787+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7b40386f0 msgr2=0x7fb7b403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.787+0000 7fb7cda17640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7b40386f0 0x7fb7b403aba0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb7c809e700 tx=0x7fb7bc008040 comp rx=0 tx=0).stop 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.787+0000 7fb7cda17640 10 monclient: shutdown 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.787+0000 7fb7cda17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.787+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 msgr2=0x7fb7c8140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009ea0 tx=0x7fb7b8036000 comp rx=0 tx=0).stop 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 shutdown_connections 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb7b40386f0 0x7fb7b403aba0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7c809caa0 0x7fb7c8068900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7c809a540 0x7fb7c8131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 --2- 172.21.15.153:0/2395872933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7c806f490 0x7fb7c8140ff0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:45:58.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 >> 172.21.15.153:0/2395872933 conn(0x7fb7c8067cd0 msgr2=0x7fb7c8098a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:45:58.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 shutdown_connections 2024-09-15T03:45:58.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:45:58.788+0000 7fb7cda17640 1 -- 172.21.15.153:0/2395872933 wait complete. 2024-09-15T03:46:01.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.904+0000 7f4be6257640 10 monclient: get_monmap_and_config 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.904+0000 7f4be6257640 10 monclient: build_initial_monmap 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.904+0000 7f4be6257640 10 monclient: monmap: 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:01.904951+0000 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:01.904951+0000 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:01.906 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:01.906 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:01.906 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:01.906 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 1 Processor -- start 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 1 -- start start 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient: init 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4be01318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f4be009ea50 0x7f4be009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient(hunting): picked mon.noname-c con 0x7f4be009ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4be0136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be0131cc0 con 0x7f4be0136c20 2024-09-15T03:46:01.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be009fbf0 con 0x7f4be01318f0 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be009fe30 con 0x7f4be009ea50 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4be6257640 10 monclient(hunting): authenticate will time out at 1759.544724s 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4bdffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4be0136c20 0x7f4be0139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50652/0 (socket says 172.21.15.153:50652) 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.905+0000 7f4bdffff640 1 -- 172.21.15.153:0/2051942343 learned_addr learned my addr 172.21.15.153:0/2051942343 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 1 --2- 172.21.15.153:0/2051942343 >> v2:172.21.15.181:3300/0 conn(0x7f4be009ea50 0x7f4be009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): get_auth_request con 0x7f4be009ea50 auth_method 0 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:01.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient(hunting): handle_auth_done global_id 204223 payload 274 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 1 -- 172.21.15.153:0/2051942343 >> v1:172.21.15.181:6789/0 conn(0x7f4be01318f0 legacy=0x7f4be0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 1 -- 172.21.15.153:0/2051942343 >> v1:172.21.15.153:6789/0 conn(0x7f4be0136c20 legacy=0x7f4be0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 1 monclient: found mon.noname-c 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:01.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.906+0000 7f4bdeffd640 1 -- 172.21.15.153:0/2051942343 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00a1210 con 0x7f4be009ea50 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bdeffd640 1 -- 172.21.15.153:0/2051942343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 msgr2=0x7f4be009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bdeffd640 1 --2- 172.21.15.153:0/2051942343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be009ee20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009850 tx=0x7f4bd40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf500611e0405c58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 1 -- 172.21.15.153:0/2051942343 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4032440 con 0x7f4be009ea50 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: dump: 2024-09-15T03:46:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 1 -- 172.21.15.153:0/2051942343 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd40325a0 con 0x7f4be009ea50 2024-09-15T03:46:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 1 -- 172.21.15.153:0/2051942343 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4032940 con 0x7f4be009ea50 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4be6257640 5 monclient: authenticate success, global_id 204223 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4be6257640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4be6257640 4 set_mon_vals no callback set 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.909+0000 7f4bde7fc640 10 monclient: dump: 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 10 monclient: shutdown 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 -- 172.21.15.153:0/2051942343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 msgr2=0x7f4be009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 --2- 172.21.15.153:0/2051942343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be009ee20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009850 tx=0x7f4bd40090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 -- 172.21.15.153:0/2051942343 shutdown_connections 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 --2- 172.21.15.153:0/2051942343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be009ee20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 -- 172.21.15.153:0/2051942343 >> 172.21.15.153:0/2051942343 conn(0x7f4be0067110 msgr2=0x7f4be0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 -- 172.21.15.153:0/2051942343 shutdown_connections 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 1 -- 172.21.15.153:0/2051942343 wait complete. 2024-09-15T03:46:01.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 10 monclient: build_initial_monmap 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.910+0000 7f4be6257640 10 monclient: monmap: 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 Processor -- start 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 -- start start 2024-09-15T03:46:01.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient: init 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be0130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): picked mon.noname-b con 0x7f4be009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be01318f0 0x7f4be0099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): picked mon.noname-c con 0x7f4be01318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 0x7f4be0099e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): picked mon.noname-a con 0x7f4be0136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be009e000 con 0x7f4be0136c20 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be011cd30 con 0x7f4be009ea50 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be009bf10 con 0x7f4be01318f0 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4be6257640 10 monclient(hunting): authenticate will time out at 1759.550863s 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdeffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be01318f0 msgr2=0x7f4be0099920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be01318f0 0x7f4be0099920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 0x7f4be0099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 0x7f4be0099e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47218/0 (socket says 172.21.15.153:47218) 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 1 -- 172.21.15.153:0/2041016661 learned_addr learned my addr 172.21.15.153:0/2041016661 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 10 monclient(hunting): get_auth_request con 0x7f4be0136c20 auth_method 0 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.911+0000 7f4bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be0130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4be009ea50 auth_method 0 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 10 monclient(hunting): handle_auth_done global_id 214155 payload 824 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 1 -- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be01318f0 msgr2=0x7f4be0099920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be01318f0 0x7f4be0099920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 1 -- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 msgr2=0x7f4be0130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:01.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be0130940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:01.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 10 monclient: _finish_hunting 0 2024-09-15T03:46:01.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 1 monclient: found mon.noname-a 2024-09-15T03:46:01.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:01.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:01.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.912+0000 7f4bdffff640 1 -- 172.21.15.153:0/2041016661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd4009390 con 0x7f4be0136c20 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bdffff640 1 -- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 msgr2=0x7f4be0099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bdffff640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 0x7f4be0099e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800ccc0 tx=0x7f4bd8005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800ce80 con 0x7f4be0136c20 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bc77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bc77fe640 10 monclient: dump: 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:01.924 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bc77fe640 10 monclient: _finish_auth 0 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4bc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4be6257640 5 monclient: authenticate success, global_id 214155 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4be6257640 10 monclient: _renew_subs 2024-09-15T03:46:01.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4be6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.914+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be011ca50 con 0x7f4be0136c20 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 10 monclient: _renew_subs 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd800aa60 con 0x7f4be0136c20 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be0141d90 con 0x7f4be0136c20 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800ae20 con 0x7f4be0136c20 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 10 monclient: dump: 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bddffb640 4 set_mon_vals no callback set 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4bd8007300 con 0x7f4be0136c20 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.915+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be004bf60 con 0x7f4be0136c20 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.916+0000 7f4bc77fe640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bcc0386f0 0x7f4bcc03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:01.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.916+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f4bd804c690 con 0x7f4be0136c20 2024-09-15T03:46:01.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.916+0000 7f4bdf7fe640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bcc0386f0 0x7f4bcc03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:01.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.916+0000 7f4bdf7fe640 10 monclient: get_auth_request con 0x7f4bcc0386f0 auth_method 0 2024-09-15T03:46:01.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.919+0000 7f4bdf7fe640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bcc0386f0 0x7f4bcc03aba0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0007fd0 tx=0x7f4bd0007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:01.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.920+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4bd801b400 con 0x7f4be0136c20 2024-09-15T03:46:01.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.920+0000 7f4bc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:01.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:01.920+0000 7f4bc77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:02.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4be6257640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:02.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4be6257640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4be6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4be00638d0 con 0x7f4be0136c20 2024-09-15T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4bc77fe640 1 -- 172.21.15.153:0/2041016661 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4bd8010300 con 0x7f4be0136c20 2024-09-15T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4bc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.138+0000 7f4bc77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:02.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bcc0386f0 msgr2=0x7f4bcc03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:02.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bcc0386f0 0x7f4bcc03aba0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0007fd0 tx=0x7f4bd0007e10 comp rx=0 tx=0).stop 2024-09-15T03:46:02.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 10 monclient: shutdown 2024-09-15T03:46:02.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 msgr2=0x7f4be0099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 0x7f4be0099e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800ccc0 tx=0x7f4bd8005370 comp rx=0 tx=0).stop 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 shutdown_connections 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4bcc0386f0 0x7f4bcc03aba0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4be0136c20 0x7f4be0099e60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4be01318f0 0x7f4be0099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 --2- 172.21.15.153:0/2041016661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be009ea50 0x7f4be0130940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 >> 172.21.15.153:0/2041016661 conn(0x7f4be0067110 msgr2=0x7f4be00a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 shutdown_connections 2024-09-15T03:46:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:02.140+0000 7f4be6257640 1 -- 172.21.15.153:0/2041016661 wait complete. 2024-09-15T03:46:05.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.256+0000 7f50bf680640 10 monclient: get_monmap_and_config 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.256+0000 7f50bf680640 10 monclient: build_initial_monmap 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.256+0000 7f50bf680640 10 monclient: monmap: 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:05.256965+0000 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:05.256965+0000 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:05.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:05.258 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:05.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.256+0000 7f50bf680640 1 Processor -- start 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 1 -- start start 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient: init 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f50b8131ab0 0x7f50b8131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): picked mon.noname-c con 0x7f50b8131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50b809e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f50b8136bb0 0x7f50b8138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): picked mon.noname-a con 0x7f50b8136bb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b809faf0 con 0x7f50b809e910 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:05.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b809fd30 con 0x7f50b8136bb0 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b80a1110 con 0x7f50b8131ab0 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bf680640 10 monclient(hunting): authenticate will time out at 1762.896712s 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f50b8136bb0 0x7f50b8138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f50b8136bb0 0x7f50b8138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39176/0 (socket says 172.21.15.153:39176) 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 learned_addr learned my addr 172.21.15.153:0/3884018721 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request con 0x7f50b8136bb0 auth_method 0 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bcc1b640 1 --1- 172.21.15.153:0/3884018721 >> v1:172.21.15.153:6789/0 conn(0x7f50b809e910 0x7f50b809ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:51966/0 (socket says 172.21.15.153:51966) 2024-09-15T03:46:05.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3884018721 >> v2:172.21.15.181:3300/0 conn(0x7f50b8131ab0 0x7f50b8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f50b8131ab0 auth_method 0 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.257+0000 7f50bd41c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_done global_id 214158 payload 274 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> v2:172.21.15.181:3300/0 conn(0x7f50b8131ab0 msgr2=0x7f50b8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> v2:172.21.15.181:3300/0 conn(0x7f50b8131ab0 0x7f50b8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> v1:172.21.15.153:6789/0 conn(0x7f50b809e910 legacy=0x7f50b809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 10 monclient: _finish_hunting 0 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 1 monclient: found mon.noname-a 2024-09-15T03:46:05.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:05.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:05.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b80a13f0 con 0x7f50b8136bb0 2024-09-15T03:46:05.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.258+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3884018721 >> v2:172.21.15.181:3300/0 conn(0x7f50b8131ab0 0x7f50b8131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:46:05.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.260+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 msgr2=0x7f50b8138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.260+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 0x7f50b8138f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009f20 tx=0x7f50b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4775980b54cce413 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 msgr2=0x7f50b8138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 msgr2=0x7f50b8138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 0x7f50b8138f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009f20 tx=0x7f50b0009840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 0x7f50b8138f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009f20 tx=0x7f50b0009840 comp rx=0 tx=0).stop 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 msgr2=0x7f50b8138f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 0x7f50b8138f90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient: _add_conns ranks=[1,2] 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50a40034c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 0x7f50a4009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient(hunting): picked mon.noname-b con 0x7f50a40073e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:05.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 1 -- 172.21.15.153:0/3884018721 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a400a180 con 0x7f50a40034c0 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 1 -- 172.21.15.153:0/3884018721 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a400b380 con 0x7f50a40073e0 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50abfff640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 msgr2=0x7f50a4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.261+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 0x7f50a4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:05.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.263+0000 7f50bd41c640 1 -- 172.21.15.153:0/3884018721 >> v1:172.21.15.181:6789/0 conn(0x7f50a40034c0 legacy=0x7f50a4005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:05.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.462+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 msgr2=0x7f50a4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:05.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.462+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 0x7f50a4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:05.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.863+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 msgr2=0x7f50a4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:05.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:05.863+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 0x7f50a4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:06.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:06.664+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 msgr2=0x7f50a4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:06.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:06.664+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 0x7f50a4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): tick 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 monclient(hunting): continuing hunt 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 msgr2=0x7f50a4009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a40073e0 0x7f50a4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 -- 172.21.15.153:0/3884018721 >> v1:172.21.15.181:6789/0 conn(0x7f50a40034c0 legacy=0x7f50a4005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 reap_dead start 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50b809e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:08.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c006db0 0x7f509c009230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): picked mon.noname-b con 0x7f509c006db0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 --2- 172.21.15.153:0/3884018721 >> v2:172.21.15.181:3300/0 conn(0x7f509c009930 0x7f509c00bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): picked mon.noname-c con 0x7f509c009930 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 -- 172.21.15.153:0/3884018721 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a400b600 con 0x7f50b809e910 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 -- 172.21.15.153:0/3884018721 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a400b380 con 0x7f509c009930 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 1 -- 172.21.15.153:0/3884018721 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c00d770 con 0x7f509c006db0 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50aaffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c006db0 msgr2=0x7f509c009230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c006db0 0x7f509c009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> v2:172.21.15.181:3300/0 conn(0x7f509c009930 0x7f509c00bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request con 0x7f509c009930 auth_method 0 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_done global_id 214158 payload 293 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c006db0 msgr2=0x7f509c009230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c006db0 0x7f509c009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> v1:172.21.15.153:6789/0 conn(0x7f50b809e910 legacy=0x7f509c005650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient: _finish_hunting 0 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 monclient: found mon.noname-c 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2024-09-15T03:46:08.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:08.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.258+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c00d930 con 0x7f509c009930 2024-09-15T03:46:08.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.260+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f509c00bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:08.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.260+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f509c00bde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50b809bdb0 tx=0x7f50b0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434d690f9124989d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:08.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 1 -- 172.21.15.153:0/3884018721 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b002aea0 con 0x7f509c009930 2024-09-15T03:46:08.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: dump: 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:08.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: _finish_auth 0 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 1 -- 172.21.15.153:0/3884018721 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b002d050 con 0x7f509c009930 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 1 -- 172.21.15.153:0/3884018721 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0007e00 con 0x7f509c009930 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50bf680640 5 monclient: authenticate success, global_id 214158 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50bf680640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50bf680640 4 set_mon_vals no callback set 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.261+0000 7f50abfff640 10 monclient: dump: 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:08.268 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 10 monclient: shutdown 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 -- 172.21.15.153:0/3884018721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f509c00bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f509c00bde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50b809bdb0 tx=0x7f50b0036000 comp rx=0 tx=0).stop 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 -- 172.21.15.153:0/3884018721 shutdown_connections 2024-09-15T03:46:08.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f509c00bde0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c006db0 0x7f509c009230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 --2- 172.21.15.153:0/3884018721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50b8136bb0 0x7f50b8138f90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 -- 172.21.15.153:0/3884018721 >> 172.21.15.153:0/3884018721 conn(0x7f50b8067d30 msgr2=0x7f50b80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 -- 172.21.15.153:0/3884018721 shutdown_connections 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 1 -- 172.21.15.153:0/3884018721 wait complete. 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 10 monclient: build_initial_monmap 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.262+0000 7f50bf680640 10 monclient: monmap: 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:08.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.263+0000 7f50bf680640 1 Processor -- start 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.263+0000 7f50bf680640 1 -- start start 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.263+0000 7f50bf680640 10 monclient: init 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:46:08.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): picked mon.noname-a con 0x7f509c006db0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): picked mon.noname-b con 0x7f509c009930 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f50b8141790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): picked mon.noname-c con 0x7f50b809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b811ccf0 con 0x7f509c006db0 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b809bde0 con 0x7f509c009930 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80a3d90 con 0x7f50b809e910 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bf680640 10 monclient(hunting): authenticate will time out at 1765.903266s 2024-09-15T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bdc1d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f50b8141790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bdc1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f50b8141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39184/0 (socket says 172.21.15.153:39184) 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 learned_addr learned my addr 172.21.15.153:0/3921154609 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request con 0x7f509c009930 auth_method 0 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:08.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.264+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:08.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f50b8141790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:08.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f50b8141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:08.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request con 0x7f509c009930 auth_method 0 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:08.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:08.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:08.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:08.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.465+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:08.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f50b8141790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f50b8141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request con 0x7f509c009930 auth_method 0 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:08.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:08.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:08.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:08.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:08.866+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f50b8141790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f50b8141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request con 0x7f509c009930 auth_method 0 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bcc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bcc1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.667+0000 7f50bcc1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.668+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.668+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.668+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:09.668+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:46:11.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): tick 2024-09-15T03:46:11.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:46:11.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:46:11.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f50b8141790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f50b8141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 msgr2=0x7f50b8141250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f509c009930 0x7f50b8141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f50b811d930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f50b811d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f5098003700 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f50b809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:11.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f509c006db0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b811ccf0 con 0x7f509c006db0 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b809bde0 con 0x7f5098003700 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80a3d90 con 0x7f50b809e910 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50a3fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f5098008170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:11.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f5098003700 auth_method 0 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:11.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:11.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:11.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:11.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:11.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.264+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:11.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f5098008170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:11.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f5098003700 auth_method 0 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:11.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.465+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:11.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f5098008170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:11.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 10 monclient(hunting): get_auth_request con 0x7f5098003700 auth_method 0 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:11.866+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bcc1b640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f5098008170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bcc1b640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request con 0x7f509c006db0 auth_method 0 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient(hunting): handle_auth_done global_id 214161 payload 824 2024-09-15T03:46:12.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 msgr2=0x7f5098008170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 msgr2=0x7f5098005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient: _finish_hunting 0 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 monclient: found mon.noname-a 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:12.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.667+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b811c400 con 0x7f509c006db0 2024-09-15T03:46:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bdc1d640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bdc1d640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f50a4003430 tx=0x7f50b003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0007610 con 0x7f509c006db0 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: dump: 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:12.673 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: _finish_auth 0 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0046430 con 0x7f509c006db0 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 5 monclient: authenticate success, global_id 214161 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 10 monclient: _renew_subs 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b809aff0 con 0x7f509c006db0 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00467b0 con 0x7f509c006db0 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50a97fa640 10 monclient: dump: 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:12.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50ab7fe640 4 set_mon_vals no callback set 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 10 monclient: _renew_subs 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.670+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b811db50 con 0x7f509c006db0 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f50b0003040 con 0x7f509c006db0 2024-09-15T03:46:12.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50a97fa640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f509c039480 0x7f509c03b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f50b006e910 con 0x7f509c006db0 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50bf680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50bf680640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50bf680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b811db50 con 0x7f509c006db0 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.671+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f509c039480 0x7f509c03b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.672+0000 7f50bd41c640 10 monclient: get_auth_request con 0x7f509c039480 auth_method 0 2024-09-15T03:46:12.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.674+0000 7f50bd41c640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f509c039480 0x7f509c03b930 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f50ac006ef0 tx=0x7f50ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:12.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.676+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f50b0036020 con 0x7f509c006db0 2024-09-15T03:46:12.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.676+0000 7f50a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:12.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.676+0000 7f50a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.895+0000 7f50bf680640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.896+0000 7f50bf680640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.896+0000 7f50bf680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.896+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f50b811db50 con 0x7f509c006db0 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.896+0000 7f50a97fa640 1 -- 172.21.15.153:0/3921154609 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f50b0040030 con 0x7f509c006db0 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.896+0000 7f50a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:12.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.896+0000 7f50a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:12.898 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f509c039480 msgr2=0x7f509c03b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f509c039480 0x7f509c03b930 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f50ac006ef0 tx=0x7f50ac008040 comp rx=0 tx=0).stop 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 10 monclient: shutdown 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 msgr2=0x7f509800a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f50a4003430 tx=0x7f50b003c000 comp rx=0 tx=0).stop 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 shutdown_connections 2024-09-15T03:46:12.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f509c039480 0x7f509c03b930 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:12.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c006db0 0x7f509800a740 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:12.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50b809e910 0x7f5098008170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:12.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 --2- 172.21.15.153:0/3921154609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5098003700 0x7f5098005b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:12.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 >> 172.21.15.153:0/3921154609 conn(0x7f50b8067d30 msgr2=0x7f50b809d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:12.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 shutdown_connections 2024-09-15T03:46:12.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:12.898+0000 7f50bf680640 1 -- 172.21.15.153:0/3921154609 wait complete. 2024-09-15T03:46:12.910 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:46:12.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:46:12.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:46:13.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.004+0000 7f511c114640 10 monclient: get_monmap_and_config 2024-09-15T03:46:13.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.004+0000 7f511c114640 10 monclient: build_initial_monmap 2024-09-15T03:46:13.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.004+0000 7f511c114640 10 monclient: monmap: 2024-09-15T03:46:13.005 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:13.005 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.005 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:13.005138+0000 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:13.005138+0000 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.004+0000 7f511c114640 1 Processor -- start 2024-09-15T03:46:13.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 1 -- start start 2024-09-15T03:46:13.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient: init 2024-09-15T03:46:13.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:13.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f511409c860 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f511409e840 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5114137cc0 0x7f511413a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient(hunting): picked mon.noname-c con 0x7f5114137cc0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511409ec40 con 0x7f511409e840 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511409b2b0 con 0x7f511409c860 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511406a320 con 0x7f5114137cc0 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f511c114640 10 monclient(hunting): authenticate will time out at 1770.644750s 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f51196af640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f511409e840 0x7f511409aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:51984/0 (socket says 172.21.15.153:51984) 2024-09-15T03:46:13.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.005+0000 7f51196af640 1 -- 172.21.15.153:0/764154546 learned_addr learned my addr 172.21.15.153:0/764154546 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 1 --2- 172.21.15.153:0/764154546 >> v2:172.21.15.181:3300/0 conn(0x7f5114137cc0 0x7f511413a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): get_auth_request con 0x7f5114137cc0 auth_method 0 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient(hunting): handle_auth_done global_id 204232 payload 274 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 1 -- 172.21.15.153:0/764154546 >> v1:172.21.15.181:6789/0 conn(0x7f511409c860 legacy=0x7f511409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 1 -- 172.21.15.153:0/764154546 >> v1:172.21.15.153:6789/0 conn(0x7f511409e840 legacy=0x7f511409aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient: _finish_hunting 0 2024-09-15T03:46:13.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 1 monclient: found mon.noname-c 2024-09-15T03:46:13.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:13.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:13.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.006+0000 7f511a6b1640 1 -- 172.21.15.153:0/764154546 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114068a20 con 0x7f5114137cc0 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f511a6b1640 1 -- 172.21.15.153:0/764154546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114137cc0 msgr2=0x7f511413a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f511a6b1640 1 --2- 172.21.15.153:0/764154546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114137cc0 0x7f511413a0a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f510c009e60 tx=0x7f510c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9236b35241fc03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 1 -- 172.21.15.153:0/764154546 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c007630 con 0x7f5114137cc0 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: dump: 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: _finish_auth 0 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 1 -- 172.21.15.153:0/764154546 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c02d050 con 0x7f5114137cc0 2024-09-15T03:46:13.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f511c114640 5 monclient: authenticate success, global_id 204232 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f511c114640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f511c114640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 1 -- 172.21.15.153:0/764154546 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c0079c0 con 0x7f5114137cc0 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f511c114640 4 set_mon_vals no callback set 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:13.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.009+0000 7f5118eae640 10 monclient: dump: 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 10 monclient: shutdown 2024-09-15T03:46:13.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:13.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 -- 172.21.15.153:0/764154546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114137cc0 msgr2=0x7f511413a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 --2- 172.21.15.153:0/764154546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114137cc0 0x7f511413a0a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f510c009e60 tx=0x7f510c009780 comp rx=0 tx=0).stop 2024-09-15T03:46:13.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 -- 172.21.15.153:0/764154546 shutdown_connections 2024-09-15T03:46:13.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 --2- 172.21.15.153:0/764154546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5114137cc0 0x7f511413a0a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 -- 172.21.15.153:0/764154546 >> 172.21.15.153:0/764154546 conn(0x7f51140a2e40 msgr2=0x7f51140a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:13.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 -- 172.21.15.153:0/764154546 shutdown_connections 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 -- 172.21.15.153:0/764154546 wait complete. 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 10 monclient: build_initial_monmap 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 10 monclient: monmap: 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.010+0000 7f511c114640 1 Processor -- start 2024-09-15T03:46:13.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 -- start start 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient: init 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f511409c860 0x7f511406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): picked mon.noname-b con 0x7f511409c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409e840 0x7f511406b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): picked mon.noname-c con 0x7f511409e840 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 0x7f511406bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): picked mon.noname-a con 0x7f5114137cc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51140a2630 con 0x7f5114137cc0 2024-09-15T03:46:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511406eaf0 con 0x7f511409c860 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511406dd50 con 0x7f511409e840 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511c114640 10 monclient(hunting): authenticate will time out at 1770.650432s 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f51196af640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409e840 msgr2=0x7f511406b8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f51196af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409e840 0x7f511406b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:13.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 0x7f511406bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 0x7f511406bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39250/0 (socket says 172.21.15.153:39250) 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 learned_addr learned my addr 172.21.15.153:0/33957250 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): get_auth_request con 0x7f5114137cc0 auth_method 0 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f5119eb0640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f511409c860 0x7f511406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient(hunting): handle_auth_done global_id 214164 payload 824 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409e840 msgr2=0x7f511406b8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f511409e840 0x7f511406b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f511409c860 msgr2=0x7f511406ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f511409c860 0x7f511406ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient: _finish_hunting 0 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 monclient: found mon.noname-a 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c009a20 con 0x7f5114137cc0 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f5119eb0640 10 monclient: get_auth_request con 0x7f511409c860 auth_method 0 2024-09-15T03:46:13.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.011+0000 7f5119eb0640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f511409c860 0x7f511406ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 msgr2=0x7f511406bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511a6b1640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 0x7f511406bdf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f510c0099f0 tx=0x7f510c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c031ac0 con 0x7f5114137cc0 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: dump: 2024-09-15T03:46:13.035 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.036 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: _finish_auth 0 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c040070 con 0x7f5114137cc0 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 5 monclient: authenticate success, global_id 214164 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 10 monclient: _renew_subs 2024-09-15T03:46:13.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511406e030 con 0x7f5114137cc0 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c031e50 con 0x7f5114137cc0 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f5109ffb640 10 monclient: dump: 2024-09-15T03:46:13.038 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.039 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f510bfff640 4 set_mon_vals no callback set 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 10 monclient: _renew_subs 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.014+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51141379e0 con 0x7f5114137cc0 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f510c042030 con 0x7f5114137cc0 2024-09-15T03:46:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f511c114640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f511c114640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 0x7f511406c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f511c114640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114065e00 con 0x7f511406ce60 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5109ffb640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f80386f0 0x7f50f803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f510c06de90 con 0x7f5114137cc0 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 0x7f511406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient: get_auth_request con 0x7f511406ce60 auth_method 0 2024-09-15T03:46:13.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 10 monclient(hunting): handle_auth_done global_id 214167 payload 824 2024-09-15T03:46:13.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f51196af640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f80386f0 0x7f50f803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f5119eb0640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f511404bf60 con 0x7f511406ce60 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.015+0000 7f51196af640 10 monclient: get_auth_request con 0x7f50f80386f0 auth_method 0 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.017+0000 7f5119eb0640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 msgr2=0x7f511406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.017+0000 7f5119eb0640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 0x7f511406c330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f511000d500 tx=0x7f5110009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110007030 con 0x7f511406ce60 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 10 monclient: dump: 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.044 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f51196af640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f80386f0 0x7f50f803aba0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f51140992d0 tx=0x7f5104007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5110007190 con 0x7f511406ce60 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 msgr2=0x7f511406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 0x7f511406c330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f511000d500 tx=0x7f5110009040 comp rx=0 tx=0).stop 2024-09-15T03:46:13.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 msgr2=0x7f511406c330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:13.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.018+0000 7f5109ffb640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511406ce60 0x7f511406c330 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.023+0000 7f511c114640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.023+0000 7f511c114640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 0x7f511412fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.023+0000 7f511c114640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.023+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114099760 con 0x7f511412fad0 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 0x7f511412fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient: get_auth_request con 0x7f511412fad0 auth_method 0 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 10 monclient(hunting): handle_auth_done global_id 214170 payload 824 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.024+0000 7f5119eb0640 1 -- 172.21.15.153:0/33957250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5114098e50 con 0x7f511412fad0 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.026+0000 7f5119eb0640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 msgr2=0x7f511412fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.026+0000 7f5119eb0640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 0x7f511412fed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f511000d500 tx=0x7f511001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.026+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110020070 con 0x7f511412fad0 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.026+0000 7f5109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.026+0000 7f5109ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:13.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.026+0000 7f5109ffb640 10 monclient: dump: 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f5110007650 con 0x7f511412fad0 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 msgr2=0x7f511412fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 0x7f511412fed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f511000d500 tx=0x7f511001a000 comp rx=0 tx=0).stop 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 msgr2=0x7f511412fed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f5109ffb640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f511412fad0 0x7f511412fed0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f511000d500 tx=0x7f511001a000 comp rx=0 tx=0).stop 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 182, 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:13.051 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:13.052 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:13.053 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:13.056 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:13.057 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.058 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:13.059 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.061 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:13.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:13.061 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:13.061 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:13.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.061 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:13.062 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:13.062 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.062 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.062 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:46:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:46:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.067 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:46:13.067 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.067 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.067 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:13.067 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:13.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f80386f0 msgr2=0x7f50f803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f50f80386f0 0x7f50f803aba0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f51140992d0 tx=0x7f5104007b80 comp rx=0 tx=0).stop 2024-09-15T03:46:13.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 10 monclient: shutdown 2024-09-15T03:46:13.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 msgr2=0x7f511406bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 --2- 172.21.15.153:0/33957250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5114137cc0 0x7f511406bdf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f510c0099f0 tx=0x7f510c037000 comp rx=0 tx=0).stop 2024-09-15T03:46:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 reap_dead start 2024-09-15T03:46:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511a6b1640 1 -- 172.21.15.153:0/33957250 reap_dead start 2024-09-15T03:46:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 shutdown_connections 2024-09-15T03:46:13.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 >> 172.21.15.153:0/33957250 conn(0x7f51140a2e40 msgr2=0x7f51140a1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:13.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 shutdown_connections 2024-09-15T03:46:13.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.027+0000 7f511c114640 1 -- 172.21.15.153:0/33957250 wait complete. 2024-09-15T03:46:13.072 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:46:13.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:46:13.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:46:13.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.166+0000 7fdcd9c80640 10 monclient: get_monmap_and_config 2024-09-15T03:46:13.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.166+0000 7fdcd9c80640 10 monclient: build_initial_monmap 2024-09-15T03:46:13.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.166+0000 7fdcd9c80640 10 monclient: monmap: 2024-09-15T03:46:13.168 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:13.168 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.168 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:13.167703+0000 2024-09-15T03:46:13.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:13.167703+0000 2024-09-15T03:46:13.169 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:13.169 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.169 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:13.169 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:13.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:13.170 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:13.170 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:13.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.167+0000 7fdcd9c80640 1 Processor -- start 2024-09-15T03:46:13.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.167+0000 7fdcd9c80640 1 -- start start 2024-09-15T03:46:13.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.167+0000 7fdcd9c80640 10 monclient: init 2024-09-15T03:46:13.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:13.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:46:13.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcd40981c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:13.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): picked mon.noname-b con 0x7fdcd4131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdcd4132170 0x7fdcd41390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): picked mon.noname-a con 0x7fdcd4132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd409ca60 con 0x7fdcd40981c0 2024-09-15T03:46:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd4131bd0 con 0x7fdcd4132170 2024-09-15T03:46:13.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd409cca0 con 0x7fdcd4131da0 2024-09-15T03:46:13.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:13.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd9c80640 10 monclient(hunting): authenticate will time out at 1770.807302s 2024-09-15T03:46:13.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd2ffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 msgr2=0x7fdcd4136b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:13.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:13.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdcd4132170 0x7fdcd41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdcd4132170 0x7fdcd41390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39266/0 (socket says 172.21.15.153:39266) 2024-09-15T03:46:13.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 -- 172.21.15.153:0/1170814754 learned_addr learned my addr 172.21.15.153:0/1170814754 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:13.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): get_auth_request con 0x7fdcd4132170 auth_method 0 2024-09-15T03:46:13.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient(hunting): handle_auth_done global_id 214173 payload 274 2024-09-15T03:46:13.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 -- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 msgr2=0x7fdcd4136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:13.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 --2- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 -- 172.21.15.153:0/1170814754 >> v1:172.21.15.181:6789/0 conn(0x7fdcd40981c0 legacy=0x7fdcd4098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:13.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:46:13.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 monclient: found mon.noname-a 2024-09-15T03:46:13.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:13.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.168+0000 7fdcd3fff640 1 -- 172.21.15.153:0/1170814754 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd409e010 con 0x7fdcd4132170 2024-09-15T03:46:13.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd3fff640 1 -- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd4132170 msgr2=0x7fdcd41390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd3fff640 1 --2- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd4132170 0x7fdcd41390c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009f20 tx=0x7fdccc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf659ad1b1d8837 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 1 -- 172.21.15.153:0/1170814754 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc007610 con 0x7fdcd4132170 2024-09-15T03:46:13.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:13.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: dump: 2024-09-15T03:46:13.200 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.200 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.201 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.201 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.201 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.201 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.201 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.202 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.202 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.202 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:13.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:13.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd9c80640 5 monclient: authenticate success, global_id 214173 2024-09-15T03:46:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 1 -- 172.21.15.153:0/1170814754 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc02d050 con 0x7fdcd4132170 2024-09-15T03:46:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd9c80640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 1 -- 172.21.15.153:0/1170814754 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0079a0 con 0x7fdcd4132170 2024-09-15T03:46:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd9c80640 4 set_mon_vals no callback set 2024-09-15T03:46:13.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:13.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.171+0000 7fdcd27fc640 10 monclient: dump: 2024-09-15T03:46:13.204 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.204 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.204 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.205 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.205 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.205 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.205 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.205 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.205 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.206 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 10 monclient: shutdown 2024-09-15T03:46:13.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:13.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 -- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd4132170 msgr2=0x7fdcd41390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd4132170 0x7fdcd41390c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009f20 tx=0x7fdccc009840 comp rx=0 tx=0).stop 2024-09-15T03:46:13.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 -- 172.21.15.153:0/1170814754 shutdown_connections 2024-09-15T03:46:13.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd4132170 0x7fdcd41390c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/1170814754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 -- 172.21.15.153:0/1170814754 >> 172.21.15.153:0/1170814754 conn(0x7fdcd40a1e30 msgr2=0x7fdcd40a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:13.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 -- 172.21.15.153:0/1170814754 shutdown_connections 2024-09-15T03:46:13.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 1 -- 172.21.15.153:0/1170814754 wait complete. 2024-09-15T03:46:13.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 10 monclient: build_initial_monmap 2024-09-15T03:46:13.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.172+0000 7fdcd9c80640 10 monclient: monmap: 2024-09-15T03:46:13.208 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:13.208 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:13.208 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:13.209 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:13.209 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:13.209 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.209 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:13.209 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:13.209 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:13.210 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 Processor -- start 2024-09-15T03:46:13.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 -- start start 2024-09-15T03:46:13.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient: init 2024-09-15T03:46:13.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:13.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:46:13.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 0x7fdcd4131450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): picked mon.noname-a con 0x7fdcd40981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:13.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4137ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): picked mon.noname-c con 0x7fdcd4131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:13.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4132170 0x7fdcd41383e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): picked mon.noname-b con 0x7fdcd4132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:13.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd411a980 con 0x7fdcd40981c0 2024-09-15T03:46:13.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd409bb60 con 0x7fdcd4132170 2024-09-15T03:46:13.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4098cb0 con 0x7fdcd4131da0 2024-09-15T03:46:13.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:13.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd9c80640 10 monclient(hunting): authenticate will time out at 1770.812956s 2024-09-15T03:46:13.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd2ffd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 msgr2=0x7fdcd4137ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:13.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.173+0000 7fdcd2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4137ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:13.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 0x7fdcd4131450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 0x7fdcd4131450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39270/0 (socket says 172.21.15.153:39270) 2024-09-15T03:46:13.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 -- 172.21.15.153:0/2346745953 learned_addr learned my addr 172.21.15.153:0/2346745953 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:13.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): get_auth_request con 0x7fdcd40981c0 auth_method 0 2024-09-15T03:46:13.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient(hunting): handle_auth_done global_id 214176 payload 824 2024-09-15T03:46:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd3fff640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4132170 0x7fdcd41383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 msgr2=0x7fdcd4137ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:13.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcd4131da0 0x7fdcd4137ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4132170 msgr2=0x7fdcd41383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4132170 0x7fdcd41383e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 monclient: found mon.noname-a 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd37fe640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc009ae0 con 0x7fdcd40981c0 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd3fff640 10 monclient: get_auth_request con 0x7fdcd4132170 auth_method 0 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.174+0000 7fdcd3fff640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4132170 0x7fdcd41383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd37fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 msgr2=0x7fdcd4131450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd37fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 0x7fdcd4131450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400c610 tx=0x7fdcc4007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4012920 con 0x7fdcd40981c0 2024-09-15T03:46:13.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:13.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: dump: 2024-09-15T03:46:13.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.222 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.222 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:46:13.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:13.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc4012aa0 con 0x7fdcd40981c0 2024-09-15T03:46:13.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 5 monclient: authenticate success, global_id 214176 2024-09-15T03:46:13.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:13.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4012e60 con 0x7fdcd40981c0 2024-09-15T03:46:13.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:13.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcbb7fe640 10 monclient: dump: 2024-09-15T03:46:13.224 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.224 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.224 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.224 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.225 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.225 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.225 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.225 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.225 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.226 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 10 monclient: _renew_subs 2024-09-15T03:46:13.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd4130940 con 0x7fdcd40981c0 2024-09-15T03:46:13.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd1ffb640 4 set_mon_vals no callback set 2024-09-15T03:46:13.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 10 monclient: _renew_subs 2024-09-15T03:46:13.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:13.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.176+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd4118870 con 0x7fdcd40981c0 2024-09-15T03:46:13.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd9c80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:13.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 0x7fdcd402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd411a4d0 con 0x7fdcd4119440 2024-09-15T03:46:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdcc4011590 con 0x7fdcd40981c0 2024-09-15T03:46:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcbb7fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcc00386f0 0x7fdcc003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdcc404c850 con 0x7fdcd40981c0 2024-09-15T03:46:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd2ffd640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 0x7fdcd402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd2ffd640 10 monclient: get_auth_request con 0x7fdcd4119440 auth_method 0 2024-09-15T03:46:13.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.177+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd3fff640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcc00386f0 0x7fdcc003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd3fff640 10 monclient: get_auth_request con 0x7fdcc00386f0 auth_method 0 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_done global_id 204235 payload 824 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.178+0000 7fdcd2ffd640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdcd411a730 con 0x7fdcd4119440 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcd2ffd640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 msgr2=0x7fdcd402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcd2ffd640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 0x7fdcd402aa40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4069ac0 tx=0x7fdcc80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcd3fff640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcc00386f0 0x7fdcc003aba0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4066560 tx=0x7fdccc036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80050f0 con 0x7fdcd4119440 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.180+0000 7fdcbb7fe640 10 monclient: dump: 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdcc8005250 con 0x7fdcd4119440 2024-09-15T03:46:13.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 msgr2=0x7fdcd402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 0x7fdcd402aa40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4069ac0 tx=0x7fdcc80091a0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 msgr2=0x7fdcd402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.181+0000 7fdcbb7fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4119440 0x7fdcd402aa40 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4069ac0 tx=0x7fdcc80091a0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.186+0000 7fdcd9c80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.186+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 0x7fdcd405e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.186+0000 7fdcd9c80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.186+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4065f90 con 0x7fdcd4138920 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 0x7fdcd405e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient: get_auth_request con 0x7fdcd4138920 auth_method 0 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 10 monclient(hunting): handle_auth_done global_id 204238 payload 824 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.187+0000 7fdcd2ffd640 1 -- 172.21.15.153:0/2346745953 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdcd4138d20 con 0x7fdcd4138920 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.189+0000 7fdcd2ffd640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 msgr2=0x7fdcd405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.189+0000 7fdcd2ffd640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 0x7fdcd405e310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4069ac0 tx=0x7fdcc8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8010e90 con 0x7fdcd4138920 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 10 monclient: dump: 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:13.233 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7fdcc8019400 con 0x7fdcd4138920 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 msgr2=0x7fdcd405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 0x7fdcd405e310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4069ac0 tx=0x7fdcc8015000 comp rx=0 tx=0).stop 2024-09-15T03:46:13.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 msgr2=0x7fdcd405e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcbb7fe640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcd4138920 0x7fdcd405e310 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 182, 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:13.235 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:13.236 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:13.237 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:13.238 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:13.239 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:13.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:13.244 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.245 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:13.247 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:13.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcc00386f0 msgr2=0x7fdcc003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdcc00386f0 0x7fdcc003aba0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4066560 tx=0x7fdccc036000 comp rx=0 tx=0).stop 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 10 monclient: shutdown 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 msgr2=0x7fdcd4131450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 0x7fdcd4131450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400c610 tx=0x7fdcc4007ed0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd3fff640 1 -- 172.21.15.153:0/2346745953 reap_dead start 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 shutdown_connections 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 --2- 172.21.15.153:0/2346745953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcd40981c0 0x7fdcd4131450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 >> 172.21.15.153:0/2346745953 conn(0x7fdcd40a1e30 msgr2=0x7fdcd409bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 shutdown_connections 2024-09-15T03:46:13.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:13.190+0000 7fdcd9c80640 1 -- 172.21.15.153:0/2346745953 wait complete. 2024-09-15T03:46:13.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:46:13.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:46:33.250 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:46:33.251 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:46:33.251 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:46:33.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:46:33.254 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:46:33.254 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:46:33.254 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:33.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.358+0000 7f0cb4ef8640 10 monclient: get_monmap_and_config 2024-09-15T03:46:33.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.359+0000 7f0cb4ef8640 10 monclient: build_initial_monmap 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.359+0000 7f0cb4ef8640 10 monclient: monmap: 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:33.359765+0000 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:33.359765+0000 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:33.360 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:33.361 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:33.361 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:33.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.359+0000 7f0cb4ef8640 1 Processor -- start 2024-09-15T03:46:33.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.359+0000 7f0cb4ef8640 1 -- start start 2024-09-15T03:46:33.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.359+0000 7f0cb4ef8640 10 monclient: init 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0cb01318f0 0x7f0cb0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f0cb01318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0cb009ea50 0x7f0cb009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f0cb009ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0cb0068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb009fbf0 con 0x7f0cb0068d90 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:33.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb0131d50 con 0x7f0cb009ea50 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb009fe30 con 0x7f0cb01318f0 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cb4ef8640 10 monclient(hunting): authenticate will time out at 1790.999390s 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0cb009ea50 0x7f0cb009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0cb009ea50 0x7f0cb009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47836/0 (socket says 172.21.15.153:47836) 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 -- 172.21.15.153:0/3848330014 learned_addr learned my addr 172.21.15.153:0/3848330014 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): get_auth_request con 0x7f0cb009ea50 auth_method 0 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0caed9d640 1 --1- 172.21.15.153:0/3848330014 >> v1:172.21.15.153:6789/0 conn(0x7f0cb0068d90 0x7f0cb00691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40380/0 (socket says 172.21.15.153:40380) 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:33.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 1 --2- 172.21.15.153:0/3848330014 >> v2:172.21.15.181:3300/0 conn(0x7f0cb01318f0 0x7f0cb0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): get_auth_request con 0x7f0cb01318f0 auth_method 0 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:33.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_done global_id 214179 payload 274 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 -- 172.21.15.153:0/3848330014 >> v2:172.21.15.181:3300/0 conn(0x7f0cb01318f0 msgr2=0x7f0cb0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 --2- 172.21.15.153:0/3848330014 >> v2:172.21.15.181:3300/0 conn(0x7f0cb01318f0 0x7f0cb0131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 -- 172.21.15.153:0/3848330014 >> v1:172.21.15.153:6789/0 conn(0x7f0cb0068d90 legacy=0x7f0cb00691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 monclient: found mon.noname-a 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:33.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.360+0000 7f0cadd9b640 1 -- 172.21.15.153:0/3848330014 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00a1210 con 0x7f0cb009ea50 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cadd9b640 1 -- 172.21.15.153:0/3848330014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb009ea50 msgr2=0x7f0cb009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cadd9b640 1 --2- 172.21.15.153:0/3848330014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb009ea50 0x7f0cb009ee20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0076600 tx=0x7f0ca40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc5c1d2a159c4b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 1 -- 172.21.15.153:0/3848330014 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4032440 con 0x7f0cb009ea50 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: dump: 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.375 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: _finish_auth 0 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 1 -- 172.21.15.153:0/3848330014 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca40325a0 con 0x7f0cb009ea50 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cb4ef8640 5 monclient: authenticate success, global_id 214179 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cb4ef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cb4ef8640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 1 -- 172.21.15.153:0/3848330014 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4032940 con 0x7f0cb009ea50 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cb4ef8640 4 set_mon_vals no callback set 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.363+0000 7f0cad59a640 10 monclient: dump: 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 10 monclient: shutdown 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/3848330014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb009ea50 msgr2=0x7f0cb009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/3848330014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb009ea50 0x7f0cb009ee20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0076600 tx=0x7f0ca40091d0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/3848330014 shutdown_connections 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/3848330014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb009ea50 0x7f0cb009ee20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/3848330014 >> v2:172.21.15.181:3300/0 conn(0x7f0cb01318f0 0x7f0cb0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/3848330014 >> 172.21.15.153:0/3848330014 conn(0x7f0cb0067110 msgr2=0x7f0cb0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/3848330014 shutdown_connections 2024-09-15T03:46:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.364+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/3848330014 wait complete. 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient: build_initial_monmap 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient: monmap: 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 Processor -- start 2024-09-15T03:46:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 -- start start 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient: init 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f0cb0068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cb009ea50 0x7f0cb0130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f0cb009ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 0x7f0cb011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f0cb01318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00a4410 con 0x7f0cb01318f0 2024-09-15T03:46:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb011e310 con 0x7f0cb009ea50 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb011e080 con 0x7f0cb0068d90 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.365+0000 7f0cb4ef8640 10 monclient(hunting): authenticate will time out at 1791.004971s 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 0x7f0cb011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45006/0 (socket says 172.21.15.153:45006) 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 0x7f0cb011b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47840/0 (socket says 172.21.15.153:47840) 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 -- 172.21.15.153:0/2271666718 learned_addr learned my addr 172.21.15.153:0/2271666718 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 10 monclient(hunting): get_auth_request con 0x7f0cb0068d90 auth_method 0 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): get_auth_request con 0x7f0cb01318f0 auth_method 0 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 msgr2=0x7f0cb009ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 msgr2=0x7f0cb009ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cae59c640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cb009ea50 0x7f0cb0130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): get_auth_request con 0x7f0cb009ea50 auth_method 0 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0cadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:33.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient(hunting): handle_auth_done global_id 214182 payload 824 2024-09-15T03:46:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 msgr2=0x7f0cb009ab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cb009ea50 msgr2=0x7f0cb0130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cb009ea50 0x7f0cb0130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:46:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 monclient: found mon.noname-a 2024-09-15T03:46:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.366+0000 7f0caed9d640 1 -- 172.21.15.153:0/2271666718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4009450 con 0x7f0cb01318f0 2024-09-15T03:46:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0caed9d640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 msgr2=0x7f0cb011b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0caed9d640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 0x7f0cb011b720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800c7c0 tx=0x7f0ca8004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8005d60 con 0x7f0cb01318f0 2024-09-15T03:46:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: dump: 2024-09-15T03:46:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:33.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:33.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:33.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:33.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:33.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:33.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:33.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:33.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0cb4ef8640 5 monclient: authenticate success, global_id 214182 2024-09-15T03:46:33.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca800a5e0 con 0x7f0cb01318f0 2024-09-15T03:46:33.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0cb4ef8640 10 monclient: _renew_subs 2024-09-15T03:46:33.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0cb4ef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:33.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb011dd20 con 0x7f0cb01318f0 2024-09-15T03:46:33.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:33.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800a9a0 con 0x7f0cb01318f0 2024-09-15T03:46:33.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:33.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:33.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0c9e7fc640 10 monclient: dump: 2024-09-15T03:46:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:33.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.368+0000 7f0cacd99640 4 set_mon_vals no callback set 2024-09-15T03:46:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 10 monclient: _renew_subs 2024-09-15T03:46:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb011ce30 con 0x7f0cb01318f0 2024-09-15T03:46:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0ca80070f0 con 0x7f0cb01318f0 2024-09-15T03:46:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0c9e7fc640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c90038740 0x7f0c9003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0ca804c260 con 0x7f0cb01318f0 2024-09-15T03:46:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.369+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb009b7c0 con 0x7f0cb01318f0 2024-09-15T03:46:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.370+0000 7f0cae59c640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c90038740 0x7f0c9003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.370+0000 7f0cae59c640 10 monclient: get_auth_request con 0x7f0c90038740 auth_method 0 2024-09-15T03:46:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.372+0000 7f0cae59c640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c90038740 0x7f0c9003abf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009550 tx=0x7f0ca0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.374+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0ca800ad90 con 0x7f0cb01318f0 2024-09-15T03:46:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.374+0000 7f0c9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.374+0000 7f0c9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:33.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0cb4ef8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:33.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0cb4ef8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:33.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0cb4ef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:33.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0cb004bf60 con 0x7f0cb01318f0 2024-09-15T03:46:33.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0c9e7fc640 1 -- 172.21.15.153:0/2271666718 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f0ca8010770 con 0x7f0cb01318f0 2024-09-15T03:46:33.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0c9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:33.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.588+0000 7f0c9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:33.590 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:33.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c90038740 msgr2=0x7f0c9003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:33.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c90038740 0x7f0c9003abf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009550 tx=0x7f0ca0008040 comp rx=0 tx=0).stop 2024-09-15T03:46:33.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 10 monclient: shutdown 2024-09-15T03:46:33.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 msgr2=0x7f0cb011b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 0x7f0cb011b720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800c7c0 tx=0x7f0ca8004e20 comp rx=0 tx=0).stop 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 shutdown_connections 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0c90038740 0x7f0c9003abf0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cb01318f0 0x7f0cb011b720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cb009ea50 0x7f0cb0130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 --2- 172.21.15.153:0/2271666718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cb0068d90 0x7f0cb009ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 >> 172.21.15.153:0/2271666718 conn(0x7f0cb0067110 msgr2=0x7f0cb00a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 shutdown_connections 2024-09-15T03:46:33.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:33.590+0000 7f0cb4ef8640 1 -- 172.21.15.153:0/2271666718 wait complete. 2024-09-15T03:46:36.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:36.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.708+0000 7f0159d09640 10 monclient: get_monmap_and_config 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.708+0000 7f0159d09640 10 monclient: build_initial_monmap 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.708+0000 7f0159d09640 10 monclient: monmap: 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:36.708783+0000 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:36.708783+0000 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:36.709 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:36.710 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:36.710 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.708+0000 7f0159d09640 1 Processor -- start 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.708+0000 7f0159d09640 1 -- start start 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.708+0000 7f0159d09640 10 monclient: init 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01541318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f015409ea50 0x7f015409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient(hunting): picked mon.noname-a con 0x7f015409ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0154136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154131cc0 con 0x7f0154136c20 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015409fbf0 con 0x7f01541318f0 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015409fe30 con 0x7f015409ea50 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0159d09640 10 monclient(hunting): authenticate will time out at 1794.348354s 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f015409ea50 0x7f015409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0153fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0154136c20 0x7f0154139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45292/0 (socket says 172.21.15.153:45292) 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0153fff640 1 -- 172.21.15.153:0/2216670101 learned_addr learned my addr 172.21.15.153:0/2216670101 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f015409ea50 0x7f015409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58500/0 (socket says 172.21.15.153:58500) 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): get_auth_request con 0x7f015409ea50 auth_method 0 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:36.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient(hunting): handle_auth_done global_id 214185 payload 274 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 1 -- 172.21.15.153:0/2216670101 >> v1:172.21.15.181:6789/0 conn(0x7f01541318f0 legacy=0x7f0154131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 1 -- 172.21.15.153:0/2216670101 >> v1:172.21.15.153:6789/0 conn(0x7f0154136c20 legacy=0x7f0154139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 1 monclient: found mon.noname-a 2024-09-15T03:46:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.709+0000 7f0152ffd640 1 -- 172.21.15.153:0/2216670101 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540a1210 con 0x7f015409ea50 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f0152ffd640 1 -- 172.21.15.153:0/2216670101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f015409ea50 msgr2=0x7f015409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f0152ffd640 1 --2- 172.21.15.153:0/2216670101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f015409ea50 0x7f015409ee20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0148009850 tx=0x7f01480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be17b8f116894a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 1 -- 172.21.15.153:0/2216670101 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148032440 con 0x7f015409ea50 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: dump: 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.721 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 1 -- 172.21.15.153:0/2216670101 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01480325a0 con 0x7f015409ea50 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f0159d09640 5 monclient: authenticate success, global_id 214185 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f0159d09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:36.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f0159d09640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 1 -- 172.21.15.153:0/2216670101 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148032940 con 0x7f015409ea50 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f0159d09640 4 set_mon_vals no callback set 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.712+0000 7f01527fc640 10 monclient: dump: 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.723 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 10 monclient: shutdown 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 -- 172.21.15.153:0/2216670101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f015409ea50 msgr2=0x7f015409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 --2- 172.21.15.153:0/2216670101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f015409ea50 0x7f015409ee20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0148009850 tx=0x7f01480090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 -- 172.21.15.153:0/2216670101 shutdown_connections 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 --2- 172.21.15.153:0/2216670101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f015409ea50 0x7f015409ee20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 -- 172.21.15.153:0/2216670101 >> 172.21.15.153:0/2216670101 conn(0x7f0154067110 msgr2=0x7f0154067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 -- 172.21.15.153:0/2216670101 shutdown_connections 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 1 -- 172.21.15.153:0/2216670101 wait complete. 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 10 monclient: build_initial_monmap 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.713+0000 7f0159d09640 10 monclient: monmap: 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:36.725 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 1 Processor -- start 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 1 -- start start 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 10 monclient: init 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f015409ea50 0x7f0154141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 10 monclient(hunting): picked mon.noname-b con 0x7f015409ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.714+0000 7f0159d09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 0x7f01541417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): picked mon.noname-a con 0x7f01541318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 0x7f0154098820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): picked mon.noname-c con 0x7f0154136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:36.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015409bf10 con 0x7f01541318f0 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015409dfe0 con 0x7f015409ea50 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015411bb00 con 0x7f0154136c20 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0159d09640 10 monclient(hunting): authenticate will time out at 1794.354033s 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 0x7f01541417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 0x7f0154098820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 0x7f01541417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58508/0 (socket says 172.21.15.153:58508) 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 -- 172.21.15.153:0/3386505679 learned_addr learned my addr 172.21.15.153:0/3386505679 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): get_auth_request con 0x7f01541318f0 auth_method 0 2024-09-15T03:46:36.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 10 monclient(hunting): get_auth_request con 0x7f0154136c20 auth_method 0 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f015409ea50 0x7f0154141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 msgr2=0x7f0154098820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 msgr2=0x7f0154098820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:36.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 0x7f0154098820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): get_auth_request con 0x7f015409ea50 auth_method 0 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0153fff640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 0x7f0154098820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f01537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient(hunting): handle_auth_done global_id 214188 payload 824 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 msgr2=0x7f0154098820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 0x7f0154098820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f015409ea50 msgr2=0x7f0154141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f015409ea50 0x7f0154141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 monclient: found mon.noname-a 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.715+0000 7f0152ffd640 1 -- 172.21.15.153:0/3386505679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148009390 con 0x7f01541318f0 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0152ffd640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 msgr2=0x7f01541417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0152ffd640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 0x7f01541417b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0148009360 tx=0x7f0148032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480318c0 con 0x7f01541318f0 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: dump: 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0148031a20 con 0x7f01541318f0 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0159d09640 5 monclient: authenticate success, global_id 214188 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0159d09640 10 monclient: _renew_subs 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148031dc0 con 0x7f01541318f0 2024-09-15T03:46:36.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0159d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015411bde0 con 0x7f01541318f0 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f013b7fe640 10 monclient: dump: 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:36.732 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.717+0000 7f0151ffb640 4 set_mon_vals no callback set 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.718+0000 7f0159d09640 10 monclient: _renew_subs 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.718+0000 7f0159d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.718+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015409af40 con 0x7f01541318f0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.718+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0148039040 con 0x7f01541318f0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f0159d09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f0159d09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f0159d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f013b7fe640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01400387c0 0x7f014003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015404bf60 con 0x7f01541318f0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f014806c7c0 con 0x7f01541318f0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f01537fe640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01400387c0 0x7f014003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:36.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.719+0000 7f01537fe640 10 monclient: get_auth_request con 0x7f01400387c0 auth_method 0 2024-09-15T03:46:36.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.722+0000 7f01537fe640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01400387c0 0x7f014003ac70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0144007fd0 tx=0x7f0144007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:36.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.723+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f014803f080 con 0x7f01541318f0 2024-09-15T03:46:36.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.723+0000 7f013b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:36.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.723+0000 7f013b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:36.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.936+0000 7f0159d09640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.936+0000 7f0159d09640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.936+0000 7f0159d09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.936+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f015404bf60 con 0x7f01541318f0 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.937+0000 7f013b7fe640 1 -- 172.21.15.153:0/3386505679 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f0148049020 con 0x7f01541318f0 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.937+0000 7f013b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.937+0000 7f013b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:36.938 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01400387c0 msgr2=0x7f014003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01400387c0 0x7f014003ac70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0144007fd0 tx=0x7f0144007e10 comp rx=0 tx=0).stop 2024-09-15T03:46:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 10 monclient: shutdown 2024-09-15T03:46:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 msgr2=0x7f01541417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 0x7f01541417b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0148009360 tx=0x7f0148032000 comp rx=0 tx=0).stop 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 shutdown_connections 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f01400387c0 0x7f014003ac70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0154136c20 0x7f0154098820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01541318f0 0x7f01541417b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 --2- 172.21.15.153:0/3386505679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f015409ea50 0x7f0154141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 >> 172.21.15.153:0/3386505679 conn(0x7f0154067110 msgr2=0x7f01540a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 shutdown_connections 2024-09-15T03:46:36.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:36.939+0000 7f0159d09640 1 -- 172.21.15.153:0/3386505679 wait complete. 2024-09-15T03:46:39.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:40.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.056+0000 7f924dcaf640 10 monclient: get_monmap_and_config 2024-09-15T03:46:40.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.056+0000 7f924dcaf640 10 monclient: build_initial_monmap 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.056+0000 7f924dcaf640 10 monclient: monmap: 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:40.056856+0000 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:40.056856+0000 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:40.058 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:40.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.056+0000 7f924dcaf640 1 Processor -- start 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.056+0000 7f924dcaf640 1 -- start start 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient: init 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92481318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f924809ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9248137c30 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9248131cc0 con 0x7f92481318f0 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924809f530 con 0x7f924809ea50 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924809fbf0 con 0x7f9248137c30 2024-09-15T03:46:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f924dcaf640 10 monclient(hunting): authenticate will time out at 1797.696533s 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f92477fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f92481318f0 0x7f9248131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45302/0 (socket says 172.21.15.153:45302) 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f92477fe640 1 -- 172.21.15.153:0/3203430084 learned_addr learned my addr 172.21.15.153:0/3203430084 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 10 monclient(hunting): get_auth_request con 0x7f9248137c30 auth_method 0 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:40.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.057+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:40.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.059+0000 7f92477fe640 1 -- 172.21.15.153:0/3203430084 >> v1:172.21.15.153:6789/0 conn(0x7f92481318f0 legacy=0x7f9248131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:40.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.060+0000 7f9246ffd640 1 -- 172.21.15.153:0/3203430084 >> v1:172.21.15.181:6789/0 conn(0x7f924809ea50 legacy=0x7f924809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 10 monclient(hunting): get_auth_request con 0x7f9248137c30 auth_method 0 2024-09-15T03:46:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:46:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.258+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:40.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:40.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 10 monclient(hunting): get_auth_request con 0x7f9248137c30 auth_method 0 2024-09-15T03:46:40.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:40.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:40.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:40.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:46:40.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:40.659+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:41.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 10 monclient(hunting): get_auth_request con 0x7f9248137c30 auth_method 0 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:41.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:41.460+0000 7f9247fff640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:46:43.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): tick 2024-09-15T03:46:43.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 monclient(hunting): continuing hunt 2024-09-15T03:46:43.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 msgr2=0x7f924813a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.153:3300/0 conn(0x7f9248137c30 0x7f924813a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 -- 172.21.15.153:0/3203430084 >> v1:172.21.15.181:6789/0 conn(0x7f924809ea50 legacy=0x7f924809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 -- 172.21.15.153:0/3203430084 >> v1:172.21.15.153:6789/0 conn(0x7f92481318f0 legacy=0x7f9248131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient: _add_conns ranks=[2,4] 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): picked mon.noname-b con 0x7f92340037f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 0x7f92340087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): picked mon.noname-c con 0x7f9234006340 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 -- 172.21.15.153:0/3203430084 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924809fbf0 con 0x7f9234006340 2024-09-15T03:46:43.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 1 -- 172.21.15.153:0/3203430084 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9234008eb0 con 0x7f92340037f0 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.057+0000 7f92457fa640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 10 monclient(hunting): get_auth_request con 0x7f92340037f0 auth_method 0 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 0x7f92340087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 msgr2=0x7f9234005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 msgr2=0x7f9234005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 10 monclient(hunting): get_auth_request con 0x7f9234006340 auth_method 0 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:43.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:43.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f92477fe640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:43.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 msgr2=0x7f92340087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:46:43.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 1 -- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 msgr2=0x7f92340087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:43.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 0x7f92340087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:43.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.058+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 0x7f92340087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 10 monclient(hunting): get_auth_request con 0x7f92340037f0 auth_method 0 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 msgr2=0x7f9234005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 msgr2=0x7f9234005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:43.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f92477fe640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3203430084 >> v2:172.21.15.181:3300/0 conn(0x7f9234006340 0x7f92340087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): get_auth_request con 0x7f9234006340 auth_method 0 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient(hunting): handle_auth_done global_id 204250 payload 274 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 msgr2=0x7f9234005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340037f0 0x7f9234005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 1 monclient: found mon.noname-c 2024-09-15T03:46:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.259+0000 7f9246ffd640 1 -- 172.21.15.153:0/3203430084 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924809fe30 con 0x7f9234006340 2024-09-15T03:46:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.262+0000 7f9246ffd640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 msgr2=0x7f92340087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:43.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.262+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f92340087f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f923c007b00 tx=0x7f923c009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c24c37052cef66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 1 -- 172.21.15.153:0/3203430084 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c02ae80 con 0x7f9234006340 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: dump: 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 1 -- 172.21.15.153:0/3203430084 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c0070d0 con 0x7f9234006340 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 1 -- 172.21.15.153:0/3203430084 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c007450 con 0x7f9234006340 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:43.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f92467fc640 10 monclient: dump: 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f924dcaf640 5 monclient: authenticate success, global_id 204250 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f924dcaf640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:43.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.263+0000 7f924dcaf640 4 set_mon_vals no callback set 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 10 monclient: shutdown 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 -- 172.21.15.153:0/3203430084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 msgr2=0x7f92340087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f92340087f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f923c007b00 tx=0x7f923c009040 comp rx=0 tx=0).stop 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f9247fff640 1 -- 172.21.15.153:0/3203430084 reap_dead start 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 -- 172.21.15.153:0/3203430084 shutdown_connections 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3203430084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f92340087f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 -- 172.21.15.153:0/3203430084 >> 172.21.15.153:0/3203430084 conn(0x7f9248067110 msgr2=0x7f9248067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 -- 172.21.15.153:0/3203430084 shutdown_connections 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.264+0000 7f924dcaf640 1 -- 172.21.15.153:0/3203430084 wait complete. 2024-09-15T03:46:43.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.265+0000 7f924dcaf640 10 monclient: build_initial_monmap 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.265+0000 7f924dcaf640 10 monclient: monmap: 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.265+0000 7f924dcaf640 1 Processor -- start 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 -- start start 2024-09-15T03:46:43.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient: init 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f9248136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9234006340 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): picked mon.noname-c con 0x7f924809a420 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 0x7f9248097420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9248096fa0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9248099a40 con 0x7f9248096fa0 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924809a000 con 0x7f9234006340 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:43.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924809d850 con 0x7f924809a420 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f924dcaf640 10 monclient(hunting): authenticate will time out at 1800.905617s 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 0x7f9248097420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 0x7f9248097420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58550/0 (socket says 172.21.15.153:58550) 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33530/0 (socket says 172.21.15.153:33530) 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 1 -- 172.21.15.153:0/3565675851 learned_addr learned my addr 172.21.15.153:0/3565675851 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): get_auth_request con 0x7f9248096fa0 auth_method 0 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 10 monclient(hunting): get_auth_request con 0x7f924809a420 auth_method 0 2024-09-15T03:46:43.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f92477fe640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f9248136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 msgr2=0x7f9248096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 msgr2=0x7f9248096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f92477fe640 10 monclient(hunting): get_auth_request con 0x7f9234006340 auth_method 0 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f92477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f92477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:43.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f92477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.266+0000 7f9246ffd640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f92477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f92477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f92477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 10 monclient(hunting): handle_auth_done global_id 214191 payload 824 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 msgr2=0x7f9248096a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 msgr2=0x7f9248136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f9248136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 10 monclient: _finish_hunting 0 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 1 monclient: found mon.noname-a 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:43.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f9247fff640 1 -- 172.21.15.153:0/3565675851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c00f040 con 0x7f9248096fa0 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.267+0000 7f92477fe640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f9248136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f9247fff640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 msgr2=0x7f9248097420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f9247fff640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 0x7f9248097420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f924806a300 tx=0x7f9240008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240013e80 con 0x7f9248096fa0 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: dump: 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:43.278 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240016460 con 0x7f9248096fa0 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 5 monclient: authenticate success, global_id 214191 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 10 monclient: _renew_subs 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9248097c30 con 0x7f9248096fa0 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:43.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240016820 con 0x7f9248096fa0 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f922f7fe640 10 monclient: dump: 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:43.280 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 10 monclient: _renew_subs 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9248099760 con 0x7f9248096fa0 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.269+0000 7f9245ffb640 4 set_mon_vals no callback set 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.270+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9240016c50 con 0x7f9248096fa0 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f924dcaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f924dcaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:43.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f924dcaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9248099c60 con 0x7f9248096fa0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f922f7fe640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234039680 0x7f923403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f924004c250 con 0x7f9248096fa0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f92477fe640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234039680 0x7f923403bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.271+0000 7f92477fe640 10 monclient: get_auth_request con 0x7f9234039680 auth_method 0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.274+0000 7f92477fe640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234039680 0x7f923403bb30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9238002250 tx=0x7f9238009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.275+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9248099c60 con 0x7f9248096fa0 2024-09-15T03:46:43.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.275+0000 7f922f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:43.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.275+0000 7f922f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f924dcaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f924dcaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f924dcaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9248099c60 con 0x7f9248096fa0 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f922f7fe640 1 -- 172.21.15.153:0/3565675851 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9248099c60 con 0x7f9248096fa0 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f922f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:43.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.496+0000 7f922f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:43.498 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":186,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:43.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234039680 msgr2=0x7f923403bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:43.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234039680 0x7f923403bb30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9238002250 tx=0x7f9238009040 comp rx=0 tx=0).stop 2024-09-15T03:46:43.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 10 monclient: shutdown 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 msgr2=0x7f9248097420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 0x7f9248097420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f924806a300 tx=0x7f9240008040 comp rx=0 tx=0).stop 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 shutdown_connections 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9234039680 0x7f923403bb30 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9248096fa0 0x7f9248097420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f924809a420 0x7f9248096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 --2- 172.21.15.153:0/3565675851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9234006340 0x7f9248136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 >> 172.21.15.153:0/3565675851 conn(0x7f9248067110 msgr2=0x7f92480a4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 shutdown_connections 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:43.498+0000 7f924dcaf640 1 -- 172.21.15.153:0/3565675851 wait complete. 2024-09-15T03:46:46.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:46.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.615+0000 7f1e50aef640 10 monclient: get_monmap_and_config 2024-09-15T03:46:46.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.615+0000 7f1e50aef640 10 monclient: build_initial_monmap 2024-09-15T03:46:46.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.615+0000 7f1e50aef640 10 monclient: monmap: 2024-09-15T03:46:46.617 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:46.617 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.617 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:46.615930+0000 2024-09-15T03:46:46.617 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:46.615930+0000 2024-09-15T03:46:46.618 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:46.618 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.618 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:46.618 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:46.618 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:46.618 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:46.619 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:46.619 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.615+0000 7f1e50aef640 1 Processor -- start 2024-09-15T03:46:46.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.615+0000 7f1e50aef640 1 -- start start 2024-09-15T03:46:46.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.615+0000 7f1e50aef640 10 monclient: init 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e4c131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1e4c09e910 0x7f1e4c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): picked mon.noname-a con 0x7f1e4c09e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 0x7f1e4c138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): picked mon.noname-b con 0x7f1e4c09f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:46.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4c09faf0 con 0x7f1e4c131ab0 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4c132590 con 0x7f1e4c09e910 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c09fd30 con 0x7f1e4c09f250 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e50aef640 10 monclient(hunting): authenticate will time out at 1804.255520s 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1e4c09e910 0x7f1e4c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 0x7f1e4c138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4a59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1e4c131ab0 0x7f1e4c131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44266/0 (socket says 172.21.15.153:44266) 2024-09-15T03:46:46.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1e4c09e910 0x7f1e4c09ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42492/0 (socket says 172.21.15.153:42492) 2024-09-15T03:46:46.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4a59c640 1 -- 172.21.15.153:0/1752219144 learned_addr learned my addr 172.21.15.153:0/1752219144 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:46.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): get_auth_request con 0x7f1e4c09e910 auth_method 0 2024-09-15T03:46:46.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 10 monclient(hunting): get_auth_request con 0x7f1e4c09f250 auth_method 0 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 1 -- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 msgr2=0x7f1e4c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 1 -- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 msgr2=0x7f1e4c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 0x7f1e4c138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e4ad9d640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 0x7f1e4c138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_done global_id 214194 payload 274 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 msgr2=0x7f1e4c138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 0x7f1e4c138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1752219144 >> v1:172.21.15.153:6789/0 conn(0x7f1e4c131ab0 legacy=0x7f1e4c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 monclient: found mon.noname-a 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.616+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1752219144 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c0a1110 con 0x7f1e4c09e910 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09e910 msgr2=0x7f1e4c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e49d9b640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09e910 0x7f1e4c09ed10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009850 tx=0x7f1e3c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d495b2020d5fe520 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 1 -- 172.21.15.153:0/1752219144 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c02ae80 con 0x7f1e4c09e910 2024-09-15T03:46:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: dump: 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: _finish_auth 0 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 1 -- 172.21.15.153:0/1752219144 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c032440 con 0x7f1e4c09e910 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e50aef640 5 monclient: authenticate success, global_id 214194 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 1 -- 172.21.15.153:0/1752219144 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c032800 con 0x7f1e4c09e910 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e50aef640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e50aef640 4 set_mon_vals no callback set 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.619+0000 7f1e4959a640 10 monclient: dump: 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.644 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 10 monclient: shutdown 2024-09-15T03:46:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 -- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09e910 msgr2=0x7f1e4c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09e910 0x7f1e4c09ed10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009850 tx=0x7f1e3c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 -- 172.21.15.153:0/1752219144 shutdown_connections 2024-09-15T03:46:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c09f250 0x7f1e4c138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1752219144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09e910 0x7f1e4c09ed10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 -- 172.21.15.153:0/1752219144 >> 172.21.15.153:0/1752219144 conn(0x7f1e4c067d30 msgr2=0x7f1e4c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 -- 172.21.15.153:0/1752219144 shutdown_connections 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.620+0000 7f1e50aef640 1 -- 172.21.15.153:0/1752219144 wait complete. 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient: build_initial_monmap 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient: monmap: 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 Processor -- start 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 -- start start 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient: init 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c09e910 0x7f1e4c134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): picked mon.noname-b con 0x7f1e4c09e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 0x7f1e4c134840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): picked mon.noname-a con 0x7f1e4c09f250 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 0x7f1e4c134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): picked mon.noname-c con 0x7f1e4c131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c1378e0 con 0x7f1e4c09f250 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c09bde0 con 0x7f1e4c09e910 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c09d730 con 0x7f1e4c131ab0 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e50aef640 10 monclient(hunting): authenticate will time out at 1804.260862s 2024-09-15T03:46:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 0x7f1e4c134840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 0x7f1e4c134840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42506/0 (socket says 172.21.15.153:42506) 2024-09-15T03:46:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1222322081 learned_addr learned my addr 172.21.15.153:0/1222322081 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e4ad9d640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 0x7f1e4c134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 10 monclient(hunting): get_auth_request con 0x7f1e4c09f250 auth_method 0 2024-09-15T03:46:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e4ad9d640 10 monclient(hunting): get_auth_request con 0x7f1e4c131ab0 auth_method 0 2024-09-15T03:46:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.621+0000 7f1e4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4ad9d640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 msgr2=0x7f1e4c134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:46:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4ad9d640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 msgr2=0x7f1e4c134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4ad9d640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 0x7f1e4c134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4ad9d640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 0x7f1e4c134d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 10 monclient(hunting): handle_auth_done global_id 214197 payload 824 2024-09-15T03:46:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4a59c640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c09e910 0x7f1e4c134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 msgr2=0x7f1e4c134d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 0x7f1e4c134d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c09e910 msgr2=0x7f1e4c134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c09e910 0x7f1e4c134300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 1 monclient: found mon.noname-a 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1222322081 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c009390 con 0x7f1e4c09f250 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4a59c640 10 monclient: get_auth_request con 0x7f1e4c09e910 auth_method 0 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.622+0000 7f1e4a59c640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c09e910 0x7f1e4c134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:46:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e49d9b640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 msgr2=0x7f1e4c134840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e49d9b640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 0x7f1e4c134840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009850 tx=0x7f1e3c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c031d40 con 0x7f1e4c09f250 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: dump: 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c03d420 con 0x7f1e4c09f250 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 5 monclient: authenticate success, global_id 214197 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 10 monclient: _renew_subs 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c137600 con 0x7f1e4c09f250 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c031ea0 con 0x7f1e4c09f250 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e427fc640 10 monclient: dump: 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.657 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 10 monclient: _renew_subs 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c1315e0 con 0x7f1e4c09f250 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.624+0000 7f1e48d99640 4 set_mon_vals no callback set 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1e3c039040 con 0x7f1e4c09f250 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e50aef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e50aef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e50aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c137b00 con 0x7f1e4c09f250 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e427fc640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1e2c038740 0x7f1e2c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.625+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1e3c06c430 con 0x7f1e4c09f250 2024-09-15T03:46:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.626+0000 7f1e4a59c640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1e2c038740 0x7f1e2c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.626+0000 7f1e4a59c640 10 monclient: get_auth_request con 0x7f1e2c038740 auth_method 0 2024-09-15T03:46:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.628+0000 7f1e4a59c640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1e2c038740 0x7f1e2c03abf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1e34005150 tx=0x7f1e34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.630+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1e4c137b00 con 0x7f1e4c09f250 2024-09-15T03:46:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.630+0000 7f1e427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.630+0000 7f1e427fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.843+0000 7f1e50aef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.844+0000 7f1e50aef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.844+0000 7f1e50aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.844+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1e4c137b00 con 0x7f1e4c09f250 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.844+0000 7f1e427fc640 1 -- 172.21.15.153:0/1222322081 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f1e3c037620 con 0x7f1e4c09f250 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.844+0000 7f1e427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.844+0000 7f1e427fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:46.845 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1e2c038740 msgr2=0x7f1e2c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1e2c038740 0x7f1e2c03abf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1e34005150 tx=0x7f1e34009040 comp rx=0 tx=0).stop 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 10 monclient: shutdown 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 msgr2=0x7f1e4c134840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 0x7f1e4c134840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009850 tx=0x7f1e3c032000 comp rx=0 tx=0).stop 2024-09-15T03:46:46.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 shutdown_connections 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1e2c038740 0x7f1e2c03abf0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c131ab0 0x7f1e4c134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c09f250 0x7f1e4c134840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 --2- 172.21.15.153:0/1222322081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c09e910 0x7f1e4c134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 >> 172.21.15.153:0/1222322081 conn(0x7f1e4c067d30 msgr2=0x7f1e4c068b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 shutdown_connections 2024-09-15T03:46:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.846+0000 7f1e50aef640 1 -- 172.21.15.153:0/1222322081 wait complete. 2024-09-15T03:46:46.857 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:46:46.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:46:46.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:46:46.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.950+0000 7f5dfcf9d640 10 monclient: get_monmap_and_config 2024-09-15T03:46:46.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.950+0000 7f5dfcf9d640 10 monclient: build_initial_monmap 2024-09-15T03:46:46.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.950+0000 7f5dfcf9d640 10 monclient: monmap: 2024-09-15T03:46:46.951 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:46.951 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:46.951502+0000 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:46.951502+0000 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:46.952 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 Processor -- start 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 -- start start 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient: init 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5df80975c0 0x7f5df8097990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5df80975c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5df812ea30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f5df812ee90 0x7f5df8131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5df812ee90 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df8063950 con 0x7f5df80975c0 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df806e9e0 con 0x7f5df812ee90 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df809d1c0 con 0x7f5df812ea30 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.951+0000 7f5dfcf9d640 10 monclient(hunting): authenticate will time out at 1804.590933s 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5df80975c0 0x7f5df8097990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43814/0 (socket says 172.21.15.153:43814) 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5df80975c0 0x7f5df8097990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42522/0 (socket says 172.21.15.153:42522) 2024-09-15T03:46:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 -- 172.21.15.153:0/923581251 learned_addr learned my addr 172.21.15.153:0/923581251 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 10 monclient(hunting): get_auth_request con 0x7f5df812ea30 auth_method 0 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): get_auth_request con 0x7f5df80975c0 auth_method 0 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 -- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 msgr2=0x7f5df8131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 -- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 msgr2=0x7f5df8131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5deffff640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df6d9d640 1 --2- 172.21.15.153:0/923581251 >> v2:172.21.15.181:3300/0 conn(0x7f5df812ee90 0x7f5df8131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df6d9d640 10 monclient(hunting): get_auth_request con 0x7f5df812ee90 auth_method 0 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient(hunting): handle_auth_done global_id 214200 payload 274 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 -- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 msgr2=0x7f5df8131950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 -- 172.21.15.153:0/923581251 >> v2:172.21.15.181:3300/0 conn(0x7f5df812ee90 msgr2=0x7f5df8131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 --2- 172.21.15.153:0/923581251 >> v2:172.21.15.181:3300/0 conn(0x7f5df812ee90 0x7f5df8131e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient: _finish_hunting 0 2024-09-15T03:46:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 monclient: found mon.noname-a 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df659c640 1 -- 172.21.15.153:0/923581251 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df8097ed0 con 0x7f5df80975c0 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.952+0000 7f5df6d9d640 1 --2- 172.21.15.153:0/923581251 >> v2:172.21.15.181:3300/0 conn(0x7f5df812ee90 0x7f5df8131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df659c640 1 -- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df80975c0 msgr2=0x7f5df8097990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df659c640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df80975c0 0x7f5df8097990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5df806f410 tx=0x7f5de80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a24ef565558c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 1 -- 172.21.15.153:0/923581251 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de802adf0 con 0x7f5df80975c0 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: dump: 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.963 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: _finish_auth 0 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 5 monclient: authenticate success, global_id 214200 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 1 -- 172.21.15.153:0/923581251 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de8032440 con 0x7f5df80975c0 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:46.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 4 set_mon_vals no callback set 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 1 -- 172.21.15.153:0/923581251 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de80327e0 con 0x7f5df80975c0 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5df5d9b640 10 monclient: dump: 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.965 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 10 monclient: shutdown 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df80975c0 msgr2=0x7f5df8097990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df80975c0 0x7f5df8097990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5df806f410 tx=0x7f5de80090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/923581251 shutdown_connections 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/923581251 >> v2:172.21.15.181:3300/0 conn(0x7f5df812ee90 0x7f5df8131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df812ea30 0x7f5df8131950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/923581251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df80975c0 0x7f5df8097990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/923581251 >> 172.21.15.153:0/923581251 conn(0x7f5df8067b50 msgr2=0x7f5df80a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/923581251 shutdown_connections 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.955+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/923581251 wait complete. 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.956+0000 7f5dfcf9d640 10 monclient: build_initial_monmap 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.956+0000 7f5dfcf9d640 10 monclient: monmap: 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.956+0000 7f5dfcf9d640 1 Processor -- start 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.956+0000 7f5dfcf9d640 1 -- start start 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.956+0000 7f5dfcf9d640 10 monclient: init 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 0x7f5df809a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5df80975c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812ea30 0x7f5df812b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5df812ea30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df812ee90 0x7f5df8141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5df812ee90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df8130c80 con 0x7f5df812ea30 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df809ce40 con 0x7f5df812ee90 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df812c500 con 0x7f5df80975c0 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5dfcf9d640 10 monclient(hunting): authenticate will time out at 1804.596173s 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 0x7f5df809a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812ea30 0x7f5df812b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 0x7f5df809a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43830/0 (socket says 172.21.15.153:43830) 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 learned_addr learned my addr 172.21.15.153:0/957583580 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 10 monclient(hunting): get_auth_request con 0x7f5df80975c0 auth_method 0 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): get_auth_request con 0x7f5df812ea30 auth_method 0 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 msgr2=0x7f5df809a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 msgr2=0x7f5df809a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 0x7f5df809a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df659c640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 0x7f5df809a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df812ee90 0x7f5df8141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): get_auth_request con 0x7f5df812ee90 auth_method 0 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5df6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient(hunting): handle_auth_done global_id 214203 payload 824 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 msgr2=0x7f5df809a810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5df80975c0 0x7f5df809a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df812ee90 msgr2=0x7f5df8141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5df812ee90 0x7f5df8141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient: _finish_hunting 0 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 monclient: found mon.noname-a 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.957+0000 7f5deffff640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8009390 con 0x7f5df812ea30 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5deffff640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812ea30 msgr2=0x7f5df812b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5deffff640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812ea30 0x7f5df812b610 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5de0006f60 tx=0x7f5de0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de000c6d0 con 0x7f5df812ea30 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: dump: 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dfcf9d640 5 monclient: authenticate success, global_id 214203 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de000c830 con 0x7f5df812ea30 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dfcf9d640 10 monclient: _renew_subs 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dfcf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df8131d00 con 0x7f5df812ea30 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de000cbf0 con 0x7f5df812ea30 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5dee7fc640 10 monclient: dump: 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.979 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 10 monclient: _renew_subs 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.959+0000 7f5df559a640 4 set_mon_vals no callback set 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df8130430 con 0x7f5df812ea30 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 0x7f5df8141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df8063720 con 0x7f5df812fa70 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 0x7f5df8141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient: get_auth_request con 0x7f5df812fa70 auth_method 0 2024-09-15T03:46:46.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.960+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5de0013410 con 0x7f5df812ea30 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.961+0000 7f5df659c640 10 monclient(hunting): handle_auth_done global_id 214206 payload 824 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.961+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5df8132220 con 0x7f5df812fa70 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.961+0000 7f5dee7fc640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd8038740 0x7f5dd803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.961+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5de004c610 con 0x7f5df812ea30 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.961+0000 7f5df6d9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd8038740 0x7f5dd803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.961+0000 7f5df6d9d640 10 monclient: get_auth_request con 0x7f5dd8038740 auth_method 0 2024-09-15T03:46:46.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 msgr2=0x7f5df8141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5df659c640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 0x7f5df8141570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009130 tx=0x7f5de8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de803d980 con 0x7f5df812fa70 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 10 monclient: dump: 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5de803dae0 con 0x7f5df812fa70 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 msgr2=0x7f5df8141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 0x7f5df8141570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009130 tx=0x7f5de8032000 comp rx=0 tx=0).stop 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 msgr2=0x7f5df8141570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5dee7fc640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812fa70 0x7f5df8141570 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009130 tx=0x7f5de8032000 comp rx=0 tx=0).stop 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.963+0000 7f5df6d9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd8038740 0x7f5dd803abf0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5df00076e0 tx=0x7f5df0009990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5dfcf9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 0x7f5df8141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5dfcf9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:46.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df809b460 con 0x7f5df812bb50 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 0x7f5df8141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient: get_auth_request con 0x7f5df812bb50 auth_method 0 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 10 monclient(hunting): handle_auth_done global_id 214209 payload 824 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.969+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5df804bf60 con 0x7f5df812bb50 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5df659c640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 msgr2=0x7f5df8141ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5df659c640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 0x7f5df8141ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5df80950c0 tx=0x7f5de803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:46.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de804b070 con 0x7f5df812bb50 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 10 monclient: dump: 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f5de803c6c0 con 0x7f5df812bb50 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 msgr2=0x7f5df8141ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 0x7f5df8141ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5df80950c0 tx=0x7f5de803d000 comp rx=0 tx=0).stop 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 msgr2=0x7f5df8141ab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dee7fc640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812bb50 0x7f5df8141ab0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5df80950c0 tx=0x7f5de803d000 comp rx=0 tx=0).stop 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 190, 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:46.987 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:46.992 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:46.993 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:46.994 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:46.995 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:46.996 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:46.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:46.998 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:46.999 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:46:47.000 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:47.001 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:47.002 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd8038740 msgr2=0x7f5dd803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5dd8038740 0x7f5dd803abf0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5df00076e0 tx=0x7f5df0009990 comp rx=0 tx=0).stop 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 10 monclient: shutdown 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812ea30 msgr2=0x7f5df812b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 --2- 172.21.15.153:0/957583580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5df812ea30 0x7f5df812b610 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5de0006f60 tx=0x7f5de0008040 comp rx=0 tx=0).stop 2024-09-15T03:46:47.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5df6d9d640 1 -- 172.21.15.153:0/957583580 reap_dead start 2024-09-15T03:46:47.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5df6d9d640 1 -- 172.21.15.153:0/957583580 reap_dead start 2024-09-15T03:46:47.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 shutdown_connections 2024-09-15T03:46:47.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 >> 172.21.15.153:0/957583580 conn(0x7f5df8067b50 msgr2=0x7f5df80955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:47.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 shutdown_connections 2024-09-15T03:46:47.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:46.972+0000 7f5dfcf9d640 1 -- 172.21.15.153:0/957583580 wait complete. 2024-09-15T03:46:47.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:46:47.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:46:47.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:46:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.098+0000 7f0a0606f640 10 monclient: get_monmap_and_config 2024-09-15T03:46:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.098+0000 7f0a0606f640 10 monclient: build_initial_monmap 2024-09-15T03:46:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.098+0000 7f0a0606f640 10 monclient: monmap: 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:47.099533+0000 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:47.099533+0000 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:47.100 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.099+0000 7f0a0606f640 1 Processor -- start 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.099+0000 7f0a0606f640 1 -- start start 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.099+0000 7f0a0606f640 10 monclient: init 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.099+0000 7f0a0606f640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.099+0000 7f0a0606f640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0a0006eb80 0x7f0a0006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 10 monclient(hunting): picked mon.noname-c con 0x7f0a0006eb80 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a00068a20 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a0006f490 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00069b80 con 0x7f0a0006f490 2024-09-15T03:46:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00069e00 con 0x7f0a00068a20 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a00099f90 con 0x7f0a0006eb80 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f0a0606f640 10 monclient(hunting): authenticate will time out at 1804.739078s 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09fffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0a0006f490 0x7f0a00138ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44276/0 (socket says 172.21.15.153:44276) 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09fffff640 1 -- 172.21.15.153:0/2550920193 learned_addr learned my addr 172.21.15.153:0/2550920193 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/2550920193 >> v2:172.21.15.181:3300/0 conn(0x7f0a0006eb80 0x7f0a0006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request con 0x7f0a0006eb80 auth_method 0 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_done global_id 204262 payload 274 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 1 -- 172.21.15.153:0/2550920193 >> v1:172.21.15.181:6789/0 conn(0x7f0a00068a20 legacy=0x7f0a0009c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 1 -- 172.21.15.153:0/2550920193 >> v1:172.21.15.153:6789/0 conn(0x7f0a0006f490 legacy=0x7f0a00138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 1 monclient: found mon.noname-c 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.100+0000 7f09ff7fe640 1 -- 172.21.15.153:0/2550920193 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0009cc50 con 0x7f0a0006eb80 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09ff7fe640 1 -- 172.21.15.153:0/2550920193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006eb80 msgr2=0x7f0a0006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/2550920193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006eb80 0x7f0a0006ef50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009850 tx=0x7f09f00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ae33a6a0c958b0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 1 -- 172.21.15.153:0/2550920193 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0032440 con 0x7f0a0006eb80 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: dump: 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f0a0606f640 5 monclient: authenticate success, global_id 204262 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f0a0606f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 1 -- 172.21.15.153:0/2550920193 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f00325a0 con 0x7f0a0006eb80 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 1 -- 172.21.15.153:0/2550920193 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0032940 con 0x7f0a0006eb80 2024-09-15T03:46:47.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f09fe7fc640 10 monclient: dump: 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f0a0606f640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:47.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.103+0000 7f0a0606f640 4 set_mon_vals no callback set 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 10 monclient: shutdown 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 -- 172.21.15.153:0/2550920193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006eb80 msgr2=0x7f0a0006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 --2- 172.21.15.153:0/2550920193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006eb80 0x7f0a0006ef50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009850 tx=0x7f09f00090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 -- 172.21.15.153:0/2550920193 shutdown_connections 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 --2- 172.21.15.153:0/2550920193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006eb80 0x7f0a0006ef50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 -- 172.21.15.153:0/2550920193 >> 172.21.15.153:0/2550920193 conn(0x7f0a00067cd0 msgr2=0x7f0a000a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 -- 172.21.15.153:0/2550920193 shutdown_connections 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 1 -- 172.21.15.153:0/2550920193 wait complete. 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 10 monclient: build_initial_monmap 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.104+0000 7f0a0606f640 10 monclient: monmap: 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:47.115 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 Processor -- start 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 -- start start 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient: init 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:46:47.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 0x7f0a0011eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): picked mon.noname-c con 0x7f0a00068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 0x7f0a0011f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0006eb80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006f490 0x7f0a0011f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0006f490 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a001225b0 con 0x7f0a0006eb80 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0009cf40 con 0x7f0a0006f490 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000a2700 con 0x7f0a00068a20 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f0a0606f640 10 monclient(hunting): authenticate will time out at 1804.744905s 2024-09-15T03:46:47.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.105+0000 7f09feffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 0x7f0a0011f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 0x7f0a0011eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 0x7f0a0011f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42548/0 (socket says 172.21.15.153:42548) 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 0x7f0a0011eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43836/0 (socket says 172.21.15.153:43836) 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 -- 172.21.15.153:0/4276716942 learned_addr learned my addr 172.21.15.153:0/4276716942 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): get_auth_request con 0x7f0a0006eb80 auth_method 0 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request con 0x7f0a00068a20 auth_method 0 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:47.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 msgr2=0x7f0a0011eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 msgr2=0x7f0a0011eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 0x7f0a0011eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 0x7f0a0011eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09fffff640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006f490 0x7f0a0011f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09fffff640 10 monclient(hunting): get_auth_request con 0x7f0a0006f490 auth_method 0 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient(hunting): handle_auth_done global_id 214212 payload 824 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 msgr2=0x7f0a0011eb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:47.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a00068a20 0x7f0a0011eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006f490 msgr2=0x7f0a0011f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a0006f490 0x7f0a0011f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 monclient: found mon.noname-a 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.106+0000 7f09feffd640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f0009390 con 0x7f0a0006eb80 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09feffd640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 msgr2=0x7f0a0011f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09feffd640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 0x7f0a0011f0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09f400c5e0 tx=0x7f09f4007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4012840 con 0x7f0a0006eb80 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: dump: 2024-09-15T03:46:47.120 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: _finish_auth 0 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f40129a0 con 0x7f0a0006eb80 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 5 monclient: authenticate success, global_id 214212 2024-09-15T03:46:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 10 monclient: _renew_subs 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a00120940 con 0x7f0a0006eb80 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4012d60 con 0x7f0a0006eb80 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09e77fe640 10 monclient: dump: 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f09fdffb640 4 set_mon_vals no callback set 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 10 monclient: _renew_subs 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.108+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00121000 con 0x7f0a0006eb80 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f0a0606f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f0a0606f640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f0a0606f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00122100 con 0x7f0a0011fb30 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f09f4011590 con 0x7f0a0006eb80 2024-09-15T03:46:47.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 10 monclient: get_auth_request con 0x7f0a0011fb30 auth_method 0 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09e77fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09ec038740 0x7f09ec03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.109+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f09f404c7b0 con 0x7f0a0006eb80 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.110+0000 7f09fffff640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09ec038740 0x7f09ec03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.110+0000 7f09fffff640 10 monclient: get_auth_request con 0x7f09ec038740 auth_method 0 2024-09-15T03:46:47.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.112+0000 7f09fffff640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09ec038740 0x7f09ec03abf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0a0006f8d0 tx=0x7f09f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 10 monclient: get_auth_request con 0x7f0a0011fb30 auth_method 0 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:47.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:47.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.310+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 10 monclient: get_auth_request con 0x7f0a0011fb30 auth_method 0 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:46:47.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:47.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:47.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:47.711+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:46:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient: get_auth_request con 0x7f0a0011fb30 auth_method 0 2024-09-15T03:46:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:46:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_done global_id 224099 payload 824 2024-09-15T03:46:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.512+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a0004bf60 con 0x7f0a0011fb30 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f09f0034000 tx=0x7f09f0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f003c9f0 con 0x7f0a0011fb30 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 10 monclient: dump: 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f09f0041070 con 0x7f0a0011fb30 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f09f0034000 tx=0x7f09f0032000 comp rx=0 tx=0).stop 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 msgr2=0x7f0a0006b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.515+0000 7f09e77fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0011fb30 0x7f0a0006b840 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f0a0606f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f0a0606f640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 0x7f0a0006c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f0a0606f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a001222e0 con 0x7f0a0006bd80 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 0x7f0a0006c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient: get_auth_request con 0x7f0a0006bd80 auth_method 0 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 10 monclient(hunting): handle_auth_done global_id 224102 payload 824 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.522+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a0006e900 con 0x7f0a0006bd80 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.524+0000 7f09ff7fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 msgr2=0x7f0a0006c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.524+0000 7f09ff7fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 0x7f0a0006c180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003080 tx=0x7f09f003f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f003d800 con 0x7f0a0006bd80 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 10 monclient: dump: 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f09f003c7e0 con 0x7f0a0006bd80 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 msgr2=0x7f0a0006c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 0x7f0a0006c180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003080 tx=0x7f09f003f070 comp rx=0 tx=0).stop 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 msgr2=0x7f0a0006c180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09e77fe640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0006bd80 0x7f0a0006c180 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003080 tx=0x7f09f003f070 comp rx=0 tx=0).stop 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 190, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09ec038740 msgr2=0x7f09ec03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f09ec038740 0x7f09ec03abf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0a0006f8d0 tx=0x7f09f8009040 comp rx=0 tx=0).stop 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 10 monclient: shutdown 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 msgr2=0x7f0a0011f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 --2- 172.21.15.153:0/4276716942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0006eb80 0x7f0a0011f0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09f400c5e0 tx=0x7f09f4007ed0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09fffff640 1 -- 172.21.15.153:0/4276716942 reap_dead start 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f09fffff640 1 -- 172.21.15.153:0/4276716942 reap_dead start 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 shutdown_connections 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 >> 172.21.15.153:0/4276716942 conn(0x7f0a00067cd0 msgr2=0x7f0a0009d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 shutdown_connections 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.525+0000 7f0a0606f640 1 -- 172.21.15.153:0/4276716942 wait complete. 2024-09-15T03:46:48.544 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:46:48.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:46:48.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:46:48.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.646+0000 7f5054034640 10 monclient: get_monmap_and_config 2024-09-15T03:46:48.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.646+0000 7f5054034640 10 monclient: build_initial_monmap 2024-09-15T03:46:48.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.646+0000 7f5054034640 10 monclient: monmap: 2024-09-15T03:46:48.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:48.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:48.647186+0000 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:48.647186+0000 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.646+0000 7f5054034640 1 Processor -- start 2024-09-15T03:46:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 1 -- start start 2024-09-15T03:46:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient: init 2024-09-15T03:46:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:46:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f504c09c9c0 0x7f504c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): picked mon.noname-a con 0x7f504c09c9c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f504c09e340 0x7f504c09e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): picked mon.noname-c con 0x7f504c09e340 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f504c12f770 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c09bdd0 con 0x7f504c12f770 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c09ec90 con 0x7f504c09c9c0 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c09a880 con 0x7f504c09e340 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5054034640 10 monclient(hunting): authenticate will time out at 1806.286751s 2024-09-15T03:46:48.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f504c09c9c0 0x7f504c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f50525d1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f504c12f770 0x7f504c12fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44282/0 (socket says 172.21.15.153:44282) 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f504c09c9c0 0x7f504c09d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42574/0 (socket says 172.21.15.153:42574) 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f50525d1640 1 -- 172.21.15.153:0/772386611 learned_addr learned my addr 172.21.15.153:0/772386611 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 10 monclient(hunting): get_auth_request con 0x7f504c09c9c0 auth_method 0 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.647+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 10 monclient(hunting): handle_auth_done global_id 214215 payload 274 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 1 -- 172.21.15.153:0/772386611 >> v2:172.21.15.181:3300/0 conn(0x7f504c09e340 msgr2=0x7f504c09e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 1 --2- 172.21.15.153:0/772386611 >> v2:172.21.15.181:3300/0 conn(0x7f504c09e340 0x7f504c09e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 1 -- 172.21.15.153:0/772386611 >> v1:172.21.15.153:6789/0 conn(0x7f504c12f770 legacy=0x7f504c12fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 10 monclient: _finish_hunting 0 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 1 monclient: found mon.noname-a 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.648+0000 7f5051dd0640 1 -- 172.21.15.153:0/772386611 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c09a440 con 0x7f504c09c9c0 2024-09-15T03:46:48.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.650+0000 7f5051dd0640 1 -- 172.21.15.153:0/772386611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09c9c0 msgr2=0x7f504c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.650+0000 7f5051dd0640 1 --2- 172.21.15.153:0/772386611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09c9c0 0x7f504c09d830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f504c09b520 tx=0x7f50480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92709a03f35ef2e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 1 -- 172.21.15.153:0/772386611 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048032440 con 0x7f504c09c9c0 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: dump: 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: _finish_auth 0 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 1 -- 172.21.15.153:0/772386611 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50480325a0 con 0x7f504c09c9c0 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 5 monclient: authenticate success, global_id 214215 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 1 -- 172.21.15.153:0/772386611 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048032940 con 0x7f504c09c9c0 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5050dce640 10 monclient: dump: 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 4 set_mon_vals no callback set 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 10 monclient: shutdown 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 -- 172.21.15.153:0/772386611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09c9c0 msgr2=0x7f504c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 --2- 172.21.15.153:0/772386611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09c9c0 0x7f504c09d830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f504c09b520 tx=0x7f50480090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 -- 172.21.15.153:0/772386611 shutdown_connections 2024-09-15T03:46:48.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 --2- 172.21.15.153:0/772386611 >> v2:172.21.15.181:3300/0 conn(0x7f504c09e340 0x7f504c09e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 --2- 172.21.15.153:0/772386611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09c9c0 0x7f504c09d830 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 -- 172.21.15.153:0/772386611 >> 172.21.15.153:0/772386611 conn(0x7f504c0a2e40 msgr2=0x7f504c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 -- 172.21.15.153:0/772386611 shutdown_connections 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.651+0000 7f5054034640 1 -- 172.21.15.153:0/772386611 wait complete. 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.652+0000 7f5054034640 10 monclient: build_initial_monmap 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.652+0000 7f5054034640 10 monclient: monmap: 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:48.664 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.652+0000 7f5054034640 1 Processor -- start 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.652+0000 7f5054034640 1 -- start start 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.652+0000 7f5054034640 10 monclient: init 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c09c9c0 0x7f504c11bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): picked mon.noname-b con 0x7f504c09c9c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 0x7f504c11c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): picked mon.noname-a con 0x7f504c09e340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f504c12f770 0x7f504c11c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): picked mon.noname-c con 0x7f504c12f770 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:48.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c11f5a0 con 0x7f504c09e340 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c11e220 con 0x7f504c09c9c0 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c099cc0 con 0x7f504c12f770 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5054034640 10 monclient(hunting): authenticate will time out at 1806.292075s 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f504c12f770 0x7f504c11c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 0x7f504c11c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f504c12f770 0x7f504c11c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43904/0 (socket says 172.21.15.153:43904) 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 0x7f504c11c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42590/0 (socket says 172.21.15.153:42590) 2024-09-15T03:46:48.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 1 -- 172.21.15.153:0/3779602093 learned_addr learned my addr 172.21.15.153:0/3779602093 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): get_auth_request con 0x7f504c12f770 auth_method 0 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): get_auth_request con 0x7f504c09e340 auth_method 0 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c09c9c0 0x7f504c11bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50525d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5051dd0640 10 monclient(hunting): get_auth_request con 0x7f504c09c9c0 auth_method 0 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5051dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5051dd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5051dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient(hunting): handle_auth_done global_id 214218 payload 824 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f504c12f770 msgr2=0x7f504c11c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f504c12f770 0x7f504c11c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c09c9c0 msgr2=0x7f504c11bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c09c9c0 0x7f504c11bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient: _finish_hunting 0 2024-09-15T03:46:48.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 monclient: found mon.noname-a 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f50515cf640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048009390 con 0x7f504c09e340 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.653+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c09c9c0 0x7f504c11bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f50515cf640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 msgr2=0x7f504c11c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f50515cf640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 0x7f504c11c3e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b560 tx=0x7f503c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c00f720 con 0x7f504c09e340 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: dump: 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: _finish_auth 0 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c00f880 con 0x7f504c09e340 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5054034640 5 monclient: authenticate success, global_id 214218 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c00fc40 con 0x7f504c09e340 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5054034640 10 monclient: _renew_subs 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5054034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c11d800 con 0x7f504c09e340 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5043fff640 4 set_mon_vals no callback set 2024-09-15T03:46:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.655+0000 7f5041ffb640 10 monclient: dump: 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 10 monclient: _renew_subs 2024-09-15T03:46:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504c11f370 con 0x7f504c09e340 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f503c011610 con 0x7f504c09e340 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.656+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c065e00 con 0x7f504c11ce60 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5041ffb640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5030038770 0x7f503003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f503c04c0c0 con 0x7f504c09e340 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient: get_auth_request con 0x7f504c11ce60 auth_method 0 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f50525d1640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5030038770 0x7f503003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f50525d1640 10 monclient: get_auth_request con 0x7f5030038770 auth_method 0 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 10 monclient(hunting): handle_auth_done global_id 204271 payload 824 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f504c11f7e0 con 0x7f504c11ce60 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 msgr2=0x7f504c1194a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 msgr2=0x7f504c1194a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f5048034000 tx=0x7f5048032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:46:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.657+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:48.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.659+0000 7f50525d1640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5030038770 0x7f503003ac20 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5044006e30 tx=0x7f5044005370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: get_auth_request con 0x7f504c11ce60 auth_method 0 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: _init_auth method 2 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: _init_auth already have auth, reseting 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: handle_auth_reply_more payload 9 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-15T03:46:48.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.858+0000 7f5051dd0640 10 monclient: handle_auth_done global_id 204271 payload 293 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.860+0000 7f5051dd0640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 msgr2=0x7f504c1194a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.860+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f504803e000 tx=0x7f504803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504803cd60 con 0x7f504c11ce60 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 10 monclient: dump: 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.862 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5048047440 con 0x7f504c11ce60 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 msgr2=0x7f504c1194a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f504803e000 tx=0x7f504803d000 comp rx=0 tx=0).stop 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 msgr2=0x7f504c1194a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:48.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.861+0000 7f5041ffb640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c11ce60 0x7f504c1194a0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f504803e000 tx=0x7f504803d000 comp rx=0 tx=0).stop 2024-09-15T03:46:48.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5054034640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:48.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5054034640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 0x7f504c119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5054034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c0990f0 con 0x7f504c1199e0 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 0x7f504c119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient: get_auth_request con 0x7f504c1199e0 auth_method 0 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:48.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.867+0000 7f5051dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:48.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.868+0000 7f5051dd0640 10 monclient(hunting): handle_auth_done global_id 204274 payload 824 2024-09-15T03:46:48.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.868+0000 7f5051dd0640 1 -- 172.21.15.153:0/3779602093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f504c04bf60 con 0x7f504c1199e0 2024-09-15T03:46:48.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.870+0000 7f5051dd0640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 msgr2=0x7f504c119de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:48.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.870+0000 7f5051dd0640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 0x7f504c119de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5048009130 tx=0x7f5048047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50480509d0 con 0x7f504c1199e0 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 10 monclient: dump: 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:48.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f504805b420 con 0x7f504c1199e0 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:48.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 msgr2=0x7f504c119de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 0x7f504c119de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5048009130 tx=0x7f5048047000 comp rx=0 tx=0).stop 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 msgr2=0x7f504c119de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5041ffb640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f504c1199e0 0x7f504c119de0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5048009130 tx=0x7f5048047000 comp rx=0 tx=0).stop 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 190, 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:48.875 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:46:48.875 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:46:48.875 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.875 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:46:48.875 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:48.875 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:48.876 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:48.877 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:48.878 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:48.878 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:48.878 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:48.878 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:48.878 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:48.878 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:48.880 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:48.880 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:48.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:48.880 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:48.880 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:48.881 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:48.881 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:48.881 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:48.881 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.881 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:48.882 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:48.882 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:48.882 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:48.882 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:48.882 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:48.883 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:48.883 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:48.883 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:48.883 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:48.883 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:48.883 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:48.884 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.885 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.886 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.887 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.888 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:48.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:48.888 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:48.888 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:48.888 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.888 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:48.889 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:48.889 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.889 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:48.889 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:48.890 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:48.892 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:48.894 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:48.895 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:48.896 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5030038770 msgr2=0x7f503003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5030038770 0x7f503003ac20 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5044006e30 tx=0x7f5044005370 comp rx=0 tx=0).stop 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 10 monclient: shutdown 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 msgr2=0x7f504c11c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 --2- 172.21.15.153:0/3779602093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f504c09e340 0x7f504c11c3e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b560 tx=0x7f503c009120 comp rx=0 tx=0).stop 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f50525d1640 1 -- 172.21.15.153:0/3779602093 reap_dead start 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f50525d1640 1 -- 172.21.15.153:0/3779602093 reap_dead start 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 shutdown_connections 2024-09-15T03:46:48.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 >> 172.21.15.153:0/3779602093 conn(0x7f504c0a2e40 msgr2=0x7f504c0975a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:48.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 shutdown_connections 2024-09-15T03:46:48.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.871+0000 7f5054034640 1 -- 172.21.15.153:0/3779602093 wait complete. 2024-09-15T03:46:48.899 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:46:48.899 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:46:48.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:46:49.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.992+0000 7f0efd754640 10 monclient: get_monmap_and_config 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.992+0000 7f0efd754640 10 monclient: build_initial_monmap 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.992+0000 7f0efd754640 10 monclient: monmap: 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:48.993533+0000 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:48.993533+0000 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:49.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 Processor -- start 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 -- start start 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient: init 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 0x7f0ef809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef809c860 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f0ef809aba0 0x7f0ef809d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef809aba0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:49.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef809dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef809afa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef809a9d0 con 0x7f0ef809c860 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef806a320 con 0x7f0ef809aba0 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8068a20 con 0x7f0ef809afa0 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.993+0000 7f0efd754640 10 monclient(hunting): authenticate will time out at 1806.632945s 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 0x7f0ef809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef809dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 0x7f0ef809cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42600/0 (socket says 172.21.15.153:42600) 2024-09-15T03:46:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 1 -- 172.21.15.153:0/2938947192 learned_addr learned my addr 172.21.15.153:0/2938947192 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): get_auth_request con 0x7f0ef809c860 auth_method 0 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request con 0x7f0ef809afa0 auth_method 0 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:49.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef67fc640 1 --2- 172.21.15.153:0/2938947192 >> v2:172.21.15.181:3300/0 conn(0x7f0ef809aba0 0x7f0ef809d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef67fc640 10 monclient(hunting): get_auth_request con 0x7f0ef809aba0 auth_method 0 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_done global_id 224108 payload 274 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 -- 172.21.15.153:0/2938947192 >> v2:172.21.15.181:3300/0 conn(0x7f0ef809aba0 msgr2=0x7f0ef809d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 --2- 172.21.15.153:0/2938947192 >> v2:172.21.15.181:3300/0 conn(0x7f0ef809aba0 0x7f0ef809d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 -- 172.21.15.153:0/2938947192 >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 msgr2=0x7f0ef809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 --2- 172.21.15.153:0/2938947192 >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 0x7f0ef809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 monclient: found mon.noname-b 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:49.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef77fe640 1 -- 172.21.15.153:0/2938947192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef8068e60 con 0x7f0ef809afa0 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.994+0000 7f0ef6ffd640 1 --2- 172.21.15.153:0/2938947192 >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 0x7f0ef809cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef77fe640 1 -- 172.21.15.153:0/2938947192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 msgr2=0x7f0ef809dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef77fe640 1 --2- 172.21.15.153:0/2938947192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef809dd70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ef806b700 tx=0x7f0ef0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481847b6c3b86ecd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 1 -- 172.21.15.153:0/2938947192 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0007760 con 0x7f0ef809afa0 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: dump: 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 1 -- 172.21.15.153:0/2938947192 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef002d050 con 0x7f0ef809afa0 2024-09-15T03:46:49.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 5 monclient: authenticate success, global_id 224108 2024-09-15T03:46:49.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:49.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:49.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:49.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 4 set_mon_vals no callback set 2024-09-15T03:46:49.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 1 -- 172.21.15.153:0/2938947192 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0007af0 con 0x7f0ef809afa0 2024-09-15T03:46:49.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0ef5ffb640 10 monclient: dump: 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:49.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 10 monclient: shutdown 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 -- 172.21.15.153:0/2938947192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 msgr2=0x7f0ef809dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 --2- 172.21.15.153:0/2938947192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef809dd70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ef806b700 tx=0x7f0ef0009780 comp rx=0 tx=0).stop 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 -- 172.21.15.153:0/2938947192 shutdown_connections 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 --2- 172.21.15.153:0/2938947192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef809dd70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 --2- 172.21.15.153:0/2938947192 >> v2:172.21.15.181:3300/0 conn(0x7f0ef809aba0 0x7f0ef809d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 --2- 172.21.15.153:0/2938947192 >> v2:172.21.15.153:3300/0 conn(0x7f0ef809c860 0x7f0ef809cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 -- 172.21.15.153:0/2938947192 >> 172.21.15.153:0/2938947192 conn(0x7f0ef80a2e40 msgr2=0x7f0ef80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.997+0000 7f0efd754640 1 -- 172.21.15.153:0/2938947192 shutdown_connections 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.998+0000 7f0efd754640 1 -- 172.21.15.153:0/2938947192 wait complete. 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.998+0000 7f0efd754640 10 monclient: build_initial_monmap 2024-09-15T03:46:49.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.998+0000 7f0efd754640 10 monclient: monmap: 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:49.017 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.998+0000 7f0efd754640 1 Processor -- start 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.998+0000 7f0efd754640 1 -- start start 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.998+0000 7f0efd754640 10 monclient: init 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 0x7f0ef813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef809aba0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef813cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef809afa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c860 0x7f0ef8140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef809c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8121f10 con 0x7f0ef809aba0 2024-09-15T03:46:49.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8099cc0 con 0x7f0ef809c860 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef80a1050 con 0x7f0ef809afa0 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0efd754640 10 monclient(hunting): authenticate will time out at 1806.638404s 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 0x7f0ef813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef813cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 0x7f0ef813d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42610/0 (socket says 172.21.15.153:42610) 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef813cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43926/0 (socket says 172.21.15.153:43926) 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 -- 172.21.15.153:0/1939020743 learned_addr learned my addr 172.21.15.153:0/1939020743 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): get_auth_request con 0x7f0ef809aba0 auth_method 0 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): get_auth_request con 0x7f0ef809afa0 auth_method 0 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef77fe640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c860 0x7f0ef8140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request con 0x7f0ef809c860 auth_method 0 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient(hunting): handle_auth_done global_id 214224 payload 824 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 -- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 msgr2=0x7f0ef813cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef813cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 -- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c860 msgr2=0x7f0ef8140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c860 0x7f0ef8140a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef6ffd640 1 -- 172.21.15.153:0/1939020743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8009450 con 0x7f0ef809aba0 2024-09-15T03:46:49.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef77fe640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c860 0x7f0ef8140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:48.999+0000 7f0ef67fc640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef813cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0ef6ffd640 1 -- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 msgr2=0x7f0ef813d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0ef6ffd640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 0x7f0ef813d4e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800ecb0 tx=0x7f0ee80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8009730 con 0x7f0ef809aba0 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: dump: 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:49.022 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: _finish_auth 0 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee800cdd0 con 0x7f0ef809aba0 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 5 monclient: authenticate success, global_id 214224 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 10 monclient: _renew_subs 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:49.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef0009a20 con 0x7f0ef809aba0 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80136a0 con 0x7f0ef809aba0 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0edeffd640 10 monclient: dump: 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 10 monclient: _renew_subs 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0ef57fa640 4 set_mon_vals no callback set 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.002+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef8121c30 con 0x7f0ef809aba0 2024-09-15T03:46:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0ee8013a50 con 0x7f0ef809aba0 2024-09-15T03:46:49.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0efd754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:49.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0efd754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:49.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0efd754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:49.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef804bf60 con 0x7f0ef809aba0 2024-09-15T03:46:49.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0edeffd640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ee4038740 0x7f0ee403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:49.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0ee804cad0 con 0x7f0ef809aba0 2024-09-15T03:46:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0ef67fc640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ee4038740 0x7f0ee403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.003+0000 7f0ef67fc640 10 monclient: get_auth_request con 0x7f0ee4038740 auth_method 0 2024-09-15T03:46:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.006+0000 7f0ef67fc640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ee4038740 0x7f0ee403abf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0ef811fde0 tx=0x7f0eec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.007+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0ee80123d0 con 0x7f0ef809aba0 2024-09-15T03:46:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.007+0000 7f0edeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.007+0000 7f0edeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.221+0000 7f0efd754640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:46:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.221+0000 7f0efd754640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:46:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.221+0000 7f0efd754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.221+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0ef80638d0 con 0x7f0ef809aba0 2024-09-15T03:46:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f0edeffd640 1 -- 172.21.15.153:0/1939020743 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0ee8011e00 con 0x7f0ef809aba0 2024-09-15T03:46:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f0edeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:46:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f0edeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:49.223 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.223 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=216726818,config=3140514009,health=62781586,kv=3235807718,logm=3323389701}) 2024-09-15T03:46:49.224 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=11,kv=7,logm=49} crc {auth=4257270871,config=3140514009,health=2895704524,kv=581790462,logm=180984965}) 2024-09-15T03:46:49.224 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.224 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=216726818,config=3140514009,health=62781586,kv=3235807718,logm=3323389701}) 2024-09-15T03:46:49.224 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=2083960889,config=3140514009,health=62781586,kv=852659224,logm=3783391969}) 2024-09-15T03:46:49.225 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.225 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885253293}) 2024-09-15T03:46:49.225 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387159069}) 2024-09-15T03:46:49.225 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.225 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885253293}) 2024-09-15T03:46:49.225 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3535716359}) 2024-09-15T03:46:49.226 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.226 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370737160}) 2024-09-15T03:46:49.226 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=460261661}) 2024-09-15T03:46:49.226 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370737160}) 2024-09-15T03:46:49.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137615323}) 2024-09-15T03:46:49.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ee4038740 msgr2=0x7f0ee403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ee4038740 0x7f0ee403abf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0ef811fde0 tx=0x7f0eec008040 comp rx=0 tx=0).stop 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 10 monclient: shutdown 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 msgr2=0x7f0ef813d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 0x7f0ef813d4e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800ecb0 tx=0x7f0ee80091d0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 shutdown_connections 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0ee4038740 0x7f0ee403abf0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef809c860 0x7f0ef8140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef809afa0 0x7f0ef813cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 --2- 172.21.15.153:0/1939020743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef809aba0 0x7f0ef813d4e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:49.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 >> 172.21.15.153:0/1939020743 conn(0x7f0ef80a2e40 msgr2=0x7f0ef8098a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:49.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 shutdown_connections 2024-09-15T03:46:49.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:49.225+0000 7f0efd754640 1 -- 172.21.15.153:0/1939020743 wait complete. 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.226+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.226+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823471629}) 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.226+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3042964625}) 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.226+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.226+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823471629}) 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.226+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769160674}) 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.228+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.228+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132756966}) 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.228+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270993807}) 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.228+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.228+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132756966}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.228+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1621141535}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.229+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.229+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705960042}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.229+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045122557}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.229+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.229+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705960042}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.229+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126427712}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.230+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.230+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485797674}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.230+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494829726}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.230+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.230+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485797674}) 2024-09-15T03:46:49.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.230+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=902726038}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.231+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.231+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488760488}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.231+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2228573193}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.231+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.231+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488760488}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.231+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043175826}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281373181}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3359656903}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281373181}) 2024-09-15T03:46:49.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=262237376}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230075087}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=22716251}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230075087}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.232+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091089096}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.233+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.233+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003797830}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.233+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791451984}) 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.233+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.233+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003797830}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.233+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399097040}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167896263}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2726075898}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167896263}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078650083}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=629690051}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2876316568}) 2024-09-15T03:46:49.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=629690051}) 2024-09-15T03:46:49.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2555652589}) 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=324309925}) 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644115989}) 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=324309925}) 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.235+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650081264}) 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.236+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.236+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652655546}) 2024-09-15T03:46:49.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.236+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968913190}) 2024-09-15T03:46:49.241 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.236+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.241 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.236+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652655546}) 2024-09-15T03:46:49.241 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.236+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699687405}) 2024-09-15T03:46:49.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.237+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.237+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933693384}) 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.237+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221263617}) 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.237+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.237+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933693384}) 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.237+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=445712185}) 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.238+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.252 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.238+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276955417}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.238+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546961421}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.238+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.238+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276955417}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.238+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270992263}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881230590}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619475179}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881230590}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815151958}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195172512}) 2024-09-15T03:46:49.253 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3377779149}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195172512}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.239+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269467410}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.240+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.240+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3098447790}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.240+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2880001090}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.240+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.240+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3098447790}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.240+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537821348}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.241+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.241+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198273565}) 2024-09-15T03:46:49.254 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.241+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810628357}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.241+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.241+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198273565}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.241+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=862079405}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502727720}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551907697}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502727720}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1965500259}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534472806}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053009341}) 2024-09-15T03:46:49.255 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534472806}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.242+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292825022}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.243+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.243+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277965587}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.243+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053268956}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.243+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.243+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277965587}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.243+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098859716}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1924150045}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551984351}) 2024-09-15T03:46:49.256 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1924150045}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688449982}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833319232}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169095288}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833319232}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.244+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568198100}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.245+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.245+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274714626}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.245+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4219375181}) 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.245+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.257 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.245+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274714626}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.245+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=837568738}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416148474}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1822205913}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416148474}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4155405265}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723132418}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751061705}) 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.258 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723132418}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.246+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1666041431}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.247+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.247+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708172621}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.247+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979919650}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.247+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.247+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708172621}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.247+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=974288698}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.248+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.248+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994122421}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.248+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887810238}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.248+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.248+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994122421}) 2024-09-15T03:46:49.259 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.248+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4163197149}) 2024-09-15T03:46:49.317 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.316+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.317 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.316+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267536741}) 2024-09-15T03:46:49.318 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.316+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1697202549}) 2024-09-15T03:46:49.318 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.316+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.318 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.316+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267536741}) 2024-09-15T03:46:49.318 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.316+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712228965}) 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.317+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.317+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129338973}) 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.317+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2741982947}) 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.317+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.317+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129338973}) 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.317+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718510175}) 2024-09-15T03:46:49.319 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.318+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.318+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081779225}) 2024-09-15T03:46:49.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.318+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464282381}) 2024-09-15T03:46:49.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.318+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.318+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081779225}) 2024-09-15T03:46:49.320 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.318+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511605167}) 2024-09-15T03:46:49.322 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.319+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.319+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275588574}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.319+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635440912}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.319+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.319+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275588574}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.319+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1165951148}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650171407}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3464985112}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650171407}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3705649777}) 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.323 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043179520}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219141956}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043179520}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.320+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1532731208}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.321+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.321+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329666996}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.321+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2226647435}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.321+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.321+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329666996}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.321+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2347626068}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455828696}) 2024-09-15T03:46:49.324 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365936086}) 2024-09-15T03:46:49.325 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.325 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455828696}) 2024-09-15T03:46:49.325 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210738411}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673338826}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801160545}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673338826}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.322+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375348710}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.323+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.323+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891420820}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.323+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505170405}) 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.323+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.328 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.323+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891420820}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.323+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2231111118}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.324+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.324+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776470412}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.324+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899934701}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.324+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.324+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776470412}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.324+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546274596}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535523323}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802619969}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535523323}) 2024-09-15T03:46:49.329 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751513674}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325708546}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157118817}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325708546}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.325+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382741751}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.326+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.326+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952847237}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.326+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1344005354}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.326+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.326+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952847237}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.326+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=891275710}) 2024-09-15T03:46:49.330 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.327+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.331 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.327+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763519818}) 2024-09-15T03:46:49.331 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.327+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482294155}) 2024-09-15T03:46:49.331 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.327+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.331 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.327+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763519818}) 2024-09-15T03:46:49.331 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.327+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577394021}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027263403}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283100155}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027263403}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2324034509}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409317959}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246652002}) 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.333 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409317959}) 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.328+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950755720}) 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.329+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.329+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817018245}) 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.329+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1370728069}) 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.329+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.329+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817018245}) 2024-09-15T03:46:49.334 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.329+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=365261509}) 2024-09-15T03:46:49.685 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.684+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.684+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549554464}) 2024-09-15T03:46:49.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.684+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521274173}) 2024-09-15T03:46:49.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.684+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.684+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549554464}) 2024-09-15T03:46:49.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.684+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2197485743}) 2024-09-15T03:46:49.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239615893}) 2024-09-15T03:46:49.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488303555}) 2024-09-15T03:46:49.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239615893}) 2024-09-15T03:46:49.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=601047152}) 2024-09-15T03:46:49.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874404227}) 2024-09-15T03:46:49.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=629794474}) 2024-09-15T03:46:49.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874404227}) 2024-09-15T03:46:49.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.905+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=426202868}) 2024-09-15T03:46:49.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.906+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.906+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025526497}) 2024-09-15T03:46:49.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.906+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442480043}) 2024-09-15T03:46:49.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.906+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.906+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025526497}) 2024-09-15T03:46:49.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.906+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332495796}) 2024-09-15T03:46:49.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.907+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.907+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813923595}) 2024-09-15T03:46:49.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.907+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619344170}) 2024-09-15T03:46:49.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.907+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.907+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813923595}) 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.907+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110553930}) 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.908+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.908+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663981375}) 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.908+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550781672}) 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.908+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.908+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663981375}) 2024-09-15T03:46:49.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.908+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404050983}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.909+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.909+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464403132}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.909+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=970091996}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.909+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.909+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464403132}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.909+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=868775118}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.910+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.910+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439103359}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.910+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111976314}) 2024-09-15T03:46:49.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.910+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.910+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439103359}) 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.910+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2731391178}) 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.911+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.911+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312349959}) 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.911+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622481210}) 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.911+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.911+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312349959}) 2024-09-15T03:46:49.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.911+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480243748}) 2024-09-15T03:46:49.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943137249}) 2024-09-15T03:46:49.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076905183}) 2024-09-15T03:46:49.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943137249}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769428569}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396182255}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.912+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826148442}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396182255}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=478609473}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009198887}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615593248}) 2024-09-15T03:46:49.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009198887}) 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.913+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894853445}) 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.914+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.914+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575559794}) 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.914+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975468248}) 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.914+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.914+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575559794}) 2024-09-15T03:46:49.917 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.914+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866962389}) 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.915+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.915+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673722103}) 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.915+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3910273400}) 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.915+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.915+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673722103}) 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.915+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3256893490}) 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145922892}) 2024-09-15T03:46:49.921 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170086346}) 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145922892}) 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481415835}) 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458029322}) 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4133051101}) 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.916+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458029322}) 2024-09-15T03:46:49.922 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945255796}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341341468}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872196769}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341341468}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.917+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414215046}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.918+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.918+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74090300}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.918+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972638909}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.918+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.918+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74090300}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.918+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780262779}) 2024-09-15T03:46:49.923 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494906001}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282732634}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494906001}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530683017}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160972198}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=319532025}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160972198}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.919+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116390831}) 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.920+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.924 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.920+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38301943}) 2024-09-15T03:46:49.925 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.920+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301984367}) 2024-09-15T03:46:49.925 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.920+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.925 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.920+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38301943}) 2024-09-15T03:46:49.925 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.920+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155676087}) 2024-09-15T03:46:49.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.921+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.921+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=105401901}) 2024-09-15T03:46:49.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.921+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3636696315}) 2024-09-15T03:46:49.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.921+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.921+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=105401901}) 2024-09-15T03:46:49.934 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.921+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386829958}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116711077}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589686276}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116711077}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272994140}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860466655}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=768311404}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860466655}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.922+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098566952}) 2024-09-15T03:46:49.935 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.923+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.923+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698231293}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.923+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348600444}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.923+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.923+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698231293}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.923+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3838193027}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159046112}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327478712}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159046112}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1601055424}) 2024-09-15T03:46:49.936 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338413965}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509822492}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338413965}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.924+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325635133}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.925+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.925+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254788136}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.925+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1241524460}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.925+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.925+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254788136}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.925+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080459967}) 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.937 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329947613}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646607581}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329947613}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3071594969}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225188883}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2232555335}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225188883}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.926+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235149417}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.927+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.927+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110367667}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.927+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792008255}) 2024-09-15T03:46:49.938 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.927+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.927+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110367667}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.927+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244799360}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.928+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.928+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=671017934}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.928+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=499203985}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.928+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.928+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=671017934}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.928+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680427119}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849086434}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456070202}) 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.939 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849086434}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3598821221}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007647200}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2013647582}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007647200}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.929+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437094411}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.930+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.930+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701883163}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.930+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119520338}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.930+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.930+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701883163}) 2024-09-15T03:46:49.940 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.930+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840126711}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989114507}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224664366}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989114507}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520239184}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597448963}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2887033847}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597448963}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.931+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624501555}) 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.932+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.941 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.932+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286444759}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.932+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=965869857}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.932+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.932+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286444759}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.932+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127258877}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629222011}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039126307}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629222011}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708578430}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486261855}) 2024-09-15T03:46:49.942 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623009897}) 2024-09-15T03:46:49.943 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.943 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486261855}) 2024-09-15T03:46:49.943 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.933+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068949440}) 2024-09-15T03:46:49.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.934+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.934+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568690188}) 2024-09-15T03:46:49.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.934+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561985585}) 2024-09-15T03:46:49.944 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.934+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.934+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568690188}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.934+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281312328}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.935+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.935+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=578721991,mdsmap=1531151339,mgr=1360396465,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3314334110,mon_config_key=4117641549,monmap=244636553,osd_metadata=540015074,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3191595281}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.935+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=46} crc {logm=578721991,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=2197403422,mon_config_key=4117641549,monmap=244636553,osd_metadata=4072215849,osd_pg_creating=1754493876,osd_snap=2673976383,osdmap=4042042337}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.935+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.935+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=578721991,mdsmap=1531151339,mgr=1360396465,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3314334110,mon_config_key=4117641549,monmap=244636553,osd_metadata=540015074,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3191595281}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.935+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=2885562317,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgrstat=2822596480,mon_config_key=4117641549,monmap=244636553,osd_metadata=1633747541,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=2972863930}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.940+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.940+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=7} crc {osdmap=2045476292}) 2024-09-15T03:46:49.945 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:46:49.940+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=7} crc {osdmap=2045476293}) 2024-09-15T03:46:50.242 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:46:50.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:50.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.334+0000 7efd9d754640 10 monclient: get_monmap_and_config 2024-09-15T03:46:50.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.334+0000 7efd9d754640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.334+0000 7efd9d754640 10 monclient: monmap: 2024-09-15T03:46:50.335 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.335 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:50.335425+0000 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:50.335425+0000 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:50.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 Processor -- start 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 -- start start 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient: init 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd9806eb80 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 0x7efd9809c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): picked mon.noname-b con 0x7efd98068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7efd9809ca80 0x7efd98138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): picked mon.noname-c con 0x7efd9809ca80 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd98069b80 con 0x7efd9806eb80 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd98099f90 con 0x7efd9809ca80 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd98069e00 con 0x7efd98068a20 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd9d754640 10 monclient(hunting): authenticate will time out at 1807.974762s 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 0x7efd9809c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 0x7efd9809c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43934/0 (socket says 172.21.15.153:43934) 2024-09-15T03:46:50.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 1 -- 172.21.15.153:0/334388665 learned_addr learned my addr 172.21.15.153:0/334388665 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): get_auth_request con 0x7efd98068a20 auth_method 0 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.335+0000 7efd967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 10 monclient(hunting): handle_auth_done global_id 224114 payload 274 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 1 -- 172.21.15.153:0/334388665 >> v2:172.21.15.181:3300/0 conn(0x7efd9809ca80 msgr2=0x7efd98138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd977fe640 1 --2- 172.21.15.153:0/334388665 >> v2:172.21.15.181:3300/0 conn(0x7efd9809ca80 0x7efd98138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 1 --2- 172.21.15.153:0/334388665 >> v2:172.21.15.181:3300/0 conn(0x7efd9809ca80 0x7efd98138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 1 -- 172.21.15.153:0/334388665 >> v1:172.21.15.181:6789/0 conn(0x7efd9806eb80 legacy=0x7efd9806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 1 monclient: found mon.noname-b 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.336+0000 7efd967fc640 1 -- 172.21.15.153:0/334388665 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9806f660 con 0x7efd98068a20 2024-09-15T03:46:50.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.338+0000 7efd967fc640 1 -- 172.21.15.153:0/334388665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 msgr2=0x7efd9809c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd967fc640 1 --2- 172.21.15.153:0/334388665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 0x7efd9809c540 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009910 tx=0x7efd8c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78435356b71d5fc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 1 -- 172.21.15.153:0/334388665 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c032440 con 0x7efd98068a20 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: dump: 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 1 -- 172.21.15.153:0/334388665 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c0325a0 con 0x7efd98068a20 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 1 -- 172.21.15.153:0/334388665 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c032940 con 0x7efd98068a20 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd95ffb640 10 monclient: dump: 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 5 monclient: authenticate success, global_id 224114 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 4 set_mon_vals no callback set 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 10 monclient: shutdown 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 1 -- 172.21.15.153:0/334388665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 msgr2=0x7efd9809c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 1 --2- 172.21.15.153:0/334388665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 0x7efd9809c540 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009910 tx=0x7efd8c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 1 -- 172.21.15.153:0/334388665 shutdown_connections 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 1 --2- 172.21.15.153:0/334388665 >> v2:172.21.15.181:3300/0 conn(0x7efd9809ca80 0x7efd98138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 1 --2- 172.21.15.153:0/334388665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd98068a20 0x7efd9809c540 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.339+0000 7efd9d754640 1 -- 172.21.15.153:0/334388665 >> 172.21.15.153:0/334388665 conn(0x7efd98067cd0 msgr2=0x7efd980a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 1 -- 172.21.15.153:0/334388665 shutdown_connections 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 1 -- 172.21.15.153:0/334388665 wait complete. 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 10 monclient: monmap: 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 1 Processor -- start 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 1 -- start start 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.340+0000 7efd9d754640 10 monclient: init 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd98068a20 0x7efd9811e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): picked mon.noname-a con 0x7efd98068a20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd9806eb80 0x7efd9811ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): picked mon.noname-b con 0x7efd9806eb80 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 0x7efd9811ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): picked mon.noname-c con 0x7efd9809ca80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd98121b40 con 0x7efd98068a20 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd98120c50 con 0x7efd9806eb80 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd98120600 con 0x7efd9809ca80 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd9d754640 10 monclient(hunting): authenticate will time out at 1807.980400s 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 0x7efd9811ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd98068a20 0x7efd9811e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 0x7efd9811ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43942/0 (socket says 172.21.15.153:43942) 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 -- 172.21.15.153:0/810068470 learned_addr learned my addr 172.21.15.153:0/810068470 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): get_auth_request con 0x7efd9809ca80 auth_method 0 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): get_auth_request con 0x7efd98068a20 auth_method 0 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd967fc640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd9806eb80 0x7efd9811ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd967fc640 10 monclient(hunting): get_auth_request con 0x7efd9806eb80 auth_method 0 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient(hunting): handle_auth_done global_id 224117 payload 824 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 -- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd9806eb80 msgr2=0x7efd9811ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd9806eb80 0x7efd9811ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 -- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd98068a20 msgr2=0x7efd9811e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd98068a20 0x7efd9811e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 monclient: found mon.noname-c 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd977fe640 1 -- 172.21.15.153:0/810068470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c009450 con 0x7efd9809ca80 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.341+0000 7efd96ffd640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd98068a20 0x7efd9811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd977fe640 1 -- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 msgr2=0x7efd9811ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd977fe640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 0x7efd9811ef70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efd9000cee0 tx=0x7efd90007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000e610 con 0x7efd9809ca80 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: dump: 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9000e770 con 0x7efd9809ca80 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000eb30 con 0x7efd9809ca80 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 5 monclient: authenticate success, global_id 224117 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 10 monclient: _renew_subs 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9809b670 con 0x7efd9809ca80 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd7effd640 10 monclient: dump: 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd957fa640 4 set_mon_vals no callback set 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 10 monclient: _renew_subs 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.344+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd98138d60 con 0x7efd9809ca80 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efd90009030 con 0x7efd9809ca80 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd9d754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd9d754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd9d754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd98121e80 con 0x7efd9809ca80 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd7effd640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd84038740 0x7efd8403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efd90012070 con 0x7efd9809ca80 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd96ffd640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd84038740 0x7efd8403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.345+0000 7efd96ffd640 10 monclient: get_auth_request con 0x7efd84038740 auth_method 0 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.348+0000 7efd96ffd640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd84038740 0x7efd8403abf0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7efd88007fd0 tx=0x7efd88007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.349+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efd98121e80 con 0x7efd9809ca80 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.349+0000 7efd7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.349+0000 7efd7effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:50.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.563+0000 7efd9d754640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:50.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.563+0000 7efd9d754640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:50.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.563+0000 7efd9d754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.563+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efd98121e80 con 0x7efd9809ca80 2024-09-15T03:46:50.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.564+0000 7efd7effd640 1 -- 172.21.15.153:0/810068470 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7efd98121e80 con 0x7efd9809ca80 2024-09-15T03:46:50.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.564+0000 7efd7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:50.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.564+0000 7efd7effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:50.565 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd84038740 msgr2=0x7efd8403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd84038740 0x7efd8403abf0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7efd88007fd0 tx=0x7efd88007e10 comp rx=0 tx=0).stop 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 10 monclient: shutdown 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 msgr2=0x7efd9811ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 0x7efd9811ef70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efd9000cee0 tx=0x7efd90007870 comp rx=0 tx=0).stop 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 shutdown_connections 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efd84038740 0x7efd8403abf0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd9809ca80 0x7efd9811ef70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd9806eb80 0x7efd9811ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 --2- 172.21.15.153:0/810068470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd98068a20 0x7efd9811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 >> 172.21.15.153:0/810068470 conn(0x7efd98067cd0 msgr2=0x7efd980a1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 shutdown_connections 2024-09-15T03:46:50.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.566+0000 7efd9d754640 1 -- 172.21.15.153:0/810068470 wait complete. 2024-09-15T03:46:50.577 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:46:50.577 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:46:50.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:46:50.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:46:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.670+0000 7f30f355d640 10 monclient: get_monmap_and_config 2024-09-15T03:46:50.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.670+0000 7f30f355d640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.670+0000 7f30f355d640 10 monclient: monmap: 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:50.671373+0000 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:50.671373+0000 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.677 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 1 Processor -- start 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 1 -- start start 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient: init 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30ec0981c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:50.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f30ec131da0 0x7f30ec136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient(hunting): picked mon.noname-c con 0x7f30ec131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30ec132170 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec09ca60 con 0x7f30ec132170 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec09cca0 con 0x7f30ec0981c0 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec131bd0 con 0x7f30ec131da0 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.671+0000 7f30f355d640 10 monclient(hunting): authenticate will time out at 1808.310922s 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f1afa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f30ec132170 0x7f30ec1390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44290/0 (socket says 172.21.15.153:44290) 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f1afa640 1 -- 172.21.15.153:0/644822943 learned_addr learned my addr 172.21.15.153:0/644822943 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 1 --2- 172.21.15.153:0/644822943 >> v2:172.21.15.181:3300/0 conn(0x7f30ec131da0 0x7f30ec136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): get_auth_request con 0x7f30ec131da0 auth_method 0 2024-09-15T03:46:50.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient(hunting): handle_auth_done global_id 204286 payload 274 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 1 -- 172.21.15.153:0/644822943 >> v1:172.21.15.181:6789/0 conn(0x7f30ec0981c0 legacy=0x7f30ec098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 1 -- 172.21.15.153:0/644822943 >> v1:172.21.15.153:6789/0 conn(0x7f30ec132170 legacy=0x7f30ec1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 1 monclient: found mon.noname-c 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:50.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.672+0000 7f30f0af8640 1 -- 172.21.15.153:0/644822943 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec09e010 con 0x7f30ec131da0 2024-09-15T03:46:50.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.675+0000 7f30f0af8640 1 -- 172.21.15.153:0/644822943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 msgr2=0x7f30ec136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.675+0000 7f30f0af8640 1 --2- 172.21.15.153:0/644822943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 0x7f30ec136b70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009850 tx=0x7f30dc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4c7e4220b89250c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.675+0000 7f30e3fff640 1 -- 172.21.15.153:0/644822943 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc032440 con 0x7f30ec131da0 2024-09-15T03:46:50.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: dump: 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 1 -- 172.21.15.153:0/644822943 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc0325a0 con 0x7f30ec131da0 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 5 monclient: authenticate success, global_id 204286 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 4 set_mon_vals no callback set 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 1 -- 172.21.15.153:0/644822943 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc032940 con 0x7f30ec131da0 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:50.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30e3fff640 10 monclient: dump: 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 10 monclient: shutdown 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 -- 172.21.15.153:0/644822943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 msgr2=0x7f30ec136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 --2- 172.21.15.153:0/644822943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 0x7f30ec136b70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009850 tx=0x7f30dc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 -- 172.21.15.153:0/644822943 shutdown_connections 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 --2- 172.21.15.153:0/644822943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 0x7f30ec136b70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 -- 172.21.15.153:0/644822943 >> 172.21.15.153:0/644822943 conn(0x7f30ec0a1e30 msgr2=0x7f30ec0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 -- 172.21.15.153:0/644822943 shutdown_connections 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.676+0000 7f30f355d640 1 -- 172.21.15.153:0/644822943 wait complete. 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient: monmap: 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:50.693 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 1 Processor -- start 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 1 -- start start 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient: init 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:46:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec0981c0 0x7f30ec1314b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient(hunting): picked mon.noname-a con 0x7f30ec0981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 0x7f30ec1380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient(hunting): picked mon.noname-b con 0x7f30ec131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 0x7f30ec1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient(hunting): picked mon.noname-c con 0x7f30ec132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.677+0000 7f30f355d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec098cb0 con 0x7f30ec0981c0 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f355d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f355d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec0a0920 con 0x7f30ec131da0 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f355d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f355d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec119810 con 0x7f30ec132170 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f355d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f355d640 10 monclient(hunting): authenticate will time out at 1808.317012s 2024-09-15T03:46:50.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec0981c0 0x7f30ec1314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 0x7f30ec1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec0981c0 0x7f30ec1314b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42630/0 (socket says 172.21.15.153:42630) 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 1 -- 172.21.15.153:0/1678033760 learned_addr learned my addr 172.21.15.153:0/1678033760 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): get_auth_request con 0x7f30ec0981c0 auth_method 0 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): get_auth_request con 0x7f30ec132170 auth_method 0 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 0x7f30ec1380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): get_auth_request con 0x7f30ec131da0 auth_method 0 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f0af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient(hunting): handle_auth_done global_id 224120 payload 824 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 msgr2=0x7f30ec1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30ec131da0 0x7f30ec1380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec0981c0 msgr2=0x7f30ec1314b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec0981c0 0x7f30ec1314b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 monclient: found mon.noname-c 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f1afa640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc009390 con 0x7f30ec132170 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.678+0000 7f30f12f9640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec0981c0 0x7f30ec1314b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f1afa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 msgr2=0x7f30ec1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f1afa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 0x7f30ec1385f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30e400cb60 tx=0x7f30e40052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e400a850 con 0x7f30ec132170 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: dump: 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 5 monclient: authenticate success, global_id 224120 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 10 monclient: _renew_subs 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec11a160 con 0x7f30ec132170 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e400a9d0 con 0x7f30ec132170 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e400ad90 con 0x7f30ec132170 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e17fa640 10 monclient: dump: 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.700 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30e37fe640 4 set_mon_vals no callback set 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 10 monclient: _renew_subs 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.680+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec139c70 con 0x7f30ec132170 2024-09-15T03:46:50.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f355d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f355d640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 0x7f30ec134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f30e4007300 con 0x7f30ec132170 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f355d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec065f90 con 0x7f30ec138b30 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 0x7f30ec134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient: get_auth_request con 0x7f30ec138b30 auth_method 0 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30e17fa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d0038740 0x7f30d003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f30e404c680 con 0x7f30ec132170 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.681+0000 7f30f12f9640 10 monclient(hunting): handle_auth_done global_id 214233 payload 824 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.682+0000 7f30f12f9640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30ec138f00 con 0x7f30ec138b30 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.682+0000 7f30f0af8640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d0038740 0x7f30d003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.682+0000 7f30f0af8640 10 monclient: get_auth_request con 0x7f30d0038740 auth_method 0 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30f12f9640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 msgr2=0x7f30ec134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30f12f9640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 0x7f30ec134650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f30e800fbb0 tx=0x7f30e8007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e800de70 con 0x7f30ec138b30 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 10 monclient: dump: 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.703 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f30e8014470 con 0x7f30ec138b30 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 msgr2=0x7f30ec134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 0x7f30ec134650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f30e800fbb0 tx=0x7f30e8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 msgr2=0x7f30ec134650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30e17fa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec138b30 0x7f30ec134650 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.684+0000 7f30f0af8640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d0038740 0x7f30d003abf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f30dc031000 tx=0x7f30dc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f355d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f355d640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 0x7f30ec134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f355d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec1196a0 con 0x7f30ec134b90 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 0x7f30ec134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient: get_auth_request con 0x7f30ec134b90 auth_method 0 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 10 monclient(hunting): handle_auth_done global_id 214236 payload 824 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.690+0000 7f30f12f9640 1 -- 172.21.15.153:0/1678033760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f30ec04bf60 con 0x7f30ec134b90 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.692+0000 7f30f12f9640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 msgr2=0x7f30ec134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.692+0000 7f30f12f9640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 0x7f30ec134f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f30e8019000 tx=0x7f30e801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e801ee10 con 0x7f30ec134b90 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 10 monclient: dump: 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f30e800dc60 con 0x7f30ec134b90 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 msgr2=0x7f30ec134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 0x7f30ec134f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f30e8019000 tx=0x7f30e801a000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 msgr2=0x7f30ec134f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:50.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30e17fa640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30ec134b90 0x7f30ec134f90 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f30e8019000 tx=0x7f30e801a000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 190, 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.710 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.711 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:50.713 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.714 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:50.716 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.717 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:50.718 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:50.720 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:50.721 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:46:50.722 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.722 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.722 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:50.722 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:50.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d0038740 msgr2=0x7f30d003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d0038740 0x7f30d003abf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f30dc031000 tx=0x7f30dc032000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 10 monclient: shutdown 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 msgr2=0x7f30ec1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f1afa640 1 -- 172.21.15.153:0/1678033760 reap_dead start 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 0x7f30ec1385f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30e400cb60 tx=0x7f30e40052b0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 shutdown_connections 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 --2- 172.21.15.153:0/1678033760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30ec132170 0x7f30ec1385f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 >> 172.21.15.153:0/1678033760 conn(0x7f30ec0a1e30 msgr2=0x7f30ec0a0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 shutdown_connections 2024-09-15T03:46:50.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.693+0000 7f30f355d640 1 -- 172.21.15.153:0/1678033760 wait complete. 2024-09-15T03:46:50.725 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:46:50.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:46:50.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.817+0000 7f7837d00640 10 monclient: get_monmap_and_config 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.817+0000 7f7837d00640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.817+0000 7f7837d00640 10 monclient: monmap: 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:50.818152+0000 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:50.818152+0000 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.817+0000 7f7837d00640 1 Processor -- start 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 1 -- start start 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient: init 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7830131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f783009fda0 0x7f783009e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient(hunting): picked mon.noname-c con 0x7f783009fda0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f783009ef00 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78300a1180 con 0x7f783009ef00 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78300a1460 con 0x7f7830131980 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7830132460 con 0x7f783009fda0 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f7837d00640 10 monclient(hunting): authenticate will time out at 1808.457641s 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f783629d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f783009ef00 0x7f7830138cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44304/0 (socket says 172.21.15.153:44304) 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f783629d640 1 -- 172.21.15.153:0/987093862 learned_addr learned my addr 172.21.15.153:0/987093862 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.818+0000 7f783529b640 1 --2- 172.21.15.153:0/987093862 >> v2:172.21.15.181:3300/0 conn(0x7f783009fda0 0x7f783009e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): get_auth_request con 0x7f783009fda0 auth_method 0 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient(hunting): handle_auth_done global_id 204292 payload 274 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 1 -- 172.21.15.153:0/987093862 >> v1:172.21.15.181:6789/0 conn(0x7f7830131980 legacy=0x7f7830131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 1 -- 172.21.15.153:0/987093862 >> v1:172.21.15.153:6789/0 conn(0x7f783009ef00 legacy=0x7f7830138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 1 monclient: found mon.noname-c 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.819+0000 7f783529b640 1 -- 172.21.15.153:0/987093862 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78301325a0 con 0x7f783009fda0 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f783529b640 1 -- 172.21.15.153:0/987093862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 msgr2=0x7f783009e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f783529b640 1 --2- 172.21.15.153:0/987093862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 0x7f783009e9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7820009850 tx=0x7f78200090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315d4ebf844577ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 1 -- 172.21.15.153:0/987093862 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820032440 con 0x7f783009fda0 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: dump: 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 1 -- 172.21.15.153:0/987093862 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78200325a0 con 0x7f783009fda0 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 5 monclient: authenticate success, global_id 204292 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 1 -- 172.21.15.153:0/987093862 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820032940 con 0x7f783009fda0 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 4 set_mon_vals no callback set 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7834a9a640 10 monclient: dump: 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 10 monclient: shutdown 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.822+0000 7f7837d00640 1 -- 172.21.15.153:0/987093862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 msgr2=0x7f783009e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 --2- 172.21.15.153:0/987093862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 0x7f783009e9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7820009850 tx=0x7f78200090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 -- 172.21.15.153:0/987093862 shutdown_connections 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 --2- 172.21.15.153:0/987093862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 0x7f783009e9c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 -- 172.21.15.153:0/987093862 >> 172.21.15.153:0/987093862 conn(0x7f7830067bd0 msgr2=0x7f78300a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 -- 172.21.15.153:0/987093862 shutdown_connections 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 -- 172.21.15.153:0/987093862 wait complete. 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 10 monclient: monmap: 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 Processor -- start 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 1 -- start start 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.823+0000 7f7837d00640 10 monclient: init 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 0x7f78301382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): picked mon.noname-c con 0x7f783009ef00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 0x7f7830134050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): picked mon.noname-b con 0x7f783009fda0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7830131980 0x7f7830134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): picked mon.noname-a con 0x7f7830131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7830136060 con 0x7f7830131980 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783009bf10 con 0x7f783009fda0 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7830139820 con 0x7f783009ef00 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7837d00640 10 monclient(hunting): authenticate will time out at 1808.463381s 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7830131980 0x7f7830134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 0x7f78301382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7830131980 0x7f7830134590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42646/0 (socket says 172.21.15.153:42646) 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): get_auth_request con 0x7f783009ef00 auth_method 0 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 1 -- 172.21.15.153:0/4052337719 learned_addr learned my addr 172.21.15.153:0/4052337719 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): get_auth_request con 0x7f7830131980 auth_method 0 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 0x7f7830134050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): get_auth_request con 0x7f783009fda0 auth_method 0 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient(hunting): handle_auth_done global_id 224123 payload 824 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 msgr2=0x7f7830134050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f783009fda0 0x7f7830134050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7830131980 msgr2=0x7f7830134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7830131980 0x7f7830134590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 monclient: found mon.noname-c 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f7835a9c640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7820009390 con 0x7f783009ef00 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.824+0000 7f783629d640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7830131980 0x7f7830134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7835a9c640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 msgr2=0x7f78301382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7835a9c640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 0x7f78301382a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f782c00c4e0 tx=0x7f782c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c0117a0 con 0x7f783009ef00 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: dump: 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782c011920 con 0x7f783009ef00 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 5 monclient: authenticate success, global_id 224123 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 10 monclient: _renew_subs 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7830135ab0 con 0x7f783009ef00 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c011ce0 con 0x7f783009ef00 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7825ffb640 10 monclient: dump: 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7827fff640 4 set_mon_vals no callback set 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 10 monclient: _renew_subs 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.827+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78301387e0 con 0x7f783009ef00 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f782c012630 con 0x7f783009ef00 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7837d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7837d00640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 0x7f7830098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7837d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7830065e00 con 0x7f7830134ad0 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7825ffb640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7814038740 0x7f781403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f782c04c1c0 con 0x7f783009ef00 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 0x7f7830098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient: get_auth_request con 0x7f7830134ad0 auth_method 0 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 10 monclient(hunting): handle_auth_done global_id 224126 payload 824 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783529b640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7830139590 con 0x7f7830134ad0 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783629d640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7814038740 0x7f781403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.828+0000 7f783629d640 10 monclient: get_auth_request con 0x7f7814038740 auth_method 0 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.830+0000 7f783529b640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 msgr2=0x7f7830098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.830+0000 7f783529b640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 0x7f7830098a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7830068aa0 tx=0x7f7820032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.830+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820031930 con 0x7f7830134ad0 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.830+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.830+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.830+0000 7f7825ffb640 10 monclient: dump: 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f783629d640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7814038740 0x7f781403abf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f782800d760 tx=0x7f7828009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7820041070 con 0x7f7830134ad0 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 msgr2=0x7f7830098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 0x7f7830098a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7830068aa0 tx=0x7f7820032000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 msgr2=0x7f7830098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.831+0000 7f7825ffb640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830134ad0 0x7f7830098a70 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7830068aa0 tx=0x7f7820032000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f7837d00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f7837d00640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 0x7f78300993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f7837d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7830137b00 con 0x7f7830098fb0 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 0x7f78300993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient: get_auth_request con 0x7f7830098fb0 auth_method 0 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.836+0000 7f783529b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.837+0000 7f783529b640 10 monclient(hunting): handle_auth_done global_id 224129 payload 824 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.837+0000 7f783529b640 1 -- 172.21.15.153:0/4052337719 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7830134ea0 con 0x7f7830098fb0 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f783529b640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 msgr2=0x7f78300993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f783529b640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 0x7f78300993b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7830068700 tx=0x7f782003f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782003d600 con 0x7f7830098fb0 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 10 monclient: dump: 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f782003d760 con 0x7f7830098fb0 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 msgr2=0x7f78300993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 0x7f78300993b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7830068700 tx=0x7f782003f070 comp rx=0 tx=0).stop 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 msgr2=0x7f78300993b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.839+0000 7f7825ffb640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7830098fb0 0x7f78300993b0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7830068700 tx=0x7f782003f070 comp rx=0 tx=0).stop 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 190, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7814038740 msgr2=0x7f781403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7814038740 0x7f781403abf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f782800d760 tx=0x7f7828009040 comp rx=0 tx=0).stop 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 10 monclient: shutdown 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 msgr2=0x7f78301382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f783629d640 1 -- 172.21.15.153:0/4052337719 reap_dead start 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 0x7f78301382a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f782c00c4e0 tx=0x7f782c007e10 comp rx=0 tx=0).stop 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 shutdown_connections 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 --2- 172.21.15.153:0/4052337719 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f783009ef00 0x7f78301382a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 >> 172.21.15.153:0/4052337719 conn(0x7f7830067bd0 msgr2=0x7f78300a4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 shutdown_connections 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.840+0000 7f7837d00640 1 -- 172.21.15.153:0/4052337719 wait complete. 2024-09-15T03:46:50.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:46:50.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:46:50.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:46:50.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.959+0000 7fe174f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:46:50.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.959+0000 7fe174f5d640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.959+0000 7fe174f5d640 10 monclient: monmap: 2024-09-15T03:46:50.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:50.960329+0000 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:50.960329+0000 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:50.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 Processor -- start 2024-09-15T03:46:50.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 -- start start 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient: init 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe170096e10 0x7fe17009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe170096e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe17009aaa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 0x7fe170135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe1700978f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17009d4a0 con 0x7fe17009aaa0 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe170099620 con 0x7fe170096e10 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe170099340 con 0x7fe1700978f0 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe174f5d640 10 monclient(hunting): authenticate will time out at 1808.599863s 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.960+0000 7fe16ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 0x7fe170135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 0x7fe170135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43976/0 (socket says 172.21.15.153:43976) 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1771679419 learned_addr learned my addr 172.21.15.153:0/1771679419 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): get_auth_request con 0x7fe1700978f0 auth_method 0 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16e59c640 1 --2- 172.21.15.153:0/1771679419 >> v2:172.21.15.181:3300/0 conn(0x7fe170096e10 0x7fe17009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16e59c640 10 monclient(hunting): get_auth_request con 0x7fe170096e10 auth_method 0 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_done global_id 224132 payload 274 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1771679419 >> v2:172.21.15.181:3300/0 conn(0x7fe170096e10 msgr2=0x7fe17009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 --2- 172.21.15.153:0/1771679419 >> v2:172.21.15.181:3300/0 conn(0x7fe170096e10 0x7fe17009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1771679419 >> v1:172.21.15.181:6789/0 conn(0x7fe17009aaa0 legacy=0x7fe1700971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 monclient: found mon.noname-b 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.961+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1771679419 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe170099900 con 0x7fe1700978f0 2024-09-15T03:46:50.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1771679419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 msgr2=0x7fe170135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16ed9d640 1 --2- 172.21.15.153:0/1771679419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 0x7fe170135b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe168009f20 tx=0x7fe168009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2db14fd0f119b795 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 1 -- 172.21.15.153:0/1771679419 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168007610 con 0x7fe1700978f0 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: dump: 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 5 monclient: authenticate success, global_id 224132 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 1 -- 172.21.15.153:0/1771679419 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16802d050 con 0x7fe1700978f0 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 4 set_mon_vals no callback set 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 1 -- 172.21.15.153:0/1771679419 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1680079e0 con 0x7fe1700978f0 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe16d59a640 10 monclient: dump: 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 10 monclient: shutdown 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 1 -- 172.21.15.153:0/1771679419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 msgr2=0x7fe170135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.964+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1771679419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 0x7fe170135b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe168009f20 tx=0x7fe168009840 comp rx=0 tx=0).stop 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 -- 172.21.15.153:0/1771679419 shutdown_connections 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1771679419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1700978f0 0x7fe170135b40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1771679419 >> v2:172.21.15.181:3300/0 conn(0x7fe170096e10 0x7fe17009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 -- 172.21.15.153:0/1771679419 >> 172.21.15.153:0/1771679419 conn(0x7fe1700672b0 msgr2=0x7fe1700676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 -- 172.21.15.153:0/1771679419 shutdown_connections 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 -- 172.21.15.153:0/1771679419 wait complete. 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 10 monclient: build_initial_monmap 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 10 monclient: monmap: 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:50.976 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 Processor -- start 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 1 -- start start 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.965+0000 7fe174f5d640 10 monclient: init 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 0x7fe170140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe170096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700978f0 0x7fe1701411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe1700978f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe17009aaa0 0x7fe170130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe17009aaa0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170122570 con 0x7fe17009aaa0 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17009b050 con 0x7fe1700978f0 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700a0310 con 0x7fe170096e10 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:50.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe174f5d640 10 monclient(hunting): authenticate will time out at 1808.605359s 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 0x7fe170140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 0x7fe170140c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:43986/0 (socket says 172.21.15.153:43986) 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 -- 172.21.15.153:0/1235001856 learned_addr learned my addr 172.21.15.153:0/1235001856 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): get_auth_request con 0x7fe170096e10 auth_method 0 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe17009aaa0 0x7fe170130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): get_auth_request con 0x7fe17009aaa0 auth_method 0 2024-09-15T03:46:50.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16dd9b640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700978f0 0x7fe1701411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16dd9b640 10 monclient(hunting): get_auth_request con 0x7fe1700978f0 auth_method 0 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient(hunting): handle_auth_done global_id 224135 payload 824 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700978f0 msgr2=0x7fe1701411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700978f0 0x7fe1701411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe17009aaa0 msgr2=0x7fe170130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe17009aaa0 0x7fe170130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 monclient: found mon.noname-c 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.966+0000 7fe16e59c640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168009ae0 con 0x7fe170096e10 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.968+0000 7fe16e59c640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 msgr2=0x7fe170140c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.968+0000 7fe16e59c640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 0x7fe170140c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe16000d8f0 tx=0x7fe160008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16000de80 con 0x7fe170096e10 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:50.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: dump: 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:50.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 5 monclient: authenticate success, global_id 224135 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16000bae0 con 0x7fe170096e10 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 10 monclient: _renew_subs 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe170120180 con 0x7fe170096e10 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16000be80 con 0x7fe170096e10 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 10 monclient: dump: 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.983 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 10 monclient: _renew_subs 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe16cd99640 4 set_mon_vals no callback set 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe170121cf0 con 0x7fe170096e10 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe160014640 con 0x7fe170096e10 2024-09-15T03:46:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.969+0000 7fe174f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 0x7fe17011ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701220c0 con 0x7fe170131250 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe15e7fc640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe150038740 0x7fe15003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe16004cc00 con 0x7fe170096e10 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 0x7fe17011ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16ed9d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe150038740 0x7fe15003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient: get_auth_request con 0x7fe170131250 auth_method 0 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16ed9d640 10 monclient: get_auth_request con 0x7fe150038740 auth_method 0 2024-09-15T03:46:50.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_done global_id 204304 payload 824 2024-09-15T03:46:50.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.970+0000 7fe16dd9b640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe17004bf60 con 0x7fe170131250 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe16ed9d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe150038740 0x7fe15003abf0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe168033000 tx=0x7fe168036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe16dd9b640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 msgr2=0x7fe17011ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe16dd9b640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 0x7fe17011ed80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe17009b2a0 tx=0x7fe164009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164007450 con 0x7fe170131250 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe15e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:50.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.972+0000 7fe15e7fc640 10 monclient: dump: 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe1640075b0 con 0x7fe170131250 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:50.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 msgr2=0x7fe17011ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 0x7fe17011ed80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe17009b2a0 tx=0x7fe164009040 comp rx=0 tx=0).stop 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 msgr2=0x7fe17011ed80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.973+0000 7fe15e7fc640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe170131250 0x7fe17011ed80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.978+0000 7fe174f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.978+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 0x7fe170029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.978+0000 7fe174f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.978+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170065f20 con 0x7fe1700298c0 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 0x7fe170029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient: get_auth_request con 0x7fe1700298c0 auth_method 0 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:50.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 10 monclient(hunting): handle_auth_done global_id 204307 payload 824 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.979+0000 7fe16dd9b640 1 -- 172.21.15.153:0/1235001856 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe170122320 con 0x7fe1700298c0 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.981+0000 7fe16dd9b640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 msgr2=0x7fe170029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.981+0000 7fe16dd9b640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 0x7fe170029cc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe17009b2a0 tx=0x7fe16401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.981+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164013e00 con 0x7fe1700298c0 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.981+0000 7fe15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.981+0000 7fe15e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.981+0000 7fe15e7fc640 10 monclient: dump: 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fe164007240 con 0x7fe1700298c0 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 msgr2=0x7fe170029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 0x7fe170029cc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe17009b2a0 tx=0x7fe16401b000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 msgr2=0x7fe170029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:46:50.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe15e7fc640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1700298c0 0x7fe170029cc0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe17009b2a0 tx=0x7fe16401b000 comp rx=0 tx=0).stop 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 190, 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:46:50.993 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.994 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.995 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:50.996 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:46:50.997 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:46:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.003 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:46:51.005 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:46:51.006 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:46:51.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:46:51.006 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:46:51.006 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:46:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe150038740 msgr2=0x7fe15003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe150038740 0x7fe15003abf0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe168033000 tx=0x7fe168036000 comp rx=0 tx=0).stop 2024-09-15T03:46:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 10 monclient: shutdown 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 msgr2=0x7fe170140c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 --2- 172.21.15.153:0/1235001856 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe170096e10 0x7fe170140c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe16000d8f0 tx=0x7fe160008040 comp rx=0 tx=0).stop 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1235001856 reap_dead start 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe16ed9d640 1 -- 172.21.15.153:0/1235001856 reap_dead start 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 shutdown_connections 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 >> 172.21.15.153:0/1235001856 conn(0x7fe1700672b0 msgr2=0x7fe17009f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 shutdown_connections 2024-09-15T03:46:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:50.982+0000 7fe174f5d640 1 -- 172.21.15.153:0/1235001856 wait complete. 2024-09-15T03:46:51.008 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:46:51.009 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:46:51.009 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:46:51.009 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:46:51.009 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:46:51.009 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:46:51.009 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:46:51.019 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:46:51.020 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:46:51.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.152+0000 7f1fcb95d640 10 monclient: get_monmap_and_config 2024-09-15T03:46:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.152+0000 7f1fcb95d640 10 monclient: build_initial_monmap 2024-09-15T03:46:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.152+0000 7f1fcb95d640 10 monclient: monmap: 2024-09-15T03:46:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:51.153503+0000 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:51.153503+0000 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:51.155 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:51.155 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:51.155 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:51.155 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:51.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.153+0000 7f1fcb95d640 1 Processor -- start 2024-09-15T03:46:51.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.153+0000 7f1fcb95d640 1 -- start start 2024-09-15T03:46:51.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.153+0000 7f1fcb95d640 10 monclient: init 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.153+0000 7f1fcb95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.153+0000 7f1fcb95d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4096e10 0x7f1fc409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc4096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1fc409aaa0 0x7f1fc40971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc409aaa0 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fc4097720 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc4099620 con 0x7f1fc4097720 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc4099340 con 0x7f1fc409aaa0 2024-09-15T03:46:51.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4099900 con 0x7f1fc4096e10 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fcb95d640 10 monclient(hunting): authenticate will time out at 1808.793075s 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc96f9640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4096e10 msgr2=0x7f1fc409a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc96f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4096e10 0x7f1fc409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1fc409aaa0 0x7f1fc40971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1fc409aaa0 0x7f1fc40971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42660/0 (socket says 172.21.15.153:42660) 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 -- 172.21.15.153:0/2026999797 learned_addr learned my addr 172.21.15.153:0/2026999797 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:51.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): get_auth_request con 0x7f1fc409aaa0 auth_method 0 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient(hunting): handle_auth_done global_id 214245 payload 274 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 -- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4096e10 msgr2=0x7f1fc409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 --2- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4096e10 0x7f1fc409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 -- 172.21.15.153:0/2026999797 >> v1:172.21.15.181:6789/0 conn(0x7f1fc4097720 legacy=0x7f1fc412e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient: _finish_hunting 0 2024-09-15T03:46:51.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 monclient: found mon.noname-a 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.154+0000 7f1fc8ef8640 1 -- 172.21.15.153:0/2026999797 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4097b10 con 0x7f1fc409aaa0 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fc8ef8640 1 -- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc409aaa0 msgr2=0x7f1fc40971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fc8ef8640 1 --2- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc409aaa0 0x7f1fc40971e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009910 tx=0x7f1fb40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e58b6df4b5edb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 1 -- 172.21.15.153:0/2026999797 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4032440 con 0x7f1fc409aaa0 2024-09-15T03:46:51.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: dump: 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: _finish_auth 0 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 1 -- 172.21.15.153:0/2026999797 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb40325a0 con 0x7f1fc409aaa0 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 5 monclient: authenticate success, global_id 214245 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 1 -- 172.21.15.153:0/2026999797 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4032940 con 0x7f1fc409aaa0 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 4 set_mon_vals no callback set 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:51.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fbbfff640 10 monclient: dump: 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:51.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 10 monclient: shutdown 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 1 -- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc409aaa0 msgr2=0x7f1fc40971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc409aaa0 0x7f1fc40971e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009910 tx=0x7f1fb40091d0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 1 -- 172.21.15.153:0/2026999797 shutdown_connections 2024-09-15T03:46:51.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc409aaa0 0x7f1fc40971e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/2026999797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4096e10 0x7f1fc409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.157+0000 7f1fcb95d640 1 -- 172.21.15.153:0/2026999797 >> 172.21.15.153:0/2026999797 conn(0x7f1fc40672b0 msgr2=0x7f1fc40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:51.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 1 -- 172.21.15.153:0/2026999797 shutdown_connections 2024-09-15T03:46:51.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 1 -- 172.21.15.153:0/2026999797 wait complete. 2024-09-15T03:46:51.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 10 monclient: build_initial_monmap 2024-09-15T03:46:51.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 10 monclient: monmap: 2024-09-15T03:46:51.177 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:51.177 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:51.177 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:51.178 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 1 Processor -- start 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 1 -- start start 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.158+0000 7f1fcb95d640 10 monclient: init 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 0x7f1fc409b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc4096e10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:51.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4097720 0x7f1fc412dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:51.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc4097720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:51.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fc409aaa0 0x7f1fc4140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:51.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc409aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:51.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:51.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc412e820 con 0x7f1fc4096e10 2024-09-15T03:46:51.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:51.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4118350 con 0x7f1fc409aaa0 2024-09-15T03:46:51.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:51.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc409c2e0 con 0x7f1fc4097720 2024-09-15T03:46:51.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:51.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc8ef8640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4097720 msgr2=0x7f1fc412dbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:51.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fcb95d640 10 monclient(hunting): authenticate will time out at 1808.798653s 2024-09-15T03:46:51.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc8ef8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4097720 0x7f1fc412dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:51.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 0x7f1fc409b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:51.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 0x7f1fc409b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42664/0 (socket says 172.21.15.153:42664) 2024-09-15T03:46:51.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 -- 172.21.15.153:0/837074958 learned_addr learned my addr 172.21.15.153:0/837074958 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:51.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): get_auth_request con 0x7f1fc4096e10 auth_method 0 2024-09-15T03:46:51.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:51.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:51.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:51.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:51.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient(hunting): handle_auth_done global_id 214248 payload 824 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 -- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4097720 msgr2=0x7f1fc412dbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4097720 0x7f1fc412dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 -- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fc409aaa0 msgr2=0x7f1fc4140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fc409aaa0 0x7f1fc4140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient: _finish_hunting 0 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 monclient: found mon.noname-a 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.159+0000 7f1fc96f9640 1 -- 172.21.15.153:0/837074958 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4009450 con 0x7f1fc4096e10 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fc96f9640 1 -- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 msgr2=0x7f1fc409b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fc96f9640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 0x7f1fc409b810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b630 tx=0x7f1fc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0015070 con 0x7f1fc4096e10 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: dump: 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:51.187 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: _finish_auth 0 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc0003d90 con 0x7f1fc4096e10 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 5 monclient: authenticate success, global_id 214248 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 10 monclient: _renew_subs 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc411e040 con 0x7f1fc4096e10 2024-09-15T03:46:51.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0006380 con 0x7f1fc4096e10 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fbb7fe640 4 set_mon_vals no callback set 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fb97fa640 10 monclient: dump: 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:51.189 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 10 monclient: _renew_subs 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.162+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc4118a30 con 0x7f1fc4096e10 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1fc0010440 con 0x7f1fc4096e10 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fcb95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fcb95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fcb95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:51.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc404bf60 con 0x7f1fc4096e10 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fb97fa640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fa80386f0 0x7f1fa803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1fc004c190 con 0x7f1fc4096e10 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fc8ef8640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fa80386f0 0x7f1fa803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.163+0000 7f1fc8ef8640 10 monclient: get_auth_request con 0x7f1fa80386f0 auth_method 0 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.166+0000 7f1fc8ef8640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fa80386f0 0x7f1fa803aba0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1fc409aff0 tx=0x7f1fb4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.167+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1fc0018ab0 con 0x7f1fc4096e10 2024-09-15T03:46:51.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.167+0000 7f1fb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:51.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.167+0000 7f1fb97fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.381+0000 7f1fcb95d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.381+0000 7f1fcb95d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.381+0000 7f1fcb95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.381+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1fc40638d0 con 0x7f1fc4096e10 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.382+0000 7f1fb97fa640 1 -- 172.21.15.153:0/837074958 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f1fc0018690 con 0x7f1fc4096e10 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.382+0000 7f1fb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.382+0000 7f1fb97fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:51.383 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:51.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fa80386f0 msgr2=0x7f1fa803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:51.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fa80386f0 0x7f1fa803aba0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1fc409aff0 tx=0x7f1fb4032000 comp rx=0 tx=0).stop 2024-09-15T03:46:51.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 10 monclient: shutdown 2024-09-15T03:46:51.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:51.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 msgr2=0x7f1fc409b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:51.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 0x7f1fc409b810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b630 tx=0x7f1fc00091a0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 shutdown_connections 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1fa80386f0 0x7f1fa803aba0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fc409aaa0 0x7f1fc4140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1fc4097720 0x7f1fc412dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 --2- 172.21.15.153:0/837074958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fc4096e10 0x7f1fc409b810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 >> 172.21.15.153:0/837074958 conn(0x7f1fc40672b0 msgr2=0x7f1fc4131780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 shutdown_connections 2024-09-15T03:46:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:51.384+0000 7f1fcb95d640 1 -- 172.21.15.153:0/837074958 wait complete. 2024-09-15T03:46:54.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.499+0000 7fb6566a8640 10 monclient: get_monmap_and_config 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.499+0000 7fb6566a8640 10 monclient: build_initial_monmap 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.499+0000 7fb6566a8640 10 monclient: monmap: 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:54.500642+0000 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:54.500642+0000 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.500+0000 7fb6566a8640 1 Processor -- start 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.500+0000 7fb6566a8640 1 -- start start 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.500+0000 7fb6566a8640 10 monclient: init 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb6500975f0 0x7fb650095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb6500975f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fb6500979c0 0x7fb650097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb6500979c0 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb65009bf50 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65009d700 con 0x7fb65009bf50 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65009d1b0 con 0x7fb6500975f0 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65009d420 con 0x7fb6500979c0 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb6566a8640 10 monclient(hunting): authenticate will time out at 1812.140421s 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb6500975f0 0x7fb650095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb654c45640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb65009bf50 0x7fb65009c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44316/0 (socket says 172.21.15.153:44316) 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb6500975f0 0x7fb650095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42680/0 (socket says 172.21.15.153:42680) 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb654c45640 1 -- 172.21.15.153:0/3338899757 learned_addr learned my addr 172.21.15.153:0/3338899757 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): get_auth_request con 0x7fb6500975f0 auth_method 0 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient(hunting): handle_auth_done global_id 214251 payload 274 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 -- 172.21.15.153:0/3338899757 >> v2:172.21.15.181:3300/0 conn(0x7fb6500979c0 msgr2=0x7fb650097dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 --2- 172.21.15.153:0/3338899757 >> v2:172.21.15.181:3300/0 conn(0x7fb6500979c0 0x7fb650097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 -- 172.21.15.153:0/3338899757 >> v1:172.21.15.153:6789/0 conn(0x7fb65009bf50 legacy=0x7fb65009c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 monclient: found mon.noname-a 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.501+0000 7fb64ffff640 1 -- 172.21.15.153:0/3338899757 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65009d9e0 con 0x7fb6500975f0 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64ffff640 1 -- 172.21.15.153:0/3338899757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 msgr2=0x7fb650095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64ffff640 1 --2- 172.21.15.153:0/3338899757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb650095ad0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb6500959e0 tx=0x7fb6400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248c2400c6ef526e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 1 -- 172.21.15.153:0/3338899757 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640032440 con 0x7fb6500975f0 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: dump: 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: _finish_auth 0 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb6566a8640 5 monclient: authenticate success, global_id 214251 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 1 -- 172.21.15.153:0/3338899757 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6400325a0 con 0x7fb6500975f0 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb6566a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb6566a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 1 -- 172.21.15.153:0/3338899757 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640032940 con 0x7fb6500975f0 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb6566a8640 4 set_mon_vals no callback set 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.504+0000 7fb64effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb64effd640 10 monclient: dump: 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 10 monclient: shutdown 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 -- 172.21.15.153:0/3338899757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 msgr2=0x7fb650095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 --2- 172.21.15.153:0/3338899757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb650095ad0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb6500959e0 tx=0x7fb6400090f0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 -- 172.21.15.153:0/3338899757 shutdown_connections 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 --2- 172.21.15.153:0/3338899757 >> v2:172.21.15.181:3300/0 conn(0x7fb6500979c0 0x7fb650097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 --2- 172.21.15.153:0/3338899757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb650095ad0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 -- 172.21.15.153:0/3338899757 >> 172.21.15.153:0/3338899757 conn(0x7fb6500678a0 msgr2=0x7fb6500a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 -- 172.21.15.153:0/3338899757 shutdown_connections 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.505+0000 7fb6566a8640 1 -- 172.21.15.153:0/3338899757 wait complete. 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.506+0000 7fb6566a8640 10 monclient: build_initial_monmap 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.506+0000 7fb6566a8640 10 monclient: monmap: 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.506+0000 7fb6566a8640 1 Processor -- start 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.506+0000 7fb6566a8640 1 -- start start 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.506+0000 7fb6566a8640 10 monclient: init 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb6500942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb6500975f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6500979c0 0x7fb650094820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb6500979c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb65009bf50 0x7fb650128960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb65009bf50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65012c380 con 0x7fb6500975f0 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65012bdc0 con 0x7fb6500979c0 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65012bae0 con 0x7fb65009bf50 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb6566a8640 10 monclient(hunting): authenticate will time out at 1812.146186s 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb654c45640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb65009bf50 msgr2=0x7fb650128960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb6500942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb654c45640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb65009bf50 0x7fb650128960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb6500942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42682/0 (socket says 172.21.15.153:42682) 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 -- 172.21.15.153:0/2729269667 learned_addr learned my addr 172.21.15.153:0/2729269667 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): get_auth_request con 0x7fb6500975f0 auth_method 0 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64f7fe640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6500979c0 0x7fb650094820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64f7fe640 10 monclient(hunting): get_auth_request con 0x7fb6500979c0 auth_method 0 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient(hunting): handle_auth_done global_id 214254 payload 824 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 -- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb65009bf50 msgr2=0x7fb650128960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb65009bf50 0x7fb650128960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 -- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6500979c0 msgr2=0x7fb650094820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6500979c0 0x7fb650094820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 monclient: found mon.noname-a 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64ffff640 1 -- 172.21.15.153:0/2729269667 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb640009390 con 0x7fb6500975f0 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.507+0000 7fb64f7fe640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6500979c0 0x7fb650094820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb64ffff640 1 -- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 msgr2=0x7fb6500942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb64ffff640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb6500942e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb640009360 tx=0x7fb640032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6400318c0 con 0x7fb6500975f0 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: dump: 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: _finish_auth 0 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640031a20 con 0x7fb6500975f0 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb6566a8640 5 monclient: authenticate success, global_id 214254 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb6566a8640 10 monclient: _renew_subs 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb6566a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65012c0a0 con 0x7fb6500975f0 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640031de0 con 0x7fb6500975f0 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb637fff640 10 monclient: dump: 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 10 monclient: _renew_subs 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb650094d60 con 0x7fb6500975f0 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.509+0000 7fb64e7fc640 4 set_mon_vals no callback set 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb640039040 con 0x7fb6500975f0 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.510+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65012c4c0 con 0x7fb6500975f0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.511+0000 7fb637fff640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb63c0386f0 0x7fb63c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.511+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb64006c750 con 0x7fb6500975f0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.511+0000 7fb64f7fe640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb63c0386f0 0x7fb63c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.511+0000 7fb64f7fe640 10 monclient: get_auth_request con 0x7fb63c0386f0 auth_method 0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.513+0000 7fb64f7fe640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb63c0386f0 0x7fb63c03aba0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb6500a2600 tx=0x7fb644008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.515+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb65012c4c0 con 0x7fb6500975f0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.515+0000 7fb637fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.515+0000 7fb637fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:54.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.729+0000 7fb6566a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:54.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.729+0000 7fb6566a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:54.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.729+0000 7fb6566a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:54.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.729+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb65012a200 con 0x7fb6500975f0 2024-09-15T03:46:54.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.730+0000 7fb637fff640 1 -- 172.21.15.153:0/2729269667 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fb64003f620 con 0x7fb6500975f0 2024-09-15T03:46:54.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.730+0000 7fb637fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:54.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.730+0000 7fb637fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:54.731 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb63c0386f0 msgr2=0x7fb63c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb63c0386f0 0x7fb63c03aba0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb6500a2600 tx=0x7fb644008040 comp rx=0 tx=0).stop 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 10 monclient: shutdown 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 msgr2=0x7fb6500942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb6500942e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb640009360 tx=0x7fb640032000 comp rx=0 tx=0).stop 2024-09-15T03:46:54.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 shutdown_connections 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb63c0386f0 0x7fb63c03aba0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb65009bf50 0x7fb650128960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6500979c0 0x7fb650094820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 --2- 172.21.15.153:0/2729269667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6500975f0 0x7fb6500942e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 >> 172.21.15.153:0/2729269667 conn(0x7fb6500678a0 msgr2=0x7fb650132a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 shutdown_connections 2024-09-15T03:46:54.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:54.732+0000 7fb6566a8640 1 -- 172.21.15.153:0/2729269667 wait complete. 2024-09-15T03:46:57.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.848+0000 7f7902860640 10 monclient: get_monmap_and_config 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.848+0000 7f7902860640 10 monclient: build_initial_monmap 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.848+0000 7f7902860640 10 monclient: monmap: 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:46:57.848932+0000 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:46:57.848932+0000 2024-09-15T03:46:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:46:57.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:57.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 1 Processor -- start 2024-09-15T03:46:57.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 1 -- start start 2024-09-15T03:46:57.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient: init 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78fc1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78fc09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f78fc137c30 0x7f78fc13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient(hunting): picked mon.noname-c con 0x7f78fc137c30 addr v2:172.21.15.181:3300/0 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc131cc0 con 0x7f78fc09ea50 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc09f530 con 0x7f78fc1318f0 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc09fbf0 con 0x7f78fc137c30 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f7902860640 10 monclient(hunting): authenticate will time out at 1815.488708s 2024-09-15T03:46:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f78fb7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f78fc09ea50 0x7f78fc09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:57060/0 (socket says 172.21.15.153:57060) 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.849+0000 7f78fb7fe640 1 -- 172.21.15.153:0/2321003142 learned_addr learned my addr 172.21.15.153:0/2321003142 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 1 --2- 172.21.15.153:0/2321003142 >> v2:172.21.15.181:3300/0 conn(0x7f78fc137c30 0x7f78fc13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): get_auth_request con 0x7f78fc137c30 auth_method 0 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient(hunting): handle_auth_done global_id 204313 payload 274 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 1 -- 172.21.15.153:0/2321003142 >> v1:172.21.15.181:6789/0 conn(0x7f78fc1318f0 legacy=0x7f78fc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 1 -- 172.21.15.153:0/2321003142 >> v1:172.21.15.153:6789/0 conn(0x7f78fc09ea50 legacy=0x7f78fc09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient: _finish_hunting 0 2024-09-15T03:46:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 1 monclient: found mon.noname-c 2024-09-15T03:46:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:46:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.850+0000 7f7900dfd640 1 -- 172.21.15.153:0/2321003142 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc09fe30 con 0x7f78fc137c30 2024-09-15T03:46:57.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f7900dfd640 1 -- 172.21.15.153:0/2321003142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 msgr2=0x7f78fc13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f7900dfd640 1 --2- 172.21.15.153:0/2321003142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc13a010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009e60 tx=0x7f78f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf836281b7662347 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 1 -- 172.21.15.153:0/2321003142 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4007630 con 0x7f78fc137c30 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: dump: 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: _finish_auth 0 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 1 -- 172.21.15.153:0/2321003142 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f402d050 con 0x7f78fc137c30 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 1 -- 172.21.15.153:0/2321003142 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f40079c0 con 0x7f78fc137c30 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f78faffd640 10 monclient: dump: 2024-09-15T03:46:57.862 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f7902860640 5 monclient: authenticate success, global_id 204313 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f7902860640 10 monclient: get_monmap_and_config success 2024-09-15T03:46:57.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.853+0000 7f7902860640 4 set_mon_vals no callback set 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 10 monclient: shutdown 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 -- 172.21.15.153:0/2321003142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 msgr2=0x7f78fc13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 --2- 172.21.15.153:0/2321003142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc13a010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009e60 tx=0x7f78f4009780 comp rx=0 tx=0).stop 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 -- 172.21.15.153:0/2321003142 shutdown_connections 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 --2- 172.21.15.153:0/2321003142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc13a010 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 -- 172.21.15.153:0/2321003142 >> 172.21.15.153:0/2321003142 conn(0x7f78fc067110 msgr2=0x7f78fc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 -- 172.21.15.153:0/2321003142 shutdown_connections 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 1 -- 172.21.15.153:0/2321003142 wait complete. 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 10 monclient: build_initial_monmap 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.854+0000 7f7902860640 10 monclient: monmap: 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 Processor -- start 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 -- start start 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient: init 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient: _reopen_session rank -1 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78fc09ea50 0x7f78fc136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): picked mon.noname-c con 0x7f78fc09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 0x7f78fc096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): picked mon.noname-a con 0x7f78fc1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): picked mon.noname-b con 0x7f78fc137c30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc099c50 con 0x7f78fc1318f0 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc09bf10 con 0x7f78fc137c30 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): start opening mon connection 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc09d7c0 con 0x7f78fc09ea50 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): _renew_subs 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f7902860640 10 monclient(hunting): authenticate will time out at 1815.494814s 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fbfff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78fc09ea50 msgr2=0x7f78fc136cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78fc09ea50 0x7f78fc136cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 0x7f78fc096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 0x7f78fc096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37446/0 (socket says 172.21.15.153:37446) 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 1 -- 172.21.15.153:0/3566180796 learned_addr learned my addr 172.21.15.153:0/3566180796 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 10 monclient(hunting): get_auth_request con 0x7f78fc1318f0 auth_method 0 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.855+0000 7f78fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): get_auth_request con 0x7f78fc137c30 auth_method 0 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f7900dfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 10 monclient(hunting): handle_auth_done global_id 214257 payload 824 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 1 -- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78fc09ea50 msgr2=0x7f78fc136cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78fc09ea50 0x7f78fc136cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 1 -- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 msgr2=0x7f78fc096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 1 monclient: found mon.noname-a 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.856+0000 7f78fb7fe640 1 -- 172.21.15.153:0/3566180796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4009a20 con 0x7f78fc1318f0 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78fb7fe640 1 -- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 msgr2=0x7f78fc096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78fb7fe640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 0x7f78fc096a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78f000c5e0 tx=0x7f78f0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0005cb0 con 0x7f78fc1318f0 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: dump: 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: _finish_auth 0 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f0005e30 con 0x7f78fc1318f0 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 5 monclient: authenticate success, global_id 214257 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 10 monclient: _renew_subs 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc0980d0 con 0x7f78fc1318f0 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0007340 con 0x7f78fc1318f0 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78e3fff640 10 monclient: dump: 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 10 monclient: _renew_subs 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f78fa7fc640 4 set_mon_vals no callback set 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.858+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc09aa50 con 0x7f78fc1318f0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f78f001b400 con 0x7f78fc1318f0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f7902860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f7902860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f7902860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc099950 con 0x7f78fc1318f0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f78e3fff640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78e80386f0 0x7f78e803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.859+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f78f004bf20 con 0x7f78fc1318f0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.860+0000 7f78fbfff640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78e80386f0 0x7f78e803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.860+0000 7f78fbfff640 10 monclient: get_auth_request con 0x7f78e80386f0 auth_method 0 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.862+0000 7f78fbfff640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78e80386f0 0x7f78e803aba0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f78ec007fd0 tx=0x7f78ec007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.864+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f78f001a960 con 0x7f78fc1318f0 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.864+0000 7f78e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:57.864+0000 7f78e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:46:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.078+0000 7f7902860640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:46:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.078+0000 7f7902860640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.078+0000 7f7902860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:46:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.078+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78fc098070 con 0x7f78fc1318f0 2024-09-15T03:46:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.079+0000 7f78e3fff640 1 -- 172.21.15.153:0/3566180796 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f78f0010300 con 0x7f78fc1318f0 2024-09-15T03:46:58.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.079+0000 7f78e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:46:58.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.079+0000 7f78e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:46:58.080 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:46:58.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78e80386f0 msgr2=0x7f78e803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:58.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78e80386f0 0x7f78e803aba0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f78ec007fd0 tx=0x7f78ec007a70 comp rx=0 tx=0).stop 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 10 monclient: shutdown 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 msgr2=0x7f78fc096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 0x7f78fc096a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78f000c5e0 tx=0x7f78f0004960 comp rx=0 tx=0).stop 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 shutdown_connections 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f78e80386f0 0x7f78e803aba0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78fc137c30 0x7f78fc096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78fc1318f0 0x7f78fc096a60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 --2- 172.21.15.153:0/3566180796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78fc09ea50 0x7f78fc136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.080+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 >> 172.21.15.153:0/3566180796 conn(0x7f78fc067110 msgr2=0x7f78fc0a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.081+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 shutdown_connections 2024-09-15T03:46:58.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:46:58.081+0000 7f7902860640 1 -- 172.21.15.153:0/3566180796 wait complete. 2024-09-15T03:47:01.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:01.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.195+0000 7f52864a8640 10 monclient: get_monmap_and_config 2024-09-15T03:47:01.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.195+0000 7f52864a8640 10 monclient: build_initial_monmap 2024-09-15T03:47:01.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.195+0000 7f52864a8640 10 monclient: monmap: 2024-09-15T03:47:01.196 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:01.196 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:01.196 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:01.196308+0000 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:01.196308+0000 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.196+0000 7f52864a8640 1 Processor -- start 2024-09-15T03:47:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.196+0000 7f52864a8640 1 -- start start 2024-09-15T03:47:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.196+0000 7f52864a8640 10 monclient: init 2024-09-15T03:47:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:01.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): picked mon.noname-c con 0x7f52800a1890 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52800975c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 0x7f5280139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5280136c80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528006e900 con 0x7f52800975c0 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5280098150 con 0x7f52800a1890 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52800a1c60 con 0x7f5280136c80 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f52864a8640 10 monclient(hunting): authenticate will time out at 1818.836113s 2024-09-15T03:47:01.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f5284a45640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 msgr2=0x7f5280139060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f5284a45640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 0x7f5280139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f5277fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f52800975c0 0x7f5280097a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:57074/0 (socket says 172.21.15.153:57074) 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f5277fff640 1 -- 172.21.15.153:0/2487390636 learned_addr learned my addr 172.21.15.153:0/2487390636 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f52800a1890 auth_method 0 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.197+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:01.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.199+0000 7f5277fff640 1 -- 172.21.15.153:0/2487390636 >> v1:172.21.15.153:6789/0 conn(0x7f52800975c0 legacy=0x7f5280097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:01.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f5284a45640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 msgr2=0x7f5280139060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:01.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f5284a45640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 0x7f5280139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:01.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f52800a1890 auth_method 0 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:01.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.398+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f5284a45640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 msgr2=0x7f5280139060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f5284a45640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 0x7f5280139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f52800a1890 auth_method 0 2024-09-15T03:47:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:01.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:01.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:01.799+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f5284a45640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 msgr2=0x7f5280139060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:02.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f5284a45640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 0x7f5280139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f52800a1890 auth_method 0 2024-09-15T03:47:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:02.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:02.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:02.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:02.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:02.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:02.600+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:04.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient(hunting): tick 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 monclient(hunting): continuing hunt 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 msgr2=0x7f5280139060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5280136c80 0x7f5280139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 msgr2=0x7f528012eae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.181:3300/0 conn(0x7f52800a1890 0x7f528012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 -- 172.21.15.153:0/2487390636 >> v1:172.21.15.153:6789/0 conn(0x7f52800975c0 legacy=0x7f5280097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient: _add_conns ranks=[3,1] 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f5260003700 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5260006260 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:04.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 -- 172.21.15.153:0/2487390636 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52800641d0 con 0x7f5260006260 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 1 -- 172.21.15.153:0/2487390636 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5280098150 con 0x7f5260003700 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527e7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f5260003700 auth_method 0 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:04.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.197+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:04.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.199+0000 7f5277fff640 1 -- 172.21.15.153:0/2487390636 >> v1:172.21.15.181:6789/0 conn(0x7f5260006260 legacy=0x7f52600086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f5260003700 auth_method 0 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:04.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.398+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f5260003700 auth_method 0 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:04.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:04.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:04.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:04.799+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:05.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.152+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449404432 0 0) 0x7f52800641d0 con 0x7f5260006260 2024-09-15T03:47:05.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.152+0000 7f527f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.152+0000 7f527f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.152+0000 7f527f7fe640 10 monclient(hunting): my global_id is 204319 2024-09-15T03:47:05.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.152+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c0034e0 con 0x7f5260006260 2024-09-15T03:47:05.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689507755 0 0) 0x7f526c0034e0 con 0x7f5260006260 2024-09-15T03:47:05.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 msgr2=0x7f5260005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:05.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 --2- 172.21.15.153:0/2487390636 >> v2:172.21.15.153:3300/0 conn(0x7f5260003700 0x7f5260005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526000b040 con 0x7f5260006260 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f52680016e0 con 0x7f5260006260 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: dump: 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.160 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 >> v1:172.21.15.181:6789/0 conn(0x7f5260006260 legacy=0x7f52600086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 1 -- 172.21.15.153:0/2487390636 reap_dead start 2024-09-15T03:47:05.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 0x7f526c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): picked mon.a con 0x7f526c0039d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c004780 0x7f526c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): picked mon.c con 0x7f526c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c005180 0x7f526c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): picked mon.b con 0x7f526c005180 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280098150 con 0x7f526c0039d0 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c00c120 con 0x7f526c005180 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526c00c2e0 con 0x7f526c004780 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:47:05.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5277fff640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c004780 msgr2=0x7f526c004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5277fff640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c004780 0x7f526c004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 0x7f526c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f526c0039d0 auth_method 0 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c005180 0x7f526c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): get_auth_request con 0x7f526c005180 auth_method 0 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f5284a45640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 10 monclient(hunting): handle_auth_done global_id 204319 payload 293 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c004780 msgr2=0x7f526c004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c004780 0x7f526c004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.156+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c005180 msgr2=0x7f526c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c005180 0x7f526c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 1 monclient: found mon.a 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c0034e0 con 0x7f526c0039d0 2024-09-15T03:47:05.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f527ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f52864a8640 5 monclient: authenticate success, global_id 204319 2024-09-15T03:47:05.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.157+0000 7f52864a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.159+0000 7f527ffff640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 msgr2=0x7f526c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.159+0000 7f527ffff640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 0x7f526c003dc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5270006ea0 tx=0x7f5270008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115345199e6e594b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527002ca20 con 0x7f526c0039d0 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: dump: 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527002cb80 con 0x7f526c0039d0 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 1 -- 172.21.15.153:0/2487390636 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270037460 con 0x7f526c0039d0 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 4 set_mon_vals no callback set 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f527f7fe640 10 monclient: dump: 2024-09-15T03:47:05.172 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 10 monclient: shutdown 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 -- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 msgr2=0x7f526c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 0x7f526c003dc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5270006ea0 tx=0x7f5270008040 comp rx=0 tx=0).stop 2024-09-15T03:47:05.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 -- 172.21.15.153:0/2487390636 shutdown_connections 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c005180 0x7f526c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c004780 0x7f526c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 --2- 172.21.15.153:0/2487390636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c0039d0 0x7f526c003dc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.160+0000 7f52864a8640 1 -- 172.21.15.153:0/2487390636 >> 172.21.15.153:0/2487390636 conn(0x7f5280067b50 msgr2=0x7f52800a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.161+0000 7f52864a8640 1 -- 172.21.15.153:0/2487390636 shutdown_connections 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.161+0000 7f52864a8640 1 -- 172.21.15.153:0/2487390636 wait complete. 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.161+0000 7f52864a8640 10 monclient: build_initial_monmap 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.161+0000 7f52864a8640 10 monclient: monmap: 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:05.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.161+0000 7f52864a8640 1 Processor -- start 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 -- start start 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient: init 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c0039d0 0x7f5280141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): picked mon.noname-c con 0x7f526c0039d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c004780 0x7f5280141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): picked mon.noname-b con 0x7f526c004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 0x7f52801281a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): picked mon.noname-a con 0x7f526c005180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280096860 con 0x7f526c005180 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528012bc30 con 0x7f526c004780 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5280129ab0 con 0x7f526c0039d0 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f527ffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c0039d0 msgr2=0x7f5280141320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f52864a8640 10 monclient(hunting): authenticate will time out at 1822.801569s 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f527ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c0039d0 0x7f5280141320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 0x7f52801281a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 0x7f52801281a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49802/0 (socket says 172.21.15.153:49802) 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 1 -- 172.21.15.153:0/3535142455 learned_addr learned my addr 172.21.15.153:0/3535142455 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): get_auth_request con 0x7f526c005180 auth_method 0 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5284a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c004780 0x7f5280141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): get_auth_request con 0x7f526c004780 auth_method 0 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.162+0000 7f5277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 10 monclient(hunting): handle_auth_done global_id 214260 payload 824 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 1 -- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c0039d0 msgr2=0x7f5280141320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c0039d0 0x7f5280141320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 1 -- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c004780 msgr2=0x7f5280141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c004780 0x7f5280141860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 1 monclient: found mon.noname-a 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.163+0000 7f5284a45640 1 -- 172.21.15.153:0/3535142455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528012b5e0 con 0x7f526c005180 2024-09-15T03:47:05.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f5284a45640 1 -- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 msgr2=0x7f52801281a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f5284a45640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 0x7f52801281a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5260009f30 tx=0x7f5278007530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52780116a0 con 0x7f526c005180 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: dump: 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.179 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278004e90 con 0x7f526c005180 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 5 monclient: authenticate success, global_id 214260 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 10 monclient: _renew_subs 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528012b8c0 con 0x7f526c005180 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278011a70 con 0x7f526c005180 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527cff9640 10 monclient: dump: 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f527effd640 4 set_mon_vals no callback set 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 10 monclient: _renew_subs 2024-09-15T03:47:05.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.165+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5280141da0 con 0x7f526c005180 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5278012460 con 0x7f526c005180 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f527cff9640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5260039160 0x7f526003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f52864a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f52864a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f52864a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f527804bd20 con 0x7f526c005180 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.166+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528004bf60 con 0x7f526c005180 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.167+0000 7f527ffff640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5260039160 0x7f526003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.167+0000 7f527ffff640 10 monclient: get_auth_request con 0x7f5260039160 auth_method 0 2024-09-15T03:47:05.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.169+0000 7f527ffff640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5260039160 0x7f526003b610 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5270006fd0 tx=0x7f5270037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.171+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f527801a870 con 0x7f526c005180 2024-09-15T03:47:05.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.171+0000 7f527cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.171+0000 7f527cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:05.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f52864a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:05.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f52864a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:05.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f52864a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5280063b60 con 0x7f526c005180 2024-09-15T03:47:05.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f527cff9640 1 -- 172.21.15.153:0/3535142455 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f527801a270 con 0x7f526c005180 2024-09-15T03:47:05.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f527cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:05.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.389+0000 7f527cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:05.392 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":194,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5260039160 msgr2=0x7f526003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5260039160 0x7f526003b610 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5270006fd0 tx=0x7f5270037000 comp rx=0 tx=0).stop 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 10 monclient: shutdown 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 msgr2=0x7f52801281a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 0x7f52801281a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5260009f30 tx=0x7f5278007530 comp rx=0 tx=0).stop 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 shutdown_connections 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5260039160 0x7f526003b610 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f526c005180 0x7f52801281a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f526c004780 0x7f5280141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 --2- 172.21.15.153:0/3535142455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f526c0039d0 0x7f5280141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 >> 172.21.15.153:0/3535142455 conn(0x7f5280067b50 msgr2=0x7f5280095560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:05.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 shutdown_connections 2024-09-15T03:47:05.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.391+0000 7f52864a8640 1 -- 172.21.15.153:0/3535142455 wait complete. 2024-09-15T03:47:05.403 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:47:05.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:47:05.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.496+0000 7f57aa8b8640 10 monclient: get_monmap_and_config 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.496+0000 7f57aa8b8640 10 monclient: build_initial_monmap 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.496+0000 7f57aa8b8640 10 monclient: monmap: 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:05.496973+0000 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:05.496973+0000 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:05.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:05.498 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:05.498 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:05.498 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:05.498 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.496+0000 7f57aa8b8640 1 Processor -- start 2024-09-15T03:47:05.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 1 -- start start 2024-09-15T03:47:05.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient: init 2024-09-15T03:47:05.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57a4131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 0x7f57a409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient(hunting): picked mon.noname-b con 0x7f57a409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57a409f220 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a409faf0 con 0x7f57a409f220 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a409fd30 con 0x7f57a4131ab0 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4132590 con 0x7f57a409e910 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57aa8b8640 10 monclient(hunting): authenticate will time out at 1823.136550s 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57a37fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 msgr2=0x7f57a409ece0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57a37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 0x7f57a409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57a8e55640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f57a409f220 0x7f57a4138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:54524/0 (socket says 172.21.15.153:54524) 2024-09-15T03:47:05.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.497+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 learned_addr learned my addr 172.21.15.153:0/3527804072 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:05.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.499+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 >> v1:172.21.15.153:6789/0 conn(0x7f57a409f220 legacy=0x7f57a4138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062945580 0 0) 0x7f57a409faf0 con 0x7f57a409f220 2024-09-15T03:47:05.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): my global_id is 214263 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57900034e0 con 0x7f57a409f220 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 1 -- 172.21.15.153:0/3527804072 >> v1:172.21.15.181:6789/0 conn(0x7f57a4131ab0 legacy=0x7f57a4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1355409871 0 0) 0x7f57900034e0 con 0x7f57a409f220 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 msgr2=0x7f57a409ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 0x7f57a409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 >> v1:172.21.15.181:6789/0 conn(0x7f57a4131ab0 legacy=0x7f57a4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:47:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40a1110 con 0x7f57a409f220 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f579c001b60 con 0x7f57a409f220 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57aa8b8640 5 monclient: authenticate success, global_id 214263 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57aa8b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: dump: 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.506 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 >> v1:172.21.15.153:6789/0 conn(0x7f57a409f220 legacy=0x7f57a4138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5790003a00 0x7f5790003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): picked mon.b con 0x7f5790003a00 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 0x7f5790004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): picked mon.c con 0x7f57a409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f5790005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): picked mon.a con 0x7f5790004de0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4132590 con 0x7f5790004de0 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579000bc60 con 0x7f5790003a00 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579000be20 con 0x7f57a409e910 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57aa8b8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a37fe640 1 -- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 msgr2=0x7f5790004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a37fe640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 0x7f5790004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f5790005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): get_auth_request con 0x7f5790004de0 auth_method 0 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5790003a00 0x7f5790003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): get_auth_request con 0x7f5790003a00 auth_method 0 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient(hunting): handle_auth_done global_id 214263 payload 293 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 msgr2=0x7f5790004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a409e910 0x7f5790004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5790003a00 msgr2=0x7f5790003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5790003a00 0x7f5790003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 monclient: found mon.a 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57900034e0 con 0x7f5790004de0 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a8e55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.500+0000 7f57a3fff640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5790003a00 0x7f5790003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 msgr2=0x7f5790005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a8e55640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f5790005250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f579c00aca0 tx=0x7f579c00a350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c3069b14c4fab2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c02b790 con 0x7f5790004de0 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: dump: 2024-09-15T03:47:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57aa8b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c02b8f0 con 0x7f5790004de0 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 1 -- 172.21.15.153:0/3527804072 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c02bcb0 con 0x7f5790004de0 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57aa8b8640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57aa8b8640 4 set_mon_vals no callback set 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.503+0000 7f57a2ffd640 10 monclient: dump: 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 10 monclient: shutdown 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 1 -- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 msgr2=0x7f5790005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 1 --2- 172.21.15.153:0/3527804072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f5790005250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f579c00aca0 tx=0x7f579c00a350 comp rx=0 tx=0).stop 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57a8e55640 1 -- 172.21.15.153:0/3527804072 reap_dead start 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 1 -- 172.21.15.153:0/3527804072 shutdown_connections 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 1 -- 172.21.15.153:0/3527804072 >> 172.21.15.153:0/3527804072 conn(0x7f57a4067d30 msgr2=0x7f57a40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 1 -- 172.21.15.153:0/3527804072 shutdown_connections 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 1 -- 172.21.15.153:0/3527804072 wait complete. 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 10 monclient: build_initial_monmap 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.504+0000 7f57aa8b8640 10 monclient: monmap: 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:05.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 Processor -- start 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 -- start start 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient: init 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:47:05.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f57a4141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): picked mon.noname-a con 0x7f5790004de0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57a4130b90 0x7f57a4130f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): picked mon.noname-b con 0x7f57a4130b90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a411b6b0 0x7f57a411bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): picked mon.noname-c con 0x7f57a411b6b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a41314c0 con 0x7f5790004de0 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4141e90 con 0x7f57a4130b90 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a411c770 con 0x7f57a411b6b0 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57aa8b8640 10 monclient(hunting): authenticate will time out at 1823.144839s 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a8e55640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a411b6b0 msgr2=0x7f57a411bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a8e55640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a411b6b0 0x7f57a411bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f57a4141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f57a4141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49814/0 (socket says 172.21.15.153:49814) 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 1 -- 172.21.15.153:0/2047195215 learned_addr learned my addr 172.21.15.153:0/2047195215 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 10 monclient(hunting): get_auth_request con 0x7f5790004de0 auth_method 0 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.505+0000 7f57a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 10 monclient(hunting): handle_auth_done global_id 214266 payload 824 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a411b6b0 msgr2=0x7f57a411bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57a411b6b0 0x7f57a411bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57a4130b90 msgr2=0x7f57a4130f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57a4130b90 0x7f57a4130f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 1 monclient: found mon.noname-a 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.506+0000 7f57a3fff640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40a4b00 con 0x7f5790004de0 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57a3fff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 msgr2=0x7f57a4141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57a3fff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f57a4141480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5794007990 tx=0x7f5794009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57940074f0 con 0x7f5790004de0 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: dump: 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.527 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5794005780 con 0x7f5790004de0 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 5 monclient: authenticate success, global_id 214266 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 10 monclient: _renew_subs 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a409b650 con 0x7f5790004de0 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794005b00 con 0x7f5790004de0 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f579bfff640 10 monclient: dump: 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57a27fc640 4 set_mon_vals no callback set 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 10 monclient: _renew_subs 2024-09-15T03:47:05.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.508+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a411dcb0 con 0x7f5790004de0 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57aa8b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57aa8b8640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 0x7f57a4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f579400d730 con 0x7f5790004de0 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57aa8b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4063740 con 0x7f57a411c060 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 0x7f57a4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient: get_auth_request con 0x7f57a411c060 auth_method 0 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f579bfff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f57840386f0 0x7f578403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f579404c780 con 0x7f5790004de0 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 10 monclient(hunting): handle_auth_done global_id 214269 payload 824 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a37fe640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57a404bf60 con 0x7f57a411c060 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.509+0000 7f57a8e55640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f57840386f0 0x7f578403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.510+0000 7f57a8e55640 10 monclient: get_auth_request con 0x7f57840386f0 auth_method 0 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.511+0000 7f57a37fe640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 msgr2=0x7f57a4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.511+0000 7f57a37fe640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 0x7f57a4096a60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f578c003940 tx=0x7f578c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c006150 con 0x7f57a411c060 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 10 monclient: dump: 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.531 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f578c0062b0 con 0x7f57a411c060 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 msgr2=0x7f57a4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 0x7f57a4096a60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f578c003940 tx=0x7f578c009120 comp rx=0 tx=0).stop 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 msgr2=0x7f57a4096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f579bfff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a411c060 0x7f57a4096a60 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f578c003940 tx=0x7f578c009120 comp rx=0 tx=0).stop 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.512+0000 7f57a8e55640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f57840386f0 0x7f578403aba0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f579c001810 tx=0x7f579c031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.517+0000 7f57aa8b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.517+0000 7f57aa8b8640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 0x7f57a40973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57aa8b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a411d920 con 0x7f57a4096fa0 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 0x7f57a40973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient: get_auth_request con 0x7f57a4096fa0 auth_method 0 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 10 monclient(hunting): handle_auth_done global_id 214272 payload 824 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.518+0000 7f57a37fe640 1 -- 172.21.15.153:0/2047195215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57a411c430 con 0x7f57a4096fa0 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f57a37fe640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 msgr2=0x7f57a40973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f57a37fe640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 0x7f57a40973a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f578c013000 tx=0x7f578c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c006810 con 0x7f57a4096fa0 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 10 monclient: dump: 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.534 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.535 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.535 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.535 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f578c0103c0 con 0x7f57a4096fa0 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 msgr2=0x7f57a40973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 0x7f57a40973a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f578c013000 tx=0x7f578c014000 comp rx=0 tx=0).stop 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 msgr2=0x7f57a40973a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.520+0000 7f579bfff640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57a4096fa0 0x7f57a40973a0 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f578c013000 tx=0x7f578c014000 comp rx=0 tx=0).stop 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:05.536 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 194, 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:05.537 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:05.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:05.539 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:05.540 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:05.542 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:05.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:05.544 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:05.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.547 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:05.548 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:05.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f57840386f0 msgr2=0x7f578403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f57840386f0 0x7f578403aba0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f579c001810 tx=0x7f579c031000 comp rx=0 tx=0).stop 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 10 monclient: shutdown 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 msgr2=0x7f57a4141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 --2- 172.21.15.153:0/2047195215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5790004de0 0x7f57a4141480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5794007990 tx=0x7f5794009040 comp rx=0 tx=0).stop 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57a8e55640 1 -- 172.21.15.153:0/2047195215 reap_dead start 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57a8e55640 1 -- 172.21.15.153:0/2047195215 reap_dead start 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 shutdown_connections 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 >> 172.21.15.153:0/2047195215 conn(0x7f57a4067d30 msgr2=0x7f57a40a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 shutdown_connections 2024-09-15T03:47:05.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.521+0000 7f57aa8b8640 1 -- 172.21.15.153:0/2047195215 wait complete. 2024-09-15T03:47:05.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:47:05.552 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:47:05.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.646+0000 7fab8e71d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.646+0000 7fab8e71d640 10 monclient: build_initial_monmap 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.646+0000 7fab8e71d640 10 monclient: monmap: 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:05.647329+0000 2024-09-15T03:47:05.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:05.647329+0000 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:05.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 1 Processor -- start 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 1 -- start start 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient: init 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab880974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 0x7fab8809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient(hunting): picked mon.noname-b con 0x7fab8809a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab88133070 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8812b210 con 0x7fab880974f0 2024-09-15T03:47:05.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab880978c0 con 0x7fab88133070 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8809ca20 con 0x7fab8809a8a0 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab8e71d640 10 monclient(hunting): authenticate will time out at 1823.286832s 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab877fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 msgr2=0x7fab8809acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab877fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 0x7fab8809acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab87fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fab880974f0 0x7fab8812ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:54530/0 (socket says 172.21.15.153:54530) 2024-09-15T03:47:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.647+0000 7fab87fff640 1 -- 172.21.15.153:0/1380610345 learned_addr learned my addr 172.21.15.153:0/1380610345 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:05.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 -- 172.21.15.153:0/1380610345 >> v1:172.21.15.153:6789/0 conn(0x7fab880974f0 legacy=0x7fab8812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109642671 0 0) 0x7fab8812b210 con 0x7fab880974f0 2024-09-15T03:47:05.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): my global_id is 214275 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab700034e0 con 0x7fab880974f0 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 1 -- 172.21.15.153:0/1380610345 >> v1:172.21.15.181:6789/0 conn(0x7fab88133070 legacy=0x7fab88135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3213380020 0 0) 0x7fab700034e0 con 0x7fab880974f0 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 msgr2=0x7fab8809acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 0x7fab8809acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 >> v1:172.21.15.181:6789/0 conn(0x7fab88133070 legacy=0x7fab88135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8809b1f0 con 0x7fab880974f0 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8e71d640 5 monclient: authenticate success, global_id 214275 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8e71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fab78001720 con 0x7fab880974f0 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: dump: 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.660 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 >> v1:172.21.15.153:6789/0 conn(0x7fab880974f0 legacy=0x7fab8812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab70003a80 0x7fab70003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): picked mon.a con 0x7fab70003a80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 0x7fab70004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): picked mon.c con 0x7fab8809a8a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab70004e70 0x7fab7000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab877fe640 1 -- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 msgr2=0x7fab70004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): picked mon.b con 0x7fab70004e70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8809ca20 con 0x7fab70003a80 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70005650 con 0x7fab70004e70 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7000bf90 con 0x7fab8809a8a0 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab877fe640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 0x7fab70004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab86ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8e71d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab70003a80 0x7fab70003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): get_auth_request con 0x7fab70003a80 auth_method 0 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab70004e70 0x7fab7000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): get_auth_request con 0x7fab70004e70 auth_method 0 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:05.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab8ccba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient(hunting): handle_auth_done global_id 214275 payload 293 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 -- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 msgr2=0x7fab70004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab8809a8a0 0x7fab70004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 -- 172.21.15.153:0/1380610345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab70004e70 msgr2=0x7fab7000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab70004e70 0x7fab7000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 monclient: found mon.a 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 1 -- 172.21.15.153:0/1380610345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700034e0 con 0x7fab70003a80 2024-09-15T03:47:05.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.650+0000 7fab87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab87fff640 1 -- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab70003a80 msgr2=0x7fab70003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab87fff640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab70003a80 0x7fab70003e90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fab78001490 tx=0x7fab880695f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675ff2e1fb078d9c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78005890 con 0x7fab70003a80 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: dump: 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.665 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab780059f0 con 0x7fab70003a80 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab8e71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab8e71d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 1 -- 172.21.15.153:0/1380610345 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78005db0 con 0x7fab70003a80 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab8e71d640 4 set_mon_vals no callback set 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.653+0000 7fab86ffd640 10 monclient: dump: 2024-09-15T03:47:05.666 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 10 monclient: shutdown 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 1 -- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab70003a80 msgr2=0x7fab70003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 1 --2- 172.21.15.153:0/1380610345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab70003a80 0x7fab70003e90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fab78001490 tx=0x7fab880695f0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8ccba640 1 -- 172.21.15.153:0/1380610345 reap_dead start 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 1 -- 172.21.15.153:0/1380610345 shutdown_connections 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 1 -- 172.21.15.153:0/1380610345 >> 172.21.15.153:0/1380610345 conn(0x7fab880672b0 msgr2=0x7fab880676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 1 -- 172.21.15.153:0/1380610345 shutdown_connections 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 1 -- 172.21.15.153:0/1380610345 wait complete. 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 10 monclient: build_initial_monmap 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.654+0000 7fab8e71d640 10 monclient: monmap: 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:05.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 Processor -- start 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 -- start start 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient: init 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:05.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab70003a80 0x7fab8812fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): picked mon.noname-c con 0x7fab70003a80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88130260 0x7fab88130630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): picked mon.noname-b con 0x7fab88130260 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 0x7fab88147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): picked mon.noname-a con 0x7fab88130b70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8812a400 con 0x7fab88130b70 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8806cb20 con 0x7fab88130260 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8806c840 con 0x7fab70003a80 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab87fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab70003a80 msgr2=0x7fab8812fd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8e71d640 10 monclient(hunting): authenticate will time out at 1823.294882s 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab87fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab70003a80 0x7fab8812fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.655+0000 7fab8ccba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 0x7fab88147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 0x7fab88147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49860/0 (socket says 172.21.15.153:49860) 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 learned_addr learned my addr 172.21.15.153:0/2154930617 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): get_auth_request con 0x7fab88130b70 auth_method 0 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88130260 0x7fab88130630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): get_auth_request con 0x7fab88130260 auth_method 0 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient(hunting): handle_auth_done global_id 214278 payload 824 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab70003a80 msgr2=0x7fab8812fd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:05.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab70003a80 0x7fab8812fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88130260 msgr2=0x7fab88130630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88130260 0x7fab88130630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient: _finish_hunting 0 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 monclient: found mon.noname-a 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.656+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab881323b0 con 0x7fab88130b70 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 msgr2=0x7fab88147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8ccba640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 0x7fab88147550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab88068900 tx=0x7fab80009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab80005d10 con 0x7fab88130b70 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: dump: 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.678 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: _finish_auth 0 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab80005e70 con 0x7fab88130b70 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab80007300 con 0x7fab88130b70 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 5 monclient: authenticate success, global_id 214278 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab6ffff640 10 monclient: dump: 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 10 monclient: _renew_subs 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8806c560 con 0x7fab88130b70 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab867fc640 4 set_mon_vals no callback set 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 10 monclient: _renew_subs 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.658+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8806f4b0 con 0x7fab88130b70 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab8e71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab8e71d640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 0x7fab88126d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab8e71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab88065f20 con 0x7fab88126970 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fab800076f0 con 0x7fab88130b70 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab6ffff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab740386f0 0x7fab7403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fab8004beb0 con 0x7fab88130b70 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.659+0000 7fab87fff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 0x7fab88126d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient: get_auth_request con 0x7fab88126970 auth_method 0 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab877fe640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab740386f0 0x7fab7403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab877fe640 10 monclient: get_auth_request con 0x7fab740386f0 auth_method 0 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 10 monclient(hunting): handle_auth_done global_id 204334 payload 824 2024-09-15T03:47:05.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.660+0000 7fab87fff640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8806cc90 con 0x7fab88126970 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab87fff640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 msgr2=0x7fab88126d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab87fff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 0x7fab88126d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fab78033000 tx=0x7fab78034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab877fe640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab740386f0 0x7fab7403aba0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fab7c003850 tx=0x7fab7c0091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab780011e0 con 0x7fab88126970 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.662+0000 7fab6ffff640 10 monclient: dump: 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.683 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fab78007720 con 0x7fab88126970 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 msgr2=0x7fab88126d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 0x7fab88126d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fab78033000 tx=0x7fab78034000 comp rx=0 tx=0).stop 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 msgr2=0x7fab88126d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:05.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.663+0000 7fab6ffff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88126970 0x7fab88126d40 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fab78033000 tx=0x7fab78034000 comp rx=0 tx=0).stop 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.668+0000 7fab8e71d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.668+0000 7fab8e71d640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 0x7fab8814bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.668+0000 7fab8e71d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.668+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8804bf60 con 0x7fab88127280 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 0x7fab8814bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient: get_auth_request con 0x7fab88127280 auth_method 0 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 10 monclient(hunting): handle_auth_done global_id 204337 payload 824 2024-09-15T03:47:05.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.669+0000 7fab87fff640 1 -- 172.21.15.153:0/2154930617 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fab880638d0 con 0x7fab88127280 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.671+0000 7fab87fff640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 msgr2=0x7fab8814bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab87fff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 0x7fab8814bab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab88098460 tx=0x7fab7803e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78005780 con 0x7fab88127280 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 10 monclient: dump: 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7fab7803b420 con 0x7fab88127280 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 msgr2=0x7fab8814bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 0x7fab8814bab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab88098460 tx=0x7fab7803e040 comp rx=0 tx=0).stop 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 msgr2=0x7fab8814bab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab6ffff640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab88127280 0x7fab8814bab0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab88098460 tx=0x7fab7803e040 comp rx=0 tx=0).stop 2024-09-15T03:47:05.687 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 194, 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:05.688 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:05.689 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:05.690 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:05.691 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:05.692 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:05.693 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.694 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:05.695 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.697 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:05.698 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.699 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:47:05.700 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:05.701 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab740386f0 msgr2=0x7fab7403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab740386f0 0x7fab7403aba0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fab7c003850 tx=0x7fab7c0091a0 comp rx=0 tx=0).stop 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 10 monclient: shutdown 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 msgr2=0x7fab88147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 --2- 172.21.15.153:0/2154930617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab88130b70 0x7fab88147550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab88068900 tx=0x7fab80009040 comp rx=0 tx=0).stop 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 reap_dead start 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8ccba640 1 -- 172.21.15.153:0/2154930617 reap_dead start 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 shutdown_connections 2024-09-15T03:47:05.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 >> 172.21.15.153:0/2154930617 conn(0x7fab880672b0 msgr2=0x7fab88125640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:05.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 shutdown_connections 2024-09-15T03:47:05.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:05.672+0000 7fab8e71d640 1 -- 172.21.15.153:0/2154930617 wait complete. 2024-09-15T03:47:05.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:47:05.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:47:25.705 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:47:25.705 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:47:25.705 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:47:25.705 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:47:25.708 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:47:25.708 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:47:25.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.811+0000 7fdda3a6f640 10 monclient: get_monmap_and_config 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.811+0000 7fdda3a6f640 10 monclient: build_initial_monmap 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.811+0000 7fdda3a6f640 10 monclient: monmap: 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:25.813199+0000 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:25.813199+0000 2024-09-15T03:47:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:25.814 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 1 Processor -- start 2024-09-15T03:47:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 1 -- start start 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient: init 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdd9c131ab0 0x7fdd9c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9c131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd9c09e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd9c136bb0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c09faf0 con 0x7fdd9c09e910 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c09fd30 con 0x7fdd9c136bb0 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9c0a1110 con 0x7fdd9c131ab0 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda3a6f640 10 monclient(hunting): authenticate will time out at 1843.452817s 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda100a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdd9c09e910 0x7fdd9c09ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55030/0 (socket says 172.21.15.153:55030) 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.812+0000 7fdda100a640 1 -- 172.21.15.153:0/2733301518 learned_addr learned my addr 172.21.15.153:0/2733301518 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 1 --2- 172.21.15.153:0/2733301518 >> v2:172.21.15.181:3300/0 conn(0x7fdd9c131ab0 0x7fdd9c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): get_auth_request con 0x7fdd9c131ab0 auth_method 0 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient(hunting): handle_auth_done global_id 204340 payload 274 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 1 -- 172.21.15.153:0/2733301518 >> v1:172.21.15.181:6789/0 conn(0x7fdd9c136bb0 legacy=0x7fdd9c138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 1 -- 172.21.15.153:0/2733301518 >> v1:172.21.15.153:6789/0 conn(0x7fdd9c09e910 legacy=0x7fdd9c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient: _finish_hunting 0 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 1 monclient: found mon.noname-c 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:47:25.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.813+0000 7fdda180b640 1 -- 172.21.15.153:0/2733301518 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c0a13f0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda180b640 1 -- 172.21.15.153:0/2733301518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c131ab0 msgr2=0x7fdd9c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda180b640 1 --2- 172.21.15.153:0/2733301518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c131e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009880 tx=0x7fdd940090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ead2c6edbed0a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 1 -- 172.21.15.153:0/2733301518 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94032440 con 0x7fdd9c131ab0 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: dump: 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:25.826 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: _finish_auth 0 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 1 -- 172.21.15.153:0/2733301518 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd940325a0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 1 -- 172.21.15.153:0/2733301518 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94032940 con 0x7fdd9c131ab0 2024-09-15T03:47:25.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 5 monclient: authenticate success, global_id 204340 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 4 set_mon_vals no callback set 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda0809640 10 monclient: dump: 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:25.828 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 10 monclient: shutdown 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 1 -- 172.21.15.153:0/2733301518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c131ab0 msgr2=0x7fdd9c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.816+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/2733301518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c131e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009880 tx=0x7fdd940090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:25.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 -- 172.21.15.153:0/2733301518 shutdown_connections 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/2733301518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c131e80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 -- 172.21.15.153:0/2733301518 >> 172.21.15.153:0/2733301518 conn(0x7fdd9c067d30 msgr2=0x7fdd9c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 -- 172.21.15.153:0/2733301518 shutdown_connections 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 -- 172.21.15.153:0/2733301518 wait complete. 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 10 monclient: build_initial_monmap 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 10 monclient: monmap: 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 Processor -- start 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 1 -- start start 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.817+0000 7fdda3a6f640 10 monclient: init 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:25.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 0x7fdd9c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9c09e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9c131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c136bb0 0x7fdd9c0974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9c136bb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c097fd0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c09be10 con 0x7fdd9c136bb0 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c099a40 con 0x7fdd9c09e910 2024-09-15T03:47:25.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda3a6f640 10 monclient(hunting): authenticate will time out at 1843.458363s 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57690/0 (socket says 172.21.15.153:57690) 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 0x7fdd9c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 -- 172.21.15.153:0/3355969552 learned_addr learned my addr 172.21.15.153:0/3355969552 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): get_auth_request con 0x7fdd9c131ab0 auth_method 0 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 10 monclient(hunting): get_auth_request con 0x7fdd9c09e910 auth_method 0 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:25.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 msgr2=0x7fdd9c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 msgr2=0x7fdd9c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 0x7fdd9c096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda180b640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 0x7fdd9c096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda200c640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c136bb0 0x7fdd9c0974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient(hunting): handle_auth_done global_id 214281 payload 824 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 msgr2=0x7fdd9c096a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 0x7fdd9c096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c136bb0 msgr2=0x7fdd9c0974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:25.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c136bb0 0x7fdd9c0974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient: _finish_hunting 0 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 monclient: found mon.noname-a 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda100a640 1 -- 172.21.15.153:0/3355969552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd94009390 con 0x7fdd9c131ab0 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda200c640 10 monclient: get_auth_request con 0x7fdd9c136bb0 auth_method 0 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.818+0000 7fdda200c640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c136bb0 0x7fdd9c0974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda100a640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 msgr2=0x7fdd9c096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda100a640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c096fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000cca0 tx=0x7fdd900053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000aa60 con 0x7fdd9c131ab0 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:25.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: dump: 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: _finish_auth 0 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:25.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 5 monclient: authenticate success, global_id 214281 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 10 monclient: _renew_subs 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c099d20 con 0x7fdd9c131ab0 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9000abc0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000aa60 con 0x7fdd9c131ab0 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd99ffb640 10 monclient: dump: 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:25.837 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 10 monclient: _renew_subs 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c09abf0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.821+0000 7fdd9bfff640 4 set_mon_vals no callback set 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.822+0000 7fdda3a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.822+0000 7fdda3a6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.822+0000 7fdda3a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:25.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.822+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c04bf60 con 0x7fdd9c131ab0 2024-09-15T03:47:25.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.822+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdd900070f0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.822+0000 7fdd99ffb640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd88038800 0x7fdd8803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:25.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.823+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdd9004c2b0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.823+0000 7fdda180b640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd88038800 0x7fdd8803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:25.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.823+0000 7fdda180b640 10 monclient: get_auth_request con 0x7fdd88038800 auth_method 0 2024-09-15T03:47:25.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.825+0000 7fdda180b640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd88038800 0x7fdd8803acb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c09bdb0 tx=0x7fdd94032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:25.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.827+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdd9001b8e0 con 0x7fdd9c131ab0 2024-09-15T03:47:25.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.827+0000 7fdd99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:25.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:25.827+0000 7fdd99ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:26.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdda3a6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:26.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdda3a6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:26.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdda3a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:26.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdd9c0a2750 con 0x7fdd9c131ab0 2024-09-15T03:47:26.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdd99ffb640 1 -- 172.21.15.153:0/3355969552 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fdd9001ba80 con 0x7fdd9c131ab0 2024-09-15T03:47:26.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdd99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:26.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.042+0000 7fdd99ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:26.045 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":194,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:26.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd88038800 msgr2=0x7fdd8803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:26.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd88038800 0x7fdd8803acb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c09bdb0 tx=0x7fdd94032000 comp rx=0 tx=0).stop 2024-09-15T03:47:26.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 10 monclient: shutdown 2024-09-15T03:47:26.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 msgr2=0x7fdd9c096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c096fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000cca0 tx=0x7fdd900053f0 comp rx=0 tx=0).stop 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 shutdown_connections 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdd88038800 0x7fdd8803acb0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd9c136bb0 0x7fdd9c0974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c131ab0 0x7fdd9c096fa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 --2- 172.21.15.153:0/3355969552 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd9c09e910 0x7fdd9c096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 >> 172.21.15.153:0/3355969552 conn(0x7fdd9c067d30 msgr2=0x7fdd9c09d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 shutdown_connections 2024-09-15T03:47:26.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:26.044+0000 7fdda3a6f640 1 -- 172.21.15.153:0/3355969552 wait complete. 2024-09-15T03:47:29.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.159+0000 7f41a471d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.159+0000 7f41a471d640 10 monclient: build_initial_monmap 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.159+0000 7f41a471d640 10 monclient: monmap: 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:29.159919+0000 2024-09-15T03:47:29.160 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:29.159919+0000 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.159+0000 7f41a471d640 1 Processor -- start 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.159+0000 7f41a471d640 1 -- start start 2024-09-15T03:47:29.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.159+0000 7f41a471d640 10 monclient: init 2024-09-15T03:47:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T03:47:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f419c131ab0 0x7f419c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): picked mon.noname-c con 0x7f419c131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:29.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f419c09e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f419c136bb0 0x7f419c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): picked mon.noname-a con 0x7f419c136bb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c09faf0 con 0x7f419c09e910 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c09fd30 con 0x7f419c136bb0 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c0a1110 con 0x7f419c131ab0 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a471d640 10 monclient(hunting): authenticate will time out at 1846.799495s 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f419c136bb0 0x7f419c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f419c136bb0 0x7f419c138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57754/0 (socket says 172.21.15.153:57754) 2024-09-15T03:47:29.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 -- 172.21.15.153:0/2231496346 learned_addr learned my addr 172.21.15.153:0/2231496346 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): get_auth_request con 0x7f419c136bb0 auth_method 0 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a1cb8640 1 --1- 172.21.15.153:0/2231496346 >> v1:172.21.15.153:6789/0 conn(0x7f419c09e910 0x7f419c09ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55046/0 (socket says 172.21.15.153:55046) 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a24b9640 1 --2- 172.21.15.153:0/2231496346 >> v2:172.21.15.181:3300/0 conn(0x7f419c131ab0 0x7f419c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient(hunting): handle_auth_done global_id 214284 payload 274 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 -- 172.21.15.153:0/2231496346 >> v2:172.21.15.181:3300/0 conn(0x7f419c131ab0 msgr2=0x7f419c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 --2- 172.21.15.153:0/2231496346 >> v2:172.21.15.181:3300/0 conn(0x7f419c131ab0 0x7f419c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 -- 172.21.15.153:0/2231496346 >> v1:172.21.15.153:6789/0 conn(0x7f419c09e910 legacy=0x7f419c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient: _finish_hunting 0 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 monclient: found mon.noname-a 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a2cba640 1 -- 172.21.15.153:0/2231496346 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0a13f0 con 0x7f419c136bb0 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a24b9640 10 monclient: get_auth_request con 0x7f419c131ab0 auth_method 0 2024-09-15T03:47:29.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.160+0000 7f41a24b9640 1 --2- 172.21.15.153:0/2231496346 >> v2:172.21.15.181:3300/0 conn(0x7f419c131ab0 0x7f419c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:47:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a2cba640 1 -- 172.21.15.153:0/2231496346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c136bb0 msgr2=0x7f419c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a2cba640 1 --2- 172.21.15.153:0/2231496346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c136bb0 0x7f419c138f90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4194009f20 tx=0x7f4194009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bbe0300207838e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 1 -- 172.21.15.153:0/2231496346 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419402aea0 con 0x7f419c136bb0 2024-09-15T03:47:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:29.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: dump: 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: _finish_auth 0 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a471d640 5 monclient: authenticate success, global_id 214284 2024-09-15T03:47:29.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 1 -- 172.21.15.153:0/2231496346 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419402d050 con 0x7f419c136bb0 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a471d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a471d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 1 -- 172.21.15.153:0/2231496346 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194007840 con 0x7f419c136bb0 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a471d640 4 set_mon_vals no callback set 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.163+0000 7f41a14b7640 10 monclient: dump: 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 10 monclient: shutdown 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 -- 172.21.15.153:0/2231496346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c136bb0 msgr2=0x7f419c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 --2- 172.21.15.153:0/2231496346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c136bb0 0x7f419c138f90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4194009f20 tx=0x7f4194009840 comp rx=0 tx=0).stop 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 -- 172.21.15.153:0/2231496346 shutdown_connections 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 --2- 172.21.15.153:0/2231496346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c136bb0 0x7f419c138f90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 --2- 172.21.15.153:0/2231496346 >> v2:172.21.15.181:3300/0 conn(0x7f419c131ab0 0x7f419c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 -- 172.21.15.153:0/2231496346 >> 172.21.15.153:0/2231496346 conn(0x7f419c067d30 msgr2=0x7f419c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 -- 172.21.15.153:0/2231496346 shutdown_connections 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 1 -- 172.21.15.153:0/2231496346 wait complete. 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 10 monclient: build_initial_monmap 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.164+0000 7f41a471d640 10 monclient: monmap: 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:29.176 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 Processor -- start 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 -- start start 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient: init 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c09e910 0x7f419c135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): picked mon.noname-b con 0x7f419c09e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 0x7f419c12f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): picked mon.noname-a con 0x7f419c131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 0x7f419c12f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): picked mon.noname-c con 0x7f419c136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:29.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c133d80 con 0x7f419c131ab0 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c133960 con 0x7f419c09e910 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c1333a0 con 0x7f419c136bb0 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a471d640 10 monclient(hunting): authenticate will time out at 1846.804751s 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 0x7f419c12f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 0x7f419c12f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57758/0 (socket says 172.21.15.153:57758) 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a2cba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 0x7f419c12f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 1 -- 172.21.15.153:0/2115612211 learned_addr learned my addr 172.21.15.153:0/2115612211 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): get_auth_request con 0x7f419c131ab0 auth_method 0 2024-09-15T03:47:29.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a2cba640 10 monclient(hunting): get_auth_request con 0x7f419c136bb0 auth_method 0 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a2cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a2cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a2cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.165+0000 7f41a1cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a2cba640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 msgr2=0x7f419c12f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a2cba640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 msgr2=0x7f419c12f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a2cba640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 0x7f419c12f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:29.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a2cba640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 0x7f419c12f550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:29.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a24b9640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c09e910 0x7f419c135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:29.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 10 monclient(hunting): handle_auth_done global_id 214287 payload 824 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 msgr2=0x7f419c12f550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 0x7f419c12f550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c09e910 msgr2=0x7f419c135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c09e910 0x7f419c135f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 10 monclient: _finish_hunting 0 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 1 monclient: found mon.noname-a 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a1cb8640 1 -- 172.21.15.153:0/2115612211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4194009ae0 con 0x7f419c131ab0 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a24b9640 10 monclient: get_auth_request con 0x7f419c09e910 auth_method 0 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.166+0000 7f41a24b9640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c09e910 0x7f419c135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:47:29.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a1cb8640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 msgr2=0x7f419c12f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a1cb8640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 0x7f419c12f010 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c5b0 tx=0x7f418c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c007510 con 0x7f419c131ab0 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: dump: 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: _finish_auth 0 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c0057d0 con 0x7f419c131ab0 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 5 monclient: authenticate success, global_id 214287 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 10 monclient: _renew_subs 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c1330c0 con 0x7f419c131ab0 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:29.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c007690 con 0x7f419c131ab0 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41927fc640 10 monclient: dump: 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:29.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 10 monclient: _renew_subs 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a0cb6640 4 set_mon_vals no callback set 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.168+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c130890 con 0x7f419c131ab0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f418c01b400 con 0x7f419c131ab0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41a471d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41a471d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41a471d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c04bf60 con 0x7f419c131ab0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41927fc640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4180038740 0x7f418003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f418c04c1d0 con 0x7f419c131ab0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.169+0000 7f41a24b9640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4180038740 0x7f418003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:29.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.170+0000 7f41a24b9640 10 monclient: get_auth_request con 0x7f4180038740 auth_method 0 2024-09-15T03:47:29.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.172+0000 7f41a24b9640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4180038740 0x7f418003abf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4198008a80 tx=0x7f4198008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:29.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.173+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f418c01b6b0 con 0x7f419c131ab0 2024-09-15T03:47:29.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.173+0000 7f41927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:29.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.173+0000 7f41927fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:29.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.389+0000 7f41a471d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:29.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.389+0000 7f41a471d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:29.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.389+0000 7f41a471d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:29.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.389+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f419c131350 con 0x7f419c131ab0 2024-09-15T03:47:29.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.390+0000 7f41927fc640 1 -- 172.21.15.153:0/2115612211 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f418c01a360 con 0x7f419c131ab0 2024-09-15T03:47:29.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.390+0000 7f41927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:29.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.390+0000 7f41927fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:29.391 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":194,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4180038740 msgr2=0x7f418003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4180038740 0x7f418003abf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4198008a80 tx=0x7f4198008040 comp rx=0 tx=0).stop 2024-09-15T03:47:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 10 monclient: shutdown 2024-09-15T03:47:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 msgr2=0x7f419c12f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 0x7f419c12f010 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c5b0 tx=0x7f418c007f00 comp rx=0 tx=0).stop 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 shutdown_connections 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4180038740 0x7f418003abf0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f419c136bb0 0x7f419c12f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f419c131ab0 0x7f419c12f010 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 --2- 172.21.15.153:0/2115612211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c09e910 0x7f419c135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 >> 172.21.15.153:0/2115612211 conn(0x7f419c067d30 msgr2=0x7f419c09c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 shutdown_connections 2024-09-15T03:47:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:29.392+0000 7f41a471d640 1 -- 172.21.15.153:0/2115612211 wait complete. 2024-09-15T03:47:32.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:32.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.507+0000 7f683519d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:32.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.507+0000 7f683519d640 10 monclient: build_initial_monmap 2024-09-15T03:47:32.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.507+0000 7f683519d640 10 monclient: monmap: 2024-09-15T03:47:32.508 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:32.508269+0000 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:32.508269+0000 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:32.509 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 Processor -- start 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 -- start start 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient: init 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): picked mon.noname-c con 0x7f683009ad90 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:32.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): picked mon.noname-b con 0x7f683009b160 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): picked mon.noname-a con 0x7f6830131960 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683009d890 con 0x7f6830131960 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68300692c0 con 0x7f683009ad90 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683009db70 con 0x7f683009b160 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f683519d640 10 monclient(hunting): authenticate will time out at 1850.147834s 2024-09-15T03:47:32.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f682f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.508+0000 7f6827fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57764/0 (socket says 172.21.15.153:57764) 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40700/0 (socket says 172.21.15.153:40700) 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 learned_addr learned my addr 172.21.15.153:0/697323413 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f6830131960 auth_method 0 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 10 monclient(hunting): get_auth_request con 0x7f683009b160 auth_method 0 2024-09-15T03:47:32.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f683009ad90 auth_method 0 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:32.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:32.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.509+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:32.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:32.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f6830131960 auth_method 0 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 10 monclient(hunting): get_auth_request con 0x7f683009b160 auth_method 0 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:32.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f683009ad90 auth_method 0 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:32.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:32.710+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f6830131960 auth_method 0 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 10 monclient(hunting): get_auth_request con 0x7f683009b160 auth_method 0 2024-09-15T03:47:33.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f683009ad90 auth_method 0 2024-09-15T03:47:33.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:33.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:33.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:33.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:33.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:33.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:33.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.111+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f6830131960 auth_method 0 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 10 monclient(hunting): get_auth_request con 0x7f683009b160 auth_method 0 2024-09-15T03:47:33.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f683009ad90 auth_method 0 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f6827fff640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:33.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:33.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:33.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:33.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:33.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:33.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:33.912+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 10 monclient(hunting): tick 2024-09-15T03:47:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 monclient(hunting): continuing hunt 2024-09-15T03:47:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:47:35.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 msgr2=0x7f683009b530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f683009b160 0x7f683009b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 msgr2=0x7f683009c5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.181:3300/0 conn(0x7f683009ad90 0x7f683009c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 -- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 msgr2=0x7f6830131dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.508+0000 7f682d59a640 1 --2- 172.21.15.153:0/697323413 >> v2:172.21.15.153:3300/0 conn(0x7f6830131960 0x7f6830131dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.509+0000 7f682d59a640 10 monclient: _add_conns ranks=[0,1] 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.509+0000 7f682d59a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f681c003650 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.509+0000 7f682d59a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f681c007350 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.509+0000 7f682d59a640 1 -- 172.21.15.153:0/697323413 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f683009d5b0 con 0x7f681c003650 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.509+0000 7f682d59a640 1 -- 172.21.15.153:0/697323413 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c00c0c0 con 0x7f681c007350 2024-09-15T03:47:35.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.509+0000 7f682d59a640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.511+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> v1:172.21.15.153:6789/0 conn(0x7f681c003650 legacy=0x7f681c005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:35.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:35.511+0000 7f6827fff640 1 -- 172.21.15.153:0/697323413 >> v1:172.21.15.181:6789/0 conn(0x7f681c007350 legacy=0x7f681c009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:36.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396733077 0 0) 0x7f681c00c0c0 con 0x7f681c007350 2024-09-15T03:47:36.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:36.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): my global_id is 204343 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68100034e0 con 0x7f681c007350 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 338456361 0 0) 0x7f68100034e0 con 0x7f681c007350 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 >> v1:172.21.15.153:6789/0 conn(0x7f681c003650 legacy=0x7f681c005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c00d2c0 con 0x7f681c007350 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: _finish_auth 0 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f681800d480 con 0x7f681c007350 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:36.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: dump: 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:47:36.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 >> v1:172.21.15.181:6789/0 conn(0x7f681c007350 legacy=0x7f681c009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682f59e640 1 -- 172.21.15.153:0/697323413 reap_dead start 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 0x7f6810003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): picked mon.c con 0x7f6810003af0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6810004780 0x7f6810004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): picked mon.a con 0x7f6810004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:36.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68100099c0 0x7f681000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): picked mon.b con 0x7f68100099c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:36.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681000c3e0 con 0x7f6810004780 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681000c550 con 0x7f68100099c0 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681000c710 con 0x7f6810003af0 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682e59c640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:47:36.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 0x7f6810003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f6810003af0 auth_method 0 2024-09-15T03:47:36.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:36.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.525+0000 7f682f59e640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68100099c0 0x7f681000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f68100099c0 auth_method 0 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:36.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient(hunting): handle_auth_done global_id 204343 payload 293 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68100099c0 msgr2=0x7f681000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68100099c0 0x7f681000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6810004780 msgr2=0x7f6810004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6810004780 0x7f6810004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 1 monclient: found mon.c 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68100034e0 con 0x7f6810003af0 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 10 monclient: _finish_auth 0 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f682ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f683519d640 5 monclient: authenticate success, global_id 204343 2024-09-15T03:47:36.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.526+0000 7f683519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:36.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682ed9d640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 msgr2=0x7f6810003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682ed9d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 0x7f6810003ee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6820006ea0 tx=0x7f6820008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e2a9842846f3ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682002b800 con 0x7f6810003af0 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: dump: 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:36.550 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f683519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682002f070 con 0x7f6810003af0 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 1 -- 172.21.15.153:0/697323413 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682002bbb0 con 0x7f6810003af0 2024-09-15T03:47:36.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f683519d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f683519d640 4 set_mon_vals no callback set 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.529+0000 7f682e59c640 10 monclient: dump: 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:36.552 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 10 monclient: shutdown 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 -- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 msgr2=0x7f6810003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 0x7f6810003ee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6820006ea0 tx=0x7f6820008040 comp rx=0 tx=0).stop 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 -- 172.21.15.153:0/697323413 shutdown_connections 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68100099c0 0x7f681000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6810004780 0x7f6810004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 --2- 172.21.15.153:0/697323413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810003af0 0x7f6810003ee0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.530+0000 7f683519d640 1 -- 172.21.15.153:0/697323413 >> 172.21.15.153:0/697323413 conn(0x7f68300678a0 msgr2=0x7f68300a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:36.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.531+0000 7f683519d640 1 -- 172.21.15.153:0/697323413 shutdown_connections 2024-09-15T03:47:36.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.531+0000 7f683519d640 1 -- 172.21.15.153:0/697323413 wait complete. 2024-09-15T03:47:36.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.531+0000 7f683519d640 10 monclient: build_initial_monmap 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.531+0000 7f683519d640 10 monclient: monmap: 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:36.555 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 Processor -- start 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 -- start start 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient: init 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:47:36.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f6830141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): picked mon.noname-b con 0x7f6810003af0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f683006e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): picked mon.noname-c con 0x7f6810004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 0x7f68301448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): picked mon.noname-a con 0x7f68100099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683006f250 con 0x7f68100099c0 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68300a0680 con 0x7f6810003af0 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683006aad0 con 0x7f6810004780 2024-09-15T03:47:36.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:36.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f683519d640 10 monclient(hunting): authenticate will time out at 1854.171845s 2024-09-15T03:47:36.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f6827fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f683006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f682f59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 0x7f68301448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.532+0000 7f6827fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f683006e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33046/0 (socket says 172.21.15.153:33046) 2024-09-15T03:47:36.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 1 -- 172.21.15.153:0/1284092155 learned_addr learned my addr 172.21.15.153:0/1284092155 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 10 monclient(hunting): get_auth_request con 0x7f6810004780 auth_method 0 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f68100099c0 auth_method 0 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f6830141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f6810003af0 auth_method 0 2024-09-15T03:47:36.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 msgr2=0x7f683006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 msgr2=0x7f683006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f683006e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:36.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f6827fff640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f683006e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient(hunting): handle_auth_done global_id 214290 payload 824 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 msgr2=0x7f683006e300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f683006e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 msgr2=0x7f6830141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f6830141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 1 monclient: found mon.noname-a 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6830141e10 con 0x7f68100099c0 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.533+0000 7f682ed9d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f6830141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 msgr2=0x7f68301448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 0x7f68301448e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68300640a0 tx=0x7f682800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 msgr2=0x7f68301448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 msgr2=0x7f68301448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 0x7f68301448e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68300640a0 tx=0x7f682800b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 0x7f68301448e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68300640a0 tx=0x7f682800b040 comp rx=0 tx=0).stop 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 msgr2=0x7f68301448e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68100099c0 0x7f68301448e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f681c002b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f6810004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f681c009300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f6810003af0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 0x7f681c00b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f681c00b0b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c00be30 con 0x7f681c00b0b0 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c006230 con 0x7f6810003af0 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0063c0 con 0x7f6810004780 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.535+0000 7f6826ffd640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 0x7f681c00b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): get_auth_request con 0x7f681c00b0b0 auth_method 0 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f681c002b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): get_auth_request con 0x7f6810004780 auth_method 0 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f681c009300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): get_auth_request con 0x7f6810003af0 auth_method 0 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f6827fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient(hunting): handle_auth_done global_id 214290 payload 293 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 msgr2=0x7f681c002b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f681c002b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 msgr2=0x7f681c009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f681c009300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 monclient: found mon.noname-a 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.536+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c006640 con 0x7f681c00b0b0 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f682f59e640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 msgr2=0x7f681c00b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f682f59e640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 0x7f681c00b520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f683006c980 tx=0x7f682801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828028070 con 0x7f681c00b0b0 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f6826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f6826ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f6826ffd640 10 monclient: dump: 2024-09-15T03:47:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f6826ffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.538+0000 7f6826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 5 monclient: authenticate success, global_id 214290 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682800f040 con 0x7f681c00b0b0 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f6826ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828018cf0 con 0x7f681c00b0b0 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 10 monclient: _renew_subs 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683006c530 con 0x7f681c00b0b0 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f6826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f6826ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f6826ffd640 10 monclient: dump: 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f682dd9b640 4 set_mon_vals no callback set 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 10 monclient: _renew_subs 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.539+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683006b7e0 con 0x7f681c00b0b0 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f6828021660 con 0x7f681c00b0b0 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f683519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f683519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f683519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f6826ffd640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68100099c0 0x7f681c043610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683004bf60 con 0x7f681c00b0b0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f682804e280 con 0x7f681c00b0b0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f682ed9d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68100099c0 0x7f681c043610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.540+0000 7f682ed9d640 10 monclient: get_auth_request con 0x7f68100099c0 auth_method 0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.543+0000 7f682ed9d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68100099c0 0x7f681c043610 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6820006fd0 tx=0x7f6820037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.545+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6828021910 con 0x7f681c00b0b0 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.545+0000 7f6826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:36.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.545+0000 7f6826ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.762+0000 7f683519d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.762+0000 7f683519d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:36.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.762+0000 7f683519d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:36.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.762+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f68300a0460 con 0x7f681c00b0b0 2024-09-15T03:47:36.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.763+0000 7f6826ffd640 1 -- 172.21.15.153:0/1284092155 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f682801f360 con 0x7f681c00b0b0 2024-09-15T03:47:36.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.763+0000 7f6826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:36.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.763+0000 7f6826ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:36.764 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":198,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68100099c0 msgr2=0x7f681c043610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68100099c0 0x7f681c043610 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6820006fd0 tx=0x7f6820037000 comp rx=0 tx=0).stop 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 10 monclient: shutdown 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 msgr2=0x7f681c00b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 0x7f681c00b520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f683006c980 tx=0x7f682801f000 comp rx=0 tx=0).stop 2024-09-15T03:47:36.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 shutdown_connections 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f68100099c0 0x7f681c043610 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f681c00b0b0 0x7f681c00b520 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6810003af0 0x7f681c009300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 --2- 172.21.15.153:0/1284092155 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6810004780 0x7f681c002b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 >> 172.21.15.153:0/1284092155 conn(0x7f68300678a0 msgr2=0x7f6830133f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 shutdown_connections 2024-09-15T03:47:36.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:36.765+0000 7f683519d640 1 -- 172.21.15.153:0/1284092155 wait complete. 2024-09-15T03:47:39.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.883+0000 7feaeaf1d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.883+0000 7feaeaf1d640 10 monclient: build_initial_monmap 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.883+0000 7feaeaf1d640 10 monclient: monmap: 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:39.883999+0000 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:39.883999+0000 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:39.884 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:39.885 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:39.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 Processor -- start 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 -- start start 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient: init 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feae4131ab0 0x7feae4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): picked mon.noname-a con 0x7feae4131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feae409e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): picked mon.noname-b con 0x7feae4136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:39.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae409faf0 con 0x7feae409e910 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae409fd30 con 0x7feae4131ab0 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae40a1110 con 0x7feae4136bb0 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feaeaf1d640 10 monclient(hunting): authenticate will time out at 1857.523646s 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feae4131ab0 0x7feae4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feae4131ab0 0x7feae4131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42712/0 (socket says 172.21.15.153:42712) 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33066/0 (socket says 172.21.15.153:33066) 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 1 -- 172.21.15.153:0/2337504503 learned_addr learned my addr 172.21.15.153:0/2337504503 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): get_auth_request con 0x7feae4131ab0 auth_method 0 2024-09-15T03:47:39.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 10 monclient(hunting): get_auth_request con 0x7feae4136bb0 auth_method 0 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae8cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 1 -- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 msgr2=0x7feae4138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 1 -- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 msgr2=0x7feae4138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.884+0000 7feae94ba640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:39.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 10 monclient(hunting): handle_auth_done global_id 214293 payload 274 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 1 -- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 msgr2=0x7feae4138f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 1 -- 172.21.15.153:0/2337504503 >> v1:172.21.15.181:6789/0 conn(0x7feae409e910 legacy=0x7feae409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 1 monclient: found mon.noname-a 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:39.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.885+0000 7feae8cb9640 1 -- 172.21.15.153:0/2337504503 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae40a13f0 con 0x7feae4131ab0 2024-09-15T03:47:39.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.887+0000 7feae8cb9640 1 -- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4131ab0 msgr2=0x7feae4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.887+0000 7feae8cb9640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4131ab0 0x7feae4131e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fead4009910 tx=0x7fead40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10289e8074d9e4df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 1 -- 172.21.15.153:0/2337504503 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4032440 con 0x7feae4131ab0 2024-09-15T03:47:39.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: dump: 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:39.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: _finish_auth 0 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 1 -- 172.21.15.153:0/2337504503 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead40325a0 con 0x7feae4131ab0 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 5 monclient: authenticate success, global_id 214293 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 1 -- 172.21.15.153:0/2337504503 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4032980 con 0x7feae4131ab0 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feae37fe640 10 monclient: dump: 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 4 set_mon_vals no callback set 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 10 monclient: shutdown 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 -- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4131ab0 msgr2=0x7feae4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4131ab0 0x7feae4131e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fead4009910 tx=0x7fead40091d0 comp rx=0 tx=0).stop 2024-09-15T03:47:39.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 -- 172.21.15.153:0/2337504503 shutdown_connections 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae4136bb0 0x7feae4138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/2337504503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4131ab0 0x7feae4131e80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 -- 172.21.15.153:0/2337504503 >> 172.21.15.153:0/2337504503 conn(0x7feae4067d30 msgr2=0x7feae40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 -- 172.21.15.153:0/2337504503 shutdown_connections 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.888+0000 7feaeaf1d640 1 -- 172.21.15.153:0/2337504503 wait complete. 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.889+0000 7feaeaf1d640 10 monclient: build_initial_monmap 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.889+0000 7feaeaf1d640 10 monclient: monmap: 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.889+0000 7feaeaf1d640 1 Processor -- start 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.889+0000 7feaeaf1d640 1 -- start start 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.889+0000 7feaeaf1d640 10 monclient: init 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): picked mon.noname-c con 0x7feae409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feae4131ab0 0x7feae4097940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:39.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): picked mon.noname-b con 0x7feae4131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 0x7feae4097e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): picked mon.noname-a con 0x7feae4136bb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae409a870 con 0x7feae4136bb0 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae409bde0 con 0x7feae4131ab0 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae409df10 con 0x7feae409e910 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feaeaf1d640 10 monclient(hunting): authenticate will time out at 1857.529267s 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 0x7feae4097e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33072/0 (socket says 172.21.15.153:33072) 2024-09-15T03:47:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 -- 172.21.15.153:0/3084390284 learned_addr learned my addr 172.21.15.153:0/3084390284 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 10 monclient(hunting): get_auth_request con 0x7feae409e910 auth_method 0 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): get_auth_request con 0x7feae4136bb0 auth_method 0 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 msgr2=0x7feae4097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 msgr2=0x7feae4097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feae4131ab0 0x7feae4097940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae8cb9640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): get_auth_request con 0x7feae4131ab0 auth_method 0 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient(hunting): handle_auth_done global_id 214296 payload 824 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 msgr2=0x7feae4097400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:39.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feae4131ab0 msgr2=0x7feae4097940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feae4131ab0 0x7feae4097940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient: _finish_hunting 0 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 monclient: found mon.noname-a 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.890+0000 7feae94ba640 1 -- 172.21.15.153:0/3084390284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4009450 con 0x7feae4136bb0 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.892+0000 7feae94ba640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 msgr2=0x7feae4097e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.892+0000 7feae94ba640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 0x7feae4097e80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feadc006ea0 tx=0x7feadc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc0098b0 con 0x7feae4136bb0 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: dump: 2024-09-15T03:47:39.907 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc009a10 con 0x7feae4136bb0 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 5 monclient: authenticate success, global_id 214296 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 10 monclient: _renew_subs 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae4099b50 con 0x7feae4136bb0 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc009dd0 con 0x7feae4136bb0 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae0ff9640 10 monclient: dump: 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:39.909 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 10 monclient: _renew_subs 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feae2ffd640 4 set_mon_vals no callback set 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.893+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae409b1c0 con 0x7feae4136bb0 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7feadc014620 con 0x7feae4136bb0 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feaeaf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feaeaf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:39.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feaeaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae404bf60 con 0x7feae4136bb0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feae0ff9640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fead0038740 0x7fead003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7feadc04cc80 con 0x7feae4136bb0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feae8cb9640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fead0038740 0x7fead003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.894+0000 7feae8cb9640 10 monclient: get_auth_request con 0x7fead0038740 auth_method 0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.897+0000 7feae8cb9640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fead0038740 0x7fead003abf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fead4009210 tx=0x7fead4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.898+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7feadc025d60 con 0x7feae4136bb0 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.898+0000 7feae0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:39.898+0000 7feae0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.114+0000 7feaeaf1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.114+0000 7feaeaf1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.114+0000 7feaeaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.114+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feae409aab0 con 0x7feae4136bb0 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.115+0000 7feae0ff9640 1 -- 172.21.15.153:0/3084390284 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7feadc018020 con 0x7feae4136bb0 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.115+0000 7feae0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.115+0000 7feae0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fead0038740 msgr2=0x7fead003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fead0038740 0x7fead003abf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fead4009210 tx=0x7fead4032000 comp rx=0 tx=0).stop 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 10 monclient: shutdown 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 msgr2=0x7feae4097e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 0x7feae4097e80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feadc006ea0 tx=0x7feadc008040 comp rx=0 tx=0).stop 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 shutdown_connections 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fead0038740 0x7fead003abf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feae4136bb0 0x7feae4097e80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feae4131ab0 0x7feae4097940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 --2- 172.21.15.153:0/3084390284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feae409e910 0x7feae4097400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 >> 172.21.15.153:0/3084390284 conn(0x7feae4067d30 msgr2=0x7feae409e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 shutdown_connections 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.117+0000 7feaeaf1d640 1 -- 172.21.15.153:0/3084390284 wait complete. 2024-09-15T03:47:40.128 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:47:40.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:47:40.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:47:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.223+0000 7f5bfd51d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:40.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.223+0000 7f5bfd51d640 10 monclient: build_initial_monmap 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.223+0000 7f5bfd51d640 10 monclient: monmap: 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:40.223773+0000 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:40.223773+0000 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:40.224 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.223+0000 7f5bfd51d640 1 Processor -- start 2024-09-15T03:47:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.223+0000 7f5bfd51d640 1 -- start start 2024-09-15T03:47:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.223+0000 7f5bfd51d640 10 monclient: init 2024-09-15T03:47:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:40.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5bf8098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient(hunting): picked mon.noname-b con 0x7f5bf80981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bf8131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bf8132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf8098ad0 con 0x7f5bf8131da0 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf8067f30 con 0x7f5bf8132170 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8131bd0 con 0x7f5bf80981c0 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bfd51d640 10 monclient(hunting): authenticate will time out at 1857.863195s 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5bf8098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf67fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5bf8131da0 0x7f5bf8067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41008/0 (socket says 172.21.15.153:41008) 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf67fc640 1 -- 172.21.15.153:0/20549291 learned_addr learned my addr 172.21.15.153:0/20549291 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 10 monclient(hunting): get_auth_request con 0x7f5bf80981c0 auth_method 0 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 msgr2=0x7f5bf8098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 msgr2=0x7f5bf8098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5bf8098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.224+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5bf8098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf67fc640 1 -- 172.21.15.153:0/20549291 >> v1:172.21.15.153:6789/0 conn(0x7f5bf8131da0 legacy=0x7f5bf8067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf77fe640 1 -- 172.21.15.153:0/20549291 >> v1:172.21.15.181:6789/0 conn(0x7f5bf8132170 legacy=0x7f5bf8067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335828466 0 0) 0x7f5bf8098ad0 con 0x7f5bf8131da0 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient(hunting): my global_id is 214299 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be00034e0 con 0x7f5bf8131da0 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463951819 0 0) 0x7f5be00034e0 con 0x7f5bf8131da0 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 msgr2=0x7f5bf8098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5bf8098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 >> v1:172.21.15.181:6789/0 conn(0x7f5bf8132170 legacy=0x7f5bf8067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: _finish_hunting 0 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:47:40.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf809ca60 con 0x7f5bf8131da0 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bfd51d640 5 monclient: authenticate success, global_id 214299 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bfd51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f5bec001720 con 0x7f5bf8131da0 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: dump: 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.240 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 >> v1:172.21.15.153:6789/0 conn(0x7f5bf8131da0 legacy=0x7f5bf8067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5be0003b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 10 monclient(hunting): picked mon.c con 0x7f5bf80981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.226+0000 7f5bf5ffb640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5be00040f0 0x7f5be0004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): picked mon.b con 0x7f5be00040f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:40.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be0004de0 0x7f5be0005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): picked mon.a con 0x7f5be0004de0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8131bd0 con 0x7f5be0004de0 2024-09-15T03:47:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be000bd50 con 0x7f5be00040f0 2024-09-15T03:47:40.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be000bf10 con 0x7f5bf80981c0 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bfd51d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5be0003b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): get_auth_request con 0x7f5bf80981c0 auth_method 0 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be0004de0 0x7f5be0005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5be00040f0 0x7f5be0004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): get_auth_request con 0x7f5be0004de0 auth_method 0 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): get_auth_request con 0x7f5be00040f0 auth_method 0 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_done global_id 214299 payload 293 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5be00040f0 msgr2=0x7f5be0004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5be00040f0 0x7f5be0004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be0004de0 msgr2=0x7f5be0005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be0004de0 0x7f5be0005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:47:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 monclient: found mon.c 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/20549291 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be00034e0 con 0x7f5bf80981c0 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.227+0000 7f5bf77fe640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be0004de0 0x7f5be0005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 msgr2=0x7f5be0003b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5be0003b00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5be8006fd0 tx=0x7f5be8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19356b261f460824 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be802aab0 con 0x7f5bf80981c0 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: dump: 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.248 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be802e070 con 0x7f5bf80981c0 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 1 -- 172.21.15.153:0/20549291 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be802ae60 con 0x7f5bf80981c0 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 4 set_mon_vals no callback set 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf5ffb640 10 monclient: dump: 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 10 monclient: shutdown 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 1 -- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 msgr2=0x7f5be0003b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 1 --2- 172.21.15.153:0/20549291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf80981c0 0x7f5be0003b00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5be8006fd0 tx=0x7f5be8008040 comp rx=0 tx=0).stop 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bf77fe640 1 -- 172.21.15.153:0/20549291 reap_dead start 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 1 -- 172.21.15.153:0/20549291 shutdown_connections 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 1 -- 172.21.15.153:0/20549291 >> 172.21.15.153:0/20549291 conn(0x7f5bf80a1e30 msgr2=0x7f5bf80a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 1 -- 172.21.15.153:0/20549291 shutdown_connections 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.230+0000 7f5bfd51d640 1 -- 172.21.15.153:0/20549291 wait complete. 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.231+0000 7f5bfd51d640 10 monclient: build_initial_monmap 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.231+0000 7f5bfd51d640 10 monclient: monmap: 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:40.251 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.231+0000 7f5bfd51d640 1 Processor -- start 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.231+0000 7f5bfd51d640 1 -- start start 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.231+0000 7f5bfd51d640 10 monclient: init 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 0x7f5bf8140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): picked mon.noname-a con 0x7f5bf80981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:40.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 0x7f5bf8130930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): picked mon.noname-c con 0x7f5bf811caf0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf8130e70 0x7f5bf811a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): picked mon.noname-b con 0x7f5bf8130e70 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf811de30 con 0x7f5bf80981c0 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8098cb0 con 0x7f5bf8130e70 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf80a0490 con 0x7f5bf811caf0 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bfd51d640 10 monclient(hunting): authenticate will time out at 1857.871408s 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 0x7f5bf8140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 0x7f5bf8130930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 0x7f5bf8140f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42734/0 (socket says 172.21.15.153:42734) 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/3917150431 learned_addr learned my addr 172.21.15.153:0/3917150431 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 10 monclient(hunting): get_auth_request con 0x7f5bf811caf0 auth_method 0 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): get_auth_request con 0x7f5bf80981c0 auth_method 0 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 msgr2=0x7f5bf8130930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 msgr2=0x7f5bf8130930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 0x7f5bf8130930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 0x7f5bf8130930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf77fe640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf8130e70 0x7f5bf811a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf77fe640 10 monclient(hunting): get_auth_request con 0x7f5bf8130e70 auth_method 0 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient(hunting): handle_auth_done global_id 214302 payload 824 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 msgr2=0x7f5bf8130930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bf811caf0 0x7f5bf8130930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf8130e70 msgr2=0x7f5bf811a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf8130e70 0x7f5bf811a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 monclient: found mon.noname-a 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf811bf90 con 0x7f5bf80981c0 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.232+0000 7f5bf77fe640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf8130e70 0x7f5bf811a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bf6ffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 msgr2=0x7f5bf8140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bf6ffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 0x7f5bf8140f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009a50 tx=0x7f5be8035000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be8043070 con 0x7f5bf80981c0 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:40.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: dump: 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:40.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be8034db0 con 0x7f5bf80981c0 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be803f670 con 0x7f5bf80981c0 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bdeffd640 10 monclient: dump: 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 5 monclient: authenticate success, global_id 214302 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 10 monclient: _renew_subs 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf811da50 con 0x7f5bf80981c0 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bf57fa640 4 set_mon_vals no callback set 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 10 monclient: _renew_subs 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.235+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf8141440 con 0x7f5bf80981c0 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5be803faa0 con 0x7f5bf80981c0 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bfd51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bfd51d640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 0x7f5bf811add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bfd51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8065f90 con 0x7f5bf811aa00 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 0x7f5bf811add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient: get_auth_request con 0x7f5bf811aa00 auth_method 0 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bdeffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5be40387c0 0x7f5be403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5be806c1e0 con 0x7f5bf80981c0 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_done global_id 214305 payload 824 2024-09-15T03:47:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf67fc640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bf8141660 con 0x7f5bf811aa00 2024-09-15T03:47:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf77fe640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5be40387c0 0x7f5be403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.236+0000 7f5bf77fe640 10 monclient: get_auth_request con 0x7f5be40387c0 auth_method 0 2024-09-15T03:47:40.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.238+0000 7f5bf67fc640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 msgr2=0x7f5bf811add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.238+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 0x7f5bf811add0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5bec001900 tx=0x7f5bec009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.238+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec011a60 con 0x7f5bf811aa00 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.238+0000 7f5bdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.238+0000 7f5bdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.238+0000 7f5bdeffd640 10 monclient: dump: 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5bec011bc0 con 0x7f5bf811aa00 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 msgr2=0x7f5bf811add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 0x7f5bf811add0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5bec001900 tx=0x7f5bec009040 comp rx=0 tx=0).stop 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 msgr2=0x7f5bf811add0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bdeffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf811aa00 0x7f5bf811add0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.239+0000 7f5bf77fe640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5be40387c0 0x7f5be403ac70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5bf809d0b0 tx=0x7f5bf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bfd51d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bfd51d640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 0x7f5bf8099710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bfd51d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf811df70 con 0x7f5bf8099310 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 0x7f5bf8099710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bf67fc640 10 monclient: get_auth_request con 0x7f5bf8099310 auth_method 0 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.244+0000 7f5bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.245+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.245+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.245+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.245+0000 7f5bf67fc640 10 monclient(hunting): handle_auth_done global_id 214308 payload 824 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.245+0000 7f5bf67fc640 1 -- 172.21.15.153:0/3917150431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5bf811b310 con 0x7f5bf8099310 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bf67fc640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 msgr2=0x7f5bf8099710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bf67fc640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 0x7f5bf8099710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5bec019000 tx=0x7f5bec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec01ec90 con 0x7f5bf8099310 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 10 monclient: dump: 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f5bec01edf0 con 0x7f5bf8099310 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 msgr2=0x7f5bf8099710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 0x7f5bf8099710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5bec019000 tx=0x7f5bec017000 comp rx=0 tx=0).stop 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 msgr2=0x7f5bf8099710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.247+0000 7f5bdeffd640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8099310 0x7f5bf8099710 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5bec019000 tx=0x7f5bec017000 comp rx=0 tx=0).stop 2024-09-15T03:47:40.268 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 200, 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:40.269 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:40.270 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.271 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:40.272 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:40.273 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:40.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:40.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:40.276 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:40.277 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:40.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:40.280 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:40.281 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:47:40.281 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:40.281 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:40.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:40.281 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:40.281 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5be40387c0 msgr2=0x7f5be403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5be40387c0 0x7f5be403ac70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5bf809d0b0 tx=0x7f5bf0009040 comp rx=0 tx=0).stop 2024-09-15T03:47:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 10 monclient: shutdown 2024-09-15T03:47:40.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 msgr2=0x7f5bf8140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 --2- 172.21.15.153:0/3917150431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf80981c0 0x7f5bf8140f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009a50 tx=0x7f5be8035000 comp rx=0 tx=0).stop 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bf77fe640 1 -- 172.21.15.153:0/3917150431 reap_dead start 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bf77fe640 1 -- 172.21.15.153:0/3917150431 reap_dead start 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 shutdown_connections 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 >> 172.21.15.153:0/3917150431 conn(0x7f5bf80a1e30 msgr2=0x7f5bf80a06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 shutdown_connections 2024-09-15T03:47:40.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.248+0000 7f5bfd51d640 1 -- 172.21.15.153:0/3917150431 wait complete. 2024-09-15T03:47:40.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:47:40.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:47:40.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:47:40.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.376+0000 7fd987e34640 10 monclient: get_monmap_and_config 2024-09-15T03:47:40.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.376+0000 7fd987e34640 10 monclient: build_initial_monmap 2024-09-15T03:47:40.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.376+0000 7fd987e34640 10 monclient: monmap: 2024-09-15T03:47:40.377 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:40.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:40.377568+0000 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:40.377568+0000 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:40.378 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.377+0000 7fd987e34640 1 Processor -- start 2024-09-15T03:47:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.377+0000 7fd987e34640 1 -- start start 2024-09-15T03:47:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.377+0000 7fd987e34640 10 monclient: init 2024-09-15T03:47:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-15T03:47:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): picked mon.noname-b con 0x7fd9800981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd980131da0 0x7fd980067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): picked mon.noname-c con 0x7fd980131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd980132170 0x7fd980067650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): picked mon.noname-a con 0x7fd980132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd980131bd0 con 0x7fd980132170 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd980098ad0 con 0x7fd980131da0 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98009ca60 con 0x7fd9800981c0 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd987e34640 10 monclient(hunting): authenticate will time out at 1858.017199s 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd980132170 0x7fd980067650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33104/0 (socket says 172.21.15.153:33104) 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 -- 172.21.15.153:0/3025187448 learned_addr learned my addr 172.21.15.153:0/3025187448 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd980132170 0x7fd980067650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42760/0 (socket says 172.21.15.153:42760) 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 10 monclient(hunting): get_auth_request con 0x7fd9800981c0 auth_method 0 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): get_auth_request con 0x7fd980132170 auth_method 0 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 -- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 msgr2=0x7fd980098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 -- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 msgr2=0x7fd980098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 1 --2- 172.21.15.153:0/3025187448 >> v2:172.21.15.181:3300/0 conn(0x7fd980131da0 0x7fd980067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd985bd0640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): get_auth_request con 0x7fd980131da0 auth_method 0 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9853cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient(hunting): handle_auth_done global_id 214311 payload 274 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 -- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 msgr2=0x7fd980098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 -- 172.21.15.153:0/3025187448 >> v2:172.21.15.181:3300/0 conn(0x7fd980131da0 msgr2=0x7fd980067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 --2- 172.21.15.153:0/3025187448 >> v2:172.21.15.181:3300/0 conn(0x7fd980131da0 0x7fd980067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient: _finish_hunting 0 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 monclient: found mon.noname-a 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.378+0000 7fd9863d1640 1 -- 172.21.15.153:0/3025187448 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98009cca0 con 0x7fd980132170 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd9863d1640 1 -- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980132170 msgr2=0x7fd980067650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd9863d1640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980132170 0x7fd980067650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd98006f880 tx=0x7fd978009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ffad6249a23c8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 1 -- 172.21.15.153:0/3025187448 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978007760 con 0x7fd980132170 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: dump: 2024-09-15T03:47:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: _finish_auth 0 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 1 -- 172.21.15.153:0/3025187448 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97802d050 con 0x7fd980132170 2024-09-15T03:47:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd987e34640 5 monclient: authenticate success, global_id 214311 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd987e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 1 -- 172.21.15.153:0/3025187448 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978007af0 con 0x7fd980132170 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd984bce640 10 monclient: dump: 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd987e34640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.381+0000 7fd987e34640 4 set_mon_vals no callback set 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 10 monclient: shutdown 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 -- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980132170 msgr2=0x7fd980067650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980132170 0x7fd980067650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd98006f880 tx=0x7fd978009780 comp rx=0 tx=0).stop 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 -- 172.21.15.153:0/3025187448 shutdown_connections 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980132170 0x7fd980067650 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 --2- 172.21.15.153:0/3025187448 >> v2:172.21.15.181:3300/0 conn(0x7fd980131da0 0x7fd980067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 --2- 172.21.15.153:0/3025187448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9800981c0 0x7fd980098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 -- 172.21.15.153:0/3025187448 >> 172.21.15.153:0/3025187448 conn(0x7fd9800a1e30 msgr2=0x7fd9800a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 -- 172.21.15.153:0/3025187448 shutdown_connections 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 1 -- 172.21.15.153:0/3025187448 wait complete. 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 10 monclient: build_initial_monmap 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.382+0000 7fd987e34640 10 monclient: monmap: 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:40.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 Processor -- start 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 -- start start 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient: init 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9800981c0 0x7fd980140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): picked mon.noname-b con 0x7fd9800981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 0x7fd9801309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): picked mon.noname-a con 0x7fd980131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 0x7fd98011a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): picked mon.noname-c con 0x7fd980132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd980098cb0 con 0x7fd980131da0 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98009b9a0 con 0x7fd9800981c0 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98011db70 con 0x7fd980132170 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd987e34640 10 monclient(hunting): authenticate will time out at 1858.022560s 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 0x7fd98011a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 0x7fd9801309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 0x7fd9801309c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42766/0 (socket says 172.21.15.153:42766) 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 -- 172.21.15.153:0/26479539 learned_addr learned my addr 172.21.15.153:0/26479539 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 0x7fd98011a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33120/0 (socket says 172.21.15.153:33120) 2024-09-15T03:47:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): get_auth_request con 0x7fd980131da0 auth_method 0 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 10 monclient(hunting): get_auth_request con 0x7fd980132170 auth_method 0 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 msgr2=0x7fd98011a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 msgr2=0x7fd98011a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:40.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 0x7fd98011a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9800981c0 0x7fd980140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9863d1640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 0x7fd98011a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd985bd0640 10 monclient(hunting): get_auth_request con 0x7fd9800981c0 auth_method 0 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd985bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd985bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd985bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient(hunting): handle_auth_done global_id 214314 payload 824 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 msgr2=0x7fd98011a510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980132170 0x7fd98011a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9800981c0 msgr2=0x7fd980140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9800981c0 0x7fd980140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient: _finish_hunting 0 2024-09-15T03:47:40.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 monclient: found mon.noname-a 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.383+0000 7fd9853cf640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978009a20 con 0x7fd980131da0 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd9853cf640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 msgr2=0x7fd9801309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd9853cf640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 0x7fd9801309c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd97000ea50 tx=0x7fd9700090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970009600 con 0x7fd980131da0 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: dump: 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.401 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: _finish_auth 0 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97000cc20 con 0x7fd980131da0 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 5 monclient: authenticate success, global_id 214314 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 10 monclient: _renew_subs 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98011e100 con 0x7fd980131da0 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970013470 con 0x7fd980131da0 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd975ffb640 10 monclient: dump: 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.403 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd977fff640 4 set_mon_vals no callback set 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 10 monclient: _renew_subs 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.386+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd980141b00 con 0x7fd980131da0 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd987e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd970013860 con 0x7fd980131da0 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd987e34640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd980065f90 con 0x7fd980130f00 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 10 monclient: get_auth_request con 0x7fd980130f00 auth_method 0 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd975ffb640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd964038740 0x7fd96403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd97004ccd0 con 0x7fd980131da0 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.387+0000 7fd9863d1640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd964038740 0x7fd96403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.388+0000 7fd9863d1640 10 monclient: get_auth_request con 0x7fd964038740 auth_method 0 2024-09-15T03:47:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.390+0000 7fd9863d1640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd964038740 0x7fd96403abf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd98009c940 tx=0x7fd978037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 10 monclient: get_auth_request con 0x7fd980130f00 auth_method 0 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.588+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient: get_auth_request con 0x7fd980130f00 auth_method 0 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:40.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 10 monclient(hunting): handle_auth_done global_id 234101 payload 824 2024-09-15T03:47:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.989+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd98011cbe0 con 0x7fd980130f00 2024-09-15T03:47:40.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.991+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:40.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.991+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd97c006f60 tx=0x7fd97c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:40.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.991+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c012be0 con 0x7fd980130f00 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.991+0000 7fd975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.991+0000 7fd975ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.991+0000 7fd975ffb640 10 monclient: dump: 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:40.993 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd97c00c040 con 0x7fd980130f00 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd97c006f60 tx=0x7fd97c008040 comp rx=0 tx=0).stop 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 msgr2=0x7fd98011aa50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.992+0000 7fd975ffb640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd980130f00 0x7fd98011aa50 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd97c006f60 tx=0x7fd97c008040 comp rx=0 tx=0).stop 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd987e34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd987e34640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 0x7fd980099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd987e34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801312d0 con 0x7fd98011af90 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 0x7fd980099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient: get_auth_request con 0x7fd98011af90 auth_method 0 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:40.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 10 monclient(hunting): handle_auth_done global_id 234104 payload 824 2024-09-15T03:47:41.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:40.998+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd98011b390 con 0x7fd98011af90 2024-09-15T03:47:41.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.000+0000 7fd985bd0640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 msgr2=0x7fd980099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.000+0000 7fd985bd0640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 0x7fd980099310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd97c01e000 tx=0x7fd97c021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.000+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0218f0 con 0x7fd98011af90 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.000+0000 7fd975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.000+0000 7fd975ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.000+0000 7fd975ffb640 10 monclient: dump: 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.002 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fd97c02a450 con 0x7fd98011af90 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 msgr2=0x7fd980099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 0x7fd980099310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd97c01e000 tx=0x7fd97c021000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 msgr2=0x7fd980099310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd975ffb640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd98011af90 0x7fd980099310 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd97c01e000 tx=0x7fd97c021000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:41.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 200, 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:41.004 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:41.005 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:41.006 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:41.007 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:41.008 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:41.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.011 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:41.012 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:41.015 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:41.015 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:41.015 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.015 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.015 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:41.016 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:41.017 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.017 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.017 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:41.017 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:41.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd964038740 msgr2=0x7fd96403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd964038740 0x7fd96403abf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd98009c940 tx=0x7fd978037000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 10 monclient: shutdown 2024-09-15T03:47:41.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:41.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 msgr2=0x7fd9801309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 --2- 172.21.15.153:0/26479539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980131da0 0x7fd9801309c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd97000ea50 tx=0x7fd9700090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd9863d1640 1 -- 172.21.15.153:0/26479539 reap_dead start 2024-09-15T03:47:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd9863d1640 1 -- 172.21.15.153:0/26479539 reap_dead start 2024-09-15T03:47:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 shutdown_connections 2024-09-15T03:47:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 >> 172.21.15.153:0/26479539 conn(0x7fd9800a1e30 msgr2=0x7fd980096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 shutdown_connections 2024-09-15T03:47:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.001+0000 7fd987e34640 1 -- 172.21.15.153:0/26479539 wait complete. 2024-09-15T03:47:41.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:47:41.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:47:41.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.115+0000 7fef3ad9d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.115+0000 7fef3ad9d640 10 monclient: build_initial_monmap 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.115+0000 7fef3ad9d640 10 monclient: monmap: 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:41.116155+0000 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:41.116155+0000 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 1 Processor -- start 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 1 -- start start 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient: init 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3409c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient(hunting): picked mon.noname-b con 0x7fef3409a7f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef3409abc0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef3409afb0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3409b410 con 0x7fef3409afb0 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef34130500 con 0x7fef3409abc0 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef340a2050 con 0x7fef3409a7f0 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3ad9d640 10 monclient(hunting): authenticate will time out at 1858.755779s 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3409c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3409c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33156/0 (socket says 172.21.15.153:33156) 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef3933a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fef3409afb0 0x7fef3412fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41010/0 (socket says 172.21.15.153:41010) 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 1 -- 172.21.15.153:0/2561486760 learned_addr learned my addr 172.21.15.153:0/2561486760 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): get_auth_request con 0x7fef3409a7f0 auth_method 0 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.116+0000 7fef38b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 10 monclient(hunting): handle_auth_done global_id 234107 payload 274 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 1 -- 172.21.15.153:0/2561486760 >> v1:172.21.15.181:6789/0 conn(0x7fef3409abc0 legacy=0x7fef3412f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 1 -- 172.21.15.153:0/2561486760 >> v1:172.21.15.153:6789/0 conn(0x7fef3409afb0 legacy=0x7fef3412fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 10 monclient: _finish_hunting 0 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 1 monclient: found mon.noname-b 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.117+0000 7fef38b39640 1 -- 172.21.15.153:0/2561486760 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3409cfb0 con 0x7fef3409a7f0 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.119+0000 7fef38b39640 1 -- 172.21.15.153:0/2561486760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 msgr2=0x7fef3409c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.119+0000 7fef38b39640 1 --2- 172.21.15.153:0/2561486760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3409c0f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef24009940 tx=0x7fef240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d4114d3c296e1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.119+0000 7fef337fe640 1 -- 172.21.15.153:0/2561486760 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24032440 con 0x7fef3409a7f0 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: dump: 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: _finish_auth 0 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 1 -- 172.21.15.153:0/2561486760 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef240325a0 con 0x7fef3409a7f0 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 5 monclient: authenticate success, global_id 234107 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 1 -- 172.21.15.153:0/2561486760 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24032940 con 0x7fef3409a7f0 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 4 set_mon_vals no callback set 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef337fe640 10 monclient: dump: 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 10 monclient: shutdown 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 -- 172.21.15.153:0/2561486760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 msgr2=0x7fef3409c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 --2- 172.21.15.153:0/2561486760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3409c0f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef24009940 tx=0x7fef240091d0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 -- 172.21.15.153:0/2561486760 shutdown_connections 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 --2- 172.21.15.153:0/2561486760 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3409c0f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 -- 172.21.15.153:0/2561486760 >> 172.21.15.153:0/2561486760 conn(0x7fef34067b50 msgr2=0x7fef340a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 -- 172.21.15.153:0/2561486760 shutdown_connections 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.120+0000 7fef3ad9d640 1 -- 172.21.15.153:0/2561486760 wait complete. 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 10 monclient: build_initial_monmap 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 10 monclient: monmap: 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 1 Processor -- start 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 1 -- start start 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 10 monclient: init 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.121+0000 7fef3ad9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): picked mon.noname-c con 0x7fef3409a7f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef3409abc0 0x7fef340951e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): picked mon.noname-a con 0x7fef3409abc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3409afb0 0x7fef34095720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): picked mon.noname-b con 0x7fef3409afb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3412f830 con 0x7fef3409abc0 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3412c090 con 0x7fef3409afb0 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3412abe0 con 0x7fef3409a7f0 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3ad9d640 10 monclient(hunting): authenticate will time out at 1858.761075s 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef3409abc0 0x7fef340951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3406ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33160/0 (socket says 172.21.15.153:33160) 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef3409abc0 0x7fef340951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42770/0 (socket says 172.21.15.153:42770) 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 -- 172.21.15.153:0/1956344661 learned_addr learned my addr 172.21.15.153:0/1956344661 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): get_auth_request con 0x7fef3409a7f0 auth_method 0 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): get_auth_request con 0x7fef3409abc0 auth_method 0 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3933a640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3409afb0 0x7fef34095720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3933a640 10 monclient(hunting): get_auth_request con 0x7fef3409afb0 auth_method 0 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3933a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3933a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef3933a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient(hunting): handle_auth_done global_id 234110 payload 824 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3409afb0 msgr2=0x7fef34095720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3409afb0 0x7fef34095720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef3409abc0 msgr2=0x7fef340951e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef3409abc0 0x7fef340951e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient: _finish_hunting 0 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 monclient: found mon.noname-c 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.122+0000 7fef38b39640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24009450 con 0x7fef3409a7f0 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef38b39640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 msgr2=0x7fef3406ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef38b39640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3406ef40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef24035040 tx=0x7fef24032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2403c960 con 0x7fef3409a7f0 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: dump: 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: _finish_auth 0 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2403cae0 con 0x7fef3409a7f0 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 5 monclient: authenticate success, global_id 234110 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2403cea0 con 0x7fef3409a7f0 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 10 monclient: _renew_subs 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3412a960 con 0x7fef3409a7f0 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef30ff9640 10 monclient: dump: 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef32ffd640 4 set_mon_vals no callback set 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 10 monclient: _renew_subs 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.124+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef34096d80 con 0x7fef3409a7f0 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef3ad9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef3ad9d640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 0x7fef3412d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef3ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef34063720 con 0x7fef34095c60 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fef24039040 con 0x7fef3409a7f0 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef30ff9640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef14038740 0x7fef1403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.125+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fef2406c4b0 con 0x7fef3409a7f0 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 0x7fef3412d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient: get_auth_request con 0x7fef34095c60 auth_method 0 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef3933a640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef14038740 0x7fef1403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef3933a640 10 monclient: get_auth_request con 0x7fef14038740 auth_method 0 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 10 monclient(hunting): handle_auth_done global_id 204367 payload 824 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.126+0000 7fef33fff640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef34096030 con 0x7fef34095c60 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef33fff640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 msgr2=0x7fef3412d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef33fff640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 0x7fef3412d460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00f780 tx=0x7fef1c007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef3933a640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef14038740 0x7fef1403abf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fef3406f4d0 tx=0x7fef2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c00d8b0 con 0x7fef34095c60 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.128+0000 7fef30ff9640 10 monclient: dump: 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fef1c00da10 con 0x7fef34095c60 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 msgr2=0x7fef3412d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 0x7fef3412d460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00f780 tx=0x7fef1c007e10 comp rx=0 tx=0).stop 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 msgr2=0x7fef3412d460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.129+0000 7fef30ff9640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef34095c60 0x7fef3412d460 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00f780 tx=0x7fef1c007e10 comp rx=0 tx=0).stop 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.134+0000 7fef3ad9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.134+0000 7fef3ad9d640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 0x7fef3412dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.134+0000 7fef3ad9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.134+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3412bcc0 con 0x7fef3412d9a0 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 0x7fef3412dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient: get_auth_request con 0x7fef3412d9a0 auth_method 0 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 10 monclient(hunting): handle_auth_done global_id 204370 payload 824 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.135+0000 7fef33fff640 1 -- 172.21.15.153:0/1956344661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fef3404bf60 con 0x7fef3412d9a0 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.137+0000 7fef33fff640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 msgr2=0x7fef3412dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.137+0000 7fef33fff640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 0x7fef3412dda0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00f780 tx=0x7fef1c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.137+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c014e20 con 0x7fef3412d9a0 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.137+0000 7fef30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.137+0000 7fef30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.137+0000 7fef30ff9640 10 monclient: dump: 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fef1c01d610 con 0x7fef3412d9a0 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 msgr2=0x7fef3412dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 0x7fef3412dda0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00f780 tx=0x7fef1c019000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 msgr2=0x7fef3412dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef30ff9640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef3412d9a0 0x7fef3412dda0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00f780 tx=0x7fef1c019000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 200, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:41.155 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:41.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:41.157 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:41.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:41.162 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.163 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:41.164 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef14038740 msgr2=0x7fef1403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fef14038740 0x7fef1403abf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fef3406f4d0 tx=0x7fef2c009040 comp rx=0 tx=0).stop 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 10 monclient: shutdown 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 msgr2=0x7fef3406ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 --2- 172.21.15.153:0/1956344661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3409a7f0 0x7fef3406ef40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef24035040 tx=0x7fef24032000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3933a640 1 -- 172.21.15.153:0/1956344661 reap_dead start 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3933a640 1 -- 172.21.15.153:0/1956344661 reap_dead start 2024-09-15T03:47:41.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 shutdown_connections 2024-09-15T03:47:41.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 >> 172.21.15.153:0/1956344661 conn(0x7fef34067b50 msgr2=0x7fef340a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:41.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 shutdown_connections 2024-09-15T03:47:41.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.138+0000 7fef3ad9d640 1 -- 172.21.15.153:0/1956344661 wait complete. 2024-09-15T03:47:41.167 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:47:41.167 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:47:41.167 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:47:41.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.259+0000 7facda8c0640 10 monclient: get_monmap_and_config 2024-09-15T03:47:41.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.259+0000 7facda8c0640 10 monclient: build_initial_monmap 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.259+0000 7facda8c0640 10 monclient: monmap: 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:41.260648+0000 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:41.260648+0000 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:41.261 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.260+0000 7facda8c0640 1 Processor -- start 2024-09-15T03:47:41.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.260+0000 7facda8c0640 1 -- start start 2024-09-15T03:47:41.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.260+0000 7facda8c0640 10 monclient: init 2024-09-15T03:47:41.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:41.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-15T03:47:41.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 0x7facd409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): picked mon.noname-a con 0x7facd4096c60 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:41.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 0x7facd4097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): picked mon.noname-b con 0x7facd4097030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:41.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facd412e5f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:41.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd409d2e0 con 0x7facd412e5f0 2024-09-15T03:47:41.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd409ac00 con 0x7facd4096c60 2024-09-15T03:47:41.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd409d6b0 con 0x7facd4097030 2024-09-15T03:47:41.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facda8c0640 10 monclient(hunting): authenticate will time out at 1858.900205s 2024-09-15T03:47:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 0x7facd409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 0x7facd4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 0x7facd409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42776/0 (socket says 172.21.15.153:42776) 2024-09-15T03:47:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 0x7facd4097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33174/0 (socket says 172.21.15.153:33174) 2024-09-15T03:47:41.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd8e5d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7facd412e5f0 0x7facd412ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41012/0 (socket says 172.21.15.153:41012) 2024-09-15T03:47:41.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 1 -- 172.21.15.153:0/3573362497 learned_addr learned my addr 172.21.15.153:0/3573362497 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:41.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): get_auth_request con 0x7facd4096c60 auth_method 0 2024-09-15T03:47:41.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): get_auth_request con 0x7facd4097030 auth_method 0 2024-09-15T03:47:41.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient(hunting): handle_auth_done global_id 234113 payload 274 2024-09-15T03:47:41.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 -- 172.21.15.153:0/3573362497 >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 msgr2=0x7facd409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:41.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 --2- 172.21.15.153:0/3573362497 >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 0x7facd409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 -- 172.21.15.153:0/3573362497 >> v1:172.21.15.153:6789/0 conn(0x7facd412e5f0 legacy=0x7facd412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:41.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:47:41.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 monclient: found mon.noname-b 2024-09-15T03:47:41.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:41.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd37fe640 1 -- 172.21.15.153:0/3573362497 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40992f0 con 0x7facd4097030 2024-09-15T03:47:41.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.261+0000 7facd3fff640 1 --2- 172.21.15.153:0/3573362497 >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 0x7facd409a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:41.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd37fe640 1 -- 172.21.15.153:0/3573362497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 msgr2=0x7facd4097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd37fe640 1 --2- 172.21.15.153:0/3573362497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 0x7facd4097430 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7facc8009850 tx=0x7facc80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4306ac4bca19bbac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 1 -- 172.21.15.153:0/3573362497 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc802adf0 con 0x7facd4097030 2024-09-15T03:47:41.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:47:41.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: dump: 2024-09-15T03:47:41.286 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.286 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.286 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.286 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:41.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:41.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 1 -- 172.21.15.153:0/3573362497 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc8032440 con 0x7facd4097030 2024-09-15T03:47:41.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facda8c0640 5 monclient: authenticate success, global_id 234113 2024-09-15T03:47:41.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facda8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:41.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:41.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facda8c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:41.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facda8c0640 4 set_mon_vals no callback set 2024-09-15T03:47:41.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 1 -- 172.21.15.153:0/3573362497 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc80327e0 con 0x7facd4097030 2024-09-15T03:47:41.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:41.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.264+0000 7facd2ffd640 10 monclient: dump: 2024-09-15T03:47:41.290 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.290 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.290 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.290 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.290 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.291 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.291 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.291 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.291 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 10 monclient: shutdown 2024-09-15T03:47:41.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:41.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 -- 172.21.15.153:0/3573362497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 msgr2=0x7facd4097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 --2- 172.21.15.153:0/3573362497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 0x7facd4097430 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7facc8009850 tx=0x7facc80090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 -- 172.21.15.153:0/3573362497 shutdown_connections 2024-09-15T03:47:41.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 --2- 172.21.15.153:0/3573362497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4097030 0x7facd4097430 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 --2- 172.21.15.153:0/3573362497 >> v2:172.21.15.153:3300/0 conn(0x7facd4096c60 0x7facd409a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 -- 172.21.15.153:0/3573362497 >> 172.21.15.153:0/3573362497 conn(0x7facd40672b0 msgr2=0x7facd40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:41.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 -- 172.21.15.153:0/3573362497 shutdown_connections 2024-09-15T03:47:41.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 -- 172.21.15.153:0/3573362497 wait complete. 2024-09-15T03:47:41.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 10 monclient: build_initial_monmap 2024-09-15T03:47:41.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 10 monclient: monmap: 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:41.294 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.265+0000 7facda8c0640 1 Processor -- start 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 -- start start 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient: init 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:41.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 0x7facd41411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): picked mon.noname-c con 0x7facd4096c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd4097030 0x7facd412daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): picked mon.noname-b con 0x7facd4097030 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd412e5f0 0x7facd412dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): picked mon.noname-a con 0x7facd412e5f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd411f250 con 0x7facd412e5f0 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd41420b0 con 0x7facd4097030 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd409add0 con 0x7facd4096c60 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facda8c0640 10 monclient(hunting): authenticate will time out at 1858.905561s 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 0x7facd41411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 0x7facd41411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33190/0 (socket says 172.21.15.153:33190) 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 -- 172.21.15.153:0/3018790734 learned_addr learned my addr 172.21.15.153:0/3018790734 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd412e5f0 0x7facd412dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): get_auth_request con 0x7facd4096c60 auth_method 0 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): get_auth_request con 0x7facd412e5f0 auth_method 0 2024-09-15T03:47:41.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient(hunting): handle_auth_done global_id 234116 payload 824 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 -- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd4097030 msgr2=0x7facd412daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd37fe640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd4097030 0x7facd412daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd4097030 0x7facd412daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 -- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd412e5f0 msgr2=0x7facd412dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd412e5f0 0x7facd412dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 monclient: found mon.noname-c 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd3fff640 1 -- 172.21.15.153:0/3018790734 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4009450 con 0x7facd4096c60 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.266+0000 7facd8e5d640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd412e5f0 0x7facd412dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:41.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facd3fff640 1 -- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 msgr2=0x7facd41411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facd3fff640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 0x7facd41411c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7facc400ece0 tx=0x7facc40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4009730 con 0x7facd4096c60 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: dump: 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.301 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: _finish_auth 0 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 5 monclient: authenticate success, global_id 234116 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 10 monclient: _renew_subs 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc400cdd0 con 0x7facd4096c60 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc8009390 con 0x7facd4096c60 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40136a0 con 0x7facd4096c60 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facbbfff640 10 monclient: dump: 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 10 monclient: _renew_subs 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd411d0d0 con 0x7facd4096c60 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.269+0000 7facd27fc640 4 set_mon_vals no callback set 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7facc4013a50 con 0x7facd4096c60 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facda8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facda8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facda8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd404bf60 con 0x7facd4096c60 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facbbfff640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7facc0038740 0x7facc003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7facc404cba0 con 0x7facd4096c60 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facd37fe640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7facc0038740 0x7facc003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.270+0000 7facd37fe640 10 monclient: get_auth_request con 0x7facc0038740 auth_method 0 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.273+0000 7facd37fe640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7facc0038740 0x7facc003abf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7facd409ecd0 tx=0x7facc8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.274+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7facc40123d0 con 0x7facd4096c60 2024-09-15T03:47:41.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.274+0000 7facbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:41.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.274+0000 7facbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.488+0000 7facda8c0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.488+0000 7facda8c0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.488+0000 7facda8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.488+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7facd40983b0 con 0x7facd4096c60 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.489+0000 7facbbfff640 1 -- 172.21.15.153:0/3018790734 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7facc4011e20 con 0x7facd4096c60 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.489+0000 7facbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.489+0000 7facbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:41.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.490+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.490+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=7,logm=53} crc {auth=1594636052,config=1228700967,health=311099025,kv=2691552684,logm=1716809428}) 2024-09-15T03:47:41.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.490+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,health=11,kv=8,logm=56} crc {auth=1489507996,health=2574758340,kv=3235807718,logm=1589723305}) 2024-09-15T03:47:41.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.490+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.490+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=7,logm=53} crc {auth=1594636052,config=1228700967,health=311099025,kv=2691552684,logm=1716809428}) 2024-09-15T03:47:41.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.490+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=8,logm=54} crc {auth=1725067733,config=3140514009,health=311099025,kv=3235807718,logm=1861718633}) 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7facc0038740 msgr2=0x7facc003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7facc0038740 0x7facc003abf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7facd409ecd0 tx=0x7facc8032000 comp rx=0 tx=0).stop 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 10 monclient: shutdown 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 msgr2=0x7facd41411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 0x7facd41411c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7facc400ece0 tx=0x7facc40091d0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 shutdown_connections 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7facc0038740 0x7facc003abf0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd412e5f0 0x7facd412dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd4097030 0x7facd412daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 --2- 172.21.15.153:0/3018790734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd4096c60 0x7facd41411c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 >> 172.21.15.153:0/3018790734 conn(0x7facd40672b0 msgr2=0x7facd412f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 shutdown_connections 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:41.492+0000 7facda8c0640 1 -- 172.21.15.153:0/3018790734 wait complete. 2024-09-15T03:47:41.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.497+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.497+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501059040}) 2024-09-15T03:47:41.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.497+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421348007}) 2024-09-15T03:47:41.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.497+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.497+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501059040}) 2024-09-15T03:47:41.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.497+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010629214}) 2024-09-15T03:47:41.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.498+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.498+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289059018}) 2024-09-15T03:47:41.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.498+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2577281222}) 2024-09-15T03:47:41.500 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.498+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.500 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289059018}) 2024-09-15T03:47:41.500 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097224204}) 2024-09-15T03:47:41.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333108374}) 2024-09-15T03:47:41.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1334698438}) 2024-09-15T03:47:41.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333108374}) 2024-09-15T03:47:41.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298768075}) 2024-09-15T03:47:41.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696254181}) 2024-09-15T03:47:41.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4146185111}) 2024-09-15T03:47:41.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696254181}) 2024-09-15T03:47:41.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322717515}) 2024-09-15T03:47:41.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589268327}) 2024-09-15T03:47:41.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1485603963}) 2024-09-15T03:47:41.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589268327}) 2024-09-15T03:47:41.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479959906}) 2024-09-15T03:47:41.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650534624}) 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159921153}) 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650534624}) 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055000964}) 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956890356}) 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280856417}) 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956890356}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=880156354}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=591871969}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=314177747}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=591871969}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102550822}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939359902}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257055623}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939359902}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392819180}) 2024-09-15T03:47:41.508 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166949439}) 2024-09-15T03:47:41.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1706073570}) 2024-09-15T03:47:41.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166949439}) 2024-09-15T03:47:41.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268007545}) 2024-09-15T03:47:41.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563436465}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004802021}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563436465}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2585366311}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769944144}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1557242619}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769944144}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609072919}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611278923}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041434636}) 2024-09-15T03:47:41.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611278923}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173378441}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201467585}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220300407}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201467585}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725699643}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752077533}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312920196}) 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.515 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752077533}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067531644}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3906307768}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652696072}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3906307768}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3817809484}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934095812}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44499389}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934095812}) 2024-09-15T03:47:41.516 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406677929}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750986467}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288616948}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750986467}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3772337406}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069767009}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107051178}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069767009}) 2024-09-15T03:47:41.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3677228964}) 2024-09-15T03:47:41.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010361238}) 2024-09-15T03:47:41.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=319569361}) 2024-09-15T03:47:41.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010361238}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259164860}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499922988}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467137081}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499922988}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776143312}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043738746}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924516817}) 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043738746}) 2024-09-15T03:47:41.521 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773480224}) 2024-09-15T03:47:41.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109059875}) 2024-09-15T03:47:41.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617592001}) 2024-09-15T03:47:41.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:41.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109059875}) 2024-09-15T03:47:41.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:41.554+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600008482}) 2024-09-15T03:47:42.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.049+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.049+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330745390}) 2024-09-15T03:47:42.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.049+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2372913241}) 2024-09-15T03:47:42.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.049+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.049+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330745390}) 2024-09-15T03:47:42.050 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.049+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4006680213}) 2024-09-15T03:47:42.052 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591625135}) 2024-09-15T03:47:42.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1506991809}) 2024-09-15T03:47:42.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591625135}) 2024-09-15T03:47:42.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7491799}) 2024-09-15T03:47:42.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3584765413}) 2024-09-15T03:47:42.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072193842}) 2024-09-15T03:47:42.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3584765413}) 2024-09-15T03:47:42.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799257813}) 2024-09-15T03:47:42.180 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.180 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780045741}) 2024-09-15T03:47:42.180 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=149624612}) 2024-09-15T03:47:42.180 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.180 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780045741}) 2024-09-15T03:47:42.180 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989298650}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341197839}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538227088}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341197839}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.499+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448877239}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093830342}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430544997}) 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093830342}) 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.500+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329258231}) 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136118439}) 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259433645}) 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136118439}) 2024-09-15T03:47:42.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.501+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395369348}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940170982}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537548883}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940170982}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1085127733}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054992642}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3346661905}) 2024-09-15T03:47:42.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054992642}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.502+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201157576}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936508488}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3053817148}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936508488}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.503+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666779302}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2686614668}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885505839}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2686614668}) 2024-09-15T03:47:42.506 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.504+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902627488}) 2024-09-15T03:47:42.507 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:47:42.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209235424}) 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711668906}) 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209235424}) 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1417544398}) 2024-09-15T03:47:42.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.505+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519371067}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4130621652}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519371067}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087654283}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471956185}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055512173}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471956185}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.506+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384357775}) 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617965711}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2363285397}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617965711}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.507+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2888846042}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654065731}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565180974}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654065731}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.508+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917892437}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876299942}) 2024-09-15T03:47:42.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546135275}) 2024-09-15T03:47:42.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876299942}) 2024-09-15T03:47:42.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998309219}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692702594}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991923118}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692702594}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.509+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267451767}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713810902}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4219467504}) 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.517 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713810902}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.510+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734384146}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406598512}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071786146}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406598512}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.511+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1825719084}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123333779}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070553330}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123333779}) 2024-09-15T03:47:42.518 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.512+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365483932}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123735052}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843842189}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123735052}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107985165}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641034882}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416641999}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641034882}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.513+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=284467379}) 2024-09-15T03:47:42.519 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901798262}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169803965}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901798262}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.514+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464229044}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613315437}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1862691770}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613315437}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.515+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257454135}) 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.516+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.520 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.516+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589373269}) 2024-09-15T03:47:42.521 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.516+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806988145}) 2024-09-15T03:47:42.521 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.516+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.521 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.516+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589373269}) 2024-09-15T03:47:42.521 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.516+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16510090}) 2024-09-15T03:47:42.523 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.517+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.517+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139901965}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.517+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943206861}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.517+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.517+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139901965}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.517+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644970643}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.518+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.518+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626067576}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.518+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=995085747}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.518+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.518+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626067576}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.518+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283161009}) 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.519+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.524 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.519+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368525424}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.519+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1415025558}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.519+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.519+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368525424}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.519+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3223377927}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.520+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.520+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381697672}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.520+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918649264}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.520+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.520+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381697672}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.520+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2208711522}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.521+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.521+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159396692}) 2024-09-15T03:47:42.525 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.521+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496636692}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.521+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.521+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159396692}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.521+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785237531}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.522+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.522+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044935108}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.522+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398030028}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.522+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.522+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044935108}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.522+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1238477414}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.523+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.523+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350103467}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.523+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3955427673}) 2024-09-15T03:47:42.526 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.523+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.527 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.523+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350103467}) 2024-09-15T03:47:42.527 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.523+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=348612251}) 2024-09-15T03:47:42.530 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.524+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.530 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.524+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362177922}) 2024-09-15T03:47:42.530 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.524+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2724426265}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.524+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.524+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362177922}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.524+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479515832}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972415901}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869937982}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972415901}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1795324196}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.525+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3608599443}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1334692866}) 2024-09-15T03:47:42.531 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3608599443}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1809196759}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.526+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495996535}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.527+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748924736}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.527+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.527+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495996535}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.527+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501439876}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782642695}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581246862}) 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.532 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782642695}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890886983}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.528+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802214486}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.529+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3213705731}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.529+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.529+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802214486}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.529+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=354348398}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622943846}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106172301}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622943846}) 2024-09-15T03:47:42.533 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1254472693}) 2024-09-15T03:47:42.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.530+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883496560}) 2024-09-15T03:47:42.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355171716}) 2024-09-15T03:47:42.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883496560}) 2024-09-15T03:47:42.537 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=745640326}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829495164}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376076686}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.531+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829495164}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75250910}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.532+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475673191}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171356254}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475673191}) 2024-09-15T03:47:42.538 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359374615}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464350635}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=100837043}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464350635}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.533+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825914446}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127411764}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.534+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2330231785}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127411764}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.535+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393657661}) 2024-09-15T03:47:42.539 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004173888}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1411450022}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004173888}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502022453}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063762127}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3726598779}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063762127}) 2024-09-15T03:47:42.540 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.536+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2933119516}) 2024-09-15T03:47:42.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976732763}) 2024-09-15T03:47:42.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1370083763}) 2024-09-15T03:47:42.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.544 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976732763}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.537+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3057528233}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1104925565}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278012669}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1104925565}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.538+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018756333}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504162321}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1950092677}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504162321}) 2024-09-15T03:47:42.545 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.539+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2450947706}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809958536}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231642348}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809958536}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699962630}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.540+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408485417}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2715800007}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408485417}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.541+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338284875}) 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.546 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045947233}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878950227}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045947233}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.542+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536462350}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819690444}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824264456}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819690444}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760256827}) 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.547 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118487124}) 2024-09-15T03:47:42.548 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685799258}) 2024-09-15T03:47:42.548 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.548 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118487124}) 2024-09-15T03:47:42.548 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.543+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324702895}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389560555}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986666043}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389560555}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.544+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777439368}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159144053}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=241328968}) 2024-09-15T03:47:42.551 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159144053}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.545+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27885845}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=910553776}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3865654438}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=910553776}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1986996052}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125290128}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3910885707}) 2024-09-15T03:47:42.552 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.546+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125290128}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3826628137}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975693981}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=243289227}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975693981}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.547+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21920229}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159980618}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690548470}) 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.553 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159980618}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.548+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918214159}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841239331}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2922796697}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841239331}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.549+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129141015}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064902118}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3882132688}) 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.554 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064902118}) 2024-09-15T03:47:42.555 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.550+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604174888}) 2024-09-15T03:47:42.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.562+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.562+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377729321}) 2024-09-15T03:47:42.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.562+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204231398}) 2024-09-15T03:47:42.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.562+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.562+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377729321}) 2024-09-15T03:47:42.563 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.562+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617649367}) 2024-09-15T03:47:42.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.563+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.563+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657126331}) 2024-09-15T03:47:42.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.563+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351937148}) 2024-09-15T03:47:42.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.563+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.563+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657126331}) 2024-09-15T03:47:42.564 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.563+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825619369}) 2024-09-15T03:47:42.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.564+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.564+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934177699}) 2024-09-15T03:47:42.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.564+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336157293}) 2024-09-15T03:47:42.565 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.564+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.564+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934177699}) 2024-09-15T03:47:42.566 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.564+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102041104}) 2024-09-15T03:47:42.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.610+0000 7f426e59d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:42.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.610+0000 7f426e59d640 10 monclient: build_initial_monmap 2024-09-15T03:47:42.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.610+0000 7f426e59d640 10 monclient: monmap: 2024-09-15T03:47:42.611 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:42.611568+0000 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:42.611568+0000 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:42.612 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.611+0000 7f426e59d640 1 Processor -- start 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.611+0000 7f426e59d640 1 -- start start 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.611+0000 7f426e59d640 10 monclient: init 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f426812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): picked mon.noname-b con 0x7f42680974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f426809a8a0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:42.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4268133070 0x7f4268135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4268133070 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42680978c0 con 0x7f426809a8a0 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426809ca20 con 0x7f4268133070 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4268098ad0 con 0x7f42680974f0 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426e59d640 10 monclient(hunting): authenticate will time out at 1860.251150s 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4268133070 0x7f4268135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f426812ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f42677fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f426809a8a0 0x7f426809acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41018/0 (socket says 172.21.15.153:41018) 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): get_auth_request con 0x7f4268133070 auth_method 0 2024-09-15T03:47:42.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f42677fe640 1 -- 172.21.15.153:0/3208274659 learned_addr learned my addr 172.21.15.153:0/3208274659 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): get_auth_request con 0x7f42680974f0 auth_method 0 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:42.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient(hunting): handle_auth_done global_id 234119 payload 274 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 1 -- 172.21.15.153:0/3208274659 >> v2:172.21.15.153:3300/0 conn(0x7f4268133070 msgr2=0x7f4268135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 1 --2- 172.21.15.153:0/3208274659 >> v2:172.21.15.153:3300/0 conn(0x7f4268133070 0x7f4268135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 1 -- 172.21.15.153:0/3208274659 >> v1:172.21.15.153:6789/0 conn(0x7f426809a8a0 legacy=0x7f426809acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 1 monclient: found mon.noname-b 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f4267fff640 1 -- 172.21.15.153:0/3208274659 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268098db0 con 0x7f42680974f0 2024-09-15T03:47:42.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.612+0000 7f426cb3a640 1 --2- 172.21.15.153:0/3208274659 >> v2:172.21.15.153:3300/0 conn(0x7f4268133070 0x7f4268135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4267fff640 1 -- 172.21.15.153:0/3208274659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 msgr2=0x7f426812ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4267fff640 1 --2- 172.21.15.153:0/3208274659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f426812ab00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4258009850 tx=0x7f42580090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77159450f72414cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 1 -- 172.21.15.153:0/3208274659 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425802adf0 con 0x7f42680974f0 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: dump: 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: _finish_auth 0 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 1 -- 172.21.15.153:0/3208274659 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4258032440 con 0x7f42680974f0 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f426e59d640 5 monclient: authenticate success, global_id 234119 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f426e59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:42.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f426e59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 1 -- 172.21.15.153:0/3208274659 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42580327e0 con 0x7f42680974f0 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f426e59d640 4 set_mon_vals no callback set 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.615+0000 7f4266ffd640 10 monclient: dump: 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.626 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 10 monclient: shutdown 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 -- 172.21.15.153:0/3208274659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 msgr2=0x7f426812ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 --2- 172.21.15.153:0/3208274659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f426812ab00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4258009850 tx=0x7f42580090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 -- 172.21.15.153:0/3208274659 shutdown_connections 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 --2- 172.21.15.153:0/3208274659 >> v2:172.21.15.153:3300/0 conn(0x7f4268133070 0x7f4268135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 --2- 172.21.15.153:0/3208274659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f426812ab00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 -- 172.21.15.153:0/3208274659 >> 172.21.15.153:0/3208274659 conn(0x7f42680672b0 msgr2=0x7f42680676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 -- 172.21.15.153:0/3208274659 shutdown_connections 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 -- 172.21.15.153:0/3208274659 wait complete. 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 10 monclient: build_initial_monmap 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 10 monclient: monmap: 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.628 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.616+0000 7f426e59d640 1 Processor -- start 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 -- start start 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient: init 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f4268127e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): picked mon.noname-c con 0x7f42680974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f426809a8a0 0x7f426806dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): picked mon.noname-a con 0x7f426809a8a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4268133070 0x7f426806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4268133070 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426812c7c0 con 0x7f426809a8a0 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426812ccf0 con 0x7f4268133070 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4268098300 con 0x7f42680974f0 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f426e59d640 10 monclient(hunting): authenticate will time out at 1860.256581s 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f4268127e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f426809a8a0 0x7f426806dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f4268127e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33230/0 (socket says 172.21.15.153:33230) 2024-09-15T03:47:42.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 -- 172.21.15.153:0/149841809 learned_addr learned my addr 172.21.15.153:0/149841809 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): get_auth_request con 0x7f42680974f0 auth_method 0 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): get_auth_request con 0x7f426809a8a0 auth_method 0 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient(hunting): handle_auth_done global_id 234122 payload 824 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 -- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4268133070 msgr2=0x7f426806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4268133070 0x7f426806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 -- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f426809a8a0 msgr2=0x7f426806dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f426809a8a0 0x7f426806dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 monclient: found mon.noname-c 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f4267fff640 1 -- 172.21.15.153:0/149841809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4260009a20 con 0x7f42680974f0 2024-09-15T03:47:42.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.617+0000 7f42677fe640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f426809a8a0 0x7f426806dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f4267fff640 1 -- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 msgr2=0x7f4268127e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f4267fff640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f4268127e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4258009850 tx=0x7f4258032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258032ea0 con 0x7f42680974f0 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: dump: 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: _finish_auth 0 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425803b820 con 0x7f42680974f0 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 5 monclient: authenticate success, global_id 234122 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 10 monclient: _renew_subs 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425803bbe0 con 0x7f42680974f0 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4258009390 con 0x7f42680974f0 2024-09-15T03:47:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f424ffff640 10 monclient: dump: 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f42667fc640 4 set_mon_vals no callback set 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 10 monclient: _renew_subs 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.620+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426806f390 con 0x7f42680974f0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4258039040 con 0x7f42680974f0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f426e59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f426e59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f426e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426804bf60 con 0x7f42680974f0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f424ffff640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4254038740 0x7f425403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f425806c6a0 con 0x7f42680974f0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f42677fe640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4254038740 0x7f425403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:42.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.621+0000 7f42677fe640 10 monclient: get_auth_request con 0x7f4254038740 auth_method 0 2024-09-15T03:47:42.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.624+0000 7f42677fe640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4254038740 0x7f425403abf0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f425c007fd0 tx=0x7f425c007f60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:42.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.626+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f425804b300 con 0x7f42680974f0 2024-09-15T03:47:42.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.626+0000 7f424ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:42.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.626+0000 7f424ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:42.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.693+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.693+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563245593}) 2024-09-15T03:47:42.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.693+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2297825268}) 2024-09-15T03:47:42.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.693+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:42.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.693+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563245593}) 2024-09-15T03:47:42.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:42.693+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712473082}) 2024-09-15T03:47:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f426e59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f426e59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f426e59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f42680638d0 con 0x7f42680974f0 2024-09-15T03:47:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f424ffff640 1 -- 172.21.15.153:0/149841809 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f425803f020 con 0x7f42680974f0 2024-09-15T03:47:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f424ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:42.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.840+0000 7f424ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:42.842 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4254038740 msgr2=0x7f425403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4254038740 0x7f425403abf0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f425c007fd0 tx=0x7f425c007f60 comp rx=0 tx=0).stop 2024-09-15T03:47:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 10 monclient: shutdown 2024-09-15T03:47:42.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 msgr2=0x7f4268127e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f4268127e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4258009850 tx=0x7f4258032000 comp rx=0 tx=0).stop 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 shutdown_connections 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4254038740 0x7f425403abf0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4268133070 0x7f426806dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f426809a8a0 0x7f426806dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 --2- 172.21.15.153:0/149841809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42680974f0 0x7f4268127e00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 >> 172.21.15.153:0/149841809 conn(0x7f42680672b0 msgr2=0x7f426809b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 shutdown_connections 2024-09-15T03:47:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.842+0000 7f426e59d640 1 -- 172.21.15.153:0/149841809 wait complete. 2024-09-15T03:47:42.854 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:47:42.854 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:47:42.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:47:42.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:47:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.988+0000 7f56cce34640 10 monclient: get_monmap_and_config 2024-09-15T03:47:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.988+0000 7f56cce34640 10 monclient: build_initial_monmap 2024-09-15T03:47:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.988+0000 7f56cce34640 10 monclient: monmap: 2024-09-15T03:47:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:42.989506+0000 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:42.989506+0000 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:42.990 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:42.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 1 Processor -- start 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 1 -- start start 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 10 monclient: init 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 0x7f56c809c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 10 monclient(hunting): picked mon.noname-b con 0x7f56c80a1ef0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56c809abd0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.989+0000 7f56cce34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f56c8136d00 0x7f56c81390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 10 monclient(hunting): picked mon.noname-c con 0x7f56c8136d00 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c806f010 con 0x7f56c809abd0 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c809d9e0 con 0x7f56c8136d00 2024-09-15T03:47:42.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c809dcc0 con 0x7f56c80a1ef0 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56cce34640 10 monclient(hunting): authenticate will time out at 1860.629047s 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 0x7f56c809c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56bffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f56c809abd0 0x7f56c809afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41024/0 (socket says 172.21.15.153:41024) 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56bffff640 1 -- 172.21.15.153:0/1747988448 learned_addr learned my addr 172.21.15.153:0/1747988448 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): get_auth_request con 0x7f56c80a1ef0 auth_method 0 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:42.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient(hunting): handle_auth_done global_id 234125 payload 274 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 1 -- 172.21.15.153:0/1747988448 >> v2:172.21.15.181:3300/0 conn(0x7f56c8136d00 msgr2=0x7f56c81390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 1 --2- 172.21.15.153:0/1747988448 >> v2:172.21.15.181:3300/0 conn(0x7f56c8136d00 0x7f56c81390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 1 -- 172.21.15.153:0/1747988448 >> v1:172.21.15.153:6789/0 conn(0x7f56c809abd0 legacy=0x7f56c809afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient: _finish_hunting 0 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 1 monclient: found mon.noname-b 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:47:42.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.990+0000 7f56c659c640 1 -- 172.21.15.153:0/1747988448 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c80a22c0 con 0x7f56c80a1ef0 2024-09-15T03:47:43.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c659c640 1 -- 172.21.15.153:0/1747988448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 msgr2=0x7f56c809c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c659c640 1 --2- 172.21.15.153:0/1747988448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 0x7f56c809c6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009850 tx=0x7f56b80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9c8377b0873966 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 1 -- 172.21.15.153:0/1747988448 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8032440 con 0x7f56c80a1ef0 2024-09-15T03:47:43.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: dump: 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 1 -- 172.21.15.153:0/1747988448 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b80325a0 con 0x7f56c80a1ef0 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 5 monclient: authenticate success, global_id 234125 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 1 -- 172.21.15.153:0/1747988448 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8032940 con 0x7f56c80a1ef0 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 4 set_mon_vals no callback set 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56c5d9b640 10 monclient: dump: 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.004 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 10 monclient: shutdown 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 1 -- 172.21.15.153:0/1747988448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 msgr2=0x7f56c809c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 1 --2- 172.21.15.153:0/1747988448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 0x7f56c809c6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009850 tx=0x7f56b80090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 1 -- 172.21.15.153:0/1747988448 shutdown_connections 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 1 --2- 172.21.15.153:0/1747988448 >> v2:172.21.15.181:3300/0 conn(0x7f56c8136d00 0x7f56c81390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 1 --2- 172.21.15.153:0/1747988448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c80a1ef0 0x7f56c809c6a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.993+0000 7f56cce34640 1 -- 172.21.15.153:0/1747988448 >> 172.21.15.153:0/1747988448 conn(0x7f56c80678a0 msgr2=0x7f56c80a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 1 -- 172.21.15.153:0/1747988448 shutdown_connections 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 1 -- 172.21.15.153:0/1747988448 wait complete. 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 10 monclient: monmap: 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:43.006 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 1 Processor -- start 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 1 -- start start 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.994+0000 7f56cce34640 10 monclient: init 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 0x7f56c812eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): picked mon.noname-c con 0x7f56c809abd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56c80a1ef0 0x7f56c8141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): picked mon.noname-b con 0x7f56c80a1ef0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:43.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8136d00 0x7f56c8134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): picked mon.noname-a con 0x7f56c8136d00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8128280 con 0x7f56c8136d00 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8128840 con 0x7f56c80a1ef0 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8126630 con 0x7f56c809abd0 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56cce34640 10 monclient(hunting): authenticate will time out at 1860.634305s 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 0x7f56c812eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 0x7f56c812eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33256/0 (socket says 172.21.15.153:33256) 2024-09-15T03:47:43.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 -- 172.21.15.153:0/3736566903 learned_addr learned my addr 172.21.15.153:0/3736566903 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): get_auth_request con 0x7f56c809abd0 auth_method 0 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8136d00 0x7f56c8134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): get_auth_request con 0x7f56c8136d00 auth_method 0 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56c80a1ef0 0x7f56c8141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): get_auth_request con 0x7f56c80a1ef0 auth_method 0 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient(hunting): handle_auth_done global_id 234128 payload 824 2024-09-15T03:47:43.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56c80a1ef0 msgr2=0x7f56c8141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56c80a1ef0 0x7f56c8141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8136d00 msgr2=0x7f56c8134bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8136d00 0x7f56c8134bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 monclient: found mon.noname-c 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c659c640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b8009390 con 0x7f56c809abd0 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.995+0000 7f56c6d9d640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8136d00 0x7f56c8134bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.997+0000 7f56c659c640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 msgr2=0x7f56c812eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.997+0000 7f56c659c640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 0x7f56c812eda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009360 tx=0x7f56b8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b80318c0 con 0x7f56c809abd0 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: dump: 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.012 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b8031a20 con 0x7f56c809abd0 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 5 monclient: authenticate success, global_id 234128 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8031dc0 con 0x7f56c809abd0 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56be7fc640 10 monclient: dump: 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 10 monclient: _renew_subs 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c8126910 con 0x7f56c809abd0 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56c559a640 4 set_mon_vals no callback set 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 10 monclient: _renew_subs 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c812f2e0 con 0x7f56c809abd0 2024-09-15T03:47:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 0x7f56c81350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.998+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8128140 con 0x7f56c8141790 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f56b8039040 con 0x7f56c809abd0 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 0x7f56c81350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient: get_auth_request con 0x7f56c8141790 auth_method 0 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56be7fc640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f56a8038740 0x7f56a803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 10 monclient(hunting): handle_auth_done global_id 214335 payload 824 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f56b806c900 con 0x7f56c809abd0 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56bffff640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56c8127d20 con 0x7f56c8141790 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56c6d9d640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f56a8038740 0x7f56a803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:42.999+0000 7f56c6d9d640 10 monclient: get_auth_request con 0x7f56a8038740 auth_method 0 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56bffff640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 msgr2=0x7f56c81350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 0x7f56c81350f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f56b000e780 tx=0x7f56b0006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000cac0 con 0x7f56c8141790 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 10 monclient: dump: 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f56b000cc20 con 0x7f56c8141790 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 msgr2=0x7f56c81350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 0x7f56c81350f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f56b000e780 tx=0x7f56b0006f50 comp rx=0 tx=0).stop 2024-09-15T03:47:43.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 msgr2=0x7f56c81350f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:43.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.001+0000 7f56be7fc640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8141790 0x7f56c81350f0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f56b000e780 tx=0x7f56b0006f50 comp rx=0 tx=0).stop 2024-09-15T03:47:43.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.002+0000 7f56c6d9d640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f56a8038740 0x7f56a803abf0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f56c8128d10 tx=0x7f56c0009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56cce34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56cce34640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56cce34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8065e00 con 0x7f56c8135630 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient: get_auth_request con 0x7f56c8135630 auth_method 0 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 msgr2=0x7f56c8131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:47:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 msgr2=0x7f56c8131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:47:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.007+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:43.052 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.052 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774708043}) 2024-09-15T03:47:43.052 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2412539638}) 2024-09-15T03:47:43.052 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774708043}) 2024-09-15T03:47:43.053 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.051+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4184315198}) 2024-09-15T03:47:43.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.054+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.054+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866481436}) 2024-09-15T03:47:43.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.054+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968748012}) 2024-09-15T03:47:43.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.054+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.054+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866481436}) 2024-09-15T03:47:43.055 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.054+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364622398}) 2024-09-15T03:47:43.178 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.178 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296813923}) 2024-09-15T03:47:43.178 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2238563866}) 2024-09-15T03:47:43.178 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.178 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296813923}) 2024-09-15T03:47:43.179 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998640872}) 2024-09-15T03:47:43.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823708193}) 2024-09-15T03:47:43.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142695718}) 2024-09-15T03:47:43.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823708193}) 2024-09-15T03:47:43.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492296861}) 2024-09-15T03:47:43.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient: get_auth_request con 0x7f56c8135630 auth_method 0 2024-09-15T03:47:43.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 10 monclient(hunting): handle_auth_done global_id 214341 payload 824 2024-09-15T03:47:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.208+0000 7f56bffff640 1 -- 172.21.15.153:0/3736566903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f56c8135a30 con 0x7f56c8135630 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.210+0000 7f56bffff640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 msgr2=0x7f56c8131500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.210+0000 7f56bffff640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f56b0019000 tx=0x7f56b001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0023050 con 0x7f56c8135630 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 10 monclient: dump: 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f56b0024040 con 0x7f56c8135630 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 msgr2=0x7f56c8131500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f56b0019000 tx=0x7f56b001a000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 msgr2=0x7f56c8131500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56be7fc640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c8135630 0x7f56c8131500 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f56b0019000 tx=0x7f56b001a000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 200, 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:43.228 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f56a8038740 msgr2=0x7f56a803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f56a8038740 0x7f56a803abf0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f56c8128d10 tx=0x7f56c0009780 comp rx=0 tx=0).stop 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 10 monclient: shutdown 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 msgr2=0x7f56c812eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 --2- 172.21.15.153:0/3736566903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56c809abd0 0x7f56c812eda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009360 tx=0x7f56b8032000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56c6d9d640 1 -- 172.21.15.153:0/3736566903 reap_dead start 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56c6d9d640 1 -- 172.21.15.153:0/3736566903 reap_dead start 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 shutdown_connections 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 >> 172.21.15.153:0/3736566903 conn(0x7f56c80678a0 msgr2=0x7f56c8097440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 shutdown_connections 2024-09-15T03:47:43.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.211+0000 7f56cce34640 1 -- 172.21.15.153:0/3736566903 wait complete. 2024-09-15T03:47:43.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:47:43.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:47:43.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:47:43.297 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.296+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.298 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.296+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434222340}) 2024-09-15T03:47:43.298 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.296+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293583124}) 2024-09-15T03:47:43.298 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.296+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.298 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.296+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434222340}) 2024-09-15T03:47:43.298 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.296+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107442370}) 2024-09-15T03:47:43.304 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.303+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.304 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.303+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354401126}) 2024-09-15T03:47:43.304 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198269606}) 2024-09-15T03:47:43.305 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.305 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354401126}) 2024-09-15T03:47:43.305 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945208826}) 2024-09-15T03:47:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004553307}) 2024-09-15T03:47:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074918065}) 2024-09-15T03:47:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.307 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004553307}) 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.304+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493107572}) 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662008325}) 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=651607207}) 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662008325}) 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642393183}) 2024-09-15T03:47:43.308 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800233570}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489428875}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.305+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800233570}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1378096272}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959719580}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2989126308}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959719580}) 2024-09-15T03:47:43.309 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.306+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969277072}) 2024-09-15T03:47:43.311 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.307+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.311 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.307+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1764004422,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977773,mgrstat=1463493728,mon_config_key=3725707551,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=4183882002}) 2024-09-15T03:47:43.311 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.307+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=38,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=2173107195,mdsmap=1531151339,mgr=2844096759,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1463493728,mon_config_key=105398142,monmap=626991505,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=584695987}) 2024-09-15T03:47:43.311 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.307+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.312 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.307+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1764004422,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977773,mgrstat=1463493728,mon_config_key=3725707551,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=4183882002}) 2024-09-15T03:47:43.312 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.307+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=25} crc {logm=1240523969,mdsmap=1531151339,mgr=1088511988,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=1717788636,mon_config_key=2726416669,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=4262329805}) 2024-09-15T03:47:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.313+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.313+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=507761345}) 2024-09-15T03:47:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.314+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=31} crc {osdmap=3232245560}) 2024-09-15T03:47:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.314+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:47:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.314+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=507761345}) 2024-09-15T03:47:43.315 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:47:43.314+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=4029877211}) 2024-09-15T03:47:43.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.334+0000 7fc6acd5d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.334+0000 7fc6acd5d640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.334+0000 7fc6acd5d640 10 monclient: monmap: 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:43.335045+0000 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:43.335045+0000 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:43.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.334+0000 7fc6acd5d640 1 Processor -- start 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.334+0000 7fc6acd5d640 1 -- start start 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.334+0000 7fc6acd5d640 10 monclient: init 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6a8131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 0x7fc6a809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc6a809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc6a809f220 0x7fc6a8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a809f220 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a809faf0 con 0x7fc6a8131ab0 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a8132590 con 0x7fc6a809f220 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a809fd30 con 0x7fc6a809e910 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6acd5d640 10 monclient(hunting): authenticate will time out at 1860.974636s 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a659c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc6a8131ab0 0x7fc6a8131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41030/0 (socket says 172.21.15.153:41030) 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a659c640 1 -- 172.21.15.153:0/1062043032 learned_addr learned my addr 172.21.15.153:0/1062043032 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/1062043032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 0x7fc6a809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): get_auth_request con 0x7fc6a809e910 auth_method 0 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_done global_id 234131 payload 274 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.335+0000 7fc6a6d9d640 1 --2- 172.21.15.153:0/1062043032 >> v2:172.21.15.181:3300/0 conn(0x7fc6a809f220 0x7fc6a8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/1062043032 >> v2:172.21.15.181:3300/0 conn(0x7fc6a809f220 msgr2=0x7fc6a8138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/1062043032 >> v2:172.21.15.181:3300/0 conn(0x7fc6a809f220 0x7fc6a8138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/1062043032 >> v1:172.21.15.153:6789/0 conn(0x7fc6a8131ab0 legacy=0x7fc6a8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 1 monclient: found mon.noname-b 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:47:43.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.336+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/1062043032 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a80a1110 con 0x7fc6a809e910 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.338+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/1062043032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 msgr2=0x7fc6a809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.338+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/1062043032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 0x7fc6a809ece0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc690009850 tx=0x7fc6900090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb755ced3eb2cb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 1 -- 172.21.15.153:0/1062043032 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690032440 con 0x7fc6a809e910 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: dump: 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 1 -- 172.21.15.153:0/1062043032 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6900325a0 con 0x7fc6a809e910 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 5 monclient: authenticate success, global_id 234131 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 1 -- 172.21.15.153:0/1062043032 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690032940 con 0x7fc6a809e910 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6a559a640 10 monclient: dump: 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:43.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 4 set_mon_vals no callback set 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 10 monclient: shutdown 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 -- 172.21.15.153:0/1062043032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 msgr2=0x7fc6a809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/1062043032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 0x7fc6a809ece0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc690009850 tx=0x7fc6900090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 -- 172.21.15.153:0/1062043032 shutdown_connections 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/1062043032 >> v2:172.21.15.181:3300/0 conn(0x7fc6a809f220 0x7fc6a8138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/1062043032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a809e910 0x7fc6a809ece0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 -- 172.21.15.153:0/1062043032 >> 172.21.15.153:0/1062043032 conn(0x7fc6a8067d30 msgr2=0x7fc6a80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 -- 172.21.15.153:0/1062043032 shutdown_connections 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.339+0000 7fc6acd5d640 1 -- 172.21.15.153:0/1062043032 wait complete. 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.340+0000 7fc6acd5d640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.340+0000 7fc6acd5d640 10 monclient: monmap: 2024-09-15T03:47:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.340+0000 7fc6acd5d640 1 Processor -- start 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.340+0000 7fc6acd5d640 1 -- start start 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.340+0000 7fc6acd5d640 10 monclient: init 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 0x7fc6a8138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc6a809e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6a809f220 0x7fc6a8134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc6a809f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a8131ab0 0x7fc6a81345d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a8131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8137a10 con 0x7fc6a809e910 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8136260 con 0x7fc6a809f220 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8131420 con 0x7fc6a8131ab0 2024-09-15T03:47:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6acd5d640 10 monclient(hunting): authenticate will time out at 1860.980215s 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 0x7fc6a8138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a8131ab0 0x7fc6a81345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 0x7fc6a8138110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42892/0 (socket says 172.21.15.153:42892) 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 -- 172.21.15.153:0/2815446458 learned_addr learned my addr 172.21.15.153:0/2815446458 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): get_auth_request con 0x7fc6a809e910 auth_method 0 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): get_auth_request con 0x7fc6a8131ab0 auth_method 0 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6a809f220 0x7fc6a8134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a5d9b640 10 monclient(hunting): get_auth_request con 0x7fc6a809f220 auth_method 0 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient(hunting): handle_auth_done global_id 214344 payload 824 2024-09-15T03:47:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a8131ab0 msgr2=0x7fc6a81345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a8131ab0 0x7fc6a81345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6a809f220 msgr2=0x7fc6a8134090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6a809f220 0x7fc6a8134090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 monclient: found mon.noname-a 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a659c640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc690009390 con 0x7fc6a809e910 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6a809f220 0x7fc6a8134090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.341+0000 7fc6a6d9d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a8131ab0 0x7fc6a81345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.343+0000 7fc6a659c640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 msgr2=0x7fc6a8138110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.343+0000 7fc6a659c640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 0x7fc6a8138110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c5e0 tx=0x7fc6a0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.343+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0005cb0 con 0x7fc6a809e910 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: dump: 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a0005e30 con 0x7fc6a809e910 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 5 monclient: authenticate success, global_id 214344 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 10 monclient: _renew_subs 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a8137cf0 con 0x7fc6a809e910 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0007340 con 0x7fc6a809e910 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 10 monclient: dump: 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.357 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6a4d99640 4 set_mon_vals no callback set 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 10 monclient: _renew_subs 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a81364f0 con 0x7fc6a809e910 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc6a001b400 con 0x7fc6a809e910 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 0x7fc6a8134b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.344+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8063740 con 0x7fc6a802aa40 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 0x7fc6a8134b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient: get_auth_request con 0x7fc6a802aa40 auth_method 0 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc69e7fc640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc688038740 0x7fc68803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc6a004c1d0 con 0x7fc6a809e910 2024-09-15T03:47:43.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_done global_id 234137 payload 824 2024-09-15T03:47:43.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6a802ae10 con 0x7fc6a802aa40 2024-09-15T03:47:43.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a6d9d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc688038740 0x7fc68803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.345+0000 7fc6a6d9d640 10 monclient: get_auth_request con 0x7fc688038740 auth_method 0 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 msgr2=0x7fc6a8134b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 0x7fc6a8134b10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc690034000 tx=0x7fc690032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc6a6d9d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc688038740 0x7fc68803abf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8138770 tx=0x7fc6980077b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690031d30 con 0x7fc6a802aa40 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc69e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc69e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.347+0000 7fc69e7fc640 10 monclient: dump: 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.363 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc690031e90 con 0x7fc6a802aa40 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:43.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 msgr2=0x7fc6a8134b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 0x7fc6a8134b10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc690034000 tx=0x7fc690032000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 msgr2=0x7fc6a8134b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.348+0000 7fc69e7fc640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a802aa40 0x7fc6a8134b10 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc690034000 tx=0x7fc690032000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.353+0000 7fc6acd5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.353+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 0x7fc6a805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.353+0000 7fc6acd5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.353+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a81360e0 con 0x7fc6a805e310 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 0x7fc6a805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient: get_auth_request con 0x7fc6a805e310 auth_method 0 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 10 monclient(hunting): handle_auth_done global_id 234140 payload 824 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.354+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/2815446458 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc6a804bf60 con 0x7fc6a805e310 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc6a5d9b640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 msgr2=0x7fc6a805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc6a5d9b640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 0x7fc6a805e710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc690009850 tx=0x7fc69003c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69003cdb0 con 0x7fc6a805e310 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 10 monclient: dump: 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.366 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fc69003b3d0 con 0x7fc6a805e310 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 msgr2=0x7fc6a805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 0x7fc6a805e710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc690009850 tx=0x7fc69003c000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 msgr2=0x7fc6a805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:43.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.356+0000 7fc69e7fc640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6a805e310 0x7fc6a805e710 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc690009850 tx=0x7fc69003c000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 200, 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:47:43.368 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.369 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:43.371 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.372 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.377 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:43.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:43.379 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc688038740 msgr2=0x7fc68803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc688038740 0x7fc68803abf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8138770 tx=0x7fc6980077b0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 10 monclient: shutdown 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 msgr2=0x7fc6a8138110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6a6d9d640 1 -- 172.21.15.153:0/2815446458 reap_dead start 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 0x7fc6a8138110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c5e0 tx=0x7fc6a0004960 comp rx=0 tx=0).stop 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 shutdown_connections 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 --2- 172.21.15.153:0/2815446458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6a809e910 0x7fc6a8138110 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 >> 172.21.15.153:0/2815446458 conn(0x7fc6a8067d30 msgr2=0x7fc6a8138940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 shutdown_connections 2024-09-15T03:47:43.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.357+0000 7fc6acd5d640 1 -- 172.21.15.153:0/2815446458 wait complete. 2024-09-15T03:47:43.382 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:47:43.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:47:43.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:47:43.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.515+0000 7fa00e15d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:43.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.515+0000 7fa00e15d640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.515+0000 7fa00e15d640 10 monclient: monmap: 2024-09-15T03:47:43.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.519 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:43.516482+0000 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:43.516482+0000 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 1 Processor -- start 2024-09-15T03:47:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 1 -- start start 2024-09-15T03:47:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 10 monclient: init 2024-09-15T03:47:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-15T03:47:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0080975f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0080979c0 0x7fa008097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0080979c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa0081349f0 0x7fa008136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.516+0000 7fa00e15d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0081349f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00809d700 con 0x7fa0080975f0 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00809d1b0 con 0x7fa0080979c0 2024-09-15T03:47:43.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0080961e0 con 0x7fa0081349f0 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa00e15d640 10 monclient(hunting): authenticate will time out at 1861.156039s 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0080979c0 0x7fa008097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa0080979c0 0x7fa008097dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42898/0 (socket says 172.21.15.153:42898) 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 -- 172.21.15.153:0/479606176 learned_addr learned my addr 172.21.15.153:0/479606176 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:43.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): get_auth_request con 0x7fa0080979c0 auth_method 0 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa007fff640 1 --2- 172.21.15.153:0/479606176 >> v2:172.21.15.181:3300/0 conn(0x7fa0081349f0 0x7fa008136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa007fff640 10 monclient(hunting): get_auth_request con 0x7fa0081349f0 auth_method 0 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa007fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient(hunting): handle_auth_done global_id 214347 payload 274 2024-09-15T03:47:43.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 -- 172.21.15.153:0/479606176 >> v2:172.21.15.181:3300/0 conn(0x7fa0081349f0 msgr2=0x7fa008136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 --2- 172.21.15.153:0/479606176 >> v2:172.21.15.181:3300/0 conn(0x7fa0081349f0 0x7fa008136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 -- 172.21.15.153:0/479606176 >> v1:172.21.15.181:6789/0 conn(0x7fa0080975f0 legacy=0x7fa008095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 monclient: found mon.noname-a 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa006ffd640 1 -- 172.21.15.153:0/479606176 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00809d420 con 0x7fa0080979c0 2024-09-15T03:47:43.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.517+0000 7fa007fff640 1 --2- 172.21.15.153:0/479606176 >> v2:172.21.15.181:3300/0 conn(0x7fa0081349f0 0x7fa008136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:47:43.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa006ffd640 1 -- 172.21.15.153:0/479606176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080979c0 msgr2=0x7fa008097dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa006ffd640 1 --2- 172.21.15.153:0/479606176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080979c0 0x7fa008097dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa0080959e0 tx=0x7f9ffc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90dad46dda32cb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 1 -- 172.21.15.153:0/479606176 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc02ae90 con 0x7fa0080979c0 2024-09-15T03:47:43.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:43.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: dump: 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 1 -- 172.21.15.153:0/479606176 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc032440 con 0x7fa0080979c0 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 5 monclient: authenticate success, global_id 214347 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 1 -- 172.21.15.153:0/479606176 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc0327e0 con 0x7fa0080979c0 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 4 set_mon_vals no callback set 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa0067fc640 10 monclient: dump: 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 10 monclient: shutdown 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 -- 172.21.15.153:0/479606176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080979c0 msgr2=0x7fa008097dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 --2- 172.21.15.153:0/479606176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080979c0 0x7fa008097dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa0080959e0 tx=0x7f9ffc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 -- 172.21.15.153:0/479606176 shutdown_connections 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 --2- 172.21.15.153:0/479606176 >> v2:172.21.15.181:3300/0 conn(0x7fa0081349f0 0x7fa008136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 --2- 172.21.15.153:0/479606176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080979c0 0x7fa008097dc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 -- 172.21.15.153:0/479606176 >> 172.21.15.153:0/479606176 conn(0x7fa0080678a0 msgr2=0x7fa0080a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.520+0000 7fa00e15d640 1 -- 172.21.15.153:0/479606176 shutdown_connections 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.521+0000 7fa00e15d640 1 -- 172.21.15.153:0/479606176 wait complete. 2024-09-15T03:47:43.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.521+0000 7fa00e15d640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.521+0000 7fa00e15d640 10 monclient: monmap: 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.521+0000 7fa00e15d640 1 Processor -- start 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.521+0000 7fa00e15d640 1 -- start start 2024-09-15T03:47:43.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.521+0000 7fa00e15d640 10 monclient: init 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 0x7fa008088fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0080975f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0080979c0 0x7fa008089500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0080979c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0081349f0 0x7fa008089a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0081349f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00808ad40 con 0x7fa0080975f0 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0080983c0 con 0x7fa0081349f0 2024-09-15T03:47:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00809f470 con 0x7fa0080979c0 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa00e15d640 10 monclient(hunting): authenticate will time out at 1861.161443s 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 0x7fa008088fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0080979c0 0x7fa008089500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 0x7fa008088fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42902/0 (socket says 172.21.15.153:42902) 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 -- 172.21.15.153:0/3885072765 learned_addr learned my addr 172.21.15.153:0/3885072765 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): get_auth_request con 0x7fa0080975f0 auth_method 0 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): get_auth_request con 0x7fa0080979c0 auth_method 0 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa007fff640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0081349f0 0x7fa008089a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa007fff640 10 monclient(hunting): get_auth_request con 0x7fa0081349f0 auth_method 0 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa007fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient(hunting): handle_auth_done global_id 214350 payload 824 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0080979c0 msgr2=0x7fa008089500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0080979c0 0x7fa008089500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0081349f0 msgr2=0x7fa008089a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0081349f0 0x7fa008089a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 monclient: found mon.noname-a 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.522+0000 7fa0077fe640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc009450 con 0x7fa0080975f0 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa0077fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 msgr2=0x7fa008088fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa0077fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 0x7fa008088fc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c8b0 tx=0x7f9ff80026c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff800cea0 con 0x7fa0080975f0 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: dump: 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.546 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff800aa60 con 0x7fa0080975f0 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 5 monclient: authenticate success, global_id 214350 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 10 monclient: _renew_subs 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00808c070 con 0x7fa0080975f0 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff800ae20 con 0x7fa0080975f0 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7f9fef7fe640 10 monclient: dump: 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.548 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 10 monclient: _renew_subs 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00808bbd0 con 0x7fa0080975f0 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa005ffb640 4 set_mon_vals no callback set 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 0x7fa00806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.525+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008065e00 con 0x7fa008089f80 2024-09-15T03:47:43.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9ff8012630 con 0x7fa0080975f0 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7f9fef7fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ff4038740 0x7f9ff403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9ff804c950 con 0x7fa0080975f0 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 0x7fa00806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient: get_auth_request con 0x7fa008089f80 auth_method 0 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa007fff640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ff4038740 0x7f9ff403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa007fff640 10 monclient: get_auth_request con 0x7f9ff4038740 auth_method 0 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 10 monclient(hunting): handle_auth_done global_id 204385 payload 824 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.526+0000 7fa006ffd640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa00804bf60 con 0x7fa008089f80 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7fa007fff640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ff4038740 0x7f9ff403abf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa000006e70 tx=0x7fa000008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7fa006ffd640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 msgr2=0x7fa00806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7fa006ffd640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 0x7fa00806c7e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc034000 tx=0x7f9ffc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc031cb0 con 0x7fa008089f80 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 10 monclient: dump: 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.551 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9ffc02ae90 con 0x7fa008089f80 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 msgr2=0x7fa00806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 0x7fa00806c7e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc034000 tx=0x7f9ffc032000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 msgr2=0x7fa00806c7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:43.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.529+0000 7f9fef7fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa008089f80 0x7fa00806c7e0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc034000 tx=0x7f9ffc032000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa00e15d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa00e15d640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 0x7fa00806d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa00e15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00808b170 con 0x7fa00806cd20 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 0x7fa00806d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient: get_auth_request con 0x7fa00806cd20 auth_method 0 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 10 monclient(hunting): handle_auth_done global_id 204388 payload 824 2024-09-15T03:47:43.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.535+0000 7fa006ffd640 1 -- 172.21.15.153:0/3885072765 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa00808a350 con 0x7fa00806cd20 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7fa006ffd640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 msgr2=0x7fa00806d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7fa006ffd640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 0x7fa00806d120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc043000 tx=0x7f9ffc03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc055070 con 0x7fa00806cd20 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 10 monclient: dump: 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.555 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f9ffc03ce20 con 0x7fa00806cd20 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 msgr2=0x7fa00806d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 0x7fa00806d120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc043000 tx=0x7f9ffc03d000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 msgr2=0x7fa00806d120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:47:43.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.538+0000 7f9fef7fe640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa00806cd20 0x7fa00806d120 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc043000 tx=0x7f9ffc03d000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 200, 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:47:43.557 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 6, 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.558 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.559 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:47:43.560 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:47:43.561 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.562 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:47:43.563 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:47:43.564 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.566 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:47:43.567 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.568 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.569 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:47:43.570 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ff4038740 msgr2=0x7f9ff403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ff4038740 0x7f9ff403abf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa000006e70 tx=0x7fa000008040 comp rx=0 tx=0).stop 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 10 monclient: shutdown 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 msgr2=0x7fa008088fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 --2- 172.21.15.153:0/3885072765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0080975f0 0x7fa008088fc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c8b0 tx=0x7f9ff80026c0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa007fff640 1 -- 172.21.15.153:0/3885072765 reap_dead start 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa007fff640 1 -- 172.21.15.153:0/3885072765 reap_dead start 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 shutdown_connections 2024-09-15T03:47:43.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 >> 172.21.15.153:0/3885072765 conn(0x7fa0080678a0 msgr2=0x7fa0080963b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 shutdown_connections 2024-09-15T03:47:43.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.539+0000 7fa00e15d640 1 -- 172.21.15.153:0/3885072765 wait complete. 2024-09-15T03:47:43.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:47:43.573 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:47:43.573 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:47:43.573 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:47:43.573 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:47:43.574 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:47:43.574 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:47:43.583 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:47:43.584 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:47:43.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:43.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.715+0000 7f185db5d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.715+0000 7f185db5d640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.715+0000 7f185db5d640 10 monclient: monmap: 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:43.715945+0000 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:43.715945+0000 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:43.716 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:43.717 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.715+0000 7f185db5d640 1 Processor -- start 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.715+0000 7f185db5d640 1 -- start start 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.715+0000 7f185db5d640 10 monclient: init 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1858131ab0 0x7f1858131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1858131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:43.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f185809e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1858136bb0 0x7f1858138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1858136bb0 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185809faf0 con 0x7f185809e910 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185809fd30 con 0x7f1858131ab0 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18580a1110 con 0x7f1858136bb0 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f185db5d640 10 monclient(hunting): authenticate will time out at 1861.355453s 2024-09-15T03:47:43.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1858131ab0 0x7f1858131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1856ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f185809e910 0x7f185809ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:41040/0 (socket says 172.21.15.153:41040) 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1856ffd640 1 -- 172.21.15.153:0/1892414094 learned_addr learned my addr 172.21.15.153:0/1892414094 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): get_auth_request con 0x7f1858131ab0 auth_method 0 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1857fff640 1 --2- 172.21.15.153:0/1892414094 >> v2:172.21.15.181:3300/0 conn(0x7f1858136bb0 0x7f1858138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1857fff640 10 monclient(hunting): get_auth_request con 0x7f1858136bb0 auth_method 0 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1857fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1857fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f1857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient(hunting): handle_auth_done global_id 214353 payload 274 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 1 -- 172.21.15.153:0/1892414094 >> v2:172.21.15.181:3300/0 conn(0x7f1858136bb0 msgr2=0x7f1858138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 1 --2- 172.21.15.153:0/1892414094 >> v2:172.21.15.181:3300/0 conn(0x7f1858136bb0 0x7f1858138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 1 -- 172.21.15.153:0/1892414094 >> v1:172.21.15.153:6789/0 conn(0x7f185809e910 legacy=0x7f185809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 1 monclient: found mon.noname-a 2024-09-15T03:47:43.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.716+0000 7f18577fe640 1 -- 172.21.15.153:0/1892414094 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580a13f0 con 0x7f1858131ab0 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.719+0000 7f18577fe640 1 -- 172.21.15.153:0/1892414094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 msgr2=0x7f1858131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.719+0000 7f18577fe640 1 --2- 172.21.15.153:0/1892414094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858131e80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1848009850 tx=0x7f18480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d88addc1ab26f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 1 -- 172.21.15.153:0/1892414094 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184802ae50 con 0x7f1858131ab0 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: dump: 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.732 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 1 -- 172.21.15.153:0/1892414094 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848032440 con 0x7f1858131ab0 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 5 monclient: authenticate success, global_id 214353 2024-09-15T03:47:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 1 -- 172.21.15.153:0/1892414094 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18480327e0 con 0x7f1858131ab0 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 4 set_mon_vals no callback set 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f18567fc640 10 monclient: dump: 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 10 monclient: shutdown 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 -- 172.21.15.153:0/1892414094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 msgr2=0x7f1858131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 --2- 172.21.15.153:0/1892414094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858131e80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1848009850 tx=0x7f18480090f0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 -- 172.21.15.153:0/1892414094 shutdown_connections 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 --2- 172.21.15.153:0/1892414094 >> v2:172.21.15.181:3300/0 conn(0x7f1858136bb0 0x7f1858138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 --2- 172.21.15.153:0/1892414094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858131e80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 -- 172.21.15.153:0/1892414094 >> 172.21.15.153:0/1892414094 conn(0x7f1858067d30 msgr2=0x7f18580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 -- 172.21.15.153:0/1892414094 shutdown_connections 2024-09-15T03:47:43.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.720+0000 7f185db5d640 1 -- 172.21.15.153:0/1892414094 wait complete. 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.721+0000 7f185db5d640 10 monclient: build_initial_monmap 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.721+0000 7f185db5d640 10 monclient: monmap: 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.721+0000 7f185db5d640 1 Processor -- start 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.721+0000 7f185db5d640 1 -- start start 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.721+0000 7f185db5d640 10 monclient: init 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185809e910 0x7f185812fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): picked mon.noname-b con 0x7f185809e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858130320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1858131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1858136bb0 0x7f1858097520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1858136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18580a2240 con 0x7f1858131ab0 2024-09-15T03:47:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185809ac40 con 0x7f185809e910 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185809af20 con 0x7f1858136bb0 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f185db5d640 10 monclient(hunting): authenticate will time out at 1861.361216s 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1857fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1858136bb0 msgr2=0x7f1858097520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1857fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1858136bb0 0x7f1858097520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858130320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858130320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42916/0 (socket says 172.21.15.153:42916) 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 -- 172.21.15.153:0/4201500068 learned_addr learned my addr 172.21.15.153:0/4201500068 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): get_auth_request con 0x7f1858131ab0 auth_method 0 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185809e910 0x7f185812fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): get_auth_request con 0x7f185809e910 auth_method 0 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f18577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient(hunting): handle_auth_done global_id 214356 payload 824 2024-09-15T03:47:43.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 -- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1858136bb0 msgr2=0x7f1858097520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1858136bb0 0x7f1858097520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 -- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185809e910 msgr2=0x7f185812fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185809e910 0x7f185812fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 monclient: found mon.noname-a 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.722+0000 7f1856ffd640 1 -- 172.21.15.153:0/4201500068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848009390 con 0x7f1858131ab0 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f1856ffd640 1 -- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 msgr2=0x7f1858130320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f1856ffd640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858130320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f184000c5e0 tx=0x7f1840004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840005cb0 con 0x7f1858131ab0 2024-09-15T03:47:43.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: dump: 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:47:43.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1840005e30 con 0x7f1858131ab0 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f185db5d640 5 monclient: authenticate success, global_id 214356 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f185db5d640 10 monclient: _renew_subs 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f185db5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1858099180 con 0x7f1858131ab0 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840007340 con 0x7f1858131ab0 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f184f7fe640 10 monclient: dump: 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:43.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f185db5d640 10 monclient: _renew_subs 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f1855ffb640 4 set_mon_vals no callback set 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.724+0000 7f185db5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.725+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1858099d50 con 0x7f1858131ab0 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.725+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f184001b400 con 0x7f1858131ab0 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.725+0000 7f185db5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.725+0000 7f185db5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.725+0000 7f185db5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.725+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185804bf60 con 0x7f1858131ab0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.726+0000 7f184f7fe640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18380386f0 0x7f183803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.726+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f184004c0f0 con 0x7f1858131ab0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.726+0000 7f18577fe640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18380386f0 0x7f183803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.726+0000 7f18577fe640 10 monclient: get_auth_request con 0x7f18380386f0 auth_method 0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.728+0000 7f18577fe640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18380386f0 0x7f183803aba0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f185809bdb0 tx=0x7f1848032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.730+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f184001a720 con 0x7f1858131ab0 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.730+0000 7f184f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:43.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.730+0000 7f184f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:43.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.944+0000 7f185db5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:43.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.944+0000 7f185db5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:43.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.944+0000 7f185db5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:43.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.944+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f18580a26a0 con 0x7f1858131ab0 2024-09-15T03:47:43.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.945+0000 7f184f7fe640 1 -- 172.21.15.153:0/4201500068 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f184001a360 con 0x7f1858131ab0 2024-09-15T03:47:43.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.945+0000 7f184f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:43.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.945+0000 7f184f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:43.946 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:43.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18380386f0 msgr2=0x7f183803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18380386f0 0x7f183803aba0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f185809bdb0 tx=0x7f1848032000 comp rx=0 tx=0).stop 2024-09-15T03:47:43.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 10 monclient: shutdown 2024-09-15T03:47:43.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:43.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 msgr2=0x7f1858130320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:43.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858130320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f184000c5e0 tx=0x7f1840004960 comp rx=0 tx=0).stop 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 shutdown_connections 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f18380386f0 0x7f183803aba0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1858136bb0 0x7f1858097520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1858131ab0 0x7f1858130320 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 --2- 172.21.15.153:0/4201500068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185809e910 0x7f185812fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 >> 172.21.15.153:0/4201500068 conn(0x7f1858067d30 msgr2=0x7f185809d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 shutdown_connections 2024-09-15T03:47:43.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:43.947+0000 7f185db5d640 1 -- 172.21.15.153:0/4201500068 wait complete. 2024-09-15T03:47:46.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.063+0000 7f270fcef640 10 monclient: get_monmap_and_config 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.064+0000 7f270fcef640 10 monclient: build_initial_monmap 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.064+0000 7f270fcef640 10 monclient: monmap: 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:47.064763+0000 2024-09-15T03:47:47.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:47.064763+0000 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.064+0000 7f270fcef640 1 Processor -- start 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.064+0000 7f270fcef640 1 -- start start 2024-09-15T03:47:47.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.064+0000 7f270fcef640 10 monclient: init 2024-09-15T03:47:47.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27081318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f270809ea50 0x7f270809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): picked mon.noname-b con 0x7f270809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2708136c20 0x7f2708139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): picked mon.noname-a con 0x7f2708136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2708132640 con 0x7f27081318f0 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2708131d00 con 0x7f2708136c20 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270809fbf0 con 0x7f270809ea50 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:47.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270fcef640 10 monclient(hunting): authenticate will time out at 1864.704498s 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270d28a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f270809ea50 msgr2=0x7f270809ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270d28a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f270809ea50 0x7f270809ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2708136c20 0x7f2708139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2708136c20 0x7f2708139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33216/0 (socket says 172.21.15.153:33216) 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 -- 172.21.15.153:0/4174602065 learned_addr learned my addr 172.21.15.153:0/4174602065 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): get_auth_request con 0x7f2708136c20 auth_method 0 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient(hunting): handle_auth_done global_id 214359 payload 274 2024-09-15T03:47:47.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 -- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f270809ea50 msgr2=0x7f270809ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 --2- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f270809ea50 0x7f270809ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 -- 172.21.15.153:0/4174602065 >> v1:172.21.15.181:6789/0 conn(0x7f27081318f0 legacy=0x7f2708131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient: _finish_hunting 0 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 monclient: found mon.noname-a 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:47.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.065+0000 7f270e28c640 1 -- 172.21.15.153:0/4174602065 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270809fe30 con 0x7f2708136c20 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.068+0000 7f270e28c640 1 -- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2708136c20 msgr2=0x7f2708139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.068+0000 7f270e28c640 1 --2- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2708136c20 0x7f2708139000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2700009f20 tx=0x7f2700009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff33c51142e3b05 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 1 -- 172.21.15.153:0/4174602065 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700007610 con 0x7f2708136c20 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: dump: 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: _finish_auth 0 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 1 -- 172.21.15.153:0/4174602065 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270002d050 con 0x7f2708136c20 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 5 monclient: authenticate success, global_id 214359 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 4 set_mon_vals no callback set 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 1 -- 172.21.15.153:0/4174602065 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27000079a0 con 0x7f2708136c20 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270ca89640 10 monclient: dump: 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:47.077 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 10 monclient: shutdown 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 -- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2708136c20 msgr2=0x7f2708139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 --2- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2708136c20 0x7f2708139000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2700009f20 tx=0x7f2700009840 comp rx=0 tx=0).stop 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 -- 172.21.15.153:0/4174602065 shutdown_connections 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 --2- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2708136c20 0x7f2708139000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 --2- 172.21.15.153:0/4174602065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f270809ea50 0x7f270809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 -- 172.21.15.153:0/4174602065 >> 172.21.15.153:0/4174602065 conn(0x7f2708067110 msgr2=0x7f2708067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 -- 172.21.15.153:0/4174602065 shutdown_connections 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.071+0000 7f270fcef640 1 -- 172.21.15.153:0/4174602065 wait complete. 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.072+0000 7f270fcef640 10 monclient: build_initial_monmap 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.072+0000 7f270fcef640 10 monclient: monmap: 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.072+0000 7f270fcef640 1 Processor -- start 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.072+0000 7f270fcef640 1 -- start start 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.072+0000 7f270fcef640 10 monclient: init 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:47:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 0x7f27081412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): picked mon.noname-a con 0x7f270809ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27081318f0 0x7f2708141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): picked mon.noname-b con 0x7f27081318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2708136c20 0x7f2708099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): picked mon.noname-c con 0x7f2708136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270809b0e0 con 0x7f270809ea50 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270809bf10 con 0x7f27081318f0 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270809de00 con 0x7f2708136c20 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:47.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270fcef640 10 monclient(hunting): authenticate will time out at 1864.712151s 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270e28c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2708136c20 msgr2=0x7f2708099510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270e28c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2708136c20 0x7f2708099510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 0x7f27081412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 0x7f27081412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33230/0 (socket says 172.21.15.153:33230) 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 -- 172.21.15.153:0/56470690 learned_addr learned my addr 172.21.15.153:0/56470690 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): get_auth_request con 0x7f270809ea50 auth_method 0 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:47.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient(hunting): handle_auth_done global_id 214362 payload 824 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 -- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2708136c20 msgr2=0x7f2708099510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2708136c20 0x7f2708099510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 -- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27081318f0 msgr2=0x7f2708141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270d28a640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27081318f0 0x7f2708141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27081318f0 0x7f2708141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient: _finish_hunting 0 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 monclient: found mon.noname-a 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:47.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:47.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270da8b640 1 -- 172.21.15.153:0/56470690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2700009ae0 con 0x7f270809ea50 2024-09-15T03:47:47.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270d28a640 10 monclient: get_auth_request con 0x7f27081318f0 auth_method 0 2024-09-15T03:47:47.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.073+0000 7f270d28a640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27081318f0 0x7f2708141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:47:47.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f270da8b640 1 -- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 msgr2=0x7f27081412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:47.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f270da8b640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 0x7f27081412c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f270400c5e0 tx=0x7f2704007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704012920 con 0x7f270809ea50 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: dump: 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: _finish_auth 0 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2704012a80 con 0x7f270809ea50 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f270fcef640 5 monclient: authenticate success, global_id 214362 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f270fcef640 10 monclient: _renew_subs 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f270fcef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270811c7f0 con 0x7f270809ea50 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704012e40 con 0x7f270809ea50 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fdffb640 10 monclient: dump: 2024-09-15T03:47:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.075+0000 7f26fffff640 4 set_mon_vals no callback set 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.076+0000 7f270fcef640 10 monclient: _renew_subs 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.076+0000 7f270fcef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.076+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2708141d40 con 0x7f270809ea50 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.076+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2704011590 con 0x7f270809ea50 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f26fdffb640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26ec038720 0x7f26ec03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f270404c7a0 con 0x7f270809ea50 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f270fcef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f270fcef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f270fcef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270804bf60 con 0x7f270809ea50 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f270d28a640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26ec038720 0x7f26ec03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.077+0000 7f270d28a640 10 monclient: get_auth_request con 0x7f26ec038720 auth_method 0 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.080+0000 7f270d28a640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26ec038720 0x7f26ec03abd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f26f8003850 tx=0x7f26f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.081+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2704016070 con 0x7f270809ea50 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.081+0000 7f26fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.081+0000 7f26fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f270fcef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f270fcef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f270fcef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f270811ced0 con 0x7f270809ea50 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f26fdffb640 1 -- 172.21.15.153:0/56470690 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f27040108d0 con 0x7f270809ea50 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f26fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.293+0000 7f26fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:47.295 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:47.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26ec038720 msgr2=0x7f26ec03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:47.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26ec038720 0x7f26ec03abd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f26f8003850 tx=0x7f26f8008040 comp rx=0 tx=0).stop 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 10 monclient: shutdown 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 msgr2=0x7f27081412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 0x7f27081412c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f270400c5e0 tx=0x7f2704007ed0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 shutdown_connections 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f26ec038720 0x7f26ec03abd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2708136c20 0x7f2708099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27081318f0 0x7f2708141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 --2- 172.21.15.153:0/56470690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f270809ea50 0x7f27081412c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 >> 172.21.15.153:0/56470690 conn(0x7f2708067110 msgr2=0x7f27080a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 shutdown_connections 2024-09-15T03:47:47.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:47.295+0000 7f270fcef640 1 -- 172.21.15.153:0/56470690 wait complete. 2024-09-15T03:47:50.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:50.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.413+0000 7f9b660e8640 10 monclient: get_monmap_and_config 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.413+0000 7f9b660e8640 10 monclient: build_initial_monmap 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.413+0000 7f9b660e8640 10 monclient: monmap: 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:50.413901+0000 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:50.413901+0000 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:50.414 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:50.415 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:50.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 Processor -- start 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 -- start start 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient: init 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b601318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b6009ea50 0x7f9b6009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b6009ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:50.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9b60136c20 0x7f9b60139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b60136c20 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b60132640 con 0x7f9b601318f0 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b60131d00 con 0x7f9b60136c20 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6009fbf0 con 0x7f9b6009ea50 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b660e8640 10 monclient(hunting): authenticate will time out at 1868.053746s 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b57fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b6009ea50 msgr2=0x7f9b6009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b6009ea50 0x7f9b6009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9b60136c20 0x7f9b60139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9b60136c20 0x7f9b60139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33234/0 (socket says 172.21.15.153:33234) 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 1 -- 172.21.15.153:0/2050845501 learned_addr learned my addr 172.21.15.153:0/2050845501 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 10 monclient(hunting): get_auth_request con 0x7f9b60136c20 auth_method 0 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.414+0000 7f9b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 10 monclient(hunting): handle_auth_done global_id 214365 payload 274 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 1 -- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b6009ea50 msgr2=0x7f9b6009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 1 --2- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b6009ea50 0x7f9b6009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 1 -- 172.21.15.153:0/2050845501 >> v1:172.21.15.181:6789/0 conn(0x7f9b601318f0 legacy=0x7f9b60131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 1 monclient: found mon.noname-a 2024-09-15T03:47:50.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:50.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:50.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.415+0000 7f9b5ffff640 1 -- 172.21.15.153:0/2050845501 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6009fe30 con 0x7f9b60136c20 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.417+0000 7f9b5ffff640 1 -- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b60136c20 msgr2=0x7f9b60139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.417+0000 7f9b5ffff640 1 --2- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b60136c20 0x7f9b60139000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9b58009f20 tx=0x7f9b58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea5edb8649cd3fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 1 -- 172.21.15.153:0/2050845501 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58007610 con 0x7f9b60136c20 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: dump: 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.426 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: _finish_auth 0 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 1 -- 172.21.15.153:0/2050845501 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5802d050 con 0x7f9b60136c20 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 5 monclient: authenticate success, global_id 214365 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:50.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 4 set_mon_vals no callback set 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 1 -- 172.21.15.153:0/2050845501 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b580079e0 con 0x7f9b60136c20 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b5effd640 10 monclient: dump: 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:50.428 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 10 monclient: shutdown 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 -- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b60136c20 msgr2=0x7f9b60139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 --2- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b60136c20 0x7f9b60139000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9b58009f20 tx=0x7f9b58009840 comp rx=0 tx=0).stop 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 -- 172.21.15.153:0/2050845501 shutdown_connections 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 --2- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b60136c20 0x7f9b60139000 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 --2- 172.21.15.153:0/2050845501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b6009ea50 0x7f9b6009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 -- 172.21.15.153:0/2050845501 >> 172.21.15.153:0/2050845501 conn(0x7f9b60067110 msgr2=0x7f9b60067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 -- 172.21.15.153:0/2050845501 shutdown_connections 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.418+0000 7f9b660e8640 1 -- 172.21.15.153:0/2050845501 wait complete. 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.419+0000 7f9b660e8640 10 monclient: build_initial_monmap 2024-09-15T03:47:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.419+0000 7f9b660e8640 10 monclient: monmap: 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:50.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.419+0000 7f9b660e8640 1 Processor -- start 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.419+0000 7f9b660e8640 1 -- start start 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.419+0000 7f9b660e8640 10 monclient: init 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 0x7f9b601418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b6009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b601318f0 0x7f9b60141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b601318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b60136c20 0x7f9b60097570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b60136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6009bf10 con 0x7f9b6009ea50 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6009d870 con 0x7f9b601318f0 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600995c0 con 0x7f9b60136c20 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b660e8640 10 monclient(hunting): authenticate will time out at 1868.059148s 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5ffff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b60136c20 msgr2=0x7f9b60097570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b60136c20 0x7f9b60097570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 0x7f9b601418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 0x7f9b601418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33248/0 (socket says 172.21.15.153:33248) 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 -- 172.21.15.153:0/892625836 learned_addr learned my addr 172.21.15.153:0/892625836 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b6009ea50 auth_method 0 2024-09-15T03:47:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b57fff640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b601318f0 0x7f9b60141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient(hunting): handle_auth_done global_id 214368 payload 824 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 -- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b60136c20 msgr2=0x7f9b60097570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b60136c20 0x7f9b60097570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 -- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b601318f0 msgr2=0x7f9b60141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b601318f0 0x7f9b60141e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:47:50.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b5f7fe640 1 -- 172.21.15.153:0/892625836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b58009ae0 con 0x7f9b6009ea50 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b57fff640 10 monclient: get_auth_request con 0x7f9b601318f0 auth_method 0 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.420+0000 7f9b57fff640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b601318f0 0x7f9b60141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b5f7fe640 1 -- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 msgr2=0x7f9b601418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b5f7fe640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 0x7f9b601418e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000c5e0 tx=0x7f9b50007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50012920 con 0x7f9b6009ea50 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 10 monclient: dump: 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:50.435 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 10 monclient: _finish_auth 0 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b50012a80 con 0x7f9b6009ea50 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b660e8640 5 monclient: authenticate success, global_id 214368 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.422+0000 7f9b660e8640 10 monclient: _renew_subs 2024-09-15T03:47:50.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b60099110 con 0x7f9b6009ea50 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50012e40 con 0x7f9b6009ea50 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b577fe640 10 monclient: dump: 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:50.437 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 10 monclient: _renew_subs 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6011bc30 con 0x7f9b6009ea50 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b5e7fc640 4 set_mon_vals no callback set 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9b50011590 con 0x7f9b6009ea50 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.423+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6004bf60 con 0x7f9b6009ea50 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.424+0000 7f9b577fe640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b40038740 0x7f9b4003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:50.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.424+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9b5004c900 con 0x7f9b6009ea50 2024-09-15T03:47:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.424+0000 7f9b57fff640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b40038740 0x7f9b4003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.424+0000 7f9b57fff640 10 monclient: get_auth_request con 0x7f9b40038740 auth_method 0 2024-09-15T03:47:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.426+0000 7f9b57fff640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b40038740 0x7f9b4003abf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9b48003940 tx=0x7f9b48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.428+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9b50010ac0 con 0x7f9b6009ea50 2024-09-15T03:47:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.428+0000 7f9b577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.428+0000 7f9b577fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:47:50.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b660e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:47:50.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b660e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:50.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b660e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:50.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9b6009a850 con 0x7f9b6009ea50 2024-09-15T03:47:50.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b577fe640 1 -- 172.21.15.153:0/892625836 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f9b50016020 con 0x7f9b6009ea50 2024-09-15T03:47:50.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:47:50.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.646+0000 7f9b577fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:47:50.648 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:47:50.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b40038740 msgr2=0x7f9b4003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b40038740 0x7f9b4003abf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9b48003940 tx=0x7f9b48008040 comp rx=0 tx=0).stop 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 10 monclient: shutdown 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 msgr2=0x7f9b601418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 0x7f9b601418e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000c5e0 tx=0x7f9b50007ed0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 shutdown_connections 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9b40038740 0x7f9b4003abf0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b60136c20 0x7f9b60097570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b601318f0 0x7f9b60141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 --2- 172.21.15.153:0/892625836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b6009ea50 0x7f9b601418e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:50.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 >> 172.21.15.153:0/892625836 conn(0x7f9b60067110 msgr2=0x7f9b600a4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:50.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 shutdown_connections 2024-09-15T03:47:50.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:50.648+0000 7f9b660e8640 1 -- 172.21.15.153:0/892625836 wait complete. 2024-09-15T03:47:53.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:47:53.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.764+0000 7f9107b1d640 10 monclient: get_monmap_and_config 2024-09-15T03:47:53.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.764+0000 7f9107b1d640 10 monclient: build_initial_monmap 2024-09-15T03:47:53.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.764+0000 7f9107b1d640 10 monclient: monmap: 2024-09-15T03:47:53.765 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:53.765 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:53.765 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:47:53.765355+0000 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:47:53.765355+0000 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:47:53.766 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 1 Processor -- start 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 1 -- start start 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient: init 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91000981c0 addr v1:172.21.15.153:6789/0 2024-09-15T03:47:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9100131da0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9100132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910009ca60 con 0x7f91000981c0 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910009cca0 con 0x7f9100131da0 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9100131bd0 con 0x7f9100132170 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.765+0000 7f9107b1d640 10 monclient(hunting): authenticate will time out at 1871.404948s 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91058b9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f91000981c0 0x7f9100098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50496/0 (socket says 172.21.15.153:50496) 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33252/0 (socket says 172.21.15.153:33252) 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91058b9640 1 -- 172.21.15.153:0/1603188829 learned_addr learned my addr 172.21.15.153:0/1603188829 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:47:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 10 monclient(hunting): get_auth_request con 0x7f9100132170 auth_method 0 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.766+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:53.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.768+0000 7f91058b9640 1 -- 172.21.15.153:0/1603188829 >> v1:172.21.15.153:6789/0 conn(0x7f91000981c0 legacy=0x7f9100098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:53.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.768+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v1:172.21.15.181:6789/0 conn(0x7f9100131da0 legacy=0x7f9100136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:53.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:53.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 10 monclient(hunting): get_auth_request con 0x7f9100132170 auth_method 0 2024-09-15T03:47:53.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:53.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:53.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:53.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:53.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:53.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:53.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:53.967+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 10 monclient(hunting): get_auth_request con 0x7f9100132170 auth_method 0 2024-09-15T03:47:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:54.368+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 10 monclient(hunting): get_auth_request con 0x7f9100132170 auth_method 0 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:55.169+0000 7f91060ba640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:56.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): tick 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 monclient(hunting): continuing hunt 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 msgr2=0x7f910013a2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f910013a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 >> v1:172.21.15.181:6789/0 conn(0x7f9100131da0 legacy=0x7f9100136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 >> v1:172.21.15.153:6789/0 conn(0x7f91000981c0 legacy=0x7f9100098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient: _add_conns ranks=[2,4] 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 0x7f90e4005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): picked mon.noname-b con 0x7f90e40037d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): picked mon.noname-c con 0x7f90e4006340 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9100131bd0 con 0x7f90e4006340 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e4008e90 con 0x7f90e40037d0 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f90f77fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91058b9640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 msgr2=0x7f90e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91058b9640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 0x7f90e4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 10 monclient(hunting): get_auth_request con 0x7f90e4006340 auth_method 0 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.766+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:56.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91058b9640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 msgr2=0x7f90e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:56.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91058b9640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 0x7f90e4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 10 monclient(hunting): get_auth_request con 0x7f90e4006340 auth_method 0 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:56.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:56.967+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:47:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91058b9640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 msgr2=0x7f90e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91058b9640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 0x7f90e4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 10 monclient(hunting): get_auth_request con 0x7f90e4006340 auth_method 0 2024-09-15T03:47:57.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:47:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:57.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:57.368+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:47:58.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91058b9640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 msgr2=0x7f90e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:58.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91058b9640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 0x7f90e4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:58.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:58.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91050b8640 10 monclient(hunting): get_auth_request con 0x7f90e4006340 auth_method 0 2024-09-15T03:47:58.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91050b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:58.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91050b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:58.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.169+0000 7f91050b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:47:58.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.170+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:47:58.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.170+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:47:58.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.170+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:47:58.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:58.170+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:47:59.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): tick 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 monclient(hunting): continuing hunt 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 msgr2=0x7f90e4005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e40037d0 0x7f90e4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 msgr2=0x7f90e40087d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4006340 0x7f90e40087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4009310 0x7f90e400b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): picked mon.noname-c con 0x7f90e4009310 addr v2:172.21.15.181:3300/0 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91060ba640 1 -- 172.21.15.153:0/1603188829 reap_dead start 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f90e40087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:59.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): picked mon.noname-a con 0x7f9100132170 addr v2:172.21.15.153:3300/0 2024-09-15T03:47:59.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9100131da0 addr v1:172.21.15.181:6789/0 2024-09-15T03:47:59.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910009e010 con 0x7f9100131da0 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9100131bd0 con 0x7f9100132170 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 1 -- 172.21.15.153:0/1603188829 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e4008e90 con 0x7f90e4009310 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f90f77fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.153:3300/0 conn(0x7f9100132170 0x7f90e40087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): get_auth_request con 0x7f9100132170 auth_method 0 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4009310 0x7f90e400b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 10 monclient(hunting): get_auth_request con 0x7f90e4009310 auth_method 0 2024-09-15T03:47:59.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91050b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.766+0000 7f91058b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91058b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 10 monclient(hunting): handle_auth_done global_id 214371 payload 274 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4009310 msgr2=0x7f90e400b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4009310 0x7f90e400b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> v1:172.21.15.181:6789/0 conn(0x7f9100131da0 legacy=0x7f90e400dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 10 monclient: _finish_hunting 0 2024-09-15T03:47:59.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 1 monclient: found mon.noname-a 2024-09-15T03:47:59.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:59.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:59.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.767+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e4003c00 con 0x7f9100132170 2024-09-15T03:47:59.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91050b8640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 msgr2=0x7f90e40087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:59.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91050b8640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f90e40087d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9100071480 tx=0x7f90f0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed89c87dab6ff291 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:59.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 1 -- 172.21.15.153:0/1603188829 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0030480 con 0x7f9100132170 2024-09-15T03:47:59.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: dump: 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:59.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: _finish_auth 0 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 1 -- 172.21.15.153:0/1603188829 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f002d030 con 0x7f9100132170 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f9107b1d640 5 monclient: authenticate success, global_id 214371 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f9107b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 1 -- 172.21.15.153:0/1603188829 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0030810 con 0x7f9100132170 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f9107b1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f9107b1d640 4 set_mon_vals no callback set 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:59.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.770+0000 7f91048b7640 10 monclient: dump: 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 10 monclient: shutdown 2024-09-15T03:47:59.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 -- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 msgr2=0x7f90e40087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f90e40087d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9100071480 tx=0x7f90f0009040 comp rx=0 tx=0).stop 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 -- 172.21.15.153:0/1603188829 shutdown_connections 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 --2- 172.21.15.153:0/1603188829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f90e40087d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 --2- 172.21.15.153:0/1603188829 >> v2:172.21.15.181:3300/0 conn(0x7f90e4009310 0x7f90e400b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 -- 172.21.15.153:0/1603188829 >> 172.21.15.153:0/1603188829 conn(0x7f91000a1e30 msgr2=0x7f91000a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 -- 172.21.15.153:0/1603188829 shutdown_connections 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 1 -- 172.21.15.153:0/1603188829 wait complete. 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 10 monclient: build_initial_monmap 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.771+0000 7f9107b1d640 10 monclient: monmap: 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:47:59.784 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 Processor -- start 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 -- start start 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient: init 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e4009310 0x7f91001410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f90e4009310 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:47:59.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9100131da0 0x7f9100130890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9100131da0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f910011a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9100132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910011dc90 con 0x7f9100132170 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9100098cb0 con 0x7f9100131da0 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91000a04c0 con 0x7f90e4009310 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f9107b1d640 10 monclient(hunting): authenticate will time out at 1877.411911s 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f91058b9640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e4009310 msgr2=0x7f91001410e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:47:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.772+0000 7f91058b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e4009310 0x7f91001410e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f910011a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f910011a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54662/0 (socket says 172.21.15.153:54662) 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 -- 172.21.15.153:0/209483861 learned_addr learned my addr 172.21.15.153:0/209483861 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): get_auth_request con 0x7f9100132170 auth_method 0 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9100131da0 0x7f9100130890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): get_auth_request con 0x7f9100131da0 auth_method 0 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:47:59.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91050b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient(hunting): handle_auth_done global_id 214374 payload 824 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 -- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e4009310 msgr2=0x7f91001410e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e4009310 0x7f91001410e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 -- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9100131da0 msgr2=0x7f9100130890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9100131da0 0x7f9100130890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient: _finish_hunting 0 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 monclient: found mon.noname-a 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:47:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.773+0000 7f91060ba640 1 -- 172.21.15.153:0/209483861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f000f040 con 0x7f9100132170 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.775+0000 7f91060ba640 1 -- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 msgr2=0x7f910011a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.775+0000 7f91060ba640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f910011a310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f910009d3a0 tx=0x7f90f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f8017890 con 0x7f9100132170 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: dump: 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:59.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f8017a10 con 0x7f9100132170 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 5 monclient: authenticate success, global_id 214374 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 10 monclient: _renew_subs 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f8017dd0 con 0x7f9100132170 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9100141f90 con 0x7f9100132170 2024-09-15T03:47:59.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:47:59.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f5ffb640 10 monclient: dump: 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:47:59.793 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f90f7fff640 4 set_mon_vals no callback set 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 10 monclient: _renew_subs 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.776+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910011bef0 con 0x7f9100132170 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f90f8003040 con 0x7f9100132170 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f9107b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f9107b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f9107b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:47:59.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910004bf60 con 0x7f9100132170 2024-09-15T03:47:59.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f90f5ffb640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90e4039b90 0x7f90e403c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:47:59.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.777+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f90f804cd60 con 0x7f9100132170 2024-09-15T03:47:59.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.778+0000 7f91058b9640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90e4039b90 0x7f90e403c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:47:59.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.778+0000 7f91058b9640 10 monclient: get_auth_request con 0x7f90e4039b90 auth_method 0 2024-09-15T03:47:59.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.780+0000 7f91058b9640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90e4039b90 0x7f90e403c040 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f90fc001460 tx=0x7f90fc001d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:47:59.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.782+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f90f8020400 con 0x7f9100132170 2024-09-15T03:47:59.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.782+0000 7f90f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:47:59.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:47:59.782+0000 7f90f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:00.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f9107b1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:00.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f9107b1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:00.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f9107b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9100098ee0 con 0x7f9100132170 2024-09-15T03:48:00.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f90f5ffb640 1 -- 172.21.15.153:0/209483861 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f90f801b080 con 0x7f9100132170 2024-09-15T03:48:00.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f90f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:00.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.000+0000 7f90f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:00.002 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":204,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:00.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90e4039b90 msgr2=0x7f90e403c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90e4039b90 0x7f90e403c040 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f90fc001460 tx=0x7f90fc001d60 comp rx=0 tx=0).stop 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 10 monclient: shutdown 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 msgr2=0x7f910011a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f910011a310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f910009d3a0 tx=0x7f90f800b040 comp rx=0 tx=0).stop 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 shutdown_connections 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f90e4039b90 0x7f90e403c040 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9100132170 0x7f910011a310 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9100131da0 0x7f9100130890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 --2- 172.21.15.153:0/209483861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90e4009310 0x7f91001410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 >> 172.21.15.153:0/209483861 conn(0x7f91000a1e30 msgr2=0x7f91000a06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:00.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 shutdown_connections 2024-09-15T03:48:00.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.002+0000 7f9107b1d640 1 -- 172.21.15.153:0/209483861 wait complete. 2024-09-15T03:48:00.014 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:48:00.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:48:00.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:48:00.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.108+0000 7ff4dab9d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.108+0000 7ff4dab9d640 10 monclient: build_initial_monmap 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.108+0000 7ff4dab9d640 10 monclient: monmap: 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:00.108819+0000 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:00.108819+0000 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:00.109 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.108+0000 7ff4dab9d640 1 Processor -- start 2024-09-15T03:48:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.108+0000 7ff4dab9d640 1 -- start start 2024-09-15T03:48:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.108+0000 7ff4dab9d640 10 monclient: init 2024-09-15T03:48:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:00.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff4d4131ab0 0x7ff4d4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d4131ab0 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff4d409e910 0x7ff4d409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d409e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409f250 0x7ff4d4068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d409f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d409faf0 con 0x7ff4d409e910 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d409fd30 con 0x7ff4d4131ab0 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40a1110 con 0x7ff4d409f250 2024-09-15T03:48:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4dab9d640 10 monclient(hunting): authenticate will time out at 1877.748291s 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4d913a640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409f250 msgr2=0x7ff4d4068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4d913a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409f250 0x7ff4d4068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff4d409e910 0x7ff4d409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff4d409e910 0x7ff4d409ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54678/0 (socket says 172.21.15.153:54678) 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 -- 172.21.15.153:0/3141699145 learned_addr learned my addr 172.21.15.153:0/3141699145 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): get_auth_request con 0x7ff4d409e910 auth_method 0 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4d8939640 1 --2- 172.21.15.153:0/3141699145 >> v2:172.21.15.181:3300/0 conn(0x7ff4d4131ab0 0x7ff4d4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_done global_id 214377 payload 274 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 -- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409f250 msgr2=0x7ff4d4068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 --2- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409f250 0x7ff4d4068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 -- 172.21.15.153:0/3141699145 >> v2:172.21.15.181:3300/0 conn(0x7ff4d4131ab0 msgr2=0x7ff4d4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 --2- 172.21.15.153:0/3141699145 >> v2:172.21.15.181:3300/0 conn(0x7ff4d4131ab0 0x7ff4d4131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 monclient: found mon.noname-a 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4cbfff640 1 -- 172.21.15.153:0/3141699145 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40a13f0 con 0x7ff4d409e910 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4d8939640 10 monclient: get_auth_request con 0x7ff4d4131ab0 auth_method 0 2024-09-15T03:48:00.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.109+0000 7ff4d8939640 1 --2- 172.21.15.153:0/3141699145 >> v2:172.21.15.181:3300/0 conn(0x7ff4d4131ab0 0x7ff4d4131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cbfff640 1 -- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d409e910 msgr2=0x7ff4d409ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cbfff640 1 --2- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d409e910 0x7ff4d409ed10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4076760 tx=0x7ff4c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73db145098e5618 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 1 -- 172.21.15.153:0/3141699145 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c402ae40 con 0x7ff4d409e910 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: dump: 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4dab9d640 5 monclient: authenticate success, global_id 214377 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 1 -- 172.21.15.153:0/3141699145 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c402f4b0 con 0x7ff4d409e910 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4dab9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4dab9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4dab9d640 4 set_mon_vals no callback set 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 1 -- 172.21.15.153:0/3141699145 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c402f850 con 0x7ff4d409e910 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.112+0000 7ff4cb7fe640 10 monclient: dump: 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 10 monclient: shutdown 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 -- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d409e910 msgr2=0x7ff4d409ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d409e910 0x7ff4d409ed10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4076760 tx=0x7ff4c40091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 -- 172.21.15.153:0/3141699145 shutdown_connections 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409f250 0x7ff4d4068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/3141699145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d409e910 0x7ff4d409ed10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/3141699145 >> v2:172.21.15.181:3300/0 conn(0x7ff4d4131ab0 0x7ff4d4131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 -- 172.21.15.153:0/3141699145 >> 172.21.15.153:0/3141699145 conn(0x7ff4d4067d30 msgr2=0x7ff4d40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 -- 172.21.15.153:0/3141699145 shutdown_connections 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 1 -- 172.21.15.153:0/3141699145 wait complete. 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 10 monclient: build_initial_monmap 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.113+0000 7ff4dab9d640 10 monclient: monmap: 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 Processor -- start 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 -- start start 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient: init 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409e910 0x7ff4d409ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d409e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4d409f250 0x7ff4d4130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d409f250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 0x7ff4d4140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d4131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40a43f0 con 0x7ff4d4131ab0 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d411dc30 con 0x7ff4d409f250 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d411df10 con 0x7ff4d409e910 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4dab9d640 10 monclient(hunting): authenticate will time out at 1877.753764s 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d8939640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409e910 msgr2=0x7ff4d409ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d8939640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409e910 0x7ff4d409ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 0x7ff4d4140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 0x7ff4d4140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54690/0 (socket says 172.21.15.153:54690) 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 learned_addr learned my addr 172.21.15.153:0/352087422 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): get_auth_request con 0x7ff4d4131ab0 auth_method 0 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.114+0000 7ff4d913a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 10 monclient(hunting): handle_auth_done global_id 214380 payload 824 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4cbfff640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4d409f250 0x7ff4d4130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409e910 msgr2=0x7ff4d409ab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4d409e910 0x7ff4d409ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4d409f250 msgr2=0x7ff4d4130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4d409f250 0x7ff4d4130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 10 monclient: _finish_hunting 0 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 1 monclient: found mon.noname-a 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.115+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4009450 con 0x7ff4d4131ab0 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 msgr2=0x7ff4d4140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4d913a640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 0x7ff4d4140f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00bf80 tx=0x7ff4cc009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc016430 con 0x7ff4d4131ab0 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4d913a640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 0x7ff4d4140f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00bf80 tx=0x7ff4cc009730 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7ff4cc0165b0 config(0 keys) v1 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 10 monclient: dump: 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 5 monclient: authenticate success, global_id 214380 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc0165b0 con 0x7ff4d4131ab0 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 10 monclient: _renew_subs 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d409b520 con 0x7ff4d4131ab0 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 10 monclient: _renew_subs 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d411c6f0 con 0x7ff4d4131ab0 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.117+0000 7ff4caffd640 4 set_mon_vals no callback set 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4dab9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 0x7ff4d41414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4063740 con 0x7ff4d411bb90 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 0x7ff4d41414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient: get_auth_request con 0x7ff4d411bb90 auth_method 0 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc016760 con 0x7ff4d4131ab0 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4c8ff9640 10 monclient: dump: 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff4cc012020 con 0x7ff4d4131ab0 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 10 monclient(hunting): handle_auth_done global_id 214383 payload 824 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4d8939640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d404bf60 con 0x7ff4d411bb90 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.118+0000 7ff4c8ff9640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4c00386f0 0x7ff4c003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.119+0000 7ff4cbfff640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4c00386f0 0x7ff4c003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.119+0000 7ff4cbfff640 10 monclient: get_auth_request con 0x7ff4c00386f0 auth_method 0 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.120+0000 7ff4d8939640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 msgr2=0x7ff4d41414d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.120+0000 7ff4d8939640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 0x7ff4d41414d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000c700 tx=0x7ff4d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.120+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0007850 con 0x7ff4d411bb90 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.120+0000 7ff4c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.120+0000 7ff4c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.120+0000 7ff4c8ff9640 10 monclient: dump: 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff4d00079b0 con 0x7ff4d411bb90 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 msgr2=0x7ff4d41414d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 0x7ff4d41414d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000c700 tx=0x7ff4d0008040 comp rx=0 tx=0).stop 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 msgr2=0x7ff4d41414d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4c8ff9640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d411bb90 0x7ff4d41414d0 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000c700 tx=0x7ff4d0008040 comp rx=0 tx=0).stop 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.121+0000 7ff4cbfff640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4c00386f0 0x7ff4c003aba0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff4d409bdb0 tx=0x7ff4c4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.126+0000 7ff4dab9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.126+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 0x7ff4d4141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.126+0000 7ff4dab9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.126+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d411e0d0 con 0x7ff4d4131050 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 0x7ff4d4141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient: get_auth_request con 0x7ff4d4131050 auth_method 0 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 10 monclient(hunting): handle_auth_done global_id 214386 payload 824 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.127+0000 7ff4d8939640 1 -- 172.21.15.153:0/352087422 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff4d4066280 con 0x7ff4d4131050 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4d8939640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 msgr2=0x7ff4d4141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4d8939640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 0x7ff4d4141a10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0016000 tx=0x7ff4d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d001f070 con 0x7ff4d4131050 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 10 monclient: dump: 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7ff4d00117b0 con 0x7ff4d4131050 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 msgr2=0x7ff4d4141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 0x7ff4d4141a10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0016000 tx=0x7ff4d0019000 comp rx=0 tx=0).stop 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 msgr2=0x7ff4d4141a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.129+0000 7ff4c8ff9640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131050 0x7ff4d4141a10 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0016000 tx=0x7ff4d0019000 comp rx=0 tx=0).stop 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 204, 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4c00386f0 msgr2=0x7ff4c003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff4c00386f0 0x7ff4c003aba0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff4d409bdb0 tx=0x7ff4c4032000 comp rx=0 tx=0).stop 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 10 monclient: shutdown 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 msgr2=0x7ff4d4140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 --2- 172.21.15.153:0/352087422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4d4131ab0 0x7ff4d4140f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00bf80 tx=0x7ff4cc009730 comp rx=0 tx=0).stop 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 reap_dead start 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4d913a640 1 -- 172.21.15.153:0/352087422 reap_dead start 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4c8ff9640 1 -- 172.21.15.153:0/352087422 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7ff4cc04c900 con 0x7ff4d4131ab0 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 shutdown_connections 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 >> 172.21.15.153:0/352087422 conn(0x7ff4d4067d30 msgr2=0x7ff4d409d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 shutdown_connections 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.130+0000 7ff4dab9d640 1 -- 172.21.15.153:0/352087422 wait complete. 2024-09-15T03:48:00.158 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:48:00.158 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:48:00.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:48:00.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.251+0000 7ff542a60640 10 monclient: get_monmap_and_config 2024-09-15T03:48:00.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.251+0000 7ff542a60640 10 monclient: build_initial_monmap 2024-09-15T03:48:00.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.251+0000 7ff542a60640 10 monclient: monmap: 2024-09-15T03:48:00.252 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:00.252384+0000 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:00.252384+0000 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:00.253 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.252+0000 7ff542a60640 1 Processor -- start 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.252+0000 7ff542a60640 1 -- start start 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.252+0000 7ff542a60640 10 monclient: init 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff53c131ab0 0x7ff53c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): picked mon.noname-a con 0x7ff53c131ab0 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09e910 0x7ff53c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): picked mon.noname-b con 0x7ff53c09e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff53c09f220 0x7ff53c068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): picked mon.noname-c con 0x7ff53c09f220 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c09faf0 con 0x7ff53c131ab0 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c09fd30 con 0x7ff53c09f220 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0a1110 con 0x7ff53c09e910 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff542a60640 10 monclient(hunting): authenticate will time out at 1877.892111s 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff533fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09e910 msgr2=0x7ff53c09ece0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff533fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09e910 0x7ff53c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff53c131ab0 0x7ff53c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff53c131ab0 0x7ff53c131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54708/0 (socket says 172.21.15.153:54708) 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 -- 172.21.15.153:0/1567150736 learned_addr learned my addr 172.21.15.153:0/1567150736 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): get_auth_request con 0x7ff53c131ab0 auth_method 0 2024-09-15T03:48:00.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff540ffd640 1 --2- 172.21.15.153:0/1567150736 >> v2:172.21.15.181:3300/0 conn(0x7ff53c09f220 0x7ff53c068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient(hunting): handle_auth_done global_id 214389 payload 274 2024-09-15T03:48:00.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 -- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09e910 msgr2=0x7ff53c09ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 --2- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09e910 0x7ff53c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 -- 172.21.15.153:0/1567150736 >> v2:172.21.15.181:3300/0 conn(0x7ff53c09f220 msgr2=0x7ff53c068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 --2- 172.21.15.153:0/1567150736 >> v2:172.21.15.181:3300/0 conn(0x7ff53c09f220 0x7ff53c068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 monclient: found mon.noname-a 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff53bfff640 1 -- 172.21.15.153:0/1567150736 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c0a13f0 con 0x7ff53c131ab0 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff540ffd640 10 monclient: get_auth_request con 0x7ff53c09f220 auth_method 0 2024-09-15T03:48:00.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.253+0000 7ff540ffd640 1 --2- 172.21.15.153:0/1567150736 >> v2:172.21.15.181:3300/0 conn(0x7ff53c09f220 0x7ff53c068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:48:00.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53bfff640 1 -- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c131ab0 msgr2=0x7ff53c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53bfff640 1 --2- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c131ab0 0x7ff53c131e80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff53c09bd80 tx=0x7ff52c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863ceb717a19788c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 1 -- 172.21.15.153:0/1567150736 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c02ae40 con 0x7ff53c131ab0 2024-09-15T03:48:00.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: dump: 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 1 -- 172.21.15.153:0/1567150736 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c02f4b0 con 0x7ff53c131ab0 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff542a60640 5 monclient: authenticate success, global_id 214389 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff542a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 1 -- 172.21.15.153:0/1567150736 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c02f850 con 0x7ff53c131ab0 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff542a60640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff542a60640 4 set_mon_vals no callback set 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.256+0000 7ff53b7fe640 10 monclient: dump: 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.268 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 10 monclient: shutdown 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 -- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c131ab0 msgr2=0x7ff53c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 --2- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c131ab0 0x7ff53c131e80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff53c09bd80 tx=0x7ff52c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 -- 172.21.15.153:0/1567150736 shutdown_connections 2024-09-15T03:48:00.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 --2- 172.21.15.153:0/1567150736 >> v2:172.21.15.181:3300/0 conn(0x7ff53c09f220 0x7ff53c068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 --2- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09e910 0x7ff53c09ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 --2- 172.21.15.153:0/1567150736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c131ab0 0x7ff53c131e80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 -- 172.21.15.153:0/1567150736 >> 172.21.15.153:0/1567150736 conn(0x7ff53c067d30 msgr2=0x7ff53c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 -- 172.21.15.153:0/1567150736 shutdown_connections 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 -- 172.21.15.153:0/1567150736 wait complete. 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 10 monclient: build_initial_monmap 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 10 monclient: monmap: 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 Processor -- start 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 1 -- start start 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.257+0000 7ff542a60640 10 monclient: init 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 0x7ff53c09acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): picked mon.noname-a con 0x7ff53c09e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09f220 0x7ff53c130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): picked mon.noname-c con 0x7ff53c09f220 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c131ab0 0x7ff53c131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): picked mon.noname-b con 0x7ff53c131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c141fa0 con 0x7ff53c09e910 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c09bde0 con 0x7ff53c131ab0 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c09dd40 con 0x7ff53c09f220 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:00.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff542a60640 10 monclient(hunting): authenticate will time out at 1877.897402s 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff533fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09f220 msgr2=0x7ff53c130d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff533fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09f220 0x7ff53c130d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 0x7ff53c09acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 0x7ff53c09acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54710/0 (socket says 172.21.15.153:54710) 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 -- 172.21.15.153:0/1152441717 learned_addr learned my addr 172.21.15.153:0/1152441717 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): get_auth_request con 0x7ff53c09e910 auth_method 0 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient(hunting): handle_auth_done global_id 214392 payload 824 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09f220 msgr2=0x7ff53c130d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c09f220 0x7ff53c130d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c131ab0 msgr2=0x7ff53c131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff540ffd640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c131ab0 0x7ff53c131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c131ab0 0x7ff53c131290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 monclient: found mon.noname-a 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.258+0000 7ff53bfff640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c009450 con 0x7ff53c09e910 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff53bfff640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 msgr2=0x7ff53c09acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff53bfff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 0x7ff53c09acd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff52c035040 tx=0x7ff52c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0329c0 con 0x7ff53c09e910 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: dump: 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.275 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: _finish_auth 0 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c032b60 con 0x7ff53c09e910 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 5 monclient: authenticate success, global_id 214392 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 10 monclient: _renew_subs 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c11e1e0 con 0x7ff53c09e910 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c03d440 con 0x7ff53c09e910 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff538ff9640 10 monclient: dump: 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff53affd640 4 set_mon_vals no callback set 2024-09-15T03:48:00.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 10 monclient: _renew_subs 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.261+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c11cf10 con 0x7ff53c09e910 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff52c039040 con 0x7ff53c09e910 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff542a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff542a60640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 0x7ff53c11b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c063740 con 0x7ff53c141510 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff538ff9640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff51c038720 0x7ff51c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff52c02ae40 con 0x7ff53c09e910 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 0x7ff53c11b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient: get_auth_request con 0x7ff53c141510 auth_method 0 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff540ffd640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff51c038720 0x7ff51c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff540ffd640 10 monclient: get_auth_request con 0x7ff51c038720 auth_method 0 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 10 monclient(hunting): handle_auth_done global_id 204403 payload 824 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.262+0000 7ff533fff640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff53c11e010 con 0x7ff53c141510 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.264+0000 7ff540ffd640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff51c038720 0x7ff51c03abd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e190 tx=0x7ff534007510 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff533fff640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 msgr2=0x7ff53c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff533fff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 0x7ff53c11b6b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e920 tx=0x7ff5240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240050f0 con 0x7ff53c141510 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 10 monclient: dump: 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff524005250 con 0x7ff53c141510 2024-09-15T03:48:00.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:00.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:00.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 msgr2=0x7ff53c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 0x7ff53c11b6b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e920 tx=0x7ff5240091a0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 msgr2=0x7ff53c11b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:00.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.265+0000 7ff538ff9640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c141510 0x7ff53c11b6b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff542a60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff542a60640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 0x7ff53c11bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff542a60640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c11c530 con 0x7ff53c11bbf0 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 0x7ff53c11bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient: get_auth_request con 0x7ff53c11bbf0 auth_method 0 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:00.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 10 monclient(hunting): handle_auth_done global_id 204406 payload 824 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.271+0000 7ff533fff640 1 -- 172.21.15.153:0/1152441717 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff53c09cd80 con 0x7ff53c11bbf0 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff533fff640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 msgr2=0x7ff53c11bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff533fff640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 0x7ff53c11bff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e920 tx=0x7ff524015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524010e60 con 0x7ff53c11bbf0 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 10 monclient: dump: 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7ff524003cb0 con 0x7ff53c11bbf0 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 msgr2=0x7ff53c11bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 0x7ff53c11bff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e920 tx=0x7ff524015000 comp rx=0 tx=0).stop 2024-09-15T03:48:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 msgr2=0x7ff53c11bff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.274+0000 7ff538ff9640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff53c11bbf0 0x7ff53c11bff0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e920 tx=0x7ff524015000 comp rx=0 tx=0).stop 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 204, 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:00.286 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:00.287 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:00.288 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:00.289 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:00.290 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:00.291 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:00.292 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:00.299 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff51c038720 msgr2=0x7ff51c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff51c038720 0x7ff51c03abd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff53c11e190 tx=0x7ff534007510 comp rx=0 tx=0).stop 2024-09-15T03:48:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 10 monclient: shutdown 2024-09-15T03:48:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 msgr2=0x7ff53c09acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 --2- 172.21.15.153:0/1152441717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c09e910 0x7ff53c09acd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff52c035040 tx=0x7ff52c032000 comp rx=0 tx=0).stop 2024-09-15T03:48:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff540ffd640 1 -- 172.21.15.153:0/1152441717 reap_dead start 2024-09-15T03:48:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff540ffd640 1 -- 172.21.15.153:0/1152441717 reap_dead start 2024-09-15T03:48:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 shutdown_connections 2024-09-15T03:48:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 >> 172.21.15.153:0/1152441717 conn(0x7ff53c067d30 msgr2=0x7ff53c0a3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 shutdown_connections 2024-09-15T03:48:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:00.275+0000 7ff542a60640 1 -- 172.21.15.153:0/1152441717 wait complete. 2024-09-15T03:48:00.301 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:48:00.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:48:20.302 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:48:20.302 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:48:20.302 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:48:20.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:48:20.306 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:48:20.306 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:48:20.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.410+0000 7f1a14ad4640 10 monclient: get_monmap_and_config 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.410+0000 7f1a14ad4640 10 monclient: build_initial_monmap 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.410+0000 7f1a14ad4640 10 monclient: monmap: 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:20.411924+0000 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:20.411924+0000 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:20.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.410+0000 7f1a14ad4640 1 Processor -- start 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.410+0000 7f1a14ad4640 1 -- start start 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.410+0000 7f1a14ad4640 10 monclient: init 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a10131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1a1009e910 0x7f1a1009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): picked mon.noname-c con 0x7f1a1009e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): picked mon.noname-b con 0x7f1a1009f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1009faf0 con 0x7f1a10131ab0 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a10132590 con 0x7f1a1009e910 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1009fd30 con 0x7f1a1009f250 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a14ad4640 10 monclient(hunting): authenticate will time out at 1898.051520s 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47858/0 (socket says 172.21.15.153:47858) 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 -- 172.21.15.153:0/1556245149 learned_addr learned my addr 172.21.15.153:0/1556245149 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 10 monclient(hunting): get_auth_request con 0x7f1a1009f250 auth_method 0 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 -- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 msgr2=0x7f1a10138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:48:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 -- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 msgr2=0x7f1a10138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/1556245149 >> v2:172.21.15.181:3300/0 conn(0x7f1a1009e910 0x7f1a1009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0ed9d640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): get_auth_request con 0x7f1a1009e910 auth_method 0 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.411+0000 7f1a0dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 10 monclient(hunting): handle_auth_done global_id 204409 payload 274 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 msgr2=0x7f1a10138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/1556245149 >> v1:172.21.15.181:6789/0 conn(0x7f1a10131ab0 legacy=0x7f1a10131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 1 monclient: found mon.noname-c 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.412+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/1556245149 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100a1110 con 0x7f1a1009e910 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/1556245149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a1009e910 msgr2=0x7f1a1009ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a1009e910 0x7f1a1009ed10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009850 tx=0x7f1a040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebeda1b3cc735c86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 1 -- 172.21.15.153:0/1556245149 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0402ae80 con 0x7f1a1009e910 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: dump: 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:20.426 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: _finish_auth 0 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 1 -- 172.21.15.153:0/1556245149 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04032440 con 0x7f1a1009e910 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 5 monclient: authenticate success, global_id 204409 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:20.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 1 -- 172.21.15.153:0/1556245149 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040327c0 con 0x7f1a1009e910 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 4 set_mon_vals no callback set 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a0d59a640 10 monclient: dump: 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.428 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 10 monclient: shutdown 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 1 -- 172.21.15.153:0/1556245149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a1009e910 msgr2=0x7f1a1009ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.415+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a1009e910 0x7f1a1009ed10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009850 tx=0x7f1a040090f0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 -- 172.21.15.153:0/1556245149 shutdown_connections 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/1556245149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a1009e910 0x7f1a1009ed10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 -- 172.21.15.153:0/1556245149 >> 172.21.15.153:0/1556245149 conn(0x7f1a10067d30 msgr2=0x7f1a100a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 -- 172.21.15.153:0/1556245149 shutdown_connections 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 -- 172.21.15.153:0/1556245149 wait complete. 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 10 monclient: build_initial_monmap 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 10 monclient: monmap: 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:20.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 Processor -- start 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 1 -- start start 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.416+0000 7f1a14ad4640 10 monclient: init 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:48:20.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 0x7f1a10134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1009e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): picked mon.noname-c con 0x7f1a1009f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a10131ab0 0x7f1a10134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): picked mon.noname-b con 0x7f1a10131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a101378e0 con 0x7f1a1009e910 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1009bde0 con 0x7f1a10131ab0 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1009d730 con 0x7f1a1009f250 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a14ad4640 10 monclient(hunting): authenticate will time out at 1898.057354s 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47864/0 (socket says 172.21.15.153:47864) 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/3583182282 learned_addr learned my addr 172.21.15.153:0/3583182282 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 10 monclient(hunting): get_auth_request con 0x7f1a1009f250 auth_method 0 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 0x7f1a10134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): get_auth_request con 0x7f1a1009e910 auth_method 0 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 msgr2=0x7f1a10134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 msgr2=0x7f1a10134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:48:20.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a10131ab0 0x7f1a10134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): get_auth_request con 0x7f1a10131ab0 auth_method 0 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:20.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient(hunting): handle_auth_done global_id 214395 payload 824 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 msgr2=0x7f1a10134840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a10131ab0 msgr2=0x7f1a10134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a10131ab0 0x7f1a10134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 monclient: found mon.noname-a 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.417+0000 7f1a0e59c640 1 -- 172.21.15.153:0/3583182282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04009390 con 0x7f1a1009e910 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a0e59c640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 msgr2=0x7f1a10134300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a0e59c640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 0x7f1a10134300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000c4e0 tx=0x7f1a00007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000117a0 con 0x7f1a1009e910 2024-09-15T03:48:20.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: dump: 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:20.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a00011900 con 0x7f1a1009e910 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 5 monclient: authenticate success, global_id 214395 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 10 monclient: _renew_subs 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a10137660 con 0x7f1a1009e910 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00011ca0 con 0x7f1a1009e910 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f19fe7fc640 10 monclient: dump: 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:20.438 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a0cd99640 4 set_mon_vals no callback set 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 10 monclient: _renew_subs 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.420+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a101352c0 con 0x7f1a1009e910 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f1a14ad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f1a14ad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:20.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f1a14ad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10137ae0 con 0x7f1a1009e910 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1a00012630 con 0x7f1a1009e910 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f19fe7fc640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19f0038740 0x7f19f003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1a0004c310 con 0x7f1a1009e910 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19f0038740 0x7f19f003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.421+0000 7f1a0dd9b640 10 monclient: get_auth_request con 0x7f19f0038740 auth_method 0 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.424+0000 7f1a0dd9b640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19f0038740 0x7f19f003abf0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009330 tx=0x7f1a04032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:20.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.425+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1a0001c460 con 0x7f1a1009e910 2024-09-15T03:48:20.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.425+0000 7f19fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:20.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.425+0000 7f19fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:20.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.642+0000 7f1a14ad4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.642+0000 7f1a14ad4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.642+0000 7f1a14ad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.642+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1a10135540 con 0x7f1a1009e910 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.643+0000 7f19fe7fc640 1 -- 172.21.15.153:0/3583182282 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f1a00010750 con 0x7f1a1009e910 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.643+0000 7f19fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.643+0000 7f19fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:20.645 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":204,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19f0038740 msgr2=0x7f19f003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19f0038740 0x7f19f003abf0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009330 tx=0x7f1a04032000 comp rx=0 tx=0).stop 2024-09-15T03:48:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 10 monclient: shutdown 2024-09-15T03:48:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 msgr2=0x7f1a10134300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:20.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 0x7f1a10134300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000c4e0 tx=0x7f1a00007e10 comp rx=0 tx=0).stop 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 shutdown_connections 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f19f0038740 0x7f19f003abf0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a10131ab0 0x7f1a10134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a1009f250 0x7f1a10134840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 --2- 172.21.15.153:0/3583182282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a1009e910 0x7f1a10134300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 >> 172.21.15.153:0/3583182282 conn(0x7f1a10067d30 msgr2=0x7f1a10068b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 shutdown_connections 2024-09-15T03:48:20.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:20.645+0000 7f1a14ad4640 1 -- 172.21.15.153:0/3583182282 wait complete. 2024-09-15T03:48:23.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.761+0000 7f8ecc59d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.761+0000 7f8ecc59d640 10 monclient: build_initial_monmap 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.761+0000 7f8ecc59d640 10 monclient: monmap: 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:23.762298+0000 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:23.762298+0000 2024-09-15T03:48:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:23.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:23.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 Processor -- start 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 -- start start 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient: init 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ec406eb80 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec4068a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ec409caf0 0x7f8ec4138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec409caf0 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec406a0e0 con 0x7f8ec406eb80 2024-09-15T03:48:23.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec406f7f0 con 0x7f8ec409caf0 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec406a7a0 con 0x7f8ec4068a20 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.762+0000 7f8ecc59d640 10 monclient(hunting): authenticate will time out at 1901.401854s 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47868/0 (socket says 172.21.15.153:47868) 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 -- 172.21.15.153:0/2835104963 learned_addr learned my addr 172.21.15.153:0/2835104963 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 10 monclient(hunting): get_auth_request con 0x7f8ec4068a20 auth_method 0 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 -- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 msgr2=0x7f8ec409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:48:23.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 -- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 msgr2=0x7f8ec409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ec9b38640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 1 --2- 172.21.15.153:0/2835104963 >> v2:172.21.15.181:3300/0 conn(0x7f8ec409caf0 0x7f8ec4138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): get_auth_request con 0x7f8ec409caf0 auth_method 0 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient(hunting): handle_auth_done global_id 204415 payload 274 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 1 -- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 msgr2=0x7f8ec409c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 1 -- 172.21.15.153:0/2835104963 >> v1:172.21.15.181:6789/0 conn(0x7f8ec406eb80 legacy=0x7f8ec406ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:23.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient: _finish_hunting 0 2024-09-15T03:48:23.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 1 monclient: found mon.noname-c 2024-09-15T03:48:23.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:23.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:23.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.763+0000 7f8ecab3a640 1 -- 172.21.15.153:0/2835104963 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec4069e00 con 0x7f8ec409caf0 2024-09-15T03:48:23.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ecab3a640 1 -- 172.21.15.153:0/2835104963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec409caf0 msgr2=0x7f8ec4138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:23.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ecab3a640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec409caf0 0x7f8ec4138ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009e60 tx=0x7f8ebc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c77419a790b37dff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:23.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 1 -- 172.21.15.153:0/2835104963 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc007760 con 0x7f8ec409caf0 2024-09-15T03:48:23.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:23.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: dump: 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: _finish_auth 0 2024-09-15T03:48:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 1 -- 172.21.15.153:0/2835104963 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc02d050 con 0x7f8ec409caf0 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 1 -- 172.21.15.153:0/2835104963 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc007af0 con 0x7f8ec409caf0 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ec9337640 10 monclient: dump: 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:23.776 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ecc59d640 5 monclient: authenticate success, global_id 204415 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ecc59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.766+0000 7f8ecc59d640 4 set_mon_vals no callback set 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 10 monclient: shutdown 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 -- 172.21.15.153:0/2835104963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec409caf0 msgr2=0x7f8ec4138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec409caf0 0x7f8ec4138ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009e60 tx=0x7f8ebc009780 comp rx=0 tx=0).stop 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 -- 172.21.15.153:0/2835104963 shutdown_connections 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec409caf0 0x7f8ec4138ac0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/2835104963 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec4068a20 0x7f8ec409c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 -- 172.21.15.153:0/2835104963 >> 172.21.15.153:0/2835104963 conn(0x7f8ec4067cd0 msgr2=0x7f8ec40a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 -- 172.21.15.153:0/2835104963 shutdown_connections 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 1 -- 172.21.15.153:0/2835104963 wait complete. 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 10 monclient: build_initial_monmap 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.767+0000 7f8ecc59d640 10 monclient: monmap: 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 Processor -- start 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 -- start start 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient: init 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec4068a20 0x7f8ec411e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec4068a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 0x7f8ec411e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec406eb80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec409caf0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:23.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec4138b10 con 0x7f8ec406eb80 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec409baa0 con 0x7f8ec4068a20 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec4121fd0 con 0x7f8ec409caf0 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecc59d640 10 monclient(hunting): authenticate will time out at 1901.407709s 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 0x7f8ec411e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47880/0 (socket says 172.21.15.153:47880) 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 0x7f8ec411e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49394/0 (socket says 172.21.15.153:49394) 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 -- 172.21.15.153:0/671600867 learned_addr learned my addr 172.21.15.153:0/671600867 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 10 monclient(hunting): get_auth_request con 0x7f8ec409caf0 auth_method 0 2024-09-15T03:48:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): get_auth_request con 0x7f8ec406eb80 auth_method 0 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 msgr2=0x7f8ec411ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 msgr2=0x7f8ec411ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ec9b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:23.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.768+0000 7f8ecab3a640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8eca339640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec4068a20 0x7f8ec411e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8eca339640 10 monclient(hunting): get_auth_request con 0x7f8ec4068a20 auth_method 0 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8eca339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8eca339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8eca339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 10 monclient(hunting): handle_auth_done global_id 214398 payload 824 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 msgr2=0x7f8ec411ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec4068a20 msgr2=0x7f8ec411e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec4068a20 0x7f8ec411e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 10 monclient: _finish_hunting 0 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 1 monclient: found mon.noname-a 2024-09-15T03:48:23.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.769+0000 7f8ec9b38640 1 -- 172.21.15.153:0/671600867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc009a20 con 0x7f8ec406eb80 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ec9b38640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 msgr2=0x7f8ec411e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ec9b38640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 0x7f8ec411e900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4006fd0 tx=0x7f8eb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400c6a0 con 0x7f8ec406eb80 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: dump: 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400c800 con 0x7f8ec406eb80 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 5 monclient: authenticate success, global_id 214398 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 10 monclient: _renew_subs 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec4121cf0 con 0x7f8ec406eb80 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:23.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400cbc0 con 0x7f8ec406eb80 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8eba7fc640 10 monclient: dump: 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:23.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 10 monclient: _renew_subs 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ec8b36640 4 set_mon_vals no callback set 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.771+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec4121ac0 con 0x7f8ec406eb80 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8ecc59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8ecc59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8ecc59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8eb4013410 con 0x7f8ec406eb80 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec404bf60 con 0x7f8ec406eb80 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8eba7fc640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ea8038770 0x7f8ea803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.772+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f8eb404c820 con 0x7f8ec406eb80 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.773+0000 7f8eca339640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ea8038770 0x7f8ea803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:23.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.773+0000 7f8eca339640 10 monclient: get_auth_request con 0x7f8ea8038770 auth_method 0 2024-09-15T03:48:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.775+0000 7f8eca339640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ea8038770 0x7f8ea803ac20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00037d0 tx=0x7f8ec0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.777+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8eb401c400 con 0x7f8ec406eb80 2024-09-15T03:48:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.777+0000 7f8eba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:23.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.777+0000 7f8eba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:23.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.995+0000 7f8ecc59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:23.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.995+0000 7f8ecc59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:23.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.995+0000 7f8ecc59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:23.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.995+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ec4122110 con 0x7f8ec406eb80 2024-09-15T03:48:23.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.996+0000 7f8eba7fc640 1 -- 172.21.15.153:0/671600867 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f8ec4122110 con 0x7f8ec406eb80 2024-09-15T03:48:23.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.996+0000 7f8eba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:23.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.996+0000 7f8eba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:23.997 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":204,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.997+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ea8038770 msgr2=0x7f8ea803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ea8038770 0x7f8ea803ac20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00037d0 tx=0x7f8ec0009040 comp rx=0 tx=0).stop 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 10 monclient: shutdown 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 msgr2=0x7f8ec411e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 0x7f8ec411e900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4006fd0 tx=0x7f8eb4008040 comp rx=0 tx=0).stop 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 shutdown_connections 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8ea8038770 0x7f8ea803ac20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ec409caf0 0x7f8ec411ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec406eb80 0x7f8ec411e900 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 --2- 172.21.15.153:0/671600867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ec4068a20 0x7f8ec411e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:23.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 >> 172.21.15.153:0/671600867 conn(0x7f8ec4067cd0 msgr2=0x7f8ec40a1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:24.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 shutdown_connections 2024-09-15T03:48:24.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:23.998+0000 7f8ecc59d640 1 -- 172.21.15.153:0/671600867 wait complete. 2024-09-15T03:48:27.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:27.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.113+0000 7fdc25099640 10 monclient: get_monmap_and_config 2024-09-15T03:48:27.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.113+0000 7fdc25099640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.113+0000 7fdc25099640 10 monclient: monmap: 2024-09-15T03:48:27.114 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.114 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:27.114586+0000 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:27.114586+0000 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:27.115 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 1 Processor -- start 2024-09-15T03:48:27.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 1 -- start start 2024-09-15T03:48:27.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 10 monclient: init 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc2009ad90 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc2009b160 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdc2009b530 0x7fdc2013a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.114+0000 7fdc25099640 10 monclient(hunting): picked mon.noname-a con 0x7fdc2009b530 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc25099640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc200692c0 con 0x7fdc2009ad90 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc25099640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2009b990 con 0x7fdc2009b160 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc25099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc25099640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2009cce0 con 0x7fdc2009b530 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc25099640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc25099640 10 monclient(hunting): authenticate will time out at 1904.754029s 2024-09-15T03:48:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdc2009b530 0x7fdc2013a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdc2009b530 0x7fdc2013a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52228/0 (socket says 172.21.15.153:52228) 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdc2009ad90 0x7fdc2009c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34494/0 (socket says 172.21.15.153:34494) 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 -- 172.21.15.153:0/873585251 learned_addr learned my addr 172.21.15.153:0/873585251 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): get_auth_request con 0x7fdc2009b530 auth_method 0 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_done global_id 214401 payload 274 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 -- 172.21.15.153:0/873585251 >> v1:172.21.15.181:6789/0 conn(0x7fdc2009b160 legacy=0x7fdc20136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 -- 172.21.15.153:0/873585251 >> v1:172.21.15.153:6789/0 conn(0x7fdc2009ad90 legacy=0x7fdc2009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.115+0000 7fdc1f59e640 1 -- 172.21.15.153:0/873585251 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2009db70 con 0x7fdc2009b530 2024-09-15T03:48:27.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1f59e640 1 -- 172.21.15.153:0/873585251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 msgr2=0x7fdc2013a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1f59e640 1 --2- 172.21.15.153:0/873585251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2013a3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009910 tx=0x7fdc180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162d27c104659930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 1 -- 172.21.15.153:0/873585251 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18032440 con 0x7fdc2009b530 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: dump: 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 1 -- 172.21.15.153:0/873585251 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc180325a0 con 0x7fdc2009b530 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc25099640 5 monclient: authenticate success, global_id 214401 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc25099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc25099640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 1 -- 172.21.15.153:0/873585251 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18032940 con 0x7fdc2009b530 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc25099640 4 set_mon_vals no callback set 2024-09-15T03:48:27.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.118+0000 7fdc1dd9b640 10 monclient: dump: 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 10 monclient: shutdown 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 -- 172.21.15.153:0/873585251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 msgr2=0x7fdc2013a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 --2- 172.21.15.153:0/873585251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2013a3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009910 tx=0x7fdc180091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 -- 172.21.15.153:0/873585251 shutdown_connections 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 --2- 172.21.15.153:0/873585251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2013a3c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 -- 172.21.15.153:0/873585251 >> 172.21.15.153:0/873585251 conn(0x7fdc200678a0 msgr2=0x7fdc200a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 -- 172.21.15.153:0/873585251 shutdown_connections 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 1 -- 172.21.15.153:0/873585251 wait complete. 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.119+0000 7fdc25099640 10 monclient: monmap: 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.129 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 Processor -- start 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 -- start start 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient: init 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:48:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc2009ad90 0x7fdc20133930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): picked mon.noname-c con 0x7fdc2009ad90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc2009b160 0x7fdc20133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): picked mon.noname-b con 0x7fdc2009b160 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2008c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): picked mon.noname-a con 0x7fdc2009b530 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2006f020 con 0x7fdc2009b530 2024-09-15T03:48:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201349a0 con 0x7fdc2009b160 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2008f720 con 0x7fdc2009ad90 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc25099640 10 monclient(hunting): authenticate will time out at 1904.759582s 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2008c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc2009ad90 0x7fdc20133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc2009ad90 0x7fdc20133930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59512/0 (socket says 172.21.15.153:59512) 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 1 -- 172.21.15.153:0/3249254514 learned_addr learned my addr 172.21.15.153:0/3249254514 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): get_auth_request con 0x7fdc2009b530 auth_method 0 2024-09-15T03:48:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): get_auth_request con 0x7fdc2009ad90 auth_method 0 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1e59c640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc2009b160 0x7fdc20133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient(hunting): handle_auth_done global_id 214404 payload 824 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 1 -- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc2009ad90 msgr2=0x7fdc20133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc2009ad90 0x7fdc20133930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 1 -- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc2009b160 msgr2=0x7fdc20133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc2009b160 0x7fdc20133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.120+0000 7fdc1f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.121+0000 7fdc1f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.121+0000 7fdc1f59e640 1 -- 172.21.15.153:0/3249254514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18009450 con 0x7fdc2009b530 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.121+0000 7fdc1e59c640 10 monclient: get_auth_request con 0x7fdc2009b160 auth_method 0 2024-09-15T03:48:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.121+0000 7fdc1e59c640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc2009b160 0x7fdc20133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc1f59e640 1 -- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 msgr2=0x7fdc2008c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc1f59e640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2008c2d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdc18031000 tx=0x7fdc18032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18031b10 con 0x7fdc2009b530 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: dump: 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc18031c70 con 0x7fdc2009b530 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1803c470 con 0x7fdc2009b530 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 5 monclient: authenticate success, global_id 214404 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 10 monclient: _renew_subs 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2008dbf0 con 0x7fdc2009b530 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc1d59a640 4 set_mon_vals no callback set 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc06ffd640 10 monclient: dump: 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 10 monclient: _renew_subs 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.123+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2008e7c0 con 0x7fdc2009b530 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdc18039040 con 0x7fdc2009b530 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc06ffd640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc0c038790 0x7fdc0c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc25099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc25099640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc25099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdc1806cdb0 con 0x7fdc2009b530 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2008fbc0 con 0x7fdc2009b530 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc1ed9d640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc0c038790 0x7fdc0c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.124+0000 7fdc1ed9d640 10 monclient: get_auth_request con 0x7fdc0c038790 auth_method 0 2024-09-15T03:48:27.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.127+0000 7fdc1ed9d640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc0c038790 0x7fdc0c03ac40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fdc10007fd0 tx=0x7fdc10004f70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.129+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fdc18040020 con 0x7fdc2009b530 2024-09-15T03:48:27.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.129+0000 7fdc06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.129+0000 7fdc06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc25099640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc25099640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc25099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc2008fbc0 con 0x7fdc2009b530 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc06ffd640 1 -- 172.21.15.153:0/3249254514 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fdc18049070 con 0x7fdc2009b530 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.343+0000 7fdc06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:27.344 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc0c038790 msgr2=0x7fdc0c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc0c038790 0x7fdc0c03ac40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fdc10007fd0 tx=0x7fdc10004f70 comp rx=0 tx=0).stop 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 10 monclient: shutdown 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 msgr2=0x7fdc2008c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2008c2d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdc18031000 tx=0x7fdc18032000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 shutdown_connections 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdc0c038790 0x7fdc0c03ac40 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc2009b530 0x7fdc2008c2d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc2009b160 0x7fdc20133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 --2- 172.21.15.153:0/3249254514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc2009ad90 0x7fdc20133930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 >> 172.21.15.153:0/3249254514 conn(0x7fdc200678a0 msgr2=0x7fdc20134bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 shutdown_connections 2024-09-15T03:48:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.345+0000 7fdc25099640 1 -- 172.21.15.153:0/3249254514 wait complete. 2024-09-15T03:48:27.356 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:48:27.356 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:48:27.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.450+0000 7f5b83754640 10 monclient: get_monmap_and_config 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.450+0000 7f5b83754640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.450+0000 7f5b83754640 10 monclient: monmap: 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:27.451256+0000 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:27.451256+0000 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.451 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:27.452 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:27.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 Processor -- start 2024-09-15T03:48:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 -- start start 2024-09-15T03:48:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient: init 2024-09-15T03:48:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5b7c0981c0 0x7f5b7c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): picked mon.noname-a con 0x7f5b7c0981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b7c131da0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7c132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c098ad0 con 0x7f5b7c131da0 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c131bd0 con 0x7f5b7c0981c0 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c09ca60 con 0x7f5b7c132170 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b83754640 10 monclient(hunting): authenticate will time out at 1905.090801s 2024-09-15T03:48:27.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b81cf1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b814f0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5b7c0981c0 0x7f5b7c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b80cef640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5b7c131da0 0x7f5b7c067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34510/0 (socket says 172.21.15.153:34510) 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b814f0640 10 monclient(hunting): get_auth_request con 0x7f5b7c0981c0 auth_method 0 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b80cef640 1 -- 172.21.15.153:0/1420104962 learned_addr learned my addr 172.21.15.153:0/1420104962 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b814f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b814f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b814f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b81cf1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59524/0 (socket says 172.21.15.153:59524) 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b81cf1640 10 monclient(hunting): get_auth_request con 0x7f5b7c132170 auth_method 0 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b81cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b81cf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.451+0000 7f5b81cf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b81cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b81cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b81cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 10 monclient(hunting): handle_auth_done global_id 214407 payload 274 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 1 -- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 msgr2=0x7f5b7c067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 1 --2- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 1 -- 172.21.15.153:0/1420104962 >> v1:172.21.15.153:6789/0 conn(0x7f5b7c131da0 legacy=0x7f5b7c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b814f0640 1 -- 172.21.15.153:0/1420104962 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c09cca0 con 0x7f5b7c0981c0 2024-09-15T03:48:27.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.452+0000 7f5b81cf1640 1 --2- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b814f0640 1 -- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c0981c0 msgr2=0x7f5b7c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b814f0640 1 --2- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c098590 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c066560 tx=0x7f5b780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38812c9a6f2d005 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 1 -- 172.21.15.153:0/1420104962 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7802adb0 con 0x7f5b7c0981c0 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: dump: 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 1 -- 172.21.15.153:0/1420104962 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78032440 con 0x7f5b7c0981c0 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 5 monclient: authenticate success, global_id 214407 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 4 set_mon_vals no callback set 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 1 -- 172.21.15.153:0/1420104962 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78032820 con 0x7f5b7c0981c0 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b73fff640 10 monclient: dump: 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 10 monclient: shutdown 2024-09-15T03:48:27.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 -- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c0981c0 msgr2=0x7f5b7c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 --2- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c098590 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c066560 tx=0x7f5b780090f0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 -- 172.21.15.153:0/1420104962 shutdown_connections 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 --2- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 --2- 172.21.15.153:0/1420104962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c098590 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 -- 172.21.15.153:0/1420104962 >> 172.21.15.153:0/1420104962 conn(0x7f5b7c0a1e30 msgr2=0x7f5b7c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 -- 172.21.15.153:0/1420104962 shutdown_connections 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.455+0000 7f5b83754640 1 -- 172.21.15.153:0/1420104962 wait complete. 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.456+0000 7f5b83754640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.456+0000 7f5b83754640 10 monclient: monmap: 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:27.469 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.456+0000 7f5b83754640 1 Processor -- start 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.456+0000 7f5b83754640 1 -- start start 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.456+0000 7f5b83754640 10 monclient: init 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c1410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7c0981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 0x7f5b7c1309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): picked mon.noname-a con 0x7f5b7c131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c11a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): picked mon.noname-c con 0x7f5b7c132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c11e0d0 con 0x7f5b7c131da0 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0a0570 con 0x7f5b7c0981c0 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c11db60 con 0x7f5b7c132170 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b83754640 10 monclient(hunting): authenticate will time out at 1905.096166s 2024-09-15T03:48:27.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 0x7f5b7c1309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c11a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c11a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59530/0 (socket says 172.21.15.153:59530) 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 0x7f5b7c1309c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52248/0 (socket says 172.21.15.153:52248) 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 -- 172.21.15.153:0/2209106960 learned_addr learned my addr 172.21.15.153:0/2209106960 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): get_auth_request con 0x7f5b7c132170 auth_method 0 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): get_auth_request con 0x7f5b7c131da0 auth_method 0 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 msgr2=0x7f5b7c11a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 msgr2=0x7f5b7c11a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c11a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b81cf1640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c11a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b814f0640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c1410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient(hunting): handle_auth_done global_id 214410 payload 824 2024-09-15T03:48:27.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 msgr2=0x7f5b7c11a570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b7c132170 0x7f5b7c11a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b7c0981c0 msgr2=0x7f5b7c1410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c1410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b80cef640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74009a20 con 0x7f5b7c131da0 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b814f0640 10 monclient: get_auth_request con 0x7f5b7c0981c0 auth_method 0 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.457+0000 7f5b814f0640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b7c0981c0 0x7f5b7c1410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b80cef640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 msgr2=0x7f5b7c1309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b80cef640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 0x7f5b7c1309c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c490 tx=0x7f5b6c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c005cb0 con 0x7f5b7c131da0 2024-09-15T03:48:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 10 monclient: dump: 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c005e30 con 0x7f5b7c131da0 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b83754640 5 monclient: authenticate success, global_id 214410 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b83754640 10 monclient: _renew_subs 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b83754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b78009390 con 0x7f5b7c131da0 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.459+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c007320 con 0x7f5b7c131da0 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b717fa640 10 monclient: dump: 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 10 monclient: _renew_subs 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c141ea0 con 0x7f5b7c131da0 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b737fe640 4 set_mon_vals no callback set 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 0x7f5b7c11aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c065f90 con 0x7f5b7c130f00 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5b6c01b3b0 con 0x7f5b7c131da0 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b814f0640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 0x7f5b7c11aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b814f0640 10 monclient: get_auth_request con 0x7f5b7c130f00 auth_method 0 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b814f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b814f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.460+0000 7f5b814f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b717fa640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b60038740 0x7f5b6003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b814f0640 10 monclient(hunting): handle_auth_done global_id 214413 payload 824 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5b6c04baf0 con 0x7f5b7c131da0 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b814f0640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b7c11e210 con 0x7f5b7c130f00 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b81cf1640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b60038740 0x7f5b6003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.461+0000 7f5b81cf1640 10 monclient: get_auth_request con 0x7f5b60038740 auth_method 0 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b814f0640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 msgr2=0x7f5b7c11aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b814f0640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 0x7f5b7c11aab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c132870 tx=0x7f5b78032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78031c50 con 0x7f5b7c130f00 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 10 monclient: dump: 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5b78031db0 con 0x7f5b7c130f00 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 msgr2=0x7f5b7c11aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 0x7f5b7c11aab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c132870 tx=0x7f5b78032000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 msgr2=0x7f5b7c11aab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:27.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.463+0000 7f5b717fa640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c130f00 0x7f5b7c11aab0 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c132870 tx=0x7f5b78032000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.464+0000 7f5b81cf1640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b60038740 0x7f5b6003abf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009f10 tx=0x7f5b7400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b83754640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b83754640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 0x7f5b7c099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b83754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c1312d0 con 0x7f5b7c11aff0 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 0x7f5b7c099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient: get_auth_request con 0x7f5b7c11aff0 auth_method 0 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 10 monclient(hunting): handle_auth_done global_id 214416 payload 824 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.469+0000 7f5b814f0640 1 -- 172.21.15.153:0/2209106960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5b7c11b3f0 con 0x7f5b7c11aff0 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b814f0640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 msgr2=0x7f5b7c099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b814f0640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 0x7f5b7c099310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c068700 tx=0x7f5b7803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7804a070 con 0x7f5b7c11aff0 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 10 monclient: dump: 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.484 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f5b78031a40 con 0x7f5b7c11aff0 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 msgr2=0x7f5b7c099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 0x7f5b7c099310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c068700 tx=0x7f5b7803c000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 msgr2=0x7f5b7c099310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b717fa640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c11aff0 0x7f5b7c099310 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:27.485 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 206, 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.487 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:27.488 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:27.489 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.490 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.491 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.492 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.493 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.494 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.495 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.496 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.497 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:48:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.498 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:27.498 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b60038740 msgr2=0x7f5b6003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5b60038740 0x7f5b6003abf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009f10 tx=0x7f5b7400b040 comp rx=0 tx=0).stop 2024-09-15T03:48:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 10 monclient: shutdown 2024-09-15T03:48:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 msgr2=0x7f5b7c1309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 --2- 172.21.15.153:0/2209106960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b7c131da0 0x7f5b7c1309c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c490 tx=0x7f5b6c007f00 comp rx=0 tx=0).stop 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b81cf1640 1 -- 172.21.15.153:0/2209106960 reap_dead start 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b81cf1640 1 -- 172.21.15.153:0/2209106960 reap_dead start 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 shutdown_connections 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 >> 172.21.15.153:0/2209106960 conn(0x7f5b7c0a1e30 msgr2=0x7f5b7c0a0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 shutdown_connections 2024-09-15T03:48:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.472+0000 7f5b83754640 1 -- 172.21.15.153:0/2209106960 wait complete. 2024-09-15T03:48:27.500 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:48:27.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:48:27.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.594+0000 7fe05f84f640 10 monclient: get_monmap_and_config 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.594+0000 7fe05f84f640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.594+0000 7fe05f84f640 10 monclient: monmap: 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:27.595156+0000 2024-09-15T03:48:27.595 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:27.595156+0000 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.594+0000 7fe05f84f640 1 Processor -- start 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 1 -- start start 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient: init 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0580975c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe05812ea30 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fe05812ee50 0x7fe05813a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient(hunting): picked mon.noname-c con 0x7fe05812ee50 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05806e9e0 con 0x7fe05812ea30 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05809d1c0 con 0x7fe0580975c0 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe058063950 con 0x7fe05812ee50 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05f84f640 10 monclient(hunting): authenticate will time out at 1905.234720s 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05cdea640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe05812ea30 0x7fe058136c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34524/0 (socket says 172.21.15.153:34524) 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.595+0000 7fe05cdea640 1 -- 172.21.15.153:0/992787714 learned_addr learned my addr 172.21.15.153:0/992787714 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 1 --2- 172.21.15.153:0/992787714 >> v2:172.21.15.181:3300/0 conn(0x7fe05812ee50 0x7fe05813a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): get_auth_request con 0x7fe05812ee50 auth_method 0 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient(hunting): handle_auth_done global_id 204421 payload 274 2024-09-15T03:48:27.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 1 -- 172.21.15.153:0/992787714 >> v1:172.21.15.181:6789/0 conn(0x7fe0580975c0 legacy=0x7fe058097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 1 -- 172.21.15.153:0/992787714 >> v1:172.21.15.153:6789/0 conn(0x7fe05812ea30 legacy=0x7fe058136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 1 monclient: found mon.noname-c 2024-09-15T03:48:27.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:27.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.596+0000 7fe05ddec640 1 -- 172.21.15.153:0/992787714 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05809d4a0 con 0x7fe05812ee50 2024-09-15T03:48:27.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05ddec640 1 -- 172.21.15.153:0/992787714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 msgr2=0x7fe05813a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05ddec640 1 --2- 172.21.15.153:0/992787714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 0x7fe05813a3b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe050009e60 tx=0x7fe050009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a7993d81dce4bf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 1 -- 172.21.15.153:0/992787714 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050007630 con 0x7fe05812ee50 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: dump: 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 1 -- 172.21.15.153:0/992787714 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05002d050 con 0x7fe05812ee50 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 5 monclient: authenticate success, global_id 204421 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 1 -- 172.21.15.153:0/992787714 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500079c0 con 0x7fe05812ee50 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 4 set_mon_vals no callback set 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe04ffff640 10 monclient: dump: 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 10 monclient: shutdown 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 1 -- 172.21.15.153:0/992787714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 msgr2=0x7fe05813a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 1 --2- 172.21.15.153:0/992787714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 0x7fe05813a3b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe050009e60 tx=0x7fe050009780 comp rx=0 tx=0).stop 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 1 -- 172.21.15.153:0/992787714 shutdown_connections 2024-09-15T03:48:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.599+0000 7fe05f84f640 1 --2- 172.21.15.153:0/992787714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 0x7fe05813a3b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 1 -- 172.21.15.153:0/992787714 >> 172.21.15.153:0/992787714 conn(0x7fe058067b50 msgr2=0x7fe0580a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 1 -- 172.21.15.153:0/992787714 shutdown_connections 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 1 -- 172.21.15.153:0/992787714 wait complete. 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 10 monclient: monmap: 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:27.610 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 1 Processor -- start 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 1 -- start start 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.600+0000 7fe05f84f640 10 monclient: init 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0580975c0 0x7fe05813a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0580975c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 0x7fe058128b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): picked mon.noname-c con 0x7fe05812ea30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 0x7fe058129080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): picked mon.noname-b con 0x7fe05812ee50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05812f9a0 con 0x7fe0580975c0 2024-09-15T03:48:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0581300a0 con 0x7fe05812ee50 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05809ceb0 con 0x7fe05812ea30 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05f84f640 10 monclient(hunting): authenticate will time out at 1905.240476s 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 0x7fe058128b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0580975c0 0x7fe05813a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 0x7fe058128b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59534/0 (socket says 172.21.15.153:59534) 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0580975c0 0x7fe05813a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52270/0 (socket says 172.21.15.153:52270) 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 -- 172.21.15.153:0/2092955969 learned_addr learned my addr 172.21.15.153:0/2092955969 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): get_auth_request con 0x7fe05812ea30 auth_method 0 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): get_auth_request con 0x7fe0580975c0 auth_method 0 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 0x7fe058129080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): get_auth_request con 0x7fe05812ee50 auth_method 0 2024-09-15T03:48:27.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient(hunting): handle_auth_done global_id 244109 payload 824 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 msgr2=0x7fe058129080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe05812ee50 0x7fe058129080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0580975c0 msgr2=0x7fe05813a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0580975c0 0x7fe05813a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 monclient: found mon.noname-c 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05cdea640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050009a20 con 0x7fe05812ea30 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.601+0000 7fe05d5eb640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0580975c0 0x7fe05813a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05cdea640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 msgr2=0x7fe058128b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05cdea640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 0x7fe058128b40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe05400c5e0 tx=0x7fe054004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe054005cb0 con 0x7fe05812ea30 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: dump: 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.615 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe054005e30 con 0x7fe05812ea30 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 5 monclient: authenticate success, global_id 244109 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe054007340 con 0x7fe05812ea30 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 10 monclient: _renew_subs 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0581302b0 con 0x7fe05812ea30 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04d7fa640 10 monclient: dump: 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe04f7fe640 4 set_mon_vals no callback set 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 10 monclient: _renew_subs 2024-09-15T03:48:27.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.604+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05812f6c0 con 0x7fe05812ea30 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fe05401b400 con 0x7fe05812ea30 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe04d7fa640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe03c038770 0x7fe03c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05f84f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe05404c180 con 0x7fe05812ea30 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05f84f640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 0x7fe0581295c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05f84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe058063720 con 0x7fe05812c190 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 0x7fe0581295c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient: get_auth_request con 0x7fe05812c190 auth_method 0 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05d5eb640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe03c038770 0x7fe03c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05d5eb640 10 monclient: get_auth_request con 0x7fe03c038770 auth_method 0 2024-09-15T03:48:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 10 monclient(hunting): handle_auth_done global_id 244112 payload 824 2024-09-15T03:48:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.605+0000 7fe05ddec640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe05812a740 con 0x7fe05812c190 2024-09-15T03:48:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.607+0000 7fe05ddec640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 msgr2=0x7fe0581295c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.607+0000 7fe05ddec640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 0x7fe0581295c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe050034000 tx=0x7fe050037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050041070 con 0x7fe05812c190 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 10 monclient: dump: 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe05d5eb640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe03c038770 0x7fe03c03ac20 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe048007fd0 tx=0x7fe048007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe05003cc50 con 0x7fe05812c190 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 msgr2=0x7fe0581295c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 0x7fe0581295c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe050034000 tx=0x7fe050037000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 msgr2=0x7fe0581295c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.608+0000 7fe04d7fa640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812c190 0x7fe0581295c0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe050034000 tx=0x7fe050037000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05f84f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05f84f640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 0x7fe05809a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05f84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05812fcb0 con 0x7fe05809a570 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 0x7fe05809a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient: get_auth_request con 0x7fe05809a570 auth_method 0 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 10 monclient(hunting): handle_auth_done global_id 244115 payload 824 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.614+0000 7fe05ddec640 1 -- 172.21.15.153:0/2092955969 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe05804bf60 con 0x7fe05809a570 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe05ddec640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 msgr2=0x7fe05809a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe05ddec640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 0x7fe05809a970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0500097c0 tx=0x7fe050041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500459e0 con 0x7fe05809a570 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 10 monclient: dump: 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fe05003b410 con 0x7fe05809a570 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 msgr2=0x7fe05809a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 0x7fe05809a970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0500097c0 tx=0x7fe050041000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 msgr2=0x7fe05809a970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.616+0000 7fe04d7fa640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05809a570 0x7fe05809a970 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0500097c0 tx=0x7fe050041000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.626 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 206, 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:27.627 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:27.628 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:27.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:27.632 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:27.633 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:27.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:27.638 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe03c038770 msgr2=0x7fe03c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fe03c038770 0x7fe03c03ac20 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe048007fd0 tx=0x7fe048007e10 comp rx=0 tx=0).stop 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 10 monclient: shutdown 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 msgr2=0x7fe058128b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 0x7fe058128b40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe05400c5e0 tx=0x7fe054004960 comp rx=0 tx=0).stop 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05ddec640 1 -- 172.21.15.153:0/2092955969 reap_dead start 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 shutdown_connections 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 --2- 172.21.15.153:0/2092955969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe05812ea30 0x7fe058128b40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 >> 172.21.15.153:0/2092955969 conn(0x7fe058067b50 msgr2=0x7fe0580a0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 shutdown_connections 2024-09-15T03:48:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.617+0000 7fe05f84f640 1 -- 172.21.15.153:0/2092955969 wait complete. 2024-09-15T03:48:27.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:48:27.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:48:27.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.734+0000 7f99962d4640 10 monclient: get_monmap_and_config 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.734+0000 7f99962d4640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.734+0000 7f99962d4640 10 monclient: monmap: 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:27.735350+0000 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:27.735350+0000 2024-09-15T03:48:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:27.736 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 1 Processor -- start 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 1 -- start start 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient: init 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f999009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient(hunting): picked mon.noname-b con 0x7f9990096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f999009aaa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99900978f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999009d4a0 con 0x7f999009aaa0 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990099620 con 0x7f99900978f0 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9990099340 con 0x7f9990096e10 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.735+0000 7f99962d4640 10 monclient(hunting): authenticate will time out at 1905.374887s 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f999009aaa0 0x7f99900971e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34538/0 (socket says 172.21.15.153:34538) 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998f7fe640 1 -- 172.21.15.153:0/2503070940 learned_addr learned my addr 172.21.15.153:0/2503070940 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f999009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 10 monclient(hunting): get_auth_request con 0x7f9990096e10 auth_method 0 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 msgr2=0x7f999009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 msgr2=0x7f999009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f999009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.736+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f999009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:27.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 1 -- 172.21.15.153:0/2503070940 >> v1:172.21.15.153:6789/0 conn(0x7f999009aaa0 legacy=0x7f99900971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116767603 0 0) 0x7f999009d4a0 con 0x7f999009aaa0 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): my global_id is 214422 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c0034e0 con 0x7f999009aaa0 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 1 -- 172.21.15.153:0/2503070940 >> v1:172.21.15.181:6789/0 conn(0x7f99900978f0 legacy=0x7f9990135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448427703 0 0) 0x7f997c0034e0 con 0x7f999009aaa0 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 msgr2=0x7f999009a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f999009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 >> v1:172.21.15.181:6789/0 conn(0x7f99900978f0 legacy=0x7f9990135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990099900 con 0x7f999009aaa0 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f99962d4640 5 monclient: authenticate success, global_id 214422 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f9980001720 con 0x7f999009aaa0 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f99962d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: dump: 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.753 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 >> v1:172.21.15.153:6789/0 conn(0x7f999009aaa0 legacy=0x7f99900971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:48:27.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f997c003b00 0x7f997c003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): picked mon.a con 0x7f997c003b00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f997c004780 0x7f997c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): picked mon.b con 0x7f997c004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f997c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): picked mon.c con 0x7f9990096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990099340 con 0x7f997c003b00 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c005570 con 0x7f997c004780 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c00bf10 con 0x7f9990096e10 2024-09-15T03:48:27.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f99962d4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f997c003b00 0x7f997c003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): get_auth_request con 0x7f997c003b00 auth_method 0 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f997c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): get_auth_request con 0x7f9990096e10 auth_method 0 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f997c004780 0x7f997c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): get_auth_request con 0x7f997c004780 auth_method 0 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f9994871640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:48:27.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 10 monclient(hunting): handle_auth_done global_id 214422 payload 293 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 msgr2=0x7f997c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9990096e10 0x7f997c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f997c004780 msgr2=0x7f997c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.738+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f997c004780 0x7f997c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 1 monclient: found mon.a 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 1 -- 172.21.15.153:0/2503070940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c0034e0 con 0x7f997c003b00 2024-09-15T03:48:27.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.739+0000 7f998ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998ffff640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f997c003b00 msgr2=0x7f997c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998ffff640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f997c003b00 0x7f997c003ed0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639268f8b40b2fb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997802add0 con 0x7f997c003b00 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: dump: 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.759 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997802e070 con 0x7f997c003b00 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f99962d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f99962d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 1 -- 172.21.15.153:0/2503070940 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978034660 con 0x7f997c003b00 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f99962d4640 4 set_mon_vals no callback set 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.741+0000 7f998effd640 10 monclient: dump: 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 10 monclient: shutdown 2024-09-15T03:48:27.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 1 -- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f997c003b00 msgr2=0x7f997c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 1 --2- 172.21.15.153:0/2503070940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f997c003b00 0x7f997c003ed0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978008040 comp rx=0 tx=0).stop 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f9994871640 1 -- 172.21.15.153:0/2503070940 reap_dead start 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 1 -- 172.21.15.153:0/2503070940 shutdown_connections 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 1 -- 172.21.15.153:0/2503070940 >> 172.21.15.153:0/2503070940 conn(0x7f99900672b0 msgr2=0x7f99900676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 1 -- 172.21.15.153:0/2503070940 shutdown_connections 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 1 -- 172.21.15.153:0/2503070940 wait complete. 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.742+0000 7f99962d4640 10 monclient: monmap: 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 Processor -- start 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 -- start start 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient: init 2024-09-15T03:48:27.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f997c003b00 0x7f9990140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): picked mon.noname-c con 0x7f997c003b00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9990120f80 0x7f999012ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): picked mon.noname-a con 0x7f9990120f80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011e7f0 0x7f999011ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): picked mon.noname-b con 0x7f999011e7f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990120470 con 0x7f9990120f80 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990120be0 con 0x7f999011e7f0 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99900a0090 con 0x7f997c003b00 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.743+0000 7f99962d4640 10 monclient(hunting): authenticate will time out at 1905.382992s 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f997c003b00 0x7f9990140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9990120f80 0x7f999012ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f997c003b00 0x7f9990140d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59572/0 (socket says 172.21.15.153:59572) 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 1 -- 172.21.15.153:0/1158935641 learned_addr learned my addr 172.21.15.153:0/1158935641 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): get_auth_request con 0x7f9990120f80 auth_method 0 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): get_auth_request con 0x7f997c003b00 auth_method 0 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f9994871640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011e7f0 0x7f999011ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient(hunting): handle_auth_done global_id 214425 payload 824 2024-09-15T03:48:27.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f997c003b00 msgr2=0x7f9990140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f997c003b00 0x7f9990140d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011e7f0 msgr2=0x7f999011ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011e7f0 0x7f999011ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998f7fe640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990121ee0 con 0x7f9990120f80 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f9994871640 10 monclient: get_auth_request con 0x7f999011e7f0 auth_method 0 2024-09-15T03:48:27.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f998ffff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f997c003b00 0x7f9990140d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.744+0000 7f9994871640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011e7f0 0x7f999011ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f998f7fe640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9990120f80 msgr2=0x7f999012ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f998f7fe640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9990120f80 0x7f999012ed60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9980001490 tx=0x7f9980007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980018030 con 0x7f9990120f80 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: dump: 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.768 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980001790 con 0x7f9990120f80 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 5 monclient: authenticate success, global_id 214425 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 10 monclient: _renew_subs 2024-09-15T03:48:27.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99901221c0 con 0x7f9990120f80 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998000fb10 con 0x7f9990120f80 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f9987fff640 10 monclient: dump: 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.770 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f998e7fc640 4 set_mon_vals no callback set 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 10 monclient: _renew_subs 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.746+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9990141a50 con 0x7f9990120f80 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f99962d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f99962d4640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 0x7f999006d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f99962d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990141c70 con 0x7f999011f1b0 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f998001b070 con 0x7f9990120f80 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f9987fff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9970038730 0x7f997003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9980019070 con 0x7f9990120f80 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.747+0000 7f998ffff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 0x7f999006d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient: get_auth_request con 0x7f999011f1b0 auth_method 0 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f9994871640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9970038730 0x7f997003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f9994871640 10 monclient: get_auth_request con 0x7f9970038730 auth_method 0 2024-09-15T03:48:27.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 10 monclient(hunting): handle_auth_done global_id 204430 payload 824 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.748+0000 7f998ffff640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9990122650 con 0x7f999011f1b0 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.750+0000 7f9994871640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9970038730 0x7f997003abe0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f999009b020 tx=0x7f9988009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.750+0000 7f998ffff640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 msgr2=0x7f999006d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.750+0000 7f998ffff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 0x7f999006d3a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9978009090 tx=0x7f9978035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997803f930 con 0x7f999011f1b0 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:27.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 10 monclient: dump: 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.774 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f997803fa90 con 0x7f999011f1b0 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 msgr2=0x7f999006d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 0x7f999006d3a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9978009090 tx=0x7f9978035000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 msgr2=0x7f999006d3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.751+0000 7f9987fff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999011f1b0 0x7f999006d3a0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9978009090 tx=0x7f9978035000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f99962d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:27.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f99962d4640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 0x7f999006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f99962d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990065f20 con 0x7f999006d8e0 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 0x7f999006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient: get_auth_request con 0x7f999006d8e0 auth_method 0 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 10 monclient(hunting): handle_auth_done global_id 204433 payload 824 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.757+0000 7f998ffff640 1 -- 172.21.15.153:0/1158935641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f999004bf60 con 0x7f999006d8e0 2024-09-15T03:48:27.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f998ffff640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 msgr2=0x7f999006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f998ffff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 0x7f999006dce0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f997803e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978047e70 con 0x7f999006d8e0 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 10 monclient: dump: 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.779 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f997804e070 con 0x7f999006d8e0 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 msgr2=0x7f999006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 0x7f999006dce0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f997803e000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 msgr2=0x7f999006dce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:27.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.760+0000 7f9987fff640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f999006d8e0 0x7f999006dce0 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f997803e000 comp rx=0 tx=0).stop 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 206, 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.781 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.782 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:27.784 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:27.785 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:27.786 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.787 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:27.790 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:27.791 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.792 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:27.793 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:27.794 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:27.794 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:27.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9970038730 msgr2=0x7f997003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9970038730 0x7f997003abe0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f999009b020 tx=0x7f9988009040 comp rx=0 tx=0).stop 2024-09-15T03:48:27.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 10 monclient: shutdown 2024-09-15T03:48:27.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9990120f80 msgr2=0x7f999012ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 --2- 172.21.15.153:0/1158935641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9990120f80 0x7f999012ed60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9980001490 tx=0x7f9980007a70 comp rx=0 tx=0).stop 2024-09-15T03:48:27.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f9994871640 1 -- 172.21.15.153:0/1158935641 reap_dead start 2024-09-15T03:48:27.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f9994871640 1 -- 172.21.15.153:0/1158935641 reap_dead start 2024-09-15T03:48:27.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 shutdown_connections 2024-09-15T03:48:27.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 >> 172.21.15.153:0/1158935641 conn(0x7f99900672b0 msgr2=0x7f999009ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 shutdown_connections 2024-09-15T03:48:27.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.761+0000 7f99962d4640 1 -- 172.21.15.153:0/1158935641 wait complete. 2024-09-15T03:48:27.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:48:27.796 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:48:27.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:48:27.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.889+0000 7f7ee1099640 10 monclient: get_monmap_and_config 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.889+0000 7f7ee1099640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.889+0000 7f7ee1099640 10 monclient: monmap: 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:27.889904+0000 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:27.889904+0000 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:27.891 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:27.891 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:27.891 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:27.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.889+0000 7f7ee1099640 1 Processor -- start 2024-09-15T03:48:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.889+0000 7f7ee1099640 1 -- start start 2024-09-15T03:48:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.889+0000 7f7ee1099640 10 monclient: init 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7edc0974f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7edc09a8a0 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7edc134080 0x7f7edc136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient(hunting): picked mon.noname-a con 0x7f7edc134080 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc12b210 con 0x7f7edc0974f0 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0978c0 con 0x7f7edc09a8a0 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7edc09ca20 con 0x7f7edc134080 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7ee1099640 10 monclient(hunting): authenticate will time out at 1905.529472s 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7edc134080 0x7f7edc136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7edc0974f0 0x7f7edc12ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34548/0 (socket says 172.21.15.153:34548) 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edad9d640 1 -- 172.21.15.153:0/1149755438 learned_addr learned my addr 172.21.15.153:0/1149755438 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): get_auth_request con 0x7f7edc134080 auth_method 0 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient(hunting): handle_auth_done global_id 214428 payload 274 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 1 -- 172.21.15.153:0/1149755438 >> v1:172.21.15.181:6789/0 conn(0x7f7edc09a8a0 legacy=0x7f7edc09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 1 -- 172.21.15.153:0/1149755438 >> v1:172.21.15.153:6789/0 conn(0x7f7edc0974f0 legacy=0x7f7edc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:27.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.890+0000 7f7edb59e640 1 -- 172.21.15.153:0/1149755438 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc098ad0 con 0x7f7edc134080 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7edb59e640 1 -- 172.21.15.153:0/1149755438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 msgr2=0x7f7edc136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7edb59e640 1 --2- 172.21.15.153:0/1149755438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc136460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400af70 tx=0x7f7ed400a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc6bf880576547b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 1 -- 172.21.15.153:0/1149755438 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4008660 con 0x7f7edc134080 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: dump: 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7ee1099640 5 monclient: authenticate success, global_id 214428 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7ee1099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 1 -- 172.21.15.153:0/1149755438 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed402e050 con 0x7f7edc134080 2024-09-15T03:48:27.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7ee1099640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7ee1099640 4 set_mon_vals no callback set 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 1 -- 172.21.15.153:0/1149755438 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed40089f0 con 0x7f7edc134080 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.893+0000 7f7eda59c640 10 monclient: dump: 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.902 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 10 monclient: shutdown 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 -- 172.21.15.153:0/1149755438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 msgr2=0x7f7edc136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 --2- 172.21.15.153:0/1149755438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc136460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400af70 tx=0x7f7ed400a890 comp rx=0 tx=0).stop 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 -- 172.21.15.153:0/1149755438 shutdown_connections 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 --2- 172.21.15.153:0/1149755438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc136460 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 -- 172.21.15.153:0/1149755438 >> 172.21.15.153:0/1149755438 conn(0x7f7edc0672b0 msgr2=0x7f7edc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 -- 172.21.15.153:0/1149755438 shutdown_connections 2024-09-15T03:48:27.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 1 -- 172.21.15.153:0/1149755438 wait complete. 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 10 monclient: build_initial_monmap 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.894+0000 7f7ee1099640 10 monclient: monmap: 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 Processor -- start 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 -- start start 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient: init 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 0x7f7edc06bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): picked mon.noname-c con 0x7f7edc0974f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7edc09a8a0 0x7f7edc06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): picked mon.noname-b con 0x7f7edc09a8a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc06c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): picked mon.noname-a con 0x7f7edc134080 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc06e580 con 0x7f7edc134080 2024-09-15T03:48:27.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc098300 con 0x7f7edc09a8a0 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc12a390 con 0x7f7edc0974f0 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7ee1099640 10 monclient(hunting): authenticate will time out at 1905.534830s 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 0x7f7edc06bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edb59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc06c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 0x7f7edc06bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59582/0 (socket says 172.21.15.153:59582) 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edb59e640 10 monclient(hunting): get_auth_request con 0x7f7edc134080 auth_method 0 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 1 -- 172.21.15.153:0/3954380085 learned_addr learned my addr 172.21.15.153:0/3954380085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 10 monclient(hunting): get_auth_request con 0x7f7edc0974f0 auth_method 0 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.895+0000 7f7edad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7ed3fff640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7edc09a8a0 0x7f7edc06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7ed3fff640 10 monclient(hunting): get_auth_request con 0x7f7edc09a8a0 auth_method 0 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7ed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 10 monclient(hunting): handle_auth_done global_id 214431 payload 824 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 1 -- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 msgr2=0x7f7edc06bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 0x7f7edc06bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 1 -- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7edc09a8a0 msgr2=0x7f7edc06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7edc09a8a0 0x7f7edc06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 10 monclient: _finish_hunting 0 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 1 monclient: found mon.noname-a 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:27.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edb59e640 1 -- 172.21.15.153:0/3954380085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed400ab30 con 0x7f7edc134080 2024-09-15T03:48:27.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.896+0000 7f7edad9d640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 0x7f7edc06bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:27.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.898+0000 7f7edb59e640 1 -- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 msgr2=0x7f7edc06c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:27.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.898+0000 7f7edb59e640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc06c830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4031000 tx=0x7f7ed4037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4032cb0 con 0x7f7edc134080 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: dump: 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.911 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ee1099640 5 monclient: authenticate success, global_id 214431 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed4032e10 con 0x7f7edc134080 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ee1099640 10 monclient: _renew_subs 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ee1099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc06d8c0 con 0x7f7edc134080 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed403c660 con 0x7f7edc134080 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed2ffd640 10 monclient: dump: 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:27.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 10 monclient: _renew_subs 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.900+0000 7f7ed9d9b640 4 set_mon_vals no callback set 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc133910 con 0x7f7edc134080 2024-09-15T03:48:27.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7ed4039070 con 0x7f7edc134080 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc04bf60 con 0x7f7edc134080 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.901+0000 7f7ed2ffd640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ebc038790 0x7f7ebc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.902+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7ed406d340 con 0x7f7edc134080 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.902+0000 7f7edad9d640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ebc038790 0x7f7ebc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.902+0000 7f7edad9d640 10 monclient: get_auth_request con 0x7f7ebc038790 auth_method 0 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.904+0000 7f7edad9d640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ebc038790 0x7f7ebc03ac40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc006fd0 tx=0x7f7ecc004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.906+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7ed404c550 con 0x7f7edc134080 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.906+0000 7f7ed2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:27.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:27.906+0000 7f7ed2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:28.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.120+0000 7f7ee1099640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:48:28.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.120+0000 7f7ee1099640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:48:28.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.120+0000 7f7ee1099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:28.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.120+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7edc0638d0 con 0x7f7edc134080 2024-09-15T03:48:28.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f7ed2ffd640 1 -- 172.21.15.153:0/3954380085 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7ed4040020 con 0x7f7edc134080 2024-09-15T03:48:28.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f7ed2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:48:28.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f7ed2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:28.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.122 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=4135475751,config=3140514009,health=317363223,kv=3235807718,logm=3830976795}) 2024-09-15T03:48:28.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=7,logm=55} crc {auth=3082670671,config=3140514009,health=1527198000,kv=1577269308,logm=1307046076}) 2024-09-15T03:48:28.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.121+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=4135475751,config=3140514009,health=317363223,kv=3235807718,logm=3830976795}) 2024-09-15T03:48:28.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.122+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=3311923517,config=3140514009,health=900616432,kv=3235807718,logm=2334513213}) 2024-09-15T03:48:28.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408771523}) 2024-09-15T03:48:28.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622943990}) 2024-09-15T03:48:28.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408771523}) 2024-09-15T03:48:28.124 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3481400899}) 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ebc038790 msgr2=0x7f7ebc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ebc038790 0x7f7ebc03ac40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc006fd0 tx=0x7f7ecc004960 comp rx=0 tx=0).stop 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 10 monclient: shutdown 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 msgr2=0x7f7edc06c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc06c830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4031000 tx=0x7f7ed4037000 comp rx=0 tx=0).stop 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 shutdown_connections 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7ebc038790 0x7f7ebc03ac40 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:28.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc134080 0x7f7edc06c830 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:28.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7edc09a8a0 0x7f7edc06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:28.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 --2- 172.21.15.153:0/3954380085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc0974f0 0x7f7edc06bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:28.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 >> 172.21.15.153:0/3954380085 conn(0x7f7edc0672b0 msgr2=0x7f7edc09b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:28.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 shutdown_connections 2024-09-15T03:48:28.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:28.123+0000 7f7ee1099640 1 -- 172.21.15.153:0/3954380085 wait complete. 2024-09-15T03:48:28.131 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.130+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.131 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.130+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1378875176}) 2024-09-15T03:48:28.131 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.130+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1525474691}) 2024-09-15T03:48:28.131 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.130+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.131 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.130+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1378875176}) 2024-09-15T03:48:28.131 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.130+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=268793775}) 2024-09-15T03:48:28.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.132+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.132+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67566989}) 2024-09-15T03:48:28.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.132+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012741530}) 2024-09-15T03:48:28.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.132+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.132+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67566989}) 2024-09-15T03:48:28.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.132+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485108011}) 2024-09-15T03:48:28.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.133+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.133+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385777040}) 2024-09-15T03:48:28.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.133+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543130596}) 2024-09-15T03:48:28.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.133+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.133+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385777040}) 2024-09-15T03:48:28.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.133+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108418724}) 2024-09-15T03:48:28.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:48:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.134+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.134+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070023122}) 2024-09-15T03:48:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.134+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333734311}) 2024-09-15T03:48:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.134+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.134+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070023122}) 2024-09-15T03:48:28.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.134+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1095966054}) 2024-09-15T03:48:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.135+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.135+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871730718}) 2024-09-15T03:48:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.135+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126494648}) 2024-09-15T03:48:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.135+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.135+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871730718}) 2024-09-15T03:48:28.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.135+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2160495538}) 2024-09-15T03:48:28.142 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.142+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.142+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662120417}) 2024-09-15T03:48:28.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.142+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479191722}) 2024-09-15T03:48:28.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.142+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.142+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662120417}) 2024-09-15T03:48:28.143 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.142+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2194505030}) 2024-09-15T03:48:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566664388}) 2024-09-15T03:48:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=709601739}) 2024-09-15T03:48:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566664388}) 2024-09-15T03:48:28.144 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173648496}) 2024-09-15T03:48:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464594717}) 2024-09-15T03:48:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1604696966}) 2024-09-15T03:48:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464594717}) 2024-09-15T03:48:28.145 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.143+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955017069}) 2024-09-15T03:48:28.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.144+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.144+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221370956}) 2024-09-15T03:48:28.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.144+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403628669}) 2024-09-15T03:48:28.147 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.144+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.144+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221370956}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.144+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080627404}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100670391}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2681784950}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100670391}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844453788}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643108284}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219477876}) 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.148 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643108284}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.145+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608744111}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722036822}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259526518}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722036822}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4253380703}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029579499}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050525317}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029579499}) 2024-09-15T03:48:28.149 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.146+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110897500}) 2024-09-15T03:48:28.154 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.147+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.154 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.147+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521919857}) 2024-09-15T03:48:28.154 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.147+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4290628333}) 2024-09-15T03:48:28.154 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.147+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.147+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521919857}) 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.147+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082281738}) 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006460630}) 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=717303646}) 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006460630}) 2024-09-15T03:48:28.155 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866086631}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481576827}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163043614}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481576827}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.148+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178480833}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217519166}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1576815304}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217519166}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103554704}) 2024-09-15T03:48:28.156 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121518789}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231996008}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121518789}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.149+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435760997}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.150+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.150+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119590359}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.150+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=538659159}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.150+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.150+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119590359}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.150+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321469409}) 2024-09-15T03:48:28.157 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3435269450}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3597909264}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3435269450}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764607554}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909185232}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2400978409}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909185232}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.151+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778949946}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721138260}) 2024-09-15T03:48:28.158 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452621246}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721138260}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850995767}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724061747}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.152+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055762410}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724061747}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447777959}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724301537}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649189893}) 2024-09-15T03:48:28.159 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.160 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724301537}) 2024-09-15T03:48:28.160 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.153+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2199551252}) 2024-09-15T03:48:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.154+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.154+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893192092}) 2024-09-15T03:48:28.168 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.154+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181228800}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.154+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.154+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893192092}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.154+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2436015820}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.155+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.155+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349002996}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.155+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951648243}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.155+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.155+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349002996}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.155+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143643390}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.156+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.156+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270906202}) 2024-09-15T03:48:28.169 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.156+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974599346}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.156+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.156+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270906202}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.156+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939162894}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.157+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.157+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67066792}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.157+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239368020}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.157+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.157+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67066792}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.157+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923049635}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.158+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.158+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2254720032}) 2024-09-15T03:48:28.170 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.158+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1999156194}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.158+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.158+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2254720032}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.158+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1368725501}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.159+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.159+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485710547}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.159+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846501878}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.159+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.159+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485710547}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.159+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384767079}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.160+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.160+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933026276}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.160+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053280130}) 2024-09-15T03:48:28.171 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.160+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.160+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933026276}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.160+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864878921}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869500393}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341964227}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869500393}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484759092}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.161+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917710212}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2156515933}) 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.172 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917710212}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170830653}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771087359}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3491018673}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.162+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771087359}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267231757}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284544812}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2355262011}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284544812}) 2024-09-15T03:48:28.173 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.163+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804490260}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.164+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.164+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241126675}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.164+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1507835469}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.164+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.164+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241126675}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.164+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732147340}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.165+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.165+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034320319}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.165+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118015442}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.165+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.165+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034320319}) 2024-09-15T03:48:28.174 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.165+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1139558136}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.166+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.166+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524978410}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.166+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4207736216}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.166+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.166+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524978410}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.166+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569372880}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.167+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.167+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038475114}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.167+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994297265}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.167+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.167+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038475114}) 2024-09-15T03:48:28.175 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.167+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212252522}) 2024-09-15T03:48:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.168+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.168+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962551438}) 2024-09-15T03:48:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.168+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047278362}) 2024-09-15T03:48:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.168+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.185 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.168+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962551438}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.168+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4281267788}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.169+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.169+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196146032}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.169+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2799768801}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.169+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.169+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196146032}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.169+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284882974}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.170+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.170+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823014589}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.170+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63442289}) 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.170+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.186 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.170+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823014589}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.170+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753780217}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693824324}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2249836860}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693824324}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.171+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890455973}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.172+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.172+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565773840}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.172+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491330407}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.172+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.172+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565773840}) 2024-09-15T03:48:28.187 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.172+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205313595}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.173+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.173+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782439476}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.173+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223647270}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.173+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.173+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782439476}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.173+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146713848}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.174+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.174+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2846944474}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.174+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500809624}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.174+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.174+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2846944474}) 2024-09-15T03:48:28.188 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.174+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=914637265}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.175+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.175+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246900942}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.175+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2019764646}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.175+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.175+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246900942}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.175+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815403670}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.176+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.176+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273232374}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.176+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545228185}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.176+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.176+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273232374}) 2024-09-15T03:48:28.189 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.176+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106204492}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280291272}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070067803}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280291272}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.177+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2507133434}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.178+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.178+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134173251}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.178+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1231314188}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.178+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.178+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134173251}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.178+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691321383}) 2024-09-15T03:48:28.190 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159379737}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=480313990}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159379737}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.179+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545027787}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.180+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.180+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682561906}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.180+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084693392}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.180+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.180+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682561906}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.180+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336634958}) 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.181+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.191 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.181+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2926633764}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.181+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2479045726}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.181+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.181+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2926633764}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.181+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267386673}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189888575}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242502301}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189888575}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=625745205}) 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.192 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000702685}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815208217}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.182+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000702685}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593797305}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4126836786}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2981949340}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4126836786}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.183+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61575713}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.184+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.184+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253127008}) 2024-09-15T03:48:28.193 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.184+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3368099123}) 2024-09-15T03:48:28.194 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.184+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.194 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.184+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253127008}) 2024-09-15T03:48:28.194 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.184+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3716573671}) 2024-09-15T03:48:28.205 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.185+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.205 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.185+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553992881}) 2024-09-15T03:48:28.205 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.185+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=87194497}) 2024-09-15T03:48:28.205 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.185+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.205 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.185+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553992881}) 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.185+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3125050140}) 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.186+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.186+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1789240743}) 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.186+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699103940}) 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.186+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.186+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1789240743}) 2024-09-15T03:48:28.206 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.186+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898370846}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.187+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.187+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440302136}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.187+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2384883856}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.187+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.187+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440302136}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.187+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320892281}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41909038}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=394610879}) 2024-09-15T03:48:28.208 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41909038}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.188+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3118053418}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.189+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.189+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973166167}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.189+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2629866242}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.189+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.189+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973166167}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.189+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155534465}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.190+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.190+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806690617}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.190+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4146423746}) 2024-09-15T03:48:28.209 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.190+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.190+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806690617}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.190+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625687540}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.191+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.191+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981416494}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.191+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789075577}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.191+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.191+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981416494}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.191+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030184760}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.192+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.192+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658815507}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.192+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752862240}) 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.192+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.210 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.192+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658815507}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.192+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3222787205}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.193+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.193+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437175458}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.193+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3290002768}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.193+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.193+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437175458}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.193+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019232777}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.194+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.194+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910577912}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.194+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=530509803}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.194+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.194+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910577912}) 2024-09-15T03:48:28.211 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.194+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342735038}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.195+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.195+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435640908}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.195+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915004419}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.195+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.195+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435640908}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.195+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728034401}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.196+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.196+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049357918}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.196+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333497121}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.196+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.196+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049357918}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.196+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036950910}) 2024-09-15T03:48:28.212 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.197+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.197+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940810789}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.197+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076940014}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.197+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.197+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940810789}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.197+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3705308185}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552650524}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068315980}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552650524}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3347653094}) 2024-09-15T03:48:28.213 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.198+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749188074}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256317498}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749188074}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376930194}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736695449}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264756240}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.199+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736695449}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2836420015}) 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.214 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946894812}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1574851797}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.200+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946894812}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193498186}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437845431}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799341163}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437845431}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.201+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1920881482}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.202+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.202+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081966397}) 2024-09-15T03:48:28.215 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.202+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4017141247}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.202+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.202+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081966397}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.202+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3378055053}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.203+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.203+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667469032}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.203+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298041721}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.203+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.203+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667469032}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.203+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1994710181}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.204+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.204+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2640530381}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.204+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387374645}) 2024-09-15T03:48:28.216 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.204+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.217 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.204+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2640530381}) 2024-09-15T03:48:28.217 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.204+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1552002067}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.205+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.205+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653556429}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.205+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3655323898}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.205+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.205+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653556429}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.205+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974095490}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.206+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.206+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604836826}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.206+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013472081}) 2024-09-15T03:48:28.229 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.206+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.206+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604836826}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.206+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330518665}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.207+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.207+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549259940}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.207+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539219847}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.207+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.207+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549259940}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.207+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120369406}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.208+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.208+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042004060}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.208+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4206859267}) 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.208+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.230 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.208+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042004060}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.208+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140985581}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.209+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.209+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275224501}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.209+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2121644131}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.209+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.209+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275224501}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.209+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408016317}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.210+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.210+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277542675}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.210+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=141210103}) 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.210+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.231 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.210+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277542675}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.210+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191809874}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.211+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.211+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094917321}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.211+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1735116258}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.211+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.211+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094917321}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.211+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4054487180}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.212+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.212+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293155375}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.212+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2684694215}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.212+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.212+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293155375}) 2024-09-15T03:48:28.232 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.212+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3933722233}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.213+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.213+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294494753}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.213+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2470198801}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.213+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.213+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294494753}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.213+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949919540}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.214+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.214+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074445557}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.214+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640482883}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.214+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.214+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074445557}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.214+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1124719320}) 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.215+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.233 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.215+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=677789625}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.215+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188619061}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.215+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.215+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=677789625}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.215+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=490561410}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.216+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.216+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573201943}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.216+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2657942385}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.216+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.216+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573201943}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.216+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637041239}) 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.217+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.234 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.217+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277446104}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.217+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491326078}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.217+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.217+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277446104}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.217+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53025491}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.218+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.218+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=363896627}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.218+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006643372}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.218+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.218+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=363896627}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.218+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734193548}) 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.219+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.235 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.219+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578767222}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.219+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=276388257}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.219+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.219+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578767222}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.219+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=490088937}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.220+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.220+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690304313}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.220+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035159573}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.220+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.220+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690304313}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.220+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267498684}) 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.236 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3514947460}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125803337}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3514947460}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746658598}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.221+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2955316974}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1706588135}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2955316974}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2337037360}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878588529}) 2024-09-15T03:48:28.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3471639514}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878588529}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.222+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559506971}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697120067}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628811914}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697120067}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.223+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2016582416}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484423307}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1324617170}) 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484423307}) 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.224+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661079331}) 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3755748280,mdsmap=2826658659}) 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679406945}) 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3755748280,mdsmap=2826658659}) 2024-09-15T03:48:28.239 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.225+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3636279367}) 2024-09-15T03:48:28.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {mdsmap=3140514009,mgr=2938112866,mgr_metadata=2280977772,mgrstat=3478737778,mon_config_key=4117641549,monmap=244636553,osd_metadata=1491145534,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3339647910}) 2024-09-15T03:48:28.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3478737778,mon_config_key=105398142,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3684676279}) 2024-09-15T03:48:28.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:48:28.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {mdsmap=3140514009,mgr=2938112866,mgr_metadata=2280977772,mgrstat=3478737778,mon_config_key=4117641549,monmap=244636553,osd_metadata=1491145534,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3339647910}) 2024-09-15T03:48:28.240 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:48:28.234+0000 7f9ec8ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mdsmap=3140514009,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=3478737778,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=1684560307}) 2024-09-15T03:48:29.136 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:48:29.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:29.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.231+0000 7f104e91d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:29.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.231+0000 7f104e91d640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.231+0000 7f104e91d640 10 monclient: monmap: 2024-09-15T03:48:29.232 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.232 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.232 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:29.231855+0000 2024-09-15T03:48:29.232 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:29.231855+0000 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:29.233 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.231+0000 7f104e91d640 1 Processor -- start 2024-09-15T03:48:29.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.231+0000 7f104e91d640 1 -- start start 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.231+0000 7f104e91d640 10 monclient: init 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10481318f0 0x7f1048131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): picked mon.noname-a con 0x7f10481318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f104809ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f1048136c20 0x7f1048139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): picked mon.noname-c con 0x7f1048136c20 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104809f530 con 0x7f104809ea50 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1048131d50 con 0x7f10481318f0 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104809fbf0 con 0x7f1048136c20 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104e91d640 10 monclient(hunting): authenticate will time out at 1906.871414s 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10481318f0 0x7f1048131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10481318f0 0x7f1048131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52338/0 (socket says 172.21.15.153:52338) 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 1 -- 172.21.15.153:0/886527654 learned_addr learned my addr 172.21.15.153:0/886527654 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): get_auth_request con 0x7f10481318f0 auth_method 0 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 1 --2- 172.21.15.153:0/886527654 >> v2:172.21.15.181:3300/0 conn(0x7f1048136c20 0x7f1048139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): get_auth_request con 0x7f1048136c20 auth_method 0 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.232+0000 7f104ceba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 10 monclient(hunting): handle_auth_done global_id 204439 payload 274 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 1 -- 172.21.15.153:0/886527654 >> v2:172.21.15.153:3300/0 conn(0x7f10481318f0 msgr2=0x7f1048131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 1 --2- 172.21.15.153:0/886527654 >> v2:172.21.15.153:3300/0 conn(0x7f10481318f0 0x7f1048131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 1 -- 172.21.15.153:0/886527654 >> v1:172.21.15.153:6789/0 conn(0x7f104809ea50 legacy=0x7f104809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 1 monclient: found mon.noname-c 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.233+0000 7f104ceba640 1 -- 172.21.15.153:0/886527654 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104809fe30 con 0x7f1048136c20 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104ceba640 1 -- 172.21.15.153:0/886527654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1048136c20 msgr2=0x7f1048139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104ceba640 1 --2- 172.21.15.153:0/886527654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1048136c20 0x7f1048139000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f104809c9e0 tx=0x7f1040009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=978c4a03186f54dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 1 -- 172.21.15.153:0/886527654 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040007760 con 0x7f1048136c20 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: dump: 2024-09-15T03:48:29.252 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.253 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 1 -- 172.21.15.153:0/886527654 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104002d050 con 0x7f1048136c20 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 5 monclient: authenticate success, global_id 204439 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 1 -- 172.21.15.153:0/886527654 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040007af0 con 0x7f1048136c20 2024-09-15T03:48:29.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 4 set_mon_vals no callback set 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f1046ffd640 10 monclient: dump: 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.255 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 10 monclient: shutdown 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 -- 172.21.15.153:0/886527654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1048136c20 msgr2=0x7f1048139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 --2- 172.21.15.153:0/886527654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1048136c20 0x7f1048139000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f104809c9e0 tx=0x7f1040009780 comp rx=0 tx=0).stop 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 -- 172.21.15.153:0/886527654 shutdown_connections 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 --2- 172.21.15.153:0/886527654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1048136c20 0x7f1048139000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 --2- 172.21.15.153:0/886527654 >> v2:172.21.15.153:3300/0 conn(0x7f10481318f0 0x7f1048131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 -- 172.21.15.153:0/886527654 >> 172.21.15.153:0/886527654 conn(0x7f1048067110 msgr2=0x7f1048067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.236+0000 7f104e91d640 1 -- 172.21.15.153:0/886527654 shutdown_connections 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.237+0000 7f104e91d640 1 -- 172.21.15.153:0/886527654 wait complete. 2024-09-15T03:48:29.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.237+0000 7f104e91d640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.237+0000 7f104e91d640 10 monclient: monmap: 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.237+0000 7f104e91d640 1 Processor -- start 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.237+0000 7f104e91d640 1 -- start start 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.237+0000 7f104e91d640 10 monclient: init 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 0x7f1048099590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): picked mon.noname-c con 0x7f104809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10481318f0 0x7f1048099ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): picked mon.noname-b con 0x7f10481318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1048136c20 0x7f104809a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): picked mon.noname-a con 0x7f1048136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104811cff0 con 0x7f1048136c20 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104809bf10 con 0x7f10481318f0 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104809dda0 con 0x7f104809ea50 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104e91d640 10 monclient(hunting): authenticate will time out at 1906.877372s 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 0x7f1048099590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 0x7f1048099590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59596/0 (socket says 172.21.15.153:59596) 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1048136c20 0x7f104809a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 -- 172.21.15.153:0/266859834 learned_addr learned my addr 172.21.15.153:0/266859834 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): get_auth_request con 0x7f104809ea50 auth_method 0 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): get_auth_request con 0x7f1048136c20 auth_method 0 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10481318f0 0x7f1048099ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): get_auth_request con 0x7f10481318f0 auth_method 0 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f10477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient(hunting): handle_auth_done global_id 244127 payload 824 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 -- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10481318f0 msgr2=0x7f1048099ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10481318f0 0x7f1048099ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 -- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1048136c20 msgr2=0x7f104809a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1048136c20 0x7f104809a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 monclient: found mon.noname-c 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f1047fff640 1 -- 172.21.15.153:0/266859834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1040009a20 con 0x7f104809ea50 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.238+0000 7f104ceba640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1048136c20 0x7f104809a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f1047fff640 1 -- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 msgr2=0x7f1048099590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f1047fff640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 0x7f1048099590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f103800ea20 tx=0x7f10380091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10380095a0 con 0x7f104809ea50 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: dump: 2024-09-15T03:48:29.262 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103800cb50 con 0x7f104809ea50 2024-09-15T03:48:29.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 5 monclient: authenticate success, global_id 244127 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 10 monclient: _renew_subs 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104809af80 con 0x7f104809ea50 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038013470 con 0x7f104809ea50 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 10 monclient: dump: 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 10 monclient: _renew_subs 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104811c770 con 0x7f104809ea50 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f10467fc640 4 set_mon_vals no callback set 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.241+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1038013820 con 0x7f104809ea50 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f104e91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f104e91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f104e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104811cda0 con 0x7f104809ea50 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f102ffff640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1034038740 0x7f103403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f103804cbd0 con 0x7f104809ea50 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f10477fe640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1034038740 0x7f103403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.242+0000 7f10477fe640 10 monclient: get_auth_request con 0x7f1034038740 auth_method 0 2024-09-15T03:48:29.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.244+0000 7f10477fe640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1034038740 0x7f103403abf0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f10481323d0 tx=0x7f103c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.246+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f103801bb90 con 0x7f104809ea50 2024-09-15T03:48:29.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.246+0000 7f102ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.246+0000 7f102ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:29.441 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=110 2024-09-15T03:48:29.441 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 110 -ne 110 ]] 2024-09-15T03:48:29.441 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:48:29.441 INFO:tasks.workunit.client.0.smithi181.stderr:+ export CEPH_ARGS= 2024-09-15T03:48:29.441 INFO:tasks.workunit.client.0.smithi181.stderr:+ CEPH_ARGS= 2024-09-15T03:48:29.441 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph auth del client.bar 2024-09-15T03:48:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.458+0000 7f104e91d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.458+0000 7f104e91d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:29.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.458+0000 7f104e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:29.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.458+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f104804bf60 con 0x7f104809ea50 2024-09-15T03:48:29.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.458+0000 7f102ffff640 1 -- 172.21.15.153:0/266859834 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f1038005970 con 0x7f104809ea50 2024-09-15T03:48:29.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.459+0000 7f102ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:29.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.459+0000 7f102ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:29.460 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1034038740 msgr2=0x7f103403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1034038740 0x7f103403abf0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f10481323d0 tx=0x7f103c007e10 comp rx=0 tx=0).stop 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 10 monclient: shutdown 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 msgr2=0x7f1048099590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 0x7f1048099590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f103800ea20 tx=0x7f10380091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 shutdown_connections 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1034038740 0x7f103403abf0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.460+0000 7f104e91d640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1048136c20 0x7f104809a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.461+0000 7f104e91d640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10481318f0 0x7f1048099ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.461+0000 7f104e91d640 1 --2- 172.21.15.153:0/266859834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f104809ea50 0x7f1048099590 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.461+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 >> 172.21.15.153:0/266859834 conn(0x7f1048067110 msgr2=0x7f10480a3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.461+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 shutdown_connections 2024-09-15T03:48:29.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.461+0000 7f104e91d640 1 -- 172.21.15.153:0/266859834 wait complete. 2024-09-15T03:48:29.472 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:48:29.472 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:48:29.472 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:48:29.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:48:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 10 monclient: get_monmap_and_config 2024-09-15T03:48:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 10 monclient: monmap: 2024-09-15T03:48:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:48:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:48:29.518533+0000 2024-09-15T03:48:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:48:29.518533+0000 2024-09-15T03:48:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:48:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:29.520 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 1 Processor -- start 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 1 -- start start 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 10 monclient: init 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.517+0000 7f4744c0f640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f474006e810 0x7f474012e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f474006e810 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4740097890 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f474009c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4740097cd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474009d750 con 0x7f4740097890 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474006ebe0 con 0x7f474006e810 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474009d470 con 0x7f4740097cd0 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f4744c0f640 10 monclient(hunting): authenticate will time out at 1914.000069s 2024-09-15T03:48:29.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f474006e810 0x7f474012e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f474006e810 0x7f474012e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:35270/0 (socket says 172.21.15.181:35270) 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 -- 172.21.15.181:0/3676034868 learned_addr learned my addr 172.21.15.181:0/3676034868 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): get_auth_request con 0x7f474006e810 auth_method 0 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 1 --2- 172.21.15.181:0/3676034868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f474009c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): get_auth_request con 0x7f4740097cd0 auth_method 0 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient(hunting): handle_auth_done global_id 204445 payload 274 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 -- 172.21.15.181:0/3676034868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 msgr2=0x7f474009c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 --2- 172.21.15.181:0/3676034868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f474009c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 -- 172.21.15.181:0/3676034868 >> v1:172.21.15.153:6789/0 conn(0x7f4740097890 legacy=0x7f474009bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 monclient: found mon.noname-c 2024-09-15T03:48:29.533 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.518+0000 7f473e59c640 1 -- 172.21.15.181:0/3676034868 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474009da30 con 0x7f474006e810 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473e59c640 1 -- 172.21.15.181:0/3676034868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474006e810 msgr2=0x7f474012e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473e59c640 1 --2- 172.21.15.181:0/3676034868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474006e810 0x7f474012e6f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4740069ec0 tx=0x7f47280091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ef9bf31c1211bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 1 -- 172.21.15.181:0/3676034868 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472802ae80 con 0x7f474006e810 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: dump: 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.534 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 1 -- 172.21.15.181:0/3676034868 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4728032440 con 0x7f474006e810 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 1 -- 172.21.15.181:0/3676034868 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728032800 con 0x7f474006e810 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f4744c0f640 5 monclient: authenticate success, global_id 204445 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f4744c0f640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f4744c0f640 4 set_mon_vals no callback set 2024-09-15T03:48:29.535 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.521+0000 7f473d59a640 10 monclient: dump: 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.536 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 10 monclient: shutdown 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 -- 172.21.15.181:0/3676034868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474006e810 msgr2=0x7f474012e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 --2- 172.21.15.181:0/3676034868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474006e810 0x7f474012e6f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4740069ec0 tx=0x7f47280091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 -- 172.21.15.181:0/3676034868 shutdown_connections 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 --2- 172.21.15.181:0/3676034868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f474009c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 --2- 172.21.15.181:0/3676034868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f474006e810 0x7f474012e6f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 -- 172.21.15.181:0/3676034868 >> 172.21.15.181:0/3676034868 conn(0x7f47400a1330 msgr2=0x7f47400a1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 -- 172.21.15.181:0/3676034868 shutdown_connections 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 1 -- 172.21.15.181:0/3676034868 wait complete. 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.522+0000 7f4744c0f640 10 monclient: monmap: 2024-09-15T03:48:29.537 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 Processor -- start 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 -- start start 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient: init 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f474006e810 0x7f4740141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f474006e810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 0x7f474012d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4740097890 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f4740130950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f4740097cd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474009cd30 con 0x7f474006e810 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47400a0d00 con 0x7f4740097890 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4740132370 con 0x7f4740097cd0 2024-09-15T03:48:29.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f4744c0f640 10 monclient(hunting): authenticate will time out at 1914.005670s 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 0x7f474012d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 0x7f474012d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:35280/0 (socket says 172.21.15.181:35280) 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 1 -- 172.21.15.181:0/1410609920 learned_addr learned my addr 172.21.15.181:0/1410609920 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): get_auth_request con 0x7f4740097890 auth_method 0 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f474006e810 0x7f4740141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f4740130950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): get_auth_request con 0x7f474006e810 auth_method 0 2024-09-15T03:48:29.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): get_auth_request con 0x7f4740097cd0 auth_method 0 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.523+0000 7f473e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 10 monclient(hunting): handle_auth_done global_id 204448 payload 824 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 1 -- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 msgr2=0x7f4740130950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f4740130950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 1 -- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f474006e810 msgr2=0x7f4740141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f474006e810 0x7f4740141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 1 monclient: found mon.noname-b 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.524+0000 7f473dd9b640 1 -- 172.21.15.181:0/1410609920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4728009450 con 0x7f4740097890 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f473dd9b640 1 -- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 msgr2=0x7f474012d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f473dd9b640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 0x7f474012d950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f473000c610 tx=0x7f4730007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730012840 con 0x7f4740097890 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: dump: 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.543 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47300129e0 con 0x7f4740097890 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 5 monclient: authenticate success, global_id 204448 2024-09-15T03:48:29.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 10 monclient: _renew_subs 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4740127b40 con 0x7f4740097890 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730012da0 con 0x7f4740097890 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f47367fc640 10 monclient: dump: 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.545 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f473cd99640 4 set_mon_vals no callback set 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 10 monclient: _renew_subs 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.526+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4740141f80 con 0x7f4740097890 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f4730011590 con 0x7f4740097890 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f4744c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f4744c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f4744c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4740066600 con 0x7f4740097890 2024-09-15T03:48:29.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f47367fc640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4720038740 0x7f472003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.527+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f473004c890 con 0x7f4740097890 2024-09-15T03:48:29.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.528+0000 7f473e59c640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4720038740 0x7f472003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.528+0000 7f473e59c640 10 monclient: get_auth_request con 0x7f4720038740 auth_method 0 2024-09-15T03:48:29.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.530+0000 7f473e59c640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4720038740 0x7f472003abf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4728034000 tx=0x7f4728032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.532+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f47300108c0 con 0x7f4740097890 2024-09-15T03:48:29.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.532+0000 7f47367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.532+0000 7f47367fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.566+0000 7f22bd31d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.566+0000 7f22bd31d640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.566+0000 7f22bd31d640 10 monclient: monmap: 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:29.567274+0000 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:29.567274+0000 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:29.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:29.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:29.568 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:29.568 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:29.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 1 Processor -- start 2024-09-15T03:48:29.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 1 -- start start 2024-09-15T03:48:29.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient: init 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22b8131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22b809e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f22b8137bc0 0x7f22b8139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient(hunting): picked mon.noname-a con 0x7f22b8137bc0 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b809faf0 con 0x7f22b8131ab0 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b809fd30 con 0x7f22b809e910 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b8132590 con 0x7f22b8137bc0 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22bd31d640 10 monclient(hunting): authenticate will time out at 1907.206610s 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f22b8137bc0 0x7f22b8139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f22b8137bc0 0x7f22b8139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52356/0 (socket says 172.21.15.153:52356) 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 1 -- 172.21.15.153:0/4117601037 learned_addr learned my addr 172.21.15.153:0/4117601037 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 10 monclient(hunting): get_auth_request con 0x7f22b8137bc0 auth_method 0 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.567+0000 7f22b6ffd640 1 --1- 172.21.15.153:0/4117601037 >> v1:172.21.15.153:6789/0 conn(0x7f22b8131ab0 0x7f22b8131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34572/0 (socket says 172.21.15.153:34572) 2024-09-15T03:48:29.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.569+0000 7f22b6ffd640 1 -- 172.21.15.153:0/4117601037 >> v1:172.21.15.153:6789/0 conn(0x7f22b8131ab0 legacy=0x7f22b8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.569+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.569+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.569+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.569+0000 7f22b67fc640 1 -- 172.21.15.153:0/4117601037 >> v1:172.21.15.181:6789/0 conn(0x7f22b809e910 legacy=0x7f22b809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 10 monclient(hunting): handle_auth_done global_id 214443 payload 274 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 1 -- 172.21.15.153:0/4117601037 >> v1:172.21.15.181:6789/0 conn(0x7f22b809e910 legacy=0x7f22b809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 1 -- 172.21.15.153:0/4117601037 >> v1:172.21.15.153:6789/0 conn(0x7f22b8131ab0 legacy=0x7f22b8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 1 monclient: found mon.noname-a 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.570+0000 7f22b77fe640 1 -- 172.21.15.153:0/4117601037 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80a1110 con 0x7f22b8137bc0 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.572+0000 7f22b77fe640 1 -- 172.21.15.153:0/4117601037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 msgr2=0x7f22b8139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.572+0000 7f22b77fe640 1 --2- 172.21.15.153:0/4117601037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8139fa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009fd0 tx=0x7f22b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159ec1bbfd078c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 1 -- 172.21.15.153:0/4117601037 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0007610 con 0x7f22b8137bc0 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: dump: 2024-09-15T03:48:29.575 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 1 -- 172.21.15.153:0/4117601037 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b0007770 con 0x7f22b8137bc0 2024-09-15T03:48:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 5 monclient: authenticate success, global_id 214443 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 1 -- 172.21.15.153:0/4117601037 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0007b10 con 0x7f22b8137bc0 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 4 set_mon_vals no callback set 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22b5ffb640 10 monclient: dump: 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 10 monclient: shutdown 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 1 -- 172.21.15.153:0/4117601037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 msgr2=0x7f22b8139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 1 --2- 172.21.15.153:0/4117601037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8139fa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009fd0 tx=0x7f22b0009840 comp rx=0 tx=0).stop 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 1 -- 172.21.15.153:0/4117601037 shutdown_connections 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 1 --2- 172.21.15.153:0/4117601037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8139fa0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 1 -- 172.21.15.153:0/4117601037 >> 172.21.15.153:0/4117601037 conn(0x7f22b8067d30 msgr2=0x7f22b80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.573+0000 7f22bd31d640 1 -- 172.21.15.153:0/4117601037 shutdown_connections 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.574+0000 7f22bd31d640 1 -- 172.21.15.153:0/4117601037 wait complete. 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.574+0000 7f22bd31d640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.574+0000 7f22bd31d640 10 monclient: monmap: 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.574+0000 7f22bd31d640 1 Processor -- start 2024-09-15T03:48:29.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.574+0000 7f22bd31d640 1 -- start start 2024-09-15T03:48:29.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.574+0000 7f22bd31d640 10 monclient: init 2024-09-15T03:48:29.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:48:29.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22b809e910 0x7f22b809a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): picked mon.noname-b con 0x7f22b809e910 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:29.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22b8131ab0 0x7f22b8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): picked mon.noname-c con 0x7f22b8131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): picked mon.noname-a con 0x7f22b8137bc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b809bde0 con 0x7f22b8137bc0 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80a53b0 con 0x7f22b809e910 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8099790 con 0x7f22b8131ab0 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22bd31d640 10 monclient(hunting): authenticate will time out at 1907.214348s 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22b8131ab0 0x7f22b8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52364/0 (socket says 172.21.15.153:52364) 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 learned_addr learned my addr 172.21.15.153:0/832222005 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): get_auth_request con 0x7f22b8137bc0 auth_method 0 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): get_auth_request con 0x7f22b8131ab0 auth_method 0 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22b809e910 0x7f22b809a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): get_auth_request con 0x7f22b809e910 auth_method 0 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient(hunting): handle_auth_done global_id 214449 payload 824 2024-09-15T03:48:29.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22b8131ab0 msgr2=0x7f22b8096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22b8131ab0 0x7f22b8096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22b809e910 msgr2=0x7f22b809a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22b809e910 0x7f22b809a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 monclient: found mon.noname-a 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b0009ae0 con 0x7f22b8137bc0 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.575+0000 7f22b67fc640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22b8131ab0 0x7f22b8096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 msgr2=0x7f22b8096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22b77fe640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8096fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009f20 tx=0x7f22b0037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b003e870 con 0x7f22b8137bc0 2024-09-15T03:48:29.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22b77fe640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8096fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009f20 tx=0x7f22b0037000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f22b003edb0 mon_map magic: 0 v1 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 10 monclient: dump: 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.590 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b003e9f0 con 0x7f22b8137bc0 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 5 monclient: authenticate success, global_id 214449 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 10 monclient: _renew_subs 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b8099a70 con 0x7f22b8137bc0 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 10 monclient: _renew_subs 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b8136d00 con 0x7f22b8137bc0 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22b57fa640 4 set_mon_vals no callback set 2024-09-15T03:48:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 0x7f22b80974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8063740 con 0x7f22b80985c0 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.578+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b003edb0 con 0x7f22b8137bc0 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f229effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f229effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f229effd640 10 monclient: dump: 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.592 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f22b0032640 con 0x7f22b8137bc0 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 0x7f22b80974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient: get_auth_request con 0x7f22b80985c0 auth_method 0 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f229effd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22a4038740 0x7f22a403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_done global_id 214452 payload 824 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b6ffd640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22b804bf60 con 0x7f22b80985c0 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b67fc640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22a4038740 0x7f22a403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.579+0000 7f22b67fc640 10 monclient: get_auth_request con 0x7f22a4038740 auth_method 0 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f22b6ffd640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 msgr2=0x7f22b80974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f22b6ffd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 0x7f22b80974e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004f70 tx=0x7f22a8007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a8012850 con 0x7f22b80985c0 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 10 monclient: dump: 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.594 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f22a80129b0 con 0x7f22b80985c0 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 msgr2=0x7f22b80974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 0x7f22b80974e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004f70 tx=0x7f22a8007cc0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 msgr2=0x7f22b80974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f229effd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b80985c0 0x7f22b80974e0 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004f70 tx=0x7f22a8007cc0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.581+0000 7f22b67fc640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22a4038740 0x7f22a403abf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f22ac007fd0 tx=0x7f22ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22bd31d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22bd31d640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 0x7f22b8132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22bd31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8097a20 con 0x7f22b81326d0 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 0x7f22b8132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient: get_auth_request con 0x7f22b81326d0 auth_method 0 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 10 monclient(hunting): handle_auth_done global_id 214455 payload 824 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.587+0000 7f22b6ffd640 1 -- 172.21.15.153:0/832222005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f22b8066280 con 0x7f22b81326d0 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22b6ffd640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 msgr2=0x7f22b8132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22b6ffd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 0x7f22b8132ad0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004f70 tx=0x7f22a8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a8021830 con 0x7f22b81326d0 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 10 monclient: dump: 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.602 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f22a8021990 con 0x7f22b81326d0 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 msgr2=0x7f22b8132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 0x7f22b8132ad0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004f70 tx=0x7f22a8015000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 msgr2=0x7f22b8132ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:29.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b81326d0 0x7f22b8132ad0 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004f70 tx=0x7f22a8015000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 206, 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:29.604 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 4, 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.605 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.606 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:29.607 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.608 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:29.610 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.611 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:29.612 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.613 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.614 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.615 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.616 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.617 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:48:29.617 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.617 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.617 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:29.617 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22a4038740 msgr2=0x7f22a403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f22a4038740 0x7f22a403abf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f22ac007fd0 tx=0x7f22ac009040 comp rx=0 tx=0).stop 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 10 monclient: shutdown 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 msgr2=0x7f22b8096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 --2- 172.21.15.153:0/832222005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b8137bc0 0x7f22b8096fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009f20 tx=0x7f22b0037000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 reap_dead start 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22b77fe640 1 -- 172.21.15.153:0/832222005 reap_dead start 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f229effd640 1 -- 172.21.15.153:0/832222005 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7f22b006cc30 con 0x7f22b8137bc0 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 shutdown_connections 2024-09-15T03:48:29.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 >> 172.21.15.153:0/832222005 conn(0x7f22b8067d30 msgr2=0x7f22b809d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 shutdown_connections 2024-09-15T03:48:29.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.590+0000 7f22bd31d640 1 -- 172.21.15.153:0/832222005 wait complete. 2024-09-15T03:48:29.620 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:48:29.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:48:29.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.715+0000 7f3dff75d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.715+0000 7f3dff75d640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.715+0000 7f3dff75d640 10 monclient: monmap: 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:29.715812+0000 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:29.715812+0000 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.716 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.715+0000 7f3dff75d640 1 Processor -- start 2024-09-15T03:48:29.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.715+0000 7f3dff75d640 1 -- start start 2024-09-15T03:48:29.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.715+0000 7f3dff75d640 10 monclient: init 2024-09-15T03:48:29.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3df81318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f3df809ea50 0x7f3df809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient(hunting): picked mon.noname-c con 0x7f3df809ea50 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3df8136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df8131cc0 con 0x7f3df81318f0 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df809fbf0 con 0x7f3df8136c20 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df809fe30 con 0x7f3df809ea50 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dff75d640 10 monclient(hunting): authenticate will time out at 1907.355428s 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfd4f9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3df81318f0 0x7f3df8131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34588/0 (socket says 172.21.15.153:34588) 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfd4f9640 1 -- 172.21.15.153:0/1309051182 learned_addr learned my addr 172.21.15.153:0/1309051182 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 1 --2- 172.21.15.153:0/1309051182 >> v2:172.21.15.181:3300/0 conn(0x7f3df809ea50 0x7f3df809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): get_auth_request con 0x7f3df809ea50 auth_method 0 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_done global_id 204457 payload 274 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 1 -- 172.21.15.153:0/1309051182 >> v1:172.21.15.181:6789/0 conn(0x7f3df8136c20 legacy=0x7f3df8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 1 -- 172.21.15.153:0/1309051182 >> v1:172.21.15.153:6789/0 conn(0x7f3df81318f0 legacy=0x7f3df8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 1 monclient: found mon.noname-c 2024-09-15T03:48:29.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.716+0000 7f3dfccf8640 1 -- 172.21.15.153:0/1309051182 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df80a1210 con 0x7f3df809ea50 2024-09-15T03:48:29.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3dfccf8640 1 -- 172.21.15.153:0/1309051182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 msgr2=0x7f3df809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3dfccf8640 1 --2- 172.21.15.153:0/1309051182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df809ee20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009850 tx=0x7f3de80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74db51e802863a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 1 -- 172.21.15.153:0/1309051182 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8032440 con 0x7f3df809ea50 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: dump: 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3dff75d640 5 monclient: authenticate success, global_id 204457 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3dff75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 1 -- 172.21.15.153:0/1309051182 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de80325a0 con 0x7f3df809ea50 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3dff75d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3dff75d640 4 set_mon_vals no callback set 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 1 -- 172.21.15.153:0/1309051182 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8032940 con 0x7f3df809ea50 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.719+0000 7f3deffff640 10 monclient: dump: 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 10 monclient: shutdown 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 -- 172.21.15.153:0/1309051182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 msgr2=0x7f3df809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 --2- 172.21.15.153:0/1309051182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df809ee20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009850 tx=0x7f3de80090f0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 -- 172.21.15.153:0/1309051182 shutdown_connections 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 --2- 172.21.15.153:0/1309051182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df809ee20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 -- 172.21.15.153:0/1309051182 >> 172.21.15.153:0/1309051182 conn(0x7f3df8067110 msgr2=0x7f3df8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 -- 172.21.15.153:0/1309051182 shutdown_connections 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 1 -- 172.21.15.153:0/1309051182 wait complete. 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.720+0000 7f3dff75d640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient: monmap: 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:29.734 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 Processor -- start 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 -- start start 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient: init 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df8130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): picked mon.noname-b con 0x7f3df809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df81318f0 0x7f3df8099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): picked mon.noname-c con 0x7f3df81318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 0x7f3df8099e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): picked mon.noname-a con 0x7f3df8136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df809b8e0 con 0x7f3df8136c20 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df809e000 con 0x7f3df809ea50 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df811bba0 con 0x7f3df81318f0 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.721+0000 7f3dff75d640 10 monclient(hunting): authenticate will time out at 1907.360955s 2024-09-15T03:48:29.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 0x7f3df8099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df81318f0 0x7f3df8099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 0x7f3df8099e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52392/0 (socket says 172.21.15.153:52392) 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df81318f0 0x7f3df8099920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59618/0 (socket says 172.21.15.153:59618) 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 learned_addr learned my addr 172.21.15.153:0/3460169865 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): get_auth_request con 0x7f3df8136c20 auth_method 0 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): get_auth_request con 0x7f3df81318f0 auth_method 0 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df8130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): get_auth_request con 0x7f3df809ea50 auth_method 0 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfccf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient(hunting): handle_auth_done global_id 214458 payload 824 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df81318f0 msgr2=0x7f3df8099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df81318f0 0x7f3df8099920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 msgr2=0x7f3df8130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df8130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 monclient: found mon.noname-a 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8009390 con 0x7f3df8136c20 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.722+0000 7f3dfd4f9640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3df809ea50 0x7f3df8130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 msgr2=0x7f3df8099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dfdcfa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 0x7f3df8099e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c890 tx=0x7f3df0002640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df000ce80 con 0x7f3df8136c20 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: dump: 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.740 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df000aa00 con 0x7f3df8136c20 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 5 monclient: authenticate success, global_id 214458 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df000ae00 con 0x7f3df8136c20 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 10 monclient: _renew_subs 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df809bf10 con 0x7f3df8136c20 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3def7fe640 4 set_mon_vals no callback set 2024-09-15T03:48:29.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3ded7fa640 10 monclient: dump: 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 10 monclient: _renew_subs 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.724+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df8141ea0 con 0x7f3df8136c20 2024-09-15T03:48:29.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dff75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dff75d640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 0x7f3df80298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dff75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8065e00 con 0x7f3df8130ec0 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 0x7f3df80298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient: get_auth_request con 0x7f3df8130ec0 auth_method 0 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.725+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3df0012630 con 0x7f3df8136c20 2024-09-15T03:48:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.726+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_done global_id 244142 payload 824 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.726+0000 7f3dfd4f9640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3df809b460 con 0x7f3df8130ec0 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.726+0000 7f3ded7fa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ddc038740 0x7f3ddc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.726+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3df004c920 con 0x7f3df8136c20 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.726+0000 7f3dfccf8640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ddc038740 0x7f3ddc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.726+0000 7f3dfccf8640 10 monclient: get_auth_request con 0x7f3ddc038740 auth_method 0 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3dfd4f9640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 msgr2=0x7f3df80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3dfd4f9640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 0x7f3df80298c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005140 tx=0x7f3df4007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df400fd50 con 0x7f3df8130ec0 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 10 monclient: dump: 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.744 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3df400feb0 con 0x7f3df8130ec0 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 msgr2=0x7f3df80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 0x7f3df80298c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005140 tx=0x7f3df4007f00 comp rx=0 tx=0).stop 2024-09-15T03:48:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 msgr2=0x7f3df80298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3ded7fa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8130ec0 0x7f3df80298c0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005140 tx=0x7f3df4007f00 comp rx=0 tx=0).stop 2024-09-15T03:48:29.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.728+0000 7f3dfccf8640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ddc038740 0x7f3ddc03abf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3df8076600 tx=0x7f3de8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dff75d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dff75d640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 0x7f3df802a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dff75d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8131290 con 0x7f3df8029e00 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 0x7f3df802a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient: get_auth_request con 0x7f3df8029e00 auth_method 0 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 10 monclient(hunting): handle_auth_done global_id 244145 payload 824 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.734+0000 7f3dfd4f9640 1 -- 172.21.15.153:0/3460169865 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3df804bf60 con 0x7f3df8029e00 2024-09-15T03:48:29.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dfd4f9640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 msgr2=0x7f3df802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dfd4f9640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 0x7f3df802a200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005140 tx=0x7f3df4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4021870 con 0x7f3df8029e00 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 10 monclient: dump: 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f3df40219d0 con 0x7f3df8029e00 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 msgr2=0x7f3df802a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 0x7f3df802a200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005140 tx=0x7f3df4018000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 msgr2=0x7f3df802a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3ded7fa640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3df8029e00 0x7f3df802a200 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3df4005140 tx=0x7f3df4018000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:29.750 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 206, 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:29.751 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.752 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:29.753 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.755 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.756 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.757 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.758 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:29.759 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.760 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:29.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:29.762 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.750+0000 7f4744c0f640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.750+0000 7f4744c0f640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.750+0000 7f4744c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.750+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f4740126c40 con 0x7f4740097890 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.754+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f47300102e0 con 0x7f4740097890 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.754+0000 7f47367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T03:48:29.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.754+0000 7f47367fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ddc038740 msgr2=0x7f3ddc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3ddc038740 0x7f3ddc03abf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3df8076600 tx=0x7f3de8032000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 10 monclient: shutdown 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 msgr2=0x7f3df8099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 --2- 172.21.15.153:0/3460169865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df8136c20 0x7f3df8099e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c890 tx=0x7f3df0002640 comp rx=0 tx=0).stop 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 reap_dead start 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dfdcfa640 1 -- 172.21.15.153:0/3460169865 reap_dead start 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 shutdown_connections 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 >> 172.21.15.153:0/3460169865 conn(0x7f3df8067110 msgr2=0x7f3df80a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 shutdown_connections 2024-09-15T03:48:29.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.737+0000 7f3dff75d640 1 -- 172.21.15.153:0/3460169865 wait complete. 2024-09-15T03:48:29.766 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:48:29.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:48:29.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:48:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.858+0000 7fcbc2900640 10 monclient: get_monmap_and_config 2024-09-15T03:48:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.858+0000 7fcbc2900640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.858+0000 7fcbc2900640 10 monclient: monmap: 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:29.859471+0000 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:29.859471+0000 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:29.860 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 1 Processor -- start 2024-09-15T03:48:29.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 1 -- start start 2024-09-15T03:48:29.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient: init 2024-09-15T03:48:29.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbbc1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcbbc09ea50 0x7fcbbc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient(hunting): picked mon.noname-a con 0x7fcbbc09ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbbc136c20 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc131cc0 con 0x7fcbbc1318f0 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc09fbf0 con 0x7fcbbc136c20 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.859+0000 7fcbc2900640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc09fe30 con 0x7fcbbc09ea50 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbc2900640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbc2900640 10 monclient(hunting): authenticate will time out at 1907.499012s 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcbbc09ea50 0x7fcbbc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcbbc09ea50 0x7fcbbc09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52396/0 (socket says 172.21.15.153:52396) 2024-09-15T03:48:29.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/2615592741 learned_addr learned my addr 172.21.15.153:0/2615592741 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): get_auth_request con 0x7fcbbc09ea50 auth_method 0 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.860+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbc0e9d640 1 -- 172.21.15.153:0/2615592741 >> v1:172.21.15.181:6789/0 conn(0x7fcbbc136c20 legacy=0x7fcbbc139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbbfff640 1 -- 172.21.15.153:0/2615592741 >> v1:172.21.15.153:6789/0 conn(0x7fcbbc1318f0 legacy=0x7fcbbc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_done global_id 214461 payload 274 2024-09-15T03:48:29.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/2615592741 >> v1:172.21.15.181:6789/0 conn(0x7fcbbc136c20 legacy=0x7fcbbc139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/2615592741 >> v1:172.21.15.153:6789/0 conn(0x7fcbbc1318f0 legacy=0x7fcbbc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 1 monclient: found mon.noname-a 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.862+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/2615592741 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0a1210 con 0x7fcbbc09ea50 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/2615592741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc09ea50 msgr2=0x7fcbbc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbb7fe640 1 --2- 172.21.15.153:0/2615592741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc09ea50 0x7fcbbc09ee20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009a10 tx=0x7fcbb00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d851984a4a12c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 1 -- 172.21.15.153:0/2615592741 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb002f4b0 con 0x7fcbbc09ea50 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: dump: 2024-09-15T03:48:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 1 -- 172.21.15.153:0/2615592741 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb002f610 con 0x7fcbbc09ea50 2024-09-15T03:48:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 5 monclient: authenticate success, global_id 214461 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 1 -- 172.21.15.153:0/2615592741 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb002f9b0 con 0x7fcbbc09ea50 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 4 set_mon_vals no callback set 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbbaffd640 10 monclient: dump: 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 10 monclient: shutdown 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 1 -- 172.21.15.153:0/2615592741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc09ea50 msgr2=0x7fcbbc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 1 --2- 172.21.15.153:0/2615592741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc09ea50 0x7fcbbc09ee20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009a10 tx=0x7fcbb00091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.865+0000 7fcbc2900640 1 -- 172.21.15.153:0/2615592741 shutdown_connections 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 1 --2- 172.21.15.153:0/2615592741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc09ea50 0x7fcbbc09ee20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 1 -- 172.21.15.153:0/2615592741 >> 172.21.15.153:0/2615592741 conn(0x7fcbbc067110 msgr2=0x7fcbbc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 1 -- 172.21.15.153:0/2615592741 shutdown_connections 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 1 -- 172.21.15.153:0/2615592741 wait complete. 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 10 monclient: build_initial_monmap 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 10 monclient: monmap: 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.866+0000 7fcbc2900640 1 Processor -- start 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 -- start start 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient: init 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbbc09ea50 0x7fcbbc141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): picked mon.noname-c con 0x7fcbbc09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 0x7fcbbc130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): picked mon.noname-a con 0x7fcbbc1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc136c20 0x7fcbbc131050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): picked mon.noname-b con 0x7fcbbc136c20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc09e020 con 0x7fcbbc1318f0 2024-09-15T03:48:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc11cc20 con 0x7fcbbc136c20 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc11cf00 con 0x7fcbbc09ea50 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc2900640 10 monclient(hunting): authenticate will time out at 1907.506496s 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 0x7fcbbc130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 0x7fcbbc130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52406/0 (socket says 172.21.15.153:52406) 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbbc09ea50 0x7fcbbc141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/3280236456 learned_addr learned my addr 172.21.15.153:0/3280236456 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): get_auth_request con 0x7fcbbc1318f0 auth_method 0 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): get_auth_request con 0x7fcbbc09ea50 auth_method 0 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc136c20 0x7fcbbc131050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): get_auth_request con 0x7fcbbc136c20 auth_method 0 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbc0e9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient(hunting): handle_auth_done global_id 214464 payload 824 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbbc09ea50 msgr2=0x7fcbbc141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbbc09ea50 0x7fcbbc141170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc136c20 msgr2=0x7fcbbc131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc136c20 0x7fcbbc131050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:48:29.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 monclient: found mon.noname-a 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.867+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb0009450 con 0x7fcbbc1318f0 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbbb7fe640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 msgr2=0x7fcbbc130b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbbb7fe640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 0x7fcbbc130b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009210 tx=0x7fcbb0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0032a30 con 0x7fcbbc1318f0 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: dump: 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.882 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: _finish_auth 0 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb0032bb0 con 0x7fcbbc1318f0 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 5 monclient: authenticate success, global_id 214464 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 10 monclient: _renew_subs 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc11d1e0 con 0x7fcbbc1318f0 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb003b400 con 0x7fcbbc1318f0 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcba3fff640 10 monclient: dump: 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.884 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 10 monclient: _renew_subs 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbba7fc640 4 set_mon_vals no callback set 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.870+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc11bf90 con 0x7fcbbc1318f0 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbc2900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbc2900640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 0x7fcbbc09a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbc2900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbb7fe640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 0x7fcbbc130b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009210 tx=0x7fcbb0032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fcbb0039040 mgrmap(e 5) v1 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc065e00 con 0x7fcbbc141710 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcbb0039040 con 0x7fcbbc1318f0 2024-09-15T03:48:29.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 0x7fcbbc09a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient: get_auth_request con 0x7fcbbc141710 auth_method 0 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcba3fff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcba8038770 0x7fcba803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbc0e9d640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcba8038770 0x7fcba803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_done global_id 204469 payload 824 2024-09-15T03:48:29.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbbbfff640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbbc136670 con 0x7fcbbc141710 2024-09-15T03:48:29.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.871+0000 7fcbc0e9d640 10 monclient: get_auth_request con 0x7fcba8038770 auth_method 0 2024-09-15T03:48:29.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.907+0000 7f4744c0f640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T03:48:29.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.907+0000 7f4744c0f640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T03:48:29.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.907+0000 7f4744c0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:29.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.907+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f47400008d0 con 0x7f4740097890 2024-09-15T03:48:29.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.908+0000 7f47367fc640 1 -- 172.21.15.181:0/1410609920 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f47300108c0 con 0x7f4740097890 2024-09-15T03:48:29.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.908+0000 7f47367fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-15T03:48:29.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.908+0000 7f47367fc640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcbbbfff640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 msgr2=0x7fcbbc09a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcbbbfff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 0x7fcbbc09a130 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00f7b0 tx=0x7fcbac009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac00fe80 con 0x7fcbbc141710 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcba3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcba3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcba3fff640 10 monclient: dump: 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.913 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.873+0000 7fcbc0e9d640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcba8038770 0x7fcba803ac20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc11d870 tx=0x7fcbb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcbac0072e0 con 0x7fcbbc141710 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 msgr2=0x7fcbbc09a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 0x7fcbbc09a130 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00f7b0 tx=0x7fcbac009040 comp rx=0 tx=0).stop 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 msgr2=0x7fcbbc09a130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:29.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.874+0000 7fcba3fff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc141710 0x7fcbbc09a130 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00f7b0 tx=0x7fcbac009040 comp rx=0 tx=0).stop 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.879+0000 7fcbc2900640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.879+0000 7fcbc2900640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 0x7fcbbc029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.879+0000 7fcbc2900640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.879+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc09b480 con 0x7fcbbc0298c0 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 0x7fcbbc029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient: get_auth_request con 0x7fcbbc0298c0 auth_method 0 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 10 monclient(hunting): handle_auth_done global_id 204472 payload 824 2024-09-15T03:48:29.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.880+0000 7fcbbbfff640 1 -- 172.21.15.153:0/3280236456 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcbbc04bf60 con 0x7fcbbc0298c0 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcbbbfff640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 msgr2=0x7fcbbc029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcbbbfff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 0x7fcbbc029cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbac017000 tx=0x7fcbac018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac021620 con 0x7fcbbc0298c0 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 10 monclient: dump: 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:29.916 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fcbac021780 con 0x7fcbbc0298c0 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:29.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 msgr2=0x7fcbbc029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 0x7fcbbc029cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbac017000 tx=0x7fcbac018000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 msgr2=0x7fcbbc029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.882+0000 7fcba3fff640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbbc0298c0 0x7fcbbc029cc0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbac017000 tx=0x7fcbac018000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 206, 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.919 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.920 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:29.921 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:29.922 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.923 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.924 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.925 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:29.926 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:29.927 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.929 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.930 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:29.931 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:29.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4720038740 msgr2=0x7f472003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4720038740 0x7f472003abf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4728034000 tx=0x7f4728032000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 10 monclient: shutdown 2024-09-15T03:48:29.933 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 msgr2=0x7f474012d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 0x7f474012d950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f473000c610 tx=0x7f4730007ed0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 shutdown_connections 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f4720038740 0x7f472003abf0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4740097cd0 0x7f4740130950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4740097890 0x7f474012d950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 --2- 172.21.15.181:0/1410609920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f474006e810 0x7f4740141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 >> 172.21.15.181:0/1410609920 conn(0x7f47400a1330 msgr2=0x7f4740134bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 shutdown_connections 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:48:29.910+0000 7f4744c0f640 1 -- 172.21.15.181:0/1410609920 wait complete. 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-15T03:48:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=13 2024-09-15T03:48:29.935 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T03:48:29.935 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T03:48:29.935 INFO:tasks.workunit.client.0.smithi181.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-15T03:48:29.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcba8038770 msgr2=0x7fcba803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcba8038770 0x7fcba803ac20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc11d870 tx=0x7fcbb4009040 comp rx=0 tx=0).stop 2024-09-15T03:48:29.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 10 monclient: shutdown 2024-09-15T03:48:29.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:29.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 msgr2=0x7fcbbc130b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 --2- 172.21.15.153:0/3280236456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc1318f0 0x7fcbbc130b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009210 tx=0x7fcbb0032000 comp rx=0 tx=0).stop 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc0e9d640 1 -- 172.21.15.153:0/3280236456 reap_dead start 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcba3fff640 1 -- 172.21.15.153:0/3280236456 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7fcbb0041030 con 0x7fcbbc1318f0 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc0e9d640 1 -- 172.21.15.153:0/3280236456 reap_dead start 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 shutdown_connections 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 >> 172.21.15.153:0/3280236456 conn(0x7fcbbc067110 msgr2=0x7fcbbc0a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 shutdown_connections 2024-09-15T03:48:29.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:29.883+0000 7fcbc2900640 1 -- 172.21.15.153:0/3280236456 wait complete. 2024-09-15T03:48:29.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:48:29.937 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:48:29.937 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:48:29.937 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:48:29.937 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:48:29.937 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:48:29.937 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:48:29.951 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:48:29.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:48:29.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.083+0000 7fd610268640 10 monclient: get_monmap_and_config 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.083+0000 7fd610268640 10 monclient: build_initial_monmap 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.083+0000 7fd610268640 10 monclient: monmap: 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:30.084310+0000 2024-09-15T03:48:30.084 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:30.084310+0000 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:30.085 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 1 Processor -- start 2024-09-15T03:48:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 1 -- start start 2024-09-15T03:48:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient: init 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd60809c860 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd60809e840 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd608137cc0 0x7fd60813a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient(hunting): picked mon.noname-c con 0x7fd608137cc0 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60809ec40 con 0x7fd60809c860 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60809b2b0 con 0x7fd60809e840 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60806a320 con 0x7fd608137cc0 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd610268640 10 monclient(hunting): authenticate will time out at 1907.723798s 2024-09-15T03:48:30.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.084+0000 7fd60e004640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fd60809c860 legacy=0x7fd60809cc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fd608137cc0 0x7fd60813a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60d803640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fd60809e840 0x7fd60809aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:54612/0 (socket says 172.21.15.153:54612) 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60d803640 1 -- 172.21.15.153:0/3057645961 learned_addr learned my addr 172.21.15.153:0/3057645961 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): get_auth_request con 0x7fd608137cc0 auth_method 0 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient(hunting): handle_auth_done global_id 204478 payload 274 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 1 -- 172.21.15.153:0/3057645961 >> v1:172.21.15.181:6789/0 conn(0x7fd60809e840 legacy=0x7fd60809aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:30.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 1 -- 172.21.15.153:0/3057645961 >> v1:172.21.15.153:6789/0 conn(0x7fd60809c860 legacy=0x7fd60809cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient: _finish_hunting 0 2024-09-15T03:48:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 1 monclient: found mon.noname-c 2024-09-15T03:48:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:30.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.085+0000 7fd60e805640 1 -- 172.21.15.153:0/3057645961 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd608068a20 con 0x7fd608137cc0 2024-09-15T03:48:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60e805640 1 -- 172.21.15.153:0/3057645961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd608137cc0 msgr2=0x7fd60813a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60e805640 1 --2- 172.21.15.153:0/3057645961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd608137cc0 0x7fd60813a0a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd600009850 tx=0x7fd6000090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265924893377553c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 1 -- 172.21.15.153:0/3057645961 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600032440 con 0x7fd608137cc0 2024-09-15T03:48:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:30.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: dump: 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: _finish_auth 0 2024-09-15T03:48:30.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 1 -- 172.21.15.153:0/3057645961 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6000325a0 con 0x7fd608137cc0 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd610268640 5 monclient: authenticate success, global_id 204478 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd610268640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 1 -- 172.21.15.153:0/3057645961 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600032940 con 0x7fd608137cc0 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd610268640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd610268640 4 set_mon_vals no callback set 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.088+0000 7fd60d002640 10 monclient: dump: 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 10 monclient: shutdown 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 -- 172.21.15.153:0/3057645961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd608137cc0 msgr2=0x7fd60813a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 --2- 172.21.15.153:0/3057645961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd608137cc0 0x7fd60813a0a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd600009850 tx=0x7fd6000090f0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 -- 172.21.15.153:0/3057645961 shutdown_connections 2024-09-15T03:48:30.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 --2- 172.21.15.153:0/3057645961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd608137cc0 0x7fd60813a0a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 -- 172.21.15.153:0/3057645961 >> 172.21.15.153:0/3057645961 conn(0x7fd6080a2e40 msgr2=0x7fd6080a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 -- 172.21.15.153:0/3057645961 shutdown_connections 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 1 -- 172.21.15.153:0/3057645961 wait complete. 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 10 monclient: build_initial_monmap 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.089+0000 7fd610268640 10 monclient: monmap: 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:30.099 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 Processor -- start 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 -- start start 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient: init 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:48:30.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd60809c860 0x7fd60806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): picked mon.noname-a con 0x7fd60809c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 0x7fd60806b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): picked mon.noname-b con 0x7fd60809e840 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 0x7fd60806bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): picked mon.noname-c con 0x7fd608137cc0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60806f2d0 con 0x7fd60809c860 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60806dc70 con 0x7fd60809e840 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60809dff0 con 0x7fd608137cc0 2024-09-15T03:48:30.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e004640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd60809c860 msgr2=0x7fd60806ed90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd610268640 10 monclient(hunting): authenticate will time out at 1907.729531s 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e004640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd60809c860 0x7fd60806ed90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 0x7fd60806bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 0x7fd60806bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59682/0 (socket says 172.21.15.153:59682) 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 1 -- 172.21.15.153:0/1067998836 learned_addr learned my addr 172.21.15.153:0/1067998836 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:30.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): get_auth_request con 0x7fd608137cc0 auth_method 0 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 0x7fd60806b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): get_auth_request con 0x7fd60809e840 auth_method 0 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:30.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient(hunting): handle_auth_done global_id 204481 payload 824 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 -- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 msgr2=0x7fd60806bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 0x7fd60806bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 -- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd60809c860 msgr2=0x7fd60806ed90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd60809c860 0x7fd60806ed90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient: _finish_hunting 0 2024-09-15T03:48:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 monclient: found mon.noname-b 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60d803640 1 -- 172.21.15.153:0/1067998836 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd600009390 con 0x7fd60809e840 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.090+0000 7fd60e805640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 0x7fd60806bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd60d803640 1 -- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 msgr2=0x7fd60806b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd60d803640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 0x7fd60806b8b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c4c0 tx=0x7fd5f8007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8005cb0 con 0x7fd60809e840 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: dump: 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f8005e30 con 0x7fd60809e840 2024-09-15T03:48:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 5 monclient: authenticate success, global_id 204481 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 10 monclient: _renew_subs 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60806cd80 con 0x7fd60809e840 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8007320 con 0x7fd60809e840 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd5fe7fc640 10 monclient: dump: 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd60c801640 4 set_mon_vals no callback set 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 10 monclient: _renew_subs 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.093+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd608099330 con 0x7fd60809e840 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd5f801b3b0 con 0x7fd60809e840 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd610268640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd610268640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:30.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd610268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608099d30 con 0x7fd60809e840 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd5fe7fc640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5ec0386f0 0x7fd5ec03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd5f804baa0 con 0x7fd60809e840 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd60e004640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5ec0386f0 0x7fd5ec03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.094+0000 7fd60e004640 10 monclient: get_auth_request con 0x7fd5ec0386f0 auth_method 0 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.097+0000 7fd60e004640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5ec0386f0 0x7fd5ec03aba0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd604003cc0 tx=0x7fd604009310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.098+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd608099d30 con 0x7fd60809e840 2024-09-15T03:48:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.098+0000 7fd5fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.098+0000 7fd5fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:30.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.310+0000 7fd610268640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:30.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.310+0000 7fd610268640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.310+0000 7fd610268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:48:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.310+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd608099d30 con 0x7fd60809e840 2024-09-15T03:48:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.311+0000 7fd5fe7fc640 1 -- 172.21.15.153:0/1067998836 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fd608099d30 con 0x7fd60809e840 2024-09-15T03:48:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.311+0000 7fd5fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:30.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.311+0000 7fd5fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:30.313 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5ec0386f0 msgr2=0x7fd5ec03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5ec0386f0 0x7fd5ec03aba0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd604003cc0 tx=0x7fd604009310 comp rx=0 tx=0).stop 2024-09-15T03:48:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 10 monclient: shutdown 2024-09-15T03:48:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 msgr2=0x7fd60806b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 0x7fd60806b8b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c4c0 tx=0x7fd5f8007f00 comp rx=0 tx=0).stop 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 shutdown_connections 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5ec0386f0 0x7fd5ec03aba0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd608137cc0 0x7fd60806bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd60809e840 0x7fd60806b8b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 --2- 172.21.15.153:0/1067998836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd60809c860 0x7fd60806ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 >> 172.21.15.153:0/1067998836 conn(0x7fd6080a2e40 msgr2=0x7fd608131c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 shutdown_connections 2024-09-15T03:48:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:30.313+0000 7fd610268640 1 -- 172.21.15.153:0/1067998836 wait complete. 2024-09-15T03:48:33.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:33.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.428+0000 7f30fc64f640 10 monclient: get_monmap_and_config 2024-09-15T03:48:33.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.428+0000 7f30fc64f640 10 monclient: build_initial_monmap 2024-09-15T03:48:33.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.428+0000 7f30fc64f640 10 monclient: monmap: 2024-09-15T03:48:33.429 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:33.429 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:33.429351+0000 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:33.429351+0000 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:33.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:33.431 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:33.431 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:33.431 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.429+0000 7f30fc64f640 1 Processor -- start 2024-09-15T03:48:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.429+0000 7f30fc64f640 1 -- start start 2024-09-15T03:48:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.429+0000 7f30fc64f640 10 monclient: init 2024-09-15T03:48:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:33.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 0x7f30f409d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): picked mon.noname-b con 0x7f30f409c9c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30f409e340 0x7f30f409e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): picked mon.noname-a con 0x7f30f409e340 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30f412f770 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f409bdd0 con 0x7f30f412f770 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f409ec90 con 0x7f30f409e340 2024-09-15T03:48:33.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f409a880 con 0x7f30f409c9c0 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fc64f640 10 monclient(hunting): authenticate will time out at 1911.069160s 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30f9bea640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f30f409e340 msgr2=0x7f30f409e750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fabec640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f30f412f770 legacy=0x7f30f412fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30f9bea640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30f409e340 0x7f30f409e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:33.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 0x7f30f409d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 0x7f30f409d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59690/0 (socket says 172.21.15.153:59690) 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 -- 172.21.15.153:0/3229687783 learned_addr learned my addr 172.21.15.153:0/3229687783 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): get_auth_request con 0x7f30f409c9c0 auth_method 0 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:33.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient(hunting): handle_auth_done global_id 244157 payload 274 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 -- 172.21.15.153:0/3229687783 >> v2:172.21.15.153:3300/0 conn(0x7f30f409e340 msgr2=0x7f30f409e750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 --2- 172.21.15.153:0/3229687783 >> v2:172.21.15.153:3300/0 conn(0x7f30f409e340 0x7f30f409e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 -- 172.21.15.153:0/3229687783 >> v1:172.21.15.153:6789/0 conn(0x7f30f412f770 legacy=0x7f30f412fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient: _finish_hunting 0 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 monclient: found mon.noname-b 2024-09-15T03:48:33.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.430+0000 7f30fa3eb640 1 -- 172.21.15.153:0/3229687783 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f409a440 con 0x7f30f409c9c0 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30fa3eb640 1 -- 172.21.15.153:0/3229687783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 msgr2=0x7f30f409d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30fa3eb640 1 --2- 172.21.15.153:0/3229687783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 0x7f30f409d830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009940 tx=0x7f30f00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec040f344cf7c542 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 1 -- 172.21.15.153:0/3229687783 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0032440 con 0x7f30f409c9c0 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:48:33.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: dump: 2024-09-15T03:48:33.444 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:33.444 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:33.444 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.444 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: _finish_auth 0 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 1 -- 172.21.15.153:0/3229687783 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f00325a0 con 0x7f30f409c9c0 2024-09-15T03:48:33.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30fc64f640 5 monclient: authenticate success, global_id 244157 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30fc64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30fc64f640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30fc64f640 4 set_mon_vals no callback set 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 1 -- 172.21.15.153:0/3229687783 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0032940 con 0x7f30f409c9c0 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.433+0000 7f30f93e9640 10 monclient: dump: 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:33.446 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 10 monclient: shutdown 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 -- 172.21.15.153:0/3229687783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 msgr2=0x7f30f409d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:33.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3229687783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 0x7f30f409d830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009940 tx=0x7f30f00091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 -- 172.21.15.153:0/3229687783 shutdown_connections 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3229687783 >> v2:172.21.15.153:3300/0 conn(0x7f30f409e340 0x7f30f409e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3229687783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f409c9c0 0x7f30f409d830 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 -- 172.21.15.153:0/3229687783 >> 172.21.15.153:0/3229687783 conn(0x7f30f40a2e40 msgr2=0x7f30f40a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 -- 172.21.15.153:0/3229687783 shutdown_connections 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 1 -- 172.21.15.153:0/3229687783 wait complete. 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 10 monclient: build_initial_monmap 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.434+0000 7f30fc64f640 10 monclient: monmap: 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 Processor -- start 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 -- start start 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient: init 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:33.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f409c9c0 0x7f30f41417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): picked mon.noname-a con 0x7f30f409c9c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f409e340 0x7f30f4141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): picked mon.noname-b con 0x7f30f409e340 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 0x7f30f4068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): picked mon.noname-c con 0x7f30f412f770 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f4121550 con 0x7f30f409c9c0 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f4121b80 con 0x7f30f409e340 2024-09-15T03:48:33.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f4120250 con 0x7f30f412f770 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fa3eb640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f409c9c0 msgr2=0x7f30f41417d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fc64f640 10 monclient(hunting): authenticate will time out at 1911.074647s 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fa3eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f409c9c0 0x7f30f41417d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 0x7f30f4068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:33.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 0x7f30f4068900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59700/0 (socket says 172.21.15.153:59700) 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 1 -- 172.21.15.153:0/3331646975 learned_addr learned my addr 172.21.15.153:0/3331646975 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): get_auth_request con 0x7f30f412f770 auth_method 0 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.435+0000 7f30fabec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 10 monclient(hunting): handle_auth_done global_id 244160 payload 824 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30f9bea640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f409e340 0x7f30f4141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 1 -- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f409e340 msgr2=0x7f30f4141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:33.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f409e340 0x7f30f4141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 1 -- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f409c9c0 msgr2=0x7f30f41417d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f409c9c0 0x7f30f41417d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 10 monclient: _finish_hunting 0 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 1 monclient: found mon.noname-c 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.436+0000 7f30fabec640 1 -- 172.21.15.153:0/3331646975 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f0009450 con 0x7f30f412f770 2024-09-15T03:48:33.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fabec640 1 -- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 msgr2=0x7f30f4068900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fabec640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 0x7f30f4068900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00bbb0 tx=0x7f30ec0097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec00ccb0 con 0x7f30f412f770 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: dump: 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:33.456 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec00ce10 con 0x7f30f412f770 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 5 monclient: authenticate success, global_id 244160 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 10 monclient: _renew_subs 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f4098e90 con 0x7f30f412f770 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec006150 con 0x7f30f412f770 2024-09-15T03:48:33.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30ea7fc640 10 monclient: dump: 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:33.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30f8be8640 4 set_mon_vals no callback set 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 10 monclient: _renew_subs 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.438+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f4120660 con 0x7f30f412f770 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f30ec012020 con 0x7f30f412f770 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30fc64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30fc64f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30fc64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f404bf60 con 0x7f30f412f770 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30ea7fc640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d8038740 0x7f30d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.439+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f30ec04c5b0 con 0x7f30f412f770 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.440+0000 7f30fa3eb640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d8038740 0x7f30d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:33.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.440+0000 7f30fa3eb640 10 monclient: get_auth_request con 0x7f30d8038740 auth_method 0 2024-09-15T03:48:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.442+0000 7f30fa3eb640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d8038740 0x7f30d803abf0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f30f0034000 tx=0x7f30f0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.444+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f30ec01dae0 con 0x7f30f412f770 2024-09-15T03:48:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.444+0000 7f30ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:33.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.444+0000 7f30ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30fc64f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30fc64f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30fc64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f30f4099ef0 con 0x7f30f412f770 2024-09-15T03:48:33.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30ea7fc640 1 -- 172.21.15.153:0/3331646975 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f30ec01b070 con 0x7f30f412f770 2024-09-15T03:48:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:33.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.657+0000 7f30ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:33.659 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d8038740 msgr2=0x7f30d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d8038740 0x7f30d803abf0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f30f0034000 tx=0x7f30f0032000 comp rx=0 tx=0).stop 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 10 monclient: shutdown 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 msgr2=0x7f30f4068900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 0x7f30f4068900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00bbb0 tx=0x7f30ec0097b0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 shutdown_connections 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f30d8038740 0x7f30d803abf0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30f412f770 0x7f30f4068900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30f409e340 0x7f30f4141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 --2- 172.21.15.153:0/3331646975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30f409c9c0 0x7f30f41417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 >> 172.21.15.153:0/3331646975 conn(0x7f30f40a2e40 msgr2=0x7f30f40977c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 shutdown_connections 2024-09-15T03:48:33.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:33.659+0000 7f30fc64f640 1 -- 172.21.15.153:0/3331646975 wait complete. 2024-09-15T03:48:36.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.776+0000 7fc054d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.776+0000 7fc054d9d640 10 monclient: build_initial_monmap 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.776+0000 7fc054d9d640 10 monclient: monmap: 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:36.777030+0000 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:36.777030+0000 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:36.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:36.778 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:36.778 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:36.778 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.776+0000 7fc054d9d640 1 Processor -- start 2024-09-15T03:48:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.776+0000 7fc054d9d640 1 -- start start 2024-09-15T03:48:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.776+0000 7fc054d9d640 10 monclient: init 2024-09-15T03:48:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-15T03:48:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc050131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc05009e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc050137bc0 0x7fc050139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc050137bc0 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05009faf0 con 0x7fc050131ab0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05009fd30 con 0x7fc05009e910 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc050132590 con 0x7fc050137bc0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc054d9d640 10 monclient(hunting): authenticate will time out at 1914.416636s 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04e59c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fc050131ab0 legacy=0x7fc050131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04ed9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc050137bc0 0x7fc050139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04dd9b640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fc05009e910 0x7fc05009ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:37184/0 (socket says 172.21.15.153:37184) 2024-09-15T03:48:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04dd9b640 1 -- 172.21.15.153:0/2769500443 learned_addr learned my addr 172.21.15.153:0/2769500443 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04ed9d640 1 --2- 172.21.15.153:0/2769500443 >> v2:172.21.15.181:3300/0 conn(0x7fc050137bc0 0x7fc050139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.153:57580/0 (socket says 172.21.15.153:57580) 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04ed9d640 10 monclient(hunting): get_auth_request con 0x7fc050137bc0 auth_method 0 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.777+0000 7fc04ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_done global_id 204490 payload 274 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 1 -- 172.21.15.153:0/2769500443 >> v1:172.21.15.181:6789/0 conn(0x7fc05009e910 legacy=0x7fc05009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 1 -- 172.21.15.153:0/2769500443 >> v1:172.21.15.153:6789/0 conn(0x7fc050131ab0 legacy=0x7fc050131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:48:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:36.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:48:36.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.778+0000 7fc04ed9d640 1 -- 172.21.15.153:0/2769500443 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0500a1110 con 0x7fc050137bc0 2024-09-15T03:48:36.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04ed9d640 1 -- 172.21.15.153:0/2769500443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 msgr2=0x7fc050139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04ed9d640 1 --2- 172.21.15.153:0/2769500443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc050139fa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc040009e90 tx=0x7fc040009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db37dc92f4086cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 1 -- 172.21.15.153:0/2769500443 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040007840 con 0x7fc050137bc0 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: dump: 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:36.792 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: _finish_auth 0 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 1 -- 172.21.15.153:0/2769500443 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04002d050 con 0x7fc050137bc0 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 1 -- 172.21.15.153:0/2769500443 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040007bd0 con 0x7fc050137bc0 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 5 monclient: authenticate success, global_id 204490 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 4 set_mon_vals no callback set 2024-09-15T03:48:36.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc04d59a640 10 monclient: dump: 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:36.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 10 monclient: shutdown 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 1 -- 172.21.15.153:0/2769500443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 msgr2=0x7fc050139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.781+0000 7fc054d9d640 1 --2- 172.21.15.153:0/2769500443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc050139fa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc040009e90 tx=0x7fc040009780 comp rx=0 tx=0).stop 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 -- 172.21.15.153:0/2769500443 shutdown_connections 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 --2- 172.21.15.153:0/2769500443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc050139fa0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 -- 172.21.15.153:0/2769500443 >> 172.21.15.153:0/2769500443 conn(0x7fc050067d30 msgr2=0x7fc0500a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 -- 172.21.15.153:0/2769500443 shutdown_connections 2024-09-15T03:48:36.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 -- 172.21.15.153:0/2769500443 wait complete. 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 10 monclient: build_initial_monmap 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 10 monclient: monmap: 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 Processor -- start 2024-09-15T03:48:36.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 1 -- start start 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.782+0000 7fc054d9d640 10 monclient: init 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc05009e910 0x7fc0501370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc05009e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 0x7fc05012df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc050131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc05012e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc050137bc0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc050130070 con 0x7fc05009e910 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05009bde0 con 0x7fc050137bc0 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04e59c640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc05009e910 msgr2=0x7fc0501370e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500a53b0 con 0x7fc050131ab0 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc054d9d640 10 monclient(hunting): authenticate will time out at 1914.422348s 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc05009e910 0x7fc0501370e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 0x7fc05012df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 0x7fc05012df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54534/0 (socket says 172.21.15.153:54534) 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 -- 172.21.15.153:0/656379370 learned_addr learned my addr 172.21.15.153:0/656379370 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): get_auth_request con 0x7fc050131ab0 auth_method 0 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc05012e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): get_auth_request con 0x7fc050137bc0 auth_method 0 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient(hunting): handle_auth_done global_id 244163 payload 824 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 -- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 msgr2=0x7fc05012e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc05012e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 -- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc05009e910 msgr2=0x7fc0501370e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc05009e910 0x7fc0501370e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 monclient: found mon.noname-c 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.783+0000 7fc04dd9b640 1 -- 172.21.15.153:0/656379370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc040009a20 con 0x7fc050131ab0 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.785+0000 7fc04dd9b640 1 -- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 msgr2=0x7fc05012df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.785+0000 7fc04dd9b640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 0x7fc05012df50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc03800c5e0 tx=0x7fc038004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038005cb0 con 0x7fc050131ab0 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: dump: 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc038005e30 con 0x7fc050131ab0 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 5 monclient: authenticate success, global_id 244163 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 10 monclient: _renew_subs 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0501313e0 con 0x7fc050131ab0 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038007340 con 0x7fc050131ab0 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 10 monclient: dump: 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.803 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc04cd99640 4 set_mon_vals no callback set 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 10 monclient: _renew_subs 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc050130480 con 0x7fc050131ab0 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.786+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc03801b400 con 0x7fc050131ab0 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc054d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:36.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc054d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc054d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05004bf60 con 0x7fc050131ab0 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc0467fc640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc0300386f0 0x7fc03003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc03804c100 con 0x7fc050131ab0 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc04e59c640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc0300386f0 0x7fc03003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.787+0000 7fc04e59c640 10 monclient: get_auth_request con 0x7fc0300386f0 auth_method 0 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.790+0000 7fc04e59c640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc0300386f0 0x7fc03003aba0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc048003940 tx=0x7fc048006d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:36.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.790+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:48:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.792+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc03801a720 con 0x7fc050131ab0 2024-09-15T03:48:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.792+0000 7fc0467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:36.792+0000 7fc0467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:37.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.006+0000 7fc054d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.006+0000 7fc054d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.006+0000 7fc054d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.006+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc050066280 con 0x7fc050131ab0 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.007+0000 7fc0467fc640 1 -- 172.21.15.153:0/656379370 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fc03801a360 con 0x7fc050131ab0 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.007+0000 7fc0467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.007+0000 7fc0467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc0300386f0 msgr2=0x7fc03003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc0300386f0 0x7fc03003aba0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc048003940 tx=0x7fc048006d80 comp rx=0 tx=0).stop 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 10 monclient: shutdown 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 msgr2=0x7fc05012df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 0x7fc05012df50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc03800c5e0 tx=0x7fc038004960 comp rx=0 tx=0).stop 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 shutdown_connections 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc0300386f0 0x7fc03003aba0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc050137bc0 0x7fc05012e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:37.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc050131ab0 0x7fc05012df50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:37.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 --2- 172.21.15.153:0/656379370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc05009e910 0x7fc0501370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:37.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 >> 172.21.15.153:0/656379370 conn(0x7fc050067d30 msgr2=0x7fc05009d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:37.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 shutdown_connections 2024-09-15T03:48:37.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:37.008+0000 7fc054d9d640 1 -- 172.21.15.153:0/656379370 wait complete. 2024-09-15T03:48:40.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.124+0000 7efe00457640 10 monclient: get_monmap_and_config 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.124+0000 7efe00457640 10 monclient: build_initial_monmap 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.124+0000 7efe00457640 10 monclient: monmap: 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:40.125503+0000 2024-09-15T03:48:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:40.125503+0000 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.125+0000 7efe00457640 1 Processor -- start 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.125+0000 7efe00457640 1 -- start start 2024-09-15T03:48:40.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.125+0000 7efe00457640 10 monclient: init 2024-09-15T03:48:40.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:40.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdf809bf50 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient(hunting): picked mon.noname-b con 0x7efdf809c320 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdf8133c00 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf8099610 con 0x7efdf8133c00 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf809d250 con 0x7efdf809bf50 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf8098a40 con 0x7efdf809c320 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efe00457640 10 monclient(hunting): authenticate will time out at 1917.765121s 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfe9f4640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7efdf8133c00 legacy=0x7efdf8135fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54540/0 (socket says 172.21.15.153:54540) 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 learned_addr learned my addr 172.21.15.153:0/400283780 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request con 0x7efdf809c320 auth_method 0 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.126+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:40.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.128+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.181:6789/0 conn(0x7efdf809bf50 legacy=0x7efdf809a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.153:6789/0 conn(0x7efdf8133c00 legacy=0x7efdf8135fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request con 0x7efdf809c320 auth_method 0 2024-09-15T03:48:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:40.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:48:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.327+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:48:40.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.153:6789/0 conn(0x7efdf8133c00 legacy=0x7efdf8135fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:40.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:40.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request con 0x7efdf809c320 auth_method 0 2024-09-15T03:48:40.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:40.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:40.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:40.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:48:40.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:40.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:40.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:40.728+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.153:6789/0 conn(0x7efdf8133c00 legacy=0x7efdf8135fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request con 0x7efdf809c320 auth_method 0 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:41.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:48:41.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:41.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:41.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:41.529+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:48:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): tick 2024-09-15T03:48:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:48:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:48:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 msgr2=0x7efdf809c6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:43.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdf809c320 0x7efdf809c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.181:6789/0 conn(0x7efdf809bf50 legacy=0x7efdf809a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.153:6789/0 conn(0x7efdf8133c00 legacy=0x7efdf8135fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient: _add_conns ranks=[4,3] 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): picked mon.noname-c con 0x7efde4003700 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 0x7efde4008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): picked mon.noname-a con 0x7efde4006260 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf8098a40 con 0x7efde4006260 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:43.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde4008dd0 con 0x7efde4003700 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdf7fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 msgr2=0x7efde4008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 0x7efde4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request con 0x7efde4003700 auth_method 0 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:48:43.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:43.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:43.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.126+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:43.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 msgr2=0x7efde4008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:43.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 0x7efde4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:48:43.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:43.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request con 0x7efde4003700 auth_method 0 2024-09-15T03:48:43.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:43.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:43.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:48:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:43.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.327+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:48:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 msgr2=0x7efde4008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 0x7efde4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:48:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request con 0x7efde4003700 auth_method 0 2024-09-15T03:48:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:43.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:43.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:48:43.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:43.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:43.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:43.728+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:48:44.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.529+0000 7efdfd9f2640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 msgr2=0x7efde4008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.529+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 0x7efde4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:48:44.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.529+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.529+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request con 0x7efde4003700 auth_method 0 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.529+0000 7efdfe1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.530+0000 7efdfe1f3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.530+0000 7efdfe1f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.530+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.530+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.530+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:48:44.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:44.530+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:48:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): tick 2024-09-15T03:48:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 msgr2=0x7efde4005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.181:3300/0 conn(0x7efde4003700 0x7efde4005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 msgr2=0x7efde4008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde4006260 0x7efde4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 reap_dead start 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde40091a0 0x7efde400b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): picked mon.noname-a con 0x7efde40091a0 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdf809bf50 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 0x7efde4005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): picked mon.noname-b con 0x7efde4008710 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:46.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf809a770 con 0x7efdf809bf50 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf8098a40 con 0x7efde40091a0 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 1 -- 172.21.15.153:0/400283780 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde4008dd0 con 0x7efde4008710 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdf7fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe1f3640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde40091a0 msgr2=0x7efde400b580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde40091a0 0x7efde400b580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 0x7efde4005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): get_auth_request con 0x7efde4008710 auth_method 0 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.126+0000 7efdfe9f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 10 monclient(hunting): handle_auth_done global_id 244169 payload 274 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde40091a0 msgr2=0x7efde400b580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde40091a0 0x7efde400b580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 >> v1:172.21.15.181:6789/0 conn(0x7efdf809bf50 legacy=0x7efde400bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 1 monclient: found mon.noname-b 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.127+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8099610 con 0x7efde4008710 2024-09-15T03:48:46.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfe9f4640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 msgr2=0x7efde4005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfe9f4640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 0x7efde4005b70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdf8069f80 tx=0x7efde80076b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecfd703e31efa41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 1 -- 172.21.15.153:0/400283780 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde802a9f0 con 0x7efde4008710 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: dump: 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efe00457640 5 monclient: authenticate success, global_id 244169 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efe00457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:46.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 1 -- 172.21.15.153:0/400283780 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8009030 con 0x7efde4008710 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efe00457640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efe00457640 4 set_mon_vals no callback set 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 1 -- 172.21.15.153:0/400283780 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde802ada0 con 0x7efde4008710 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.130+0000 7efdfd1f1640 10 monclient: dump: 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.139 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 10 monclient: shutdown 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 -- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 msgr2=0x7efde4005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 0x7efde4005b70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdf8069f80 tx=0x7efde80076b0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 -- 172.21.15.153:0/400283780 shutdown_connections 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 --2- 172.21.15.153:0/400283780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde4008710 0x7efde4005b70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 --2- 172.21.15.153:0/400283780 >> v2:172.21.15.153:3300/0 conn(0x7efde40091a0 0x7efde400b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 -- 172.21.15.153:0/400283780 >> 172.21.15.153:0/400283780 conn(0x7efdf80672b0 msgr2=0x7efdf80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 -- 172.21.15.153:0/400283780 shutdown_connections 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 1 -- 172.21.15.153:0/400283780 wait complete. 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 10 monclient: build_initial_monmap 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.131+0000 7efe00457640 10 monclient: monmap: 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:46.141 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 Processor -- start 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 -- start start 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient: init 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde4008710 0x7efdf8141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): picked mon.noname-a con 0x7efde4008710 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 0x7efdf8132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): picked mon.noname-c con 0x7efde40091a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdf809bf50 0x7efdf812bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): picked mon.noname-b con 0x7efdf809bf50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf80a2230 con 0x7efde4008710 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf8142080 con 0x7efdf809bf50 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf812f030 con 0x7efde40091a0 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efe00457640 10 monclient(hunting): authenticate will time out at 1923.771954s 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efdfe1f3640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde4008710 msgr2=0x7efdf8141180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.132+0000 7efdfe1f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde4008710 0x7efdf8141180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 0x7efdf8132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 0x7efdf8132b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40510/0 (socket says 172.21.15.153:40510) 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 -- 172.21.15.153:0/1671427735 learned_addr learned my addr 172.21.15.153:0/1671427735 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request con 0x7efde40091a0 auth_method 0 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfe9f4640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdf809bf50 0x7efdf812bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfe9f4640 10 monclient(hunting): get_auth_request con 0x7efdf809bf50 auth_method 0 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfe9f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfe9f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfe9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient(hunting): handle_auth_done global_id 244172 payload 824 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 -- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdf809bf50 msgr2=0x7efdf812bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdf809bf50 0x7efdf812bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 -- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde4008710 msgr2=0x7efdf8141180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde4008710 0x7efdf8141180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 monclient: found mon.noname-c 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfd9f2640 1 -- 172.21.15.153:0/1671427735 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde800b040 con 0x7efde40091a0 2024-09-15T03:48:46.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.133+0000 7efdfe9f4640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdf809bf50 0x7efdf812bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:48:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.135+0000 7efdfd9f2640 1 -- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 msgr2=0x7efdf8132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.135+0000 7efdfd9f2640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 0x7efdf8132b40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efdec011b50 tx=0x7efdec012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec011e50 con 0x7efde40091a0 2024-09-15T03:48:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: dump: 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec0122b0 con 0x7efde40091a0 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 5 monclient: authenticate success, global_id 244172 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec00ee00 con 0x7efde40091a0 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdf67fc640 10 monclient: dump: 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.150 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 10 monclient: _renew_subs 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efdfc9f0640 4 set_mon_vals no callback set 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf812edb0 con 0x7efde40091a0 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 10 monclient: _renew_subs 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.136+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf8133080 con 0x7efde40091a0 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7efdec01b3e0 con 0x7efde40091a0 2024-09-15T03:48:46.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efe00457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efe00457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efe00457640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf804bf60 con 0x7efde40091a0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efdf67fc640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efde4039ef0 0x7efde403c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.137+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efdec04d290 con 0x7efde40091a0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.138+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efde4039ef0 0x7efde403c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.138+0000 7efdfe1f3640 10 monclient: get_auth_request con 0x7efde4039ef0 auth_method 0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.140+0000 7efdfe1f3640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efde4039ef0 0x7efde403c3a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000f60 tx=0x7efdf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.142+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efdec00fd80 con 0x7efde40091a0 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.142+0000 7efdf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.142+0000 7efdf67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:46.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.363+0000 7efe00457640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:48:46.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.363+0000 7efe00457640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:46.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.363+0000 7efe00457640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.363+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efdf812f170 con 0x7efde40091a0 2024-09-15T03:48:46.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.364+0000 7efdf67fc640 1 -- 172.21.15.153:0/1671427735 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7efdf812f170 con 0x7efde40091a0 2024-09-15T03:48:46.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.364+0000 7efdf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:48:46.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.364+0000 7efdf67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:46.365 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":208,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efde4039ef0 msgr2=0x7efde403c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efde4039ef0 0x7efde403c3a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000f60 tx=0x7efdf0009040 comp rx=0 tx=0).stop 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 10 monclient: shutdown 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 msgr2=0x7efdf8132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 0x7efdf8132b40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efdec011b50 tx=0x7efdec012040 comp rx=0 tx=0).stop 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 shutdown_connections 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7efde4039ef0 0x7efde403c3a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdf809bf50 0x7efdf812bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde40091a0 0x7efdf8132b40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 --2- 172.21.15.153:0/1671427735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde4008710 0x7efdf8141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 >> 172.21.15.153:0/1671427735 conn(0x7efdf80672b0 msgr2=0x7efdf80a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:46.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 shutdown_connections 2024-09-15T03:48:46.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.366+0000 7efe00457640 1 -- 172.21.15.153:0/1671427735 wait complete. 2024-09-15T03:48:46.377 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:48:46.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:48:46.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.472+0000 7f9d06f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.472+0000 7f9d06f1d640 10 monclient: build_initial_monmap 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.472+0000 7f9d06f1d640 10 monclient: monmap: 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:46.473045+0000 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:46.473045+0000 2024-09-15T03:48:46.473 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:46.474 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 Processor -- start 2024-09-15T03:48:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 -- start start 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient: init 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9d000981c0 0x7f9d00098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d000981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 0x7f9d00067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d00131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f9d00132170 0x7f9d00067650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d00132170 addr v2:172.21.15.181:3300/0 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d00131bd0 con 0x7f9d000981c0 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d00098ad0 con 0x7f9d00132170 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0009ca60 con 0x7f9d00131da0 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d06f1d640 10 monclient(hunting): authenticate will time out at 1924.112593s 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d04cb9640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f9d000981c0 msgr2=0x7f9d00098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9d04cb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9d000981c0 0x7f9d00098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 0x7f9d00067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 0x7f9d00067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40514/0 (socket says 172.21.15.153:40514) 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 -- 172.21.15.153:0/1255125686 learned_addr learned my addr 172.21.15.153:0/1255125686 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:46.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): get_auth_request con 0x7f9d00131da0 auth_method 0 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient(hunting): handle_auth_done global_id 244175 payload 274 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 -- 172.21.15.153:0/1255125686 >> v2:172.21.15.181:3300/0 conn(0x7f9d00132170 msgr2=0x7f9d00067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1255125686 >> v2:172.21.15.181:3300/0 conn(0x7f9d00132170 0x7f9d00067650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 -- 172.21.15.153:0/1255125686 >> v2:172.21.15.153:3300/0 conn(0x7f9d000981c0 msgr2=0x7f9d00098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1255125686 >> v2:172.21.15.153:3300/0 conn(0x7f9d000981c0 0x7f9d00098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 monclient: found mon.noname-b 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.473+0000 7f9cfffff640 1 -- 172.21.15.153:0/1255125686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0009cca0 con 0x7f9d00131da0 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.476+0000 7f9cfffff640 1 -- 172.21.15.153:0/1255125686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 msgr2=0x7f9d00067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.476+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1255125686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 0x7f9d00067110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d0009c940 tx=0x7f9cec0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f50c961e49c0e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 1 -- 172.21.15.153:0/1255125686 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec032440 con 0x7f9d00131da0 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: dump: 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 1 -- 172.21.15.153:0/1255125686 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec0325a0 con 0x7f9d00131da0 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 5 monclient: authenticate success, global_id 244175 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:46.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 1 -- 172.21.15.153:0/1255125686 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec032940 con 0x7f9d00131da0 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 4 set_mon_vals no callback set 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9cff7fe640 10 monclient: dump: 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.487 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 10 monclient: shutdown 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1255125686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 msgr2=0x7f9d00067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1255125686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 0x7f9d00067110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d0009c940 tx=0x7f9cec0091d0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1255125686 shutdown_connections 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1255125686 >> v2:172.21.15.181:3300/0 conn(0x7f9d00132170 0x7f9d00067650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1255125686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d00131da0 0x7f9d00067110 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1255125686 >> v2:172.21.15.153:3300/0 conn(0x7f9d000981c0 0x7f9d00098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1255125686 >> 172.21.15.153:0/1255125686 conn(0x7f9d000a1e30 msgr2=0x7f9d000a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1255125686 shutdown_connections 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.477+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1255125686 wait complete. 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.478+0000 7f9d06f1d640 10 monclient: build_initial_monmap 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.478+0000 7f9d06f1d640 10 monclient: monmap: 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:46.489 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.478+0000 7f9d06f1d640 1 Processor -- start 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.478+0000 7f9d06f1d640 1 -- start start 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.478+0000 7f9d06f1d640 10 monclient: init 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 0x7f9d0011de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d000981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d00131da0 0x7f9d0011e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d00131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d00132170 0x7f9d0011e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d00132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00098cb0 con 0x7f9d00131da0 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d000a07f0 con 0x7f9d00132170 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00121500 con 0x7f9d000981c0 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d06f1d640 10 monclient(hunting): authenticate will time out at 1924.118100s 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9cfffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d00131da0 msgr2=0x7f9d0011e3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9cfffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d00131da0 0x7f9d0011e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 0x7f9d0011de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 0x7f9d0011de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40520/0 (socket says 172.21.15.153:40520) 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 -- 172.21.15.153:0/1834125972 learned_addr learned my addr 172.21.15.153:0/1834125972 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): get_auth_request con 0x7f9d000981c0 auth_method 0 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d054ba640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d00132170 0x7f9d0011e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d054ba640 10 monclient(hunting): get_auth_request con 0x7f9d00132170 auth_method 0 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d054ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d054ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d054ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient(hunting): handle_auth_done global_id 244178 payload 824 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d00132170 msgr2=0x7f9d0011e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d00132170 0x7f9d0011e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d00131da0 msgr2=0x7f9d0011e3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:46.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d00131da0 0x7f9d0011e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 monclient: found mon.noname-c 2024-09-15T03:48:46.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.479+0000 7f9d04cb9640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec009450 con 0x7f9d000981c0 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d04cb9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 msgr2=0x7f9d0011de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d04cb9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 0x7f9d0011de80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400b5c0 tx=0x7f9cf4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4003d90 con 0x7f9d000981c0 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: dump: 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.503 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4006150 con 0x7f9d000981c0 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4006510 con 0x7f9d000981c0 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 5 monclient: authenticate success, global_id 244178 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 10 monclient: _renew_subs 2024-09-15T03:48:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0011f7c0 con 0x7f9d000981c0 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfeffd640 4 set_mon_vals no callback set 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9cfcff9640 10 monclient: dump: 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.505 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 10 monclient: _renew_subs 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.481+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d00131450 con 0x7f9d000981c0 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9d06f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 0x7f9d0011a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00065f90 con 0x7f9d0011ee40 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9cf40103c0 con 0x7f9d000981c0 2024-09-15T03:48:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 0x7f9d0011a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient: get_auth_request con 0x7f9d0011ee40 auth_method 0 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.482+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.483+0000 7f9cfcff9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ce40386f0 0x7f9ce403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.483+0000 7f9cfffff640 10 monclient(hunting): handle_auth_done global_id 244181 payload 824 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.483+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9cf404bd20 con 0x7f9d000981c0 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.483+0000 7f9cfffff640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d0004bf60 con 0x7f9d0011ee40 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.483+0000 7f9d054ba640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ce40386f0 0x7f9ce403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.483+0000 7f9d054ba640 10 monclient: get_auth_request con 0x7f9ce40386f0 auth_method 0 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfffff640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 msgr2=0x7f9d0011a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 0x7f9d0011a4a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d00066560 tx=0x7f9cec032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec032440 con 0x7f9d0011ee40 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 10 monclient: dump: 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.508 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9cec006e80 con 0x7f9d0011ee40 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 msgr2=0x7f9d0011a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 0x7f9d0011a4a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d00066560 tx=0x7f9cec032000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 msgr2=0x7f9d0011a4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9cfcff9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011ee40 0x7f9d0011a4a0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d00066560 tx=0x7f9cec032000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.485+0000 7f9d054ba640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ce40386f0 0x7f9ce403aba0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8001250 tx=0x7f9cf8009620 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9d06f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 0x7f9d0011ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9d06f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00098ee0 con 0x7f9d0011a9e0 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 0x7f9d0011ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient: get_auth_request con 0x7f9d0011a9e0 auth_method 0 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 10 monclient(hunting): handle_auth_done global_id 244184 payload 824 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.491+0000 7f9cfffff640 1 -- 172.21.15.153:0/1834125972 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d000638d0 con 0x7f9d0011a9e0 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfffff640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 msgr2=0x7f9d0011ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfffff640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 0x7f9d0011ade0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009210 tx=0x7f9cec03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec046670 con 0x7f9d0011a9e0 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 10 monclient: dump: 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.511 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f9cec0467d0 con 0x7f9d0011a9e0 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 msgr2=0x7f9d0011ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 0x7f9d0011ade0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009210 tx=0x7f9cec03d000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 msgr2=0x7f9d0011ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9cfcff9640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d0011a9e0 0x7f9d0011ade0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009210 tx=0x7f9cec03d000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 208, 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:46.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:46.514 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:46.515 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:46.517 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.518 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:46.519 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:46.520 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.521 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:46.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:46.523 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ce40386f0 msgr2=0x7f9ce403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9ce40386f0 0x7f9ce403aba0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8001250 tx=0x7f9cf8009620 comp rx=0 tx=0).stop 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 10 monclient: shutdown 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 msgr2=0x7f9d0011de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 --2- 172.21.15.153:0/1834125972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d000981c0 0x7f9d0011de80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400b5c0 tx=0x7f9cf4009120 comp rx=0 tx=0).stop 2024-09-15T03:48:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d054ba640 1 -- 172.21.15.153:0/1834125972 reap_dead start 2024-09-15T03:48:46.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d054ba640 1 -- 172.21.15.153:0/1834125972 reap_dead start 2024-09-15T03:48:46.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 shutdown_connections 2024-09-15T03:48:46.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 >> 172.21.15.153:0/1834125972 conn(0x7f9d000a1e30 msgr2=0x7f9d0009b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:46.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 shutdown_connections 2024-09-15T03:48:46.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.494+0000 7f9d06f1d640 1 -- 172.21.15.153:0/1834125972 wait complete. 2024-09-15T03:48:46.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:48:46.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:48:46.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:48:46.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.622+0000 7fbb164af640 10 monclient: get_monmap_and_config 2024-09-15T03:48:46.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.622+0000 7fbb164af640 10 monclient: build_initial_monmap 2024-09-15T03:48:46.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.622+0000 7fbb164af640 10 monclient: monmap: 2024-09-15T03:48:46.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:46.623 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:48:46.623326+0000 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:48:46.623326+0000 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:48:46.624 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 1 Processor -- start 2024-09-15T03:48:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 1 -- start start 2024-09-15T03:48:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient: init 2024-09-15T03:48:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-15T03:48:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbb10067110 0x7fbb1012f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient(hunting): picked mon.noname-a con 0x7fbb10067110 addr v2:172.21.15.153:3300/0 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb1009b010 addr v1:172.21.15.181:6789/0 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb1009b430 addr v1:172.21.15.153:6789/0 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb10130540 con 0x7fbb1009b430 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb0ffff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7fbb10067110 msgr2=0x7fbb1012f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb10099800 con 0x7fbb1009b010 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb10067ed0 con 0x7fbb10067110 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb0ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbb10067110 0x7fbb1012f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb164af640 10 monclient(hunting): authenticate will time out at 1924.262932s 2024-09-15T03:48:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.623+0000 7fbb14a4c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fbb1009b430 legacy=0x7fbb1009cf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-15T03:48:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.624+0000 7fbb0f7fe640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fbb1009b010 0x7fbb1009c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.153:50240/0 (socket says 172.21.15.153:50240) 2024-09-15T03:48:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.624+0000 7fbb0f7fe640 1 -- 172.21.15.153:0/110426253 learned_addr learned my addr 172.21.15.153:0/110426253 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:48:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0f7fe640 1 -- 172.21.15.153:0/110426253 >> v1:172.21.15.181:6789/0 conn(0x7fbb1009b010 legacy=0x7fbb1009c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126617448 0 0) 0x7fbb10099800 con 0x7fbb1009b010 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient(hunting): my global_id is 204505 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbafc0034e0 con 0x7fbb1009b010 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3259313753 0 0) 0x7fbafc0034e0 con 0x7fbb1009b010 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 >> v2:172.21.15.153:3300/0 conn(0x7fbb10067110 msgr2=0x7fbb1012f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 --2- 172.21.15.153:0/110426253 >> v2:172.21.15.153:3300/0 conn(0x7fbb10067110 0x7fbb1012f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 >> v1:172.21.15.153:6789/0 conn(0x7fbb1009b430 legacy=0x7fbb1009cf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.181:6789/0 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb100674e0 con 0x7fbb1009b010 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fbaf80017e0 con 0x7fbb1009b010 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb164af640 5 monclient: authenticate success, global_id 204505 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb164af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: dump: 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.644 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] but i'm connected to v1:172.21.15.181:6789/0, reconnecting 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 >> v1:172.21.15.181:6789/0 conn(0x7fbb1009b010 legacy=0x7fbb1009c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.626+0000 7fbb0effd640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc0039d0 0x7fbafc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): picked mon.b con 0x7fbafc0039d0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbafc004780 0x7fbafc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): picked mon.a con 0x7fbafc004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:46.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 0x7fbafc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): picked mon.c con 0x7fbafc005280 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc00c210 con 0x7fbafc004780 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc00c380 con 0x7fbafc0039d0 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc00c540 con 0x7fbafc005280 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb164af640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0f7fe640 1 -- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbafc004780 msgr2=0x7fbafc004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0f7fe640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbafc004780 0x7fbafc004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 0x7fbafc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): get_auth_request con 0x7fbafc005280 auth_method 0 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc0039d0 0x7fbafc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): get_auth_request con 0x7fbafc0039d0 auth_method 0 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_done global_id 204505 payload 293 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 -- 172.21.15.153:0/110426253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc0039d0 msgr2=0x7fbafc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:48:46.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc0039d0 0x7fbafc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 -- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbafc004780 msgr2=0x7fbafc004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbafc004780 0x7fbafc004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 monclient: found mon.c 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 1 -- 172.21.15.153:0/110426253 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0034e0 con 0x7fbafc005280 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.627+0000 7fbb14a4c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.628+0000 7fbb14a4c640 1 -- 172.21.15.153:0/110426253 reap_dead start 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb14a4c640 1 -- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 msgr2=0x7fbafc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 0x7fbafc00b9d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009f90 tx=0x7fbb080095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7021eea646efeae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08032430 con 0x7fbafc005280 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: dump: 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0802d070 con 0x7fbafc005280 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 10 monclient: get_monmap_and_config success 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 4 set_mon_vals no callback set 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 1 -- 172.21.15.153:0/110426253 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08032820 con 0x7fbafc005280 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb0effd640 10 monclient: dump: 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.651 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 10 monclient: shutdown 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 1 -- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 msgr2=0x7fbafc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 0x7fbafc00b9d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009f90 tx=0x7fbb080095d0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 1 -- 172.21.15.153:0/110426253 shutdown_connections 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 1 --2- 172.21.15.153:0/110426253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbafc005280 0x7fbafc00b9d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 1 -- 172.21.15.153:0/110426253 >> 172.21.15.153:0/110426253 conn(0x7fbb100a1e30 msgr2=0x7fbb100a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:46.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.630+0000 7fbb164af640 1 -- 172.21.15.153:0/110426253 shutdown_connections 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.631+0000 7fbb164af640 1 -- 172.21.15.153:0/110426253 wait complete. 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.631+0000 7fbb164af640 10 monclient: build_initial_monmap 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.631+0000 7fbb164af640 10 monclient: monmap: 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:48:46.653 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.631+0000 7fbb164af640 1 Processor -- start 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.631+0000 7fbb164af640 1 -- start start 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.631+0000 7fbb164af640 10 monclient: init 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient: _reopen_session rank -1 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc005280 0x7fbb10140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): picked mon.noname-b con 0x7fbafc005280 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb1012e850 0x7fbb1012ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): picked mon.noname-a con 0x7fbb1012e850 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 0x7fbb10096ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): picked mon.noname-c con 0x7fbb10096a60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb100a0090 con 0x7fbb1012e850 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1012f160 con 0x7fbafc005280 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1011dc70 con 0x7fbb10096a60 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): _renew_subs 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb164af640 10 monclient(hunting): authenticate will time out at 1924.271446s 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0f7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb1012e850 msgr2=0x7fbb1012ec20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb1012e850 0x7fbb1012ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 0x7fbb10096ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 0x7fbb10096ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40548/0 (socket says 172.21.15.153:40548) 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 learned_addr learned my addr 172.21.15.153:0/1977407321 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): get_auth_request con 0x7fbb10096a60 auth_method 0 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc005280 0x7fbb10140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0ffff640 10 monclient(hunting): get_auth_request con 0x7fbafc005280 auth_method 0 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient(hunting): handle_auth_done global_id 244187 payload 824 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc005280 msgr2=0x7fbb10140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc005280 0x7fbb10140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb1012e850 msgr2=0x7fbb1012ec20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb1012e850 0x7fbb1012ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient: _finish_hunting 0 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 monclient: found mon.noname-c 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1011d990 con 0x7fbb10096a60 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.632+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbafc005280 0x7fbb10140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 msgr2=0x7fbb10096ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb14a4c640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 0x7fbb10096ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb0802a7b0 tx=0x7fbb08035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb080052e0 con 0x7fbb10096a60 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: dump: 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: _finish_auth 0 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb08005460 con 0x7fbb10096a60 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08005820 con 0x7fbb10096a60 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 5 monclient: authenticate success, global_id 244187 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb07fff640 10 monclient: dump: 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.659 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 10 monclient: _renew_subs 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb100981e0 con 0x7fbb10096a60 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb0e7fc640 4 set_mon_vals no callback set 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 10 monclient: _renew_subs 2024-09-15T03:48:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.635+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb10141380 con 0x7fbb10096a60 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb164af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb164af640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 0x7fbb1006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb164af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb10065f90 con 0x7fbb10097410 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbb08039070 con 0x7fbb10096a60 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 0x7fbb1006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient: get_auth_request con 0x7fbb10097410 auth_method 0 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb07fff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbaf00386f0 0x7fbaf003aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbb08006e40 con 0x7fbb10096a60 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0f7fe640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbaf00386f0 0x7fbaf003aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0f7fe640 10 monclient: get_auth_request con 0x7fbaf00386f0 auth_method 0 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_done global_id 204511 payload 824 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.636+0000 7fbb0ffff640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1011dde0 con 0x7fbb10097410 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.638+0000 7fbb0ffff640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 msgr2=0x7fbb1006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.638+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 0x7fbb1006aa40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009580 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb0f7fe640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbaf00386f0 0x7fbaf003aba0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8001490 tx=0x7fbaf8001e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00015980 con 0x7fbb10097410 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 10 monclient: dump: 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbb00015ae0 con 0x7fbb10097410 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 msgr2=0x7fbb1006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 0x7fbb1006aa40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009580 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 msgr2=0x7fbb1006aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.639+0000 7fbb07fff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb10097410 0x7fbb1006aa40 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009580 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb164af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb164af640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 0x7fbb1006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb164af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1009e3c0 con 0x7fbb1006af80 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 0x7fbb1006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient: get_auth_request con 0x7fbb1006af80 auth_method 0 2024-09-15T03:48:46.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 10 monclient(hunting): handle_auth_done global_id 204514 payload 824 2024-09-15T03:48:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.645+0000 7fbb0ffff640 1 -- 172.21.15.153:0/1977407321 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb1004bf60 con 0x7fbb1006af80 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.647+0000 7fbb0ffff640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 msgr2=0x7fbb1006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.647+0000 7fbb0ffff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 0x7fbb1006b380 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009580 tx=0x7fbb0001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00005e10 con 0x7fbb1006af80 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 10 monclient: dump: 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:48:46.669 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7fbb00006260 con 0x7fbb1006af80 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:48:46.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 msgr2=0x7fbb1006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 0x7fbb1006b380 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009580 tx=0x7fbb0001b000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 msgr2=0x7fbb1006b380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb07fff640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1006af80 0x7fbb1006b380 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009580 tx=0x7fbb0001b000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:48:46.671 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 208, 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:48:46.672 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:46.673 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:46.674 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:48:46.675 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:46.676 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.677 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:48:46.680 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:48:46.681 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:48:46.682 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:46.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:48:46.685 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbaf00386f0 msgr2=0x7fbaf003aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbaf00386f0 0x7fbaf003aba0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8001490 tx=0x7fbaf8001e90 comp rx=0 tx=0).stop 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 10 monclient: shutdown 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 msgr2=0x7fbb10096ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 --2- 172.21.15.153:0/1977407321 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb10096a60 0x7fbb10096ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb0802a7b0 tx=0x7fbb08035000 comp rx=0 tx=0).stop 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 reap_dead start 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb14a4c640 1 -- 172.21.15.153:0/1977407321 reap_dead start 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 shutdown_connections 2024-09-15T03:48:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 >> 172.21.15.153:0/1977407321 conn(0x7fbb100a1e30 msgr2=0x7fbb10132700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:48:46.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 shutdown_connections 2024-09-15T03:48:46.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:48:46.648+0000 7fbb164af640 1 -- 172.21.15.153:0/1977407321 wait complete. 2024-09-15T03:48:46.688 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:48:46.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:49:06.689 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:49:06.690 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:49:06.690 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:49:06.690 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:49:06.693 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:49:06.693 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:49:06.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.796+0000 7f3172e6f640 10 monclient: get_monmap_and_config 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.796+0000 7f3172e6f640 10 monclient: build_initial_monmap 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.796+0000 7f3172e6f640 10 monclient: monmap: 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:06.798229+0000 2024-09-15T03:49:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:06.798229+0000 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:06.799 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:06.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 Processor -- start 2024-09-15T03:49:06.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 -- start start 2024-09-15T03:49:06.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient: init 2024-09-15T03:49:06.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:06.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:49:06.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): picked mon.noname-a con 0x7f316c09c860 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:06.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 0x7f316c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f316c09aba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:06.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f316c09afa0 0x7f316c09dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): picked mon.noname-c con 0x7f316c09afa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:06.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:06.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c09a9d0 con 0x7f316c09c860 2024-09-15T03:49:06.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:06.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c06a320 con 0x7f316c09afa0 2024-09-15T03:49:06.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:06.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c068a20 con 0x7f316c09aba0 2024-09-15T03:49:06.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:06.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3172e6f640 10 monclient(hunting): authenticate will time out at 1944.437838s 2024-09-15T03:49:06.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3170c0b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f316bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 0x7f316c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f316bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 0x7f316c09d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54124/0 (socket says 172.21.15.153:54124) 2024-09-15T03:49:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.797+0000 7f3170c0b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60646/0 (socket says 172.21.15.153:60646) 2024-09-15T03:49:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 -- 172.21.15.153:0/4240292412 learned_addr learned my addr 172.21.15.153:0/4240292412 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): get_auth_request con 0x7f316c09aba0 auth_method 0 2024-09-15T03:49:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:06.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:06.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 10 monclient(hunting): get_auth_request con 0x7f316c09c860 auth_method 0 2024-09-15T03:49:06.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:06.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:06.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:06.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:06.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:06.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 1 -- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 msgr2=0x7f316c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:49:06.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 1 -- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 msgr2=0x7f316c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:06.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:06.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:06.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f3170c0b640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:06.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f317140c640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.181:3300/0 conn(0x7f316c09afa0 0x7f316c09dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f317140c640 10 monclient(hunting): get_auth_request con 0x7f316c09afa0 auth_method 0 2024-09-15T03:49:06.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f317140c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:06.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f317140c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:06.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f317140c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:06.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient(hunting): handle_auth_done global_id 244202 payload 274 2024-09-15T03:49:06.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 -- 172.21.15.153:0/4240292412 >> v2:172.21.15.181:3300/0 conn(0x7f316c09afa0 msgr2=0x7f316c09dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:06.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.181:3300/0 conn(0x7f316c09afa0 0x7f316c09dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 -- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 msgr2=0x7f316c09cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:06.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:06.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 monclient: found mon.noname-b 2024-09-15T03:49:06.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:06.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:49:06.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.798+0000 7f316bfff640 1 -- 172.21.15.153:0/4240292412 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c068e60 con 0x7f316c09aba0 2024-09-15T03:49:06.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316bfff640 1 -- 172.21.15.153:0/4240292412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 msgr2=0x7f316c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:06.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316bfff640 1 --2- 172.21.15.153:0/4240292412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 0x7f316c09d830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f316c09a0a0 tx=0x7f31580091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a03083c522fef2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:06.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 1 -- 172.21.15.153:0/4240292412 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158032440 con 0x7f316c09aba0 2024-09-15T03:49:06.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:06.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:06.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: dump: 2024-09-15T03:49:06.824 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:06.824 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:06.825 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.825 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.825 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:06.825 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:06.825 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:06.825 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:06.826 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:06.826 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:06.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:49:06.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:06.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 5 monclient: authenticate success, global_id 244202 2024-09-15T03:49:06.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 1 -- 172.21.15.153:0/4240292412 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31580325a0 con 0x7f316c09aba0 2024-09-15T03:49:06.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:06.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:06.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:06.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 4 set_mon_vals no callback set 2024-09-15T03:49:06.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 1 -- 172.21.15.153:0/4240292412 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158032940 con 0x7f316c09aba0 2024-09-15T03:49:06.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f316b7fe640 10 monclient: dump: 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:06.829 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 10 monclient: shutdown 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 -- 172.21.15.153:0/4240292412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 msgr2=0x7f316c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 --2- 172.21.15.153:0/4240292412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 0x7f316c09d830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f316c09a0a0 tx=0x7f31580091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 -- 172.21.15.153:0/4240292412 shutdown_connections 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.181:3300/0 conn(0x7f316c09afa0 0x7f316c09dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 --2- 172.21.15.153:0/4240292412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09aba0 0x7f316c09d830 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 --2- 172.21.15.153:0/4240292412 >> v2:172.21.15.153:3300/0 conn(0x7f316c09c860 0x7f316c09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 -- 172.21.15.153:0/4240292412 >> 172.21.15.153:0/4240292412 conn(0x7f316c0a2e40 msgr2=0x7f316c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 -- 172.21.15.153:0/4240292412 shutdown_connections 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.801+0000 7f3172e6f640 1 -- 172.21.15.153:0/4240292412 wait complete. 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.802+0000 7f3172e6f640 10 monclient: build_initial_monmap 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.802+0000 7f3172e6f640 10 monclient: monmap: 2024-09-15T03:49:06.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.802+0000 7f3172e6f640 1 Processor -- start 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.802+0000 7f3172e6f640 1 -- start start 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.802+0000 7f3172e6f640 10 monclient: init 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:06.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f316c09aba0 0x7f316c1410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f316c09aba0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 0x7f316c133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): picked mon.noname-c con 0x7f316c09afa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): picked mon.noname-a con 0x7f316c09c860 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c121370 con 0x7f316c09c860 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c099cc0 con 0x7f316c09aba0 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c141f60 con 0x7f316c09afa0 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:06.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3172e6f640 10 monclient(hunting): authenticate will time out at 1944.443397s 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 0x7f316c133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 0x7f316c133c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54126/0 (socket says 172.21.15.153:54126) 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60654/0 (socket says 172.21.15.153:60654) 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 -- 172.21.15.153:0/1844164549 learned_addr learned my addr 172.21.15.153:0/1844164549 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): get_auth_request con 0x7f316c09afa0 auth_method 0 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 10 monclient(hunting): get_auth_request con 0x7f316c09c860 auth_method 0 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:06.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f316c09aba0 0x7f316c1410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 msgr2=0x7f316c11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 msgr2=0x7f316c11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): get_auth_request con 0x7f316c09aba0 auth_method 0 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f317140c640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:06.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f3170c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient(hunting): handle_auth_done global_id 244205 payload 824 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f316c09aba0 msgr2=0x7f316c1410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f316c09aba0 0x7f316c1410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 msgr2=0x7f316c11d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 monclient: found mon.noname-c 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.803+0000 7f316bfff640 1 -- 172.21.15.153:0/1844164549 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009450 con 0x7f316c09afa0 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f316bfff640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 msgr2=0x7f316c133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:06.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f316bfff640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 0x7f316c133c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3158009940 tx=0x7f3158032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580319c0 con 0x7f316c09afa0 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: dump: 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:06.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:06.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: _finish_auth 0 2024-09-15T03:49:06.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:06.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 5 monclient: authenticate success, global_id 244205 2024-09-15T03:49:06.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 10 monclient: _renew_subs 2024-09-15T03:49:06.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:06.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c120db0 con 0x7f316c09afa0 2024-09-15T03:49:06.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 10 monclient: _renew_subs 2024-09-15T03:49:06.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c121140 con 0x7f316c09afa0 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3158031b20 con 0x7f316c09afa0 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f316affd640 4 set_mon_vals no callback set 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580319c0 con 0x7f316c09afa0 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.806+0000 7f3168ff9640 10 monclient: dump: 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:06.839 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3158039040 con 0x7f316c09afa0 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3172e6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3172e6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3172e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316c04bf60 con 0x7f316c09afa0 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3168ff9640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f31500387a0 0x7f315003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:06.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f315806c000 con 0x7f316c09afa0 2024-09-15T03:49:06.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3170c0b640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f31500387a0 0x7f315003ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:06.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.807+0000 7f3170c0b640 10 monclient: get_auth_request con 0x7f31500387a0 auth_method 0 2024-09-15T03:49:06.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.810+0000 7f3170c0b640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f31500387a0 0x7f315003ac50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3160006fd0 tx=0x7f3160008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:06.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.811+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3158030570 con 0x7f316c09afa0 2024-09-15T03:49:06.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.811+0000 7f3168ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:06.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:06.811+0000 7f3168ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:07.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3172e6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:49:07.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3172e6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:07.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3172e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:07.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f316c069c10 con 0x7f316c09afa0 2024-09-15T03:49:07.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3168ff9640 1 -- 172.21.15.153:0/1844164549 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f3158037610 con 0x7f316c09afa0 2024-09-15T03:49:07.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3168ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:07.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.027+0000 7f3168ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:07.030 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":208,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:49:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f31500387a0 msgr2=0x7f315003ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f31500387a0 0x7f315003ac50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3160006fd0 tx=0x7f3160008040 comp rx=0 tx=0).stop 2024-09-15T03:49:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 10 monclient: shutdown 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 msgr2=0x7f316c133c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 0x7f316c133c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3158009940 tx=0x7f3158032000 comp rx=0 tx=0).stop 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 shutdown_connections 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f31500387a0 0x7f315003ac50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f316c09c860 0x7f316c11d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f316c09afa0 0x7f316c133c90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 --2- 172.21.15.153:0/1844164549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f316c09aba0 0x7f316c1410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 >> 172.21.15.153:0/1844164549 conn(0x7f316c0a2e40 msgr2=0x7f316c098b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 shutdown_connections 2024-09-15T03:49:07.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:07.029+0000 7f3172e6f640 1 -- 172.21.15.153:0/1844164549 wait complete. 2024-09-15T03:49:10.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:49:10.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.148+0000 7f784815d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.148+0000 7f784815d640 10 monclient: build_initial_monmap 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.148+0000 7f784815d640 10 monclient: monmap: 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:10.150075+0000 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:10.150075+0000 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:10.151 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.148+0000 7f784815d640 1 Processor -- start 2024-09-15T03:49:10.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 -- start start 2024-09-15T03:49:10.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient: init 2024-09-15T03:49:10.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:10.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-15T03:49:10.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 0x7f784009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): picked mon.noname-b con 0x7f784009c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): picked mon.noname-a con 0x7f784009aba0 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f784009af70 0x7f784009dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): picked mon.noname-c con 0x7f784009af70 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784009a9d0 con 0x7f784009aba0 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784006a320 con 0x7f784009af70 2024-09-15T03:49:10.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7840068a20 con 0x7f784009c860 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f784815d640 10 monclient(hunting): authenticate will time out at 1947.789777s 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f7845ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 0x7f784009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60666/0 (socket says 172.21.15.153:60666) 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 1 -- 172.21.15.153:0/417459445 learned_addr learned my addr 172.21.15.153:0/417459445 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 10 monclient(hunting): get_auth_request con 0x7f784009aba0 auth_method 0 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f78456f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:10.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f7845ef9640 10 monclient(hunting): get_auth_request con 0x7f784009c860 auth_method 0 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f7845ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f7845ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.149+0000 7f7845ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.181:3300/0 conn(0x7f784009af70 0x7f784009dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): get_auth_request con 0x7f784009af70 auth_method 0 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78456f8640 1 -- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 msgr2=0x7f784009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78456f8640 1 -- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 msgr2=0x7f784009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78456f8640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78456f8640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f78466fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:10.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 10 monclient(hunting): handle_auth_done global_id 244208 payload 274 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 1 -- 172.21.15.153:0/417459445 >> v2:172.21.15.181:3300/0 conn(0x7f784009af70 msgr2=0x7f784009dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.181:3300/0 conn(0x7f784009af70 0x7f784009dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 1 -- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 msgr2=0x7f784009d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 10 monclient: _finish_hunting 0 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 1 monclient: found mon.noname-b 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.150+0000 7f7845ef9640 1 -- 172.21.15.153:0/417459445 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7840068e60 con 0x7f784009c860 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7845ef9640 1 -- 172.21.15.153:0/417459445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 msgr2=0x7f784009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7845ef9640 1 --2- 172.21.15.153:0/417459445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 0x7f784009cc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7840068db0 tx=0x7f7838009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801eddc8d46c046b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 1 -- 172.21.15.153:0/417459445 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838007760 con 0x7f784009c860 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:10.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: dump: 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:10.167 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: _finish_auth 0 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 1 -- 172.21.15.153:0/417459445 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802d050 con 0x7f784009c860 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 5 monclient: authenticate success, global_id 244208 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 1 -- 172.21.15.153:0/417459445 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838007af0 con 0x7f784009c860 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 4 set_mon_vals no callback set 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:10.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f7844ef7640 10 monclient: dump: 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 10 monclient: shutdown 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:10.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 -- 172.21.15.153:0/417459445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 msgr2=0x7f784009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 --2- 172.21.15.153:0/417459445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 0x7f784009cc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7840068db0 tx=0x7f7838009780 comp rx=0 tx=0).stop 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 -- 172.21.15.153:0/417459445 shutdown_connections 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.181:3300/0 conn(0x7f784009af70 0x7f784009dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 --2- 172.21.15.153:0/417459445 >> v2:172.21.15.153:3300/0 conn(0x7f784009aba0 0x7f784009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 --2- 172.21.15.153:0/417459445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009c860 0x7f784009cc30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.153+0000 7f784815d640 1 -- 172.21.15.153:0/417459445 >> 172.21.15.153:0/417459445 conn(0x7f78400a2e40 msgr2=0x7f78400a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 1 -- 172.21.15.153:0/417459445 shutdown_connections 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 1 -- 172.21.15.153:0/417459445 wait complete. 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 10 monclient: build_initial_monmap 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 10 monclient: monmap: 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:10.170 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 1 Processor -- start 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 1 -- start start 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.154+0000 7f784815d640 10 monclient: init 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f7840133940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): picked mon.noname-a con 0x7f784009aba0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:10.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): picked mon.noname-c con 0x7f784009af70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f784011ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): picked mon.noname-b con 0x7f784009c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7840099cc0 con 0x7f784009aba0 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7840134aa0 con 0x7f784009c860 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784011f4b0 con 0x7f784009af70 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:10.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f784815d640 10 monclient(hunting): authenticate will time out at 1947.795528s 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f7840133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54180/0 (socket says 172.21.15.153:54180) 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 10 monclient(hunting): get_auth_request con 0x7f784009aba0 auth_method 0 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 learned_addr learned my addr 172.21.15.153:0/2524944247 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:10.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): get_auth_request con 0x7f784009af70 auth_method 0 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 msgr2=0x7f7840133940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 msgr2=0x7f7840133940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f7840133940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f7845ef9640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f7840133940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient(hunting): handle_auth_done global_id 244211 payload 824 2024-09-15T03:49:10.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78466fa640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f784011ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 msgr2=0x7f784011ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f784011ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 msgr2=0x7f7840133940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f7840133940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient: _finish_hunting 0 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 monclient: found mon.noname-c 2024-09-15T03:49:10.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.155+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838009a20 con 0x7f784009af70 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 msgr2=0x7f784011d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f78456f8640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f783400e870 tx=0x7f7834008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783400c6d0 con 0x7f784009af70 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: dump: 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:10.176 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: _finish_auth 0 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783400c830 con 0x7f784009af70 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 5 monclient: authenticate success, global_id 244211 2024-09-15T03:49:10.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 10 monclient: _renew_subs 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7840121020 con 0x7f784009af70 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783400cbf0 con 0x7f784009af70 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783dffb640 10 monclient: dump: 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.178 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 10 monclient: _renew_subs 2024-09-15T03:49:10.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784011fb70 con 0x7f784009af70 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.158+0000 7f783ffff640 4 set_mon_vals no callback set 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7834013410 con 0x7f784009af70 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f784815d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f784815d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f784815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784011f000 con 0x7f784009af70 2024-09-15T03:49:10.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f783dffb640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f782c038770 0x7f782c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f783404c5a0 con 0x7f784009af70 2024-09-15T03:49:10.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f7845ef9640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f782c038770 0x7f782c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.159+0000 7f7845ef9640 10 monclient: get_auth_request con 0x7f782c038770 auth_method 0 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.162+0000 7f7845ef9640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f782c038770 0x7f782c03ac20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f784009ef00 tx=0x7f7838037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 msgr2=0x7f784011d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f78456f8640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 msgr2=0x7f784011d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f78456f8640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f783400e870 tx=0x7f7834008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f78456f8640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f783400e870 tx=0x7f7834008040 comp rx=0 tx=0).stop 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 msgr2=0x7f784011d8b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f784009af70 0x7f784011d8b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f782c042f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): picked mon.b con 0x7f784009c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 0x7f782c045c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): picked mon.c con 0x7f782c043800 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f782c048760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): picked mon.a con 0x7f784009aba0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0490e0 con 0x7f784009aba0 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c049220 con 0x7f784009c860 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0493e0 con 0x7f782c043800 2024-09-15T03:49:10.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.163+0000 7f783dffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 0x7f782c045c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f782c048760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): get_auth_request con 0x7f782c043800 auth_method 0 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): get_auth_request con 0x7f784009aba0 auth_method 0 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:10.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f782c042f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): get_auth_request con 0x7f784009c860 auth_method 0 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78456f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:49:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient(hunting): handle_auth_done global_id 244211 payload 293 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 msgr2=0x7f782c042f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f782c042f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 msgr2=0x7f782c048760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f782c048760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient: _finish_hunting 0 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 monclient: found mon.c 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f782c049660 con 0x7f782c043800 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7830009ae0 con 0x7f782c043800 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 10 monclient: _finish_auth 0 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f78466fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.164+0000 7f7845ef9640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f782c048760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f78466fa640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 msgr2=0x7f782c045c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f78466fa640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 0x7f782c045c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7830009ef0 tx=0x7f7830009f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830010d90 con 0x7f782c043800 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783dffb640 10 monclient: dump: 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:10.191 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783001d460 con 0x7f782c043800 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.166+0000 7f783f7fe640 4 set_mon_vals no callback set 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.167+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f78300143f0 con 0x7f782c043800 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.171+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7830011260 con 0x7f782c043800 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.171+0000 7f783dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:10.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.171+0000 7f783dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:10.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f784815d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f784815d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f784815d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f784004bf60 con 0x7f782c043800 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f783dffb640 1 -- 172.21.15.153:0/2524944247 <== mon.2 v2:172.21.15.153:3301/0 5 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f7830011260 con 0x7f782c043800 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f783dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.385+0000 7f783dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:10.387 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":208,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f782c038770 msgr2=0x7f782c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f782c038770 0x7f782c03ac20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f784009ef00 tx=0x7f7838037000 comp rx=0 tx=0).stop 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 10 monclient: shutdown 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 msgr2=0x7f782c045c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 0x7f782c045c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7830009ef0 tx=0x7f7830009f20 comp rx=0 tx=0).stop 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 shutdown_connections 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f784009aba0 0x7f782c048760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f782c043800 0x7f782c045c80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f784009c860 0x7f782c042f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 --2- 172.21.15.153:0/2524944247 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f782c038770 0x7f782c03ac20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:10.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 >> 172.21.15.153:0/2524944247 conn(0x7f78400a2e40 msgr2=0x7f7840098b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:10.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 shutdown_connections 2024-09-15T03:49:10.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:10.387+0000 7f784815d640 1 -- 172.21.15.153:0/2524944247 wait complete. 2024-09-15T03:49:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=13 2024-09-15T03:49:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 13 -ne 13 ]] 2024-09-15T03:49:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:49:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-15T03:49:13.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:49:13.405 INFO:tasks.workunit.client.0.smithi181.stderr:+ cat 2024-09-15T03:49:13.405 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph auth import -i - 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.476+0000 7f20b924f640 10 monclient: get_monmap_and_config 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.476+0000 7f20b924f640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.476+0000 7f20b924f640 10 monclient: monmap: 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:49:13.478120+0000 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:49:13.478120+0000 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:13.478 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:13.479 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:13.479 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.476+0000 7f20b924f640 1 Processor -- start 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.476+0000 7f20b924f640 1 -- start start 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.476+0000 7f20b924f640 10 monclient: init 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20b412e770 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20b412eb40 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f20b4134d40 0x7f20b4137120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient(hunting): picked mon.noname-c con 0x7f20b4134d40 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:13.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b409ce10 con 0x7f20b412eb40 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b4099850 con 0x7f20b412e770 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b4099570 con 0x7f20b4134d40 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b924f640 10 monclient(hunting): authenticate will time out at 1957.959482s 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f20b4134d40 0x7f20b4137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b2d9d640 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f20b412e770 0x7f20b409aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:40792/0 (socket says 172.21.15.181:40792) 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f20b4134d40 0x7f20b4137120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48882/0 (socket says 172.21.15.181:48882) 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b2d9d640 1 -- 172.21.15.181:0/606192505 learned_addr learned my addr 172.21.15.181:0/606192505 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-09-15T03:49:13.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): get_auth_request con 0x7f20b4134d40 auth_method 0 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient(hunting): handle_auth_done global_id 204550 payload 274 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 1 -- 172.21.15.181:0/606192505 >> v1:172.21.15.181:6789/0 conn(0x7f20b412e770 legacy=0x7f20b409aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 1 -- 172.21.15.181:0/606192505 >> v1:172.21.15.153:6789/0 conn(0x7f20b412eb40 legacy=0x7f20b412ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 1 monclient: found mon.noname-c 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.483 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.477+0000 7f20b359e640 1 -- 172.21.15.181:0/606192505 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4099b30 con 0x7f20b4134d40 2024-09-15T03:49:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.480+0000 7f20b359e640 1 -- 172.21.15.181:0/606192505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b4134d40 msgr2=0x7f20b4137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.480+0000 7f20b359e640 1 --2- 172.21.15.181:0/606192505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b4134d40 0x7f20b4137120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009e60 tx=0x7f20ac009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ed89610df8aefc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 1 -- 172.21.15.181:0/606192505 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac007630 con 0x7f20b4134d40 2024-09-15T03:49:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.489 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: dump: 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.490 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 1 -- 172.21.15.181:0/606192505 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ac02d050 con 0x7f20b4134d40 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 5 monclient: authenticate success, global_id 204550 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 1 -- 172.21.15.181:0/606192505 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac0079c0 con 0x7f20b4134d40 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 4 set_mon_vals no callback set 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b1d9b640 10 monclient: dump: 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.491 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 10 monclient: shutdown 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 -- 172.21.15.181:0/606192505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b4134d40 msgr2=0x7f20b4137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 --2- 172.21.15.181:0/606192505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b4134d40 0x7f20b4137120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009e60 tx=0x7f20ac009780 comp rx=0 tx=0).stop 2024-09-15T03:49:13.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 -- 172.21.15.181:0/606192505 shutdown_connections 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 --2- 172.21.15.181:0/606192505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b4134d40 0x7f20b4137120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 -- 172.21.15.181:0/606192505 >> 172.21.15.181:0/606192505 conn(0x7f20b40a18b0 msgr2=0x7f20b409d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 -- 172.21.15.181:0/606192505 shutdown_connections 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.481+0000 7f20b924f640 1 -- 172.21.15.181:0/606192505 wait complete. 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.482+0000 7f20b924f640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.482+0000 7f20b924f640 10 monclient: monmap: 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.493 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.482+0000 7f20b924f640 1 Processor -- start 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.482+0000 7f20b924f640 1 -- start start 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.482+0000 7f20b924f640 10 monclient: init 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20b412e770 0x7f20b4117f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): picked mon.noname-c con 0x7f20b412e770 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 0x7f20b406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.494 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): picked mon.noname-b con 0x7f20b412eb40 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20b4134d40 0x7f20b406c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): picked mon.noname-a con 0x7f20b4134d40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b409b280 con 0x7f20b4134d40 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40a02a0 con 0x7f20b412eb40 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406df90 con 0x7f20b412e770 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b924f640 10 monclient(hunting): authenticate will time out at 1957.965258s 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 0x7f20b406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 0x7f20b406bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48884/0 (socket says 172.21.15.181:48884) 2024-09-15T03:49:13.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 -- 172.21.15.181:0/350637065 learned_addr learned my addr 172.21.15.181:0/350637065 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): get_auth_request con 0x7f20b412eb40 auth_method 0 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20b412e770 0x7f20b4117f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20b4134d40 0x7f20b406c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): get_auth_request con 0x7f20b412e770 auth_method 0 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): get_auth_request con 0x7f20b4134d40 auth_method 0 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient(hunting): handle_auth_done global_id 204553 payload 824 2024-09-15T03:49:13.497 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 -- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20b412e770 msgr2=0x7f20b4117f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20b412e770 0x7f20b4117f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 -- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20b4134d40 msgr2=0x7f20b406c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20b4134d40 0x7f20b406c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 monclient: found mon.noname-b 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.483+0000 7f20b259c640 1 -- 172.21.15.181:0/350637065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac009a20 con 0x7f20b412eb40 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.485+0000 7f20b259c640 1 -- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 msgr2=0x7f20b406bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.485+0000 7f20b259c640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 0x7f20b406bea0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20a800c5e0 tx=0x7f20a8007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a8012840 con 0x7f20b412eb40 2024-09-15T03:49:13.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: dump: 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 5 monclient: authenticate success, global_id 204553 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a80129a0 con 0x7f20b412eb40 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 10 monclient: _renew_subs 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b406f540 con 0x7f20b412eb40 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a8012da0 con 0x7f20b412eb40 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f209affd640 10 monclient: dump: 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.500 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 10 monclient: _renew_subs 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b406e650 con 0x7f20b412eb40 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.486+0000 7f20b159a640 4 set_mon_vals no callback set 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f20a8011590 con 0x7f20b412eb40 2024-09-15T03:49:13.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f20b924f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f20b924f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f20b924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b406e8d0 con 0x7f20b412eb40 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f209affd640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20a0038770 0x7f20a003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f20a804c680 con 0x7f20b412eb40 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f20b2d9d640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20a0038770 0x7f20a003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.487+0000 7f20b2d9d640 10 monclient: get_auth_request con 0x7f20a0038770 auth_method 0 2024-09-15T03:49:13.502 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.489+0000 7f20b2d9d640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20a0038770 0x7f20a003ac20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f20a4007fd0 tx=0x7f20a4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.491+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f20b406e8d0 con 0x7f20b412eb40 2024-09-15T03:49:13.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.491+0000 7f209affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.491+0000 7f209affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.508+0000 7fba4239d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.508+0000 7fba4239d640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.508+0000 7fba4239d640 10 monclient: monmap: 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:13.509992+0000 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:13.509992+0000 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:13.511 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 1 Processor -- start 2024-09-15T03:49:13.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 1 -- start start 2024-09-15T03:49:13.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient: init 2024-09-15T03:49:13.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba3c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba3c09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 0x7fba3c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient(hunting): picked mon.noname-b con 0x7fba3c137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c132640 con 0x7fba3c09ea50 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c131cc0 con 0x7fba3c1318f0 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c09f530 con 0x7fba3c137c30 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4239d640 10 monclient(hunting): authenticate will time out at 1951.149642s 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba3b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fba3c09ea50 0x7fba3c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60580/0 (socket says 172.21.15.153:60580) 2024-09-15T03:49:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba3b7fe640 1 -- 172.21.15.153:0/1528658801 learned_addr learned my addr 172.21.15.153:0/1528658801 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 1 --2- 172.21.15.153:0/1528658801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 0x7fba3c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): get_auth_request con 0x7fba3c137c30 auth_method 0 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.509+0000 7fba4093a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 10 monclient(hunting): handle_auth_done global_id 244220 payload 274 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 1 -- 172.21.15.153:0/1528658801 >> v1:172.21.15.181:6789/0 conn(0x7fba3c1318f0 legacy=0x7fba3c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 1 -- 172.21.15.153:0/1528658801 >> v1:172.21.15.153:6789/0 conn(0x7fba3c09ea50 legacy=0x7fba3c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 1 monclient: found mon.noname-b 2024-09-15T03:49:13.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:49:13.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.510+0000 7fba4093a640 1 -- 172.21.15.153:0/1528658801 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c09fbf0 con 0x7fba3c137c30 2024-09-15T03:49:13.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.512+0000 7fba4093a640 1 -- 172.21.15.153:0/1528658801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 msgr2=0x7fba3c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.512+0000 7fba4093a640 1 --2- 172.21.15.153:0/1528658801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 0x7fba3c13a010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba34009e90 tx=0x7fba34009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aacbb1f39afa308 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 1 -- 172.21.15.153:0/1528658801 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34007840 con 0x7fba3c137c30 2024-09-15T03:49:13.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: dump: 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 5 monclient: authenticate success, global_id 244220 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 1 -- 172.21.15.153:0/1528658801 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3402d050 con 0x7fba3c137c30 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 1 -- 172.21.15.153:0/1528658801 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34007bd0 con 0x7fba3c137c30 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 4 set_mon_vals no callback set 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba3affd640 10 monclient: dump: 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 10 monclient: shutdown 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 -- 172.21.15.153:0/1528658801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 msgr2=0x7fba3c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 --2- 172.21.15.153:0/1528658801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 0x7fba3c13a010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba34009e90 tx=0x7fba34009780 comp rx=0 tx=0).stop 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 -- 172.21.15.153:0/1528658801 shutdown_connections 2024-09-15T03:49:13.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 --2- 172.21.15.153:0/1528658801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c137c30 0x7fba3c13a010 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 -- 172.21.15.153:0/1528658801 >> 172.21.15.153:0/1528658801 conn(0x7fba3c067110 msgr2=0x7fba3c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 -- 172.21.15.153:0/1528658801 shutdown_connections 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.513+0000 7fba4239d640 1 -- 172.21.15.153:0/1528658801 wait complete. 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.514+0000 7fba4239d640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.514+0000 7fba4239d640 10 monclient: monmap: 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:13.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.514+0000 7fba4239d640 1 Processor -- start 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.514+0000 7fba4239d640 1 -- start start 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient: init 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 0x7fba3c136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): picked mon.noname-c con 0x7fba3c09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c1318f0 0x7fba3c12fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): picked mon.noname-b con 0x7fba3c1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): picked mon.noname-a con 0x7fba3c137c30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c1347f0 con 0x7fba3c137c30 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c13a2c0 con 0x7fba3c1318f0 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c134f20 con 0x7fba3c09ea50 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4239d640 10 monclient(hunting): authenticate will time out at 1951.155467s 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 0x7fba3c136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60700/0 (socket says 172.21.15.153:60700) 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 -- 172.21.15.153:0/1342603093 learned_addr learned my addr 172.21.15.153:0/1342603093 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 10 monclient(hunting): get_auth_request con 0x7fba3c137c30 auth_method 0 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): get_auth_request con 0x7fba3c09ea50 auth_method 0 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 msgr2=0x7fba3c130410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 msgr2=0x7fba3c130410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c1318f0 0x7fba3c12fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba4093a640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): get_auth_request con 0x7fba3c1318f0 auth_method 0 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient(hunting): handle_auth_done global_id 244223 payload 824 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c1318f0 msgr2=0x7fba3c12fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c1318f0 0x7fba3c12fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 msgr2=0x7fba3c130410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 monclient: found mon.noname-c 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.515+0000 7fba3bfff640 1 -- 172.21.15.153:0/1342603093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34009a20 con 0x7fba3c09ea50 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba3bfff640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 msgr2=0x7fba3c136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba3bfff640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 0x7fba3c136310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c4e0 tx=0x7fba2c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0117a0 con 0x7fba3c09ea50 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: dump: 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.530 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 5 monclient: authenticate success, global_id 244223 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c011920 con 0x7fba3c09ea50 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 10 monclient: _renew_subs 2024-09-15T03:49:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c011ce0 con 0x7fba3c09ea50 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c134510 con 0x7fba3c09ea50 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba33fff640 10 monclient: dump: 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba39ffb640 4 set_mon_vals no callback set 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 10 monclient: _renew_subs 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.518+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c137300 con 0x7fba3c09ea50 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fba2c012630 con 0x7fba3c09ea50 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba4239d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba4239d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba4239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c04bf60 con 0x7fba3c09ea50 2024-09-15T03:49:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba33fff640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba1c038740 0x7fba1c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fba2c04c1b0 con 0x7fba3c09ea50 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba3b7fe640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba1c038740 0x7fba1c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.519+0000 7fba3b7fe640 10 monclient: get_auth_request con 0x7fba1c038740 auth_method 0 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.522+0000 7fba3b7fe640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba1c038740 0x7fba1c03abf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fba24007fd0 tx=0x7fba24007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.524+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fba2c0128e0 con 0x7fba3c09ea50 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.524+0000 7fba33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.524+0000 7fba33fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.709+0000 7f20b924f640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-15T03:49:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.709+0000 7f20b924f640 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-15T03:49:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.709+0000 7f20b924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.709+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7f20b406e8d0 con 0x7f20b412eb40 2024-09-15T03:49:13.711 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:13.710+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-15T03:49:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.713+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth import"}]=0 v37) v1 ==== 59+0+0 (secure 0 0 0) 0x7f20b406e8d0 con 0x7f20b412eb40 2024-09-15T03:49:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.713+0000 7f209affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-15T03:49:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.713+0000 7f209affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba4239d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba4239d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba4239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fba3c0638d0 con 0x7fba3c09ea50 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba33fff640 1 -- 172.21.15.153:0/1342603093 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fba2c010750 con 0x7fba3c09ea50 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:13.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.739+0000 7fba33fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:13.742 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":212,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:49:13.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba1c038740 msgr2=0x7fba1c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba1c038740 0x7fba1c03abf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fba24007fd0 tx=0x7fba24007f00 comp rx=0 tx=0).stop 2024-09-15T03:49:13.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 10 monclient: shutdown 2024-09-15T03:49:13.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 msgr2=0x7fba3c136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 0x7fba3c136310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c4e0 tx=0x7fba2c007e10 comp rx=0 tx=0).stop 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 shutdown_connections 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fba1c038740 0x7fba1c03abf0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c137c30 0x7fba3c130410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c1318f0 0x7fba3c12fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 --2- 172.21.15.153:0/1342603093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba3c09ea50 0x7fba3c136310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 >> 172.21.15.153:0/1342603093 conn(0x7fba3c067110 msgr2=0x7fba3c12b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 shutdown_connections 2024-09-15T03:49:13.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.741+0000 7fba4239d640 1 -- 172.21.15.153:0/1342603093 wait complete. 2024-09-15T03:49:13.754 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:49:13.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:49:13.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:49:13.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.846+0000 7f8ab715d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.846+0000 7f8ab715d640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.846+0000 7f8ab715d640 10 monclient: monmap: 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:13.847806+0000 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:13.847806+0000 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:13.849 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:13.849 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.846+0000 7f8ab715d640 1 Processor -- start 2024-09-15T03:49:13.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.846+0000 7f8ab715d640 1 -- start start 2024-09-15T03:49:13.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.846+0000 7f8ab715d640 10 monclient: init 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ab0131ab0 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f8ab009e910 0x7f8ab009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab009e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8ab009f250 0x7f8ab0138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab009f250 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab009faf0 con 0x7f8ab0131ab0 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab0132590 con 0x7f8ab009f250 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab009fd30 con 0x7f8ab009e910 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab715d640 10 monclient(hunting): authenticate will time out at 1951.487489s 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab4ef9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8ab0131ab0 0x7f8ab0131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60584/0 (socket says 172.21.15.153:60584) 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab4ef9640 1 -- 172.21.15.153:0/1939792297 learned_addr learned my addr 172.21.15.153:0/1939792297 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8ab009f250 0x7f8ab0138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): get_auth_request con 0x7f8ab009f250 auth_method 0 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 1 --2- 172.21.15.153:0/1939792297 >> v2:172.21.15.181:3300/0 conn(0x7f8ab009e910 0x7f8ab009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): get_auth_request con 0x7f8ab009e910 auth_method 0 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8aaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_done global_id 254106 payload 274 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 1 -- 172.21.15.153:0/1939792297 >> v2:172.21.15.181:3300/0 conn(0x7f8ab009e910 msgr2=0x7f8ab009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 1 --2- 172.21.15.153:0/1939792297 >> v2:172.21.15.181:3300/0 conn(0x7f8ab009e910 0x7f8ab009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 1 -- 172.21.15.153:0/1939792297 >> v1:172.21.15.153:6789/0 conn(0x7f8ab0131ab0 legacy=0x7f8ab0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 1 monclient: found mon.noname-a 2024-09-15T03:49:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:13.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.847+0000 7f8ab56fa640 1 -- 172.21.15.153:0/1939792297 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab00a1110 con 0x7f8ab009f250 2024-09-15T03:49:13.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8ab56fa640 1 -- 172.21.15.153:0/1939792297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 msgr2=0x7f8ab0138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8ab56fa640 1 --2- 172.21.15.153:0/1939792297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab0138bc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8ab009bdb0 tx=0x7f8aa80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d94543c13fbbf65 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 1 -- 172.21.15.153:0/1939792297 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa802ae00 con 0x7f8ab009f250 2024-09-15T03:49:13.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: dump: 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 1 -- 172.21.15.153:0/1939792297 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa8032440 con 0x7f8ab009f250 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8ab715d640 5 monclient: authenticate success, global_id 254106 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8ab715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8ab715d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8ab715d640 4 set_mon_vals no callback set 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 1 -- 172.21.15.153:0/1939792297 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8032820 con 0x7f8ab009f250 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.850+0000 7f8aaf7fe640 10 monclient: dump: 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.865 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 10 monclient: shutdown 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 -- 172.21.15.153:0/1939792297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 msgr2=0x7f8ab0138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 --2- 172.21.15.153:0/1939792297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab0138bc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8ab009bdb0 tx=0x7f8aa80090f0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 -- 172.21.15.153:0/1939792297 shutdown_connections 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 --2- 172.21.15.153:0/1939792297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab0138bc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 --2- 172.21.15.153:0/1939792297 >> v2:172.21.15.181:3300/0 conn(0x7f8ab009e910 0x7f8ab009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 -- 172.21.15.153:0/1939792297 >> 172.21.15.153:0/1939792297 conn(0x7f8ab0067d30 msgr2=0x7f8ab00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 -- 172.21.15.153:0/1939792297 shutdown_connections 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 1 -- 172.21.15.153:0/1939792297 wait complete. 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.851+0000 7f8ab715d640 10 monclient: monmap: 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:13.867 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 1 Processor -- start 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 1 -- start start 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient: init 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab009e910 0x7f8ab0098340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab009a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab009f250 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab0131ab0 0x7f8ab0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab0131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:13.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.852+0000 7f8ab715d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0099a80 con 0x7f8ab009f250 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab715d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0099810 con 0x7f8ab0131ab0 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab715d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0099d60 con 0x7f8ab009e910 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab715d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab715d640 10 monclient(hunting): authenticate will time out at 1951.493021s 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab009a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab009e910 0x7f8ab0098340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab009a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60720/0 (socket says 172.21.15.153:60720) 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 -- 172.21.15.153:0/594619618 learned_addr learned my addr 172.21.15.153:0/594619618 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:13.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): get_auth_request con 0x7f8ab009f250 auth_method 0 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): get_auth_request con 0x7f8ab009e910 auth_method 0 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab0131ab0 0x7f8ab0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): get_auth_request con 0x7f8ab0131ab0 auth_method 0 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab56fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient(hunting): handle_auth_done global_id 254109 payload 824 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab009e910 msgr2=0x7f8ab0098340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab009e910 0x7f8ab0098340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab0131ab0 msgr2=0x7f8ab0096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ab0131ab0 0x7f8ab0096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 monclient: found mon.noname-a 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8aaffff640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8009390 con 0x7f8ab009f250 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.853+0000 7f8ab4ef9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ab009e910 0x7f8ab0098340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aaffff640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 msgr2=0x7f8ab009a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aaffff640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab009a380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800ea20 tx=0x7f8a980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98009600 con 0x7f8ab009f250 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: dump: 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 5 monclient: authenticate success, global_id 254109 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9800cc20 con 0x7f8ab009f250 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 10 monclient: _renew_subs 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab0097b50 con 0x7f8ab009f250 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98013470 con 0x7f8ab009f250 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aacff9640 10 monclient: dump: 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.874 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8aaeffd640 4 set_mon_vals no callback set 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 10 monclient: _renew_subs 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.855+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab01311c0 con 0x7f8ab009f250 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab715d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f8a98013860 con 0x7f8ab009f250 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab715d640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 0x7f8ab0097370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0063740 con 0x7f8ab0096fa0 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 0x7f8ab0097370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient: get_auth_request con 0x7f8ab0096fa0 auth_method 0 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8aacff9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a90038740 0x7f8a9003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f8a9804c940 con 0x7f8ab009f250 2024-09-15T03:49:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_done global_id 254112 payload 824 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab4ef9640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ab00978b0 con 0x7f8ab0096fa0 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.856+0000 7f8ab56fa640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a90038740 0x7f8a9003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.857+0000 7f8ab56fa640 10 monclient: get_auth_request con 0x7f8a90038740 auth_method 0 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8ab4ef9640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 msgr2=0x7f8ab0097370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8ab4ef9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 0x7f8ab0097370 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000d3b0 tx=0x7f8aa0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0007320 con 0x7f8ab0096fa0 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 10 monclient: dump: 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.877 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8ab56fa640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a90038740 0x7f8a9003abf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8034000 tx=0x7f8aa8032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8aa0007480 con 0x7f8ab0096fa0 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 msgr2=0x7f8ab0097370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 0x7f8ab0097370 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000d3b0 tx=0x7f8aa0009040 comp rx=0 tx=0).stop 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 msgr2=0x7f8ab0097370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:13.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.859+0000 7f8aacff9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab0096fa0 0x7f8ab0097370 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000d3b0 tx=0x7f8aa0009040 comp rx=0 tx=0).stop 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.868+0000 7f20b924f640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.868+0000 7f20b924f640 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.868+0000 7f20b924f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.868+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7f20b406e8d0 con 0x7f20b412eb40 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.874+0000 7f209affd640 1 -- 172.21.15.181:0/350637065 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "auth import"}]=0 v38) v1 ==== 59+0+0 (secure 0 0 0) 0x7f20b406e8d0 con 0x7f20b412eb40 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.874+0000 7f209affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.874+0000 7f209affd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20a0038770 msgr2=0x7f20a003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20a0038770 0x7f20a003ac20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f20a4007fd0 tx=0x7f20a4007f00 comp rx=0 tx=0).stop 2024-09-15T03:49:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 10 monclient: shutdown 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 msgr2=0x7f20b406bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 0x7f20b406bea0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20a800c5e0 tx=0x7f20a8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 shutdown_connections 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20a0038770 0x7f20a003ac20 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20b4134d40 0x7f20b406c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20b412eb40 0x7f20b406bea0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 --2- 172.21.15.181:0/350637065 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20b412e770 0x7f20b4117f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.876+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 >> 172.21.15.181:0/350637065 conn(0x7f20b40a18b0 msgr2=0x7f20b41344d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.877+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 shutdown_connections 2024-09-15T03:49:13.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.877+0000 7f20b924f640 1 -- 172.21.15.181:0/350637065 wait complete. 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab715d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab715d640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 0x7f8ab009d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab715d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0131440 con 0x7f8ab009ce40 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 0x7f8ab009d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient: get_auth_request con 0x7f8ab009ce40 auth_method 0 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 10 monclient(hunting): handle_auth_done global_id 254115 payload 824 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.865+0000 7f8ab4ef9640 1 -- 172.21.15.153:0/594619618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8ab004bf60 con 0x7f8ab009ce40 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.867+0000 7f8ab4ef9640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 msgr2=0x7f8ab009d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.867+0000 7f8ab4ef9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 0x7f8ab009d240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000d3b0 tx=0x7f8aa0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.867+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0023600 con 0x7f8ab009ce40 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.867+0000 7f8aacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.867+0000 7f8aacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.867+0000 7f8aacff9640 10 monclient: dump: 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.884 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f8aa0023760 con 0x7f8ab009ce40 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 msgr2=0x7f8ab009d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 0x7f8ab009d240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000d3b0 tx=0x7f8aa0017000 comp rx=0 tx=0).stop 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 msgr2=0x7f8ab009d240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:13.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8aacff9640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009ce40 0x7f8ab009d240 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000d3b0 tx=0x7f8aa0017000 comp rx=0 tx=0).stop 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 212, 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:49:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:13.889 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:13.890 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:13.891 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:13.891 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:13.891 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:13.891 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:13.891 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.892 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:13.893 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:13.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:13.895 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:13.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:13.898 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:49:13.899 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:13.899 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:13.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:13.899 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:13.899 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:13.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a90038740 msgr2=0x7f8a9003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8a90038740 0x7f8a9003abf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8034000 tx=0x7f8aa8032000 comp rx=0 tx=0).stop 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 10 monclient: shutdown 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 msgr2=0x7f8ab009a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 --2- 172.21.15.153:0/594619618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ab009f250 0x7f8ab009a380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800ea20 tx=0x7f8a980090f0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab56fa640 1 -- 172.21.15.153:0/594619618 reap_dead start 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab56fa640 1 -- 172.21.15.153:0/594619618 reap_dead start 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 shutdown_connections 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 >> 172.21.15.153:0/594619618 conn(0x7f8ab0067d30 msgr2=0x7f8ab00a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 shutdown_connections 2024-09-15T03:49:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.868+0000 7f8ab715d640 1 -- 172.21.15.153:0/594619618 wait complete. 2024-09-15T03:49:13.903 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:49:13.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:49:13.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:49:13.904 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph health 2024-09-15T03:49:13.904 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep AUTH_BAD_CAP 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.952+0000 7ff6b18e8640 10 monclient: get_monmap_and_config 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.952+0000 7ff6b18e8640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.952+0000 7ff6b18e8640 10 monclient: monmap: 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:49:13.954551+0000 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:49:13.954551+0000 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.965 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 1 Processor -- start 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 1 -- start start 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 10 monclient: init 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6ac12e770 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:13.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6ac12eb40 0x7ff6ac12ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.953+0000 7ff6b18e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ac12eb40 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac133d30 0x7ff6ac136110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ac133d30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac09ce10 con 0x7ff6ac12e770 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ac099850 con 0x7ff6ac12eb40 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac099570 con 0x7ff6ac133d30 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6b18e8640 10 monclient(hunting): authenticate will time out at 1958.436072s 2024-09-15T03:49:13.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6ac12eb40 0x7ff6ac12ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ff6ac12eb40 0x7ff6ac12ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48900/0 (socket says 172.21.15.181:48900) 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 -- 172.21.15.181:0/3197550081 learned_addr learned my addr 172.21.15.181:0/3197550081 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): get_auth_request con 0x7ff6ac12eb40 auth_method 0 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 1 --2- 172.21.15.181:0/3197550081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac133d30 0x7ff6ac136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff6ac133d30 auth_method 0 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient(hunting): handle_auth_done global_id 204565 payload 274 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 -- 172.21.15.181:0/3197550081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac133d30 msgr2=0x7ff6ac136110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 --2- 172.21.15.181:0/3197550081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac133d30 0x7ff6ac136110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 -- 172.21.15.181:0/3197550081 >> v1:172.21.15.153:6789/0 conn(0x7ff6ac12e770 legacy=0x7ff6ac09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 monclient: found mon.noname-c 2024-09-15T03:49:13.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.954+0000 7ff6aa7fc640 1 -- 172.21.15.181:0/3197550081 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac099b30 con 0x7ff6ac12eb40 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6aa7fc640 1 -- 172.21.15.181:0/3197550081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12eb40 msgr2=0x7ff6ac12ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6aa7fc640 1 --2- 172.21.15.181:0/3197550081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12eb40 0x7ff6ac12ef40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009940 tx=0x7ff6a00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2b60954af67a404 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 1 -- 172.21.15.181:0/3197550081 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0032440 con 0x7ff6ac12eb40 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: dump: 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 1 -- 172.21.15.181:0/3197550081 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a00325a0 con 0x7ff6ac12eb40 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 5 monclient: authenticate success, global_id 204565 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 1 -- 172.21.15.181:0/3197550081 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0032940 con 0x7ff6ac12eb40 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6a9ffb640 10 monclient: dump: 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.972 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 4 set_mon_vals no callback set 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 10 monclient: shutdown 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 -- 172.21.15.181:0/3197550081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12eb40 msgr2=0x7ff6ac12ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/3197550081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12eb40 0x7ff6ac12ef40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009940 tx=0x7ff6a00091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 -- 172.21.15.181:0/3197550081 shutdown_connections 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/3197550081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac133d30 0x7ff6ac136110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/3197550081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12eb40 0x7ff6ac12ef40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 -- 172.21.15.181:0/3197550081 >> 172.21.15.181:0/3197550081 conn(0x7ff6ac0a18b0 msgr2=0x7ff6ac09d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.957+0000 7ff6b18e8640 1 -- 172.21.15.181:0/3197550081 shutdown_connections 2024-09-15T03:49:13.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.958+0000 7ff6b18e8640 1 -- 172.21.15.181:0/3197550081 wait complete. 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.958+0000 7ff6b18e8640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.958+0000 7ff6b18e8640 10 monclient: monmap: 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.958+0000 7ff6b18e8640 1 Processor -- start 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 -- start start 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient: init 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 0x7ff6ac12e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ac12e770 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac12eb40 0x7ff6ac130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ac12eb40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6ac133d30 0x7ff6ac1311b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ac133d30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac0a0860 con 0x7ff6ac133d30 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac129e80 con 0x7ff6ac12e770 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac1325b0 con 0x7ff6ac12eb40 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6b18e8640 10 monclient(hunting): authenticate will time out at 1958.441433s 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 0x7ff6ac12e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 0x7ff6ac12e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48916/0 (socket says 172.21.15.181:48916) 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 -- 172.21.15.181:0/603417811 learned_addr learned my addr 172.21.15.181:0/603417811 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): get_auth_request con 0x7ff6ac12e770 auth_method 0 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aa7fc640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac12eb40 0x7ff6ac130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6ab7fe640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6ac133d30 0x7ff6ac1311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aa7fc640 10 monclient(hunting): get_auth_request con 0x7ff6ac12eb40 auth_method 0 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff6ac133d30 auth_method 0 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:13.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient(hunting): handle_auth_done global_id 204568 payload 824 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 -- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac12eb40 msgr2=0x7ff6ac130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac12eb40 0x7ff6ac130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 -- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6ac133d30 msgr2=0x7ff6ac1311b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6ac133d30 0x7ff6ac1311b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient: _finish_hunting 0 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 monclient: found mon.noname-b 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aaffd640 1 -- 172.21.15.181:0/603417811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a0009450 con 0x7ff6ac12e770 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6aa7fc640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac12eb40 0x7ff6ac130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.959+0000 7ff6ab7fe640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6ac133d30 0x7ff6ac1311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.961+0000 7ff6aaffd640 1 -- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 msgr2=0x7ff6ac12e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6aaffd640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 0x7ff6ac12e130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00c8b0 tx=0x7ff69c0026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00cea0 con 0x7ff6ac12e770 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: dump: 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: _finish_auth 0 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69c00aa60 con 0x7ff6ac12e770 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 5 monclient: authenticate success, global_id 204568 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 10 monclient: _renew_subs 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ac1291c0 con 0x7ff6ac12e770 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00ae20 con 0x7ff6ac12e770 2024-09-15T03:49:13.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff692ffd640 10 monclient: dump: 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:13.981 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 10 monclient: _renew_subs 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6a97fa640 4 set_mon_vals no callback set 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.962+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ac12d620 con 0x7ff6ac12e770 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff69c012630 con 0x7ff6ac12e770 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff6b18e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff6b18e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff6b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ac066600 con 0x7ff6ac12e770 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff692ffd640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff698038740 0x7ff69803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff69c04c470 con 0x7ff6ac12e770 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff6aa7fc640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff698038740 0x7ff69803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:13.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.963+0000 7ff6aa7fc640 10 monclient: get_auth_request con 0x7ff698038740 auth_method 0 2024-09-15T03:49:13.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.965+0000 7ff6aa7fc640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff698038740 0x7ff69803abf0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac076b90 tx=0x7ff6a0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:13.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.967+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff69c018850 con 0x7ff6ac12e770 2024-09-15T03:49:13.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.967+0000 7ff692ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:13.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:13.967+0000 7ff692ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:13.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.997+0000 7fdff115d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:13.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.997+0000 7fdff115d640 10 monclient: build_initial_monmap 2024-09-15T03:49:13.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.997+0000 7fdff115d640 10 monclient: monmap: 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:13.998934+0000 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:13.998934+0000 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:13.999 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.997+0000 7fdff115d640 1 Processor -- start 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.997+0000 7fdff115d640 1 -- start start 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.997+0000 7fdff115d640 10 monclient: init 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fdfec1318f0 0x7fdfec131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfec1318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfec09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfec136c20 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfec09f530 con 0x7fdfec136c20 2024-09-15T03:49:14.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfec09fbf0 con 0x7fdfec09ea50 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfec131d50 con 0x7fdfec1318f0 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdff115d640 10 monclient(hunting): authenticate will time out at 1951.638394s 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfeb59e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdfec136c20 0x7fdfec139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60600/0 (socket says 172.21.15.153:60600) 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfeb59e640 1 -- 172.21.15.153:0/3778984704 learned_addr learned my addr 172.21.15.153:0/3778984704 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 1 --2- 172.21.15.153:0/3778984704 >> v2:172.21.15.181:3300/0 conn(0x7fdfec1318f0 0x7fdfec131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): get_auth_request con 0x7fdfec1318f0 auth_method 0 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient(hunting): handle_auth_done global_id 204571 payload 274 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 1 -- 172.21.15.153:0/3778984704 >> v1:172.21.15.181:6789/0 conn(0x7fdfec09ea50 legacy=0x7fdfec09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 1 -- 172.21.15.153:0/3778984704 >> v1:172.21.15.153:6789/0 conn(0x7fdfec136c20 legacy=0x7fdfec139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 1 monclient: found mon.noname-c 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:13.998+0000 7fdfead9d640 1 -- 172.21.15.153:0/3778984704 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec09fe30 con 0x7fdfec1318f0 2024-09-15T03:49:14.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.001+0000 7fdfead9d640 1 -- 172.21.15.153:0/3778984704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec1318f0 msgr2=0x7fdfec131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.001+0000 7fdfead9d640 1 --2- 172.21.15.153:0/3778984704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec1318f0 0x7fdfec131f30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009850 tx=0x7fdfdc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f61f73b3bd505ca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 1 -- 172.21.15.153:0/3778984704 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc032440 con 0x7fdfec1318f0 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: dump: 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.011 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 1 -- 172.21.15.153:0/3778984704 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc0325a0 con 0x7fdfec1318f0 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 1 -- 172.21.15.153:0/3778984704 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc032940 con 0x7fdfec1318f0 2024-09-15T03:49:14.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 5 monclient: authenticate success, global_id 204571 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 4 set_mon_vals no callback set 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdfe9d9b640 10 monclient: dump: 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.013 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 10 monclient: shutdown 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 -- 172.21.15.153:0/3778984704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec1318f0 msgr2=0x7fdfec131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 --2- 172.21.15.153:0/3778984704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec1318f0 0x7fdfec131f30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009850 tx=0x7fdfdc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 -- 172.21.15.153:0/3778984704 shutdown_connections 2024-09-15T03:49:14.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 --2- 172.21.15.153:0/3778984704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec1318f0 0x7fdfec131f30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 -- 172.21.15.153:0/3778984704 >> 172.21.15.153:0/3778984704 conn(0x7fdfec067110 msgr2=0x7fdfec067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 -- 172.21.15.153:0/3778984704 shutdown_connections 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.002+0000 7fdff115d640 1 -- 172.21.15.153:0/3778984704 wait complete. 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.003+0000 7fdff115d640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.003+0000 7fdff115d640 10 monclient: monmap: 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:14.015 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.003+0000 7fdff115d640 1 Processor -- start 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.003+0000 7fdff115d640 1 -- start start 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.003+0000 7fdff115d640 10 monclient: init 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:49:14.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec09ea50 0x7fdfec1329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfec09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 0x7fdfec132f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfec1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfec136c20 0x7fdfec133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfec136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec133ef0 con 0x7fdfec136c20 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec136220 con 0x7fdfec09ea50 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec135f40 con 0x7fdfec1318f0 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdff115d640 10 monclient(hunting): authenticate will time out at 1951.644161s 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfec136c20 0x7fdfec133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 0x7fdfec132f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec09ea50 0x7fdfec1329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfec136c20 0x7fdfec133470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60756/0 (socket says 172.21.15.153:60756) 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 1 -- 172.21.15.153:0/1222361238 learned_addr learned my addr 172.21.15.153:0/1222361238 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): get_auth_request con 0x7fdfec136c20 auth_method 0 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): get_auth_request con 0x7fdfec1318f0 auth_method 0 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): get_auth_request con 0x7fdfec09ea50 auth_method 0 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient(hunting): handle_auth_done global_id 244235 payload 824 2024-09-15T03:49:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec09ea50 msgr2=0x7fdfec1329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfec09ea50 0x7fdfec1329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfec136c20 msgr2=0x7fdfec133470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfec136c20 0x7fdfec133470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 monclient: found mon.noname-c 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfea59c640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc009390 con 0x7fdfec1318f0 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.004+0000 7fdfeb59e640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfec136c20 0x7fdfec133470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfea59c640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 msgr2=0x7fdfec132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfea59c640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 0x7fdfec132f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000c510 tx=0x7fdfe0007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe00117a0 con 0x7fdfec1318f0 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 10 monclient: dump: 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe0011920 con 0x7fdfec1318f0 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdff115d640 5 monclient: authenticate success, global_id 244235 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0011ce0 con 0x7fdfec1318f0 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdff115d640 10 monclient: _renew_subs 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdff115d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec136500 con 0x7fdfec1318f0 2024-09-15T03:49:14.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfd2ffd640 10 monclient: dump: 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.024 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.006+0000 7fdfe959a640 4 set_mon_vals no callback set 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 10 monclient: _renew_subs 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec131430 con 0x7fdfec1318f0 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 0x7fdfec0a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec1343c0 con 0x7fdfec12d160 2024-09-15T03:49:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 0x7fdfec0a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fdfe0012630 con 0x7fdfec1318f0 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient: get_auth_request con 0x7fdfec12d160 auth_method 0 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.007+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.008+0000 7fdfd2ffd640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfd8038740 0x7fdfd803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.008+0000 7fdfead9d640 10 monclient(hunting): handle_auth_done global_id 244238 payload 824 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.008+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdfe004c1e0 con 0x7fdfec1318f0 2024-09-15T03:49:14.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.008+0000 7fdfead9d640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfec134b20 con 0x7fdfec12d160 2024-09-15T03:49:14.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.008+0000 7fdfeb59e640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfd8038740 0x7fdfd803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.008+0000 7fdfeb59e640 10 monclient: get_auth_request con 0x7fdfd8038740 auth_method 0 2024-09-15T03:49:14.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfead9d640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 msgr2=0x7fdfec0a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfead9d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 0x7fdfec0a3d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdfec09c9e0 tx=0x7fdfdc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc031930 con 0x7fdfec12d160 2024-09-15T03:49:14.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 10 monclient: dump: 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfeb59e640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfd8038740 0x7fdfd803abf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400d760 tx=0x7fdfe4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdfdc041070 con 0x7fdfec12d160 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 msgr2=0x7fdfec0a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 0x7fdfec0a3d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdfec09c9e0 tx=0x7fdfdc032000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 msgr2=0x7fdfec0a3d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.010+0000 7fdfd2ffd640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec12d160 0x7fdfec0a3d90 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdfec09c9e0 tx=0x7fdfdc032000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdff115d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdff115d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 0x7fdfec0a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdff115d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec065e00 con 0x7fdfec0a42d0 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 0x7fdfec0a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient: get_auth_request con 0x7fdfec0a42d0 auth_method 0 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 10 monclient(hunting): handle_auth_done global_id 244241 payload 824 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.016+0000 7fdfead9d640 1 -- 172.21.15.153:0/1222361238 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdfec1316b0 con 0x7fdfec0a42d0 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfead9d640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 msgr2=0x7fdfec0a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfead9d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 0x7fdfec0a46d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009130 tx=0x7fdfdc03f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc03d600 con 0x7fdfec0a42d0 2024-09-15T03:49:14.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 10 monclient: dump: 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fdfdc03d760 con 0x7fdfec0a42d0 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 msgr2=0x7fdfec0a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 0x7fdfec0a46d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009130 tx=0x7fdfdc03f070 comp rx=0 tx=0).stop 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 msgr2=0x7fdfec0a46d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfd2ffd640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec0a42d0 0x7fdfec0a46d0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009130 tx=0x7fdfdc03f070 comp rx=0 tx=0).stop 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:14.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 212, 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:14.036 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:14.037 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:14.038 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:14.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.040 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:14.041 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.042 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:14.043 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:14.044 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.045 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.047 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:14.047 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfd8038740 msgr2=0x7fdfd803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdfd8038740 0x7fdfd803abf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400d760 tx=0x7fdfe4009040 comp rx=0 tx=0).stop 2024-09-15T03:49:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 10 monclient: shutdown 2024-09-15T03:49:14.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 msgr2=0x7fdfec132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 0x7fdfec132f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000c510 tx=0x7fdfe0007e10 comp rx=0 tx=0).stop 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdfeb59e640 1 -- 172.21.15.153:0/1222361238 reap_dead start 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 shutdown_connections 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 --2- 172.21.15.153:0/1222361238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdfec1318f0 0x7fdfec132f30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 >> 172.21.15.153:0/1222361238 conn(0x7fdfec067110 msgr2=0x7fdfec12bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 shutdown_connections 2024-09-15T03:49:14.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.019+0000 7fdff115d640 1 -- 172.21.15.153:0/1222361238 wait complete. 2024-09-15T03:49:14.049 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:49:14.049 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:49:14.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:49:14.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.143+0000 7fb1f8294640 10 monclient: get_monmap_and_config 2024-09-15T03:49:14.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.143+0000 7fb1f8294640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.143+0000 7fb1f8294640 10 monclient: monmap: 2024-09-15T03:49:14.145 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:14.145148+0000 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:14.145148+0000 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:14.146 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:14.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.143+0000 7fb1f8294640 1 Processor -- start 2024-09-15T03:49:14.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 1 -- start start 2024-09-15T03:49:14.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient: init 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 0x7fb1f0098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f00981c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb1f0131da0 0x7fb1f0067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f0131da0 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1f0132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f0098ad0 con 0x7fb1f0132170 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f0131bd0 con 0x7fb1f0131da0 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f009ca60 con 0x7fb1f00981c0 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f8294640 10 monclient(hunting): authenticate will time out at 1951.784664s 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 0x7fb1f0098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb1f0131da0 0x7fb1f0067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 0x7fb1f0098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54270/0 (socket says 172.21.15.153:54270) 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 1 -- 172.21.15.153:0/1761330021 learned_addr learned my addr 172.21.15.153:0/1761330021 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): get_auth_request con 0x7fb1f0131da0 auth_method 0 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): get_auth_request con 0x7fb1f00981c0 auth_method 0 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f582f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.144+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 10 monclient(hunting): handle_auth_done global_id 254124 payload 274 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 1 -- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 msgr2=0x7fb1f0098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 1 --2- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 0x7fb1f0098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 1 -- 172.21.15.153:0/1761330021 >> v1:172.21.15.181:6789/0 conn(0x7fb1f0132170 legacy=0x7fb1f0067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 1 monclient: found mon.noname-a 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f582f640 1 -- 172.21.15.153:0/1761330021 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f009cca0 con 0x7fb1f0131da0 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.145+0000 7fb1f6030640 1 --2- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 0x7fb1f0098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.147+0000 7fb1f582f640 1 -- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f0131da0 msgr2=0x7fb1f0067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.147+0000 7fb1f582f640 1 --2- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f0131da0 0x7fb1f0067110 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009e60 tx=0x7fb1e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38cace26324a656 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 1 -- 172.21.15.153:0/1761330021 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4007760 con 0x7fb1f0131da0 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: dump: 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 1 -- 172.21.15.153:0/1761330021 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e402d050 con 0x7fb1f0131da0 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 5 monclient: authenticate success, global_id 254124 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 1 -- 172.21.15.153:0/1761330021 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4007af0 con 0x7fb1f0131da0 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 4 set_mon_vals no callback set 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f502e640 10 monclient: dump: 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.162 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 10 monclient: shutdown 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 -- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f0131da0 msgr2=0x7fb1f0067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 --2- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f0131da0 0x7fb1f0067110 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009e60 tx=0x7fb1e4009780 comp rx=0 tx=0).stop 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 -- 172.21.15.153:0/1761330021 shutdown_connections 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 --2- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f0131da0 0x7fb1f0067110 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 --2- 172.21.15.153:0/1761330021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f00981c0 0x7fb1f0098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 -- 172.21.15.153:0/1761330021 >> 172.21.15.153:0/1761330021 conn(0x7fb1f00a1e30 msgr2=0x7fb1f00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 -- 172.21.15.153:0/1761330021 shutdown_connections 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.148+0000 7fb1f8294640 1 -- 172.21.15.153:0/1761330021 wait complete. 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 10 monclient: monmap: 2024-09-15T03:49:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 1 Processor -- start 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 1 -- start start 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 10 monclient: init 2024-09-15T03:49:14.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.149+0000 7fb1f8294640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f00981c0 0x7fb1f011dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f00981c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f0131da0 0x7fb1f011e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f0131da0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0132170 0x7fb1f011e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f0132170 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f011f870 con 0x7fb1f00981c0 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f0098ce0 con 0x7fb1f0132170 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f0120490 con 0x7fb1f0131da0 2024-09-15T03:49:14.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f8294640 10 monclient(hunting): authenticate will time out at 1951.790060s 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f00981c0 0x7fb1f011dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f0131da0 0x7fb1f011e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f00981c0 0x7fb1f011dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60764/0 (socket says 172.21.15.153:60764) 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 1 -- 172.21.15.153:0/3121230947 learned_addr learned my addr 172.21.15.153:0/3121230947 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): get_auth_request con 0x7fb1f00981c0 auth_method 0 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): get_auth_request con 0x7fb1f0131da0 auth_method 0 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0132170 0x7fb1f011e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): get_auth_request con 0x7fb1f0132170 auth_method 0 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6831640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient(hunting): handle_auth_done global_id 244247 payload 824 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0132170 msgr2=0x7fb1f011e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0132170 0x7fb1f011e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f00981c0 msgr2=0x7fb1f011dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f00981c0 0x7fb1f011dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 monclient: found mon.noname-c 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f582f640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8009450 con 0x7fb1f0131da0 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.150+0000 7fb1f6030640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1f00981c0 0x7fb1f011dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f582f640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f0131da0 msgr2=0x7fb1f011e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f582f640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f0131da0 0x7fb1f011e200 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4031000 tx=0x7fb1e4037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4000fc0 con 0x7fb1f0131da0 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: dump: 2024-09-15T03:49:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 5 monclient: authenticate success, global_id 244247 2024-09-15T03:49:14.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e4040070 con 0x7fb1f0131da0 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 10 monclient: _renew_subs 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e4009a20 con 0x7fb1f0131da0 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4031d10 con 0x7fb1f0131da0 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1ee7fc640 10 monclient: dump: 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.171 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f482d640 4 set_mon_vals no callback set 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 10 monclient: _renew_subs 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.152+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f0131300 con 0x7fb1f0131da0 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f8294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f8294640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 0x7fb1f011a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f0121760 con 0x7fb1f011ec80 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fb1e4042030 con 0x7fb1f0131da0 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 0x7fb1f011a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1ee7fc640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1dc038740 0x7fb1dc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient: get_auth_request con 0x7fb1f011ec80 auth_method 0 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb1e406d9d0 con 0x7fb1f0131da0 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.153+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.154+0000 7fb1f6831640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1dc038740 0x7fb1dc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.154+0000 7fb1f6831640 10 monclient: get_auth_request con 0x7fb1dc038740 auth_method 0 2024-09-15T03:49:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.154+0000 7fb1f6030640 10 monclient(hunting): handle_auth_done global_id 204580 payload 824 2024-09-15T03:49:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.154+0000 7fb1f6030640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1f01314f0 con 0x7fb1f011ec80 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff6b18e8640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff6b18e8640 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff6b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7ff6ac1293b0 con 0x7ff6ac12e770 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (secure 0 0 0) 0x7ff69c016070 con 0x7ff6ac12e770 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff692ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-15T03:49:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.161+0000 7ff692ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1f6030640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 msgr2=0x7fb1f011a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1f6030640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 0x7fb1f011a4a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f4a0 tx=0x7fb1e80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1f6831640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1dc038740 0x7fb1dc03abf0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0120110 tx=0x7fb1e0007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8013a40 con 0x7fb1f011ec80 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 10 monclient: dump: 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.178 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb1e8013ba0 con 0x7fb1f011ec80 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:14.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 msgr2=0x7fb1f011a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 0x7fb1f011a4a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f4a0 tx=0x7fb1e80091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 msgr2=0x7fb1f011a4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.156+0000 7fb1ee7fc640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011ec80 0x7fb1f011a4a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.162+0000 7fb1f8294640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.162+0000 7fb1f8294640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 0x7fb1f011ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.162+0000 7fb1f8294640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.162+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f0065f90 con 0x7fb1f011a9e0 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 0x7fb1f011ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient: get_auth_request con 0x7fb1f011a9e0 auth_method 0 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 10 monclient(hunting): handle_auth_done global_id 204583 payload 824 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.163+0000 7fb1f6030640 1 -- 172.21.15.153:0/3121230947 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1f01213f0 con 0x7fb1f011a9e0 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.165+0000 7fb1f6030640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 msgr2=0x7fb1f011ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.165+0000 7fb1f6030640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 0x7fb1f011ade0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f4a0 tx=0x7fb1e801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.165+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e801dd30 con 0x7fb1f011a9e0 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.165+0000 7fb1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.165+0000 7fb1ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.165+0000 7fb1ee7fc640 10 monclient: dump: 2024-09-15T03:49:14.181 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fb1e8013830 con 0x7fb1f011a9e0 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:14.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 msgr2=0x7fb1f011ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 0x7fb1f011ade0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f4a0 tx=0x7fb1e801c000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 msgr2=0x7fb1f011ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1ee7fc640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f011a9e0 0x7fb1f011ade0 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800f4a0 tx=0x7fb1e801c000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:14.183 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 212, 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:14.184 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.186 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:14.187 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:14.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:14.192 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.194 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.195 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:14.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:14.197 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:14.197 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1dc038740 msgr2=0x7fb1dc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fb1dc038740 0x7fb1dc03abf0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0120110 tx=0x7fb1e0007e10 comp rx=0 tx=0).stop 2024-09-15T03:49:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 10 monclient: shutdown 2024-09-15T03:49:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f0131da0 msgr2=0x7fb1f011e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 --2- 172.21.15.153:0/3121230947 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1f0131da0 0x7fb1f011e200 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4031000 tx=0x7fb1e4037000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f6831640 1 -- 172.21.15.153:0/3121230947 reap_dead start 2024-09-15T03:49:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f6831640 1 -- 172.21.15.153:0/3121230947 reap_dead start 2024-09-15T03:49:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 shutdown_connections 2024-09-15T03:49:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 >> 172.21.15.153:0/3121230947 conn(0x7fb1f00a1e30 msgr2=0x7fb1f0096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 shutdown_connections 2024-09-15T03:49:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.166+0000 7fb1f8294640 1 -- 172.21.15.153:0/3121230947 wait complete. 2024-09-15T03:49:14.199 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:49:14.199 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:49:14.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:49:14.289 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.287+0000 7ff6b18e8640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-15T03:49:14.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.287+0000 7ff6b18e8640 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-15T03:49:14.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.287+0000 7ff6b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.287+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7ff6ac0008d0 con 0x7ff6ac12e770 2024-09-15T03:49:14.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.288+0000 7ff692ffd640 1 -- 172.21.15.181:0/603417811 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (secure 0 0 0) 0x7ff69c0182c0 con 0x7ff6ac12e770 2024-09-15T03:49:14.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.288+0000 7ff692ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-15T03:49:14.290 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.288+0000 7ff692ffd640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:49:14.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff698038740 msgr2=0x7ff69803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff698038740 0x7ff69803abf0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac076b90 tx=0x7ff6a0032000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 10 monclient: shutdown 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 msgr2=0x7ff6ac12e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 0x7ff6ac12e130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00c8b0 tx=0x7ff69c0026c0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 shutdown_connections 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff698038740 0x7ff69803abf0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6ac133d30 0x7ff6ac1311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6ac12eb40 0x7ff6ac130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 --2- 172.21.15.181:0/603417811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6ac12e770 0x7ff6ac12e130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 >> 172.21.15.181:0/603417811 conn(0x7ff6ac0a18b0 msgr2=0x7ff6ac0983f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.289+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 shutdown_connections 2024-09-15T03:49:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.290+0000 7ff6b18e8640 1 -- 172.21.15.181:0/603417811 wait complete. 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.290+0000 7f309544f640 10 monclient: get_monmap_and_config 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.290+0000 7f309544f640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.290+0000 7f309544f640 10 monclient: monmap: 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:14.292642+0000 2024-09-15T03:49:14.295 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:14.292642+0000 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.291+0000 7f309544f640 1 Processor -- start 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.291+0000 7f309544f640 1 -- start start 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.291+0000 7f309544f640 10 monclient: init 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30901318f0 0x7f3090131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): picked mon.noname-a con 0x7f30901318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f309009ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): picked mon.noname-b con 0x7f3090136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3090131cc0 con 0x7f309009ea50 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309009f530 con 0x7f30901318f0 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309009fbf0 con 0x7f3090136c20 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f309544f640 10 monclient(hunting): authenticate will time out at 1951.932182s 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30901318f0 0x7f3090131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f30901318f0 0x7f3090131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60774/0 (socket says 172.21.15.153:60774) 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 -- 172.21.15.153:0/2826960911 learned_addr learned my addr 172.21.15.153:0/2826960911 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): get_auth_request con 0x7f30901318f0 auth_method 0 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): get_auth_request con 0x7f3090136c20 auth_method 0 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient(hunting): handle_auth_done global_id 254130 payload 274 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 -- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 msgr2=0x7f3090139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 -- 172.21.15.153:0/2826960911 >> v1:172.21.15.181:6789/0 conn(0x7f309009ea50 legacy=0x7f309009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 monclient: found mon.noname-a 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308effd640 1 -- 172.21.15.153:0/2826960911 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309009fe30 con 0x7f30901318f0 2024-09-15T03:49:14.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.292+0000 7f308f7fe640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:14.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308effd640 1 -- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30901318f0 msgr2=0x7f3090131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308effd640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30901318f0 0x7f3090131f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3080009910 tx=0x7f30800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6091e8429408d606 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 1 -- 172.21.15.153:0/2826960911 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308002aea0 con 0x7f30901318f0 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: dump: 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 1 -- 172.21.15.153:0/2826960911 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080032440 con 0x7f30901318f0 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f309544f640 5 monclient: authenticate success, global_id 254130 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f309544f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 1 -- 172.21.15.153:0/2826960911 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30800327e0 con 0x7f30901318f0 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f309544f640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f309544f640 4 set_mon_vals no callback set 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.295+0000 7f308dffb640 10 monclient: dump: 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 10 monclient: shutdown 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 -- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30901318f0 msgr2=0x7f3090131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30901318f0 0x7f3090131f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3080009910 tx=0x7f30800091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 -- 172.21.15.153:0/2826960911 shutdown_connections 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 --2- 172.21.15.153:0/2826960911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30901318f0 0x7f3090131f30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 -- 172.21.15.153:0/2826960911 >> 172.21.15.153:0/2826960911 conn(0x7f3090067110 msgr2=0x7f3090067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 -- 172.21.15.153:0/2826960911 shutdown_connections 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 1 -- 172.21.15.153:0/2826960911 wait complete. 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.296+0000 7f309544f640 10 monclient: monmap: 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:14.309 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 Processor -- start 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 -- start start 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient: init 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 0x7f3090130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): picked mon.noname-a con 0x7f309009ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30901318f0 0x7f30900997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): picked mon.noname-b con 0x7f30901318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090099cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): picked mon.noname-c con 0x7f3090136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3090141f20 con 0x7f309009ea50 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309009b2e0 con 0x7f30901318f0 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309011cd60 con 0x7f3090136c20 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.297+0000 7f309544f640 10 monclient(hunting): authenticate will time out at 1951.937943s 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090099cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 0x7f3090130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090099cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:54302/0 (socket says 172.21.15.153:54302) 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 0x7f3090130970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60776/0 (socket says 172.21.15.153:60776) 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 1 -- 172.21.15.153:0/1491621177 learned_addr learned my addr 172.21.15.153:0/1491621177 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): get_auth_request con 0x7f3090136c20 auth_method 0 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): get_auth_request con 0x7f309009ea50 auth_method 0 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30901318f0 0x7f30900997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): get_auth_request con 0x7f30901318f0 auth_method 0 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient(hunting): handle_auth_done global_id 254133 payload 824 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 -- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 msgr2=0x7f3090099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090099cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 -- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30901318f0 msgr2=0x7f30900997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30901318f0 0x7f30900997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 monclient: found mon.noname-a 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308effd640 1 -- 172.21.15.153:0/1491621177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088009a20 con 0x7f309009ea50 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308f7fe640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.298+0000 7f308e7fc640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30901318f0 0x7f30900997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f308effd640 1 -- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 msgr2=0x7f3090130970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f308effd640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 0x7f3090130970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30900a5020 tx=0x7f3080032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308002aea0 con 0x7f309009ea50 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: dump: 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.315 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080031c70 con 0x7f309009ea50 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f309544f640 5 monclient: authenticate success, global_id 254133 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308003d420 con 0x7f309009ea50 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f309544f640 10 monclient: _renew_subs 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f309544f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3080009450 con 0x7f309009ea50 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f3076ffd640 10 monclient: dump: 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:14.316 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.300+0000 7f308d7fa640 4 set_mon_vals no callback set 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.301+0000 7f309544f640 10 monclient: _renew_subs 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.301+0000 7f309544f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.301+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309011b720 con 0x7f309009ea50 2024-09-15T03:49:14.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.301+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3080039040 con 0x7f309009ea50 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f309544f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f309544f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f309544f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309004bf60 con 0x7f309009ea50 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f3076ffd640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f307c038730 0x7f307c03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f308006c310 con 0x7f309009ea50 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f308e7fc640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f307c038730 0x7f307c03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.302+0000 7f308e7fc640 10 monclient: get_auth_request con 0x7f307c038730 auth_method 0 2024-09-15T03:49:14.319 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph health detail 2024-09-15T03:49:14.319 INFO:tasks.workunit.client.0.smithi181.stderr:+ grep client.bad 2024-09-15T03:49:14.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.304+0000 7f308e7fc640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f307c038730 0x7f307c03abe0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f30900a4ef0 tx=0x7f3084007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.306+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f30800398f0 con 0x7f309009ea50 2024-09-15T03:49:14.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.306+0000 7f3076ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.306+0000 7f3076ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:14.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.366+0000 7f027eb9d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.366+0000 7f027eb9d640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.366+0000 7f027eb9d640 10 monclient: monmap: 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:49:14.368416+0000 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:49:14.368416+0000 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:14.380 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 Processor -- start 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 -- start start 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient: init 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.381 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f027809e1d0 0x7f027809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f027809e1d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f027812e770 0x7f027812eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f027812e770 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f027812f0b0 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02780995e0 con 0x7f027812f0b0 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027809cda0 con 0x7f027812e770 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0278099b30 con 0x7f027809e1d0 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.382 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027eb9d640 10 monclient(hunting): authenticate will time out at 1958.849849s 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f027809e1d0 0x7f027809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f027809e1d0 0x7f027809aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48922/0 (socket says 172.21.15.181:48922) 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 1 -- 172.21.15.181:0/3889828530 learned_addr learned my addr 172.21.15.181:0/3889828530 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 10 monclient(hunting): get_auth_request con 0x7f027809e1d0 auth_method 0 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.367+0000 7f027c939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f0277fff640 1 --2- 172.21.15.181:0/3889828530 >> v2:172.21.15.153:3300/0 conn(0x7f027812e770 0x7f027812eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f0277fff640 10 monclient(hunting): get_auth_request con 0x7f027812e770 auth_method 0 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f0277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f0277fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f0277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 10 monclient(hunting): handle_auth_done global_id 204589 payload 274 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 1 -- 172.21.15.181:0/3889828530 >> v2:172.21.15.153:3300/0 conn(0x7f027812e770 msgr2=0x7f027812eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 1 --2- 172.21.15.181:0/3889828530 >> v2:172.21.15.153:3300/0 conn(0x7f027812e770 0x7f027812eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 1 -- 172.21.15.181:0/3889828530 >> v1:172.21.15.153:6789/0 conn(0x7f027812f0b0 legacy=0x7f027812b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 1 monclient: found mon.noname-c 2024-09-15T03:49:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.368+0000 7f027c939640 1 -- 172.21.15.181:0/3889828530 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027809afa0 con 0x7f027809e1d0 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027c939640 1 -- 172.21.15.181:0/3889828530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027809e1d0 msgr2=0x7f027809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027c939640 1 --2- 172.21.15.181:0/3889828530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027809e1d0 0x7f027809aa60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f02780606b0 tx=0x7f026c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c0bbb607c47dfff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 1 -- 172.21.15.181:0/3889828530 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c032440 con 0x7f027809e1d0 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: dump: 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.385 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 1 -- 172.21.15.181:0/3889828530 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c0325a0 con 0x7f027809e1d0 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 5 monclient: authenticate success, global_id 204589 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 4 set_mon_vals no callback set 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 1 -- 172.21.15.181:0/3889828530 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c032940 con 0x7f027809e1d0 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f02777fe640 10 monclient: dump: 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 10 monclient: shutdown 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 1 -- 172.21.15.181:0/3889828530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027809e1d0 msgr2=0x7f027809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3889828530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027809e1d0 0x7f027809aa60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f02780606b0 tx=0x7f026c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 1 -- 172.21.15.181:0/3889828530 shutdown_connections 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3889828530 >> v2:172.21.15.153:3300/0 conn(0x7f027812e770 0x7f027812eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3889828530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027809e1d0 0x7f027809aa60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.371+0000 7f027eb9d640 1 -- 172.21.15.181:0/3889828530 >> 172.21.15.181:0/3889828530 conn(0x7f02780a18b0 msgr2=0x7f027809d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 1 -- 172.21.15.181:0/3889828530 shutdown_connections 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 1 -- 172.21.15.181:0/3889828530 wait complete. 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 10 monclient: monmap: 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:49:14.389 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 1 Processor -- start 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 1 -- start start 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.372+0000 7f027eb9d640 10 monclient: init 2024-09-15T03:49:14.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f027809e1d0 0x7f027813dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f027809e1d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 0x7f027812d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f027812e770 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f027812f0b0 0x7f0278118390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f027812f0b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027809b280 con 0x7f027809e1d0 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027811bfc0 con 0x7f027812e770 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027811bce0 con 0x7f027812f0b0 2024-09-15T03:49:14.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027eb9d640 10 monclient(hunting): authenticate will time out at 1958.855430s 2024-09-15T03:49:14.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 0x7f027812d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 0x7f027812d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48938/0 (socket says 172.21.15.181:48938) 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 -- 172.21.15.181:0/3065031495 learned_addr learned my addr 172.21.15.181:0/3065031495 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): get_auth_request con 0x7f027812e770 auth_method 0 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f027809e1d0 0x7f027813dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): get_auth_request con 0x7f027809e1d0 auth_method 0 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f027812f0b0 0x7f0278118390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): get_auth_request con 0x7f027812f0b0 auth_method 0 2024-09-15T03:49:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027c939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f027d13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient(hunting): handle_auth_done global_id 204592 payload 824 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 -- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f027812f0b0 msgr2=0x7f0278118390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f027812f0b0 0x7f0278118390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 -- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f027809e1d0 msgr2=0x7f027813dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f027809e1d0 0x7f027813dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 monclient: found mon.noname-b 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.373+0000 7f0277fff640 1 -- 172.21.15.181:0/3065031495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c009450 con 0x7f027812e770 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0277fff640 1 -- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 msgr2=0x7f027812d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0277fff640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 0x7f027812d8d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f026400d890 tx=0x7f0264008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026400de20 con 0x7f027812e770 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: dump: 2024-09-15T03:49:14.397 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026400bae0 con 0x7f027812e770 2024-09-15T03:49:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 5 monclient: authenticate success, global_id 204592 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 10 monclient: _renew_subs 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027813ea10 con 0x7f027812e770 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026400be80 con 0x7f027812e770 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 10 monclient: dump: 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0276ffd640 4 set_mon_vals no callback set 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 10 monclient: _renew_subs 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027811ac40 con 0x7f027812e770 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.376+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0264014640 con 0x7f027812e770 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f0274ff9640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f025c038740 0x7f025c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f027eb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f027eb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f027eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f026404cc40 con 0x7f027812e770 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027811bb40 con 0x7f027812e770 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f027c939640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f025c038740 0x7f025c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.377+0000 7f027c939640 10 monclient: get_auth_request con 0x7f025c038740 auth_method 0 2024-09-15T03:49:14.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.379+0000 7f027c939640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f025c038740 0x7f025c03abf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f027809a020 tx=0x7f026c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.381+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f027811bb40 con 0x7f027812e770 2024-09-15T03:49:14.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.381+0000 7f0274ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.381+0000 7f0274ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:14.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f309544f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:49:14.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f309544f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:49:14.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f309544f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:14.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f30900638d0 con 0x7f309009ea50 2024-09-15T03:49:14.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f3076ffd640 1 -- 172.21.15.153:0/1491621177 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3080037080 con 0x7f309009ea50 2024-09-15T03:49:14.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f3076ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:49:14.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.529+0000 7f3076ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f307c038730 msgr2=0x7f307c03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f307c038730 0x7f307c03abe0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f30900a4ef0 tx=0x7f3084007e10 comp rx=0 tx=0).stop 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 10 monclient: shutdown 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 msgr2=0x7f3090130970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 0x7f3090130970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30900a5020 tx=0x7f3080032000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 shutdown_connections 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f307c038730 0x7f307c03abe0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3090136c20 0x7f3090099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30901318f0 0x7f30900997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 --2- 172.21.15.153:0/1491621177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f309009ea50 0x7f3090130970 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 >> 172.21.15.153:0/1491621177 conn(0x7f3090067110 msgr2=0x7f30900a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 shutdown_connections 2024-09-15T03:49:14.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:14.532+0000 7f309544f640 1 -- 172.21.15.153:0/1491621177 wait complete. 2024-09-15T03:49:14.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f027eb9d640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f027eb9d640 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f027eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f027811bb40 con 0x7f027812e770 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (secure 0 0 0) 0x7f027811bb40 con 0x7f027812e770 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f0274ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-15T03:49:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.596+0000 7f0274ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:14.599 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.597+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:14.599 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.597+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=1641470795,config=3140514009,health=2074139178,kv=3235807718,logm=3909386444}) 2024-09-15T03:49:14.599 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.597+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=32,config=2,health=11,kv=7,logm=48} crc {auth=1411273318,config=3140514009,health=2046447255,kv=2145712985,logm=1794056746}) 2024-09-15T03:49:14.599 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.597+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:14.599 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.597+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=1641470795,config=3140514009,health=2074139178,kv=3235807718,logm=3909386444}) 2024-09-15T03:49:14.599 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.597+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=33,config=2,health=10,kv=6,logm=49} crc {auth=359810687,config=3140514009,health=2125639532,kv=3991602445,logm=174949450}) 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.746+0000 7f027eb9d640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.746+0000 7f027eb9d640 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.746+0000 7f027eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.746+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f027811bb40 con 0x7f027812e770 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.747+0000 7f0274ff9640 1 -- 172.21.15.181:0/3065031495 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (secure 0 0 0) 0x7f027811bb40 con 0x7f027812e770 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.747+0000 7f0274ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-15T03:49:14.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.747+0000 7f0274ff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:49:14.750 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.748+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:14.750 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.748+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231760629}) 2024-09-15T03:49:14.750 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.748+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994413852}) 2024-09-15T03:49:14.750 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.748+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:14.750 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.748+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231760629}) 2024-09-15T03:49:14.750 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:14.748+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573271541}) 2024-09-15T03:49:14.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f025c038740 msgr2=0x7f025c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f025c038740 0x7f025c03abf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f027809a020 tx=0x7f026c032000 comp rx=0 tx=0).stop 2024-09-15T03:49:14.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 10 monclient: shutdown 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 msgr2=0x7f027812d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 0x7f027812d8d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f026400d890 tx=0x7f0264008040 comp rx=0 tx=0).stop 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 shutdown_connections 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f025c038740 0x7f025c03abf0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f027812f0b0 0x7f0278118390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f027812e770 0x7f027812d8d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 --2- 172.21.15.181:0/3065031495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f027809e1d0 0x7f027813dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 >> 172.21.15.181:0/3065031495 conn(0x7f02780a18b0 msgr2=0x7f02780a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 shutdown_connections 2024-09-15T03:49:14.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.749+0000 7f027eb9d640 1 -- 172.21.15.181:0/3065031495 wait complete. 2024-09-15T03:49:14.759 INFO:tasks.workunit.client.0.smithi181.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-15T03:49:14.759 INFO:tasks.workunit.client.0.smithi181.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-15T03:49:14.759 INFO:tasks.workunit.client.0.smithi181.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-15T03:49:14.759 INFO:tasks.workunit.client.0.smithi181.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-15T03:49:14.760 INFO:tasks.workunit.client.0.smithi181.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-15T03:49:14.760 INFO:tasks.workunit.client.0.smithi181.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-15T03:49:14.760 INFO:tasks.workunit.client.0.smithi181.stderr:+ ceph auth rm client.bad 2024-09-15T03:49:14.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.826+0000 7fa406c34640 10 monclient: get_monmap_and_config 2024-09-15T03:49:14.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.826+0000 7fa406c34640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.826+0000 7fa406c34640 10 monclient: monmap: 2024-09-15T03:49:14.828 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:14.828 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:49:14.828496+0000 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:49:14.828496+0000 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:14.829 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:14.832 INFO:tasks.workunit.client.0.smithi181.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:14.832 INFO:tasks.workunit.client.0.smithi181.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:14.832 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 Processor -- start 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 -- start start 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient: init 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa4001302d0 0x7fa40009ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): picked mon.noname-a con 0x7fa4001302d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009b760 0x7fa40009d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): picked mon.noname-b con 0x7fa40009b760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa40009d5b0 0x7fa40009d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): picked mon.noname-c con 0x7fa40009d5b0 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:14.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40009df30 con 0x7fa4001302d0 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4000980f0 con 0x7fa40009d5b0 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa400097850 con 0x7fa40009b760 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.827+0000 7fa406c34640 10 monclient(hunting): authenticate will time out at 1959.309907s 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa40009d5b0 0x7fa40009d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fa40009d5b0 0x7fa40009d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48942/0 (socket says 172.21.15.181:48942) 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 -- 172.21.15.181:0/612320663 learned_addr learned my addr 172.21.15.181:0/612320663 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): get_auth_request con 0x7fa40009d5b0 auth_method 0 2024-09-15T03:49:14.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4049d0640 1 --2- 172.21.15.181:0/612320663 >> v2:172.21.15.153:3300/0 conn(0x7fa4001302d0 0x7fa40009ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa3fffff640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009b760 0x7fa40009d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4049d0640 10 monclient(hunting): get_auth_request con 0x7fa4001302d0 auth_method 0 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4049d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4049d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4049d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient(hunting): handle_auth_done global_id 204595 payload 274 2024-09-15T03:49:14.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 -- 172.21.15.181:0/612320663 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009b760 msgr2=0x7fa40009d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009b760 0x7fa40009d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 -- 172.21.15.181:0/612320663 >> v2:172.21.15.153:3300/0 conn(0x7fa4001302d0 msgr2=0x7fa40009ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 --2- 172.21.15.181:0/612320663 >> v2:172.21.15.153:3300/0 conn(0x7fa4001302d0 0x7fa40009ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 monclient: found mon.noname-c 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa4051d1640 1 -- 172.21.15.181:0/612320663 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000983b0 con 0x7fa40009d5b0 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa3fffff640 10 monclient: get_auth_request con 0x7fa40009b760 auth_method 0 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.828+0000 7fa3fffff640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009b760 0x7fa40009d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa4051d1640 1 -- 172.21.15.181:0/612320663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009d5b0 msgr2=0x7fa40009d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa4051d1640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009d5b0 0x7fa40009d9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa400067b60 tx=0x7fa3f8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da9c863649f01ffc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 1 -- 172.21.15.181:0/612320663 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8007610 con 0x7fa40009d5b0 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: dump: 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 1 -- 172.21.15.181:0/612320663 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f802d050 con 0x7fa40009d5b0 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 5 monclient: authenticate success, global_id 204595 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 1 -- 172.21.15.181:0/612320663 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80079a0 con 0x7fa40009d5b0 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 4 set_mon_vals no callback set 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa3ff7fe640 10 monclient: dump: 2024-09-15T03:49:14.848 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 10 monclient: shutdown 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 -- 172.21.15.181:0/612320663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009d5b0 msgr2=0x7fa40009d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009d5b0 0x7fa40009d9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa400067b60 tx=0x7fa3f8009840 comp rx=0 tx=0).stop 2024-09-15T03:49:14.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 -- 172.21.15.181:0/612320663 shutdown_connections 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009d5b0 0x7fa40009d9f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 --2- 172.21.15.181:0/612320663 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009b760 0x7fa40009d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 --2- 172.21.15.181:0/612320663 >> v2:172.21.15.153:3300/0 conn(0x7fa4001302d0 0x7fa40009ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 -- 172.21.15.181:0/612320663 >> 172.21.15.181:0/612320663 conn(0x7fa4000a20c0 msgr2=0x7fa40009e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 -- 172.21.15.181:0/612320663 shutdown_connections 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.831+0000 7fa406c34640 1 -- 172.21.15.181:0/612320663 wait complete. 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.832+0000 7fa406c34640 10 monclient: build_initial_monmap 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.832+0000 7fa406c34640 10 monclient: monmap: 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 0 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 0.000000 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:created 0.000000 2024-09-15T03:49:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.832+0000 7fa406c34640 1 Processor -- start 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.832+0000 7fa406c34640 1 -- start start 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.832+0000 7fa406c34640 10 monclient: init 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 0x7fa40012cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): picked mon.noname-b con 0x7fa40009b760 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009d5b0 0x7fa40013ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): picked mon.noname-c con 0x7fa40009d5b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4001302d0 0x7fa40013f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): picked mon.noname-a con 0x7fa4001302d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40012a050 con 0x7fa4001302d0 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40012a770 con 0x7fa40009b760 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000a0fd0 con 0x7fa40009d5b0 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa406c34640 10 monclient(hunting): authenticate will time out at 1959.315424s 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 0x7fa40012cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 0x7fa40012cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:48958/0 (socket says 172.21.15.181:48958) 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 -- 172.21.15.181:0/733539005 learned_addr learned my addr 172.21.15.181:0/733539005 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): get_auth_request con 0x7fa40009b760 auth_method 0 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4001302d0 0x7fa40013f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): get_auth_request con 0x7fa4001302d0 auth_method 0 2024-09-15T03:49:14.853 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009d5b0 0x7fa40013ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): get_auth_request con 0x7fa40009d5b0 auth_method 0 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4051d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient(hunting): handle_auth_done global_id 204598 payload 824 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 -- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009d5b0 msgr2=0x7fa40013ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009d5b0 0x7fa40013ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 -- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4001302d0 msgr2=0x7fa40013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4001302d0 0x7fa40013f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient: _finish_hunting 0 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 monclient: found mon.noname-b 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.833+0000 7fa4049d0640 1 -- 172.21.15.181:0/733539005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8009ae0 con 0x7fa40009b760 2024-09-15T03:49:14.855 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa4049d0640 1 -- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 msgr2=0x7fa40012cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa4049d0640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 0x7fa40012cc30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000d890 tx=0x7fa3f0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f000de20 con 0x7fa40009b760 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: dump: 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.856 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: _finish_auth 0 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 5 monclient: authenticate success, global_id 204598 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f000bae0 con 0x7fa40009b760 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 10 monclient: _renew_subs 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40012baa0 con 0x7fa40009b760 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f000be80 con 0x7fa40009b760 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:14.857 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3fcff9640 10 monclient: dump: 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:epoch 1 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:election_strategy: 1 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 10 monclient: _renew_subs 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40013f650 con 0x7fa40009b760 2024-09-15T03:49:14.858 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.836+0000 7fa3feffd640 4 set_mon_vals no callback set 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa3f0014640 con 0x7fa40009b760 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa406c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa406c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa406c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa400066600 con 0x7fa40009b760 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa3fcff9640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3ec038770 0x7fa3ec03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa3f004cc40 con 0x7fa40009b760 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa3fffff640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3ec038770 0x7fa3ec03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.837+0000 7fa3fffff640 10 monclient: get_auth_request con 0x7fa3ec038770 auth_method 0 2024-09-15T03:49:14.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.840+0000 7fa3fffff640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3ec038770 0x7fa3ec03ac20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400acc0 tx=0x7fa3f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.842+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa3f001eb40 con 0x7fa40009b760 2024-09-15T03:49:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.842+0000 7fa3fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:14.842+0000 7fa3fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:15.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713590582}) 2024-09-15T03:49:15.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410908565}) 2024-09-15T03:49:15.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713590582}) 2024-09-15T03:49:15.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1720919760}) 2024-09-15T03:49:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.055+0000 7fa406c34640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T03:49:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.055+0000 7fa406c34640 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T03:49:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.055+0000 7fa406c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:15.057 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.055+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fa400067ee0 con 0x7fa40009b760 2024-09-15T03:49:15.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.056+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.056+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572797208}) 2024-09-15T03:49:15.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.056+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228637336}) 2024-09-15T03:49:15.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.056+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.056+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572797208}) 2024-09-15T03:49:15.058 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.056+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869639214}) 2024-09-15T03:49:15.060 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.058+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.060 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.058+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74055848}) 2024-09-15T03:49:15.060 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.058+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=785440064}) 2024-09-15T03:49:15.060 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.058+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.060 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.058+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74055848}) 2024-09-15T03:49:15.060 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.058+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2453744498}) 2024-09-15T03:49:15.063 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.061+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.063 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.061+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131369248}) 2024-09-15T03:49:15.063 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.061+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1713349205}) 2024-09-15T03:49:15.063 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.061+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.063 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.061+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131369248}) 2024-09-15T03:49:15.063 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.061+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207576371}) 2024-09-15T03:49:15.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.061+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v39) v1 ==== 79+0+0 (secure 0 0 0) 0x7fa3f001dbb0 con 0x7fa40009b760 2024-09-15T03:49:15.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.061+0000 7fa3fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T03:49:15.064 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.061+0000 7fa3fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:15.103 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.101+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.103 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.101+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981136983}) 2024-09-15T03:49:15.103 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.101+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669429345}) 2024-09-15T03:49:15.103 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.101+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.103 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.101+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981136983}) 2024-09-15T03:49:15.103 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.101+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376416434}) 2024-09-15T03:49:15.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.107+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.107+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506828910}) 2024-09-15T03:49:15.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.107+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1594400744}) 2024-09-15T03:49:15.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.107+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.109 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.107+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506828910}) 2024-09-15T03:49:15.110 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.107+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419447613}) 2024-09-15T03:49:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.213+0000 7fa406c34640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T03:49:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.213+0000 7fa406c34640 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T03:49:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.213+0000 7fa406c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.181:3300/0 2024-09-15T03:49:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.213+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fa4001284d0 con 0x7fa40009b760 2024-09-15T03:49:15.217 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.215+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.217 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.215+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644336966}) 2024-09-15T03:49:15.217 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.215+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089867535}) 2024-09-15T03:49:15.217 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.215+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.218 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.215+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644336966}) 2024-09-15T03:49:15.218 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.215+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184331380}) 2024-09-15T03:49:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.216+0000 7fa3fcff9640 1 -- 172.21.15.181:0/733539005 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v39) v1 ==== 79+0+0 (secure 0 0 0) 0x7fa3f0011340 con 0x7fa40009b760 2024-09-15T03:49:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.216+0000 7fa3fcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-15T03:49:15.218 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.216+0000 7fa3fcff9640 10 monclient: _finish_command 3 = system:0 2024-09-15T03:49:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3ec038770 msgr2=0x7fa3ec03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3ec038770 0x7fa3ec03ac20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400acc0 tx=0x7fa3f4008040 comp rx=0 tx=0).stop 2024-09-15T03:49:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 10 monclient: shutdown 2024-09-15T03:49:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 msgr2=0x7fa40012cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.220 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 0x7fa40012cc30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000d890 tx=0x7fa3f0008040 comp rx=0 tx=0).stop 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 shutdown_connections 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa3ec038770 0x7fa3ec03ac20 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4001302d0 0x7fa40013f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa40009d5b0 0x7fa40013ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 --2- 172.21.15.181:0/733539005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa40009b760 0x7fa40012cc30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 >> 172.21.15.181:0/733539005 conn(0x7fa4000a20c0 msgr2=0x7fa40009a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 shutdown_connections 2024-09-15T03:49:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-15T03:49:15.218+0000 7fa406c34640 1 -- 172.21.15.181:0/733539005 wait complete. 2024-09-15T03:49:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-15T03:49:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-15T03:49:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:+ expected_ret=1 2024-09-15T03:49:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-15T03:49:15.229 INFO:tasks.workunit.client.0.smithi181.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-15T03:49:15.229 INFO:tasks.workunit.client.0.smithi181.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797730981}) 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134422683}) 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797730981}) 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426470680}) 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.365+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.365+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474365692}) 2024-09-15T03:49:15.376 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.365+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=543587014}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.365+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.365+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474365692}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.365+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1998494748}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.366+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.366+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873563452}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.366+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50802830}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.366+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.366+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873563452}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.366+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179526497}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670533440}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551645394}) 2024-09-15T03:49:15.377 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670533440}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1135712959}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405015863}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3206098977}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405015863}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.367+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165842992}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.368+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.368+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793270547}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.368+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089520443}) 2024-09-15T03:49:15.378 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.368+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.368+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793270547}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.368+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=566397746}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46178811}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258146347}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46178811}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416562138}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.369+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828973408}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3683583155}) 2024-09-15T03:49:15.379 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828973408}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532147788}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870340580}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2302369315}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870340580}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.370+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799524819}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.371+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.371+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115570427}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.371+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960313377}) 2024-09-15T03:49:15.380 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.371+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.371+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115570427}) 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.371+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208077450}) 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062584212}) 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866491604}) 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.381 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062584212}) 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578875828}) 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578059960}) 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=574725607}) 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578059960}) 2024-09-15T03:49:15.382 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.372+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095334907}) 2024-09-15T03:49:15.383 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.373+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.383 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.373+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189523216}) 2024-09-15T03:49:15.383 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.373+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59445350}) 2024-09-15T03:49:15.383 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.373+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.383 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.373+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189523216}) 2024-09-15T03:49:15.383 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.373+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2208356620}) 2024-09-15T03:49:15.528 INFO:tasks.workunit.client.0.smithi181.stderr:+ ret=1 2024-09-15T03:49:15.528 INFO:tasks.workunit.client.0.smithi181.stderr:+ [[ 1 -ne 1 ]] 2024-09-15T03:49:15.528 INFO:tasks.workunit.client.0.smithi181.stderr:+ return 0 2024-09-15T03:49:15.528 INFO:tasks.workunit.client.0.smithi181.stdout:OK 2024-09-15T03:49:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:+ echo OK 2024-09-15T03:49:15.529 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:49:15.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:49:15.545 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:49:15.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:49:15.553 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-15T03:49:15.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T03:49:15.630 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.628+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.630 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.628+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129500641}) 2024-09-15T03:49:15.630 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.628+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3395386502}) 2024-09-15T03:49:15.630 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.628+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.630 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.628+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129500641}) 2024-09-15T03:49:15.630 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.628+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337135228}) 2024-09-15T03:49:15.631 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.629+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.631 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.629+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3971465922}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.629+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4119706290}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.629+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.629+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3971465922}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.629+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462693438}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192846631}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351406046}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192846631}) 2024-09-15T03:49:15.632 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2159671586}) 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505362051}) 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1038143579}) 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505362051}) 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.630+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=506014250}) 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.634 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168583791}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550093314}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168583791}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711405943}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144886821}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2434921313}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144886821}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.631+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151728975}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.632+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.632+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469052212}) 2024-09-15T03:49:15.635 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.632+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487835727}) 2024-09-15T03:49:15.636 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.632+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.636 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.632+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469052212}) 2024-09-15T03:49:15.636 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.632+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383941808}) 2024-09-15T03:49:15.644 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.644 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769442496}) 2024-09-15T03:49:15.644 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2600584484}) 2024-09-15T03:49:15.645 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.645 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769442496}) 2024-09-15T03:49:15.645 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746569381}) 2024-09-15T03:49:15.645 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.646 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883875374}) 2024-09-15T03:49:15.646 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1667300564}) 2024-09-15T03:49:15.646 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.647 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883875374}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.633+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726021700}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.634+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.634+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763504041}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.634+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175668624}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.634+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.634+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763504041}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.634+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1973858207}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477164013}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950804880}) 2024-09-15T03:49:15.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477164013}) 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259249114}) 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407368788}) 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1557409137}) 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407368788}) 2024-09-15T03:49:15.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.635+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216534126}) 2024-09-15T03:49:15.650 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.650 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076332388}) 2024-09-15T03:49:15.650 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278234684}) 2024-09-15T03:49:15.650 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.650 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076332388}) 2024-09-15T03:49:15.650 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855128577}) 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247298611}) 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=885172665}) 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247298611}) 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.636+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2224694981}) 2024-09-15T03:49:15.651 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.637+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.637+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973062571}) 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.637+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3263901185}) 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.637+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.637+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973062571}) 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.637+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3168808092}) 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3111003637}) 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494601906}) 2024-09-15T03:49:15.652 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3111003637}) 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862275603}) 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071898160}) 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970950637}) 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071898160}) 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.638+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1225736820}) 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815461121}) 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3973103825}) 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815461121}) 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649252188}) 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207901759}) 2024-09-15T03:49:15.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385453344}) 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207901759}) 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.639+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380358219}) 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.640+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.640+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022027946}) 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.640+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3636950343}) 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.640+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.655 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.640+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022027946}) 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.640+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780987818}) 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329302992}) 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1790925884}) 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329302992}) 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844336439}) 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.656 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822133363}) 2024-09-15T03:49:15.657 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312636456}) 2024-09-15T03:49:15.657 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.657 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822133363}) 2024-09-15T03:49:15.657 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.641+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520405567}) 2024-09-15T03:49:15.658 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.642+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.658 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.642+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125409938}) 2024-09-15T03:49:15.658 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.642+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=200641223}) 2024-09-15T03:49:15.658 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.642+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.658 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.642+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125409938}) 2024-09-15T03:49:15.658 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.642+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415935931}) 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.644+0000 7fd5ec83f640 10 monclient: get_monmap_and_config 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.644+0000 7fd5ec83f640 10 monclient: build_initial_monmap 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.644+0000 7fd5ec83f640 10 monclient: monmap: 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:15.645769+0000 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:15.645769+0000 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:15.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.644+0000 7fd5ec83f640 1 Processor -- start 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.644+0000 7fd5ec83f640 1 -- start start 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.644+0000 7fd5ec83f640 10 monclient: init 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5e8131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:15.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd5e809e910 0x7fd5e809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e809e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809f250 0x7fd5e8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e809f250 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e809faf0 con 0x7fd5e8131ab0 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e8132590 con 0x7fd5e809e910 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e809fd30 con 0x7fd5e809f250 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5ec83f640 10 monclient(hunting): authenticate will time out at 1953.285369s 2024-09-15T03:49:15.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd5e809e910 0x7fd5e809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809f250 0x7fd5e8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd5e809e910 0x7fd5e809ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49080/0 (socket says 172.21.15.153:49080) 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 -- 172.21.15.153:0/585164518 learned_addr learned my addr 172.21.15.153:0/585164518 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): get_auth_request con 0x7fd5e809f250 auth_method 0 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): get_auth_request con 0x7fd5e809e910 auth_method 0 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:15.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient(hunting): handle_auth_done global_id 254151 payload 274 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 -- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809f250 msgr2=0x7fd5e8138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 --2- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809f250 0x7fd5e8138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 -- 172.21.15.153:0/585164518 >> v1:172.21.15.181:6789/0 conn(0x7fd5e8131ab0 legacy=0x7fd5e8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 monclient: found mon.noname-a 2024-09-15T03:49:15.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5dffff640 1 -- 172.21.15.153:0/585164518 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e80a1110 con 0x7fd5e809e910 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.645+0000 7fd5e6d9d640 1 --2- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809f250 0x7fd5e8138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.648+0000 7fd5dffff640 1 -- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e809e910 msgr2=0x7fd5e809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.648+0000 7fd5dffff640 1 --2- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e809e910 0x7fd5e809ed10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009910 tx=0x7fd5d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6daf79967ff2e1c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.648+0000 7fd5e5d9b640 1 -- 172.21.15.153:0/585164518 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0032440 con 0x7fd5e809e910 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.648+0000 7fd5e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.648+0000 7fd5e5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.648+0000 7fd5e5d9b640 10 monclient: dump: 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:15.672 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 10 monclient: _finish_auth 0 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 1 -- 172.21.15.153:0/585164518 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d00325a0 con 0x7fd5e809e910 2024-09-15T03:49:15.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 5 monclient: authenticate success, global_id 254151 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 1 -- 172.21.15.153:0/585164518 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0032940 con 0x7fd5e809e910 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 4 set_mon_vals no callback set 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5e5d9b640 10 monclient: dump: 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 10 monclient: shutdown 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 -- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e809e910 msgr2=0x7fd5e809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e809e910 0x7fd5e809ed10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009910 tx=0x7fd5d00091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 -- 172.21.15.153:0/585164518 shutdown_connections 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809f250 0x7fd5e8138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/585164518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e809e910 0x7fd5e809ed10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 -- 172.21.15.153:0/585164518 >> 172.21.15.153:0/585164518 conn(0x7fd5e8067d30 msgr2=0x7fd5e80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 -- 172.21.15.153:0/585164518 shutdown_connections 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.649+0000 7fd5ec83f640 1 -- 172.21.15.153:0/585164518 wait complete. 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient: build_initial_monmap 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient: monmap: 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 Processor -- start 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 -- start start 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient: init 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 0x7fd5e8134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e809f250 0x7fd5e81345d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e809f250 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e8131ab0 0x7fd5e8134b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e8131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e8137850 con 0x7fd5e8131ab0 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e81315d0 con 0x7fd5e809f250 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e8138730 con 0x7fd5e809e910 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.650+0000 7fd5ec83f640 10 monclient(hunting): authenticate will time out at 1953.290960s 2024-09-15T03:49:15.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 0x7fd5e8134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 0x7fd5e8134090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41002/0 (socket says 172.21.15.153:41002) 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 -- 172.21.15.153:0/2196118544 learned_addr learned my addr 172.21.15.153:0/2196118544 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): get_auth_request con 0x7fd5e809e910 auth_method 0 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e8131ab0 0x7fd5e8134b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): get_auth_request con 0x7fd5e8131ab0 auth_method 0 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e809f250 0x7fd5e81345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:15.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): get_auth_request con 0x7fd5e809f250 auth_method 0 2024-09-15T03:49:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient(hunting): handle_auth_done global_id 244268 payload 824 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 -- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e809f250 msgr2=0x7fd5e81345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e809f250 0x7fd5e81345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 -- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e8131ab0 msgr2=0x7fd5e8134b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e8131ab0 0x7fd5e8134b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient: _finish_hunting 0 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 monclient: found mon.noname-c 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e659c640 1 -- 172.21.15.153:0/2196118544 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e0009a20 con 0x7fd5e809e910 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.651+0000 7fd5e6d9d640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e8131ab0 0x7fd5e8134b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5e659c640 1 -- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 msgr2=0x7fd5e8134090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:15.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5e659c640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 0x7fd5e8134090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800c5e0 tx=0x7fd5d8007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80117a0 con 0x7fd5e809e910 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: dump: 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: _finish_auth 0 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d8011900 con 0x7fd5e809e910 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 5 monclient: authenticate success, global_id 244268 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8011cc0 con 0x7fd5e809e910 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 10 monclient: _renew_subs 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d0009450 con 0x7fd5e809e910 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5de7fc640 10 monclient: dump: 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5e559a640 4 set_mon_vals no callback set 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 10 monclient: _renew_subs 2024-09-15T03:49:15.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.653+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e8135ac0 con 0x7fd5e809e910 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd5d8012630 con 0x7fd5e809e910 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5ec83f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5ec83f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5ec83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5de7fc640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5c8038740 0x7fd5c803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e804bf60 con 0x7fd5e809e910 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd5d804c310 con 0x7fd5e809e910 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5dffff640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5c8038740 0x7fd5c803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:15.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.654+0000 7fd5dffff640 10 monclient: get_auth_request con 0x7fd5c8038740 auth_method 0 2024-09-15T03:49:15.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.657+0000 7fd5dffff640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5c8038740 0x7fd5c803abf0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0031000 tx=0x7fd5d0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:15.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.659+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd5d80108e0 con 0x7fd5e809e910 2024-09-15T03:49:15.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.659+0000 7fd5de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:15.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.659+0000 7fd5de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:15.853 DEBUG:teuthology.parallel:result is None 2024-09-15T03:49:15.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.871+0000 7fd5ec83f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.871+0000 7fd5ec83f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.871+0000 7fd5ec83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.871+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd5e809a140 con 0x7fd5e809e910 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.872+0000 7fd5de7fc640 1 -- 172.21.15.153:0/2196118544 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fd5d80102e0 con 0x7fd5e809e910 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.872+0000 7fd5de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:15.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.872+0000 7fd5de7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:15.875 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":212,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.873+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5c8038740 msgr2=0x7fd5c803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.873+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5c8038740 0x7fd5c803abf0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0031000 tx=0x7fd5d0032000 comp rx=0 tx=0).stop 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.873+0000 7fd5ec83f640 10 monclient: shutdown 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.873+0000 7fd5ec83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.873+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 msgr2=0x7fd5e8134090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.873+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 0x7fd5e8134090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800c5e0 tx=0x7fd5d8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 shutdown_connections 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd5c8038740 0x7fd5c803abf0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e8131ab0 0x7fd5e8134b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e809f250 0x7fd5e81345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 --2- 172.21.15.153:0/2196118544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e809e910 0x7fd5e8134090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:15.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 >> 172.21.15.153:0/2196118544 conn(0x7fd5e8067d30 msgr2=0x7fd5e80a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:15.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 shutdown_connections 2024-09-15T03:49:15.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.874+0000 7fd5ec83f640 1 -- 172.21.15.153:0/2196118544 wait complete. 2024-09-15T03:49:15.878 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:49:15.878 DEBUG:teuthology.orchestra.run.smithi181:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:49:15.886 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:49:15.886 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:49:15.886 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:49:15.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:49:15.956 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.954+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.957 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.954+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=682298872}) 2024-09-15T03:49:15.957 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.954+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865242028}) 2024-09-15T03:49:15.957 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.954+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.957 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.954+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=682298872}) 2024-09-15T03:49:15.957 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.954+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785789560}) 2024-09-15T03:49:15.958 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.956+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.958 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.956+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080417810}) 2024-09-15T03:49:15.958 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.956+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=77863990}) 2024-09-15T03:49:15.958 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.956+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.958 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.956+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080417810}) 2024-09-15T03:49:15.958 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.956+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409609423}) 2024-09-15T03:49:15.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.957+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.957+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76291642}) 2024-09-15T03:49:15.959 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.957+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866771412}) 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.957+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.957+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76291642}) 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.957+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738801444}) 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.958+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.958+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155232139}) 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.958+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1532043887}) 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.958+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.958+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155232139}) 2024-09-15T03:49:15.960 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.958+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3799644383}) 2024-09-15T03:49:15.961 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.961 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497543056}) 2024-09-15T03:49:15.961 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141586890}) 2024-09-15T03:49:15.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497543056}) 2024-09-15T03:49:15.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961898135}) 2024-09-15T03:49:15.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.959+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191367425}) 2024-09-15T03:49:15.962 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539415679}) 2024-09-15T03:49:15.963 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:15.963 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191367425}) 2024-09-15T03:49:15.963 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797854904}) 2024-09-15T03:49:15.978 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:49:15.979 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-15T03:49:15.993 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485377123}) 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057895737}) 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485377123}) 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.960+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=225064377}) 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.961+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.001 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.961+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232459715}) 2024-09-15T03:49:16.002 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.961+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991895674}) 2024-09-15T03:49:16.002 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.961+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.002 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.961+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232459715}) 2024-09-15T03:49:16.002 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.961+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3772560963}) 2024-09-15T03:49:16.002 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.962+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.002 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.962+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450464791}) 2024-09-15T03:49:16.003 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.962+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394704300}) 2024-09-15T03:49:16.003 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.962+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.003 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.962+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450464791}) 2024-09-15T03:49:16.003 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.962+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329577589}) 2024-09-15T03:49:16.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.963+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.963+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274869866}) 2024-09-15T03:49:16.004 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.963+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1165250386}) 2024-09-15T03:49:16.005 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.963+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.005 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.963+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274869866}) 2024-09-15T03:49:16.005 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.963+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812678801}) 2024-09-15T03:49:16.005 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.964+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.964+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692635724}) 2024-09-15T03:49:16.006 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.964+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2378342737}) 2024-09-15T03:49:16.007 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.964+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.007 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.964+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692635724}) 2024-09-15T03:49:16.007 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.964+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1161297683}) 2024-09-15T03:49:16.007 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.965+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.008 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.965+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248779172}) 2024-09-15T03:49:16.008 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.965+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082765499}) 2024-09-15T03:49:16.008 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.965+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.008 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.965+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248779172}) 2024-09-15T03:49:16.008 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.965+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899896322}) 2024-09-15T03:49:16.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.966+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.966+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262385321}) 2024-09-15T03:49:16.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.966+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465735504}) 2024-09-15T03:49:16.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.966+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.009 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.966+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262385321}) 2024-09-15T03:49:16.010 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.966+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2953593891}) 2024-09-15T03:49:16.010 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.967+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.010 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.967+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345527705}) 2024-09-15T03:49:16.012 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.967+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2083927654}) 2024-09-15T03:49:16.012 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.967+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.012 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.967+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345527705}) 2024-09-15T03:49:16.012 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.967+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375488895}) 2024-09-15T03:49:16.012 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.968+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.012 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.968+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991236755}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.968+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2432075556}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.968+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.968+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991236755}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.968+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120377789}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.969+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.969+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033210089}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.969+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3064216974}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.969+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.969+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033210089}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.969+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45642964}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.970+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.970+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941081101}) 2024-09-15T03:49:16.013 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.970+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1212007695}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.970+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.970+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941081101}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.970+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702614915}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.971+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.971+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2866661492}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.971+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654952920}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.971+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.971+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2866661492}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.971+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363225288}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.972+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.972+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886280559}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.972+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2732661338}) 2024-09-15T03:49:16.014 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.972+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.972+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886280559}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.972+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184164412}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.973+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.973+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374296423}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.973+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254202449}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.973+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.973+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374296423}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.973+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3466883598}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.974+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.974+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455547385}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.974+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573111196}) 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.974+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.015 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.974+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455547385}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.974+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939038858}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.975+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.975+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2253940010}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.975+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3562005325}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.975+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.975+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2253940010}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.975+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866949410}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.976+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.976+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044904049}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.976+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2841279863}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.976+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.976+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044904049}) 2024-09-15T03:49:16.016 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.976+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954185432}) 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.977+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.977+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094439806}) 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.977+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221562064}) 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.977+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.977+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094439806}) 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.977+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720821565}) 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.978+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.978+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026184887}) 2024-09-15T03:49:16.017 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.978+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878817588}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.978+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.978+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026184887}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.978+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1877658544}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.979+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.979+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679036105}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.979+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801525837}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.979+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.979+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679036105}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.979+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3429694319}) 2024-09-15T03:49:16.018 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.980+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.980+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346469012}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.980+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485924053}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.980+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.980+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346469012}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.980+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769051841}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.981+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.981+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341880784}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.981+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425453686}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.981+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.981+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341880784}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.981+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2488608243}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.982+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.982+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1432226677}) 2024-09-15T03:49:16.019 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.982+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930460852}) 2024-09-15T03:49:16.020 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.982+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.020 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.982+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1432226677}) 2024-09-15T03:49:16.020 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:15.982+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132762667}) 2024-09-15T03:49:16.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.982+0000 7fbd1f468640 10 monclient: get_monmap_and_config 2024-09-15T03:49:16.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.982+0000 7fbd1f468640 10 monclient: build_initial_monmap 2024-09-15T03:49:16.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.982+0000 7fbd1f468640 10 monclient: monmap: 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:15.984395+0000 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:15.984395+0000 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 1 Processor -- start 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 1 -- start start 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 10 monclient: init 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbd181318f0 0x7fbd18131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 10 monclient(hunting): picked mon.noname-c con 0x7fbd181318f0 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbd1809ea50 0x7fbd1809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1809ea50 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd18068d90 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.983+0000 7fbd1f468640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1809fbf0 con 0x7fbd18068d90 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1f468640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd18131d50 con 0x7fbd1809ea50 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1f468640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd1809fe30 con 0x7fbd181318f0 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1f468640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1f468640 10 monclient(hunting): authenticate will time out at 1953.624031s 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1da05640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbd18068d90 0x7fbd180691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46986/0 (socket says 172.21.15.153:46986) 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1da05640 1 -- 172.21.15.153:0/3722515448 learned_addr learned my addr 172.21.15.153:0/3722515448 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 1 --2- 172.21.15.153:0/3722515448 >> v2:172.21.15.181:3300/0 conn(0x7fbd181318f0 0x7fbd18131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3722515448 >> v2:172.21.15.153:3300/0 conn(0x7fbd1809ea50 0x7fbd1809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): get_auth_request con 0x7fbd181318f0 auth_method 0 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): get_auth_request con 0x7fbd1809ea50 auth_method 0 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1d204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_done global_id 254157 payload 274 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3722515448 >> v2:172.21.15.181:3300/0 conn(0x7fbd181318f0 msgr2=0x7fbd18131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:16.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3722515448 >> v2:172.21.15.181:3300/0 conn(0x7fbd181318f0 0x7fbd18131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3722515448 >> v1:172.21.15.153:6789/0 conn(0x7fbd18068d90 legacy=0x7fbd180691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient: _finish_hunting 0 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 1 monclient: found mon.noname-a 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.984+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3722515448 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180a1210 con 0x7fbd1809ea50 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3722515448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1809ea50 msgr2=0x7fbd1809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3722515448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1809ea50 0x7fbd1809ee20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd18076600 tx=0x7fbd080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7158e55742edcf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 1 -- 172.21.15.153:0/3722515448 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0802ae00 con 0x7fbd1809ea50 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: dump: 2024-09-15T03:49:16.035 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: _finish_auth 0 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd1f468640 5 monclient: authenticate success, global_id 254157 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd1f468640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 1 -- 172.21.15.153:0/3722515448 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd08032440 con 0x7fbd1809ea50 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd1f468640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd1f468640 4 set_mon_vals no callback set 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 1 -- 172.21.15.153:0/3722515448 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd080327e0 con 0x7fbd1809ea50 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.987+0000 7fbd0ffff640 10 monclient: dump: 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.037 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 10 monclient: shutdown 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 -- 172.21.15.153:0/3722515448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1809ea50 msgr2=0x7fbd1809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3722515448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1809ea50 0x7fbd1809ee20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd18076600 tx=0x7fbd080090f0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 -- 172.21.15.153:0/3722515448 shutdown_connections 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3722515448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1809ea50 0x7fbd1809ee20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3722515448 >> v2:172.21.15.181:3300/0 conn(0x7fbd181318f0 0x7fbd18131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 -- 172.21.15.153:0/3722515448 >> 172.21.15.153:0/3722515448 conn(0x7fbd18067110 msgr2=0x7fbd18067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 -- 172.21.15.153:0/3722515448 shutdown_connections 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.988+0000 7fbd1f468640 1 -- 172.21.15.153:0/3722515448 wait complete. 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.989+0000 7fbd1f468640 10 monclient: build_initial_monmap 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.989+0000 7fbd1f468640 10 monclient: monmap: 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:16.039 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.989+0000 7fbd1f468640 1 Processor -- start 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.989+0000 7fbd1f468640 1 -- start start 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.989+0000 7fbd1f468640 10 monclient: init 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 0x7fbd1809ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): picked mon.noname-a con 0x7fbd18068d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd1809ea50 0x7fbd18130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd181318f0 0x7fbd1811b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): picked mon.noname-c con 0x7fbd181318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180a4410 con 0x7fbd18068d90 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1811e310 con 0x7fbd1809ea50 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1811e080 con 0x7fbd181318f0 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1f468640 10 monclient(hunting): authenticate will time out at 1953.630107s 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 0x7fbd1809ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 0x7fbd1809ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49118/0 (socket says 172.21.15.153:49118) 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 -- 172.21.15.153:0/3705940043 learned_addr learned my addr 172.21.15.153:0/3705940043 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): get_auth_request con 0x7fbd18068d90 auth_method 0 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd181318f0 0x7fbd1811b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): get_auth_request con 0x7fbd181318f0 auth_method 0 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd1809ea50 0x7fbd18130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1da05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1ca03640 10 monclient(hunting): get_auth_request con 0x7fbd1809ea50 auth_method 0 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1ca03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1ca03640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1ca03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient(hunting): handle_auth_done global_id 254160 payload 824 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd181318f0 msgr2=0x7fbd1811b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd181318f0 0x7fbd1811b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd1809ea50 msgr2=0x7fbd18130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd1809ea50 0x7fbd18130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient: _finish_hunting 0 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 monclient: found mon.noname-a 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1d204640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08009390 con 0x7fbd18068d90 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.990+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd1809ea50 0x7fbd18130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1d204640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 msgr2=0x7fbd1809ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1d204640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 0x7fbd1809ab70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400ea50 tx=0x7fbd140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14009600 con 0x7fbd18068d90 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:16.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: dump: 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:16.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 5 monclient: authenticate success, global_id 254160 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1400cbc0 con 0x7fbd18068d90 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 10 monclient: _renew_subs 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1811dd20 con 0x7fbd18068d90 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14013470 con 0x7fbd18068d90 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0d7fa640 10 monclient: dump: 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.046 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 10 monclient: _renew_subs 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd0f7fe640 4 set_mon_vals no callback set 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.992+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1811ce30 con 0x7fbd18068d90 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1f468640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 0x7fbd1811bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18065e00 con 0x7fbd18131140 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 0x7fbd1811bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient: get_auth_request con 0x7fbd18131140 auth_method 0 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.993+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbd14013860 con 0x7fbd18068d90 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_done global_id 254163 payload 824 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd1804bf60 con 0x7fbd18131140 2024-09-15T03:49:16.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd0d7fa640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcfc038740 0x7fbcfc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbd1404c940 con 0x7fbd18068d90 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd1da05640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcfc038740 0x7fbcfc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.994+0000 7fbd1da05640 10 monclient: get_auth_request con 0x7fbcfc038740 auth_method 0 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 msgr2=0x7fbd1811bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 0x7fbd1811bc60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd08034000 tx=0x7fbd08032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08031ce0 con 0x7fbd18131140 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 10 monclient: dump: 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.049 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbd08031e40 con 0x7fbd18131140 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 msgr2=0x7fbd1811bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 0x7fbd1811bc60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd08034000 tx=0x7fbd08032000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 msgr2=0x7fbd1811bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd0d7fa640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18131140 0x7fbd1811bc60 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd08034000 tx=0x7fbd08032000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:15.996+0000 7fbd1da05640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcfc038740 0x7fbcfc03abf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbd10006d70 tx=0x7fbd100052b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1f468640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 0x7fbd18096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1f468640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18131510 con 0x7fbd1811c1a0 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 0x7fbd18096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient: get_auth_request con 0x7fbd1811c1a0 auth_method 0 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 10 monclient(hunting): handle_auth_done global_id 254166 payload 824 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.002+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3705940043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbd1811c5a0 con 0x7fbd1811c1a0 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1ca03640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 msgr2=0x7fbd18096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1ca03640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 0x7fbd18096a60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd1811d0a0 tx=0x7fbd0803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08047a40 con 0x7fbd1811c1a0 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 10 monclient: dump: 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.052 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fbd08051460 con 0x7fbd1811c1a0 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 msgr2=0x7fbd18096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 0x7fbd18096a60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd1811d0a0 tx=0x7fbd0803c000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 msgr2=0x7fbd18096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd0d7fa640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1811c1a0 0x7fbd18096a60 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd1811d0a0 tx=0x7fbd0803c000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 212, 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.055 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.056 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:16.057 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.058 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:16.059 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.060 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:16.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:16.064 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.066 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:16.066 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcfc038740 msgr2=0x7fbcfc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbcfc038740 0x7fbcfc03abf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbd10006d70 tx=0x7fbd100052b0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 10 monclient: shutdown 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 msgr2=0x7fbd1809ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 --2- 172.21.15.153:0/3705940043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18068d90 0x7fbd1809ab70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400ea50 tx=0x7fbd140090f0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1da05640 1 -- 172.21.15.153:0/3705940043 reap_dead start 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1da05640 1 -- 172.21.15.153:0/3705940043 reap_dead start 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 shutdown_connections 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 >> 172.21.15.153:0/3705940043 conn(0x7fbd18067110 msgr2=0x7fbd180a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 shutdown_connections 2024-09-15T03:49:16.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.005+0000 7fbd1f468640 1 -- 172.21.15.153:0/3705940043 wait complete. 2024-09-15T03:49:16.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193073561}) 2024-09-15T03:49:16.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280581819}) 2024-09-15T03:49:16.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193073561}) 2024-09-15T03:49:16.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.014+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011303336}) 2024-09-15T03:49:16.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:49:16.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:49:16.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:49:16.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.099+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.099+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942990617}) 2024-09-15T03:49:16.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.099+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1697337279}) 2024-09-15T03:49:16.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.099+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.099+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942990617}) 2024-09-15T03:49:16.101 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.099+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337031941}) 2024-09-15T03:49:16.107 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.105+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.107 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.105+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456086622}) 2024-09-15T03:49:16.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.105+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2404283003}) 2024-09-15T03:49:16.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.105+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.105+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456086622}) 2024-09-15T03:49:16.108 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.105+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111511845}) 2024-09-15T03:49:16.117 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.115+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.115+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912372325}) 2024-09-15T03:49:16.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.115+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583102316}) 2024-09-15T03:49:16.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.115+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.115+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912372325}) 2024-09-15T03:49:16.118 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.115+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399836812}) 2024-09-15T03:49:16.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.119+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.119+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59899300}) 2024-09-15T03:49:16.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.119+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73046587}) 2024-09-15T03:49:16.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.119+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.119+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59899300}) 2024-09-15T03:49:16.121 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.119+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950823646}) 2024-09-15T03:49:16.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.121+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.121+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194375617}) 2024-09-15T03:49:16.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.121+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766536613}) 2024-09-15T03:49:16.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.121+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.121+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194375617}) 2024-09-15T03:49:16.123 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.121+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188191791}) 2024-09-15T03:49:16.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.130+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.130+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75619097}) 2024-09-15T03:49:16.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.130+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906438815}) 2024-09-15T03:49:16.132 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.130+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.130+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75619097}) 2024-09-15T03:49:16.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.130+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384373754}) 2024-09-15T03:49:16.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.131+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.131+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892973240}) 2024-09-15T03:49:16.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.131+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2185469370}) 2024-09-15T03:49:16.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.131+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.131+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892973240}) 2024-09-15T03:49:16.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.131+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333492658}) 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.132+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.132+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642401150}) 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.132+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786032478}) 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.132+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.132+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642401150}) 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.132+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430392574}) 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.133+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.136 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.133+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72550461}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.133+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3910073113}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.133+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.133+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72550461}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.133+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2687396632}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.134+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.134+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234568560}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.134+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=425273708}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.134+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.134+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234568560}) 2024-09-15T03:49:16.137 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.134+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006711188}) 2024-09-15T03:49:16.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.135+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.135+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998315735}) 2024-09-15T03:49:16.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.135+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2080031774}) 2024-09-15T03:49:16.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.135+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.135+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998315735}) 2024-09-15T03:49:16.138 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.135+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515505963}) 2024-09-15T03:49:16.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.172+0000 7f2d35f5d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:16.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.172+0000 7f2d35f5d640 10 monclient: build_initial_monmap 2024-09-15T03:49:16.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.172+0000 7f2d35f5d640 10 monclient: monmap: 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:16.174705+0000 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:16.174705+0000 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:16.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.173+0000 7f2d35f5d640 1 Processor -- start 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.173+0000 7f2d35f5d640 1 -- start start 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.173+0000 7f2d35f5d640 10 monclient: init 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d30131ab0 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 0x7f2d3009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d3009e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f2d3009f220 0x7f2d30138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d3009f220 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3009faf0 con 0x7f2d30131ab0 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d30132590 con 0x7f2d3009f220 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3009fd30 con 0x7f2d3009e910 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d35f5d640 10 monclient(hunting): authenticate will time out at 1953.814418s 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 0x7f2d3009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 0x7f2d3009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41038/0 (socket says 172.21.15.153:41038) 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 -- 172.21.15.153:0/3271940248 learned_addr learned my addr 172.21.15.153:0/3271940248 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): get_auth_request con 0x7f2d3009e910 auth_method 0 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d2ffff640 1 --2- 172.21.15.153:0/3271940248 >> v2:172.21.15.181:3300/0 conn(0x7f2d3009f220 0x7f2d30138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d2ffff640 10 monclient(hunting): get_auth_request con 0x7f2d3009f220 auth_method 0 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient(hunting): handle_auth_done global_id 244274 payload 274 2024-09-15T03:49:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 -- 172.21.15.153:0/3271940248 >> v2:172.21.15.181:3300/0 conn(0x7f2d3009f220 msgr2=0x7f2d30138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 --2- 172.21.15.153:0/3271940248 >> v2:172.21.15.181:3300/0 conn(0x7f2d3009f220 0x7f2d30138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 -- 172.21.15.153:0/3271940248 >> v1:172.21.15.181:6789/0 conn(0x7f2d30131ab0 legacy=0x7f2d30131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient: _finish_hunting 0 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 monclient: found mon.noname-b 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:49:16.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.174+0000 7f2d27fff640 1 -- 172.21.15.153:0/3271940248 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d300a1110 con 0x7f2d3009e910 2024-09-15T03:49:16.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d27fff640 1 -- 172.21.15.153:0/3271940248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 msgr2=0x7f2d3009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d27fff640 1 --2- 172.21.15.153:0/3271940248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 0x7f2d3009ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009910 tx=0x7f2d180091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f70eb427f08c347 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 1 -- 172.21.15.153:0/3271940248 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18032440 con 0x7f2d3009e910 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: dump: 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.189 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: _finish_auth 0 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 1 -- 172.21.15.153:0/3271940248 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d180325a0 con 0x7f2d3009e910 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d35f5d640 5 monclient: authenticate success, global_id 244274 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d35f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d35f5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 1 -- 172.21.15.153:0/3271940248 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18032940 con 0x7f2d3009e910 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d35f5d640 4 set_mon_vals no callback set 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.177+0000 7f2d2effd640 10 monclient: dump: 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 10 monclient: shutdown 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:16.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 -- 172.21.15.153:0/3271940248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 msgr2=0x7f2d3009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/3271940248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 0x7f2d3009ece0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009910 tx=0x7f2d180091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 -- 172.21.15.153:0/3271940248 shutdown_connections 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/3271940248 >> v2:172.21.15.181:3300/0 conn(0x7f2d3009f220 0x7f2d30138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/3271940248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d3009e910 0x7f2d3009ece0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 -- 172.21.15.153:0/3271940248 >> 172.21.15.153:0/3271940248 conn(0x7f2d30067d30 msgr2=0x7f2d300a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 -- 172.21.15.153:0/3271940248 shutdown_connections 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 1 -- 172.21.15.153:0/3271940248 wait complete. 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 10 monclient: build_initial_monmap 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.178+0000 7f2d35f5d640 10 monclient: monmap: 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 Processor -- start 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 -- start start 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient: init 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:49:16.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 0x7f2d301382c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d3009e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d3009f220 0x7f2d30133f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d3009f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30131ab0 0x7f2d301344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d30131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3011b6b0 con 0x7f2d3009e910 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3009bde0 con 0x7f2d3009f220 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30137bc0 con 0x7f2d30131ab0 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d35f5d640 10 monclient(hunting): authenticate will time out at 1953.819634s 2024-09-15T03:49:16.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30131ab0 0x7f2d301344a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 0x7f2d301382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30131ab0 0x7f2d301344a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41044/0 (socket says 172.21.15.153:41044) 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 0x7f2d301382c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49150/0 (socket says 172.21.15.153:49150) 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 1 -- 172.21.15.153:0/2155428627 learned_addr learned my addr 172.21.15.153:0/2155428627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d3009e910 auth_method 0 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): get_auth_request con 0x7f2d30131ab0 auth_method 0 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d3009f220 0x7f2d30133f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): get_auth_request con 0x7f2d3009f220 auth_method 0 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.179+0000 7f2d2f7fe640 10 monclient(hunting): handle_auth_done global_id 254169 payload 824 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30131ab0 msgr2=0x7f2d301344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30131ab0 0x7f2d301344a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d3009f220 msgr2=0x7f2d30133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d3009f220 0x7f2d30133f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 1 monclient: found mon.noname-a 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2f7fe640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18009450 con 0x7f2d3009e910 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.180+0000 7f2d2ffff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30131ab0 0x7f2d301344a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:16.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d2f7fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 msgr2=0x7f2d301382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d2f7fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 0x7f2d301382c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c4e0 tx=0x7f2d20007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200117a0 con 0x7f2d3009e910 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: dump: 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.198 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: _finish_auth 0 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 5 monclient: authenticate success, global_id 254169 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d20011920 con 0x7f2d3009e910 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 10 monclient: _renew_subs 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d30135fc0 con 0x7f2d3009e910 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20011ce0 con 0x7f2d3009e910 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d277fe640 10 monclient: dump: 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 10 monclient: _renew_subs 2024-09-15T03:49:16.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d2e7fc640 4 set_mon_vals no callback set 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.182+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d30131640 con 0x7f2d3009e910 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d35f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 0x7f2d30141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30063740 con 0x7f2d301349e0 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2d20012630 con 0x7f2d3009e910 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 0x7f2d30141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient: get_auth_request con 0x7f2d301349e0 auth_method 0 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d10038790 0x7f2d1003ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 10 monclient(hunting): handle_auth_done global_id 244280 payload 824 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d30137690 con 0x7f2d301349e0 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2d2004c1a0 con 0x7f2d3009e910 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d2ffff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d10038790 0x7f2d1003ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.183+0000 7f2d2ffff640 10 monclient: get_auth_request con 0x7f2d10038790 auth_method 0 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.185+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 msgr2=0x7f2d30141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.185+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 0x7f2d30141680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2d18034000 tx=0x7f2d18032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.185+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18000d10 con 0x7f2d301349e0 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.185+0000 7f2d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.185+0000 7f2d277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:16.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.185+0000 7f2d277fe640 10 monclient: dump: 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2d18031c50 con 0x7f2d301349e0 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 msgr2=0x7f2d30141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 0x7f2d30141680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2d18034000 tx=0x7f2d18032000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 msgr2=0x7f2d30141680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d2ffff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d10038790 0x7f2d1003ac40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2d28007a90 tx=0x7f2d28009900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.186+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d301349e0 0x7f2d30141680 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d35f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:16.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d35f5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30137970 con 0x7f2d30141bc0 2024-09-15T03:49:16.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient: get_auth_request con 0x7f2d30141bc0 auth_method 0 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 10 monclient(hunting): handle_auth_done global_id 244283 payload 824 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.192+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d30134db0 con 0x7f2d30141bc0 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 msgr2=0x7f2d30141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009910 tx=0x7f2d1803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 msgr2=0x7f2d30141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 msgr2=0x7f2d30141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:49:16.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009910 tx=0x7f2d1803d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009910 tx=0x7f2d1803d000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 msgr2=0x7f2d30141f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 0x7f2d10043580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 msgr2=0x7f2d30141f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d30141bc0 0x7f2d30141f90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10043b90 con 0x7f2d100411a0 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 0x7f2d10043580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient: get_auth_request con 0x7f2d100411a0 auth_method 0 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.194+0000 7f2d27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.195+0000 7f2d27fff640 10 monclient(hunting): handle_auth_done global_id 244286 payload 824 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.195+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d18031c50 con 0x7f2d100411a0 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d27fff640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 msgr2=0x7f2d10043580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d27fff640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 0x7f2d10043580 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d30137ab0 tx=0x7f2d1804d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1805c880 con 0x7f2d100411a0 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 10 monclient: dump: 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f2d18046600 con 0x7f2d100411a0 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 msgr2=0x7f2d10043580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 0x7f2d10043580 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d30137ab0 tx=0x7f2d1804d000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 msgr2=0x7f2d10043580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d277fe640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100411a0 0x7f2d10043580 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d2ffff640 1 -- 172.21.15.153:0/2155428627 reap_dead start 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 212, 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:16.212 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.213 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.214 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:16.215 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.216 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.218 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:16.219 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:16.221 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:16.222 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:16.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:16.224 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d10038790 msgr2=0x7f2d1003ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d10038790 0x7f2d1003ac40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2d28007a90 tx=0x7f2d28009900 comp rx=0 tx=0).stop 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.197+0000 7f2d35f5d640 10 monclient: shutdown 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 msgr2=0x7f2d301382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 0x7f2d301382c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c4e0 tx=0x7f2d20007e10 comp rx=0 tx=0).stop 2024-09-15T03:49:16.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 shutdown_connections 2024-09-15T03:49:16.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d10038790 0x7f2d1003ac40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 --2- 172.21.15.153:0/2155428627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d3009e910 0x7f2d301382c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 >> 172.21.15.153:0/2155428627 conn(0x7f2d30067d30 msgr2=0x7f2d3009ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:16.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 shutdown_connections 2024-09-15T03:49:16.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f2d35f5d640 1 -- 172.21.15.153:0/2155428627 wait complete. 2024-09-15T03:49:16.226 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711612779}) 2024-09-15T03:49:16.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614509135}) 2024-09-15T03:49:16.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:49:16.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711612779}) 2024-09-15T03:49:16.227 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:49:16.198+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1566896067}) 2024-09-15T03:49:16.228 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:49:16.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:49:16.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:49:16.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.360+0000 7f729148b640 10 monclient: get_monmap_and_config 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.360+0000 7f729148b640 10 monclient: build_initial_monmap 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.360+0000 7f729148b640 10 monclient: monmap: 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:16.361791+0000 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:16.361791+0000 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:16.362 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:16.363 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:16.363 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:16.363 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:16.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.360+0000 7f729148b640 1 Processor -- start 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.360+0000 7f729148b640 1 -- start start 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.360+0000 7f729148b640 10 monclient: init 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f728c1318f0 0x7f728c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): picked mon.noname-a con 0x7f728c1318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f728c09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): picked mon.noname-b con 0x7f728c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c131cc0 con 0x7f728c09ea50 2024-09-15T03:49:16.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728c09f530 con 0x7f728c1318f0 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c09fbf0 con 0x7f728c136c20 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f729148b640 10 monclient(hunting): authenticate will time out at 1954.001224s 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f728c1318f0 0x7f728c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f728c1318f0 0x7f728c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49162/0 (socket says 172.21.15.153:49162) 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 -- 172.21.15.153:0/1890651140 learned_addr learned my addr 172.21.15.153:0/1890651140 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): get_auth_request con 0x7f728c1318f0 auth_method 0 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41078/0 (socket says 172.21.15.153:41078) 2024-09-15T03:49:16.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): get_auth_request con 0x7f728c136c20 auth_method 0 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient(hunting): handle_auth_done global_id 254172 payload 274 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 -- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 msgr2=0x7f728c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 --2- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 -- 172.21.15.153:0/1890651140 >> v1:172.21.15.181:6789/0 conn(0x7f728c09ea50 legacy=0x7f728c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient: _finish_hunting 0 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 monclient: found mon.noname-a 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728affd640 1 -- 172.21.15.153:0/1890651140 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c09fe30 con 0x7f728c1318f0 2024-09-15T03:49:16.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.361+0000 7f728b7fe640 1 --2- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728affd640 1 -- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 msgr2=0x7f728c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728affd640 1 --2- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c131f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f727c009910 tx=0x7f727c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e029fccc488db2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 1 -- 172.21.15.153:0/1890651140 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c032440 con 0x7f728c1318f0 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: dump: 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 1 -- 172.21.15.153:0/1890651140 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c0325a0 con 0x7f728c1318f0 2024-09-15T03:49:16.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f729148b640 5 monclient: authenticate success, global_id 254172 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f729148b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f729148b640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 1 -- 172.21.15.153:0/1890651140 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c032940 con 0x7f728c1318f0 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f729148b640 4 set_mon_vals no callback set 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.364+0000 7f728a7fc640 10 monclient: dump: 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.381 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 10 monclient: shutdown 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 -- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 msgr2=0x7f728c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 --2- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c131f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f727c009910 tx=0x7f727c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 -- 172.21.15.153:0/1890651140 shutdown_connections 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 --2- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 --2- 172.21.15.153:0/1890651140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c131f30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 -- 172.21.15.153:0/1890651140 >> 172.21.15.153:0/1890651140 conn(0x7f728c067110 msgr2=0x7f728c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 -- 172.21.15.153:0/1890651140 shutdown_connections 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 1 -- 172.21.15.153:0/1890651140 wait complete. 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 10 monclient: build_initial_monmap 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.365+0000 7f729148b640 10 monclient: monmap: 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:16.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 Processor -- start 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 -- start start 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient: init 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c09ea50 0x7f728c130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): picked mon.noname-b con 0x7f728c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c0997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): picked mon.noname-a con 0x7f728c1318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c099cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): picked mon.noname-c con 0x7f728c136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:16.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c141f20 con 0x7f728c1318f0 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c09b2e0 con 0x7f728c09ea50 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c11cd60 con 0x7f728c136c20 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f729148b640 10 monclient(hunting): authenticate will time out at 1954.006627s 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c0997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c099cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c0997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49174/0 (socket says 172.21.15.153:49174) 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c099cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41092/0 (socket says 172.21.15.153:41092) 2024-09-15T03:49:16.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 1 -- 172.21.15.153:0/519806271 learned_addr learned my addr 172.21.15.153:0/519806271 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): get_auth_request con 0x7f728c1318f0 auth_method 0 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): get_auth_request con 0x7f728c136c20 auth_method 0 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c09ea50 0x7f728c130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): get_auth_request con 0x7f728c09ea50 auth_method 0 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.366+0000 7f72827fc640 10 monclient(hunting): handle_auth_done global_id 254175 payload 824 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 msgr2=0x7f728c099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c099cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c09ea50 msgr2=0x7f728c130970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c09ea50 0x7f728c130970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 10 monclient: _finish_hunting 0 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 1 monclient: found mon.noname-a 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f72827fc640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7284009a20 con 0x7f728c1318f0 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.367+0000 7f728b7fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f728c136c20 0x7f728c099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72827fc640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 msgr2=0x7f728c0997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72827fc640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c0997b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f727400c610 tx=0x7f7274007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72740117a0 con 0x7f728c1318f0 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: dump: 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: _finish_auth 0 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274011920 con 0x7f728c1318f0 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 5 monclient: authenticate success, global_id 254175 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 10 monclient: _renew_subs 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c009450 con 0x7f728c1318f0 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274011ce0 con 0x7f728c1318f0 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f72837fe640 10 monclient: dump: 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f7289ffb640 4 set_mon_vals no callback set 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 10 monclient: _renew_subs 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.369+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c11b720 con 0x7f728c1318f0 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f729148b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:16.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f729148b640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 0x7f728c09a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c11b9a0 con 0x7f728c1374b0 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7274012630 con 0x7f728c1318f0 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 0x7f728c09a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient: get_auth_request con 0x7f728c1374b0 auth_method 0 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f72837fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f726c038740 0x7f726c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f727404c2f0 con 0x7f728c1318f0 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728b7fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f726c038740 0x7f726c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 10 monclient(hunting): handle_auth_done global_id 204625 payload 824 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728affd640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f728c04bf60 con 0x7f728c1374b0 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.370+0000 7f728b7fe640 10 monclient: get_auth_request con 0x7f726c038740 auth_method 0 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f728affd640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 msgr2=0x7f728c09a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f728affd640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 0x7f728c09a230 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f727c034000 tx=0x7f727c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f728b7fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f726c038740 0x7f726c03abf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f728400eb00 tx=0x7f728400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c031a70 con 0x7f728c1374b0 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 10 monclient: dump: 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f727c031bd0 con 0x7f728c1374b0 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 msgr2=0x7f728c09a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 0x7f728c09a230 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f727c034000 tx=0x7f727c032000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 msgr2=0x7f728c09a230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.373+0000 7f72837fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c1374b0 0x7f728c09a230 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f727c034000 tx=0x7f727c032000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f729148b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:16.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f729148b640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 0x7f728c09a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:16.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f729148b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c11cc20 con 0x7f728c130eb0 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 0x7f728c09a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient: get_auth_request con 0x7f728c130eb0 auth_method 0 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 10 monclient(hunting): handle_auth_done global_id 204628 payload 824 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.379+0000 7f728affd640 1 -- 172.21.15.153:0/519806271 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f728c0638d0 con 0x7f728c130eb0 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.381+0000 7f728affd640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 msgr2=0x7f728c09a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.381+0000 7f728affd640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 0x7f728c09a770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f728c11d130 tx=0x7f727c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c03ce00 con 0x7f728c130eb0 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 10 monclient: dump: 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f727c046810 con 0x7f728c130eb0 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 msgr2=0x7f728c09a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 0x7f728c09a770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f728c11d130 tx=0x7f727c03d000 comp rx=0 tx=0).stop 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 msgr2=0x7f728c09a770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f72837fe640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f728c130eb0 0x7f728c09a770 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:16.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 212, 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-15T03:49:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:16.412 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f726c038740 msgr2=0x7f726c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f726c038740 0x7f726c03abf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f728400eb00 tx=0x7f728400b040 comp rx=0 tx=0).stop 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 10 monclient: shutdown 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 msgr2=0x7f728c0997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 --2- 172.21.15.153:0/519806271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f728c1318f0 0x7f728c0997b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f727400c610 tx=0x7f7274007ed0 comp rx=0 tx=0).stop 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f728b7fe640 1 -- 172.21.15.153:0/519806271 reap_dead start 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f728b7fe640 1 -- 172.21.15.153:0/519806271 reap_dead start 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 shutdown_connections 2024-09-15T03:49:16.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 >> 172.21.15.153:0/519806271 conn(0x7f728c067110 msgr2=0x7f728c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:16.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 shutdown_connections 2024-09-15T03:49:16.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:16.382+0000 7f729148b640 1 -- 172.21.15.153:0/519806271 wait complete. 2024-09-15T03:49:16.415 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:49:16.415 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:49:16.415 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-15T03:49:16.415 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:49:16.415 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:49:16.416 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:49:16.416 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:16.446 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:49:16.446 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T03:49:16.447 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T03:49:16.447 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:49:31.448 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T03:49:31.449 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T03:49:31.449 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:49:31.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:49:31.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.553+0000 7f20d8d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:31.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.553+0000 7f20d8d5d640 10 monclient: build_initial_monmap 2024-09-15T03:49:31.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.553+0000 7f20d8d5d640 10 monclient: monmap: 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:31.554648+0000 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:31.554648+0000 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:31.555 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:31.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.554+0000 7f20d8d5d640 1 Processor -- start 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.554+0000 7f20d8d5d640 1 -- start start 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.554+0000 7f20d8d5d640 10 monclient: init 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20d40981c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20d4131da0 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f20d4132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d409ca60 con 0x7f20d40981c0 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d4131bd0 con 0x7f20d4131da0 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d409cca0 con 0x7f20d4132170 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d8d5d640 10 monclient(hunting): authenticate will time out at 1969.194265s 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d259c640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f20d40981c0 legacy=0x7f20d4098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35118/0 (socket says 172.21.15.153:35118) 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 learned_addr learned my addr 172.21.15.153:0/1067408567 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request con 0x7f20d4131da0 auth_method 0 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:31.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request con 0x7f20d4132170 auth_method 0 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:31.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.555+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request con 0x7f20d4132170 auth_method 0 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:31.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v1:172.21.15.181:6789/0 conn(0x7f20d40981c0 legacy=0x7f20d4098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request con 0x7f20d4131da0 auth_method 0 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:31.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:49:31.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:31.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:31.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:49:31.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:31.756+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request con 0x7f20d4132170 auth_method 0 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request con 0x7f20d4131da0 auth_method 0 2024-09-15T03:49:32.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v1:172.21.15.181:6789/0 conn(0x7f20d40981c0 legacy=0x7f20d4098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:49:32.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.157+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:49:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v1:172.21.15.181:6789/0 conn(0x7f20d40981c0 legacy=0x7f20d4098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:49:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request con 0x7f20d4132170 auth_method 0 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request con 0x7f20d4131da0 auth_method 0 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:49:32.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:32.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:49:32.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:49:32.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:32.958+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:49:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient(hunting): tick 2024-09-15T03:49:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 monclient(hunting): continuing hunt 2024-09-15T03:49:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:49:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20d41390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20d41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 msgr2=0x7f20d4136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.153:3300/0 conn(0x7f20d4131da0 0x7f20d4136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 -- 172.21.15.153:0/1067408567 >> v1:172.21.15.181:6789/0 conn(0x7f20d40981c0 legacy=0x7f20d4098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient: _add_conns ranks=[4,0] 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 0x7f20b4005aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f20b4003670 addr v2:172.21.15.181:3300/0 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20b4006190 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 -- 172.21.15.153:0/1067408567 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d409e010 con 0x7f20b4006190 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 1 -- 172.21.15.153:0/1067408567 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d4131bd0 con 0x7f20b4003670 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20c3fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 msgr2=0x7f20b4005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:34.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.555+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 0x7f20b4005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:34.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.557+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> v1:172.21.15.153:6789/0 conn(0x7f20b4006190 legacy=0x7f20b4008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:34.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.756+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 msgr2=0x7f20b4005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:34.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:34.756+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 0x7f20b4005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:49:35.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:35.157+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 msgr2=0x7f20b4005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:35.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:35.157+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 0x7f20b4005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:49:35.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:35.958+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 msgr2=0x7f20b4005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:35.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:35.958+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 0x7f20b4005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:49:36.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.474+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739847274 0 0) 0x7f20d409e010 con 0x7f20b4006190 2024-09-15T03:49:36.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.474+0000 7f20d159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.474+0000 7f20d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.474+0000 7f20d159a640 10 monclient(hunting): my global_id is 254178 2024-09-15T03:49:36.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.474+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20bc0034e0 con 0x7f20b4006190 2024-09-15T03:49:36.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2514007064 0 0) 0x7f20bc0034e0 con 0x7f20b4006190 2024-09-15T03:49:36.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 msgr2=0x7f20b4005aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:36.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 --2- 172.21.15.153:0/1067408567 >> v2:172.21.15.181:3300/0 conn(0x7f20b4003670 0x7f20b4005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d409ca60 con 0x7f20b4006190 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f20c8009860 con 0x7f20b4006190 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: dump: 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 >> v1:172.21.15.153:6789/0 conn(0x7f20b4006190 legacy=0x7f20b4008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:49:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.475+0000 7f20d159a640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20bc0039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): picked mon.c con 0x7f20d4132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc0040c0 0x7f20bc004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20bc0039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d2d9d640 1 -- 172.21.15.153:0/1067408567 reap_dead start 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): picked mon.b con 0x7f20bc0040c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20bc004cc0 0x7f20bc005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): picked mon.a con 0x7f20bc004cc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4131bd0 con 0x7f20bc004cc0 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc00bc30 con 0x7f20bc0040c0 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc00bdf0 con 0x7f20d4132170 2024-09-15T03:49:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d159a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d1d9b640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc0040c0 msgr2=0x7f20bc004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d1d9b640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc0040c0 0x7f20bc004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d259c640 10 monclient(hunting): get_auth_request con 0x7f20d4132170 auth_method 0 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.476+0000 7f20d259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient(hunting): handle_auth_done global_id 254178 payload 293 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc0040c0 msgr2=0x7f20bc004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc0040c0 0x7f20bc004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20bc004cc0 msgr2=0x7f20bc005130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20bc004cc0 0x7f20bc005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 1 monclient: found mon.c 2024-09-15T03:49:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d409ca60 con 0x7f20d4132170 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d8d5d640 5 monclient: authenticate success, global_id 254178 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.477+0000 7f20d8d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.480+0000 7f20d259c640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20bc0039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.480+0000 7f20d259c640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20bc0039d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009f70 tx=0x7f20cc0095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f997c06acdb33c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc02c650 con 0x7f20d4132170 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: dump: 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20cc02c7f0 con 0x7f20d4132170 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 1 -- 172.21.15.153:0/1067408567 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc02cbb0 con 0x7f20d4132170 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d8d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d8d5d640 4 set_mon_vals no callback set 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.483+0000 7f20d159a640 10 monclient: dump: 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.486 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.487 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.487 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.487 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 10 monclient: shutdown 2024-09-15T03:49:36.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:36.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 -- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 msgr2=0x7f20bc0039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20bc0039d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009f70 tx=0x7f20cc0095b0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 -- 172.21.15.153:0/1067408567 shutdown_connections 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20bc004cc0 0x7f20bc005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc0040c0 0x7f20bc004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/1067408567 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20d4132170 0x7f20bc0039d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 -- 172.21.15.153:0/1067408567 >> 172.21.15.153:0/1067408567 conn(0x7f20d40a1e30 msgr2=0x7f20d40a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 -- 172.21.15.153:0/1067408567 shutdown_connections 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.484+0000 7f20d8d5d640 1 -- 172.21.15.153:0/1067408567 wait complete. 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.485+0000 7f20d8d5d640 10 monclient: build_initial_monmap 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.485+0000 7f20d8d5d640 10 monclient: monmap: 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:36.493 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.486+0000 7f20d8d5d640 1 Processor -- start 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.486+0000 7f20d8d5d640 1 -- start start 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.486+0000 7f20d8d5d640 10 monclient: init 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20bc0040c0 0x7f20d4140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f20bc0040c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc004cc0 0x7f20d4130910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f20bc004cc0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 0x7f20d411a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20d4132170 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d411dd10 con 0x7f20d4132170 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d411da30 con 0x7f20bc004cc0 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d411dff0 con 0x7f20bc0040c0 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d8d5d640 10 monclient(hunting): authenticate will time out at 1974.126289s 2024-09-15T03:49:36.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d1d9b640 1 -- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc004cc0 msgr2=0x7f20d4130910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 0x7f20d411a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20bc0040c0 0x7f20d4140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d1d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc004cc0 0x7f20d4130910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 0x7f20d411a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59124/0 (socket says 172.21.15.153:59124) 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 1 -- 172.21.15.153:0/502659170 learned_addr learned my addr 172.21.15.153:0/502659170 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request con 0x7f20d4132170 auth_method 0 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): get_auth_request con 0x7f20bc0040c0 auth_method 0 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.487+0000 7f20d2d9d640 10 monclient(hunting): handle_auth_done global_id 254181 payload 824 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 1 -- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20bc0040c0 msgr2=0x7f20d4140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20bc0040c0 0x7f20d4140be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 1 -- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc004cc0 msgr2=0x7f20d4130910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc004cc0 0x7f20d4130910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 1 monclient: found mon.noname-a 2024-09-15T03:49:36.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d2d9d640 1 -- 172.21.15.153:0/502659170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d411cb40 con 0x7f20d4132170 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.488+0000 7f20d259c640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20bc0040c0 0x7f20d4140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d2d9d640 1 -- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 msgr2=0x7f20d411a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d2d9d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 0x7f20d411a3e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f20b4006100 tx=0x7f20c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4013e70 con 0x7f20d4132170 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: dump: 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.498 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 5 monclient: authenticate success, global_id 254181 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4016460 con 0x7f20d4132170 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 10 monclient: _renew_subs 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d411cd90 con 0x7f20d4132170 2024-09-15T03:49:36.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4016820 con 0x7f20d4132170 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20c27fc640 10 monclient: dump: 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d0d99640 4 set_mon_vals no callback set 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 10 monclient: _renew_subs 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.490+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d41412c0 con 0x7f20d4132170 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.491+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f20c4016c90 con 0x7f20d4132170 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.491+0000 7f20d8d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.491+0000 7f20d8d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.491+0000 7f20d8d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.491+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d404bf60 con 0x7f20d4132170 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.491+0000 7f20c27fc640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20b4038df0 0x7f20b403b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.492+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f20c404c780 con 0x7f20d4132170 2024-09-15T03:49:36.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.492+0000 7f20d259c640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20b4038df0 0x7f20b403b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.492+0000 7f20d259c640 10 monclient: get_auth_request con 0x7f20b4038df0 auth_method 0 2024-09-15T03:49:36.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.494+0000 7f20d259c640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20b4038df0 0x7f20b403b2a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f20cc02f000 tx=0x7f20cc036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.496+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f20c4014b30 con 0x7f20d4132170 2024-09-15T03:49:36.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.496+0000 7f20c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.496+0000 7f20c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20d8d5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20d8d5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20d8d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f20d4098ee0 con 0x7f20d4132170 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20c27fc640 1 -- 172.21.15.153:0/502659170 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f20c4014350 con 0x7f20d4132170 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:49:36.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.713+0000 7f20c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:36.715 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":216,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:49:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20b4038df0 msgr2=0x7f20b403b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20b4038df0 0x7f20b403b2a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f20cc02f000 tx=0x7f20cc036000 comp rx=0 tx=0).stop 2024-09-15T03:49:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 10 monclient: shutdown 2024-09-15T03:49:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 msgr2=0x7f20d411a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 0x7f20d411a3e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f20b4006100 tx=0x7f20c4008040 comp rx=0 tx=0).stop 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 shutdown_connections 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f20b4038df0 0x7f20b403b2a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20d4132170 0x7f20d411a3e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20bc004cc0 0x7f20d4130910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 --2- 172.21.15.153:0/502659170 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20bc0040c0 0x7f20d4140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 >> 172.21.15.153:0/502659170 conn(0x7f20d40a1e30 msgr2=0x7f20d40a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 shutdown_connections 2024-09-15T03:49:36.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.715+0000 7f20d8d5d640 1 -- 172.21.15.153:0/502659170 wait complete. 2024-09-15T03:49:36.727 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:49:36.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:49:36.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:49:36.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.820+0000 7f298515d640 10 monclient: get_monmap_and_config 2024-09-15T03:49:36.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.820+0000 7f298515d640 10 monclient: build_initial_monmap 2024-09-15T03:49:36.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.820+0000 7f298515d640 10 monclient: monmap: 2024-09-15T03:49:36.821 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:36.821673+0000 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:36.821673+0000 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:36.822 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.821+0000 7f298515d640 1 Processor -- start 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.821+0000 7f298515d640 1 -- start start 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.821+0000 7f298515d640 10 monclient: init 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.821+0000 7f298515d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.821+0000 7f298515d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.821+0000 7f298515d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 0x7f298009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): picked mon.noname-b con 0x7f298009c860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f298009aba0 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 0x7f298009df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): picked mon.noname-a con 0x7f298009af70 addr v2:172.21.15.153:3300/0 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298006a320 con 0x7f298009aba0 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298009a9d0 con 0x7f298009af70 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2980068a20 con 0x7f298009c860 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f298515d640 10 monclient(hunting): authenticate will time out at 1974.461063s 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 0x7f298009df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 0x7f298009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 0x7f298009cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38534/0 (socket says 172.21.15.153:38534) 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 0x7f298009df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59138/0 (socket says 172.21.15.153:59138) 2024-09-15T03:49:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f298009aba0 0x7f298009d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39714/0 (socket says 172.21.15.153:39714) 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 -- 172.21.15.153:0/1435555872 learned_addr learned my addr 172.21.15.153:0/1435555872 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): get_auth_request con 0x7f298009c860 auth_method 0 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): get_auth_request con 0x7f298009af70 auth_method 0 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient(hunting): handle_auth_done global_id 244298 payload 274 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 -- 172.21.15.153:0/1435555872 >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 msgr2=0x7f298009df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 --2- 172.21.15.153:0/1435555872 >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 0x7f298009df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 -- 172.21.15.153:0/1435555872 >> v1:172.21.15.153:6789/0 conn(0x7f298009aba0 legacy=0x7f298009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 monclient: found mon.noname-b 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297ed9d640 1 -- 172.21.15.153:0/1435555872 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980068e60 con 0x7f298009c860 2024-09-15T03:49:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.822+0000 7f297f59e640 1 --2- 172.21.15.153:0/1435555872 >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 0x7f298009df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297ed9d640 1 -- 172.21.15.153:0/1435555872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 msgr2=0x7f298009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297ed9d640 1 --2- 172.21.15.153:0/1435555872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 0x7f298009cc30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2970009910 tx=0x7f29700091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=127bd15f33ee7712 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 1 -- 172.21.15.153:0/1435555872 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297002aea0 con 0x7f298009c860 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: dump: 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 1 -- 172.21.15.153:0/1435555872 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970032440 con 0x7f298009c860 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 5 monclient: authenticate success, global_id 244298 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:36.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 1 -- 172.21.15.153:0/1435555872 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29700327e0 con 0x7f298009c860 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 4 set_mon_vals no callback set 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f297dd9b640 10 monclient: dump: 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.838 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 10 monclient: shutdown 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 -- 172.21.15.153:0/1435555872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 msgr2=0x7f298009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 --2- 172.21.15.153:0/1435555872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 0x7f298009cc30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2970009910 tx=0x7f29700091d0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 -- 172.21.15.153:0/1435555872 shutdown_connections 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 --2- 172.21.15.153:0/1435555872 >> v2:172.21.15.153:3300/0 conn(0x7f298009af70 0x7f298009df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 --2- 172.21.15.153:0/1435555872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009c860 0x7f298009cc30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 -- 172.21.15.153:0/1435555872 >> 172.21.15.153:0/1435555872 conn(0x7f29800a2e40 msgr2=0x7f29800a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 -- 172.21.15.153:0/1435555872 shutdown_connections 2024-09-15T03:49:36.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.825+0000 7f298515d640 1 -- 172.21.15.153:0/1435555872 wait complete. 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.826+0000 7f298515d640 10 monclient: build_initial_monmap 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.826+0000 7f298515d640 10 monclient: monmap: 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.826+0000 7f298515d640 1 Processor -- start 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.826+0000 7f298515d640 1 -- start start 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.826+0000 7f298515d640 10 monclient: init 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 0x7f2980131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): picked mon.noname-c con 0x7f298009aba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298009af70 0x7f298006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): picked mon.noname-a con 0x7f298009af70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f298009c860 0x7f298006dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): picked mon.noname-b con 0x7f298009c860 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2980134ac0 con 0x7f298009af70 2024-09-15T03:49:36.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298011ecd0 con 0x7f298009c860 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298011efb0 con 0x7f298009aba0 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f298515d640 10 monclient(hunting): authenticate will time out at 1974.466451s 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 0x7f2980131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298009af70 0x7f298006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 0x7f2980131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38544/0 (socket says 172.21.15.153:38544) 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 -- 172.21.15.153:0/1316530674 learned_addr learned my addr 172.21.15.153:0/1316530674 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): get_auth_request con 0x7f298009aba0 auth_method 0 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): get_auth_request con 0x7f298009af70 auth_method 0 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297f59e640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f298009c860 msgr2=0x7f298006dca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297f59e640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f298009c860 0x7f298006dca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient(hunting): handle_auth_done global_id 244301 payload 824 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f298009c860 msgr2=0x7f298006dca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f298009c860 0x7f298006dca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298009af70 msgr2=0x7f298006d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298009af70 0x7f298006d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 monclient: found mon.noname-c 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297ed9d640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978009a20 con 0x7f298009aba0 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.827+0000 7f297e59c640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298009af70 0x7f298006d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f297ed9d640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 msgr2=0x7f2980131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f297ed9d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 0x7f2980131b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2970009910 tx=0x7f2970032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297002aea0 con 0x7f298009aba0 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: dump: 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.845 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970000d10 con 0x7f298009aba0 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 5 monclient: authenticate success, global_id 244301 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970031ea0 con 0x7f298009aba0 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 10 monclient: _renew_subs 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2970009450 con 0x7f298009aba0 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f2966ffd640 10 monclient: dump: 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f297d59a640 4 set_mon_vals no callback set 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 10 monclient: _renew_subs 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.830+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298006f1d0 con 0x7f298009aba0 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f298515d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f298515d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 0x7f298006e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2980065e00 con 0x7f29801320d0 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 0x7f298006e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2970039040 con 0x7f298009aba0 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient: get_auth_request con 0x7f29801320d0 auth_method 0 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 10 monclient(hunting): handle_auth_done global_id 254190 payload 824 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297e59c640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f298004bf60 con 0x7f29801320d0 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f2966ffd640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f296c0386f0 0x7f296c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f297006c070 con 0x7f298009aba0 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297f59e640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f296c0386f0 0x7f296c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.831+0000 7f297f59e640 10 monclient: get_auth_request con 0x7f296c0386f0 auth_method 0 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f297e59c640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 msgr2=0x7f298006e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f297e59c640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 0x7f298006e1e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f298006f860 tx=0x7f2974007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297400fd50 con 0x7f29801320d0 2024-09-15T03:49:36.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 10 monclient: dump: 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f297400feb0 con 0x7f29801320d0 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 msgr2=0x7f298006e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 0x7f298006e1e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f298006f860 tx=0x7f2974007e40 comp rx=0 tx=0).stop 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 msgr2=0x7f298006e1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.833+0000 7f2966ffd640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29801320d0 0x7f298006e1e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.834+0000 7f297f59e640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f296c0386f0 0x7f296c03aba0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2978009c50 tx=0x7f297800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f298515d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f298515d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 0x7f2980097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f298515d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298006eb20 con 0x7f298006e720 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 0x7f2980097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient: get_auth_request con 0x7f298006e720 auth_method 0 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.839+0000 7f297e59c640 10 monclient(hunting): handle_auth_done global_id 254193 payload 824 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.840+0000 7f297e59c640 1 -- 172.21.15.153:0/1316530674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f29801363c0 con 0x7f298006e720 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f297e59c640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 msgr2=0x7f2980097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f297e59c640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 0x7f2980097a70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f298006f860 tx=0x7f2974015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297401ca50 con 0x7f298006e720 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 10 monclient: dump: 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.854 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4399 (secure 0 0 0) 0x7f297400fb40 con 0x7f298006e720 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 msgr2=0x7f2980097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 0x7f2980097a70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f298006f860 tx=0x7f2974015000 comp rx=0 tx=0).stop 2024-09-15T03:49:36.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 msgr2=0x7f2980097a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:36.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f2966ffd640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f298006e720 0x7f2980097a70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.856 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:49:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 216, 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-15T03:49:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:36.868 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:36.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f296c0386f0 msgr2=0x7f296c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f296c0386f0 0x7f296c03aba0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2978009c50 tx=0x7f297800b040 comp rx=0 tx=0).stop 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 10 monclient: shutdown 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 msgr2=0x7f2980131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 0x7f2980131b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2970009910 tx=0x7f2970032000 comp rx=0 tx=0).stop 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f297f59e640 1 -- 172.21.15.153:0/1316530674 reap_dead start 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 shutdown_connections 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 --2- 172.21.15.153:0/1316530674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f298009aba0 0x7f2980131b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 >> 172.21.15.153:0/1316530674 conn(0x7f29800a2e40 msgr2=0x7f2980134cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 shutdown_connections 2024-09-15T03:49:36.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.842+0000 7f298515d640 1 -- 172.21.15.153:0/1316530674 wait complete. 2024-09-15T03:49:36.871 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:49:36.871 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:49:36.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.964+0000 7f3a81480640 10 monclient: get_monmap_and_config 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.964+0000 7f3a81480640 10 monclient: build_initial_monmap 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.964+0000 7f3a81480640 10 monclient: monmap: 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:49:36.964877+0000 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:49:36.964877+0000 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.965 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.964+0000 7f3a81480640 1 Processor -- start 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.964+0000 7f3a81480640 1 -- start start 2024-09-15T03:49:36.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.964+0000 7f3a81480640 10 monclient: init 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a7c1318f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a7c09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 0x7f3a7c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient(hunting): picked mon.noname-b con 0x7f3a7c137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c132640 con 0x7f3a7c09ea50 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c131cc0 con 0x7f3a7c1318f0 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c09f530 con 0x7f3a7c137c30 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a81480640 10 monclient(hunting): authenticate will time out at 1974.604283s 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7affd640 1 -- >> v1:172.21.15.181:6789/0 conn(0x7f3a7c1318f0 legacy=0x7f3a7c131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.181:6789/0 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7a7fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3a7c09ea50 0x7f3a7c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39718/0 (socket says 172.21.15.153:39718) 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7a7fc640 1 -- 172.21.15.153:0/3964151489 learned_addr learned my addr 172.21.15.153:0/3964151489 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 1 --2- 172.21.15.153:0/3964151489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 0x7f3a7c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f3a7c137c30 auth_method 0 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_done global_id 244304 payload 274 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 1 -- 172.21.15.153:0/3964151489 >> v1:172.21.15.181:6789/0 conn(0x7f3a7c1318f0 legacy=0x7f3a7c131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 1 -- 172.21.15.153:0/3964151489 >> v1:172.21.15.153:6789/0 conn(0x7f3a7c09ea50 legacy=0x7f3a7c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.965+0000 7f3a7b7fe640 1 -- 172.21.15.153:0/3964151489 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c09fbf0 con 0x7f3a7c137c30 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a7b7fe640 1 -- 172.21.15.153:0/3964151489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 msgr2=0x7f3a7c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a7b7fe640 1 --2- 172.21.15.153:0/3964151489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 0x7f3a7c13a010 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009880 tx=0x7f3a6c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eac0e8219290011e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 1 -- 172.21.15.153:0/3964151489 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c032440 con 0x7f3a7c137c30 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: dump: 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.978 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 1 -- 172.21.15.153:0/3964151489 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6c0325a0 con 0x7f3a7c137c30 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a81480640 5 monclient: authenticate success, global_id 244304 2024-09-15T03:49:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a81480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a81480640 10 monclient: get_monmap_and_config success 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 1 -- 172.21.15.153:0/3964151489 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c032940 con 0x7f3a7c137c30 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a81480640 4 set_mon_vals no callback set 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.968+0000 7f3a79ffb640 10 monclient: dump: 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 10 monclient: shutdown 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 -- 172.21.15.153:0/3964151489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 msgr2=0x7f3a7c13a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 --2- 172.21.15.153:0/3964151489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 0x7f3a7c13a010 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009880 tx=0x7f3a6c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 -- 172.21.15.153:0/3964151489 shutdown_connections 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 --2- 172.21.15.153:0/3964151489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c137c30 0x7f3a7c13a010 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 -- 172.21.15.153:0/3964151489 >> 172.21.15.153:0/3964151489 conn(0x7f3a7c067110 msgr2=0x7f3a7c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 -- 172.21.15.153:0/3964151489 shutdown_connections 2024-09-15T03:49:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 1 -- 172.21.15.153:0/3964151489 wait complete. 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 10 monclient: build_initial_monmap 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.969+0000 7f3a81480640 10 monclient: monmap: 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 Processor -- start 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 -- start start 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient: init 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient: _reopen_session rank -1 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a7c09ea50 0x7f3a7c1362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): picked mon.noname-b con 0x7f3a7c09ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 0x7f3a7c12fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): picked mon.noname-c con 0x7f3a7c1318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a7c137c30 0x7f3a7c12e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): picked mon.noname-a con 0x7f3a7c137c30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c132b50 con 0x7f3a7c137c30 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c132870 con 0x7f3a7c09ea50 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1313a0 con 0x7f3a7c1318f0 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): _renew_subs 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a81480640 10 monclient(hunting): authenticate will time out at 1974.609636s 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 0x7f3a7c12fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a7c137c30 0x7f3a7c12e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 0x7f3a7c12fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:38556/0 (socket says 172.21.15.153:38556) 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 -- 172.21.15.153:0/2262039939 learned_addr learned my addr 172.21.15.153:0/2262039939 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): get_auth_request con 0x7f3a7c1318f0 auth_method 0 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7affd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a7c09ea50 msgr2=0x7f3a7c1362d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.181:3300/0 2024-09-15T03:49:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f3a7c137c30 auth_method 0 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7affd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a7c09ea50 0x7f3a7c1362d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient(hunting): handle_auth_done global_id 244307 payload 824 2024-09-15T03:49:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a7c09ea50 msgr2=0x7f3a7c1362d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a7c09ea50 0x7f3a7c1362d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a7c137c30 msgr2=0x7f3a7c12e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a7c137c30 0x7f3a7c12e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient: _finish_hunting 0 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 monclient: found mon.noname-c 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.970+0000 7f3a7a7fc640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c009390 con 0x7f3a7c1318f0 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a7a7fc640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 msgr2=0x7f3a7c12fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a7a7fc640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 0x7f3a7c12fe80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000cca0 tx=0x7f3a700053f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7000aa60 con 0x7f3a7c1318f0 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: dump: 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: _finish_auth 0 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7000abe0 con 0x7f3a7c1318f0 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 5 monclient: authenticate success, global_id 244307 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a700070f0 con 0x7f3a7c1318f0 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 10 monclient: _renew_subs 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c131680 con 0x7f3a7c1318f0 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a62ffd640 10 monclient: dump: 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a797fa640 4 set_mon_vals no callback set 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 10 monclient: _renew_subs 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:49:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.973+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c137350 con 0x7f3a7c1318f0 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f3a70007520 con 0x7f3a7c1318f0 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a81480640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a81480640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 0x7f3a7c12e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a81480640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c065e00 con 0x7f3a7c12e580 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a62ffd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3a7004c830 con 0x7f3a7c1318f0 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 0x7f3a7c12e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient: get_auth_request con 0x7f3a7c12e580 auth_method 0 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 10 monclient(hunting): handle_auth_done global_id 244310 payload 824 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7affd640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a7c04bf60 con 0x7f3a7c12e580 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7b7fe640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.974+0000 7f3a7b7fe640 10 monclient: get_auth_request con 0x7f3a68038740 auth_method 0 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.976+0000 7f3a7affd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 msgr2=0x7f3a7c12e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.976+0000 7f3a7affd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 0x7f3a7c12e950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b480 tx=0x7f3a740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a740050f0 con 0x7f3a7c12e580 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 10 monclient: dump: 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3a74005250 con 0x7f3a7c12e580 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 msgr2=0x7f3a7c12e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 0x7f3a7c12e950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b480 tx=0x7f3a740091a0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 msgr2=0x7f3a7c12e950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:36.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a62ffd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c12e580 0x7f3a7c12e950 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:36.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.977+0000 7f3a7b7fe640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c06f600 tx=0x7f3a6c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.982+0000 7f3a81480640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.982+0000 7f3a81480640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 0x7f3a7c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.982+0000 7f3a81480640 10 monclient(hunting): start opening mon connection 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.982+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c132cf0 con 0x7f3a7c0a3d60 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 0x7f3a7c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient: get_auth_request con 0x7f3a7c0a3d60 auth_method 0 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:49:36.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 10 monclient(hunting): handle_auth_done global_id 244313 payload 824 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.983+0000 7f3a7affd640 1 -- 172.21.15.153:0/2262039939 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3a7c12ee90 con 0x7f3a7c0a3d60 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a7affd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 msgr2=0x7f3a7c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a7affd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 0x7f3a7c0a4160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b480 tx=0x7f3a74015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a740059e0 con 0x7f3a7c0a3d60 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 10 monclient: dump: 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:49:36.997 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f3a74003cb0 con 0x7f3a7c0a3d60 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 msgr2=0x7f3a7c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 0x7f3a7c0a4160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b480 tx=0x7f3a74015000 comp rx=0 tx=0).stop 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 msgr2=0x7f3a7c0a4160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:49:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.985+0000 7f3a62ffd640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c0a3d60 0x7f3a7c0a4160 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b480 tx=0x7f3a74015000 comp rx=0 tx=0).stop 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 216, 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:36.999 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:37.000 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:49:37.001 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:49:37.002 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.003 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.006 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:49:37.007 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:49:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:49:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:49:37.011 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:49:37.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 msgr2=0x7f3a6803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:37.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f3a68038740 0x7f3a6803abf0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c06f600 tx=0x7f3a6c032000 comp rx=0 tx=0).stop 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 10 monclient: shutdown 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 msgr2=0x7f3a7c12fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 0x7f3a7c12fe80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000cca0 tx=0x7f3a700053f0 comp rx=0 tx=0).stop 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a7b7fe640 1 -- 172.21.15.153:0/2262039939 reap_dead start 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 shutdown_connections 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 --2- 172.21.15.153:0/2262039939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a7c1318f0 0x7f3a7c12fe80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 >> 172.21.15.153:0/2262039939 conn(0x7f3a7c067110 msgr2=0x7f3a7c09d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 shutdown_connections 2024-09-15T03:49:37.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:49:36.986+0000 7f3a81480640 1 -- 172.21.15.153:0/2262039939 wait complete. 2024-09-15T03:49:37.013 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:49:37.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-15T03:49:57.016 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:49:57.016 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:49:57.016 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:49:57.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:49:57.019 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:49:57.019 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-15T03:49:57.020 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-15T03:49:57.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:50:12.020 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-15T03:50:12.021 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-15T03:50:12.021 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:50:12.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.126+0000 7fcc490d4640 10 monclient: get_monmap_and_config 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.126+0000 7fcc490d4640 10 monclient: build_initial_monmap 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.126+0000 7fcc490d4640 10 monclient: monmap: 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:12.127810+0000 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:12.127810+0000 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.126+0000 7fcc490d4640 1 Processor -- start 2024-09-15T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.126+0000 7fcc490d4640 1 -- start start 2024-09-15T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.126+0000 7fcc490d4640 10 monclient: init 2024-09-15T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): picked mon.noname-a con 0x7fcc441318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): picked mon.noname-b con 0x7fcc4409ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc44068d90 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc44131cc0 con 0x7fcc44068d90 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4409fbf0 con 0x7fcc441318f0 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4409fe30 con 0x7fcc4409ea50 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc490d4640 10 monclient(hunting): authenticate will time out at 2009.767254s 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44086/0 (socket says 172.21.15.153:44086) 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47606/0 (socket says 172.21.15.153:47606) 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 learned_addr learned my addr 172.21.15.153:0/1619373742 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc4409ea50 auth_method 0 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request con 0x7fcc441318f0 auth_method 0 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.127+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.129+0000 7fcc4359e640 1 -- 172.21.15.153:0/1619373742 >> v1:172.21.15.181:6789/0 conn(0x7fcc44068d90 legacy=0x7fcc440691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc4409ea50 auth_method 0 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request con 0x7fcc441318f0 auth_method 0 2024-09-15T03:50:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:50:12.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.328+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:50:12.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:12.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:12.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request con 0x7fcc441318f0 auth_method 0 2024-09-15T03:50:12.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc4409ea50 auth_method 0 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:12.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:50:12.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:12.729+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:50:13.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc4409ea50 auth_method 0 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request con 0x7fcc441318f0 auth_method 0 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:13.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:50:13.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:13.531+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient(hunting): tick 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 monclient(hunting): continuing hunt 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 msgr2=0x7fcc4409ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc4409ea50 0x7fcc4409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 msgr2=0x7fcc44131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.153:3300/0 conn(0x7fcc441318f0 0x7fcc44131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 -- 172.21.15.153:0/1619373742 >> v1:172.21.15.181:6789/0 conn(0x7fcc44068d90 legacy=0x7fcc440691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient: _add_conns ranks=[0,4] 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc30003720 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient(hunting): picked mon.noname-c con 0x7fcc30007420 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 -- 172.21.15.153:0/1619373742 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc440a1210 con 0x7fcc30003720 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 1 -- 172.21.15.153:0/1619373742 --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4409fbf0 con 0x7fcc30007420 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc40d99640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc30007420 auth_method 0 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:15.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.128+0000 7fcc4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:15.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.129+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:15.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.129+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:15.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.129+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:15.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.129+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:15.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.130+0000 7fcc42d9d640 1 -- 172.21.15.153:0/1619373742 >> v1:172.21.15.153:6789/0 conn(0x7fcc30003720 legacy=0x7fcc30005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:15.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:15.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc30007420 auth_method 0 2024-09-15T03:50:15.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:15.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:15.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:15.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:15.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:15.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:15.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.330+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc30007420 auth_method 0 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:15.731+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.532+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc30007420 auth_method 0 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:50:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:16.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:16.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:16.533+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:50:17.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607571110 0 0) 0x7fcc440a1210 con 0x7fcc30003720 2024-09-15T03:50:17.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:17.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient(hunting): my global_id is 254199 2024-09-15T03:50:17.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0034e0 con 0x7fcc30003720 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875641899 0 0) 0x7fcc2c0034e0 con 0x7fcc30003720 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 msgr2=0x7fcc300098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 --2- 172.21.15.153:0/1619373742 >> v2:172.21.15.181:3300/0 conn(0x7fcc30007420 0x7fcc300098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3000b020 con 0x7fcc30003720 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: _finish_auth 0 2024-09-15T03:50:17.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fcc34009a00 con 0x7fcc30003720 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: dump: 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:17.045 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:17.045 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:17.045 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:17.045 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:17.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:50:17.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:17.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 >> v1:172.21.15.153:6789/0 conn(0x7fcc30003720 legacy=0x7fcc30005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:17.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc4359e640 1 -- 172.21.15.153:0/1619373742 reap_dead start 2024-09-15T03:50:17.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:17.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c003a20 0x7fcc2c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.039+0000 7fcc41d9b640 10 monclient(hunting): picked mon.c con 0x7fcc2c003a20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 0x7fcc2c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): picked mon.a con 0x7fcc2c004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c005330 0x7fcc2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): picked mon.b con 0x7fcc2c005330 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4409fbf0 con 0x7fcc2c004780 2024-09-15T03:50:17.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:17.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c00c260 con 0x7fcc2c005330 2024-09-15T03:50:17.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:17.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c00c420 con 0x7fcc2c003a20 2024-09-15T03:50:17.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:17.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc41d9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c003a20 0x7fcc2c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 0x7fcc2c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc2c004780 auth_method 0 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request con 0x7fcc2c003a20 auth_method 0 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:17.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4359e640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c005330 0x7fcc2c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:17.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4359e640 10 monclient(hunting): get_auth_request con 0x7fcc2c005330 auth_method 0 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient(hunting): handle_auth_done global_id 254199 payload 293 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c003a20 msgr2=0x7fcc2c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c003a20 0x7fcc2c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c005330 msgr2=0x7fcc2c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c005330 0x7fcc2c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient: _finish_hunting 0 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 monclient: found mon.a 2024-09-15T03:50:17.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:17.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:17.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0034e0 con 0x7fcc2c004780 2024-09-15T03:50:17.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 10 monclient: _finish_auth 0 2024-09-15T03:50:17.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc4259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:17.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc490d4640 5 monclient: authenticate success, global_id 254199 2024-09-15T03:50:17.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.040+0000 7fcc490d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:17.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.043+0000 7fcc4259c640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 msgr2=0x7fcc2c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:17.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.043+0000 7fcc4259c640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 0x7fcc2c004bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc38006fd0 tx=0x7fcc38008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13b9336979023d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3802c7e0 con 0x7fcc2c004780 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: dump: 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:17.062 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc490d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3800c040 con 0x7fcc2c004780 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc490d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 1 -- 172.21.15.153:0/1619373742 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3802cb70 con 0x7fcc2c004780 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc490d4640 4 set_mon_vals no callback set 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.044+0000 7fcc41d9b640 10 monclient: dump: 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:17.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 10 monclient: shutdown 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 -- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 msgr2=0x7fcc2c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 0x7fcc2c004bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc38006fd0 tx=0x7fcc38008040 comp rx=0 tx=0).stop 2024-09-15T03:50:17.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 -- 172.21.15.153:0/1619373742 shutdown_connections 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c005330 0x7fcc2c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c004780 0x7fcc2c004bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 --2- 172.21.15.153:0/1619373742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c003a20 0x7fcc2c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 -- 172.21.15.153:0/1619373742 >> 172.21.15.153:0/1619373742 conn(0x7fcc44067110 msgr2=0x7fcc44067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 -- 172.21.15.153:0/1619373742 shutdown_connections 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.045+0000 7fcc490d4640 1 -- 172.21.15.153:0/1619373742 wait complete. 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.046+0000 7fcc490d4640 10 monclient: build_initial_monmap 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.046+0000 7fcc490d4640 10 monclient: monmap: 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:17.065 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 Processor -- start 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 -- start start 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient: init 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c003a20 0x7fcc4409a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2c003a20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 0x7fcc4411bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2c004780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 0x7fcc44130a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2c005330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4411e140 con 0x7fcc2c005330 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4411dbf0 con 0x7fcc2c003a20 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4411de60 con 0x7fcc2c004780 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc490d4640 10 monclient(hunting): authenticate will time out at 2014.686685s 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 0x7fcc4411bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 0x7fcc44130a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 0x7fcc44130a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33600/0 (socket says 172.21.15.153:33600) 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 0x7fcc4411bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41318/0 (socket says 172.21.15.153:41318) 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 1 -- 172.21.15.153:0/3793375389 learned_addr learned my addr 172.21.15.153:0/3793375389 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): get_auth_request con 0x7fcc2c005330 auth_method 0 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 10 monclient(hunting): get_auth_request con 0x7fcc2c004780 auth_method 0 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:17.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.047+0000 7fcc4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c003a20 0x7fcc4409a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request con 0x7fcc2c003a20 auth_method 0 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 10 monclient(hunting): handle_auth_done global_id 244316 payload 824 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 1 -- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c003a20 msgr2=0x7fcc4409a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c003a20 0x7fcc4409a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 1 -- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 msgr2=0x7fcc44130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:17.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 0x7fcc44130a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 10 monclient: _finish_hunting 0 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 1 monclient: found mon.noname-c 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4259c640 1 -- 172.21.15.153:0/3793375389 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4411cd20 con 0x7fcc2c004780 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.048+0000 7fcc4359e640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 0x7fcc44130a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc4259c640 1 -- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 msgr2=0x7fcc4411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc4259c640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 0x7fcc4411bbf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc38002000 tx=0x7fcc38036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38044070 con 0x7fcc2c004780 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: dump: 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:17.070 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: _finish_auth 0 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc380349a0 con 0x7fcc2c004780 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 5 monclient: authenticate success, global_id 244316 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 10 monclient: _renew_subs 2024-09-15T03:50:17.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc44130fa0 con 0x7fcc2c004780 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38034d40 con 0x7fcc2c004780 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc2affd640 10 monclient: dump: 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:17.072 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 10 monclient: _renew_subs 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc44140ab0 con 0x7fcc2c004780 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.050+0000 7fcc4159a640 4 set_mon_vals no callback set 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fcc38040690 con 0x7fcc2c004780 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc2affd640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc30039090 0x7fcc3003b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc490d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc490d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcc3806e170 con 0x7fcc2c004780 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc490d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:17.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.051+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4411e280 con 0x7fcc2c004780 2024-09-15T03:50:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.052+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc30039090 0x7fcc3003b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.052+0000 7fcc42d9d640 10 monclient: get_auth_request con 0x7fcc30039090 auth_method 0 2024-09-15T03:50:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.054+0000 7fcc42d9d640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc30039090 0x7fcc3003b540 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcc34000f80 tx=0x7fcc3400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.056+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcc4411e280 con 0x7fcc2c004780 2024-09-15T03:50:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.056+0000 7fcc2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.056+0000 7fcc2affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.273+0000 7fcc490d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:50:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.273+0000 7fcc490d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:50:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.273+0000 7fcc490d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.273+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcc4411e280 con 0x7fcc2c004780 2024-09-15T03:50:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.274+0000 7fcc2affd640 1 -- 172.21.15.153:0/3793375389 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fcc4411e280 con 0x7fcc2c004780 2024-09-15T03:50:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.274+0000 7fcc2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:50:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.274+0000 7fcc2affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:17.275 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":220,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc30039090 msgr2=0x7fcc3003b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc30039090 0x7fcc3003b540 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcc34000f80 tx=0x7fcc3400b040 comp rx=0 tx=0).stop 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 10 monclient: shutdown 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 msgr2=0x7fcc4411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 0x7fcc4411bbf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc38002000 tx=0x7fcc38036000 comp rx=0 tx=0).stop 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 shutdown_connections 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fcc30039090 0x7fcc3003b540 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc2c005330 0x7fcc44130a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc2c004780 0x7fcc4411bbf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 --2- 172.21.15.153:0/3793375389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc2c003a20 0x7fcc4409a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 >> 172.21.15.153:0/3793375389 conn(0x7fcc44067110 msgr2=0x7fcc44134670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 shutdown_connections 2024-09-15T03:50:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:17.276+0000 7fcc490d4640 1 -- 172.21.15.153:0/3793375389 wait complete. 2024-09-15T03:50:20.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.392+0000 7f1067889640 10 monclient: get_monmap_and_config 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.392+0000 7f1067889640 10 monclient: build_initial_monmap 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.392+0000 7f1067889640 10 monclient: monmap: 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:20.393145+0000 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:20.393145+0000 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:20.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:20.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:20.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:20.394 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:20.394 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:20.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.392+0000 7f1067889640 1 Processor -- start 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 1 -- start start 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient: init 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 0x7f1060096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): picked mon.noname-b con 0x7f106009a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10600971e0 0x7f10600975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): picked mon.noname-a con 0x7f10600971e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:20.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f106012f670 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106006ee10 con 0x7f106012f670 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106009cb20 con 0x7f10600971e0 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1060097af0 con 0x7f106009a510 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1067889640 10 monclient(hunting): authenticate will time out at 2018.032747s 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10600971e0 0x7f10600975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065625640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 0x7f1060096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10600971e0 0x7f10600975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33612/0 (socket says 172.21.15.153:33612) 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065625640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 0x7f1060096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41346/0 (socket says 172.21.15.153:41346) 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 1 -- 172.21.15.153:0/1035252203 learned_addr learned my addr 172.21.15.153:0/1035252203 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 10 monclient(hunting): get_auth_request con 0x7f10600971e0 auth_method 0 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065e26640 1 --1- 172.21.15.153:0/1035252203 >> v1:172.21.15.153:6789/0 conn(0x7f106012f670 0x7f106012fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40214/0 (socket says 172.21.15.153:40214) 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1064e24640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065625640 10 monclient(hunting): get_auth_request con 0x7f106009a510 auth_method 0 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065625640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065625640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.393+0000 7f1065625640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1065625640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1065625640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1065625640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 10 monclient(hunting): handle_auth_done global_id 254205 payload 274 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 1 -- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 msgr2=0x7f1060096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 1 --2- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 0x7f1060096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 1 -- 172.21.15.153:0/1035252203 >> v1:172.21.15.153:6789/0 conn(0x7f106012f670 legacy=0x7f106012fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 1 monclient: found mon.noname-a 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1064e24640 1 -- 172.21.15.153:0/1035252203 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106009a8e0 con 0x7f10600971e0 2024-09-15T03:50:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.394+0000 7f1065625640 1 --2- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 0x7f1060096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:20.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1064e24640 1 -- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10600971e0 msgr2=0x7f10600975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1064e24640 1 --2- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10600971e0 0x7f10600975b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f105000dd40 tx=0x7f105000d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6c3a8cef40ca31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 1 -- 172.21.15.153:0/1035252203 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105000d2e0 con 0x7f10600971e0 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: dump: 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 5 monclient: authenticate success, global_id 254205 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 1 -- 172.21.15.153:0/1035252203 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1050031050 con 0x7f10600971e0 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:20.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 4 set_mon_vals no callback set 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 1 -- 172.21.15.153:0/1035252203 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105000b890 con 0x7f10600971e0 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1057fff640 10 monclient: dump: 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.409 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 10 monclient: shutdown 2024-09-15T03:50:20.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 1 -- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10600971e0 msgr2=0x7f10600975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.397+0000 7f1067889640 1 --2- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10600971e0 0x7f10600975b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f105000dd40 tx=0x7f105000d660 comp rx=0 tx=0).stop 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 -- 172.21.15.153:0/1035252203 shutdown_connections 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 --2- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10600971e0 0x7f10600975b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 --2- 172.21.15.153:0/1035252203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f106009a510 0x7f1060096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 -- 172.21.15.153:0/1035252203 >> 172.21.15.153:0/1035252203 conn(0x7f10600672b0 msgr2=0x7f10600676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 -- 172.21.15.153:0/1035252203 shutdown_connections 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 -- 172.21.15.153:0/1035252203 wait complete. 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 10 monclient: build_initial_monmap 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 10 monclient: monmap: 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 Processor -- start 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 1 -- start start 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.398+0000 7f1067889640 10 monclient: init 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 0x7f106013cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): picked mon.noname-c con 0x7f10600971e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106009a510 0x7f106012e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): picked mon.noname-a con 0x7f106009a510 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106012f670 0x7f106011a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): picked mon.noname-b con 0x7f106012f670 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106009c3a0 con 0x7f106009a510 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1060098720 con 0x7f106012f670 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106013dae0 con 0x7f10600971e0 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1067889640 10 monclient(hunting): authenticate will time out at 2018.038609s 2024-09-15T03:50:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 0x7f106013cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106009a510 0x7f106012e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 0x7f106013cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41354/0 (socket says 172.21.15.153:41354) 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106009a510 0x7f106012e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33620/0 (socket says 172.21.15.153:33620) 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 -- 172.21.15.153:0/3533314169 learned_addr learned my addr 172.21.15.153:0/3533314169 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): get_auth_request con 0x7f10600971e0 auth_method 0 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): get_auth_request con 0x7f106009a510 auth_method 0 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1064e24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient(hunting): handle_auth_done global_id 244322 payload 824 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 -- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106012f670 msgr2=0x7f106011a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106012f670 0x7f106011a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 -- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106009a510 msgr2=0x7f106012e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106009a510 0x7f106012e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 monclient: found mon.noname-c 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.399+0000 7f1065625640 1 -- 172.21.15.153:0/3533314169 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c009390 con 0x7f10600971e0 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1065625640 1 -- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 msgr2=0x7f106013cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1065625640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 0x7f106013cbb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f105c00ec80 tx=0x7f105c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00eec0 con 0x7f10600971e0 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: dump: 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c00cdd0 con 0x7f10600971e0 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 5 monclient: authenticate success, global_id 244322 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 10 monclient: _renew_subs 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105000d900 con 0x7f10600971e0 2024-09-15T03:50:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c013680 con 0x7f10600971e0 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10557fa640 10 monclient: dump: 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 10 monclient: _renew_subs 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f10577fe640 4 set_mon_vals no callback set 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.402+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106011dd30 con 0x7f10600971e0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f105c0139f0 con 0x7f10600971e0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f1067889640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f1067889640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f1067889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f10557fa640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1044038740 0x7f104403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106013d150 con 0x7f10600971e0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f105c04cb70 con 0x7f10600971e0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f1064e24640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1044038740 0x7f104403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.403+0000 7f1064e24640 10 monclient: get_auth_request con 0x7f1044038740 auth_method 0 2024-09-15T03:50:20.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.406+0000 7f1064e24640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1044038740 0x7f104403abf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f106009b210 tx=0x7f105003a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.408+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f105c0234d0 con 0x7f10600971e0 2024-09-15T03:50:20.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.408+0000 7f10557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.408+0000 7f10557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f1067889640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f1067889640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f1067889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f106004bf60 con 0x7f10600971e0 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f10557fa640 1 -- 172.21.15.153:0/3533314169 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f105c0232f0 con 0x7f10600971e0 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f10557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:50:20.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.621+0000 7f10557fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:20.623 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:50:20.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1044038740 msgr2=0x7f104403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1044038740 0x7f104403abf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f106009b210 tx=0x7f105003a000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 10 monclient: shutdown 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 msgr2=0x7f106013cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 0x7f106013cbb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f105c00ec80 tx=0x7f105c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 shutdown_connections 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1044038740 0x7f104403abf0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106012f670 0x7f106011a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106009a510 0x7f106012e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 --2- 172.21.15.153:0/3533314169 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10600971e0 0x7f106013cbb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 >> 172.21.15.153:0/3533314169 conn(0x7f10600672b0 msgr2=0x7f106009faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 shutdown_connections 2024-09-15T03:50:20.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.623+0000 7f1067889640 1 -- 172.21.15.153:0/3533314169 wait complete. 2024-09-15T03:50:20.635 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:50:20.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:50:20.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.729+0000 7f95da19d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.729+0000 7f95da19d640 10 monclient: build_initial_monmap 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.729+0000 7f95da19d640 10 monclient: monmap: 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:20.730065+0000 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:20.730065+0000 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:20.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:20.731 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:20.731 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:20.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 Processor -- start 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 -- start start 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient: init 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 0x7f95d409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): picked mon.noname-c con 0x7f95d4096e10 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95d409aaa0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:20.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f95d40978f0 0x7f95d4135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): picked mon.noname-a con 0x7f95d40978f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d409d4a0 con 0x7f95d409aaa0 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d4099620 con 0x7f95d40978f0 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d4099340 con 0x7f95d4096e10 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95da19d640 10 monclient(hunting): authenticate will time out at 2018.369613s 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f95d40978f0 0x7f95d4135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f95d40978f0 0x7f95d4135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33636/0 (socket says 172.21.15.153:33636) 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 1 -- 172.21.15.153:0/3517539425 learned_addr learned my addr 172.21.15.153:0/3517539425 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): get_auth_request con 0x7f95d40978f0 auth_method 0 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d37fe640 1 -- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 msgr2=0x7f95d409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d37fe640 1 -- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 msgr2=0x7f95d409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d37fe640 1 --2- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 0x7f95d409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d37fe640 1 --2- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 0x7f95d409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.730+0000 7f95d3fff640 10 monclient(hunting): handle_auth_done global_id 254211 payload 274 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 1 -- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 msgr2=0x7f95d409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 1 --2- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 0x7f95d409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 1 -- 172.21.15.153:0/3517539425 >> v1:172.21.15.181:6789/0 conn(0x7f95d409aaa0 legacy=0x7f95d40971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:20.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 1 monclient: found mon.noname-a 2024-09-15T03:50:20.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:20.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.731+0000 7f95d3fff640 1 -- 172.21.15.153:0/3517539425 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d4099900 con 0x7f95d40978f0 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.733+0000 7f95d3fff640 1 -- 172.21.15.153:0/3517539425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 msgr2=0x7f95d4135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.733+0000 7f95d3fff640 1 --2- 172.21.15.153:0/3517539425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d4135b40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f95d4076be0 tx=0x7f95cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843586d71b7b524e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 1 -- 172.21.15.153:0/3517539425 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc007610 con 0x7f95d40978f0 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: dump: 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 1 -- 172.21.15.153:0/3517539425 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc02d050 con 0x7f95d40978f0 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 1 -- 172.21.15.153:0/3517539425 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0079a0 con 0x7f95d40978f0 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 5 monclient: authenticate success, global_id 254211 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:20.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 4 set_mon_vals no callback set 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95d27fc640 10 monclient: dump: 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 10 monclient: shutdown 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 -- 172.21.15.153:0/3517539425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 msgr2=0x7f95d4135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 --2- 172.21.15.153:0/3517539425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d4135b40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f95d4076be0 tx=0x7f95cc009840 comp rx=0 tx=0).stop 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 -- 172.21.15.153:0/3517539425 shutdown_connections 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 --2- 172.21.15.153:0/3517539425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d4135b40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 --2- 172.21.15.153:0/3517539425 >> v2:172.21.15.181:3300/0 conn(0x7f95d4096e10 0x7f95d409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 -- 172.21.15.153:0/3517539425 >> 172.21.15.153:0/3517539425 conn(0x7f95d40672b0 msgr2=0x7f95d40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 -- 172.21.15.153:0/3517539425 shutdown_connections 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.734+0000 7f95da19d640 1 -- 172.21.15.153:0/3517539425 wait complete. 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.735+0000 7f95da19d640 10 monclient: build_initial_monmap 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.735+0000 7f95da19d640 10 monclient: monmap: 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:20.745 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.735+0000 7f95da19d640 1 Processor -- start 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.735+0000 7f95da19d640 1 -- start start 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.735+0000 7f95da19d640 10 monclient: init 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:20.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95d4096e10 0x7f95d409b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): picked mon.noname-c con 0x7f95d4096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d411d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): picked mon.noname-a con 0x7f95d40978f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d409aaa0 0x7f95d411d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): picked mon.noname-b con 0x7f95d409aaa0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d411e530 con 0x7f95d40978f0 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4120730 con 0x7f95d409aaa0 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d406f5a0 con 0x7f95d4096e10 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95da19d640 10 monclient(hunting): authenticate will time out at 2018.375187s 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95d4096e10 0x7f95d409b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d411d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95d4096e10 0x7f95d409b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41368/0 (socket says 172.21.15.153:41368) 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 1 -- 172.21.15.153:0/3159317306 learned_addr learned my addr 172.21.15.153:0/3159317306 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): get_auth_request con 0x7f95d40978f0 auth_method 0 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): get_auth_request con 0x7f95d4096e10 auth_method 0 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d3fff640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d409aaa0 0x7f95d411d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d3fff640 10 monclient(hunting): get_auth_request con 0x7f95d409aaa0 auth_method 0 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient(hunting): handle_auth_done global_id 254214 payload 824 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95d4096e10 msgr2=0x7f95d409b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95d4096e10 0x7f95d409b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d409aaa0 msgr2=0x7f95d411d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d409aaa0 0x7f95d411d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 monclient: found mon.noname-a 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.736+0000 7f95d2ffd640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc009ae0 con 0x7f95d40978f0 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95d2ffd640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 msgr2=0x7f95d411d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95d2ffd640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d411d470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c4e0 tx=0x7f95bc007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0117a0 con 0x7f95d40978f0 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: dump: 2024-09-15T03:50:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc011920 con 0x7f95d40978f0 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95da19d640 5 monclient: authenticate success, global_id 254214 2024-09-15T03:50:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95da19d640 10 monclient: _renew_subs 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc011ce0 con 0x7f95d40978f0 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95da19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d4135050 con 0x7f95d40978f0 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95d1ffb640 4 set_mon_vals no callback set 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.738+0000 7f95cb7fe640 10 monclient: dump: 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 10 monclient: _renew_subs 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d409c440 con 0x7f95d40978f0 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f95bc012630 con 0x7f95d40978f0 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 0x7f95d411def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4065f20 con 0x7f95d411eb80 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 0x7f95d411def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient: get_auth_request con 0x7f95d411eb80 auth_method 0 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.739+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.740+0000 7f95cb7fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95b4038740 0x7f95b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.740+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f95bc04c230 con 0x7f95d40978f0 2024-09-15T03:50:20.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.740+0000 7f95d37fe640 10 monclient(hunting): handle_auth_done global_id 254217 payload 824 2024-09-15T03:50:20.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.740+0000 7f95d37fe640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95d404bf60 con 0x7f95d411eb80 2024-09-15T03:50:20.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.740+0000 7f95d3fff640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95b4038740 0x7f95b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.740+0000 7f95d3fff640 10 monclient: get_auth_request con 0x7f95b4038740 auth_method 0 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95d37fe640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 msgr2=0x7f95d411def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95d37fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 0x7f95d411def0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f95c400e830 tx=0x7f95c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c400ccd0 con 0x7f95d411eb80 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 10 monclient: dump: 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.758 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f95c400ce30 con 0x7f95d411eb80 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 msgr2=0x7f95d411def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 0x7f95d411def0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f95c400e830 tx=0x7f95c40091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 msgr2=0x7f95d411def0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95cb7fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d411eb80 0x7f95d411def0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.742+0000 7f95d3fff640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95b4038740 0x7f95b403abf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f95d409aff0 tx=0x7f95cc036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95da19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95da19d640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 0x7f95d4117880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95da19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d411fee0 con 0x7f95d4117480 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 0x7f95d4117880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient: get_auth_request con 0x7f95d4117480 auth_method 0 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 10 monclient(hunting): handle_auth_done global_id 254220 payload 824 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.748+0000 7f95d37fe640 1 -- 172.21.15.153:0/3159317306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95d40638d0 con 0x7f95d4117480 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95d37fe640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 msgr2=0x7f95d4117880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95d37fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 0x7f95d4117880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95c4017000 tx=0x7f95c4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c401cd00 con 0x7f95d4117480 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 10 monclient: dump: 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.761 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f95c400cac0 con 0x7f95d4117480 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 msgr2=0x7f95d4117880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 0x7f95d4117880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95c4017000 tx=0x7f95c4015000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 msgr2=0x7f95d4117880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:20.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95cb7fe640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d4117480 0x7f95d4117880 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95c4017000 tx=0x7f95c4015000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 226, 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-15T03:50:20.763 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.765 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:50:20.766 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:50:20.769 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:20.772 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:50:20.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:50:20.774 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95b4038740 msgr2=0x7f95b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f95b4038740 0x7f95b403abf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f95d409aff0 tx=0x7f95cc036000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 10 monclient: shutdown 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 msgr2=0x7f95d411d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 --2- 172.21.15.153:0/3159317306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95d40978f0 0x7f95d411d470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c4e0 tx=0x7f95bc007e10 comp rx=0 tx=0).stop 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95d3fff640 1 -- 172.21.15.153:0/3159317306 reap_dead start 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95d3fff640 1 -- 172.21.15.153:0/3159317306 reap_dead start 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 shutdown_connections 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 >> 172.21.15.153:0/3159317306 conn(0x7f95d40672b0 msgr2=0x7f95d4069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 shutdown_connections 2024-09-15T03:50:20.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.751+0000 7f95da19d640 1 -- 172.21.15.153:0/3159317306 wait complete. 2024-09-15T03:50:20.777 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:50:20.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:50:20.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.871+0000 7f513391d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.871+0000 7f513391d640 10 monclient: build_initial_monmap 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.871+0000 7f513391d640 10 monclient: monmap: 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:20.872039+0000 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:20.872039+0000 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:20.877 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:20.877 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:20.877 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:20.877 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:20.877 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.871+0000 7f513391d640 1 Processor -- start 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 1 -- start start 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient: init 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f512c1318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f512c09ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 0x7f512c13a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient(hunting): picked mon.noname-c con 0x7f512c137c30 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:20.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c131cc0 con 0x7f512c1318f0 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c09f530 con 0x7f512c09ea50 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c09fbf0 con 0x7f512c137c30 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f513391d640 10 monclient(hunting): authenticate will time out at 2018.511581s 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f51316b9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f512c1318f0 0x7f512c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40226/0 (socket says 172.21.15.153:40226) 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f51316b9640 1 -- 172.21.15.153:0/4007484359 learned_addr learned my addr 172.21.15.153:0/4007484359 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f5131eba640 1 --2- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 0x7f512c13a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f5131eba640 10 monclient(hunting): get_auth_request con 0x7f512c137c30 auth_method 0 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f5131eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f5131eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.872+0000 7f5131eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.874+0000 7f51316b9640 1 -- 172.21.15.153:0/4007484359 >> v1:172.21.15.153:6789/0 conn(0x7f512c1318f0 legacy=0x7f512c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.874+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847185837 0 0) 0x7f512c131cc0 con 0x7f512c1318f0 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.874+0000 7f5123fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.874+0000 7f5123fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.874+0000 7f5123fff640 10 monclient(hunting): my global_id is 254223 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.874+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51180034e0 con 0x7f512c1318f0 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 1 -- 172.21.15.153:0/4007484359 >> v1:172.21.15.181:6789/0 conn(0x7f512c09ea50 legacy=0x7f512c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 1 -- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 msgr2=0x7f512c13a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 1 -- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 msgr2=0x7f512c13a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:20.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 1 --2- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 0x7f512c13a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 1 --2- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 0x7f512c13a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616426179 0 0) 0x7f51180034e0 con 0x7f512c1318f0 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 msgr2=0x7f512c13a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 --2- 172.21.15.153:0/4007484359 >> v2:172.21.15.181:3300/0 conn(0x7f512c137c30 0x7f512c13a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 >> v1:172.21.15.181:6789/0 conn(0x7f512c09ea50 legacy=0x7f512c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c09fe30 con 0x7f512c1318f0 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f513391d640 5 monclient: authenticate success, global_id 254223 2024-09-15T03:50:20.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f513391d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7f5128001720 con 0x7f512c1318f0 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: dump: 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.894 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 >> v1:172.21.15.153:6789/0 conn(0x7f512c1318f0 legacy=0x7f512c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 0x7f5118003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): picked mon.c con 0x7f5118003b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5118004780 0x7f5118004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): picked mon.a con 0x7f5118004780 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51180053b0 0x7f511800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): picked mon.b con 0x7f51180053b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c09fbf0 con 0x7f5118004780 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511800c380 con 0x7f51180053b0 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511800c540 con 0x7f5118003b20 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5123fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f513391d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 0x7f5118003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): get_auth_request con 0x7f5118003b20 auth_method 0 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5118004780 0x7f5118004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): get_auth_request con 0x7f5118004780 auth_method 0 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51180053b0 0x7f511800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): get_auth_request con 0x7f51180053b0 auth_method 0 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient(hunting): handle_auth_done global_id 254223 payload 293 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 -- 172.21.15.153:0/4007484359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51180053b0 msgr2=0x7f511800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51180053b0 0x7f511800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 -- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5118004780 msgr2=0x7f5118004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5118004780 0x7f5118004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 monclient: found mon.c 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 1 -- 172.21.15.153:0/4007484359 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180034e0 con 0x7f5118003b20 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5131eba640 1 -- 172.21.15.153:0/4007484359 reap_dead start 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f51316b9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.875+0000 7f5130eb8640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5118004780 0x7f5118004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f51316b9640 1 -- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 msgr2=0x7f5118003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f51316b9640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 0x7f5118003f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5128001490 tx=0x7f5128007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e30957616883ef8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128005890 con 0x7f5118003b20 2024-09-15T03:50:20.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: dump: 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.900 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f513391d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5128005a10 con 0x7f5118003b20 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f513391d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f513391d640 4 set_mon_vals no callback set 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 1 -- 172.21.15.153:0/4007484359 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128005dd0 con 0x7f5118003b20 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.878+0000 7f5123fff640 10 monclient: dump: 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.901 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 10 monclient: shutdown 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 -- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 msgr2=0x7f5118003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 0x7f5118003f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5128001490 tx=0x7f5128007a50 comp rx=0 tx=0).stop 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 -- 172.21.15.153:0/4007484359 shutdown_connections 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 --2- 172.21.15.153:0/4007484359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5118003b20 0x7f5118003f30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 -- 172.21.15.153:0/4007484359 >> 172.21.15.153:0/4007484359 conn(0x7f512c067110 msgr2=0x7f512c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 -- 172.21.15.153:0/4007484359 shutdown_connections 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.879+0000 7f513391d640 1 -- 172.21.15.153:0/4007484359 wait complete. 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.880+0000 7f513391d640 10 monclient: build_initial_monmap 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.880+0000 7f513391d640 10 monclient: monmap: 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.880+0000 7f513391d640 1 Processor -- start 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.880+0000 7f513391d640 1 -- start start 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.880+0000 7f513391d640 10 monclient: init 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5118003b20 0x7f512c141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): picked mon.noname-b con 0x7f5118003b20 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f512c141bc0 0x7f512c0989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): picked mon.noname-a con 0x7f512c141bc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c098f00 0x7f512c099370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): picked mon.noname-c con 0x7f512c098f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c09bf10 con 0x7f512c141bc0 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c09deb0 con 0x7f5118003b20 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c09a490 con 0x7f512c098f00 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f513391d640 10 monclient(hunting): authenticate will time out at 2018.520130s 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f512c141bc0 0x7f512c0989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c098f00 0x7f512c099370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f512c141bc0 0x7f512c0989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33698/0 (socket says 172.21.15.153:33698) 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): get_auth_request con 0x7f512c098f00 auth_method 0 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 1 -- 172.21.15.153:0/684201352 learned_addr learned my addr 172.21.15.153:0/684201352 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): get_auth_request con 0x7f512c141bc0 auth_method 0 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5118003b20 0x7f512c141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): get_auth_request con 0x7f5118003b20 auth_method 0 2024-09-15T03:50:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient(hunting): handle_auth_done global_id 244328 payload 824 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5118003b20 msgr2=0x7f512c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5118003b20 0x7f512c141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f512c141bc0 msgr2=0x7f512c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f512c141bc0 0x7f512c0989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient: _finish_hunting 0 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 monclient: found mon.noname-c 2024-09-15T03:50:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5131eba640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c11bb00 con 0x7f512c098f00 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.881+0000 7f5130eb8640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f512c141bc0 0x7f512c0989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f5131eba640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c098f00 msgr2=0x7f512c099370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f5131eba640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c098f00 0x7f512c099370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f512c06a540 tx=0x7f5124009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124007040 con 0x7f512c098f00 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: dump: 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: _finish_auth 0 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 5 monclient: authenticate success, global_id 244328 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512400f8e0 con 0x7f512c098f00 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 10 monclient: _renew_subs 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512c11bde0 con 0x7f512c098f00 2024-09-15T03:50:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512400fc80 con 0x7f512c098f00 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51217fa640 10 monclient: dump: 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.911 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f51237fe640 4 set_mon_vals no callback set 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 10 monclient: _renew_subs 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.883+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512c131020 con 0x7f512c098f00 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5124014600 con 0x7f512c098f00 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f513391d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f513391d640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 0x7f512c0998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f513391d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c065e00 con 0x7f512c02aa40 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 0x7f512c0998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient: get_auth_request con 0x7f512c02aa40 auth_method 0 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51217fa640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5110038740 0x7f511003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.884+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.885+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f512404c820 con 0x7f512c098f00 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.885+0000 7f51316b9640 10 monclient(hunting): handle_auth_done global_id 244331 payload 824 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.885+0000 7f51316b9640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f512c02ae10 con 0x7f512c02aa40 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.885+0000 7f5130eb8640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5110038740 0x7f511003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.885+0000 7f5130eb8640 10 monclient: get_auth_request con 0x7f5110038740 auth_method 0 2024-09-15T03:50:20.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51316b9640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 msgr2=0x7f512c0998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51316b9640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 0x7f512c0998b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5128007e50 tx=0x7f5128034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128041070 con 0x7f512c02aa40 2024-09-15T03:50:20.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 10 monclient: dump: 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.915 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5128007510 con 0x7f512c02aa40 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 msgr2=0x7f512c0998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 0x7f512c0998b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5128007e50 tx=0x7f5128034000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 msgr2=0x7f512c0998b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:20.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.887+0000 7f51217fa640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c02aa40 0x7f512c0998b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:20.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.888+0000 7f5130eb8640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5110038740 0x7f511003abf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f511c001eb0 tx=0x7f511c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f513391d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:50:20.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f513391d640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 0x7f512c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f513391d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c099df0 con 0x7f512c05e310 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 0x7f512c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient: get_auth_request con 0x7f512c05e310 auth_method 0 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 10 monclient(hunting): handle_auth_done global_id 244334 payload 824 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.893+0000 7f51316b9640 1 -- 172.21.15.153:0/684201352 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f512c11c020 con 0x7f512c05e310 2024-09-15T03:50:20.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51316b9640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 msgr2=0x7f512c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51316b9640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 0x7f512c05e710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f512802ae50 tx=0x7f5128041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128039e40 con 0x7f512c05e310 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 10 monclient: dump: 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:20.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f5128007300 con 0x7f512c05e310 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 msgr2=0x7f512c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 0x7f512c05e710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f512802ae50 tx=0x7f5128041000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 msgr2=0x7f512c05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f51217fa640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c05e310 0x7f512c05e710 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f512802ae50 tx=0x7f5128041000 comp rx=0 tx=0).stop 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:20.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 226, 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:20.923 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:20.924 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:50:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:20.926 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:50:20.927 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.928 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:20.929 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.930 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:50:20.931 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:20.932 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:20.933 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.933 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.933 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:50:20.933 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:50:20.933 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-15T03:50:20.934 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:50:20.935 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:20.936 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:20.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:20.936 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:50:20.936 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5110038740 msgr2=0x7f511003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5110038740 0x7f511003abf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f511c001eb0 tx=0x7f511c009040 comp rx=0 tx=0).stop 2024-09-15T03:50:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 10 monclient: shutdown 2024-09-15T03:50:20.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c098f00 msgr2=0x7f512c099370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 --2- 172.21.15.153:0/684201352 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f512c098f00 0x7f512c099370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f512c06a540 tx=0x7f5124009720 comp rx=0 tx=0).stop 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f5131eba640 1 -- 172.21.15.153:0/684201352 reap_dead start 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f5131eba640 1 -- 172.21.15.153:0/684201352 reap_dead start 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 shutdown_connections 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 >> 172.21.15.153:0/684201352 conn(0x7f512c067110 msgr2=0x7f512c0a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 shutdown_connections 2024-09-15T03:50:20.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:20.896+0000 7f513391d640 1 -- 172.21.15.153:0/684201352 wait complete. 2024-09-15T03:50:20.938 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:50:20.938 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:50:20.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.032+0000 7fc02b59d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.032+0000 7fc02b59d640 10 monclient: build_initial_monmap 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.032+0000 7fc02b59d640 10 monclient: monmap: 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:21.033045+0000 2024-09-15T03:50:21.033 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:21.033045+0000 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:21.034 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.032+0000 7fc02b59d640 1 Processor -- start 2024-09-15T03:50:21.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 1 -- start start 2024-09-15T03:50:21.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient: init 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc0240981c0 0x7fc024098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): picked mon.noname-a con 0x7fc0240981c0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fc024131da0 0x7fc024067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): picked mon.noname-c con 0x7fc024131da0 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc024132170 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc024098ad0 con 0x7fc024132170 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc024131bd0 con 0x7fc0240981c0 2024-09-15T03:50:21.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02409ca60 con 0x7fc024131da0 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc02b59d640 10 monclient(hunting): authenticate will time out at 2018.672583s 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc0240981c0 0x7fc024098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc0240981c0 0x7fc024098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33712/0 (socket says 172.21.15.153:33712) 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 -- 172.21.15.153:0/741187357 learned_addr learned my addr 172.21.15.153:0/741187357 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): get_auth_request con 0x7fc0240981c0 auth_method 0 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc028b38640 1 --2- 172.21.15.153:0/741187357 >> v2:172.21.15.181:3300/0 conn(0x7fc024131da0 0x7fc024067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc028b38640 10 monclient(hunting): get_auth_request con 0x7fc024131da0 auth_method 0 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc028b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc028b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc028b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient(hunting): handle_auth_done global_id 254229 payload 274 2024-09-15T03:50:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 -- 172.21.15.153:0/741187357 >> v2:172.21.15.181:3300/0 conn(0x7fc024131da0 msgr2=0x7fc024067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 --2- 172.21.15.153:0/741187357 >> v2:172.21.15.181:3300/0 conn(0x7fc024131da0 0x7fc024067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 -- 172.21.15.153:0/741187357 >> v1:172.21.15.181:6789/0 conn(0x7fc024132170 legacy=0x7fc024067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 10 monclient: _finish_hunting 0 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.033+0000 7fc029339640 1 monclient: found mon.noname-a 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.034+0000 7fc029339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.034+0000 7fc029339640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:21.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.034+0000 7fc029339640 1 -- 172.21.15.153:0/741187357 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02409cca0 con 0x7fc0240981c0 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc029339640 1 -- 172.21.15.153:0/741187357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0240981c0 msgr2=0x7fc024098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc029339640 1 --2- 172.21.15.153:0/741187357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0240981c0 0x7fc024098590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc024132870 tx=0x7fc0200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92531e91d4adedd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 1 -- 172.21.15.153:0/741187357 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020032440 con 0x7fc0240981c0 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: dump: 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.048 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: _finish_auth 0 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 1 -- 172.21.15.153:0/741187357 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0200325a0 con 0x7fc0240981c0 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc02b59d640 5 monclient: authenticate success, global_id 254229 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc02b59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:21.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc02b59d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 1 -- 172.21.15.153:0/741187357 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020032940 con 0x7fc0240981c0 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc02b59d640 4 set_mon_vals no callback set 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.036+0000 7fc01bfff640 10 monclient: dump: 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 10 monclient: shutdown 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 -- 172.21.15.153:0/741187357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0240981c0 msgr2=0x7fc024098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 --2- 172.21.15.153:0/741187357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0240981c0 0x7fc024098590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc024132870 tx=0x7fc0200091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 -- 172.21.15.153:0/741187357 shutdown_connections 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 --2- 172.21.15.153:0/741187357 >> v2:172.21.15.181:3300/0 conn(0x7fc024131da0 0x7fc024067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 --2- 172.21.15.153:0/741187357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0240981c0 0x7fc024098590 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 -- 172.21.15.153:0/741187357 >> 172.21.15.153:0/741187357 conn(0x7fc0240a1e30 msgr2=0x7fc0240a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:21.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 -- 172.21.15.153:0/741187357 shutdown_connections 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 1 -- 172.21.15.153:0/741187357 wait complete. 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 10 monclient: build_initial_monmap 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.037+0000 7fc02b59d640 10 monclient: monmap: 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:21.052 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 Processor -- start 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 -- start start 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient: init 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0240981c0 0x7fc02411db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): picked mon.noname-b con 0x7fc0240981c0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024131da0 0x7fc02411e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): picked mon.noname-a con 0x7fc024131da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc024132170 0x7fc02411e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): picked mon.noname-c con 0x7fc024132170 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0241202f0 con 0x7fc024131da0 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc024120120 con 0x7fc0240981c0 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc024098cb0 con 0x7fc024132170 2024-09-15T03:50:21.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.038+0000 7fc02b59d640 10 monclient(hunting): authenticate will time out at 2018.677947s 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024131da0 0x7fc02411e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024131da0 0x7fc02411e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33724/0 (socket says 172.21.15.153:33724) 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc024132170 0x7fc02411e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 1 -- 172.21.15.153:0/914944326 learned_addr learned my addr 172.21.15.153:0/914944326 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): get_auth_request con 0x7fc024131da0 auth_method 0 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): get_auth_request con 0x7fc024132170 auth_method 0 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029339640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0240981c0 0x7fc02411db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029339640 10 monclient(hunting): get_auth_request con 0x7fc0240981c0 auth_method 0 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient(hunting): handle_auth_done global_id 244337 payload 824 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0240981c0 msgr2=0x7fc02411db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0240981c0 0x7fc02411db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024131da0 msgr2=0x7fc02411e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024131da0 0x7fc02411e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient: _finish_hunting 0 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 monclient: found mon.noname-c 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc029b3a640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc020009450 con 0x7fc024132170 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.039+0000 7fc028b38640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc024131da0 0x7fc02411e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc029b3a640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc024132170 msgr2=0x7fc02411e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc029b3a640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc024132170 0x7fc02411e5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00c7e0 tx=0x7fc01c0025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c00a850 con 0x7fc024132170 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:21.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: dump: 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: _finish_auth 0 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 5 monclient: authenticate success, global_id 244337 2024-09-15T03:50:21.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01c00a9d0 con 0x7fc024132170 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 10 monclient: _renew_subs 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0240a0850 con 0x7fc024132170 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c00ad90 con 0x7fc024132170 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc0197fa640 10 monclient: dump: 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.060 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc01b7fe640 4 set_mon_vals no callback set 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 10 monclient: _renew_subs 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.041+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc024121290 con 0x7fc024132170 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fc01c012630 con 0x7fc024132170 2024-09-15T03:50:21.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc02b59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc02b59d640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 0x7fc02411a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc024065f90 con 0x7fc02411eb00 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc0197fa640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc008038770 0x7fc00803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.042+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc01c04c870 con 0x7fc024132170 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 0x7fc02411a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc028b38640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc008038770 0x7fc00803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient: get_auth_request con 0x7fc02411eb00 auth_method 0 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc028b38640 10 monclient: get_auth_request con 0x7fc008038770 auth_method 0 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 10 monclient(hunting): handle_auth_done global_id 264109 payload 824 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.043+0000 7fc029339640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc02404bf60 con 0x7fc02411eb00 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc028b38640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc008038770 0x7fc00803ac20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc014010370 tx=0x7fc014008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc029339640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 msgr2=0x7fc02411a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc029339640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 0x7fc02411a4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc020034000 tx=0x7fc020032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200319a0 con 0x7fc02411eb00 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc0197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc0197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:50:21.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.045+0000 7fc0197fa640 10 monclient: dump: 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc020031b00 con 0x7fc02411eb00 2024-09-15T03:50:21.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:21.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:21.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 msgr2=0x7fc02411a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 0x7fc02411a4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc020034000 tx=0x7fc020032000 comp rx=0 tx=0).stop 2024-09-15T03:50:21.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 msgr2=0x7fc02411a4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:21.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.046+0000 7fc0197fa640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411eb00 0x7fc02411a4a0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc020034000 tx=0x7fc020032000 comp rx=0 tx=0).stop 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.051+0000 7fc02b59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.051+0000 7fc02b59d640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 0x7fc02411ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.051+0000 7fc02b59d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.051+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc024098ee0 con 0x7fc02411a9e0 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 0x7fc02411ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient: get_auth_request con 0x7fc02411a9e0 auth_method 0 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 10 monclient(hunting): handle_auth_done global_id 264112 payload 824 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.052+0000 7fc029339640 1 -- 172.21.15.153:0/914944326 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc02411eed0 con 0x7fc02411a9e0 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.054+0000 7fc029339640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 msgr2=0x7fc02411ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.054+0000 7fc029339640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 0x7fc02411ade0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc024068680 tx=0x7fc02003c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02003c470 con 0x7fc02411a9e0 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 10 monclient: dump: 2024-09-15T03:50:21.068 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fc02003c5d0 con 0x7fc02411a9e0 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:21.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 msgr2=0x7fc02411ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 0x7fc02411ade0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc024068680 tx=0x7fc02003c000 comp rx=0 tx=0).stop 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 msgr2=0x7fc02411ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc0197fa640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc02411a9e0 0x7fc02411ade0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc024068680 tx=0x7fc02003c000 comp rx=0 tx=0).stop 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 226, 2024-09-15T03:50:21.070 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2449958755906961412", 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:21.071 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:21.072 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "7dd05962-040e-4682-bebc-5f3c2ef6922e", 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:50:21.073 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-15T03:27:58.706914Z", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 17, 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-15T03:50:21.074 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy" 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.075 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:21.076 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:21.077 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:21.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:3300", 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:21.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789", 2024-09-15T03:50:21.084 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:21.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:21.084 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:21.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.084 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6789/0", 2024-09-15T03:50:21.084 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6789/0", 2024-09-15T03:50:21.085 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:21.085 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:21.085 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:21.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.086 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-15T03:50:21.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-15T03:50:21.086 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-15T03:50:21.086 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-15T03:50:21.086 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.087 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-15T03:50:21.087 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-15T03:50:21.087 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:21.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.087 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-15T03:50:21.088 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-15T03:50:21.088 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-15T03:50:21.088 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:21.088 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:21.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.088 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-15T03:50:21.089 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-15T03:50:21.089 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-15T03:50:21.089 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-15T03:50:21.089 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-15T03:50:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-15T03:50:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-15T03:50:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.091 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:21.091 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:21.091 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:50:21.091 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:21.092 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-15T03:50:21.092 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-15T03:50:21.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-15T03:50:21.092 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:50:21.092 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "luminous", 2024-09-15T03:50:21.093 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-15T03:50:21.093 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-15T03:50:21.093 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-15T03:50:21.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-15T03:50:21.093 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-15T03:50:21.093 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:21.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc008038770 msgr2=0x7fc00803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc02b59d640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fc008038770 0x7fc00803ac20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc014010370 tx=0x7fc014008040 comp rx=0 tx=0).stop 2024-09-15T03:50:21.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc02b59d640 10 monclient: shutdown 2024-09-15T03:50:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc02b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc024132170 msgr2=0x7fc02411e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc02b59d640 1 --2- 172.21.15.153:0/914944326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc024132170 0x7fc02411e5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00c7e0 tx=0x7fc01c0025b0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc029b3a640 1 -- 172.21.15.153:0/914944326 reap_dead start 2024-09-15T03:50:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.055+0000 7fc029b3a640 1 -- 172.21.15.153:0/914944326 reap_dead start 2024-09-15T03:50:21.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.056+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 shutdown_connections 2024-09-15T03:50:21.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.056+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 >> 172.21.15.153:0/914944326 conn(0x7fc0240a1e30 msgr2=0x7fc02409bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:21.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.056+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 shutdown_connections 2024-09-15T03:50:21.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.056+0000 7fc02b59d640 1 -- 172.21.15.153:0/914944326 wait complete. 2024-09-15T03:50:21.098 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:50:21.099 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:50:21.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.191+0000 7fde4179d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.191+0000 7fde4179d640 10 monclient: build_initial_monmap 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.191+0000 7fde4179d640 10 monclient: monmap: 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:21.192697+0000 2024-09-15T03:50:21.193 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:21.192697+0000 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:21.194 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:21.195 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.192+0000 7fde4179d640 1 Processor -- start 2024-09-15T03:50:21.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.192+0000 7fde4179d640 1 -- start start 2024-09-15T03:50:21.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.192+0000 7fde4179d640 10 monclient: init 2024-09-15T03:50:21.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:21.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-15T03:50:21.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 0x7fde3c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): picked mon.noname-b con 0x7fde3c131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fde3c09e910 0x7fde3c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): picked mon.noname-c con 0x7fde3c09e910 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde3c09f220 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3c09faf0 con 0x7fde3c09f220 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3c09fd30 con 0x7fde3c09e910 2024-09-15T03:50:21.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c0a1110 con 0x7fde3c131ab0 2024-09-15T03:50:21.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:21.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde4179d640 10 monclient(hunting): authenticate will time out at 2018.832111s 2024-09-15T03:50:21.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 0x7fde3c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 0x7fde3c131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41406/0 (socket says 172.21.15.153:41406) 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 -- 172.21.15.153:0/2209260214 learned_addr learned my addr 172.21.15.153:0/2209260214 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): get_auth_request con 0x7fde3c131ab0 auth_method 0 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3a7fc640 1 --2- 172.21.15.153:0/2209260214 >> v2:172.21.15.181:3300/0 conn(0x7fde3c09e910 0x7fde3c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3a7fc640 10 monclient(hunting): get_auth_request con 0x7fde3c09e910 auth_method 0 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient(hunting): handle_auth_done global_id 244340 payload 274 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 -- 172.21.15.153:0/2209260214 >> v2:172.21.15.181:3300/0 conn(0x7fde3c09e910 msgr2=0x7fde3c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 --2- 172.21.15.153:0/2209260214 >> v2:172.21.15.181:3300/0 conn(0x7fde3c09e910 0x7fde3c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 -- 172.21.15.153:0/2209260214 >> v1:172.21.15.181:6789/0 conn(0x7fde3c09f220 legacy=0x7fde3c068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 monclient: found mon.noname-b 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.193+0000 7fde3affd640 1 -- 172.21.15.153:0/2209260214 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c0a13f0 con 0x7fde3c131ab0 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde3affd640 1 -- 172.21.15.153:0/2209260214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 msgr2=0x7fde3c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde3affd640 1 --2- 172.21.15.153:0/2209260214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 0x7fde3c131e80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fde34009f20 tx=0x7fde34009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16e181028233eb58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 1 -- 172.21.15.153:0/2209260214 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34007610 con 0x7fde3c131ab0 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: dump: 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.209 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: _finish_auth 0 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:21.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 1 -- 172.21.15.153:0/2209260214 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3402d050 con 0x7fde3c131ab0 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 5 monclient: authenticate success, global_id 244340 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 1 -- 172.21.15.153:0/2209260214 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde340079e0 con 0x7fde3c131ab0 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 4 set_mon_vals no callback set 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde39ffb640 10 monclient: dump: 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 10 monclient: shutdown 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 1 -- 172.21.15.153:0/2209260214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 msgr2=0x7fde3c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.196+0000 7fde4179d640 1 --2- 172.21.15.153:0/2209260214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 0x7fde3c131e80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fde34009f20 tx=0x7fde34009840 comp rx=0 tx=0).stop 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 -- 172.21.15.153:0/2209260214 shutdown_connections 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 --2- 172.21.15.153:0/2209260214 >> v2:172.21.15.181:3300/0 conn(0x7fde3c09e910 0x7fde3c09ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 --2- 172.21.15.153:0/2209260214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c131ab0 0x7fde3c131e80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 -- 172.21.15.153:0/2209260214 >> 172.21.15.153:0/2209260214 conn(0x7fde3c067d30 msgr2=0x7fde3c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 -- 172.21.15.153:0/2209260214 shutdown_connections 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 -- 172.21.15.153:0/2209260214 wait complete. 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 10 monclient: build_initial_monmap 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 10 monclient: monmap: 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.197+0000 7fde4179d640 1 Processor -- start 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 -- start start 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient: init 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 0x7fde3c11de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): picked mon.noname-c con 0x7fde3c09e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde3c09f220 0x7fde3c11e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): picked mon.noname-b con 0x7fde3c09f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde3c131ab0 0x7fde3c11e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): picked mon.noname-a con 0x7fde3c131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c120540 con 0x7fde3c131ab0 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c09bde0 con 0x7fde3c09f220 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c1219f0 con 0x7fde3c09e910 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde4179d640 10 monclient(hunting): authenticate will time out at 2018.837477s 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 0x7fde3c11de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 0x7fde3c11de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41416/0 (socket says 172.21.15.153:41416) 2024-09-15T03:50:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 -- 172.21.15.153:0/501788651 learned_addr learned my addr 172.21.15.153:0/501788651 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): get_auth_request con 0x7fde3c09e910 auth_method 0 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde3c131ab0 0x7fde3c11e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): get_auth_request con 0x7fde3c131ab0 auth_method 0 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde3c09f220 0x7fde3c11e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): get_auth_request con 0x7fde3c09f220 auth_method 0 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient(hunting): handle_auth_done global_id 244343 payload 824 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 -- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde3c09f220 msgr2=0x7fde3c11e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde3c09f220 0x7fde3c11e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 -- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde3c131ab0 msgr2=0x7fde3c11e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde3c131ab0 0x7fde3c11e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 monclient: found mon.noname-c 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3affd640 1 -- 172.21.15.153:0/501788651 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34009ae0 con 0x7fde3c09e910 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.198+0000 7fde3b7fe640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde3c131ab0 0x7fde3c11e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde3affd640 1 -- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 msgr2=0x7fde3c11de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde3affd640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 0x7fde3c11de20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fde34009f20 tx=0x7fde34036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34007e90 con 0x7fde3c09e910 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: dump: 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: _finish_auth 0 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 5 monclient: authenticate success, global_id 244343 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3403c840 con 0x7fde3c09e910 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 10 monclient: _renew_subs 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3c121710 con 0x7fde3c09e910 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3403cc20 con 0x7fde3c09e910 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:21.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde32ffd640 10 monclient: dump: 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde397fa640 4 set_mon_vals no callback set 2024-09-15T03:50:21.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 10 monclient: _renew_subs 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.201+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3c11f970 con 0x7fde3c09e910 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fde34031400 con 0x7fde3c09e910 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde4179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde4179d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde4179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3c121b30 con 0x7fde3c09e910 2024-09-15T03:50:21.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde32ffd640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde1c038740 0x7fde1c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:21.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fde3406d380 con 0x7fde3c09e910 2024-09-15T03:50:21.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde3a7fc640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde1c038740 0x7fde1c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:21.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.202+0000 7fde3a7fc640 10 monclient: get_auth_request con 0x7fde1c038740 auth_method 0 2024-09-15T03:50:21.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.205+0000 7fde3a7fc640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde1c038740 0x7fde1c03abf0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fde24007ba0 tx=0x7fde24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.206+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fde3c121b30 con 0x7fde3c09e910 2024-09-15T03:50:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.206+0000 7fde32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.206+0000 7fde32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:21.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.424+0000 7fde4179d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:50:21.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.424+0000 7fde4179d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:50:21.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.424+0000 7fde4179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:21.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.424+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fde3c121b30 con 0x7fde3c09e910 2024-09-15T03:50:21.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.425+0000 7fde32ffd640 1 -- 172.21.15.153:0/501788651 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fde3c121b30 con 0x7fde3c09e910 2024-09-15T03:50:21.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.425+0000 7fde32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:50:21.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.425+0000 7fde32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:21.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.425+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.425+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=8,logm=54} crc {auth=2771215803,config=1228700967,health=1649607816,kv=3235807718,logm=499749010}) 2024-09-15T03:50:21.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.425+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=7,logm=54} crc {auth=1588356158,config=3140514009,health=813996049,kv=1577269308,logm=345348805}) 2024-09-15T03:50:21.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.425+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.425+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=8,logm=54} crc {auth=2771215803,config=1228700967,health=1649607816,kv=3235807718,logm=499749010}) 2024-09-15T03:50:21.427 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.425+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=3094052539,config=3140514009,health=2615813559,kv=3235807718,logm=847518061}) 2024-09-15T03:50:21.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.427+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.427+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259142753}) 2024-09-15T03:50:21.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.427+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369055653}) 2024-09-15T03:50:21.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.427+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.427+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259142753}) 2024-09-15T03:50:21.428 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.427+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3487496266}) 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde1c038740 msgr2=0x7fde1c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde1c038740 0x7fde1c03abf0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fde24007ba0 tx=0x7fde24008040 comp rx=0 tx=0).stop 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 10 monclient: shutdown 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 msgr2=0x7fde3c11de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 0x7fde3c11de20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fde34009f20 tx=0x7fde34036000 comp rx=0 tx=0).stop 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 shutdown_connections 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fde1c038740 0x7fde1c03abf0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde3c131ab0 0x7fde3c11e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde3c09f220 0x7fde3c11e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 --2- 172.21.15.153:0/501788651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde3c09e910 0x7fde3c11de20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:21.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 >> 172.21.15.153:0/501788651 conn(0x7fde3c067d30 msgr2=0x7fde3c09e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:21.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 shutdown_connections 2024-09-15T03:50:21.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:21.427+0000 7fde4179d640 1 -- 172.21.15.153:0/501788651 wait complete. 2024-09-15T03:50:21.432 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.428+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.428+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835675229}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.428+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284816362}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.428+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.428+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835675229}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.428+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221995051}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.430+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.430+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339316065}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.430+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057364523}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.430+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.430+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339316065}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.430+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448037338}) 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.431+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.433 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.431+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073178908}) 2024-09-15T03:50:21.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.431+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3479201912}) 2024-09-15T03:50:21.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.431+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.431+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073178908}) 2024-09-15T03:50:21.434 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.431+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3340722159}) 2024-09-15T03:50:21.437 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.432+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.432+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128209798}) 2024-09-15T03:50:21.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.432+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239826323}) 2024-09-15T03:50:21.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.432+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.432+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128209798}) 2024-09-15T03:50:21.438 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.432+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236478777}) 2024-09-15T03:50:21.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.433+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.433+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575187446}) 2024-09-15T03:50:21.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.433+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967507996}) 2024-09-15T03:50:21.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.433+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.439 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.433+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575187446}) 2024-09-15T03:50:21.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.433+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417332158}) 2024-09-15T03:50:21.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.434+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.434+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213390437}) 2024-09-15T03:50:21.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.434+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4018774990}) 2024-09-15T03:50:21.440 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.434+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.441 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.434+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213390437}) 2024-09-15T03:50:21.441 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.434+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948199563}) 2024-09-15T03:50:21.441 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.435+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.435+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446911498}) 2024-09-15T03:50:21.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.435+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434151314}) 2024-09-15T03:50:21.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.435+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.442 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.435+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446911498}) 2024-09-15T03:50:21.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.435+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324090646}) 2024-09-15T03:50:21.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.436+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.436+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820507450}) 2024-09-15T03:50:21.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.436+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1392463474}) 2024-09-15T03:50:21.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.436+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.443 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.436+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820507450}) 2024-09-15T03:50:21.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.436+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392953066}) 2024-09-15T03:50:21.444 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.437+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.437+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495775332}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.437+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1368239766}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.437+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.437+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495775332}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.437+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821292258}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.438+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.438+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243892883}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.438+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996475551}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.438+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.438+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243892883}) 2024-09-15T03:50:21.445 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.438+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=259042027}) 2024-09-15T03:50:21.447 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:50:21.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.439+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.439+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553229101}) 2024-09-15T03:50:21.452 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.439+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50433156}) 2024-09-15T03:50:21.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.439+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.439+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553229101}) 2024-09-15T03:50:21.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.439+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839083464}) 2024-09-15T03:50:21.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.440+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.440+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050385571}) 2024-09-15T03:50:21.453 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.440+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610176000}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.440+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.440+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050385571}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.440+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147065384}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.441+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.441+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013703822}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.441+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527170943}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.441+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.441+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013703822}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.441+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2406835816}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.442+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.442+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106368581}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.442+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2695064717}) 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.442+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.454 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.442+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106368581}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.442+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710806019}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.443+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.443+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000459241}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.443+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966483137}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.443+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.443+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000459241}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.443+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1351537505}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.444+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.444+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800316108}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.444+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1199050092}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.444+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.444+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800316108}) 2024-09-15T03:50:21.455 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.444+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269858506}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.445+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.445+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754941048}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.445+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878527745}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.445+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.445+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754941048}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.445+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271444371}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.446+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.446+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783885076}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.446+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677025908}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.446+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.446+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783885076}) 2024-09-15T03:50:21.456 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.446+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3304392469}) 2024-09-15T03:50:21.870 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.869+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.870 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.869+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829573810}) 2024-09-15T03:50:21.870 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.869+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2180561238}) 2024-09-15T03:50:21.870 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.869+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.870 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.869+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829573810}) 2024-09-15T03:50:21.870 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.869+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814632475}) 2024-09-15T03:50:21.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.872+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.872+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4205023800}) 2024-09-15T03:50:21.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.872+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614512531}) 2024-09-15T03:50:21.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.872+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.872+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4205023800}) 2024-09-15T03:50:21.873 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.872+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578601830}) 2024-09-15T03:50:21.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.877+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.877+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035490550}) 2024-09-15T03:50:21.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.877+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953015073}) 2024-09-15T03:50:21.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.877+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:21.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.877+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035490550}) 2024-09-15T03:50:21.878 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:21.877+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3514089727}) 2024-09-15T03:50:22.024 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.023+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.024 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.023+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2641978319}) 2024-09-15T03:50:22.025 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.023+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273651130}) 2024-09-15T03:50:22.025 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.023+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.025 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.023+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2641978319}) 2024-09-15T03:50:22.025 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.023+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449534732}) 2024-09-15T03:50:22.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069544535}) 2024-09-15T03:50:22.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778830616}) 2024-09-15T03:50:22.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069544535}) 2024-09-15T03:50:22.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=433765622}) 2024-09-15T03:50:22.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92330527}) 2024-09-15T03:50:22.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=876382341}) 2024-09-15T03:50:22.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92330527}) 2024-09-15T03:50:22.036 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3504061535}) 2024-09-15T03:50:22.040 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.040 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246890668}) 2024-09-15T03:50:22.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3457037782}) 2024-09-15T03:50:22.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246890668}) 2024-09-15T03:50:22.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082805816}) 2024-09-15T03:50:22.066 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.066+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.066+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250418924}) 2024-09-15T03:50:22.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.066+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=401295466}) 2024-09-15T03:50:22.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.066+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.066+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250418924}) 2024-09-15T03:50:22.067 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.066+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608559487}) 2024-09-15T03:50:22.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.067+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.067+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967898171}) 2024-09-15T03:50:22.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.067+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135607269}) 2024-09-15T03:50:22.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.067+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.067+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967898171}) 2024-09-15T03:50:22.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.067+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353384450}) 2024-09-15T03:50:22.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.068+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.068+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103250372}) 2024-09-15T03:50:22.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.068+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2212907358}) 2024-09-15T03:50:22.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.068+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.068+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103250372}) 2024-09-15T03:50:22.069 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.068+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592887883}) 2024-09-15T03:50:22.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.069+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.069+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974918777}) 2024-09-15T03:50:22.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.069+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839068315}) 2024-09-15T03:50:22.070 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.069+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.071 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.069+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974918777}) 2024-09-15T03:50:22.071 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.069+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230010675}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.070+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.070+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089445264}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.070+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094133164}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.070+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.070+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089445264}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.070+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967474765}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.071+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.071+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221082951}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.071+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=55191548}) 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.071+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.073 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.071+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221082951}) 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.071+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556718994}) 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.072+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.072+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685665528}) 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.072+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2693570567}) 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.072+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.072+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685665528}) 2024-09-15T03:50:22.074 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.072+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179227011}) 2024-09-15T03:50:22.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.073+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.073+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701032988}) 2024-09-15T03:50:22.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.073+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872285257}) 2024-09-15T03:50:22.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.073+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.073+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701032988}) 2024-09-15T03:50:22.075 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.073+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501311270}) 2024-09-15T03:50:22.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.074+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.074+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1669788581}) 2024-09-15T03:50:22.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.074+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510370838}) 2024-09-15T03:50:22.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.074+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.074+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1669788581}) 2024-09-15T03:50:22.076 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.074+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=427403961}) 2024-09-15T03:50:22.115 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.114+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.114+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276190158}) 2024-09-15T03:50:22.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.114+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2083913914}) 2024-09-15T03:50:22.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.114+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.114+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276190158}) 2024-09-15T03:50:22.116 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.114+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3181920847}) 2024-09-15T03:50:22.219 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.218+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.219 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.218+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805404995}) 2024-09-15T03:50:22.219 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.218+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771587185}) 2024-09-15T03:50:22.219 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.218+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.220 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.218+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805404995}) 2024-09-15T03:50:22.220 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.218+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133004562}) 2024-09-15T03:50:22.365 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.366 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028891881}) 2024-09-15T03:50:22.366 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315038644}) 2024-09-15T03:50:22.366 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.366 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028891881}) 2024-09-15T03:50:22.367 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.364+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913913659}) 2024-09-15T03:50:22.448 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:50:22.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:50:22.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.542+0000 7f528b03f640 10 monclient: get_monmap_and_config 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.542+0000 7f528b03f640 10 monclient: build_initial_monmap 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.542+0000 7f528b03f640 10 monclient: monmap: 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:22.542919+0000 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:22.542919+0000 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:22.543 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:22.544 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:22.544 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.542+0000 7f528b03f640 1 Processor -- start 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.542+0000 7f528b03f640 1 -- start start 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.542+0000 7f528b03f640 10 monclient: init 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 0x7f5284131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): picked mon.noname-b con 0x7f5284131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f528409e910 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 0x7f5284138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): picked mon.noname-a con 0x7f5284136bb0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528409faf0 con 0x7f528409e910 2024-09-15T03:50:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528409fd30 con 0x7f5284136bb0 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52840a1110 con 0x7f5284131ab0 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f528b03f640 10 monclient(hunting): authenticate will time out at 2020.182436s 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 0x7f5284138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 0x7f5284131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 0x7f5284138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33732/0 (socket says 172.21.15.153:33732) 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 1 -- 172.21.15.153:0/2757577390 learned_addr learned my addr 172.21.15.153:0/2757577390 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5283fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f528409e910 0x7f528409ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40232/0 (socket says 172.21.15.153:40232) 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): get_auth_request con 0x7f5284136bb0 auth_method 0 2024-09-15T03:50:22.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 --2- 172.21.15.153:0/2757577390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 0x7f5284131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41424/0 (socket says 172.21.15.153:41424) 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): get_auth_request con 0x7f5284131ab0 auth_method 0 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient(hunting): handle_auth_done global_id 244346 payload 274 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 -- 172.21.15.153:0/2757577390 >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 msgr2=0x7f5284138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 --2- 172.21.15.153:0/2757577390 >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 0x7f5284138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 -- 172.21.15.153:0/2757577390 >> v1:172.21.15.153:6789/0 conn(0x7f528409e910 legacy=0x7f528409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient: _finish_hunting 0 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 monclient: found mon.noname-b 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f5288ddb640 1 -- 172.21.15.153:0/2757577390 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52840a13f0 con 0x7f5284131ab0 2024-09-15T03:50:22.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.543+0000 7f52895dc640 1 --2- 172.21.15.153:0/2757577390 >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 0x7f5284138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:22.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.546+0000 7f5288ddb640 1 -- 172.21.15.153:0/2757577390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 msgr2=0x7f5284131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:22.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.546+0000 7f5288ddb640 1 --2- 172.21.15.153:0/2757577390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 0x7f5284131e80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f527c009850 tx=0x7f527c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ebca6c9ebfb9a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:22.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 1 -- 172.21.15.153:0/2757577390 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c02ae80 con 0x7f5284131ab0 2024-09-15T03:50:22.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:22.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: dump: 2024-09-15T03:50:22.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.556 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 1 -- 172.21.15.153:0/2757577390 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527c032440 con 0x7f5284131ab0 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 5 monclient: authenticate success, global_id 244346 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 1 -- 172.21.15.153:0/2757577390 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c0327c0 con 0x7f5284131ab0 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 4 set_mon_vals no callback set 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f52837fe640 10 monclient: dump: 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 10 monclient: shutdown 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 -- 172.21.15.153:0/2757577390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 msgr2=0x7f5284131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 --2- 172.21.15.153:0/2757577390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 0x7f5284131e80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f527c009850 tx=0x7f527c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 -- 172.21.15.153:0/2757577390 shutdown_connections 2024-09-15T03:50:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 --2- 172.21.15.153:0/2757577390 >> v2:172.21.15.153:3300/0 conn(0x7f5284136bb0 0x7f5284138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 --2- 172.21.15.153:0/2757577390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284131ab0 0x7f5284131e80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 -- 172.21.15.153:0/2757577390 >> 172.21.15.153:0/2757577390 conn(0x7f5284067d30 msgr2=0x7f52840a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 -- 172.21.15.153:0/2757577390 shutdown_connections 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.547+0000 7f528b03f640 1 -- 172.21.15.153:0/2757577390 wait complete. 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.548+0000 7f528b03f640 10 monclient: build_initial_monmap 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.548+0000 7f528b03f640 10 monclient: monmap: 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.548+0000 7f528b03f640 1 Processor -- start 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.548+0000 7f528b03f640 1 -- start start 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.548+0000 7f528b03f640 10 monclient: init 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 0x7f5284097c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): picked mon.noname-a con 0x7f528409e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5284131ab0 0x7f52840981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): picked mon.noname-b con 0x7f5284131ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 0x7f52840986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): picked mon.noname-c con 0x7f5284136bb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528409aa60 con 0x7f528409e910 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5284131650 con 0x7f5284131ab0 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528409d900 con 0x7f5284136bb0 2024-09-15T03:50:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f528b03f640 10 monclient(hunting): authenticate will time out at 2020.188185s 2024-09-15T03:50:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 0x7f5284097c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 0x7f5284097c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33746/0 (socket says 172.21.15.153:33746) 2024-09-15T03:50:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 0x7f52840986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 1 -- 172.21.15.153:0/2537090727 learned_addr learned my addr 172.21.15.153:0/2537090727 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): get_auth_request con 0x7f528409e910 auth_method 0 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): get_auth_request con 0x7f5284136bb0 auth_method 0 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5283fff640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5284131ab0 0x7f52840981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient(hunting): handle_auth_done global_id 244349 payload 824 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 -- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5284131ab0 msgr2=0x7f52840981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5284131ab0 0x7f52840981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 -- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 msgr2=0x7f5284097c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 0x7f5284097c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient: _finish_hunting 0 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 monclient: found mon.noname-c 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:22.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f52895dc640 1 -- 172.21.15.153:0/2537090727 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5274009450 con 0x7f5284136bb0 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5283fff640 10 monclient: get_auth_request con 0x7f5284131ab0 auth_method 0 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5288ddb640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 0x7f5284097c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.549+0000 7f5283fff640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5284131ab0 0x7f52840981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f52895dc640 1 -- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 msgr2=0x7f52840986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f52895dc640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 0x7f52840986e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f527400e950 tx=0x7f52740091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527400cb50 con 0x7f5284136bb0 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 10 monclient: dump: 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 10 monclient: _finish_auth 0 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527400ccd0 con 0x7f5284136bb0 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f528b03f640 5 monclient: authenticate success, global_id 244349 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.551+0000 7f5280ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52740136c0 con 0x7f5284136bb0 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f5280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f5280ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f5280ff9640 10 monclient: dump: 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 10 monclient: _renew_subs 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f5282ffd640 4 set_mon_vals no callback set 2024-09-15T03:50:22.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c009390 con 0x7f5284136bb0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 10 monclient: _renew_subs 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528409b720 con 0x7f5284136bb0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f5274013a30 con 0x7f5284136bb0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.552+0000 7f528b03f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.553+0000 7f528b03f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.553+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528404bf60 con 0x7f5284136bb0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.553+0000 7f5280ff9640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5264038740 0x7f526403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.553+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f527404c6c0 con 0x7f5284136bb0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.553+0000 7f5288ddb640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5264038740 0x7f526403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.553+0000 7f5288ddb640 10 monclient: get_auth_request con 0x7f5264038740 auth_method 0 2024-09-15T03:50:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.555+0000 7f5288ddb640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5264038740 0x7f526403abf0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f527c009130 tx=0x7f527c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.557+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f527401ab40 con 0x7f5284136bb0 2024-09-15T03:50:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.557+0000 7f5280ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.557+0000 7f5280ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:22.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f528b03f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:50:22.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f528b03f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:50:22.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f528b03f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:22.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f528409ad70 con 0x7f5284136bb0 2024-09-15T03:50:22.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f5280ff9640 1 -- 172.21.15.153:0/2537090727 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f5274020620 con 0x7f5284136bb0 2024-09-15T03:50:22.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f5280ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:50:22.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.770+0000 7f5280ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:22.773 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","modified":"2024-09-15T03:27:58.706914Z","created":"2024-09-15T03:27:58.706914Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5264038740 msgr2=0x7f526403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5264038740 0x7f526403abf0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f527c009130 tx=0x7f527c032000 comp rx=0 tx=0).stop 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 10 monclient: shutdown 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 msgr2=0x7f52840986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 0x7f52840986e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f527400e950 tx=0x7f52740091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 shutdown_connections 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f5264038740 0x7f526403abf0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5284136bb0 0x7f52840986e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5284131ab0 0x7f52840981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 --2- 172.21.15.153:0/2537090727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f528409e910 0x7f5284097c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 >> 172.21.15.153:0/2537090727 conn(0x7f5284067d30 msgr2=0x7f52840a3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:22.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 shutdown_connections 2024-09-15T03:50:22.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.772+0000 7f528b03f640 1 -- 172.21.15.153:0/2537090727 wait complete. 2024-09-15T03:50:22.784 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:50:22.784 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T03:50:22.793 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T03:50:22.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:50:22.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.874+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.875 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.874+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64721914}) 2024-09-15T03:50:22.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.874+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3676084512}) 2024-09-15T03:50:22.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.874+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:22.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.874+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64721914}) 2024-09-15T03:50:22.876 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:22.874+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744716355}) 2024-09-15T03:50:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.887+0000 7f34d4646640 10 monclient: get_monmap_and_config 2024-09-15T03:50:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.887+0000 7f34d4646640 10 monclient: build_initial_monmap 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.887+0000 7f34d4646640 10 monclient: monmap: 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:22.887787+0000 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:22.887787+0000 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:22.888 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:22.889 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:22.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.887+0000 7f34d4646640 1 Processor -- start 2024-09-15T03:50:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.887+0000 7f34d4646640 1 -- start start 2024-09-15T03:50:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.887+0000 7f34d4646640 10 monclient: init 2024-09-15T03:50:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:50:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34cc096e10 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34cc0971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f34cc134b40 0x7f34cc136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient(hunting): picked mon.noname-a con 0x7f34cc134b40 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc09d4a0 con 0x7f34cc096e10 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc099620 con 0x7f34cc0971e0 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc099340 con 0x7f34cc134b40 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d4646640 10 monclient(hunting): authenticate will time out at 2020.527291s 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d23e2640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f34cc096e10 0x7f34cc09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40236/0 (socket says 172.21.15.153:40236) 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d23e2640 1 -- 172.21.15.153:0/3129165410 learned_addr learned my addr 172.21.15.153:0/3129165410 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 1 --2- 172.21.15.153:0/3129165410 >> v2:172.21.15.153:3300/0 conn(0x7f34cc134b40 0x7f34cc136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): get_auth_request con 0x7f34cc134b40 auth_method 0 2024-09-15T03:50:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient(hunting): handle_auth_done global_id 254244 payload 274 2024-09-15T03:50:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 1 -- 172.21.15.153:0/3129165410 >> v1:172.21.15.181:6789/0 conn(0x7f34cc0971e0 legacy=0x7f34cc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 1 -- 172.21.15.153:0/3129165410 >> v1:172.21.15.153:6789/0 conn(0x7f34cc096e10 legacy=0x7f34cc09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient: _finish_hunting 0 2024-09-15T03:50:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 1 monclient: found mon.noname-a 2024-09-15T03:50:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.888+0000 7f34d2be3640 1 -- 172.21.15.153:0/3129165410 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc099900 con 0x7f34cc134b40 2024-09-15T03:50:22.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.891+0000 7f34d2be3640 1 -- 172.21.15.153:0/3129165410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 msgr2=0x7f34cc136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.891+0000 7f34d2be3640 1 --2- 172.21.15.153:0/3129165410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc136f20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f34cc09b020 tx=0x7f34c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a44e245d7b49d6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.891+0000 7f34d13e0640 1 -- 172.21.15.153:0/3129165410 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4007630 con 0x7f34cc134b40 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: dump: 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: _finish_auth 0 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 5 monclient: authenticate success, global_id 254244 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 1 -- 172.21.15.153:0/3129165410 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c402d050 con 0x7f34cc134b40 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 4 set_mon_vals no callback set 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 1 -- 172.21.15.153:0/3129165410 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40079c0 con 0x7f34cc134b40 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d13e0640 10 monclient: dump: 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 10 monclient: shutdown 2024-09-15T03:50:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 -- 172.21.15.153:0/3129165410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 msgr2=0x7f34cc136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 --2- 172.21.15.153:0/3129165410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc136f20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f34cc09b020 tx=0x7f34c4009780 comp rx=0 tx=0).stop 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 -- 172.21.15.153:0/3129165410 shutdown_connections 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 --2- 172.21.15.153:0/3129165410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc136f20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 -- 172.21.15.153:0/3129165410 >> 172.21.15.153:0/3129165410 conn(0x7f34cc0672b0 msgr2=0x7f34cc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 -- 172.21.15.153:0/3129165410 shutdown_connections 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.892+0000 7f34d4646640 1 -- 172.21.15.153:0/3129165410 wait complete. 2024-09-15T03:50:22.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 10 monclient: build_initial_monmap 2024-09-15T03:50:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 10 monclient: monmap: 2024-09-15T03:50:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 1 Processor -- start 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 1 -- start start 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 10 monclient: init 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.893+0000 7f34d4646640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34cc096e10 0x7f34cc12ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): picked mon.noname-b con 0x7f34cc096e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34cc0971e0 0x7f34cc09b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): picked mon.noname-c con 0x7f34cc0971e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc09bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): picked mon.noname-a con 0x7f34cc134b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc09b050 con 0x7f34cc134b40 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc09e5f0 con 0x7f34cc096e10 2024-09-15T03:50:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc133bf0 con 0x7f34cc0971e0 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d4646640 10 monclient(hunting): authenticate will time out at 2020.533077s 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc09bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34cc0971e0 0x7f34cc09b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc09bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33778/0 (socket says 172.21.15.153:33778) 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 -- 172.21.15.153:0/1197839076 learned_addr learned my addr 172.21.15.153:0/1197839076 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): get_auth_request con 0x7f34cc134b40 auth_method 0 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): get_auth_request con 0x7f34cc0971e0 auth_method 0 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34cc096e10 0x7f34cc12ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): get_auth_request con 0x7f34cc096e10 auth_method 0 2024-09-15T03:50:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d23e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient(hunting): handle_auth_done global_id 254247 payload 824 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 -- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34cc0971e0 msgr2=0x7f34cc09b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34cc0971e0 0x7f34cc09b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 -- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34cc096e10 msgr2=0x7f34cc12ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34cc096e10 0x7f34cc12ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient: _finish_hunting 0 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 monclient: found mon.noname-a 2024-09-15T03:50:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d2be3640 1 -- 172.21.15.153:0/1197839076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4009a20 con 0x7f34cc134b40 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.894+0000 7f34d1be1640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34cc0971e0 0x7f34cc09b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d2be3640 1 -- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 msgr2=0x7f34cc09bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d2be3640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc09bd10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009de0 tx=0x7f34c4037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4007e80 con 0x7f34cc134b40 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: dump: 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d4646640 5 monclient: authenticate success, global_id 254247 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4040070 con 0x7f34cc134b40 2024-09-15T03:50:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4031cf0 con 0x7f34cc134b40 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d4646640 10 monclient: _renew_subs 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d4646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc132010 con 0x7f34cc134b40 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34d0bdf640 4 set_mon_vals no callback set 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.896+0000 7f34c27fc640 10 monclient: dump: 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 10 monclient: _renew_subs 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc131b00 con 0x7f34cc134b40 2024-09-15T03:50:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f34c4042030 con 0x7f34cc134b40 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc04bf60 con 0x7f34cc134b40 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.897+0000 7f34c27fc640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b0038740 0x7f34b003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.898+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f34c406de90 con 0x7f34cc134b40 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.898+0000 7f34d23e2640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b0038740 0x7f34b003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.898+0000 7f34d23e2640 10 monclient: get_auth_request con 0x7f34b0038740 auth_method 0 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.900+0000 7f34d23e2640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b0038740 0x7f34b003abf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f34c8007fd0 tx=0x7f34c8007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.902+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f34c404e440 con 0x7f34cc134b40 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.902+0000 7f34c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:22.902+0000 7f34c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:23.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.030+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.032 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.030+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233783291}) 2024-09-15T03:50:23.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.030+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112784238}) 2024-09-15T03:50:23.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.030+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233783291}) 2024-09-15T03:50:23.033 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.031+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430503429}) 2024-09-15T03:50:23.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.033+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.033+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252486831}) 2024-09-15T03:50:23.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.033+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=515384270}) 2024-09-15T03:50:23.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.033+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.033+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252486831}) 2024-09-15T03:50:23.034 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.033+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567300682}) 2024-09-15T03:50:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.061+0000 7f34d4646640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:50:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.061+0000 7f34d4646640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:50:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.061+0000 7f34d4646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.061+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34cc04bf60 con 0x7f34cc134b40 2024-09-15T03:50:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.062+0000 7f34c27fc640 1 -- 172.21.15.153:0/1197839076 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+20424 (secure 0 0 0) 0x7f34c4057fe0 con 0x7f34cc134b40 2024-09-15T03:50:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.062+0000 7f34c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:50:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.062+0000 7f34c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:23.065 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:23.066 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":259,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:38:20.477637+0000","last_up_change":"2024-09-15T03:28:32.859989+0000","last_in_change":"2024-09-15T03:33:16.513275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":183,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:28:33.881566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:28:34.684266+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-15T03:33:32.211473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-15T03:33:32.992291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":31922},{"type":"v1","addr":"172.21.15.153:6801","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":31922},{"type":"v1","addr":"172.21.15.153:6803","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":31922},{"type":"v1","addr":"172.21.15.153:6807","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":31922},{"type":"v1","addr":"172.21.15.153:6805","nonce":31922}]},"public_addr":"172.21.15.153:6801/31922","cluster_addr":"172.21.15.153:6803/31922","heartbeat_back_addr":"172.21.15.153:6807/31922","heartbeat_front_addr":"172.21.15.153:6805/31922","state":["exists","up"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":31923},{"type":"v1","addr":"172.21.15.153:6809","nonce":31923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":31923},{"type":"v1","addr":"172.21.15.153:6811","nonce":31923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":31923},{"type":"v1","addr":"172.21.15.153:6815","nonce":31923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":31923},{"type":"v1","addr":"172.21.15.153:6813","nonce":31923}]},"public_addr":"172.21.15.153:6809/31923","cluster_addr":"172.21.15.153:6811/31923","heartbeat_back_addr":"172.21.15.153:6815/31923","heartbeat_front_addr":"172.21.15.153:6813/31923","state":["exists","up"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":31924},{"type":"v1","addr":"172.21.15.153:6817","nonce":31924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":31924},{"type":"v1","addr":"172.21.15.153:6819","nonce":31924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":31924},{"type":"v1","addr":"172.21.15.153:6823","nonce":31924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":31924},{"type":"v1","addr":"172.21.15.153:6821","nonce":31924}]},"public_addr":"172.21.15.153:6817/31924","cluster_addr":"172.21.15.153:6819/31924","heartbeat_back_addr":"172.21.15.153:6823/31924","heartbeat_front_addr":"172.21.15.153:6821/31924","state":["exists","up"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":31874},{"type":"v1","addr":"172.21.15.181:6809","nonce":31874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":31874},{"type":"v1","addr":"172.21.15.181:6811","nonce":31874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":31874},{"type":"v1","addr":"172.21.15.181:6815","nonce":31874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":31874},{"type":"v1","addr":"172.21.15.181:6813","nonce":31874}]},"public_addr":"172.21.15.181:6809/31874","cluster_addr":"172.21.15.181:6811/31874","heartbeat_back_addr":"172.21.15.181:6815/31874","heartbeat_front_addr":"172.21.15.181:6813/31874","state":["exists","up"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":31875},{"type":"v1","addr":"172.21.15.181:6801","nonce":31875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":31875},{"type":"v1","addr":"172.21.15.181:6803","nonce":31875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":31875},{"type":"v1","addr":"172.21.15.181:6807","nonce":31875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":31875},{"type":"v1","addr":"172.21.15.181:6805","nonce":31875}]},"public_addr":"172.21.15.181:6801/31875","cluster_addr":"172.21.15.181:6803/31875","heartbeat_back_addr":"172.21.15.181:6807/31875","heartbeat_front_addr":"172.21.15.181:6805/31875","state":["exists","up"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":31876},{"type":"v1","addr":"172.21.15.181:6817","nonce":31876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":31876},{"type":"v1","addr":"172.21.15.181:6819","nonce":31876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":31876},{"type":"v1","addr":"172.21.15.181:6823","nonce":31876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":31876},{"type":"v1","addr":"172.21.15.181:6821","nonce":31876}]},"public_addr":"172.21.15.181:6817/31876","cluster_addr":"172.21.15.181:6819/31876","heartbeat_back_addr":"172.21.15.181:6823/31876","heartbeat_front_addr":"172.21.15.181:6821/31876","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"ecab9f08-f922-45d9-8393-9f53fccdb802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.685491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.637606+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.740014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.612976+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.627273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.624592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:50:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b0038740 msgr2=0x7f34b003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b0038740 0x7f34b003abf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f34c8007fd0 tx=0x7f34c8007f00 comp rx=0 tx=0).stop 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 10 monclient: shutdown 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 msgr2=0x7f34cc09bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc09bd10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009de0 tx=0x7f34c4037000 comp rx=0 tx=0).stop 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 shutdown_connections 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b0038740 0x7f34b003abf0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34cc134b40 0x7f34cc09bd10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34cc0971e0 0x7f34cc09b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 --2- 172.21.15.153:0/1197839076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34cc096e10 0x7f34cc12ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 >> 172.21.15.153:0/1197839076 conn(0x7f34cc0672b0 msgr2=0x7f34cc1297a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 shutdown_connections 2024-09-15T03:50:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f34d4646640 1 -- 172.21.15.153:0/1197839076 wait complete. 2024-09-15T03:50:23.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508406753}) 2024-09-15T03:50:23.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489747845}) 2024-09-15T03:50:23.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508406753}) 2024-09-15T03:50:23.068 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.065+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353052042}) 2024-09-15T03:50:23.076 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:50:23.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:50:23.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.169+0000 7fd9f2f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.169+0000 7fd9f2f1d640 10 monclient: build_initial_monmap 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.169+0000 7fd9f2f1d640 10 monclient: monmap: 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:23.169868+0000 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:23.169868+0000 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:23.170 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.171 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:23.171 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:23.171 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:23.171 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:23.171 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:23.171 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.169+0000 7fd9f2f1d640 1 Processor -- start 2024-09-15T03:50:23.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.169+0000 7fd9f2f1d640 1 -- start start 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.169+0000 7fd9f2f1d640 10 monclient: init 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9ec1318f0 0x7fd9ec131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9ec1318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9ec09ea50 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9ec136c20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec131cc0 con 0x7fd9ec09ea50 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ec09f530 con 0x7fd9ec1318f0 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec09fbf0 con 0x7fd9ec136c20 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f2f1d640 10 monclient(hunting): authenticate will time out at 2020.809448s 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9ec1318f0 0x7fd9ec131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9ec1318f0 0x7fd9ec131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33790/0 (socket says 172.21.15.153:33790) 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/2327803993 learned_addr learned my addr 172.21.15.153:0/2327803993 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): get_auth_request con 0x7fd9ec1318f0 auth_method 0 2024-09-15T03:50:23.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9ebfff640 1 --1- 172.21.15.153:0/2327803993 >> v1:172.21.15.153:6789/0 conn(0x7fd9ec09ea50 0x7fd9ec09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40240/0 (socket says 172.21.15.153:40240) 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): get_auth_request con 0x7fd9ec136c20 auth_method 0 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_done global_id 254250 payload 274 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec136c20 msgr2=0x7fd9ec139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 --2- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/2327803993 >> v1:172.21.15.153:6789/0 conn(0x7fd9ec09ea50 legacy=0x7fd9ec09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 monclient: found mon.noname-a 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:23.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/2327803993 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec09fe30 con 0x7fd9ec1318f0 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.170+0000 7fd9f14ba640 1 --2- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec1318f0 msgr2=0x7fd9ec131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9f0cb9640 1 --2- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec131f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc009910 tx=0x7fd9dc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c4394c9c0bb5d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 1 -- 172.21.15.153:0/2327803993 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc032440 con 0x7fd9ec1318f0 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: dump: 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.180 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 1 -- 172.21.15.153:0/2327803993 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc0325a0 con 0x7fd9ec1318f0 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9f2f1d640 5 monclient: authenticate success, global_id 254250 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9f2f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9f2f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:23.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 1 -- 172.21.15.153:0/2327803993 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc032940 con 0x7fd9ec1318f0 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9f2f1d640 4 set_mon_vals no callback set 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.173+0000 7fd9eb7fe640 10 monclient: dump: 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 10 monclient: shutdown 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec1318f0 msgr2=0x7fd9ec131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec131f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc009910 tx=0x7fd9dc0091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/2327803993 shutdown_connections 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/2327803993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec131f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/2327803993 >> 172.21.15.153:0/2327803993 conn(0x7fd9ec067110 msgr2=0x7fd9ec067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/2327803993 shutdown_connections 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/2327803993 wait complete. 2024-09-15T03:50:23.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 10 monclient: build_initial_monmap 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.174+0000 7fd9f2f1d640 10 monclient: monmap: 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:23.184 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:23.185 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 Processor -- start 2024-09-15T03:50:23.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 -- start start 2024-09-15T03:50:23.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient: init 2024-09-15T03:50:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:50:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 0x7fd9ec130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9ec09ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec0997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9ec1318f0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec136c20 0x7fd9ec099cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9ec136c20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec141f20 con 0x7fd9ec136c20 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec09b2e0 con 0x7fd9ec1318f0 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec11cd60 con 0x7fd9ec09ea50 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f2f1d640 10 monclient(hunting): authenticate will time out at 2020.814733s 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 0x7fd9ec130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec136c20 0x7fd9ec099cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 0x7fd9ec130970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41462/0 (socket says 172.21.15.153:41462) 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/3873137167 learned_addr learned my addr 172.21.15.153:0/3873137167 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): get_auth_request con 0x7fd9ec09ea50 auth_method 0 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): get_auth_request con 0x7fd9ec136c20 auth_method 0 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.175+0000 7fd9f14ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 10 monclient(hunting): handle_auth_done global_id 244358 payload 824 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9ec1318f0 msgr2=0x7fd9ec0997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9ebfff640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec0997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec0997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec136c20 msgr2=0x7fd9ec099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec136c20 0x7fd9ec099cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 10 monclient: _finish_hunting 0 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 1 monclient: found mon.noname-c 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/3873137167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4009a20 con 0x7fd9ec09ea50 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.176+0000 7fd9f14ba640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec136c20 0x7fd9ec099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f0cb9640 1 -- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 msgr2=0x7fd9ec130970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f0cb9640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 0x7fd9ec130970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc02a6e0 tx=0x7fd9dc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc031a90 con 0x7fd9ec09ea50 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: dump: 2024-09-15T03:50:23.193 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.194 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.194 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.194 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.194 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.194 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.194 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc031c30 con 0x7fd9ec09ea50 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 5 monclient: authenticate success, global_id 244358 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 10 monclient: _renew_subs 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc03d420 con 0x7fd9ec09ea50 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc009450 con 0x7fd9ec09ea50 2024-09-15T03:50:23.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9e8ff9640 10 monclient: dump: 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9eaffd640 4 set_mon_vals no callback set 2024-09-15T03:50:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 10 monclient: _renew_subs 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.178+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec11b720 con 0x7fd9ec09ea50 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fd9dc039040 con 0x7fd9ec09ea50 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9f2f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9f2f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9f2f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec11b910 con 0x7fd9ec09ea50 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9e8ff9640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd9d8038740 0x7fd9d803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd9dc06c360 con 0x7fd9ec09ea50 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9ebfff640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd9d8038740 0x7fd9d803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.179+0000 7fd9ebfff640 10 monclient: get_auth_request con 0x7fd9d8038740 auth_method 0 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.182+0000 7fd9ebfff640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd9d8038740 0x7fd9d803abf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0a4ef0 tx=0x7fd9e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.184+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd9dc03d8b0 con 0x7fd9ec09ea50 2024-09-15T03:50:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.184+0000 7fd9e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.184+0000 7fd9e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.337+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9ec04bf60 con 0x7fd9d8038740 2024-09-15T03:50:23.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.341+0000 7fd9e8ff9640 1 -- 172.21.15.153:0/3873137167 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98135 (secure 0 0 0) 0x7fd9ec04bf60 con 0x7fd9d8038740 2024-09-15T03:50:23.343 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:23.344 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd9d8038740 msgr2=0x7fd9d803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd9d8038740 0x7fd9d803abf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0a4ef0 tx=0x7fd9e0009040 comp rx=0 tx=0).stop 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 10 monclient: shutdown 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 msgr2=0x7fd9ec130970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 0x7fd9ec130970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc02a6e0 tx=0x7fd9dc032000 comp rx=0 tx=0).stop 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 shutdown_connections 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fd9d8038740 0x7fd9d803abf0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ec136c20 0x7fd9ec099cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9ec1318f0 0x7fd9ec0997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 --2- 172.21.15.153:0/3873137167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ec09ea50 0x7fd9ec130970 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 >> 172.21.15.153:0/3873137167 conn(0x7fd9ec067110 msgr2=0x7fd9ec0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 shutdown_connections 2024-09-15T03:50:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.344+0000 7fd9f2f1d640 1 -- 172.21.15.153:0/3873137167 wait complete. 2024-09-15T03:50:23.355 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":908,"stamp":"2024-09-15T03:50:22.114741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1809672,"kb_used_data":8896,"kb_used_omap":9,"kb_used_meta":1800694,"kb_avail":564421368,"statfs":{"total":579820584960,"available":577967480832,"internally_reserved":0,"allocated":9109504,"data_stored":7285462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843910672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":171,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253295+0000","last_change":"2024-09-15T03:35:36.177407+0000","last_active":"2024-09-15T03:38:16.253295+0000","last_peered":"2024-09-15T03:38:16.253295+0000","last_clean":"2024-09-15T03:38:16.253295+0000","last_became_active":"2024-09-15T03:35:36.177073+0000","last_became_peered":"2024-09-15T03:35:36.177073+0000","last_unstale":"2024-09-15T03:38:16.253295+0000","last_undegraded":"2024-09-15T03:38:16.253295+0000","last_fullsized":"2024-09-15T03:38:16.253295+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:01:53.230196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412884+0000","last_change":"2024-09-15T03:35:36.171117+0000","last_active":"2024-09-15T03:38:15.412884+0000","last_peered":"2024-09-15T03:38:15.412884+0000","last_clean":"2024-09-15T03:38:15.412884+0000","last_became_active":"2024-09-15T03:35:36.170838+0000","last_became_peered":"2024-09-15T03:35:36.170838+0000","last_unstale":"2024-09-15T03:38:15.412884+0000","last_undegraded":"2024-09-15T03:38:15.412884+0000","last_fullsized":"2024-09-15T03:38:15.412884+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:12:13.052343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":146,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866090+0000","last_change":"2024-09-15T03:35:36.180940+0000","last_active":"2024-09-15T03:38:16.866090+0000","last_peered":"2024-09-15T03:38:16.866090+0000","last_clean":"2024-09-15T03:38:16.866090+0000","last_became_active":"2024-09-15T03:35:36.180590+0000","last_became_peered":"2024-09-15T03:35:36.180590+0000","last_unstale":"2024-09-15T03:38:16.866090+0000","last_undegraded":"2024-09-15T03:38:16.866090+0000","last_fullsized":"2024-09-15T03:38:16.866090+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:06.005311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412806+0000","last_change":"2024-09-15T03:35:36.173953+0000","last_active":"2024-09-15T03:38:15.412806+0000","last_peered":"2024-09-15T03:38:15.412806+0000","last_clean":"2024-09-15T03:38:15.412806+0000","last_became_active":"2024-09-15T03:35:36.173628+0000","last_became_peered":"2024-09-15T03:35:36.173628+0000","last_unstale":"2024-09-15T03:38:15.412806+0000","last_undegraded":"2024-09-15T03:38:15.412806+0000","last_fullsized":"2024-09-15T03:38:15.412806+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:56:12.188821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":131,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412847+0000","last_change":"2024-09-15T03:35:42.048298+0000","last_active":"2024-09-15T03:38:15.412847+0000","last_peered":"2024-09-15T03:38:15.412847+0000","last_clean":"2024-09-15T03:38:15.412847+0000","last_became_active":"2024-09-15T03:35:42.047949+0000","last_became_peered":"2024-09-15T03:35:42.047949+0000","last_unstale":"2024-09-15T03:38:15.412847+0000","last_undegraded":"2024-09-15T03:38:15.412847+0000","last_fullsized":"2024-09-15T03:38:15.412847+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:49:06.838016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":147,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253319+0000","last_change":"2024-09-15T03:35:36.179570+0000","last_active":"2024-09-15T03:38:16.253319+0000","last_peered":"2024-09-15T03:38:16.253319+0000","last_clean":"2024-09-15T03:38:16.253319+0000","last_became_active":"2024-09-15T03:35:36.179302+0000","last_became_peered":"2024-09-15T03:35:36.179302+0000","last_unstale":"2024-09-15T03:38:16.253319+0000","last_undegraded":"2024-09-15T03:38:16.253319+0000","last_fullsized":"2024-09-15T03:38:16.253319+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:26:04.635839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411506+0000","last_change":"2024-09-15T03:35:36.179910+0000","last_active":"2024-09-15T03:38:15.411506+0000","last_peered":"2024-09-15T03:38:15.411506+0000","last_clean":"2024-09-15T03:38:15.411506+0000","last_became_active":"2024-09-15T03:35:36.179540+0000","last_became_peered":"2024-09-15T03:35:36.179540+0000","last_unstale":"2024-09-15T03:38:15.411506+0000","last_undegraded":"2024-09-15T03:38:15.411506+0000","last_fullsized":"2024-09-15T03:38:15.411506+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:29.604427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510296+0000","last_change":"2024-09-15T03:35:36.176521+0000","last_active":"2024-09-15T03:38:16.510296+0000","last_peered":"2024-09-15T03:38:16.510296+0000","last_clean":"2024-09-15T03:38:16.510296+0000","last_became_active":"2024-09-15T03:35:36.176169+0000","last_became_peered":"2024-09-15T03:35:36.176169+0000","last_unstale":"2024-09-15T03:38:16.510296+0000","last_undegraded":"2024-09-15T03:38:16.510296+0000","last_fullsized":"2024-09-15T03:38:16.510296+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:17:56.952574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":114,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412674+0000","last_change":"2024-09-15T03:35:36.176184+0000","last_active":"2024-09-15T03:38:15.412674+0000","last_peered":"2024-09-15T03:38:15.412674+0000","last_clean":"2024-09-15T03:38:15.412674+0000","last_became_active":"2024-09-15T03:35:36.175786+0000","last_became_peered":"2024-09-15T03:35:36.175786+0000","last_unstale":"2024-09-15T03:38:15.412674+0000","last_undegraded":"2024-09-15T03:38:15.412674+0000","last_fullsized":"2024-09-15T03:38:15.412674+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:57.476041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":109,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411493+0000","last_change":"2024-09-15T03:35:36.174198+0000","last_active":"2024-09-15T03:38:15.411493+0000","last_peered":"2024-09-15T03:38:15.411493+0000","last_clean":"2024-09-15T03:38:15.411493+0000","last_became_active":"2024-09-15T03:35:36.173941+0000","last_became_peered":"2024-09-15T03:35:36.173941+0000","last_unstale":"2024-09-15T03:38:15.411493+0000","last_undegraded":"2024-09-15T03:38:15.411493+0000","last_fullsized":"2024-09-15T03:38:15.411493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.690762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":122,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866041+0000","last_change":"2024-09-15T03:35:36.173396+0000","last_active":"2024-09-15T03:38:16.866041+0000","last_peered":"2024-09-15T03:38:16.866041+0000","last_clean":"2024-09-15T03:38:16.866041+0000","last_became_active":"2024-09-15T03:35:36.173027+0000","last_became_peered":"2024-09-15T03:35:36.173027+0000","last_unstale":"2024-09-15T03:38:16.866041+0000","last_undegraded":"2024-09-15T03:38:16.866041+0000","last_fullsized":"2024-09-15T03:38:16.866041+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:28:33.565728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510475+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510475+0000","last_peered":"2024-09-15T03:38:16.510475+0000","last_clean":"2024-09-15T03:38:16.510475+0000","last_became_active":"2024-09-15T03:35:36.176264+0000","last_became_peered":"2024-09-15T03:35:36.176264+0000","last_unstale":"2024-09-15T03:38:16.510475+0000","last_undegraded":"2024-09-15T03:38:16.510475+0000","last_fullsized":"2024-09-15T03:38:16.510475+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510499+0000","last_change":"2024-09-15T03:35:36.179783+0000","last_active":"2024-09-15T03:38:16.510499+0000","last_peered":"2024-09-15T03:38:16.510499+0000","last_clean":"2024-09-15T03:38:16.510499+0000","last_became_active":"2024-09-15T03:35:36.179473+0000","last_became_peered":"2024-09-15T03:35:36.179473+0000","last_unstale":"2024-09-15T03:38:16.510499+0000","last_undegraded":"2024-09-15T03:38:16.510499+0000","last_fullsized":"2024-09-15T03:38:16.510499+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:11.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":178,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253392+0000","last_change":"2024-09-15T03:35:36.179775+0000","last_active":"2024-09-15T03:38:16.253392+0000","last_peered":"2024-09-15T03:38:16.253392+0000","last_clean":"2024-09-15T03:38:16.253392+0000","last_became_active":"2024-09-15T03:35:36.179476+0000","last_became_peered":"2024-09-15T03:35:36.179476+0000","last_unstale":"2024-09-15T03:38:16.253392+0000","last_undegraded":"2024-09-15T03:38:16.253392+0000","last_fullsized":"2024-09-15T03:38:16.253392+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:33.988609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866135+0000","last_change":"2024-09-15T03:35:41.930950+0000","last_active":"2024-09-15T03:38:16.866135+0000","last_peered":"2024-09-15T03:38:16.866135+0000","last_clean":"2024-09-15T03:38:16.866135+0000","last_became_active":"2024-09-15T03:35:41.930552+0000","last_became_peered":"2024-09-15T03:35:41.930552+0000","last_unstale":"2024-09-15T03:38:16.866135+0000","last_undegraded":"2024-09-15T03:38:16.866135+0000","last_fullsized":"2024-09-15T03:38:16.866135+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.602540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510466+0000","last_change":"2024-09-15T03:35:36.176671+0000","last_active":"2024-09-15T03:38:16.510466+0000","last_peered":"2024-09-15T03:38:16.510466+0000","last_clean":"2024-09-15T03:38:16.510466+0000","last_became_active":"2024-09-15T03:35:36.176131+0000","last_became_peered":"2024-09-15T03:35:36.176131+0000","last_unstale":"2024-09-15T03:38:16.510466+0000","last_undegraded":"2024-09-15T03:38:16.510466+0000","last_fullsized":"2024-09-15T03:38:16.510466+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.630912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":332,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411395+0000","last_change":"2024-09-15T03:35:36.180042+0000","last_active":"2024-09-15T03:38:15.411395+0000","last_peered":"2024-09-15T03:38:15.411395+0000","last_clean":"2024-09-15T03:38:15.411395+0000","last_became_active":"2024-09-15T03:35:36.179501+0000","last_became_peered":"2024-09-15T03:35:36.179501+0000","last_unstale":"2024-09-15T03:38:15.411395+0000","last_undegraded":"2024-09-15T03:38:15.411395+0000","last_fullsized":"2024-09-15T03:38:15.411395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:32:25.282577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":254,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412719+0000","last_change":"2024-09-15T03:35:36.182382+0000","last_active":"2024-09-15T03:38:15.412719+0000","last_peered":"2024-09-15T03:38:15.412719+0000","last_clean":"2024-09-15T03:38:15.412719+0000","last_became_active":"2024-09-15T03:35:36.175947+0000","last_became_peered":"2024-09-15T03:35:36.175947+0000","last_unstale":"2024-09-15T03:38:15.412719+0000","last_undegraded":"2024-09-15T03:38:15.412719+0000","last_fullsized":"2024-09-15T03:38:15.412719+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:04:51.063277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411377+0000","last_change":"2024-09-15T03:35:36.177331+0000","last_active":"2024-09-15T03:38:15.411377+0000","last_peered":"2024-09-15T03:38:15.411377+0000","last_clean":"2024-09-15T03:38:15.411377+0000","last_became_active":"2024-09-15T03:35:36.177043+0000","last_became_peered":"2024-09-15T03:35:36.177043+0000","last_unstale":"2024-09-15T03:38:15.411377+0000","last_undegraded":"2024-09-15T03:38:15.411377+0000","last_fullsized":"2024-09-15T03:38:15.411377+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:52:08.123750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866082+0000","last_change":"2024-09-15T03:35:36.177279+0000","last_active":"2024-09-15T03:38:16.866082+0000","last_peered":"2024-09-15T03:38:16.866082+0000","last_clean":"2024-09-15T03:38:16.866082+0000","last_became_active":"2024-09-15T03:35:36.176997+0000","last_became_peered":"2024-09-15T03:35:36.176997+0000","last_unstale":"2024-09-15T03:38:16.866082+0000","last_undegraded":"2024-09-15T03:38:16.866082+0000","last_fullsized":"2024-09-15T03:38:16.866082+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:05:52.287110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":251,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412676+0000","last_change":"2024-09-15T03:35:36.189329+0000","last_active":"2024-09-15T03:38:15.412676+0000","last_peered":"2024-09-15T03:38:15.412676+0000","last_clean":"2024-09-15T03:38:15.412676+0000","last_became_active":"2024-09-15T03:35:36.176519+0000","last_became_peered":"2024-09-15T03:35:36.176519+0000","last_unstale":"2024-09-15T03:38:15.412676+0000","last_undegraded":"2024-09-15T03:38:15.412676+0000","last_fullsized":"2024-09-15T03:38:15.412676+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:40:12.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510504+0000","last_change":"2024-09-15T03:35:42.047866+0000","last_active":"2024-09-15T03:38:16.510504+0000","last_peered":"2024-09-15T03:38:16.510504+0000","last_clean":"2024-09-15T03:38:16.510504+0000","last_became_active":"2024-09-15T03:35:42.047571+0000","last_became_peered":"2024-09-15T03:35:42.047571+0000","last_unstale":"2024-09-15T03:38:16.510504+0000","last_undegraded":"2024-09-15T03:38:16.510504+0000","last_fullsized":"2024-09-15T03:38:16.510504+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:44.835315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411503+0000","last_change":"2024-09-15T03:35:36.180602+0000","last_active":"2024-09-15T03:38:15.411503+0000","last_peered":"2024-09-15T03:38:15.411503+0000","last_clean":"2024-09-15T03:38:15.411503+0000","last_became_active":"2024-09-15T03:35:36.180366+0000","last_became_peered":"2024-09-15T03:35:36.180366+0000","last_unstale":"2024-09-15T03:38:15.411503+0000","last_undegraded":"2024-09-15T03:38:15.411503+0000","last_fullsized":"2024-09-15T03:38:15.411503+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:51:20.797295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510310+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510310+0000","last_peered":"2024-09-15T03:38:16.510310+0000","last_clean":"2024-09-15T03:38:16.510310+0000","last_became_active":"2024-09-15T03:35:36.176265+0000","last_became_peered":"2024-09-15T03:35:36.176265+0000","last_unstale":"2024-09-15T03:38:16.510310+0000","last_undegraded":"2024-09-15T03:38:16.510310+0000","last_fullsized":"2024-09-15T03:38:16.510310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:07:54.095941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411431+0000","last_change":"2024-09-15T03:35:36.176300+0000","last_active":"2024-09-15T03:38:15.411431+0000","last_peered":"2024-09-15T03:38:15.411431+0000","last_clean":"2024-09-15T03:38:15.411431+0000","last_became_active":"2024-09-15T03:35:36.175986+0000","last_became_peered":"2024-09-15T03:35:36.175986+0000","last_unstale":"2024-09-15T03:38:15.411431+0000","last_undegraded":"2024-09-15T03:38:15.411431+0000","last_fullsized":"2024-09-15T03:38:15.411431+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:58:16.529865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63700000000000001,"5min":0.873,"15min":0.873},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59599999999999997,"5min":0.68400000000000005,"15min":0.751},"last":0.38200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.505},"min":{"1min":0.379,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73799999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.33800000000000002},"max":{"1min":0.60899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71199999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.74199999999999999},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.248,"5min":0.247,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.62},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.379},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":0.45200000000000001,"5min":0.63100000000000001,"15min":0.64400000000000002},"last":0.26800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.39000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.64400000000000002},"last":0.316},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.42199999999999999},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.77000000000000002},"last":0.42899999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.69099999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.499},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.64100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":0.59899999999999998,"5min":0.67700000000000005,"15min":0.70499999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.32800000000000001},"max":{"1min":0.629,"5min":0.66000000000000003,"15min":0.71199999999999997},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34599999999999997},"max":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.72199999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35299999999999998,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.72299999999999998},"last":0.53900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.627},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22700000000000001},"max":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.60099999999999998,"5min":0.63,"15min":0.63},"last":0.34699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640520,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51400000000000001,"15min":0.48999999999999999},"min":{"1min":0.35699999999999998,"5min":0.31,"15min":0.28000000000000003},"max":{"1min":0.68400000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54200000000000004,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.28699999999999998,"15min":0.17799999999999999},"max":{"1min":0.66400000000000003,"5min":0.76000000000000001,"15min":0.79700000000000004},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52600000000000002,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.29699999999999999},"max":{"1min":0.68300000000000005,"5min":0.755,"15min":0.75700000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.505,"15min":0.51400000000000001},"min":{"1min":0.34000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.71899999999999997,"15min":1.597},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.748,"15min":0.84199999999999997},"last":0.48899999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19600000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.65400000000000003},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.222},"max":{"1min":0.54300000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34399999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.25,"5min":0.23899999999999999,"15min":0.221},"max":{"1min":0.54700000000000004,"5min":0.63200000000000001,"15min":0.65500000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.44800000000000001,"15min":0.42699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":0.57399999999999995,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.31900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31,"15min":0.308},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.58799999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.72699999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":0.48699999999999999,"5min":0.51200000000000001,"15min":0.59599999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.22},"max":{"1min":0.50600000000000001,"5min":0.57499999999999996,"15min":0.68500000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.35299999999999998,"15min":0.26700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56100000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.314},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":5.6269999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.29199999999999998},"max":{"1min":0.65800000000000003,"5min":0.77500000000000002,"15min":0.84999999999999998},"last":0.5},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72799999999999998,"5min":0.77300000000000002,"15min":0.82599999999999996},"last":0.52200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640519,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32500000000000001,"15min":0.32700000000000001},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":0.41999999999999998,"5min":0.52800000000000002,"15min":0.874},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.48999999999999999,"15min":0.55900000000000005},"last":0.26700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.215},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.77800000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.50900000000000001,"15min":0.747},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.73199999999999998,"15min":0.89800000000000002},"last":0.5},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.88300000000000001},"last":0.432},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.76600000000000001,"15min":0.81799999999999995},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.93799999999999994},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.432,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.91300000000000003},"last":0.45300000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640520,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32200000000000001,"15min":0.32000000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":0.55000000000000004,"15min":0.625},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33300000000000002,"15min":0.33400000000000002},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":0.52400000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.219},"max":{"1min":0.47499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.28899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.433},{"interface":"front","average":{"1min":0.499,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":0.85699999999999998},"last":0.378}]},{"osd":4,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.29699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.34899999999999998,"15min":0.34200000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.81200000000000006},"last":0.44700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":0.69999999999999996,"5min":0.76000000000000001,"15min":0.78000000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.84099999999999997,"15min":0.84799999999999998},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:50:23.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.449+0000 7f0b9c15d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.449+0000 7f0b9c15d640 10 monclient: build_initial_monmap 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.449+0000 7f0b9c15d640 10 monclient: monmap: 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:23.449905+0000 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:23.449905+0000 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:23.451 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:23.451 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:23.451 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:23.451 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.449+0000 7f0b9c15d640 1 Processor -- start 2024-09-15T03:50:23.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.449+0000 7f0b9c15d640 1 -- start start 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.449+0000 7f0b9c15d640 10 monclient: init 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b941318f0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b9409ea50 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 0x7f0b9413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b94137c30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b94132640 con 0x7f0b941318f0 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b94131cc0 con 0x7f0b9409ea50 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9409f530 con 0x7f0b94137c30 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:23.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9c15d640 10 monclient(hunting): authenticate will time out at 2021.089406s 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 0x7f0b9413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b99ef9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0b941318f0 0x7f0b94131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40254/0 (socket says 172.21.15.153:40254) 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b99ef9640 1 -- 172.21.15.153:0/3805745956 learned_addr learned my addr 172.21.15.153:0/3805745956 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): get_auth_request con 0x7f0b94137c30 auth_method 0 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient(hunting): handle_auth_done global_id 244361 payload 274 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 1 -- 172.21.15.153:0/3805745956 >> v1:172.21.15.181:6789/0 conn(0x7f0b9409ea50 legacy=0x7f0b9409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:23.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 1 -- 172.21.15.153:0/3805745956 >> v1:172.21.15.153:6789/0 conn(0x7f0b941318f0 legacy=0x7f0b94131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:23.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient: _finish_hunting 0 2024-09-15T03:50:23.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 1 monclient: found mon.noname-b 2024-09-15T03:50:23.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:23.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.450+0000 7f0b9a6fa640 1 -- 172.21.15.153:0/3805745956 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9409fbf0 con 0x7f0b94137c30 2024-09-15T03:50:23.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b9a6fa640 1 -- 172.21.15.153:0/3805745956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 msgr2=0x7f0b9413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:23.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b9a6fa640 1 --2- 172.21.15.153:0/3805745956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 0x7f0b9413a010 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009880 tx=0x7f0b840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e623085cf36bc015 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 1 -- 172.21.15.153:0/3805745956 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84032440 con 0x7f0b94137c30 2024-09-15T03:50:23.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: dump: 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: _finish_auth 0 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 1 -- 172.21.15.153:0/3805745956 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b840325a0 con 0x7f0b94137c30 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b9c15d640 5 monclient: authenticate success, global_id 244361 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b9c15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 1 -- 172.21.15.153:0/3805745956 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84032940 con 0x7f0b94137c30 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b9c15d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b9c15d640 4 set_mon_vals no callback set 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.453+0000 7f0b98ef7640 10 monclient: dump: 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 10 monclient: shutdown 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 -- 172.21.15.153:0/3805745956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 msgr2=0x7f0b9413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/3805745956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 0x7f0b9413a010 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009880 tx=0x7f0b840090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 -- 172.21.15.153:0/3805745956 shutdown_connections 2024-09-15T03:50:23.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/3805745956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b94137c30 0x7f0b9413a010 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 -- 172.21.15.153:0/3805745956 >> 172.21.15.153:0/3805745956 conn(0x7f0b94067110 msgr2=0x7f0b94067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 -- 172.21.15.153:0/3805745956 shutdown_connections 2024-09-15T03:50:23.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 1 -- 172.21.15.153:0/3805745956 wait complete. 2024-09-15T03:50:23.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 10 monclient: build_initial_monmap 2024-09-15T03:50:23.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.454+0000 7f0b9c15d640 10 monclient: monmap: 2024-09-15T03:50:23.465 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:23.466 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 Processor -- start 2024-09-15T03:50:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 -- start start 2024-09-15T03:50:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient: init 2024-09-15T03:50:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:50:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 0x7f0b94137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9409ea50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 0x7f0b9412df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b941318f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b94137c30 0x7f0b9412e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b94137c30 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94131340 con 0x7f0b9409ea50 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94132870 con 0x7f0b94137c30 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94132ea0 con 0x7f0b941318f0 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:23.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9c15d640 10 monclient(hunting): authenticate will time out at 2021.094724s 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 0x7f0b94137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 0x7f0b9412df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 0x7f0b94137060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33808/0 (socket says 172.21.15.153:33808) 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 1 -- 172.21.15.153:0/2547215388 learned_addr learned my addr 172.21.15.153:0/2547215388 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): get_auth_request con 0x7f0b9409ea50 auth_method 0 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): get_auth_request con 0x7f0b941318f0 auth_method 0 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b99ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b996f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.455+0000 7f0b9a6fa640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b94137c30 0x7f0b9412e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b9a6fa640 10 monclient(hunting): get_auth_request con 0x7f0b94137c30 auth_method 0 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b9a6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b9a6fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b9a6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 10 monclient(hunting): handle_auth_done global_id 244364 payload 824 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 1 -- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b94137c30 msgr2=0x7f0b9412e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b94137c30 0x7f0b9412e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 1 -- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 msgr2=0x7f0b94137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 0x7f0b94137060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 10 monclient: _finish_hunting 0 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 1 monclient: found mon.noname-c 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b996f8640 1 -- 172.21.15.153:0/2547215388 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84009390 con 0x7f0b941318f0 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.456+0000 7f0b99ef9640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 0x7f0b94137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b996f8640 1 -- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 msgr2=0x7f0b9412df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b996f8640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 0x7f0b9412df80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800c860 tx=0x7f0b88002630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8800a850 con 0x7f0b941318f0 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: dump: 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.473 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: _finish_auth 0 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8800a9b0 con 0x7f0b941318f0 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 5 monclient: authenticate success, global_id 244364 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 10 monclient: _renew_subs 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b94130450 con 0x7f0b941318f0 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8800ad70 con 0x7f0b941318f0 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b91ffb640 10 monclient: dump: 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 10 monclient: _renew_subs 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b93fff640 4 set_mon_vals no callback set 2024-09-15T03:50:23.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.458+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b94131620 con 0x7f0b941318f0 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f0b88012630 con 0x7f0b941318f0 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b9c15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b9c15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b9c15d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9413a270 con 0x7f0b941318f0 2024-09-15T03:50:23.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b91ffb640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b80038740 0x7f0b8003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.459+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0b8804c9a0 con 0x7f0b941318f0 2024-09-15T03:50:23.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.460+0000 7f0b99ef9640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b80038740 0x7f0b8003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.460+0000 7f0b99ef9640 10 monclient: get_auth_request con 0x7f0b80038740 auth_method 0 2024-09-15T03:50:23.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.462+0000 7f0b99ef9640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b80038740 0x7f0b8003abf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c006fd0 tx=0x7f0b8c004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.464+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0b88022d70 con 0x7f0b941318f0 2024-09-15T03:50:23.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.464+0000 7f0b91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.464+0000 7f0b91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:23.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.614+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b9404bf60 con 0x7f0b80038740 2024-09-15T03:50:23.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.617+0000 7f0b91ffb640 1 -- 172.21.15.153:0/2547215388 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98135 (secure 0 0 0) 0x7f0b9404bf60 con 0x7f0b80038740 2024-09-15T03:50:23.619 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:23.619 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:50:23.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.619+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b80038740 msgr2=0x7f0b8003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.619+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b80038740 0x7f0b8003abf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c006fd0 tx=0x7f0b8c004960 comp rx=0 tx=0).stop 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.619+0000 7f0b9c15d640 10 monclient: shutdown 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.619+0000 7f0b9c15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.619+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 msgr2=0x7f0b9412df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.619+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 0x7f0b9412df80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800c860 tx=0x7f0b88002630 comp rx=0 tx=0).stop 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 shutdown_connections 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f0b80038740 0x7f0b8003abf0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b94137c30 0x7f0b9412e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b941318f0 0x7f0b9412df80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 --2- 172.21.15.153:0/2547215388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9409ea50 0x7f0b94137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 >> 172.21.15.153:0/2547215388 conn(0x7f0b94067110 msgr2=0x7f0b9409d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 shutdown_connections 2024-09-15T03:50:23.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.620+0000 7f0b9c15d640 1 -- 172.21.15.153:0/2547215388 wait complete. 2024-09-15T03:50:23.631 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":908,"stamp":"2024-09-15T03:50:22.114741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1809672,"kb_used_data":8896,"kb_used_omap":9,"kb_used_meta":1800694,"kb_avail":564421368,"statfs":{"total":579820584960,"available":577967480832,"internally_reserved":0,"allocated":9109504,"data_stored":7285462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843910672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":171,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253295+0000","last_change":"2024-09-15T03:35:36.177407+0000","last_active":"2024-09-15T03:38:16.253295+0000","last_peered":"2024-09-15T03:38:16.253295+0000","last_clean":"2024-09-15T03:38:16.253295+0000","last_became_active":"2024-09-15T03:35:36.177073+0000","last_became_peered":"2024-09-15T03:35:36.177073+0000","last_unstale":"2024-09-15T03:38:16.253295+0000","last_undegraded":"2024-09-15T03:38:16.253295+0000","last_fullsized":"2024-09-15T03:38:16.253295+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:01:53.230196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412884+0000","last_change":"2024-09-15T03:35:36.171117+0000","last_active":"2024-09-15T03:38:15.412884+0000","last_peered":"2024-09-15T03:38:15.412884+0000","last_clean":"2024-09-15T03:38:15.412884+0000","last_became_active":"2024-09-15T03:35:36.170838+0000","last_became_peered":"2024-09-15T03:35:36.170838+0000","last_unstale":"2024-09-15T03:38:15.412884+0000","last_undegraded":"2024-09-15T03:38:15.412884+0000","last_fullsized":"2024-09-15T03:38:15.412884+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:12:13.052343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":146,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866090+0000","last_change":"2024-09-15T03:35:36.180940+0000","last_active":"2024-09-15T03:38:16.866090+0000","last_peered":"2024-09-15T03:38:16.866090+0000","last_clean":"2024-09-15T03:38:16.866090+0000","last_became_active":"2024-09-15T03:35:36.180590+0000","last_became_peered":"2024-09-15T03:35:36.180590+0000","last_unstale":"2024-09-15T03:38:16.866090+0000","last_undegraded":"2024-09-15T03:38:16.866090+0000","last_fullsized":"2024-09-15T03:38:16.866090+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:06.005311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412806+0000","last_change":"2024-09-15T03:35:36.173953+0000","last_active":"2024-09-15T03:38:15.412806+0000","last_peered":"2024-09-15T03:38:15.412806+0000","last_clean":"2024-09-15T03:38:15.412806+0000","last_became_active":"2024-09-15T03:35:36.173628+0000","last_became_peered":"2024-09-15T03:35:36.173628+0000","last_unstale":"2024-09-15T03:38:15.412806+0000","last_undegraded":"2024-09-15T03:38:15.412806+0000","last_fullsized":"2024-09-15T03:38:15.412806+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:56:12.188821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":131,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412847+0000","last_change":"2024-09-15T03:35:42.048298+0000","last_active":"2024-09-15T03:38:15.412847+0000","last_peered":"2024-09-15T03:38:15.412847+0000","last_clean":"2024-09-15T03:38:15.412847+0000","last_became_active":"2024-09-15T03:35:42.047949+0000","last_became_peered":"2024-09-15T03:35:42.047949+0000","last_unstale":"2024-09-15T03:38:15.412847+0000","last_undegraded":"2024-09-15T03:38:15.412847+0000","last_fullsized":"2024-09-15T03:38:15.412847+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:49:06.838016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":147,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253319+0000","last_change":"2024-09-15T03:35:36.179570+0000","last_active":"2024-09-15T03:38:16.253319+0000","last_peered":"2024-09-15T03:38:16.253319+0000","last_clean":"2024-09-15T03:38:16.253319+0000","last_became_active":"2024-09-15T03:35:36.179302+0000","last_became_peered":"2024-09-15T03:35:36.179302+0000","last_unstale":"2024-09-15T03:38:16.253319+0000","last_undegraded":"2024-09-15T03:38:16.253319+0000","last_fullsized":"2024-09-15T03:38:16.253319+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:26:04.635839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411506+0000","last_change":"2024-09-15T03:35:36.179910+0000","last_active":"2024-09-15T03:38:15.411506+0000","last_peered":"2024-09-15T03:38:15.411506+0000","last_clean":"2024-09-15T03:38:15.411506+0000","last_became_active":"2024-09-15T03:35:36.179540+0000","last_became_peered":"2024-09-15T03:35:36.179540+0000","last_unstale":"2024-09-15T03:38:15.411506+0000","last_undegraded":"2024-09-15T03:38:15.411506+0000","last_fullsized":"2024-09-15T03:38:15.411506+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:29.604427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510296+0000","last_change":"2024-09-15T03:35:36.176521+0000","last_active":"2024-09-15T03:38:16.510296+0000","last_peered":"2024-09-15T03:38:16.510296+0000","last_clean":"2024-09-15T03:38:16.510296+0000","last_became_active":"2024-09-15T03:35:36.176169+0000","last_became_peered":"2024-09-15T03:35:36.176169+0000","last_unstale":"2024-09-15T03:38:16.510296+0000","last_undegraded":"2024-09-15T03:38:16.510296+0000","last_fullsized":"2024-09-15T03:38:16.510296+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:17:56.952574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":114,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412674+0000","last_change":"2024-09-15T03:35:36.176184+0000","last_active":"2024-09-15T03:38:15.412674+0000","last_peered":"2024-09-15T03:38:15.412674+0000","last_clean":"2024-09-15T03:38:15.412674+0000","last_became_active":"2024-09-15T03:35:36.175786+0000","last_became_peered":"2024-09-15T03:35:36.175786+0000","last_unstale":"2024-09-15T03:38:15.412674+0000","last_undegraded":"2024-09-15T03:38:15.412674+0000","last_fullsized":"2024-09-15T03:38:15.412674+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:57.476041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":109,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411493+0000","last_change":"2024-09-15T03:35:36.174198+0000","last_active":"2024-09-15T03:38:15.411493+0000","last_peered":"2024-09-15T03:38:15.411493+0000","last_clean":"2024-09-15T03:38:15.411493+0000","last_became_active":"2024-09-15T03:35:36.173941+0000","last_became_peered":"2024-09-15T03:35:36.173941+0000","last_unstale":"2024-09-15T03:38:15.411493+0000","last_undegraded":"2024-09-15T03:38:15.411493+0000","last_fullsized":"2024-09-15T03:38:15.411493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.690762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":122,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866041+0000","last_change":"2024-09-15T03:35:36.173396+0000","last_active":"2024-09-15T03:38:16.866041+0000","last_peered":"2024-09-15T03:38:16.866041+0000","last_clean":"2024-09-15T03:38:16.866041+0000","last_became_active":"2024-09-15T03:35:36.173027+0000","last_became_peered":"2024-09-15T03:35:36.173027+0000","last_unstale":"2024-09-15T03:38:16.866041+0000","last_undegraded":"2024-09-15T03:38:16.866041+0000","last_fullsized":"2024-09-15T03:38:16.866041+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:28:33.565728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510475+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510475+0000","last_peered":"2024-09-15T03:38:16.510475+0000","last_clean":"2024-09-15T03:38:16.510475+0000","last_became_active":"2024-09-15T03:35:36.176264+0000","last_became_peered":"2024-09-15T03:35:36.176264+0000","last_unstale":"2024-09-15T03:38:16.510475+0000","last_undegraded":"2024-09-15T03:38:16.510475+0000","last_fullsized":"2024-09-15T03:38:16.510475+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510499+0000","last_change":"2024-09-15T03:35:36.179783+0000","last_active":"2024-09-15T03:38:16.510499+0000","last_peered":"2024-09-15T03:38:16.510499+0000","last_clean":"2024-09-15T03:38:16.510499+0000","last_became_active":"2024-09-15T03:35:36.179473+0000","last_became_peered":"2024-09-15T03:35:36.179473+0000","last_unstale":"2024-09-15T03:38:16.510499+0000","last_undegraded":"2024-09-15T03:38:16.510499+0000","last_fullsized":"2024-09-15T03:38:16.510499+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:11.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":178,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253392+0000","last_change":"2024-09-15T03:35:36.179775+0000","last_active":"2024-09-15T03:38:16.253392+0000","last_peered":"2024-09-15T03:38:16.253392+0000","last_clean":"2024-09-15T03:38:16.253392+0000","last_became_active":"2024-09-15T03:35:36.179476+0000","last_became_peered":"2024-09-15T03:35:36.179476+0000","last_unstale":"2024-09-15T03:38:16.253392+0000","last_undegraded":"2024-09-15T03:38:16.253392+0000","last_fullsized":"2024-09-15T03:38:16.253392+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:33.988609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866135+0000","last_change":"2024-09-15T03:35:41.930950+0000","last_active":"2024-09-15T03:38:16.866135+0000","last_peered":"2024-09-15T03:38:16.866135+0000","last_clean":"2024-09-15T03:38:16.866135+0000","last_became_active":"2024-09-15T03:35:41.930552+0000","last_became_peered":"2024-09-15T03:35:41.930552+0000","last_unstale":"2024-09-15T03:38:16.866135+0000","last_undegraded":"2024-09-15T03:38:16.866135+0000","last_fullsized":"2024-09-15T03:38:16.866135+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.602540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510466+0000","last_change":"2024-09-15T03:35:36.176671+0000","last_active":"2024-09-15T03:38:16.510466+0000","last_peered":"2024-09-15T03:38:16.510466+0000","last_clean":"2024-09-15T03:38:16.510466+0000","last_became_active":"2024-09-15T03:35:36.176131+0000","last_became_peered":"2024-09-15T03:35:36.176131+0000","last_unstale":"2024-09-15T03:38:16.510466+0000","last_undegraded":"2024-09-15T03:38:16.510466+0000","last_fullsized":"2024-09-15T03:38:16.510466+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.630912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":332,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411395+0000","last_change":"2024-09-15T03:35:36.180042+0000","last_active":"2024-09-15T03:38:15.411395+0000","last_peered":"2024-09-15T03:38:15.411395+0000","last_clean":"2024-09-15T03:38:15.411395+0000","last_became_active":"2024-09-15T03:35:36.179501+0000","last_became_peered":"2024-09-15T03:35:36.179501+0000","last_unstale":"2024-09-15T03:38:15.411395+0000","last_undegraded":"2024-09-15T03:38:15.411395+0000","last_fullsized":"2024-09-15T03:38:15.411395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:32:25.282577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":254,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412719+0000","last_change":"2024-09-15T03:35:36.182382+0000","last_active":"2024-09-15T03:38:15.412719+0000","last_peered":"2024-09-15T03:38:15.412719+0000","last_clean":"2024-09-15T03:38:15.412719+0000","last_became_active":"2024-09-15T03:35:36.175947+0000","last_became_peered":"2024-09-15T03:35:36.175947+0000","last_unstale":"2024-09-15T03:38:15.412719+0000","last_undegraded":"2024-09-15T03:38:15.412719+0000","last_fullsized":"2024-09-15T03:38:15.412719+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:04:51.063277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411377+0000","last_change":"2024-09-15T03:35:36.177331+0000","last_active":"2024-09-15T03:38:15.411377+0000","last_peered":"2024-09-15T03:38:15.411377+0000","last_clean":"2024-09-15T03:38:15.411377+0000","last_became_active":"2024-09-15T03:35:36.177043+0000","last_became_peered":"2024-09-15T03:35:36.177043+0000","last_unstale":"2024-09-15T03:38:15.411377+0000","last_undegraded":"2024-09-15T03:38:15.411377+0000","last_fullsized":"2024-09-15T03:38:15.411377+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:52:08.123750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866082+0000","last_change":"2024-09-15T03:35:36.177279+0000","last_active":"2024-09-15T03:38:16.866082+0000","last_peered":"2024-09-15T03:38:16.866082+0000","last_clean":"2024-09-15T03:38:16.866082+0000","last_became_active":"2024-09-15T03:35:36.176997+0000","last_became_peered":"2024-09-15T03:35:36.176997+0000","last_unstale":"2024-09-15T03:38:16.866082+0000","last_undegraded":"2024-09-15T03:38:16.866082+0000","last_fullsized":"2024-09-15T03:38:16.866082+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:05:52.287110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":251,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412676+0000","last_change":"2024-09-15T03:35:36.189329+0000","last_active":"2024-09-15T03:38:15.412676+0000","last_peered":"2024-09-15T03:38:15.412676+0000","last_clean":"2024-09-15T03:38:15.412676+0000","last_became_active":"2024-09-15T03:35:36.176519+0000","last_became_peered":"2024-09-15T03:35:36.176519+0000","last_unstale":"2024-09-15T03:38:15.412676+0000","last_undegraded":"2024-09-15T03:38:15.412676+0000","last_fullsized":"2024-09-15T03:38:15.412676+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:40:12.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510504+0000","last_change":"2024-09-15T03:35:42.047866+0000","last_active":"2024-09-15T03:38:16.510504+0000","last_peered":"2024-09-15T03:38:16.510504+0000","last_clean":"2024-09-15T03:38:16.510504+0000","last_became_active":"2024-09-15T03:35:42.047571+0000","last_became_peered":"2024-09-15T03:35:42.047571+0000","last_unstale":"2024-09-15T03:38:16.510504+0000","last_undegraded":"2024-09-15T03:38:16.510504+0000","last_fullsized":"2024-09-15T03:38:16.510504+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:44.835315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411503+0000","last_change":"2024-09-15T03:35:36.180602+0000","last_active":"2024-09-15T03:38:15.411503+0000","last_peered":"2024-09-15T03:38:15.411503+0000","last_clean":"2024-09-15T03:38:15.411503+0000","last_became_active":"2024-09-15T03:35:36.180366+0000","last_became_peered":"2024-09-15T03:35:36.180366+0000","last_unstale":"2024-09-15T03:38:15.411503+0000","last_undegraded":"2024-09-15T03:38:15.411503+0000","last_fullsized":"2024-09-15T03:38:15.411503+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:51:20.797295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510310+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510310+0000","last_peered":"2024-09-15T03:38:16.510310+0000","last_clean":"2024-09-15T03:38:16.510310+0000","last_became_active":"2024-09-15T03:35:36.176265+0000","last_became_peered":"2024-09-15T03:35:36.176265+0000","last_unstale":"2024-09-15T03:38:16.510310+0000","last_undegraded":"2024-09-15T03:38:16.510310+0000","last_fullsized":"2024-09-15T03:38:16.510310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:07:54.095941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411431+0000","last_change":"2024-09-15T03:35:36.176300+0000","last_active":"2024-09-15T03:38:15.411431+0000","last_peered":"2024-09-15T03:38:15.411431+0000","last_clean":"2024-09-15T03:38:15.411431+0000","last_became_active":"2024-09-15T03:35:36.175986+0000","last_became_peered":"2024-09-15T03:35:36.175986+0000","last_unstale":"2024-09-15T03:38:15.411431+0000","last_undegraded":"2024-09-15T03:38:15.411431+0000","last_fullsized":"2024-09-15T03:38:15.411431+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:58:16.529865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63700000000000001,"5min":0.873,"15min":0.873},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59599999999999997,"5min":0.68400000000000005,"15min":0.751},"last":0.38200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.505},"min":{"1min":0.379,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73799999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.33800000000000002},"max":{"1min":0.60899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71199999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.74199999999999999},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.248,"5min":0.247,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.62},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.379},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":0.45200000000000001,"5min":0.63100000000000001,"15min":0.64400000000000002},"last":0.26800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.39000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.64400000000000002},"last":0.316},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.42199999999999999},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.77000000000000002},"last":0.42899999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.69099999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.499},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.64100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":0.59899999999999998,"5min":0.67700000000000005,"15min":0.70499999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.32800000000000001},"max":{"1min":0.629,"5min":0.66000000000000003,"15min":0.71199999999999997},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34599999999999997},"max":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.72199999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35299999999999998,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.72299999999999998},"last":0.53900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.627},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22700000000000001},"max":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.60099999999999998,"5min":0.63,"15min":0.63},"last":0.34699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640520,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51400000000000001,"15min":0.48999999999999999},"min":{"1min":0.35699999999999998,"5min":0.31,"15min":0.28000000000000003},"max":{"1min":0.68400000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54200000000000004,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.28699999999999998,"15min":0.17799999999999999},"max":{"1min":0.66400000000000003,"5min":0.76000000000000001,"15min":0.79700000000000004},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52600000000000002,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.29699999999999999},"max":{"1min":0.68300000000000005,"5min":0.755,"15min":0.75700000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.505,"15min":0.51400000000000001},"min":{"1min":0.34000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.71899999999999997,"15min":1.597},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.748,"15min":0.84199999999999997},"last":0.48899999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19600000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.65400000000000003},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.222},"max":{"1min":0.54300000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34399999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.25,"5min":0.23899999999999999,"15min":0.221},"max":{"1min":0.54700000000000004,"5min":0.63200000000000001,"15min":0.65500000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.44800000000000001,"15min":0.42699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":0.57399999999999995,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.31900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31,"15min":0.308},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.58799999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.72699999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":0.48699999999999999,"5min":0.51200000000000001,"15min":0.59599999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.22},"max":{"1min":0.50600000000000001,"5min":0.57499999999999996,"15min":0.68500000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.35299999999999998,"15min":0.26700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56100000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.314},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":5.6269999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.29199999999999998},"max":{"1min":0.65800000000000003,"5min":0.77500000000000002,"15min":0.84999999999999998},"last":0.5},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72799999999999998,"5min":0.77300000000000002,"15min":0.82599999999999996},"last":0.52200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640519,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32500000000000001,"15min":0.32700000000000001},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":0.41999999999999998,"5min":0.52800000000000002,"15min":0.874},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.48999999999999999,"15min":0.55900000000000005},"last":0.26700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.215},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.77800000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.50900000000000001,"15min":0.747},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.73199999999999998,"15min":0.89800000000000002},"last":0.5},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.88300000000000001},"last":0.432},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.76600000000000001,"15min":0.81799999999999995},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.93799999999999994},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.432,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.91300000000000003},"last":0.45300000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640520,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32200000000000001,"15min":0.32000000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":0.55000000000000004,"15min":0.625},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33300000000000002,"15min":0.33400000000000002},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":0.52400000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.219},"max":{"1min":0.47499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.28899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.433},{"interface":"front","average":{"1min":0.499,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":0.85699999999999998},"last":0.378}]},{"osd":4,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.29699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.34899999999999998,"15min":0.34200000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.81200000000000006},"last":0.44700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":0.69999999999999996,"5min":0.76000000000000001,"15min":0.78000000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.84099999999999997,"15min":0.84799999999999998},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:50:23.634 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:50:23.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:50:23.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.673+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.673+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999216378}) 2024-09-15T03:50:23.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.673+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220938007}) 2024-09-15T03:50:23.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.673+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.673+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999216378}) 2024-09-15T03:50:23.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.673+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019100609}) 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486749860}) 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602234173}) 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486749860}) 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=985153239}) 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830624418}) 2024-09-15T03:50:23.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692529984}) 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830624418}) 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.674+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724797639}) 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779336780}) 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2498163717}) 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779336780}) 2024-09-15T03:50:23.680 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3350259933}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448185586}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.675+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2615975091}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448185586}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190848407}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208703512}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064354098}) 2024-09-15T03:50:23.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208703512}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.676+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352103347}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900129856}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753709459}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900129856}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928339471}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413350088}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4259928312}) 2024-09-15T03:50:23.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413350088}) 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.677+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1510086492}) 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.678+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.678+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2883215054}) 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.678+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3251181909}) 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.678+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.678+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2883215054}) 2024-09-15T03:50:23.683 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.678+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2368969287}) 2024-09-15T03:50:23.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487711140}) 2024-09-15T03:50:23.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=788806491}) 2024-09-15T03:50:23.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487711140}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3981769866}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826835957}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422956507}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826835957}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.679+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1356811051}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.680+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.680+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103104409}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.680+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=953288304}) 2024-09-15T03:50:23.694 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.680+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.680+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103104409}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.680+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492466419}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.681+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.681+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454211288}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.681+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=411762022}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.681+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.681+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454211288}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.681+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1246395656}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599804959}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769383312}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599804959}) 2024-09-15T03:50:23.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638272734}) 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810179011}) 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3746860366}) 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810179011}) 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.682+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4238888227}) 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.683+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.683+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157665438}) 2024-09-15T03:50:23.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.683+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096572839}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.683+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.683+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157665438}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.683+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460342066}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707571275}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=624905664}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707571275}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1551571913}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417231593}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043075038}) 2024-09-15T03:50:23.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417231593}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.684+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765704169}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.685+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.685+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754456763}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.685+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4171320716}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.685+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.685+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754456763}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.685+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=238538276}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=199108134}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655078739}) 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=199108134}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3471870902}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.686+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867714074}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3263084004}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867714074}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1326466114}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784743943}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286384578}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784743943}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.687+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404250858}) 2024-09-15T03:50:23.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.688+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.688+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507274602}) 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.688+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=524820363}) 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.688+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.688+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507274602}) 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.688+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152663663}) 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918566980}) 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3187710742}) 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918566980}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1387753360}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715936315}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.689+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1448807293}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715936315}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2085042957}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989881142}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910792440}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989881142}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.690+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562814691}) 2024-09-15T03:50:23.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.691+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.691+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1674083595}) 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.691+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052149321}) 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.691+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.691+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1674083595}) 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.691+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3031935902}) 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141904987}) 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2114482510}) 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141904987}) 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3634161032}) 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111206939}) 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738477011}) 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111206939}) 2024-09-15T03:50:23.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.692+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855159241}) 2024-09-15T03:50:23.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.693+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.705 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.693+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556812598}) 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.693+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570324323}) 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.693+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.693+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556812598}) 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.693+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679194992}) 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3672972163}) 2024-09-15T03:50:23.706 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3901682311}) 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3672972163}) 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550408016}) 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795113512}) 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453591198}) 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795113512}) 2024-09-15T03:50:23.707 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.694+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3812004640}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.695+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.695+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042166162}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.695+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4195463667}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.695+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.695+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042166162}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.695+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563565482}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.696+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.696+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387286815}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.696+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2006304124}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.696+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.696+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387286815}) 2024-09-15T03:50:23.708 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:23.696+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1189662047}) 2024-09-15T03:50:23.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.730+0000 7f9da1d5d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:23.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.730+0000 7f9da1d5d640 10 monclient: build_initial_monmap 2024-09-15T03:50:23.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.730+0000 7f9da1d5d640 10 monclient: monmap: 2024-09-15T03:50:23.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:23.732 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.732 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:23.731194+0000 2024-09-15T03:50:23.732 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:23.731194+0000 2024-09-15T03:50:23.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:23.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.733 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:23.733 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:23.733 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:23.733 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:23.733 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:23.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 1 Processor -- start 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 1 -- start start 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient: init 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d9c131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d9c09fda0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d9c132290 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c0a1180 con 0x7f9d9c09fda0 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c0a1460 con 0x7f9d9c132290 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c09f0d0 con 0x7f9d9c131980 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9da1d5d640 10 monclient(hunting): authenticate will time out at 2021.370682s 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9d9c09fda0 0x7f9d9c09e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40256/0 (socket says 172.21.15.153:40256) 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9affd640 1 -- 172.21.15.153:0/490247903 learned_addr learned my addr 172.21.15.153:0/490247903 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 1 --2- 172.21.15.153:0/490247903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): get_auth_request con 0x7f9d9c131980 auth_method 0 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.731+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_done global_id 244367 payload 274 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 1 -- 172.21.15.153:0/490247903 >> v1:172.21.15.181:6789/0 conn(0x7f9d9c132290 legacy=0x7f9d9c138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 1 -- 172.21.15.153:0/490247903 >> v1:172.21.15.153:6789/0 conn(0x7f9d9c09fda0 legacy=0x7f9d9c09e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.732+0000 7f9d9b7fe640 1 -- 172.21.15.153:0/490247903 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c09f210 con 0x7f9d9c131980 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.734+0000 7f9d9b7fe640 1 -- 172.21.15.153:0/490247903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 msgr2=0x7f9d9c131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:23.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.734+0000 7f9d9b7fe640 1 --2- 172.21.15.153:0/490247903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c131d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009880 tx=0x7f9d8c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ba0d0a34742650 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 1 -- 172.21.15.153:0/490247903 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c032440 con 0x7f9d9c131980 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: dump: 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 5 monclient: authenticate success, global_id 244367 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 1 -- 172.21.15.153:0/490247903 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c0325a0 con 0x7f9d9c131980 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 1 -- 172.21.15.153:0/490247903 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c032940 con 0x7f9d9c131980 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 4 set_mon_vals no callback set 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9d9a7fc640 10 monclient: dump: 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 10 monclient: shutdown 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 -- 172.21.15.153:0/490247903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 msgr2=0x7f9d9c131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/490247903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c131d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009880 tx=0x7f9d8c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 -- 172.21.15.153:0/490247903 shutdown_connections 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/490247903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c131d50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 -- 172.21.15.153:0/490247903 >> 172.21.15.153:0/490247903 conn(0x7f9d9c067bd0 msgr2=0x7f9d9c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 -- 172.21.15.153:0/490247903 shutdown_connections 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.735+0000 7f9da1d5d640 1 -- 172.21.15.153:0/490247903 wait complete. 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient: build_initial_monmap 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient: monmap: 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:23.744 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 Processor -- start 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 -- start start 2024-09-15T03:50:23.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient: init 2024-09-15T03:50:23.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:23.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:50:23.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 0x7f9d9c138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d9c09fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c134050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d9c131980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d9c132290 0x7f9d9c134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d9c132290 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1354c0 con 0x7f9d9c09fda0 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c139880 con 0x7f9d9c132290 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c137820 con 0x7f9d9c131980 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.736+0000 7f9da1d5d640 10 monclient(hunting): authenticate will time out at 2021.375987s 2024-09-15T03:50:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c134050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 0x7f9d9c138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c134050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41502/0 (socket says 172.21.15.153:41502) 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 0x7f9d9c138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33824/0 (socket says 172.21.15.153:33824) 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 -- 172.21.15.153:0/1935540973 learned_addr learned my addr 172.21.15.153:0/1935540973 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): get_auth_request con 0x7f9d9c131980 auth_method 0 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): get_auth_request con 0x7f9d9c09fda0 auth_method 0 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d9c132290 0x7f9d9c134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): get_auth_request con 0x7f9d9c132290 auth_method 0 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:23.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient(hunting): handle_auth_done global_id 244370 payload 824 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 -- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d9c132290 msgr2=0x7f9d9c134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d9c132290 0x7f9d9c134590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 -- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 msgr2=0x7f9d9c138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 0x7f9d9c138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 monclient: found mon.noname-c 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9affd640 1 -- 172.21.15.153:0/1935540973 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c009390 con 0x7f9d9c131980 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.737+0000 7f9d9b7fe640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 0x7f9d9c138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d9affd640 1 -- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 msgr2=0x7f9d9c134050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d9affd640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c134050 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000c5e0 tx=0x7f9d90004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90005cb0 con 0x7f9d9c131980 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: dump: 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.755 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90005e30 con 0x7f9d9c131980 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 5 monclient: authenticate success, global_id 244370 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90007340 con 0x7f9d9c131980 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 10 monclient: _renew_subs 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c135a80 con 0x7f9d9c131980 2024-09-15T03:50:23.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d837fe640 10 monclient: dump: 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9d99ffb640 4 set_mon_vals no callback set 2024-09-15T03:50:23.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 10 monclient: _renew_subs 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.739+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c136780 con 0x7f9d9c131980 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f9d9001b400 con 0x7f9d9c131980 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9da1d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9da1d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9da1d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c04bf60 con 0x7f9d9c131980 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9d837fe640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d88038790 0x7f9d8803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.740+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9d9004c180 con 0x7f9d9c131980 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.741+0000 7f9d9b7fe640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d88038790 0x7f9d8803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.741+0000 7f9d9b7fe640 10 monclient: get_auth_request con 0x7f9d88038790 auth_method 0 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.743+0000 7f9d9b7fe640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d88038790 0x7f9d8803ac40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0a4f60 tx=0x7f9d8c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.745+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9d9001b6b0 con 0x7f9d9c131980 2024-09-15T03:50:23.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.745+0000 7f9d837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:23.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.745+0000 7f9d837fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:23.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.900+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d9c0638d0 con 0x7f9d88038790 2024-09-15T03:50:23.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.903+0000 7f9d837fe640 1 -- 172.21.15.153:0/1935540973 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98135 (secure 0 0 0) 0x7f9d9c0638d0 con 0x7f9d88038790 2024-09-15T03:50:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:23.905 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:50:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d88038790 msgr2=0x7f9d8803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d88038790 0x7f9d8803ac40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0a4f60 tx=0x7f9d8c032000 comp rx=0 tx=0).stop 2024-09-15T03:50:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 10 monclient: shutdown 2024-09-15T03:50:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:23.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 msgr2=0x7f9d9c134050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c134050 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000c5e0 tx=0x7f9d90004960 comp rx=0 tx=0).stop 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 shutdown_connections 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f9d88038790 0x7f9d8803ac40 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d9c132290 0x7f9d9c134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d9c131980 0x7f9d9c134050 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 --2- 172.21.15.153:0/1935540973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d9c09fda0 0x7f9d9c138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 >> 172.21.15.153:0/1935540973 conn(0x7f9d9c067bd0 msgr2=0x7f9d9c09d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 shutdown_connections 2024-09-15T03:50:23.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:23.905+0000 7f9da1d5d640 1 -- 172.21.15.153:0/1935540973 wait complete. 2024-09-15T03:50:23.916 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":908,"stamp":"2024-09-15T03:50:22.114741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1809672,"kb_used_data":8896,"kb_used_omap":9,"kb_used_meta":1800694,"kb_avail":564421368,"statfs":{"total":579820584960,"available":577967480832,"internally_reserved":0,"allocated":9109504,"data_stored":7285462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843910672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":171,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253295+0000","last_change":"2024-09-15T03:35:36.177407+0000","last_active":"2024-09-15T03:38:16.253295+0000","last_peered":"2024-09-15T03:38:16.253295+0000","last_clean":"2024-09-15T03:38:16.253295+0000","last_became_active":"2024-09-15T03:35:36.177073+0000","last_became_peered":"2024-09-15T03:35:36.177073+0000","last_unstale":"2024-09-15T03:38:16.253295+0000","last_undegraded":"2024-09-15T03:38:16.253295+0000","last_fullsized":"2024-09-15T03:38:16.253295+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:01:53.230196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412884+0000","last_change":"2024-09-15T03:35:36.171117+0000","last_active":"2024-09-15T03:38:15.412884+0000","last_peered":"2024-09-15T03:38:15.412884+0000","last_clean":"2024-09-15T03:38:15.412884+0000","last_became_active":"2024-09-15T03:35:36.170838+0000","last_became_peered":"2024-09-15T03:35:36.170838+0000","last_unstale":"2024-09-15T03:38:15.412884+0000","last_undegraded":"2024-09-15T03:38:15.412884+0000","last_fullsized":"2024-09-15T03:38:15.412884+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:12:13.052343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":146,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866090+0000","last_change":"2024-09-15T03:35:36.180940+0000","last_active":"2024-09-15T03:38:16.866090+0000","last_peered":"2024-09-15T03:38:16.866090+0000","last_clean":"2024-09-15T03:38:16.866090+0000","last_became_active":"2024-09-15T03:35:36.180590+0000","last_became_peered":"2024-09-15T03:35:36.180590+0000","last_unstale":"2024-09-15T03:38:16.866090+0000","last_undegraded":"2024-09-15T03:38:16.866090+0000","last_fullsized":"2024-09-15T03:38:16.866090+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:06.005311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412806+0000","last_change":"2024-09-15T03:35:36.173953+0000","last_active":"2024-09-15T03:38:15.412806+0000","last_peered":"2024-09-15T03:38:15.412806+0000","last_clean":"2024-09-15T03:38:15.412806+0000","last_became_active":"2024-09-15T03:35:36.173628+0000","last_became_peered":"2024-09-15T03:35:36.173628+0000","last_unstale":"2024-09-15T03:38:15.412806+0000","last_undegraded":"2024-09-15T03:38:15.412806+0000","last_fullsized":"2024-09-15T03:38:15.412806+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:56:12.188821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":131,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412847+0000","last_change":"2024-09-15T03:35:42.048298+0000","last_active":"2024-09-15T03:38:15.412847+0000","last_peered":"2024-09-15T03:38:15.412847+0000","last_clean":"2024-09-15T03:38:15.412847+0000","last_became_active":"2024-09-15T03:35:42.047949+0000","last_became_peered":"2024-09-15T03:35:42.047949+0000","last_unstale":"2024-09-15T03:38:15.412847+0000","last_undegraded":"2024-09-15T03:38:15.412847+0000","last_fullsized":"2024-09-15T03:38:15.412847+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:49:06.838016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":147,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253319+0000","last_change":"2024-09-15T03:35:36.179570+0000","last_active":"2024-09-15T03:38:16.253319+0000","last_peered":"2024-09-15T03:38:16.253319+0000","last_clean":"2024-09-15T03:38:16.253319+0000","last_became_active":"2024-09-15T03:35:36.179302+0000","last_became_peered":"2024-09-15T03:35:36.179302+0000","last_unstale":"2024-09-15T03:38:16.253319+0000","last_undegraded":"2024-09-15T03:38:16.253319+0000","last_fullsized":"2024-09-15T03:38:16.253319+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:26:04.635839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411506+0000","last_change":"2024-09-15T03:35:36.179910+0000","last_active":"2024-09-15T03:38:15.411506+0000","last_peered":"2024-09-15T03:38:15.411506+0000","last_clean":"2024-09-15T03:38:15.411506+0000","last_became_active":"2024-09-15T03:35:36.179540+0000","last_became_peered":"2024-09-15T03:35:36.179540+0000","last_unstale":"2024-09-15T03:38:15.411506+0000","last_undegraded":"2024-09-15T03:38:15.411506+0000","last_fullsized":"2024-09-15T03:38:15.411506+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:29.604427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510296+0000","last_change":"2024-09-15T03:35:36.176521+0000","last_active":"2024-09-15T03:38:16.510296+0000","last_peered":"2024-09-15T03:38:16.510296+0000","last_clean":"2024-09-15T03:38:16.510296+0000","last_became_active":"2024-09-15T03:35:36.176169+0000","last_became_peered":"2024-09-15T03:35:36.176169+0000","last_unstale":"2024-09-15T03:38:16.510296+0000","last_undegraded":"2024-09-15T03:38:16.510296+0000","last_fullsized":"2024-09-15T03:38:16.510296+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:17:56.952574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":114,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412674+0000","last_change":"2024-09-15T03:35:36.176184+0000","last_active":"2024-09-15T03:38:15.412674+0000","last_peered":"2024-09-15T03:38:15.412674+0000","last_clean":"2024-09-15T03:38:15.412674+0000","last_became_active":"2024-09-15T03:35:36.175786+0000","last_became_peered":"2024-09-15T03:35:36.175786+0000","last_unstale":"2024-09-15T03:38:15.412674+0000","last_undegraded":"2024-09-15T03:38:15.412674+0000","last_fullsized":"2024-09-15T03:38:15.412674+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:57.476041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":109,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411493+0000","last_change":"2024-09-15T03:35:36.174198+0000","last_active":"2024-09-15T03:38:15.411493+0000","last_peered":"2024-09-15T03:38:15.411493+0000","last_clean":"2024-09-15T03:38:15.411493+0000","last_became_active":"2024-09-15T03:35:36.173941+0000","last_became_peered":"2024-09-15T03:35:36.173941+0000","last_unstale":"2024-09-15T03:38:15.411493+0000","last_undegraded":"2024-09-15T03:38:15.411493+0000","last_fullsized":"2024-09-15T03:38:15.411493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.690762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":122,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866041+0000","last_change":"2024-09-15T03:35:36.173396+0000","last_active":"2024-09-15T03:38:16.866041+0000","last_peered":"2024-09-15T03:38:16.866041+0000","last_clean":"2024-09-15T03:38:16.866041+0000","last_became_active":"2024-09-15T03:35:36.173027+0000","last_became_peered":"2024-09-15T03:35:36.173027+0000","last_unstale":"2024-09-15T03:38:16.866041+0000","last_undegraded":"2024-09-15T03:38:16.866041+0000","last_fullsized":"2024-09-15T03:38:16.866041+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:28:33.565728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510475+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510475+0000","last_peered":"2024-09-15T03:38:16.510475+0000","last_clean":"2024-09-15T03:38:16.510475+0000","last_became_active":"2024-09-15T03:35:36.176264+0000","last_became_peered":"2024-09-15T03:35:36.176264+0000","last_unstale":"2024-09-15T03:38:16.510475+0000","last_undegraded":"2024-09-15T03:38:16.510475+0000","last_fullsized":"2024-09-15T03:38:16.510475+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510499+0000","last_change":"2024-09-15T03:35:36.179783+0000","last_active":"2024-09-15T03:38:16.510499+0000","last_peered":"2024-09-15T03:38:16.510499+0000","last_clean":"2024-09-15T03:38:16.510499+0000","last_became_active":"2024-09-15T03:35:36.179473+0000","last_became_peered":"2024-09-15T03:35:36.179473+0000","last_unstale":"2024-09-15T03:38:16.510499+0000","last_undegraded":"2024-09-15T03:38:16.510499+0000","last_fullsized":"2024-09-15T03:38:16.510499+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:11.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":178,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253392+0000","last_change":"2024-09-15T03:35:36.179775+0000","last_active":"2024-09-15T03:38:16.253392+0000","last_peered":"2024-09-15T03:38:16.253392+0000","last_clean":"2024-09-15T03:38:16.253392+0000","last_became_active":"2024-09-15T03:35:36.179476+0000","last_became_peered":"2024-09-15T03:35:36.179476+0000","last_unstale":"2024-09-15T03:38:16.253392+0000","last_undegraded":"2024-09-15T03:38:16.253392+0000","last_fullsized":"2024-09-15T03:38:16.253392+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:33.988609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866135+0000","last_change":"2024-09-15T03:35:41.930950+0000","last_active":"2024-09-15T03:38:16.866135+0000","last_peered":"2024-09-15T03:38:16.866135+0000","last_clean":"2024-09-15T03:38:16.866135+0000","last_became_active":"2024-09-15T03:35:41.930552+0000","last_became_peered":"2024-09-15T03:35:41.930552+0000","last_unstale":"2024-09-15T03:38:16.866135+0000","last_undegraded":"2024-09-15T03:38:16.866135+0000","last_fullsized":"2024-09-15T03:38:16.866135+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.602540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510466+0000","last_change":"2024-09-15T03:35:36.176671+0000","last_active":"2024-09-15T03:38:16.510466+0000","last_peered":"2024-09-15T03:38:16.510466+0000","last_clean":"2024-09-15T03:38:16.510466+0000","last_became_active":"2024-09-15T03:35:36.176131+0000","last_became_peered":"2024-09-15T03:35:36.176131+0000","last_unstale":"2024-09-15T03:38:16.510466+0000","last_undegraded":"2024-09-15T03:38:16.510466+0000","last_fullsized":"2024-09-15T03:38:16.510466+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.630912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":332,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411395+0000","last_change":"2024-09-15T03:35:36.180042+0000","last_active":"2024-09-15T03:38:15.411395+0000","last_peered":"2024-09-15T03:38:15.411395+0000","last_clean":"2024-09-15T03:38:15.411395+0000","last_became_active":"2024-09-15T03:35:36.179501+0000","last_became_peered":"2024-09-15T03:35:36.179501+0000","last_unstale":"2024-09-15T03:38:15.411395+0000","last_undegraded":"2024-09-15T03:38:15.411395+0000","last_fullsized":"2024-09-15T03:38:15.411395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:32:25.282577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":254,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412719+0000","last_change":"2024-09-15T03:35:36.182382+0000","last_active":"2024-09-15T03:38:15.412719+0000","last_peered":"2024-09-15T03:38:15.412719+0000","last_clean":"2024-09-15T03:38:15.412719+0000","last_became_active":"2024-09-15T03:35:36.175947+0000","last_became_peered":"2024-09-15T03:35:36.175947+0000","last_unstale":"2024-09-15T03:38:15.412719+0000","last_undegraded":"2024-09-15T03:38:15.412719+0000","last_fullsized":"2024-09-15T03:38:15.412719+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:04:51.063277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411377+0000","last_change":"2024-09-15T03:35:36.177331+0000","last_active":"2024-09-15T03:38:15.411377+0000","last_peered":"2024-09-15T03:38:15.411377+0000","last_clean":"2024-09-15T03:38:15.411377+0000","last_became_active":"2024-09-15T03:35:36.177043+0000","last_became_peered":"2024-09-15T03:35:36.177043+0000","last_unstale":"2024-09-15T03:38:15.411377+0000","last_undegraded":"2024-09-15T03:38:15.411377+0000","last_fullsized":"2024-09-15T03:38:15.411377+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:52:08.123750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866082+0000","last_change":"2024-09-15T03:35:36.177279+0000","last_active":"2024-09-15T03:38:16.866082+0000","last_peered":"2024-09-15T03:38:16.866082+0000","last_clean":"2024-09-15T03:38:16.866082+0000","last_became_active":"2024-09-15T03:35:36.176997+0000","last_became_peered":"2024-09-15T03:35:36.176997+0000","last_unstale":"2024-09-15T03:38:16.866082+0000","last_undegraded":"2024-09-15T03:38:16.866082+0000","last_fullsized":"2024-09-15T03:38:16.866082+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:05:52.287110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":251,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412676+0000","last_change":"2024-09-15T03:35:36.189329+0000","last_active":"2024-09-15T03:38:15.412676+0000","last_peered":"2024-09-15T03:38:15.412676+0000","last_clean":"2024-09-15T03:38:15.412676+0000","last_became_active":"2024-09-15T03:35:36.176519+0000","last_became_peered":"2024-09-15T03:35:36.176519+0000","last_unstale":"2024-09-15T03:38:15.412676+0000","last_undegraded":"2024-09-15T03:38:15.412676+0000","last_fullsized":"2024-09-15T03:38:15.412676+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:40:12.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510504+0000","last_change":"2024-09-15T03:35:42.047866+0000","last_active":"2024-09-15T03:38:16.510504+0000","last_peered":"2024-09-15T03:38:16.510504+0000","last_clean":"2024-09-15T03:38:16.510504+0000","last_became_active":"2024-09-15T03:35:42.047571+0000","last_became_peered":"2024-09-15T03:35:42.047571+0000","last_unstale":"2024-09-15T03:38:16.510504+0000","last_undegraded":"2024-09-15T03:38:16.510504+0000","last_fullsized":"2024-09-15T03:38:16.510504+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:44.835315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411503+0000","last_change":"2024-09-15T03:35:36.180602+0000","last_active":"2024-09-15T03:38:15.411503+0000","last_peered":"2024-09-15T03:38:15.411503+0000","last_clean":"2024-09-15T03:38:15.411503+0000","last_became_active":"2024-09-15T03:35:36.180366+0000","last_became_peered":"2024-09-15T03:35:36.180366+0000","last_unstale":"2024-09-15T03:38:15.411503+0000","last_undegraded":"2024-09-15T03:38:15.411503+0000","last_fullsized":"2024-09-15T03:38:15.411503+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:51:20.797295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510310+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510310+0000","last_peered":"2024-09-15T03:38:16.510310+0000","last_clean":"2024-09-15T03:38:16.510310+0000","last_became_active":"2024-09-15T03:35:36.176265+0000","last_became_peered":"2024-09-15T03:35:36.176265+0000","last_unstale":"2024-09-15T03:38:16.510310+0000","last_undegraded":"2024-09-15T03:38:16.510310+0000","last_fullsized":"2024-09-15T03:38:16.510310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:07:54.095941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411431+0000","last_change":"2024-09-15T03:35:36.176300+0000","last_active":"2024-09-15T03:38:15.411431+0000","last_peered":"2024-09-15T03:38:15.411431+0000","last_clean":"2024-09-15T03:38:15.411431+0000","last_became_active":"2024-09-15T03:35:36.175986+0000","last_became_peered":"2024-09-15T03:35:36.175986+0000","last_unstale":"2024-09-15T03:38:15.411431+0000","last_undegraded":"2024-09-15T03:38:15.411431+0000","last_fullsized":"2024-09-15T03:38:15.411431+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:58:16.529865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63700000000000001,"5min":0.873,"15min":0.873},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59599999999999997,"5min":0.68400000000000005,"15min":0.751},"last":0.38200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.505},"min":{"1min":0.379,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73799999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.33800000000000002},"max":{"1min":0.60899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71199999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.74199999999999999},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.248,"5min":0.247,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.62},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.379},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":0.45200000000000001,"5min":0.63100000000000001,"15min":0.64400000000000002},"last":0.26800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.39000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.64400000000000002},"last":0.316},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.42199999999999999},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.77000000000000002},"last":0.42899999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.69099999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.499},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.64100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":0.59899999999999998,"5min":0.67700000000000005,"15min":0.70499999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.32800000000000001},"max":{"1min":0.629,"5min":0.66000000000000003,"15min":0.71199999999999997},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34599999999999997},"max":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.72199999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35299999999999998,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.72299999999999998},"last":0.53900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.627},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22700000000000001},"max":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.60099999999999998,"5min":0.63,"15min":0.63},"last":0.34699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640520,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51400000000000001,"15min":0.48999999999999999},"min":{"1min":0.35699999999999998,"5min":0.31,"15min":0.28000000000000003},"max":{"1min":0.68400000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54200000000000004,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.28699999999999998,"15min":0.17799999999999999},"max":{"1min":0.66400000000000003,"5min":0.76000000000000001,"15min":0.79700000000000004},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52600000000000002,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.29699999999999999},"max":{"1min":0.68300000000000005,"5min":0.755,"15min":0.75700000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.505,"15min":0.51400000000000001},"min":{"1min":0.34000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.71899999999999997,"15min":1.597},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.748,"15min":0.84199999999999997},"last":0.48899999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.19600000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.65400000000000003},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.222},"max":{"1min":0.54300000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34399999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.25,"5min":0.23899999999999999,"15min":0.221},"max":{"1min":0.54700000000000004,"5min":0.63200000000000001,"15min":0.65500000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.44800000000000001,"15min":0.42699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":0.57399999999999995,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.31900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31,"15min":0.308},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.58799999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.72699999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":0.48699999999999999,"5min":0.51200000000000001,"15min":0.59599999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.22},"max":{"1min":0.50600000000000001,"5min":0.57499999999999996,"15min":0.68500000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.35299999999999998,"15min":0.26700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56100000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.314},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":5.6269999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.29199999999999998},"max":{"1min":0.65800000000000003,"5min":0.77500000000000002,"15min":0.84999999999999998},"last":0.5},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72799999999999998,"5min":0.77300000000000002,"15min":0.82599999999999996},"last":0.52200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640519,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32500000000000001,"15min":0.32700000000000001},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":0.41999999999999998,"5min":0.52800000000000002,"15min":0.874},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.48999999999999999,"15min":0.55900000000000005},"last":0.26700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.215},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.77800000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.50900000000000001,"15min":0.747},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.73199999999999998,"15min":0.89800000000000002},"last":0.5},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.88300000000000001},"last":0.432},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.76600000000000001,"15min":0.81799999999999995},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.93799999999999994},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.432,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.91300000000000003},"last":0.45300000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640520,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32200000000000001,"15min":0.32000000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":0.55000000000000004,"15min":0.625},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33300000000000002,"15min":0.33400000000000002},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":0.52400000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.219},"max":{"1min":0.47499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.28899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.433},{"interface":"front","average":{"1min":0.499,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":0.85699999999999998},"last":0.378}]},{"osd":4,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.29699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.34899999999999998,"15min":0.34200000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.81200000000000006},"last":0.44700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":0.69999999999999996,"5min":0.76000000000000001,"15min":0.78000000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.84099999999999997,"15min":0.84799999999999998},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:50:23.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:50:24.028 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.027+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.028 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.027+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767537109}) 2024-09-15T03:50:24.028 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.027+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4196099932}) 2024-09-15T03:50:24.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.027+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.027+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767537109}) 2024-09-15T03:50:24.029 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.027+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415182430}) 2024-09-15T03:50:24.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722427263}) 2024-09-15T03:50:24.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1003909867}) 2024-09-15T03:50:24.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722427263}) 2024-09-15T03:50:24.035 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.034+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038599103}) 2024-09-15T03:50:24.040 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510114585}) 2024-09-15T03:50:24.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=948976322}) 2024-09-15T03:50:24.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.039+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510114585}) 2024-09-15T03:50:24.041 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.040+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176518544}) 2024-09-15T03:50:24.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.052+0000 7f58d6060640 10 monclient: get_monmap_and_config 2024-09-15T03:50:24.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.052+0000 7f58d6060640 10 monclient: build_initial_monmap 2024-09-15T03:50:24.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.052+0000 7f58d6060640 10 monclient: monmap: 2024-09-15T03:50:24.053 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:24.053 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:24.053617+0000 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:24.053617+0000 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:24.054 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:24.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.053+0000 7f58d6060640 1 Processor -- start 2024-09-15T03:50:24.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.053+0000 7f58d6060640 1 -- start start 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.053+0000 7f58d6060640 10 monclient: init 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f58d009caa0 0x7f58d009ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): picked mon.noname-c con 0x7f58d009caa0 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 0x7f58d006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): picked mon.noname-b con 0x7f58d009a540 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58d006f490 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d009abd0 con 0x7f58d006f490 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d009ed10 con 0x7f58d009caa0 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d009d3b0 con 0x7f58d009a540 2024-09-15T03:50:24.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58d6060640 10 monclient(hunting): authenticate will time out at 2021.693197s 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 0x7f58d006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 0x7f58d006ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41510/0 (socket says 172.21.15.153:41510) 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 -- 172.21.15.153:0/2246301243 learned_addr learned my addr 172.21.15.153:0/2246301243 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): get_auth_request con 0x7f58d009a540 auth_method 0 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58cf7fe640 1 --2- 172.21.15.153:0/2246301243 >> v2:172.21.15.181:3300/0 conn(0x7f58d009caa0 0x7f58d009ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient(hunting): handle_auth_done global_id 244373 payload 274 2024-09-15T03:50:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 -- 172.21.15.153:0/2246301243 >> v2:172.21.15.181:3300/0 conn(0x7f58d009caa0 msgr2=0x7f58d009ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 --2- 172.21.15.153:0/2246301243 >> v2:172.21.15.181:3300/0 conn(0x7f58d009caa0 0x7f58d009ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 -- 172.21.15.153:0/2246301243 >> v1:172.21.15.181:6789/0 conn(0x7f58d006f490 legacy=0x7f58d009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 monclient: found mon.noname-b 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58ceffd640 1 -- 172.21.15.153:0/2246301243 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d009b920 con 0x7f58d009a540 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58cf7fe640 10 monclient: get_auth_request con 0x7f58d009caa0 auth_method 0 2024-09-15T03:50:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.054+0000 7f58cf7fe640 1 --2- 172.21.15.153:0/2246301243 >> v2:172.21.15.181:3300/0 conn(0x7f58d009caa0 0x7f58d009ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ceffd640 1 -- 172.21.15.153:0/2246301243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 msgr2=0x7f58d006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ceffd640 1 --2- 172.21.15.153:0/2246301243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 0x7f58d006ef50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009910 tx=0x7f58c40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131efe28e304c81e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 1 -- 172.21.15.153:0/2246301243 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c402ae00 con 0x7f58d009a540 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: dump: 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58d6060640 5 monclient: authenticate success, global_id 244373 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 1 -- 172.21.15.153:0/2246301243 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c402f4b0 con 0x7f58d009a540 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58d6060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58d6060640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58d6060640 4 set_mon_vals no callback set 2024-09-15T03:50:24.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 1 -- 172.21.15.153:0/2246301243 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c402f850 con 0x7f58d009a540 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.057+0000 7f58ce7fc640 10 monclient: dump: 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:24.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 10 monclient: shutdown 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 -- 172.21.15.153:0/2246301243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 msgr2=0x7f58d006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 --2- 172.21.15.153:0/2246301243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 0x7f58d006ef50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009910 tx=0x7f58c40091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 -- 172.21.15.153:0/2246301243 shutdown_connections 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 --2- 172.21.15.153:0/2246301243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009a540 0x7f58d006ef50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 --2- 172.21.15.153:0/2246301243 >> v2:172.21.15.181:3300/0 conn(0x7f58d009caa0 0x7f58d009ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 -- 172.21.15.153:0/2246301243 >> 172.21.15.153:0/2246301243 conn(0x7f58d0067cd0 msgr2=0x7f58d00a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 -- 172.21.15.153:0/2246301243 shutdown_connections 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 1 -- 172.21.15.153:0/2246301243 wait complete. 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 10 monclient: build_initial_monmap 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.058+0000 7f58d6060640 10 monclient: monmap: 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:24.068 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 Processor -- start 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 -- start start 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient: init 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58d006f490 0x7f58d013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:24.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): picked mon.noname-b con 0x7f58d006f490 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 0x7f58d012f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): picked mon.noname-a con 0x7f58d009a540 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 0x7f58d0119530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): picked mon.noname-c con 0x7f58d009caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d0099da0 con 0x7f58d009a540 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d011cc60 con 0x7f58d006f490 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d013fae0 con 0x7f58d009caa0 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58d6060640 10 monclient(hunting): authenticate will time out at 2021.698619s 2024-09-15T03:50:24.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 0x7f58d0119530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 0x7f58d012f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 0x7f58d012f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33828/0 (socket says 172.21.15.153:33828) 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 1 -- 172.21.15.153:0/4097539261 learned_addr learned my addr 172.21.15.153:0/4097539261 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): get_auth_request con 0x7f58d009a540 auth_method 0 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 0x7f58d0119530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:41524/0 (socket says 172.21.15.153:41524) 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): get_auth_request con 0x7f58d009caa0 auth_method 0 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cf7fe640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58d006f490 0x7f58d013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cf7fe640 10 monclient(hunting): get_auth_request con 0x7f58d006f490 auth_method 0 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.059+0000 7f58cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 10 monclient(hunting): handle_auth_done global_id 244376 payload 824 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 1 -- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58d006f490 msgr2=0x7f58d013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58d006f490 0x7f58d013ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 1 -- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 msgr2=0x7f58d012f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 0x7f58d012f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 1 monclient: found mon.noname-c 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58cffff640 1 -- 172.21.15.153:0/4097539261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c4009450 con 0x7f58d009caa0 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.060+0000 7f58ceffd640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 0x7f58d012f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58cffff640 1 -- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 msgr2=0x7f58d0119530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58cffff640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 0x7f58d0119530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58c800c7e0 tx=0x7f58c80025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c800a850 con 0x7f58d009caa0 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: dump: 2024-09-15T03:50:24.073 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c800a9d0 con 0x7f58d009caa0 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 5 monclient: authenticate success, global_id 244376 2024-09-15T03:50:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c800ad90 con 0x7f58d009caa0 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 10 monclient: _renew_subs 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d011c950 con 0x7f58d009caa0 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58b77fe640 10 monclient: dump: 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58cdffb640 4 set_mon_vals no callback set 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 10 monclient: _renew_subs 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.062+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d013f1a0 con 0x7f58d009caa0 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f58c8012630 con 0x7f58d009caa0 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58b77fe640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58bc038740 0x7f58bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58d6060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58d6060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58d6060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f58c804c840 con 0x7f58d009caa0 2024-09-15T03:50:24.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d004bf60 con 0x7f58d009caa0 2024-09-15T03:50:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58cf7fe640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58bc038740 0x7f58bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.063+0000 7f58cf7fe640 10 monclient: get_auth_request con 0x7f58bc038740 auth_method 0 2024-09-15T03:50:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.066+0000 7f58cf7fe640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58bc038740 0x7f58bc03abf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f58c000ab30 tx=0x7f58c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.067+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f58c8016070 con 0x7f58d009caa0 2024-09-15T03:50:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.068+0000 7f58b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.068+0000 7f58b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:24.126 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2197765998}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550703052}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2197765998}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260507224}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.123+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.124+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269308932}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.124+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376647313}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.124+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.124+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269308932}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.124+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487781016}) 2024-09-15T03:50:24.127 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846293216}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581594131}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846293216}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621753883}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969249442}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341361748}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969249442}) 2024-09-15T03:50:24.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.125+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2816653761}) 2024-09-15T03:50:24.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.230+0000 7f58d6060640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:50:24.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.230+0000 7f58d6060640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:50:24.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.230+0000 7f58d6060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:24.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.230+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58d011ae90 con 0x7f58d009caa0 2024-09-15T03:50:24.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.231+0000 7f58b77fe640 1 -- 172.21.15.153:0/4097539261 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+20424 (secure 0 0 0) 0x7f58c8022950 con 0x7f58d009caa0 2024-09-15T03:50:24.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.231+0000 7f58b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:50:24.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.231+0000 7f58b77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:50:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:24.233 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":259,"fsid":"7dd05962-040e-4682-bebc-5f3c2ef6922e","created":"2024-09-15T03:28:23.739325+0000","modified":"2024-09-15T03:38:20.477637+0000","last_up_change":"2024-09-15T03:28:32.859989+0000","last_in_change":"2024-09-15T03:33:16.513275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":183,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:28:33.881566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:28:34.684266+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-15T03:33:32.211473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-15T03:33:32.992291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"42ca15e3-1d0f-4207-acde-15c6655103b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":31922},{"type":"v1","addr":"172.21.15.153:6801","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":31922},{"type":"v1","addr":"172.21.15.153:6803","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":31922},{"type":"v1","addr":"172.21.15.153:6807","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":31922},{"type":"v1","addr":"172.21.15.153:6805","nonce":31922}]},"public_addr":"172.21.15.153:6801/31922","cluster_addr":"172.21.15.153:6803/31922","heartbeat_back_addr":"172.21.15.153:6807/31922","heartbeat_front_addr":"172.21.15.153:6805/31922","state":["exists","up"]},{"osd":1,"uuid":"d77d5d67-15e9-427a-a58d-2dbfc5eded5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":31923},{"type":"v1","addr":"172.21.15.153:6809","nonce":31923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":31923},{"type":"v1","addr":"172.21.15.153:6811","nonce":31923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":31923},{"type":"v1","addr":"172.21.15.153:6815","nonce":31923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":31923},{"type":"v1","addr":"172.21.15.153:6813","nonce":31923}]},"public_addr":"172.21.15.153:6809/31923","cluster_addr":"172.21.15.153:6811/31923","heartbeat_back_addr":"172.21.15.153:6815/31923","heartbeat_front_addr":"172.21.15.153:6813/31923","state":["exists","up"]},{"osd":2,"uuid":"de4c4fc9-0d8c-444b-8f99-bef942a203db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":31924},{"type":"v1","addr":"172.21.15.153:6817","nonce":31924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":31924},{"type":"v1","addr":"172.21.15.153:6819","nonce":31924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":31924},{"type":"v1","addr":"172.21.15.153:6823","nonce":31924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":31924},{"type":"v1","addr":"172.21.15.153:6821","nonce":31924}]},"public_addr":"172.21.15.153:6817/31924","cluster_addr":"172.21.15.153:6819/31924","heartbeat_back_addr":"172.21.15.153:6823/31924","heartbeat_front_addr":"172.21.15.153:6821/31924","state":["exists","up"]},{"osd":3,"uuid":"8f6b762b-6dd0-4337-a983-9f53a45dde62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":31874},{"type":"v1","addr":"172.21.15.181:6809","nonce":31874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":31874},{"type":"v1","addr":"172.21.15.181:6811","nonce":31874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":31874},{"type":"v1","addr":"172.21.15.181:6815","nonce":31874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":31874},{"type":"v1","addr":"172.21.15.181:6813","nonce":31874}]},"public_addr":"172.21.15.181:6809/31874","cluster_addr":"172.21.15.181:6811/31874","heartbeat_back_addr":"172.21.15.181:6815/31874","heartbeat_front_addr":"172.21.15.181:6813/31874","state":["exists","up"]},{"osd":4,"uuid":"d9b93903-4d7a-471d-a4f8-16f4895771f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":31875},{"type":"v1","addr":"172.21.15.181:6801","nonce":31875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":31875},{"type":"v1","addr":"172.21.15.181:6803","nonce":31875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":31875},{"type":"v1","addr":"172.21.15.181:6807","nonce":31875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":31875},{"type":"v1","addr":"172.21.15.181:6805","nonce":31875}]},"public_addr":"172.21.15.181:6801/31875","cluster_addr":"172.21.15.181:6803/31875","heartbeat_back_addr":"172.21.15.181:6807/31875","heartbeat_front_addr":"172.21.15.181:6805/31875","state":["exists","up"]},{"osd":5,"uuid":"90de00c0-aaec-41ca-85a8-5693f2fe4e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":31876},{"type":"v1","addr":"172.21.15.181:6817","nonce":31876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":31876},{"type":"v1","addr":"172.21.15.181:6819","nonce":31876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":31876},{"type":"v1","addr":"172.21.15.181:6823","nonce":31876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":31876},{"type":"v1","addr":"172.21.15.181:6821","nonce":31876}]},"public_addr":"172.21.15.181:6817/31876","cluster_addr":"172.21.15.181:6819/31876","heartbeat_back_addr":"172.21.15.181:6823/31876","heartbeat_front_addr":"172.21.15.181:6821/31876","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"ecab9f08-f922-45d9-8393-9f53fccdb802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.685491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.637606+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.740014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.612976+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.627273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:28:31.624592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:50:24.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58bc038740 msgr2=0x7f58bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:24.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58bc038740 0x7f58bc03abf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f58c000ab30 tx=0x7f58c0008040 comp rx=0 tx=0).stop 2024-09-15T03:50:24.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 10 monclient: shutdown 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 msgr2=0x7f58d0119530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 0x7f58d0119530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58c800c7e0 tx=0x7f58c80025b0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 shutdown_connections 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f58bc038740 0x7f58bc03abf0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58d009caa0 0x7f58d0119530 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d009a540 0x7f58d012f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 --2- 172.21.15.153:0/4097539261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58d006f490 0x7f58d013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 >> 172.21.15.153:0/4097539261 conn(0x7f58d0067cd0 msgr2=0x7f58d00a1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 shutdown_connections 2024-09-15T03:50:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:24.233+0000 7f58d6060640 1 -- 172.21.15.153:0/4097539261 wait complete. 2024-09-15T03:50:24.460 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.459+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.460 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.459+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463013958}) 2024-09-15T03:50:24.460 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.459+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132264613}) 2024-09-15T03:50:24.460 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.459+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.460 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.459+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463013958}) 2024-09-15T03:50:24.461 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.459+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587141100}) 2024-09-15T03:50:24.462 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.460+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.462 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.460+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669674607}) 2024-09-15T03:50:24.462 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.460+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974743900}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.460+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.460+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669674607}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.460+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328894556}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134902720}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057678420}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134902720}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725533173}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551098347}) 2024-09-15T03:50:24.463 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908773658}) 2024-09-15T03:50:24.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551098347}) 2024-09-15T03:50:24.464 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.461+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428968285}) 2024-09-15T03:50:24.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.462+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.462+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643819117}) 2024-09-15T03:50:24.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.462+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340135015}) 2024-09-15T03:50:24.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.462+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.462+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643819117}) 2024-09-15T03:50:24.476 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.462+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3150516482}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.463+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.463+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672146984}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.463+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142997624}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.463+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.463+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672146984}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.463+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293707563}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119668219}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419501982}) 2024-09-15T03:50:24.477 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119668219}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023720978}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503261409}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775535536}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503261409}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.464+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1572996452}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.465+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.465+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867425189}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.465+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3828689037}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.465+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.465+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867425189}) 2024-09-15T03:50:24.478 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.465+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532769079}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986045047}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1864151265}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986045047}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669488836}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2450225859}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1676356815}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2450225859}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.466+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=214407318}) 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.467+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.467+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722281723}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.467+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=736294702}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.467+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.467+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722281723}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.467+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003376714}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460891529}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3771802129}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460891529}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1976807216}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734652389}) 2024-09-15T03:50:24.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=416944926}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734652389}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.468+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048709867}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.469+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.469+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145605418}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.469+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=394130911}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.469+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.469+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145605418}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.469+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026067101}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.470+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.470+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522575032}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.470+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384875712}) 2024-09-15T03:50:24.481 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.470+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.470+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522575032}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.470+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693828626}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606853285}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615665392}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606853285}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2377510135}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324808397}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3173393109}) 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324808397}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.471+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672602549}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.472+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.472+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187292658}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.472+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1858794998}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.472+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.472+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187292658}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.472+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225999286}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301838686}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3654486195}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301838686}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648522006}) 2024-09-15T03:50:24.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790346195}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=69515238}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790346195}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.473+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=73048443}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.474+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.474+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702042133}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.474+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3415739567}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.474+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.474+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702042133}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.474+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375590513}) 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604371100}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208160427}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604371100}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160281759}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278648893}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634296105}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278648893}) 2024-09-15T03:50:24.485 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.475+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449579153}) 2024-09-15T03:50:24.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.476+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.476+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985331830}) 2024-09-15T03:50:24.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.476+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466404925}) 2024-09-15T03:50:24.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.476+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.476+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985331830}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.476+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3413044130}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.477+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.477+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787514203}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.477+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1347357088}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.477+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.477+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787514203}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.477+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=88095151}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250353546}) 2024-09-15T03:50:24.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266345131}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250353546}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=249184716}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425187545}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512637067}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425187545}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.478+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1529356627}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.479+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.479+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407622120}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.479+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3672126083}) 2024-09-15T03:50:24.495 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.479+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.479+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407622120}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.479+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440775896}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630943902}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259825097}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630943902}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209866069}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693883360}) 2024-09-15T03:50:24.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671306113}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693883360}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.480+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625507318}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959781751}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309278541}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959781751}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2620957524}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170476933}) 2024-09-15T03:50:24.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271781719}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170476933}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.481+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893141088}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.482+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.482+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204176293}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.482+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536168977}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.482+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.482+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204176293}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.482+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526997544}) 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.483+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.483+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=3214622005,mdsmap=1531151339,mgr=1013486720,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=818079576,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3121057786}) 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.483+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=7828386,mdsmap=1531151339,mgr=2938112866,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=818079576,mon_config_key=4117641549,monmap=244636553,osd_metadata=2347033642,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=2028975641}) 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.483+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.483+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=3214622005,mdsmap=1531151339,mgr=1013486720,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=818079576,mon_config_key=4117641549,monmap=244636553,osd_metadata=851455416,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3121057786}) 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.483+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=26,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=39} crc {logm=3318859391,mdsmap=1531151339,mgr=2572398688,mgr_command_descs=3545737492,mgr_metadata=2280977772,mgrstat=4079236905,mon_config_key=4117641549,monmap=626991505,osd_metadata=1633747541,osd_pg_creating=1754493876,osd_snap=454132730,osdmap=3514888776}) 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.487+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.487+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=1789405924}) 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.487+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=18} crc {osdmap=1625426874}) 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.487+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:50:24.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.487+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=1789405924}) 2024-09-15T03:50:24.500 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-15T03:50:24.487+0000 7f39749f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=18} crc {osdmap=1625426874}) 2024-09-15T03:50:25.245 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T03:50:25.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.349+0000 7f1c4579d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.349+0000 7f1c4579d640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.349+0000 7f1c4579d640 10 monclient: monmap: 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:25.350169+0000 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:25.350169+0000 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.351 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:25.351 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:25.351 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:25.351 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:25.351 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:25.351 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 Processor -- start 2024-09-15T03:50:25.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 -- start start 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient: init 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1c40131980 0x7f1c40131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c40131980 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c40132290 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 0x7f1c40138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4009f140 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4009c530 con 0x7f1c40132290 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c40132690 con 0x7f1c40131980 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4009f560 con 0x7f1c4009f140 2024-09-15T03:50:25.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c4579d640 10 monclient(hunting): authenticate will time out at 2022.989564s 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1c40131980 0x7f1c40131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 0x7f1c40138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1c40131980 0x7f1c40131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55466/0 (socket says 172.21.15.153:55466) 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 0x7f1c40138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34314/0 (socket says 172.21.15.153:34314) 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 1 -- 172.21.15.153:0/3090251901 learned_addr learned my addr 172.21.15.153:0/3090251901 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): get_auth_request con 0x7f1c40131980 auth_method 0 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): get_auth_request con 0x7f1c4009f140 auth_method 0 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.350+0000 7f1c3effd640 10 monclient(hunting): handle_auth_done global_id 254265 payload 274 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 1 -- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 msgr2=0x7f1c40138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 1 --2- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 0x7f1c40138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 1 -- 172.21.15.153:0/3090251901 >> v1:172.21.15.181:6789/0 conn(0x7f1c40132290 legacy=0x7f1c4009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 1 monclient: found mon.noname-a 2024-09-15T03:50:25.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3effd640 1 -- 172.21.15.153:0/3090251901 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4009fe10 con 0x7f1c40131980 2024-09-15T03:50:25.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.351+0000 7f1c3f7fe640 1 --2- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 0x7f1c40138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:25.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.353+0000 7f1c3effd640 1 -- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c40131980 msgr2=0x7f1c40131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.353+0000 7f1c3effd640 1 --2- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c40131980 0x7f1c40131d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009940 tx=0x7f1c300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86af4e08a3d7b58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.353+0000 7f1c3e7fc640 1 -- 172.21.15.153:0/3090251901 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032440 con 0x7f1c40131980 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.353+0000 7f1c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.353+0000 7f1c3e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.353+0000 7f1c3e7fc640 10 monclient: dump: 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.363 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 1 -- 172.21.15.153:0/3090251901 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c300325a0 con 0x7f1c40131980 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 5 monclient: authenticate success, global_id 254265 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 1 -- 172.21.15.153:0/3090251901 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032940 con 0x7f1c40131980 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 4 set_mon_vals no callback set 2024-09-15T03:50:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c3e7fc640 10 monclient: dump: 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 10 monclient: shutdown 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 -- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c40131980 msgr2=0x7f1c40131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 --2- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c40131980 0x7f1c40131d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009940 tx=0x7f1c300091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 -- 172.21.15.153:0/3090251901 shutdown_connections 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 --2- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c4009f140 0x7f1c40138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 --2- 172.21.15.153:0/3090251901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c40131980 0x7f1c40131d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 -- 172.21.15.153:0/3090251901 >> 172.21.15.153:0/3090251901 conn(0x7f1c400a5ad0 msgr2=0x7f1c40067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 -- 172.21.15.153:0/3090251901 shutdown_connections 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.354+0000 7f1c4579d640 1 -- 172.21.15.153:0/3090251901 wait complete. 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient: monmap: 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:25.366 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 1 Processor -- start 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 1 -- start start 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient: init 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 0x7f1c401342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4009f140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c40131980 0x7f1c401347e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.355+0000 7f1c4579d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c40131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c40132290 0x7f1c40134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 10 monclient(hunting): picked mon.noname-c con 0x7f1c40132290 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40131560 con 0x7f1c4009f140 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401362c0 con 0x7f1c40131980 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c401378c0 con 0x7f1c40132290 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c4579d640 10 monclient(hunting): authenticate will time out at 2022.995041s 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 0x7f1c401342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 0x7f1c401342a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55470/0 (socket says 172.21.15.153:55470) 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 -- 172.21.15.153:0/15591554 learned_addr learned my addr 172.21.15.153:0/15591554 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): get_auth_request con 0x7f1c4009f140 auth_method 0 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c40132290 0x7f1c40134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): get_auth_request con 0x7f1c40132290 auth_method 0 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c367fc640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c40131980 0x7f1c401347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c367fc640 10 monclient(hunting): get_auth_request con 0x7f1c40131980 auth_method 0 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c367fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient(hunting): handle_auth_done global_id 254268 payload 824 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 -- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c40132290 msgr2=0x7f1c40134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c40132290 0x7f1c40134d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 -- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c40131980 msgr2=0x7f1c401347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c40131980 0x7f1c401347e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 monclient: found mon.noname-a 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.356+0000 7f1c3effd640 1 -- 172.21.15.153:0/15591554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38009a20 con 0x7f1c4009f140 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c3effd640 1 -- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 msgr2=0x7f1c401342a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c3effd640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 0x7f1c401342a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1c30035040 tx=0x7f1c30032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30031a10 con 0x7f1c4009f140 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: dump: 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.372 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c30031bb0 con 0x7f1c4009f140 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 5 monclient: authenticate success, global_id 254268 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 10 monclient: _renew_subs 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c30009450 con 0x7f1c4009f140 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3003d420 con 0x7f1c4009f140 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c377fe640 10 monclient: dump: 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.374 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c3dffb640 4 set_mon_vals no callback set 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 10 monclient: _renew_subs 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.358+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c40135d90 con 0x7f1c4009f140 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.359+0000 7f1c4579d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.359+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c4005ed70 con 0x7f1c4009f140 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.359+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f1c30039040 con 0x7f1c4009f140 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.359+0000 7f1c377fe640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c20038740 0x7f1c2003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.359+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1c3006c410 con 0x7f1c4009f140 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c377fe640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7f1c200415f0 0x7f1c20043a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 --> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c20044110 con 0x7f1c200415f0 2024-09-15T03:50:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=259) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c3006e910 con 0x7f1c4009f140 2024-09-15T03:50:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c377fe640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-15T03:50:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c367fc640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c20038740 0x7f1c2003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c367fc640 10 monclient: get_auth_request con 0x7f1c20038740 auth_method 0 2024-09-15T03:50:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c3f7fe640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7f1c200415f0 0x7f1c20043a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.360+0000 7f1c3f7fe640 10 monclient: get_auth_request con 0x7f1c200415f0 auth_method 0 2024-09-15T03:50:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.362+0000 7f1c367fc640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c20038740 0x7f1c2003abf0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1c280037d0 tx=0x7f1c28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.364+0000 7f1c3f7fe640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7f1c200415f0 0x7f1c20043a40 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.365+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== osd.0 v2:172.21.15.153:6800/31922 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1c20044110 con 0x7f1c200415f0 2024-09-15T03:50:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.376+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 --> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1c401388a0 con 0x7f1c200415f0 2024-09-15T03:50:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.376+0000 7f1c377fe640 1 -- 172.21.15.153:0/15591554 <== osd.0 v2:172.21.15.153:6800/31922 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1c401388a0 con 0x7f1c200415f0 2024-09-15T03:50:25.378 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:25.378 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:50:25.378 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7f1c200415f0 msgr2=0x7f1c20043a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:6800/31922,v1:172.21.15.153:6801/31922] conn(0x7f1c200415f0 0x7f1c20043a40 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c20038740 msgr2=0x7f1c2003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f1c20038740 0x7f1c2003abf0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1c280037d0 tx=0x7f1c28009040 comp rx=0 tx=0).stop 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 10 monclient: shutdown 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 msgr2=0x7f1c401342a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 --2- 172.21.15.153:0/15591554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4009f140 0x7f1c401342a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1c30035040 tx=0x7f1c30032000 comp rx=0 tx=0).stop 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c3f7fe640 1 -- 172.21.15.153:0/15591554 reap_dead start 2024-09-15T03:50:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 shutdown_connections 2024-09-15T03:50:25.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 >> 172.21.15.153:0/15591554 conn(0x7f1c400a5ad0 msgr2=0x7f1c400a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 shutdown_connections 2024-09-15T03:50:25.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.377+0000 7f1c4579d640 1 -- 172.21.15.153:0/15591554 wait complete. 2024-09-15T03:50:25.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.480+0000 7fab86e4f640 10 monclient: get_monmap_and_config 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.480+0000 7fab86e4f640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.480+0000 7fab86e4f640 10 monclient: monmap: 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:25.481233+0000 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:25.481233+0000 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:25.482 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:25.482 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:25.482 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:25.482 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:25.482 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 1 Processor -- start 2024-09-15T03:50:25.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 1 -- start start 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient: init 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab8009ad90 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fab8009b160 0x7fab8009b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient(hunting): picked mon.noname-c con 0x7fab8009b160 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab80136c90 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800692c0 con 0x7fab8009ad90 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8009cce0 con 0x7fab80136c90 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8009db70 con 0x7fab8009b160 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab86e4f640 10 monclient(hunting): authenticate will time out at 2023.120743s 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab84beb640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fab8009ad90 0x7fab8009c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50082/0 (socket says 172.21.15.153:50082) 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab84beb640 1 -- 172.21.15.153:0/239558661 learned_addr learned my addr 172.21.15.153:0/239558661 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.481+0000 7fab7ffff640 1 --2- 172.21.15.153:0/239558661 >> v2:172.21.15.181:3300/0 conn(0x7fab8009b160 0x7fab8009b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): get_auth_request con 0x7fab8009b160 auth_method 0 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.482+0000 7fab7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab84beb640 1 -- 172.21.15.153:0/239558661 >> v1:172.21.15.153:6789/0 conn(0x7fab8009ad90 legacy=0x7fab8009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab853ec640 1 -- 172.21.15.153:0/239558661 >> v1:172.21.15.181:6789/0 conn(0x7fab80136c90 legacy=0x7fab80139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 10 monclient(hunting): handle_auth_done global_id 264136 payload 274 2024-09-15T03:50:25.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 1 -- 172.21.15.153:0/239558661 >> v1:172.21.15.181:6789/0 conn(0x7fab80136c90 legacy=0x7fab80139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 1 -- 172.21.15.153:0/239558661 >> v1:172.21.15.153:6789/0 conn(0x7fab8009ad90 legacy=0x7fab8009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 1 monclient: found mon.noname-c 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7f7fe640 1 -- 172.21.15.153:0/239558661 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535339034 0 0) 0x7fab800692c0 con 0x7fab8009ad90 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.181:3300/0 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7ffff640 1 -- 172.21.15.153:0/239558661 --> v2:172.21.15.181:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8009d5b0 con 0x7fab8009b160 2024-09-15T03:50:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.484+0000 7fab7f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7ffff640 1 -- 172.21.15.153:0/239558661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009b160 msgr2=0x7fab8009b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7ffff640 1 --2- 172.21.15.153:0/239558661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009b160 0x7fab8009b530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009950 tx=0x7fab6c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=493d32627c7a49b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 1 -- 172.21.15.153:0/239558661 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c02f4b0 con 0x7fab8009b160 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: dump: 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.492 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 5 monclient: authenticate success, global_id 264136 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 1 -- 172.21.15.153:0/239558661 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c02f610 con 0x7fab8009b160 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 1 -- 172.21.15.153:0/239558661 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c02f9b0 con 0x7fab8009b160 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 4 set_mon_vals no callback set 2024-09-15T03:50:25.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab7f7fe640 10 monclient: dump: 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 10 monclient: shutdown 2024-09-15T03:50:25.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 1 -- 172.21.15.153:0/239558661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009b160 msgr2=0x7fab8009b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.487+0000 7fab86e4f640 1 --2- 172.21.15.153:0/239558661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009b160 0x7fab8009b530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009950 tx=0x7fab6c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 1 -- 172.21.15.153:0/239558661 shutdown_connections 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 1 --2- 172.21.15.153:0/239558661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009b160 0x7fab8009b530 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 1 -- 172.21.15.153:0/239558661 >> 172.21.15.153:0/239558661 conn(0x7fab800678a0 msgr2=0x7fab800a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 1 -- 172.21.15.153:0/239558661 shutdown_connections 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 1 -- 172.21.15.153:0/239558661 wait complete. 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 10 monclient: monmap: 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:25.495 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.488+0000 7fab86e4f640 1 Processor -- start 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 -- start start 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient: init 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009ad90 0x7fab80135950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): picked mon.noname-b con 0x7fab8009ad90 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 0x7fab80135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): picked mon.noname-a con 0x7fab8009b160 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:25.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab80136c90 0x7fab8008cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): picked mon.noname-c con 0x7fab80136c90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:25.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80090b40 con 0x7fab8009b160 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8006f020 con 0x7fab8009ad90 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab80098e30 con 0x7fab80136c90 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab86e4f640 10 monclient(hunting): authenticate will time out at 2023.128456s 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 0x7fab80135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 0x7fab80135e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55474/0 (socket says 172.21.15.153:55474) 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 -- 172.21.15.153:0/1398100401 learned_addr learned my addr 172.21.15.153:0/1398100401 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab80136c90 0x7fab8008cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): get_auth_request con 0x7fab8009b160 auth_method 0 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): get_auth_request con 0x7fab80136c90 auth_method 0 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009ad90 0x7fab80135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): get_auth_request con 0x7fab8009ad90 auth_method 0 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab84beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient(hunting): handle_auth_done global_id 254274 payload 824 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 -- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab80136c90 msgr2=0x7fab8008cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab80136c90 0x7fab8008cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 -- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009ad90 msgr2=0x7fab80135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009ad90 0x7fab80135950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 monclient: found mon.noname-a 2024-09-15T03:50:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab7ffff640 1 -- 172.21.15.153:0/1398100401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c009390 con 0x7fab8009b160 2024-09-15T03:50:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.489+0000 7fab853ec640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab80136c90 0x7fab8008cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:25.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7ffff640 1 -- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 msgr2=0x7fab80135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7ffff640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 0x7fab80135e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009950 tx=0x7fab6c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c032ab0 con 0x7fab8009b160 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: dump: 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.506 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 5 monclient: authenticate success, global_id 254274 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 10 monclient: _renew_subs 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c032c10 con 0x7fab8009b160 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8008ea30 con 0x7fab8009b160 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c03d440 con 0x7fab8009b160 2024-09-15T03:50:25.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7cff9640 10 monclient: dump: 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab7effd640 4 set_mon_vals no callback set 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 10 monclient: _renew_subs 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.492+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab800917c0 con 0x7fab8009b160 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fab6c039040 con 0x7fab8009b160 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab86e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab86e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab86e4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8004bf60 con 0x7fab8009b160 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab7cff9640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab64038770 0x7fab6403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fab6c06c180 con 0x7fab8009b160 2024-09-15T03:50:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab84beb640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab64038770 0x7fab6403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.493+0000 7fab84beb640 10 monclient: get_auth_request con 0x7fab64038770 auth_method 0 2024-09-15T03:50:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.496+0000 7fab84beb640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab64038770 0x7fab6403ac20 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fab74005840 tx=0x7fab74007b40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.497+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fab6c030b30 con 0x7fab8009b160 2024-09-15T03:50:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.497+0000 7fab7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.497+0000 7fab7cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:25.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.653+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fab80090840 con 0x7fab64038770 2024-09-15T03:50:25.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.654+0000 7fab7cff9640 1 -- 172.21.15.153:0/1398100401 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fab80090840 con 0x7fab64038770 2024-09-15T03:50:25.655 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T03:50:25.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab64038770 msgr2=0x7fab6403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab64038770 0x7fab6403ac20 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fab74005840 tx=0x7fab74007b40 comp rx=0 tx=0).stop 2024-09-15T03:50:25.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 10 monclient: shutdown 2024-09-15T03:50:25.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 msgr2=0x7fab80135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 0x7fab80135e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009950 tx=0x7fab6c032000 comp rx=0 tx=0).stop 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 shutdown_connections 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fab64038770 0x7fab6403ac20 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab80136c90 0x7fab8008cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab8009b160 0x7fab80135e90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 --2- 172.21.15.153:0/1398100401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8009ad90 0x7fab80135950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 >> 172.21.15.153:0/1398100401 conn(0x7fab800678a0 msgr2=0x7fab80097440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 shutdown_connections 2024-09-15T03:50:25.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.656+0000 7fab86e4f640 1 -- 172.21.15.153:0/1398100401 wait complete. 2024-09-15T03:50:25.668 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T03:50:25.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.765+0000 7f2db3a6f640 10 monclient: get_monmap_and_config 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.765+0000 7f2db3a6f640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.765+0000 7f2db3a6f640 10 monclient: monmap: 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:25.766176+0000 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:25.766176+0000 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.766 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:25.767 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:25.767 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:25.767 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:25.767 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:25.767 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.765+0000 7f2db3a6f640 1 Processor -- start 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 1 -- start start 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient: init 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2dac131ac0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2dac09fd00 0x7f2dac09e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient(hunting): picked mon.noname-a con 0x7f2dac09fd00 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2dac09eec0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac0a13c0 con 0x7f2dac131ac0 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac0a10e0 con 0x7f2dac09eec0 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac1325c0 con 0x7f2dac09fd00 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db3a6f640 10 monclient(hunting): authenticate will time out at 2023.405698s 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2dac09fd00 0x7f2dac09e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2dac09fd00 0x7f2dac09e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55488/0 (socket says 172.21.15.153:55488) 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 1 -- 172.21.15.153:0/2830474296 learned_addr learned my addr 172.21.15.153:0/2830474296 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): get_auth_request con 0x7f2dac09fd00 auth_method 0 2024-09-15T03:50:25.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db180b640 1 --1- 172.21.15.153:0/2830474296 >> v1:172.21.15.153:6789/0 conn(0x7f2dac131ac0 0x7f2dac131e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50086/0 (socket says 172.21.15.153:50086) 2024-09-15T03:50:25.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.766+0000 7f2db100a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 10 monclient(hunting): handle_auth_done global_id 254277 payload 274 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 1 -- 172.21.15.153:0/2830474296 >> v1:172.21.15.181:6789/0 conn(0x7f2dac09eec0 legacy=0x7f2dac138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 1 -- 172.21.15.153:0/2830474296 >> v1:172.21.15.153:6789/0 conn(0x7f2dac131ac0 legacy=0x7f2dac131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 1 monclient: found mon.noname-a 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.767+0000 7f2db100a640 1 -- 172.21.15.153:0/2830474296 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac09f320 con 0x7f2dac09fd00 2024-09-15T03:50:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.769+0000 7f2db100a640 1 -- 172.21.15.153:0/2830474296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 msgr2=0x7f2dac09e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.769+0000 7f2db100a640 1 --2- 172.21.15.153:0/2830474296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac09e980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2dac09c860 tx=0x7f2d9c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b06bdd24350eee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 1 -- 172.21.15.153:0/2830474296 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c032440 con 0x7f2dac09fd00 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: dump: 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 1 -- 172.21.15.153:0/2830474296 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c0325a0 con 0x7f2dac09fd00 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 5 monclient: authenticate success, global_id 254277 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 1 -- 172.21.15.153:0/2830474296 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c032940 con 0x7f2dac09fd00 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 4 set_mon_vals no callback set 2024-09-15T03:50:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db0809640 10 monclient: dump: 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 10 monclient: shutdown 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 -- 172.21.15.153:0/2830474296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 msgr2=0x7f2dac09e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 --2- 172.21.15.153:0/2830474296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac09e980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2dac09c860 tx=0x7f2d9c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 -- 172.21.15.153:0/2830474296 shutdown_connections 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 --2- 172.21.15.153:0/2830474296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac09e980 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 -- 172.21.15.153:0/2830474296 >> 172.21.15.153:0/2830474296 conn(0x7f2dac0a4f60 msgr2=0x7f2dac0a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 -- 172.21.15.153:0/2830474296 shutdown_connections 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.770+0000 7f2db3a6f640 1 -- 172.21.15.153:0/2830474296 wait complete. 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.771+0000 7f2db3a6f640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.771+0000 7f2db3a6f640 10 monclient: monmap: 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.771+0000 7f2db3a6f640 1 Processor -- start 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.771+0000 7f2db3a6f640 1 -- start start 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.771+0000 7f2db3a6f640 10 monclient: init 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 0x7f2dac1394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f2dac09eec0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): picked mon.noname-a con 0x7f2dac09fd00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac131ac0 0x7f2dac134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f2dac131ac0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac09bce0 con 0x7f2dac09fd00 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac137c50 con 0x7f2dac131ac0 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac135450 con 0x7f2dac09eec0 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db3a6f640 10 monclient(hunting): authenticate will time out at 2023.411093s 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 0x7f2dac1394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 0x7f2dac1394c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34328/0 (socket says 172.21.15.153:34328) 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 -- 172.21.15.153:0/1972253639 learned_addr learned my addr 172.21.15.153:0/1972253639 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): get_auth_request con 0x7f2dac09fd00 auth_method 0 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): get_auth_request con 0x7f2dac09eec0 auth_method 0 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac131ac0 0x7f2dac134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): get_auth_request con 0x7f2dac131ac0 auth_method 0 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db200c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient(hunting): handle_auth_done global_id 244388 payload 824 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 -- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac131ac0 msgr2=0x7f2dac134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dac131ac0 0x7f2dac134530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 -- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 msgr2=0x7f2dac133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac133ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 monclient: found mon.noname-c 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db180b640 1 -- 172.21.15.153:0/1972253639 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c009450 con 0x7f2dac09eec0 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.772+0000 7f2db100a640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac09fd00 0x7f2dac133ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2db180b640 1 -- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 msgr2=0x7f2dac1394c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2db180b640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 0x7f2dac1394c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2da800c5e0 tx=0x7f2da8007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8007510 con 0x7f2dac09eec0 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: dump: 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da80057d0 con 0x7f2dac09eec0 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8007690 con 0x7f2dac09eec0 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2db3a6f640 5 monclient: authenticate success, global_id 244388 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2db3a6f640 10 monclient: _renew_subs 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2db3a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac137690 con 0x7f2dac09eec0 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da3fff640 4 set_mon_vals no callback set 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:25.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.774+0000 7f2da1ffb640 10 monclient: dump: 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.775+0000 7f2db3a6f640 10 monclient: _renew_subs 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.775+0000 7f2db3a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:25.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.775+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac136940 con 0x7f2dac09eec0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.775+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f2da801b400 con 0x7f2dac09eec0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.775+0000 7f2db3a6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.775+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2dac05ed70 con 0x7f2dac09eec0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2da1ffb640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d90038770 0x7f2d9003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2da804c010 con 0x7f2dac09eec0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2da1ffb640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7f2d900415a0 0x7f2d900439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 --> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d900440c0 con 0x7f2d900415a0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=259) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2da801b6b0 con 0x7f2dac09eec0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2da1ffb640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2db100a640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d90038770 0x7f2d9003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2db200c640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7f2d900415a0 0x7f2d900439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2db100a640 10 monclient: get_auth_request con 0x7f2d90038770 auth_method 0 2024-09-15T03:50:25.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.776+0000 7f2db200c640 10 monclient: get_auth_request con 0x7f2d900415a0 auth_method 0 2024-09-15T03:50:25.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.778+0000 7f2db100a640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d90038770 0x7f2d9003ac20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f2dac064eb0 tx=0x7f2d9c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.780+0000 7f2db200c640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7f2d900415a0 0x7f2d900439f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.781+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== osd.1 v2:172.21.15.153:6808/31923 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2d900440c0 con 0x7f2d900415a0 2024-09-15T03:50:25.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.792+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 --> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2dac138350 con 0x7f2d900415a0 2024-09-15T03:50:25.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.792+0000 7f2da1ffb640 1 -- 172.21.15.153:0/1972253639 <== osd.1 v2:172.21.15.153:6808/31923 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2dac138350 con 0x7f2d900415a0 2024-09-15T03:50:25.794 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:25.794 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:50:25.794 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.792+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7f2d900415a0 msgr2=0x7f2d900439f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.792+0000 7f2db3a6f640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:6808/31923,v1:172.21.15.153:6809/31923] conn(0x7f2d900415a0 0x7f2d900439f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d90038770 msgr2=0x7f2d9003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f2d90038770 0x7f2d9003ac20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f2dac064eb0 tx=0x7f2d9c032000 comp rx=0 tx=0).stop 2024-09-15T03:50:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 10 monclient: shutdown 2024-09-15T03:50:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 msgr2=0x7f2dac1394c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 --2- 172.21.15.153:0/1972253639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dac09eec0 0x7f2dac1394c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2da800c5e0 tx=0x7f2da8007f00 comp rx=0 tx=0).stop 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db200c640 1 -- 172.21.15.153:0/1972253639 reap_dead start 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 shutdown_connections 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 >> 172.21.15.153:0/1972253639 conn(0x7f2dac0a4f60 msgr2=0x7f2dac0a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 shutdown_connections 2024-09-15T03:50:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.793+0000 7f2db3a6f640 1 -- 172.21.15.153:0/1972253639 wait complete. 2024-09-15T03:50:25.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.897+0000 7fde02817640 10 monclient: get_monmap_and_config 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.897+0000 7fde02817640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.897+0000 7fde02817640 10 monclient: monmap: 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:25.898259+0000 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:25.898259+0000 2024-09-15T03:50:25.898 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:25.899 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 1 Processor -- start 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 1 -- start start 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient: init 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddfc09caa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:25.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 0x7fddfc09ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient(hunting): picked mon.noname-b con 0x7fddfc09a540 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddfc1346c0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddfc09abd0 con 0x7fddfc09caa0 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddfc09b920 con 0x7fddfc1346c0 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc09ed10 con 0x7fddfc09a540 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fde02817640 10 monclient(hunting): authenticate will time out at 2023.537850s 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 0x7fddfc09ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfbfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fddfc09caa0 0x7fddfc06ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50092/0 (socket says 172.21.15.153:50092) 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfbfff640 1 -- 172.21.15.153:0/2093751596 learned_addr learned my addr 172.21.15.153:0/2093751596 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfb7fe640 10 monclient(hunting): get_auth_request con 0x7fddfc09a540 auth_method 0 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.898+0000 7fddfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_done global_id 244391 payload 274 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 1 -- 172.21.15.153:0/2093751596 >> v1:172.21.15.181:6789/0 conn(0x7fddfc1346c0 legacy=0x7fddfc136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 1 -- 172.21.15.153:0/2093751596 >> v1:172.21.15.153:6789/0 conn(0x7fddfc09caa0 legacy=0x7fddfc06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 1 monclient: found mon.noname-b 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:25.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.899+0000 7fddfb7fe640 1 -- 172.21.15.153:0/2093751596 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc09c620 con 0x7fddfc09a540 2024-09-15T03:50:25.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfb7fe640 1 -- 172.21.15.153:0/2093751596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 msgr2=0x7fddfc09ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfb7fe640 1 --2- 172.21.15.153:0/2093751596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 0x7fddfc09ce70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009880 tx=0x7fdde40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c105c2153dfe2f8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 1 -- 172.21.15.153:0/2093751596 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4032440 con 0x7fddfc09a540 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: dump: 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.910 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 5 monclient: authenticate success, global_id 244391 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 1 -- 172.21.15.153:0/2093751596 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde40325a0 con 0x7fddfc09a540 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 4 set_mon_vals no callback set 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 1 -- 172.21.15.153:0/2093751596 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4032940 con 0x7fddfc09a540 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fddfaffd640 10 monclient: dump: 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.912 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 10 monclient: shutdown 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 -- 172.21.15.153:0/2093751596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 msgr2=0x7fddfc09ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 --2- 172.21.15.153:0/2093751596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 0x7fddfc09ce70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009880 tx=0x7fdde40090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 -- 172.21.15.153:0/2093751596 shutdown_connections 2024-09-15T03:50:25.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 --2- 172.21.15.153:0/2093751596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09a540 0x7fddfc09ce70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 -- 172.21.15.153:0/2093751596 >> 172.21.15.153:0/2093751596 conn(0x7fddfc067cd0 msgr2=0x7fddfc0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 -- 172.21.15.153:0/2093751596 shutdown_connections 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.902+0000 7fde02817640 1 -- 172.21.15.153:0/2093751596 wait complete. 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.903+0000 7fde02817640 10 monclient: build_initial_monmap 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.903+0000 7fde02817640 10 monclient: monmap: 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.914 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.903+0000 7fde02817640 1 Processor -- start 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.903+0000 7fde02817640 1 -- start start 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.903+0000 7fde02817640 10 monclient: init 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 0x7fddfc068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): picked mon.noname-b con 0x7fddfc09a540 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:25.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 0x7fddfc068e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): picked mon.noname-c con 0x7fddfc09caa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 0x7fddfc069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): picked mon.noname-a con 0x7fddfc1346c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc1339c0 con 0x7fddfc1346c0 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc137eb0 con 0x7fddfc09a540 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc069a80 con 0x7fddfc09caa0 2024-09-15T03:50:25.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde02817640 10 monclient(hunting): authenticate will time out at 2023.543307s 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 0x7fddfc068e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 0x7fddfc069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 0x7fddfc068e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34340/0 (socket says 172.21.15.153:34340) 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 1 -- 172.21.15.153:0/3796322902 learned_addr learned my addr 172.21.15.153:0/3796322902 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): get_auth_request con 0x7fddfc1346c0 auth_method 0 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): get_auth_request con 0x7fddfc09caa0 auth_method 0 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfbfff640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 0x7fddfc068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfbfff640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 msgr2=0x7fddfc068900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfbfff640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 msgr2=0x7fddfc068900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfbfff640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 0x7fddfc068900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfbfff640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 0x7fddfc068900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient(hunting): handle_auth_done global_id 254283 payload 824 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 msgr2=0x7fddfc068e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 0x7fddfc068e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 msgr2=0x7fddfc068900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 0x7fddfc068900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient: _finish_hunting 0 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 monclient: found mon.noname-a 2024-09-15T03:50:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fde00db4640 1 -- 172.21.15.153:0/3796322902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4009390 con 0x7fddfc1346c0 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.904+0000 7fddfb7fe640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 0x7fddfc068e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.906+0000 7fde00db4640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 msgr2=0x7fddfc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.906+0000 7fde00db4640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 0x7fddfc069380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fddf400c7e0 tx=0x7fddf40025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf400a850 con 0x7fddfc1346c0 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:25.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: dump: 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: _finish_auth 0 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf400a9d0 con 0x7fddfc1346c0 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 5 monclient: authenticate success, global_id 254283 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf400ad90 con 0x7fddfc1346c0 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 10 monclient: _renew_subs 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc099d50 con 0x7fddfc1346c0 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddfa7fc640 4 set_mon_vals no callback set 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fddf3fff640 10 monclient: dump: 2024-09-15T03:50:25.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:25.923 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:25.923 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.923 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:25.923 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:25.923 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:25.923 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 10 monclient: _renew_subs 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.907+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddfc11e350 con 0x7fddfc1346c0 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fddf4012630 con 0x7fddfc1346c0 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fddf3fff640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdddc038740 0x7fdddc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fddf404c950 con 0x7fddfc1346c0 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fde02817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fde02817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:25.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fde02817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddfc04bf60 con 0x7fddfc1346c0 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fddfbfff640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdddc038740 0x7fdddc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.908+0000 7fddfbfff640 10 monclient: get_auth_request con 0x7fdddc038740 auth_method 0 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.911+0000 7fddfbfff640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdddc038740 0x7fdddc03abf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fddec0079a0 tx=0x7fddec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.912+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fddf4016030 con 0x7fddfc1346c0 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.912+0000 7fddf3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:25.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:25.912+0000 7fddf3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:26.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.067+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fddfc06a630 con 0x7fdddc038740 2024-09-15T03:50:26.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.067+0000 7fddf3fff640 1 -- 172.21.15.153:0/3796322902 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fddfc06a630 con 0x7fdddc038740 2024-09-15T03:50:26.068 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdddc038740 msgr2=0x7fdddc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdddc038740 0x7fdddc03abf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fddec0079a0 tx=0x7fddec009040 comp rx=0 tx=0).stop 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 10 monclient: shutdown 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 msgr2=0x7fddfc069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 0x7fddfc069380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fddf400c7e0 tx=0x7fddf40025b0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 shutdown_connections 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fdddc038740 0x7fdddc03abf0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddfc1346c0 0x7fddfc069380 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fddfc09caa0 0x7fddfc068e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 --2- 172.21.15.153:0/3796322902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddfc09a540 0x7fddfc068900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.069+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 >> 172.21.15.153:0/3796322902 conn(0x7fddfc067cd0 msgr2=0x7fddfc06dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.070+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 shutdown_connections 2024-09-15T03:50:26.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.070+0000 7fde02817640 1 -- 172.21.15.153:0/3796322902 wait complete. 2024-09-15T03:50:26.081 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T03:50:26.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:50:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.178+0000 7ff60d6c0640 10 monclient: get_monmap_and_config 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.178+0000 7ff60d6c0640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.178+0000 7ff60d6c0640 10 monclient: monmap: 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:26.179346+0000 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:26.179346+0000 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.392 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 Processor -- start 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 -- start start 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient: init 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:50:26.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff608131980 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 0x7ff60809ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff60809ea30 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 0x7ff608139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff608136cb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60809c530 con 0x7ff608131980 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60809fe10 con 0x7ff60809ea30 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6080a11f0 con 0x7ff608136cb0 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff60d6c0640 10 monclient(hunting): authenticate will time out at 2023.818932s 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.179+0000 7ff5fffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 0x7ff60809ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 0x7ff608139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 0x7ff60809ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55520/0 (socket says 172.21.15.153:55520) 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 1 -- 172.21.15.153:0/947737437 learned_addr learned my addr 172.21.15.153:0/947737437 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): get_auth_request con 0x7ff60809ea30 auth_method 0 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 --2- 172.21.15.153:0/947737437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 0x7ff608139090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34342/0 (socket says 172.21.15.153:34342) 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): get_auth_request con 0x7ff608136cb0 auth_method 0 2024-09-15T03:50:26.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient(hunting): handle_auth_done global_id 244397 payload 274 2024-09-15T03:50:26.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 -- 172.21.15.153:0/947737437 >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 msgr2=0x7ff60809ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 --2- 172.21.15.153:0/947737437 >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 0x7ff60809ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 -- 172.21.15.153:0/947737437 >> v1:172.21.15.181:6789/0 conn(0x7ff608131980 legacy=0x7ff608131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 monclient: found mon.noname-b 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff6077fe640 1 -- 172.21.15.153:0/947737437 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080a14d0 con 0x7ff608136cb0 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.180+0000 7ff5fffff640 1 --2- 172.21.15.153:0/947737437 >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 0x7ff60809ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6077fe640 1 -- 172.21.15.153:0/947737437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 msgr2=0x7ff608139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6077fe640 1 --2- 172.21.15.153:0/947737437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 0x7ff608139090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff600009e60 tx=0x7ff600009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=124a7a75e6321c86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 1 -- 172.21.15.153:0/947737437 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600007760 con 0x7ff608136cb0 2024-09-15T03:50:26.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: dump: 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.398 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 1 -- 172.21.15.153:0/947737437 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60002d050 con 0x7ff608136cb0 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 5 monclient: authenticate success, global_id 244397 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 1 -- 172.21.15.153:0/947737437 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600007b30 con 0x7ff608136cb0 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff6067fc640 10 monclient: dump: 2024-09-15T03:50:26.399 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 4 set_mon_vals no callback set 2024-09-15T03:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 10 monclient: shutdown 2024-09-15T03:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 -- 172.21.15.153:0/947737437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 msgr2=0x7ff608139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 --2- 172.21.15.153:0/947737437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 0x7ff608139090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff600009e60 tx=0x7ff600009780 comp rx=0 tx=0).stop 2024-09-15T03:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 -- 172.21.15.153:0/947737437 shutdown_connections 2024-09-15T03:50:26.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 --2- 172.21.15.153:0/947737437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff608136cb0 0x7ff608139090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 --2- 172.21.15.153:0/947737437 >> v2:172.21.15.153:3300/0 conn(0x7ff60809ea30 0x7ff60809ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 -- 172.21.15.153:0/947737437 >> 172.21.15.153:0/947737437 conn(0x7ff6080a5ad0 msgr2=0x7ff608067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 -- 172.21.15.153:0/947737437 shutdown_connections 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.183+0000 7ff60d6c0640 1 -- 172.21.15.153:0/947737437 wait complete. 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.184+0000 7ff60d6c0640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.184+0000 7ff60d6c0640 10 monclient: monmap: 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:26.408 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.184+0000 7ff60d6c0640 1 Processor -- start 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.184+0000 7ff60d6c0640 1 -- start start 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.184+0000 7ff60d6c0640 10 monclient: init 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 0x7ff608140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff60809ea30 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff608131980 0x7ff608141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff608131980 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff608136cb0 0x7ff6081416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff608136cb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60809bf10 con 0x7ff608131980 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff608069710 con 0x7ff608136cb0 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6080992c0 con 0x7ff60809ea30 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 0x7ff608140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff60d6c0640 10 monclient(hunting): authenticate will time out at 2023.824319s 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff608131980 0x7ff608141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 0x7ff608140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34358/0 (socket says 172.21.15.153:34358) 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 -- 172.21.15.153:0/2526741475 learned_addr learned my addr 172.21.15.153:0/2526741475 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): get_auth_request con 0x7ff60809ea30 auth_method 0 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): get_auth_request con 0x7ff608131980 auth_method 0 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient(hunting): handle_auth_done global_id 244400 payload 824 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff6077fe640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff608136cb0 0x7ff6081416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 -- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff608136cb0 msgr2=0x7ff6081416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff608136cb0 0x7ff6081416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 -- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff608131980 msgr2=0x7ff608141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff608131980 0x7ff608141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 monclient: found mon.noname-c 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff606ffd640 1 -- 172.21.15.153:0/2526741475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0009450 con 0x7ff60809ea30 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff6077fe640 10 monclient: get_auth_request con 0x7ff608136cb0 auth_method 0 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff6077fe640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff608136cb0 0x7ff6081416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.185+0000 7ff5fffff640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff608131980 0x7ff608141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.187+0000 7ff606ffd640 1 -- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 msgr2=0x7ff608140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.187+0000 7ff606ffd640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 0x7ff608140c40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c5e0 tx=0x7ff5f8007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.187+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8012840 con 0x7ff60809ea30 2024-09-15T03:50:26.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.187+0000 7ff5feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: dump: 2024-09-15T03:50:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f80129a0 con 0x7ff60809ea30 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 5 monclient: authenticate success, global_id 244400 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8012da0 con 0x7ff60809ea30 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 10 monclient: _renew_subs 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff600009a20 con 0x7ff60809ea30 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 10 monclient: dump: 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff605ffb640 4 set_mon_vals no callback set 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 10 monclient: _renew_subs 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff608098db0 con 0x7ff60809ea30 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff5f8011590 con 0x7ff60809ea30 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.188+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff60809b010 con 0x7ff60809ea30 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5feffd640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff5e8038740 0x7ff5e803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff5f804c710 con 0x7ff60809ea30 2024-09-15T03:50:26.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5feffd640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7ff5e8041570 0x7ff5e80439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 --> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5e8044090 con 0x7ff5e8041570 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=259) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5f8011840 con 0x7ff60809ea30 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5feffd640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff6077fe640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7ff5e8041570 0x7ff5e80439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5fffff640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff5e8038740 0x7ff5e803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff6077fe640 10 monclient: get_auth_request con 0x7ff5e8041570 auth_method 0 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.189+0000 7ff5fffff640 10 monclient: get_auth_request con 0x7ff5e8038740 auth_method 0 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.191+0000 7ff5fffff640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff5e8038740 0x7ff5e803abf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0006c90 tx=0x7ff5f00091d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.193+0000 7ff6077fe640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7ff5e8041570 0x7ff5e80439c0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.194+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== osd.2 v2:172.21.15.153:6816/31924 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff5e8044090 con 0x7ff5e8041570 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.205+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 --> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff60805ed70 con 0x7ff5e8041570 2024-09-15T03:50:26.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.205+0000 7ff5feffd640 1 -- 172.21.15.153:0/2526741475 <== osd.2 v2:172.21.15.153:6816/31924 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff60805ed70 con 0x7ff5e8041570 2024-09-15T03:50:26.419 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:26.419 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:50:26.419 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.205+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7ff5e8041570 msgr2=0x7ff5e80439c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.205+0000 7ff60d6c0640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:6816/31924,v1:172.21.15.153:6817/31924] conn(0x7ff5e8041570 0x7ff5e80439c0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff5e8038740 msgr2=0x7ff5e803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff5e8038740 0x7ff5e803abf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0006c90 tx=0x7ff5f00091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 10 monclient: shutdown 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 msgr2=0x7ff608140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 --2- 172.21.15.153:0/2526741475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff60809ea30 0x7ff608140c40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c5e0 tx=0x7ff5f8007ed0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff6077fe640 1 -- 172.21.15.153:0/2526741475 reap_dead start 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 shutdown_connections 2024-09-15T03:50:26.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 >> 172.21.15.153:0/2526741475 conn(0x7ff6080a5ad0 msgr2=0x7ff60809e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 shutdown_connections 2024-09-15T03:50:26.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.206+0000 7ff60d6c0640 1 -- 172.21.15.153:0/2526741475 wait complete. 2024-09-15T03:50:26.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.554+0000 7f85b8034640 10 monclient: get_monmap_and_config 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.554+0000 7f85b8034640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.554+0000 7f85b8034640 10 monclient: monmap: 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:26.554970+0000 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:26.554970+0000 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.555 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:26.556 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:26.556 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:26.556 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:26.556 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:26.556 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.554+0000 7f85b8034640 1 Processor -- start 2024-09-15T03:50:26.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.554+0000 7f85b8034640 1 -- start start 2024-09-15T03:50:26.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.554+0000 7f85b8034640 10 monclient: init 2024-09-15T03:50:26.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-15T03:50:26.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85b009a510 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:26.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85b00973b0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 0x7f85b01380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient(hunting): picked mon.noname-b con 0x7f85b00977b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b006ee10 con 0x7f85b009a510 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b009cb00 con 0x7f85b00973b0 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b009a8e0 con 0x7f85b00977b0 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b8034640 10 monclient(hunting): authenticate will time out at 2024.194481s 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b5dd0640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f85b009a510 0x7f85b0096ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50102/0 (socket says 172.21.15.153:50102) 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b5dd0640 1 -- 172.21.15.153:0/4238823525 learned_addr learned my addr 172.21.15.153:0/4238823525 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 0x7f85b01380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): get_auth_request con 0x7f85b00977b0 auth_method 0 2024-09-15T03:50:26.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient(hunting): handle_auth_done global_id 244403 payload 274 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 1 -- 172.21.15.153:0/4238823525 >> v1:172.21.15.181:6789/0 conn(0x7f85b00973b0 legacy=0x7f85b01349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 1 -- 172.21.15.153:0/4238823525 >> v1:172.21.15.153:6789/0 conn(0x7f85b009a510 legacy=0x7f85b0096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 1 monclient: found mon.noname-b 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.555+0000 7f85b65d1640 1 -- 172.21.15.153:0/4238823525 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b009d590 con 0x7f85b00977b0 2024-09-15T03:50:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b65d1640 1 -- 172.21.15.153:0/4238823525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 msgr2=0x7f85b01380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b65d1640 1 --2- 172.21.15.153:0/4238823525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 0x7f85b01380d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e90 tx=0x7f85a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1affcfb6b53d0d3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 1 -- 172.21.15.153:0/4238823525 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8007840 con 0x7f85b00977b0 2024-09-15T03:50:26.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: dump: 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.571 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 1 -- 172.21.15.153:0/4238823525 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a802d050 con 0x7f85b00977b0 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b8034640 5 monclient: authenticate success, global_id 244403 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b8034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b8034640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b8034640 4 set_mon_vals no callback set 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 1 -- 172.21.15.153:0/4238823525 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8007bd0 con 0x7f85b00977b0 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:26.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.558+0000 7f85b4dce640 10 monclient: dump: 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 10 monclient: shutdown 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 -- 172.21.15.153:0/4238823525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 msgr2=0x7f85b01380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 --2- 172.21.15.153:0/4238823525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 0x7f85b01380d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e90 tx=0x7f85a8009780 comp rx=0 tx=0).stop 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 -- 172.21.15.153:0/4238823525 shutdown_connections 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 --2- 172.21.15.153:0/4238823525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b00977b0 0x7f85b01380d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 -- 172.21.15.153:0/4238823525 >> 172.21.15.153:0/4238823525 conn(0x7f85b00672b0 msgr2=0x7f85b00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 -- 172.21.15.153:0/4238823525 shutdown_connections 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.559+0000 7f85b8034640 1 -- 172.21.15.153:0/4238823525 wait complete. 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient: monmap: 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:26.574 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 Processor -- start 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 -- start start 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient: init 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85b013ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): picked mon.noname-b con 0x7f85b00973b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:26.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f85b01319f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): picked mon.noname-a con 0x7f85b00977b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): picked mon.noname-c con 0x7f85b009a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b011d5a0 con 0x7f85b00977b0 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b011f100 con 0x7f85b00973b0 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b013fbe0 con 0x7f85b009a510 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.560+0000 7f85b8034640 10 monclient(hunting): authenticate will time out at 2024.199953s 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f85b01319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34372/0 (socket says 172.21.15.153:34372) 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 learned_addr learned my addr 172.21.15.153:0/147522497 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): get_auth_request con 0x7f85b00977b0 auth_method 0 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): get_auth_request con 0x7f85b009a510 auth_method 0 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85b013ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): get_auth_request con 0x7f85b00973b0 auth_method 0 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b5dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient(hunting): handle_auth_done global_id 244406 payload 824 2024-09-15T03:50:26.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 msgr2=0x7f85b013ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85b013ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 msgr2=0x7f85b01319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f85b01319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 monclient: found mon.noname-c 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a8009a20 con 0x7f85b009a510 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.561+0000 7f85b55cf640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f85b01319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 msgr2=0x7f85b011b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85b65d1640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e90 tx=0x7f85a8037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8007ec0 con 0x7f85b009a510 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: dump: 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.581 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8040070 con 0x7f85b009a510 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85b8034640 5 monclient: authenticate success, global_id 244406 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8045430 con 0x7f85b009a510 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85b8034640 10 monclient: _renew_subs 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85b8034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b011edf0 con 0x7f85b009a510 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a3fff640 4 set_mon_vals no callback set 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.563+0000 7f85a1ffb640 10 monclient: dump: 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.584 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 10 monclient: _renew_subs 2024-09-15T03:50:26.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b011eb10 con 0x7f85b009a510 2024-09-15T03:50:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85a8042030 con 0x7f85b009a510 2024-09-15T03:50:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85a1ffb640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8594038740 0x7f859403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f85a802ae40 con 0x7f85b009a510 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.564+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b009c500 con 0x7f85b009a510 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 msgr2=0x7f85b011b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 msgr2=0x7f85b011b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e90 tx=0x7f85a8037000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e90 tx=0x7f85a8037000 comp rx=0 tx=0).stop 2024-09-15T03:50:26.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 msgr2=0x7f85b011b4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85b009a510 0x7f85b011b4a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f8594042dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): picked mon.a con 0x7f85b00977b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:26.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85940456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): picked mon.b con 0x7f85b00973b0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8594046050 0x7f85940484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): picked mon.c con 0x7f8594046050 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594048f40 con 0x7f85b00977b0 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594049080 con 0x7f85b00973b0 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8594049240 con 0x7f8594046050 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85a1ffb640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f8594042dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8594038740 0x7f859403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): get_auth_request con 0x7f8594038740 auth_method 0 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): get_auth_request con 0x7f85b00977b0 auth_method 0 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8594046050 0x7f85940484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85940456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): get_auth_request con 0x7f8594046050 auth_method 0 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): get_auth_request con 0x7f85b00973b0 auth_method 0 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b5dd0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b65d1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient(hunting): handle_auth_done global_id 244406 payload 293 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8594046050 msgr2=0x7f85940484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8594046050 0x7f85940484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 msgr2=0x7f85940456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85940456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 monclient: found mon.a 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f85940494c0 con 0x7f85b00977b0 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac002d60 con 0x7f85b00977b0 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.565+0000 7f85b55cf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.567+0000 7f85b5dd0640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8594038740 0x7f859403abf0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f85a40078e0 tx=0x7f85a4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.567+0000 7f85b55cf640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 msgr2=0x7f8594042dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.567+0000 7f85b55cf640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f8594042dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00f460 tx=0x7f85ac00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac00fe90 con 0x7f85b00977b0 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a1ffb640 10 monclient: dump: 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.594 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac007030 con 0x7f85b00977b0 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.568+0000 7f85a37fe640 4 set_mon_vals no callback set 2024-09-15T03:50:26.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.569+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f85ac0071a0 con 0x7f85b00977b0 2024-09-15T03:50:26.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.572+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85ac01a320 con 0x7f85b00977b0 2024-09-15T03:50:26.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.572+0000 7f85a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:26.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.572+0000 7f85a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.727+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f85b011f2d0 con 0x7f8594038740 2024-09-15T03:50:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.728+0000 7f85a1ffb640 1 -- 172.21.15.153:0/147522497 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f85b011f2d0 con 0x7f8594038740 2024-09-15T03:50:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T03:50:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8594038740 msgr2=0x7f859403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8594038740 0x7f859403abf0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f85a40078e0 tx=0x7f85a4007f00 comp rx=0 tx=0).stop 2024-09-15T03:50:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 10 monclient: shutdown 2024-09-15T03:50:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 msgr2=0x7f8594042dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f8594042dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00f460 tx=0x7f85ac00f490 comp rx=0 tx=0).stop 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 shutdown_connections 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8594046050 0x7f85940484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85b00973b0 0x7f85940456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85b00977b0 0x7f8594042dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 --2- 172.21.15.153:0/147522497 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f8594038740 0x7f859403abf0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 >> 172.21.15.153:0/147522497 conn(0x7f85b00672b0 msgr2=0x7f85b01328c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 shutdown_connections 2024-09-15T03:50:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.730+0000 7f85b8034640 1 -- 172.21.15.153:0/147522497 wait complete. 2024-09-15T03:50:26.742 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T03:50:26.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:50:26.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.838+0000 7fef0c280640 10 monclient: get_monmap_and_config 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.838+0000 7fef0c280640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.838+0000 7fef0c280640 10 monclient: monmap: 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:26.839517+0000 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:26.839517+0000 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:26.840 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:26.841 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:26.841 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 1 Processor -- start 2024-09-15T03:50:26.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 1 -- start start 2024-09-15T03:50:26.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient: init 2024-09-15T03:50:26.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T03:50:26.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef04098360 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:26.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04136c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient(hunting): picked mon.noname-b con 0x7fef04098730 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef04098b60 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef04096c70 con 0x7fef04098b60 2024-09-15T03:50:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef04132620 con 0x7fef04098360 2024-09-15T03:50:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0409e0b0 con 0x7fef04098730 2024-09-15T03:50:26.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.839+0000 7fef0c280640 10 monclient(hunting): authenticate will time out at 2024.478980s 2024-09-15T03:50:26.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0a81d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fef04098b60 0x7fef04139180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50104/0 (socket says 172.21.15.153:50104) 2024-09-15T03:50:26.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0a81d640 1 -- 172.21.15.153:0/661221256 learned_addr learned my addr 172.21.15.153:0/661221256 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:26.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 1 --2- 172.21.15.153:0/661221256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04136c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34382/0 (socket says 172.21.15.153:34382) 2024-09-15T03:50:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): get_auth_request con 0x7fef04098730 auth_method 0 2024-09-15T03:50:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient(hunting): handle_auth_done global_id 244409 payload 274 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 1 -- 172.21.15.153:0/661221256 >> v1:172.21.15.181:6789/0 conn(0x7fef04098360 legacy=0x7fef04131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 1 -- 172.21.15.153:0/661221256 >> v1:172.21.15.153:6789/0 conn(0x7fef04098b60 legacy=0x7fef04139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 1 monclient: found mon.noname-b 2024-09-15T03:50:26.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.840+0000 7fef0981b640 1 -- 172.21.15.153:0/661221256 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0409ced0 con 0x7fef04098730 2024-09-15T03:50:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0981b640 1 -- 172.21.15.153:0/661221256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 msgr2=0x7fef04136c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0981b640 1 --2- 172.21.15.153:0/661221256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04136c30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feef4009880 tx=0x7feef40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04a6f01ec548c8a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 1 -- 172.21.15.153:0/661221256 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4032440 con 0x7fef04098730 2024-09-15T03:50:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: dump: 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.857 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 1 -- 172.21.15.153:0/661221256 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef40325a0 con 0x7fef04098730 2024-09-15T03:50:26.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 5 monclient: authenticate success, global_id 244409 2024-09-15T03:50:26.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:26.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 1 -- 172.21.15.153:0/661221256 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4032940 con 0x7fef04098730 2024-09-15T03:50:26.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0901a640 10 monclient: dump: 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.860 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.861 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.861 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.861 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.861 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.861 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 4 set_mon_vals no callback set 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 10 monclient: shutdown 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 1 -- 172.21.15.153:0/661221256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 msgr2=0x7fef04136c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 1 --2- 172.21.15.153:0/661221256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04136c30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feef4009880 tx=0x7feef40090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 1 -- 172.21.15.153:0/661221256 shutdown_connections 2024-09-15T03:50:26.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 1 --2- 172.21.15.153:0/661221256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04136c30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 1 -- 172.21.15.153:0/661221256 >> 172.21.15.153:0/661221256 conn(0x7fef040a1d20 msgr2=0x7fef040a20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.843+0000 7fef0c280640 1 -- 172.21.15.153:0/661221256 shutdown_connections 2024-09-15T03:50:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.844+0000 7fef0c280640 1 -- 172.21.15.153:0/661221256 wait complete. 2024-09-15T03:50:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.844+0000 7fef0c280640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.844+0000 7fef0c280640 10 monclient: monmap: 2024-09-15T03:50:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.864 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:26.864 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:26.864 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:26.864 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.864 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.864 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.844+0000 7fef0c280640 1 Processor -- start 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.844+0000 7fef0c280640 1 -- start start 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.844+0000 7fef0c280640 10 monclient: init 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 0x7fef04141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): picked mon.noname-a con 0x7fef04098360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04130010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): picked mon.noname-c con 0x7fef04098730 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef04098b60 0x7fef04119b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): picked mon.noname-b con 0x7fef04098b60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:26.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0409d400 con 0x7fef04098360 2024-09-15T03:50:26.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef04130f10 con 0x7fef04098b60 2024-09-15T03:50:26.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0411d130 con 0x7fef04098730 2024-09-15T03:50:26.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0c280640 10 monclient(hunting): authenticate will time out at 2024.484345s 2024-09-15T03:50:26.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 0x7fef04141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04130010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 0x7fef04141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55560/0 (socket says 172.21.15.153:55560) 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 -- 172.21.15.153:0/458463143 learned_addr learned my addr 172.21.15.153:0/458463143 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): get_auth_request con 0x7fef04098360 auth_method 0 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): get_auth_request con 0x7fef04098730 auth_method 0 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0981b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a81d640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef04098b60 0x7fef04119b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a81d640 10 monclient(hunting): get_auth_request con 0x7fef04098b60 auth_method 0 2024-09-15T03:50:26.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a81d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a81d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a81d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient(hunting): handle_auth_done global_id 254295 payload 824 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 -- 172.21.15.153:0/458463143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 msgr2=0x7fef04130010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef04098730 0x7fef04130010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 -- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef04098b60 msgr2=0x7fef04119b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef04098b60 0x7fef04119b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 monclient: found mon.noname-a 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a01c640 1 -- 172.21.15.153:0/458463143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4009390 con 0x7fef04098360 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.845+0000 7fef0a81d640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef04098b60 0x7fef04119b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.847+0000 7fef0a01c640 1 -- 172.21.15.153:0/458463143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 msgr2=0x7fef04141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.847+0000 7fef0a01c640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 0x7fef04141150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fef0000c5e0 tx=0x7fef00007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00012840 con 0x7fef04098360 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: dump: 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.872 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef000129a0 con 0x7fef04098360 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 5 monclient: authenticate success, global_id 254295 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 10 monclient: _renew_subs 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef04141f80 con 0x7fef04098360 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00012da0 con 0x7fef04098360 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 10 monclient: dump: 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef08819640 4 set_mon_vals no callback set 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 10 monclient: _renew_subs 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0411bab0 con 0x7fef04098360 2024-09-15T03:50:26.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef0405ed70 con 0x7fef04098360 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.848+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fef00011590 con 0x7fef04098360 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7feefa7fc640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feee8038740 0x7feee803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fef0004c6f0 con 0x7fef04098360 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7feefa7fc640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7feee80415f0 0x7feee8043a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 --> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feee8044110 con 0x7feee80415f0 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=259) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef0004ec10 con 0x7fef04098360 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7feefa7fc640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7fef0981b640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feee8038740 0x7feee803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7fef0981b640 10 monclient: get_auth_request con 0x7feee8038740 auth_method 0 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7fef0a81d640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7feee80415f0 0x7feee8043a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.849+0000 7fef0a81d640 10 monclient: get_auth_request con 0x7feee80415f0 auth_method 0 2024-09-15T03:50:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.851+0000 7fef0981b640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feee8038740 0x7feee803abf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7feef4034000 tx=0x7feef4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.854+0000 7fef0a81d640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7feee80415f0 0x7feee8043a40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.854+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== osd.3 v2:172.21.15.181:6808/31874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7feee8044110 con 0x7feee80415f0 2024-09-15T03:50:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 --> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fef04141690 con 0x7feee80415f0 2024-09-15T03:50:26.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7feefa7fc640 1 -- 172.21.15.153:0/458463143 <== osd.3 v2:172.21.15.181:6808/31874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fef04141690 con 0x7feee80415f0 2024-09-15T03:50:26.877 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:26.877 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:50:26.877 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7feee80415f0 msgr2=0x7feee8043a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6808/31874,v1:172.21.15.181:6809/31874] conn(0x7feee80415f0 0x7feee8043a40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feee8038740 msgr2=0x7feee803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7feee8038740 0x7feee803abf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7feef4034000 tx=0x7feef4032000 comp rx=0 tx=0).stop 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 10 monclient: shutdown 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 msgr2=0x7fef04141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 --2- 172.21.15.153:0/458463143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef04098360 0x7fef04141150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fef0000c5e0 tx=0x7fef00007ed0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0a81d640 1 -- 172.21.15.153:0/458463143 reap_dead start 2024-09-15T03:50:26.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 shutdown_connections 2024-09-15T03:50:26.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.866+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 >> 172.21.15.153:0/458463143 conn(0x7fef040a1d20 msgr2=0x7fef040970b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.867+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 shutdown_connections 2024-09-15T03:50:26.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.867+0000 7fef0c280640 1 -- 172.21.15.153:0/458463143 wait complete. 2024-09-15T03:50:26.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T03:50:26.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.974+0000 7f765ea57640 10 monclient: get_monmap_and_config 2024-09-15T03:50:26.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.974+0000 7f765ea57640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.974+0000 7f765ea57640 10 monclient: monmap: 2024-09-15T03:50:26.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.975 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:26.975241+0000 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:26.975241+0000 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:26.976 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 1 Processor -- start 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 1 -- start start 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient: init 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7658131980 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f765809fda0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f765809f0d0 0x7f7658139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient(hunting): picked mon.noname-a con 0x7f765809f0d0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765809f510 con 0x7f7658131980 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76580a1180 con 0x7f765809fda0 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76580a1460 con 0x7f765809f0d0 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765ea57640 10 monclient(hunting): authenticate will time out at 2024.614871s 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765cff4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f765809f0d0 0x7f7658139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f7657fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7658131980 0x7f7658131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50108/0 (socket says 172.21.15.153:50108) 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f7657fff640 1 -- 172.21.15.153:0/1832670431 learned_addr learned my addr 172.21.15.153:0/1832670431 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765cff4640 10 monclient(hunting): get_auth_request con 0x7f765809f0d0 auth_method 0 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765cff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765cff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.975+0000 7f765cff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 10 monclient(hunting): handle_auth_done global_id 254298 payload 274 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 1 -- 172.21.15.153:0/1832670431 >> v1:172.21.15.181:6789/0 conn(0x7f765809fda0 legacy=0x7f765809e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 1 -- 172.21.15.153:0/1832670431 >> v1:172.21.15.153:6789/0 conn(0x7f7658131980 legacy=0x7f7658131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 1 monclient: found mon.noname-a 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.976+0000 7f765cff4640 1 -- 172.21.15.153:0/1832670431 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658132460 con 0x7f765809f0d0 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.978+0000 7f765cff4640 1 -- 172.21.15.153:0/1832670431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809f0d0 msgr2=0x7f7658139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.978+0000 7f765cff4640 1 --2- 172.21.15.153:0/1832670431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809f0d0 0x7f7658139cd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7650009e60 tx=0x7f7650009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3278b2adda53675e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 1 -- 172.21.15.153:0/1832670431 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650007630 con 0x7f765809f0d0 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: dump: 2024-09-15T03:50:26.987 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 1 -- 172.21.15.153:0/1832670431 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765002d050 con 0x7f765809f0d0 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 5 monclient: authenticate success, global_id 254298 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 1 -- 172.21.15.153:0/1832670431 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76500079c0 con 0x7f765809f0d0 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 4 set_mon_vals no callback set 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f7656ffd640 10 monclient: dump: 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.990 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 10 monclient: shutdown 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 -- 172.21.15.153:0/1832670431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809f0d0 msgr2=0x7f7658139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 --2- 172.21.15.153:0/1832670431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809f0d0 0x7f7658139cd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7650009e60 tx=0x7f7650009780 comp rx=0 tx=0).stop 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 -- 172.21.15.153:0/1832670431 shutdown_connections 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 --2- 172.21.15.153:0/1832670431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809f0d0 0x7f7658139cd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 -- 172.21.15.153:0/1832670431 >> 172.21.15.153:0/1832670431 conn(0x7f7658067bd0 msgr2=0x7f76580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 -- 172.21.15.153:0/1832670431 shutdown_connections 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.979+0000 7f765ea57640 1 -- 172.21.15.153:0/1832670431 wait complete. 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.980+0000 7f765ea57640 10 monclient: build_initial_monmap 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.980+0000 7f765ea57640 10 monclient: monmap: 2024-09-15T03:50:26.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.980+0000 7f765ea57640 1 Processor -- start 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.980+0000 7f765ea57640 1 -- start start 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.980+0000 7f765ea57640 10 monclient: init 2024-09-15T03:50:26.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 0x7f76581383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): picked mon.noname-c con 0x7f765809f0d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 0x7f7658133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): picked mon.noname-a con 0x7f765809fda0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7658131980 0x7f76581341c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): picked mon.noname-b con 0x7f7658131980 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765809bf10 con 0x7f765809fda0 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7658139fd0 con 0x7f7658131980 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:26.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7658137fe0 con 0x7f765809f0d0 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765ea57640 10 monclient(hunting): authenticate will time out at 2024.620166s 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 0x7f7658133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 0x7f76581383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 0x7f7658133c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55574/0 (socket says 172.21.15.153:55574) 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 0x7f76581383d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34396/0 (socket says 172.21.15.153:34396) 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 -- 172.21.15.153:0/2060367710 learned_addr learned my addr 172.21.15.153:0/2060367710 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): get_auth_request con 0x7f765809fda0 auth_method 0 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): get_auth_request con 0x7f765809f0d0 auth_method 0 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7658131980 0x7f76581341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): get_auth_request con 0x7f7658131980 auth_method 0 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:26.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f765cff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient(hunting): handle_auth_done global_id 254301 payload 824 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 -- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 msgr2=0x7f76581383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 0x7f76581383d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 -- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7658131980 msgr2=0x7f76581341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7658131980 0x7f76581341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient: _finish_hunting 0 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 monclient: found mon.noname-a 2024-09-15T03:50:26.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f76577fe640 1 -- 172.21.15.153:0/2060367710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7650009a20 con 0x7f765809fda0 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.981+0000 7f7657fff640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 0x7f76581383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f76577fe640 1 -- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 msgr2=0x7f7658133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f76577fe640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 0x7f7658133c80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f764000c5e0 tx=0x7f7640007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640012840 con 0x7f765809fda0 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 10 monclient: dump: 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 10 monclient: _finish_auth 0 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76400129a0 con 0x7f765809fda0 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f765ea57640 5 monclient: authenticate success, global_id 254301 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f765ea57640 10 monclient: _renew_subs 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f765ea57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7658135dd0 con 0x7f765809fda0 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640012d60 con 0x7f765809fda0 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.983+0000 7f764ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f764ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f764ffff640 10 monclient: dump: 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f76567fc640 4 set_mon_vals no callback set 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 10 monclient: _renew_subs 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7658136430 con 0x7f765809fda0 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f7640011590 con 0x7f765809fda0 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.984+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765804bf60 con 0x7f765809fda0 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.985+0000 7f764ffff640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7638038740 0x7f763803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.985+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f764004c630 con 0x7f765809fda0 2024-09-15T03:50:27.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.985+0000 7f7657fff640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7638038740 0x7f763803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.985+0000 7f7657fff640 10 monclient: get_auth_request con 0x7f7638038740 auth_method 0 2024-09-15T03:50:27.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.987+0000 7f7657fff640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7638038740 0x7f763803abf0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7648007fd0 tx=0x7f7648007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.989+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f7640010ac0 con 0x7f765809fda0 2024-09-15T03:50:27.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.989+0000 7f764ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:26.989+0000 7f764ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:27.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.147+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7658135dd0 con 0x7f7638038740 2024-09-15T03:50:27.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.147+0000 7f764ffff640 1 -- 172.21.15.153:0/2060367710 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7658135dd0 con 0x7f7638038740 2024-09-15T03:50:27.148 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T03:50:27.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7638038740 msgr2=0x7f763803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7638038740 0x7f763803abf0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7648007fd0 tx=0x7f7648007f00 comp rx=0 tx=0).stop 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 10 monclient: shutdown 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 msgr2=0x7f7658133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 0x7f7658133c80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f764000c5e0 tx=0x7f7640007ed0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 shutdown_connections 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f7638038740 0x7f763803abf0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7658131980 0x7f76581341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f765809fda0 0x7f7658133c80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 --2- 172.21.15.153:0/2060367710 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f765809f0d0 0x7f76581383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 >> 172.21.15.153:0/2060367710 conn(0x7f7658067bd0 msgr2=0x7f76580a3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 shutdown_connections 2024-09-15T03:50:27.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.149+0000 7f765ea57640 1 -- 172.21.15.153:0/2060367710 wait complete. 2024-09-15T03:50:27.161 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T03:50:27.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:50:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.256+0000 7f69e1ca8640 10 monclient: get_monmap_and_config 2024-09-15T03:50:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.256+0000 7f69e1ca8640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.256+0000 7f69e1ca8640 10 monclient: monmap: 2024-09-15T03:50:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:27.257429+0000 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:27.257429+0000 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:27.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.257+0000 7f69e1ca8640 1 Processor -- start 2024-09-15T03:50:27.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.257+0000 7f69e1ca8640 1 -- start start 2024-09-15T03:50:27.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.257+0000 7f69e1ca8640 10 monclient: init 2024-09-15T03:50:27.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f69dc12d8d0 0x7f69dc09a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc12d8d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69dc12dca0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 0x7f69dc097720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc12e0a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0999b0 con 0x7f69dc12dca0 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69dc12e4a0 con 0x7f69dc12d8d0 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0996d0 con 0x7f69dc12e0a0 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69e1ca8640 10 monclient(hunting): authenticate will time out at 2024.897118s 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69d3fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f69dc12dca0 0x7f69dc097010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50124/0 (socket says 172.21.15.153:50124) 2024-09-15T03:50:27.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69d3fff640 1 -- 172.21.15.153:0/3949222086 learned_addr learned my addr 172.21.15.153:0/3949222086 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 1 --2- 172.21.15.153:0/3949222086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 0x7f69dc097720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): get_auth_request con 0x7f69dc12e0a0 auth_method 0 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69db7fe640 1 --2- 172.21.15.153:0/3949222086 >> v2:172.21.15.181:3300/0 conn(0x7f69dc12d8d0 0x7f69dc09a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69db7fe640 10 monclient(hunting): get_auth_request con 0x7f69dc12d8d0 auth_method 0 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient(hunting): handle_auth_done global_id 244418 payload 274 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 1 -- 172.21.15.153:0/3949222086 >> v2:172.21.15.181:3300/0 conn(0x7f69dc12d8d0 msgr2=0x7f69dc09a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 1 --2- 172.21.15.153:0/3949222086 >> v2:172.21.15.181:3300/0 conn(0x7f69dc12d8d0 0x7f69dc09a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 1 -- 172.21.15.153:0/3949222086 >> v1:172.21.15.153:6789/0 conn(0x7f69dc12dca0 legacy=0x7f69dc097010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 1 monclient: found mon.noname-b 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.258+0000 7f69dbfff640 1 -- 172.21.15.153:0/3949222086 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc09d480 con 0x7f69dc12e0a0 2024-09-15T03:50:27.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69dbfff640 1 -- 172.21.15.153:0/3949222086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 msgr2=0x7f69dc097720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69dbfff640 1 --2- 172.21.15.153:0/3949222086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 0x7f69dc097720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009e60 tx=0x7f69d4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a4a4eba7fc58436 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 1 -- 172.21.15.153:0/3949222086 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4007760 con 0x7f69dc12e0a0 2024-09-15T03:50:27.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: dump: 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69e1ca8640 5 monclient: authenticate success, global_id 244418 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 1 -- 172.21.15.153:0/3949222086 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d402d050 con 0x7f69dc12e0a0 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69e1ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69e1ca8640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 1 -- 172.21.15.153:0/3949222086 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4007b30 con 0x7f69dc12e0a0 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69e1ca8640 4 set_mon_vals no callback set 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.261+0000 7f69daffd640 10 monclient: dump: 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 10 monclient: shutdown 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 -- 172.21.15.153:0/3949222086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 msgr2=0x7f69dc097720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 --2- 172.21.15.153:0/3949222086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 0x7f69dc097720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009e60 tx=0x7f69d4009780 comp rx=0 tx=0).stop 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 -- 172.21.15.153:0/3949222086 shutdown_connections 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 --2- 172.21.15.153:0/3949222086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12e0a0 0x7f69dc097720 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 --2- 172.21.15.153:0/3949222086 >> v2:172.21.15.181:3300/0 conn(0x7f69dc12d8d0 0x7f69dc09a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 -- 172.21.15.153:0/3949222086 >> 172.21.15.153:0/3949222086 conn(0x7f69dc0a1150 msgr2=0x7f69dc0a1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 -- 172.21.15.153:0/3949222086 shutdown_connections 2024-09-15T03:50:27.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 -- 172.21.15.153:0/3949222086 wait complete. 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 10 monclient: monmap: 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 Processor -- start 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 1 -- start start 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.262+0000 7f69e1ca8640 10 monclient: init 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12d8d0 0x7f69dc09b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc12d8d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc12dca0 0x7f69dc119ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc12dca0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc12e0a0 0x7f69dc12c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc12e0a0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc11beb0 con 0x7f69dc12dca0 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc11a020 con 0x7f69dc12e0a0 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc11b960 con 0x7f69dc12d8d0 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69e1ca8640 10 monclient(hunting): authenticate will time out at 2024.902430s 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12d8d0 0x7f69dc09b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc12dca0 0x7f69dc119ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12d8d0 0x7f69dc09b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34408/0 (socket says 172.21.15.153:34408) 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): get_auth_request con 0x7f69dc12dca0 auth_method 0 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 1 -- 172.21.15.153:0/2958569732 learned_addr learned my addr 172.21.15.153:0/2958569732 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): get_auth_request con 0x7f69dc12d8d0 auth_method 0 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69dbfff640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc12e0a0 0x7f69dc12c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69dbfff640 10 monclient(hunting): get_auth_request con 0x7f69dc12e0a0 auth_method 0 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient(hunting): handle_auth_done global_id 254304 payload 824 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 -- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12d8d0 msgr2=0x7f69dc09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12d8d0 0x7f69dc09b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 -- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc12e0a0 msgr2=0x7f69dc12c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc12e0a0 0x7f69dc12c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 monclient: found mon.noname-a 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69d3fff640 1 -- 172.21.15.153:0/2958569732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4009a20 con 0x7f69dc12dca0 2024-09-15T03:50:27.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.263+0000 7f69db7fe640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69dc12d8d0 0x7f69dc09b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d3fff640 1 -- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc12dca0 msgr2=0x7f69dc119ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d3fff640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc12dca0 0x7f69dc119ae0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f69c400c490 tx=0x7f69c4007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4005cb0 con 0x7f69dc12dca0 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: dump: 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.278 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.279 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.279 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.279 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 5 monclient: authenticate success, global_id 254304 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c4005e30 con 0x7f69dc12dca0 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 10 monclient: _renew_subs 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc11ac00 con 0x7f69dc12dca0 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4007320 con 0x7f69dc12dca0 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69d37fe640 10 monclient: dump: 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 10 monclient: _renew_subs 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69da7fc640 4 set_mon_vals no callback set 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.266+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc11bc80 con 0x7f69dc12dca0 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69e1ca8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f69dc05ed70 con 0x7f69dc12dca0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f69c401b3b0 con 0x7f69dc12dca0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69bc038740 0x7f69bc03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f69c404baf0 con 0x7f69dc12dca0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f69bc0415f0 0x7f69bc043a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 --> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69bc044110 con 0x7f69bc0415f0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=259) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69c40502a0 con 0x7f69dc12dca0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69d37fe640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69db7fe640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69bc038740 0x7f69bc03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69db7fe640 10 monclient: get_auth_request con 0x7f69bc038740 auth_method 0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69dbfff640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f69bc0415f0 0x7f69bc043a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.267+0000 7f69dbfff640 10 monclient: get_auth_request con 0x7f69bc0415f0 auth_method 0 2024-09-15T03:50:27.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.270+0000 7f69db7fe640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69bc038740 0x7f69bc03abf0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f69cc010370 tx=0x7f69cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.272+0000 7f69dbfff640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f69bc0415f0 0x7f69bc043a40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.272+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== osd.4 v2:172.21.15.181:6800/31875 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f69bc044110 con 0x7f69bc0415f0 2024-09-15T03:50:27.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.283+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 --> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f69dc12cf90 con 0x7f69bc0415f0 2024-09-15T03:50:27.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.284+0000 7f69d37fe640 1 -- 172.21.15.153:0/2958569732 <== osd.4 v2:172.21.15.181:6800/31875 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f69dc12cf90 con 0x7f69bc0415f0 2024-09-15T03:50:27.285 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:27.285 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:50:27.285 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.284+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f69bc0415f0 msgr2=0x7f69bc043a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.284+0000 7f69e1ca8640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6800/31875,v1:172.21.15.181:6801/31875] conn(0x7f69bc0415f0 0x7f69bc043a40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69bc038740 msgr2=0x7f69bc03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f69bc038740 0x7f69bc03abf0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f69cc010370 tx=0x7f69cc008040 comp rx=0 tx=0).stop 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 10 monclient: shutdown 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc12dca0 msgr2=0x7f69dc119ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 --2- 172.21.15.153:0/2958569732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc12dca0 0x7f69dc119ae0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f69c400c490 tx=0x7f69c4007f00 comp rx=0 tx=0).stop 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69dbfff640 1 -- 172.21.15.153:0/2958569732 reap_dead start 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 shutdown_connections 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 >> 172.21.15.153:0/2958569732 conn(0x7f69dc0a1150 msgr2=0x7f69dc130f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 shutdown_connections 2024-09-15T03:50:27.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.285+0000 7f69e1ca8640 1 -- 172.21.15.153:0/2958569732 wait complete. 2024-09-15T03:50:27.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T03:50:27.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.388+0000 7fa02224f640 10 monclient: get_monmap_and_config 2024-09-15T03:50:27.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.388+0000 7fa02224f640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.388+0000 7fa02224f640 10 monclient: monmap: 2024-09-15T03:50:27.389 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.389 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.389 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:27.389522+0000 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:27.389522+0000 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:27.390 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.389+0000 7fa02224f640 1 Processor -- start 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.389+0000 7fa02224f640 1 -- start start 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.389+0000 7fa02224f640 10 monclient: init 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): picked mon.noname-b con 0x7fa01c096e10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa01c09aaa0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa01c0978f0 0x7fa01c135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): picked mon.noname-a con 0x7fa01c0978f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c09d4a0 con 0x7fa01c09aaa0 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c099620 con 0x7fa01c0978f0 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c099340 con 0x7fa01c096e10 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa02224f640 10 monclient(hunting): authenticate will time out at 2025.029212s 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa01c0978f0 0x7fa01c135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34410/0 (socket says 172.21.15.153:34410) 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 -- 172.21.15.153:0/1254507996 learned_addr learned my addr 172.21.15.153:0/1254507996 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 10 monclient(hunting): get_auth_request con 0x7fa01c096e10 auth_method 0 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa01c0978f0 auth_method 0 2024-09-15T03:50:27.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 -- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 msgr2=0x7fa01c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 -- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 msgr2=0x7fa01c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient(hunting): handle_auth_done global_id 254307 payload 274 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 1 -- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 msgr2=0x7fa01c09a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 1 -- 172.21.15.153:0/1254507996 >> v1:172.21.15.153:6789/0 conn(0x7fa01c09aaa0 legacy=0x7fa01c0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 1 monclient: found mon.noname-a 2024-09-15T03:50:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.390+0000 7fa01bfff640 1 -- 172.21.15.153:0/1254507996 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c099900 con 0x7fa01c0978f0 2024-09-15T03:50:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01bfff640 1 -- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 msgr2=0x7fa01c135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01bfff640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c135b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa014009e60 tx=0x7fa014009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634651802e09bd13 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 1 -- 172.21.15.153:0/1254507996 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014007760 con 0x7fa01c0978f0 2024-09-15T03:50:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: dump: 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa02224f640 5 monclient: authenticate success, global_id 254307 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 1 -- 172.21.15.153:0/1254507996 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01402d050 con 0x7fa01c0978f0 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa02224f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa02224f640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 1 -- 172.21.15.153:0/1254507996 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014007af0 con 0x7fa01c0978f0 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa02224f640 4 set_mon_vals no callback set 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.393+0000 7fa01a7fc640 10 monclient: dump: 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 10 monclient: shutdown 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 -- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 msgr2=0x7fa01c135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c135b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa014009e60 tx=0x7fa014009780 comp rx=0 tx=0).stop 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 -- 172.21.15.153:0/1254507996 shutdown_connections 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c135b40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 --2- 172.21.15.153:0/1254507996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c096e10 0x7fa01c09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 -- 172.21.15.153:0/1254507996 >> 172.21.15.153:0/1254507996 conn(0x7fa01c0672b0 msgr2=0x7fa01c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 -- 172.21.15.153:0/1254507996 shutdown_connections 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 1 -- 172.21.15.153:0/1254507996 wait complete. 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.394+0000 7fa02224f640 10 monclient: monmap: 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:27.405 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 Processor -- start 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 -- start start 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient: init 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 0x7fa01c13c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): picked mon.noname-b con 0x7fa01c096e10 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c12e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): picked mon.noname-a con 0x7fa01c0978f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 0x7fa01c11a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): picked mon.noname-c con 0x7fa01c09aaa0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c13d7a0 con 0x7fa01c0978f0 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0a0190 con 0x7fa01c096e10 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c11d9d0 con 0x7fa01c09aaa0 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa02224f640 10 monclient(hunting): authenticate will time out at 2025.034643s 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 0x7fa01c11a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c12e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 0x7fa01c11a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34414/0 (socket says 172.21.15.153:34414) 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 1 -- 172.21.15.153:0/1843871396 learned_addr learned my addr 172.21.15.153:0/1843871396 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c12e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55608/0 (socket says 172.21.15.153:55608) 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa01c09aaa0 auth_method 0 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): get_auth_request con 0x7fa01c0978f0 auth_method 0 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.395+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 0x7fa01c13c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 10 monclient(hunting): get_auth_request con 0x7fa01c096e10 auth_method 0 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 msgr2=0x7fa01c13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 10 monclient(hunting): handle_auth_done global_id 254310 payload 824 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 msgr2=0x7fa01c13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 0x7fa01c13c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 0x7fa01c13c8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 msgr2=0x7fa01c11a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 0x7fa01c11a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 msgr2=0x7fa01c13c8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 0x7fa01c13c8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 1 monclient: found mon.noname-a 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01affd640 1 -- 172.21.15.153:0/1843871396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014009a20 con 0x7fa01c0978f0 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.396+0000 7fa01bfff640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 0x7fa01c11a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa01affd640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 msgr2=0x7fa01c12e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa01affd640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c12e890 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa01000c4e0 tx=0x7fa010007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0100117a0 con 0x7fa01c0978f0 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: dump: 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa010011920 con 0x7fa01c0978f0 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 5 monclient: authenticate success, global_id 254310 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 10 monclient: _renew_subs 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010011ce0 con 0x7fa01c0978f0 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c11dcb0 con 0x7fa01c0978f0 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa0037fe640 10 monclient: dump: 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa019ffb640 4 set_mon_vals no callback set 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 10 monclient: _renew_subs 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.398+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c13cf80 con 0x7fa01c0978f0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fa010012630 con 0x7fa01c0978f0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa02224f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa02224f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa02224f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa0037fe640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa008038740 0x7fa00803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c11dea0 con 0x7fa01c0978f0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa01004c1b0 con 0x7fa01c0978f0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa008038740 0x7fa00803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.399+0000 7fa01b7fe640 10 monclient: get_auth_request con 0x7fa008038740 auth_method 0 2024-09-15T03:50:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.402+0000 7fa01b7fe640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa008038740 0x7fa00803abf0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa00c006fd0 tx=0x7fa00c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.403+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa01000e290 con 0x7fa01c0978f0 2024-09-15T03:50:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.404+0000 7fa0037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.404+0000 7fa0037fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:27.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.559+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa01c11e090 con 0x7fa008038740 2024-09-15T03:50:27.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.560+0000 7fa0037fe640 1 -- 172.21.15.153:0/1843871396 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa01c11e090 con 0x7fa008038740 2024-09-15T03:50:27.561 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T03:50:27.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa008038740 msgr2=0x7fa00803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa008038740 0x7fa00803abf0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa00c006fd0 tx=0x7fa00c008040 comp rx=0 tx=0).stop 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 10 monclient: shutdown 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 msgr2=0x7fa01c12e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c12e890 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa01000c4e0 tx=0x7fa010007e10 comp rx=0 tx=0).stop 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 shutdown_connections 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fa008038740 0x7fa00803abf0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa01c09aaa0 0x7fa01c11a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c0978f0 0x7fa01c12e890 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.561+0000 7fa02224f640 1 --2- 172.21.15.153:0/1843871396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa01c096e10 0x7fa01c13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.562+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 >> 172.21.15.153:0/1843871396 conn(0x7fa01c0672b0 msgr2=0x7fa01c09ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.562+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 shutdown_connections 2024-09-15T03:50:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.562+0000 7fa02224f640 1 -- 172.21.15.153:0/1843871396 wait complete. 2024-09-15T03:50:27.573 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T03:50:27.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.669+0000 7ff37835d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.669+0000 7ff37835d640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.669+0000 7ff37835d640 10 monclient: monmap: 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:27.670207+0000 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:27.670207+0000 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:27.671 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 Processor -- start 2024-09-15T03:50:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 -- start start 2024-09-15T03:50:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient: init 2024-09-15T03:50:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3700972e0 0x7ff3700957c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3700972e0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3700976b0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff370097ab0 0x7ff37009c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): picked mon.noname-b con 0x7ff370097ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff370064200 con 0x7ff3700976b0 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37009d920 con 0x7ff3700972e0 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff370097ef0 con 0x7ff370097ab0 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff37835d640 10 monclient(hunting): authenticate will time out at 2025.309682s 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3700972e0 0x7ff3700957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff370097ab0 0x7ff37009c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff3700972e0 0x7ff3700957c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55618/0 (socket says 172.21.15.153:55618) 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3758f8640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff3700976b0 0x7ff37009bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50144/0 (socket says 172.21.15.153:50144) 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 1 -- 172.21.15.153:0/1167940798 learned_addr learned my addr 172.21.15.153:0/1167940798 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): get_auth_request con 0x7ff370097ab0 auth_method 0 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): get_auth_request con 0x7ff3700972e0 auth_method 0 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3768fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.670+0000 7ff3760f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 10 monclient(hunting): handle_auth_done global_id 254313 payload 274 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 1 -- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff370097ab0 msgr2=0x7ff37009c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 1 --2- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff370097ab0 0x7ff37009c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 1 -- 172.21.15.153:0/1167940798 >> v1:172.21.15.153:6789/0 conn(0x7ff3700976b0 legacy=0x7ff37009bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 1 monclient: found mon.noname-a 2024-09-15T03:50:27.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:50:27.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3760f9640 1 -- 172.21.15.153:0/1167940798 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370069250 con 0x7ff3700972e0 2024-09-15T03:50:27.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.671+0000 7ff3768fa640 1 --2- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff370097ab0 0x7ff37009c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:27.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.673+0000 7ff3760f9640 1 -- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700972e0 msgr2=0x7ff3700957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.673+0000 7ff3760f9640 1 --2- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700972e0 0x7ff3700957c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009910 tx=0x7ff36c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3d6b7b75c2ac7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 1 -- 172.21.15.153:0/1167940798 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c032440 con 0x7ff3700972e0 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: dump: 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 1 -- 172.21.15.153:0/1167940798 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c0325a0 con 0x7ff3700972e0 2024-09-15T03:50:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 5 monclient: authenticate success, global_id 254313 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 1 -- 172.21.15.153:0/1167940798 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c032940 con 0x7ff3700972e0 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 4 set_mon_vals no callback set 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:50:27.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff3750f7640 10 monclient: dump: 2024-09-15T03:50:27.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.693 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.693 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.693 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.693 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.693 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 10 monclient: shutdown 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 -- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700972e0 msgr2=0x7ff3700957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 --2- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700972e0 0x7ff3700957c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009910 tx=0x7ff36c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 -- 172.21.15.153:0/1167940798 shutdown_connections 2024-09-15T03:50:27.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 --2- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff370097ab0 0x7ff37009c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 --2- 172.21.15.153:0/1167940798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700972e0 0x7ff3700957c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 -- 172.21.15.153:0/1167940798 >> 172.21.15.153:0/1167940798 conn(0x7ff3700a1400 msgr2=0x7ff3700a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 -- 172.21.15.153:0/1167940798 shutdown_connections 2024-09-15T03:50:27.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.674+0000 7ff37835d640 1 -- 172.21.15.153:0/1167940798 wait complete. 2024-09-15T03:50:27.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient: monmap: 2024-09-15T03:50:27.696 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.696 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:27.697 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:27.697 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:27.697 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.698 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.698 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 Processor -- start 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 -- start start 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient: init 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 0x7ff370131cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3700972e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700976b0 0x7ff37013ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3700976b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff370097ab0 0x7ff37013f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): picked mon.noname-b con 0x7ff370097ab0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37012bba0 con 0x7ff3700976b0 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37012adf0 con 0x7ff370097ab0 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37012d1b0 con 0x7ff3700972e0 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff37835d640 10 monclient(hunting): authenticate will time out at 2025.314921s 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.675+0000 7ff3760f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 0x7ff370131cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700976b0 0x7ff37013ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 0x7ff370131cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34420/0 (socket says 172.21.15.153:34420) 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 -- 172.21.15.153:0/332097679 learned_addr learned my addr 172.21.15.153:0/332097679 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): get_auth_request con 0x7ff3700972e0 auth_method 0 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): get_auth_request con 0x7ff3700976b0 auth_method 0 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3768fa640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff370097ab0 0x7ff37013f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3768fa640 10 monclient(hunting): get_auth_request con 0x7ff370097ab0 auth_method 0 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3768fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3768fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3768fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient(hunting): handle_auth_done global_id 244430 payload 824 2024-09-15T03:50:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 -- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff370097ab0 msgr2=0x7ff37013f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff370097ab0 0x7ff37013f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 -- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700976b0 msgr2=0x7ff37013ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700976b0 0x7ff37013ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 monclient: found mon.noname-c 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3760f9640 1 -- 172.21.15.153:0/332097679 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368009a20 con 0x7ff3700972e0 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.676+0000 7ff3758f8640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3700976b0 0x7ff37013ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3760f9640 1 -- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 msgr2=0x7ff370131cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3760f9640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 0x7ff370131cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff36c035040 tx=0x7ff36c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c031a90 con 0x7ff3700972e0 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: dump: 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c031c10 con 0x7ff3700972e0 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 5 monclient: authenticate success, global_id 244430 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 10 monclient: _renew_subs 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c03d440 con 0x7ff3700972e0 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c009450 con 0x7ff3700972e0 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3667fc640 10 monclient: dump: 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff3748f6640 4 set_mon_vals no callback set 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 10 monclient: _renew_subs 2024-09-15T03:50:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.678+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37012c090 con 0x7ff3700972e0 2024-09-15T03:50:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff37835d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff37005ed70 con 0x7ff3700972e0 2024-09-15T03:50:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ff36c039040 con 0x7ff3700972e0 2024-09-15T03:50:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff3667fc640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff354038740 0x7ff35403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff36c06bf70 con 0x7ff3700972e0 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff3667fc640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7ff3540415f0 0x7ff354043a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.679+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff354044110 con 0x7ff3540415f0 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.680+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=259) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff36c06e500 con 0x7ff3700972e0 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.680+0000 7ff3667fc640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.680+0000 7ff3758f8640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff354038740 0x7ff35403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.680+0000 7ff3758f8640 10 monclient: get_auth_request con 0x7ff354038740 auth_method 0 2024-09-15T03:50:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.680+0000 7ff3768fa640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7ff3540415f0 0x7ff354043a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.680+0000 7ff3768fa640 10 monclient: get_auth_request con 0x7ff3540415f0 auth_method 0 2024-09-15T03:50:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.682+0000 7ff3758f8640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff354038740 0x7ff35403abf0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff37012b7d0 tx=0x7ff360007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.684+0000 7ff3768fa640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7ff3540415f0 0x7ff354043a40 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.685+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== osd.5 v2:172.21.15.181:6816/31876 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff354044110 con 0x7ff3540415f0 2024-09-15T03:50:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.696+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 --> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff370132450 con 0x7ff3540415f0 2024-09-15T03:50:27.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff3667fc640 1 -- 172.21.15.153:0/332097679 <== osd.5 v2:172.21.15.181:6816/31876 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff370132450 con 0x7ff3540415f0 2024-09-15T03:50:27.711 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-15T03:50:27.711 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:50:27.711 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-15T03:50:27.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7ff3540415f0 msgr2=0x7ff354043a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6816/31876,v1:172.21.15.181:6817/31876] conn(0x7ff3540415f0 0x7ff354043a40 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff354038740 msgr2=0x7ff35403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ff354038740 0x7ff35403abf0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff37012b7d0 tx=0x7ff360007e40 comp rx=0 tx=0).stop 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 10 monclient: shutdown 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 msgr2=0x7ff370131cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 --2- 172.21.15.153:0/332097679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3700972e0 0x7ff370131cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff36c035040 tx=0x7ff36c032000 comp rx=0 tx=0).stop 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff3768fa640 1 -- 172.21.15.153:0/332097679 reap_dead start 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 shutdown_connections 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.697+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 >> 172.21.15.153:0/332097679 conn(0x7ff3700a1400 msgr2=0x7ff37006c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.698+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 shutdown_connections 2024-09-15T03:50:27.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.698+0000 7ff37835d640 1 -- 172.21.15.153:0/332097679 wait complete. 2024-09-15T03:50:27.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.807+0000 7ffaa9caf640 10 monclient: get_monmap_and_config 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.807+0000 7ffaa9caf640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.807+0000 7ffaa9caf640 10 monclient: monmap: 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:27.808103+0000 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:27.808103+0000 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:27.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:27.809 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:27.809 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 Processor -- start 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 -- start start 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient: init 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7ffaa409a510 0x7ffaa4096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): picked mon.noname-c con 0x7ffaa409a510 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffaa40971e0 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:27.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 0x7ffaa412fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): picked mon.noname-b con 0x7ffaa40975e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa406ee10 con 0x7ffaa40971e0 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa409cb00 con 0x7ffaa409a510 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa409a8e0 con 0x7ffaa40975e0 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa9caf640 10 monclient(hunting): authenticate will time out at 2025.447605s 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 0x7ffaa412fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 0x7ffaa412fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34432/0 (socket says 172.21.15.153:34432) 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 1 -- 172.21.15.153:0/2985144938 learned_addr learned my addr 172.21.15.153:0/2985144938 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request con 0x7ffaa40975e0 auth_method 0 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa37fe640 1 --2- 172.21.15.153:0/2985144938 >> v2:172.21.15.181:3300/0 conn(0x7ffaa409a510 0x7ffaa4096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa37fe640 10 monclient(hunting): get_auth_request con 0x7ffaa409a510 auth_method 0 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_done global_id 244433 payload 274 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 1 -- 172.21.15.153:0/2985144938 >> v2:172.21.15.181:3300/0 conn(0x7ffaa409a510 msgr2=0x7ffaa4096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 1 --2- 172.21.15.153:0/2985144938 >> v2:172.21.15.181:3300/0 conn(0x7ffaa409a510 0x7ffaa4096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.808+0000 7ffaa3fff640 1 -- 172.21.15.153:0/2985144938 >> v1:172.21.15.181:6789/0 conn(0x7ffaa40971e0 legacy=0x7ffaa412f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.809+0000 7ffaa3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.809+0000 7ffaa3fff640 1 monclient: found mon.noname-b 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.809+0000 7ffaa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.809+0000 7ffaa3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.809+0000 7ffaa3fff640 1 -- 172.21.15.153:0/2985144938 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa409d590 con 0x7ffaa40975e0 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa3fff640 1 -- 172.21.15.153:0/2985144938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 msgr2=0x7ffaa412fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa3fff640 1 --2- 172.21.15.153:0/2985144938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 0x7ffaa412fd80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009f20 tx=0x7ffa9c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a6808c13ec9da8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 1 -- 172.21.15.153:0/2985144938 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c007610 con 0x7ffaa40975e0 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: dump: 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.826 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 1 -- 172.21.15.153:0/2985144938 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c02d050 con 0x7ffaa40975e0 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa9caf640 5 monclient: authenticate success, global_id 244433 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa9caf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 1 -- 172.21.15.153:0/2985144938 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c0079a0 con 0x7ffaa40975e0 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa9caf640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa9caf640 4 set_mon_vals no callback set 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.811+0000 7ffaa27fc640 10 monclient: dump: 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 10 monclient: shutdown 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 -- 172.21.15.153:0/2985144938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 msgr2=0x7ffaa412fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/2985144938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 0x7ffaa412fd80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009f20 tx=0x7ffa9c009840 comp rx=0 tx=0).stop 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 -- 172.21.15.153:0/2985144938 shutdown_connections 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/2985144938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa40975e0 0x7ffaa412fd80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/2985144938 >> v2:172.21.15.181:3300/0 conn(0x7ffaa409a510 0x7ffaa4096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 -- 172.21.15.153:0/2985144938 >> 172.21.15.153:0/2985144938 conn(0x7ffaa40672b0 msgr2=0x7ffaa40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 -- 172.21.15.153:0/2985144938 shutdown_connections 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 1 -- 172.21.15.153:0/2985144938 wait complete. 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.812+0000 7ffaa9caf640 10 monclient: build_initial_monmap 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient: monmap: 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:27.830 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 Processor -- start 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 -- start start 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient: init 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaa40971e0 0x7ffaa4140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): picked mon.noname-b con 0x7ffaa40971e0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaa40975e0 0x7ffaa41318f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): picked mon.noname-a con 0x7ffaa40975e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 0x7ffaa4098320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): picked mon.noname-c con 0x7ffaa409a510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4099ed0 con 0x7ffaa40975e0 2024-09-15T03:50:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4120d10 con 0x7ffaa40971e0 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4120a30 con 0x7ffaa409a510 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.813+0000 7ffaa9caf640 10 monclient(hunting): authenticate will time out at 2025.452959s 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 0x7ffaa4098320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 0x7ffaa4098320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34438/0 (socket says 172.21.15.153:34438) 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 -- 172.21.15.153:0/1356178688 learned_addr learned my addr 172.21.15.153:0/1356178688 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaa40975e0 0x7ffaa41318f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request con 0x7ffaa409a510 auth_method 0 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): get_auth_request con 0x7ffaa40975e0 auth_method 0 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaa40971e0 0x7ffaa4140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): get_auth_request con 0x7ffaa40971e0 auth_method 0 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_done global_id 244436 payload 824 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 -- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaa40971e0 msgr2=0x7ffaa4140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaa40971e0 0x7ffaa4140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 -- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaa40975e0 msgr2=0x7ffaa41318f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaa40975e0 0x7ffaa41318f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 monclient: found mon.noname-c 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa3fff640 1 -- 172.21.15.153:0/1356178688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c009ae0 con 0x7ffaa409a510 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.814+0000 7ffaa2ffd640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaa40975e0 0x7ffaa41318f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa3fff640 1 -- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 msgr2=0x7ffaa4098320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa3fff640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 0x7ffaa4098320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009ab0 tx=0x7ffa9c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c032c60 con 0x7ffaa409a510 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: dump: 2024-09-15T03:50:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c032dc0 con 0x7ffaa409a510 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 5 monclient: authenticate success, global_id 244436 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 10 monclient: _renew_subs 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa4121050 con 0x7ffaa409a510 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c03b640 con 0x7ffaa409a510 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffa9b7fe640 10 monclient: dump: 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa1ffb640 4 set_mon_vals no callback set 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 10 monclient: _renew_subs 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.816+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa411f570 con 0x7ffaa409a510 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7ffa9c038050 con 0x7ffaa409a510 2024-09-15T03:50:27.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffaa9caf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffaa9caf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffaa9caf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa404bf60 con 0x7ffaa409a510 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffa9b7fe640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa84038740 0x7ffa8403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.817+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ffa9c06d290 con 0x7ffaa409a510 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.818+0000 7ffaa37fe640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa84038740 0x7ffa8403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.818+0000 7ffaa37fe640 10 monclient: get_auth_request con 0x7ffa84038740 auth_method 0 2024-09-15T03:50:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.820+0000 7ffaa37fe640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa84038740 0x7ffa8403abf0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ffa94007ba0 tx=0x7ffa94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:27.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.822+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ffa9c03b9f0 con 0x7ffaa409a510 2024-09-15T03:50:27.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.822+0000 7ffa9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:27.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.822+0000 7ffa9b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:27.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.978+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaa4120ed0 con 0x7ffa84038740 2024-09-15T03:50:27.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.979+0000 7ffa9b7fe640 1 -- 172.21.15.153:0/1356178688 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffa94010d50 con 0x7ffa84038740 2024-09-15T03:50:27.980 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T03:50:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa84038740 msgr2=0x7ffa8403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa84038740 0x7ffa8403abf0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ffa94007ba0 tx=0x7ffa94008040 comp rx=0 tx=0).stop 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 10 monclient: shutdown 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 msgr2=0x7ffaa4098320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 0x7ffaa4098320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009ab0 tx=0x7ffa9c036000 comp rx=0 tx=0).stop 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 shutdown_connections 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7ffa84038740 0x7ffa8403abf0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffaa409a510 0x7ffaa4098320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffaa40975e0 0x7ffaa41318f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 --2- 172.21.15.153:0/1356178688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaa40971e0 0x7ffaa4140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 >> 172.21.15.153:0/1356178688 conn(0x7ffaa40672b0 msgr2=0x7ffaa4069e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 shutdown_connections 2024-09-15T03:50:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:27.981+0000 7ffaa9caf640 1 -- 172.21.15.153:0/1356178688 wait complete. 2024-09-15T03:50:27.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.086+0000 7f34de84f640 10 monclient: get_monmap_and_config 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.086+0000 7f34de84f640 10 monclient: build_initial_monmap 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.086+0000 7f34de84f640 10 monclient: monmap: 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:28.086959+0000 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:28.086959+0000 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:28.088 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.086+0000 7f34de84f640 1 Processor -- start 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 -- start start 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient: init 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f34d81318f0 0x7f34d8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): picked mon.noname-a con 0x7f34d81318f0 addr v2:172.21.15.153:3300/0 2024-09-15T03:50:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 0x7f34d809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): picked mon.noname-b con 0x7f34d809ea50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7f34d8068d90 0x7f34d80691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): picked mon.noname-c con 0x7f34d8068d90 addr v2:172.21.15.181:3300/0 2024-09-15T03:50:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d8131cc0 con 0x7f34d81318f0 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d809fbf0 con 0x7f34d8068d90 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d809fe30 con 0x7f34d809ea50 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34de84f640 10 monclient(hunting): authenticate will time out at 2025.726590s 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 0x7f34d809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f34d81318f0 0x7f34d8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 0x7f34d809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34446/0 (socket says 172.21.15.153:34446) 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 1 -- 172.21.15.153:0/986908514 learned_addr learned my addr 172.21.15.153:0/986908514 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): get_auth_request con 0x7f34d809ea50 auth_method 0 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): get_auth_request con 0x7f34d81318f0 auth_method 0 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34dcdec640 1 --2- 172.21.15.153:0/986908514 >> v2:172.21.15.181:3300/0 conn(0x7f34d8068d90 0x7f34d80691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34dcdec640 10 monclient(hunting): get_auth_request con 0x7f34d8068d90 auth_method 0 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34dcdec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34dcdec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34dcdec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 10 monclient(hunting): handle_auth_done global_id 244439 payload 274 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 1 -- 172.21.15.153:0/986908514 >> v2:172.21.15.181:3300/0 conn(0x7f34d8068d90 msgr2=0x7f34d80691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.087+0000 7f34d77fe640 1 --2- 172.21.15.153:0/986908514 >> v2:172.21.15.181:3300/0 conn(0x7f34d8068d90 0x7f34d80691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 1 -- 172.21.15.153:0/986908514 >> v2:172.21.15.153:3300/0 conn(0x7f34d81318f0 msgr2=0x7f34d8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 1 --2- 172.21.15.153:0/986908514 >> v2:172.21.15.153:3300/0 conn(0x7f34d81318f0 0x7f34d8131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 1 monclient: found mon.noname-b 2024-09-15T03:50:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d77fe640 1 -- 172.21.15.153:0/986908514 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80a1210 con 0x7f34d809ea50 2024-09-15T03:50:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.088+0000 7f34d7fff640 1 --2- 172.21.15.153:0/986908514 >> v2:172.21.15.153:3300/0 conn(0x7f34d81318f0 0x7f34d8131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:28.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d77fe640 1 -- 172.21.15.153:0/986908514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 msgr2=0x7f34d809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:28.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d77fe640 1 --2- 172.21.15.153:0/986908514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 0x7f34d809ee20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f34d8064050 tx=0x7f34c00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2345b2badd2b9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:28.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 1 -- 172.21.15.153:0/986908514 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0032440 con 0x7f34d809ea50 2024-09-15T03:50:28.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:28.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: dump: 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:28.103 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:28.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 1 -- 172.21.15.153:0/986908514 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c00325a0 con 0x7f34d809ea50 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34de84f640 5 monclient: authenticate success, global_id 244439 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34de84f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34de84f640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 1 -- 172.21.15.153:0/986908514 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0032940 con 0x7f34d809ea50 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34de84f640 4 set_mon_vals no callback set 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.090+0000 7f34d6ffd640 10 monclient: dump: 2024-09-15T03:50:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 10 monclient: shutdown 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 -- 172.21.15.153:0/986908514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 msgr2=0x7f34d809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:28.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 --2- 172.21.15.153:0/986908514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 0x7f34d809ee20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f34d8064050 tx=0x7f34c00091d0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 -- 172.21.15.153:0/986908514 shutdown_connections 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 --2- 172.21.15.153:0/986908514 >> v2:172.21.15.181:3300/0 conn(0x7f34d8068d90 0x7f34d80691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 --2- 172.21.15.153:0/986908514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d809ea50 0x7f34d809ee20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 --2- 172.21.15.153:0/986908514 >> v2:172.21.15.153:3300/0 conn(0x7f34d81318f0 0x7f34d8131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 -- 172.21.15.153:0/986908514 >> 172.21.15.153:0/986908514 conn(0x7f34d8067110 msgr2=0x7f34d8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 -- 172.21.15.153:0/986908514 shutdown_connections 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.091+0000 7f34de84f640 1 -- 172.21.15.153:0/986908514 wait complete. 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient: build_initial_monmap 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient: monmap: 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:28.106 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 1 Processor -- start 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 1 -- start start 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient: init 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 0x7f34d809aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient(hunting): picked mon.noname-c con 0x7f34d8068d90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:28.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d809ea50 0x7f34d8130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 10 monclient(hunting): picked mon.noname-b con 0x7f34d809ea50 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.092+0000 7f34de84f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d81318f0 0x7f34d8140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 10 monclient(hunting): picked mon.noname-a con 0x7f34d81318f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d809bf10 con 0x7f34d81318f0 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d809e750 con 0x7f34d809ea50 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d811cf10 con 0x7f34d8068d90 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34de84f640 10 monclient(hunting): authenticate will time out at 2025.732023s 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 0x7f34d809aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d81318f0 0x7f34d8140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 0x7f34d809aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:34452/0 (socket says 172.21.15.153:34452) 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): get_auth_request con 0x7f34d81318f0 auth_method 0 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 -- 172.21.15.153:0/2404144349 learned_addr learned my addr 172.21.15.153:0/2404144349 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): get_auth_request con 0x7f34d8068d90 auth_method 0 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d809ea50 0x7f34d8130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): get_auth_request con 0x7f34d809ea50 auth_method 0 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient(hunting): handle_auth_done global_id 244442 payload 824 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 -- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d809ea50 msgr2=0x7f34d8130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d809ea50 0x7f34d8130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 -- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d81318f0 msgr2=0x7f34d8140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d81318f0 0x7f34d8140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 monclient: found mon.noname-c 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34d7fff640 1 -- 172.21.15.153:0/2404144349 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009390 con 0x7f34d8068d90 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.093+0000 7f34dcdec640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d81318f0 0x7f34d8140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34d7fff640 1 -- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 msgr2=0x7f34d809aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34d7fff640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 0x7f34d809aae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f34c800ecb0 tx=0x7f34c80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c800cdd0 con 0x7f34d8068d90 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: dump: 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:28.112 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: _finish_auth 0 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c8013470 con 0x7f34d8068d90 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 5 monclient: authenticate success, global_id 244442 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 10 monclient: _renew_subs 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c0009450 con 0x7f34d8068d90 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:28.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8013810 con 0x7f34d8068d90 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34cffff640 10 monclient: dump: 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:28.114 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34d67fc640 4 set_mon_vals no callback set 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 10 monclient: _renew_subs 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.095+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d811bbf0 con 0x7f34d8068d90 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f34c8013b80 con 0x7f34d8068d90 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34de84f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34de84f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34de84f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d811e2b0 con 0x7f34d8068d90 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34cffff640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b8038740 0x7f34b803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.096+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f34c804cb20 con 0x7f34d8068d90 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.097+0000 7f34d77fe640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b8038740 0x7f34b803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.097+0000 7f34d77fe640 10 monclient: get_auth_request con 0x7f34b8038740 auth_method 0 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.099+0000 7f34d77fe640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b8038740 0x7f34b803abf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f34d8076600 tx=0x7f34c0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:28.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.101+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f34c80123d0 con 0x7f34d8068d90 2024-09-15T03:50:28.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.101+0000 7f34cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:28.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.101+0000 7f34cffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:28.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.259+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34d811df70 con 0x7f34b8038740 2024-09-15T03:50:28.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.262+0000 7f34cffff640 1 -- 172.21.15.153:0/2404144349 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98025 (secure 0 0 0) 0x7f34d811df70 con 0x7f34b8038740 2024-09-15T03:50:28.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:28.263 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:50:28.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b8038740 msgr2=0x7f34b803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:28.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b8038740 0x7f34b803abf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f34d8076600 tx=0x7f34c0032000 comp rx=0 tx=0).stop 2024-09-15T03:50:28.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 10 monclient: shutdown 2024-09-15T03:50:28.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:28.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 msgr2=0x7f34d809aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 0x7f34d809aae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f34c800ecb0 tx=0x7f34c80090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 shutdown_connections 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f34b8038740 0x7f34b803abf0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34d81318f0 0x7f34d8140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d809ea50 0x7f34d8130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 --2- 172.21.15.153:0/2404144349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8068d90 0x7f34d809aae0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 >> 172.21.15.153:0/2404144349 conn(0x7f34d8067110 msgr2=0x7f34d80a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 shutdown_connections 2024-09-15T03:50:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:28.264+0000 7f34de84f640 1 -- 172.21.15.153:0/2404144349 wait complete. 2024-09-15T03:50:28.275 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":911,"stamp":"2024-09-15T03:50:28.116028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1809672,"kb_used_data":8896,"kb_used_omap":9,"kb_used_meta":1800694,"kb_avail":564421368,"statfs":{"total":579820584960,"available":577967480832,"internally_reserved":0,"allocated":9109504,"data_stored":7285462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843910672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002884"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":171,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253295+0000","last_change":"2024-09-15T03:35:36.177407+0000","last_active":"2024-09-15T03:38:16.253295+0000","last_peered":"2024-09-15T03:38:16.253295+0000","last_clean":"2024-09-15T03:38:16.253295+0000","last_became_active":"2024-09-15T03:35:36.177073+0000","last_became_peered":"2024-09-15T03:35:36.177073+0000","last_unstale":"2024-09-15T03:38:16.253295+0000","last_undegraded":"2024-09-15T03:38:16.253295+0000","last_fullsized":"2024-09-15T03:38:16.253295+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:01:53.230196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412884+0000","last_change":"2024-09-15T03:35:36.171117+0000","last_active":"2024-09-15T03:38:15.412884+0000","last_peered":"2024-09-15T03:38:15.412884+0000","last_clean":"2024-09-15T03:38:15.412884+0000","last_became_active":"2024-09-15T03:35:36.170838+0000","last_became_peered":"2024-09-15T03:35:36.170838+0000","last_unstale":"2024-09-15T03:38:15.412884+0000","last_undegraded":"2024-09-15T03:38:15.412884+0000","last_fullsized":"2024-09-15T03:38:15.412884+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:12:13.052343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":146,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866090+0000","last_change":"2024-09-15T03:35:36.180940+0000","last_active":"2024-09-15T03:38:16.866090+0000","last_peered":"2024-09-15T03:38:16.866090+0000","last_clean":"2024-09-15T03:38:16.866090+0000","last_became_active":"2024-09-15T03:35:36.180590+0000","last_became_peered":"2024-09-15T03:35:36.180590+0000","last_unstale":"2024-09-15T03:38:16.866090+0000","last_undegraded":"2024-09-15T03:38:16.866090+0000","last_fullsized":"2024-09-15T03:38:16.866090+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:39:06.005311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":90,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412806+0000","last_change":"2024-09-15T03:35:36.173953+0000","last_active":"2024-09-15T03:38:15.412806+0000","last_peered":"2024-09-15T03:38:15.412806+0000","last_clean":"2024-09-15T03:38:15.412806+0000","last_became_active":"2024-09-15T03:35:36.173628+0000","last_became_peered":"2024-09-15T03:35:36.173628+0000","last_unstale":"2024-09-15T03:38:15.412806+0000","last_undegraded":"2024-09-15T03:38:15.412806+0000","last_fullsized":"2024-09-15T03:38:15.412806+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:56:12.188821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":131,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412847+0000","last_change":"2024-09-15T03:35:42.048298+0000","last_active":"2024-09-15T03:38:15.412847+0000","last_peered":"2024-09-15T03:38:15.412847+0000","last_clean":"2024-09-15T03:38:15.412847+0000","last_became_active":"2024-09-15T03:35:42.047949+0000","last_became_peered":"2024-09-15T03:35:42.047949+0000","last_unstale":"2024-09-15T03:38:15.412847+0000","last_undegraded":"2024-09-15T03:38:15.412847+0000","last_fullsized":"2024-09-15T03:38:15.412847+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:49:06.838016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":147,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253319+0000","last_change":"2024-09-15T03:35:36.179570+0000","last_active":"2024-09-15T03:38:16.253319+0000","last_peered":"2024-09-15T03:38:16.253319+0000","last_clean":"2024-09-15T03:38:16.253319+0000","last_became_active":"2024-09-15T03:35:36.179302+0000","last_became_peered":"2024-09-15T03:35:36.179302+0000","last_unstale":"2024-09-15T03:38:16.253319+0000","last_undegraded":"2024-09-15T03:38:16.253319+0000","last_fullsized":"2024-09-15T03:38:16.253319+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:26:04.635839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411506+0000","last_change":"2024-09-15T03:35:36.179910+0000","last_active":"2024-09-15T03:38:15.411506+0000","last_peered":"2024-09-15T03:38:15.411506+0000","last_clean":"2024-09-15T03:38:15.411506+0000","last_became_active":"2024-09-15T03:35:36.179540+0000","last_became_peered":"2024-09-15T03:35:36.179540+0000","last_unstale":"2024-09-15T03:38:15.411506+0000","last_undegraded":"2024-09-15T03:38:15.411506+0000","last_fullsized":"2024-09-15T03:38:15.411506+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:29.604427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510296+0000","last_change":"2024-09-15T03:35:36.176521+0000","last_active":"2024-09-15T03:38:16.510296+0000","last_peered":"2024-09-15T03:38:16.510296+0000","last_clean":"2024-09-15T03:38:16.510296+0000","last_became_active":"2024-09-15T03:35:36.176169+0000","last_became_peered":"2024-09-15T03:35:36.176169+0000","last_unstale":"2024-09-15T03:38:16.510296+0000","last_undegraded":"2024-09-15T03:38:16.510296+0000","last_fullsized":"2024-09-15T03:38:16.510296+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:17:56.952574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":114,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412674+0000","last_change":"2024-09-15T03:35:36.176184+0000","last_active":"2024-09-15T03:38:15.412674+0000","last_peered":"2024-09-15T03:38:15.412674+0000","last_clean":"2024-09-15T03:38:15.412674+0000","last_became_active":"2024-09-15T03:35:36.175786+0000","last_became_peered":"2024-09-15T03:35:36.175786+0000","last_unstale":"2024-09-15T03:38:15.412674+0000","last_undegraded":"2024-09-15T03:38:15.412674+0000","last_fullsized":"2024-09-15T03:38:15.412674+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:57.476041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":109,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411493+0000","last_change":"2024-09-15T03:35:36.174198+0000","last_active":"2024-09-15T03:38:15.411493+0000","last_peered":"2024-09-15T03:38:15.411493+0000","last_clean":"2024-09-15T03:38:15.411493+0000","last_became_active":"2024-09-15T03:35:36.173941+0000","last_became_peered":"2024-09-15T03:35:36.173941+0000","last_unstale":"2024-09-15T03:38:15.411493+0000","last_undegraded":"2024-09-15T03:38:15.411493+0000","last_fullsized":"2024-09-15T03:38:15.411493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.690762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":122,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866041+0000","last_change":"2024-09-15T03:35:36.173396+0000","last_active":"2024-09-15T03:38:16.866041+0000","last_peered":"2024-09-15T03:38:16.866041+0000","last_clean":"2024-09-15T03:38:16.866041+0000","last_became_active":"2024-09-15T03:35:36.173027+0000","last_became_peered":"2024-09-15T03:35:36.173027+0000","last_unstale":"2024-09-15T03:38:16.866041+0000","last_undegraded":"2024-09-15T03:38:16.866041+0000","last_fullsized":"2024-09-15T03:38:16.866041+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:28:33.565728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510475+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510475+0000","last_peered":"2024-09-15T03:38:16.510475+0000","last_clean":"2024-09-15T03:38:16.510475+0000","last_became_active":"2024-09-15T03:35:36.176264+0000","last_became_peered":"2024-09-15T03:35:36.176264+0000","last_unstale":"2024-09-15T03:38:16.510475+0000","last_undegraded":"2024-09-15T03:38:16.510475+0000","last_fullsized":"2024-09-15T03:38:16.510475+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510499+0000","last_change":"2024-09-15T03:35:36.179783+0000","last_active":"2024-09-15T03:38:16.510499+0000","last_peered":"2024-09-15T03:38:16.510499+0000","last_clean":"2024-09-15T03:38:16.510499+0000","last_became_active":"2024-09-15T03:35:36.179473+0000","last_became_peered":"2024-09-15T03:35:36.179473+0000","last_unstale":"2024-09-15T03:38:16.510499+0000","last_undegraded":"2024-09-15T03:38:16.510499+0000","last_fullsized":"2024-09-15T03:38:16.510499+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:11.139885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":178,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.253392+0000","last_change":"2024-09-15T03:35:36.179775+0000","last_active":"2024-09-15T03:38:16.253392+0000","last_peered":"2024-09-15T03:38:16.253392+0000","last_clean":"2024-09-15T03:38:16.253392+0000","last_became_active":"2024-09-15T03:35:36.179476+0000","last_became_peered":"2024-09-15T03:35:36.179476+0000","last_unstale":"2024-09-15T03:38:16.253392+0000","last_undegraded":"2024-09-15T03:38:16.253392+0000","last_fullsized":"2024-09-15T03:38:16.253392+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:33.988609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866135+0000","last_change":"2024-09-15T03:35:41.930950+0000","last_active":"2024-09-15T03:38:16.866135+0000","last_peered":"2024-09-15T03:38:16.866135+0000","last_clean":"2024-09-15T03:38:16.866135+0000","last_became_active":"2024-09-15T03:35:41.930552+0000","last_became_peered":"2024-09-15T03:35:41.930552+0000","last_unstale":"2024-09-15T03:38:16.866135+0000","last_undegraded":"2024-09-15T03:38:16.866135+0000","last_fullsized":"2024-09-15T03:38:16.866135+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:33.614703+0000","last_clean_scrub_stamp":"2024-09-15T03:33:33.614703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.602540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510466+0000","last_change":"2024-09-15T03:35:36.176671+0000","last_active":"2024-09-15T03:38:16.510466+0000","last_peered":"2024-09-15T03:38:16.510466+0000","last_clean":"2024-09-15T03:38:16.510466+0000","last_became_active":"2024-09-15T03:35:36.176131+0000","last_became_peered":"2024-09-15T03:35:36.176131+0000","last_unstale":"2024-09-15T03:38:16.510466+0000","last_undegraded":"2024-09-15T03:38:16.510466+0000","last_fullsized":"2024-09-15T03:38:16.510466+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:33:32.611587+0000","last_clean_scrub_stamp":"2024-09-15T03:33:32.611587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.630912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":332,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411395+0000","last_change":"2024-09-15T03:35:36.180042+0000","last_active":"2024-09-15T03:38:15.411395+0000","last_peered":"2024-09-15T03:38:15.411395+0000","last_clean":"2024-09-15T03:38:15.411395+0000","last_became_active":"2024-09-15T03:35:36.179501+0000","last_became_peered":"2024-09-15T03:35:36.179501+0000","last_unstale":"2024-09-15T03:38:15.411395+0000","last_undegraded":"2024-09-15T03:38:15.411395+0000","last_fullsized":"2024-09-15T03:38:15.411395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:33.881688+0000","last_clean_scrub_stamp":"2024-09-15T03:28:33.881688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:32:25.282577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":254,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412719+0000","last_change":"2024-09-15T03:35:36.182382+0000","last_active":"2024-09-15T03:38:15.412719+0000","last_peered":"2024-09-15T03:38:15.412719+0000","last_clean":"2024-09-15T03:38:15.412719+0000","last_became_active":"2024-09-15T03:35:36.175947+0000","last_became_peered":"2024-09-15T03:35:36.175947+0000","last_unstale":"2024-09-15T03:38:15.412719+0000","last_undegraded":"2024-09-15T03:38:15.412719+0000","last_fullsized":"2024-09-15T03:38:15.412719+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:04:51.063277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411377+0000","last_change":"2024-09-15T03:35:36.177331+0000","last_active":"2024-09-15T03:38:15.411377+0000","last_peered":"2024-09-15T03:38:15.411377+0000","last_clean":"2024-09-15T03:38:15.411377+0000","last_became_active":"2024-09-15T03:35:36.177043+0000","last_became_peered":"2024-09-15T03:35:36.177043+0000","last_unstale":"2024-09-15T03:38:15.411377+0000","last_undegraded":"2024-09-15T03:38:15.411377+0000","last_fullsized":"2024-09-15T03:38:15.411377+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:52:08.123750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.866082+0000","last_change":"2024-09-15T03:35:36.177279+0000","last_active":"2024-09-15T03:38:16.866082+0000","last_peered":"2024-09-15T03:38:16.866082+0000","last_clean":"2024-09-15T03:38:16.866082+0000","last_became_active":"2024-09-15T03:35:36.176997+0000","last_became_peered":"2024-09-15T03:35:36.176997+0000","last_unstale":"2024-09-15T03:38:16.866082+0000","last_undegraded":"2024-09-15T03:38:16.866082+0000","last_fullsized":"2024-09-15T03:38:16.866082+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:05:52.287110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":251,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.412676+0000","last_change":"2024-09-15T03:35:36.189329+0000","last_active":"2024-09-15T03:38:15.412676+0000","last_peered":"2024-09-15T03:38:15.412676+0000","last_clean":"2024-09-15T03:38:15.412676+0000","last_became_active":"2024-09-15T03:35:36.176519+0000","last_became_peered":"2024-09-15T03:35:36.176519+0000","last_unstale":"2024-09-15T03:38:15.412676+0000","last_undegraded":"2024-09-15T03:38:15.412676+0000","last_fullsized":"2024-09-15T03:38:15.412676+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:40:12.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510504+0000","last_change":"2024-09-15T03:35:42.047866+0000","last_active":"2024-09-15T03:38:16.510504+0000","last_peered":"2024-09-15T03:38:16.510504+0000","last_clean":"2024-09-15T03:38:16.510504+0000","last_became_active":"2024-09-15T03:35:42.047571+0000","last_became_peered":"2024-09-15T03:35:42.047571+0000","last_unstale":"2024-09-15T03:38:16.510504+0000","last_undegraded":"2024-09-15T03:38:16.510504+0000","last_fullsized":"2024-09-15T03:38:16.510504+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:05:44.835315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411503+0000","last_change":"2024-09-15T03:35:36.180602+0000","last_active":"2024-09-15T03:38:15.411503+0000","last_peered":"2024-09-15T03:38:15.411503+0000","last_clean":"2024-09-15T03:38:15.411503+0000","last_became_active":"2024-09-15T03:35:36.180366+0000","last_became_peered":"2024-09-15T03:35:36.180366+0000","last_unstale":"2024-09-15T03:38:15.411503+0000","last_undegraded":"2024-09-15T03:38:15.411503+0000","last_fullsized":"2024-09-15T03:38:15.411503+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:51:20.797295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:16.510310+0000","last_change":"2024-09-15T03:35:36.176603+0000","last_active":"2024-09-15T03:38:16.510310+0000","last_peered":"2024-09-15T03:38:16.510310+0000","last_clean":"2024-09-15T03:38:16.510310+0000","last_became_active":"2024-09-15T03:35:36.176265+0000","last_became_peered":"2024-09-15T03:35:36.176265+0000","last_unstale":"2024-09-15T03:38:16.510310+0000","last_undegraded":"2024-09-15T03:38:16.510310+0000","last_fullsized":"2024-09-15T03:38:16.510310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:07:54.095941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-15T03:38:15.411431+0000","last_change":"2024-09-15T03:35:36.176300+0000","last_active":"2024-09-15T03:38:15.411431+0000","last_peered":"2024-09-15T03:38:15.411431+0000","last_clean":"2024-09-15T03:38:15.411431+0000","last_became_active":"2024-09-15T03:35:36.175986+0000","last_became_peered":"2024-09-15T03:35:36.175986+0000","last_unstale":"2024-09-15T03:38:15.411431+0000","last_undegraded":"2024-09-15T03:38:15.411431+0000","last_fullsized":"2024-09-15T03:38:15.411431+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:58:16.529865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63700000000000001,"5min":0.873,"15min":0.873},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59599999999999997,"5min":0.68400000000000005,"15min":0.751},"last":0.63}]},{"osd":1,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.505},"min":{"1min":0.379,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73799999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.33800000000000002},"max":{"1min":0.60899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71199999999999997},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.74199999999999999},"last":0.69599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.248,"5min":0.247,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.62},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.379},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":0.45200000000000001,"5min":0.63100000000000001,"15min":0.64400000000000002},"last":0.41199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.39000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.64400000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.42199999999999999},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.77000000000000002},"last":0.435}]}]},{"osd":4,"up_from":11,"seq":47244640521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.69099999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.499},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.64100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":0.59899999999999998,"5min":0.67700000000000005,"15min":0.70499999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.32800000000000001},"max":{"1min":0.629,"5min":0.66000000000000003,"15min":0.71199999999999997},"last":0.41799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34599999999999997},"max":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.72199999999999998},"last":0.497},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35299999999999998,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.72299999999999998},"last":0.39000000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.627},"last":0.31},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22700000000000001},"max":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.36799999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.60099999999999998,"5min":0.63,"15min":0.63},"last":0.44400000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640521,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.50800000000000001,"15min":0.49099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61499999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.51800000000000002},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.76000000000000001,"15min":0.79700000000000004},"last":0.46999999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61899999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53100000000000003,"15min":0.503},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63900000000000001,"5min":0.755,"15min":0.75700000000000001},"last":0.5}]},{"osd":2,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":1.597},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.73299999999999998,"15min":0.84199999999999997},"last":0.56599999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.19600000000000001},"max":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.65400000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"min":{"1min":0.28999999999999998,"5min":0.26800000000000002,"15min":0.222},"max":{"1min":0.53200000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.311}]},{"osd":5,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.379,"15min":0.38200000000000001},"min":{"1min":0.248,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58699999999999997,"5min":0.63200000000000001,"15min":0.65500000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.441,"15min":0.42599999999999999},"min":{"1min":0.29999999999999999,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.34499999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073176,"statfs":{"total":96636764160,"available":96330932224,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31,"15min":0.308},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.58799999999999997},"last":0.217},{"interface":"front","average":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.72699999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":0.48699999999999999,"5min":0.51200000000000001,"15min":0.59599999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.22},"max":{"1min":0.50600000000000001,"5min":0.57499999999999996,"15min":0.68500000000000005},"last":0.28399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"last":0.495},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.35299999999999998,"15min":0.26700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.47799999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.314},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":5.6269999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.42799999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.29199999999999998},"max":{"1min":0.65800000000000003,"5min":0.77500000000000002,"15min":0.84999999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72799999999999998,"5min":0.77300000000000002,"15min":0.82599999999999996},"last":0.45300000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640520,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302828,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069012,"statfs":{"total":96636764160,"available":96326668288,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32500000000000001,"15min":0.32700000000000001},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":0.41999999999999998,"5min":0.52800000000000002,"15min":0.874},"last":0.246},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.48999999999999999,"15min":0.55900000000000005},"last":0.28100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.215},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.77800000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.50900000000000001,"15min":0.747},"last":0.30599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.73199999999999998,"15min":0.89800000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57299999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.88300000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.76600000000000001,"15min":0.81799999999999995},"last":0.56399999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.93799999999999994},"last":0.499},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.432,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.91300000000000003},"last":0.58199999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640521,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303344,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068496,"statfs":{"total":96636764160,"available":96326139904,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32200000000000001,"15min":0.32000000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":0.55000000000000004,"15min":0.625},"last":0.223},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33300000000000002,"15min":0.33400000000000002},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.246}]},{"osd":2,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":0.52400000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.219},"max":{"1min":0.47499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.26400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.443},{"interface":"front","average":{"1min":0.499,"5min":0.50900000000000001,"15min":0.51000000000000001},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":0.85699999999999998},"last":0.46400000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.29699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.34899999999999998,"15min":0.34200000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.81200000000000006},"last":0.5}]},{"osd":5,"last update":"Sun Sep 15 03:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":0.69999999999999996,"5min":0.76000000000000001,"15min":0.78000000000000003},"last":0.433},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.84099999999999997,"15min":0.84799999999999998},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:50:28.280 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.280 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.280 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.280 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.281 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-15T03:33:33.614703+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-15T03:33:32.611587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=33, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T03:28:33.881688+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.282 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.283 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.283 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.283 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.283 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:28.283 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:50:48.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:50:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.389+0000 7fca5e39d640 10 monclient: get_monmap_and_config 2024-09-15T03:50:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.389+0000 7fca5e39d640 10 monclient: build_initial_monmap 2024-09-15T03:50:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.389+0000 7fca5e39d640 10 monclient: monmap: 2024-09-15T03:50:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:50:48.389797+0000 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:50:48.389797+0000 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:50:48.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:50:48.392 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:50:48.392 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:50:48.392 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:50:48.392 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:48.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.389+0000 7fca5e39d640 1 Processor -- start 2024-09-15T03:50:48.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 1 -- start start 2024-09-15T03:50:48.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient: init 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 0x7fca58131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient(hunting): picked mon.noname-b con 0x7fca58131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca5809e910 addr v1:172.21.15.181:6789/0 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca58136bb0 addr v1:172.21.15.153:6789/0 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5809faf0 con 0x7fca58136bb0 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5809fd30 con 0x7fca5809e910 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca580a1110 con 0x7fca58131ab0 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5e39d640 10 monclient(hunting): authenticate will time out at 2046.029630s 2024-09-15T03:50:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 0x7fca58131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5c93a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fca58136bb0 0x7fca58138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:54116/0 (socket says 172.21.15.153:54116) 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 0x7fca58131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33360/0 (socket says 172.21.15.153:33360) 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca5c93a640 1 -- 172.21.15.153:0/1649008859 learned_addr learned my addr 172.21.15.153:0/1649008859 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): get_auth_request con 0x7fca58131ab0 auth_method 0 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.390+0000 7fca57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 10 monclient(hunting): handle_auth_done global_id 244445 payload 274 2024-09-15T03:50:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 1 -- 172.21.15.153:0/1649008859 >> v1:172.21.15.181:6789/0 conn(0x7fca5809e910 legacy=0x7fca5809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 1 -- 172.21.15.153:0/1649008859 >> v1:172.21.15.153:6789/0 conn(0x7fca58136bb0 legacy=0x7fca58138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 1 monclient: found mon.noname-b 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.391+0000 7fca57fff640 1 -- 172.21.15.153:0/1649008859 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca580a13f0 con 0x7fca58131ab0 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.393+0000 7fca57fff640 1 -- 172.21.15.153:0/1649008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 msgr2=0x7fca58131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.393+0000 7fca57fff640 1 --2- 172.21.15.153:0/1649008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 0x7fca58131e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca48009880 tx=0x7fca480090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93652a97604007 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 1 -- 172.21.15.153:0/1649008859 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48032440 con 0x7fca58131ab0 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:48.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: dump: 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: _finish_auth 0 2024-09-15T03:50:48.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 5 monclient: authenticate success, global_id 244445 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 1 -- 172.21.15.153:0/1649008859 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca480325a0 con 0x7fca58131ab0 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 10 monclient: get_monmap_and_config success 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 1 -- 172.21.15.153:0/1649008859 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48032940 con 0x7fca58131ab0 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 4 set_mon_vals no callback set 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca577fe640 10 monclient: dump: 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:48.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 10 monclient: shutdown 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 -- 172.21.15.153:0/1649008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 msgr2=0x7fca58131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 --2- 172.21.15.153:0/1649008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 0x7fca58131e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca48009880 tx=0x7fca480090f0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 -- 172.21.15.153:0/1649008859 shutdown_connections 2024-09-15T03:50:48.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 --2- 172.21.15.153:0/1649008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca58131ab0 0x7fca58131e80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 -- 172.21.15.153:0/1649008859 >> 172.21.15.153:0/1649008859 conn(0x7fca58067d30 msgr2=0x7fca580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 -- 172.21.15.153:0/1649008859 shutdown_connections 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.394+0000 7fca5e39d640 1 -- 172.21.15.153:0/1649008859 wait complete. 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 10 monclient: build_initial_monmap 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 10 monclient: monmap: 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:50:48.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 1 Processor -- start 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 1 -- start start 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 10 monclient: init 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 10 monclient: _reopen_session rank -1 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.395+0000 7fca5e39d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 0x7fca58096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): picked mon.noname-c con 0x7fca5809e910 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca58131ab0 0x7fca58096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): picked mon.noname-a con 0x7fca58131ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca58136bb0 0x7fca580974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): picked mon.noname-b con 0x7fca58136bb0 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5813a4a0 con 0x7fca58131ab0 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5809a110 con 0x7fca58136bb0 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca58099870 con 0x7fca5809e910 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): _renew_subs 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5e39d640 10 monclient(hunting): authenticate will time out at 2046.035053s 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 0x7fca58096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 0x7fca58096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:33372/0 (socket says 172.21.15.153:33372) 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca58131ab0 0x7fca58096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 -- 172.21.15.153:0/3593872269 learned_addr learned my addr 172.21.15.153:0/3593872269 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): get_auth_request con 0x7fca5809e910 auth_method 0 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): get_auth_request con 0x7fca58131ab0 auth_method 0 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:50:48.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5c93a640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca58136bb0 0x7fca580974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5c93a640 10 monclient(hunting): get_auth_request con 0x7fca58136bb0 auth_method 0 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5c93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5c93a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5c93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient(hunting): handle_auth_done global_id 244448 payload 824 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 -- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca58136bb0 msgr2=0x7fca580974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca58136bb0 0x7fca580974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 -- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca58131ab0 msgr2=0x7fca58096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca58131ab0 0x7fca58096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient: _finish_hunting 0 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 monclient: found mon.noname-c 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:50:48.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca57fff640 1 -- 172.21.15.153:0/3593872269 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48009390 con 0x7fca5809e910 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca5c93a640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca58136bb0 0x7fca580974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.396+0000 7fca4ffff640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca58131ab0 0x7fca58096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca57fff640 1 -- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 msgr2=0x7fca58096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca57fff640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 0x7fca58096a60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca48009880 tx=0x7fca48032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48031910 con 0x7fca5809e910 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: dump: 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: _finish_auth 0 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca48031a90 con 0x7fca5809e910 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca5e39d640 5 monclient: authenticate success, global_id 244448 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48031e50 con 0x7fca5809e910 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca5e39d640 10 monclient: _renew_subs 2024-09-15T03:50:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca5e39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca58099b50 con 0x7fca5809e910 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca56ffd640 4 set_mon_vals no callback set 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.398+0000 7fca54ff9640 10 monclient: dump: 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:50:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 10 monclient: _renew_subs 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca58097d40 con 0x7fca5809e910 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fca48043030 con 0x7fca5809e910 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.399+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5804bf60 con 0x7fca5809e910 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.400+0000 7fca54ff9640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca38038740 0x7fca3803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:50:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.400+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fca48041070 con 0x7fca5809e910 2024-09-15T03:50:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.400+0000 7fca4ffff640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca38038740 0x7fca3803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:50:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.400+0000 7fca4ffff640 10 monclient: get_auth_request con 0x7fca38038740 auth_method 0 2024-09-15T03:50:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.402+0000 7fca4ffff640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca38038740 0x7fca3803abf0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fca5813a370 tx=0x7fca40006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:50:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.404+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fca4804e950 con 0x7fca5809e910 2024-09-15T03:50:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.404+0000 7fca54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:50:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.404+0000 7fca54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:50:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.555+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca5804bf60 con 0x7fca38038740 2024-09-15T03:50:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.559+0000 7fca54ff9640 1 -- 172.21.15.153:0/3593872269 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98413 (secure 0 0 0) 0x7fca5804bf60 con 0x7fca38038740 2024-09-15T03:50:48.560 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:50:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:50:48.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca38038740 msgr2=0x7fca3803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca38038740 0x7fca3803abf0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fca5813a370 tx=0x7fca40006f50 comp rx=0 tx=0).stop 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 10 monclient: shutdown 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 msgr2=0x7fca58096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 0x7fca58096a60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca48009880 tx=0x7fca48032000 comp rx=0 tx=0).stop 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 shutdown_connections 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fca38038740 0x7fca3803abf0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca58136bb0 0x7fca580974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca58131ab0 0x7fca58096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 --2- 172.21.15.153:0/3593872269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca5809e910 0x7fca58096a60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:50:48.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 >> 172.21.15.153:0/3593872269 conn(0x7fca58067d30 msgr2=0x7fca580a47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:50:48.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 shutdown_connections 2024-09-15T03:50:48.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:50:48.561+0000 7fca5e39d640 1 -- 172.21.15.153:0/3593872269 wait complete. 2024-09-15T03:50:48.572 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":921,"stamp":"2024-09-15T03:50:48.121017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1809648,"kb_used_data":8896,"kb_used_omap":9,"kb_used_meta":1800694,"kb_avail":564421392,"statfs":{"total":579820584960,"available":577967505408,"internally_reserved":0,"allocated":9109504,"data_stored":7285462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843910672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002932"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":182,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:33.030467+0000","last_change":"2024-09-15T03:50:33.030419+0000","last_active":"2024-09-15T03:50:33.030467+0000","last_peered":"2024-09-15T03:50:33.030467+0000","last_clean":"2024-09-15T03:50:33.030467+0000","last_became_active":"2024-09-15T03:35:36.177073+0000","last_became_peered":"2024-09-15T03:35:36.177073+0000","last_unstale":"2024-09-15T03:50:33.030467+0000","last_undegraded":"2024-09-15T03:50:33.030467+0000","last_fullsized":"2024-09-15T03:50:33.030467+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:33.030284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:33.030284+0000","last_clean_scrub_stamp":"2024-09-15T03:50:33.030284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:25:32.127112+0000","scrub_duration":0.0018146010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":101,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:34.491280+0000","last_change":"2024-09-15T03:50:34.491250+0000","last_active":"2024-09-15T03:50:34.491280+0000","last_peered":"2024-09-15T03:50:34.491280+0000","last_clean":"2024-09-15T03:50:34.491280+0000","last_became_active":"2024-09-15T03:35:36.170838+0000","last_became_peered":"2024-09-15T03:35:36.170838+0000","last_unstale":"2024-09-15T03:50:34.491280+0000","last_undegraded":"2024-09-15T03:50:34.491280+0000","last_fullsized":"2024-09-15T03:50:34.491280+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:34.491161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:34.491161+0000","last_clean_scrub_stamp":"2024-09-15T03:50:34.491161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:48:38.374243+0000","scrub_duration":0.0015124820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":157,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:27.030711+0000","last_change":"2024-09-15T03:50:27.030681+0000","last_active":"2024-09-15T03:50:27.030711+0000","last_peered":"2024-09-15T03:50:27.030711+0000","last_clean":"2024-09-15T03:50:27.030711+0000","last_became_active":"2024-09-15T03:35:36.180590+0000","last_became_peered":"2024-09-15T03:35:36.180590+0000","last_unstale":"2024-09-15T03:50:27.030711+0000","last_undegraded":"2024-09-15T03:50:27.030711+0000","last_fullsized":"2024-09-15T03:50:27.030711+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:27.030599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:27.030599+0000","last_clean_scrub_stamp":"2024-09-15T03:50:27.030599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:06:36.467272+0000","scrub_duration":0.0014349160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":101,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.537028+0000","last_change":"2024-09-15T03:50:31.536997+0000","last_active":"2024-09-15T03:50:31.537028+0000","last_peered":"2024-09-15T03:50:31.537028+0000","last_clean":"2024-09-15T03:50:31.537028+0000","last_became_active":"2024-09-15T03:35:36.173628+0000","last_became_peered":"2024-09-15T03:35:36.173628+0000","last_unstale":"2024-09-15T03:50:31.537028+0000","last_undegraded":"2024-09-15T03:50:31.537028+0000","last_fullsized":"2024-09-15T03:50:31.537028+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.536914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.536914+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.536914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:04:41.679502+0000","scrub_duration":0.00189146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":142,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:38.465233+0000","last_change":"2024-09-15T03:50:38.465181+0000","last_active":"2024-09-15T03:50:38.465233+0000","last_peered":"2024-09-15T03:50:38.465233+0000","last_clean":"2024-09-15T03:50:38.465233+0000","last_became_active":"2024-09-15T03:35:42.047949+0000","last_became_peered":"2024-09-15T03:35:42.047949+0000","last_unstale":"2024-09-15T03:50:38.465233+0000","last_undegraded":"2024-09-15T03:50:38.465233+0000","last_fullsized":"2024-09-15T03:50:38.465233+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:38.465085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:38.465085+0000","last_clean_scrub_stamp":"2024-09-15T03:50:38.465085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:21:41.366790+0000","scrub_duration":0.0017380150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":158,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:30.974493+0000","last_change":"2024-09-15T03:50:30.974458+0000","last_active":"2024-09-15T03:50:30.974493+0000","last_peered":"2024-09-15T03:50:30.974493+0000","last_clean":"2024-09-15T03:50:30.974493+0000","last_became_active":"2024-09-15T03:35:36.179302+0000","last_became_peered":"2024-09-15T03:35:36.179302+0000","last_unstale":"2024-09-15T03:50:30.974493+0000","last_undegraded":"2024-09-15T03:50:30.974493+0000","last_fullsized":"2024-09-15T03:50:30.974493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:30.974370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:30.974370+0000","last_clean_scrub_stamp":"2024-09-15T03:50:30.974370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:17:55.985403+0000","scrub_duration":0.001722765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":96,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:33.257345+0000","last_change":"2024-09-15T03:50:33.257279+0000","last_active":"2024-09-15T03:50:33.257345+0000","last_peered":"2024-09-15T03:50:33.257345+0000","last_clean":"2024-09-15T03:50:33.257345+0000","last_became_active":"2024-09-15T03:35:36.179540+0000","last_became_peered":"2024-09-15T03:35:36.179540+0000","last_unstale":"2024-09-15T03:50:33.257345+0000","last_undegraded":"2024-09-15T03:50:33.257345+0000","last_fullsized":"2024-09-15T03:50:33.257345+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:33.257156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:33.257156+0000","last_clean_scrub_stamp":"2024-09-15T03:50:33.257156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:41:55.192997+0000","scrub_duration":0.002146897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":92,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:30.052690+0000","last_change":"2024-09-15T03:50:30.052663+0000","last_active":"2024-09-15T03:50:30.052690+0000","last_peered":"2024-09-15T03:50:30.052690+0000","last_clean":"2024-09-15T03:50:30.052690+0000","last_became_active":"2024-09-15T03:35:36.176169+0000","last_became_peered":"2024-09-15T03:35:36.176169+0000","last_unstale":"2024-09-15T03:50:30.052690+0000","last_undegraded":"2024-09-15T03:50:30.052690+0000","last_fullsized":"2024-09-15T03:50:30.052690+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:30.052572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:30.052572+0000","last_clean_scrub_stamp":"2024-09-15T03:50:30.052572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:27:20.777490+0000","scrub_duration":0.001622662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":125,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:33.523390+0000","last_change":"2024-09-15T03:50:33.523332+0000","last_active":"2024-09-15T03:50:33.523390+0000","last_peered":"2024-09-15T03:50:33.523390+0000","last_clean":"2024-09-15T03:50:33.523390+0000","last_became_active":"2024-09-15T03:35:36.175786+0000","last_became_peered":"2024-09-15T03:35:36.175786+0000","last_unstale":"2024-09-15T03:50:33.523390+0000","last_undegraded":"2024-09-15T03:50:33.523390+0000","last_fullsized":"2024-09-15T03:50:33.523390+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:33.523234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:33.523234+0000","last_clean_scrub_stamp":"2024-09-15T03:50:33.523234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:12:24.120105+0000","scrub_duration":0.0015875430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":120,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:34.286441+0000","last_change":"2024-09-15T03:50:34.286411+0000","last_active":"2024-09-15T03:50:34.286441+0000","last_peered":"2024-09-15T03:50:34.286441+0000","last_clean":"2024-09-15T03:50:34.286441+0000","last_became_active":"2024-09-15T03:35:36.173941+0000","last_became_peered":"2024-09-15T03:35:36.173941+0000","last_unstale":"2024-09-15T03:50:34.286441+0000","last_undegraded":"2024-09-15T03:50:34.286441+0000","last_fullsized":"2024-09-15T03:50:34.286441+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:34.286316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:34.286316+0000","last_clean_scrub_stamp":"2024-09-15T03:50:34.286316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:34:43.710252+0000","scrub_duration":0.0016205499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":133,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.355177+0000","last_change":"2024-09-15T03:50:31.355124+0000","last_active":"2024-09-15T03:50:31.355177+0000","last_peered":"2024-09-15T03:50:31.355177+0000","last_clean":"2024-09-15T03:50:31.355177+0000","last_became_active":"2024-09-15T03:35:36.173027+0000","last_became_peered":"2024-09-15T03:35:36.173027+0000","last_unstale":"2024-09-15T03:50:31.355177+0000","last_undegraded":"2024-09-15T03:50:31.355177+0000","last_fullsized":"2024-09-15T03:50:31.355177+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.355029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.355029+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.355029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:30.305206+0000","scrub_duration":0.001910449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":92,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:40.948679+0000","last_change":"2024-09-15T03:50:40.948653+0000","last_active":"2024-09-15T03:50:40.948679+0000","last_peered":"2024-09-15T03:50:40.948679+0000","last_clean":"2024-09-15T03:50:40.948679+0000","last_became_active":"2024-09-15T03:35:36.176264+0000","last_became_peered":"2024-09-15T03:35:36.176264+0000","last_unstale":"2024-09-15T03:50:40.948679+0000","last_undegraded":"2024-09-15T03:50:40.948679+0000","last_fullsized":"2024-09-15T03:50:40.948679+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:40.948568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:40.948568+0000","last_clean_scrub_stamp":"2024-09-15T03:50:40.948568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:14:32.007649+0000","scrub_duration":0.001366894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":93,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:36.033602+0000","last_change":"2024-09-15T03:50:36.033537+0000","last_active":"2024-09-15T03:50:36.033602+0000","last_peered":"2024-09-15T03:50:36.033602+0000","last_clean":"2024-09-15T03:50:36.033602+0000","last_became_active":"2024-09-15T03:35:36.179473+0000","last_became_peered":"2024-09-15T03:35:36.179473+0000","last_unstale":"2024-09-15T03:50:36.033602+0000","last_undegraded":"2024-09-15T03:50:36.033602+0000","last_fullsized":"2024-09-15T03:50:36.033602+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:36.033440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:36.033440+0000","last_clean_scrub_stamp":"2024-09-15T03:50:36.033440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:33:54.062758+0000","scrub_duration":0.0016963340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":189,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:27.950420+0000","last_change":"2024-09-15T03:50:27.950400+0000","last_active":"2024-09-15T03:50:27.950420+0000","last_peered":"2024-09-15T03:50:27.950420+0000","last_clean":"2024-09-15T03:50:27.950420+0000","last_became_active":"2024-09-15T03:35:36.179476+0000","last_became_peered":"2024-09-15T03:35:36.179476+0000","last_unstale":"2024-09-15T03:50:27.950420+0000","last_undegraded":"2024-09-15T03:50:27.950420+0000","last_fullsized":"2024-09-15T03:50:27.950420+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:27.950332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:27.950332+0000","last_clean_scrub_stamp":"2024-09-15T03:50:27.950332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:42:09.314865+0000","scrub_duration":0.001136597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":89,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.995010+0000","last_change":"2024-09-15T03:50:31.994958+0000","last_active":"2024-09-15T03:50:31.995010+0000","last_peered":"2024-09-15T03:50:31.995010+0000","last_clean":"2024-09-15T03:50:31.995010+0000","last_became_active":"2024-09-15T03:35:41.930552+0000","last_became_peered":"2024-09-15T03:35:41.930552+0000","last_unstale":"2024-09-15T03:50:31.995010+0000","last_undegraded":"2024-09-15T03:50:31.995010+0000","last_fullsized":"2024-09-15T03:50:31.995010+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.994864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.994864+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.994864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:20:33.211826+0000","scrub_duration":0.001696489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":105,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:28.067374+0000","last_change":"2024-09-15T03:50:28.067354+0000","last_active":"2024-09-15T03:50:28.067374+0000","last_peered":"2024-09-15T03:50:28.067374+0000","last_clean":"2024-09-15T03:50:28.067374+0000","last_became_active":"2024-09-15T03:35:36.176131+0000","last_became_peered":"2024-09-15T03:35:36.176131+0000","last_unstale":"2024-09-15T03:50:28.067374+0000","last_undegraded":"2024-09-15T03:50:28.067374+0000","last_fullsized":"2024-09-15T03:50:28.067374+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:28.067292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:28.067292+0000","last_clean_scrub_stamp":"2024-09-15T03:50:28.067292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:54:55.824622+0000","scrub_duration":0.0016256549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"259'74","reported_seq":345,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:39.282505+0000","last_change":"2024-09-15T03:50:39.282468+0000","last_active":"2024-09-15T03:50:39.282505+0000","last_peered":"2024-09-15T03:50:39.282505+0000","last_clean":"2024-09-15T03:50:39.282505+0000","last_became_active":"2024-09-15T03:35:36.179501+0000","last_became_peered":"2024-09-15T03:35:36.179501+0000","last_unstale":"2024-09-15T03:50:39.282505+0000","last_undegraded":"2024-09-15T03:50:39.282505+0000","last_fullsized":"2024-09-15T03:50:39.282505+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'74","last_scrub_stamp":"2024-09-15T03:50:39.282340+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2024-09-15T03:50:39.282340+0000","last_clean_scrub_stamp":"2024-09-15T03:50:39.282340+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:04:02.953146+0000","scrub_duration":0.0050212859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"259'2","reported_seq":266,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:36.530041+0000","last_change":"2024-09-15T03:50:36.529979+0000","last_active":"2024-09-15T03:50:36.530041+0000","last_peered":"2024-09-15T03:50:36.530041+0000","last_clean":"2024-09-15T03:50:36.530041+0000","last_became_active":"2024-09-15T03:35:36.175947+0000","last_became_peered":"2024-09-15T03:35:36.175947+0000","last_unstale":"2024-09-15T03:50:36.530041+0000","last_undegraded":"2024-09-15T03:50:36.530041+0000","last_fullsized":"2024-09-15T03:50:36.530041+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'2","last_scrub_stamp":"2024-09-15T03:50:36.529880+0000","last_deep_scrub":"259'2","last_deep_scrub_stamp":"2024-09-15T03:50:36.529880+0000","last_clean_scrub_stamp":"2024-09-15T03:50:36.529880+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:36:25.081029+0000","scrub_duration":0.0034767890000000001,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:28.251180+0000","last_change":"2024-09-15T03:50:28.251151+0000","last_active":"2024-09-15T03:50:28.251180+0000","last_peered":"2024-09-15T03:50:28.251180+0000","last_clean":"2024-09-15T03:50:28.251180+0000","last_became_active":"2024-09-15T03:35:36.177043+0000","last_became_peered":"2024-09-15T03:35:36.177043+0000","last_unstale":"2024-09-15T03:50:28.251180+0000","last_undegraded":"2024-09-15T03:50:28.251180+0000","last_fullsized":"2024-09-15T03:50:28.251180+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:28.251055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:28.251055+0000","last_clean_scrub_stamp":"2024-09-15T03:50:28.251055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:00.307485+0000","scrub_duration":0.0014808110000000001,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.006445+0000","last_change":"2024-09-15T03:50:31.006409+0000","last_active":"2024-09-15T03:50:31.006445+0000","last_peered":"2024-09-15T03:50:31.006445+0000","last_clean":"2024-09-15T03:50:31.006445+0000","last_became_active":"2024-09-15T03:35:36.176997+0000","last_became_peered":"2024-09-15T03:35:36.176997+0000","last_unstale":"2024-09-15T03:50:31.006445+0000","last_undegraded":"2024-09-15T03:50:31.006445+0000","last_fullsized":"2024-09-15T03:50:31.006445+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.006319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.006319+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.006319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:42.065334+0000","scrub_duration":0.0013913899999999999,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"259'3","reported_seq":263,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:37.485554+0000","last_change":"2024-09-15T03:50:37.485527+0000","last_active":"2024-09-15T03:50:37.485554+0000","last_peered":"2024-09-15T03:50:37.485554+0000","last_clean":"2024-09-15T03:50:37.485554+0000","last_became_active":"2024-09-15T03:35:36.176519+0000","last_became_peered":"2024-09-15T03:35:36.176519+0000","last_unstale":"2024-09-15T03:50:37.485554+0000","last_undegraded":"2024-09-15T03:50:37.485554+0000","last_fullsized":"2024-09-15T03:50:37.485554+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'3","last_scrub_stamp":"2024-09-15T03:50:37.485450+0000","last_deep_scrub":"259'3","last_deep_scrub_stamp":"2024-09-15T03:50:37.485450+0000","last_clean_scrub_stamp":"2024-09-15T03:50:37.485450+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:47:17.086755+0000","scrub_duration":0.0031258979999999998,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":238,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:38:21.831659+0000","last_change":"2024-09-15T03:35:42.047866+0000","last_active":"2024-09-15T03:38:21.831659+0000","last_peered":"2024-09-15T03:38:21.831659+0000","last_clean":"2024-09-15T03:38:21.831659+0000","last_became_active":"2024-09-15T03:35:42.047571+0000","last_became_peered":"2024-09-15T03:35:42.047571+0000","last_unstale":"2024-09-15T03:38:21.831659+0000","last_undegraded":"2024-09-15T03:38:21.831659+0000","last_fullsized":"2024-09-15T03:38:21.831659+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":85,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:38:21.831515+0000","last_change":"2024-09-15T03:35:36.180602+0000","last_active":"2024-09-15T03:38:21.831515+0000","last_peered":"2024-09-15T03:38:21.831515+0000","last_clean":"2024-09-15T03:38:21.831515+0000","last_became_active":"2024-09-15T03:35:36.180366+0000","last_became_peered":"2024-09-15T03:35:36.180366+0000","last_unstale":"2024-09-15T03:38:21.831515+0000","last_undegraded":"2024-09-15T03:38:21.831515+0000","last_fullsized":"2024-09-15T03:38:21.831515+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:28:35.665354+0000","last_clean_scrub_stamp":"2024-09-15T03:28:35.665354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":92,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:29.049992+0000","last_change":"2024-09-15T03:50:29.049969+0000","last_active":"2024-09-15T03:50:29.049992+0000","last_peered":"2024-09-15T03:50:29.049992+0000","last_clean":"2024-09-15T03:50:29.049992+0000","last_became_active":"2024-09-15T03:35:36.176265+0000","last_became_peered":"2024-09-15T03:35:36.176265+0000","last_unstale":"2024-09-15T03:50:29.049992+0000","last_undegraded":"2024-09-15T03:50:29.049992+0000","last_fullsized":"2024-09-15T03:50:29.049992+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:29.049878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:29.049878+0000","last_clean_scrub_stamp":"2024-09-15T03:50:29.049878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:16:52.760560+0000","scrub_duration":0.001864816,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:43.333366+0000","last_change":"2024-09-15T03:50:43.333332+0000","last_active":"2024-09-15T03:50:43.333366+0000","last_peered":"2024-09-15T03:50:43.333366+0000","last_clean":"2024-09-15T03:50:43.333366+0000","last_became_active":"2024-09-15T03:35:36.175986+0000","last_became_peered":"2024-09-15T03:35:36.175986+0000","last_unstale":"2024-09-15T03:50:43.333366+0000","last_undegraded":"2024-09-15T03:50:43.333366+0000","last_fullsized":"2024-09-15T03:50:43.333366+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:43.333243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:43.333243+0000","last_clean_scrub_stamp":"2024-09-15T03:50:43.333243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:43:41.149745+0000","scrub_duration":0.0014005630000000001,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302824,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069016,"statfs":{"total":96636764160,"available":96326672384,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63700000000000001,"5min":0.873,"15min":0.873},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59599999999999997,"5min":0.68400000000000005,"15min":0.751},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.505},"min":{"1min":0.379,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73799999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.33800000000000002},"max":{"1min":0.60899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71199999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.74199999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.248,"5min":0.247,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.62},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.379},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":0.45200000000000001,"5min":0.63100000000000001,"15min":0.64400000000000002},"last":0.32100000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:50:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39800000000000002,"15min":0.39100000000000001},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.38},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"min":{"1min":0.35299999999999998,"5min":0.27200000000000002,"15min":0.22500000000000001},"max":{"1min":0.58199999999999996,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298660,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073180,"statfs":{"total":96636764160,"available":96330936320,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.69099999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.499},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.64100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.56399999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":0.59899999999999998,"5min":0.67700000000000005,"15min":0.70499999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.32800000000000001},"max":{"1min":0.629,"5min":0.66000000000000003,"15min":0.71199999999999997},"last":0.58999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34599999999999997},"max":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.72199999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35299999999999998,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.72299999999999998},"last":0.66200000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.627},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22700000000000001},"max":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.46800000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.44},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.60099999999999998,"5min":0.63,"15min":0.63},"last":0.53600000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640525,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303340,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068500,"statfs":{"total":96636764160,"available":96326144000,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.50800000000000001,"15min":0.49099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61499999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.51800000000000002},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.76000000000000001,"15min":0.79700000000000004},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61899999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53100000000000003,"15min":0.503},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63900000000000001,"5min":0.755,"15min":0.75700000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":1.597},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.73299999999999998,"15min":0.84199999999999997},"last":0.60399999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.19600000000000001},"max":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.65400000000000003},"last":0.436},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"min":{"1min":0.28999999999999998,"5min":0.26800000000000002,"15min":0.222},"max":{"1min":0.53200000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.40100000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.379,"15min":0.38200000000000001},"min":{"1min":0.248,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58699999999999997,"5min":0.63200000000000001,"15min":0.65500000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.441,"15min":0.42599999999999999},"min":{"1min":0.29999999999999999,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.46600000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640527,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298660,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073180,"statfs":{"total":96636764160,"available":96330936320,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31,"15min":0.308},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.58799999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.72699999999999998},"last":0.379}]},{"osd":1,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":0.48699999999999999,"5min":0.51200000000000001,"15min":0.59599999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.22},"max":{"1min":0.50600000000000001,"5min":0.57499999999999996,"15min":0.68500000000000005},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"last":0.622},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.35299999999999998,"15min":0.26700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.60799999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.314},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":5.6269999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.54200000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.29199999999999998},"max":{"1min":0.65800000000000003,"5min":0.77500000000000002,"15min":0.84999999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72799999999999998,"5min":0.77300000000000002,"15min":0.82599999999999996},"last":0.52600000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640524,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302824,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069016,"statfs":{"total":96636764160,"available":96326672384,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32500000000000001,"15min":0.32700000000000001},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":0.41999999999999998,"5min":0.52800000000000002,"15min":0.874},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.48999999999999999,"15min":0.55900000000000005},"last":0.30499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.215},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.77800000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.50900000000000001,"15min":0.747},"last":0.377}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.73199999999999998,"15min":0.89800000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51200000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.88300000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.76600000000000001,"15min":0.81799999999999995},"last":0.54900000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.93799999999999994},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.432,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.91300000000000003},"last":0.52000000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640525,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303340,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068500,"statfs":{"total":96636764160,"available":96326144000,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.625},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33300000000000002},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.497,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.36699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":0.52600000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34799999999999998,"15min":0.35499999999999998},"min":{"1min":0.26400000000000001,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.47399999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.29299999999999998},"max":{"1min":0.79000000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.495},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.72599999999999998,"5min":0.77900000000000003,"15min":0.85699999999999998},"last":0.46899999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.42099999999999999,"5min":0.313,"15min":0.312},"max":{"1min":0.72499999999999998,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.441,"5min":0.34899999999999998,"15min":0.34200000000000003},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.81200000000000006},"last":0.54300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.78000000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73799999999999999,"5min":0.84099999999999997,"15min":0.84799999999999998},"last":0.51200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:50:48.575 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:48.575 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T03:28:35.665354+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=28, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=50, tm_sec=24, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:50:48.575 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:51:08.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:51:08.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.680+0000 7f79f771d640 10 monclient: get_monmap_and_config 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.680+0000 7f79f771d640 10 monclient: build_initial_monmap 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.680+0000 7f79f771d640 10 monclient: monmap: 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:51:08.681875+0000 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:51:08.681875+0000 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:51:08.682 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:51:08.683 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.680+0000 7f79f771d640 1 Processor -- start 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 1 -- start start 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient: init 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient: _reopen_session rank -1 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): picked mon.noname-b con 0x7f79f0131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f79f009e910 0x7f79f009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): picked mon.noname-a con 0x7f79f009e910 addr v2:172.21.15.153:3300/0 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79f009f220 addr v1:172.21.15.181:6789/0 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f009faf0 con 0x7f79f009f220 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f009fd30 con 0x7f79f009e910 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f00a1110 con 0x7f79f0131ab0 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): _renew_subs 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f771d640 10 monclient(hunting): authenticate will time out at 2066.321575s 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f79f009e910 0x7f79f009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f79f009e910 0x7f79f009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35338/0 (socket says 172.21.15.153:35338) 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45944/0 (socket says 172.21.15.153:45944) 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 1 -- 172.21.15.153:0/2099593964 learned_addr learned my addr 172.21.15.153:0/2099593964 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:51:08.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): get_auth_request con 0x7f79f009e910 auth_method 0 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): get_auth_request con 0x7f79f0131ab0 auth_method 0 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f4cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.681+0000 7f79f54b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 10 monclient(hunting): handle_auth_done global_id 254331 payload 274 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 1 -- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 msgr2=0x7f79f0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 1 --2- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 1 -- 172.21.15.153:0/2099593964 >> v1:172.21.15.181:6789/0 conn(0x7f79f009f220 legacy=0x7f79f0068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 10 monclient: _finish_hunting 0 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 1 monclient: found mon.noname-a 2024-09-15T03:51:08.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:51:08.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:51:08.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.682+0000 7f79f4cb8640 1 -- 172.21.15.153:0/2099593964 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f00a13f0 con 0x7f79f009e910 2024-09-15T03:51:08.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.684+0000 7f79f4cb8640 1 -- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 msgr2=0x7f79f009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:51:08.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.684+0000 7f79f4cb8640 1 --2- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009ece0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009910 tx=0x7f79e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf60e3c0e6906a49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:51:08.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 1 -- 172.21.15.153:0/2099593964 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0032440 con 0x7f79f009e910 2024-09-15T03:51:08.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: dump: 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: _finish_auth 0 2024-09-15T03:51:08.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 1 -- 172.21.15.153:0/2099593964 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e00325a0 con 0x7f79f009e910 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 5 monclient: authenticate success, global_id 254331 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 10 monclient: get_monmap_and_config success 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 1 -- 172.21.15.153:0/2099593964 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0032940 con 0x7f79f009e910 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 4 set_mon_vals no callback set 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79e7fff640 10 monclient: dump: 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.699 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 10 monclient: shutdown 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 1 -- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 msgr2=0x7f79f009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 1 --2- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009ece0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009910 tx=0x7f79e00091d0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 1 -- 172.21.15.153:0/2099593964 shutdown_connections 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 1 --2- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009ece0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 1 --2- 172.21.15.153:0/2099593964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.685+0000 7f79f771d640 1 -- 172.21.15.153:0/2099593964 >> 172.21.15.153:0/2099593964 conn(0x7f79f0067d30 msgr2=0x7f79f00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 1 -- 172.21.15.153:0/2099593964 shutdown_connections 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 1 -- 172.21.15.153:0/2099593964 wait complete. 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 10 monclient: build_initial_monmap 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 10 monclient: monmap: 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:51:08.701 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 1 Processor -- start 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 1 -- start start 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.686+0000 7f79f771d640 10 monclient: init 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient: _reopen_session rank -1 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): picked mon.noname-a con 0x7f79f009e910 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f009f220 0x7f79f0130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): picked mon.noname-b con 0x7f79f009f220 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:51:08.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): picked mon.noname-c con 0x7f79f0131ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f009b640 con 0x7f79f009e910 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f00a4530 con 0x7f79f009f220 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f011c630 con 0x7f79f0131ab0 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): _renew_subs 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f771d640 10 monclient(hunting): authenticate will time out at 2066.327552s 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35344/0 (socket says 172.21.15.153:35344) 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 -- 172.21.15.153:0/768218767 learned_addr learned my addr 172.21.15.153:0/768218767 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:51:08.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): get_auth_request con 0x7f79f009e910 auth_method 0 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): get_auth_request con 0x7f79f0131ab0 auth_method 0 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f4cb8640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f009f220 0x7f79f0130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f4cb8640 10 monclient(hunting): get_auth_request con 0x7f79f009f220 auth_method 0 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f4cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f4cb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f4cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient(hunting): handle_auth_done global_id 254334 payload 824 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 -- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 msgr2=0x7f79f0140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 -- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f009f220 msgr2=0x7f79f0130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f009f220 0x7f79f0130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient: _finish_hunting 0 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 monclient: found mon.noname-a 2024-09-15T03:51:08.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f54b9640 1 -- 172.21.15.153:0/768218767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0009450 con 0x7f79f009e910 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.687+0000 7f79f5cba640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f54b9640 1 -- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 msgr2=0x7f79f009aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f54b9640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009aaf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009ef0 tx=0x7f79ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00ce10 con 0x7f79f009e910 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: dump: 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.706 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: _finish_auth 0 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec00f040 con 0x7f79f009e910 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 5 monclient: authenticate success, global_id 254334 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 10 monclient: _renew_subs 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0148c0 con 0x7f79f009e910 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:51:08.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f011dca0 con 0x7f79f009e910 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e57fa640 10 monclient: dump: 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79e77fe640 4 set_mon_vals no callback set 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 10 monclient: _renew_subs 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.690+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f011cd40 con 0x7f79f009e910 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7f79ec014c30 con 0x7f79f009e910 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79f771d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79f771d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79f771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f004bf60 con 0x7f79f009e910 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79e57fa640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79d4038740 0x7f79d403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f79ec04d630 con 0x7f79f009e910 2024-09-15T03:51:08.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79f4cb8640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79d4038740 0x7f79d403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.691+0000 7f79f4cb8640 10 monclient: get_auth_request con 0x7f79d4038740 auth_method 0 2024-09-15T03:51:08.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.694+0000 7f79f4cb8640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79d4038740 0x7f79d403abf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f79f009bd80 tx=0x7f79e0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:51:08.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.696+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f79ec0133e0 con 0x7f79f009e910 2024-09-15T03:51:08.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.696+0000 7f79e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:51:08.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.696+0000 7f79e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:51:08.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.847+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 --> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79f0066280 con 0x7f79d4038740 2024-09-15T03:51:08.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.851+0000 7f79e57fa640 1 -- 172.21.15.153:0/768218767 <== mgr.4105 v2:172.21.15.181:6824/31449 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98517 (secure 0 0 0) 0x7f79f0066280 con 0x7f79d4038740 2024-09-15T03:51:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-15T03:51:08.854 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-15T03:51:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79d4038740 msgr2=0x7f79d403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79d4038740 0x7f79d403abf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f79f009bd80 tx=0x7f79e0032000 comp rx=0 tx=0).stop 2024-09-15T03:51:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 10 monclient: shutdown 2024-09-15T03:51:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:51:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 msgr2=0x7f79f009aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009aaf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009ef0 tx=0x7f79ec00b040 comp rx=0 tx=0).stop 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 shutdown_connections 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7f79d4038740 0x7f79d403abf0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79f0131ab0 0x7f79f0140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f009f220 0x7f79f0130b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 --2- 172.21.15.153:0/768218767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f009e910 0x7f79f009aaf0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 >> 172.21.15.153:0/768218767 conn(0x7f79f0067d30 msgr2=0x7f79f009c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 shutdown_connections 2024-09-15T03:51:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.853+0000 7f79f771d640 1 -- 172.21.15.153:0/768218767 wait complete. 2024-09-15T03:51:08.865 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":931,"stamp":"2024-09-15T03:51:08.125016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1809648,"kb_used_data":8896,"kb_used_omap":9,"kb_used_meta":1800694,"kb_avail":564421392,"statfs":{"total":579820584960,"available":577967505408,"internally_reserved":0,"allocated":9109504,"data_stored":7285462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843910672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002409"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":182,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:33.030467+0000","last_change":"2024-09-15T03:50:33.030419+0000","last_active":"2024-09-15T03:50:33.030467+0000","last_peered":"2024-09-15T03:50:33.030467+0000","last_clean":"2024-09-15T03:50:33.030467+0000","last_became_active":"2024-09-15T03:35:36.177073+0000","last_became_peered":"2024-09-15T03:35:36.177073+0000","last_unstale":"2024-09-15T03:50:33.030467+0000","last_undegraded":"2024-09-15T03:50:33.030467+0000","last_fullsized":"2024-09-15T03:50:33.030467+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:33.030284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:33.030284+0000","last_clean_scrub_stamp":"2024-09-15T03:50:33.030284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:25:32.127112+0000","scrub_duration":0.0018146010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":101,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:34.491280+0000","last_change":"2024-09-15T03:50:34.491250+0000","last_active":"2024-09-15T03:50:34.491280+0000","last_peered":"2024-09-15T03:50:34.491280+0000","last_clean":"2024-09-15T03:50:34.491280+0000","last_became_active":"2024-09-15T03:35:36.170838+0000","last_became_peered":"2024-09-15T03:35:36.170838+0000","last_unstale":"2024-09-15T03:50:34.491280+0000","last_undegraded":"2024-09-15T03:50:34.491280+0000","last_fullsized":"2024-09-15T03:50:34.491280+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:34.491161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:34.491161+0000","last_clean_scrub_stamp":"2024-09-15T03:50:34.491161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:48:38.374243+0000","scrub_duration":0.0015124820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":157,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:27.030711+0000","last_change":"2024-09-15T03:50:27.030681+0000","last_active":"2024-09-15T03:50:27.030711+0000","last_peered":"2024-09-15T03:50:27.030711+0000","last_clean":"2024-09-15T03:50:27.030711+0000","last_became_active":"2024-09-15T03:35:36.180590+0000","last_became_peered":"2024-09-15T03:35:36.180590+0000","last_unstale":"2024-09-15T03:50:27.030711+0000","last_undegraded":"2024-09-15T03:50:27.030711+0000","last_fullsized":"2024-09-15T03:50:27.030711+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:27.030599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:27.030599+0000","last_clean_scrub_stamp":"2024-09-15T03:50:27.030599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:06:36.467272+0000","scrub_duration":0.0014349160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":101,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.537028+0000","last_change":"2024-09-15T03:50:31.536997+0000","last_active":"2024-09-15T03:50:31.537028+0000","last_peered":"2024-09-15T03:50:31.537028+0000","last_clean":"2024-09-15T03:50:31.537028+0000","last_became_active":"2024-09-15T03:35:36.173628+0000","last_became_peered":"2024-09-15T03:35:36.173628+0000","last_unstale":"2024-09-15T03:50:31.537028+0000","last_undegraded":"2024-09-15T03:50:31.537028+0000","last_fullsized":"2024-09-15T03:50:31.537028+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.536914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.536914+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.536914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:04:41.679502+0000","scrub_duration":0.00189146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":142,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:38.465233+0000","last_change":"2024-09-15T03:50:38.465181+0000","last_active":"2024-09-15T03:50:38.465233+0000","last_peered":"2024-09-15T03:50:38.465233+0000","last_clean":"2024-09-15T03:50:38.465233+0000","last_became_active":"2024-09-15T03:35:42.047949+0000","last_became_peered":"2024-09-15T03:35:42.047949+0000","last_unstale":"2024-09-15T03:50:38.465233+0000","last_undegraded":"2024-09-15T03:50:38.465233+0000","last_fullsized":"2024-09-15T03:50:38.465233+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:38.465085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:38.465085+0000","last_clean_scrub_stamp":"2024-09-15T03:50:38.465085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:21:41.366790+0000","scrub_duration":0.0017380150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":158,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:30.974493+0000","last_change":"2024-09-15T03:50:30.974458+0000","last_active":"2024-09-15T03:50:30.974493+0000","last_peered":"2024-09-15T03:50:30.974493+0000","last_clean":"2024-09-15T03:50:30.974493+0000","last_became_active":"2024-09-15T03:35:36.179302+0000","last_became_peered":"2024-09-15T03:35:36.179302+0000","last_unstale":"2024-09-15T03:50:30.974493+0000","last_undegraded":"2024-09-15T03:50:30.974493+0000","last_fullsized":"2024-09-15T03:50:30.974493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:30.974370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:30.974370+0000","last_clean_scrub_stamp":"2024-09-15T03:50:30.974370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:17:55.985403+0000","scrub_duration":0.001722765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":96,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:33.257345+0000","last_change":"2024-09-15T03:50:33.257279+0000","last_active":"2024-09-15T03:50:33.257345+0000","last_peered":"2024-09-15T03:50:33.257345+0000","last_clean":"2024-09-15T03:50:33.257345+0000","last_became_active":"2024-09-15T03:35:36.179540+0000","last_became_peered":"2024-09-15T03:35:36.179540+0000","last_unstale":"2024-09-15T03:50:33.257345+0000","last_undegraded":"2024-09-15T03:50:33.257345+0000","last_fullsized":"2024-09-15T03:50:33.257345+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:33.257156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:33.257156+0000","last_clean_scrub_stamp":"2024-09-15T03:50:33.257156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:41:55.192997+0000","scrub_duration":0.002146897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":92,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:30.052690+0000","last_change":"2024-09-15T03:50:30.052663+0000","last_active":"2024-09-15T03:50:30.052690+0000","last_peered":"2024-09-15T03:50:30.052690+0000","last_clean":"2024-09-15T03:50:30.052690+0000","last_became_active":"2024-09-15T03:35:36.176169+0000","last_became_peered":"2024-09-15T03:35:36.176169+0000","last_unstale":"2024-09-15T03:50:30.052690+0000","last_undegraded":"2024-09-15T03:50:30.052690+0000","last_fullsized":"2024-09-15T03:50:30.052690+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:30.052572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:30.052572+0000","last_clean_scrub_stamp":"2024-09-15T03:50:30.052572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:27:20.777490+0000","scrub_duration":0.001622662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":125,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:33.523390+0000","last_change":"2024-09-15T03:50:33.523332+0000","last_active":"2024-09-15T03:50:33.523390+0000","last_peered":"2024-09-15T03:50:33.523390+0000","last_clean":"2024-09-15T03:50:33.523390+0000","last_became_active":"2024-09-15T03:35:36.175786+0000","last_became_peered":"2024-09-15T03:35:36.175786+0000","last_unstale":"2024-09-15T03:50:33.523390+0000","last_undegraded":"2024-09-15T03:50:33.523390+0000","last_fullsized":"2024-09-15T03:50:33.523390+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:33.523234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:33.523234+0000","last_clean_scrub_stamp":"2024-09-15T03:50:33.523234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:12:24.120105+0000","scrub_duration":0.0015875430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":120,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:34.286441+0000","last_change":"2024-09-15T03:50:34.286411+0000","last_active":"2024-09-15T03:50:34.286441+0000","last_peered":"2024-09-15T03:50:34.286441+0000","last_clean":"2024-09-15T03:50:34.286441+0000","last_became_active":"2024-09-15T03:35:36.173941+0000","last_became_peered":"2024-09-15T03:35:36.173941+0000","last_unstale":"2024-09-15T03:50:34.286441+0000","last_undegraded":"2024-09-15T03:50:34.286441+0000","last_fullsized":"2024-09-15T03:50:34.286441+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:34.286316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:34.286316+0000","last_clean_scrub_stamp":"2024-09-15T03:50:34.286316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:34:43.710252+0000","scrub_duration":0.0016205499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":133,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.355177+0000","last_change":"2024-09-15T03:50:31.355124+0000","last_active":"2024-09-15T03:50:31.355177+0000","last_peered":"2024-09-15T03:50:31.355177+0000","last_clean":"2024-09-15T03:50:31.355177+0000","last_became_active":"2024-09-15T03:35:36.173027+0000","last_became_peered":"2024-09-15T03:35:36.173027+0000","last_unstale":"2024-09-15T03:50:31.355177+0000","last_undegraded":"2024-09-15T03:50:31.355177+0000","last_fullsized":"2024-09-15T03:50:31.355177+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.355029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.355029+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.355029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:30.305206+0000","scrub_duration":0.001910449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":92,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:40.948679+0000","last_change":"2024-09-15T03:50:40.948653+0000","last_active":"2024-09-15T03:50:40.948679+0000","last_peered":"2024-09-15T03:50:40.948679+0000","last_clean":"2024-09-15T03:50:40.948679+0000","last_became_active":"2024-09-15T03:35:36.176264+0000","last_became_peered":"2024-09-15T03:35:36.176264+0000","last_unstale":"2024-09-15T03:50:40.948679+0000","last_undegraded":"2024-09-15T03:50:40.948679+0000","last_fullsized":"2024-09-15T03:50:40.948679+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:40.948568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:40.948568+0000","last_clean_scrub_stamp":"2024-09-15T03:50:40.948568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:14:32.007649+0000","scrub_duration":0.001366894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":93,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:36.033602+0000","last_change":"2024-09-15T03:50:36.033537+0000","last_active":"2024-09-15T03:50:36.033602+0000","last_peered":"2024-09-15T03:50:36.033602+0000","last_clean":"2024-09-15T03:50:36.033602+0000","last_became_active":"2024-09-15T03:35:36.179473+0000","last_became_peered":"2024-09-15T03:35:36.179473+0000","last_unstale":"2024-09-15T03:50:36.033602+0000","last_undegraded":"2024-09-15T03:50:36.033602+0000","last_fullsized":"2024-09-15T03:50:36.033602+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:36.033440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:36.033440+0000","last_clean_scrub_stamp":"2024-09-15T03:50:36.033440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:33:54.062758+0000","scrub_duration":0.0016963340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":189,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:27.950420+0000","last_change":"2024-09-15T03:50:27.950400+0000","last_active":"2024-09-15T03:50:27.950420+0000","last_peered":"2024-09-15T03:50:27.950420+0000","last_clean":"2024-09-15T03:50:27.950420+0000","last_became_active":"2024-09-15T03:35:36.179476+0000","last_became_peered":"2024-09-15T03:35:36.179476+0000","last_unstale":"2024-09-15T03:50:27.950420+0000","last_undegraded":"2024-09-15T03:50:27.950420+0000","last_fullsized":"2024-09-15T03:50:27.950420+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:27.950332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:27.950332+0000","last_clean_scrub_stamp":"2024-09-15T03:50:27.950332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:42:09.314865+0000","scrub_duration":0.001136597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":89,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.995010+0000","last_change":"2024-09-15T03:50:31.994958+0000","last_active":"2024-09-15T03:50:31.995010+0000","last_peered":"2024-09-15T03:50:31.995010+0000","last_clean":"2024-09-15T03:50:31.995010+0000","last_became_active":"2024-09-15T03:35:41.930552+0000","last_became_peered":"2024-09-15T03:35:41.930552+0000","last_unstale":"2024-09-15T03:50:31.995010+0000","last_undegraded":"2024-09-15T03:50:31.995010+0000","last_fullsized":"2024-09-15T03:50:31.995010+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.994864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.994864+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.994864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:20:33.211826+0000","scrub_duration":0.001696489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":105,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:28.067374+0000","last_change":"2024-09-15T03:50:28.067354+0000","last_active":"2024-09-15T03:50:28.067374+0000","last_peered":"2024-09-15T03:50:28.067374+0000","last_clean":"2024-09-15T03:50:28.067374+0000","last_became_active":"2024-09-15T03:35:36.176131+0000","last_became_peered":"2024-09-15T03:35:36.176131+0000","last_unstale":"2024-09-15T03:50:28.067374+0000","last_undegraded":"2024-09-15T03:50:28.067374+0000","last_fullsized":"2024-09-15T03:50:28.067374+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:28.067292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:28.067292+0000","last_clean_scrub_stamp":"2024-09-15T03:50:28.067292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:54:55.824622+0000","scrub_duration":0.0016256549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"259'74","reported_seq":345,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:39.282505+0000","last_change":"2024-09-15T03:50:39.282468+0000","last_active":"2024-09-15T03:50:39.282505+0000","last_peered":"2024-09-15T03:50:39.282505+0000","last_clean":"2024-09-15T03:50:39.282505+0000","last_became_active":"2024-09-15T03:35:36.179501+0000","last_became_peered":"2024-09-15T03:35:36.179501+0000","last_unstale":"2024-09-15T03:50:39.282505+0000","last_undegraded":"2024-09-15T03:50:39.282505+0000","last_fullsized":"2024-09-15T03:50:39.282505+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'74","last_scrub_stamp":"2024-09-15T03:50:39.282340+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2024-09-15T03:50:39.282340+0000","last_clean_scrub_stamp":"2024-09-15T03:50:39.282340+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:04:02.953146+0000","scrub_duration":0.0050212859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"259'2","reported_seq":266,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:36.530041+0000","last_change":"2024-09-15T03:50:36.529979+0000","last_active":"2024-09-15T03:50:36.530041+0000","last_peered":"2024-09-15T03:50:36.530041+0000","last_clean":"2024-09-15T03:50:36.530041+0000","last_became_active":"2024-09-15T03:35:36.175947+0000","last_became_peered":"2024-09-15T03:35:36.175947+0000","last_unstale":"2024-09-15T03:50:36.530041+0000","last_undegraded":"2024-09-15T03:50:36.530041+0000","last_fullsized":"2024-09-15T03:50:36.530041+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'2","last_scrub_stamp":"2024-09-15T03:50:36.529880+0000","last_deep_scrub":"259'2","last_deep_scrub_stamp":"2024-09-15T03:50:36.529880+0000","last_clean_scrub_stamp":"2024-09-15T03:50:36.529880+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:36:25.081029+0000","scrub_duration":0.0034767890000000001,"objects_trimmed":0,"snaptrim_duration":0.00032055099999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:28.251180+0000","last_change":"2024-09-15T03:50:28.251151+0000","last_active":"2024-09-15T03:50:28.251180+0000","last_peered":"2024-09-15T03:50:28.251180+0000","last_clean":"2024-09-15T03:50:28.251180+0000","last_became_active":"2024-09-15T03:35:36.177043+0000","last_became_peered":"2024-09-15T03:35:36.177043+0000","last_unstale":"2024-09-15T03:50:28.251180+0000","last_undegraded":"2024-09-15T03:50:28.251180+0000","last_fullsized":"2024-09-15T03:50:28.251180+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:28.251055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:28.251055+0000","last_clean_scrub_stamp":"2024-09-15T03:50:28.251055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:00.307485+0000","scrub_duration":0.0014808110000000001,"objects_trimmed":0,"snaptrim_duration":0.00051635400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:31.006445+0000","last_change":"2024-09-15T03:50:31.006409+0000","last_active":"2024-09-15T03:50:31.006445+0000","last_peered":"2024-09-15T03:50:31.006445+0000","last_clean":"2024-09-15T03:50:31.006445+0000","last_became_active":"2024-09-15T03:35:36.176997+0000","last_became_peered":"2024-09-15T03:35:36.176997+0000","last_unstale":"2024-09-15T03:50:31.006445+0000","last_undegraded":"2024-09-15T03:50:31.006445+0000","last_fullsized":"2024-09-15T03:50:31.006445+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:31.006319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:31.006319+0000","last_clean_scrub_stamp":"2024-09-15T03:50:31.006319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:42.065334+0000","scrub_duration":0.0013913899999999999,"objects_trimmed":0,"snaptrim_duration":0.00045240500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"259'3","reported_seq":263,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:37.485554+0000","last_change":"2024-09-15T03:50:37.485527+0000","last_active":"2024-09-15T03:50:37.485554+0000","last_peered":"2024-09-15T03:50:37.485554+0000","last_clean":"2024-09-15T03:50:37.485554+0000","last_became_active":"2024-09-15T03:35:36.176519+0000","last_became_peered":"2024-09-15T03:35:36.176519+0000","last_unstale":"2024-09-15T03:50:37.485554+0000","last_undegraded":"2024-09-15T03:50:37.485554+0000","last_fullsized":"2024-09-15T03:50:37.485554+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'3","last_scrub_stamp":"2024-09-15T03:50:37.485450+0000","last_deep_scrub":"259'3","last_deep_scrub_stamp":"2024-09-15T03:50:37.485450+0000","last_clean_scrub_stamp":"2024-09-15T03:50:37.485450+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:47:17.086755+0000","scrub_duration":0.0031258979999999998,"objects_trimmed":0,"snaptrim_duration":0.000739016,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":248,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:56.935503+0000","last_change":"2024-09-15T03:50:56.935470+0000","last_active":"2024-09-15T03:50:56.935503+0000","last_peered":"2024-09-15T03:50:56.935503+0000","last_clean":"2024-09-15T03:50:56.935503+0000","last_became_active":"2024-09-15T03:35:42.047571+0000","last_became_peered":"2024-09-15T03:35:42.047571+0000","last_unstale":"2024-09-15T03:50:56.935503+0000","last_undegraded":"2024-09-15T03:50:56.935503+0000","last_fullsized":"2024-09-15T03:50:56.935503+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:56.935366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:56.935366+0000","last_clean_scrub_stamp":"2024-09-15T03:50:56.935366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:11.830106+0000","scrub_duration":0.001477332,"objects_trimmed":0,"snaptrim_duration":0.00031302199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:47.379714+0000","last_change":"2024-09-15T03:50:47.379681+0000","last_active":"2024-09-15T03:50:47.379714+0000","last_peered":"2024-09-15T03:50:47.379714+0000","last_clean":"2024-09-15T03:50:47.379714+0000","last_became_active":"2024-09-15T03:35:36.180366+0000","last_became_peered":"2024-09-15T03:35:36.180366+0000","last_unstale":"2024-09-15T03:50:47.379714+0000","last_undegraded":"2024-09-15T03:50:47.379714+0000","last_fullsized":"2024-09-15T03:50:47.379714+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:47.379582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:47.379582+0000","last_clean_scrub_stamp":"2024-09-15T03:50:47.379582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:35:20.737443+0000","scrub_duration":0.0019651619999999999,"objects_trimmed":0,"snaptrim_duration":0.00034721099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":92,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:29.049992+0000","last_change":"2024-09-15T03:50:29.049969+0000","last_active":"2024-09-15T03:50:29.049992+0000","last_peered":"2024-09-15T03:50:29.049992+0000","last_clean":"2024-09-15T03:50:29.049992+0000","last_became_active":"2024-09-15T03:35:36.176265+0000","last_became_peered":"2024-09-15T03:35:36.176265+0000","last_unstale":"2024-09-15T03:50:29.049992+0000","last_undegraded":"2024-09-15T03:50:29.049992+0000","last_fullsized":"2024-09-15T03:50:29.049992+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:29.049878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:29.049878+0000","last_clean_scrub_stamp":"2024-09-15T03:50:29.049878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:16:52.760560+0000","scrub_duration":0.001864816,"objects_trimmed":0,"snaptrim_duration":0.00040382900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-15T03:50:43.333366+0000","last_change":"2024-09-15T03:50:43.333332+0000","last_active":"2024-09-15T03:50:43.333366+0000","last_peered":"2024-09-15T03:50:43.333366+0000","last_clean":"2024-09-15T03:50:43.333366+0000","last_became_active":"2024-09-15T03:35:36.175986+0000","last_became_peered":"2024-09-15T03:35:36.175986+0000","last_unstale":"2024-09-15T03:50:43.333366+0000","last_undegraded":"2024-09-15T03:50:43.333366+0000","last_fullsized":"2024-09-15T03:50:43.333366+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:50:43.333243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:50:43.333243+0000","last_clean_scrub_stamp":"2024-09-15T03:50:43.333243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:43:41.149745+0000","scrub_duration":0.0014005630000000001,"objects_trimmed":0,"snaptrim_duration":0.00038281300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302824,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069016,"statfs":{"total":96636764160,"available":96326672384,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63700000000000001,"5min":0.873,"15min":0.873},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59599999999999997,"5min":0.68400000000000005,"15min":0.751},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.505},"min":{"1min":0.379,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.73799999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.33800000000000002},"max":{"1min":0.60899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71199999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.74199999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.248,"5min":0.247,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.62},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.379},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":0.45200000000000001,"5min":0.63100000000000001,"15min":0.64400000000000002},"last":0.53600000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:50:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39800000000000002,"15min":0.39100000000000001},"min":{"1min":0.29399999999999998,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"min":{"1min":0.35299999999999998,"5min":0.27200000000000002,"15min":0.22500000000000001},"max":{"1min":0.58199999999999996,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63300000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640529,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298660,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073180,"statfs":{"total":96636764160,"available":96330936320,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.27700000000000002},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.69099999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.499},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.64100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":0.59899999999999998,"5min":0.67700000000000005,"15min":0.70499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.501,"15min":0.502},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.32800000000000001},"max":{"1min":0.629,"5min":0.66000000000000003,"15min":0.71199999999999997},"last":0.5}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34599999999999997},"max":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.72199999999999998},"last":0.437},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35299999999999998,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.72299999999999998},"last":0.47499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.57799999999999996,"15min":0.627},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.22700000000000001},"max":{"1min":0.57999999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.41699999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.60099999999999998,"5min":0.63,"15min":0.63},"last":0.46200000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640529,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303340,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068500,"statfs":{"total":96636764160,"available":96326144000,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.50800000000000001,"15min":0.49099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61499999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.371},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.51800000000000002},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.76000000000000001,"15min":0.79700000000000004},"last":0.51400000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61899999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53100000000000003,"15min":0.503},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63900000000000001,"5min":0.755,"15min":0.75700000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":1.597},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.73299999999999998,"15min":0.84199999999999997},"last":0.46100000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.19600000000000001},"max":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.65400000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"min":{"1min":0.28999999999999998,"5min":0.26800000000000002,"15min":0.222},"max":{"1min":0.53200000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.379}]},{"osd":5,"last update":"Sun Sep 15 03:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.379,"15min":0.38200000000000001},"min":{"1min":0.248,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58699999999999997,"5min":0.63200000000000001,"15min":0.65500000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.441,"15min":0.42599999999999999},"min":{"1min":0.29999999999999999,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.432}]}]},{"osd":2,"up_from":11,"seq":47244640531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298660,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073180,"statfs":{"total":96636764160,"available":96330936320,"internally_reserved":0,"allocated":1339392,"data_stored":1037928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31,"15min":0.308},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.58799999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.72699999999999998},"last":0.28399999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32800000000000001,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":0.48699999999999999,"5min":0.51200000000000001,"15min":0.59599999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.22},"max":{"1min":0.50600000000000001,"5min":0.57499999999999996,"15min":0.68500000000000005},"last":0.373}]},{"osd":3,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.35299999999999998,"15min":0.26700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57399999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.314},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":5.6269999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.51200000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.29199999999999998},"max":{"1min":0.65800000000000003,"5min":0.77500000000000002,"15min":0.84999999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72799999999999998,"5min":0.77300000000000002,"15min":0.82599999999999996},"last":0.52900000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302824,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069016,"statfs":{"total":96636764160,"available":96326672384,"internally_reserved":0,"allocated":1343488,"data_stored":1037947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32500000000000001,"15min":0.32700000000000001},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":0.41999999999999998,"5min":0.52800000000000002,"15min":0.874},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.48999999999999999,"15min":0.55900000000000005},"last":0.32500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.215},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.77800000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.50900000000000001,"15min":0.747},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.73199999999999998,"15min":0.89800000000000002},"last":0.504},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.436}]},{"osd":4,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.75600000000000001,"15min":0.88300000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.76600000000000001,"15min":0.81799999999999995},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.93799999999999994},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.54700000000000004},"min":{"1min":0.432,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.91300000000000003},"last":0.46700000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303340,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068500,"statfs":{"total":96636764160,"available":96326144000,"internally_reserved":0,"allocated":1871872,"data_stored":1566856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.625},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33300000000000002},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.497,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":0.52600000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34799999999999998,"15min":0.35499999999999998},"min":{"1min":0.26400000000000001,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.47399999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.40100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.29299999999999998},"max":{"1min":0.79000000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":0.72599999999999998,"5min":0.77900000000000003,"15min":0.85699999999999998},"last":0.60099999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.42099999999999999,"5min":0.313,"15min":0.312},"max":{"1min":0.72499999999999998,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.441,"5min":0.34899999999999998,"15min":0.34200000000000003},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.81200000000000006},"last":0.59199999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.78000000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73799999999999999,"5min":0.84099999999999997,"15min":0.84799999999999998},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:51:08.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T03:51:08.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient: get_monmap_and_config 2024-09-15T03:51:08.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient: build_initial_monmap 2024-09-15T03:51:08.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient: monmap: 2024-09-15T03:51:08.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:51:08.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:51:08.960387+0000 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:51:08.960387+0000 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.181:6789/0 mon.noname-c-legacy 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.181:3300/0 mon.noname-c 2024-09-15T03:51:08.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 1 Processor -- start 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 1 -- start start 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient: init 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient: _reopen_session rank -1 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc180974f0 addr v1:172.21.15.181:6789/0 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 1 --2- >> v2:172.21.15.181:3300/0 conn(0x7fbc1809c9d0 0x7fbc1809beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1809c9d0 addr v2:172.21.15.181:3300/0 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc1806f340 addr v1:172.21.15.153:6789/0 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc180978c0 con 0x7fbc1806f340 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1809b220 con 0x7fbc180974f0 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 1 -- --> v2:172.21.15.181:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc18099b20 con 0x7fbc1809c9d0 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient(hunting): _renew_subs 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.959+0000 7fbc1e034640 10 monclient(hunting): authenticate will time out at 2066.599938s 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc17fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbc1806f340 0x7fbc181355c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33774/0 (socket says 172.21.15.153:33774) 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc17fff640 1 -- 172.21.15.153:0/2928316165 learned_addr learned my addr 172.21.15.153:0/2928316165 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 1 --2- 172.21.15.153:0/2928316165 >> v2:172.21.15.181:3300/0 conn(0x7fbc1809c9d0 0x7fbc1809beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): get_auth_request con 0x7fbc1809c9d0 auth_method 0 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.960+0000 7fbc16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 1 -- 172.21.15.153:0/2928316165 >> v1:172.21.15.153:6789/0 conn(0x7fbc1806f340 legacy=0x7fbc181355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:51:08.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109327798 0 0) 0x7fbc180978c0 con 0x7fbc1806f340 2024-09-15T03:51:08.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): my global_id is 254337 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc000034e0 con 0x7fbc1806f340 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 -- 172.21.15.153:0/2928316165 >> v1:172.21.15.181:6789/0 conn(0x7fbc180974f0 legacy=0x7fbc1809b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4191095184 0 0) 0x7fbc000034e0 con 0x7fbc1806f340 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 >> v2:172.21.15.181:3300/0 conn(0x7fbc1809c9d0 msgr2=0x7fbc1809beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 --2- 172.21.15.153:0/2928316165 >> v2:172.21.15.181:3300/0 conn(0x7fbc1809c9d0 0x7fbc1809beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 >> v1:172.21.15.181:6789/0 conn(0x7fbc180974f0 legacy=0x7fbc1809b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: _finish_hunting 0 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-15T03:51:08.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc180988b0 con 0x7fbc1806f340 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: _finish_auth 0 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1104885554 0 0) 0x7fbc10001aa0 con 0x7fbc1806f340 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc1e034640 5 monclient: authenticate success, global_id 254337 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc1e034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: dump: 2024-09-15T03:51:08.977 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 >> v1:172.21.15.153:6789/0 conn(0x7fbc1806f340 legacy=0x7fbc181355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:51:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 0x7fbc00003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): picked mon.a con 0x7fbc00003b20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00004780 0x7fbc00004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): picked mon.b con 0x7fbc00004780 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00005330 0x7fbc0000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): picked mon.c con 0x7fbc00005330 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18099b20 con 0x7fbc00003b20 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0000c2d0 con 0x7fbc00004780 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0000c490 con 0x7fbc00005330 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc167fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:51:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc1e034640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 0x7fbc00003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): get_auth_request con 0x7fbc00003b20 auth_method 0 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00005330 0x7fbc0000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): get_auth_request con 0x7fbc00005330 auth_method 0 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00004780 0x7fbc00004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): get_auth_request con 0x7fbc00004780 auth_method 0 2024-09-15T03:51:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient(hunting): handle_auth_done global_id 254337 payload 293 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 -- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00005330 msgr2=0x7fbc0000b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:51:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00005330 0x7fbc0000b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 -- 172.21.15.153:0/2928316165 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00004780 msgr2=0x7fbc00004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00004780 0x7fbc00004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient: _finish_hunting 0 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 monclient: found mon.a 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 1 -- 172.21.15.153:0/2928316165 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000034e0 con 0x7fbc00003b20 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 10 monclient: _finish_auth 0 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.963+0000 7fbc17fff640 1 -- 172.21.15.153:0/2928316165 reap_dead start 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.965+0000 7fbc177fe640 1 -- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 msgr2=0x7fbc00003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.965+0000 7fbc177fe640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 0x7fbc00003f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc08001b00 tx=0x7fbc08007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90b1c138c96d2b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:51:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0802ac50 con 0x7fbc00003b20 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: dump: 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0802adb0 con 0x7fbc00003b20 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 10 monclient: get_monmap_and_config success 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 4 set_mon_vals no callback set 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 1 -- 172.21.15.153:0/2928316165 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08007340 con 0x7fbc00003b20 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc167fc640 10 monclient: dump: 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 10 monclient: shutdown 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 -- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 msgr2=0x7fbc00003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 0x7fbc00003f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc08001b00 tx=0x7fbc08007c40 comp rx=0 tx=0).stop 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 -- 172.21.15.153:0/2928316165 shutdown_connections 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 --2- 172.21.15.153:0/2928316165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc00003b20 0x7fbc00003f30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 -- 172.21.15.153:0/2928316165 >> 172.21.15.153:0/2928316165 conn(0x7fbc180a0f50 msgr2=0x7fbc180a1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:51:08.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 -- 172.21.15.153:0/2928316165 shutdown_connections 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.966+0000 7fbc1e034640 1 -- 172.21.15.153:0/2928316165 wait complete. 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.967+0000 7fbc1e034640 10 monclient: build_initial_monmap 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.967+0000 7fbc1e034640 10 monclient: monmap: 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.noname-b 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.967+0000 7fbc1e034640 1 Processor -- start 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.967+0000 7fbc1e034640 1 -- start start 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.967+0000 7fbc1e034640 10 monclient: init 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient: _reopen_session rank -1 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 0x7fbc1812f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): picked mon.noname-c con 0x7fbc00003b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1812fa60 0x7fbc181353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): picked mon.noname-b con 0x7fbc1812fa60 addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc181358e0 0x7fbc18135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): picked mon.noname-a con 0x7fbc181358e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc181178d0 con 0x7fbc181358e0 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc181175f0 con 0x7fbc1812fa60 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): start opening mon connection 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18117bb0 con 0x7fbc00003b20 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): _renew_subs 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc1e034640 10 monclient(hunting): authenticate will time out at 2066.608254s 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 0x7fbc1812f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc181358e0 0x7fbc18135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 0x7fbc1812f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:45968/0 (socket says 172.21.15.153:45968) 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 -- 172.21.15.153:0/3513301049 learned_addr learned my addr 172.21.15.153:0/3513301049 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): get_auth_request con 0x7fbc00003b20 auth_method 0 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): get_auth_request con 0x7fbc181358e0 auth_method 0 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc16ffd640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1812fa60 0x7fbc181353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc16ffd640 10 monclient(hunting): get_auth_request con 0x7fbc1812fa60 auth_method 0 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:51:08.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient(hunting): handle_auth_done global_id 244457 payload 824 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 -- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1812fa60 msgr2=0x7fbc181353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1812fa60 0x7fbc181353a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 -- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc181358e0 msgr2=0x7fbc18135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc181358e0 0x7fbc18135d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient: _finish_hunting 0 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 monclient: found mon.noname-c 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc177fe640 1 -- 172.21.15.153:0/3513301049 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc18130490 con 0x7fbc00003b20 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc17fff640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc181358e0 0x7fbc18135d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.968+0000 7fbc16ffd640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1812fa60 0x7fbc181353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:51:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.970+0000 7fbc177fe640 1 -- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 msgr2=0x7fbc1812f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.970+0000 7fbc177fe640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 0x7fbc1812f520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc080077b0 tx=0x7fbc08030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08005d70 con 0x7fbc00003b20 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: dump: 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: _finish_auth 0 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc08038070 con 0x7fbc00003b20 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 5 monclient: authenticate success, global_id 244457 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 10 monclient: _renew_subs 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc18139160 con 0x7fbc00003b20 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0803e6a0 con 0x7fbc00003b20 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 10 monclient: dump: 2024-09-15T03:51:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 7dd05962-040e-4682-bebc-5f3c2ef6922e 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-15T03:27:58.706914+0000 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 10 monclient: _renew_subs 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1813fa50 con 0x7fbc00003b20 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbc15ffb640 4 set_mon_vals no callback set 2024-09-15T03:51:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.971+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) v1 ==== 45010+0+0 (secure 0 0 0) 0x7fbc08030070 con 0x7fbc00003b20 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbbff7fe640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc04038740 0x7fbc0403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbc1e034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbc1e034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbc1e034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(259..259 src has 234..259) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbc0806c0c0 con 0x7fbc00003b20 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc18117fe0 con 0x7fbc00003b20 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbc16ffd640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc04038740 0x7fbc0403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:51:08.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.972+0000 7fbc16ffd640 10 monclient: get_auth_request con 0x7fbc04038740 auth_method 0 2024-09-15T03:51:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.974+0000 7fbc16ffd640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc04038740 0x7fbc0403abf0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c006d30 tx=0x7fbc0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:51:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.976+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fbc08043020 con 0x7fbc00003b20 2024-09-15T03:51:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.976+0000 7fbbff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:51:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:08.976+0000 7fbbff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:51:09.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.130+0000 7fbc1e034640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T03:51:09.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.130+0000 7fbc1e034640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T03:51:09.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.130+0000 7fbc1e034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-15T03:51:09.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.130+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbc1804bf60 con 0x7fbc00003b20 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbc0803c370 con 0x7fbc00003b20 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbbff7fe640 10 monclient: handle_config config(1 keys) v1 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbc15ffb640 4 set_mon_vals no callback set 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbc15ffb640 10 set_mon_vals mon_health_to_clog = false 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbbff7fe640 1 -- 172.21.15.153:0/3513301049 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbc08031970 con 0x7fbc00003b20 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbbff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T03:51:09.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.137+0000 7fbbff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:51:09.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc04038740 msgr2=0x7fbc0403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:09.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc04038740 0x7fbc0403abf0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c006d30 tx=0x7fbc0c008040 comp rx=0 tx=0).stop 2024-09-15T03:51:09.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 10 monclient: shutdown 2024-09-15T03:51:09.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:51:09.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 msgr2=0x7fbc1812f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 0x7fbc1812f520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc080077b0 tx=0x7fbc08030000 comp rx=0 tx=0).stop 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 shutdown_connections 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:6824/31449,v1:172.21.15.181:6825/31449] conn(0x7fbc04038740 0x7fbc0403abf0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc181358e0 0x7fbc18135d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1812fa60 0x7fbc181353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 --2- 172.21.15.153:0/3513301049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc00003b20 0x7fbc1812f520 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 >> 172.21.15.153:0/3513301049 conn(0x7fbc180a0f50 msgr2=0x7fbc18099cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 shutdown_connections 2024-09-15T03:51:09.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-15T03:51:09.139+0000 7fbc1e034640 1 -- 172.21.15.153:0/3513301049 wait complete. 2024-09-15T03:51:09.152 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T03:51:09.152 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T03:51:09.152 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T03:51:09.152 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.185 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T03:51:09.186 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T03:51:09.186 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.265 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T03:51:09.265 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T03:51:09.265 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.335 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T03:51:09.335 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:51:09.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.368 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:51:09.368 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T03:51:09.368 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.437 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T03:51:09.437 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T03:51:09.437 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.506 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T03:51:09.507 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T03:51:09.507 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T03:51:09.507 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.574 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T03:51:09.575 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T03:51:09.575 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:51:09.575 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.588 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:51:09.588 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:51:09.588 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.641 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:51:09.641 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:51:09.641 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:51:09.654 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:51:09.654 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T03:51:09.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-15T03:51:09.719 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:51:09.720 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T03:51:09.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:51:09.902 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T03:51:09.967 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:51:09.967 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T03:51:10.026 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:51:10.026 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T03:51:10.128 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:51:10.128 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T03:51:10.225 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:51:10.225 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T03:51:10.282 INFO:tasks.ceph:Archiving mon data... 2024-09-15T03:51:10.283 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/data/mon.a.tgz 2024-09-15T03:51:10.284 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-15T03:51:10.299 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.ixWNxMXHeo 2024-09-15T03:51:10.300 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ixWNxMXHeo 2024-09-15T03:51:10.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.ixWNxMXHeo 2024-09-15T03:51:10.593 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.ixWNxMXHeo is 883KB 2024-09-15T03:51:10.700 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.ixWNxMXHeo 2024-09-15T03:51:10.716 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/data/mon.c.tgz 2024-09-15T03:51:10.716 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-15T03:51:10.770 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.roMyCWHt8a 2024-09-15T03:51:10.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.roMyCWHt8a 2024-09-15T03:51:10.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.roMyCWHt8a 2024-09-15T03:51:11.045 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.roMyCWHt8a is 883KB 2024-09-15T03:51:11.148 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.roMyCWHt8a 2024-09-15T03:51:11.163 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/data/mon.b.tgz 2024-09-15T03:51:11.164 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-15T03:51:11.183 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.ehWdCGIdKE 2024-09-15T03:51:11.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ehWdCGIdKE 2024-09-15T03:51:11.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.ehWdCGIdKE 2024-09-15T03:51:11.490 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.ehWdCGIdKE is 885KB 2024-09-15T03:51:11.595 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.ehWdCGIdKE 2024-09-15T03:51:11.611 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T03:51:11.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:51:11.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:51:11.675 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T03:51:11.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:51:11.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:51:11.704 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:51:11.743 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:51:11.747 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T03:51:11.749 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/remote/smithi153/crash 2024-09-15T03:51:11.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:51:11.774 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/remote/smithi181/crash 2024-09-15T03:51:11.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:51:11.812 INFO:tasks.ceph:Compressing logs... 2024-09-15T03:51:11.813 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:51:11.818 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:51:11.840 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T03:51:11.840 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38860.log 2024-09-15T03:51:11.840 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36027.log 2024-09-15T03:51:12.101 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35346.log 2024-09-15T03:51:12.101 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38003.log 2024-09-15T03:51:12.101 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36678.log 2024-09-15T03:51:12.101 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40858.log 2024-09-15T03:51:12.101 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35346.log.gz 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38099.log 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38003.log.gz 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38324.log 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36678.log.gz 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40858.log.gz 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36586.log 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38099.log.gz 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38324.log: /var/log/ceph/ceph-client.admin.41388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37308.log 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38324.log.gz 2024-09-15T03:51:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38149.log 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40508.log 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36586.log.gz 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41845.log 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37308.log.gz 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36822.log 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38149.log.gz 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37402.log 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40508.log.gz 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42076.log 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41845.log.gz 2024-09-15T03:51:12.103 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37761.log 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36822.log.gz 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37402.log.gz 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42076.log.gz 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31583.log 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37761.log.gz 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36117.log 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37424.log 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-09-15T03:51:12.104 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35144.log 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31583.log.gz 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36117.log.gz 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39485.log 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37424.log.gz 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38297.log 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35144.log.gz 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39175.log 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36016.log 2024-09-15T03:51:12.105 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39485.log.gz 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38297.log.gz 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42594.log 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39175.log.gz 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35434.log 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36016.log.gz 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bar.42594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42699.log 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.bar.42594.log.gz 2024-09-15T03:51:12.106 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36067.log 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35434.log.gz 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41773.log 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36067.log: gzip -5 --verbose 89.1% -- /var/log/ceph/ceph-client.admin.40724.log -- replaced with /var/log/ceph/ceph-client.admin.42699.log.gz 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38590.log 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36067.log.gz 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41773.log.gz 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37857.log 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2024-09-15T03:51:12.107 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39668.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38590.log.gz 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35022.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42723.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37857.log.gz 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35847.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39668.log.gz 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bar.42723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38348.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35022.log.gz 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42480.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35991.log 2024-09-15T03:51:12.108 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35847.log.gz 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42810.log 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% 93.5% -- replaced with /var/log/ceph/ceph-client.bar.42723.log.gz -- replaced with /var/log/ceph/ceph-client.admin.38348.log.gz 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35775.log 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42480.log.gz 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35943.log 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35991.log.gz 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42810.log.gz 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39459.log 2024-09-15T03:51:12.109 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35775.log.gz 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40484.log 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35943.log.gz 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38446.log 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39224.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39459.log.gz 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38783.log 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40484.log.gz 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40652.log 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36894.log 2024-09-15T03:51:12.110 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38446.log.gz 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38783.log.gz 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42572.log 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40652.log.gz 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35618.log 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36894.log.gz 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bar.42572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36093.log 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35618.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.bar.42572.log.gz /var/log/ceph/ceph-client.admin.38396.log 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:51:12.111 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34885.log 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36093.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35618.log.gz 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35521.log 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34885.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36093.log.gz 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36339.log 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38396.log.gz 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34885.log.gz 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41363.log 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35521.log.gz 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40123.log 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36339.log.gz 2024-09-15T03:51:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38900.log 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40388.log 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41363.log.gz 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40123.log.gz 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37931.log 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38900.log.gz 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36560.log 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40388.log.gz 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38197.log 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-09-15T03:51:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36632.log 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37931.log.gz 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40099.log 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36560.log.gz 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38053.log 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38197.log.gz 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36632.log.gz 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38950.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40099.log.gz 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38053.log.gz 2024-09-15T03:51:12.114 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38125.log 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36966.log 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36364.log 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38125.log.gz 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36942.log 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36966.log.gz 2024-09-15T03:51:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36314.log 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36364.log.gz 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38612.log 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36942.log.gz 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31714.log 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36314.log.gz 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9%/var/log/ceph/ceph-client.admin.31714.log: -- replaced with /var/log/ceph/ceph-client.admin.38612.log.gz 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37522.log 2024-09-15T03:51:12.116 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40796.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31714.log.gz 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31794.log 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37522.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35727.log 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37522.log.gz 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42359.log 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31794.log.gz 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34699.log 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.foo.42359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35545.log 2024-09-15T03:51:12.117 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35727.log.gz 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41026.log 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34699.log.gz 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39695.log 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35545.log.gz 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.foo.42359.log.gz 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41557.log 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42162.log 2024-09-15T03:51:12.118 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39695.log.gz 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42162.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41557.log.gz 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37809.log 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35390.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42162.log.gz 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42221.log 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37809.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gz 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42004.log 2024-09-15T03:51:12.119 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37809.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41749.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42221.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36654.log 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37497.log 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42004.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37497.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42004.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36654.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42337.log 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37497.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41339.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-09-15T03:51:12.120 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39920.log 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36702.log 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.foo.42337.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39719.log 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36702.log: 87.6% -- replaced with /var/log/ceph/ceph-client.foo.42337.log.gz 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41956.log 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39920.log.gz 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36702.log.gz 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41956.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39719.log.gz 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37547.log 2024-09-15T03:51:12.121 INFO:teuthology.orchestra.run.smithi181.stderr: 90.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37907.log 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41581.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41956.log.gz 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42504.log 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39848.log 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37547.log.gz 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bar.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37881.log 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37907.log.gz 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39641.log 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.bar.42504.log.gz 2024-09-15T03:51:12.122 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40292.log 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39848.log.gz 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41267.log 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37881.log.gz 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bazar.42186.log 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39641.log.gz 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40292.log: /var/log/ceph/ceph-client.admin.41267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36436.log 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bazar.42186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40292.log.gz 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr: 83.6% -- replaced with /var/log/ceph/ceph-client.bazar.42186.log.gz 88.0% 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41267.log.gz 2024-09-15T03:51:12.123 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36042.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36436.log.gz 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35497.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39743.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31623.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36042.log.gz 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39743.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35497.log.gz 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35412.log 2024-09-15T03:51:12.124 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39743.log.gz 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37472.log 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31623.log.gz 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35412.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39000.log 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37472.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35412.log.gz 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37472.log.gz 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37356.log 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39000.log.gz 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39126.log 2024-09-15T03:51:12.125 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38688.log 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34863.log 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37356.log.gz 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41908.log 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39126.log.gz 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38688.log.gz 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41908.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34863.log.gz 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36846.log 2024-09-15T03:51:12.126 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41908.log.gz 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31667.log 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37979.log 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39539.log 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36846.log.gz 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39101.log 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31667.log.gz 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36488.log 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39539.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37979.log.gz 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35001.log 2024-09-15T03:51:12.127 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36488.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39539.log.gz 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39200.log 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39101.log.gz 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.35001.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42834.log 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36488.log.gz 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36512.log 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35001.log.gz 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42269.log 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39200.log.gz 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38223.log 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42834.log.gz 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.foo.42269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 2024-09-15T03:51:12.128 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36512.log.gz 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38223.log: gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.foo.42269.log.gz -- 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.40748.log 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38223.log.gz 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37260.log 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gz 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41797.log 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41195.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37833.log 2024-09-15T03:51:12.129 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37260.log.gz 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35703.log 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40532.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41797.log.gz 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35703.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37833.log.gz 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37140.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35703.log.gz 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37188.log 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41533.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34772.log 2024-09-15T03:51:12.130 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35823.log 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37188.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37188.log.gz 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40316.log 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34772.log.gz 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42785.log 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35823.log.gz 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42528.log 2024-09-15T03:51:12.131 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40316.log.gz 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42785.log.gz 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39248.log 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35455.log: gzip -5 --verbose/var/log/ceph/ceph-client.bar.42528.log: -- /var/log/ceph/ceph-client.admin.38808.log 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39248.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39273.log 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.bar.42528.log.gz 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39896.log 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39248.log.gz 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34977.log 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37595.log 2024-09-15T03:51:12.132 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39273.log.gz 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36870.log 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39896.log.gz 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41412.log 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34977.log.gz 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34909.log 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37595.log.gz 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36870.log.gz 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41243.log 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41412.log.gz 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2024-09-15T03:51:12.133 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34909.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37785.log 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35895.log 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41701.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41243.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37166.log 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36412.log 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37785.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37234.log 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35895.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37166.log.gz 2024-09-15T03:51:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35871.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35118.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36412.log.gz 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37234.log.gz 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36918.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35118.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35871.log.gz 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42760.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36918.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35118.log.gz 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35070.log 2024-09-15T03:51:12.135 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42760.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36918.log.gz 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41051.log 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38542.log 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42760.log.gz 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36388.log 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35070.log.gz 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36192.log 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41051.log.gz 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36388.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38542.log.gz 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34931.log 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35967.log 2024-09-15T03:51:12.136 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36388.log.gz 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41821.log 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36192.log.gz 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38664.log 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34931.log.gz 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41821.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35967.log.gz 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37044.log 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37737.log 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41821.log.gz 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41605.log 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38664.log.gz 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39297.log 2024-09-15T03:51:12.137 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37044.log.gz 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37737.log.gz 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37689.log 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39297.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41605.log.gz 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39025.log 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37689.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41123.log 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37689.log.gz 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39025.log: /var/log/ceph/ceph-client.admin.41123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36143.log 2024-09-15T03:51:12.138 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36289.log 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41123.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39025.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36143.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36289.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36798.log 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-09-15T03:51:12.139 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36798.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37448.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36798.log.gz 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz -5 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41099.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38758.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37448.log.gz 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41099.log.gz 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37643.log 2024-09-15T03:51:12.140 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38758.log.gz 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40676.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz -5 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39321.log 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37643.log: /var/log/ceph/ceph-client.admin.40820.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37643.log.gz 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38878.log 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40195.log 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39321.log.gz 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42245.log 2024-09-15T03:51:12.141 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40195.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38878.log.gz 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40340.log 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.foo.42245.log: /var/log/ceph/ceph-client.admin.41219.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40195.log.gz 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34723.log 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.foo.42245.log.gz 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37092.log 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34723.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40340.log.gz 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37571.log 2024-09-15T03:51:12.142 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34723.log.gz 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37116.log 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37571.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37092.log.gz 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34841.log 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37571.log.gz 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38518.log 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.34841.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37116.log.gz 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35368.log 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38856.log 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34841.log.gz 2024-09-15T03:51:12.143 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42100.log 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35368.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38518.log.gz 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39614.log 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35368.log.gz 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42100.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38856.log.gz 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36216.log 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42100.log.gz 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42469.log 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39614.log.gz 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36240.log 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-09-15T03:51:12.144 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42052.log 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36216.log.gz 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.foo.42469.log: /var/log/ceph/ceph-client.admin.36240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37713.log 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr: 81.5% -- replaced with /var/log/ceph/ceph-client.foo.42469.log.gz 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42052.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36240.log.gz 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36992.log 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42052.log.gz 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35799.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37713.log.gz 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31754.log 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36992.log: /var/log/ceph/ceph-client.admin.38832.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2024-09-15T03:51:12.145 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40700.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36536.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40700.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36992.log.gz 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31754.log.gz 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42616.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36536.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40700.log.gz 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39512.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bar.42616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35174.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36536.log.gz 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35919.log 2024-09-15T03:51:12.146 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39512.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35174.log: /var/log/ceph/ceph-client.admin.41147.log: 93.2% -- replaced with /var/log/ceph/ceph-client.bar.42616.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35174.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38924.log 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38924.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35919.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38924.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40556.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-09-15T03:51:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38974.log 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40906.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42550.log 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38974.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34797.log 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.bar.42550.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38974.log.gz 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42315.log 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.bar.42550.log.gz 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40075.log 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41437.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34797.log.gz 2024-09-15T03:51:12.148 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.foo.42315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35316.log 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:gzip 89.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35230.log 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.foo.42315.log.gz 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40075.log: /var/log/ceph/ceph-client.admin.35316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41315.log 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40436.log 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35316.log.gz 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41315.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40075.log.gz 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35230.log.gz 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41315.log.gz 2024-09-15T03:51:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36608.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34675.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.34675.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37284.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36608.log.gz 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39872.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34675.log.gz 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37284.log: /var/log/ceph/ceph-client.admin.39872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39590.log 2024-09-15T03:51:12.150 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37380.log 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37284.log.gz 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39872.log.gz 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39590.log.gz 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38370.log 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37380.log.gz 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38027.log 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36462.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37665.log 2024-09-15T03:51:12.151 INFO:teuthology.orchestra.run.smithi181.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38027.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38370.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37955.log 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37665.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38027.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34748.log 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34955.log 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37955.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37665.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34748.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37955.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34748.log.gz 2024-09-15T03:51:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37619.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34955.log.gz 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40147.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37018.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38710.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37619.log.gz 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40628.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37018.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40147.log.gz 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2024-09-15T03:51:12.153 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37018.log.gz 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37070.log 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40628.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38710.log.gz 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38271.log 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40628.log.gz 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37070.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42293.log 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39794.log 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37070.log.gz 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38271.log: /var/log/ceph/ceph-client.foo.42293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41075.log 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39794.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T03:51:12.154 INFO:teuthology.orchestra.run.smithi181.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.foo.42293.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41075.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38271.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39794.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41075.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36750.log 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.40171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38247.log 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36167.log 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36750.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36750.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36167.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38247.log.gz 2024-09-15T03:51:12.155 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37210.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36167.log.gz 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35046.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35570.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37210.log.gz 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35046.log: /var/log/ceph/ceph-client.admin.35570.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35046.log.gz 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35570.log.gz 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37334.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40243.log 2024-09-15T03:51:12.156 INFO:teuthology.orchestra.run.smithi181.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42736.log 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37334.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40243.log: /var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42736.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40243.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42736.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36726.log 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41002.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2024-09-15T03:51:12.157 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36726.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-09-15T03:51:12.158 INFO:teuthology.orchestra.run.smithi181.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36726.log.gz 2024-09-15T03:51:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36069.log 2024-09-15T03:51:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38860.log: /var/log/ceph/ceph-client.admin.36027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35361.log 2024-09-15T03:51:12.225 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38860.log.gz 2024-09-15T03:51:12.225 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37398.log 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36027.log.gz 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38003.log 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36069.log.gz 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39322.log 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35361.log.gz 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51383.log 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37398.log.gz 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37440.log 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38003.log.gz 2024-09-15T03:51:12.226 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39322.log.gz 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42539.log 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51383.log.gz 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45988.log 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37440.log.gz 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42539.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39196.log 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34865.log 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42539.log.gz 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44861.log 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34865.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.45988.log.gz 2024-09-15T03:51:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50400.log 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39196.log.gz 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44861.log: gzip 88.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz -- 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.44609.log 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.44861.log.gz 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50921.log 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50400.log.gz 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43835.log 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44609.log.gz 2024-09-15T03:51:12.228 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50921.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46553.log 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49711.log 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50921.log.gz 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48728.log 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49209.log 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.43835.log.gz 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46553.log.gz 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44819.log 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49711.log.gz 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52552.log 2024-09-15T03:51:12.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49209.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48728.log.gz 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49209.log.gz 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53942.log 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44819.log.gz 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45029.log 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52552.log.gz 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53942.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49837.log 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46719.log 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53942.log.gz 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51173.log 2024-09-15T03:51:12.230 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45029.log.gz 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49837.log.gz 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52889.log 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38944.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.46719.log.gz 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38902.log 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51173.log.gz 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37692.log 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43877.log 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52889.log.gz 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51904.log 2024-09-15T03:51:12.231 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38902.log.gz 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37692.log.gz/var/log/ceph/ceph-client.admin.43877.log: 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48434.log 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35119.log 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43877.log.gz 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40142.log 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51904.log.gz 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph-client.admin.35119.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45509.log 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48434.log.gz 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47240.log 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35119.log.gz 2024-09-15T03:51:12.232 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40142.log.gz 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45425.log 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45509.log.gz 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41430.log 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47240.log.gz 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36927.log 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53312.log 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36927.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.45425.log.gz 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45719.log 2024-09-15T03:51:12.233 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41430.log.gz 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53312.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.41514.log 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36927.log.gz 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38339.log 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53312.log.gz 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52030.log 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45719.log.gz 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37146.log 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52030.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47660.log 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38339.log.gz 2024-09-15T03:51:12.234 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37230.log 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52030.log.gz 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51215.log 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37146.log.gz 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38297.log 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47660.log.gz 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37230.log.gz 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52283.log 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51215.log.gz 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39974.log 2024-09-15T03:51:12.235 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38297.log.gz 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53144.log 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39974.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52283.log.gz 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35691.log 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53144.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46761.log 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39974.log.gz 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35691.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53144.log.gz 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38733.log 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46761.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35691.log.gz 2024-09-15T03:51:12.236 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42625.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46761.log.gz 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38733.log.gz 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40581.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46677.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40581.log: /var/log/ceph/ceph-client.admin.39154.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42625.log.gz 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49375.log 2024-09-15T03:51:12.237 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40581.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31921.log 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46677.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36591.log 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50274.log 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49375.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35114.log 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31921.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40749.log 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36591.log.gz 2024-09-15T03:51:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50274.log.gz 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51425.log 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35114.log.gz 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35775.log 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41052.log: /var/log/ceph/ceph-client.admin.51425.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40749.log.gz 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39238.log 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48182.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.51425.log.gz 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35775.log.gz 2024-09-15T03:51:12.239 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35943.log 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39238.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51341.log 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39238.log.gz 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52510.log 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35943.log: /var/log/ceph/ceph-client.admin.51341.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49106.log 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35943.log.gz 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52510.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.51341.log.gz 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37524.log 2024-09-15T03:51:12.240 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43919.log 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37524.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52510.log.gz 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37482.log 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49106.log.gz 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44357.log 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37524.log.gz 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44315.log 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43919.log.gz 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37482.log.gz 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37314.log 2024-09-15T03:51:12.241 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44357.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53606.log 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.44315.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51694.log 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51651.log 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37314.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50836.log 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53606.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53102.log 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51694.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50836.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51651.log.gz 2024-09-15T03:51:12.242 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40016.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49753.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50836.log.gz 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53102.log: /var/log/ceph/ceph-client.admin.40016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52425.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41682.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1% 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40016.log.gz -- replaced with /var/log/ceph/ceph-client.admin.53102.log.gz 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50652.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49753.log.gz 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41682.log: /var/log/ceph/ceph-client.admin.45383.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.52425.log.gz 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47492.log 2024-09-15T03:51:12.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36279.log 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45383.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41682.log.gz 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.50652.log.gz 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45761.log 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45383.log.gz 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.36279.log: -- /var/log/ceph/ceph-client.admin.40665.log 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47492.log.gz 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41766.log 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36279.log.gz 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45946.log 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45761.log.gz 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46030.log 2024-09-15T03:51:12.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41766.log: /var/log/ceph/ceph-client.admin.45946.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40665.log.gz 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47198.log 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46030.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41766.log.gz 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43330.log 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45946.log.gz 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34683.log 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43330.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46030.log.gz 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53774.log 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34683.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43330.log.gz 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 91.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.47198.log.gz 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.35541.log 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2024-09-15T03:51:12.245 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34683.log.gz 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53984.log 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53774.log.gz 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40331.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35541.log.gz 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53732.log 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47618.log 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53984.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36885.log 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47618.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53984.log.gz 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53732.log.gz 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49459.log 2024-09-15T03:51:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36885.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47618.log.gz 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52931.log 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36885.log.gz 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49459.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz -5 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49417.log 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52846.log 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49417.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49459.log.gz 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43203.log 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52931.log.gz 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52468.log 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49417.log.gz 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36464.log 2024-09-15T03:51:12.247 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52468.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.52846.log.gz 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41724.log 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43203.log.gz 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41934.log 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52468.log.gz 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41724.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36464.log.gz 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48896.log 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49501.log 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41934.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41724.log.gz 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42919.log 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49501.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41934.log.gz 2024-09-15T03:51:12.248 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45173.log 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.48896.log.gz 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36507.log 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44735.log 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42919.log.gz 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53858.log 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45173.log.gz 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53438.log 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39364.log 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.36507.log.gz 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44735.log.gz 2024-09-15T03:51:12.249 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44693.log 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53858.log.gz 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38045.log 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53438.log.gz 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51257.log 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44693.log: /var/log/ceph/ceph-client.admin.38045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.39364.log.gz 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51257.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44693.log.gz 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50021.log 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38045.log.gz 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54108.log 2024-09-15T03:51:12.250 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51257.log.gz 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34867.log 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.54108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51131.log 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50021.log.gz 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47366.log 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54108.log.gz 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34867.log.gz 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47366.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51131.log.gz 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2024-09-15T03:51:12.251 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35985.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47366.log.gz 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53018.log 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36237.log 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48014.log 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53018.log.gz 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40917.log 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40539.log 2024-09-15T03:51:12.252 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39678.log 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36237.log.gz 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48014.log.gz 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40539.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40917.log.gz 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41262.log 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39678.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40539.log.gz 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46492.log 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48518.log 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41262.log: /var/log/ceph/ceph-client.admin.46492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52678.log 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48518.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.41262.log.gz 2024-09-15T03:51:12.253 INFO:teuthology.orchestra.run.smithi153.stderr:gzip 89.8% -- replaced with /var/log/ceph/ceph-client.admin.46492.log.gz 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35607.log 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52678.log.gz 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38817.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.48518.log.gz 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47970.log 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42835.log 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35607.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44567.log 2024-09-15T03:51:12.254 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35607.log.gz 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42835.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47970.log.gz 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52072.log 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43541.log 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42835.log.gz 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52594.log 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37097.log 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52072.log.gz 91.7% -- replaced with /var/log/ceph/ceph-client.admin.44567.log.gz 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52594.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.43541.log.gz 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49290.log 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47450.log 2024-09-15T03:51:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37097.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52594.log.gz/var/log/ceph/ceph-client.admin.49290.log: 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42961.log 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37097.log.gz 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47450.log: gzip -5 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49290.log.gz 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41220.log 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41178.log 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47450.log.gz 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42961.log.gz 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41178.log: gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41220.log.gz 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.48476.log 2024-09-15T03:51:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41472.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41178.log.gz 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44441.log 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48476.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49669.log 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48476.log.gz 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35859.log 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35271.log: gzip -5 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44441.log.gz 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38087.log 89.5% 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49669.log.gz 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35859.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz --verbose 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.49543.log 2024-09-15T03:51:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38087.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35859.log.gz 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51005.log 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44987.log 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38087.log.gz 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48854.log 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.51005.log.gz 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51946.log 2024-09-15T03:51:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48854.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44987.log.gz 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46156.log 88.9% 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51946.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.48854.log.gz 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36111.log 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47282.log 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46156.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51946.log.gz 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37356.log 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.46156.log.gz 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34864.log 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36111.log.gz 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53648.log 2024-09-15T03:51:12.259 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47282.log.gz 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35649.log 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34863.log 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37356.log.gz 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34864.log.gz 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53648.log.gz 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46324.log 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34863.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35649.log.gz 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34863.log.gz 2024-09-15T03:51:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46324.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52762.log 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46324.log.gz 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph-client.admin.40959.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53480.log 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42018.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.52762.log.gz 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49585.log 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-09-15T03:51:12.261 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53396.log 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49585.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53480.log.gz 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35481.log 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49585.log.gz 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47097.log 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40833.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53396.log.gz 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53228.log 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47097.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35481.log.gz 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50568.log 2024-09-15T03:51:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47013.log 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50568.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47097.log.gz 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53228.log.gz 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47013.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50568.log.gz 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36843.log 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41976.log: /var/log/ceph/ceph-client.admin.36843.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47013.log.gz 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz/var/log/ceph/ceph-client.admin.42412.log: 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53564.log 2024-09-15T03:51:12.263 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36843.log.gz 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44483.log 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53564.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45071.log 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44483.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53564.log.gz 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32004.log 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48266.log 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44483.log.gz 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51862.log 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45071.log.gz 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48266.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32004.log.gz 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36321.log 2024-09-15T03:51:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37734.log 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51862.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48266.log.gz 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50358.log 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51862.log.gz 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37734.log: 89.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.36321.log.gz -5 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43793.log 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42270.log 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43793.log: gzip -5 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37734.log.gz 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42455.log 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50358.log.gz 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42270.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43793.log.gz 2024-09-15T03:51:12.265 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43709.log 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53522.log 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42455.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42270.log.gz 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42582.log 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53522.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43709.log.gz 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37188.log 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42582.log: gzip 92.1% -- replaced with /var/log/ceph/ceph-client.admin.42455.log.gz 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48980.log 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53522.log.gz 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42793.log 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42582.log.gz 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48980.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37188.log.gz 2024-09-15T03:51:12.266 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51736.log 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46635.log 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48980.log.gz 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36153.log 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42793.log.gz 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39804.log 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51736.log.gz 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43981.log 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39804.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36153.log.gz 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.46635.log.gz 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52804.log 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43981.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39804.log.gz 2024-09-15T03:51:12.267 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54068.log 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44273.log 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52804.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43981.log.gz 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.54068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44273.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.52804.log.gz 88.7% 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54068.log.gz 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52636.log 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph-client.admin.39070.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51820.log 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7%/var/log/ceph/ceph-client.admin.52636.log: -- replaced with /var/log/ceph/ceph-client.admin.44273.log.gz 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose 88.9% -- /var/log/ceph/ceph-client.admin.53816.log -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.268 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52636.log.gz 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46803.log 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42667.log 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53816.log: /var/log/ceph/ceph-client.admin.46803.log: gzip 90.1% -- replaced with /var/log/ceph/ceph-client.admin.51820.log.gz 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51467.log 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2%/var/log/ceph/ceph-client.admin.42667.log: -- replaced with /var/log/ceph/ceph-client.admin.53816.log.gz 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46803.log.gz 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48938.log 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41808.log 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51467.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42667.log.gz 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50526.log 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41808.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51467.log.gz 2024-09-15T03:51:12.269 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48938.log.gz 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46366.log 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41808.log.gz 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44651.log 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50526.log.gz 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36549.log 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44651.log: gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.46366.log.gz --verbose 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.50442.log 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48770.log 2024-09-15T03:51:12.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36549.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44651.log.gz 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47889.log 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36549.log.gz 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48770.log: gzip 89.9% -- replaced with /var/log/ceph/ceph-client.admin.50442.log.gz 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38381.log 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45215.log 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48770.log.gz 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51299.log 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47889.log.gz 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45215.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38381.log.gz 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46845.log 2024-09-15T03:51:12.271 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36717.log 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51299.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.45215.log.gz 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35116.log 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51299.log.gz 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36717.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46845.log.gz 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36195.log 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35116.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36717.log.gz 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41640.log 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35116.log.gz 2024-09-15T03:51:12.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.a.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36195.log.gz 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46929.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48308.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41640.log: /var/log/ceph/ceph-client.admin.46929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50232.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48308.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41640.log.gz 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46929.log.gz 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43003.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48308.log.gz 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50232.log: /var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37608.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47744.log 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50232.log.gz 2024-09-15T03:51:12.273 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37608.log: 87.9% /var/log/ceph/ceph-client.admin.38986.log 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43003.log.gz 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34641.log 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37608.log.gz 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37960.log 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47744.log.gz 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35118.log 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38986.log.gz 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34641.log: /var/log/ceph/ceph-client.admin.37960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46887.log 2024-09-15T03:51:12.274 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37960.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52344.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34641.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35118.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45467.log 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46887.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45467.log: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49940.log 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46887.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34725.log 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45467.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49940.log: --verbose -- /var/log/ceph/ceph-client.admin.37818.log 2024-09-15T03:51:12.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45299.log 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49940.log.gz 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37818.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34725.log.gz 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48686.log 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45299.log: gzip -5 --verbose 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37818.log.gz 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.47324.log 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48686.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45299.log.gz 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42751.log 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35901.log 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48686.log.gz 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42751.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47324.log.gz 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T03:51:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph-client.admin.35901.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44399.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42751.log.gz 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35901.log.gz 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.42228.log 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44063.log 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42228.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44399.log.gz 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38549.log 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44063.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42228.log.gz 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36759.log 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6%/var/log/ceph/ceph-client.admin.38549.log: -- replaced with /var/log/ceph/ceph-client.admin.44063.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50064.log 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38549.log.gz 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36759.log: /var/log/ceph/ceph-client.admin.50064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36675.log 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47576.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.50064.log.gz 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36759.log.gz 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47786.log 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36675.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36675.log.gz 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45904.log 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48812.log 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45822.log 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47786.log.gz 2024-09-15T03:51:12.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48812.log: gzip -5 --verbose 90.9% -- /var/log/ceph/ceph-client.admin.41850.log 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45904.log.gz 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45822.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27303.log 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48812.log.gz 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41850.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45822.log.gz 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36969.log 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.27303.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27303.log.gz 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48140.log 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50694.log 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35733.log 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36969.log.gz 2024-09-15T03:51:12.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35117.log 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50694.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48140.log.gz 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36801.log 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35733.log.gz 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50694.log.gz 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35331.log 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35117.log.gz 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53900.log 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35331.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36801.log.gz 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51509.log 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34860.log 2024-09-15T03:51:12.280 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35331.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53900.log: /var/log/ceph/ceph-client.admin.51509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34860.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53900.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51509.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45593.log 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34860.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36383.log 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40707.log: /var/log/ceph/ceph-client.admin.45593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45635.log 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36383.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45593.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40058.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36383.log.gz 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40058.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45635.log.gz 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42877.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40058.log.gz 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42877.log: /var/log/ceph/ceph-client.admin.42497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43414.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40455.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42877.log.gz 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52156.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz/var/log/ceph/ceph-client.admin.43414.log: 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49022.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43751.log 2024-09-15T03:51:12.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52156.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.43414.log.gz 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37776.log 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43751.log: 88.1% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52156.log.gz 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49022.log.gz 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49148.log 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43751.log.gz 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49795.log 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49148.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37776.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46971.log 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49795.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49148.log.gz 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35501.log 2024-09-15T03:51:12.283 INFO:teuthology.orchestra.run.smithi153.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.49795.log.gz 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46971.log.gz 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.46408.log 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52974.log 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46408.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38507.log 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52974.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.46408.log.gz 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- 91.4% -- replaced with /var/log/ceph/ceph-client.admin.52974.log.gz 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.54026.log 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50879.log 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38507.log: /var/log/ceph/ceph-client.admin.54026.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38507.log.gz 2024-09-15T03:51:12.284 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54026.log.gz 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37011.log 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37011.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.50879.log.gz 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37011.log.gz 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48098.log 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45341.log 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48098.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49627.log 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48098.log.gz 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49627.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45341.log.gz 2024-09-15T03:51:12.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43625.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49627.log.gz 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44903.log 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42144.log 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44903.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44903.log.gz 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40791.log 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40791.log: /var/log/ceph/ceph-client.admin.42144.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42144.log.gz 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40791.log.gz 2024-09-15T03:51:12.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37566.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44231.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40268.log: /var/log/ceph/ceph-client.admin.37566.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37566.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52720.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44777.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52720.log: /var/log/ceph/ceph-client.admin.44231.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44231.log.gz 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52720.log.gz 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49879.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42186.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44777.log: /var/log/ceph/ceph-client.admin.49879.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44777.log.gz 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48350.log 2024-09-15T03:51:12.287 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49879.log.gz 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38465.log 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42186.log: /var/log/ceph/ceph-client.admin.48350.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48350.log.gz 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42186.log.gz 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50148.log 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2%/var/log/ceph/ceph-client.admin.50148.log: -- replaced with /var/log/ceph/ceph-client.admin.38465.log.gz 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39448.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.50148.log.gz 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44189.log 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53186.log 2024-09-15T03:51:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44189.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43372.log 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44189.log.gz 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36633.log 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43372.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53186.log.gz 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52114.log 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43372.log.gz 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40100.log 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36633.log: /var/log/ceph/ceph-client.admin.52114.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52114.log.gz 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36633.log.gz 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42709.log 2024-09-15T03:51:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39112.log 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42709.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40100.log.gz 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37650.log 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39112.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42709.log.gz 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53060.log 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39112.log.gz 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37650.log: /var/log/ceph/ceph-client.admin.53060.log: 88.8% 90.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37650.log.gz 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53060.log.gz 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47055.log 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48560.log 2024-09-15T03:51:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47055.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47828.log 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47055.log.gz 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48560.log: /var/log/ceph/ceph-client.admin.42060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39512.log 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47828.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48560.log.gz 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35291.log 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47828.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52199.log 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39512.log: /var/log/ceph/ceph-client.admin.35291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50106.log 2024-09-15T03:51:12.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52199.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39512.log.gz 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35291.log.gz 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52199.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.51047.log 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50106.log: /var/log/ceph/ceph-client.admin.40413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38171.log 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51047.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50106.log.gz 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46198.log 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51047.log.gz 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52241.log 2024-09-15T03:51:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44525.log 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46198.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38171.log.gz 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.52241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50610.log 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46198.log.gz 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52241.log.gz 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50610.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44525.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41556.log 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49333.log 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34550.log: /var/log/ceph/ceph-client.admin.41556.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.50610.log.gz 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48602.log 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9%/var/log/ceph/ceph-client.admin.49333.log: -- replaced with /var/log/ceph/ceph-client.admin.41556.log.gz 2024-09-15T03:51:12.293 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50755.log 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48602.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.49333.log.gz 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45257.log 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.48602.log.gz 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45257.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50755.log.gz 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46282.log 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46240.log 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1%/var/log/ceph/ceph-client.admin.40226.log: -- replaced with /var/log/ceph/ceph-client.admin.45257.log.gz 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2024-09-15T03:51:12.294 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46240.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46282.log.gz 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48644.log 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46240.log.gz/var/log/ceph/ceph-client.admin.39280.log: 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37055.log 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2%/var/log/ceph/ceph-client.admin.48644.log: -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34866.log 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40184.log 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48644.log.gz 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37055.log: /var/log/ceph/ceph-client.admin.34866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35115.log 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40184.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37055.log.gz 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34866.log.gz 2024-09-15T03:51:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53354.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51988.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40184.log.gz 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53354.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35115.log.gz 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43142.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47408.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53354.log.gz 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43142.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51988.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43057.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50190.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43142.log.gz 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43057.log: /var/log/ceph/ceph-client.admin.47408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43099.log 2024-09-15T03:51:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50190.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47408.log.gz 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr:gzip 91.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.43057.log.gz -- 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.47534.log 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50190.log.gz/var/log/ceph/ceph-client.admin.43099.log: 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43099.log.gz 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50963.log 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47534.log.gz 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41304.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44105.log 2024-09-15T03:51:12.297 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43498.log 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44105.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.50963.log.gz 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51778.log 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48392.log 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51778.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44105.log.gz 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43498.log.gz 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45551.log 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51778.log.gz 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50484.log 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48392.log: /var/log/ceph/ceph-client.admin.45551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41892.log 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50484.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45551.log.gz 2024-09-15T03:51:12.298 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.48392.log.gzgzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50484.log.gz 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44945.log 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44147.log 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44945.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41892.log.gz 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31698.log 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44945.log.gz 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.37272.log 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31698.log: /var/log/ceph/ceph-client.admin.44147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40623.log 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37272.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.44147.log.gz 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31698.log.gz 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35391.log 2024-09-15T03:51:12.299 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37272.log.gz 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37879.log 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40623.log: /var/log/ceph/ceph-client.admin.35391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48056.log 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37879.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35391.log.gz 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40623.log.gz 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37879.log.gz 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38213.log 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42331.log 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53690.log 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.48056.log.gz 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-09-15T03:51:12.300 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38213.log.gz 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53690.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42331.log.gz 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38423.log 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53690.log.gz 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46114.log 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53270.log 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46114.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46450.log 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.53270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45677.log 2024-09-15T03:51:12.301 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3%/var/log/ceph/ceph-client.admin.46450.log: -- replaced with /var/log/ceph/ceph-client.admin.46114.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53270.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43667.log 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46450.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46072.log 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45677.log: /var/log/ceph/ceph-client.admin.43667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34525.log 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46072.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45677.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46072.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35817.log 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51089.log 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35817.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34525.log.gz 2024-09-15T03:51:12.302 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50316.log 2024-09-15T03:51:12.303 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-09-15T03:51:12.303 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.51089.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35817.log.gz 2024-09-15T03:51:12.303 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50316.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51089.log.gz 2024-09-15T03:51:12.303 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39888.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50316.log.gz 2024-09-15T03:51:12.303 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-09-15T03:51:12.303 INFO:teuthology.orchestra.run.smithi181.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T03:51:12.503 INFO:teuthology.orchestra.run.smithi153.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T03:51:12.533 INFO:teuthology.orchestra.run.smithi181.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T03:51:12.866 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T03:51:12.878 INFO:teuthology.orchestra.run.smithi181.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T03:51:12.901 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T03:51:12.916 INFO:teuthology.orchestra.run.smithi181.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T03:51:12.960 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T03:51:12.969 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T03:51:12.970 INFO:teuthology.orchestra.run.smithi181.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T03:51:12.972 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T03:51:12.972 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m1.097s 2024-09-15T03:51:12.973 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m3.879s 2024-09-15T03:51:12.973 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.342s 2024-09-15T03:51:12.973 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-15T03:51:12.973 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m1.141s 2024-09-15T03:51:12.973 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m4.862s 2024-09-15T03:51:12.973 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.471s 2024-09-15T03:51:12.973 INFO:tasks.ceph:Archiving logs... 2024-09-15T03:51:12.973 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/remote/smithi153/log 2024-09-15T03:51:12.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:51:14.177 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/remote/smithi181/log 2024-09-15T03:51:14.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:51:15.113 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:51:15.123 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:51:15.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:51:15.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:51:15.211 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:51:15.268 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:51:15.268 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:51:15.271 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:51:15.286 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:51:15.288 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:51:15.291 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:51:15.291 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-15T03:51:15.291 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +192us[ +195us] +/- 58ms 2024-09-15T03:51:15.291 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -89us[ -87us] +/- 73ms 2024-09-15T03:51:15.291 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -194us[ -194us] +/- 84ms 2024-09-15T03:51:15.291 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:51:15.293 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:51:15.293 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-15T03:51:15.293 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +189us[ +188us] +/- 58ms 2024-09-15T03:51:15.293 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 31 -80us[ -81us] +/- 73ms 2024-09-15T03:51:15.294 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -201us[ -202us] +/- 84ms 2024-09-15T03:51:15.294 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:51:15.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:51:15.343 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:51:15.343 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:51:15.397 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:51:15.475 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:51:15.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:51:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:51:15.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:51:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:51:15.604 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:51:15.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:51:15.633 INFO:teuthology.task.internal:Duration was 1751.591923 seconds 2024-09-15T03:51:15.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:51:15.645 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:51:15.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:51:15.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:51:15.685 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:51:15.688 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:51:16.035 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:51:16.036 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-09-15T03:51:16.036 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:51:16.056 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T03:51:16.057 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:51:16.102 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:51:16.102 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:51:16.106 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:51:16.170 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:51:16.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:51:16.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:51:16.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:51:16.326 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:51:16.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:51:16.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:51:16.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:51:16.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:51:16.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:51:16.417 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-09-15T03:51:16.441 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-15T03:51:16.458 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:51:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:51:16.512 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:51:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:51:16.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:51:16.538 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:51:16.540 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/remote/smithi153 2024-09-15T03:51:16.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:51:16.589 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905270/remote/smithi181 2024-09-15T03:51:16.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:51:16.623 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:51:16.623 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:51:16.625 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:51:16.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:51:16.685 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:51:16.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:51:16.704 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:51:16.705 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:51:16.707 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:51:16.719 INFO:teuthology.orchestra.run.smithi153.stdout: 265329 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:51 /home/ubuntu/cephtest 2024-09-15T03:51:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:51 /home/ubuntu/cephtest 2024-09-15T03:51:16.730 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:51:16.740 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:51:16.805 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} duration: 1751.5919225215912 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:51:16.806 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:51:16.885 INFO:teuthology.run:pass